Storage container reconciliation overview
Container reconciliation identifies corrupted or diverged container replicas and repairs them to ensure consistency and identical data across all replicas.
By design, container replicas are identical and contain the same data. Container reconciliation is a repair mechanism that allows administrators to detect and repair container replicas that are corrupted or have diverged from each other. The data within each replica is summarized as a single data checksum. If the replica data diverges, these checksums will differ. After reconciliation,, the replicas will contain the same data with matching checksums. This checksum remains constant even as data is deleted from the container in the background.
he system summarizes the data within each replica as a single data checksum. If the replica data diverges, these checksums will differ. After reconciliation, the replicas are restored to an identical state with matching checksums. This checksum remains constant even as data is deleted from the container in the background.
Restrictions and limitations
- Reconciliation is not supported for containers that are in the
OPEN,DELETING, orDELETEDstates. Reconciling containers in these states fails immediately without any modifications. Reconciliation can only be used to repair data inUNHEALTHYreplicas by updating the replica data and corresponding checksums, but not their state. - Reconciliation is supported for Ratis-replicated containers only. Reconciling an Erasure Coded (EC) container fails immediately without any modifications.
- Containers with a replication factor of 1 cannot be reconciled because no peer replicas exist to compare against. Similarly, containers with a replication factor of 3 that have zero or one replica available cannot be reconciled.
- After running the
ozone admin container reconcilecommand on aCLOSEDRATIS/THREEcontainer, checksums might match, but the replica state does not change, even if the container wasUNHEALTHY. If the container is corrupted again later and the checksums diverge once more, you can run theozone admin container reconcilecommand multiple times until the container is repaired. - Storage Container Manager (SCM) can only use replication to automatically
recover from failures. SCM will not automatically schedule reconciliation. Since replication
requires at least one source replica not in the
UNHEALTHYstate, reconciliation will need to be run manually if all replicas areUNHEALTHY.
