Storage container reconciliation overview

Container reconciliation identifies corrupted or diverged container replicas and repairs them to ensure consistency and identical data across all replicas.

By design, container replicas are identical and contain the same data. Container reconciliation is a repair mechanism that allows administrators to detect and repair container replicas that are corrupted or have diverged from each other. The data within each replica is summarized as a single data checksum. If the replica data diverges, these checksums will differ. After reconciliation,, the replicas will contain the same data with matching checksums. This checksum remains constant even as data is deleted from the container in the background.

he system summarizes the data within each replica as a single data checksum. If the replica data diverges, these checksums will differ. After reconciliation, the replicas are restored to an identical state with matching checksums. This checksum remains constant even as data is deleted from the container in the background.

Restrictions and limitations

Consider the following constraints and behavioral limitations before you use container reconciliation:
  • Reconciliation is not supported for containers that are in the OPEN, DELETING, or DELETED states. Reconciling containers in these states fails immediately without any modifications. Reconciliation can only be used to repair data in UNHEALTHY replicas by updating the replica data and corresponding checksums, but not their state.
  • Reconciliation is supported for Ratis-replicated containers only. Reconciling an Erasure Coded (EC) container fails immediately without any modifications.
  • Containers with a replication factor of 1 cannot be reconciled because no peer replicas exist to compare against. Similarly, containers with a replication factor of 3 that have zero or one replica available cannot be reconciled.
  • After running the ozone admin container reconcile command on a CLOSED RATIS/THREE container, checksums might match, but the replica state does not change, even if the container was UNHEALTHY. If the container is corrupted again later and the checksums diverge once more, you can run the ozone admin container reconcile command multiple times until the container is repaired.
  • Storage Container Manager (SCM) can only use replication to automatically recover from failures. SCM will not automatically schedule reconciliation. Since replication requires at least one source replica not in the UNHEALTHY state, reconciliation will need to be run manually if all replicas are UNHEALTHY.