Reconciling storage containers

Understand the process of storage container reconciliation to detect and repair corrupted or inconsistent container replicas, ensuring data consistency and uniformity across all replicas.

Detecting containers that need reconciliation

You can detect the container replicas that are corrupted or have otherwise diverged from each other by using one of the following methods:

Using Recon web UI:
1. Sign in to Cloudera Manager.
2. Click Clusters in the left navigation.
3. Select the Ozone service.
4. Click the Web UI dropdown menu and click Recon Web UI. The Recon UI page appears.
5. Click the Containers in the left navigation.
6. Click Mismatched Replicas tab. The Mismatched Replicas tab displays all containers where replicas have different data checksums. Containers appearing in this tab are eligible for reconciliation.
Using the command line:
1. Open the Terminal on your device.
2. Run the following command to see the reconciliation status of specific containers:
```
ozone admin container reconcile --status <CONTAINER_ID> [<CONTAINER_ID> ...]
```
  This command displays detailed information about each container and its replicas, including:
  - Container state
  - Replication configuration
  - Data checksum for each replica
  - Whether the data checksums of all the container's replicas match
  - DataNode details such as ID, hostname, and IP address for each replica

Triggering container reconciliation

To trigger reconciliation for one or more containers, do the following steps:

Open the Terminal on your device.
Run the following command to trigger reconciliation for one or more containers:
```
ozone admin container reconcile <***CONTAINER-ID***> [<***CONTAINER-ID***> ...]
```

When the reconciliation is triggered:

The Storage Container Manager sends a reconciliation command to all the DataNodes hosting replicas of the container.
Each DataNode compares its container data checksums with those of peer replicas.
DataNodes automatically repairs the missing or the corrupt data by doing the following:
- Copy missing blocks from peer replicas
- Retrieve missing chunks within blocks
- Repair chunks with incorrect checksums
Scans the container's data in the background to verify the integrity of the repaired data.

Monitoring reconciliation progress

After triggering reconciliation, you can monitor the progress by using one of the following methods:

Using Recon web UI:
1. Sign in to Cloudera Manager.
2. Click Clusters in the left navigation.
3. Select the Ozone service.
4. Click the Web UI dropdown menu and click Recon Web UI. The Recon UI appears.
5. Click Containers in the left navigation.
6. Click the Mismatched Replicas tab.
7. Verify that the container no longer appears in the Mismatched Replicas tab.
Using the command line:
1. Open the Terminal on your device.
2. Run the following command to see the reconciliation status of specific containers:
```
ozone admin container reconcile --status <***CONTAINER-ID***> [<***CONTAINER-ID***> ...]
```
  This command displays detailed information about each container and its replicas, including:
  - Container state
  - Replication configuration
  - Data checksum for each replica
  - Whether the data checksums of all the container's replicas match
  - DataNode details such as ID, hostname, and IP address for each replica
Reviewing DataNode logs: Check the DataNode logs for the reconciliation progress messages.