Reconciling storage containers
Reconcile storage containers by detecting, repairing and, monitoring corrupted or inconsistent container replicas to ensure data consistency and uniformity across all replicas.
Detecting corrupted or diverged containers
- Using Recon web UI:
- Sign in to Cloudera Manager.
- Click Clusters in the left navigation.
- Select the Ozone service.
- In the Web UI drop-down menu, select Recon Web UI. The Recon UI page is displayed.
- Click Containers in the left navigation.
- Click the Mismatched Replicas tab. The Mismatched Replicas tab displays all containers whose replicas have different data checksums. Containers displayed in this tab are eligible for reconciliation.
- Using the command line:
- Open the Terminal on your device.
- Run the following command to see the reconciliation status of specific
containers:
ozone admin container reconcile --status <CONTAINER_ID> [<CONTAINER_ID> ...]This command displays the following information about each container and its replicas:- Container state
- Replication configuration
- Data checksum for each replica
- Whether the data checksums of all the container's replicas match
- DataNode details such as ID, hostname, and IP address for each replica
Triggering container reconciliation
- Open the Terminal on your device.
- Run the following command to trigger reconciliation for one or more
containers:
ozone admin container reconcile <***CONTAINER-ID***> [<***CONTAINER-ID***> ...]
Result:
When
you
trigger
reconciliation,
the system performs following actions:
- The Storage Container Manager sends a reconciliation command to all the DataNodes hosting replicas of the container.
- Each DataNode compares its container data checksums with those of the peer replicas.
- The
DataNodes automatically
repair the
missing or the corrupt data by
performing
the following
actions:
- Copying missing blocks from peer replicas
- Retrieving missing chunks within blocks
- Repairing chunks with incorrect checksums
- The Data Nodes scan the container data in the background to verify the integrity of the repaired data.
Monitoring reconciliation progress
- Using Recon web UI:
- Sign in to Cloudera Manager.
- Click Clusters in the left navigation.
- Select the Ozone service.
- In the Web UI drop-down menu, select Recon Web UI. The Recon UI is displayed.
- Click Containers in the left navigation.
- Click the Mismatched Replicas tab.
- Verify that the container is no longer displayed in the Mismatched Replicas tab.
- Using the command line:
- Open the Terminal on your device.
- Run the following command to see the reconciliation status of specific
containers:
ozone admin container reconcile --status <***CONTAINER-ID***> [<***CONTAINER-ID***> ...]This command displays the following information about each container and its replicas:- Container state
- Replication configuration
- Data checksum for each replica
- Whether the data checksums of all the container replicas match
- DataNode details such as ID, hostname, and IP address for each replica
- Review DataNode logs for the reconciliation progress messages.
