Reconciling storage containers

Reconcile storage containers by detecting, repairing and, monitoring corrupted or inconsistent container replicas to ensure data consistency and uniformity across all replicas.

Detecting corrupted or diverged containers

  • Using Recon web UI:
    1. Sign in to Cloudera Manager.
    2. Click Clusters in the left navigation.
    3. Select the Ozone service.
    4. In the Web UI drop-down menu, select Recon Web UI. The Recon UI page is displayed.
    5. Click Containers in the left navigation.
    6. Click the Mismatched Replicas tab. The Mismatched Replicas tab displays all containers whose replicas have different data checksums. Containers displayed in this tab are eligible for reconciliation.
  • Using the command line:
    1. Open the Terminal on your device.
    2. Run the following command to see the reconciliation status of specific containers:
      ozone admin container reconcile --status <CONTAINER_ID> [<CONTAINER_ID> ...]
      This command displays the following information about each container and its replicas:
      • Container state
      • Replication configuration
      • Data checksum for each replica
      • Whether the data checksums of all the container's replicas match
      • DataNode details such as ID, hostname, and IP address for each replica

Triggering container reconciliation

  1. Open the Terminal on your device.
  2. Run the following command to trigger reconciliation for one or more containers:
    ozone admin container reconcile <***CONTAINER-ID***> [<***CONTAINER-ID***> ...]
    
Result: When you trigger reconciliation, the system performs following actions:
  1. The Storage Container Manager sends a reconciliation command to all the DataNodes hosting replicas of the container.
  2. Each DataNode compares its container data checksums with those of the peer replicas.
  3. The DataNodes automatically repair the missing or the corrupt data by performing the following actions:
    • Copying missing blocks from peer replicas
    • Retrieving missing chunks within blocks
    • Repairing chunks with incorrect checksums
  4. The Data Nodes scan the container data in the background to verify the integrity of the repaired data.

Monitoring reconciliation progress

  • Using Recon web UI:
    1. Sign in to Cloudera Manager.
    2. Click Clusters in the left navigation.
    3. Select the Ozone service.
    4. In the Web UI drop-down menu, select Recon Web UI. The Recon UI is displayed.
    5. Click Containers in the left navigation.
    6. Click the Mismatched Replicas tab.
    7. Verify that the container is no longer displayed in the Mismatched Replicas tab.
  • Using the command line:
    1. Open the Terminal on your device.
    2. Run the following command to see the reconciliation status of specific containers:
      ozone admin container reconcile --status <***CONTAINER-ID***> [<***CONTAINER-ID***> ...]
      This command displays the following information about each container and its replicas:
      • Container state
      • Replication configuration
      • Data checksum for each replica
      • Whether the data checksums of all the container replicas match
      • DataNode details such as ID, hostname, and IP address for each replica
  • Review DataNode logs for the reconciliation progress messages.