You can use the output of hdfs fsck or hdfs dfsadmin
-report commands for information about inconsistencies with the HDFS data blocks
such as missing, misreplicated, or underreplicated blocks. You can adopt different methods to
address these inconsistencies.
Missing blocks: Ensure that the DataNodes in your cluster and the disks running on
them are healthy. This should help in recovering those blocks that have recoverable replicas
(indicated as missing blocks). If a file contains corrupt or missing blocks that cannot be
recovered, then the file would be missing data, and all this data starting from the missing
block becomes inaccessible through the CLI tools and FileSystem API. In most cases, the only
solution is to delete the data file (by using the hdfs fsck <path>
-delete command) and recover the data from another source.
Underreplicated blocks: HDFS automatically attempts to fix this issue by replicating
the underreplicated blocks to other DataNodes and match the replication factor. If the
automatic replication does not work, you can run the HDFS Balancer to address the issue.
Misreplicated blocks: Run the hdfs fsck -replicate command to
trigger the replication of misreplicated blocks. This ensures that the blocks are correctly
replicated across racks in the cluster.
