Verify rack awareness
You must perform a series of checks to verify if rack awareness is activated on the cluster.
-
Check the NameNode logs located in
/var/log/hadoop/hdfs/
for the addition of nodes.014-01-13 15:58:08,495 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /rack01/<ipaddress>
-
Run the hdfs fsck command to ensure that there are no
inconsistencies.
For a cluster with two racks, the fsck command returns a status similar to the following:
Status: HEALTHY Total size: 123456789 B Total dirs: 0 Total files: 1 Total blocks (validated): 1 (avg. block size 123456789 B) Minimally replicated blocks: 1 (100.0 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 0 (0.0 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 3 Average block replication: 3.0 Corrupt blocks: 0 Missing replicas: 0 (0.0 %) Number of data-nodes: 40 Number of racks: 2 FSCK ended at Mon Jan 13 17:10:51 UTC 2014 in 1 milliseconds
-
Run the dfsadmin -report command for a report that includes the
rack name next to each DataNode.
The dfsadmin -report command returns a report similar to the following excerpted example:
Configured Capacity: 19010409390080 (17.29 TB) Present Capacity: 18228294160384 (16.58 TB) DFS Remaining: 5514620928000 (5.02 TB) DFS Used: 12713673232384 (11.56 TB) DFS Used%: 69.75% Under replicated blocks: 181 Blocks with corrupt replicas: 0 Missing blocks: 0 ------------------------------------------------- Datanodes available: 5 (5 total, 0 dead) Name: 192.168.90.231:50010 (h2d1.hdp.local) Hostname: h2d1.hdp.local Rack: /default/rack_02 Decommission Status : Normal Configured Capacity: 15696052224 (14.62 GB) DFS Used: 314380288 (299.82 MB) Non DFS Used: 3238612992 (3.02 GB) DFS Remaining: 12143058944 (11.31 GB) DFS Used%: 2.00% DFS Remaining%: 77.36% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Last contact: Thu Jun 12 11:39:51 EDT 2014