DSSD DataNode Health Tests

This DSSD DataNode health test checks that the number of file descriptors used does not rise above some percentage of the DSSD DataNode file descriptor limit. A failure of this health test may indicate a bug in either Hadoop or Cloudera Manager. Contact Cloudera support. This test can be configured using the File Descriptor Monitoring Thresholds DSSD DataNode monitoring setting.

Short Name: File Descriptors

This DSSD DataNode health test checks that the filesystem containing the heap dump directory of this DSSD DataNode has sufficient free space. This test can be configured using the Heap Dump Directory Free Space Monitoring Absolute Thresholds and Heap Dump Directory Free Space Monitoring Percentage Thresholds DSSD DataNode monitoring settings.

Short Name: Heap Dump Directory Free Space

This DSSD DataNode health test factors in the health of the host upon which the DSSD DataNode is running. A failure of this test means that the host running the DSSD DataNode is experiencing some problem. See that host's status page for more details.This test can be enabled or disabled using the DSSD DataNode Host Health Test DSSD DataNode monitoring setting.

Short Name: Host Health

This is a DataNode health test that checks that all the running NameNodes in the HDFS service consider the DataNode alive. A failure of this health test may indicate that the DataNode is having trouble communicating with some or all NameNodes in the service. Look in the DataNode logs for more details. This test can be enabled or disabled using the DataNode Connectivity Health Test DataNode monitoring setting. The DataNode Connectivity Tolerance at Startup DataNode monitoring setting and the Health Check Startup Tolerance NameNode monitoring setting can be used to control the test's tolerance windows around DataNode and NameNode restarts respectively.

Short Name: NameNode Connectivity

This DSSD DataNode health test checks that the DSSD DataNode threads are not experiencing long scheduling pauses. The test uses a pause monitoring thread in the DSSD DataNode that tracks scheduling delay by noting if it is run on its requested schedule. If the thread is not run on its requested schedule, the delay is noted and considered pause time. The health test checks that no more than some percentage of recent time is spent paused. A failure of this health test may indicate that the DSSD DataNode is not getting enough CPU resources, or that it is spending too much time doing garbage collection. Inspect the DSSD DataNode logs for any pause monitor output and check garbage collection metrics exposed by the DSSD DataNode. This test can be configured using the Pause Duration Thresholds and Pause Duration Monitoring Period DSSD DataNode monitoring settings.

Short Name: Pause Duration

This DSSD DataNode health test checks that the Cloudera Manager Agent on the DSSD DataNode host is heart beating correctly and that the process associated with the DSSD DataNode role is in the state expected by Cloudera Manager. A failure of this health test may indicate a problem with the DSSD DataNode process, a lack of connectivity to the Cloudera Manager Agent on the DSSD DataNode host, or a problem with the Cloudera Manager Agent. This test can fail either because the DSSD DataNode has crashed or because the DSSD DataNode will not start or stop in a timely fashion. Check the DSSD DataNode logs for more details. If the test fails because of problems communicating with the Cloudera Manager Agent on the DSSD DataNode host, check the status of the Cloudera Manager Agent by running /etc/init.d/cloudera-scm-agent status on the DSSD DataNode host, or look in the Cloudera Manager Agent logs on the DSSD DataNode host for more details. This test can be enabled or disabled using the DSSD DataNode Process Health Test DSSD DataNode monitoring setting.

Short Name: Process Status

This DSSD DataNode health test checks the amount of swap memory in use by the role. A failure of this health test may indicate that your machine is overloaded. This test can be configured using the Process Swap Memory Thresholds monitoring settings.

Short Name: Swap Memory Usage

This is a DataNode health test that checks for whether the DataNode transceivers usage is nearing the configured transceivers limit. A transceiver is a thread used to transfer data into or out of the DataNode. If a DataNode hits its transceivers limit, HDFS reads and writes begin to fail. This test can be configured using the DataNode Transceivers Usage Thresholds DataNode monitoring setting. The Maximum Number of Transfer Threads DataNode setting controls the DataNode transceivers limit.

Short Name: Transceiver Usage

This DSSD DataNode health test checks that the DSSD DataNode has not recently exited unexpectedly. The test returns "Bad" health if the number of unexpected exits exceeds a critical threshold. For example, if this test is configured with a critical threshold of 1, this test returns "Good" health if there have been no unexpected exits recently. If 1 or more unexpected exits occured recently, this test returns "Bad" health. The test also indicates whether any of the exits were caused by an OutOfMemory error if the Cloudera Manager Kill When Out of Memory monitoring setting is enabled. This test can be configured using the Unexpected Exits Thresholds and Unexpected Exits Monitoring Period DSSD DataNode monitoring settings.

Short Name: Unexpected Exits

This health test checks that the role's web server is responding quickly to requests by the Cloudera Manager Agent, and that the Cloudera Manager Agent can collect metrics from the web server. Failure of this health test may indicate a problem with the web server of the DSSD DataNode, a misconfiguration of the DSSD DataNode, or a problem with the Cloudera Manager Agent. Consult the Cloudera Manager Agent logs and the logs of the DSSD DataNode for more detail. If the test failure message indicates a communication problem, the Cloudera Manager Agent's HTTP requests to the DSSD DataNode's web server are failing or timing out. If the test's failure message indicates an unexpected response, the DSSD DataNode's web server responded to the Cloudera Manager Agent's request, but the response could not be interpreted for some reason. This test can be configured using the Web Metric Collection DSSD DataNode monitoring setting.

Short Name: Web Server Status