Host Health Tests

This is a host health test that checks that the filesystem containing the Agent's log directory has sufficient free space. This test can be configured using the Agent Log Directory Free Space Monitoring Absolute Thresholds and Agent Log Directory Free Space Monitoring Percentage Thresholds host monitoring settings.

Short Name: Agent Log Directory

This is a host health test that checks whether the filesystem containing the Agent's parcel directory has sufficient free space. This test can be configured using the Agent Parcel Directory Free Space Monitoring Absolute Thresholds and Agent Parcel Directory Free Space Monitoring Percentage Thresholds host monitoring settings.

Short Name: Agent Parcel Directory

This is a host health test that checks that the filesystem containing the Agent's process directory has sufficient free space. The process directory contains the configuration files for the processes which the Agent starts. This test can be configured using the Agent Process Directory Free Space Monitoring Absolute Thresholds and Agent Process Directory Free Space Monitoring Percentage Thresholds host monitoring settings.

Short Name: Agent Process Directory

This is a host health test that checks that the host's Agent is heartbeating correctly to the Server, is in contact with the Host Monitor, and has the correct software version. A failure of this health test may indicate a lack of connectivity between the host's Agent and the Server, a lack of connectivity betwen the host's Agent and the Host Monitor, or that the Agent or Host Monitor software is out of date. Check the status of the Agent by running /etc/init.d/cloudera-scm-agent status on the host, or look in the host's Agent logs for more details. If this test reports a software version mismatch between the Agent and the Host Monitor, check the version of each component by consulting the appropriate logs or the appropriate status web pages. This test can be enabled or disabled using the Host Process Health Test host configuration setting.

Short Name: Agent Status

This health test checks the expiry time of the TLS certificate for the Agent on the given host. This test can be configured using the Agent TLS Certificate Expiry Thresholds host monitoring setting. This test alerts when the certificate is close to expiry.

Short Name: Certificate Expiration

This is a host health test that checks whether the host's system clock appears to be out-of-sync with its NTP server(s). The test uses either the 'ntpq -np' (if ntpd is running) or 'chronyc sources' (if chronyd is running) command to check that whether the host is synchronized to an NTP peer and that the absolute value of the host's clock offset from that peer is not too large. The test returns "Bad" health If the command fails to find a synchronized NTP peer or an offset between the peer and the host time is too large. To troubleshoot, run 'ntpq -np' or 'chronyc sources' on the host. The output should contain a row for each of the host's NTP servers. The row starting with a '*' (if ntpdc) or '^*' (if chronyc) contains the peer to which the host is currently synchronized. If no row starts with a '*' or '^*', this indicates that the host is not currently synchronized. Make sure you have a pool of valid ntp servers in your ntp.conf (if ntpdc) or chrony.conf (if chronyc). Make sure that UDP port 123 is open on any firewall that is in use. Check the system log for ntpd or chronyd messages related to configuration errors. If the issue persists or the offset between the peer and the host time is too large, please seek assistance from your system administrator. If NTP is not in use on the host, disable this check by setting both 'Warning' and 'Critical' thresholds to 'NEVER' using the configuration options shown below for the host. recommends using NTP for time synchronization of Hadoop clusters. This test can be configured using the Host Clock Offset Thresholds host configuration setting.

Short Name: Clock Offset

This is a host health test that checks that the host's hostname and canonical name are consistent when checked from a Java process, and that the DNS resolution completes in a timely manner. The DNS resolution duration is calculated by measuring the time taken to call getLocalHost in a Java process on this host. Note that DNS information may be cached on the host, and this caching can affect the reported resolution duration. A failure of this health test may indicate that the host's DNS configuration is not correct or the host's DNS server is responding slowly. Check the Agent log for the names that were detected by this test or for errors running the Java process. The hostname and canonical name are considered to be consistent if the hostname or the hostname plus a domain name is the same as the canonical name. This health test uses domain names from the domain and search lines in /etc/resolv.conf. This health test does not consult /etc/nsswitch.conf and may give incorrect results if /etc/resolv.conf is not used by the host. There may be a delay of up to 5 minutes before this health test picks up changes to /etc/resolv.conf. This test can be configured using the Hostname and Canonical Name Health Check host configuration setting.

Short Name: DNS Resolution

This is a health test that checks that the host in a cluster has sufficient entropy. A failure of this health may indicate that entropy pool is depleted, which might cause applications waiting for the randomness to pause until more entropy is available. Random connection timeout issues usually happen when the available entropy count reaches zero. Increase the entropy using rng-tools for the host. This test can be configured using the Host Entropy Thresholds host configuration settings.

Short Name: Entropy

This is a host health test that checks for network frame errors across all network interfaces. A failure of this health test may indicate a problem with network hardware (e.g. switches) and can potentially cause other service or role-level performance problems. Check the host and network hardware logs for more details. This test can be configured using the Host Network Frame Error Percentage Thresholds, Host Network Frame Error Check Window, Host Network Frame Error Test Minimum Required Packets host configuration settings.

Short Name: Frame Errors

This is a host health test that checks for network interfaces that appear to be operating at less than full speed. The Agent uses ethtool ioctl to determine the network interface speed and duplex mode. A failure of this health test may indicate that network interface(s) may be configured incorrectly and may be causing performance problems. Use the ethtool command to check and configure the host's network interfaces to use the fastest available link speed and duplex mode. For inactive interfaces or for interfaces that do not support ethtool ioctl, the Agent cannot collect speed metrics. If the Agent fails to collect speed metrics for all network interfaces on the host, the test is disabled. This test can be configured using the Host's Network Interfaces Slow Link Modes Thresholds, Network Interface Expected Link Speed and Network Interface Expected Duplex Mode host configuration settings.

Short Name: Network Interface Speed

This is a health test that checks that the host has not swapped out more than a certain number of pages over the last fifteen minutes. A failure of this health test may indicate misconfiguration of the host operating system, or too many processes running on the host. Try reducing vm.swappiness, or add more memory to the host. This test can be configured using the Host Memory Swapping Thresholds, Host Memory Swapping Check Window host configuration settings.

Short Name: Swapping