Configuring Monitoring Settings

Minimum Required Role: Configurator (also provided by Cluster Administrator, Full Administrator)

There are several types of monitoring settings you can configure in Cloudera Manager:

  • For a service or role for which monitoring is provided, you can enable and disable selected health tests and events, configure how those health tests factor into the overall health of the service, and modify thresholds for the status of certain health tests. Cloudera Manager supports this type of monitoring configuration for HDFS, MapReduce, YARN, HBase, Impala, ZooKeeper, and Flume. For hosts you can disable or enable selected health tests, modify thresholds, and enable or disable health alerts.
  • For hosts, you can set threshold-based monitoring of free space in the various directories on the hosts Cloudera Manager monitors.
  • For MapReduce, YARN, and Impala services, you can configure aspects of how Cloudera Manager monitors activities, applications, and queries.
  • For the Cloudera Management Service you can configure monitoring settings for the monitoring roles—enable and disable health tests on the monitoring processes as well as configuring some general settings related to events and alerts (specifically with the Event Server and Alert Publisher). Each of the Cloudera Management Service roles has its own parameters that can be modified in order to specify how much data is retained by that service. For some monitoring functions, the amount of retained data can grow very large, so it may become necessary to adjust the limits.

For general information about modifying configuration settings, see Modifying Configuration Properties.

This section covers the following topics:

Configuring Health Monitoring

The initial health monitoring configuration is handled during the installation and configuration of your cluster, and most monitoring parameters have default settings. However, you can set or modify these at any time.

Depending on the service or role you select, and the configuration category, you can enable or disable health tests, determine when health tests cause alerts, or determine whether specific health tests are used in computing the overall health of a role or service. In most cases you can disable these "roll-up" health tests separately from the individual health tests.

As a rule, a health tests whose result is considered "Concerning" or "Bad" will be forwarded as an event to the Event Server. That includes health tests whose results are based on configured Warning or Critical thresholds, as well pass-fail type health tests. An event will also be published when the health test result returns to normal.

You can control when an individual health test will be forwarded as an event or an alert by modifying the threshold values for the relevant health test.

Configuring Service Monitoring

  1. Select Clusters > Service.
  2. Click the Configuration tab.
  3. Expand the Service-Wide category.
  4. Click the Monitoring category.
  5. Select the category of properties to change (these are organized as service-wide and by role group).
  6. Configure the property.
  7. Click Save Changes to commit the changes.

Configuring Host Monitoring

  1. Click the Hosts tab.
  2. Select a host.
  3. Click the Configuration tab.
  4. Expand the Service-Wide category.
  5. Click the Monitoring category.
  6. Configure the property.
  7. Click Save Changes to commit the changes. You can add a note that will be included with the change in the Configuration History.

Configuring Directory Monitoring

Cloudera Manager can perform threshold-based monitoring of free space in the various directories on the hosts it monitors — such as log directories or checkpoint directories (for the Secondary NameNode).

These thresholds can be set in one of two ways — as absolute thresholds (in terms of MiB and GiB, and so on) or as percentages of space. As with other threshold properties, you can set values that will trigger events at both the Warning and Critical levels.

If you set both thresholds, the Absolute Threshold setting will be used.

Configuring Activity Monitoring

The Activity Monitor monitors the MapReduce MRv1 jobs running on your cluster. This also includes the higher-level activities, such as Pig, Hive, and Oozie workflows that run as MapReduce tasks.

You can monitor for slow-running jobs or jobs that fail, and alert on these events. To detect jobs that are running too slowly, you must configure a set of activity duration rules that specify what jobs to monitor, and what the limits on duration are for those jobs. A "slow activity" event occurs when a job exceeds the duration limit configured for it in an activity duration rule. Activity duration rules are not defined by default; you must configure these rules if you want to see events for jobs that exceed the duration defined by these rules.

To configure Activity Monitor settings:

  1. Click the Clusters tab.
  2. Select the MapReduce service instance.
  3. Click the Configuration tab.
  4. Expand the Service-Wide category.
  5. Click the Monitoring category.
  6. Specify one or more activity duration rules.
  7. Click Save Changes to commit the changes.

Activity Duration Rules

An activity duration rule is a regular expression (used to match an activity name (that is, a Job ID)) combined with a run time limit which the job should not exceed. You can add as many rules as you like, one per line, in the Activity Duration Rules property.

The format of each rule is regex=number where the regex is a regular expression to match against the activity name, and number is the job duration limit, in minutes. When a new activity starts, each regex expression is tested against the name of the activity for a match.

The list of rules is tested in order, and the first match found is used. For example, if the rule set is:

foo=10
bar=20
any activity named "foo" would be marked slow if it ran for more than 10 minutes. Any activity named "bar" would be marked slow if it ran for more than 20 minutes.

Since Java regular expressions can be used, if the rule set is:

foo.*=10
bar=20
any activity with a name that starts with "foo" (for example, fool, food, foot) will match the first rule.

If there is no match for an activity, then that activity will not be monitored for job duration. However, you can add a "catch-all" as the last rule which will always match any name:

foo.*=10
bar=20
baz=30
.*=60

In this case, any job that runs longer than 60 minutes will be marked slow and will generate an event.

Configuring YARN Application Monitoring

You can configure the visibility of the YARN application monitoring results.

Configuring Application Visibility

To configure whether admin and non-admin users can view all applications, only that user's applications, or no applications:

  1. Click the Clusters tab.
  2. Select the YARN service instance.
  3. Click the Configuration tab.
  4. Expand the Service-Wide category.
  5. Click the Monitoring category.
  6. Set the Visibility Settings properties for admin and non-admin users.
  7. Click Save Changes to commit the changes. You can add a note that will be included with the change in the Configuration History.

Configuring Impala Query Monitoring

You can configure the visibility of the Impala query results and the size of the storage allocated to Impala query results.

Configuring Query Visibility

To configure whether admin and non-admin users can view all queries, only that user's queries, or no queries:

  1. Click the Clusters tab.
  2. Select the Impala service instance.
  3. Click the Configuration tab.
  4. Expand the Service-Wide category.
  5. Click the Monitoring category.
  6. Set the Visibility Settings properties for admin and non-admin users.
  7. Click Save Changes to commit the changes.

Configuring Impala Query Data Store Maximum Size

The query store stores enough information to make the query searchable through the filter language.
  1. Click the Clusters tab.
  2. Do one of the following:
    • Select Clusters > Cloudera Management Service > Cloudera Management Service.
    • On the Status tab of the Home page, in Cloudera Management Service table, click the Cloudera Management Service link.
  3. Click the Configuration tab.
  4. Click the Service Monitor Default Group category.
  5. In the Impala Storage section, set the firehose_impala_storage_bytes property. The default is 1 GiB.
  6. Click Save Changes to commit the changes.
  7. Restart the Service Monitor.

The firehose_impala_storage_bytes property determines the approximate amount of disk space dedicated to storing Impala query data. Once the store reaches its maximum size, older data is deleted to make room for newer queries. The disk usage is approximate because data deletion begins only when the limit has been reached.

Configuring Alerts

The following topics describe how to configure when alerts are raised and how they are delivered:

Enabling Activity Monitor Alerts

You can enable alerts when an activity runs too slowly or fails.

  1. Click the Clusters tab.
  2. Select the MapReduce service instance.
  3. Click the Configuration tab.
  4. Expand the Service-Wide category.
  5. Click the Monitoring category.
  6. Check the Alert on Slow Activities or Alert on Activity Failure checkboxes.
  7. Click Save Changes to commit the changes. You can add a note that will be included with the change in the Configuration History.

Enabling Configuration Change Alerts

Configuration change alerts can be set service wide, or on specific roles for the service.
  1. Click a service, role, or host.
  2. Click the Configuration tab.
  3. Check the Enable Configuration Change Alerts checkbox.
  4. Click Save Changes to commit the changes. You can add a note that will be included with the change in the Configuration History.

Enabling HBase Alerts

  1. Click the Clusters tab.
  2. Select the HBase service instance.
  3. Click the Configuration tab.
  4. Expand the Service-Wide category.
  5. Click the Monitoring category.
  6. Set one of the region or Hbck alerts:
    • Hbck Region Error Count
    • Hbck Error Count
    • Hbck Alert Error Codes
    • Hbck Slow Run
    • Region Health Canary Slow Run
    • Canary Unhealthy Region Count
    • Canary Unhealthy Region Percentage
  7. Click Save Changes to commit the changes.

Configuring Health Alerts

Enabling Health Alerts

You can enable alerts when the health of a service or role crosses a threshold.

  1. Select Clusters > Service.
  2. Click the Configuration tab.
  3. Expand the Service-Wide category.
  4. Click the Monitoring category.
  5. Check the Enable Health Alerts for this Role checkbox.
  6. Click Save Changes to commit the changes.

Modifying the Health Threshold

You can configure the threshold when a health alert is raised.

  1. Select Administration > Alerts.
  2. Click to the right of Health Alert Threshold.
  3. In the category Event Server Default Group, click the Health Alert Threshold value, and choose the Bad or Concerning radio button.
  4. Click Save Changes to commit the changes. You can add a note that will be included with the change in the Configuration History.

Configuring Alerts Transitioning Out of Alerting Health Threshold

You can configure an alert when a service or role instance transitions from an alerting to a non-alerting health threshold.

  1. Select Administration > Alerts.
  2. Click to the right of Alert on Transitions out of Alerting Health.
  3. In the category Event Server Default Group, check the Alert on Transitions out of Alerting Health checkbox.
  4. Click Save Changes to commit the changes. You can add a note that will be included with the change in the Configuration History.

Configuring Log Alerts

You can configure an alert when a daemon emits a log message that matches a specified regular expression. See Configuring Log Alerts.

Configuring Alert Delivery

You can configure alerts to be delivered by email or sent as SNMP traps. If you choose email delivery, you can add to or modify the list of alert recipient email addresses. You can also send a test alert e-mail. See Managing Alerts.

Configuring Log Events

You can enable or disable the forwarding of selected log events to the Event Server. This is enabled by default, and is a service-wide setting (Enable Log Event Capture) for each service for which monitoring is provided. You can enable and disable event capture for CDH services or for the Cloudera Management Service.

Configuring Logs

  1. Go to a service.
  2. Click the Configuration tab.
  3. Select Role Default Group > Logs.
  4. Edit a log property.
  5. Click Save Changes to commit the changes.
  6. Restart the role.

Configuring Log Directories

  1. Do one of the following:
      1. On the Home page, click a cluster name.
      2. Select Configuration > All Log Directories.
      3. Edit a Role Log Directory property.
      1. Go to a service.
      2. Click the Configuration tab.
      3. Select RoleName Default Group > Logs.
      4. Edit the Log Directory property.
  2. Click Save Changes to commit the changes.
  3. Restart the role.

Enabling and Disabling Log Event Capture

  1. Select Clusters > Service.
  2. Click the Configuration tab.
  3. Expand the Service-Wide category.
  4. Click the Monitoring category.
  5. Modify the Enable Log Event Capture setting.
  6. Click Save Changes to commit the changes. You can add a note that will be included with the change in the Configuration History.

You can also modify the rules that determine how log messages are turned into events. Editing these rules is not recommended.

For each role, there are rules that govern how its log messages are turned into events by the custom log4j appender for the role. These are defined in the Rules to Extract Events from Log Files property for each HDFS, MapReduce and HBase role, and for ZooKeeper, Flume agent, and monitoring roles as well.

Configuring Which Log Messages Become Events

  1. Select Clusters > Service.
  2. Expand the Service-Wide category.
  3. Click the Monitoring category.
  4. Select the role group for the Role for which you want to configure log events, or search for "Rules to Extract Events from Log Files". Note that for some roles there may be more than one role group, and you may need to modify all of them. The easiest way to ensure that you have found all occurrences of the property you need to modify is to search for the property by name. Cloudera Manager will show all copies of the property that match the search filter.
  5. Edit these rules as needed.
  6. Click Save Changes to commit the changes.

Cloudera defines a number of rules by default. For example:

  • The line {"rate": 10, "threshold":"FATAL"}, means log entries with severity FATAL should be forwarded as events, up to 10 a minute.
  • The line {"rate": 0, "exceptiontype": "java.io.EOFException"}, means log entries with the exception java.io.EOFException should always be forwarded as an event.

The syntax for these rules is defined in the Description field for this property: the syntax lets you create rules that identify log messages based on log4j severity, message content matching, or the exception type. These rules must result in valid JSON.

Configuring Log Alerts

You specify that a log event should generate an alert (by setting "alert":true in the rule). If you specify a content match, the entire content must match — if you want to match on a partial string, you must provide wildcards as appropriate to allow matching the entire string.