Diagnostic Data Collection

Cloudera Manager log files provide important information for Cloudera Support to use in analyzing problems or unexpected behavior with Cloudera Manager deployments or CDH clusters. Cloudera Director triggers the collection of diagnostic data for the deployments and clusters it manages. This helps avoid situations where a failed cluster has been terminated but Cloudera Support has no diagnostic data or log files to help them identify the cause of the failure. For users with a Cloudera Enterprise or Cloudera Enterprise Trial license, diagnostic data is collected and sent to Cloudera Support automatically upon cluster bootstrap or update failure. By default, diagnostic data is also downloaded to the Cloudera Director instance.

If Cloudera Manager is unable to collect diagnostic data, no information will be sent to Cloudera Support and the Cloudera Manager service logs will be downloaded to Cloudera Director instead of the diagnostic data. The logs contain less information than the diagnostic data, but can still be useful to Cloudera Support for analyzing deployment and cluster behavior.

In addition to the automatic collection of diagnostic data, you can initiate diagnostic data collection manually through the Cloudera Director web UI or API. You can trigger manual collection of diagnostic data for an entire Cloudera Manager deployment or for a specific CDH cluster, as described below.

For more information on how diagnostic data collection works on the Cloudera Manager side, see the Cloudera Manager documentation page Sending Usage and Diagnostic Data to Cloudera.

Manual Collection of Diagnostic Data

You can manually trigger the collection of diagnostic data using either the Cloudera Director web UI or the Cloudera Director API.

Using the Web UI

To trigger diagnostic data collection for a Cloudera Manager deployment or cluster, go to the web UI page for the deployment or cluster, and click the down arrow on the dropdown list to the right of the deployment or cluster name. In the dropdown list, click Collect Diagnostic Data.

Cloudera Director will make an API call to the Cloudera Manager API collectDiagnosticData. If successful, Cloudera Manager will send the diagnostic data to Cloudera Support and, if the download diagnostic data property is set to true in the Cloudera Director application.properties file, will also download a zip file containing the diagnostic data for the deployment or cluster to the Cloudera Director EC2 instance. If diagnostic data collection is unsuccessful, and the download diagnostic data property is set to true, Cloudera Manager will download the Cloudera Manager service logs to Cloudera Director.

Manually Triggering Collection of Diagnostic Data at Cluster Termination

When you terminate a Cloudera Manager deployment or CDH cluster in the web UI, the screen for confirming the termination includes a button that triggers collection of diagnostic data before terminating.

Here is the button as seen on the screen for terminating a Cloudera Manager deployment:


Using the API

To manually trigger collection of diagnostic data for Cloudera Manager deployments, use the API at http://cloudera_director_ip:port_number/api-console/index.html#!/deployments/collectDiagnosticData.

To manually trigger collection of diagnostic data for CDH clusters, use the API at http://cloudera_director_ip:port_number/api-console/index.html#!/clusters/collectDiagnosticData.

Status for Data Collection

While diagnostic data collection is in progress, the status of the deployment or cluster changes from its current state, for example, Ready or Bootstrap Failed, to Updating: Collecting diagnostic data when you mouse-over the Status bar:


The process does not involve an update of the cluster status; the updating message is displayed simply to inform you that diagnostic data collection is in progress. Diagnostic data collection does not change the status of the cluster, so when the data collection is complete, the deployment or cluster status changes back to what it was before diagnostic data collection began.

If you click View Log Status on the deployment or cluster screen, the Diagnostic Log Summary is displayed, showing information about the last diagnostic data collection:


If diagnostic data has never been collected for the deployment or cluster, the Diagnostic Log Status value is Not Collected and there is no link to open the Diagnostic Log Status screen.

Configuring Diagnostic Data Collection

By default, Cloudera Manager sends diagnostic data to Cloudera Support and to Cloudera Director. Whether the collected diagnostic data is sent to Cloudera Support is controlled by a configuration property in Cloudera Manager. Whether diagnostic data (or the Cloudera Manager logs, if diagnostic data collection is unsuccessful) are downloaded to Cloudera Director is controlled by Cloudera Director server configuration properties. You can configure diagnostic data collection on Cloudera Manager and Cloudera Director using the procedures described in this section.

Configuring the Upload of Diagnostic Data to Cloudera Support

The Cloudera Manager server property that determines whether diagnostic data is automatically sent to Cloudera Support has the display name Send Diagnostic Data to Cloudera Automatically and the API name phone_home. The default value for this property is true. To disable diagnostic data collection in Cloudera Manager, set this property to false. You can set the property in the Cloudera Manager web UI by following these steps:
  1. In Cloudera Manager, click Administration > Settings.
  2. In the list of Filters in the lefthand pane, click Support.
  3. Click the checkbox for the property Send Diagnostic Data to Cloudera Automatically to toggle the setting between true and false.

For more information on the phone_home property, see the table in the Support section of Cloudera Manager Server Properties.

Configuring the Download of Diagnostic Data to Cloudera Director

There are several Cloudera Director server configuration properties that affect the way diagnostic data is handled. These properties can be set in the application.properties file or at the command line. The application.properties file is located at /etc/cloudera-director-server/ on the Cloudera Director instance.
  • lp.debug.collectDiagnosticDataOnFailure: Determines whether automatic collection of diagnostic data will occur for cluster bootstrap or update failures. The default value is true.
  • lp.debug.downloadDiagnosticData: Determines whether or not, in addition to being sent to Cloudera Support, the diagnostic data is also downloaded to the Cloudera Director instance. The default value is true.
  • lp.remote.terminate.assumeYes: Determines whether Cloudera Director skips prompting the user to confirm termination when the terminate-remote command is invoked. If you set the property to true, termination proceeds even if diagnostic data collection has failed. The default setting is false
  • lp.debug.diagnosticDataDownloadDirectory: Sets a non-default path for the download of diagnostic data for deployments and clusters. The default location is /tmp. The directory where diagnostic data has been downloaded appears in the File Path field in the Diagnostic Log Summary.
  • lp.debug.createDiagnosticDataDownloadDirectory: A setting of true allows Cloudera Director to create the non-default download directory specified in lp.debug.diagnosticDataDownloadDirectory if it does not exist. The default value is true.

For information about setting Cloudera Director properties by using the CLI or editing the application.properties file, see Setting Cloudera Director Properties.