Troubleshooting Cloudera Director

This topic contains information on issues, causes, and solutions for problems you might face when setting up, configuring, or using Cloudera Director.

Viewing Cloudera Director Logs

To help you troubleshoot problems, you can view the Cloudera Director logs. Log files can be found in the following locations:
  • Cloudera Director Client
    • One shared log file per user account: 
      $HOME/.cloudera-director/logs/application.log
  • Cloudera Director Server
    • One file for all clusters: 
      /var/log/cloudera-director-server/application.log

Backing Up the H2 Embedded Database

By default, Cloudera Director uses an H2 embedded database to store environment and cluster data. The H2 embedded database file is located at: 
/var/lib/cloudera-director-server/state.h2.db

Back up the state.h2.db file to avoid losing environment and cluster data. To ensure that your backup copy can be restored, you should use the H2 backup tools and rather than simply copying the file. For more information, see the H2 Tutorial.

Cloudera Director Cannot Manage a Cluster That Was Kerberized Through Cloudera Manager

Symptom

Cloudera Director cannot manage a cluster after Cloudera Manager is used to enable Kerberos on the cluster.

Cause

Once a cluster is deployed through Cloudera Director, some changes to the cluster that are made using Cloudera Manager cause Cloudera Director to be out of sync, and hence unable to manage the cluster. See Cloudera Director and Cloudera Manager Usage.

Solution

Deploy a new kerberized cluster, use distcp to transfer data from the old cluster to the new one, and then destroy the old cluster.

RDS Name Conflicts

Symptom

RDS name conflicts occur when creating multiple clusters with the same configuration file.

Cause

The most common reason for this problem is that deletion of an older RDS instance has not completed when the user tries to launch a new cluster using the same configuration file, and hence the same RDS name.

Solution

Allow more time for an RDS instance to be completely removed before creating a new cluster with the same configuration file, or change the name of the RDS instance in the configuration files for new clusters.

New Cluster Fails to Start Because of Missing Roles

Symptom

A new cluster will not start because roles are missing.

Cause

Cloudera Director does not validate that all necessary roles are assigned when provisioning a cluster. This can lead to failures during the intial run of a new cluster. For example, if the gateway instance group was removed but the Flume Agent and Kafka Broker were assigned to roles in that group, the cluster will fail to start.

Solution

Ensure that all required role types for the CDH services included in the cluster are assigned to instances before starting the cluster.

Cloudera Director Server Will Not Start with Unsupported Java Version

Symptom

Cloudera Director server will not start, and /var/log/cloudera-director-server/cloudera-director-server.out has the following error: 
Exception in thread "main" java.lang.UnsupportedClassVersionError: com/cloudera/launchpad/Server : Unsupported major.minor version 51.0

Cause

You are running Cloudera Director server against an older, unsupported version of the Oracle Java SE Development Kit (JDK).

Solution

Update to Oracle JDK version 7 or 8.

Error Occurs if Tags Contain Unquoted Special Characters

Symptom

When using the configuration file with the bootstrap command to start Cloudera Director client, or using the bootstrap-remote command to set up a cluster with Cloudera Director server, an error message is displayed. This applies to HOCON characters, and includes periods. If the added configuration is in the form x.y, for example, the following error message may be displayed: "com.typesafe.config.ConfigException$WrongType: ... <x> has type OBJECT rather than STRING". This means that x.y must be in quotes, as in "x.y". 
com.typesafe.config.ConfigException$WrongType: ... <x> has type OBJECT rather than STRING

Cause

Cloudera Director validation checks to ensure that special characters in configurations are enclosed in double quotes.

Solution

Use double quotes for special characters in configurations. An example of a configuration that would require double quotes is "log.dirs" in Kafka.

DNS Issues

Symptom

Director fails to bootstrap a cluster with a DNS error.

Cause

This can be caused by a couple of things:
  • The Edit DNS Hostnames is not set to Yes the VPC settings.
  • The Amazon Virtual Private Cloud (VPC) is not set up for forward and reverse hostname resolution. Functional forward and reverse DNS resolution is a key requirement for many components of the Cloudera EDH platform, including Cloudera Director.

Solutions

In the AWS Management Console, go to Services > Networking and click VPC. In the VPC Dashboard, select your VPC and click Action. In the shortcut menu, click Edit DNS Hostnames and click Yes. If this does not fix the issue, continue with the instructions that follow to configure forward and reverse hostname resolution.

Configure the VPC for forward and reverse hostname resolution. You can verify if DNS is working as expected on a host by issuing the following one-line Python command: 
python -c "import socket; print socket.getfqdn(); print socket.gethostbyname(socket.getfqdn())"

For more information on DNS and Amazon VPCs, see DHCP Options Sets in the Amazon VPC documentation.

If you are using Amazon-provided DNS, perform these steps to configure DHCP options:

  1. Log in to the AWS Management Console.

  2. Select VPC from the Services navigation list box.

  3. In the left pane, click Your VPCs. A list of currently configured VPCs appears.

  4. Select the VPC you are using and note the DHCP options set ID.

  5. In the left pane, click DHCP Option Sets. A list of currently configured DHCP Option Sets appears.

  6. Select the option set used by the VPC.

  7. Check for an entry similar to the following and make sure the domain-name is specified. For example:

    domain-name = ec2.internal
domain-name-servers = AmazonProvidedDNS

  8. If it is not configured correctly, create a new DHCP option set for the specified region and assign it to the VPC. For information on how to specify the correct domain name, see the AWS Documentation.

Server Does Not Start

Symptom

The Cloudera Director server does not start or quickly exits with an Out of Memory exception.

Cause

The Cloudera Director server is running on a machine with insufficient memory.

Solution

Run Cloudera Director on an instance that has at least 1GB of free memory. See Resource Requirements for more details on Cloudera Director hardware requirements.

Problem When Removing Hosts from a Cluster

Symptom

A Modify Cluster operation fails to complete.

Cause

You are trying to shrink the cluster below the HDFS replication factor. See Removing Instances from a Cluster (Note paragraph) for more information about replication factors.

Solution

Do not attempt to shrink a cluster below the HDFS replication factor. Doing so can result in a loss of data.

Problems Connecting to Cloudera Director Server

Symptom

You are unable to connect to the Cloudera Director server.

Cause

Configuration of security group and iptables settings. For more information about configuring security groups, see Setting up the AWS Environment. For commands to turn off iptables, see either Installing Cloudera Director Server and Client on the EC2 Instance or Installing Cloudera Director Server and Client on Google Compute Engine. Some operating systems have IP tables turned on by default, and they must be turned off.

Solution

Check security group and iptables settings and reconfigure if necessary.