Troubleshooting Cloudera Director

This topic contains information on issues, causes, and solutions for problems you might face when setting up, configuring, or using Cloudera Director.

Error Occurs if Tags Contain Unquoted Special Characters

If you use the configuration file with the bootstrap command to start Cloudera Director client, or use the bootstrap-remote command to set up a cluster with Cloudera Director server, and you have added a configuration to the config file that includes special characters without quotes, you may receive an error. This applies to HOCON characters, and includes periods. If the added configuration is in the form x.y, for example, the following error message may be displayed: "com.typesafe.config.ConfigException$WrongType: ... <x> has type OBJECT rather than STRING". This means that x.y must be in quotes, as in "x.y". An example of a configuration that would require quoting is "log.dirs" in Kafka.

Viewing Cloudera Director Logs

To help you troubleshoot problems, you can view the Cloudera Director logs. Log files can be found in the following locations:
  • Cloudera Director Client
    • One shared log file per user account: 
      $HOME/.cloudera-director/logs/application.log
  • Cloudera Director Server
    • One file for all clusters: 
      /var/log/cloudera-director-server/application.log

Location of H2 Embedded Database

Cloudera Director uses an H2 embedded database to store environment and cluster data. The H2 embedded database file is located at: 
/var/lib/cloudera-director-server/state.h2.db

DNS Issues

Symptom

Director fails to bootstrap a cluster with a DNS error.

Cause

The Amazon Virtual Private Cloud (VPC) is not set up for forward and reverse hostname resolution. Functional forward and reverse DNS resolution is a key requirement for many components of the Cloudera EDH platform, including Cloudera Director.

Solution

Configure the VPC for forward and reverse hostname resolution. You can verify if DNS is working as expected on a host by issuing the following one-line Python command: 
python -c "import socket; print socket.getfqdn(); print socket.gethostbyname(socket.getfqdn())"

For more information on DNS and Amazon VPCs, see DHCP Options Sets in the Amazon VPC documentation.

If you are using Amazon-provided DNS, perform these steps to configure DHCP options:

  1. Log in to the AWS Management Console.

  2. Select VPC from the Services navigation list box.

  3. In the left pane, click Your VPCs. A list of currently configured VPCs appears.

  4. Select the VPC you are using and note the DHCP options set ID.

  5. In the left pane, click DHCP Option Sets. A list of currently configured DHCP Option Sets appears.

  6. Select the option set used by the VPC.

  7. Check for an entry similar to the following and make sure the domain-name is specified. For example:

    domain-name = ec2.internal
domain-name-servers = AmazonProvidedDNS

  8. If it is not configured correctly, create a new DHCP option set for the specified region and assign it to the VPC. For information on how to specify the correct domain name, see the AWS Documentation.

Server doesn't start

Symptom

The Cloudera Director server doesn't start or quickly exits with an Out of Memory exception.

Cause

The Cloudera Director server is running on a machine with insufficient memory.

Solution

Run Cloudera Director on an instance that has at least 1GB of free memory. See Requirements and Supported Versions for more details on Cloudera Director hardware requirements.

Problem When Removing Hosts from a Cluster

Symptom

A Modify Cluster operation fails to complete.

Cause

You are trying to shrink the cluster below the HDFS replication factor. See Removing or Repairing Hosts in a Cluster for more information about replication factors.

Solution

Do not attempt to shrink a cluster below the HDFS replication factor. Doing so can result in a loss of data.

Problems Connecting to Cloudera Director Server

Symptom

You are unable to connect to the Cloudera Director server.

Cause

Configuration of security group and iptables settings. For more information about configuring security groups and turning off iptables, see Setting Up a VPC. Some operating systems have IP tables turned on by default, and they must be turned off.

Solution

Check security group and iptables settings and reconfigure if necessary.