5. Resolving General Problems

5.1. Problem: After upgrading to Ambari 2.1.2, you receive File Does Not Exist alerts.

After upgrading to Ambari 2.1.2, you receive alerts for "DataNode Unmounted Data Dir" that the /var/lib/ambari-agent/data/datanode/dfs_data_dir_mount.hist file does not exist. The hadoop-env/dfs.datanode.data.dir.mount.file configuration property is no longer customizable from Ambari. The original default value of /etc/hadoop/conf/dfs_data_dir_mount.hist is now /var/lib/ambari-agent/data/datanode/dfs_data_dir_mount.hist, which is not customizable. On Ambari Agent upgrade, Ambari will automatically move the file from /etc/hadoop/conf/dfs_data_dir_mount.hist to /var/lib/ambari-agent/data/datanode/dfs_data_dir_mount.hist. If you have not modified this configuration property, no action is required.

5.1.1. Solution:

If you had previously modified the hadoop-env/dfs.datanode.data.dir.mount.file value to a custom location, after upgrading to Ambari 2.1.2, you must restart your DataNodes for the file to written to be the new location.

5.2. During Enable Kerberos, the Check Kerberos operation fails.

When enabling Kerberos using the wizard, the Check Kerberos operation fails. In /var/log/ambari-server/ambari-server.log, you see a message: 02:45:44,490 WARN [qtp567239306-238] MITKerberosOperationHandler:384 - Failed to execute kadmin:

5.2.1. Solution 1:

Check that NTP is running and confirm your hosts and the KDC times are in sync. A time skew as little as 5 minutes can cause Kerberos authentication to fail.

5.2.2. Solution 2: (on RHEL/CentOS/Oracle Linux)

Check that the Kerberos Admin principal being used has the necessary KDC ACL rights as set in /var/kerberos/krb5kdc/kadm5.acl .

5.3. Problem: Hive developers may encounter an exception error message during Hive Service Check

MySQL is the default database used by the Hive metastore. Depending on several factors, such as the version and configuration of MySQL, a Hive developer may see an exception message similar to the following one:

An exception was thrown while adding/validating classes) : Specified key was too long; max key length is 767 bytes

5.3.1. Solution

Administrators can resolve this issue by altering the Hive metastore database to use the Latin1 character set, as shown in the following example: mysql> ALTER DATABASE<metastore.database.name>character set latin1;

5.4. Problem: API calls for PUT, POST, DELETE respond with a "400 - Bad Request"

When attempting to perform a REST API call, you receive a 400 error response. REST API calls require the "X-Requested-By" header.

5.4.1. Solution

Starting with Ambari 1.4.2, you must include the "X-Requested-By" header with the REST API calls.

For example, if using curl, include the -H "X-Requested-By: ambari" option. curl -u admin:admin -H "X-Requested-By: ambari" -X DELETE http://<ambari-host>:8080/api/v1/hosts/host1

5.5. Problem: Ambari is checking disk full on non-local disks; causing a high number of auto-mounted home directories

When Ambari issues it's check to detect local disk capacity and use for each Ambari Agent, it uses df by default instead of df -l to only check local disks. If using NFS auto-mounted home directories, this can lead to a high number of home directories being mounted on each host; causing shutdown delays and disk capacity check delays.

5.5.1. Solution:

On the Ambari Server, edit the /etc/ambari-server/conf/ambari.properties and add the following property to only check locally mounted devices.

agent.check.remote.mounts=false

5.6. Problem: Links in pdf documentation not working.

Links in 2.1.1. pdf documentation not working.

5.6.1. Solution:

Use working links from 2.1.1 html documentation.

5.7. Problem: Ambari Web shows Storm summary values as N/A in a Kerberized cluster

With a Kerberos-enabled cluster that includes Storm, in Ambari Web > Services > Storm, the Summary values for Slots, Tasks, Executors and Topologies show as "n/a". Ambari Server log also includes the following ERROR:

24 Mar 2015 13:32:41,
288 ERROR [pool-2-thread-362] 
AppCookieManager:122 - 
SPNego authentication failed,
cannot get hadoop.auth cookie for URL: 
http: //c6402.ambari.apache.org:8744/api/
v1/topology/summary?field=topologies

5.7.1. Solution:

When Kerberos is enabled, Storm API requires SPNEGO authentication. Refer to the Ambari Security Guide to Set Up Ambari for Kerberos to enable Ambari to authenticate against the Storm API via SPNEGO.

5.8. Problem: kadmin running Ambari Server as non-root, cannot open log file.

When running Ambari Server as non-root, when enabling Kerberos, if kadmin fails to authenticate, you will see the following error in ambari-server.log if Ambari cannot access the kadmind.log.

STDERR: Couldn't open log file /var/log/kadmind.log: Permission denied 
kadmin: GSS-API (or Kerberos) error while initializing kadmin interface

5.8.1. Solution:

Be sure the user that Ambari Server is configured to run has permissions to write to the kadmind.log.

Legal notices