6.1. Known Issues for Hadoop

Hadoop fsck /, after upgrade to HDP 1.3.2 from HDP 1.0, errors / ""ERROR security.UserGroupInformation: PriviledgedActionException as:hdfs cause:java.io.IOException: Premature EOF"

PROBLEM: After you upgrade to HDP 1.3.2 from HDP 1.0, running the hadoop fsck / command causes the following error to display:

ERROR security.UserGroupInformation: PriviledgedActionException as:hdfs cause:java.io.IOException: Premature EOF

There is an issue in the way the CLI parses the HTTP response from the fsck servlet. With no options specified, the fsck servlet is writing a stream of '.' characters to indicate progress, with a line break every 100 characters. The client is using BufferedReader#readLine to parse it. This is potentially a very large response, so the HTTP response is using chunked transfer encoding.

Workaround: Instead of running hadoop fsck / on its own, run the command with the blocks, files, and location options:

hadoop fsck / -blocks -files -location

These options let the fsck servlet indicate progress in a readable way.

JobTracker UI, JMX queries, and RPC calls somtimes hang in HA mode.

Problem: JobTracker becomes slow and non-responsive in HA mode because dfs.client.retry.policy.enabled is not set to 'final' and 'false' for the JobTracker.

Workaround:Set dfs.client.retry.policy.enabled to 'final' and 'false' only for the JobTracker. Clients (such as MapReduce, Pig, Hive, Oozie) should still be set to 'true' in HA mode.

Problem: While uploading files to NFS-MountDir, the following error is reported in the DataNode log file:

INFO org.apache.hadoop.hdfs.nfs.nfs3.OpenFileCtx: requesed offset=4980736 and current filesize=0

Workaround: On some environments, especially for virtualized environments, copying large files of size close to 1GB fails intermittently. This issue is expected to be addressed in the upcoming release.

Use of init.d scripts for starting or stopping Hadoop services, is not recommended.