In HDP 2.2.8, behavioral changes affect the following Hadoop components:
| Component | Summary | Details |
|---|---|---|
| HDFS | HDFS-8549: Abort the balancer if an upgrade is in progress | As of HDP 2.2.8, the balancer cannot run during upgrade. Running the balancer during an ongoing upgrade has a negative affect, since DNs do not actually delete blocks. This means the balancer is making lots of extra replicas and not actually reducing the disk utilization of over-utilized nodes. |
| Slider | SLIDER-931: Security permissions on set up ZK path are too lax | Zookeeper paths created for applications such as HBase are created with stricter permissions. |
| MapReduce | MAPREDUCE-6454: A hive query run from a hive client running from a mapreduce job doesn't pick up the classpath of the hive client | Added jars in distributed cache to $HADOOP_CLASSPATH (previously only added to CLASSPATH). Now, when Hive spawns a local child task, it can add them on classpath automatically (as Hadoop shell only takes HADOOP_CLASSPATH env). |
| Hue | Hue now shows log messages for create and insert statements. | N/A |
| Hue | Support dfs.umaskmode and fs.permissions.umask-mode when creating files or directories | When creating files or directories in File Browser, Hue does not respect fs.permissions.umask-mode and defaults always to 022. With this change now it is possible to set umask property in Hue configuration file with an octal value that will be used as default one for files and directories creation. |
| YARN | YARN-3626: If we set mapreduce.job.user.classpath.first as true, the localized resources should appear before system classpath. But in windows, it does not work. We have fixed this in YARN-3626. | In response to the mapreduce.job.user.classpath.first setting the classpath is ordered differently so that localized resources will appear before system classpath resources when tasks execute. On Windows this does not work because the localized resources are not linked into their final location when the classpath jar is created. To compensate for that localized jar resources are added directly to the classpath generated for the jar rather than being discovered from the localized directories. Unfortunately, they are always appended to the classpath, and so are never preferred over system resources. Workaround:To resolve this, the situation should be detected and, when applicable, localized resources should be put at the beginning of the classpath rather than the end. |
| YARN | YARN-3978: Users of the Application Timeline Service can now turn off collection of detailed container-level metrics by the use of a new configuration parameter yarn.generic-application-history.save-non-am-container-meta-info and setting it to false. | The default value of yarn.generic-application-history.save-non-am-container-meta-info is true, so there is no change in behavior. For clusters with more than 100 nodes, we recommend this configuration value be set to false to reduce the load on the Application Timeline Service. |
