Configuring CDH Components for Hue

Hue can use either of the following to access HDFS data:

• WebHDFS provides high-speed data transfer with good locality because clients talk directly to the DataNodes inside the Hadoop cluster.

• HttpFS is a proxy service appropriate for integration with external systems that are not behind the cluster's firewall.

Both WebHDFS and HttpFS use the HTTP REST API so they are fully interoperable, but Hue must be configured to use one or the other. For HDFS HA deployments, you must use HttpFS.

To configure Hue to use either WebHDFS or HttpFS, do the following steps:

1. For WebHDFS only:
   1. Add the following property in hdfs-site.xml to enable WebHDFS in the NameNode and DataNodes:

      <property>
        <name>dfs.webhdfs.enabled</name>
        <value>true</value>
      </property>

   2. Restart your HDFS cluster.
2. Configure Hue as a proxy user for all other users and groups, meaning it may submit a request on behalf of any other user:

   WebHDFS: Add to core-site.xml:

   <!-- Hue WebHDFS proxy user setting -->
   <property>
     <name>hadoop.proxyuser.hue.hosts</name>
     <value>*</value>
   </property>
   <property>
     <name>hadoop.proxyuser.hue.groups</name>
     <value>*</value>
   </property>

   HttpFS: Verify that /etc/hadoop-httpfs/conf/httpfs-site.xml has the following configuration:

   <!-- Hue HttpFS proxy user setting -->
   <property>
     <name>httpfs.proxyuser.hue.hosts</name>
     <value>*</value>
   </property>
   <property>
     <name>httpfs.proxyuser.hue.groups</name>
     <value>*</value>
   </property>

   If the configuration is not present, add it to /etc/hadoop-httpfs/conf/httpfs-site.xml and restart the HttpFS daemon.
3. Verify that core-site.xml has the following configuration:

   <property>  
   <name>hadoop.proxyuser.httpfs.hosts</name>  
   <value>*</value>  
   </property>  
   <property>  
   <name>hadoop.proxyuser.httpfs.groups</name>  
   <value>*</value>  
   </property>  

   If the configuration is not present, add it to /etc/hadoop/conf/core-site.xml and restart Hadoop.
4. With root privileges, update hadoop.hdfs_clusters.default.webhdfs_url in hue.ini to point to the address of either WebHDFS or HttpFS.

   [hadoop]
   [[hdfs_clusters]]
   [[[default]]]
   # Use WebHdfs/HttpFs as the communication mechanism.

   WebHDFS:

   ...
   webhdfs_url=http://FQDN:50070/webhdfs/v1/

   HttpFS:

   ...
   webhdfs_url=http://FQDN:14000/webhdfs/v1/

If your JobTracker and Hue Server are located on the same host, copy the file over. If you are currently using CDH 4, your MapReduce library directory might be in /usr/lib/hadoop/lib.

$ cd /usr/lib/hue
$ cp desktop/libs/hadoop/java-lib/hue-plugins-*.jar /usr/lib/hadoop-0.20-mapreduce/lib

If your JobTracker runs on a different host, scp the Hue plugins .jar file to the JobTracker host.

Add the following properties to mapred-site.xml:

<property>
  <name>jobtracker.thrift.address</name>
  <value>0.0.0.0:9290</value>
</property>
<property>
  <name>mapred.jobtracker.plugins</name>
  <value>org.apache.hadoop.thriftfs.ThriftJobTrackerPlugin</value>
  <description>Comma-separated list of jobtracker plug-ins to be activated.</description>
</property>

You can confirm that the plugins are running correctly by tailing the daemon logs:

$ tail --lines=500 /var/log/hadoop-0.20-mapreduce/hadoop*jobtracker*.log | grep ThriftPlugin
2009-09-28 16:30:44,337 INFO org.apache.hadoop.thriftfs.ThriftPluginServer: Starting Thrift server
2009-09-28 16:30:44,419 INFO org.apache.hadoop.thriftfs.ThriftPluginServer:
Thrift server listening on 0.0.0.0:9290

In order to run DistCp, Streaming, Pig, Sqoop, and Hive jobs in Job Designer or the Oozie Editor/Dashboard application, you must make sure the Oozie shared libraries are installed for the correct version of MapReduce (MRv1 or YARN). See Installing the Oozie ShareLib in Hadoop HDFS for instructions.

<!-- Default proxyuser configuration for Hue -->
<property>
    <name>oozie.service.ProxyUserService.proxyuser.hue.hosts</name>
    <value>*</value>
</property>
<property>
    <name>oozie.service.ProxyUserService.proxyuser.hue.groups</name>
    <value>*</value>
</property>

See Search Configuration for details on how to configure the Search application for Hue.

The Beeswax daemon has been replaced by HiveServer2. Hue should therefore point to a running HiveServer2. This change involved the following major updates to the [beeswax] section of the Hue configuration file, hue.ini.

[beeswax]
  # Host where Hive server Thrift daemon is running.
  # If Kerberos security is enabled, use fully-qualified domain name (FQDN).
  ## hive_server_host=<FQDN of HiveServer2>

  # Port where HiveServer2 Thrift server runs on.
  ## hive_server_port=10000