Using CDH with Isilon Storage

The following versions of Cloudera and Isilon products are supported:

The following features of HDFS are not implemented with Isilon OneFS:

• HDFS caching
• HDFS encryption
• HDFS ACLs

Before installing a Cloudera Manager cluster to use Isilon storage, perform the following steps on the Isilon OneFS system. For detailed information on setting up Isilon OneFS for Cloudera Manager, see the Isilon documentation at https://community.emc.com/docs/DOC-39529.

1. Create an Isilon access zone with HDFS support.

   Example:
   /ifs/your-access-zone/hdfs

2. Create two directories that will be used by all CDH services:
   1. Create a tmp directory in the access zone.
      • Create supergroup group and hdfs user.
      • Create a tmp directory and set ownership to hdfs:supergroup, and permissions to 1777.

        Example:
        cd hdfs_root_directory
        isi_run -z zone_id mkdir tmp
        isi_run -z zone_id chown hdfs:supergroup tmp
        isi_run -z zone_id chmod 1777 tmp

   2. Create a user directory in the access zone and set ownership to hdfs:supergroup, and permissions to 755

      Example:
      cd hdfs_root_directory
      isi_run -z zone_id mkdir user
      isi_run -z zone_id chown hdfs:supergroup user
      isi_run -z zone_id chmod 755 user

3. Create the service-specific users, groups, or directories for each CDH service you plan to use. Create the directories under the access zone you have created.
   • ZooKeeper: nothing required.
   • HBase
     • Create hbase group with hbase user.
     • Create the root directory for HBase:

       Example:
       hdfs_root_directory/hbase hbase:hbase 755

   • YARN (MR2)
     • Create mapred group with mapred user.
     • Create history directory for YARN:

       Example:
       hdfs_root_directory/user/history mapred:hadoop 777

     • Create the remote application log directory for YARN:

       Example:
       hdfs_root_directory/tmp/logs mapred:hadoop 775

   • Oozie
     • Create oozie group with oozie user.
     • Create the user directory for Oozie:

       Example:
       hdfs_root_directory/user/oozie oozie:oozie 775

   • Flume
     • Create flume group with flume user.
     • Create the user directory for Flume:

       Example:
       hdfs_root_directory/user/flume flume:flume 775

   • Hive
     • Create hive group with hive user.
     • Create the user directory for Hive:

       Example:
       hdfs_root_directory/user/hive hive:hive 775

     • Create the warehouse directory for Hive:

       Example:
       hdfs_root_directory/user/hive/warehouse hive:hive 1777

     • Create a temporary directory for Hive:

       Example:
       hdfs_root_directory/tmp/hive hive:supergroup 777

   • Solr
     • Create solr group with solr user.
     • Create the data directory for Solr:

       Example:
       hdfs_root_directory/solr solr:solr 775
       

   • Sqoop
     • Create sqoop group with sqoop2 user.
     • Create the user directory for Sqoop:

       Example:
       hdfs_root_directory/user/sqoop2 sqoop2:sqoop 775

   • Hue
     • Create hue group with hue user.
     • Create sample group with sample user.
   • Spark
     • Create spark group with spark user.
     • Create the user directory for Spark:

       Example:
       hdfs_root_directory/user/spark spark:spark 751

     • Create application history directory for Spark:

       Example:
       hdfs_root_directory/user/spark/applicationHistory spark:spark 1777

Once the users, groups, and directories are created in Isilon OneFS, you are ready to install Cloudera Manager.

If you choose parcel installation on the Cluster Installation screen, the installation wizard will point to the latest parcels of CDH available.

On the installation wizard's Cluster Setup page, choose Custom Services, and choose the services you want installed in the cluster. Be sure to choose Isilon among the selected services, do not select the HDFS service, and do not check Include Cloudera Navigator at the bottom of the Cluster Setup page. Also, on the Role Assignments page, be sure to specify the hosts that will serve as gateway roles for the Isilon service. You can add gateway roles to one, some, or all nodes in the cluster.

You can use Impala to query data files that reside on EMC Isilon storage devices, rather than in HDFS. This capability allows convenient query access to a storage system where you might already be managing large volumes of data. The combination of the Impala query engine and Isilon storage is certified on CDH 5.4.4 through CDH 5.15.

isi hdfs settings modify --default-block-size=256MB

The typical use case for Impala and Isilon together is to use Isilon for the default filesystem, replacing HDFS entirely. In this configuration, when you create a database, table, or partition, the data always resides on Isilon storage and you do not need to specify any special LOCATION attribute. If you do specify a LOCATION attribute, its value refers to a path within the Isilon filesystem. For example:

-- If the default filesystem is Isilon, all Impala data resides there
-- and all Impala databases and tables are located there.
CREATE TABLE t1 (x INT, s STRING);

-- You can specify LOCATION for database, table, or partition,
-- using values from the Isilon filesystem.
CREATE DATABASE d1 LOCATION '/some/path/on/isilon/server/d1.db';
CREATE TABLE d1.t2 (a TINYINT, b BOOLEAN);

Impala can write to, delete, and rename data files and database, table, and partition directories on Isilon storage. Therefore, Impala statements such as CREATE TABLE, DROP TABLE, CREATE DATABASE, DROP DATABASE, ALTER TABLE, and INSERT work the same with Isilon storage as with HDFS.

When the Impala spill-to-disk feature is activated by a query that approaches the memory limit, Impala writes all the temporary data to a local (not Isilon) storage device. Because the I/O bandwidth for the temporary data depends on the number of local disks, and clusters using Isilon storage might not have as many local disks attached, pay special attention on Isilon-enabled clusters to any queries that use the spill-to-disk feature. Where practical, tune the queries or allocate extra memory for Impala to avoid spilling. Although you can specify an Isilon storage device as the destination for the temporary data for the spill-to-disk feature, that configuration is not recommended due to the need to transfer the data both ways using remote I/O.

When tuning Impala queries on HDFS, you typically try to avoid any remote reads. When the data resides on Isilon storage, all the I/O consists of remote reads. Do not be alarmed when you see non-zero numbers for remote read measurements in query profile output. The benefit of the Impala and Isilon integration is primarily convenience of not having to move or copy large volumes of data to HDFS, rather than raw query performance. You can increase the performance of Impala I/O for Isilon systems by increasing the value for the num_remote_hdfs_io_threads configuration parameter, in the Cloudera Manager user interface for clusters using Cloudera Manager, or through the --num_remote_hdfs_io_threads startup option for the impalad daemon on clusters not using Cloudera Manager.

For information about managing Isilon storage devices through Cloudera Manager, see Using CDH with Isilon Storage.