DSSD D5 Installation Path C - Manual Installation Using Cloudera Manager Tarballs

See the EMC DSSD D5 document DSSD D5 Installation and Service Guide for more information about these tasks.

$ sudo mkdir /opt/cloudera-manager

$ sudo tar xzf cloudera-manager*.tar.gz -C /opt/cloudera-manager

The files are extracted to a subdirectory named according to the Cloudera Manager version being extracted. For example, files could be extracted to /opt/cloudera-manager/cm-5.0/. This full path is needed later and is referred to as tarball_root directory.

$ sudo useradd --system --home=/opt/cloudera-manager/cm-5.6.0/run/cloudera-scm-server --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm

1. On the Cloudera Manager Server host, create a parcel repository directory:

   $ sudo mkdir -p /opt/cloudera/parcel-repo

2. Change the directory ownership to be the username you are using to run Cloudera Manager:

   $ sudo chown username:groupname /opt/cloudera/parcel-repo

   where username and groupname are the user and group names (respectively) you are using to run Cloudera Manager. For example, if you use the default username cloudera-scm, you would run the command:

   $ sudo chown cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo

3. On each cluster host, create a parcels directory:

   $ sudo mkdir -p /opt/cloudera/parcels

4. Change the directory ownership to be the username you are using to run Cloudera Manager:

   $ sudo chown username:groupname /opt/cloudera/parcels

   where username and groupname are the user and group names (respectively) you are using to run Cloudera Manager. For example, if you use the default username cloudera-scm, you would run the command:

   $ sudo chown cloudera-scm:cloudera-scm /opt/cloudera/parcels

If the Cloudera Manager Server does not start, see Troubleshooting Installation and Upgrade Problems.

When you install with tarballs and parcels, some services may require additional dependencies that are not provided by Cloudera. On each host, install the required packages:

1. Click the Cloudera Manager logo to open the Home page.
2. Click Administration > Settings.
3. Type DSSD in the Search box.
4. Select the DSSD Mode property.
5. Click Save Changes to commit the changes.

   Cloudera Manager reconfigures the system for DSSD mode, which may take several minutes.

6. Click the Cloudera Manager logo to open the Home page.
7. Click Add Cluster to continue with the installation.
8. The Cloudera Manager End User License Terms and Conditions page displays. Read the terms and conditions and then select Yes to accept them.
9. Click Continue.
10. The EMC Software License Agreement page displays. Read the terms and conditions and then select Yes to accept them.
11. Click Continue.

    The Welcome to Cloudera Manager page displays.

From the Welcome to Cloudera Manager page, you can select the edition of Cloudera Manager to install and, optionally, install a license:

1. Choose which edition to install:
   • Cloudera Express, which does not require a license, but provides a limited set of features.
   • Cloudera Enterprise Enterprise Data Hub Edition Trial, which does not require a license, but expires after 60 days and cannot be renewed.
   • Cloudera Enterprise with one of the following license types:
     • Basic Edition
     • Flex Edition
     • Enterprise Data Hub Edition
   If you choose Cloudera Express or Cloudera Enterprise Enterprise Data Hub Edition Trial, you can upgrade the license at a later time. See Managing Licenses.
2. If you elect Cloudera Enterprise, install a license:
   1. Click Upload License.
   2. Click the document icon to the left of the Select a License File text field.
   3. Go to the location of your license file, click the file, and click Open.
   4. Click Upload.
3. Information is displayed indicating what the CDH installation includes. At this point, you can click the Support drop-down menu to access online Help or the Support Portal.
4. Click Continue to proceed with the installation.

1. Click the Currently Managed Hosts tab.
2. Choose the hosts to add to the cluster.
3. Click Continue.

   The Cluster Installation Select Repository screen displays.

1. Choose the CDH and managed service version:
   1. Choose the parcels to install. The choices depend on the repositories you have chosen; a repository can contain multiple parcels. Only the parcels for the latest supported service versions are configured by default. Select the following parcels:
      • CDH 5
      • DSSD version 1.2
      • DSSD_SCR version 1.2 - This parcel enables short-circuit reads for HBase and Impala. Select this parcel even if you intend to disable short-circuit reads. (See DSSD D5 and Short-Circuit Reads.)
      • Any additional parcels required for your deployment (for example: Accumulo, Spark, or Keytrustee) .
      You can add additional parcels for previous versions by specifying custom repositories. For example, you can find the locations of the previous CDH 5 parcels at https://archive.cloudera.com/cdh5/parcels/.

      1. To specify the parcel directory, specify the local parcel repository, add a parcel repository, or specify the properties of a proxy server through which parcels are downloaded, click the More Options button and do one or more of the following:
         • Parcel Directory and Local Parcel Repository Path - Specify the location of parcels on cluster hosts and the Cloudera Manager Server host. If you change the default value for Parcel Directory and have already installed and started Cloudera Manager Agents, restart the Agents:

           sudo service cloudera-scm-agent restart

         • Parcel Repository - In the Remote Parcel Repository URLs field, click the button and enter the URL of the repository. The URL you specify is added to the list of repositories listed in the Configuring Cloudera Manager Server Parcel Settings page and a parcel is added to the list of parcels on the Select Repository page. If you have multiple repositories configured, you see all the unique parcels contained in all your repositories.
         • Proxy Server - Specify the properties of a proxy server.
      2. Click OK.
   2. If you are using Cloudera Manager to install software, select the release of Cloudera Manager Agent. You can choose either the version that matches the Cloudera Manager Server you are currently using or specify a version in a custom repository. If you opted to use custom repositories for installation files, you can provide a GPG key URL that applies for all repositories.
2. Click Continue. Cloudera Manager installs the CDH and managed service parcels. During parcel installation, progress is indicated for the phases of the parcel installation process in separate progress bars. If you are installing multiple parcels, you see progress bars for each parcel. When the Continue button at the bottom of the screen turns blue, the installation process is completed. Click Continue.
3. Click Continue.

   The Host Inspector runs to validate the installation and provides a summary of what it finds, including all the versions of the installed components. If the validation is successful, click Finish.

1. In the first page of the Add Services wizard, choose the combination of services to install and whether to install Cloudera Navigator:
   • Select the combination of services to install:
     • Core Hadoop - HDFS, YARN (includes MapReduce 2), ZooKeeper, Oozie, Hive, and Hue
     • Core with HBase
     • Core with Impala
     • Core with Search
     • Core with Spark
     • All Services - HDFS, YARN (includes MapReduce 2), ZooKeeper, Oozie, Hive, Hue, HBase, Impala, Solr, Spark, and Key-Value Store Indexer
     • Custom Services - Any combination of services.
     As you select services, keep the following in mind:
     • Some services depend on other services; for example, HBase requires HDFS and ZooKeeper. Cloudera Manager tracks dependencies and installs the correct combination of services.
     • In a Cloudera Manager deployment of a CDH 4 cluster, the MapReduce service is the default MapReduce computation framework. Choose Custom Services to install YARN, or use the Add Service functionality to add YARN after installation completes.
     • In a Cloudera Manager deployment of a CDH 5 cluster, the YARN service is the default MapReduce computation framework. Choose Custom Services to install MapReduce, or use the Add Service functionality to add MapReduce after installation completes.
     • The Flume service can be added only after your cluster has been set up.
   • If you have chosen Enterprise Data Hub Edition Trial or Cloudera Enterprise, optionally select the Include Cloudera Navigator checkbox to enable Cloudera Navigator. See Cloudera Navigator 2 Overview.
2. Click Continue.
3. Customize the assignment of role instances to hosts. The wizard evaluates the hardware configurations of the hosts to determine the best hosts for each role. The DataNode role is only assigned to hosts that are connected to the DSSD D5. The wizard assigns all worker roles to the same set of hosts to which the HDFS DataNode role is assigned. You can reassign role instances if necessary.

   Click a field below a role to display a dialog box containing a list of hosts. If you click a field containing multiple hosts, you can also select All Hosts to assign the role to all hosts, or Custom to display the pageable hosts dialog box.

   The following shortcuts for specifying hostname patterns are supported:

   • Range of hostnames (without the domain portion)

     Range Definition	Matching Hosts
     10.1.1.[1-4]	10.1.1.1, 10.1.1.2, 10.1.1.3, 10.1.1.4
     host[1-3].company.com	host1.company.com, host2.company.com, host3.company.com
     host[07-10].company.com	host07.company.com, host08.company.com, host09.company.com, host10.company.com

   • IP addresses
   • Rack name

   Click the View By Host button for an overview of the role assignment by hostname ranges.

4. When you are satisfied with the assignments, click Continue.

From the Cluster Setup Review Changes page:

1. Review the configuration changes to be applied. Confirm the settings entered for file system paths. The file paths required vary based on the services to be installed. If you chose to add the Sqoop service, indicate whether to use the default Derby database or the embedded PostgreSQL database. If the latter, type the database name, host, and user credentials that you specified when you created the database.

   The configuration properties that display on this page are somewhat different from those that display when configuring non-DSSD D5 DataNodes. Some properties, such as the DataNode directory have been removed because they do not apply to a cluster that uses DSSD D5 DataNodes. Other properties, such as the Flood Volume Name are specific to the DSSD D5 DataNode role.

2. (Required) In the Flood Volume Name field, enter the name of the Flood Volume as configured in the DSSD D5 appliance. If you are deploying multiple DSSD D5 appliances, note that you must specify this property for each appliance using a Role Group.
3. (Optional) If you are not using the entire capacity of the DSSD D5 for this cluster, set the Usable Capacity property. For most deployments, the default value (100 TB) is correct. See the EMC document DSSD Hadoop Plugin Installation Guide for more information on setting this property.
4. (Optional) Set the value of the HDFS Block Size parameter. The default value for this parameter is 512 MB when in DSSD Mode. You may want to change this for some types of work loads. See Tuning the HDFS Block Size for DSSD Mode.
5. Click Continue.

   The wizard starts the services.

6. When all of the services are started, click Continue.

   You see a success message indicating that your cluster has been successfully started.

7. Click Finish to proceed to the Cloudera Manager Admin Console Home Page.
8. If you see a message indicating that you need to restart Cloudera Management Services, restart the Cloudera Management Service:
   1. Do one of the following:
      • 1. Select Clusters > Cloudera Management Service > Cloudera Management Service.
        2. Select Actions > Restart.
      • On the Home > Status tab, click to the right of Cloudera Management Service and select Restart.
   2. Click Restart to confirm. The Command Details window shows the progress of stopping and then starting the roles.
   3. When Command completed with n/n successful subcommands appears, the task is complete. Click Close.
9. Choose Cluster > HDFS > Configuration and then in the filter section, select Scope > DSSD DataNode to view the DSSD D5 DataNode-specific properties.

   See the Cloudera Manager 5.8 Configuration Properties configuration reference for descriptions of these properties.

   See the EMC document DSSD Hadoop Plugin Installation Guide for information about setting these properties.

10. (Recommended for best performance) Set the Libflood CPU ID property.

    The value to use for this parameter should have been determined during the set up of the DSSD D5 appliance. See “Identify CPUs and NUMA Nodes” in the EMC document DSSD Hadoop Plugin Installation Guide. The value you set for this parameter can effect the performance of your cluster.

11. (Optional) Set the following properties to tune the performance of your cluster:
    • Libflood Command Queues
    • Libflood Command Queue Depth
12. (Optional) Set the Java heap size for the NameNode.
    1. Choose Clusters > HDFS > Configuration.
    2. Type Java heap in the search box.
    3. Set the Java Heap Size of NameNode in Bytes parameter:

       Cloudera Manager automatically sets the value of this parameter to 4 GB (If there are not adequate resources in the cluster, Cloudera Manager may set a smaller value.) Cloudera recommends that you manually set the value of this parameter by calculating the number of HDFS blocks in the cluster and including 1 GB of Java heap for each 1 million HDFS blocks. For more information on HDFS block size and the DSSD D5, see Tuning the HDFS Block Size for DSSD Mode.

    4. Set the Java Heap Size of Secondary NameNode in Bytes parameter to the same value as the Java Heap Size of NameNode in Bytes parameter.
    5. Restart the NameNode:
       1. Choose Clusters > HDFS > Instances.
       2. In the table of roles, select the NameNode (Active) and SecondaryNameNode role types.
       3. Click Actions for Selected > Restart.

If you added an Oozie service, you can change your Oozie configuration to control when data is purged to improve performance, cut down on database disk usage, or to keep the history for a longer period of time. Limiting the size of the Oozie database can also improve performance during upgrades. See Configuring Oozie Data Purge Settings Using Cloudera Manager.

To increase capacity and performance, you can configure a cluster that uses multiple DSSD D5 storage appliances. You configure the cluster by assigning all hosts connected to a DSSD D5 appliance to a single "rack" and select one of three modes to provide policies used by the NameNode to satisfy the configured replication factor. If you are only configuring a single DSSD D5 appliance, skip this section.

You can also move hosts between appliances. See Moving Existing Hosts to a New DSSD D5

You can test the installation following the instructions in Testing the Installation.