Use the following instructions to manually configure the cluster properties file for deploying HDP from the command-line or in a script:
Create a
clusterproperties.txt
file or use the sample clusterproperties.txt file extracted from the HDP Installation zip file.Add the properties to the
clusterproperties.txt
file as described in the table below:Important All properties in the
clusterproperties.txt
file must be separated by a newline character.Directory paths cannot contain whitespace characters.
For example,
C:\Program Files\Hadoop
is an invalid directory path for HDP.Use Fully Qualified Domain Names (FQDN) for specifying the network host name for each cluster host. The FQDN is a DNS name that uniquely identifies the computer on the network. By default, it is a concatenation of the host name, the primary DNS suffix, and a period.
When specifying the host lists in the
clusterproperties.txt
file, if the hosts are multi-homed or have multiple NIC cards, make sure that each name or IP address by which you specify the hosts is the preferred name or IP address by which the hosts can communicate among themselves. In other words, these should be the addresses used internal to the cluster, not those used for addressing cluster nodes from outside the cluster.To Enable NameNode HA, you must include the HA properties and exclude the SECONDARY_NAMENODE_HOST definition.
Configuration Values for MSI Installer
Configuration Property Name Description Example value Mandatory/ Optional/ Conditional HDP_LOG_DIR HDP's operational logs are written to this directory on each cluster host. Ensure that you have sufficient disk space for storing these log files. d:\hadoop\logs
Mandatory HDP_DATA_DIR HDP data will be stored in this directory on each cluster node. You can add multiple comma-separated data locations for multiple data directories. d:\hdp\data
Mandatory NAMENODE_HOST The FQDN for the cluster node that will run the NameNode master service. NAMENODE-MASTER.acme.com
Mandatory SECONDARY_NAMENODE_HOST The FQDN for the cluster node that will run the Secondary NameNode master service. SECONDARY-NN-MASTER.acme.com
Mandatory when no HA RESOURCEMANAGER_HOST The FQDN for the cluster node that will run the YARN Resource Manager master service. RESOURCE-MANAGER.acme.com
Mandatory HIVE_SERVER_HOST The FQDN for the cluster node that will run the Hive Server master service. HIVE-SERVER-MASTER.acme.com
Mandatory OOZIE_SERVER_HOST The FQDN for the cluster node that will run the Oozie Server master service. OOZIE-SERVER-MASTER.acme.com
Mandatory WEBHCAT_HOST The FQDN for the cluster node that will run the WebHCat master service. WEBHCAT-MASTER.acme.com
Mandatory FLUME_HOSTS A comma-separated list of FQDN for those cluster nodes that will run the Flume service. FLUME-SERVICE1.acme.com, FLUME-SERVICE2.acme.com, FLUME-SERVICE3.acme.com
Mandatory HBASE_MASTER The FQDN for the cluster node that will run the HBase master. HBASE-MASTER.acme.com
Mandatory HBASE_REGIONSERVERS A comma-separated list of FQDN for those cluster nodes that will run the HBase Region Server services. slave1.acme.com, slave2.acme.com, slave3.acme.com
Mandatory SLAVE_HOSTS A comma-separated list of FQDN for those cluster nodes that will run the DataNode and TaskTracker services. slave1.acme.com, slave2.acme.com, slave3.acme.com
Mandatory ZOOKEEPER_HOSTS A comma-separated list of FQDN for those cluster nodes that will run the ZooKeeper hosts. ZOOKEEEPER-HOST.acme.com
Mandatory FALCON_HOSTS A comma-separated list of FQDN for those cluster nodes that will run the Falcon hosts. falcon.acme.com, falcon1.acme.com, falcon2.acme.com
Optional KNOX_HOST The FQDN of the Knox Gateway host. KNOX-HOST.acme.com
Optional IS_TEZ Install the Tez component on Hive host. YES
orNO
Optional IS_PHOENIX Installs Phoenix on the HBase hosts. YES
orNO
Optional ENABLE_LZO Enables the LZO codec for compression in HBase cells. YES
orNO
Optional DB_FLAVOR Database type for Hive and Oozie metastores (allowed databases are SQL Server and Derby). To use default embedded Derby instance, set the value of this property to derby
. To use an existing SQL Server instance as the metastore DB, set the value asmssql
.mssql
orderby
Mandatory DB_HOSTNAME FQDN for the node where the metastore database service is installed. If using SQL Server, set the value to your SQL Server hostname. If using Derby for Hive metastore, set the value to HIVE_SERVER_HOST. sqlserver1.acme.com
Mandatory DB_PORT This is an optional property required only if you are using SQL Server for Hive and Oozie metastores. By default, the database port is set to 1433
.1433 Optional HIVE_DB_NAME Database for Hive metastore. If using SQL Server, ensure that you create the database on the SQL Server instance. hivedb
Mandatory HIVE_DB_USERNAME User account credentials for Hive metastore database instance. Ensure that this user account has appropriate permissions. hive_user
Mandatory HIVE_DB_PASSWORD hive_pass
Mandatory OOZIE_DB_NAME Database for Oozie metastore. If using SQL Server, ensure that you create the database on the SQL Server instance. ooziedb
Mandatory OOZIE_DB_USERNAME User account credentials for Oozie metastore database instance. Ensure that this user account has appropriate permissions. oozie_user
Mandatory OOZIE_DB_PASSWORD oozie_pass
Mandatory The following snapshot illustrates a sample
clusterproperties.txt
file:#Log directory HDP_LOG_DIR=c:\hadoop\logs #Data directory HDP_DATA_DIR=c:\hdpdata #hosts NAMENODE_HOST=nn-host.acme.com SECONDARY_NAMENODE_HOST=sec-nn-host.acme.com RESOURCEMANAGER_HOST=resourcemgr-host.acme.com HIVE_SERVER_HOST=hive-host.acme.com OOZIE_SERVER_HOST=oozie-host.acme.com WEBHCAT_HOST=webhcat-host.acme.com SLAVE_HOSTS=slave-host.acme.com,slave1-host.acme.com, slave2-host.acme.com ZOOKEEPER_HOSTS=zookeeper-host.acme.com CLIENT_HOSTS=client-host.acme.com,client2-host.acme.com IS_TEZ=yes ENABLE_LZO=yes HBASE_MASTER=hbase-host.acme.com HBASE_REGIONSERVERS=hbase-host.acme.com,hbase2-host.acme.com FLUME_HOSTS=flume-host.acme.com FALCON_HOST=falcon-host.acme.com KNOX_HOST=knox-host.acme.com STORM_NIMBUS=storm-host.acme.com STORM_SUPERVISORS=stormsup-host.acme.com IS_PHOENIX=yes #Database host DB_FLAVOR=DERBY DB_HOSTNAME=hive-host.acme.com DB_PORT=1527 #Hive properties HIVE_DB_NAME=hive HIVE_DB_USERNAME=hive HIVE_DB_PASSWORD=hive3 #Oozie properties OOZIE_DB_NAME=oozie OOZIE_DB_USERNAME=oozie OOZIE_DB_PASSWORD=oozie