Setting custom properties

Data Hub allows you to set custom properties dynamically for each cluster during cluster creation.

To set custom properties, do the following:

1. Prepare a cluster template that includes the appropriate mustache templating for setting custom properties. Use the "{{{ }}}" syntax. For example, this is a cluster template where the values for the Ranger plugin service name/repository and the HDFS directory used for the WAL will be dynamically replaced and set during cluster creation:

   {
     "cdhVersion": "7.2.8",
     "displayName": "test-cluster",
     "services": [
       {
         "refName": "zookeeper",
         "serviceType": "ZOOKEEPER"
       },
       {
         "refName": "hdfs",
         "serviceType": "HDFS",
         "roleConfigGroups": [
           {
             "refName": "hdfs-NAMENODE-BASE",
             "roleType": "NAMENODE",
             "base": true
           },
           {
             "refName": "hdfs-SECONDARYNAMENODE-BASE",
             "roleType": "SECONDARYNAMENODE",
             "base": true
           },
           {
             "refName": "hdfs-DATANODE-BASE",
             "roleType": "DATANODE",
             "base": true
           },
           {
             "refName": "hdfs-FAILOVERCONTROLLER-BASE",
             "roleType": "FAILOVERCONTROLLER",
             "base": true
           },
           {
             "refName": "hdfs-JOURNALNODE-BASE",
             "roleType": "JOURNALNODE",
             "base": true
           },
           {
             "refName": "hdfs-GATEWAY-BASE",
             "roleType": "GATEWAY",
             "base": true
           }
         ]
       },
       {
         "refName": "hbase",
         "serviceType": "HBASE",
         "serviceConfigs": [
           {
             "name": "hbase_wal_dir",
             "value": "{{{wal.dir}}}",
             "autoConfig": false
           },
           {
             "name": "ranger_plugin_service_name",
             "value": "{{{test.cluster.rangersvc_name}}}",
             "autoConfig": false
           }
         ],
         "roleConfigGroups": [
           {
             "refName": "hbase-MASTER-BASE",
             "roleType": "MASTER",
             "base": true
           },
           {
             "refName": "hbase-GATEWAY-BASE",
             "roleType": "GATEWAY",
             "base": true
           },
           {
             "refName": "hbase-REGIONSERVER-BASE",
             "roleType": "REGIONSERVER",
             "base": true
           },
           {
             "refName": "hbase-HBASETHRIFTSERVER-BASE",
             "roleType": "HBASETHRIFTSERVER",
             "base": true
           },
           {
             "refName": "hbase-HBASERESTSERVER-BASE",
             "roleType": "HBASERESTSERVER",
             "base": true
           }
         ]
       },
       {
         "refName": "phoenix",
         "serviceType": "PHOENIX",
         "roleConfigGroups": [
           {
             "refName": "phoenix-PHOENIX_QUERY_SERVER-BASE",
             "roleType": "PHOENIX_QUERY_SERVER",
             "base": true
           }
         ]
       },
       {
         "refName": "yarn",
         "serviceType": "YARN",
         "serviceConfigs": [],
         "roleConfigGroups": [
           {
             "refName": "yarn-RESOURCEMANAGER-BASE",
             "roleType": "RESOURCEMANAGER",
             "base": true
           },
           {
             "refName": "yarn-NODEMANAGER-BASE",
             "roleType": "NODEMANAGER",
             "base": true
           },
           {
             "refName": "yarn-JOBHISTORY-BASE",
             "roleType": "JOBHISTORY",
             "base": true
           },
           {
             "refName": "yarn-GATEWAY-BASE",
             "roleType": "GATEWAY",
             "base": true
           }
         ]
       },
       {
         "refName": "hue",
         "serviceType": "HUE",
         "roleConfigGroups": [
           {
             "refName": "hue-HUE_SERVER-BASE",
             "roleType": "HUE_SERVER",
             "base": true
           },
           {
             "refName": "hue-HUE_LOAD_BALANCER-BASE",
             "roleType": "HUE_LOAD_BALANCER",
             "base": true
           }
         ]
       }
     ],
     "hostTemplates": [
       {
         "refName": "gateway",
         "roleConfigGroupsRefNames": [
           "hbase-GATEWAY-BASE",
           "hdfs-GATEWAY-BASE",
           "yarn-GATEWAY-BASE",
           "hue-HUE_SERVER-BASE",
           "hue-HUE_LOAD_BALANCER-BASE"
         ]
       },
       {
         "refName": "master",
         "roleConfigGroupsRefNames": [
           "hbase-GATEWAY-BASE",
           "hbase-MASTER-BASE",
           "hdfs-FAILOVERCONTROLLER-BASE",
           "hdfs-GATEWAY-BASE",
           "hdfs-JOURNALNODE-BASE",
           "hdfs-NAMENODE-BASE",
           "yarn-GATEWAY-BASE",
           "zookeeper-SERVER-BASE"
         ]
       },
       {
         "refName": "leader",
         "roleConfigGroupsRefNames": [
           "hbase-GATEWAY-BASE",
           "hbase-HBASETHRIFTSERVER-BASE",
           "hbase-HBASERESTSERVER-BASE",
           "hdfs-GATEWAY-BASE",
           "hdfs-JOURNALNODE-BASE",
           "yarn-JOBHISTORY-BASE",
           "yarn-RESOURCEMANAGER-BASE",
           "yarn-GATEWAY-BASE",
           "zookeeper-SERVER-BASE"
         ]
       },
       {
         "refName": "worker",
         "roleConfigGroupsRefNames": [
           "hbase-GATEWAY-BASE",
           "hbase-REGIONSERVER-BASE",
           "hdfs-DATANODE-BASE",
           "hdfs-GATEWAY-BASE",
           "phoenix-PHOENIX_QUERY_SERVER-BASE",
           "yarn-GATEWAY-BASE",
           "yarn-NODEMANAGER-BASE"
         ]
       }
     ]
   }

   If you are customizing Cloudera Manager properties, refer to the Cloudera Manager Configuration Properties Reference. CDP-specific custom properties can be found in Recipe and cluster template parameters.
2. Prepare a JSON file that includes the custom property names and values that you want to set during cluster creation. In the template above, the custom property names that you provided ("wal.dir" and "test.cluster.rangersvc_name") will be replaced with the values that you give in this file. You can format the custom properties JSON in two different ways:

   {
     "wal.dir": "/hadoopfs/fs1",
     "test.cluster.rangersvc_name": "test-cluster-ranger"
   }

   or

   {
     "wal": {
       "dir": "/hadoopfs/fs1"
     },
     "test": {
       "cluster": {
         "rangersvc_name": "test-cluster-ranger"
       }
     }
   }

3. Register the cluster template that you created in the Management Console. Click Shared Resources > Cluster Templates > Create Cluster Template. For more information on registering custom templates, see Upload a cluster template.
4. Once you have the template and the custom properties JSON ready, you can set the custom properties on a per-cluster basis by using the options available on the Cluster Extensions page in the create cluster wizard:
   1. From the left-side navigation menu, click Data Hub Clusters > Create Data Hub and then the Custom radio button.
   2. Under Cluster Template, select the cluster template that you prepared and registered.
   3. Enter a name for the cluster and then click Advanced Options > Cluster Extensions.
   4. In the Custom Properties box, paste the custom properties JSON file that you prepared. During the cluster creation phase, the properties will be set to the values that you provided.

      

   5. Click Provision Cluster.