Advanced Cluster Options
Also available as:
PDF

Recipe parameters

The following supported parameters can be specified as variables in recipes by using mustache kind of templating with "{{{ }}}" syntax.

The parameter keys listed below follow the following general conventions:

  • { } indicates that the parameter key has multiple supported values, which are provided in this documentation. For example {fileSystemType} can be one of the following: s3, adls, adls_gen_2, wasb, or gcs.
  • [index] indicates that the parameter includes an index value for example sharedService.datalakeComponents.[index] can be "sharedService.datalakeComponents.[0]", "sharedService.datalakeComponents.[1]", and so on. There is no easy way to find out what the index will be, but you may still be able to use these parameters (for example by creating a condition to filter them).

Custom properties

Any custom property specified in the blueprint can be used as a recipe parameter. Refer to Custom properties documentation.

General

The general parameter group includes parameters related to general cluster configuration.

Parameter key Description Example key Example value
general.email Email of the Cloudbreak user. general.email cloudbreak@hortonworks.com
general.gatewayInstanceMetadataPresented Flag indicating if gateway instance metadata is present. general.gatewayInstanceMetadataPresented true
general.instanceGroupsPresented Flag indicates that instance groups are presented. general.instanceGroupsPresented true
general.clusterName Name of cluster. general.clusterName testcluster
general.stackName Name of stack. general.stackName teststack
general.uuid UUID of cluster. general.uuid 9aab7fdb-8940-454b-bc0a-62f04bce6519
general.userName Ambari username. general.userName admin
general.password Ambari password. general.password admin1234
general.executorType Type of execution. Possible values: DEFAULT or CONTAINER. general.executorType DEFAULT
general.ambariIp Ambari IP. general.ambariIp 127.0.0.1
general.orchestratorType Type of cluster orchestration. Possible values: HOST or CONTAINER. general.orchestratorType HOST
general.containerExecutor Flag indicates that the cluster is running containers. general.containerExecutor false
general.nodeCount Number of nodes. general.nodeCount 5
general.primaryGatewayInstanceDiscoveryFQDN FQDN of primary gateway instance. general.primaryGatewayInstanceDiscoveryFQDN ip-10-0-88-28.example.com
general.kafkaReplicationFactor Number indicating the Kafka replication factor (3 or 1). general.kafkaReplicationFactor 1

Attached cluster

The following parameters are only used with clusters attached to a data lake.

Parameter key Description Example key Example value
REMOTE_CLUSTER_NAME Name of data lake cluster to which the workload cluster is attached. REMOTE_CLUSTER_NAME testclusterdatalake
remoteClusterName Name of data lake cluster to which the workload cluster is attached. remoteClusterName testclusterdatalake
remote.cluster.name Name of data lake cluster to which the workload cluster is attached. remote.cluster.name testclusterdatalake
cluster_name Cluster name. cluster_name testcluster
cluster.name Cluster name. cluster.name testcluster
ranger.audit.solr.zookeepers Ranger Audit URL. ranger.audit.solr.zookeepers ip-10-0-137-205.eu-west-1.compute.internal:2181/infra-solr
atlas.rest.address Atlas component REST address. atlas.rest.address http://ip-10-0-137-205.eu-west-1.compute.internal:21000
atlas.kafka.bootstrap.servers Bootstrap server URL for Atlas Kafka. atlas.kafka.bootstrap.servers ip-10-0-137-205.eu-west-1.compute.internal:6667
ranger_admin_username Username of Ranger admin. ranger_admin_username amb_ranger_admin
policymgr_external_url Load balancer URL of Ranger. policymgr_external_url http://ip-10-0-137-205.eu-west-1.compute.internal:6080

Blueprint

The bleuprint parameter group includes parameters related to blueprint configuration.

Parameter key Description Example key Example value
blueprint.blueprintText Blueprint text in JSON format. blueprint.blueprintText
blueprint.version Version of blueprint. blueprint.version 3.2
blueprint.type Type of blueprint. blueprint.type HDF
blueprint.components.[index] Components in the blueprint. blueprint.components.[0] TEZ_CLIENT

Blueprint components

The components represented in a blueprint can be used as a recipe parameters. Possible uses of these values are to list the blueprint's components or check if this list contains a specific component.

Parameter key Description Example key Example value
components.[index] The components represented in a blueprint. components.[0] TEZ_CLIENT

Cloud storage

The fileSystemConfigs parameter group includes parameters related to cloud storage configuration.

When forming the parameter keys, the {fileSystemType} should be replaced with an actual cloud storage type such as "s3", "adls", "adls_gen_2", "wasb", or "gcs".

Parameter key Description Example key Example value
File system common configurations
fileSystemConfigs.{fileSystemType}.storageContainer Name of container in Azure storage account (Cloudbreak + stackId). fileSystemConfigs.s3.storageContainer cloudbreak123
fileSystemConfigs.{fileSystemType}.type Type of filesystem. fileSystemConfigs.s3.type S3
fileSystemConfigs.{fileSystemType}.defaultFs Flag to indicate if the file system is the default filesystem. fileSystemConfigs.s3.defaultFs false
fileSystemConfigs.{fileSystemType}.locations.[index].configFile Configuration file used to configure the filesystem. fileSystemConfigs.s3.locations.[0].configFile hbase-site
fileSystemConfigs.{fileSystemType}.locations.[index].property Property key of filesystem path in defined config. fileSystemConfigs.s3.locations.[0].property hbase.rootdir
fileSystemConfigs.{fileSystemType}.locations.[index].value Value of filesystem path in defined config. fileSystemConfigs.s3.locations.[0].value s3a://ahorvathtestranger/testrecipe2/apps/hbase/data
Amazon S3 configurations
fileSystemConfigs.s3.instanceProfile ARN of related instance profile in AWS fileSystemConfigs.s3.instanceProfile arn:aws:iam::980678866538:instance-profile/CloudbreakRole
WASB configurations
fileSystemConfigs.wasb.accountKey Access key of the corresponding Azure storage account. fileSystemConfigs.wasb.accountKey 81a9blll-bebf-436f-a333-f67b29880f1z
fileSystemConfigs.wasb.accountName Name of the corresponding Azure storage account. fileSystemConfigs.wasb.accountName teststorageaccount
fileSystemConfigs.wasb.secure Flag indicating that the file system is secure. fileSystemConfigs.wasb.secure true
fileSystemConfigs.wasb.resourceGroupName Name of the corresponding Azure resource group. fileSystemConfigs.wasb.resourceGroupName testresourcegroup
fileSystemConfigs.wasb.storageContainerName Name of container in Azure storage account. fileSystemConfigs.wasb.storageContainerName testcontainer
ADLS Gen1 configurations
fileSystemConfigs.adls.accountName Name of the corresponding Azure storage account. fileSystemConfigs.adls.accountName teststorageaccount
fileSystemConfigs.adls.clientId The corresponding Azure client ID. fileSystemConfigs.adls.clientId a9a9a88e-28dc-4851-ad3d-182a08c44666
fileSystemConfigs.adls.tenantId Tenant ID of Azure account. fileSystemConfigs.adls.tenantId d85131e4-1763-42d6-b9c7-b6bad64b3a51
fileSystemConfigs.adls.resourceGroupName Name of the corresponding Azure resource group. fileSystemConfigs.adls.resourceGroupName testresourcegroup
ADLS Gen2 configurations
fileSystemConfigs.adls_gen_2.accountName Name of the corresponding Azure storage account. fileSystemConfigs.adls_gen_2.accountName teststorageaccount
fileSystemConfigs.adls_gen_2.accountKey Access key of the corresponding Azure storage account. fileSystemConfigs.adls_gen_2.accountKey 81a9blll-bebf-436f-a333-f67b29880f1z
fileSystemConfigs.adls_gen_2.storageContainerName Name of container in Azure storage account. fileSystemConfigs.adls_gen_2.storageContainerName testcontainer
GCS configurations
fileSystemConfigs.gcs.serviceAccountEmail Email of the user's GCS account. fileSystemConfigs.gcs.serviceAccountEmail test@gmail.com

External authentication source

The ldap parameter group includes parameters related to external authentication source configuration.

Parameter key Description Example key Example value
ldap.bindDn LDAP Bind DN. ldap.bindDn Admin2@AD.HWX.COM
ldap.bindPassword Root Distinguished Name (Bind DN) password. ldap.bindPassword Admin1234
ldap.directoryType Directory type. Possible values: LDAP or ACTIVE_DIRECTORY. ldap.directoryType ACTIVE_DIRECTORY
ldap.userSearchBase LDAP user search base. This defines the location in the directory from which the LDAP search begins. ldap.userSearchBase OU=Users,OU=AD,DC=AD,DC=HWX,DC=COM
ldap.userNameAttribute The attribute for which to conduct a search on the user base. ldap.userNameAttribute sAMAccountName
ldap.userObjectClass Directory object class for users. ldap.userObjectClass person
ldap.groupSearchBase LDAP group search base. This defines the location in the directory from which the LDAP search begins. ldap.groupSearchBase OU=Users,OU=AD,DC=AD,DC=HWX,DC=COM
ldap.groupNameAttribute The attribute for which to conduct search on groups. ldap.groupNameAttribute cn
ldap.groupObjectClass The directory object class for groups. ldap.groupObjectClass group
ldap.groupMemberAttribute The attribute on the group object class that represents members. ldap.groupMemberAttribute member
ldap.domain Domain of LDAP. ldap.domain ad.hwx.com
ldap.protocol Protocol used by the LDAP: LDAP or LDAPS. ldap.protocol ldap
ldap.adminGroup Name of the admin group. ldap.adminGroup cloudbreak
ldap.userDnPattern LDAP User DN Pattern, which is used to bind an LDAP user. ldap.userDnPattern CN={0},OU=Users,OU=AD,DC=AD,DC=HWX,DC=COM
ldap.connectionURL Full connection URL of the authentication source. ldap.connectionURL ldap://hwxmsad-bd87e95aa9775a71.elb.eu-west-1.amazonaws.com:389
ldap.host Host of the authentication source (without protocol). ldap.host hwxmsad-bd87e95aa9775a71.elb.eu-west-1.amazonaws.com
ldap.port Port of the authentication source. ldap.port 389

External database

The rds parameter group includes parameters related to external database configuration.

When forming the parameter keys, the {rdsType} should be replaced with the actual database type such as "ambari", "beacon", "druid", "hive", "oozie", "ranger", "superset", or some other user-defined type.

Parameter key Description Example key Example value
rds.{rdsType}.connectionURL JDBC connection URL. rds.hive.connectionURL Value is specified in the following format: jdbc:postgresql://host:port/database
rds.{rdsType}.connectionDriver JDBC driver used for connection. rds.hive.connectionDriver org.postgresql.Driver
rds.{rdsType}.connectionUserName Username used for the JDBC connection. rds.hive.connectionUserName testuser
rds.{rdsType}.connectionPassword Password used for the JDBC connection. rds.hive.connectionPassword TestPssword123
rds.{rdsType}.databaseName Target database of the JDBC connection. rds.hive.databaseName myhivedb
rds.{rdsType}.host Host of the JDBC connection. rds.hive.host mydbhost
rds.{rdsType}.hostWithPortWithJdbc Host of JDBC connection with port and JDBC prefix. rds.hive.hostWithPortWithJdbc Value is specified in the following format: jdbc:postgresql://host:port
rds.{rdsType}.subprotocol Sub-protocol from the JDBC URL. rds.hive.subprotocol postgresql
rds.{rdsType}.connectionString URL of JDBC the connection. In case of Ranger, this does not contain the port. rds.hive.connectionString Value is specified in the following format: jdbc:postgresql://host:port/database
rds.{rdsType}.databaseVendor Database vendor. rds.hive.databaseVendor POSTGRES
rds.{rdsType}.withoutJDBCPrefix URL of the JDBC connection without JDBC prefix. rds.hive.withoutJDBCPrefix Value is specified in the following format: host:port/database

Gateway

The gateway parameter group includes parameters related to gateway configuration.

Parameter key Description Example key Example value
gateway.gatewayType Type of gateway. Possible values: CENTRAL/INDIVIDUAL. gateway.gatewayType CENTRAL
gateway.path Base path of gateway (typically this is the name of the cluster). gateway.path test
gateway.ssoType Type of SSO. Possible values: SSO_PROVIDER/NONE. gateway.ssoType SSO_PROVIDER
gateway.ssoConfigured Flag indicating if SSO is provided. gateway.ssoConfigured true
gateway.ssoProvider Path to the SSO provider. gateway.ssoProvider /test/sso/api/v1/websso
gateway.signKey Base64 encoded signing key. gateway.signKey
gateway.signPub Signing certificate (x509 format). gateway.signPub
gateway.signCert Public SSH key used for signing (standard public key format). gateway.signCert
gateway.gatewayTopologies.{topologyName} List of exposed services in a specific topology. Value is specified in JSON format. gateway.gatewayTopologies.dp-proxy {"services":["AMBARI","SPARK2HISTORYUI","LIVYSERVER"]}

HDF

The hdf parameter group includes parameters related to HDF configuration.

Parameter key Description Example key Example value
hdf.nodeEntities NiFi node entities content (needed in nifi-ambari-ssl-config configuration). hdf.nodeEntities <property name="Node Identity 1">CN=ip-10-0-85-196.eu-west-1.compute.internal, OU=NIFI</property>
hdf.registryNodeEntities NiFi registry node entities content (needed in nifi-registry-ambari-ssl-config configuration). hdf.registryNodeEntities <property name="NiFi Identity 1">CN=ip-10-0-85-196.eu-west-1.compute.internal, OU=NIFI</property>
hdf.nodeUserEntities NiFi node user entities content. hdf.nodeUserEntities <property name="Initial User Identity 1">CN=ip-10-0-85-196.eu-west-1.compute.internal, OU=NIFI</property>
hdf.proxyHosts List of proxy hosts (needed in nifi-properties configuration). hdf.proxyHosts 34.244.122.193:9091

Shared services

The sharedService parameter group includes parameters related to data lake configuration.

Parameter key Description Example key Example value
sharedService.rangerAdminPassword Admin password of the Ranger component. sharedService.rangerAdminPassword Admin1234
sharedService.attachedCluster Flag indicating that the cluster is attached to a data lake cluster. sharedService.attachedCluster true
sharedService.datalakeCluster Flag indicating that the cluster is a data lake cluster. sharedService.datalakeCluster true
sharedService.rangerAdminPort Admin port of the Ranger component. sharedService.rangerAdminPort 6080
sharedService.datalakeAmbariIp Ambari IP of data lake cluster. sharedService.datalakeAmbariIp 127.0.0.1
sharedService.datalakeAmbariFqdn Ambari FQDN of data lake cluster (or the IP if FQDN cannot be found). sharedService.datalakeAmbariFqdn ip-10-0-88-28.example.com
sharedService.datalakeComponents.[index] Data lake component list. sharedService.datalakeComponents.[0] METRICS_COLLECTOR

Stack Version

Parameter key Description Example key Example value
stack_version Stack (HDP or HDF) version. 3.2