Recipe parameters
The following supported parameters can be specified as variables in recipes by using mustache kind of templating with "{{{ }}}" syntax.
The parameter keys listed below follow the following general conventions:
{ }indicates that the parameter key has multiple supported values, which are provided in this documentation. For example{fileSystemType}can be one of the following: s3, adls, adls_gen_2, wasb, or gcs.[index]indicates that the parameter includes an index value for examplesharedService.datalakeComponents.[index]can be "sharedService.datalakeComponents.[0]", "sharedService.datalakeComponents.[1]", and so on. There is no easy way to find out what the index will be, but you may still be able to use these parameters (for example by creating a condition to filter them).
Custom properties
Any custom property specified in the blueprint can be used as a recipe parameter. Refer to Custom properties documentation.
General
The general parameter group includes parameters related to general cluster
configuration.
| Parameter key | Description | Example key | Example value |
|---|---|---|---|
| general.email | Email of the Cloudbreak user. | general.email | cloudbreak@hortonworks.com |
| general.gatewayInstanceMetadataPresented | Flag indicating if gateway instance metadata is present. | general.gatewayInstanceMetadataPresented | true |
| general.instanceGroupsPresented | Flag indicates that instance groups are presented. | general.instanceGroupsPresented | true |
| general.clusterName | Name of cluster. | general.clusterName | testcluster |
| general.stackName | Name of stack. | general.stackName | teststack |
| general.uuid | UUID of cluster. | general.uuid | 9aab7fdb-8940-454b-bc0a-62f04bce6519 |
| general.userName | Ambari username. | general.userName | admin |
| general.password | Ambari password. | general.password | admin1234 |
| general.executorType | Type of execution. Possible values: DEFAULT or CONTAINER. | general.executorType | DEFAULT |
| general.ambariIp | Ambari IP. | general.ambariIp | 127.0.0.1 |
| general.orchestratorType | Type of cluster orchestration. Possible values: HOST or CONTAINER. | general.orchestratorType | HOST |
| general.containerExecutor | Flag indicates that the cluster is running containers. | general.containerExecutor | false |
| general.nodeCount | Number of nodes. | general.nodeCount | 5 |
| general.primaryGatewayInstanceDiscoveryFQDN | FQDN of primary gateway instance. | general.primaryGatewayInstanceDiscoveryFQDN | ip-10-0-88-28.example.com |
| general.kafkaReplicationFactor | Number indicating the Kafka replication factor (3 or 1). | general.kafkaReplicationFactor | 1 |
Attached cluster
The following parameters are only used with clusters attached to a data lake.
| Parameter key | Description | Example key | Example value |
|---|---|---|---|
| REMOTE_CLUSTER_NAME | Name of data lake cluster to which the workload cluster is attached. | REMOTE_CLUSTER_NAME | testclusterdatalake |
| remoteClusterName | Name of data lake cluster to which the workload cluster is attached. | remoteClusterName | testclusterdatalake |
| remote.cluster.name | Name of data lake cluster to which the workload cluster is attached. | remote.cluster.name | testclusterdatalake |
| cluster_name | Cluster name. | cluster_name | testcluster |
| cluster.name | Cluster name. | cluster.name | testcluster |
| ranger.audit.solr.zookeepers | Ranger Audit URL. | ranger.audit.solr.zookeepers | ip-10-0-137-205.eu-west-1.compute.internal:2181/infra-solr |
| atlas.rest.address | Atlas component REST address. | atlas.rest.address | http://ip-10-0-137-205.eu-west-1.compute.internal:21000 |
| atlas.kafka.bootstrap.servers | Bootstrap server URL for Atlas Kafka. | atlas.kafka.bootstrap.servers | ip-10-0-137-205.eu-west-1.compute.internal:6667 |
| ranger_admin_username | Username of Ranger admin. | ranger_admin_username | amb_ranger_admin |
| policymgr_external_url | Load balancer URL of Ranger. | policymgr_external_url | http://ip-10-0-137-205.eu-west-1.compute.internal:6080 |
Blueprint
The bleuprint parameter group includes parameters related to blueprint
configuration.
| Parameter key | Description | Example key | Example value |
|---|---|---|---|
| blueprint.blueprintText | Blueprint text in JSON format. | blueprint.blueprintText | |
| blueprint.version | Version of blueprint. | blueprint.version | 3.2 |
| blueprint.type | Type of blueprint. | blueprint.type | HDF |
| blueprint.components.[index] | Components in the blueprint. | blueprint.components.[0] | TEZ_CLIENT |
Blueprint components
The components represented in a blueprint can be used as a recipe parameters. Possible uses of these values are to list the blueprint's components or check if this list contains a specific component.
| Parameter key | Description | Example key | Example value |
|---|---|---|---|
| components.[index] | The components represented in a blueprint. | components.[0] | TEZ_CLIENT |
Cloud storage
The fileSystemConfigs parameter group includes parameters related to cloud
storage configuration.
When forming the parameter keys, the {fileSystemType} should be replaced with
an actual cloud storage type such as "s3", "adls", "adls_gen_2", "wasb", or "gcs".
| Parameter key | Description | Example key | Example value |
|---|---|---|---|
| File system common configurations | |||
| fileSystemConfigs.{fileSystemType}.storageContainer | Name of container in Azure storage account (Cloudbreak + stackId). | fileSystemConfigs.s3.storageContainer | cloudbreak123 |
| fileSystemConfigs.{fileSystemType}.type | Type of filesystem. | fileSystemConfigs.s3.type | S3 |
| fileSystemConfigs.{fileSystemType}.defaultFs | Flag to indicate if the file system is the default filesystem. | fileSystemConfigs.s3.defaultFs | false |
| fileSystemConfigs.{fileSystemType}.locations.[index].configFile | Configuration file used to configure the filesystem. | fileSystemConfigs.s3.locations.[0].configFile | hbase-site |
| fileSystemConfigs.{fileSystemType}.locations.[index].property | Property key of filesystem path in defined config. | fileSystemConfigs.s3.locations.[0].property | hbase.rootdir |
| fileSystemConfigs.{fileSystemType}.locations.[index].value | Value of filesystem path in defined config. | fileSystemConfigs.s3.locations.[0].value | s3a://ahorvathtestranger/testrecipe2/apps/hbase/data |
| Amazon S3 configurations | |||
| fileSystemConfigs.s3.instanceProfile | ARN of related instance profile in AWS | fileSystemConfigs.s3.instanceProfile | arn:aws:iam::980678866538:instance-profile/CloudbreakRole |
| WASB configurations | |||
| fileSystemConfigs.wasb.accountKey | Access key of the corresponding Azure storage account. | fileSystemConfigs.wasb.accountKey | 81a9blll-bebf-436f-a333-f67b29880f1z |
| fileSystemConfigs.wasb.accountName | Name of the corresponding Azure storage account. | fileSystemConfigs.wasb.accountName | teststorageaccount |
| fileSystemConfigs.wasb.secure | Flag indicating that the file system is secure. | fileSystemConfigs.wasb.secure | true |
| fileSystemConfigs.wasb.resourceGroupName | Name of the corresponding Azure resource group. | fileSystemConfigs.wasb.resourceGroupName | testresourcegroup |
| fileSystemConfigs.wasb.storageContainerName | Name of container in Azure storage account. | fileSystemConfigs.wasb.storageContainerName | testcontainer |
| ADLS Gen1 configurations | |||
| fileSystemConfigs.adls.accountName | Name of the corresponding Azure storage account. | fileSystemConfigs.adls.accountName | teststorageaccount |
| fileSystemConfigs.adls.clientId | The corresponding Azure client ID. | fileSystemConfigs.adls.clientId | a9a9a88e-28dc-4851-ad3d-182a08c44666 |
| fileSystemConfigs.adls.tenantId | Tenant ID of Azure account. | fileSystemConfigs.adls.tenantId | d85131e4-1763-42d6-b9c7-b6bad64b3a51 |
| fileSystemConfigs.adls.resourceGroupName | Name of the corresponding Azure resource group. | fileSystemConfigs.adls.resourceGroupName | testresourcegroup |
| ADLS Gen2 configurations | |||
| fileSystemConfigs.adls_gen_2.accountName | Name of the corresponding Azure storage account. | fileSystemConfigs.adls_gen_2.accountName | teststorageaccount |
| fileSystemConfigs.adls_gen_2.accountKey | Access key of the corresponding Azure storage account. | fileSystemConfigs.adls_gen_2.accountKey | 81a9blll-bebf-436f-a333-f67b29880f1z |
| fileSystemConfigs.adls_gen_2.storageContainerName | Name of container in Azure storage account. | fileSystemConfigs.adls_gen_2.storageContainerName | testcontainer |
| GCS configurations | |||
| fileSystemConfigs.gcs.serviceAccountEmail | Email of the user's GCS account. | fileSystemConfigs.gcs.serviceAccountEmail | test@gmail.com |
External authentication source
The ldap parameter group includes parameters related to external
authentication source configuration.
| Parameter key | Description | Example key | Example value |
|---|---|---|---|
| ldap.bindDn | LDAP Bind DN. | ldap.bindDn | Admin2@AD.HWX.COM |
| ldap.bindPassword | Root Distinguished Name (Bind DN) password. | ldap.bindPassword | Admin1234 |
| ldap.directoryType | Directory type. Possible values: LDAP or ACTIVE_DIRECTORY. | ldap.directoryType | ACTIVE_DIRECTORY |
| ldap.userSearchBase | LDAP user search base. This defines the location in the directory from which the LDAP search begins. | ldap.userSearchBase | OU=Users,OU=AD,DC=AD,DC=HWX,DC=COM |
| ldap.userNameAttribute | The attribute for which to conduct a search on the user base. | ldap.userNameAttribute | sAMAccountName |
| ldap.userObjectClass | Directory object class for users. | ldap.userObjectClass | person |
| ldap.groupSearchBase | LDAP group search base. This defines the location in the directory from which the LDAP search begins. | ldap.groupSearchBase | OU=Users,OU=AD,DC=AD,DC=HWX,DC=COM |
| ldap.groupNameAttribute | The attribute for which to conduct search on groups. | ldap.groupNameAttribute | cn |
| ldap.groupObjectClass | The directory object class for groups. | ldap.groupObjectClass | group |
| ldap.groupMemberAttribute | The attribute on the group object class that represents members. | ldap.groupMemberAttribute | member |
| ldap.domain | Domain of LDAP. | ldap.domain | ad.hwx.com |
| ldap.protocol | Protocol used by the LDAP: LDAP or LDAPS. | ldap.protocol | ldap |
| ldap.adminGroup | Name of the admin group. | ldap.adminGroup | cloudbreak |
| ldap.userDnPattern | LDAP User DN Pattern, which is used to bind an LDAP user. | ldap.userDnPattern | CN={0},OU=Users,OU=AD,DC=AD,DC=HWX,DC=COM |
| ldap.connectionURL | Full connection URL of the authentication source. | ldap.connectionURL | ldap://hwxmsad-bd87e95aa9775a71.elb.eu-west-1.amazonaws.com:389 |
| ldap.host | Host of the authentication source (without protocol). | ldap.host | hwxmsad-bd87e95aa9775a71.elb.eu-west-1.amazonaws.com |
| ldap.port | Port of the authentication source. | ldap.port | 389 |
External database
The rds parameter group includes parameters related to external database
configuration.
When forming the parameter keys, the {rdsType} should be replaced with the
actual database type such as "ambari", "beacon", "druid", "hive", "oozie", "ranger", "superset",
or some other user-defined type.
| Parameter key | Description | Example key | Example value |
|---|---|---|---|
| rds.{rdsType}.connectionURL | JDBC connection URL. | rds.hive.connectionURL | Value is specified in the following format: jdbc:postgresql://host:port/database |
| rds.{rdsType}.connectionDriver | JDBC driver used for connection. | rds.hive.connectionDriver | org.postgresql.Driver |
| rds.{rdsType}.connectionUserName | Username used for the JDBC connection. | rds.hive.connectionUserName | testuser |
| rds.{rdsType}.connectionPassword | Password used for the JDBC connection. | rds.hive.connectionPassword | TestPssword123 |
| rds.{rdsType}.databaseName | Target database of the JDBC connection. | rds.hive.databaseName | myhivedb |
| rds.{rdsType}.host | Host of the JDBC connection. | rds.hive.host | mydbhost |
| rds.{rdsType}.hostWithPortWithJdbc | Host of JDBC connection with port and JDBC prefix. | rds.hive.hostWithPortWithJdbc | Value is specified in the following format: jdbc:postgresql://host:port |
| rds.{rdsType}.subprotocol | Sub-protocol from the JDBC URL. | rds.hive.subprotocol | postgresql |
| rds.{rdsType}.connectionString | URL of JDBC the connection. In case of Ranger, this does not contain the port. | rds.hive.connectionString | Value is specified in the following format: jdbc:postgresql://host:port/database |
| rds.{rdsType}.databaseVendor | Database vendor. | rds.hive.databaseVendor | POSTGRES |
| rds.{rdsType}.withoutJDBCPrefix | URL of the JDBC connection without JDBC prefix. | rds.hive.withoutJDBCPrefix | Value is specified in the following format: host:port/database |
Gateway
The gateway parameter group includes parameters related to gateway
configuration.
| Parameter key | Description | Example key | Example value |
|---|---|---|---|
| gateway.gatewayType | Type of gateway. Possible values: CENTRAL/INDIVIDUAL. | gateway.gatewayType | CENTRAL |
| gateway.path | Base path of gateway (typically this is the name of the cluster). | gateway.path | test |
| gateway.ssoType | Type of SSO. Possible values: SSO_PROVIDER/NONE. | gateway.ssoType | SSO_PROVIDER |
| gateway.ssoConfigured | Flag indicating if SSO is provided. | gateway.ssoConfigured | true |
| gateway.ssoProvider | Path to the SSO provider. | gateway.ssoProvider | /test/sso/api/v1/websso |
| gateway.signKey | Base64 encoded signing key. | gateway.signKey | |
| gateway.signPub | Signing certificate (x509 format). | gateway.signPub | |
| gateway.signCert | Public SSH key used for signing (standard public key format). | gateway.signCert | |
| gateway.gatewayTopologies.{topologyName} | List of exposed services in a specific topology. Value is specified in JSON format. | gateway.gatewayTopologies.dp-proxy | {"services":["AMBARI","SPARK2HISTORYUI","LIVYSERVER"]} |
HDF
The hdf parameter group includes parameters related to HDF configuration.
| Parameter key | Description | Example key | Example value |
|---|---|---|---|
| hdf.nodeEntities | NiFi node entities content (needed in nifi-ambari-ssl-config configuration). | hdf.nodeEntities | <property name="Node Identity 1">CN=ip-10-0-85-196.eu-west-1.compute.internal, OU=NIFI</property> |
| hdf.registryNodeEntities | NiFi registry node entities content (needed in nifi-registry-ambari-ssl-config configuration). | hdf.registryNodeEntities | <property name="NiFi Identity 1">CN=ip-10-0-85-196.eu-west-1.compute.internal, OU=NIFI</property> |
| hdf.nodeUserEntities | NiFi node user entities content. | hdf.nodeUserEntities | <property name="Initial User Identity 1">CN=ip-10-0-85-196.eu-west-1.compute.internal, OU=NIFI</property> |
| hdf.proxyHosts | List of proxy hosts (needed in nifi-properties configuration). | hdf.proxyHosts | 34.244.122.193:9091 |
Shared services
The sharedService parameter group includes parameters related to data lake
configuration.
| Parameter key | Description | Example key | Example value |
|---|---|---|---|
| sharedService.rangerAdminPassword | Admin password of the Ranger component. | sharedService.rangerAdminPassword | Admin1234 |
| sharedService.attachedCluster | Flag indicating that the cluster is attached to a data lake cluster. | sharedService.attachedCluster | true |
| sharedService.datalakeCluster | Flag indicating that the cluster is a data lake cluster. | sharedService.datalakeCluster | true |
| sharedService.rangerAdminPort | Admin port of the Ranger component. | sharedService.rangerAdminPort | 6080 |
| sharedService.datalakeAmbariIp | Ambari IP of data lake cluster. | sharedService.datalakeAmbariIp | 127.0.0.1 |
| sharedService.datalakeAmbariFqdn | Ambari FQDN of data lake cluster (or the IP if FQDN cannot be found). | sharedService.datalakeAmbariFqdn | ip-10-0-88-28.example.com |
| sharedService.datalakeComponents.[index] | Data lake component list. | sharedService.datalakeComponents.[0] | METRICS_COLLECTOR |
Stack Version
| Parameter key | Description | Example key | Example value |
|---|---|---|---|
| stack_version | Stack (HDP or HDF) version. | 3.2 |