Recipe and cluster template parameters
The following supported parameters can be specified as variables in recipes or cluster templates by using mustache kind of templating with "{{{ }}}" syntax.
The parameter keys listed below follow the following general conventions:
{ }indicates that the parameter key has multiple supported values, which are provided in this documentation. For example{fileSystemType}can be one of the following: s3, adls_gen_2, or wasb.[index]indicates that the parameter includes an index value for examplesharedService.datalakeComponents.[index]can be "sharedService.datalakeComponents.[0]", "sharedService.datalakeComponents.[1]", and so on. There is no easy way to find out what the index will be, but you may still be able to use these parameters (for example by creating a condition to filter them).
Custom properties
Any custom property specified in the cluster template can be used as a recipe parameter. Refer to Custom properties documentation.
General
The general parameter group includes parameters related to general cluster
configuration.
| Parameter key | Description | Example key | Example value |
|---|---|---|---|
| general.gatewayInstanceMetadataPresented | Flag indicating if gateway instance metadata is present | general.gatewayInstanceMetadataPresented | true |
| general.instanceGroupsPresented | Flag indicating that instance groups are presented | general.instanceGroupsPresented | true |
| general.clusterName | Name of cluster | general.clusterName | testcluster |
| general.stackName | Name of stack | general.stackName | teststack |
| general.uuid | UUID of cluster | general.uuid | 9aab7fdb-8940-454b-bc0a-62f04bce6519 |
| general.executorType | Type of execution. Possible values: DEFAULT or CONTAINER | general.executorType | DEFAULT |
| general.clusterManagerIp | Cloudera Manager IP | general.clusterManagerIp | 127.0.0.1 |
| general.orchestratorType | Type of cluster orchestration. Possible values: HOST or CONTAINER | general.orchestratorType | HOST |
| general.containerExecutor | Flag indicating that the cluster is running containers | general.containerExecutor | false |
| general.nodeCount | Number of nodes | general.nodeCount | 5 |
| general.primaryGatewayInstanceDiscoveryFQDN | FQDN of primary gateway instance | general.primaryGatewayInstanceDiscoveryFQDN | ip-10-0-88-28.example.com |
| general.kafkaReplicationFactor | Number indicating the Kafka replication factor (3 or 1) | general.kafkaReplicationFactor | 1 |
Blueprint
The blueprint parameter group includes parameters related to cluster template
configuration.
| Parameter key | Description | Example key | Example value |
|---|---|---|---|
| blueprint.blueprintText | Blueprint text in JSON format | blueprint.blueprintText | |
| blueprint.version | Version of blueprint | blueprint.version | 3.2 |
| blueprint.type | Type of blueprint | blueprint.type | HDF |
Cloud storage
The fileSystemConfigs parameter group includes parameters related to cloud
storage configuration.
When forming the parameter keys, the {fileSystemType} should be replaced with
an actual cloud storage type such as "s3", "adls_gen_2", or "wasb".
| Parameter key | Description | Example key | Example value |
|---|---|---|---|
| File system common configurations | |||
| fileSystemConfigs.{fileSystemType}.storageContainer | Name of container in Azure storage account (Cloudbreak + stackId) | fileSystemConfigs.s3.storageContainer | cloudbreak123 |
| fileSystemConfigs.{fileSystemType}.type | Type of filesystem | fileSystemConfigs.s3.type | S3 |
| fileSystemConfigs.{fileSystemType}.defaultFs | Flag to indicate if the file system is the default filesystem | fileSystemConfigs.s3.defaultFs | false |
| fileSystemConfigs.{fileSystemType}.locations.[index].configFile | Configuration file used to configure the filesystem | fileSystemConfigs.s3.locations.[0].configFile | hbase-site |
| fileSystemConfigs.{fileSystemType}.locations.[index].property | Property key of filesystem path in defined config | fileSystemConfigs.s3.locations.[0].property | hbase.rootdir |
| fileSystemConfigs.{fileSystemType}.locations.[index].value | Value of filesystem path in defined config | fileSystemConfigs.s3.locations.[0].value | s3a://ahorvathtestranger/testrecipe2/apps/hbase/data |
| Amazon S3 configurations | |||
| fileSystemConfigs.s3.instanceProfile | ARN of related instance profile in AWS | fileSystemConfigs.s3.instanceProfile | arn:aws:iam::980678866538:instance-profile/CloudbreakRole |
| fileSystemConfigs.s3.storageContainer | Generated name (cloudbreak + stack id number) | fileSystemConfigs.s3.storageContainer | cloudbreak7941 |
| fileSystemConfigs.s3.locations.[index] | JSON which contains all the value below for one hadoop component | fileSystemConfigs.s3.locations.[0] | {configFile=zeppelin-site, property=zeppelin.notebook.dir, value=s3a://storagename/clustername/zeppelin/notebook} |
| fileSystemConfigs.s3.locations.[index].configFile | Hadoop component configuration file | fileSystemConfigs.s3.locations.[0].configFile | zeppelin-site |
| fileSystemConfigs.s3.locations.[index].property | Component property name | fileSystemConfigs.s3.locations.[0].property | zeppelin.notebook.dir |
| fileSystemConfigs.s3.locations.[index].value | Component property value | fileSystemConfigs.s3.locations.[0].value | s3a://storagename/clustername/zeppelin/notebook |
| WASB configurations | |||
| fileSystemConfigs.wasb.accountKey | Access key of the corresponding Azure storage account | fileSystemConfigs.wasb.accountKey | 81a9blll-bebf-436f-a333-f67b29880f1z |
| fileSystemConfigs.wasb.accountName | Name of the corresponding Azure storage account | fileSystemConfigs.wasb.accountName | teststorageaccount |
| fileSystemConfigs.wasb.secure | Flag indicating that the file system is secure | fileSystemConfigs.wasb.secure | true |
| fileSystemConfigs.wasb.resourceGroupName | Name of the corresponding Azure resource group | fileSystemConfigs.wasb.resourceGroupName | testresourcegroup |
| fileSystemConfigs.wasb.storageContainerName | Name of container in Azure storage account | fileSystemConfigs.wasb.storageContainerName | testcontainer |
| ADLS Gen2 configurations | |||
| fileSystemConfigs.adls_gen_2.accountName | Name of the corresponding Azure storage account | fileSystemConfigs.adls_gen_2.accountName | teststorageaccount |
| fileSystemConfigs.adls_gen_2.accountKey | Access key of the corresponding Azure storage account | fileSystemConfigs.adls_gen_2.accountKey | 81a9blll-bebf-436f-a333-f67b29880f1z |
| fileSystemConfigs.adls_gen_2.storageContainerName | Name of container in Azure storage account | fileSystemConfigs.adls_gen_2.storageContainerName | testcontainer |
External database
The rds parameter group includes parameters related to external database
configuration.
When forming the parameter keys, the {rdsType} should be replaced with the
actual database type such as "cloudera_manager", "beacon", "druid", "hive", "oozie", "ranger",
"superset", or some other user-defined type.
| Parameter key | Description | Example key | Example value |
|---|---|---|---|
| rds.{rdsType}.connectionURL | JDBC connection URL | rds.hive.connectionURL | Value is specified in the following format: jdbc:postgresql://host:port/database |
| rds.{rdsType}.connectionDriver | JDBC driver used for connection | rds.hive.connectionDriver | org.postgresql.Driver |
| rds.{rdsType}.connectionUserName | Username used for the JDBC connection | rds.hive.connectionUserName | testuser |
| rds.{rdsType}.connectionPassword | Password used for the JDBC connection | rds.hive.connectionPassword | TestPssword123 |
| rds.{rdsType}.databaseName | Target database of the JDBC connection | rds.hive.databaseName | myhivedb |
| rds.{rdsType}.host | Host of the JDBC connection | rds.hive.host | mydbhost |
| rds.{rdsType}.hostWithPortWithJdbc | Host of JDBC connection with port and JDBC prefix | rds.hive.hostWithPortWithJdbc | Value is specified in the following format: jdbc:postgresql://host:port |
| rds.{rdsType}.subprotocol | Sub-protocol from the JDBC URL | rds.hive.subprotocol | postgresql |
| rds.{rdsType}.connectionString | URL of JDBC the connection. In case of Ranger, this does not contain the port | rds.hive.connectionString | Value is specified in the following format: jdbc:postgresql://host:port/database |
| rds.{rdsType}.databaseVendor | Database vendor | rds.hive.databaseVendor | POSTGRES |
| rds.{rdsType}.withoutJDBCPrefix | URL of the JDBC connection without JDBC prefix | rds.hive.withoutJDBCPrefix | Value is specified in the following format: host:port/database |
Gateway
The gateway parameter group includes parameters related to Knox gateway
configuration.
| Parameter key | Description | Example key | Example value |
|---|---|---|---|
| gateway.gatewayType | Type of gateway. Possible values: CENTRAL/INDIVIDUAL | gateway.gatewayType | CENTRAL |
| gateway.path | Base path of gateway (typically this is the name of the cluster) | gateway.path | test |
| gateway.ssoType | Type of SSO. Possible values: SSO_PROVIDER/NONE | gateway.ssoType | SSO_PROVIDER |
| gateway.ssoConfigured | Flag indicating if SSO is provided | gateway.ssoConfigured | true |
| gateway.ssoProvider | Path to the SSO provider | gateway.ssoProvider | /test/sso/api/v1/websso |
| gateway.signKey | Base64 encoded signing key | gateway.signKey | |
| gateway.signPub | Signing certificate (x509 format) | gateway.signPub | |
| gateway.signCert | Public SSH key used for signing (standard public key format) | gateway.signCert |
Shared services
The sharedService parameter group includes parameters related to Data Lake
configuration.
| Parameter key | Description | Example key | Example value |
|---|---|---|---|
| sharedService.rangerAdminPassword | Admin password of the Ranger component | sharedService.rangerAdminPassword | Admin1234! |
| sharedService.attachedCluster | Flag indicating that the cluster is attached to a data lake cluster | sharedService.attachedCluster | true |
| sharedService.datalakeCluster | Flag indicating that the cluster is a data lake cluster | sharedService.datalakeCluster | true |
| sharedService.rangerAdminPort | Admin port of the Ranger component | sharedService.rangerAdminPort | 6080 |
| sharedService.datalakeClusterManagerIp | Cloudera Manager IP of data lake cluster | sharedService.datalakeClusterManagerIp | 127.0.0.1 |
| sharedService.datalakeClusterManagerFqdn | Cloudera Manager FQDN of data lake cluster (or the IP if FQDN cannot be found) | sharedService.datalakeClusterManagerFqdn | ip-10-0-88-28.example.com |
| sharedService.datalakeComponents.[index] | Data lake component list | sharedService.datalakeComponents.[0] | METRICS_COLLECTOR |
