Deploying a flow definition using the CLI
Deploy a flow definition to run NiFi flows as flow deployments in Cloudera DataFlow. To do this, launch the CDP CLI and specify your
environment, parameters, sizing, and KPIs.
- You have installed CDP CLI.
- Run
cdp df list-services
to get theservice-crn
value. - Run
cdp df list-flows
to get theflow-version-crn
value.
To deploy a flow, enter:
cdp df create-deployment --service-crn [***service-crn-value***] --flow-version-crn [***flow-version-crn-value***] --deployment-name [***flow-deployment-name***] [--cluster-size-name [EXTRA_SMALL|SMALL|MEDIUM|LARGE]] [--static-node-count [***value***]] [--auto-scaling-enabled | --no-auto-scaling-enabled] [--auto-scale-min-nodes [***number-min-nodes***]] [--auto-scale-max-nodes [***number-max-nodes***]] [--cfm-nifi-version [***nifi-version***]] [--auto-start-flow | --no-auto-start-flow] [--parameter-groups [***json-file-locatione***]] [--kpis [***json-file-locatione***]] [---node-storage-profile-name [STANDARD_AWS|STANDARD_AZURE|PERFORMANCE_AWS|PERFORMANCE_AZURE]]
Where:
- --service-crn
- Specifies the
service-crn
value you obtained when completing the prerequisites. - --flow-version-crn
- Specifies the
flow-version-crn
value you obtained when completing the prerequisites. - --deployment-name
- Specifies a unique name for your flow deployment.
- [--cluster-size-name]
- Specifies the cluster size. Valid values are:
- EXTRA_SMALL
- SMALL
- MEDIUM
- LARGE
- [--static-node-count]
- Specifies the number of NiFi nodes when autoscaling is not enabled. You can select between 1 and 32 nodes. The default value is 1.
- [--auto-scaling-enabled | --no-auto-scaling-enabled]
- Specifies whether you want to enable autoscaling. The default is to
disable autoscaling.
- [--auto-scale-min-nodes]
- Specifies the minimum nodes when you have autoscaling enabled. If you have autoscaling enabled, this parameter is required.
- [--auto-scale-max-nodes]
- Specifies the maximum nodes when autoscaling is enabled. If you have autoscaling enabled, this parameter is required.
- [--cfm-nifi-version]
- Specifies the NiFi runtime version. The default is the latest version.
- [--auto-start-flow | --no-auto-start-flow]
- Specifies whether you want to automatically start your flow once it has been deployed. The default is to enable the automatic start.
- [--parameter-groups]
- Specifies the location of the parameter group JSON file, if you are using one for this flow deployment.
- [--kpis]
- Specifies the location of the KPIs JSON file, if you are providing KPIs for this flow.
- --node-storage-profile-name
- Specifies the storage profile. Valid values are:
- STANDARD_AWS
- STANDARD_AZURE
- PERFORMANCE_AWS
- PERFORMANCE_AZURE
Example parameter group file:
[ { "name": "kafka-filter-to-kafka", "parameters": [ { "name": "CDP Workload User", "assetReferences": [], "value": "srv_nifi-machine-ingest" }, { "name": "CDP Workload User Password", "assetReferences": [], "value": "<<CDP_MISSING_SENSITIVE_VALUE>>" }, { "name": "CSV Delimiter", "assetReferences": [], "value": "," }, { "name": "Data Input Format", "assetReferences": [], "value": "CSV" }, { "name": "Data Output Format", "assetReferences": [], "value": "JSON" }, { "name": "Filter Rule", "assetReferences": [], "value": "SELECT * FROM FLOWFILE" }, { "name": "Kafka Broker Endpoint", "assetReferences": [], "value": "streams-messaging-broker0.pm-sandb.a465-9q4k.cloudera.site:9093" }, { "name": "Kafka Consumer Group ID", "assetReferences": [], "value": "cdf" }, { "name": "Kafka Destination Topic", "assetReferences": [], "value": "MachineDataJSON" }, { "name": "Kafka Producer ID", "assetReferences": [], "value": "cdf" }, { "name": "Kafka Source Topic", "assetReferences": [], "value": "MachineDataCSV" }, { "name": "Schema Name", "assetReferences": [], "value": "SensorReading" }, { "name": "Schema Registry Hostname", "assetReferences": [], "value": "streams-messaging-master0.pm-sandb.a465-9q4k.cloudera.site" } ] } ]
Example KPI file:
[ { "metricId": "rateBytesReceived", "alert": { "thresholdLessThan": { "unitId": "kilobytesPerSecond", "value": 150 }, "frequencyTolerance": { "unit": { "id": "MINUTES" }, "value": 1 } } }, { "metricId": "processorAmountBytesSent", "alert": {}, "componentId": "a7f7df1c-a32d-3c25-9b09-e1d1036dcc04;a33a1b48-005b-32dd-bb88-b63230bb8525" } ]
Successfully deploying a flow results in output similar to:
{ "crn": "deployment-crn" }