Deploying a flow definition using the CLI

Deploy a flow definition to run NiFi flows as flow deployments in Cloudera DataFlow. To do this, launch the CDP CLI and specify your environment, parameters, sizing, and KPIs.

You have installed CDP CLI.
Run cdp df list-services to get the service-crn value.
Run cdp df list-flows to get the flow-version-crn value.

To deploy a flow, enter:

cdp df create-deployment
          --service-crn [***service-crn-value***]          
          --flow-version-crn [***flow-version-crn-value***]
          --deployment-name [***flow-deployment-name***]
          [--cluster-size-name [EXTRA_SMALL|SMALL|MEDIUM|LARGE]]
          [--static-node-count [***value***]]
          [--auto-scaling-enabled | --no-auto-scaling-enabled]
          [--auto-scale-min-nodes [***number-min-nodes***]]
          [--auto-scale-max-nodes [***number-max-nodes***]]
          [--cfm-nifi-version [***nifi-version***]]
          [--auto-start-flow | --no-auto-start-flow]
          [--parameter-groups [***json-file-locatione***]]
          [--kpis [***json-file-locatione***]]
          [---node-storage-profile-name [STANDARD_AWS|STANDARD_AZURE|PERFORMANCE_AWS|PERFORMANCE_AZURE]]

Where:

--service-crn

Specifies the service-crn value you obtained when completing the prerequisites.

--flow-version-crn

Specifies the flow-version-crn value you obtained when completing the prerequisites.

--deployment-name

Specifies a unique name for your flow deployment.

[--cluster-size-name]

Specifies the cluster size. Valid values are:

EXTRA_SMALL
SMALL
MEDIUM
LARGE

The default value is EXTRA_SMALL.

[--static-node-count]

Specifies the number of NiFi nodes when autoscaling is not enabled. You can select between 1 and 32 nodes. The default value is 1.

[--auto-scaling-enabled | --no-auto-scaling-enabled]

Specifies whether you want to enable autoscaling. The default is to disable autoscaling.

[--auto-scale-min-nodes]: Specifies the minimum nodes when you have autoscaling enabled. If you have autoscaling enabled, this parameter is required.
[--auto-scale-max-nodes]: Specifies the maximum nodes when autoscaling is enabled. If you have autoscaling enabled, this parameter is required.

[--cfm-nifi-version]

Specifies the NiFi runtime version. The default is the latest version.

[--auto-start-flow | --no-auto-start-flow]

Specifies whether you want to automatically start your flow once it has been deployed. The default is to enable the automatic start.

[--parameter-groups]

Specifies the location of the parameter group JSON file, if you are using one for this flow deployment.

[--kpis]

Specifies the location of the KPIs JSON file, if you are providing KPIs for this flow.

--node-storage-profile-name

Specifies the storage profile. Valid values are:

STANDARD_AWS
STANDARD_AZURE
PERFORMANCE_AWS
PERFORMANCE_AZURE

The default values are STANDARD_AWS and STANDARD_AZURE, depending on the cloud provider.

Example parameter group file:

[
  {
    "name": "kafka-filter-to-kafka",
    "parameters": [
      {
        "name": "CDP Workload User",
        "assetReferences": [],
        "value": "srv_nifi-machine-ingest"
      },
      {
        "name": "CDP Workload User Password",
        "assetReferences": [],
        "value": "<<CDP_MISSING_SENSITIVE_VALUE>>"
      },
      {
        "name": "CSV Delimiter",
        "assetReferences": [],
        "value": ","
      },
      {
        "name": "Data Input Format",
        "assetReferences": [],
        "value": "CSV"
      },
      {
        "name": "Data Output Format",
        "assetReferences": [],
        "value": "JSON"
      },
      {
        "name": "Filter Rule",
        "assetReferences": [],
        "value": "SELECT * FROM FLOWFILE"
      },
      {
        "name": "Kafka Broker Endpoint",
        "assetReferences": [],
        "value": "streams-messaging-broker0.pm-sandb.a465-9q4k.cloudera.site:9093"
      },
      {
        "name": "Kafka Consumer Group ID",
        "assetReferences": [],
        "value": "cdf"
      },
      {
        "name": "Kafka Destination Topic",
        "assetReferences": [],
        "value": "MachineDataJSON"
      },
      {
        "name": "Kafka Producer ID",
        "assetReferences": [],
        "value": "cdf"
      },
      {
        "name": "Kafka Source Topic",
        "assetReferences": [],
        "value": "MachineDataCSV"
      },
      {
        "name": "Schema Name",
        "assetReferences": [],
        "value": "SensorReading"
      },
      {
        "name": "Schema Registry Hostname",
        "assetReferences": [],
        "value": "streams-messaging-master0.pm-sandb.a465-9q4k.cloudera.site"
      }
    ]
  }
]

Example KPI file:

[
  {
    "metricId": "rateBytesReceived",
    "alert": {
      "thresholdLessThan": {
        "unitId": "kilobytesPerSecond",
        "value": 150
      },
      "frequencyTolerance": {
        "unit": {
          "id": "MINUTES"
        },
        "value": 1
      }
    }
  },
  {
    "metricId": "processorAmountBytesSent",
    "alert": {},
    "componentId": "a7f7df1c-a32d-3c25-9b09-e1d1036dcc04;a33a1b48-005b-32dd-bb88-b63230bb8525"
  }
]

Successfully deploying a flow results in output similar to:


{
  "crn": "deployment-crn"
}