Cloudera Docs

Testing your data flow

It is important to test the data flow you have built before trying to deploy it.

Any data flow that is to be used by Azure Function Apps requires an Input Port and most often contains at least one failure Output Port. You can simply move up to the Parent Process Group within NiFi and create a GenerateFlowFile processor. You can then set the "Custom Text" property to any value that you may wish to feed into the data flow. For example, to simulate an Event Grid event indicating that data was added to a Storage Container:

  1. Set the "Custom Text" property to the value:

    {
  "topic": "/subscriptions/73db9d72-863b-45e9-9a1a-890cff0ac0e6/resourceGroups/my-rg/providers/Microsoft.Storage/storageAccounts/mystorageaccount",
  "subject": "/blobServices/default/containers/my-container/blobs/input/testfile.json",
  "eventType": "Microsoft.Storage.BlobCreated",
  "id": "d4c93b9a-901e-0078-1dd4-d34b8d06bbad",
  "data": {
    "api": "DeleteBlob",
    "requestId": "9d4c93b9-901e-0078-1dd4-d34b8d000000",
    "eTag": "0xD9A1EBFB6C05A58",
    "contentType": "application/json",
    "contentLength": 136,
    "blobType": "BlockBlob",
    "url": "https://mystorageaccount.blob.core.windows.net/my-container/input/testfile.json",
    "sequencer": "000000000000000000000000000004F50000000003e3cc87",
    "storageDiagnostics": {
      "batchId": "69a23459-7006-003d-00d4-d39e6e000000"
    }
  },
  "dataVersion": "",
  "metadataVersion": "1",
  "eventTime": "2021-11-07T12:41:55.7182885Z"
}

  2. Connect this GenerateFlowFile processor to the Input Port of our Process Group that you will run in the Function App.

    The "Run Once" feature of NiFi makes it easy to create this FlowFile and send it to the Process Group.

  3. Start the Process Group and ensure that the data processes as expected.

    If there are any issues, you can fix the data flow and trigger the GenerateFlowFile processor again, until you have properly handled the data.

When the test is successful, you are ready to download the flow from NiFi and upload the flow definition to the Cloudera DataFlow Catalog.