Ingesting data into Azure Data Lake Storage

You can use an Apache NiFi data flow to ingest data into Azure Data Lake Storage (ADLS) in CDP Public Cloud by following these steps.

Understand the use case
Learn how you can use a Flow Management cluster connected to a Streams Messaging cluster to build an end-to-end flow that ingests data to Azure Data Lake Storage (ADLS). This example use case shows you how to use Apache NiFi to move data from Kafka to ADLS.
Meet the prerequisites
Use this checklist to make sure that you meet all the requirements before you start building your data flow.
Build the data flow
Learn how you can create an ingest data flow to move data from Kafka to ADLS. This involves opening Apache NiFi in your Flow Management cluster, adding processors and other data flow objects to your canvas, and connecting your data flow elements.
Create IDBroker mapping
Learn how you can create the IDBroker mapping for the PutHDFS processor for your data flow. To enable your CDP user to utilize the central authentication features CDP provides and to exchange credentials for AWS access tokens, you have to map your CDP user to the correct IAM role.
Create controller services for your data flow
Learn how you can create and configure controller services for an ADLS ingest data flow in CDP Public Cloud. Controller services provide shared services that can be used by the processors in your data flow. You will use these Controller Services later when you configure your processors.
Configure the processor for your data source
Learn how you can configure the ConsumeKafkaRecord_2_0 data source processor for your ADLS ingest data flow. You can set up a data flow to move data to Azure Data Lake Storage from many different locations. This example assumes that you are streaming data from Kafka and shows you the configuration for the relevant data source processor.
Configure the processor for merging records
Learn how you can configure the MergeRecord processor for your ADLS ingest data flow. You can use it to merge together multiple record-oriented flow files into a large flow file that contains all records of your Kafka data input.
Configure the processor for your data target
Learn how you can configure the data target processor for your ADLS ingest data flow. This example assumes that you are moving data to Azure Data Lake Storage and shows you how to configure the corresponding processors.
Start the data flow
When your flow is ready, you can begin ingesting data into Azure Data Lake Storage folders. Learn how to start your ADLS ingest data flow.
Verify data flow operation
Learn how you can verify the operation of your ADLS ingest data flow.