Ingesting data into Amazon S3

You can use an Apache NiFi data flow to ingest data into Amazon S3 object stores in CDP Public Cloud by following these steps.

Understand the use case
Learn how you can use a Flow Management cluster connected to a Streams Messaging cluster to build an end-to-end flow that ingests data to Amazon S3 storage. This example use case shows you how to use Apache NiFi to move data from Kafka to S3 buckets.
Meet the prerequisites
Use this checklist to make sure that you meet all the requirements before you start building your data flow.
Build the data flow
Learn how you can create an ingest data flow to move data from Kafka to S3 buckets. This involves opening Apache NiFi in your Flow Management cluster, adding processors and other data flow objects to your canvas, and connecting your data flow elements.
Set up AWS for your ingest data flow
Learn what you need to set on the AWS side to prepare for data ingestion.
Create IDBroker mapping
Learn how you can create the IDBroker mapping for the target data processors (PutS3Object and PutHDFS) of your data flow. To enable your CDP user to utilize the central authentication features CDP provides and to exchange credentials for AWS access tokens, you have to map your CDP user to the correct IAM role.
Create controller services for your data flow
Learn how you can create and configure controller services for an S3 ingest data flow in CDP Public Cloud. Controller services provide shared services that can be used by the processors in your data flow. You will use these Controller Services later when you configure your processors.
Configure the processor for your data source
Learn how you can configure the ConsumeKafkaRecord_2_0 data source processor for your S3 ingest data flow. You can set up a data flow to move data to Amazon S3 from many different locations. This example assumes that you are streaming data from Kafka and shows you the configuration for the relevant data source processor.
Configure the processor for merging records
Learn how you can configure the MergeRecord processor for your S3 ingest data flow. You can use it to merge together multiple record-oriented flow files into a large flow file that contains all records of your Kafka data input.
Configure the processor for your data target
Learn how you can configure the data target processor for your S3 ingest data flow. This example assumes that you are moving data to AWS S3 and shows you how to configure the corresponding processors.
Start the data flow
When your flow is ready, you can begin ingesting data into Amazon S3 buckets. Learn how to start your S3 ingest data flow.
Verify data flow operation
Learn how you can verify the operation of your S3 ingest data flow.