Ingesting data into Google Cloud Storage

You can use an Apache NiFi data flow to ingest data into Google Cloud Storage (GCS) object stores in CDP Public Cloud by following these steps.

Understand the use case
Learn how you can use a Flow Management cluster in CDP Public Cloud to build a flow that ingests data to Google Cloud Storage (GCS). This example use case shows you how to use Apache NiFi to move data into GCS buckets.
Meet the prerequisites
Use this checklist to make sure that you meet all the requirements before you start building your data flow.
Build the data flow
Learn how you can create an ingest data flow to move data to Google Cloud Storage (GCS) buckets. This involves opening Apache NiFi in your Flow Management cluster, adding processors and other data flow objects to your canvas, and connecting your data flow elements.
Create IDBroker mapping
You must create IDBroker mapping for a user or group to access cloud storage. As a part of Knox, the IDBroker allows a user to exchange cluster authentication for temporary cloud credentials. To enable your CDP user to utilize the central authentication features CDP provides and to exchange credentials for Google Cloud access tokens, you have to map your CDP user to the correct IAM role. Learn how you can create the IDBroker mapping for the PutHDFS processor for your data flow.
Create controller services for your data flow
Controller services provide shared services that can be used by the processors in your data flow. You will use these Controller Services later when you configure your processors. Learn how you can create and configure the controller services for a Google Cloud ingest data flow in CDP Public Cloud.
Configure the processor for your data source
You can set up a data flow to move data to Google Cloud Storage (GCS) from many different locations. This example assumes that you are streaming the data that you are tailing from a local file on the NiFi host. Learn how you can configure the TailFile data source processor for your GCS ingest data flow.
Configure the processor for merging records
You can use merge together multiple record-oriented flow files into a large flow file that contains all records of your data input. Learn how you can configure the MergeContent processor for your Google Cloud Storage (GCS) ingest data flow.
Configure the processor for your data target
This example use case shows you two options for ingesting data to Google Cloud Storage (GCS). Learn how you can configure these data NiFi processors for your GCS ingest flow.
Start the data flow
When your flow is ready, you can begin ingesting data into Google Cloud Storage buckets. Learn how to start your GCS ingest data flow.
Verify data flow operation
Learn how you can verify the operation of your Google Cloud Storage (GCS) ingest data flow.