Collecting configuration information for ReadyFlow

Learn how to collect the information you need to configure the "Kafka to S3 Avro" ReadyFlow.

The following sections help you to gather configuration information such as Schema Registry host name, Kafka brokers, and other details before you deploy NiFi flows:

For your ReadyFlow data ingest source

You have the Schema Registry Host Name.

From the Management Console, go to Data Hub Clusters and select the Streams Messaging cluster you are using.
Navigate to the Hardware tab to locate the Master Node FQDN. Schema Registry is always running on the Master node, so copy the Master node FQDN.

You have the Kafka broker end points.

From the Management Console, click Data Hub Clusters.
Select the Streams Messaging cluster from which you want to ingest data.
Click the Hardware tab.
Note the Kafka Broker FQDNs for each node in your cluster.
Construct your Kafka Broker Endpoints by using the FQDN and Port number 9093 separated by a colon. Separate endpoints by a comma. For example:
```
broker1.fqdn:9093,broker2.fqdn:9093,broker3.fqdn:9093
```
Kafka broker FQDNs are listed under the Core_broker section.

You have the Kafka Consumer Group ID.

This ID is defined by the user. Pick an ID and then create a Ranger policy for it. Use the ID when deploying the flow in DataFlow.

For your ReadyFlow data ingest target

Perform one of the following to configure access to S3 buckets:

You have configured access to S3 buckets with a RAZ enabled environment.
It is a best practice to enable RAZ to control access to your object store buckets. This allows you to use your CDP credentials to access S3 buckets, increases auditability, and makes object store data ingest workflows portable across cloud providers.
1. Ensure that Fine-grained access control is enabled for your DataFlow environment.
2. From the Ranger UI, navigate to the S3 repository.
3. Create a policy to govern access to the S3 bucket and path used in your ingest workflow. For example: kafka-to-s3-avro-ingest
  tip
  
  The Path field must begin with a forward slash ( / ).
4. Add the machine user that you have created for your ingest workflow to ingest the policy you just created.
For more information, see Creating Ranger policy to use in RAZ-enabled AWS environment.
You have configured access to S3 buckets using ID Broker mapping.
If your environment is not RAZ-enabled, you can configure access to S3 buckets using ID Broker mapping.
1. Access IDBroker mappings.
  1. To access IDBroker mappings in your environment, click Actions > Manage Access.
  2. Choose the IDBroker Mappings tab where you can provide mappings for users or groups and click Edit.
2. Add your CDP Workload User and the corresponding AWS role that provides write access to your folder in your S3 bucket to the Current Mappings section by clicking the blue + sign.
  note
  You can get the AWS IAM role ARN from the Roles Summary page in AWS and can copy it into the IDBroker role field. The selected AWS IAM role must have a trust policy allowing IDBroker to assume this role.
3. Click Save and Sync.