Prerequisites

1. Navigate to Management Console > Environments and select your environment.
2. Select your Streams Messaging cluster.
3. Click on the Streams Messaging Manager icon.
4. Navigate to the Topics page.
5. Click Add New and provide the following information:
   • Topic name
   • Number of partitions
   • Level of availability
   • Cleanup policy
6. Click Save.

For information on how to create a new schema, see Creating a new schemaCreating a new schema. For example:

{
   "type":"record",
   "name":"SensorReading",
   "namespace":"com.cloudera.example",
   "doc":"This is a sample sensor reading",
   "fields":[
      {
         "name":"sensor_id",
         "doc":"Sensor identification number.",
         "type":"int"
      },
      {
         "name":"sensor_ts",
         "doc":"Timestamp of the collected readings.",
         "type":"long"
      },
      {
         "name":"sensor_0",
         "doc":"Reading #0.",
         "type":"int"
      },
      {
         "name":"sensor_1",
         "doc":"Reading #1.",
         "type":"int"
      },
      {
         "name":"sensor_2",
         "doc":"Reading #2.",
         "type":"int"
      },
      {
         "name":"sensor_3",
         "doc":"Reading #3.",
         "type":"int"
      }
   ]
}

1. From the Management Console, go to Data Hub Clusters and select the Streams Messaging cluster you are using.
2. Navigate to the Hardware tab to locate the Master Node FQDN. Schema Registry is always running on the Master node, so copy the Master node FQDN.

1. From the Management Console, click Data Hub Clusters.
2. Select the Streams Messaging cluster from which you want to ingest data.
3. Click the Hardware tab.
4. Note the Kafka Broker FQDNs for each node in your cluster.
5. Construct your Kafka Broker Endpoints by using the FQDN and Port number 9093 separated by a colon. Separate endpoints by a comma. For example:

   broker1.fqdn:9093,broker2.fqdn:9093,broker3.fqdn:9093

   Kafka broker FQDNs are listed under the Core_broker section.

1. Navigate to Management Console > Environments, and select the environment where you have created your cluster.
2. Select Ranger. You are redirected to the Ranger Service Manager page.
3. Select your Streams Messaging cluster under the Kafka folder.
4. Create a policy to enable your Workload User to access the Kafka source topic.
5. On the Create Policy page, give the policy a name, select topic from the drop-down list, add the user, and assign the Consume permission.
6. Create another policy to give your Workload User access to the consumer group ID.
7. On the Create Policy page, give the policy a name, select consumergroup from the drop-down list, add the user, and assign the Consume permission.

1. Navigate to Management Console > Environments, and select the environment where you have created your cluster.
2. Select Ranger. You are redirected to the Ranger Service Manager page.
3. Select your Streams Messaging cluster under the Schema Registry folder.
4. Click Add New Policy.
5. On the Create Policy page, give the policy a name, specify the schema details, add the user, and assign the Read permission.

1. From the Management Console, go to the environment for which Cloudera DataFlow is enabled.
2. From the Actions drop down, click Manage Access.
3. Identify the user you want to use as a Workload User.
4. Give that user EnvironmentUser role.

1. Give a user permission to add the ReadyFlow to the Catalog.
   1. From the Management Console, click User Management.
   2. Enter the name of the user or group you wish to authorize in the Search field.
   3. Select the user or group from the list that displays.
   4. Click Roles > Update Roles.
   5. From Update Roles, select DFCatalogAdmin and click Update.
2. Give your user or group permission to deploy flow definitions.
   1. From the Management Console, click Environments to display the Environment List page.
   2. Select the environment to which you want your user or group to deploy flow definitions.
   3. Click Actions > Manage Access to display the Environment Access page.
   4. Enter the name of your user or group you wish to authorize in the Search field.
   5. Select your user or group and click Update Roles.
   6. Select DFFlowAdmin from the list of roles.
   7. Click Update Roles.
3. Give your user or group access to the Project where the ReadyFlow will be deployed.
   1. Go to DataFlow > Projects.
   2. Select the project where you want to manage access rights and click More > Manage Access.
4. Start typing the name of the user or group you want to add and select them from the list.
5. Select the Resource Roles you want to grant.
6. Click Update Roles.
7. Click Synchronize Users.

1. From Management Console, click Data Hub Clusters.
2. Select the Real-Time Data Mart cluster to which you want to ingest data into.
3. Click the Hardware tab.
4. Copy the FQDN for each Kudu Master.

1. Navigate to your Real Time Data Mart cluster and click Hue from the Services pane.
2. Click the Tables icon on the left pane.
3. Select the default database, and click + New to create a new table.
4. In the Type field, select Manually and click Next.
5. Provide the table Name, Format, Primary keys, and any partitions.
6. Click Submit. The newly created table displays in the default database Tables pane.
7. Check the Kudu UI Tables tab for the name of the table you created. You will need this table name when you use the Cloudera DataFlow Deployment wizard to deploy the ReadyFlow.

1. From the base cluster on Cloudera Public Cloud, select Ranger.
2. Select your Real Time Data Mart cluster from the Kudu folder.
3. Click Add New Policy policy.
4. On the Create Policy page, enter the Kudu table name in the topic field.
5. Add the Cloudera Workload User in the Select User field.
6. Add the Insert and Select permissions in the Permissions field.
7. Click Save.