Create controller services for your data flow

Learn how you can create and configure controller services for an ADLS ingest data flow in CDP Public Cloud. Controller services provide shared services that can be used by the processors in your data flow. You will use these Controller Services later when you configure your processors.

  1. To add a Controller Service to your flow, right-click on the Canvas and select Configure from the pop-up menu. This displays the Controller Services Configuration window.
  2. Select the Controller Services tab.
  3. Click the + button to display the Add Controller Service dialog.
  4. Select the required Controller Service and click Add.
  5. Perform any necessary Controller Service configuration tasks by clicking the Configure icon in the right-hand column.
  6. When you have finished configuring the options you need, save the changes by clicking the Apply button.
  7. Enable the Controller Service by clicking the Enable button (flash) in the far-right column of the Controller Services tab.
In this example we are using the following controller services:
AvroReader Controller Service
Table 1. AvroReader Controller Service properties
Property Description Example value for ingest data flow

Schema Access Strategy

Specify how to obtain the schema to be used for interpreting the data.

HWX Content-Encoded Schema Reference

Schema Registry

Specify the Controller Service to use for the Schema Registry.

CDPSchemaRegistry

Schema Name

Specify the name of the schema to look up in the Schema Registry property.

customer

CSVReader Controller Service
Table 2. CSVReader Controller Service Properties
Property Description Example value for ingest data flow

Schema Access Strategy

Specify how to obtain the schema to be used for interpreting the data.

Use String Fields From Header

Treat First Line as Header

Specify whether the first line of CSV should be considered a header or a record.

true

CSVRecordSetWriter Controller Service
Table 3. CSVRecordSetWriter Controller Service Properties
Property Description Example value for ingest data flow

Schema Write Strategy

Specify how the schema for a record should be added to the data.

Do Not Write Schema

Schema Access Strategy

Specify how to obtain the schema to be used for interpreting the data.

Use 'Schema Name' Property

Schema Name

Specify the name of the schema to look up in the Schema Registry property.

See the Appendix for an example schema.

customer

Configure the processors in your data flow.