Cloudera Docs

Creating Kafka tables using Console wizard

After registering a Kafka data provider, you can use the Add table wizard in Streaming SQL Console to create a Kafka table.

  • Make sure that you have registered Kafka as a Data Provider.
  • Make sure that you have created topics in Kafka.
  • Make sure there is generated data in the Kafka topic.
  • Make sure that you have the right permissions set in Ranger.
  1. Navigate to the Streaming SQL Console.
    1. Go to your cluster in Cloudera Manager.
    2. Select SQL Stream Builder from the list of services.
    3. Click SQLStreamBuilder Console.
    The Streaming SQL Console opens in a new window.
  2. Click Create Job or select a previous job on the Getting Started page.
    You are redirected to the Console page.
  3. Click Add table > Apache Kafka.
    The Kafka Table window appears.
  4. Provide a Table Name.
  5. Select a registered Kafka provider as Kafka cluster.
  6. Select the Data format.
    • You can select JSON as data format.
    • You can select AVRO as data format.
  7. Select a Kafka topic from the list.
  8. Determine the Schema for the Kafka table.
    1. Add a customized schema to the Schema Definition field.
    2. Click Detect Schema to read a sample of the JSON messages and automatically infer the schema.
  9. Customize your Kafka Table with the following options:
    1. Configure the Event Time if you do not want to use the default Kafka Timestamps.
      1. Disable the Use Kafka Timestamps feature.
      2. Provide the name of the Input Timestamp Column.
      3. Add a name for the Event Time Column.
      4. Add a value to the Watermark Seconds.
    2. Configure an Input Transform on the Data Transformations.
    3. Configure any Kafka properties required on the Properties tab.
    4. Select a policy for deserialization errors on the Deserialization tab.
    For more information about how to configure the Kafka table, see the Configuring Kafka tables section.
  10. Click Create and Review.
The Kafka Table is ready to be used for the SQL job either at the FROM or at the INSERT INTO statements.