Cloudera Docs

Adding Kafka Data Provider

You need to register Kafka as a Data Provider using the Streaming SQL Console to create Kafka tables in SQL Stream Builder (SSB).

  • Make sure that you have Kafka service on your cluster.
  • Make sure that you have the right permissions set in Ranger.
  1. Navigate to the Streaming SQL Console.
    1. Go to your cluster in Cloudera Manager.
    2. Select SQL Stream Builder from the list of services.
    3. Click SQLStreamBuilder Console.
    The Streaming SQL Console opens in a new window.
  2. Click Data Providers from the main menu.
  3. Click Register Kafka Provider.
    The Add Kafka Provider window appears.
  4. Add a Name to your Kafka provider.
  5. Add the broker host name(s) to Brokers.
    You need to copy the Kafka broker name(s) from Cloudera Manager.
    1. Go to your cluster in Cloudera Manager.
    2. Click Kafka from the list of services.
    3. Click Instances.
    4. Copy the hostname of the Kafka broker(s) you want to use.
    5. Go back to the Add Kafka Provider page.
    6. Paste the broker hostname to the Brokers field.
    7. Add the default Kafka port after the hostname(s).
      Example:
      docs-test-1.vpc.cloudera.com:9092, 
docs-test-2.vpc.cloudera.com:9092
      docs-test-1.vpc.cloudera.com:9093, 
docs-test-2.vpc.cloudera.com:9093
  6. Select the security Protocol.
    The connection protocol must be the same as it is configured for the Kafka cluster in Cloudera Manager.

    You can choose from the following protocols:

    1. Click Create.
    1. Provide the path to the Kafka TrustStore.
    2. Click Create.
    1. Provide the path to the Kafka TrustStore.
    2. Choose an SASL Mechanism.
    3. Provide the Username for SASL.
    4. Provide the Password for SASL.
    5. Click Create.
    1. Choose an SASL Mechanism.
    2. Provide the Username for SASL.
    3. Provide the Password for SASL.
    4. Click Create.
You have registered Kafka as a data provider to be able to add Kafka as a table in your SQL query. The already existing Kafka topics can be selected when adding Kafka as a table.