Cloudera Docs

Deserialization tab

When creating a Kafka table, you can configure how to handle errors due to schema mismatch using DDL or the Kafka wizard.

You can configure every supported type of Kafka connectors (local-kafka, kafka or upsert) how to handle if a message fails to deserialize which can result in job submission error. You can choose from the following configurations:

Fail
In this case an exception is thrown, and the job submission fails
Ignore
In this case the error message is ignored without any log, and the job submission is successful
Ignore and Log
In this case the error message is ignored, and the job submission is successful
Save to DLQ
In this case the error message is ignored, but you can store it in a dead-letter queue (DLQ) Kafka topic

Using the Kafka wizard

When you create the Kafka table using the wizard on the Streaming SQL Console, you can configure the error handling with the following steps:
  1. Navigate to the Streaming SQL Console.
    1. Go to your cluster in Cloudera Manager.
    2. Select SQL Stream Builder from the list of services.
    3. Click SQLStreamBuilder Console.
  2. Click Create Job or select a previous job on the Getting Started page.

    You are redirected to the Console page.

  3. Select Add tables > Apache Kafka.

    The Add Kafka table window appears

  4. Select Deserialization tab.
  5. Choose from the following policy options under Deserialization Policy:
    • Fail
    • Ignore
    • Ignore and Log
    • Save to DLQ

      If you choose the Save to DLQ option, you need to create a dedicated Kafka topic where you store the error message. After selecting this option, you need to further select the created DLQ topic.

    • Click Create and Review.

Using DDL

When you create the Kafka table using DDL on the Streaming SQL Console, you can configure the error handling with the following optional arguments:
  1. Navigate to the Streaming SQL Console.
    1. Go to your cluster in Cloudera Manager.
    2. Select SQL Stream Builder from the list of services.
    3. Click Web UI > SQLStreamBuilder Console.
  2. Click Create Job or select a previous job on the Getting Started page.

    You are redirected to the Console page.

  3. Choose one of the Kafka template types from Templates.
  4. Select any type of data format.

    The predefined CREATE TABLE statement is imported to the SQL Editor.

  5. Fill out the Kafka template based on your requirements.
  6. Search for the deserialization.failure.policy.
  7. Provide the value for the error handling from the following options:
    1. ‘error’
    2. ‘ignore’
    3. ‘ignore_and_log’
    4. ‘dlq’

      If you choose the dlq option, you need to create a dedicated Kafka topic where you store the error message. After selecting this option, you need to further provide the name of the created DLQ topic.

  8. Click Execute.