Cloudera Docs

Deserialization tab

When creating a Kafka table, you can configure how to handle errors due to schema mismatch using DDL or the Kafka wizard.

You can configure every supported type of Kafka connectors (local-kafka, kafka or upsert) how to handle if a message fails to deserialize which can result in job submission error. You can choose from the following configurations:

Fail
In this case an exception is thrown, and the job submission fails
Ignore
In this case the error message is ignored without any log, and the job submission is successful
Ignore and Log
In this case the error message is ignored, and the job submission is successful
Save to DLQ
In this case the error message is ignored, but you can store it in a dead-letter queue (DLQ) Kafka topic

Using the Kafka wizard

When you create the Kafka table using the wizard on the Streaming SQL Console, you can configure the error handling with the following steps:
  1. Navigate to the Streaming SQL Console.
    1. Navigate to Management Console > Environments, and select the environment where you have created your cluster.
    2. Select the Streaming Analytics cluster from the list of Data Hub clusters.
    3. Select SQL Stream Builder from the list of services.
  2. Click Create Job or select a previous job on the Getting Started page.

    You are redirected to the Console page.

  3. Select Add tables > Apache Kafka.

    The Add Kafka table window appears

  4. Select Deserialization tab.
  5. Choose from the following policy options under Deserialization Policy:
    • Fail
    • Ignore
    • Ignore and Log
    • Save to DLQ

      If you choose the Save to DLQ option, you need to create a dedicated Kafka topic where you store the error message. After selecting this option, you need to further select the created DLQ topic.

    • Click Create and Review.