Deserialization tab
When creating a Kafka table, you can configure how to handle errors due to schema mismatch using DDL or the Kafka wizard.
You can configure every supported type of Kafka connectors (local-kafka, kafka or upsert) how to handle if a message fails to deserialize which can result in job submission error. You can choose from the following configurations:
- Fail
- In this case an exception is thrown, and the job submission fails
- Ignore
- In this case the error message is ignored without any log, and the job submission is successful
- Ignore and Log
- In this case the error message is ignored, and the job submission is successful
- Save to DLQ
- In this case the error message is ignored, but you can store it in a dead-letter queue (DLQ) Kafka topic
Using the Kafka wizard
When you create the Kafka table using the wizard on the Streaming SQL Console, you can
configure the error handling with the following steps:
- Navigate to the Streaming SQL Console.
- Go to your cluster in Cloudera Manager.
- Select SQL Stream Builder from the list of services.
- Click .
- Click Create Job or select a previous job on the
Getting Started page.
You are redirected to the Console page.
- Select
The Add Kafka table window appears
. - Select Deserialization tab.
- Choose from the following policy options under Deserialization
Policy:
- Fail
- Ignore
- Ignore and Log
- Save to DLQ
If you choose the Save to DLQ option, you need to create a dedicated Kafka topic where you store the error message. In this case, you must select the dedicated DLQ topic from DLQ Topic Name.
- Click Create and Review.