Error handling for Avro deserialization
When creating a Kafka table with Avro data type, you can configure how to handle schema mismatch errors using DDL or the Kafka wizard.
You can configure every supported type of Kafka connectors (local-kafka, kafka or upsert) how to handle if a message fails to deserialize which can result in job submission error. You can choose from the following configurations:
- Throw an exception: In this case an exception is thrown, and the job submission fails
- Ignore the message: In this case the error message is ignored without any log, and the job submission is successful
- Ignore the message and log the error: In this case the error message is ignored, and the job submission is successful
- Ignore the message and store it in a topic: In this case the error message is ignored, but you can store it in a dead-letter queue Kafka topic
Using the Kafka wizard
When you create the Kafka table using the wizard on the Streaming SQL Console, you can configure the error handling with the following steps:
Using DDL
When you create the Kafka table using DDL on the Streaming SQL Console, you can configure the
error handling with the following optional arguments:
- Navigate to the Streaming SQL Console.
- Click Compose tab.
- Choose one of the Kafka template types from Templates.
- Select avro as data format.
The predefined CREATE TABLE statement is imported to the SQL Window.
- Fill out the Kafka template based on your requirements.
- Search for the
deserialization.failure.policy. - Provide the value for the error handling from the following options:
‘error’‘ignore’‘ignore_and_log’‘dlq’
- Click Execute.
