When using the Add Kafka table wizard on the Streaming SQL Console, you can configure
the event time under the Event Time tab.
You can specify Watermark Definitions when adding a Kafka table. Watermarks use an event time
attribute and have a watermark strategy, and can be used for various time-based operations. The
Event Time tab provides the following properties to configure the event time field and watermark
for the Kafka stream:
Input Timestamp Column: name of the timestamp column in the Kafka
topic from where the watermarks are mapped to the Event Time Column of the Kafka table
Event Time Column: default or custom name of the resulting timestamp
column where the watermarks are going to be mapped in the created Kafka table
Watermark seconds: number of seconds used in the watermark strategy.
The watermark is defined by the current event timestamp minus this value.
You have the following options to configure the watermark strategy for the Kafka tables:
Using the default Kafka Timestamps setting
Using the default Kafka Timestamps setting, but providing custom name
for the Event Time Column
Not using the default Kafka Timestamps setting, and providing all of
the Kafka timestamp information manually
Not using watermark strategy for the Kafka table
Using the default Kafka Timestamp setting
By default, the Use Kafka Timestamps checkbox is selected when you
create the Kafka table. In the Event Time Column, the new event time
field is extracted from the Kafka message header with the 'eventTimestamp'
predefined column name.
After saving your changes, you can view the created DDL syntax for the Table on the right side
under the DDL tab. You can review the generated watermark strategy for your table that was set
on the Watermark Definition tab.
The following DDL example shows the default setting of the Event Time Column and Watermark
Seconds where the corresponding fields were not
modified.
'eventTimestamp' TIMESTAMPS(3) METADATA FROM 'timestamp',
WATERMARK FOR 'eventTimestamp' AS 'eventTimestamp' - INTERVAL '3' SECOND
Using the default Kafka Timestamp setting with custom Event Time Column name
When you want to modify the timestamp field of the DDL from the stream itself, you must
provide a custom name of the Event Time Column. You can also add a custom
value to the Watermark Seconds. The following example shows that
'ets' is the custom name for the Event Time Column,
and '4' is the custom value for the Watermark
Seconds.
The Event Time Column can only be modified if the following
requirements are met for the timestamp field of the Input Timestamp
Column:
The column type must be "long".
The format must be in epoch (in milliseconds since January 1, 1970).
The DDL syntax should reflect the changes made for the watermark strategy as shown in the
following
example:
'ets' TIMESTAMP(3) METADATA FROM 'timestamp',
WATERMARK FOR 'ets' - INTERVAL '4' SECOND
Manually providing the Kafka timestamp information
When you want to manually configure the watermark strategy of the Kafka table, you can provide
the timestamp column name from the Kafka source, and add a custom column name for the resulting
Kafka table. Make sure that you provide the correct column name for the Input
Timestamp Column that exactly matches the column name in the Kafka source data.
To manually provide information for the watermark strategy, unselect the Use Kafka
Timestamps checkbox, and provide the following information to the column name
fields:
Input Timestamp Column: name of the timestamp field in the Kafka
source
Event Time Column: predefined 'eventTimestamp'
name or custome column name of the timestamp field in the created Kafka table
As an example, you have a timestamp column in the source Kafka topic named as
'ts', and want to add a new timestamp column in your Kafka table as
'event_time'. You provide the original timestamp column name in the
Input Timestamp Column as 'ts', and add the custom
'event_time' name to the Event Time Column. This results in that the watermarks from the 'ts' column is going to be
mapped to the 'event_time' column of the created Kafka table. As
'event_time' will become the timestamp column name in the Kafka table, you
must use the custom name (in this example the 'event_time') when querying the
Kafka stream. This configuration of the timestamp columns are optional.
The Event Time Column can only be modified if the following
requirements are met for the timestamp field of the Input Timestamp
Column:
The column type must be "long".
The format must be in epoch (in milliseconds since January 1, 1970).
Not using watermark strategy for Kafka table
In case you do not need any watermark strategies, unselect the Use Kafka
Timestamps checkbox, and leave the column and seconds field empty.
