Data Lake upgrade - Kafka consumer not available yet

After the Data Lake upgrade, the Atlas Hook does not function as the Kafka service is not available yet.

Condition

After upgrading the Data Lake to 7.3.1 or one of its patch versions, sometimes Atlas Hook does not function when Apache Atlas and Apache Kafka are started at the same time, thus Atlas is unable to connect to Kafka while Kafka is still being set up. When this happens, the following exception can be seen:

Exception in getKafkaConsumer ,WakeupException: null

The Kafka consumer creation should be retried if the Kafka service is unavailable during Atlas startup.

Cause

Atlas performs only three attempts to restart, but during this time, the Kafka service might not be available yet. This causes the Atlas Hook to not function and none of the messages from the Kafka topics are consumed.

Remedy

Wait until the Kafka service is available after the upgrade.
Restart Atlas to trigger the reconnection to Kafka.