Cloudera Docs

Activating Concurrent Message Ingestion

Atlas hook message processing can take a long time with a high number of messages, greatly increasing wait times. This can improved by turning on Concurrent Message Ingestion.

Atlas Hook message processing has linear complexity for consuming the messages. This can lead to the following:
  • Enforcing authorization policies takes longer.
  • Metadata showing up in Atlas takes an unpredictable amount of time.
Concurrent Message Ingestion can offset these issues by the enabling the following after determining dependencies within incoming messages:
  • Dependent messages are processed serially.
  • Messages without dependencies are processed concurrently.

Enabling Concurrent Message Ingestion

  1. Go to Cloudera Manager > Clusters > Atlas > Configuration.
  2. Add the following property atlas.notifications.concurrent=true.