Distributed notification processing framework

The distributed notification processing framework enables parallel processing of metadata events through a two-phase architecture to improve throughput and prevent head-of-line blocking in high-volume environments.

Apache Atlas splits the ingestion lifecycle into two phases: Preprocessing (Routing) and Persistence (Processing). This design shifts Atlas from a single-threaded sequential pipeline to a parallel architecture that significantly improves horizontal scalability and reduces ingestion backlog. By splitting the workload across multiple independent topics, it enables concurrent processing of metadata and lineage events while maintaining strict entity and lineage consistency. It achieves this by deterministically routing related entities (for example, a database table and its columns) to the same topic, preventing duplication and referential integrity issues without requiring global coordination.

Logical architecture

The distributed notification processing framework operates through the following sequence:

Ingress: External sources push HookNotification messages to ingress topics such as ATLAS_HOOK and ATLAS_SPARK_HOOK. Standardizing ingress through these topics supports lineage ingestion from any external query engine.
Phase 1 (Routing): A NotificationPreProcessor consumes the message, extracts entities, calculates a routing key from a deterministic field such as dbName.tableName, and produces new messages to metadata and lineage sub-topics. The atlas.notification.processor.metadata.topic.count and atlas.notification.processor.lineage.topic.count properties define how many metadata and lineage Kafka topics Atlas prepares for consumption. You must derive the topic names based on your desired counts and configure them explicitly in atlas.notification.hook.consumer.topic.names.

Atlas creates one consumer thread per topic name in atlas.notification.hook.consumer.topic.names (effective parallelism is capped by that list length), so hook-consumer parallelism scales with how many hook topics you list.

Routing enforces a db.table.column hierarchy, ensuring that all entities belonging to the same table, including columns, are assigned to the same topic. This design decision guarantees referential integrity and preserves lineage correctness across parallel processing threads.
Phase 2 (Processing): Consumers pull these messages from the sub-topics and translate them into graph database transactions (creating vertices for entities and edges for relationships) in JanusGraph. By isolating database transactions within dedicated, independent topic consumers, Atlas avoids the need for global coordination, external dependency graphs, or locks. JanusGraph is the underlying graph database that Apache Atlas uses to permanently store all metadata entities and their relationships (lineage).

Atlas determines the correct topic assignment for each message by normalizing the qualified names of the entities and applying a stable mathematical hash. Because this hashing process is deterministic, related entities (such as a database table and its associated columns) always resolve to the same routing key and are consistently assigned to the same topic, even across service restarts.

By distributing the ingestion workloads across these multiple independent topics, overall throughput scales efficiently based on the diversity of processed entities (the routing keys) and the number of active consumer threads, which significantly reduces ingestion latency. Even in worst-case scenarios where a workload is dominated by a single entity family, forcing that specific workload to be processed serially on a single topic, the system guarantees that data correctness and lineage consistency are fully preserved.

Limitations

When running the distributed notification processing framework, be aware of the following system behaviors and limitations:

Rename scenarios: Explicit support for resolving historical entity lineage during rename operations is omitted. This feature would significantly increase system complexity to address an edge case that occurs only rarely in production environments.
Temporal lineage behavior: In rare timing scenarios, lineage might link to a deleted entity instead of a newly recreated one. For example, if a table is recreated, a lineage message processed while the metadata for the recreated entity is still in the lag buffer will connect to the originally created (now technically deleted) vertex that was active at that precise timestamp. This is expected behavior due to the independent nature of the parallelized message streams.
Graph locking: You might observe a higher frequency of graph locking errors in the Atlas logs. This occurs when parallel events attempt to modify the same graph data simultaneously. The framework uses standard Atlas retry logic to handle these contentions and ensure data persistence.