Enable and configure the distributed notification processing framework to process
high-throughput metadata ingestion by routing messages across multiple Kafka
topics.
Configure this feature by setting the following properties in Cloudera Manager > Clusters > Atlas > Configuration > Atlas Server Advanced Configuration Snippet (Safety Valve) for
conf/atlas-application.properties.
Set atlas.notification.parallel.processing.enabled to
true to activate the two-phase (Router/Processor) pipeline
architecture.
Set atlas.notification.processor.metadata.topic.count to define the
number of parallel topics for metadata (entity/attribute) updates.
Example: 3
Set atlas.notification.processor.lineage.topic.count to define the
number of parallel topics specifically for relationship/lineage events.
Example: 3
Set atlas.notification.parallel.processing.input.topics with a
comma-separated list of ingress topics that the preprocessor listens to.
You can continue to use serial processing for some of your ingress topics by omitting
them from this list. The advantage of staying with serial processing is simplicity and
lower operational overhead when the existing workload is already processed efficiently
without consumer lag.
Compared to serial processing, parallel processing
introduces:
Increased Kafka storage footprint due to internal message
duplication/re-publishing
Additional operational complexity from managing multiple topics and consumer
threads
Default: ATLAS_HOOK, ATLAS_SPARK_HOOK
Set atlas.notification.hook.consumer.topic.names with an explicit
comma-separated list of the sub-topics that the serial processor threads consume
from.
