Cloudera Docs

Running Incremental Extraction

With incremental extraction, you can keep Atlas updated with changes happening within ADLS.

You must configure the queue for the storage account so that extraction receives all blob, create, delete, and rename events.

In addition, you can optionally set the timeout or retry time so the extractor does not keep running even when it is not able to connect to the queue.

The following command line example runs the incremental extraction. Assuming the mandatory properties are set in the default configuration file, only the parameter to enable incremental mode is required:

/opt/cloudera/parcels/CDH/lib/atlas/extractors/bin/adls-extractor.sh -e INCREMENTAL

You must plan whether to set up a repeating job to run incremental extraction. Assuming you have a lot of changes that you want to track.