Extractor configuration properties
Some of the extractor configuration properties.
|Configuration Parameter (prefixed with
Required. AWS S3 region for extraction. CDP Public Cloud: provided by IDBroker integration.
CDP Data Center: the region must be provided.
AWS S3 credentials.
CDP Public Cloud: provided by IDBroker integration.
CDP Data Center:
Set of data assets to include in metadata extraction. If no paths are
included, the extractor extracts metadata for all buckets. Buckets inside the
region and accessible by the credential used to run the extractor. For public
cloud, this is defined in
The list can include buckets or paths; separate multiple values with commas. See Defining what assets to extract metadata for.
|Set of data assets to exclude from metadata extraction. The list can include buckets or paths; separate multiple values with commas. See Defining what assets to extract metadata for.
|Number of objects returned in one call to AWS S3 in bulk extraction. Defaults to 1000.
|Enable bulk extraction resume. Disabled by default.
|Name of the progress file that tracks bulk extraction that will be used if the
extraction has to be resumed mid-way after a failure. This file is created in the
Atlas log directory. Defaults to
|Number of times the extractor will try to reconnect to AWS in case of
credentials expiring or
retryable exception from AWS S3. Defaults
|File system scheme used in case of AWS S3. At this time,
is the only option.
|Required for incremental mode. AWS SQS Queue name configured to get the AWS S3 bucket events.
|The number of messages the extractor will try to get from the SQS queue in one call. Defaults to 10.
|The interval that the extractor will wait to collect messages so long as the limit set in .incremental.messagesPerRequest hasn't been met. Defaults to 20 seconds.
|Maximum retry count if the SQS queue doesn't respond during an extractor request. Defaults to 20.