AWS S3 entity metadata migration

S3 metadata entities are migrated from Navigator to Atlas.

The following sections describe how metadata is mapped from Navigator to Atlas; if Atlas requires metadata that wasn't available in Navigator, the migration notes describe how the Atlas metadata values are generated.

Migrated entities include: For entity metadata that is common to all entities, see System metadata migration.

S3 Bucket

Navigator s3_bucket entities are migrated to Atlas aws_s3_v2_bucket entities.

Navigator Metadata Atlas Metadata Migration Notes
encryption attributes.encryption
eTag attributes.eTag
owner attributes.owner
ownerId attributes.ownerId
region attributes.region
attributes.qualifiedName Generated as a string in the format <bucket_name>@<cluster_name>. For example yellow_bucket@clustercolor.

S3 Object: Directory

Navigator s3_object entities of type=DIRECTORY is converted to aws_s3_v2_directory entities.

Navigator Metadata Atlas Metadata Migration Notes
bucketName attributes.bucketName
depth customAttributes Added to the Atlas entity custom attributes as a key value pair with the Navigator name as the key.
encryption attributes.encryption
eTag attributes.eTag
fileSystemPath attributes.storageLocation
firstClassParentId relationshipAttributes.container Points to the parent directory or bucket as a relationship attribute.
implicit attributes.implicit
newObject customAttributes Added to the Atlas entity custom attributes as a key value pair with the Navigator name as the key.
owner attributes.owner
ownerId attributes.ownerId
parentPath attributes.objectPrefix The Atlas value is derived from the Navigator value (no one-to-one migration).
region attributes.region
sequencer customAttributes Added to the Atlas entity custom attributes as a key value pair with the Navigator name as the key.
size attributes.size
storageClass attributes.storageClass .
type Used to determine the Atlas entity type.
attributes.qualifiedName Generated as a string in the format <object_prefix>://<bucket_name>/<directory_name>@<cluster_name>. For example s3a://yellow_bucket/hive_storage_color_table_dir@clustercolor.

S3 Object: File

Navigator s3_object entities of type=file is converted to aws_s3_v2_object entities.

Navigator Metadata Atlas Metadata Migration Notes
bucketName attributes.bucketName
depth customAttributes Added to the Atlas entity custom attributes as a key value pair with the Navigator name as the key.
encryption attributes.encryption
eTag attributes.eTag
fileSystemPath customAttributes Added to the Atlas entity custom attributes as a key value pair with the Navigator name as the key.
firstClassParentId relationshipAttributes.container Points to the parent directory or bucket as a relationship attribute.
implicit attributes.implicit
newObject customAttributes Added to the Atlas entity custom attributes as a key value pair with the Navigator name as the key.
owner attributes.owner
ownerId attributes.ownerId
parentPath attributes.objectPrefix The Atlas value is derived from the Navigator value (no one-to-one migration).
region attributes.region
size attributes.size
storageClass attributes.storageClass
type Used to determine the Atlas entity type.
attributes.qualifiedName Generated as a string in the format <object_prefix>://<bucket_name>/<directory_name>/<file_name>@<cluster_name>. For example s3a://yellow_bucket/hive_storage_color_table_dir/.hive-staging_hive_2020-03-20_01-43-14_862_995309273321986325-1@clustercolor.