AWS S3 entity metadata migration
S3 metadata entities are migrated from Navigator to Atlas.
The following sections describe how metadata is mapped from Navigator to Atlas; if Atlas requires metadata that wasn't available in Navigator, the migration notes describe how the Atlas metadata values are generated.
Migrated entities include: For entity metadata that is common to all entities, see System metadata migration.
S3 Bucket
Navigator s3_bucket entities are migrated to Atlas aws_s3_v2_bucket entities.
| Navigator Metadata | Atlas Metadata | Migration Notes |
|---|---|---|
| encryption | attributes.encryption | |
| eTag | attributes.eTag | |
| owner | attributes.owner | |
| ownerId | attributes.ownerId | |
| region | attributes.region | |
| attributes.qualifiedName | Generated as a string in the format
<bucket_name>@<cluster_name>. For
example yellow_bucket@clustercolor. |
S3 Object: Directory
Navigator s3_object entities of type=DIRECTORY
is converted to aws_s3_v2_directory entities.
| Navigator Metadata | Atlas Metadata | Migration Notes |
|---|---|---|
| bucketName | attributes.bucketName | |
| depth | customAttributes | Added to the Atlas entity custom attributes as a key value pair with the Navigator name as the key. |
| encryption | attributes.encryption | |
| eTag | attributes.eTag | |
| fileSystemPath | attributes.storageLocation | |
| firstClassParentId | relationshipAttributes.container | Points to the parent directory or bucket as a relationship attribute. |
| implicit | attributes.implicit | |
| newObject | customAttributes | Added to the Atlas entity custom attributes as a key value pair with the Navigator name as the key. |
| owner | attributes.owner | |
| ownerId | attributes.ownerId | |
| parentPath | attributes.objectPrefix | The Atlas value is derived from the Navigator value (no one-to-one migration). |
| region | attributes.region | |
| sequencer | customAttributes | Added to the Atlas entity custom attributes as a key value pair with the Navigator name as the key. |
| size | attributes.size | |
| storageClass | attributes.storageClass | . |
| type | Used to determine the Atlas entity type. | |
| attributes.qualifiedName | Generated as a string in the format
<object_prefix>://<bucket_name>/<directory_name>@<cluster_name>.
For example
s3a://yellow_bucket/hive_storage_color_table_dir@clustercolor. |
S3 Object: File
Navigator s3_object entities of type=file is converted to aws_s3_v2_object entities.
| Navigator Metadata | Atlas Metadata | Migration Notes |
|---|---|---|
| bucketName | attributes.bucketName | |
| depth | customAttributes | Added to the Atlas entity custom attributes as a key value pair with the Navigator name as the key. |
| encryption | attributes.encryption | |
| eTag | attributes.eTag | |
| fileSystemPath | customAttributes | Added to the Atlas entity custom attributes as a key value pair with the Navigator name as the key. |
| firstClassParentId | relationshipAttributes.container | Points to the parent directory or bucket as a relationship attribute. |
| implicit | attributes.implicit | |
| newObject | customAttributes | Added to the Atlas entity custom attributes as a key value pair with the Navigator name as the key. |
| owner | attributes.owner | |
| ownerId | attributes.ownerId | |
| parentPath | attributes.objectPrefix | The Atlas value is derived from the Navigator value (no one-to-one migration). |
| region | attributes.region | |
| size | attributes.size | |
| storageClass | attributes.storageClass | |
| type | Used to determine the Atlas entity type. | |
| attributes.qualifiedName | Generated as a string in the format
<object_prefix>://<bucket_name>/<directory_name>/<file_name>@<cluster_name>.
For example
s3a://yellow_bucket/hive_storage_color_table_dir/.hive-staging_hive_2020-03-20_01-43-14_862_995309273321986325-1@clustercolor. |
