HDFS entity metadata migration
HDFS metadata entities are migrated from Navigator to Atlas when they appear in a lineage relationship from Hive, Impala, or Spark processes.
The following sections describe how metadata is mapped from Navigator to Atlas; if Atlas requires metadata that wasn't available in Navigator, the migration notes describe how the Atlas metadata values are generated.
Migrated entities include: For entity metadata that is common to all entities, see System metadata migration.
HDFS Directory
Navigator fselement entities of type=DIRECTORY
are migrated to Atlas hdfs_path entities with the
isFile
attribute set to false.
Navigator Metadata | Atlas Metadata | Migration Notes |
---|---|---|
blockSize | Null for directories. | |
created | attributes.createTime | Converted to date type. |
ezKeyName | Null for directories. | |
fileSystemPath | attributes.path | |
group | attributes.group | |
lastAccessed | attributes.modifiedTime | Converted to date type. |
mimeType | Null for directories. | |
owner | attributes.owner | |
permissions | attributes.posixPermissions | Converted to Atlas values. |
replication | attributes.numberOfReplicas | Null for directories. |
size | attributes.fileSize | Null for directories. |
type | attributes.isFile | The Navigator type=DIRECTORY property is
converted to isFile=FALSE . |
attributes.isSymLink | Defaults to FALSE. | |
attributes.nameServiceId | Optional in Atlas. | |
attributes.qualifiedName | Generated as a string in the format
<path>@clustername . |
HDFS File
Navigator fselement entities of type=FILE
are
migrated to Atlas hdfs_path entities with the
isFile
attribute set to true.
Navigator Metadata | Atlas Metadata | Migration Notes |
---|---|---|
blockSize | Added to the Atlas entity custom attributes as a key value pair with the Navigator name as the key. | |
created | attributes.createTime | Converted to date type. |
ezKeyName | Added to the Atlas entity custom attributes as a key value pair with the Navigator name as the key. | |
fileSystemPath | attributes.path | |
group | attributes.group | |
lastAccessed | attributes.modifiedTime | Converted to date type. |
mimeType | Added to the Atlas entity custom attributes as a key value pair with the Navigator name as the key. | |
owner | attributes.owner | |
parentPath | attributes.extendedAttributes | |
permissions | attributes.posixPermissions | Converted to Atlas values. |
replication | attributes.numberOfReplicas | |
size | attributes.fileSize | |
type | attributes.isFile | The Navigator type=FILE property is
converted to isFile=TRUE . |
attributes.isSymLink | Defaults to FALSE. | |
attributes.nameServiceId | Optional in Atlas. | |
attributes.qualifiedName | Generated as a string in the format
<path>@clustername . |