Spark entity metadata migration

Spark metadata entities are fully migrated from Navigator to Atlas.

The following sections describe how metadata is mapped from Navigator to Atlas; if Atlas requires metadata that wasn't available in Navigator, the migration notes describe how the Atlas metadata values are generated.

Migrated entities include:

Spark Process
Spark Process Execution

For entity metadata that is common to all entities, see System metadata migration.

Spark Process

Navigator spark_operation entities are migrated to Atlas spark_process entities.


Navigator Metadata	Atlas Metadata	Migration Notes
inputs	relationshipAttributes.inputs	Points to the input directories as relationship attributes.
outputs	relationshipAttributes.outputs	Points to the output directory as a relationship attribute.
principal		Not mapped.
	attributes.currUser	Defaults to null.
	attributes.details	Defaults to null.
	attributes.executionId	Defaults to null.
	attributes.remoteUser	Defaults to null.
	attributes.sparkPlanDescription
	attributes.qualifiedName	Generated as a string in the format `process_id@clustername`. For example, `application_1589303388872_0001@clustercolor.20324`.

Spark Process Execution

Navigator spark_operation_execution entities are migrated to Atlas spark_process_execution entities.


Navigator Metadata	Atlas Metadata	Migration Notes
inputs	relationshipAttributes.inputs	Points to the input files as relationship attributes.
outputs	relationshipAttributes.outputs	Points to the output files as relationship attributes.
ended	attributes.endTime
operation	attributes.process	Points to the parent `spark_process` entity as a relationship attribute.
originalName	attributes.queryText
principal	attributes.userName
started	attributes.startTime
	attributes.hostName	Defaults to null.
	attributes.qualifiedName	Generated as a string in the format `process_id-exec.timestamp@clustername`. For example, `application_1589303388872_0001-exec.1589331800580@clustercolor`.