Spark entities created in Apache Atlas
Each Spark entity in Atlas includes detailed metadata collected from Spark.
The following diagrams show a summary of the entities created in Atlas for Spark operations. The data assets that Spark operations act upon are collected through HMS. The supertypes that contribute attributes to the entity types are shaded.
The metadata collected for each entity type is as follows:
|guid||System generated ID. This value is used to identify the entity in the Atlas Dashboard URL.|
The generated ID is distinct from the GUID.
|description||Metadata from Spark.|
|owner||Metadata from Spark.|
|ownerType||Metadata from Spark.|
|inputs||List of the input tables or views, including each entity’s type name and the qualified name.|
|outputs||List of the output objects, including each entity’s type name and the qualified name.|
|executionId||Metadata from Spark.|
|currUser||Metadata from Spark. In a Kerberized environment, this value contains the principal name.|
|remoteUser||Metadata from Spark. In a Kerberized environment, this value contains the principal name.|
|executionTime||Metadata from Spark.|
|details||Query plan text, including parsed logical plan, analyzed logical plan, optimized logical plan, and physical plan.|
|sparkPlanDescription||Physical plan text.|