Introduction to Cloudera Navigator Lineage Diagrams
A lineage diagram is a directed graph that depicts an entity and its relations with other entities. A lineage diagram is limited to 1000 entities and 2000 relations.
There are two types of lineage diagrams:
- Template - represents an entity that is a model for other entities
- Instance - represents an instance or execution of a template
Entities
In a lineage diagram, entity types are represented by icons:
| HDFS | Oozie | ||
|
|
|
|
| Hive | Pig | ||
|
|
|
|
| MapReduce | SQOOP | ||
|
|
|
|
| YARN | |||
|
|
||
Parent entities are represented by a white box enclosing other entities. The following lineage diagram illustrates the relations between the YARN job aggregate.pig and Pig script aggregate.pig invoked by the parent Pig Oozie workflow and its source file and destination folder:
Relations
Relations between the entities are represented graphically by gray lines, with arrows indicating the direction of the data flow. There are the following types of relations:
| Relation Type | Description |
|---|---|
| Data flow | Describes a relation between data and a processing activity. For example, between a file and a MapReduce job or vice versa. |
| Alias | Describes an alias relation. For example, from a table to a synonym. |
| Parent-child | Describes a parent-child relation. For example, between a directory and a file. |
| Logical-physical | Describes the relation between a logical entity and its physical entity. For example, between a Hive query and a MapReduce job. |
| Conjoint | Describes a non-directional relation. For example, between an table and an index. |
| Instance of | Describes the relation between a template and its instance. For example, an operation execution is an instance of operation. |
| Control flow | Describes a relation where the source entity controls the data flow of the target entity. For example, between the columns used in an insert clause and the where clause of a Hive query. |
For lines connecting database columns, a dashed line indicates that the column is in the where clause; a solid line indicates that the column is in the select clause.
Manipulating Lineage Diagrams
You can click a parent entity to display its child entities. For example, you can click the Pig script to display its child tables:
- To improve the layout of a lineage diagram you can drag and drop entities (in this case sonnets.txt and out) located outside a parent box.
- You can use the mouse scroll wheel to zoom the lineage diagram in and out.
- You can move the lineage diagram in the lineage pane by pressing the mouse button and dragging it.
Displaying a Template Lineage Diagram
A template lineage diagram contains template entities, such as jobs and queries, that can be instantiated and the input and output entities to which they are related.
To display a template lineage diagram:
- Perform a metadata search.
- In the list of results, click a result entry. For example, when you click the sample_09 result entry:
the Search screen is replaced with a page that displays the entity property sheet on the left and lineage diagram on the right:
This example lineage diagram illustrates the relations between a Hive query execution entity and its source and
destination tables:
When you click each entity icon, columns and lines connecting the source and destination columns display:
If you hover over a part, the source and destination columns are highlighted:
When you click each entity icon, columns and lines connecting the source and destination columns display:
If you hover over a part, the source and destination columns are highlighted:
Displaying an Instance Lineage Diagram
An instance lineage diagram displays instance entities, such as job and query executions, and the input and output entities to which they are related.
To display an instance lineage diagram:- Display a template lineage diagram. For example:
- Click the Instances tab, which contains a list of links to instances of the template.
- Click a link to display an instance lineage diagram. The job instance job_1396040412350_0003 replaces the wordcount job
template.
Displaying the Template Lineage Diagram for an Instance Lineage Diagram
Downloading a Lineage File
Lineage is externalized in a lineage file in JSON format.
- Display a template or instance lineage diagram.
- Click the
icon at the top left of the diagram.
A lineage file named lineage.json is downloaded. For example, the lineage file representing job_1396040412350_0003 from the preceding section is:{ "entities": { "d212538318276a6ad8abdd308a4487cc": { "level": 1, "physical": [], "logical": [], "aliasOf": [], "aliases": [], "instances": [], "children": [], "workflows": [], "identity": "d212538318276a6ad8abdd308a4487cc", "originalName": "file0", "sourceId": "012437f9eeb3c23dc69e679ac94a7fa2", "firstClassParentId": null, "parentPath": "/user/hdfs/input", "extractorRunId": null, "name": "file0", "description": null, "tags": null, "fileSystemPath": "/user/hdfs/input/file0", "type": "FILE", "size": 22, "created": "2014-07-08T22:03:32.126Z", "lastModified": "2014-07-08T22:03:32.126Z", "lastAccessed": "2014-07-08T22:03:31.612Z", "permissions": 420, "owner": "hdfs", "group": "supergroup", "blockSize": null, "mimeType": "application/octet-stream", "deleted": false, "sourceType": "HDFS", "replication": null, "internalType": "fselement", "nameField": "originalName", "sourceName": "HDFS", "isScript": false, "hasDownstream": true, "parent": "e461de8de38511a3ac6740dd7d51b8d0", "activeChildren": [] }, "57c993fc305b553dae82210090d5da7a": { "level": 1, "physical": [], "logical": [], "aliasOf": [], "aliases": [], "instances": [ "fc671272b2c7518ec738d6b17f08b7ed" ], "children": [], "workflows": [], "identity": "57c993fc305b553dae82210090d5da7a", "originalName": "wordcount", "sourceId": "262c2128425eabeb29bb1c96ee0eb35f", "firstClassParentId": null, "parentPath": null, "extractorRunId": null, "name": "wordcount", "description": null, "tags": null, "wfIds": null, "inputFormat": "org.apache.hadoop.mapred.TextInputFormat", "outputFormat": "org.apache.hadoop.mapred.TextOutputFormat", "outputKey": "org.apache.hadoop.io.Text", "outputValue": "org.apache.hadoop.io.IntWritable", "mapper": "org.myorg.WordCount$Map", "reducer": "org.myorg.WordCount$Reduce", "sourceType": "YARN", "type": "OPERATION", "internalType": "mrjobspec", "nameField": "name", "sourceName": "YARN (MR2 Included)", "isScript": false, "unorchestrated": true, "active": true, "column": 0, "renderOrdinal": 0, "activeChildren": [], "x": 0, "y": -52.3046875 }, "2b5a90798459c168d9fb5ecc2fc46aa5": { "level": 1, "physical": [], "logical": [], "aliasOf": [], "aliases": [], "instances": [], "children": [], "workflows": [], "identity": "2b5a90798459c168d9fb5ecc2fc46aa5", "originalName": "part-00001", "sourceId": "012437f9eeb3c23dc69e679ac94a7fa2", "firstClassParentId": null, "parentPath": "/user/hdfs/output", "extractorRunId": null, "name": "part-00001", "description": null, "tags": null, "fileSystemPath": "/user/hdfs/output/part-00001", "type": "FILE", "size": 6, "created": "2014-07-08T22:08:53.602Z", "lastModified": "2014-07-08T22:08:53.602Z", "lastAccessed": "2014-07-08T22:08:52.923Z", "permissions": 420, "owner": "hdfs", "group": "supergroup", "blockSize": null, "mimeType": "application/octet-stream", "deleted": false, "sourceType": "HDFS", "replication": null, "internalType": "fselement", "nameField": "originalName", "sourceName": "HDFS", "isScript": false, "parent": "e9b3d1f7aee29134338df3de4cae1100", "hasUpstream": true, "activeChildren": [] }, "fe5445fbd070d97c418d96200a218cae": { "level": 1, "physical": [], "logical": [], "aliasOf": [], "aliases": [], "instances": [], "children": [], "workflows": [], "identity": "fe5445fbd070d97c418d96200a218cae", "originalName": "part-00002", "sourceId": "012437f9eeb3c23dc69e679ac94a7fa2", "firstClassParentId": null, "parentPath": "/user/hdfs/output", "extractorRunId": null, "name": "part-00002", "description": null, "tags": null, "fileSystemPath": "/user/hdfs/output/part-00002", "type": "FILE", "size": 9, "created": "2014-07-08T22:08:53.619Z", "lastModified": "2014-07-08T22:08:53.619Z", "lastAccessed": "2014-07-08T22:08:52.464Z", "permissions": 420, "owner": "hdfs", "group": "supergroup", "blockSize": null, "mimeType": "application/octet-stream", "deleted": false, "sourceType": "HDFS", "replication": null, "internalType": "fselement", "nameField": "originalName", "sourceName": "HDFS", "isScript": false, "parent": "e9b3d1f7aee29134338df3de4cae1100", "hasUpstream": true, "activeChildren": [] }, "916b141fb9ce45094df8ef97ecdde41c": { "level": 1, "physical": [], "logical": [], "aliasOf": [], "aliases": [], "instances": [], "children": [], "workflows": [], "identity": "916b141fb9ce45094df8ef97ecdde41c", "originalName": "file1._COPYING_", "sourceId": "012437f9eeb3c23dc69e679ac94a7fa2", "firstClassParentId": null, "parentPath": "/user/hdfs/input", "extractorRunId": null, "name": "file1._COPYING_", "description": null, "tags": null, "fileSystemPath": "/user/hdfs/input/file1._COPYING_", "type": "FILE", "size": 28, "created": "2014-07-08T22:03:32.171Z", "lastModified": "2014-07-08T22:03:32.244Z", "lastAccessed": "2014-07-08T22:03:32.171Z", "permissions": 420, "owner": "hdfs", "group": "supergroup", "blockSize": null, "mimeType": "application/octet-stream", "deleted": true, "sourceType": "HDFS", "replication": null, "internalType": "fselement", "nameField": "originalName", "sourceName": "HDFS", "isScript": false, "parent": "e461de8de38511a3ac6740dd7d51b8d0", "activeChildren": [] }, "66bff6eeac3c17c5b7eb7c9035704eef": { "level": 1, "physical": [], "logical": [], "aliasOf": [], "aliases": [], "instances": [], "children": [], "workflows": [], "identity": "66bff6eeac3c17c5b7eb7c9035704eef", "originalName": "part-00005", "sourceId": "012437f9eeb3c23dc69e679ac94a7fa2", "firstClassParentId": null, "parentPath": "/user/hdfs/output", "extractorRunId": null, "name": "part-00005", "description": null, "tags": null, "fileSystemPath": "/user/hdfs/output/part-00005", "type": "FILE", "size": 8, "created": "2014-07-08T22:08:53.668Z", "lastModified": "2014-07-08T22:08:53.668Z", "lastAccessed": "2014-07-08T22:08:52.419Z", "permissions": 420, "owner": "hdfs", "group": "supergroup", "blockSize": null, "mimeType": "application/octet-stream", "deleted": false, "sourceType": "HDFS", "replication": null, "internalType": "fselement", "nameField": "originalName", "sourceName": "HDFS", "isScript": false, "hasUpstream": true, "parent": "e9b3d1f7aee29134338df3de4cae1100", "activeChildren": [] }, "51db076ee11470b7f968bd5f33429e6b": { "level": 1, "physical": [], "logical": [], "aliasOf": [], "aliases": [], "instances": [], "children": [], "workflows": [], "identity": "51db076ee11470b7f968bd5f33429e6b", "originalName": "file0._COPYING_", "sourceId": "012437f9eeb3c23dc69e679ac94a7fa2", "firstClassParentId": null, "parentPath": "/user/hdfs/input", "extractorRunId": null, "name": "file0._COPYING_", "description": null, "tags": null, "fileSystemPath": "/user/hdfs/input/file0._COPYING_", "type": "FILE", "size": 22, "created": "2014-07-08T22:03:31.612Z", "lastModified": "2014-07-08T22:03:32.100Z", "lastAccessed": "2014-07-08T22:03:31.612Z", "permissions": 420, "owner": "hdfs", "group": "supergroup", "blockSize": null, "mimeType": "application/octet-stream", "deleted": true, "sourceType": "HDFS", "replication": null, "internalType": "fselement", "nameField": "originalName", "sourceName": "HDFS", "isScript": false, "parent": "e461de8de38511a3ac6740dd7d51b8d0", "activeChildren": [] }, "c864e9c3bc3f5f7a99315c56830e811d": { "level": 1, "physical": [], "logical": [], "aliasOf": [], "aliases": [], "instances": [], "children": [], "workflows": [], "identity": "c864e9c3bc3f5f7a99315c56830e811d", "originalName": "_SUCCESS", "sourceId": "012437f9eeb3c23dc69e679ac94a7fa2", "firstClassParentId": null, "parentPath": "/user/hdfs/output", "extractorRunId": null, "name": "_SUCCESS", "description": null, "tags": null, "fileSystemPath": "/user/hdfs/output/_SUCCESS", "type": "FILE", "size": 0, "created": "2014-07-08T22:08:53.689Z", "lastModified": "2014-07-08T22:08:53.694Z", "lastAccessed": "2014-07-08T22:08:53.689Z", "permissions": 420, "owner": "hdfs", "group": "supergroup", "blockSize": null, "mimeType": "application/octet-stream", "deleted": false, "sourceType": "HDFS", "replication": null, "internalType": "fselement", "nameField": "originalName", "sourceName": "HDFS", "isScript": false, "parent": "e9b3d1f7aee29134338df3de4cae1100", "hasUpstream": true, "activeChildren": [] }, "9d7ca9e46a9e4624df0d5d10949a5fc6": { "level": 1, "physical": [], "logical": [], "aliasOf": [], "aliases": [], "instances": [], "children": [], "workflows": [], "identity": "9d7ca9e46a9e4624df0d5d10949a5fc6", "originalName": "part-00004", "sourceId": "012437f9eeb3c23dc69e679ac94a7fa2", "firstClassParentId": null, "parentPath": "/user/hdfs/output", "extractorRunId": null, "name": "part-00004", "description": null, "tags": null, "fileSystemPath": "/user/hdfs/output/part-00004", "type": "FILE", "size": 10, "created": "2014-07-08T22:08:53.651Z", "lastModified": "2014-07-08T22:08:53.651Z", "lastAccessed": "2014-07-08T22:08:52.856Z", "permissions": 420, "owner": "hdfs", "group": "supergroup", "blockSize": null, "mimeType": "application/octet-stream", "deleted": false, "sourceType": "HDFS", "replication": null, "internalType": "fselement", "nameField": "originalName", "sourceName": "HDFS", "isScript": false, "hasUpstream": true, "parent": "e9b3d1f7aee29134338df3de4cae1100", "activeChildren": [] }, "8c40fbf6b4584a0b213a257be2cae679": { "level": 1, "physical": [], "logical": [], "aliasOf": [], "aliases": [], "instances": [], "children": [], "workflows": [], "identity": "8c40fbf6b4584a0b213a257be2cae679", "originalName": "file1", "sourceId": "012437f9eeb3c23dc69e679ac94a7fa2", "firstClassParentId": null, "parentPath": "/user/hdfs/input", "extractorRunId": null, "name": "file1", "description": null, "tags": null, "fileSystemPath": "/user/hdfs/input/file1", "type": "FILE", "size": 28, "created": "2014-07-08T22:03:32.262Z", "lastModified": "2014-07-08T22:03:32.262Z", "lastAccessed": "2014-07-08T22:03:32.171Z", "permissions": 420, "owner": "hdfs", "group": "supergroup", "blockSize": null, "mimeType": "application/octet-stream", "deleted": false, "sourceType": "HDFS", "replication": null, "internalType": "fselement", "nameField": "originalName", "sourceName": "HDFS", "isScript": false, "hasDownstream": true, "parent": "e461de8de38511a3ac6740dd7d51b8d0", "activeChildren": [] }, "2690183198a2d6568188426ea9a82793": { "level": 1, "physical": [], "logical": [], "aliasOf": [], "aliases": [], "instances": [], "children": [], "workflows": [], "identity": "2690183198a2d6568188426ea9a82793", "originalName": "part-00000", "sourceId": "012437f9eeb3c23dc69e679ac94a7fa2", "firstClassParentId": null, "parentPath": "/user/hdfs/output", "extractorRunId": null, "name": "part-00000", "description": null, "tags": null, "fileSystemPath": "/user/hdfs/output/part-00000", "type": "FILE", "size": 0, "created": "2014-07-08T22:08:53.580Z", "lastModified": "2014-07-08T22:08:53.580Z", "lastAccessed": "2014-07-08T22:08:52.568Z", "permissions": 420, "owner": "hdfs", "group": "supergroup", "blockSize": null, "mimeType": "application/octet-stream", "deleted": false, "sourceType": "HDFS", "replication": null, "internalType": "fselement", "nameField": "originalName", "sourceName": "HDFS", "isScript": false, "hasUpstream": true, "parent": "e9b3d1f7aee29134338df3de4cae1100", "activeChildren": [] }, "e9b3d1f7aee29134338df3de4cae1100": { "level": 1, "physical": [], "logical": [], "aliasOf": [], "aliases": [], "instances": [], "children": [ "fe5445fbd070d97c418d96200a218cae", "c864e9c3bc3f5f7a99315c56830e811d", "2b5a90798459c168d9fb5ecc2fc46aa5", "4374529a34441bc8db30ee001bbb3563", "2690183198a2d6568188426ea9a82793", "66bff6eeac3c17c5b7eb7c9035704eef", "9d7ca9e46a9e4624df0d5d10949a5fc6", "e5586cdbc89ecef7552e125acb91c2cf" ], "workflows": [], "identity": "e9b3d1f7aee29134338df3de4cae1100", "originalName": "output", "sourceId": "012437f9eeb3c23dc69e679ac94a7fa2", "firstClassParentId": null, "parentPath": "/user/hdfs", "extractorRunId": null, "name": "output", "description": null, "tags": null, "fileSystemPath": "/user/hdfs/output", "type": "DIRECTORY", "size": null, "created": "2014-07-08T22:08:31.569Z", "lastModified": "2014-07-08T22:08:53.689Z", "lastAccessed": null, "permissions": 493, "owner": "hdfs", "group": "supergroup", "blockSize": null, "mimeType": null, "deleted": false, "sourceType": "HDFS", "replication": null, "internalType": "fselement", "nameField": "originalName", "sourceName": "HDFS", "isScript": false, "hasUpstream": true, "column": 1, "renderOrdinal": 2, "activeChildren": [ { "level": 1, "physical": [], "logical": [], "aliasOf": [], "aliases": [], "instances": [], "children": [], "workflows": [], "identity": "c864e9c3bc3f5f7a99315c56830e811d", "originalName": "_SUCCESS", "sourceId": "012437f9eeb3c23dc69e679ac94a7fa2", "firstClassParentId": null, "parentPath": "/user/hdfs/output", "extractorRunId": null, "name": "_SUCCESS", "description": null, "tags": null, "fileSystemPath": "/user/hdfs/output/_SUCCESS", "type": "FILE", "size": 0, "created": "2014-07-08T22:08:53.689Z", "lastModified": "2014-07-08T22:08:53.694Z", "lastAccessed": "2014-07-08T22:08:53.689Z", "permissions": 420, "owner": "hdfs", "group": "supergroup", "blockSize": null, "mimeType": "application/octet-stream", "deleted": false, "sourceType": "HDFS", "replication": null, "internalType": "fselement", "nameField": "originalName", "sourceName": "HDFS", "isScript": false, "parent": "e9b3d1f7aee29134338df3de4cae1100", "hasUpstream": true, "activeChildren": [] }, { "level": 1, "physical": [], "logical": [], "aliasOf": [], "aliases": [], "instances": [], "children": [], "workflows": [], "identity": "2690183198a2d6568188426ea9a82793", "originalName": "part-00000", "sourceId": "012437f9eeb3c23dc69e679ac94a7fa2", "firstClassParentId": null, "parentPath": "/user/hdfs/output", "extractorRunId": null, "name": "part-00000", "description": null, "tags": null, "fileSystemPath": "/user/hdfs/output/part-00000", "type": "FILE", "size": 0, "created": "2014-07-08T22:08:53.580Z", "lastModified": "2014-07-08T22:08:53.580Z", "lastAccessed": "2014-07-08T22:08:52.568Z", "permissions": 420, "owner": "hdfs", "group": "supergroup", "blockSize": null, "mimeType": "application/octet-stream", "deleted": false, "sourceType": "HDFS", "replication": null, "internalType": "fselement", "nameField": "originalName", "sourceName": "HDFS", "isScript": false, "hasUpstream": true, "parent": "e9b3d1f7aee29134338df3de4cae1100", "activeChildren": [] }, { "level": 1, "physical": [], "logical": [], "aliasOf": [], "aliases": [], "instances": [], "children": [], "workflows": [], "identity": "2b5a90798459c168d9fb5ecc2fc46aa5", "originalName": "part-00001", "sourceId": "012437f9eeb3c23dc69e679ac94a7fa2", "firstClassParentId": null, "parentPath": "/user/hdfs/output", "extractorRunId": null, "name": "part-00001", "description": null, "tags": null, "fileSystemPath": "/user/hdfs/output/part-00001", "type": "FILE", "size": 6, "created": "2014-07-08T22:08:53.602Z", "lastModified": "2014-07-08T22:08:53.602Z", "lastAccessed": "2014-07-08T22:08:52.923Z", "permissions": 420, "owner": "hdfs", "group": "supergroup", "blockSize": null, "mimeType": "application/octet-stream", "deleted": false, "sourceType": "HDFS", "replication": null, "internalType": "fselement", "nameField": "originalName", "sourceName": "HDFS", "isScript": false, "parent": "e9b3d1f7aee29134338df3de4cae1100", "hasUpstream": true, "activeChildren": [] }, { "level": 1, "physical": [], "logical": [], "aliasOf": [], "aliases": [], "instances": [], "children": [], "workflows": [], "identity": "fe5445fbd070d97c418d96200a218cae", "originalName": "part-00002", "sourceId": "012437f9eeb3c23dc69e679ac94a7fa2", "firstClassParentId": null, "parentPath": "/user/hdfs/output", "extractorRunId": null, "name": "part-00002", "description": null, "tags": null, "fileSystemPath": "/user/hdfs/output/part-00002", "type": "FILE", "size": 9, "created": "2014-07-08T22:08:53.619Z", "lastModified": "2014-07-08T22:08:53.619Z", "lastAccessed": "2014-07-08T22:08:52.464Z", "permissions": 420, "owner": "hdfs", "group": "supergroup", "blockSize": null, "mimeType": "application/octet-stream", "deleted": false, "sourceType": "HDFS", "replication": null, "internalType": "fselement", "nameField": "originalName", "sourceName": "HDFS", "isScript": false, "parent": "e9b3d1f7aee29134338df3de4cae1100", "hasUpstream": true, "activeChildren": [] }, { "level": 1, "physical": [], "logical": [], "aliasOf": [], "aliases": [], "instances": [], "children": [], "workflows": [], "identity": "4374529a34441bc8db30ee001bbb3563", "originalName": "part-00003", "sourceId": "012437f9eeb3c23dc69e679ac94a7fa2", "firstClassParentId": null, "parentPath": "/user/hdfs/output", "extractorRunId": null, "name": "part-00003", "description": null, "tags": null, "fileSystemPath": "/user/hdfs/output/part-00003", "type": "FILE", "size": 8, "created": "2014-07-08T22:08:53.636Z", "lastModified": "2014-07-08T22:08:53.636Z", "lastAccessed": "2014-07-08T22:08:52.499Z", "permissions": 420, "owner": "hdfs", "group": "supergroup", "blockSize": null, "mimeType": "application/octet-stream", "deleted": false, "sourceType": "HDFS", "replication": null, "internalType": "fselement", "nameField": "originalName", "sourceName": "HDFS", "isScript": false, "parent": "e9b3d1f7aee29134338df3de4cae1100", "hasUpstream": true, "activeChildren": [] }, { "level": 1, "physical": [], "logical": [], "aliasOf": [], "aliases": [], "instances": [], "children": [], "workflows": [], "identity": "9d7ca9e46a9e4624df0d5d10949a5fc6", "originalName": "part-00004", "sourceId": "012437f9eeb3c23dc69e679ac94a7fa2", "firstClassParentId": null, "parentPath": "/user/hdfs/output", "extractorRunId": null, "name": "part-00004", "description": null, "tags": null, "fileSystemPath": "/user/hdfs/output/part-00004", "type": "FILE", "size": 10, "created": "2014-07-08T22:08:53.651Z", "lastModified": "2014-07-08T22:08:53.651Z", "lastAccessed": "2014-07-08T22:08:52.856Z", "permissions": 420, "owner": "hdfs", "group": "supergroup", "blockSize": null, "mimeType": "application/octet-stream", "deleted": false, "sourceType": "HDFS", "replication": null, "internalType": "fselement", "nameField": "originalName", "sourceName": "HDFS", "isScript": false, "hasUpstream": true, "parent": "e9b3d1f7aee29134338df3de4cae1100", "activeChildren": [] }, { "level": 1, "physical": [], "logical": [], "aliasOf": [], "aliases": [], "instances": [], "children": [], "workflows": [], "identity": "66bff6eeac3c17c5b7eb7c9035704eef", "originalName": "part-00005", "sourceId": "012437f9eeb3c23dc69e679ac94a7fa2", "firstClassParentId": null, "parentPath": "/user/hdfs/output", "extractorRunId": null, "name": "part-00005", "description": null, "tags": null, "fileSystemPath": "/user/hdfs/output/part-00005", "type": "FILE", "size": 8, "created": "2014-07-08T22:08:53.668Z", "lastModified": "2014-07-08T22:08:53.668Z", "lastAccessed": "2014-07-08T22:08:52.419Z", "permissions": 420, "owner": "hdfs", "group": "supergroup", "blockSize": null, "mimeType": "application/octet-stream", "deleted": false, "sourceType": "HDFS", "replication": null, "internalType": "fselement", "nameField": "originalName", "sourceName": "HDFS", "isScript": false, "hasUpstream": true, "parent": "e9b3d1f7aee29134338df3de4cae1100", "activeChildren": [] }, { "level": 1, "physical": [], "logical": [], "aliasOf": [], "aliases": [], "instances": [], "children": [], "workflows": [], "identity": "e5586cdbc89ecef7552e125acb91c2cf", "parent": "e9b3d1f7aee29134338df3de4cae1100", "activeChildren": [] } ], "x": 182, "y": -52.3046875 }, "4374529a34441bc8db30ee001bbb3563": { "level": 1, "physical": [], "logical": [], "aliasOf": [], "aliases": [], "instances": [], "children": [], "workflows": [], "identity": "4374529a34441bc8db30ee001bbb3563", "originalName": "part-00003", "sourceId": "012437f9eeb3c23dc69e679ac94a7fa2", "firstClassParentId": null, "parentPath": "/user/hdfs/output", "extractorRunId": null, "name": "part-00003", "description": null, "tags": null, "fileSystemPath": "/user/hdfs/output/part-00003", "type": "FILE", "size": 8, "created": "2014-07-08T22:08:53.636Z", "lastModified": "2014-07-08T22:08:53.636Z", "lastAccessed": "2014-07-08T22:08:52.499Z", "permissions": 420, "owner": "hdfs", "group": "supergroup", "blockSize": null, "mimeType": "application/octet-stream", "deleted": false, "sourceType": "HDFS", "replication": null, "internalType": "fselement", "nameField": "originalName", "sourceName": "HDFS", "isScript": false, "parent": "e9b3d1f7aee29134338df3de4cae1100", "hasUpstream": true, "activeChildren": [] }, "e461de8de38511a3ac6740dd7d51b8d0": { "level": 1, "physical": [], "logical": [], "aliasOf": [], "aliases": [], "instances": [], "children": [ "916b141fb9ce45094df8ef97ecdde41c", "8c40fbf6b4584a0b213a257be2cae679", "51db076ee11470b7f968bd5f33429e6b", "d212538318276a6ad8abdd308a4487cc" ], "workflows": [], "identity": "e461de8de38511a3ac6740dd7d51b8d0", "originalName": "input", "sourceId": "012437f9eeb3c23dc69e679ac94a7fa2", "firstClassParentId": null, "parentPath": "/user/hdfs", "extractorRunId": null, "name": "input", "description": null, "tags": null, "fileSystemPath": "/user/hdfs/input", "type": "DIRECTORY", "size": null, "created": "2014-07-08T22:02:26.664Z", "lastModified": "2014-07-08T22:03:32.262Z", "lastAccessed": null, "permissions": 493, "owner": "hdfs", "group": "supergroup", "blockSize": null, "mimeType": null, "deleted": false, "sourceType": "HDFS", "replication": null, "internalType": "fselement", "nameField": "originalName", "sourceName": "HDFS", "isScript": false, "hasDownstream": true, "column": -1, "renderOrdinal": 1, "activeChildren": [ { "level": 1, "physical": [], "logical": [], "aliasOf": [], "aliases": [], "instances": [], "children": [], "workflows": [], "identity": "d212538318276a6ad8abdd308a4487cc", "originalName": "file0", "sourceId": "012437f9eeb3c23dc69e679ac94a7fa2", "firstClassParentId": null, "parentPath": "/user/hdfs/input", "extractorRunId": null, "name": "file0", "description": null, "tags": null, "fileSystemPath": "/user/hdfs/input/file0", "type": "FILE", "size": 22, "created": "2014-07-08T22:03:32.126Z", "lastModified": "2014-07-08T22:03:32.126Z", "lastAccessed": "2014-07-08T22:03:31.612Z", "permissions": 420, "owner": "hdfs", "group": "supergroup", "blockSize": null, "mimeType": "application/octet-stream", "deleted": false, "sourceType": "HDFS", "replication": null, "internalType": "fselement", "nameField": "originalName", "sourceName": "HDFS", "isScript": false, "hasDownstream": true, "parent": "e461de8de38511a3ac6740dd7d51b8d0", "activeChildren": [] }, { "level": 1, "physical": [], "logical": [], "aliasOf": [], "aliases": [], "instances": [], "children": [], "workflows": [], "identity": "51db076ee11470b7f968bd5f33429e6b", "originalName": "file0._COPYING_", "sourceId": "012437f9eeb3c23dc69e679ac94a7fa2", "firstClassParentId": null, "parentPath": "/user/hdfs/input", "extractorRunId": null, "name": "file0._COPYING_", "description": null, "tags": null, "fileSystemPath": "/user/hdfs/input/file0._COPYING_", "type": "FILE", "size": 22, "created": "2014-07-08T22:03:31.612Z", "lastModified": "2014-07-08T22:03:32.100Z", "lastAccessed": "2014-07-08T22:03:31.612Z", "permissions": 420, "owner": "hdfs", "group": "supergroup", "blockSize": null, "mimeType": "application/octet-stream", "deleted": true, "sourceType": "HDFS", "replication": null, "internalType": "fselement", "nameField": "originalName", "sourceName": "HDFS", "isScript": false, "parent": "e461de8de38511a3ac6740dd7d51b8d0", "activeChildren": [] }, { "level": 1, "physical": [], "logical": [], "aliasOf": [], "aliases": [], "instances": [], "children": [], "workflows": [], "identity": "8c40fbf6b4584a0b213a257be2cae679", "originalName": "file1", "sourceId": "012437f9eeb3c23dc69e679ac94a7fa2", "firstClassParentId": null, "parentPath": "/user/hdfs/input", "extractorRunId": null, "name": "file1", "description": null, "tags": null, "fileSystemPath": "/user/hdfs/input/file1", "type": "FILE", "size": 28, "created": "2014-07-08T22:03:32.262Z", "lastModified": "2014-07-08T22:03:32.262Z", "lastAccessed": "2014-07-08T22:03:32.171Z", "permissions": 420, "owner": "hdfs", "group": "supergroup", "blockSize": null, "mimeType": "application/octet-stream", "deleted": false, "sourceType": "HDFS", "replication": null, "internalType": "fselement", "nameField": "originalName", "sourceName": "HDFS", "isScript": false, "hasDownstream": true, "parent": "e461de8de38511a3ac6740dd7d51b8d0", "activeChildren": [] }, { "level": 1, "physical": [], "logical": [], "aliasOf": [], "aliases": [], "instances": [], "children": [], "workflows": [], "identity": "916b141fb9ce45094df8ef97ecdde41c", "originalName": "file1._COPYING_", "sourceId": "012437f9eeb3c23dc69e679ac94a7fa2", "firstClassParentId": null, "parentPath": "/user/hdfs/input", "extractorRunId": null, "name": "file1._COPYING_", "description": null, "tags": null, "fileSystemPath": "/user/hdfs/input/file1._COPYING_", "type": "FILE", "size": 28, "created": "2014-07-08T22:03:32.171Z", "lastModified": "2014-07-08T22:03:32.244Z", "lastAccessed": "2014-07-08T22:03:32.171Z", "permissions": 420, "owner": "hdfs", "group": "supergroup", "blockSize": null, "mimeType": "application/octet-stream", "deleted": true, "sourceType": "HDFS", "replication": null, "internalType": "fselement", "nameField": "originalName", "sourceName": "HDFS", "isScript": false, "parent": "e461de8de38511a3ac6740dd7d51b8d0", "activeChildren": [] } ], "x": -182, "y": -52.3046875 }, "fc671272b2c7518ec738d6b17f08b7ed": { "level": 1, "physical": [], "logical": [], "aliasOf": [], "aliases": [], "instances": [], "children": [], "workflows": [], "identity": "fc671272b2c7518ec738d6b17f08b7ed", "originalName": "job_1396040412350_0003", "sourceId": "262c2128425eabeb29bb1c96ee0eb35f", "firstClassParentId": null, "parentPath": null, "extractorRunId": null, "name": "job_1396040412350_0003", "description": null, "tags": null, "started": "2014-07-08T22:08:31.543Z", "ended": "2014-07-08T22:08:53.722Z", "principal": "hdfs", "inputs": [ "hdfs://tcdn501-1.ent.cloudera.com:8020/user/hdfs/input" ], "outputs": [ "hdfs://tcdn501-1.ent.cloudera.com:8020/user/hdfs/output" ], "wfInstId": null, "jobID": "job_1396040412350_0003", "sourceType": "YARN", "inputRecursive": false, "type": "OPERATION_EXECUTION", "internalType": "mrjobinstance", "nameField": "originalName", "sourceName": "YARN (MR2 Included)", "isScript": false, "hasUpstream": true, "template": "57c993fc305b553dae82210090d5da7a", "hasDownstream": true }, "e5586cdbc89ecef7552e125acb91c2cf": { "level": 1, "physical": [], "logical": [], "aliasOf": [], "aliases": [], "instances": [], "children": [], "workflows": [], "identity": "e5586cdbc89ecef7552e125acb91c2cf", "parent": "e9b3d1f7aee29134338df3de4cae1100", "activeChildren": [] } }, "relations": { "57bd1a6ddc6ff99ce878f9c024759bbf": { "identity": "57bd1a6ddc6ff99ce878f9c024759bbf", "type": "PARENT_CHILD", "propagatorId": null, "extractorRunId": "NOT_STORED", "parent": { "entityId": "e9b3d1f7aee29134338df3de4cae1100" }, "children": { "entityIds": [ "fe5445fbd070d97c418d96200a218cae" ] }, "unlinked": false, "propagatable": false }, "bc51b7d63696a428f0546e4d6088c936": { "identity": "bc51b7d63696a428f0546e4d6088c936", "type": "PARENT_CHILD", "propagatorId": null, "extractorRunId": "NOT_STORED", "parent": { "entityId": "e461de8de38511a3ac6740dd7d51b8d0" }, "children": { "entityIds": [ "916b141fb9ce45094df8ef97ecdde41c" ] }, "unlinked": false, "propagatable": false }, "d9568c3415368847afaecd4b88478e3f": { "identity": "d9568c3415368847afaecd4b88478e3f", "type": "DATA_FLOW", "propagatorId": "0e1f4f973761cf05b3781934945b8098", "extractorRunId": "NOT_STORED", "targets": { "entityIds": [ "fc671272b2c7518ec738d6b17f08b7ed" ] }, "sources": { "entityIds": [ "d212538318276a6ad8abdd308a4487cc", "8c40fbf6b4584a0b213a257be2cae679" ] }, "unlinked": false, "propagatable": false }, "f505a6ea5efe8d0d58f61e2e19e09644": { "identity": "f505a6ea5efe8d0d58f61e2e19e09644", "type": "PARENT_CHILD", "propagatorId": null, "extractorRunId": "NOT_STORED", "parent": { "entityId": "e9b3d1f7aee29134338df3de4cae1100" }, "children": { "entityIds": [ "c864e9c3bc3f5f7a99315c56830e811d" ] }, "unlinked": false, "propagatable": false }, "da1b64977bca48ed1d4e0f74bbcacea7": { "identity": "da1b64977bca48ed1d4e0f74bbcacea7", "type": "PARENT_CHILD", "propagatorId": null, "extractorRunId": "NOT_STORED", "parent": { "entityId": "e9b3d1f7aee29134338df3de4cae1100" }, "children": { "entityIds": [ "2b5a90798459c168d9fb5ecc2fc46aa5" ] }, "unlinked": false, "propagatable": false }, "74f685545f94873904c8a5ca5b7c9eaa": { "identity": "74f685545f94873904c8a5ca5b7c9eaa", "type": "INSTANCE_OF", "propagatorId": null, "extractorRunId": "NOT_STORED", "instances": { "entityIds": [ "fc671272b2c7518ec738d6b17f08b7ed" ] }, "template": { "entityId": "57c993fc305b553dae82210090d5da7a" }, "unlinked": false, "propagatable": false }, "2e3c88897fcc1df2853fa4490c286207": { "identity": "2e3c88897fcc1df2853fa4490c286207", "type": "PARENT_CHILD", "propagatorId": null, "extractorRunId": "NOT_STORED", "parent": { "entityId": "e461de8de38511a3ac6740dd7d51b8d0" }, "children": { "entityIds": [ "8c40fbf6b4584a0b213a257be2cae679" ] }, "unlinked": false, "propagatable": false }, "c35093d842e090c17b3374ef22fe5383": { "identity": "c35093d842e090c17b3374ef22fe5383", "type": "PARENT_CHILD", "propagatorId": null, "extractorRunId": "NOT_STORED", "parent": { "entityId": "e9b3d1f7aee29134338df3de4cae1100" }, "children": { "entityIds": [ "4374529a34441bc8db30ee001bbb3563" ] }, "unlinked": false, "propagatable": false }, "29caebf3d0a7a8d38d9a4a49bb2c016c": { "identity": "29caebf3d0a7a8d38d9a4a49bb2c016c", "type": "DATA_FLOW", "propagatorId": "cf0c7b96382b834021f51638b135da9e", "extractorRunId": "NOT_STORED", "targets": { "entityIds": [ "2690183198a2d6568188426ea9a82793", "2b5a90798459c168d9fb5ecc2fc46aa5", "fe5445fbd070d97c418d96200a218cae", "4374529a34441bc8db30ee001bbb3563", "9d7ca9e46a9e4624df0d5d10949a5fc6", "66bff6eeac3c17c5b7eb7c9035704eef", "c864e9c3bc3f5f7a99315c56830e811d" ] }, "sources": { "entityIds": [ "fc671272b2c7518ec738d6b17f08b7ed" ] }, "unlinked": false, "propagatable": false }, "a2a35d94ba084495469290f8413e91ee": { "identity": "a2a35d94ba084495469290f8413e91ee", "type": "DATA_FLOW", "propagatorId": "cf0c7b96382b834021f51638b135da9e", "extractorRunId": "NOT_STORED", "targets": { "entityIds": [ "fe5445fbd070d97c418d96200a218cae", "2b5a90798459c168d9fb5ecc2fc46aa5", "2690183198a2d6568188426ea9a82793", "4374529a34441bc8db30ee001bbb3563", "66bff6eeac3c17c5b7eb7c9035704eef", "9d7ca9e46a9e4624df0d5d10949a5fc6" ] }, "sources": { "entityIds": [ "fc671272b2c7518ec738d6b17f08b7ed" ] }, "unlinked": false, "propagatable": false }, "3a817d057f264d972c8c0e8dda4f9c3e": { "identity": "3a817d057f264d972c8c0e8dda4f9c3e", "type": "PARENT_CHILD", "propagatorId": null, "extractorRunId": "NOT_STORED", "parent": { "entityId": "e9b3d1f7aee29134338df3de4cae1100" }, "children": { "entityIds": [ "2690183198a2d6568188426ea9a82793" ] }, "unlinked": false, "propagatable": false }, "e0dfacd87d398fdc8ffb91f30d675e43": { "identity": "e0dfacd87d398fdc8ffb91f30d675e43", "type": "PARENT_CHILD", "propagatorId": null, "extractorRunId": "NOT_STORED", "parent": { "entityId": "e9b3d1f7aee29134338df3de4cae1100" }, "children": { "entityIds": [ "66bff6eeac3c17c5b7eb7c9035704eef" ] }, "unlinked": false, "propagatable": false }, "cf0c7b96382b834021f51638b135da9e": { "identity": "cf0c7b96382b834021f51638b135da9e", "type": "DATA_FLOW", "propagatorId": null, "extractorRunId": "NOT_STORED", "targets": { "entityIds": [ "e9b3d1f7aee29134338df3de4cae1100" ] }, "sources": { "entityIds": [ "fc671272b2c7518ec738d6b17f08b7ed" ] }, "unlinked": false, "propagatable": false }, "16384c6acdb0c1f4fd81624ec5e00de2": { "identity": "16384c6acdb0c1f4fd81624ec5e00de2", "type": "PARENT_CHILD", "propagatorId": null, "extractorRunId": "NOT_STORED", "parent": { "entityId": "e9b3d1f7aee29134338df3de4cae1100" }, "children": { "entityIds": [ "9d7ca9e46a9e4624df0d5d10949a5fc6" ] }, "unlinked": false, "propagatable": false }, "3745ae37cf1da17592754b9aed5d6f21": { "identity": "3745ae37cf1da17592754b9aed5d6f21", "type": "PARENT_CHILD", "propagatorId": null, "extractorRunId": "NOT_STORED", "parent": { "entityId": "e461de8de38511a3ac6740dd7d51b8d0" }, "children": { "entityIds": [ "51db076ee11470b7f968bd5f33429e6b" ] }, "unlinked": false, "propagatable": false }, "0432c75f4c29379500dc51dc5887e196": { "identity": "0432c75f4c29379500dc51dc5887e196", "type": "PARENT_CHILD", "propagatorId": null, "extractorRunId": "NOT_STORED", "parent": { "entityId": "e461de8de38511a3ac6740dd7d51b8d0" }, "children": { "entityIds": [ "d212538318276a6ad8abdd308a4487cc" ] }, "unlinked": false, "propagatable": false }, "989443d7728154fb577d239ae0e0d677": { "identity": "989443d7728154fb577d239ae0e0d677", "type": "PARENT_CHILD", "propagatorId": null, "extractorRunId": "NOT_STORED", "parent": { "entityId": "e9b3d1f7aee29134338df3de4cae1100" }, "children": { "entityIds": [ "e5586cdbc89ecef7552e125acb91c2cf" ] }, "unlinked": false, "propagatable": false }, "3043bc0b3969889eaba53d74da1df08b": { "identity": "3043bc0b3969889eaba53d74da1df08b", "type": "DATA_FLOW", "propagatorId": "cf0c7b96382b834021f51638b135da9e", "extractorRunId": "NOT_STORED", "targets": { "entityIds": [ "fe5445fbd070d97c418d96200a218cae", "2b5a90798459c168d9fb5ecc2fc46aa5", "2690183198a2d6568188426ea9a82793", "4374529a34441bc8db30ee001bbb3563", "66bff6eeac3c17c5b7eb7c9035704eef", "9d7ca9e46a9e4624df0d5d10949a5fc6" ] }, "sources": { "entityIds": [ "fc671272b2c7518ec738d6b17f08b7ed" ] }, "unlinked": false, "propagatable": false }, "0e1f4f973761cf05b3781934945b8098": { "identity": "0e1f4f973761cf05b3781934945b8098", "type": "DATA_FLOW", "propagatorId": null, "extractorRunId": "NOT_STORED", "targets": { "entityIds": [ "fc671272b2c7518ec738d6b17f08b7ed" ] }, "sources": { "entityIds": [ "e461de8de38511a3ac6740dd7d51b8d0" ] }, "unlinked": false, "propagatable": false } } }Tracing through the relations shows that job_1396040412350_0003, which has the identity fc671272b2c7518ec738d6b17f08b7ed, has the INSTANCE_OF relation with wordcount and the DATA_FLOW relation with /user/hdfs/input and /user/hdfs/output.