Cloudera Docs

Custom Lineage Connector for ETLs

Use the Custom Lineage Connector (formerly Universal Connector) Links CSV template to describe ETL processes, tasks, and column-level data flows so that Cloudera Data Lineage can show lineage for custom or unsupported ETL and orchestration tools.

The Cloudera Data Lineage platform supports data movement and transformation scenarios across many tools. The Custom Lineage Connector Links CSV template integrates metadata from custom or unsupported ETL and orchestration tools beyond native connectors. You can build a more complete view of your data ecosystem, with data lineage, discovery, and catalog content aligned and connected with the rest of your environment.

This guide explains the Custom Lineage Connector Links CSV structure. It complements database object metadata. When you also need tables and columns defined outside the ETL graph, see Custom Lineage Connector for Database Objects.

How to use the ETLs template file

  1. Download the template file: Cloudera Data Lineage Custom Lineage Connector Links
  2. Fill in the required fields using the table below.
Column Name Description Required
Process Name Name of the process that wraps the task, for example “Workflow” in Informatica or “Package” in SSIS No
Process Path Path of the process – for example, the path where the SSIS package is stored, including the package name and suffix (aaa\bbb\ccc\Package Name.dtsx). No
Process Type The type of process – job, map, package, and so forth. Yes
Process Description Short process description to be identified clearly in the lineages. No
Task Name The task name – the atomic unit that holds the data flow within the process. Yes
Task Path The path of the task – the location of the atomic unit that runs the process (for example, aaa\bbb\ccc\Package Name\container\Task Name). No
Source Provider Name The type of database that the source object connects to (for example, Oracle or SQL Server). See Supported script parsing providers. No
Source Component The name of the logic component in the ETL tool. Example: for Informatica, the name of the aggregator in the map. When there is no component, enter the table name. No
Source Server Server name of the source object. No
Source Database Database name of the source object. Yes (optional when Source Sql is populated and specifies the database)
Source Schema Schema name of the source object. Yes (optional when Source Sql is populated and specifies the schema)
Source Object Name of the source object. Leave empty when Source Sql is populated; any value entered is ignored. Yes (leave empty when Source Sql is populated)
Source Column When Source Sql is populated, the column name parsed from the source SQL that maps to the target column on the same row. Otherwise, the column name in the source object. Yes
Source Data Type Data type of the column. No
Source Precision Precision of the column. No
Source Scale Scale of the column. No
Source Type Type of source object, such as Table or View. Optional when you use a source SQL script; use this field only to distinguish components that share the same name. Required when Source Sql is not populated. Yes (optional when Source Sql is populated)
Source Sql The SQL query that retrieves the source data. When populated, Cloudera Data Lineage parses the script regardless of values in other source fields. If the script specifies schema, database, or server explicitly, those values take precedence. Otherwise, populated template columns are required and used. No
Target Provider Name The type of database that the target object connects to (for example, Oracle or SQL Server). See Supported script parsing providers. No
Target Component Name of the logic component in the ETL tool. Example: for Informatica, the name of the aggregator in the map. When there is no component, enter the table name. No
Target Server Server name of the target object. No
Target Database Database name of the target object. Yes
Target Schema Schema name of the target object. Yes
Target Object Name of the target object. Yes
Target Column Column name in the target object. Yes
Target Data Type Data type of the column. No
Target Precision Precision of the column. No
Target Scale Scale of the column. No
Target Object Type The type of target object, such as Table, View, or Stored Procedure (SP). Yes
Expression Formula or transformation between source column and target column. No
Link Type DataFlow or ImpactAnalysis. No (default = DataFlow)
Link Description Documentation about the link. No (default = empty string)

Supported script parsing providers

Supported parsing providers for scripts:

  • DB2
  • HIVE
  • IMPALA
  • MSSQL
  • SYNAPSE (MSSQL)
  • NETEZZA
  • ORACLE
  • POSTGRESQL
  • REDSHIFT
  • TERADATA
  • SNOWFLAKE
  • VERTICA
  • BIGQUERY
  • HANA

Example for ETL process on cross-system lineage

The Custom Lineage Connector links the source and the target for the task name as the main object.