Code-based Lineage Connector

Supported file types

The following file types are supported:

• Upload (Discovery display): python file (.py), scala file (.scala)
• Fully Analyzed (All modules supported - AI parsed): python file (.py), scala file (.scala)

Tool permissions prerequisites

Enable read permission for Cloudera Data Lineage Windows NT user to the folder (which contains the code-based files).

Setting up Code Based Lineage metadata source

Metadata sources are set on the Octopai Client.

1. From the Octopai Client, click the Code Based Lineage connector.

   
   
   

2. In the New Metadata Source wizard, choose the authentication method between a Local Folder or a Github Folder, and enter the appropriate information:
   • Local Folder: Choose the Local Folder option if you want to upload files from a local directory for data lineage analysis.
   • Github Folder: Choose the Github Folder option if you want to connect to your Github repository to extract the required files for data lineage analysis.
   • Local Folder
   • Github Folder

   1. Connection Name: Provide a meaningful name for this connection to help you easily identify it later in the Cloudera Data Lineage application.
   2. Tool Name: Specify the tool or language environment you are using. For example, enter "Python" or "Scala" depending on the language you are uploading.
   3. Source Folder: Click the blank field to select the folder containing the files you want to upload.
   4. Exclude Files or Strings: Enter file names, suffixes, strings, or patterns you want to exclude from the export (for example, test scripts, local configuration files).
   5. Exclude Folders: Specify folder names to exclude from the export if you have multiple subfolders (for example, venv, __pycache__, .git).

   
   
   

   1. Connection Name: Provide a meaningful name for this connection to help you easily identify it later in the Cloudera Octopai application.
   2. Tool Name: Specify the tool or language environment you are using. For example, enter "Python" or "Scala" depending on the language you are uploading.
   3. Personal Access Token (PAT): Enter your Github Personal Access Token to authenticate and authorize the connector to access your repository.
   4. Repository (owner/repo): Enter the repository name in the format 'owner/repository-name'.
   5. Branch: Specify the branch within the repository from which you want to extract files (for example, 'main').
   6. Folder path inside the repository: Provide the specific path to the directory within the repository that contains the files you want to include.
   7. Exclude Files or Strings: Enter file names, suffixes, strings, or patterns you want to exclude from the export (for example, test scripts, local configuration files).
   8. Exclude Folders: Specify folder names to exclude from the export if you have multiple subfolders (for example, venv, __pycache__, .git).

   
   
   

3. Click Next > Finish > Run.