Textual Files or Folder connector

The Textual Files or Folder connector allows you to extract file metadata from directories. You can set this connector up by configuring metadata sources and validating permissions.

Supported file types

The following file types are supported:

  • Upload (Discovery Display): text file, CSV file, SQL file, log file, XML file, JSON file, bash file, output file, BTEQ file, BTQ file, control file, report file, Perl file, KornShell file, C# file, Siddhi file
  • Fully Analyzed (All Modules Supported): SQL file, BTEQ file, BTQ file, control file, Siddhi file

Limitations

The following limitations apply to the Textual Files or Folder connector:

  • Archive files, including .zip and other archive formats, are not supported. The connector does not extract archives, and nested archives (for example, a .zip inside another .zip) are also not extracted.
  • Only files placed directly in the configured Source Folder are analyzed. Files that remain inside an archive are ignored, even if the archive itself is in the source folder.
  • Uploading a .zip or other archive through the Cloudera Data Lineage (formerly Octopai) portal does not trigger connector processing. The archive must be fully extracted, and the individual files placed in the source folder, before the connector runs.

Tool Permissions Prerequisites

Enable read permission for Cloudera Data Lineage Windows NT User to the folder (which contains the textual files).

Setting up Textual Files Metadata Source

Metadata Sources are set on the Octopai Client

  1. From the Octopai Client, click the Folder connector.
  2. In the New Metadata Source wizard, choose the authentication method between a Local Folder or a Github Folder, and enter the appropriate information:
    • Local Folder: Choose the Local Folder option if you want to upload files from a local directory for data lineage analysis.
    • Github Folder: Choose the Github Folder option if you want to connect to your Github repository to extract the required files for data lineage analysis.
    1. Connection Name: Provide a meaningful name for this connection to help you easily identify it later in the Cloudera Data Lineage application.
    2. Tool Name: Specify the tool or language environment you are using. For example, enter "JSON" if you are uploading JSON scripts.
    3. Source Folder: Click the blank field to select the folder containing the files you want to analyze. The folder must contain individual, extracted files of a supported type. Archives (for example, .zip files) and nested archives are not extracted. Extract them first and place the raw files (for example, .sql) directly in this folder. Remove any archives and unrelated files from the source folder.
    4. Exclude Files or Strings: Enter file names, suffixes, strings, or patterns you want to exclude from the export.
    5. Exclude Folders: Specify folder names to exclude from the export if you have multiple subfolders.

    New metadata source wizard to choose local folder option for Folder connector
    1. Connection Name: Provide a meaningful name for this connection to help you easily identify it later in the Cloudera Data Lineage application.
    2. Tool Name: Specify the tool or language environment you are using. For example, enter "JSON" if you are uploading JSON scripts.
    3. Personal Access Token (PAT): Enter your Github Personal Access Token to authenticate and authorize the connector to access your repository.
    4. Repository (owner/repo): Enter the repository name in the format 'owner/repository-name'.
    5. Branch: Specify the branch within the repository from which you want to extract files (for example, 'main').
    6. Folder path inside the repository: Provide the specific path to the directory within the repository that contains the files you want to include. The path must contain individual, extracted files of a supported type — archives (for example, .zip files) are not extracted.
    7. Exclude Files or Strings: Enter file names, suffixes, strings, or patterns you want to exclude from the export.
    8. Exclude Folders: Specify folder names to exclude from the export if you have multiple subfolders.

    New metadata source wizard to choose Github folder option for Folder connector
  3. Click Next > Finish > Run.

Troubleshooting

The connector run completes successfully but returns no scripts (empty result). This usually means the Source Folder did not contain any individually extracted, supported files at the time of the run. To resolve this issue:

  1. Confirm the source folder contains individual files of a supported type (for example, .sql) and not .zip or other archives. Fully extract any archives, including nested archives, and place the raw files directly in the source folder.
  2. Remove archives and unrelated files from the source folder.
  3. Verify that the Cloudera Data Lineage Windows NT user has read permission on the Source Folder and its contents (see Tool Permissions Prerequisites).
  4. Rerun the connector.