Cloudera Docs

Migrating Airflow DAGs from to a Cloudera Workflow Orchestrator Git repository [Technical Preview]

Export curated Cloudera Data Engineering Airflow DAGs and bundled resources into a Cloudera Workflow Orchestrator-managed Git repository with automated Pull Request or Merge Request generation without modifying the remaining Cloudera Data Engineering clusters.

This workflow automatically performs the following actions:

  • Opens GitHub/GitHub Enterprise Pull Requests, GitLab Merge Requests, or plain Git branch pushes reflecting your selections.

  • Places DAG code under /dags/<dagId>/ paths.

  • Stages supporting artifacts under /include/<service>/<vc>/<pool>/ prefixes mirrored from Cloudera platform namespaces.

Cloudera Data Engineering remains read-only for migrated assets. The target Git repository governs all future merges.

Administrators must ensure that the following conditions are met before migration:

  • Cloudera Data Engineering deployments must be displayed under the Cloudera Data Services badge and Git repositories as Git on the Clusters page.

  • The Cloudera Data Engineering-connected cluster must expose valid Cloudera Control Plane credential pairs.

  • Git clusters must advertise repository HTTPS URLs, personal access tokens, and base-branch names.

For operational setup instructions, see Setting up to Cloudera Workflow Orchestrator Git migration.

Step 1 — Scan the cluster

  1. Identify the cluster whose name suffix reads -cde and open it.
  2. Click Start Scanning.
  3. Select Scan Cloudera Data Engineering for Airflow.
  4. Click Scan selected and wait until the progress indicator displays success.

    Reading scan results

    Discovery displays CDE Service → Virtual Cluster → Airflow DAGs. Expand each branch to inspect individual items.

    Table 1. Discovery page columns
    Column Description
    DAG Name Cloudera Data Engineering job display name.
    DAG ID Unique ID (dag_id) inside the Python script.
    Cron Configured schedule expressions.
    Active Indicator of the DAG schedule status.
    Operators Inferred operators, such as BashOperator.
    Triggers Outbound trigger graph.
    Triggered By Inbound trigger graph.
    Resource Pools Cloudera Data Engineering resource pools allocated for mounted artifacts.

    You can use the following interface controls to change how the data is displayed:

    • Data – Switches to a file-centric tree, which is ideal before creating mappings.

    • Metadata — Switches to the logical view of DAG summaries.

    • List/tree toggle – Switches between hierarchical tree and flattened browsing list.

Step 2 — Scan the Git Repository

  1. Open clusters ending in -git.
  2. Click Start Scanning.
  3. Select Git Repository Scan, then click Scan selected.

    The repository tree displays Name and Size metadata. Empty repositories are acceptable because executions bootstrap directories.

Step 3 — Create a Migration Plan

  1. On the Plans tab, click Create Plan.
  2. Select a descriptive name aligned with Git branch naming rules (cma/<planName> prefixes created branches).
  3. Select the -cde source and -git target clusters.
  4. Select Cloudera Data Engineering DAGs to Git Repository as transition type, then finalize creation.

Step 4 — Label DAGs on the Collections tab

Collections act as a filter that controls which DAGs are visible on downstream configuration tabs.

  1. In the Plan tab, go to Collections.
  2. Expand the Cloudera Data Engineering hierarchies.
  3. Select DAG rows to migrate.
  4. Click Add to collection, or click the + icon to create a new collection.

Step 5 — Map DAG Metadata

  1. On the Metadata tab, review split panes listing Cloudera Data Engineering (left) and Git (right) DAG stubs.
  2. Map individually (Map) or bulk-select unfinished rows (Copy to Git).
  3. Defaults land at /dags/<dagId>/<dagId>.py plus /include/<service>/<vc>/<pool>/... artifact bundles.

    Unmap removes targets while preserving shared artifacts when safe.

Step 6 — Verify File Paths on the Data tab

  1. Go to Data to compare mirrored trees.
  2. Verify that DAG payloads and supporting files have the expected prefixes.

    If you notice any incorrect file paths, go to the Metadata tab to unmap or remap those files.

Step 7 — Review DAG Source (Workloads tab)

  1. Open Workloads whenever duplicate-ID badges block executions.
  2. Select DAG entries to load Monaco editors with read-only scanned copy displayed in the left pane, editable version in the right pane.
  3. Edit the right pane to rename dag_id, revise connections, or prune unsupported operators and save afterward.
  4. Ensure that warnings clear across every DAG affected by overlaps.

Step 8 — Execute Git Publishing

  1. In Execution, click Create Execution.
  2. Monitor the sequential pipeline as it automatically processes the following stages:

    1. Upload Cloudera Data Engineering resources – The CMA Agents retrieve scanned artifacts locally (skipped when unnecessary).

    2. Prepare Cloudera Data Engineering resources – The CMA Master merges edited DAG payloads with staging artifacts.

    3. Create Git change – The Git-role agents clone staging work, publish branch cma/<planName>, commit files, then open Pull Requests, or Merge Requests or fallback pushes depending on your Git configuration.

Step 9 — Inspect Provider Output

  1. Expand the completed Create Git Change rows to view direct links to your repository.
  2. Click the outbound links to open the generated Pull Requests or Merge Requests in your Git provider interface.

    Merged repositories carry DAG artifacts on tracked base branches and Cloudera Data Engineering workspaces continue governing runtime workloads.

For agent connectivity issues, duplication errors, or Git credential regressions, see Troubleshooting Airflow DAG migration to Git.