Cloudera Docs

Migrating Airflow DAGs from to Git end to end [Technical Preview]

Deploy Cloudera Migration Assistant, configure Cloudera Data Engineering and Git sources, and migrate DAGs to a Cloudera Workflow Orchestrator Git repository.

This end-to-end workflow combines deployment, platform preparation, discovery, and workload steps. Detailed reference procedures are reused from the Administrator and Migration guides.

Figure 1. Migration flow

Cloudera Migration Assistant migration flow

Verify supported platforms and migration paths in Supported platforms and migration paths before you start.

This workflow is technical preview. See Setting up to Cloudera Workflow Orchestrator Git migration.

Phase 1 — Deploy the CMA Master and CMA Agents

    1. Download both CSD files to the /opt/cloudera/csd/ directory on the Server host. 
      wget -P /opt/cloudera/csd/ https://archive.cloudera.com/cma/version/csd/CMA_MASTER-version-build.jar
      wget -P /opt/cloudera/csd/ https://archive.cloudera.com/cma/version/csd/CMA_AGENT-version-build.jar
      automatically detects the CSD files.
    2. Change the ownership of the CSD files. 
      chown cloudera-scm:cloudera-scm /opt/cloudera/csd/CMA_MASTER-version-build.jar /opt/cloudera/csd/CMA_AGENT-version-build.jar
    3. Restart for the changes to take effect. 
      systemctl restart cloudera-scm-server
    4. Log into .
    5. Restart the Cloudera Management Service.
    6. Go to Hosts > Parcels.
    7. Click Parcel Repositories & Network Settings.
    8. Add the Remote Parcel Repository URL for . 
      https://archive.cloudera.com/cma/version/parcels/
    9. Click Save & Verify Configuration to commit the change.
    10. Click Close.
      You are redirected to the Parcels page.
    11. Search for , and click Download to download the parcel to the local repository.
    12. After download completes, click Distribute to distribute the parcel to all clusters.
      Figure 2. Distribute the parcel

      Parcels page showing parcel distribution
    13. Click Activate to activate the parcel.
    14. Click OK when confirmation is required.
    15. Click Clusters in the left navigation pane.
    16. Select Add Service from the drop-down menu to the right of your cluster.
    17. From the list, select the service type to add, then click Continue. The single parcel provides two service types: CMA_MASTER (central control node) and CMA_AGENT (cluster-side executor).

      The Add Service wizard opens.

      Figure 3. Add service

      Add Service wizard showing service types
    18. Assign the service roles to the hosts where Python 3.11 and JDK 17+ are installed, and click Continue.
    19. Review service configurations and click Continue. At minimum, set OAuth2 client credentials (cma_client_secret, cma_agent_client_secret). If the is on a different cluster from the , also set Gateway URL (cma_gateway_url).

      The first run of the service starts. When the command finishes, the service is added to the cluster.

      Figure 4. service started

      showing service first run completed
    20. Repeat steps 16–19 to add the other service type if needed (for example, add CMA_AGENT after CMA_MASTER, or vice versa).
    21. Go back to the cluster homepage, open the service page, and click the Server UI tab to open the UI.
      Figure 5. landing page

      web UI landing page after parcel deployment
    22. Set the service dependency to none in the service configuration in Cloudera Manager because Cloudera Manager cannot discover the Master automatically across clusters.
    23. Set the Gateway URL to cma_gateway_url in the service configuration. 
      curl -sk https://<master-host>:8093/api/agent/config

    Phase 2 — Configure Cloudera Replication Manager on the target cluster

      1. Enable the HDFS to Ozone flag on the Server host.
        For HDFS to Ozone migrations, requires the CMF_FF_API_H2O_REPLICATION feature flag to be enabled. Without this flag, the API endpoints for HDFS to Ozone replication are not available.
        1. Append the following line to /etc/default/cloudera-scm-server on the Server host: 
          export CMF_FF_API_H2O_REPLICATION=true
        2. Restart the Server. 
          sudo systemctl restart cloudera-scm-server
        3. Wait for to become available again on port 7180 before proceeding with the setup.
      2. Register the instance in the UI to allow to locate and communicate with the replication service.
        1. In the Web UI, go to Settings.
        2. Go to the App Access section.
        3. Configure your connection fields using one of the following validation methods:

          • If has already detected a service on one of the registered clusters, select it from the Suggestions drop-down list to auto-fill the form.

          • Manually enter your target replication parameters in the following fields:

            • App URL — The full URL of the REST API, including protocol, host, and port, for example, https://rm-host.example.com:9011.

            • App username — The username for authenticating with the API.

            • App password — The password for authenticating with the API.

        4. Click Save and Push to Agents.

      Prepare Cloudera Data Engineering and Git sources (before Phase 2b)

        1. Enable the cma.cde-git-migration-enabled flag.
          1. In the UI, go to Settings.
          2. Click the Feature Flags tab.
          3. Locate the cma.cde-git-migration-enabled flag and set it to true.
          4. Click Save.
          5. Go to the service in .
          6. Click the Configuration tab.
          7. Search for CMA_AGENT_SERVER_role_env_safety_valve.
          8. Add the Cloudera Control Plane and Git repository environment variables listed in the tables in the next step.
          9. Click Save to save the advanced configuration snippet configuration.
          10. Restart the service.
          11. In the UI, go to Clusters.
          12. Confirm that two new clusters are displayed. If the clusters are not displayed within a few minutes, check the log for registration errors.
        2. Configure the advanced configuration snippet in on the cluster where the is deployed:

          and Git credentials are passed to the through the CMA_AGENT_SERVER_role_env_safety_valve advanced configuration snippet. These environment variables are read at startup. The agent uses them to register the and Git clusters.

          The Cloudera Control Plane settings allow the to authenticate with the and the Knox gateway.

          Table 1. Cloudera Control Plane environment variables
          Variable Description Example
          CMA_AGENT_CLUSTER_CLOUD_CONTROL_DATACENTER API endpoint URL. https://api.us-west-1.cdp.cloudera.com
          CMA_AGENT_CLUSTER_CLOUD_CONTROL_ACCESS_KEY_ID access key ID for Control Plane authentication. Obtained from the in User Management > Access Keys. a1b2c3d4-...
          CMA_AGENT_CLUSTER_CLOUD_CONTROL_PRIVATE_KEY private key corresponding to the access key ID. <YOUR_CDP_PRIVATE_KEY>
          CMA_AGENT_CLUSTER_CLOUD_CONTROL_USERNAME username. Must be an LDAP user. Used by the to authenticate against the Knox gateway when fetching DAG metadata. admin
          CMA_AGENT_CLUSTER_CLOUD_CONTROL_PASSWORD password corresponding to the username. <CDE_PASSWORD>

          The Git repository settings allow the Git Agent to scan the target repository and create Pull Requests or Merge Requests.

          Table 2. Git repository environment variables
          Variable Description Example
          CMA_AGENT_CLUSTER_GIT_REPOURL HTTPS URL of the target Git repository. SSH URLs are not supported. https://github.com/your-org/your-repo.git
          CMA_AGENT_CLUSTER_GIT_TOKEN Personal access token for the Git repository. For GitHub, the token requires the repo scope. For GitLab, it requires api or read_repository + write_repository. <YOUR_GIT_PAT>
          CMA_AGENT_CLUSTER_GIT_BRANCH Base branch name. creates a source branch named cma/<planName> from this branch and opens a Pull Request or Merge Request back to it. Defaults to main if not set. main
        3. Save and restart the .

          After the restarts, it registers both clusters with automatically. No manual steps are required in the UI.

        4. Verify the cluster registration.
          Table 3. Verification values for auto-registered clusters
          Cluster Name Platform Label Purpose
          <clusterName>-cde Cloudera Data Services source that is used for scanning DAGs
          <clusterName>-git Git Git target that is used for scanning the repository and creating Pull Requests and Merge Requests

        Configure user connections

        Add per-user Cloudera Data Engineering and Git credentials in the UI. See Managing user connections.

        Phase 3 — Register source and target clusters

          1. In on the source cluster, select Add Service and select CMA_AGENT.
          2. Configure the required properties.
            Table 4. configuration properties
            Property Description
            cma_gateway_url URL of the Gateway, for example, https://master-host:8093. Required when the is on a different cluster than the . The agent uses this URL to discover the Master host, gRPC port, and Auth Server URI automatically through the discovery endpoint (GET /api/agent/config). When connecting from a different cluster, set the service dependency to none in .
            cm_username administrator user name.
            cm_password administrator password.
            cma_user_home Home directory for the cma-agent user, for example, /var/lib/cma-agent.
          3. Set cma_gateway_url when the is on a different cluster than the .
          4. Start the service.
          5. Wait for the agent to connect to the and finish the registration automatically.

          Phase 4 — Scan clusters

            1. Navigate to the Clusters page and click the cluster you want to scan.
              Figure 6. Clusters page

              Clusters page
            2. On the cluster detail page, find the Scanning section, which lists the available scan types for your cluster. Click the configuration icon on the right side of the scan type you want to trigger.
              Figure 7. cluster scanning

              cluster scanning
            3. In the scan configuration drawer, configure the scan parameters (for example, scan path, scan type, scan depth) and click Scan Selected.
              Figure 8. scan configuration

              scan configuration
            4. The scan starts and the status changes to Initializing. Track scan progress by clicking the Commands tab, which shows a badge with the number of active commands.
              Figure 9. scan in progress

              scan in progress
            5. On the Commands tab, expand a command to view its Output Log, Error Log, and Configuration.
              Figure 10. scan commands

              scan commands

            Phase 5 — Create collections (optional)

              1. Click the + icon next to Collections in the sidebar.
              2. In the Collection Details drawer, enter a Collection name and select a Color for identification.
                Figure 11. create collection

                create collection
              3. Click Create.

                The new collection appears in the sidebar with 0 items.

              4. Browse the data tree to find the items you want to migrate. Click folder names to navigate deeper into the hierarchy.
              5. Select items using the checkboxes next to each item.
              6. Click the collection icon that appears on the selected item row.
              7. In the Add/Remove Collections popup, check the collections you want to assign the item to.
              8. Click Apply.
                Figure 12. assign to collection

                assign to collection
              9. Click a collection name in the sidebar to view its contents. The data browser switches to show only the items assigned to that collection.
                Figure 13. view collection

                view collection

              Phase 6 — Run the migration workload

                Step 1 — Scan the cluster

                1. Identify the cluster whose name suffix reads -cde and open it.
                2. Click Start Scanning.
                3. Select Scan for Airflow.
                4. Click Scan selected and wait until the progress indicator displays success.

                  Reading scan results

                  Discovery displays CDE Service → Virtual Cluster → Airflow DAGs. Expand each branch to inspect individual items.

                  Table 5. Discovery page columns
                  Column Description
                  DAG Name job display name.
                  DAG ID Unique ID (dag_id) inside the Python script.
                  Cron Configured schedule expressions.
                  Active Indicator of the DAG schedule status.
                  Operators Inferred operators, such as BashOperator.
                  Triggers Outbound trigger graph.
                  Triggered By Inbound trigger graph.
                  Resource Pools resource pools allocated for mounted artifacts.

                  You can use the following interface controls to change how the data is displayed:

                  • Data – Switches to a file-centric tree, which is ideal before creating mappings.

                  • Metadata — Switches to the logical view of DAG summaries.

                  • List/tree toggle – Switches between hierarchical tree and flattened browsing list.

                Step 2 — Scan the Git Repository

                1. Open clusters ending in -git.
                2. Click Start Scanning.
                3. Select Git Repository Scan, then click Scan selected.

                  The repository tree displays Name and Size metadata. Empty repositories are acceptable because executions bootstrap directories.

                Step 3 — Create a Migration Plan

                1. On the Plans tab, click Create Plan.
                2. Select a descriptive name aligned with Git branch naming rules (cma/<planName> prefixes created branches).
                3. Select the -cde source and -git target clusters.
                4. Select DAGs to Git Repository as transition type, then finalize creation.

                Step 4 — Label DAGs on the Collections tab

                Collections act as a filter that controls which DAGs are visible on downstream configuration tabs.

                1. In the Plan tab, go to Collections.
                2. Expand the hierarchies.
                3. Select DAG rows to migrate.
                4. Click Add to collection, or click the + icon to create a new collection.

                Step 5 — Map DAG Metadata

                1. On the Metadata tab, review split panes listing (left) and Git (right) DAG stubs.
                2. Map individually (Map) or bulk-select unfinished rows (Copy to Git).
                3. Defaults land at /dags/<dagId>/<dagId>.py plus /include/<service>/<vc>/<pool>/... artifact bundles.

                  Unmap removes targets while preserving shared artifacts when safe.

                Step 6 — Verify File Paths on the Data tab

                1. Go to Data to compare mirrored trees.
                2. Verify that DAG payloads and supporting files have the expected prefixes.

                  If you notice any incorrect file paths, go to the Metadata tab to unmap or remap those files.

                Step 7 — Review DAG Source (Workloads tab)

                1. Open Workloads whenever duplicate-ID badges block executions.
                2. Select DAG entries to load Monaco editors with read-only scanned copy displayed in the left pane, editable version in the right pane.
                3. Edit the right pane to rename dag_id, revise connections, or prune unsupported operators and save afterward.
                4. Ensure that warnings clear across every DAG affected by overlaps.

                Step 8 — Execute Git Publishing

                1. In Execution, click Create Execution.
                2. Monitor the sequential pipeline as it automatically processes the following stages:

                  1. Upload resources – The retrieve scanned artifacts locally (skipped when unnecessary).

                  2. Prepare resources – The merges edited DAG payloads with staging artifacts.

                  3. Create Git change – The Git-role agents clone staging work, publish branch cma/<planName>, commit files, then open Pull Requests, or Merge Requests or fallback pushes depending on your Git configuration.

                Step 9 — Inspect Provider Output

                1. Expand the completed Create Git Change rows to view direct links to your repository.
                2. Click the outbound links to open the generated Pull Requests or Merge Requests in your Git provider interface.

                  Merged repositories carry DAG artifacts on tracked base branches and workspaces continue governing runtime workloads.