Cloudera Docs

Setting up Cloudera Data Engineering to Cloudera Workflow Orchestrator Git migration [Technical Preview]

CMA can migrate Airflow DAGs from Cloudera Data Engineering virtual clusters into version-controlled Cloudera Workflow Orchestrator Git repositories. The migration produces a Pull Request (GitHub), Merge Request (GitLab), or branch (plain Git), which you review and merge — the source Cloudera Data Engineering environment is never modified.

Before migration engineers can use this feature, you must configure the CMA Agent with credentials for both the Cloudera Control Plane (Cloudera Data Engineering source) and the target Cloudera Workflow Orchestrator Git repository. CMA then automatically registers the required clusters.

When the CMA Agent starts with the required configuration, it automatically registers the following additional managed clusters in CMA Master:

  • Cloudera Data Engineering cluster (<clusterName>-cde) — Represents the Cloudera Data Engineering environment. The agent uses Cloudera Control Plane API credentials to discover Cloudera Data Engineering services, virtual clusters, and Airflow DAGs.

  • Git cluster (<clusterName>-git) — Represents the target Cloudera Workflow Orchestrator Git repository. The agent uses the repository URL and access token to scan the repository and create Pull Requests or Merge Requests.

Both clusters are displayed in the CMA Master UI and can be used as source and target in a migration plan.

Known limitations

  • No SSH support – Plain Git connections support only HTTPS repository URLs.

  • No incremental migration – Each migration execution creates a full Pull Request or Merge Request containing all mapped DAGs.

  • No automated rollback – Once a Pull Request or Merge Request is created, it must be manually closed or reverted in the Git provider.

  • Single branch per plan – All DAGs in a migration plan target one source branch (cma/<planName>).

  • You must deploy CMA in parcel deployment mode. For more information, see Deployment.

  • You must install and run the CMA Agent service on the cluster (Public Cloud or Cloudera platform Private Cloud Base).

  • The CMA Agent must have network access to the Cloudera Control Plane API endpoint.

  • The CMA Agent must have network access to Cloudera Data Engineering.

  • The CMA Agent must have outbound HTTPS access to the Git provider (GitHub, GitLab, or self-hosted).

  • You must have Cloudera Control Plane access key credentials (access key ID and private key) for the Control Plane.

  • You must have a personal access token (PAT) for the target Cloudera Workflow Orchestrator Git repository with permissions to read the repository and create Pull Requests or Merge Requests.

  • The target Cloudera Workflow Orchestrator Git repository must exist and the base branch, for example, main, must be present.

  • If a firewall is in place between CMA components and external services, the following ports must be open:

    Table 1. Network requirements
    Source Destination Port Protocol Purpose
    CMA Agent (Cloudera Data Engineering) Cloudera Control Plane API 443 HTTPS Discover Cloudera Data Engineering services and virtual clusters
    CMA Agent (Cloudera Data Engineering) Cloudera Data Engineering Knox gateway 443 HTTPS Fetch DAG metadata and download DAG files
    CMA Agent (Git) GitHub or GitLab or self-hosted Git 443 HTTPS Scan repository, create Pull Requests or Merge Requests
    CMA Agent (Cloudera Data Engineering and Git) CMA Master gRPC port 8091 gRPC Agent-to-Master communication
  1. Enable the cma.cde-git-migration-enabled flag.
    1. In the CMA Master UI, go to Settings.
    2. Click the Feature Flags tab.
    3. Locate the cma.cde-git-migration-enabled flag and set it to true.
    4. Click Save.
    5. Go to the CMA Agent service in Cloudera Manager.
    6. Click the Configuration tab.
    7. Search for CMA_AGENT_SERVER_role_env_safety_valve.
    8. Add the Cloudera Control Plane and Git repository environment variables listed in the tables in the next step.
    9. Click Save to save the advanced configuration snippet configuration.
    10. Restart the CMA Agent service.
    11. In the CMA Master UI, go to Clusters.
    12. Confirm that two new clusters are displayed. If the clusters are not displayed within a few minutes, check the CMA Agent log for registration errors.
  2. Configure the CMA Agent advanced configuration snippet in Cloudera Manager on the cluster where the CMA Agent is deployed:

    Cloudera Data Engineering and Git credentials are passed to the CMA Agent through the Cloudera Manager CMA_AGENT_SERVER_role_env_safety_valve advanced configuration snippet. These environment variables are read at startup. The agent uses them to register the Cloudera Data Engineering and Git clusters.

    The Cloudera Control Plane settings allow the CMA Agent to authenticate with the Cloudera Control Plane and the Cloudera Data Engineering Knox gateway.

    Table 2. Cloudera Control Plane environment variables
    Variable Description Example
    CMA_AGENT_CLUSTER_CLOUD_CONTROL_DATACENTER Cloudera Control Plane API endpoint URL. https://api.us-west-1.cdp.cloudera.com
    CMA_AGENT_CLUSTER_CLOUD_CONTROL_ACCESS_KEY_ID Cloudera Control Plane access key ID for Control Plane authentication. Obtained from the Cloudera Management Console in User Management > Access Keys. a1b2c3d4-...
    CMA_AGENT_CLUSTER_CLOUD_CONTROL_PRIVATE_KEY Cloudera Control Plane private key corresponding to the access key ID. <YOUR_CDP_PRIVATE_KEY>
    CMA_AGENT_CLUSTER_CLOUD_CONTROL_USERNAME Cloudera Data Engineering username. Must be an LDAP user. Used by the CMA Agent to authenticate against the Cloudera Data Engineering Knox gateway when fetching DAG metadata. admin
    CMA_AGENT_CLUSTER_CLOUD_CONTROL_PASSWORD Cloudera Data Engineering password corresponding to the username. <CDE_PASSWORD>

    The Cloudera Workflow Orchestrator Git repository settings allow the Git Agent to scan the target repository and create Pull Requests or Merge Requests.

    Table 3. Cloudera Workflow Orchestrator Git repository environment variables
    Variable Description Example
    CMA_AGENT_CLUSTER_GIT_REPOURL HTTPS URL of the target Git repository. SSH URLs are not supported. https://github.com/your-org/your-repo.git
    CMA_AGENT_CLUSTER_GIT_TOKEN Personal access token for the Git repository. For GitHub, the token requires the repo scope. For GitLab, it requires api or read_repository + write_repository. <YOUR_GIT_PAT>
    CMA_AGENT_CLUSTER_GIT_BRANCH Base branch name. CMA creates a source branch named cma/<planName> from this branch and opens a Pull Request or Merge Request back to it. Defaults to main if not set. main
  3. Save and restart the CMA Agent.

    After the CMA Agent restarts, it registers both clusters with CMA Master automatically. No manual steps are required in the CMA UI.

  4. Verify the cluster registration.
    Table 4. Verification values for auto-registered clusters
    Cluster Name Platform Label Purpose
    <clusterName>-cde Cloudera Data Services Cloudera Data Engineering source that is used for scanning DAGs
    <clusterName>-git Git Git target that is used for scanning the repository and creating Pull Requests and Merge Requests

CMA automatically selects the correct Git provider based on the repository URL. You do not need to configure the provider type manually in most cases.

Table 5. Git provider auto-detection rules
Detection rule Provider Change type
URL contains github.com GitHub Pull Request
URL contains gitlab.com GitLab Merge Request
HTTP probe responds to /api/v4/version GitLab (self-hosted) Merge Request
HTTP probe responds to /api/v3/meta GitHub Enterprise Pull Request
None of the above Plain Git Branch push

To override auto-detection, set the git.providerType cluster configuration property to one of GITHUB, GITLAB, or JGIT.

Table 6. Troubleshooting the migration
Symptom Possible cause Resolution
Cloudera Data Engineering or Git clusters are not displayed in the CMA UI CMA Agent did not restart, or advanced configuration snippet variables are missing or misspelled. Restart the CMA Agent service and check the Agent log for Cluster registration failed entries.
Cloudera Data Engineering scan fails with authentication error CMA_AGENT_CLUSTER_CLOUD_CONTROL_USERNAME or CMA_AGENT_CLUSTER_CLOUD_CONTROL_PASSWORD are incorrect, or the Cloudera Data Engineering Knox gateway is unreachable. Verify that the Cloudera Manager credentials are correct and that the Agent can reach the Cloudera Data Engineering Knox gateway on port 443.
Cloudera Data Engineering scan fails and no Cloudera Data Engineering are services found CMA_AGENT_CLUSTER_CLOUD_CONTROL_DATACENTER points to the wrong region, or the access key credentials are invalid. Verify the Control Plane URL and confirm that the access key is valid in the Cloudera Management Console.
Git scan fails with authentication error CMA_AGENT_CLUSTER_GIT_TOKEN is missing, expired, or lacks the required permissions. Regenerate the PAT in the Git provider and update CMA_AGENT_CLUSTER_GIT_TOKEN in the advanced configuration snippet.
Git scan fails, the branch is not found CMA_AGENT_CLUSTER_GIT_BRANCH refers to a branch that does not exist. Create the branch in the Git repository, or update the variable to point to an existing branch.
Cloudera Data Engineering cluster registered but no virtual clusters are displayed after the scan A specific virtual cluster could not be reached (Knox auth failure or empty vcApiUrl). Check the Cloudera Data Engineering scan command log in the CMA UI for SKIPPED entries with the affected virtual cluster name.