Cloudera Docs

Adding or updating custom operators and libraries using API

You can add or update custom python packages for Airflow with Cloudera Data Engineering (CDE). Cloudera provides access to the open source packages that you can use for your Airflow jobs using the API.

While you can install the operator, if additional runtime dependencies are required such as additional setup with binaries on the path, environment configuration like Kerberos and Cloud credentials, and so on, then the operator will not work.
  1. Start a maintenance session by calling the /admin/airflow/env maintenance. It is a synchronous call and the session is created upon success. 
    curl -X POST \
"https://pmjkrgn5.cde-czlmkz4y.na-01.xvp2-7p8o.cloudera.site/dex/api/v1/admin/airflow/env/maintenance" \
-H "Authorization: Bearer ${CDE_TOKEN}" \
-H "accept: application/json"
  2. Create and define the pip repositories by calling the /admin/airflow/env/maintenance/repos. This is a sync operation.

    The JSON payload to create a job is structured as follows:
    {
"extraPipRepositories": [ 
	{
  	"caCerts": "string",  
  	"credential": {
    	  "password": "string",
    	  "username": "string"
  	},
  	"skipCertValidation": false,
  	"url": "string"
	}
  ],
  "pipRepository": {
	"caCerts": "string",
	"credential": {
  	  "password": "string",
  	  "username": "string"
	},
	"skipCertValidation": false,
	"url": "string"
  }
}
    For example: 
    curl -X POST \
"https://pmjkrgn5.cde-czlmkz4y.na-01.xvp2-7p8o.cloudera.site/dex/api/v1/admin/airflow/env/maintenance/repos" \
-H "Authorization: Bearer ${CDE_TOKEN}" \
-H "accept: application/json" \
-H "Content-Type: application/json" \
-d '{
  "pipRepository": {
	"url": "https://pypi.org/simple/"
  }
}' 

curl -X GET \
"https://pmjkrgn5.cde-czlmkz4y.na-01.xvp2-7p8o.cloudera.site/dex/api/v1/admin/airflow/env/maintenance/status" \
-H "Authorization: Bearer ${CDE_TOKEN}" \
-H "accept: application/json"
  3. Build the Python environment by calling /admin/airflow/env/maintenance/build.
    1. To monitor results, issue GET /admin/airflow/env/maintenance/status.
      Required build starting states Success state Failure state

      pip-repos-defined

      built

      build-failed

      		 built build-failed
      The payload to create a job for a multi-part/form input is as follows:
      # requirements.txt should be a file on the local file system and a well formatted python requirements.txt
                        
curl -X POST \
"https://pmjkrgn5.cde-czlmkz4y.na-01.xvp2-7p8o.cloudera.site/dex/api/v1/admin/airflow/env/maintenance/build" \
-H "Authorization: Bearer ${CDE_TOKEN}" \
-H "accept: application/json" \
--form file="@requirements.txt"  
                        
curl -X GET \
"https://pmjkrgn5.cde-czlmkz4y.na-01.xvp2-7p8o.cloudera.site/dex/api/v1/admin/airflow/env/maintenance/status" \
-H "Authorization: Bearer ${CDE_TOKEN}" \
-H "accept: application/json"
  4. Activate the Python environment by calling /admin/airflow/env/maintenance/activate.
    1. To monitor results, issue GET /admin/airflow/env/maintenance/status.
      Required activation starting states Success state Failure state
      built

      activation-failed

      N/A, request should fail with 404, since maintenance is done

      /admin/airflow/env/status should have the “activated” status and the description of your environment should be correct and updated

      		activation-failed
      For example:
      curl -X POST \
"https://pmjkrgn5.cde-czlmkz4y.na-01.xvp2-7p8o.cloudera.site/dex/api/v1/admin/airflow/env/maintenance/activate" \
-H "Authorization: Bearer ${CDE_TOKEN}" \
-H "accept: application/json"
                          
curl -X GET \
"https://pmjkrgn5.cde-czlmkz4y.na-01.xvp2-7p8o.cloudera.site/dex/api/v1/admin/airflow/env/maintenance/status" \
-H "Authorization: Bearer ${CDE_TOKEN}" \
-H "accept: application/json"

      You can now use your DAGs with custom code.