Cloudera Docs

Cloudera Machine Learning API v2

Cloudera Machine Learning exposes a REST API that you can use to perform operations related to projects, jobs, and runs. You can use API commands to integrate Cloudera Machine Learning with third-party workflow tools or to control Cloudera Machine Learning from the command line.

API v2 supersedes the existing Jobs API. For more information on the Jobs API, see Jobs API in the Related information section, below.

How to view the API Specification

You can view the comprehensive API specification on the REST API v2 Reference page. See Related information, below, for the link.

You can also obtain the specification of the available API commands directly from Cloudera Machine Learning. In a browser, enter the following addresses:
  • REST API: https://<domain name of Cloudera Machine Learning instance>/api/v2/swagger.html
  • Python API: https://<domain name of Cloudera Machine Learning instance>/api/v2/python.html
You can also get json formatted output, by specifying swagger.json.

Quickstart

API key authentication

To get started, generate an API key. The API key is a randomly generated token that is unique to each user. It must be treated as highly sensitive information because it can be used to start jobs via the API. You need this API key to use in API calls.
  1. Sign in to Cloudera Machine Learning.
  2. In User Settings > API Keys, click Create API Key.
  3. Copy this API key to the clipboard.

Using curl from the command line

To use the curl command, it is convenient to store the domain and API key in environmental variables, as shown here:
  1. Copy the API key.
  2. Open a terminal, and store it to a variable. On unix-based systems:

    export API_KEY=<paste the API key value here>

  3. Copy the domain name, which is in the address bar of the browser. On unix-based systems: export CDSW_DOMAIN=<domain> (a value like: ml-xxxx123456.com).

Example commands

If you have some projects, jobs, and runs already set up in your Cloudera Machine Learning Workspace, here are some commands to try:
  • List available projects: 
    curl -X GET -H "authorization: Bearer $API_KEY" https://$CDSW_DOMAIN/api/v2/projects | jq

    You can format the output for readability by piping through jq, a formatting utility.

  • You can filter the output like so: 
    curl -X GET -H “authorization: Bearer $API_KEY” https://$CDSW_DOMAIN/api/v2/projects?searchFilter=demo |  jq

    The output is limited to those projects that have the word “demo” in them.

    You can also paginate the output, for example by limiting each page to two projects. To do this, replace the string starting from the ‘?’ character with this: ?pageSize=2

    The output ends with a next_page_token and a string value. To get the next page use this: ?pageSize=2&pageToken=<token>

Using the Python client library

To use the Python API in your own code, first install the Python API client and point it to your cluster.

pip3 install https://$CDSW_DOMAIN/api/v2/python.tar.gz

Include the following code, and specify the values for <CDSW_DOMAIN> and <API_KEY> with variables or values for your installation.

# In a session:
    api_instance = default_client()
   # Outside a session:
   default_client("https://"+cluster, APIKEY)

Then you can use commands in your script, such as a call to list projects:

projects = api_instance.list_projects()

The API returns objects that store values as attributes. To access the values, use dot notation. Do not use bracket notation as you would with a dictionary. For example:

myproj = client.create_project(...)

# This doesn't work:
myproj["id"]

# But this does
myproj.id

Check the Python documentation for a full list of the available API commands.

Using the Python client library in the REPL

Here is an example of a stub Python script that contains the environmental variables for your installation. Save it to your computer and run it locally to get a Python prompt for API commands.

demo.py

import clap
import argparse
    
parser = argparse.ArgumentParser(description=‘Test the generated python package.’)
parser.add_argument(“—host”, type=str, help=‘The host name of your workspace”)
parser.add_argument(“—token”, type=str, help=‘Your API key”)
args = parser.parse_args()
    
config = clap.Configuration()
config.host = ars.host
client = cmlapi.ApiClient(config)
client.set_default_header(“authorization”, “Bearer “ + args.token)
api = cmlapi.Apiapi(client)
Run the script from your command line: 
python3 -i demo.py —host https://$CDSW_DOMAIN —token $API_KEY
This returns a Python prompt with api available. You can run api calls from the prompt, as follows: 
>>> api
<cmlapi.api.api_api.ApiApi object at 0xlasjoid>
>>> api.api_list_projects()
You can specify a search filter, such as the following: 
api.api_list_projects(searchFilter=‘demo’)
     
api.api_list_projects(page_size=2)
     
api.api_list_projects(page_size=2, page_token=‘<token value>’)