Cloudera Docs

Adding ML Runtimes using Runtime Repo files

Runtime Repo files are JSON files that contain all the details of ML Runtimes that are needed by Cloudera AI to add these ML Runtimes to the Runtime Catalog. When these files are hosted on URLs accessible to Cloudera AI, these URLs can be registered in Cloudera AI.

If Enable Runtime Updates is selected in Site Administration > Runtime, ML Runtimes that appear in the Runtime Repo files will be automatically added to the Runtime Catalog within 24 hours. Additionally, you can click Update Runtimes now to immediately update the Runtime Catalog. Using Runtime Repo files, both Custom and Cloudera provided Runtimes can be added to Cloudera AI Workbench.

Cloudera provided Runtime Repo files

Cloudera AI hosts two public Runtime Repo files that contain the details of the latest released ML Runtimes and Data Visualization Runtimes.

Default repositories (latest Runtimes only)

By default, Cloudera AI Workbenches are registered with Cloudera-hosted repos pointing to the latest released runtime versions:
Name: Cloudera ML Runtimes
URL: https://archive.cloudera.com/ml-runtimes/latest/artifacts/repo-assembly.json
Name: Cloudera DataViz Runtimes
URL: https://archive.cloudera.com/cdv/latest/artifacts/repo-assembly.json

These Runtime Repo files are not added to Cloudera AI on premises clusters by default. However, if your cluster is not air gapped, you can add these Runtime Repo files to your Cloudera AI Workbench to keep your Runtime Catalog automatically updated with the latest released Cloudera Runtimes. Note, that these Runtimes will be added to your Runtime Catalog from Cloudera hosted registries, even if your on premises is configured to use a different docker registry.

If your cluster is air-gapped, you can still add new Cloudera-built ML Runtimes to your Cloudera AI Workbench by pushing these Runtimes into a docker registry that is hosted within your air-gapped environment, and then adding a Runtime Repo file to Cloudera AI. The image_identifier fields in the file must point to the self-hosted docker registry instead of any Cloudera-hosted registries.

Adding specific or older Cloudera Runtime versions

If your environment or workload requires a specific or older version of a Cloudera ML Runtime that is not available in the latest repository, you can manually register a version-specific Runtime Repo URL

To register a specific runtime version:
  1. Obtain the required release version string (for example, 2024.10.1).
  2. Construct the version-specific URL using the following format:

    https://archive.cloudera.com/ml-runtimes/<VERSION>/artifacts/repo-assembly.json

    Example for the 2024.10.1 release:

    https://archive.cloudera.com/ml-runtimes/2024.10.1/artifacts/repo-assembly.json

  3. In the Cloudera console, click the Cloudera AI tile.

    The Cloudera AI Workbenches page displays.

  4. Click on the workbench name.

    The Workbenches Home page displays.

  5. Click Site Administration in the left navigation menu.

    The Site Administration page displays.

  6. Navigate to Runtime.

  7. Add the version-specific URL to the Runtime Repo files list in the Runtime Updates section.

  8. Click Update Runtimes now to immediately fetch and add the specific runtime version to your Runtime Catalog.

Managing Runtime Repo files

To add, edit, or remove Runtime Repo files

  1. Log in as administrator.
  2. Navigate to Site Administration > Runtime.
  3. Add, edit or delete Runtime Repo files in the Runtime Updates section.

Self created Runtime Repo files

You can create your own Runtime Repo files and register them in Cloudera AI. Cloudera AI checks these Runtime Repo files for changes every 24 hours and adds any new ML Runtimes found in these files automatically to the Runtime Catalog.

To create a Repo assembly file:

  1. Create a JSON file with the same structure as the Cloudera provided one: 
    {
    "assembly_metadata_version": 1,
    "runtimes": [
         { 
           "image_identifier": string,
            "runtime_metadata_version": int,
            "editor": string,
            "edition": string,
            "description": string,
            "kernel": string,
            "full_version": string,
            "short_version": string,
            "maintenance_version": int,
            "git_hash": string,
            "gbn": int
         } , 
    ]
}
  2. Fill in the details of one or more ML Runtimes. If you are adding Cloudera-created Runtimes, use the values from the Cloudera-provided Runtime Repo files. For Custom Runtimes, git_hash should be an empty string, and gbn should be set to zero. All other fields should be filled according to the information in Metadata for Custom ML Runtimes.
  3. Host the JSON file on an URL that Cloudera AI is able to access.
  4. Add the Runtime Repo file to Cloudera AI on the Site Administration > Runtime page.