Updating ML Runtime images on Cloudera Machine Learning installations
In Private Cloud ML Runtimes, the collection of docker images are bundled as part of the package. However, you can upgrade Runtime images to the latest version any time and even if you have an air-gapped installation, that is, there is no or only limited Internet access.
- Airgapped installations (with no or limited Internet access)
- Non-airgapped installations (with Internet access)
- Download ML Runtimes from the locations specified in the
repo-assembly.json
file:https://archive.cloudera.com/ml-runtimes/[***RUNTIMES VERSION***]/artifacts/repo-assembly.json
export ml_runtime_version=[***RUNTIME VERSION***] wget -O repo-assembly.json https://archive.cloudera.com/ml-runtimes/${ml_runtime_version}/artifacts/repo-assembly.json
- Get the list of images for a particular release, that is
${ml_runtime_version}
using the following command:curl https://archive.cloudera.com/ml-runtimes/${ml_runtime_version}/artifacts/repo-assembly.json | jq -r '.runtimes[] | .image_identifier'
- Ensure that docker is running on your machine or environment.
- Use the docker pull and docker save commands
for each of the images in the
.json
manifest file to create atar.gz
file, that you can copy to your air-gapped environment.# ensure you have enough space in image_dir image_dir=<path to image dir in non-airgapped environment> mkdir -p ${image_dir} for i in $(jq -r '.runtimes[].image_identifier' repo-assembly.json) do image=$(echo $i | awk -F'/' ' { printf("%s",$NF)} ') image_file="${image}.tar.gz" echo "docker pull ${i}" docker pull ${i} echo "docker save ${i} -o \"${image_dir}/${image_file}\"" docker save ${i} -o "${image_dir}/${image_file}" done
- Transfer the
tar.gz
files to the air-gapped environment. - Upload the images to the existing docker registry of the air-gapped environment.
- Use the docker load, docker tag and
docker push commands to save the images to the air-gapped
environment.
# you will need a copy of the repo-assembly.json used above remote_image_dir=<path to image dir in airgapped environment> DOCKER_REGISTRY_DEST=<custom docker repository:port> for i in $(jq -r '.runtimes[].image_identifier' repo-assembly.json) do image=$(echo $i | awk -F'/' ' { printf("%s",$NF)} ') image_file="${image}.tar.gz" dest_path=$(echo $i | sed "s#docker.repository.cloudera.com#${DOCKER_REGISTRY_DEST}#") docker load -i ${image_file} docker tag ${i} "${dest_path}" docker push "${dest_path}" done
- Make the newly added Runtime images available by following the instructions in Adding ML Runtimes using Runtime Repo files.
In non-airgapped
installations (with available Internet access) upgrade ML Runtimes by setting and enabling
the following options:
- Download the most recent manifest and populate the Runtime Catalog by selecting .
- Enable automated daily upgrades to download ML Runtime images by selecting the checkbox at Site AdministrationRuntimeEnable Runtime Updates.