Distributing the Machine Learning Runtime Image
Select a method to distribute a custom Machine Learning Runtime to all the hosts.
Once you have built a new custom Machine Learning Runtime, use one of the following methods to distribute the new image to all your Cloudera Machine Learning hosts:
-
Option 1 - Push the image to a public registry such as DockerHub.
This option is recommended for user-created runtime images without any proprietary code or settings. Typically, this is the quickest and easiest way to start using custom runtimes. For details on pushing image to Docker registry, see Docker image push.
-
Option 2 - Push the image to your company's Docker registry.
You can install your own custom docker registry by following the CNCF Distribution - Distribution Registry. Also see, Creating an internal secure Docker Registry for CDSW/CML runtimes.
- Tag your image with the following schema:
docker tag [***IMAGENAME***] [***COMPANY REGISTRY***]/[***USERNAME***]/[***IMAGENAME***]:[***TAG***]
- Once the image has been tagged properly, push the image with the following
command:
docker push [***COMPANY REGISTRY***]/[***USERNAME***]/[***IMAGENAME***]:[***TAG***]
- Add your docker registry certificate to Cloudera Machine Learning if necessary.
- Tag your image with the following schema:
-
Option 3 - Push the image to the Cloudera Embedded Container Service docker registry.
This is not recommended for production environment.
It is possible to use the Cloudera Embedded Container Service docker registry that the rest of the your Cloudera Machine Learning cluster uses for the internal and system images:
- Find the host where the registry is running. Run this command on any
host:
ls -al /etc/docker/certs.d
This will contain an entry for the host running the registry:[root@host-1.company.com ~]$ ls -al /etc/docker/certs.d/ total 0 drwxr-xr-x 4 8536 8536 101 Mar 4 13:23 . drwxr-xr-x 3 root root 37 Jul 28 2023 .. drwx------ 2 8536 8536 26 Mar 4 13:23 docker-private.infra.cloudera.com drwxr-xr-x 2 8536 8536 40 Mar 4 13:23 host-2.company.com:5000
- View the Kubernetes secret, which contains the username and password for this
registry:
[root@host-1.company.com ~]$ kubectl get secret cdp-private-installer-docker-pull-secret -n [***CML worspace namespace***] -o jsonpath="{.data.\.dockerconfigjson}" | base64 -d {"auths":{[***AUTH***]:{[***USERNAME***],[***PASSWORD***],[***AUTH***]}}}
- Tag your custom Runtime against the registry, log into the registry, and push
the image:
[root@host-1.company.com ~] docker tag username/imagename:1 host-2.company.com:5000/username/imagename:1 [root@host-1.company.com ~] docker login host-2.company.com:5000 Username: [***USERNAME***] Password: [***PASSWORD***] [root@host-1.company.com ~] docker push host-2.company.com:5000/username/imagename:1
- Follow the documentation in Add Docker registry
credentials and certificates to add both the certificate (in this
example, from
/etc/docker/certs.d/host-2.company.com:5000/ca.crt
) and the username and password to Cloudera Machine Learning.
- Find the host where the registry is running. Run this command on any
host: