Cloudera Docs

Cloudera AI Inference service cannot discover AI Registry

Learn about scenarios or issues that may be resolved by using the steps in the Manually updating AI Registry configuration solution.

  • ML Serving installed before AI Registry

    If the AI Registry has not been created yet, when ML Serving is installed it will not be able to generate the ConfigMap.

  • AI Registry upgraded after ML Serving installed

    When the AI Registry is upgraded, it will require an updated ConfigMap.

  • AI Registry certificate expired
    After 90 days, AI Registry will have a certificate update, it will require an updated ConfigMap.

Manually updating AI Registry configuration

If you upgrade the AI Registry after creating your Cloudera AI Inference service cluster, the AI Registry configuration stored by the Cloudera AI Inference service will get out of synchronization. Follow the steps below to manually reconcile the configuration, so that Cloudera AI Inference service will be able to connect to AI Registry.

  1. Get KubeConfig for ML Serving Cluster.
    1. In a terminal where Cloudera Data Platform CLI is installed, run the cdp ml list-ml-serving-apps command.
    2. Find the appCrn of the ML Serving instance you wish to update.
    3. Retrieve the KubeConfig with this command:
      cdp ml get-ml-serving-app-kubeconfig --app-crn <YOUR-APP-CRN> |jq .kubeconfig|yq --prettyPrint

      Here:

      • cdp ml get-ml-serving-app-kubeconfig --app-crn <YOUR-APP-CRN> : returns raw json response containing KubeConfig.

      • |jq .kubeconfig: returns only the actual KubeConfig (removes json wrapper).

      • |yq --prettyPrint: prints the kKubeConfig response into a format both machine and user-readable.
    4. Direct the output to the standard KubeConfig location:
      cdp ml get-ml-serving-app-kubeconfig --app-crn <YOUR-APP-CRN> |jq .kubeconfig|yq --prettyPrint>~/.kube/config

      Make sure this is the active KubeConfig by running the kubectl config view command. The output of the above command shall now show the KubeConfig data returned above.

  2. Grant your user ML Serving App access.
    Add your cloud provider user identifier to the ML Serving App access control list. Use the following command, with your ARN, or other cloud provider user identifier.
    cdp ml grant-ml-serving-app-access --resource-crn crn:cdp:ml:us-west-1:9d74eee4-1cad-45d7-b645-7ccf9edbb73d:mlserving:2c982e74-c60e-43b9-be85-357cffb1d419 --identifier <Your Cloud Provider User ID>
  3. Get new ConfigMap data.
    1. Find the new AI Registry ConfigMap. Use the grant-model-registry-access API in Cloudera Data Platform CLI to add your user name to the new AI Registry, or use the UI, as shown:
    2. After your user ARN has been granted access to the AI Registry, get the ConfigMap data in the following way:
      • Download the AI Registry KubeConfig.
      • Set the AI Registry KubeConfig (save it to ~/.kube/config).
      • kubectl get cm -n mlx: lists all the available ConfigMaps.
      • kubectl describe cm jwks-rootca -n mlx: Returns the TLS Certificate for AI Registry.
      • cdp ml list-model-registries: The response will contain the domain for the updated AI Registry.
    3. Copy the entire rootca.pem output from the command.
      kubectl describe cm jwks-rootca -n mlx
      The domain of the new AI Registry is contained in the list-model-registries response:
  4. Apply ConfigMap Update.
    1. Update the KubeConfig back to the ML Serving KubeConfig.
    2. Edit the ConfigMap of the ML Serving Cluster:
      • kubectl edit cm modelregistry-config-controlplane -n serving: Update tls.crt with the data from above.

      • kubectl describe cm api-config -n serving: Update model.registry.url with the data from above.

    3. Restart the deployment to force the Cloudera Manager changes to take effect.
      • kubectl scale deployment api -n serving --replicas=0
      • kubectl scale deployment api -n serving --replicas=1