Running Models on GPU
Follow the guidelines for running Models on GPU.
To run a Model Endpoint on GPU, specify the number of GPUs to use per model replica, as
follows. You must specify the GPU count as a string.
# cat ./examples/mlflow/model-spec-cml-registry.json
{ "namespace": "serving-default", "name": "mlflow-wine-test-from-registry-onnx", "source": { "registry_source": { "version": 1, "model_id": "yf0o-hrxq-l0xj-8tk9" } }, "resources": { "num_gpus": "1" } }
The resources field supports specifying the following properties:
- req_cpu - the requested number of CPU cores, which is a string as per Kubernetes standards.
- req_memory - the requested amount of memory, which again follows Kubernetes conventions.
- num_gpus - the requested number of GPUs.
