This section calls out some important guidelines you should keep in mind when you start deploying models with Cloudera Machine Learning.
- Model Code
Models in Cloudera Machine Learning are designed to run any code that is wrapped into a function. This means you can potentially deploy a model that returns the result of a
SELECT *query on a very large table. However, Cloudera strongly recommends against using the models feature for such use cases.
As a best practice, your models should be returning simple JSON responses in near-real time speeds (within a fraction of a second). If you have a long-running operation that requires extensive computing and takes more than 15 seconds to complete, consider using batch jobs instead.
- Model Artifacts
Once you start building larger models, make sure you are storing these model artifacts in HDFS, S3, or any other external storage. Do not use the project filesystem to store large output artifacts.
In general, any project files larger than 50 MB must be part of your project's
.gitignorefile so that they are not included in Engines for Experiments and Models for future experiments/model builds. Note that in case your models require resources that are stored outside the model itself, it is up to you to ensure that these resources are available and immutable as model replicas may be restarted at any time.
- Resource Consumption and Scaling
Models should be treated as any other long-running applications that are continuously consuming memory and computing resources. If you are unsure about your resource requirements when you first deploy the model, start with a single replica, monitor its usage, and scale as needed.
If you notice that your models are getting stuck in various stages of the deployment process, check the Monitoring Active Models page to make sure that the cluster has sufficient resources to complete the deployment operation.
- Security Considerations
As stated previously, models do not impose any limitations on the code they can run. Additionally, models run with the permissions of the user that creates the model (same as sessions and jobs). Therefore, be conscious of potential data leaks especially when querying underlying data sets to serve predictions.
Cloudera Machine Learning models are not public by default. Each model has an access key associated with it. Only users/applications who have this key can make calls to the model. Be careful with who has permission to view this key.
Cloudera Machine Learning also prints
stdoutlogs from models to an output pane in the UI. Make sure you are not writing any sensitive information to these logs.
- Deployment Considerations
Models deployed using Cloudera Machine Learning Private Cloud are highly available subject to the following limitations:
- Model high availability is dependent on the high availability of the Kubernetes service. If using a third-party Kubernetes service to host CDP Private Cloud, please refer to your chosen provider for precise SLAs.
- In the event that the Kubernetes pod running either the model proxy service or the load balancer becomes unavailable, the Model may be unavailable for multiple seconds during failover.
There can only be one active deployment per model at any given time. This means you should plan for model downtime if you want to deploy a new build of the model or re-deploy with more or fewer replicas.
Keep in mind that models that have been developed and trained using Cloudera Machine Learning are essentially Python or R code that can easily be persisted and exported to external environments using popular serialization formats such as Pickle, PMML, ONNX, and so on.