Cloudera Docs

Model import limitations with Cloudera AI Registry

Model import processing in Cloudera AI Registry operates on a first-come, first-served queue with a maximum of two concurrent execution slots and no automatic timeout mechanism.

When you import machine learning models, the system translates each request into a Kubernetes job. To prevent resource exhaustion, the platform restricts backend concurrent job processing while permitting unlimited request submissions in the pending queue.

Understanding these queueing behaviors and scheduling constraints helps platform administrators and model developers manage model deployment workflows effectively.

Concurrency and queue behavior

Model import workflows operate under the following execution and queueing constraints:

  • Two concurrent import limit: Although there is no limitation to importing models at a time, the backend executes a maximum of two parallel import jobs simultaneously. Additional import requests remain in a pending state until an execution slot becomes available.
  • Unrestricted queue capacity: The system does not enforce a limit on the number of import jobs that can exist in a pending state. Users can trigger multiple concurrent import requests without encountering submission errors.
  • Indefinite pending duration: Queued jobs remain in a pending state indefinitely until execution slots or required system resources become available, or until an administrator or user manually terminates the job.

Scheduling and queue ordering

Cloudera AI Inference manages job scheduling and queue ordering through the following operational rules:

  • First-come, first-served processing: The system processes import jobs strictly in the order of submission. The platform does not enforce fairness controls or per-user submission quotas, meaning a single user submitting multiple imports can occupy both execution slots and the pending queue.
  • Equal job priority: The scheduler does not differentiate between new model imports and retries of failed import requests. Both request types launch as identical Kubernetes jobs without assigned priority classes.
  • No size-based prioritization: The system does not prioritize smaller model imports over larger model imports. Import requests execute based solely on submission order regardless of model size or resource requirements.

Resource allocation and timeout constraints

Model import execution relies on underlying compute and storage availability:

  • Indefinite resource waiting: If the required volume storage (PVC), CPU, or memory resources are unavailable, the job remains in a pending state rather than failing or timing out.
  • Absence of execution timeouts: Long-running or stalled import tasks do not encounter an automated timeout and continue to run until completion or manual termination.

Status reporting and log retention

Monitoring model import jobs requires distinguishing between the user interface (UI) state and backend pod telemetry:

  • UI status representation: The user interface displays an Uploading status for jobs in a backend pending state while they await execution slots or infrastructure resources. To view the true operational state of an import job, inspect the pod logs.
  • Log retention timeline: Detailed import execution logs are retained in the Kubernetes pod for approximately five days. The system persists only the final error message in the database for long-term troubleshooting.

Storage allocation and model size limitations

Model import execution depends on underlying Persistent Volume Claim (PVC) storage allocation:

  • Fixed PVC storage limit for job-based imports: The system allocates a fixed 500 GB Persistent Volume Claim (PVC) per job-based model import request. Dynamic PVC sizing and pre-validation checks are deferred and not included in this release.
  • 500 GB threshold for job-based imports: For standard job-based model imports, ensure the model’s on-disk footprint is 500 GB or less to prevent import job failures due to volume exhaustion.
  • Air-gap tooling requirement for models larger than 500 GB: Models with an on-disk footprint larger than 500 GB must be imported using the air-gap tooling workflow (or an equivalent offline import procedure) to bypass the fixed PVC size constraint.