Considerations for Trino Auto-scaling and Auto-suspend

Risk of query failure

When auto-suspend and/or auto-scaling are enabled, Trino worker pods may shut down automatically, even if queries are still running. During a scale-down event, the system enforces a strict time limit based on the configured grace period. If the grace period expires while a worker is still processing a query, the worker pod is forcibly terminated, regardless of whether the task is complete. As a result, any active queries running on that worker will fail and cannot be automatically recovered.

Workaround: Increase the value of the Grace Period for Shutting Down Worker property. Set this value to exceed the duration of your longest expected query to ensure the worker has sufficient time to complete its tasks before being terminated.

Asynchronous scaling up and scaling down processes

Both scaling up and scaling down are asynchronous processes. This means there is a delay between the system making a decision and the action actually completing.

When the conditions for scaling up are met, a scaling request is sent to the Kubernetes cluster to initiate the creation of a new worker node. This process continues even if all queries are completed or aborted in the meantime. As a result, Trino Virtual Warehouses may continue scaling up even when idle, such as when all queries have been completed or aborted.

Similarly, the scale-down process is also asynchronous and may experience delays.

Temporary lock after manual operations

Trino's auto-suspend and auto-scaling processes do not account for manual operations performed by users, such as Start or Update. To prevent conflicts between manual and automatic operations, a 1-minute delay is introduced, during which the lease of the Trino Virtual Warehouse is held after a manual operation is executed.

During this delay, attempting another manual operation may result in the error message: "Compute entity is currently ‘leased’ by another internal operation."

Workaround: This behavior is intentional and ensures system stability. Wait for 1 minute after completing a manual operation before initiating another.

Metadata Retrieval during Hue Session and auto-suspended cluster

Unlike Hive or Impala, Trino uses coordinators to process metadata requests (like SHOW TABLES) and occasionally uses worker nodes for parallel processing when the metadata is large, for example, Iceberg file metadata. The coordinator creates splits of this metadata and assigns them to worker nodes for parallel processing. Workers retrieve their assigned metadata "splits" from the connector/source, execute operations such as filtering and joining metadata, and return the results to the coordinator. The coordinator then combines the metadata fragments received from the workers and delivers the final, consolidated result (for example, a list of tables) to the client.

When a Hue session starts, Hue sends requests to the Trino coordinator to fetch database metadata. For large metadata tables, the coordinator divides the metadata request into "splits" and assigns them to worker nodes. However, if the Trino Virtual Warehouse is auto-suspended, all worker nodes are stopped, causing the metadata request to be queued. This triggers the auto-start process to spin up the worker nodes. Since it can take a few minutes to start worker nodes in a Kubernetes cluster, Hue may take a long time to load metadata for an auto-suspended Trino Virtual Warehouse.

Workaround: Disable auto-suspend for the Trino Virtual Warehouse and enable auto-scaling with a minimum of one worker node to ensure at least one worker is always available.