Configure SQL AI Assistant using vLLM

vLLM is an open-source library for LLM inference and serving. This topic describes how to set up SQL AI Assistant with a model hosted using vLLM.

To know more about the installation and its requirements, see the vLLM documentation.

Log in to the Cloudera Management Console as an Administrator.
Go to Environments and select your environment.
Go to the Data Lake tab and click on the CM URL to open Cloudera Manager.
Go to Clusters > Hue service > Configuration and select add the following lines in the Hue Service Advanced Configuration Snippet (Safety Valve) for hue_safety_valve.ini field:
```
[[ai_interface]]
service='vllm'
model_name='[**Place MODEL name here**]'
base_url="https://[***RESOURCE***]/v1"
token='[***API-KEY***]'
```
note
To secure a vLLM endpoint by using a self-signed or internal CA certificate, you cannot configure file paths inside the hue_safety_valve.ini file. Instead, you must set the CURL_CA_BUNDLE environment variable to point to the CA certificate file, such as /etc/hue/models/cert.pem, through the Hue Service Environment Advanced Configuration Snippet (Safety Valve) field in Cloudera Manager.
Click Save Changes.

You see

Assistant on the Hue SQL editor, and the SQL AI Assistant connects to the specified model hosted using vLLM.