Configure SQL AI Assistant using vLLM

vLLM is an open-source library for LLM inference and serving. This topic describes how to set up SQL AI Assistant with a model hosted using vLLM.

To know more about the installation and its requirements, see the vLLM documentation.

Log in to the CDP Management Console as an Administrator.
Go to Environments and select your environment.
Go to the Data Lake tab and click on the CM URL to open Cloudera Manager.
Go to Clusters > Hue service > Configuration and select add the following lines in the Hue Service Advanced Configuration Snippet (Safety Valve) for hue_safety_valve.ini field:
```
[[ai_interface]]
  service='vllm'
  model_name='[**Place MODEL name here**]'
  base_url="https://[***RESOURCE***]/v1"
  token='[***API-KEY***]'
```
Click Save Changes.

You see

Assistant on the Hue SQL editor, and the SQL AI Assistant connects to the specified model hosted using vLLM.