Managing Low-Rank Adaptation adapters

Upload or delete Low-Rank Adaptation (LoRA) adapters in the Cloudera AI Registry S3 bucket for Cloudera AI Inference service to deploy custom model variants.

Cloudera AI Inference service expects fine-tuned Low-Rank Adaptation (LoRA) adapters stored in the Cloudera AI Registry S3 bucket under the peft-adapters/<adapter-name>/ directory structure.

Before uploading an adapter, ensure the followings:

You have access to a Cloudera AI session.
The AWS Command Line Interface (CLI) is installed in your environment.
Locate Cloudera AI Registry credentials

Cloudera AI Registry S3 credentials and endpoint details are available as environment variables in any Cloudera AI session running on the cluster. To locate your credentials, run the following command in a Cloudera AI session terminal:
```
env | grep -i MODEL_REGISTRY
```
The command returns an output, similar to the following example:
```
MODEL_REGISTRY_AWS_ACCESS_KEY_ID=<service-principal>
MODEL_REGISTRY_AWS_SECRET_ACCESS_KEY=<credential>
MODEL_REGISTRY_BUCKET=<bucket-name>
MODEL_REGISTRY_ENDPOINT=https://<ozone-host>:9879
MODEL_REGISTRY_REGION=auto
```

To ensure successful deployment, your local adapter directory must follow the required structure before uploading.

Table 1. Required and optional files for an adapter directory
File	Requirement	Description
adapter_config.json	Required	Contains PEFT configuration settings, including PEFT type, rank, base model, and target modules.
adapter_model.safetensors	Required	Contains LoRA weight matrices for A and B decompositions.
tokenizer_config.json	Optional	Specifies tokenizer settings. Cloudera recommends including this file if the adapter modified the base tokenizer.
tokenizer.json	Optional	Contains the full tokenizer vocabulary definitions.
special_tokens_map.json	Optional	Defines special token mappings for the tokenizer.

Do not include checkpoint directories, such as checkpoint-*/, training arguments, such as training_args.bin, optimizer states, such as optimizer.pt, backup files, such as *.bak, or README.md files in your adapter directory.

Export the Cloudera AI Registry environment variables to configure the AWS CLI:

export AWS_ACCESS_KEY_ID=$(env | grep MODEL_REGISTRY_AWS_ACCESS_KEY_ID | cut -d= -f2-)
export AWS_SECRET_ACCESS_KEY=$(env | grep MODEL_REGISTRY_AWS_SECRET_ACCESS_KEY | cut -d= -f2-)
export AWS_DEFAULT_REGION=auto
export S3_ENDPOINT=$(env | grep MODEL_REGISTRY_ENDPOINT | cut -d= -f2-)
export MR_BUCKET=$(env | grep MODEL_REGISTRY_BUCKET | cut -d= -f2-)

Upload your local adapter directory to the S3 bucket by running the aws s3 cp command:
```
aws s3 cp ./<adapter-dir>/ s3://${MR_BUCKET}/peft-adapters/<adapter-name>/ \
  --endpoint-url ${S3_ENDPOINT} \
  --no-verify-ssl \
  --region auto \
  --recursive \
  --exclude "*.bak" \
  --exclude ".git/*" \
  --exclude "checkpoint-*"
```
important
You must include the --recursive flag when uploading an adapter directory. Omitting the --recursive flag causes a stream is not seekable upload error.

Verify that the adapter files uploaded successfully by listing the destination S3 path:

aws s3 ls s3://${MR_BUCKET}/peft-adapters/<adapter-name>/ \
  --endpoint-url ${S3_ENDPOINT} \
  --no-verify-ssl \
  --region auto \
  --recursive

Deleting an adapter from Model Registry

If the adapter is no longer needed, delete the adapter from the Cloudera AI Registry S3 bucket.

To remove an adapter directory, run the aws s3 rm command:

aws s3 rm s3://${MR_BUCKET}/peft-adapters/<adapter-name>/ \
  --endpoint-url ${S3_ENDPOINT} \
  --no-verify-ssl \
  --region auto \
  --recursive