Cloudera Docs

Evaluating generated datasets for fine-tuning LLMs

Learn how to evaluate synthetic data generated in previous steps using the Large Language Model as the evaluator.

  1. In the Cloudera console, click the Cloudera AI tile.

    The Cloudera AI Workbenches page displays.

  2. Click on the name of the workbench.

    The workbenches Home page displays.

  3. Click AI Studios.
  4. Under AI Studios, click Synthetic Data Studio.
  5. Under Evaluate, click Getting Started. The Evaluator page is displayed.
  6. In Display Name, provide a name for the evaluation result.
  7. In Prompt, enter your custom prompt, or click Restore Default Prompt to reset to the default prompt.
  8. In Model Provider, select either AWS Bedrock or Cloudera AI Inference as a model provider.
  9. In Model, select your desired model for the evaluation.
  10. In Evaluation Examples, update the justification text and score provided by the LLM as needed.
  11. Under Parameters, modify the model parameters for Temperature, Top K, and Top P to customize the model’s behavior.
  12. Click Evalute to start the evaluation.

    After completing the evaluation, you can access the generated evaluation report. Click on the Actions column to re-evaluate the dataset, view the evaluation results, or remove the evaluation report.