Introduction to Cloudera Semantic Search [Technical Preview]

Overview

Cloudera Semantic Search integrates hybrid multimodal search functionality into the enterprise data estate. It serves as the retrieval foundation for autonomous AI agents and Retrieval-Augmented Generation (RAG) by transforming knowledge sources into searchable vector representations. This service maintains data gravity by enabling searches across internal documentation, support systems, and data lakehouses without transferring information outside the organization.

Core functionalities

The service includes the following primary features for managing vector data:

• Indexing: Mechanism for storing and searching high-dimensional vector data.
• K-NN search: Functionality to find the top documents semantically closest to an input query.
• Neural search: Extension of K-NN that supports semantic, multimodal, and hybrid searches using foundation models.

Retrieval-Augmented Generation workflow

The search functionality follows a structured process to enhance large language models with enterprise-specific context:

1. Indexing: An embedding model is selected to convert structured and unstructured data into k-NN vectors for indexing in the Cloudera Semantic Search service.
2. Querying: The same model generates an embedding for a user input query.
3. Retrieval: The search engine identifies documents semantically similar to the query embedding and returns the resultset.

Component roles

The service uses several node types to manage operations and data:

• Cluster_manager nodes: Manages cluster state, tracks node joins, and allocates shards.
• Data nodes: Store and search data while performing indexing and aggregation operations.
• Coordinating nodes: Delegate client requests to shards and aggregate final results.
• ML nodes: Execute machine learning tasks and resource calculations for AI models.
• Ingest nodes: Pre-process and transform data through an ingest pipeline before indexing.
• Dashboard nodes: Provide a visual interface to interact with and monitor search data.

Additional details

For more information on Cloudera Semantic Search, see Cloudera Semantic Search user guide for Cloudera on cloud environments, technical preview documentation.