Getting started
Learn the core concepts of the service, including vector indexing, neural search, retrieval-augmented generation workflows, and component roles.
Core functionalities
The service includes the following primary features for managing vector data:
- Indexing: Mechanism for storing and searching high-dimensional vector data.
- K-NN search: Functionality to find the top documents semantically closest to an input query.
- Neural search: Extension of K-NN that supports semantic, multimodal, and hybrid searches using foundation models.
Retrieval-Augmented Generation workflow
The search functionality follows a structured process to enhance large language models with enterprise-specific context:
- Indexing: An embedding model generates vector representations of documents, images, or video frames.
- Querying: The same model generates an embedding for a user input query.
- Retrieval: The search engine identifies documents semantically similar to the query embedding and returns the resultset.
Component roles
The service uses several node types to manage operations and data:
- Cluster_manager nodes: Manages cluster state, tracks node joins, and allocates shards.
- Data nodes: Store and search data while performing indexing and aggregation operations.
- Coordinating nodes: Delegate client requests to shards and aggregate final results.
- ML nodes: Execute machine learning tasks and resource calculations for AI models.
- Ingest nodes: Pre-process and transform data through an ingest pipeline before indexing.
- Dashboard nodes: Provide a visual interface to interact with and monitor search data.
Additional details
For more information on Cloudera Semantic Search, see Cloudera Semantic Search user guide for Cloudera on premises environments, technical preview documentation.
