Cloudera Search and Other Cloudera Components

Cloudera Search interacts with other Cloudera components to solve different problems. The following table lists Cloudera components that contribute to the Search process and describes how they interact with Cloudera Search:

Component Contribution Applicable To
HDFS Stores source documents. Search indexes source documents to make them searchable. Files that support Cloudera Search, such as Lucene index files and write-ahead logs, are also stored in HDFS. Using HDFS provides simpler provisioning on a larger base, redundancy, and fault tolerance. With HDFS, Cloudera Search servers are essentially stateless, so host failures have minimal consequences. HDFS also provides snapshotting, inter-cluster replication, and disaster recovery. All cases
MapReduce Search includes a pre-built MapReduce-based job. This job can be used for on-demand or scheduled indexing of any supported data set stored in HDFS. This job utilizes cluster resources for scalable batch indexing. Many cases
Flume Search includes a Flume sink that enables writing events directly to indexers deployed on the cluster, allowing data indexing during ingestion. Many cases
Hue Search includes a Hue front-end search application that uses standard Solr APIs. The application can interact with data indexed in HDFS. The application provides support for the Solr standard query language, visualization of faceted search functionality, and a typical full text search GUI-based. Many cases
ZooKeeper Coordinates distribution of data and metadata, also known as shards. It provides automatic failover to increase service resiliency. Many cases
HBase Supports indexing of stored data, extracting columns, column families, and key information as fields. Because HBase does not use secondary indexing, Cloudera Search can complete full-text searches of content in rows and tables in HBase. Some cases
Cloudera Manager Deploys, configures, manages, and monitors Cloudera Search processes and resource utilization across services on the cluster. Cloudera Manager helps simplify Cloudera Search administration, but it is not required. Some cases
Oozie Automates scheduling and management of indexing jobs. Oozie can check for new data and begin indexing jobs as required. Some cases
Impala Further analyzes search results. Some cases
Hive Further analyzes search results. Some cases
Avro Includes metadata that Cloudera Search can use for indexing. Some cases
Sqoop Ingests data in batch and enables data availability for batch indexing. Some cases
Mahout Applies machine-learning processing to search results. Some cases