CSP Architecture

Cloudera Stream Processing (CSP) provides advanced messaging, stream processing and analytics capabilities powered by Apache Kafka as the core stream processing engine. It also provides streams management capabilities. Two streams management services have been added for Kafka around monitoring and replication. Streams Messaging Manager (SMM) provides a single monitoring dashboard for a Kafka cluster. Streams Replication Manager (SRM) provides enterprises with the ability to implement cross-cluster Kafka topic replication.

CSP is fully integrated with the enterprise platform services including Cloudera Manager (CM) for cluster management and monitoring, Apache Sentry for role based authorization and a new Schema Registry service that provide governance and schema management capabilities.

CSP consists of the following components and features:

New Kafka monitoring services powered by Streams Messaging Manager (SMM) delivered via a parcel
New Kafka replication services powered by Streams Replication Manager (SRM) delivered via parcel
A Schema Registry parcel that provides a centralized registry to manage schema for Apache Kafka and NiFi
Support for Apache Kafka 2.2.0 through CDH 6.3.0 or CDK 4.1

note

It is strongly recommended that you use CDH 6.3.0 or later with Apache Kafka included. If you are using CDH 5.13.0, you may use CDK 4.1.0 or later.
Support for Kafka Streams
Cluster Management and monitoring with Cloudera Manager
Rich CDH cluster access control with Apache Sentry

CSP Parcels

CSP 2.0 was designed in modular form providing flexibility for the user to choose which services to install on CM managed cluster. CSP 2.0 provides two sets of capabilities: stream processing and streams management. Each set provides its own set of parcels.

CSP Stream Processing Parcels

Kafka from CDH 6.3.0 or the CDK 4.1 Parcel
Schema Registry Parcel

CSP Stream Management Parcels

SMM Parcel
SRM Parcel