CSP Architecture

Cloudera Stream Processing (CSP) provides advanced messaging, stream processing and analytics capabilities powered by Apache Kafka as the core stream processing engine. It also provides streams management capabilities. Two streams management services have been added for Kafka around monitoring and replication. Streams Messaging Manager (SMM) provides a single monitoring dashboard for a Kafka cluster. Streams Replication Manager (SRM) provides enterprises with the ability to implement cross-cluster Kafka topic replication.

CSP is fully integrated with the enterprise platform services including Cloudera Manager (CM) for cluster management and monitoring, Apache Sentry for role based authorization and a new Schema Registry service that provide governance and schema management capabilities.

CSP consists of the following components and features:

  • New Kafka monitoring services powered by Streams Messaging Manager (SMM) delivered via a parcel

  • New Kafka replication services powered by Streams Replication Manager (SRM) delivered via parcel

  • A Schema Registry parcel that provides a centralized registry to manage schema for Apache Kafka and NiFi

  • Support for Apache Kafka 2.2.0 through CDH 6.3.0 or CDK 4.1

  • Support for Kafka Streams

  • Cluster Management and monitoring with Cloudera Manager

  • Rich CDH cluster access control with Apache Sentry

CSP Parcels

CSP 2.0 was designed in modular form providing flexibility for the user to choose which services to install on CM managed cluster. CSP 2.0 provides two sets of capabilities: stream processing and streams management. Each set provides its own set of parcels.

CSP Stream Processing Parcels
  • Kafka from CDH 6.3.0 or the CDK 4.1 Parcel

  • Schema Registry Parcel

CSP Stream Management Parcels
  • SMM Parcel

  • SRM Parcel