Cloudera Docs

Installing Streams Messaging Manager on an existing CDP Private Cloud Base cluster

Learn how to install Streams Messaging Manager (SMM) on an already running CDP Private Cloud Base cluster.

Overview

The following sections guide you through the process of installing SMM on an already running cluster. Installing SMM involves adding the service to the cluster and enabling optional integration between SMM and other services like Schema Registry and Streams Replication Manager (SRM), or service roles like Kafka Connect.

Integration with other services

SMM is capable of integrating with other services. Enabling integration enables additional features and functions on the SMM UI. For example, enabling integration with SRM enables the Cluster Replications section in SMM, which you can use to monitor SRM replications.

If other services that SMM integrates with are located on the same cluster as SMM, enabling the integration happens when you add SMM to your cluster. However, SMM can integrate with some services, like SRM, even if the service is located on a remote cluster. In this case, the integration between SMM and the remote service is configured after you add SMM to your cluster. Additionally, the integration with other services can be enabled, disabled, or further configured separately following installation. As a result, the service integration steps are also documented as separate tasks.

Deployment recommendations

The SMM service is comprised of two service roles, which are as follows:

  • Streams Messaging Manager Rest Admin Server: This role is responsible for processing Rest API requests from the SMM UI and collecting metrics information from Cloudera Manager and Kafka.
  • Streams Messaging Manager UI Server: This role provides the SMM UI. You use the UI to manage and monitor Kafka and related services.

Cloudera recommends that you add the SMM service to a management host on your cluster. Cloudera does not recommend adding the SMM service to hosts where Kafka broker roles are running. Each cluster can only have a single instance of the SMM service deployed on it. In addition, the two SMM roles must be installed on the same host.

Cloudera Manager and Cloudera Service Monitor requirements

SMM requires high levels of Cloudera Manager and Service Monitor Service (SMON) memory usage. Ensure that both are tuned according to Cloudera recommendations.

Prometheus

By default SMM uses Cloudera Manager as its metrics store. This means that the metrics that are available for monitoring on the SMM UI are sourced from Cloudera Manager. If required, you can create a deployment where instead of Cloudera Manager, SMM uses Prometheus as its metrics store. Prometheus is an open-source systems monitoring and alerting toolkit that stores its metrics as time series data.

Prometheus supports a significantly larger number of time series entities compared to Cloudera Manager. As a result, Cloudera recommends that you use Prometheus instead of Cloudera Manager if you need to monitor a large number of Kafka entities (topics, partitions and so on). Additionally, using Prometheus enables you to configure the roll up policy, delete specific time series entities, and configure scrape intervals and metrics retention periods. Prometheus setup for SMM is done after adding SMM service to your cluster.

Adding the Streams Messaging Manager service to an existing cluster

Learn how to add the Streams Messaging Manager (SMM) service to an already running cluster.

The following list of steps walk you through how you can add SMM on an already existing CDP Private Cloud Base Cluster. The following steps are aimed to provide you with the basic process of installation and do not go into detail how you can set up or configure security for the service.

  • Ensure that a Kafka service is installed on your cluster. Additionally, ensure that the Enable Producer Metrics Kafka service property is selected. This property is enabled by default.
  • SMM requires a backend database to function. The database is used to store Kafka metadata, metrics, and alert definitions. As a result, a database and database user must be set up for SMM before adding the service. The following SQL snippet gives an example of the setup that you are required to do.
    create database streamsmsgmgr;
CREATE USER 'streamsmsgmgr'@'localhost' IDENTIFIED BY 'streamsmsgmgr';
GRANT ALL PRIVILEGES ON streamsmsgmgr.* TO 'streamsmsgmgr'@'localhost' WITH GRANT OPTION;
CREATE USER 'streamsmsgmgr'@'%' IDENTIFIED BY 'streamsmsgmgr';
GRANT ALL PRIVILEGES ON streamsmsgmgr.* TO 'streamsmsgmgr'@'%' WITH GRANT OPTION;
    For more information on database setup as well as supported databases. Review the following resources:
  • SMM is capable of establishing a secure (encrypted) connection with its database if the database has TLS 1.2/TCPS enabled. This is done by specifying the database connection using a JDBC URL. If you plan on using an encrypted connection, ensure that you have a valid JDBC URL on hand and have completed additional prerequisites. For more information and example JDBC URLs, see Database setup details for Streams Messaging Manager for TLS 1.2/TCPS-enabled databases.
  1. In Cloudera Manager, select the cluster that you want to install SMM on.
  2. Click Actions > Add Service.
  3. Select Streams Messaging Manager from the list of services and click Continue.
  4. Select service dependencies.
    The options on the Select Dependencies page differ depending on what other services are available on your cluster. Review the following and select the dependencies based on your requirements.
    Schema Registry
    If selected, integration between SMM and Schema Registry will be enabled. Schema Registry related features become available on the SMM UI. This is a recommended optional dependency.
    Streams Replication Manager
    If selected, integration between SMM and SRM will be enabled. SRM integration enables the Cluster Replications section on the UI, which you use to monitor replications. This is a recommended optional dependency.
    Kafka
    SMM requires a Kafka service as a dependency. Kafka services will only be visible on this page if there are multiple Kafka services available on the cluster. Otherwise, if only a single Kafka is available, SMM automatically selects it as a dependency as Kafka is a mandatory dependency.
  5. Click Continue.
  6. Assign role instances to hosts.
    On the Assign Roles page you select which host you want to install SMM roles on. Because both SMM roles must be installed on the same host, you can only select a host for the Rest Admin Server role. The UI Server role is installed on the same host.

    A host is selected by default. If required you can change the host by doing the following:

    1. Click the field below Streams Messaging Manager Rest Admin Server to display a dialog containing a list of hosts.
    2. Select a single host and click Ok.
  7. Click Continue.
  8. Set up the database connection.
    The database that you specify here is the one that you already set up for SMM prior to installation.
    1. Select the database type and enter the database name, username, and password.
    2. Click Test Connection.
      If the connection test fails, review your configuration, fix any errors, and rerun the connection test. You can only continue if the validation check is successful.
  9. Click Continue.
  10. Review changes.
    The Review Changes pages lists default and suggested settings for several configuration properties. Review and if required configure these properties.
  11. Click Finish.

The SMM service is added to your cluster, however, it is not yet started.

Depending on your requirements, you have multiple options on how to continue. Choose one of the following.

  • Start the SMM service. To do this, click Actions > Start on the homepage of the SMM service.
  • Learn more about or enable integration with other services and service roles. Continue with service specific integration steps.
  • Set up Prometheus as the metrics store. Continue with Setting up Prometheus for Streams Messaging Manager.

Integrating Schema Registry with Streams Messaging Manager

Learn how to configure Streams Messaging Manager (SMM) to connect and integrate with Schema Registry. Enabling integration between the two services enables Schema Registry related functionality within the SMM UI.

SMM is capable of connecting to and integrating with a Schema Registry service. Enabling integration makes it possible for the SMM UI Data Explorer to use schemas stored in Schema Registry to deserialize Kafka messages. Additionally, the Data Explorer will include a quick access link to the schema associated with the topic you are viewing.

  • In most cases Schema Registry integration is enabled when you add the SMM service to the cluster. As a result, it is highly likely that integration is already enabled. If integration is enabled, you do not need to complete the following steps.
  • SMM can only integrate with a Schema Registry service if the service is co-located with SMM. Specifying remote Schema Registry services is not possible.
  1. In Cloudera Manager, select the SMM service.
  2. Go to Configuration.
  3. Find and select the Schema Registry Service property.
  4. Click Save Changes.
  5. Restart SMM.
Integration between SMM and Schema Registry is enabled. Schema Registry related functionality is now available in the Data Explorer of the SMM UI.

Integrating Streams Replication Manager with Streams Messaging Manager

Learn how to configure Streams Messaging Manager (SMM) to connect and integrate with Streams Replication Manager (SRM). Enabling integration between the two services enables SRM related functionality within the SMM UI.

SMM is capable of integrating with SRM. Enabling integration between the two services enables the Cluster Replications section within the SMM UI. You can use this section of the UI to monitor the replication flows you create with SRM.

Service integration is enabled in either of two ways. If SRM is located on the same cluster as SMM, you integrate the two services by enabling a service dependency. If the two services are located remote to each other (on different clusters), you need to manually specify the host, port, and protocol of one or multiple SRM Service role instances that SMM should interface with.

  • Ensure that a co-located or remote SRM service is available.
  • If SRM is co-located with SMM, it is likely that integration between the two services was enabled when you added SMM to your cluster. Check if the Cluster Replications section is available in the SMM UI. If it is, you do not need to complete the following steps.
  1. In Cloudera Manager, select the SMM service.
  2. Go to Configuration.
  3. Enable integration between the two services.
    How you complete this step depends on whether SRM is colocated with SMM. Use a service dependency if the two services are colocated. Otherwise, specify the connection using dedicated properties.
    1. Find and select the STREAMS_REPLICATION_MANAGER Service property.
    1. Find and select the Configure Streams Replication Manager Manually property.
    2. Find and configure the Streams Replication Manager Rest Protocol property.

      Select http or https depending on configuration of SRM. If Enable TLS/SSL for SRM Service is selected, select https, otherwise, select http.

    3. Find and configure the Streams Replication Manager Rest HostPorts property.

      Enter the host and port pairs of one or multiple the SRM Service roles. You can find the hostname by going to SRM > Instances. For the port, use the value set in the SRM Service Https Port or SRM Service Port properties. If the SRM Service roles are TLS/SSL enabled (Enable TLS/SSL for SRM Service is selected), use the port in SRM Service Https Port. Otherwise, use the port in SRM Service Port.

  4. Click Save Changes.
  5. Restart SMM.
Integration between SMM and SRM is enabled. The Cluster Replications tab is now available on the SMM UI.
Go to the SMM UI and access the Cluster Replications section from the navigation sidebar.

Integrating Kafka Connect with Streams Messaging Manager

Learn how to set up Streams Messaging Manager (SMM) for monitoring and managing Kafka Connect.

Cloudera recommends that you use SMM to manage and monitor Kafka Connect. In order for SMM to be able to interact with Kafka Connect, it must be configured and provided with the host, port, and protocol of the Kafka Connect role instances. This is done by configuring the Kafka Connect Rest HostPort and Kafka Connect Protocol SMM configuration properties. Configuring these properties enables the Connect section in SMM, which allows you to interact with Kafka Connect through a UI interface. Additionally, configuring these properties also enables you to use the SMM REST API to interact with Kafka Connect.

If the Kafka Connect Rest HostPort property is left empty (default), SMM is automatically configured with the host, port, and protocol of all Kafka Connect role instances belonging to the Kafka service selected with the Kafka Service SMM property. This means that if you want SMM to monitor and manage the Kafka Connect instance that belongs to the Kafka service that SMM depends on, you do not need to complete the following steps.

  1. Select the Streams Messaging Manager service.
  2. Go to Configuration.
  3. Find and configure the following properties.
    • Kafka Connect Rest HostPort

      Enter the host and port pairs of the Kafka Connect roles. If you have multiple instances of the Kafka Connect role, Cloudera recommends that you add all of them for high availability. If there are multiple host and port pairs specified, SMM will automatically failover between the specified endpoints. For the port, use the value set in the Kafka Connect Rest Port or Secure Kafka Connect Rest Port Kafka Connect properties. If the Kafka Connect role is not TLS/SSL enabled, use the port specified in Kafka Connect Rest Port. If TLS/SSL is enabled, use the port specified in Secure Kafka Connect Rest Port.

    • Kafka Connect Protocol

      Select http if SSL/TLS is not enabled for the Kafka Connect roles. Select https if SSL/TLS is enabled for the Kafka Connect roles.

  4. Click Save.
  5. Restart the service.
SMM is configured and is able to manage and interact with Kafka Connect.
Go to the SMM UI and access the Connect section from the left column. Alternatively, use the SMM REST API to interact with Kafka Connect.