Installing Kafka

Required Role:

Kafka is distributed as a parcel, separate from the CDH parcel. Cloudera Manager 5.4 or higher includes the Kafka Custom Service Descriptor (CSD). To install, you download Kafka using CM, then distribute Kafka to the cluster, activate the new parcel, and add the service to the cluster.

To install Kafka on a 5.2 or 5.3 cluster, you first install the Kafka CSD, and then use the CSD to install the Kafka parcel. See Installing the Kafka CSD (Cloudera Manager 5.2 and 5.3).

Kafka is supported only on Cloudera Manager 5.2 and higher on a parcel-deployed cluster. Do not use it with lower versions of Cloudera Manager or CDH, nor on a cluster deployed using packages or a tarball.

If you installed a Cloudera Labs version of Kafka, you must download a new CSD and parcel. The Cloudera Labs CSD cannot install the GA Kafka parcel.

Cloudera recommends that you deploy Kafka on dedicated hosts that are not used for other cluster roles.

Installing the Kafka CSD (Cloudera Manager 5.2 and 5.3)

Cloudera Manager 5.4 and higher includes the Kafka CSD. Use the built-in CSD. Do not download a different version when using Cloudera Manager 5.4.

If you are using Cloudera Manager 5.2 or 5.3, download the CSD, and then add a new parcel repository to your Cloudera Manager configuration:
  1. Download the Kafka CSD here.
  2. Install the Kafka CSD into Cloudera Manager. See Custom Service Descriptor Files.

Installing the Kafka Parcel

  1. In Cloudera Manager, download the Kafka parcel, distribute the parcel to the hosts in your cluster, and then activate the parcel. See Managing Parcels. After you activate the Kafka parcel, Cloudera Manager prompts you to restart the cluster. You do not need to restart the cluster after installing Kafka. Click Close to ignore this prompt.
  2. Add the Kafka service to your cluster. See Adding a Service.