Cloudera Docs

Installing Connectors

Learn how to install connectors for use with Streams Messaging Manager.

Kafka Connect connectors are distributed as:
  • A directory of JAR files:

    The directory includes the JAR for the connector itself, as well as all its dependencies.

  • An uber JAR/FAT JAR/JAR with dependencies file:

    This a single JAR file that contains the connector, as well as its dependencies.

In CDP all connectors that do not come prepackaged with the Runtime distribution have to be installed manually. This is done by making the connector JAR files available on the cluster hosts in a specific location.
Which host?
The connector files have to be made available on all hosts that are running Kafka Connect roles.
What location?
Kafka Connect discovers connectors by looking at a specific directory path on the host machine. The path it checks is determined by the Kafka Connect role’s plugin.path property. By default the plugin.path property is set to /var/lib/kafka. This means that by default any connector placed in this directory will be discovered by Kafka Connect. Cloudera recommends that you use the default path.
  1. Log in to a host that is running a Kafka Connect role.
  2. Make the connector files available in or readable from /var/lib/kafka.
    There are multiple ways you can achieve this, how you choose to complete this step will largely depend on your cluster environment. For example:
    • You can download or copy the files directly to /var/lib/kafka.
    • You can choose to place connector files in a location different from /var/lib/kafka and create symlinks that point to the location where the connector files are available.

    Regardless of what method you choose, this step is considered complete once the connector files are readable from /var/lib/kafka.

  3. Restart all Kafka Connect roles
    1. In Cloudera Manager, select the Kafka service.
    2. Go to Instances.
    3. Select all Kafka Connect instances by checking the checkbox next to each instance.
    4. Click Actions for selected > Restart.
    5. Click Restart to confirm.

      The roles are restarted once a Finished status is displayed.

The connectors are installed and available for use. You are now able to deploy and manage the new connector from the SMM UI.