Cloudera Docs

Using the Kafka Connect REST API

Kafka Connect offers a REST API that you can use to manage and monitor connectors. Learn about the REST API, available endpoints, and recommended use. Additionally learn about connect_shell.sh, which is a command line tool that you can use to establish quick access to the REST API.

The Kafka Connect REST API is available as a ClusterIP type Kubernetes service. The service is named [****CONNECT CLUSTER NAME***]-connect-api. Its default port is 8083.

[****CONNECT CLUSTER NAME***] is the name of your Kafka Connect cluster. The name is specified in the metadata.name property of the KafkaConnect resource used to deploy the cluster. The service is created when you deploy the cluster.

The REST API offers various endpoints and operations that you can use to manage (create, update, delete) as well as to monitor the connectors running in your Kafka Connect cluster. You can find a comprehensive reference in the Kafka Connect Rest API reference.

API access and security

By default the Kafka Connect API is only accessible from within the Kubernetes Cluster. Additionally, the default network policies only allow access by the Strimzi Cluster Operator and Kafka Connect pods. This is done because the REST API is insecure by default and it cannot be secured. As a result Cloudera recommends the following:

  • Do not expose the REST API to applications running outside the Kubernetes cluster.
  • Use KafkaConnector resources to manage connectors instead of the REST API.

Recommended use

Cloudera recommends that you use the API selectively for specific use-cases. In general for any connector management operations, use KafkaConnector resources. However, you can use any endpoints or operations that return information about the cluster and connectors For example, you can use the GET /connector-plugins endpoint with connectorsOnly set to false to list all plugins that are installed in the Kafka Connect cluster.

If you want to query the REST API, Cloudera recommends that you use the connect_shell.sh tool.

Using connect_shell.sh

Use connect_shell.sh to set up a pod that allows easy access to the Kafka Connect REST API. The pod created with this tool includes preset configurations, such as the $CONNECT_REST_URL environment variable, which is set to the base URL of the API.

  • Ensure that you have access to your Cloudera credentials (username and password).
  • Ensure that the environment where you run the tool has the following:
    • Bash 4 or higher.
    • GNU utilities:
      • echo
      • grep
      • sed
      • head
    • kubectl or oc
    • kubeconfig configured to target Kubernetes cluster
  1. Download the tool. 
    curl --user [***USERNAME***] \
  https://archive.cloudera.com/p/csm-operator/1.2/tools/connect_shell.sh \
  --output connect_shell.sh --location \
&& chmod +x connect_shell.sh
    Replace [***USERNAME***] with your Cloudera username. Enter your Cloudera password when prompted.
  2. Use the tool.
    You have two choices. You can either use the tool interactively. In this case, you run the tool which opens an interactive shell window where you run queries. Alternatively, you can use pipe ( | ) to run queries one at a time.
    1. Run the tool.
      ./connect_shell.sh \
  --namespace=[***CONNECT CLUSTER NAMESPACE***] \
  --cluster=[***CONNECT CLUSTER NAME***]
    2. Query the REST API.

      For example, you can list your topics with the following command.

      curl $CONNECT_REST_URL/connector-plugins

      This example queries /connector-plugins endpoint which returns available connector plugins in the cluster.

      The pod is deleted when you exit the interactive shell.

    To run one-off queries, pipe them into connect_shell.sh. 
    echo 'curl $CONNECT_REST_URL/connector-plugins' \
| ./connect_shell.sh --namespace=[***CONNECT CLUSTER NAMESPACE***] \
  --cluster=[***CONNECT CLUSTER NAME***]