Configuring the Kafka Connect connector in Cloudera Data Lineage

Learn how to configure the Kafka Connect connector in Octopai Client using Kerberos authentication (SASL/GSSAPI).

Before configuring the Kafka Connect connector in Octopai Client, ensure the following components are available and properly configured:

Kerberos infrastructure: Active Kerberos Key Distribution Center (KDC), valid Kerberos realm configuration, and network connectivity from the Octopai Client host to the KDC.
MIT Kerberos for Windows: Install MIT Kerberos for Windows on the machine running Octopai Client. The default installation path is C:\Program Files\MIT\Kerberos\bin\kinit.exe. Octopai Client uses kinit to acquire Kerberos tickets.

Kerberos configuration file: The Kerberos configuration file must exist at C:\ProgramData\MIT\Kerberos5\krb5.ini.

Example configuration:

[libdefaults]
                            default_realm = ROOT.COMOPS.SITE
                            dns_lookup_realm = false
                            dns_lookup_kdc = false
                            ticket_lifetime = 24h
                            renew_lifetime = 7d
                            forwardable = true
                            
                            [realms]
                            ROOT.COMOPS.SITE = {
                            kdc = ccycloud-1.cdp.root.comops.site
                            admin_server = ccycloud-1.cdp.root.comops.site
                            }
                            
                            [domain_realm]
                            .root.comops.site = ROOT.COMOPS.SITE
                            root.comops.site = ROOT.COMOPS.SITE

Kerberos credentials: Obtain a Kerberos principal (for example, kafka-user@REALM) and its associated keytab file (for example, C:\octopai\kafka-user.keytab). Ensure the keytab file is securely stored and accessible.
Kafka cluster configuration: Ensure the Kafka cluster is configured with SASL/GSSAPI enabled, the Kafka service principal is configured on the brokers (for example, kafka/hostname@REALM), and the broker hostnames are resolvable using fully qualified domain names.

Create a keytab file.
On a Kerberos administration server, create a keytab file for the Kerberos principal used by Octopai Client using this command:
```
ktutil
                        addent -password -p kafka-user@REALM -k 1 -e aes256-cts
                        wkt /path/to/kafka-user.keytab
                        quit
```
Or request the keytab file from your Kerberos administrator using this command:
```
kadmin -q "ktadd -k /path/to/kafka-user.keytab kafka-user@REALM"
```
Copy the keytab file securely to the Windows server running the Octopai Client.
Verify Kerberos authentication before configuring the connector.
1. Open PowerShell.
2. Navigate to the Kerberos binaries directory:
```
cd "C:\Program Files\MIT\Kerberos\bin"
```
3. Obtain a Kerberos ticket:
```
.\kinit.exe -kt "C:\octopai\kafka-user.keytab" kafka-user@REALM
```
4. Confirm the validity of the ticket:
```
.\klist.exe
```
  note
  A valid ticket confirms that Kerberos is configured correctly.
Configure the Kafka Connect connector in Octopai Client.
1. Start a new connection:
  1. Open Octopai Client.
  2. In the New Metadata Source wizard, on Metadata Source Type, select Kafka Connect.
2. Select the authentication method: None or Kerberos.
  
  note
  The fields available on the form depend on the authentication method you select.
3. Complete the connection fields:
  Figure 1. Kafka Connect metadata source details with no authentication method selected
  - Connection Name (Required): Enter a meaningful name. The name is displayed to Cloudera Data Lineage platform users.
  - Kafka Connect URL (Required): Kafka Connect REST endpoint.
  - Bootstrap Servers (Required): Kafka broker hostnames and ports (for example, kafka1.example.com:9092).
  - Schema Registry URL (Optional): Schema Registry endpoint (for example, http://schema-registry.example.com:8081).
  - Parse, Analyze, and Tag the Views and Stored Procedures as ETLs (Optional): Select this option when you want Cloudera Data Lineage to treat views and stored procedures as ETL objects during analysis. This option is enabled by default.
  note
  
  The connector retrieves topic metadata from Kafka using Bootstrap Servers, and all other metadata from Kafka Connect using the Kafka Connect URL.
4. If you selected Kerberos, enter the following additional Kerberos settings.
  Figure 2. Kafka Connect metadata source details with Kerberos authentication selected
  - Keytab Path (Required): Full path to the keytab file (for example, C:\octopai\kafka-user.keytab).
  - Kerberos Principal (Required): for example, kafka-user@REALM.
  important
  Realm names are case-sensitive.
  
  The keytab file must be readable by the Octopai Client service account.
  
  Broker addresses must use fully qualified domain names.
5. Click Next to advance to the Test & Save step.
6. To test the connection, click Test Connection.
7. Click Save to store the connection.
Verify the extracted metadata files.
1. After the extraction completes, navigate to this folder:
```
C:\Program Files (x86)\Octopai\Service\TGT
```
2. Open the ZIP file matching the connector name.
3. Verify the presence and structure of the extracted files.

Kerberos authentication errors:

Verify the principal format: username@REALM
Confirm the keytab path and permissions.
Validate the realm configuration in krb5.ini

Kafka connectivity issues:

Verify the network connectivity.
Use fully qualified domain names.
Confirm the SASL listener configuration.

Ticket expiration:

Verify the ticket_lifetime and renew_lifetime values.
Adjust the renewal configuration, if required.

Clock skew:

Synchronize the system time:

w32tm /resync /force

Error during the extraction:

Collect the logs from C:\Program Files (x86)\Octopai\Service\log
Send the logs with the connector number and name to Cloudera Support.