Setting up Atlas Kafka import tool

Configure the Atlas-Kafka import tool to map Kafka topics to Apache Atlas entities by managing client configurations and Ranger security policies.

You must first set up the tool and later run the tool manually to create or update the Kafka topic entities in Atlas. The tool uses client configuration to fetch the required configuration, like the Atlas endpoint and Zookeeper.

The Kafka type definition in Atlas contains multiple fields. The import tool fills out the following field values:

clusterName: Contains the value provided by the atlas.metadata.namespace property in the application configuration file. The default value is cm.
topic, name, description, URI: Contains the topic name.
qualifiedName: Contains the topic name and the clusterName, joined by @. For example, my-topic@cm. This serves as the unique identifier for topics.
partitionCount: Displays the number of partitions of the topic.

To set up the Atlas - Kafka import tool, follow these steps:

Select the Atlas service in Cloudera Manager.
Deploy client configs: Actions > Deploy Client Configuration.
note
- If Kerberos authentication is enabled in Apache Zookeeper, you must run the tool as the Kafka Kerberos user and set up the Ranger policies for the Kafka user.
If Apache Ranger is enabled, create a Ranger policy with the user running the tool in cm_atlas to allow the <user> to create, update, delete, and read Kafka entities:
- Log into Ranger.
- Navigate to cm_atlas policies.
- Select the policy which has create_entity and delete_entity permissions.
- Add the Kerberos user that is used with the import-kafka.sh tool.
SSH into a host where the client configuration is deployed.
Run kinit as Kafka if Kerberos is used to set up the Ticket Granting Ticket (TGT).
Set JAVA_HOME.
note
Use export JAVA_HOME=/usr/java/default in this scenario.
Run the command /opt/cloudera/parcels/CDH/lib/atlas/hook-bin/import-kafka.sh to import Kafka topics into Atlas.