Configure the processor for your data source
You can set up a data flow to move data to Apache Solr from many different locations. This example assumes that you are transferring data from Kafka using the ConsumeKafkaRecord_2_0 processor. If you are moving your data from a location other than Kafka, see Getting Started with Apache NiFi for general information about how to build a data flow and about other data consumption processor options. You can also check out the other ingest use cases in the Cloudera Data Flow for Data Hub library.
- Ensure that you have the Ranger policies required to access the Kafka consumer group.
- This use case demonstrates a data flow using data in Kafka and moving it to Solr. To review how to ingest data to Kafka, see Ingesting Data to Apache Kafka.
Launch the Configure Processor window, by right clicking the
ConsumeKafkaRecord_2_0 processor and selecting
Configure.You can see a configuration dialog with the following tabs: Settings, Scheduling, Properties, Comments.
- Configure the processor according to the behavior you expect in your data
- Click the Properties tab.
- Configure ConsumeKafkaRecord_2_0 with the required
values.The following table includes a description and example values for the properties required for this Kafka to Solr ingest data flow. For a complete list of ConsumeKafkaRecord_2_0 options, see the processor documentation in Getting Started with Apache NiFi.
Property Description Value to set for ingest to Solr data flow Kafka Brokers Provide a comma-separated list of known Kafka Brokers in the format
Topic Name(s) Enter the name of the Kafka topic from which you want to get data. customer Topic Name Format Specify whether the topics are a comma separated list of topic names or a single regular expression. names Record Reader Specify the record reader to use for incoming flowfiles. This can be a range of data formats. AvroReader Record Writer
Specify the format you want to use to serialize data before for sending it to the data target.
AvroRecordSetWriter Honor Transactions Specify whether or not NiFi should honor transactional guarantees when communicating with Kafka. false Security Protocol Specify the protocol used to communicate with brokers. This value corresponds to Kafka's 'security.protocol' property. SASL_SSL SASL Mechanism Specify the SASL mechanism to be used for authentication. This value corresponds to Kafka's 'sasl.mechanism' property. plain Username Specify the username when the SASL Mechanism is PLAIN or SCRAM-SHA-256. srv_nifi-solr-ingest Password Specify the password for the given username when the SASL Mechanism is PLAIN or SCRAM-SHA-256. Password1! SSL Context Service Specify the SSL Context Service to be used for communicating with Kafka. default Group ID Specify the Group ID used to identify consumers that are within the same consumer group. This value corresponds to Kafka's
- When you have finished configuring the options you need, save the changes by