Cloudera Docs

Configure the processor for your data source

You can set up a data flow to move data to Apache Solr from many different locations. This example assumes that you are transferring data from Kafka using the ConsumeKafkaRecord_2_0 processor.

  • Ensure that you have the Ranger policies required to access the Kafka consumer group.
  • This use case demonstrates a data flow using data in Kafka and moving it to Solr. To review how to ingest data to Kafka, see Ingesting Data to Apache Kafka.
  1. Launch the Configure Processor window, by right clicking the ConsumeKafkaRecord_2_0 processor and selecting Configure.
    You can see a configuration dialog with the following tabs: Settings, Scheduling, Properties, Comments.
  2. Configure the processor according to the behavior you expect in your data flow.
    1. Click the Properties tab.
    2. Configure ConsumeKafkaRecord_2_0 with the required values.
      The following table includes a description and example values for the properties required for this Kafka to Solr ingest data flow. For a complete list of ConsumeKafkaRecord_2_0 options, see the processor documentation in Getting Started with Apache NiFi.
      Property Description Value to set for ingest to Solr data flow
      Kafka Brokers Provide a comma-separated list of known Kafka Brokers in the format <host>:<port>.

      Docs-messaging-broker1.cdf-docs.a465-9q4k.cloudera.site:9093,

      docs-messaging-broker2.cdf-docs.a465-9q4k.cloudera.site:9093,

      docs-messaging-broker3.cdf-docs.a465-9q4k.cloudera.site:9093
      Topic Name(s) Enter the name of the Kafka topic from which you want to get data. customer
      Topic Name Format Specify whether the topics are a comma separated list of topic names or a single regular expression. names
      Record Reader Specify the record reader to use for incoming flowfiles. This can be a range of data formats. AvroReader
      Record Writer

      Specify the format you want to use to serialize data before for sending it to the data target.

      		 AvroRecordSetWriter
      Honor Transactions Specify whether or not NiFi should honor transactional guarantees when communicating with Kafka. false
      Security Protocol Specify the protocol used to communicate with brokers. This value corresponds to Kafka's 'security.protocol' property. SASL_SSL
      SASL Mechanism Specify the SASL mechanism to be used for authentication. This value corresponds to Kafka's 'sasl.mechanism' property. plain
      Username Specify the username when the SASL Mechanism is PLAIN or SCRAM-SHA-256. srv_nifi-solr-ingest
      Password Specify the password for the given username when the SASL Mechanism is PLAIN or SCRAM-SHA-256. Password1!
      SSL Context Service Specify the SSL Context Service to be used for communicating with Kafka. default
      Group ID Specify the Group ID used to identify consumers that are within the same consumer group. This value corresponds to Kafka's group.id property. nifi-solr-ingest
  3. When you have finished configuring the options you need, save the changes by clicking Apply.

    If you want to move data to Solr from a location other than Kafka, see Getting Started with Apache NiFi for general information about how to build a data flow and about other data consumption processor options. You can also check the other CDP Public Cloud ingest use cases for more information.

Configure the processor for your data target.