Configuring the Flume Solr Sink

  1. Edit /etc/flume-ng/conf/flume.conf to specify the Flume source details and set up the flow. You must set the relative or absolute path to the morphline configuration file:
    agent.sinks.solrSink.morphlineFile = /etc/flume-ng/conf/morphline.conf
  2. Edit /etc/flume-ng/conf/morphline.conf to specify the Solr location details using a SOLR_LOCATOR. The snippet that includes the SOLR_LOCATOR might appear as follows:
    SOLR_LOCATOR : {
      # Name of solr collection
      collection : collection
    
      # ZooKeeper ensemble
      zkHost : "$ZK_HOST"
    }
    
    morphlines : [
      {
        id : morphline1
        importCommands : ["org.kitesdk.**", "org.apache.solr.**"]
        commands : [
          { generateUUID { field : id } }
    
          { # Remove record fields that are unknown to Solr schema.xml.
            # Recall that Solr throws an exception on any attempt to load a document that
            # contains a field that isn't specified in schema.xml.
            sanitizeUnknownSolrFields {
              solrLocator : ${SOLR_LOCATOR} # Location from which to fetch Solr schema
            }
          }
    
          { logDebug { format : "output record: {}", args : ["@{}"] } }
    
          {
            loadSolr {
              solrLocator : ${SOLR_LOCATOR}
            }
          }
        ]
      }
    ]
  3. Copy flume-env.sh.template to flume-env.sh:
    $ sudo cp /etc/flume-ng/conf/flume-env.sh.template \
    /etc/flume-ng/conf/flume-env.sh
  4. Edit /etc/flume-ng/conf/flume-env.sh, inserting or replacing JAVA_OPTS as follows:
    JAVA_OPTS="-Xmx500m"
  5. (Optional) Modify Flume logging settings to facilitate monitoring and debugging:
    $ sudo bash -c 'echo "log4j.logger.org.apache.flume.sink.solr=DEBUG" >> \
    /etc/flume-ng/conf/log4j.properties'
    $ sudo bash -c 'echo "log4j.logger.org.kitesdk.morphline=TRACE" >> \
    /etc/flume-ng/conf/log4j.properties'
  6. (Optional) You can configure the location at which Flume finds Cloudera Search dependencies for Flume Solr Sink using SEARCH_HOME. For example, if you installed Flume from a tarball package, you can configure it to find required files by setting SEARCH_HOME. To set SEARCH_HOME use a command of the form:
    • Parcel-based Installation:
      $ export SEARCH_HOME=/opt/cloudera/parcels/CDH/lib/search
    • Package-based Installation:
      $ export SEARCH_HOME=/usr/lib/search