Flume Morphline Interceptor Configuration Options

Flume can modify and drop events in-flight with the help of Interceptors, which can be attached to any Flume source. Flume MorphlineInterceptor runs the transformations of a morphline on intercepted events. For example the morphline can ignore events or alter or insert certain event headers using regular expression-based pattern matching, or it can auto-detect and set a MIME type using Apache Tika on events that are intercepted. This packet sniffing can be used for content-based routing in a Flume topology.

Flume supports multiplexing the event flow to destinations by defining a flow multiplexer that can replicate or selectively route an event to channels. This example shows a source from agent “foo” fanning out the flow to three different channels. This fan out can be replicating or multiplexing. In replicating, each event is sent to all three channels. In multiplexing, an event is delivered to a subset of available channels when that event's attribute matches a preconfigured value. For example, if an event attribute called stream.type is set to application/pdf, it goes to channel1 and channel3. If the attribute is set to avro/binary, it goes to channel2. If that channel is unavailable then an exception is thrown and the event is replayed later when the channel becomes available again. You can set the mapping in the flume.conf file.

Flume MorphlineInterceptor provides the following configuration options in the flume.conf file:

Property Name

Default

Description

type

 
The FQCN of this class: 
org.apache.flume.sink.solr.
morphline.MorphlineInterceptor$Builder

morphlineFile

n/a

The location of the morphline configuration file.

  • In a Cloudera Manager deployment, use: 
    agent.sources.avroSrc.interceptors.
morphlineinterceptor.morphlineFile = morphlines.conf
  • In unmanaged deployments, provide the relative or absolute path on the local filesystem to the morphline configuration file. For example, /etc/flume-ng/conf/morphline.conf.

morphlineId

null

The name used to identify a morphline if a config file has multiple morphlines.
This example shows a flume.conf section for a MorphlineInterceptor for the agent named "agent": 
agent.sources.avroSrc.interceptors = morphlineinterceptor
agent.sources.avroSrc.interceptors.morphlineinterceptor.type = org.apache.flume.sink.solr.morphline.MorphlineInterceptor$Builder
agent.sources.avroSrc.interceptors.morphlineinterceptor.morphlineFile = /etc/flume-ng/conf/morphline.conf
agent.sources.avroSrc.interceptors.morphlineinterceptor.morphlineId = morphline1