This is the documentation for CDH 5.0.x. Documentation for other versions is available at Cloudera Documentation.

Flume Morphline Interceptor Configuration Options

Flume has the capability to modify and drop events in-flight. This is done with the help of a Interceptors, which can be attached to any Flume Source. Flume MorphlineInterceptor is an Interceptor that executes the transformations of a morphline on events that are intercepted. For example the morphline can ignore events or alter or insert certain event headers using regular expression-based pattern matching, or it can auto-detect and set a MIME type using Apache Tika on events that are intercepted. For example, this kind of packet sniffing can be used for content based routing in a Flume topology.

Flume supports multiplexing the event flow to one or more destinations. This is achieved by defining a flow multiplexer that can replicate or selectively route an event to one or more channels. This example shows a source from agent “foo” fanning out the flow to three different channels. This fan out can be replicating or multiplexing. In case of replicating flow, each event is sent to all three channels. For the multiplexing case, an event is delivered to a subset of available channels when that event's attribute matches a preconfigured value. For example, if an event attribute called “stream.type” is set to “application/pdf”, then it should go to channel1 and channel3. If the attribute is set to “avro/binary”, then it should go to channel2, otherwise channel3. You can set the mapping in the flume.conf file.

Flume MorphlineInterceptor provides the following configuration options in the flume.conf file:

Property Name	Default	Description
type		The FQCN of this class: org.apache.flume.sink.solr. morphline.MorphlineInterceptor$Builder
morphlineFile	n/a	The location of the morphline configuration file. In a Cloudera Managed Deployment, use: agent.sources.avroSrc.interceptors. morphlineinterceptor.morphlineFile = morphlines.conf In unmanaged deployments, provide the relative or absolute path on the local file system to the morphline configuration file. For example, `/etc/flume-ng/conf/morphline.conf`.
morphlineId	null	Name used to identify a morphline if there are multiple morphlines in a morphline config file

Property Name

Default

Description

type

The FQCN of this class:

org.apache.flume.sink.solr.
morphline.MorphlineInterceptor$Builder

morphlineFile

n/a

The location of the morphline configuration file.

In a Cloudera Managed Deployment, use:
```
agent.sources.avroSrc.interceptors.
morphlineinterceptor.morphlineFile = morphlines.conf
```
In unmanaged deployments, provide the relative or absolute path on the local file system to the morphline configuration file. For example, /etc/flume-ng/conf/morphline.conf.

morphlineId

null

Name used to identify a morphline if there are multiple morphlines in a morphline config file

For example, here is a flume.conf section for a MorphlineInterceptor for the agent named "agent":

agent.sources.avroSrc.interceptors = morphlineinterceptor
agent.sources.avroSrc.interceptors.morphlineinterceptor.type = org.apache.flume.sink.solr.morphline.MorphlineInterceptor$Builder
agent.sources.avroSrc.interceptors.morphlineinterceptor.morphlineFile = /etc/flume-ng/conf/morphline.conf
agent.sources.avroSrc.interceptors.morphlineinterceptor.morphlineId = morphline1

Note: Currently a morphline interceptor can not generate more than one output record for each input event.

Page generated September 3, 2015.