Morphline commands overview

Converts a byte stream to a string.

Extracts zero or more records from the input stream of bytes representing a Comma Separated Values (CSV) file.

Emits one record per line in the input stream.

Log parser that collapses multiple input lines into a single record, based on regular expression pattern matching.

Adds the result of System.currentTimeMillis() to a given output field.

Adds the name or IP of the local host to a given output field.

Adds a list of values (or the contents of another field) to a given field if not already contained.

Implements recursion for extracting data from container data formats.

Returns whether or not a given value is contained in a given field.

Converts the timestamps in a given field from one of a set of input date formats to an output date format.

Converts a Base64 encoded String to a byte[].

Silently consumes records without ever emitting any record. Think /dev/null.

Succeeds if all field values of the given named fields are equal to the the given values and fails otherwise.

Extracts subcomponents such as host, port, path, query, etc from a URI.

Extracts a particular subcomponent from a URI.

Extracts the query parameters with a given name from a URI.

Examines each string value in a given field and replaces each substring of the string value that matches the given string literal or grok pattern with the given replacement.

Sets a universally unique identifier on all records that are intercepted.

Uses regular expression pattern matching to extract structured fields from unstructured log or text data.

Ignores all input records beyond the N-th record, akin to the Unix head command.

Implements if-then-else conditional control flow.

Scripting support for Java. Dynamically compiles and executes the given Java code block.

Logs a message at the given log level to SLF4J.

Inverts the boolean return value of a nested command.

Pipes a record through a chain of commands.

Removes all record fields for which the field name matches a blacklist but not a whitelist.

Replaces all record field values for which the field name and value matches a blacklist but not a whitelist.

Forwards each input record with a given probability to its child command.

Emits one separate output record for each attachment in the input record's list of attachments.

Assigns a given list of values (or the contents of another field) to a given field.

Divides a string into substrings, by recognizing a separator (a.k.a. "delimiter") which can be expressed as a single character, literal string, regular expression, or grok pattern.

Splits key-value pairs where the key and value are separated by the given separator, and adds the pair's value to the record field named after the pair's key.

Starts periodically appending the metrics of all commands to a set of CSV files.

Starts publishing the metrics of all commands to JMX.

Starts periodically logging the metrics of all morphline commands to SLF4J.

Converts a String to the byte array representation of a given charset.

Converts a Java object to it's string representation; optionally also removes leading and trailing whitespace.

Replace a string with the replacement value defined in a given dictionary aka lookup hash table.

Simple rule engine for handling a list of heterogeneous input data formats.

Parses an Apache Avro binary container and emits a morphline record for each contained Avro datum.

Parses containerless Avro and emits a morphline record for each contained Avro datum.

Recursively walks an Avro tree and extracts all data into a single morphline record.

Extracts specific values from an Avro object, akin to a simple form of XPath.

Converts a morphline record to an Avro record.

Serializes Avro records into a byte array.

Parses JSON and emits a morphline record for each contained JSON object, using the Jackson library.

Extracts specific values from a JSON object, akin to a simple form of XPath.

Downloads, on startup, zero or more files or directory trees from HDFS to the local file system.

Opens an HDFS file for read and returns a corresponding Java InputStream.

Parses a Hadoop Parquet file and emits a morphline record for each contained Avro datum.

Parses an Apache Hadoop RCFile and emits morphline records row-wise or column-wise.

Parses an Apache Hadoop SequenceFile and emits a morphline record for each contained key-value pair.

Returns Geolocation information for a given IP address, using an efficient in-memory Maxmind database lookup.

Registers metrics that are related to the Java Virtual Machine with the MorphlineContext.

Exposes liveness status, health check status, metrics state and thread dumps via a set of HTTP URLs served by Jetty, using the AdminServlet.

Parses an InputStream that contains protobuf data and emits a morphline record containing the protobuf object as an attachment.

Extracts specific values from a protobuf object, akin to a simple form of XPath.

Uses Apache Tika to autodetect the MIME type of binary data.

Decompresses gzip and bzip2 format.

Unpacks tar, zip, and jar format.

Converts any HTML to XHTML, using the TagSoup Java library.

Parses XML and runs the given W3C XQuery over it, using the Saxon Java library.

Parses XML and runs the given W3C XSL Transform over it, using the Saxon Java library.

Specifies a set of configuration parameters that identify the location and schema of a Solr server or SolrCloud.

Inserts, updates or deletes records into a Solr server or MapReduce Reducer.

Assigns a unique key that is the concatenation of a field and a running count of the record number within the current session.

Removes record fields that are unknown to Solr schema.xml, or moves them to fields with a given prefix.

Uses the embedded Solr/Lucene Analyzer library to generate tokens from a text string, without sending data to a Solr server.

Uses Apache Tika to parse data, then maps the Tika output back to a record using Apache SolrCell.

Parses a user agent string and returns structured higher level data like user agent family, operating system, version, and device type.