HTTP Sink connector
The HTTP Sink connector is a Stateless NiFi dataflow developed by Cloudera that is running in the Kafka Connect framework. Learn about the connector, its properties, and configuration.
The HTTP Sink connector obtains messages from a Kafka topic and transfers their content in a
HTTP POST requests to a specified endpoint. The topic the connector receives messages from is
determined by the value of the topics
parameter in the configuration. The
connector can forward the data it reads from Kafka as is (raw data) or can be configured to
execute record processing. When record processing is enabled, the connector expects the
incoming data in Avro or JSON format. In case of Avro, the connector can either read the
record schema from the Avro file it receives (provided that the schema is embedded) or it can
fetch the schema from Schema Registry. In case of JSON, it can either infer the schema or
fetch it from Schema Registry. What strategy is used to retrieve the schema is determined by
the Schema Access Strategy
property.
If Schema Registry is used, and it is on a Kerberized cluster, the krb5.file
property must point to the krb5.conf
file that provides access to the cluster
on which Schema Registry is present. This means that the krb5.conf
file must
be on the same cluster node that the connector runs on. The connections to Schema Registry and
the HTTP server can be secured by TLS. The keystore and truststore files necessary for
securing these connections must also be on the same cluster node that the connector runs
on.
Properties and configuration
Configuration is passed to the connector in a JSON file during creation. The properties of the connector can be categorized into three groups. These are as follows:
- Common connector properties
- These are the properties of the Kafka Connect framework that are accepted by all connectors. For a comprehensive list of these properties, see the Apache Kafka documentation.
- Stateless NiFi Sink properties
- These are the properties that are specific to the Stateless NiFi Sink connector. All Stateless NiFi Sink connectors share and accept these properties. For a comprehensive list of these properties, see the Stateless NiFi Sink property reference.
- Connector/dataflow-specific properties
- These properties are unique to this specific connector. Or to be more precise, unique
to the dataflow running within the connector. These properties use the following
prefix:
For a comprehensive list of these properties, see the HTTP Sink properties reference.parameter.[***CONNECTOR NAME***] Parameters:
Notes and limitations
- Required properties must be assigned a valid value even if they are not used in the particular configuration. If a required property is not used, either leave its default value, or completely remove the property from the configuration JSON.
- If a property that has a default value is completely removed from the configuration JSON, the system uses the default value.
- Properties not marked as required must be completely removed from the configuration JSON if not set.
Schema Branch
andSchema Version
can not be specified at the same time.- The value of the
Schema Access Strategy
property is not independent of the value of theKafka Message Data Format
property. As a result, you must exercise caution when configuringSchema Access Strategy
.- If the value of
Kafka Message Data Format
isAVRO
, the possible values forSchema Access Strategy
areSchema Registry
orEmbedded Schema
. - If the value of
Kafka Message Data Format
isJSON
, the possible values forSchema Access Strategy
areSchema Registry
orInfer Schema
.
- If the value of
Configuration example
In this example, the connector receives data in any format and forwards the raw data as the content of an HTTP POST request.
{
"connector.class": "org.apache.nifi.kafka.connect.StatelessNiFiSinkConnector",
"meta.smm.predefined.flow.name": "HTTP Sink",
"meta.smm.predefined.flow.version": "1.0.0",
"key.converter": "org.apache.kafka.connect.converters.ByteArrayConverter",
"value.converter": "org.apache.kafka.connect.converters.ByteArrayConverter",
"tasks.max": "1",
"nexus.url": "https://repository.cloudera.com/artifactory/repo",
"extensions.directory": "/tmp/nifi-stateless-extensions",
"working.directory": "/tmp/nifi-stateless-working",
"input.port": "Input from Kafka",
"failure.ports": "Failure",
"topics": "[***KAFKA TOPIC NAME***]",
"parameter.HTTP Sink Parameters:Forward Raw Data": "true",
"parameter.HTTP Sink Parameters:Remote URL": "http://[***SERVER HOSTNAME***]:[***PORT***]/[***PATH***]"
}
topics
- The name of the Kafka topic the connector fetches messages from.
Forward Raw Data
- Specifies whether messages from Kafka should be forwarded as is or converted to
JSON. In this example, the property is set to
true
, meaning that the connector does not process any records. It forwards incoming data as is. Remote URL
- Identifies the HTTP endpoint that receives the messages sent by this connector. In
this example, SSL is not used. As a result, the URL starts with
http
nothttps
. For example,http://my.http-server.com:22000/contentListener
.