Fastcapa-Kafka Configuration File
The Kafka configuration file defines how Fastcapa interacts with librdkafka.
You specify the path to the
configuration file with the -c command-line argument. The file can
contain any global or topic-specific, producer-focused configuration values accepted by
Librdkafka.
The configuration file is a .ini-like Glib configuration file. Place
the global configuration values under a [kafka-global] header and place
topic-specific values under [kafka-topic].
A minimally viable configuration file only needs to include the Kafka broker to connect to:
[kafka-global]
metadata.broker.list = kafka-broker1:9092, kafka-broker2:9092
The configuration parameters that are important for either basic functioning or performance tuning of Fastcapa include the following.
| Name | Description | Default |
|---|---|---|
| metadata.broker.list | Initial list of brokers as a CSV list of broker host or host:port | NA |
| client.id | Client identifier. | |
| queue.buffering.max.messages | Maximum number of messages allowed on the producer queue | 100000 |
| queue.buffering.max.ms | Maximum time, in milliseconds, for buffering data on the producer queue | 1000 |
| message.copy.max.bytes | Maximum size for the message to be copied to buffer. Messages larger than this are passed by reference (zero-copy) at the expense of larger iovecs. | 65535 |
| batch.num.messages | Maximum number of messages batched in one MessageSet | 10000 |
| statistics.interval.ms | How often statistics are emitted; 0 = never | 0 |
| compression.codec | Compression codec to use for compressing message sets: none, gzip, snappy, lz4 | none |
Local global configuration values under the [kafka-global]
header.
Locate topic configuration values under the [kafka-topic]
header.
| Description | Default | |
|---|---|---|
| compression.codec | Compression codec to use for compressing message sets: none, gzip, snappy, lz4 | none |
| request.required.acks | How many acknowledgements the leader broker must receive from ISR brokers before responding to the request; { 0 = no ack, 1 = leader ack, -1 = all ISRs } | 1 |
| message.timeout.ms | Local message timeout. This value is only enforced locally and limits the time a produced message waits for successful delivery. A value of 0 represents infinity. | 300000 |
| queue.buffering.max.kbytes | Maximum total message size sum allowed on the producer queue | none |