Fastcapa-Kafka Configuration File
The path to the configuration file is specified with the -c command
line argument. The file can contain any global or topic-specific, producer-focused
configuration values accepted by Librdkafka.
The configuration file is a .ini-like Glib configuration file. The
global configuration values should be placed under a [kafka-global]
header and topic-specific values should be placed under
[kafka-topic].
A minimally viable configuration file would only need to include the Kafka broker to connect to.
[kafka-global] metadata.broker.list = kafka-broker1:9092, kafka-broker2:9092
The configuration parameters that are important for either basic functioning or performance tuning of Fastcapa include the following.
Global configuration values that should be located under the
[kafka-global] header.
| Name | Description | Default |
|---|---|---|
| metadata.broker.list | Initial list of brokers as a CSV list of broker host or host:port | |
| client.id | Client identifier. | |
| queue.buffering.max.messages | Maximum number of messages allowed on the producer queue | 100000 |
| queue.buffering.max.ms | Maximum time, in milliseconds, for buffering data on the producer queue | 1000 |
| message.copy.max.bytes | Maximum size for message to be copied to buffer. Messages larger than this will be passed by reference (zero-copy) at the expense of larger iovecs. | 65535 |
| batch.num.messages | Maximum number of messages batched in one MessageSet | 10000 |
| statistics.interval.ms | How often statistics are emitted; 0 = never | 0 |
| compression.codec | Compression codec to use for compressing message sets; {none, gzip, snappy, lz4 } | none |
Topic configuration values that should be located under the
[kafka-topic] header.
| Name | Description | Default |
|---|---|---|
| compression.codec | Compression codec to use for compressing message sets; {none, gzip, snappy, lz4 } | none |
| request.required.acks | How many acknowledgements the leader broker must receive from ISR brokers before responding to the request; { 0 = no ack, 1 = leader ack, -1 = all ISRs } | 1 |
| message.timeout.ms | Local message timeout. This value is only enforced locally and limits the time a produced message waits for successful delivery. A time of 0 is infinite. | 300000 |
| queue.buffering.max.kbytes | Maximum total message size sum allowed on the producer queue |