Kafka Properties in CDH 5.1.0

Role groups:

kafkabrokerdefaultgroup

Advanced

Display Name Description Related Name Default Value API Name Required
Kafka Broker Environment Advanced Configuration Snippet (Safety Valve) For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration. KAFKA_BROKER_role_env_safety_valve false
Kafka Broker Advanced Configuration Snippet (Safety Valve) for kafka-monitoring.properties For advanced use only, a string to be inserted into kafka-monitoring.properties for this role only. kafka-monitoring.properties_role_safety_valve false
Kafka Broker Advanced Configuration Snippet (Safety Valve) for kafka.properties For advanced use only, a string to be inserted into kafka.properties for this role only. kafka.properties_role_safety_valve false
Kafka Broker Logging Advanced Configuration Snippet (Safety Valve) For advanced use only, a string to be inserted into log4j.properties for this role only. log4j_safety_valve false
Automatically Restart Process When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure. false process_auto_restart true

Logs

Display Name Description Related Name Default Value API Name Required
Kafka Broker Log Directory The log directory for log files of the role Kafka Broker. kafka.log4j.dir /var/log/kafka log_dir false
Kafka Broker Logging Threshold The minimum log level for Kafka Broker logs INFO log_threshold false
Kafka Broker Maximum Log File Backups The maximum number of rolled log files to keep for Kafka Broker logs. Typically used by log4j or logback. 10 max_log_backup_index false
Kafka Broker Max Log Size The maximum size, in megabytes, per log file for Kafka Broker logs. Typically used by log4j or logback. 200 MiB max_log_size false

Monitoring

Display Name Description Related Name Default Value API Name Required
Enable Health Alerts for this Role When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold true enable_alerts false
Enable Configuration Change Alerts When set, Cloudera Manager will send alerts when this entity's configuration changes. false enable_config_alerts false
File Descriptor Monitoring Thresholds The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit. Warning: 50.0 %, Critical: 70.0 % kafka_broker_fd_thresholds false
Kafka Broker Host Health Test When computing the overall Kafka Broker health, consider the host's health. true kafka_broker_host_health_enabled false
Kafka Broker Process Health Test Enables the health test that the Kafka Broker's process state is consistent with the role configuration true kafka_broker_scm_health_enabled false
Process Swap Memory Thresholds The health test thresholds on the swap memory usage of the process. Warning: Any, Critical: Never process_swap_memory_thresholds false
Role Triggers The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed. Each trigger has all of the following fields:
  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger will not be evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here may lead to inconsistencies.
For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file-descriptors opened:[{"triggerName": "sample-trigger", "triggerExpression": "IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad", "streamThreshold": 0, "enabled": "true"}]See the trigger rules documentation for more details on how to write triggers using tsquery.The JSON format is evolving and may change in the future and, as a result, backward compatibility is not guaranteed between releases at this time.		 [] role_triggers true
Unexpected Exits Thresholds The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role. Warning: Never, Critical: Any unexpected_exits_thresholds false
Unexpected Exits Monitoring Period The period to review when computing unexpected exits. 5 minute(s) unexpected_exits_window false

Other

Display Name Description Related Name Default Value API Name Required
Advertised Host If this is set this is the hostname that will be given out to producers, consumers, and other brokers to connect to. Never set this property on service level, it should always be overriden on instance level. advertised.host.name advertised.host.name false
Broker ID ID uniquely identifying each broker. Never set this property on service level, it should always be overriden on instance level. broker.id broker.id false
Java Heap Size of Broker in Megabytes Maximum size for the Java process heap memory. Passed to Java -Xmx. Measured in megabytes. broker_max_heap_size 256 MiB broker_max_heap_size false
HTTP Metric Report Host Host the HTTP metric reporter will bind to. kafka.http.metrics.host 0.0.0.0 kafka.http.metrics.host false
Data Directory A comma-separated list of one or more directories in which Kafka data is stored.. Each new partition that is created will be placed in the directory which currently has the fewest partitions.. Each directory should be on it's own separate drive. log.dirs /var/local/kafka/data log.dirs true
Data Retention Size The amount of data to retain in the log for each topic-partitions. Note that this is the limit per-partition so multiply by the number of partitions to get the total data retained for the topic. Also note that if both log.retention.hours and log.retention.bytes are both set, segments will be deleted when either limit is exceeded. log.retention.bytes -1 B log.retention.bytes false
Data Retention Time The number of hours to keep a log segment before it is deleted, i.e. the default data retention window for all topics. Note that if both log.retention.hours and log.retention.bytes are both set, segments are deleted when either limit is exceeded. log.retention.hours 168 log.retention.hours false
Time Based Roll Time in hours to roll a new segment file. This setting will force Kafka to roll a new log segment even if the log.segment.bytes size has not been reached. log.roll.hours 7 day(s) log.roll.hours false
Segment File Size The log for a topic partition is stored as a directory of segment files. This setting controls the size to which a segment file will grow before a new segment is rolled over in the log. log.segment.bytes 102 MiB log.segment.bytes false
Maximum Connections per IP Address Maximum number of connections allowed from each ip address. max.connections.per.ip 10 max.connections.per.ip false

Performance

Display Name Description Related Name Default Value API Name Required
Maximum Process File Descriptors If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value. rlimit_fds false

Ports and Addresses

Display Name Description Related Name Default Value API Name Required
Advertised Port The port to give out to producers, consumers, and other brokers to use in establishing connections. This only needs to be set if this port is different from the port the server should bind to. Never set this property on service level, it should always be overriden on instance level. advertised.port advertised.port false
JMX Port Port for JMX. jmx_port 9393 jmx_port false
HTTP Metric Report Port Port the HTTP metric reporter will listen on. kafka.http.metrics.port 24042 kafka.http.metrics.port false
TCP Port Kafka broker port. port 9092 port false

Resource Management

Display Name Description Related Name Default Value API Name Required
Cgroup CPU Shares Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager. cpu.shares 1024 rm_cpu_shares true
Cgroup I/O Weight Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager. blkio.weight 500 rm_io_weight true
Cgroup Memory Hard Limit Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit. memory.limit_in_bytes -1 MiB rm_memory_hard_limit true
Cgroup Memory Soft Limit Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit. memory.soft_limit_in_bytes -1 MiB rm_memory_soft_limit true

kafkamirrormakerdefaultgroup

Advanced

Display Name Description Related Name Default Value API Name Required
Kafka MirrorMaker Environment Advanced Configuration Snippet (Safety Valve) For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of this role except client configuration. KAFKA_MIRROR_MAKER_role_env_safety_valve false
Kafka MirrorMaker Logging Advanced Configuration Snippet (Safety Valve) For advanced use only, a string to be inserted into log4j.properties for this role only. log4j_safety_valve false
Kafka MirrorMaker Advanced Configuration Snippet (Safety Valve) for mirror_maker_consumers.properties For advanced use only, a string to be inserted into mirror_maker_consumers.properties for this role only. mirror_maker_consumers.properties_role_safety_valve false
Kafka MirrorMaker Advanced Configuration Snippet (Safety Valve) for mirror_maker_producers.properties For advanced use only, a string to be inserted into mirror_maker_producers.properties for this role only. mirror_maker_producers.properties_role_safety_valve false
Automatically Restart Process When set, this role's process is automatically (and transparently) restarted in the event of an unexpected failure. false process_auto_restart true

Logs

Display Name Description Related Name Default Value API Name Required
Kafka MirrorMaker Log Directory The log directory for log files of the role Kafka MirrorMaker. kafka_mirrormaker.log4j.dir /var/log/kafka log_dir false
Kafka MirrorMaker Logging Threshold The minimum log level for Kafka MirrorMaker logs INFO log_threshold false
Kafka MirrorMaker Maximum Log File Backups The maximum number of rolled log files to keep for Kafka MirrorMaker logs. Typically used by log4j or logback. 10 max_log_backup_index false
Kafka MirrorMaker Max Log Size The maximum size, in megabytes, per log file for Kafka MirrorMaker logs. Typically used by log4j or logback. 200 MiB max_log_size false

Monitoring

Display Name Description Related Name Default Value API Name Required
Enable Health Alerts for this Role When set, Cloudera Manager will send alerts when the health of this role reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold true enable_alerts false
Enable Configuration Change Alerts When set, Cloudera Manager will send alerts when this entity's configuration changes. false enable_config_alerts false
File Descriptor Monitoring Thresholds The health test thresholds of the number of file descriptors used. Specified as a percentage of file descriptor limit. Warning: 50.0 %, Critical: 70.0 % kafka_mirror_maker_fd_thresholds false
Kafka MirrorMaker Host Health Test When computing the overall Kafka MirrorMaker health, consider the host's health. true kafka_mirror_maker_host_health_enabled false
Kafka MirrorMaker Process Health Test Enables the health test that the Kafka MirrorMaker's process state is consistent with the role configuration true kafka_mirror_maker_scm_health_enabled false
Process Swap Memory Thresholds The health test thresholds on the swap memory usage of the process. Warning: Any, Critical: Never process_swap_memory_thresholds false
Role Triggers The configured triggers for this role. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed. Each trigger has all of the following fields:
  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific role.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger will not be evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here may lead to inconsistencies.
For example, the following JSON formatted trigger configured for a DataNode fires if the DataNode has more than 1500 file-descriptors opened:[{"triggerName": "sample-trigger", "triggerExpression": "IF (SELECT fd_open WHERE roleName=$ROLENAME and last(fd_open) > 1500) DO health:bad", "streamThreshold": 0, "enabled": "true"}]See the trigger rules documentation for more details on how to write triggers using tsquery.The JSON format is evolving and may change in the future and, as a result, backward compatibility is not guaranteed between releases at this time.		 [] role_triggers true
Unexpected Exits Thresholds The health test thresholds for unexpected exits encountered within a recent period specified by the unexpected_exits_window configuration for the role. Warning: Never, Critical: Any unexpected_exits_thresholds false
Unexpected Exits Monitoring Period The period to review when computing unexpected exits. 5 minute(s) unexpected_exits_window false

Other

Display Name Description Related Name Default Value API Name Required
Topic Blacklist Regular expression that represent a set of topics to avoid mirroring. Note that whitelist and blacklist parameters are mutually exclusive. If both are defined, only the whilelist is used. blacklist blacklist false
Destination Broker List List of brokers on destination cluster. This should be more than one, for high availability, but there's no need to list all brokers. bootstrap.servers bootstrap.servers true
Consumer Group ID Name of the consumer group used by MirrorMaker. When multiple role instances are configured with the same topics and same group ID, the role instances will load-balance replication for the topics. When multiple role instances are configured with with the same topics but different group ID, each role instance will replicate all the events for those topics - this can be used to replicate the source cluster into multiple destination clusters. group.id cloudera_mirrormaker group.id false
Avoid Data Loss Run with MirrorMaker settings that eliminate potential loss of data. This impacts performance but is highly recommended. no.data.loss true no.data.loss false
Number of Producers Number of producer instances. num.producers 1 num.producers false
Number of Consumer Threads Number of consumer threads. num.streams 1 num.streams false
Queue Size Maximum bytes that can be buffered between producer and consumer. queue.byte.size 100000000 B queue.byte.size false
Message Queue Size Number of messages that are buffered between producer and consumer. queue.size 10000 queue.size false
Topic Whitelist Regular expression that represent a set of topics to mirror. Note that whitelist and blacklist parameters are mutually exclusive. If both are defined, only the whilelist is used. whitelist whitelist false

Performance

Display Name Description Related Name Default Value API Name Required
Maximum Process File Descriptors If configured, overrides the process soft and hard rlimits (also called ulimits) for file descriptors to the configured value. rlimit_fds false

Ports and Addresses

Display Name Description Related Name Default Value API Name Required
JMX Port Port for JMX. jmx_port 9394 jmx_port false

Resource Management

Display Name Description Related Name Default Value API Name Required
Cgroup CPU Shares Number of CPU shares to assign to this role. The greater the number of shares, the larger the share of the host's CPUs that will be given to this role when the host experiences CPU contention. Must be between 2 and 262144. Defaults to 1024 for processes not managed by Cloudera Manager. cpu.shares 1024 rm_cpu_shares true
Cgroup I/O Weight Weight for the read I/O requests issued by this role. The greater the weight, the higher the priority of the requests when the host experiences I/O contention. Must be between 100 and 1000. Defaults to 1000 for processes not managed by Cloudera Manager. blkio.weight 500 rm_io_weight true
Cgroup Memory Hard Limit Hard memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit. memory.limit_in_bytes -1 MiB rm_memory_hard_limit true
Cgroup Memory Soft Limit Soft memory limit to assign to this role, enforced by the Linux kernel. When the limit is reached, the kernel will reclaim pages charged to the process if and only if the host is facing memory pressure. If reclaiming fails, the kernel may kill the process. Both anonymous as well as page cache pages contribute to the limit. Use a value of -1 B to specify no limit. By default processes not managed by Cloudera Manager will have no limit. memory.soft_limit_in_bytes -1 MiB rm_memory_soft_limit true

service_wide

Advanced

Display Name Description Related Name Default Value API Name Required
Kafka Service Environment Advanced Configuration Snippet (Safety Valve) For advanced use only, key-value pairs (one on each line) to be inserted into a role's environment. Applies to configurations of all roles in this service except client configuration. KAFKA_service_env_safety_valve false
System Group The group that this service's processes should run as. kafka process_groupname true
System User The user that this service's processes should run as. kafka process_username true

Monitoring

Display Name Description Related Name Default Value API Name Required
Enable Service Level Health Alerts When set, Cloudera Manager will send alerts when the health of this service reaches the threshold specified by the EventServer setting eventserver_health_events_alert_threshold true enable_alerts false
Enable Configuration Change Alerts When set, Cloudera Manager will send alerts when this entity's configuration changes. false enable_config_alerts false
Service Triggers The configured triggers for this service. This is a JSON formatted list of triggers. These triggers are evaluated as part as the health system. Every trigger expression is parsed, and if the trigger condition is met, the list of actions provided in the trigger expression is executed. Each trigger has all of the following fields:
  • triggerName (mandatory) - The name of the trigger. This value must be unique for the specific service.
  • triggerExpression (mandatory) - A tsquery expression representing the trigger.
  • streamThreshold (optional) - The maximum number of streams that can satisfy a condition of a trigger before the condition fires. By default set to 0, and any stream returned causes the condition to fire.
  • enabled (optional) - By default set to 'true'. If set to 'false', the trigger will not be evaluated.
  • expressionEditorConfig (optional) - Metadata for the trigger editor. If present, the trigger should only be edited from the Edit Trigger page; editing the trigger here may lead to inconsistencies.
For example, the followig JSON formatted trigger fires if there are more than 10 DataNodes with more than 500 file-descriptors opened:[{"triggerName": "sample-trigger", "triggerExpression": "IF (SELECT fd_open WHERE roleType = DataNode and last(fd_open) > 500) DO health:bad", "streamThreshold": 10, "enabled": "true"}]See the trigger rules documentation for more details on how to write triggers using tsquery.The JSON format is evolving and may change in the future and, as a result, backward compatibility is not guaranteed between releases at this time.		 [] service_triggers true
Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) For advanced use only, a list of derived configuration properties that will be used by the Service Monitor instead of the default ones. smon_derived_configs_safety_valve false

Other

Display Name Description Related Name Default Value API Name Required
Topic Auto Creation Enable auto creation of topic on the server. If this is set to true then attempts to produce, consume, or fetch metadata for a non-existent topic will automatically create it with the default replication factor and number of partitions. auto.create.topics.enable true auto.create.topics.enable false
Enable Automatic Leader Rebalancing If this is enabled the controller will automatically try to balance leadership for partitions among the brokers by periodically returning leadership to the preferred replica for each partition if it is available. auto.leader.rebalance.enable true auto.leader.rebalance.enable false
Controlled Shutdown Enable controlled shutdown of the broker. If enabled, the broker will move all leaders on it to some other brokers before shutting itself down. This reduces the unavailability window during shutdown. controlled.shutdown.enable true controlled.shutdown.enable false
Replication Factor The default replication factor for automatically created topics. default.replication.factor 1 default.replication.factor false
Enables Delete Topic When delete topic is disabled, deleting topics through the admin tools has no effect. delete.topic.enable true delete.topic.enable false
List of Metric Reporters Comma-separated list of class names of metric reporters. HTTP reporter is included by default. kafka.metrics.reporters nl.techop.kafka.KafkaHttpMetricsReporter kafka.metrics.reporters false
Maximal Message Size The maximum size of a message that the server can receive. It is important that this property be in sync with the maximum fetch size the consumers use or else an unruly producer will be able to publish messages too large for consumers to consume. message.max.bytes 1000000 B message.max.bytes false
Minimum Number of Replicas in ISR Define the minimum number of replicas in ISR needed to satisfy a produce request with required.acks=-1 (or all). min.insync.replicas 1 min.insync.replicas false
Enable Kafka Monitoring (Note: Requires Kafka-1.3.0 parcel or higher) Enable Kafka monitoring? (Note: Requires Kafka-1.3.0 parcel or higher). monitoring.enabled true monitoring.enabled false
Default Number of Partitions The default number of partitions per topic. num.partitions 1 num.partitions false
Replica Fetch Size The number of byes of messages to attempt to fetch for each partition in the fetch requests the replicas send to the leader. replica.fetch.max.bytes 1 MiB replica.fetch.max.bytes false
Allowed Replica Lag If a replica falls more than this many messages behind the leader, the leader will remove the follower from ISR and treat it as dead. replica.lag.max.messages 4000 replica.lag.max.messages false
Enable Unclean Leader Election Enable replicas not in the ISR set to be elected as leader as a last resort, even though doing so may result in data loss. unclean.leader.election.enable false unclean.leader.election.enable false
Zookeeper Root ZNode in zookeeper that should be used as a root for this Kafka cluster. zookeeper.chroot zookeeper.chroot false
Zookeeper Session Timeout Zookeeper session timeout. If the consumer fails to heartbeat to zookeeper for this period of time it is considered dead and a rebalance will occur. zookeeper.session.timeout.ms 6 second(s) zookeeper.session.timeout.ms false
ZooKeeper Service Name of the ZooKeeper service that this Kafka service instance depends on zookeeper_service true