Monitoring log size information

The SMM UI shows log size related information about brokers, topics, and partitions. Furthermore, warning messages appear when log directory related errors happen.

The feature uses a cache mechanism. The following cache attributes are set by default. You can modify these attributes in the SMM configuration page of the Cloudera Manager UI:
  • Kafka Logdir Cache Expiration ms

    Sets cache expiration time.

  • Kafka Logdir Cache Maximum Size

    Sets maximum cache size.

  • Kafka Logdir Call Timeout ms

    Sets timeout of the Kafka query.

The following image shows the properties in Cloudera Manager:

Log size details for topics

On the Overview page of the SMM UI, the Topics tab contains a sortable column called Current Log Size which shows the data with the measure unit. This log size is calculated by the leader partitions; so, the follower partitions are not included. The column contains a tooltip which is a question-mark symbol. The topic related log-size improvements or information are also available on the Topics page.

The Topics tab on the Overview page represents topics and you can select one of them. In this case the topic’s leader partitions are viewable and here you can find the leader partition log-size. If you click a partition, a pop-up window appears, where log-size information is displayed.

If you click Profile for a topic, log-size of replica-leaders appears, hosted by the broker.

Log size details for brokers

On the Overview page, the Brokers tab contains two sortable columns called Log Size and Remaining Storage. The log-size is calculated for all the partitions including the follower partitions. The remaining storage column contains the available capacity in all of the configured log directories. The broker related log-size improvements or information are also available on the Brokers page.

If you click the arrow icon for a broker, the details of that broker are displayed. The Leader Partition Log-Size shows the sum of leader partitions’ log-size. The All Partitions Log-Size field’s value is equal to the sortable column’s value called Log Size (naming is different to make the sortable column names as short as possible). The Remaining Storage sortable column and field is the same (value and name).

If you click a specific partition, a pop-up window appears where log-size information is displayed.

If you click Profile of a broker, then the metrics details for that broker are displayed. The log-size information per partition is available. Furthermore, the sum of leader partitions’ log-size, remaining storage size, and all partitions log-size details are also available. The Metrics tab on the Brokers page contains the available storage size per log directory.

If there is any issue with the query on the log directory information, then a warning message appears. There can be multiple warnings if more than one broker is related. But no new warning message from a specific broker appears until the previous message disappears.

Cloudera Manager chart builder

For historical information, you can use the chart builder provided in Cloudera Manager. The following command and screenshot show Kafka log-size data from the past where the broker ID and topic name with the topic partition are specified.
SELECT counter_sum(kafka_bytes_received_by_partition_rate) WHERE serviceName=KafkaServiceName AND kafkaBrokerId = KafkaBrokerId AND kafkaTopicName = KafkaTopicName AND kafkaPartitionId = kafkapartitionId