Cloudera Docs

JBOD setup

Learn how to set up JBOD in your Kafka environment.

Consider the following before using JBOD support in Kafka:
  • Manual operation and administration: Monitoring offline directories and JBOD related metrics is done through Cloudera Manager. However, identifying failed disks and rebalancing partitions between disks is done manually.
  • Manual load balancing between disks: Unlike with RAID-10, JBOD does not automatically distribute data across disks. The process is fully manual.

To provide robust JBOD support in Kafka, changes in the Kafka protocol have been made. When performing an upgrade to a new version of Kafka, make sure that you follow the recommended rolling upgrade process.

For more information regarding the JBOD related Kafka protocol changes, see KIP-112 and KIP-113.

  1. Mount the required number of disks on your system.
  2. In Cloudera Manager, set up log directories for all Kafka brokers:
    1. Go to the Kafka service, select Instances and select the broker.
    2. Go to Configuration and find the Data Directories property.
    3. Modify the path of the log directories so that they correspond with the newly mounted disks.
    4. Enter a Reason for change, and then click Save Changes to commit the changes.
  3. Go to the Kafka service and select Configuration.
  4. Find and configure the following properties depending on your system and use case.
    • Number of I/O Threads
    • Number of Network Threads
    • Number of Replica Fetchers
    • Minimum Number of Replicas in ISR
  5. Set replication factor to at least 3.
  6. Restart the service:
    1. Return to the home page by clicking hte Cloudera Manager logo.
    2. Go to the Kafka service and select Actions > Rolling Restart
    3. Check the Restart roles with stale configurations only checkbox and click Rolling restart.
    4. Click Close when the restart has finished.
JBOD disks are set up in your Kafka environment.