Component Tuning Variables Overview

There are a number of services that you can use to tune the performance of your CCP cluster. These services include Kafka, Storm, and HDFS. Within these services, you can modify parsers, enrichment, and indexing (Elasticsearch or Solr).

When you consider tuning your CCP architecture, it is important to note where you can modify settings. For example, Storm gives you the ability to independently set tasks in executors for parser topologies. This is important if you want to set the number of tasks higher than the number of executors to accommodate for future performance tuning and rebalancing without the need to bring down your topologies. However, for enrichment and indexing topologies, CCP uses Flux, and there is no method for specifying the number of tasks from the number of executors in Flux. By default, the number of tasks equals the number of executors.

The following lists the major properties for each service that you can modify to tune your cluster:

• Kafka partitions

• Storm

  • Kafka spout

    • Polling frequency

    • Polling timeouts

    • Offset commit period

    • Max uncommitted offsets

  • Storm workers (OS processes)

  • Storm executors (threads in a process)

  • Storm ackers

  • Max spout pending

  • Spout and bolt parallelism

• HDFS

  • Batch size

  • Replication factor

• Indexing
  • Elasticsearch templates
  • Solr