Using Apache Storm
Also available as:
PDF

KafkaSpout Limitations

Limitations

The current version of the Kafka spout contains the following limitations:

  • Does not support Kafka 0.7x brokers.

  • Storm developers must include ${STORM_HOME}/lib/* in the CLASSPATH environment variable from the command line when running kafka-topology in local mode. Otherwise, developers will likely receive a java.lang.NoClassDefFoundError exception:

    java -cp "/usr/lib/storm/contrib/storm-kafka-example-0.9.1.2.1.1.0-320-jar-with-dependencies.jar:
     /usr/lib/storm/lib/*" org.apache.storm.kafka.TestKafkaTopology <zookeeper_host>
  • Secure Hadoop clusters must comment out the following statement from ${STORM_HOME}/bin/kafka-server-start.sh:

    EXTRA_ARGS="-name kafkaServer -loggc"
  • Core-storm API Constructor

    public SpoutConfig(BrokerHosts hosts, String topic, String zkRoot, String id)

    Table 5.5. SpoutConfig Parameters

    SpoutConfig Parameter

    Description

    hosts

    Any implementation of the BrokerHosts interface, currently either ZkHosts or StaticHosts.

    topic

    Name of the Kafka topic.

    zkroot

    Root directory in ZooKeeper where all topics and partition information is stored. By default, this is /brokers.

    id

    Unique identifier for this spout.


  • Trident API Constructors

    public TridentKafkaConfig(BrokerHosts hosts, String topic)
    public TridentKafkaConfig(BrokerHosts hosts, String topic, String id)

    Table 5.6. TridentKafkaConfig Parameters

    TridentKafkaConfig

    Description

    hosts

    Any implementation of the BrokerHosts interface, currently either ZkHosts or StaticHosts.

    topic

    Name of the Kafka topic.

    clientid

    Unique identifier for this spout.