Chapter 8. Packaging Storm Topologies

Storm developers should verify that the following conditions are met when packaging their topology into a .jar file:

Use the maven-shade-plugin, rather than the maven-assembly-plugin to package your Apache Storm topologies. The maven-shade-plugin provides the ability to merge JAR manifest entries, which are used by the Hadoop client to resolve URL schemes.
Include a dependency for the Hadoop version used in the Hadoop cluster.
Include both of the Hadoop configuration files, hdfs-site.xml and core-site.xml, in the .jar file. In addition, include any configuration files for HDP components used in your Storm topology, such as hive-site.xml and hbase-site.xml. This is the easiest way to meet the requirement that all required configuration files appear in the CLASSPATH of your Storm topology at runtime.

Maven Shade Plugin

Use the maven-shade-plugin, rather than the maven-assembly-plugin to package your Apache Storm topologies. The maven-shade-plugin provides the ability to merge JAR manifest entries, which are used by the Hadoop client to resolve URL schemes.

Use the following Maven configuration file to package your topology:

 <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-shade-plugin</artifactId>
      <version>1.4</version>
      <configuration>
            <createDependencyReducedPom>true</createDependencyReducedPom>
      </configuration>
      <executions>
           <execution>
                <phase>package</phase>
                <goals>
                     <goal>shade</goal>
                </goals>
                <configuration>
                     <transformers>
                          <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
                          <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                <mainClass></mainClass>
                          </transformer>
                     </transformers>
                </configuration>
           </execution>
      </executions>
</plugin>

Hadoop Dependency

Include a dependency for the Hadoop version used in the Hadoop cluster; for example:

<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-client</artifactId>
    <version>2.7.1.2.3.2.0-2950</version>
    <exclusions>
        <exclusion>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-log4j12</artifactId>
        </exclusion>
    </exclusions>
</dependency>

Troubleshooting

The following table describes common packaging errors.

Table 8.1. Topology Packing Errors

Error	Description
`com.google.protobuf. InvalidProtocolBufferException: Protocol message contained an invalid tag (zero)`	Hadoop client version incompatibility
`java.lang.RuntimeException: Error preparing HdfsBolt: No FileSystem for scheme: hdfs`	The .jar manifest files have not properly merged in the topology.jar

​Chapter 8. Packaging Storm Topologies

Chapter 8. Packaging Storm Topologies