Storm-HDFS: Trident APIs
The Trident API implements a StateFactory
class with an API that
resembles the methods from the storm-code
API, as shown in the
following code sample:
... Fields hdfsFields = new Fields("field1", "field2"); FileNameFormat fileNameFormat = new DefaultFileNameFormat() .withPrefix("trident") .withExtension(".txt") .withPath("/trident"); RecordFormat recordFormat = new DelimitedRecordFormat() .withFields(hdfsFields); FileRotationPolicy rotationPolicy = new FileSizeRotationPolicy(5.0f, FileSizeRotationPolicy.Units.MB); HdfsState.Options options = new HdfsState.HdfsFileOptions() .withFileNameFormat(fileNameFormat) .withRecordFormat(recordFormat) .withRotationPolicy(rotationPolicy) .withFsUrl("hdfs://localhost:8020"); StateFactory factory = new HdfsStateFactory().withOptions(options); TridentState state = stream.partitionPersist(factory, hdfsFields, new HdfsUpdater(), new Fields());
See the javadoc for the Trident API, included with the storm-hdfs
connector, for more information.
Limitations
Directory and file names changes are limited to a prepackaged file name format based on a timestamp.