Registering a Custom Source in SAM for AWS Kinesis
To register any custom source in SAM, there are three artifacts you need:
Artifact 1: Code for the custom source using the underlying streaming engine. Since SAM today supports Storm as the Streaming engine, you can refer to the following artifacts for the custom source:
Artifact 2: Code for mapping the SAM configs to the custom source/spout. Refer to the following artifacts for this mapping code:
Artifact 3: Flux mapping file to map the SAM config to the Kinesis Spout. Refer to the following artifacts
More Details on implementing a custom source and registering with SAM can be found here: https://github.com/hortonworks/streamline/tree/master/examples/sources
To register the custom Kinesis Source in SAM using the above three artifacts, perform the following steps:
Download the Sam-Custom-Extensions.zip to the host where SAM is installed (if you haven’t done it in a past step)
Unzip the contents. We will call the unzipped folder $SAM_EXTENSIONS
Switch to user streamline:
sudo su streamline
Install Artifact 1 (the custom source code) on host’s local maven repo
cd $SAM_EXTENSIONS/custom-source/kinesis/ mvn install:install-file -Dfile=storm-kinesis-1.1.0.5.jar \ -DgroupId=org.apache.storm \ -DartifactId=storm-kinesis \ -Dversion=1.1.0.5 \ -Dpackaging=jar
Register the custom source via SAM REST call. Replace SAM_HOST and SAM_PORT.
curl -sS -X POST -i -F \ topologyComponentBundle=@config/kinesis-source-topology-component.json -F \ bundleJar=@sam-custom-source-kinesis.jar \ http://SAM_HOST:SAM_PORT/api/v1/catalog/streams/componentbundles/SOURCE
If the registration was successful, you should see a message like the following by the REST response:
HTTP/1.1 201 Created Date: Wed, 03 Jan 2018 20:26:22 GMT Content-Type: application/json Content-Length: 4569
On the SAM Application Canvas Palette, you should now see KINESIS source.
Dragging the kinesis source onto the canvas and double clicking it, you should see the following kinesis dialog. The dialog properties comes from the topologyComponentBundle flux config you used to register the custom source.
