Upload Custom Processors

Steps for Uploading the ENRICH-PHOENIX Custom Processor

1. From the left-hand menu, click Configuration, then Application Resources.
2. Select Custom Processor and click the + sign to create the ENRICH-PHOENIX processor.

   Configure the processor with the following values. This processor can be used to enriched streams with data from Phoenix based on a user supplied SQL statement. The java class for this processor using the SAM SDK can be found here.

   

   ENRICH-PHOENIX Configuration Values

   • Streaming Engine – Storm

   • Name – ENRICH-PHOENIX

   • Description – Enriches the input schema with data from Phoenix based on user supplied SQL

   • ClassName – hortonworks.hdf.sam.custom.processor.enrich.phoenix.PhoenixEnrichmentSecureProcessor

   • Upload Jar – The jar for this custom processor can be found under SAM_EXTENSIONS/custom-processor/sam-custom-processor-jar-with-dependencies.jar

   Click the Add Config Fields button and the following three configuration fields:

   • Add a config field called zkServerUrl with the following values:

     1. Field Name – zkServerUrl
     2. UI Name – Phoenix ZooKeeper Connection URL
     3. Optional – false
     4. Type – string
     5. ToolTip – ZooKeeper server URL in the format of $FQDN_ZK_HOST:$ZK_PORT

   • Add a config field called enrichmentSQL with the following values:

     1. Field Name – enrichmentSQL
     2. UI Name – Enrichment SQL
     3. Optional – false
     4. Type – string
     5. ToolTip – SQL to execute for the enriched values

   • Add a config field called enrichedOutputFields with the following values:

     1. Field Name – enrichedOutputFields
     2. UI Name – Enrichment Output Fields
     3. Optional – false
     4. Type – string
     5. ToolTip – The output field names to store new enriched values

   • Add a config field called secureCluster with the following values:

     1. Field Name – secureCluster
     2. UI Name – Secure Cluster
     3. Optional – false
     4. Type – boolean
     5. ToolTip – Check if connecting to a secure HBase/Phoenix Cluster

   • Add a config field called kerberosClientPrincipal with the following values:

     1. Field Name – kerberosClientPrincipal
     2. UI Name – Kerberos Client Principal
     3. Optional – true
     4. Type – string
     5. ToolTip – The principal uses to connect to secure HBase/Phoenix Cluster. Required if secureCluster is checked.

   • Add a config field called kerberosKeyTabFile with the following values:

     1. Field Name – kerberosKeyTabFile
     2. UI Name – Kerberos Key Tab File
     3. Optional – true
     4. Type – string
     5. ToolTip – Kerberos Key Tab File location on each of the worker nodes for the configured principal

Steps for Uploading the ENRICH-WEATHER Custom Processor

1. Select Custom Processor and click the + sign to create the ENRICH-WEATHER processor. This processor can be used to enrich streams with weather data based on time and lat/long location. The java class for this processor using the SAM SDK can be found here.
2. Configure the processor with the following values.

   

ENRICH-WEATHER Configuration Values

• Streaming Engine – Storm

• Name – ENRICH-WEATHER

• Description – Enrichment with normalized weather data for a geo location

• ClassName – hortonworks.hdf.sam.custom.processor.enrich.weather.WeatherEnrichmentProcessor

• Upload Jar – The jar for this custom processor can be found under SAM_EXTENSIONS/custom-processor/sam-custom-processor.jar

Click the Add Config Fields button and a configuration field with the following values:

• Field Name – weatherServiceURL

• UI Name – Weather Web Service URL

• Optional – false

• Type – string

• Tooltip – The URL to the Weather Web Service

Input and Output Schema for ENRICH-WEATHER

• Copy this input schema and paste into the INPUT SCHEMA text area box

• Copy this output schema and paste into the OUTPUT SCHEMA text area box

Steps for Uploading the NORMALIZE-MODEL-FEATURES Custom Processor

1. Select the Custom Processor tab and click the + sign to create the NORMALIZE-MODEL-FEATURES processor. This processor normalizes the enriched fields to a format that the model is expecting.
2. Configure the processor with the following values:

   

NORMALIZE-MODEL-FEATURES Configuration Values

• Streaming Engine – Storm

• Name – NORMALIZE-MODEL-FEATURES

• Description – Normalize the features of the model before passing it to model

• ClassName – hortonworks.hdf.sam.custom.processor.enrich.driver.predictivemodel.FeatureNormalizationProcessor

• Upload Jar – The jar for this custom processor can be found under SAM_EXTENSIONS/custom-processor/sam-custom-processor.jar

Input and Output Schema for NORMALIZE-MODEL-FEATURES

• Copy this output schema and paste into the OUTPUT SCHEMA text area box

Result

You have uploaded three custom processors required to do enrichment of the stream and normalization of the enriched values to feed into the model.

If you go back to the Stream Builder, you will see three new custom processors on the palette.