Upload Custom Processors

« Prev

Steps for Uploading the ENRICH-PHOENIX Custom Processor

From the left-hand menu, click Configuration, then Application Resources.
Select Custom Processor and click the + sign to create the ENRICH-PHOENIX processor. Configure the processor with the following values. This processor can be used to enriched streams with data from Phoenix based on a user supplied sql statement. The java class for this processor using the SAM SDK can be found here.

ENRICH-PHOENIX Configuration Values
- Streaming Engine – Storm
- Name – ENRICH-PHOENIX
- Description – Enriches the input schema with data from Phoenix based on user supplied SQL
- ClassName – hortonworks.hdf.sam.custom.processor.enrich.phoenix.PhoenixEnrichmentSecureProcessor
- Upload Jar – The jar for this custom processor can be found under SAM_EXTENSIONS/sam-custom-processor-0.0.5-jar-with-dependencies.jar
Click the Add Config Fields button and and the following 3 configuration fields
- Add a config field called zkServerUrl with the following values:
  1. Field Name – zkServerUrl
  2. UI Name – Phoenix ZooKeeper Connection URL
  3. Optional – false
  4. Type – string
  5. ToolTip – ZooKeeper server url in the format of $FQDN_ZK_HOST:$ZK_PORT
- Add a config field called enrichmentSQL with the following values
  1. Field Name – enrichmentSQL
  2. UI Name – Enrichment SQL
  3. Optional – false
  4. Type – string
  5. ToolTip – SQL to execute for the enriched values
- Add a config field called enrichedOutputFields with the following values:
  1. Field Name – enrichedOutputFields
  2. UI Name – Enrichment Output Fields
  3. Optional – false
  4. Type – string
  5. ToolTip – The output field names to store new enriched values
- Add a config field called secureCluster with the following values:
  1. Field Name – secureCluster
  2. UI Name – Secure Cluster
  3. Optional – false
  4. Type – boolean
  5. ToolTip – Check if connecting to a secure HBase/Phoenix Cluster
- Add a config field called kerberosClientPrincipal with the following values:
  1. Field Name – kerberosClientPrincipal
  2. UI Name – Kerberos Client Principal
  3. Optional – true
  4. Type – string
  5. ToolTip – The principal uses to connect to secure HBase/PHoenix Cluster. Required if secureCluster is checked
- Add a config field called kerberosKeyTabFile with the following values:
  1. Field Name – kerberosKeyTabFile
  2. UI Name – Kerberos Key Tab File
  3. Optional – true
  4. Type – string
  5. ToolTip – Kerberos Key Tab File location on each of the worker nodes for thee principal configured
Input and Output Schema for ENRICH-PHOENIX
- Copy this input schema and paste into the INPUT SCHEMA text area box
- Copy this output schema and paste into the OUTPUT SCHEMA text area box

Steps for Uploading the ENRICH-WEATHER Custom Processor

Select Custom Processor and click the + sign to create the ENRICH-WEATHER processor. This processor can be used to enrich streams with weather data based on time and lat/long location. The java class for this processor using the SAM SDK can be found here.
Configure the processor with the following values.

ENRICH-WEATHER Configuration Values

Streaming Engine – Storm
Name – ENRICH-WEATHER
Description – Enrichment with normalized weather data for a geo location
ClassName – hortonworks.hdf.sam.custom.processor.enrich.weather.WeatherEnrichmentProcessor
Upload Jar – The jar for this custom processor can be found under SAM_EXTENSIONS/sam-custom-processor-0.0.5.jar

Click the Add Config Fields button and and a configuration field with the following values:

Field Name – weatherServiceURL
UI Name – Weather Web Service URL
Optional – false
Type – string
Tooltip – The URL to the Weather Web Service

Input and Output Schema for ENRICH-WEATHER

Copy this input schema and paste into the INPUT SCHEMA text area box
Copy this output schema and paste into the OUTPUT SCHEMA text area box

Steps for Uploading the NORMALIZE-MODEL-FEATURES Custom Processor

Select the Custom Processor Tab and click the + sign to create the NORMALIZE-MODEL-FEATURES processor. This processor is to normalized the enriched fields to format that the model is expecting.
Configure the processor with the following values.

NORMALIZE-MODEL-FEATURES Configuration Values

Streaming Engine – Storm
Name – NORMALIZE-MODEL-FEATURES
Description – Normalize the features of the model before passing it to model
ClassName – hortonworks.hdf.sam.custom.processor.enrich.driver.predictivemodel.FeatureNormalizationProcessor
Upload Jar – The jar for this custom processor can be found under SAM_EXTENSIONS/sam-custom-processor-0.0.5a.jar

Input and Output Schema for NORMALIZE-MODEL-FEATURES

Copy this input schema and paste into the INPUT SCHEMA text area box
Copy this output schema and paste into the OUTPUT SCHEMA text area box

Result

We have uploaded three custom processors required to do enrichment of the stream and normalization of the enriched values to feed into the model.

If you go back to the Stream Builder, you will see three new custom processors on palette.

​Upload Custom Processors

Upload Custom Processors