Setting up an Enrichment Store, Creating an HBase Table, and Creating an HDFS Directory
To prepare to perform predictive analytics on streams , you need some HBase and Phoenix tables. This section gives you instructions on setting up the HBase and Phoenix tables timesheet and drivers, loading them with reference data, and downloading the custom UDFs and processors to perform the enrichment and normalization.
Install HBase/Phoenix and download the sam-extensions
- If HBase is not installed, install/add an HBase service.
- Ensure that Phoenix is enabled on the HBase Cluster.
- Download the Sam-Custom-Extensions.zip and save it to your local machine.
- Unzip the contents. Name the unzipped folder $SAM_EXTENSIONS.
Steps for Creating Phoenix Tables and Loading Reference Data
- Copy the $SAM_EXTENSIONS/custom-processor/scripts.tar.gz to a node where HBase/Phoenix client is installed.
- On that node, untar the scripts.tar.gz. Name the directory
$SCRIPTS.
tar -zxvf scripts.tar.gz
- Navigate to the directory where the phoenix script is located which will create the
phoenix tables for enrichment and load it with reference data.
cd $SCRIPTS/phoenix
- Open the file phoenix_create.sh and replace <ZK_HOST> with the FQDN of your ZooKeeper host.
- Make the phoenix_create.sh script executable and execute it. Make
sure you add the script to JAVA_HOME.
./phoenix_create.sh
Steps for Verifying Data has Populated Phoenix Tables
- Start up the sqlline Phoenix
client.
cd /usr/hdp/current/phoenix-client/bin ./sqlline.py $ZK_HOST:2181:/hbase-unsecure
- List all the tables in Phoenix.
!tables
- Query the drivers and timesheet
tables.
select * from drivers; select * from timesheet;
Steps for Starting HBase and Creating an HBase Table
- This can be easily done by adding the HDP HBase Service using Ambari.
- Create a new HBase table by logging into an node where Hbase client is installed then
execute the following
commands:
cd /usr/hdp/current/hbase-client/bin /hbase shell create 'violation_events', {NAME=> 'events', VERSIONS => 3} ;
Steps for Creating an HDFS Directory
Create the following directory in HDFS and give it access to all users.
- Log into a node where HDFS client is installed.
- Execute the following
commands:
su hdfs hadoop fs -mkdir /apps/trucking-app hadoop fs -chmod 777 /apps/trucking-app