Getting Started with Streaming Analytics
Also available as:
PDF
loading table of contents...

Contents

1. Building an End-to-End Stream Application
Understanding the Use Case
Reference Architecture
2. Prepare Your Environment
Deploying Your HDF Clusters
Registering Schemas in Schema Registry
Create the Kafka Topics
Register Schemas for the Kafka Topics
Setting up an Enrichment Store, Creating an HBase Table, and Creating an HDFS Directory
3. Creating a Dataflow Application
Data Producer Application Generates Events
NiFi: Create a Dataflow Application
NiFi Controller Services
NiFi Ingests the Raw Sensor Events
Publish Enriched Events to Kafka for Consumption by Analytics Applications
Start the NiFi Flow
4. Creating a Stream Analytics Application
Two Options for Creating the Streaming Analytics Applications
Setting up the Stream Analytics App using the TruckingRefAppEnvEnviornmentBuilder
Configuring and Deploying the Reference Application
Creating a Service Pool and Environment
Creating Your First Application
Creating and Configuring the Kafka Source Stream
Connecting Components
Joining Multiple Streams
Filtering Events in a Stream using Rules
Using Aggregate Functions over Windows
Implementing Business Rules on the Stream
Transforming Data using a Projection Processor
Streaming Alerts to an Analytics Engine for Dashboarding
Streaming Violation Events to an Analytics Engine for Descriptive Analytics
Streaming Violation Events into a Data Lake and Operational Data Store
5. Deploy an Application
Configure Deployment Settings
Deploy the App
Running the Stream Simulator
6. Advanced: Performing Predictive Analytics on the Stream
Logistical Regression Model
Export the Model into SAM's Model Registry
Enrichment and Normalization of Model Features
Upload Custom Processors and UDFs for Enrichment and Normalization
Upload Custom UDFs
Upload Custom Processors
Scoring the Model in the Stream using a Streaming Split Join Pattern
Streaming Split Join Pattern
Score the Model Using the PMML Processor and Alert
7. Creating Visualizations Using Superset
Creating Insight Slices
Adding Insight Slices to a Dashboard
Dashboards for the Trucking IOT App
8. SAM Test Mode
Four Test Cases using SAM’s Test Mode
Test Case 1: Testing Normal Event with No Violation Prediction
Analyzing Test Case 1 Results
Test Case 2: Testing Normal Event with Yes Violation Prediction
Analyzing Test Case 2 Results
Test Case 3: Testing Violation Event
Analyzing Test Case 3 Results
Test Case 4: Testing Multiple-Speeding-Events
Analyzing Test Case 4 Results
Running SAM Test Cases as Junit Tests in CI Pipelines
9. Creating Custom Sources and Sinks
Cloud Use Case: Integration with AWS Kinesis and S3
Registering a Custom Source in SAM for AWS Kinesis
Registering a Custom Sink in SAM for AWS S3
Implementing the SAM App with Kinesis Source and S3 Sink
10. Stream Operations
My Applications View
Application Performance Monitoring
Exporting and Importing Stream Applications
Troubleshooting and Debugging a Stream Application
Monitoring SAM Apps and Identifying Performance Issues
Identifying Processor Performance Bottlenecks
Debugging an Application through Distributed Log Search
Debugging an Application through Sampling