Running the Stream Simulator
Now that you have developed and deployed the NiFi Flow Application and the Stream Analytics Application, you can run a data simulator that generates truck geo events and sensor events for the apps to process.
To generate the raw truck events serialized into Avro objects using the Schema registry and publish them into the raw Kafka topics, do the following:
- Download the Data-Loader.
- Unzip the Data Loader file and copy it to the cluster. Lets call the directory to which you unzipped the file as $DATA_LOADER_HOME.
Change into the Data Loader directory:
Untar the route.tar.gz file:
tar -zxvf $DATA_LOADER_HOME/routes.tar.gz
Open the file startTruckGenerators.sh and make the following changes:
- Modify the kafkaBrokers value based on your cluster
- If your cluster is not secure, set the value of SECURE_MODE to NONSECURE and set JAAS_CONFIG to an empty space
- Set the value of ROUTES_LOCATION to the location where you untar the routes in step 4 and then (e.g: $DATA_LOADER_HOME/routes/midwest)
Run the simulator/data generator:
You should see events being sent to the gateway kafka topics. NiFi should consume the events, enrich them, and then push them into the syndicate topics. SAM, Spark Streaming and other apps should consume from these syndicate topics