Parse the Squid Telemetry Event

Parsers transform raw data into JSON messages suitable for downstream enrichment and indexing by HCP. There is one parser for each data source and HCP pipes the information to the Enrichment/Threat Intelligence topology. You can transform the field output in the JSON messages into information and formats that make the output more useful. For example, you can change the timestamp field output from GMT to your timezone.

1. Launch the Management UI:
   1. From the Ambari Dashboard panel, click Metron.
      Make sure you have the Summary tab selected.
   2. Select Management UI from Quick Links.
      The Management UI tool should display in a separate browser tab.
2. Click Sensors on the left side of the window, under Operations.
3. Click (add button) in the lower right corner of the screen.
   The Management UI displays a panel used to create the new sensor.
   
4. Enter the following information:

   Name: mysquid
   Kafka Topic: mysquid
   Parser Type: Grok

   If a Kafka topic already exists for the sensor name, the UI displays a message similar to Kafka Topic Exists. Emitting and displays the name of the Parser Type. You can skip to the next section, Verify Events are Indexed.
5. Create a Grok statement for the new parser:
   1. In the Grok Statement box, click the (expand window) to display the Grok Validator panel.
   2. Choose or enter a name for the Grok statement in the PATTERN LABEL field.
      For our example, we chose MYSQUID.
   3. Enter a sample log entry for the data source.
      To create sample log entries, see Step 15 in Create a NiFi Flow to Stream Events to HCP.

      
      
      

   4. Refer to the format of the log entries you determined in Step 11d in Create a NiFi Flow to Stream Events to HCP.
      For example, the log entry format for Squid is:

      timestamp | time elapsed | remotehost | code/status | bytes | method | URL rfc931 peerstatus/peerhost | type

   5. In the STATEMENT field, enter the first element in the sensor log file.
      For Squid, the first element in the sensor log file is timestamp which is a number, so we enter %{NUMBER:timestamp}.

      	Note
      	The Management UI automatically completes partial words in your Grok statement as you enter them.

      
      
      

   6. Click TEST.
      If the validator finds an error, it displays the error information. If the validation succeeds, it displays the valid mapping in the PREVIEW field.
      Because you entered the timestamp element, the validator parses the timestamp correctly and leaves the rest of the information as random data.

      
      
      

   7. Click Save.
   8. Verify that the sensor name and topic name are "mysquid" with NO extra spaces or special characters.
6. Click Save on the mysquid sensor.
   The mysquid sensor appears in the Management UI. You might need to refresh your screen to see the new sensor.
7. Click the pencil icon to edit the mysquid sensor.
8. Scroll down to the Parser Config section.
9. In the first open field, indicated by enter field, enter timestampField.
   
10. In next open field, enter timestamp.
11. Click Save.
12. Continue to build and test the Grok statement until you have entries for each element in the log entry.
    For example, for Squid, the second element is elapsed, so we enter %{INT:elapsed} and click TEST again.

    
    
    

13. When your Grok statement is complete and valid, click SAVE to save the Grok statement for the sensor.
14. Click SAVE to save the sensor information and add it to the list of Sensors.
    This new data source processor topology ingests from the $Kafka topic and then parses the event with the HCP Grok framework using the Grok pattern. The result is a standard JSON Metron structure that then is added to the "enrichment" Kafka topic for further processing.
15. Test that a Kafka topic has been created for the Squid parser:
    1. Navigate to the following directory:

       /usr/hdp/current/kafka-broker/bin

    2. List all of the Kafka topics:

       ./kafka-topics.sh --zookeeper localhost:2181 --list

       You should see the following list of Kafka topics:
       • bro
       • enrichments
       • ubdexubg
       • snort
       • mysquid