Adding a New Telemetry Data Source
Also available as:


Before you add a new telemetry device, you must meet both the HCP and your user requirements.

  • Install HDP and HDF, and then install HCP.

  • The following use case assumes the following user requirements:

    • Proxy events from Squid logs must be ingested in real-time.

    • Proxy logs must be parsed into a standardized JSON structure suitable for analysis by Metron.

    • In real-time, the Squid proxy events must be enriched so that the domain names contain the IP information.

    • In real-time, the IP within the proxy event must be checked against for threat intelligence feeds.

    • If there is a threat intelligence hit, an alert must be raised.

    • The SOC analyst must be able to view new telemetry events and alerts from Squid.

    • HCP must profile the Squid data and incorporate statistical features of the profile into the threat triage rules.

    • HCP must be able to deploy a machine learning model that derives additional insights from the stream.

    • HCP must incorporate user's machine learning model into user's threat triage rule along with threat intel, static rules, and statistical rules.

    • Set HCP values.

      When you install HCP, you will set up several hosts. You will need the locations of these hosts, along with port numbers, and the Metron version. These values are listed below.

      • KAFKA_HOST = The host where a Kafka broker is installed.

      • ZOOKEEPER_HOST = The host where a ZooKeeper server is installed.

      • PROBE_HOST = The host where your sensor, probes are installed. If don't have any sensors installed, pick the host where a Storm supervisor is running.

      • SQUID_HOST = The host where you want to install SQUID. If you don't care, install SQUID on the PROBE_HOST.

      • NIFI_HOST = Host where you will install NIFI. This should be the same host on which you installed Squid.

      • HOST_WITH_ENRICHMENT_TAG = The host in your inventory hosts file that you put under the group "enrichment."

      • SEARCH_HOST = The host where you have Elastic or Solr running. This is the host in your inventory hosts file that you put under the group "search". Pick one of the search hosts.

      • SEARCH_HOST_PORT = The port of the search host where indexing is configured (for example, 9300).

      • METRON_UI_HOST = The host where your Metron UI web application is running. This is the host in your inventory hosts file that you put under the group "web."

      • METRON_VERSION = The release of the Metron binaries you are working with (for example, 0.2.0BETA-RC2).