Metadata Extraction and Indexing

Metadata Extraction

The Navigator Metadata Server extracts metadata for the following resource types from the listed servers:
  • HDFS - Extracts HDFS metadata at the next scheduled extraction run after an HDFS checkpoint. However, if you have high availability enabled, metadata is extracted as soon as it is written to the JournalNodes.
  • Hive - Extracts database, table, and query metadata from Hive lineage logs. See Managing Hive and Impala Lineage Properties.
  • Impala - Extracts database, table, and query metadata from the Impala Daemon lineage logs. See Managing Hive and Impala Lineage Properties.
  • MapReduce - Extracts job metadata from the JobTracker. The default setting in Cloudera Manager retains a maximum of five jobs, which means if you run more than five jobs between Navigator extractions, the Navigator Metadata Server would extract the five most recent jobs.
  • Oozie - Extracts Oozie workflows from the Oozie Server.
  • Pig - Extracts Pig script runs from the JobTracker or Job History Server.
  • Spark - Extracts Spark job metadata from YARN logs. (Unsupported and disabled by default. To enable, see Enabling Spark Metadata Extraction.)
  • Sqoop 1 - Extracts database and table metadata from Hive lineage logs. Extracts job runs from the JobTracker or Job History Server.
  • YARN - Extracts job metadata from the ResourceManager.

If an entity is created at time t0 in the system, that entity will be extracted and linked in Navigator after the extraction poll period (default 10 minutes) plus a service-specific interval as follows:

  • HDFS: t0 + extraction poll period + HDFS checkpoint interval (default 1 hour)
  • HDFS + HA: t0 + extraction poll period
  • Hive: t0 + extraction poll period + Hive maximum wait time (default 60 minutes)
  • Impala: t0 + extraction poll period

Metadata Indexing

After metadata is extracted it is indexed and made available for searching by an embedded Solr engine. The Solr schema indexes two types of metadata: entity properties and relationships between entities.

You can search entity metadata using the Navigator UI and API. Relationship metadata is implicitly visible in lineage diagrams and explicitly available by downloading the lineage using the Cloudera Navigator Data Management API.