Data Catalog Terminology
An overview of terminology used in Data Catalog service.
- Enables the Data Catalog service to gather and view information about different relevant characteristics of data such as shape, distribution, quality, and sensitivity which are important to understand and use the data effectively. For example, view the distribution between males and females in column “Gender”, or min/max/mean/null values in a column named “avg_income”. Profiled data is generated on a periodic basis from the profilers, which run at regularly scheduled intervals. Works with data sourced from Apache Ranger Audit Logs, Apache Atlas Metadata Store, and Hive.
- Data Lake
- A trusted and governed data repository that stores, processes, and provides access to many kinds of enterprise data to support data discovery, data preparation, analytics, insights, and predictive analytics. In the context of Hortonworks DP, a data lake can be realized in practice with an Apache Ambari managed Hadoop cluster that runs Apache Atlas for metadata and governance services, and Apache Knox and Apache Ranger for security services.
- Data Asset
- A data asset is a physical asset located in the Hadoop ecosystem such as a Hive table which contains business or technical data. A data asset could include a specific instance of an Apache Hive database, table, or column. An asset can belong to multiple asset collections. Data assets are equivalent to “entities” in Apache Atlas.
- Datasets allow users of Data Catalog to manage and govern
various kinds of data objects as a single unit through a unified interface. Asset
collections help organize and curate information about many assets based on many
facets including data content and metadata, such as size/schema/tags/alterations,
lineage, and impact on processes and downstream objects in addition to the display of
security and governance policies.
You can launch a Profiler cluster for a Data Lake. Adding new assets to (or removing from) a dataset must be done manually.