Modifying Custom Metadata

The Cloudera Navigator Metadata component allows you to add and modify the following custom metadata associated with entities: display name, description, tags, and user-defined name-value pairs. You can modify custom metadata using the Navigator Metadata UI, MapReduce service and job properties, Navigator metadata files, and the Navigator Metadata API.

Required Role:

Modifying Custom Metadata Using the Navigator UI

1. Run a search in the Navigator UI.
2. Click an entity link returned in the search. The metadata pane displays on the left and the lineage page displays on the right.
3. In the top-right of the metadata pane, click . The Editing entity dialog box drops down.
4. Edit any of the fields as instructed. Press Enter or Tab to create new tag entries. For example, a Description, the tags occupations and salaries, and property year with value 2012 have been added to the file sample_07.csv:
   
   
   
   
5. Click Save. The new metadata appears in the metadata pane:
   
   
   
   

Modifying MapReduce Custom Metadata

You can associate custom metadata with arbitrary configuration parameters for MapReduce jobs and job executions. The specific configuration parameters to be extracted by Navigator can be specified statically or dynamically.

Modifying HDFS Custom Metadata Using Metadata Files

You can add tags and properties to HDFS entities using metadata files. The reasons to use metadata files are to assign metadata to entities in bulk and to create metadata before the metadata is extracted. A metadata file is a JSON file with the following structure:

{
  "name" : "aName",
  "description" : "a description",
  "properties" : {
    "prop1" : "value1", "prop2" : "value2"
  },
  "tags" : [ "tag1" ]
}

Modifying HDFS and Hive Custom Metadata Using the Navigator API

You can use the Cloudera Navigator API to modify the metadata of HDFS or Hive entities whether or not the entities have been extracted. If an entity has been extracted at the time the API is called, the metadata will be applied immediately. If the entity has not been extracted, you can preregister metadata which is then applied once the entity is extracted. Metadata is saved regardless of whether or not a matching entity is extracted, and Navigator does not perform any cleanup of unused metadata.

curl http://hostname:port/api/v2/entities/?query=-internalType:* -u username:password -X GET 

The metadata provided via the API overwrites existing metadata. If, for example, you call the API with an empty name and description, empty array for tags, and empty dictionary for properties, the call removes this metadata. If you leave out the tags or properties fields, the existing values remain unchanged.

Modifying metadata using HDFS metadata files and the metadata API at the same time is not supported. You must use one or the other, because the two methods behave slightly differently. Metadata specified in files is merged with existing metadata whereas the API overwrites metadata. Also, the updates provided by metadata files wait in a queue before being merged, but API changes are committed immediately. This means there may be some inconsistency if a metadata file is being merged around the same time the API is in use.

• PUT

  curl http://hostname:port/api/v2/entities/identity -u username:password -X PUT -H
  "Content-Type: application/json" -d '{properties}'

  where properties are:
  • name: name metadata
  • description: description metadata
  • tags: tag metadata
  • properties: property metadata
  All existing naming rules apply, and if any value is invalid, the entire request will be denied.
• POST

  curl http://hostname:port/api/v2/entities/ -u username:password -X POST -H
  "Content-Type: application/json" -d '{properties}'

  where properties are:
  • sourceId (required): The source ID must match an existing source ID. After the first extraction, you can retrieve the source IDs using the call

    curl http://hostname:port/api/v2/entities/?query=type:SOURCE -u username:password -X GET

    For example:

    [ ...
    {
      "identity" : "a09b0233cc58ff7d601eaa68673a20c6",
      "originalName" : "HDFS-1",
      "sourceId" : null,
      "firstClassParentId" : null,
      "parentPath" : null,
      "extractorRunId" : null,
      "name" : "HDFS-1",
      "description" : null,
      "tags" : null,
      "properties" : null,
      "clusterName" : "Cluster 1",
      "sourceUrl" : "hdfs://hostname:8020",
      "sourceType" : "HDFS",
      "sourceExtractIteration" : 4935,
      "type" : "SOURCE",
      "internalType" : "source"
    }, ...

    If you have HDFS-1 and HDFS-2, you must specify the source that contains the entity you’re expecting for it to match.
  • parentPath: The path of the parent entity, as defined below:
    • HDFS file or directory: fileSystemPath of the parent directory (do not provide this field if the entity being affected is the root directory). Example parentPath for /user/admin/input_dir: /user/admin. If you apply metadata to a directory, the metadata does not propagate to any files and folders in that directory.
    • Hive database: If you are updating database metadata, you do not specify this field.
    • Hive table or view: Name of database containing this table or view. Example for table in default database: default.
    • Hive column: database name/table/view name. Example for column in sample_07 table: default/sample_07.
  • originalName (required): The name as defined by the source system.
    • HDFS file or directory: name of file or directory (ROOT if the entity is the root directory). Example originalName for /user/admin/input_dir: input_dir.
    • Hive database, table, view, or column: name of the database, table, view, or column.
      • Example for default database: default
      • Example for sample_07 table: sample_07
  • name: name metadata
  • description: description metadata
  • tags: tag metadata
  • properties: property metadata
  All existing naming rules apply, and if any value is invalid, the entire request will be denied.