Cloudera Docs

Ozone FS namespace optimization with prefix

Ozone now supports FS namespace optimization with prefix that provides atomicity and consistency in renaming and deleting files and subdirectories under a directory. Ozone now handles partial failures and performance is now deterministic.

FSO feature helps in performing the rename or delete metadata operations for the directories which have large sub-trees or sub-paths. With this feature, Ozone handles partial failures and provides atomicity and consistency in renaming and deleting each and every file and subdirectory under a directory. Performance is deterministic now and is similar to HDFS, especially for the delete and rename metadata operations, and if you are running the Spark or Hive like big data queries.

Highlights of this feature:

  • Provide an efficient Hierarchical FileSystem Namespace view with intermediate directories similar to HDFS.
  • Support for Atomic Rename and Deletes. This helps Hive, Impala, and Spark for job and task commits.
  • Strong consistency guarantees without any partial results in case of directory rename or delete failures.
  • Rename, move, and recursive directory delete operations should have deterministic performance numbers irrespective of the large set of subpaths (directories/files) contained within it.

Changes you can observe by using this feature:

  • Apache Hive drop table query, recursive directory deletion, and directory moving operations becomes faster and consistent without any partial results in case of any failure.
  • Dropping a managed Impala table should be efficient without requiring O(n) RPC calls where n is the number of file system objects for the table.
  • Job Committers of Hive, Impala, and Spark often rename their temporary output files to a final output location at the end of the job. The performance of the job is directly impacted by how quickly the rename operation is completed.
  • ACL support through Apache Ranger.
  • For more information on understanding the performance capabilities between Apache Ozone and S3 API and how to natively integrate workloads, see High Performance Object Store for CDP Private Cloud and High Performance Object Store for CDP Private Cloud.