Offloading Application Logs to Ozone

In order to offload logs from HDFS to Ozone, a small Ozone cluster can be deployed through Cloudera Manager and then by configuring Remote App Log Directory in YARN configs, customers can offload logs from HDFS to Ozone. This will help in freeing up space in HDFS metadata for more user data. After changing the configuration, logs for all YARN, Hive on Tez and Spark on Yarn are automatically redirected to Ozone.

HDFS is used as the primary storage engine by most of the Big Data applications like Hive, YARN, Spark, and so on. Currently, it stores both the important data like Hive and Spark tables and logs generated by these applications.

The YARN Log Aggregation feature enables you to move local log files of any application onto HDFS or Apache Ozone depending on the cluster configuration. Fore more information, see YARN Log Aggregation Overview

Changing configuration

• In Cloudera Manager, select the service.
• Click the Configuration tab. Search for “remote app log”.
• In the Filters pane, under Scope, select NodeManager.
• In the Remote App Log Directory (yarn.nodemanager.remote-app-log-dir) field, add one of the following:

  ofs://ozone1/logvol/logbuck/temp_log_dir