Cloudera Docs

Managing Logs

You must configure the Impala log settings to change the default log locations, to rotate logs, or to log verbose levels.

  1. To change log file locations:
    1. In Cloudera Manager, navigate to Impala service > Configuration.
    2. In the search field, type log_dir.
    3. Specify the new log directories for Impala Daemon, Catalog Server, or StateStore in the respective fields:
      • Impala Daemon Log Directory
      • Catalog Server Log Directory
      • StateStore Log Directory
    4. Click Save Changes and restart Impala.
  2. To set up log rotation:
    1. In Cloudera Manager, navigate to Impala service > Configuration.
    2. In the search field, type max_log_files.
    3. Specify the values for Impala Daemon, Catalog Server, or StateStore in the respective fields:
      • Impala Daemon Maximum Log Files
      • Catalog Server Maximum Log Files
      • StateStore Maximum Log Files
    4. Click Save Changes and restart Impala.

    The above configuration option specifies how many log files to keep at each severity level (INFO, WARNING, ERROR, and FATAL).

    • A value of 0 preserves all log files, in which case you would set up set up manual log rotation using your Linux tool or technique of choice.
    • A value of 1 preserves only the very latest log file.
    • The default value is 10.

    For some log levels, Impala logs are first temporarily buffered in memory and only written to disk periodically. The --logbufsecs setting controls the maximum time that log messages are buffered for. For example, with the default value of 5 seconds, there may be up to a 5 second delay before a logged message shows up in the log file.

    It is not recommended that you set --logbufsecs to 0 as the setting makes the Impala daemon to spin in the thread that tries to delete old log files.

  3. To specify how often the log information is written to disk:
    1. In Cloudera Manager, navigate to Impala service > Configuration.
    2. In the search field, type logbuflevel.
    3. Specify the values for Impala Daemon, Catalog Server, or StateStore in the respective fields:
      • Impala Daemon Log Buffer Level
      • Catalog Server Log Buffer Level
      • StateStore Log Buffer Level
      The default is 0, meaning that the log is immediately flushed to disk when Impala outputs an important messages such as a warning or an error, but less important messages such as informational ones are buffered in memory rather than being flushed to disk immediately.
    4. Click Save Changes and restart Impala.
  4. To set the logging verbose levels:
    1. In Cloudera Manager, navigate to Impala service > Configuration.
    2. In the search field, type GLOG_v.
    3. Specify the values for Impala Daemon, Catalog Server, or StateStore in the respective fields:
      • Impala Daemon Verbose Log Level
      • Catalog Server Verbose Log Level
      • StateStore Verbose Log Level
    4. Click Save Changes and restart Impala.

    As logging levels increase, the categories of information logged are cumulative. For example, GLOG_v=2 records everything GLOG_v=1 records, as well as additional information.

    Increasing logging levels imposes performance overhead and increases log size. Cloudera recommends using GLOG_v=1 for most cases: this level has minimal performance impact but still captures useful troubleshooting information.

    Additional information logged at each level of GLOG_v is as follows:

    • 1: The default level. Logs information about each connection and query that is initiated to an impalad instance, including runtime profiles.
    • 2: Everything from the previous level plus information for each RPC initiated. This level also records query execution progress information, including details on each file that is read.
    • 3: Everything from the previous level plus logging of every row that is read. This level is only applicable for the most serious troubleshooting and tuning scenarios, because it can produce exceptionally large and detailed log files, potentially leading to its own set of performance and capacity problems.