Cloudera Docs

Locations of Impala Log Files in S3

This topic describes how to identify the Amazon S3 locations of Impala logs for the different Impala components.

The Cloudera Data Warehouse service collects logs from Impala Virtual Warehouses and uploads them to an Amazon S3 location. This S3 log location is configured under an external warehouse directory so that the logs are preserved even if the Virtual Warehouse they are collected from is destroyed.

To identify the location of the logs in S3, you must have the environment name, impala ID identifiers, and S3 bucket name.

Finding the environment name, and impala ID identifiers
  1. In the Data Warehouse service, expand the Environment column by clicking .
  2. From the Overview page, note down the environment name, and impala ID identifiers.
    Impala ID Good Health
Identifying the bucket name
  1. In the Management console navigate to the Environment page.
  2. On the Environment page, click on your Environment and then on the Summary tab.
  3. On the Logs Storage and Audits section. You can find your log bucket and prefix under the Storage Location.
    This name is required to identify the S3 location of the logs.
    Impala S3 Log Bucket
Log locations in S3
  1. Now that you have identified the S3 bucket name, environment name, and impala ID identifiers, use the following prefix to find the logs generated by specific components in the following directories. Use the different directories listed here to view Impala/Hue logs. 
    PREFIX =
s3://<s3_bucket_name>/<prefix>/cdw/clusters/<environment_name>/logs/dt=<date_stamp>/ns=<impala_ID>/
    Impala component S3 directory location
    impalad PREFIX + “app=impala-executor-log”
    catalogd PREFIX + “app=catalogd-log”
    coordinator PREFIX + “app=coordinator-log”
    auto-scaler PREFIX + “app=impala-autoscaler-log”
    Hue

    PREFIX + “app=huebackend-log”

    PREFIX + “app=hue-huedb-create-job-log”

    PREFIX + “app=huefrontend-log”
    statestored PREFIX + “app=statestored-log”

    The impalad executor logs for 25 August 2024 are located in the following S3 location:

    PREFIX = s3://<s3_bucket_name>/<prefix>/cdw/clusters/<environment_name>/logs/dt=2024-08-25/app=impala-executor-log/

    In the above location, you can find multiple logs that were generated on the specified day.

Impala Minidumps
  1. Impala minidumps can be found under the debug-artifacts/impala directory 
    /cdw/clusters/<environment_name>/logs/debug-artifacts/impala/<impala_ID>/minidump/<pod_name>/
Impala Query Profiles
  1. Impala query profiles are written in thrift encoded format in this location:
    Impala component S3 directory location
    Impala query profiles PREFIX + “app=impala-profiles”
    Use the binary tool to decode thrift to text. This binary tool is provided with the upstream runtime Impala 4.0 as a docker image. Run the following command to use this tool.
    docker run -i apache/impala:4.0.0-impala_profile_tool < name of the thrift encoded file to decode
    You can use the docker image available here to use this decoding tool.