Locations of Impala Log Files in Azure
This topic describes how to identify the Azure locations of Impala logs for the different Impala components.
The Cloudera Data Warehouse service collects logs from Impala Virtual Warehouses and uploads them to the Azure storage account that was provided while registering the Environment. This ABFS log location is configured under an external warehouse directory so that the logs are preserved even if the Virtual Warehouse they are collected from is destroyed.
To identify the location of the logs in Azure, you must have the environment_ID, database_catalog_ID, and impala_ID identifiers and to access the logs from the Azure Portal you must know your storage account name.
- In the Data Warehouse service, expand the Environments column by clicking More….
- From the Overview page, note down the environment_ID, database_catalog_ID, and impala_ID identifiers.
- In the Management Console navigate to the Environments page.
- On the Environments page, click on your Environment and click on the Summary tab.
Scroll down to the Logs Storage and Audits section.
Note down your storage account name.
- Log in to the Azure Portal and search for your storage account name using the Search bar.
- On the Overview page of your storage account, click on the Containers menu.
Click on the file system you used during the Environment registration.
Use the environment_ID,
database_catalog_ID, and impala_ID identifiers, in the following prefix to find
the logs generated by specific components in the following directories. Use the different directories listed here to view Impala/Hue logs
PREFIX = /clusters/<environment_ID>/<database_catalog_ID>/warehouse/tablespace/external/hive/sys.db/logs/dt=<date_stamp>/ns=<impala_ID>/
Impala component ABFS directory location impalad PREFIX + “
catalogd PREFIX + “
coordinator PREFIX + “
auto-scaler PREFIX + “
PREFIX + “
PREFIX + “
PREFIX + “
statestored PREFIX + “
The impalad logs for 8 March 2020 are located in the following ABFS location:
In the above location, you can find multiple logs that were generated on the specified day.
Impala minidumps can be found under the ‘debug-artifacts/impala’ directory
Impala query profiles are written in thrift encoded format in this location:
Impala component S3 directory location Impala query profiles PREFIX + “
app=impala-profiles”Use the binary tool to decode thrift to text. This binary tool is provided with the upstream runtime Impala 4.0 as a docker image. Run the following command to use this tool.
docker run -i apache/impala:4.0.0-impala_profile_tool < name of the thrift encoded file to decodeYou can use the docker image available here to use this decoding tool.