Verify the EC Policies
You can verify the EC policies you set up on the File / Table / Partition level using the Impala commands.
File level
If you set EC policies on the file level, use the
SHOW FILES <TABLE-NAME>
command to display if a file is an EC file, and what the EC policy is. If a file is erasure
coded, you will notice an additional column “EC policy” with the related
information.show files in ect
+---------------------------------------------------------------------------------------------------------------------+------+---------------------+--------------+
| Path | Size | Partition | EC Policy |
+---------------------------------------------------------------------------------------------------------------------+------+---------------------+--------------+
| hdfs://localhost:20500/ec-warehouse/ect/date_col=2009-01-01/1d46b6cbf747c7a8-a818238e00000000_1347890556_data.0.txt | 338B | date_col=2009-01-01 | RS-3-2-1024k |
| hdfs://localhost:20500/ec-warehouse/ect/date_col=2009-01-02/1d46b6cbf747c7a8-a818238e00000000_588711848_data.0.txt | 353B | date_col=2009-01-02 | RS-3-2-1024k |
| hdfs://localhost:20500/ec-warehouse/ect/date_col=2009-01-03/1d46b6cbf747c7a8-a818238e00000000_140483252_data.0.txt | 353B | date_col=2009-01-03 | RS-3-2-1024k |
| hdfs://localhost:20500/ec-warehouse/ect/date_col=2009-01-04/1d46b6cbf747c7a8-a818238e00000000_795616328_data.0.txt | 353B | date_col=2009-01-04 | RS-3-2-1024k |
| hdfs://localhost:20500/ec-warehouse/ect/date_col=2009-01-05/1d46b6cbf747c7a8-a818238e00000000_234666509_data.0.txt | 353B | date_col=2009-01-05 | RS-3-2-1024k |
+---------------------------------------------------------------------------------------------------------------------+------+---------------------+--------------+
Fetched 5 row(s) in 0.07sTable level
If you set the EC policies on a table level directory, use the
DESCRIBE
EXTENDED <TABLE-NAME> statement to verify if the table had an EC policy set or not.
If the directory has an erasure coding policy, it will be listed under storage
information.describe extended ect
+------------------------------+------------------------------------------------------------+----------------------+
| name | type | comment |
+------------------------------+------------------------------------------------------------+----------------------+
| # col_name | data_type | comment |
| | NULL | NULL |
| id | int | Add a comment |
| bool_col | boolean | NULL |
| tinyint_col | tinyint | NULL |
| smallint_col | smallint | NULL |
| int_col | int | NULL |
| bigint_col | bigint | NULL |
| float_col | float | NULL |
| double_col | double | NULL |
| string_col | string | NULL |
| timestamp_col | timestamp | NULL |
| | NULL | NULL |
| # Partition Information | NULL | NULL |
| # col_name | data_type | comment |
| | NULL | NULL |
| date_col | date | NULL |
| | NULL | NULL |
| # Detailed Table Information | NULL | NULL |
| Database: | default | NULL |
| OwnerType: | USER | NULL |
| Owner: | michael | NULL |
| CreateTime: | Tue Nov 15 15:10:21 PST 2022 | NULL |
| LastAccessTime: | UNKNOWN | NULL |
| Retention: | 0 | NULL |
| Location: | hdfs://localhost:20500/ec-warehouse/ect | NULL |
| Erasure Coding Policy: | RS-3-2-1024k | NULL |
| Table Type: | EXTERNAL_TABLE | NULL |
| Table Parameters: | NULL | NULL |
| | DO_NOT_UPDATE_STATS | true |
| | EXTERNAL | TRUE |
| | OBJCAPABILITIES | EXTREAD,EXTWRITE |
| | STATS_GENERATED | TASK |
| | TRANSLATED_TO_EXTERNAL | TRUE |
| | external.table.purge | TRUE |
| | impala.lastComputeStatsTime | 1668549634 |
| | numRows | -1 |
| | totalSize | 27923 |
| | transient_lastDdlTime | 1668548361 |
| | NULL | NULL |
| # Storage Information | NULL | NULL |
| SerDe Library: | org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe | NULL |
| InputFormat: | org.apache.hadoop.mapred.TextInputFormat | NULL |
| OutputFormat: | org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat | NULL |
| Compressed: | No | NULL |
| Num Buckets: | 0 | NULL |
| Bucket Columns: | [] | NULL |
| Sort Columns: | [] | NULL |
| | NULL | NULL |
| # Constraints | NULL | NULL |
+------------------------------+------------------------------------------------------------+----------------------+
Fetched 50 row(s) in 0.02s
Partition level
For partitioned tables, use the
SHOW PARTITIONS statement to list out the EC
policy of each partition directory. If you set up EC policy at the partition level, you will
find the policy information under the new column EC
policy.show partitions part_text;
+-------+-------+-----+-------+--------+---------+--------------+-------------------+--------+--------------+-------------------+
| year | month | day | #Rows | #Files | Size | Bytes Cached | Cache Replication | Format | EC Policy | Incremental stats |
+-------+-------+-----+-------+--------+---------+--------------+-------------------+--------+--------------+-------------------+
| 2014 | 1 | 1 | -1 | 4 | 25.16MB | NOT CACHED | NOT CACHED | TEXT | RS-3-2-1024k | false |
| 2014 | 1 | 2 | -1 | 4 | 26.22MB | NOT CACHED | NOT CACHED | TEXT | RS-3-2-1024k | false |
| 2014 | 1 | 3 | -1 | 0 | 0B | NOT CACHED | NOT CACHED | TEXT | RS-3-2-1024k | false |
| 2014 | 1 | 4 | -1 | 0 | 0B | NOT CACHED | NOT CACHED | TEXT | RS-3-2-1024k | false |
| Total | | | -1 | 8 | 51.38MB | 0B | | | | |
+-------+-------+-----+-------+--------+---------+--------------+-------------------+--------+--------------+-------------------+
