Table properties support
Table properties are taken into consideration while creating the table.
Type of change: Configuration/Spark Core Changes
Spark 1.6 - 2.3
Parquet and ORC Hive tables are converted to Parquet or ORC by default, but table properties
are ignored. For example, the compression table property is ignored:
CREATE TABLE t(id int) STORED AS PARQUET TBLPROPERTIES (parquet.compression 'NONE')
This
command generates Snappy Parquet files.Spark 2.4
Table properties are supported. For example, if no compression is required, set the
TBLPROPERTIES as follows: (parquet.compression 'NONE')
.
This command generates uncompressed Parquet files.
Action Required
Check and set the desired TBLPROPERTIES.