LZO compression is a lossless data compression library favoring speed over compression ratio; LZO compression is recommended for temporary tables. You can enable LZO compression for HDP to optimize Hive query speed.
LZO compression is not enabled automatically. To enable it, perform the following steps on each node in your cluster:
Copy the
hadoop-lzo.jar
file from your installation zip package to%HADOOP_COMMON_HOME%\share\hadoop\common
.Copy
gplcompression.dll
andlzo2.dll
from your installation zip package to the same bin folder ashadoop.dll
.Ensure that the following configuration properties are set in
core-site.xml
:<property> <name>io.compression.codecs</name> <value>org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,com.hadoop.compression.lzo.LzoCodec,com.hadoop.compression.lzo.LzopCodec,org.apache.hadoop.io.compress.SnappyCodec</value> </property> <property> <name>io.compression.codec.lzo.class</name> <value>com.hadoop.compression.lzo.LzoCodec</value> </property>