This is the documentation for CDH 5.0.x. Documentation for other versions is available at Cloudera Documentation.

How Impala Works with Hadoop File Formats

Impala supports several familiar file formats used in Apache Hadoop. Impala can load and query data files produced by other Hadoop components such as Pig or MapReduce, and data files produced by Impala can be used by other components also. The following sections discuss the procedures, limitations, and performance considerations for using each file format with Impala.

The file format used for an Impala table has significant performance consequences. Some file formats include compression support that affects the size of data on the disk and, consequently, the amount of I/O and CPU resources required to deserialize data. The amounts of I/O and CPU resources required can be a limiting factor in query performance since querying often begins with moving and decompressing data. To reduce the potential impact of this part of the process, data is often compressed. By compressing data, a smaller total number of bytes are transferred from disk to memory. This reduces the amount of time taken to transfer the data, but a tradeoff occurs when the CPU decompresses the content.

Impala can query files encoded with most of the popular file formats and compression codecs used in Hadoop. Impala can create and insert data into tables that use some file formats but not others; for file formats that Impala cannot write to, create the table in Hive, issue the INVALIDATE METADATA statement in impala-shell, and query the table through Impala. File formats can be structured, in which case they may include metadata and built-in compression. Supported formats include:

Table 1. File Format Support in Impala
File Type Format Compression Codecs Impala Can CREATE? Impala Can INSERT?
Parquet Structured Snappy, GZIP; currently Snappy by default Yes. Yes: CREATE TABLE, INSERT, LOAD DATA, and query.
Text Unstructured LZO Yes. For CREATE TABLE with no STORED AS clause, the default file format is uncompressed text, with values separated by ASCII 0x01 characters (typically represented as Ctrl-A). Yes: CREATE TABLE, INSERT, LOAD DATA, and query. If LZO compression is used, you must create the table and load data in Hive.
Avro Structured Snappy, GZIP, deflate, BZIP2 No, create using Hive. No. Load data through LOAD DATA on data files already in the right format, or use INSERT in Hive.
RCFile Structured Snappy, GZIP, deflate, BZIP2 Yes. No. Load data through LOAD DATA on data files already in the right format, or use INSERT in Hive.
SequenceFile Structured Snappy, GZIP, deflate, BZIP2 Yes. No. Load data through LOAD DATA on data files already in the right format, or use INSERT in Hive.

Impala supports the following compression codecs:

  • Snappy. Recommended for its effective balance between compression ratio and decompression speed. Snappy compression is very fast, but GZIP provides greater space savings. Not supported for text files.
  • GZIP. Recommended when achieving the highest level of compression (and therefore greatest disk-space savings) is desired. Not supported for text files.
  • Deflate. Not supported for text files.
  • BZIP2. Not supported for text files.

  • LZO, for Text files only. Impala can query LZO-compressed Text tables, but currently cannot create them or insert data into them; perform these operations in Hive.

Choosing the File Format for a Table

Different file formats and compression codecs work better for different data sets. While Impala typically provides performance gains regardless of file format, choosing the proper format for your data can yield further performance improvements. Use the following considerations to decide which combination of file format and compression to use for a particular table:

  • If you are working with existing files that are already in a supported file format, use the same format for the Impala table where practical. If the original format does not yield acceptable query performance or resource usage, consider creating a new Impala table with different file format or compression characteristics, and doing a one-time conversion by copying the data to the new table using the INSERT statement. Depending on the file format, you might run the INSERT statement in impala-shell or in Hive.
  • Text files are convenient to produce through many different tools, and are human-readable for ease of verification and debugging. Those characteristics are why text is the default format for an Impala CREATE TABLE statement. When performance and resource usage are the primary considerations, use one of the other file formats and consider using compression. A typical workflow might involve bringing data into an Impala table by copying CSV or TSV files into the appropriate data directory, and then using the INSERT ... SELECT syntax to copy the data into a table using a different, more compact file format.
  • If your architecture involves storing data to be queried in memory, do not compress the data. There is no I/O savings since the data does not need to be moved from disk, but there is a CPU cost to decompress the data.
Page generated September 3, 2015.