Incompatible Changes in CDH 6.3.4

Changing the table file format from ORC to another file format with the ALTER TABLE command is not supported in CDH 6 (it returns an error).

Prior to this change, Hive performed implicit casts when data types from different type groups were specified in queries that use UNION ALL. For example, before CDH 6.0, if you had the two following tables:

Table "one"

+------------+------------+------------+--+
| one.col_1  | one.col_2  | one.col_3  |
+------------+------------+------------+--+
| 21         | hello_all  | b          |
+------------+------------+------------+--+
        

Where col_1 datatype is int, col_2 datatype is string, and col_3 datatype is char(1).

Table "two"

+------------+------------+------------+--+
| two.col_4  | two.col_5  | two.col_6  |
+------------+------------+------------+--+
| 75.0       | abcde      | 45         |
+------------+------------+------------+--+
        

Where col_4 datatype is double, col_5 datatype is varchar(5), and col_6 datatype is int.

And you ran the following UNION ALL query against these two tables:

SELECT * FROM one UNION ALL SELECT col_4 AS col_1, col_5 AS col_2, col_6 AS
col_3 FROM two;
        

You received the following result set:

+------------+------------+------------+--+
| _u1.col_1  | _u1.col_2  | _u1.col_3  |
+------------+------------+------------+--+
| 75.0       | abcde      | 4          |
| 21.0       | hello      | b          |
+------------+------------+------------+--+
        

Note that this statement implicitly casts the values from table one with the following errors resulting in data loss:

• one.col_1 is cast to a double datatype
• one.col_2 is cast to a varchar(5) datatype, which truncates the original value from hello_all to hello
• one.col_3 is cast to a char(1) datatype, which truncates the original value from 45 to 4

In CDH 6.0, no implicit cast is performed across different type groups. For example, STRING, CHAR, and VARCHAR are in one type group, and INT, BIGINT, and DECIMAL are in another type group, and so on. So, in CDH 6.0 and later, the above query that uses UNION ALL, returns an exception for the columns that contain datatypes that are not part of a type group. In CDH 6.0 and later, Hive performs the implicit cast only within type groups and not across different type groups. For more information, see HIVE-14251.

Support has been added for the UNION DISTINCT clause in Hive. See HIVE-9039 and the Apache wiki for more details. This feature introduces the following incompatible changes to Hive:

• Behavior in CDH 5:

  • SORT BY, CLUSTER BY, ORDER BY, LIMIT, and DISTRIBUTE BY can be specified without delineating parentheses either before a UNION ALL clause or at the end of the query, resulting in the following behaviors:

    • When specified before, these clauses are applied to the query before UNION ALL is applied.
    • When specified at the end of the query, these clauses are applied to the query after UNION ALL is applied.
  • The UNION clause is equivalent to UNION ALL, in which no duplicates are removed.

• Behavior in CDH 6:

  • SORT BY, CLUSTER BY, ORDER BY, LIMIT, and DISTRIBUTE BY can be specified without delineating parentheses only at the end of the query, resulting in the following behaviors:

    • These clauses are applied to the entire query.
    • Specifying these clauses before the UNION ALL clause results in a parsing error.
  • The UNION clause is equivalent to UNION DISTINCT, in which all duplicates are removed.

Support for Hive table and partition protection options have been removed in CDH 6.0, which includes removal of the following functionality:

• Support has been removed for:

  • ENABLE | DISABLE NO_DROP [CASCADE]
  • ENABLE | DISABLE OFFLINE
  • ALTER TABLE … IGNORE PROTECTION

• The following support has also been removed from the HiveMetastoreClient class:

  The ignoreProtection parameter has been removed from the dropPartitions methods in the IMetaStoreClient interface.

For more information, see HIVE-11145.

Cloudera recommends that you use Apache Sentry to replace most of this functionality. Although Sentry governs permissions on ALTER TABLE, it does not include permissions that are specific to a partition. See Authorization Privilege Model for Hive and Impala and Configuring the Sentry Service.

In CDH 6.0 syntax has changed for DESCRIBE queries as follows:

• DESCRIBE queries where the column name is separated by the table name using a period is no longer supported:

  DESCRIBE testTable.testColumn;
              

  Instead, the table name and column name must be separated with a space:

  DESCRIBE testTable testColumn;
              

• The partition_spec must appear after the table name, but before the optional column name:

  DESCRIBE default.testTable PARTITION (part_col = 100) testColumn;
              

For more details, see the Apache wiki and HIVE-12184.

In CDH 6.0, CREATE TABLE statements fail if any of the specified column names contain a period or a colon. For more information, see HIVE-10120 and the Apache wiki.

Hive reserved and non-reserved keywords have changed in CDH 6.0. Reserved keywords cannot be used as table or column names unless they are enclosed with back ticks (for example, `data`). Non-reserved keywords can be used as table or column names without enclosing them with back ticks. Non-reserved keywords have proscribed meanings in HiveQL, but can still be used as table or column names. For more information about the changes to reserved and non-reserved words listed below, see HIVE-6617 and HIVE-14872.

In CDH 6.0, the following changes have been introduced to Hive reserved and non-reserved keywords and are not backwards compatible:

The following changes have been introduced to the Hive API in CDH 6.0, and are not backwards compatible:

The following configuration property changes have been introduced to Hive in CDH 6.0, and are not backwards compatible:

HiveServer2 Thrift API code has been repackaged in CDH 6.0, resulting in the following changes:

• All files generated by the Thrift API for HiveServer2 have moved from the following old namespace:

  org.apache.hive.service.cli.thrift

  To the following new namespace:

  org.apache.hive.service.rpc.thrift

• All files generated by the Thrift API for HiveServer2 have moved into a separate jar file called service-rpc.

As a result of these changes, all Java classes such as TCLIService.java, TOpenSessionReq.java, TSessionHandle.java, and TGetSchemasReq.java have changed locations. For more information, see HIVE-12442.

Decimal values that are returned in query results are now padded with trailing zeroes to match the specified scale of the corresponding column. For example, before this change, when Hive read a decimal column with a specified scale of 5, the value returned for zero was returned as 0. Now, the value returned for zero is 0.00000. For more information, see HIVE-12063.

The logging framework for Hive has switched to SLF4J (Simple Logging Facade for Java) and now uses Log4j 2 by default. Use of Log4j 1.x, Apache Commons Logging, and java.util.logging have been removed. To accommodate this change, write all Log4j configuration files to be compatible with Log4j 2.

For more information, see HIVE-12237, HIVE-11304, and the Apache wiki.

The deprecated parquet classes, parquet.hive.DeprecatedParquetInputFormat and parquet.hive.DeprecatedParquetOutputFormat have been removed from Hive because they resided outside of the org.apache namespace. Any existing tables that use these classes are automatically migrated to the new SerDe classes when the metastore is upgraded.

Use one of the following options for specifying the Parquet SerDe for new Hive tables:

• Specify in the CREATE TABLE statement that you want it stored as Parquet. For example:

  CREATE TABLE <parquet_table_name> (col1 INT, col2 STRING) STORED AS PARQUET;
              

• Set the INPUTFORMAT to org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat and set the OUTPUTFORMAT to org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat. For example:

  CREATE TABLE <parquet_table_name> (col1 INT, col2 STRING)
  STORED AS INPUTFORMAT "org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat" OUTPUTFORMAT "org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat";
              

For more information, see HIVE-6757 and the Apache wiki.

Support for JDBC, counter-based, and HBase-based statistics collection mechanisms has been removed from Hive. The following configuration properties are no longer supported:

• hive.stats.dbclass
• hive.stats.retries.wait
• hive.stats.retries.max
• hive.stats.jdbc.timeout
• hive.stats.dbconnectionstring
• hive.stats.jdbcdrive
• hive.stats.key.prefix.reserve.length

This change also removed the cleanUp(String keyPrefix) method from the StatsAggregator interface.

Now all Hive statistics are collected on the default file system. For more information, see HIVE-12164, HIVE-12411, HIVE-12005, and the Apache wiki.

The S3N connector, which is used to connect to the Amazon S3 file system from Hive has been removed from CDH 6.0. To connect to the S3 file system from Hive in CDH 6.0, you must now use the S3A connector. There are a number of differences between the S3N and the S3A connectors, including configuration differences. See the Apache wiki page on integrating with Amazon Web Services for details.

Migration involves making the following changes:

• Changing all metastore data containing URIs that start with s3n:// to s3a://. This change is performed automatically when you upgrade the Hive metastore.
• Changing all scripts containing URIs that start with s3n:// to s3a://. You must perform this change manually.

Six additional columns have been added to the TRowSet that is returned by the TCLIService#GetTables request. These columns were added to comply with the official JDBC API. For more information, see the documentation for java.sql.DatabaseMetaData.

The columns added are:

For more information, see HIVE-7575.

Support has been added for escaping carriage returns and new line characters in text files by modifying the LazySimpleSerDe class. Without this change, carriage returns and new line characters are interpreted as delimiters, which causes incorrect query results.

This feature is controlled by the SerDe property serialization.escape.crlf. It is enabled (set to true) by default. If serialization.escape.crlf is enabled, 'r' or 'n' cannot be used as separators or field delimiters.

This change only affects text files and removes the getNullString method from the LazySerDeParameters class. For more information, see HIVE-11785.

The configuration properties hive.enforce.sorting and hive.enforce.bucketing have been removed. When set to false, these configurations disabled enforcement of sorted and bucketed tables when data was inserted into a table. Removing these configuration properties effectively sets these properties to true. In CDH 6.0, bucketing and sorting are enforced on Hive tables during insertions and cannot be turned off. For more information, see the Apache wiki topic on hive.enforce.bucketing and the topic on hive.enforce.sorting.

Apache DataFu Pig is a collection of user-defined functions that can be used with Pig for data mining and statistical analysis on large-scale data. The DataFu JAR was included in CDH 5, but due to very low adoption rates, the JAR was deprecated in CDH 5.9 and is being removed from CDH 6, starting with CDH 6.0. It is no longer supported.

Using the --split-by option with a CHAR or VARCHAR column does not always work properly, so Sqoop now requires the user to set the org.apache.sqoop.splitter.allow_text_splitter property to true to confirm that they are aware of this risk.

sqoop import -Dorg.apache.sqoop.splitter.allow_text_splitter=true --connect $MYCONN
--username $MYUSER --password $MYPSWD --table "test_table" --split-by "string_column"

For more information, see SQOOP-2910.

The --direct option is only supported with the following databases: MySQL, PostgreSQL, Oracle, Netezza.

In earlier releases, Sqoop silently ignored this option if it was specified for other databases, but it now throws an error.

sqoop import --connect $MYCONN --username $MYUSER --password $MYPSWD --table "direct_import" --direct

Was called with the --direct option, but no direct connector available.

For more information, see SQOOP-2913.

The --create-hive-table option is not supported when the user imports into Hive in Parquet format. Earlier this option was silently ignored, and the data was imported even if the Hive table existed. Sqoop will now fail if the --create-hive-table option is used with the --as-parquetfile option.

sqoop import --connect $MYCONN --username $MYUSER --password $MYPSWD --table
"test_table" --hive-import --as-parquetfile --create-hive-table

Hive import and create hive table is not compatible with importing into ParquetFile format.

For more information, see SQOOP-3010.

Importing into and exporting from a Hive view using HCatalog is not supported by Sqoop. A fail fast check was introduced so that now Sqoop throws a descriptive error message if the user specified a Hive view in the value of the --hcatalog-table option.

sqoop import --connect $MYCONN --username $MYUSER --password $MYPSWD --table "test_table"
--hcatalog-table "test_view"

Reads/Writes from and to Views are not supported by HCatalog

For more information, see SQOOP-3027.

Simplify Unicode character support in source files (introduced by SQOOP-3074) by defining explicit locales instead of using EscapeUtils. The Java source files generated by Sqoop will be encoded in UTF-8 format.

For more information, see SQOOP-3075.

If we export from HDFS to an RDBMS table and the file on HDFS has no value for some of the columns defined in the table, Sqoop will use the values of --input-null-string and --input-null-non-string options. Earlier this scenario was not supported and Sqoop failed.

For more information, see SQOOP-3158.

The MySQL direct connector does not support the --null-string, --null-non-string, --input-null-string, and --input-null-non-string options. These options were silently ignored earlier, but Sqoop now throws an error if these options are used with MySQL direct imports and exports.

For more information, see SQOOP-3206.