HDP-2.2.9 Release Notes
Also available as:
PDF

Hive 0.14.0

HDP 2.2.9 provides Hive 0.14.0 and the following additional Apache patches:

  • HIVE-10083: SMBJoin fails in case one table is uninitialized.

  • HIVE-10178: DateWritable incorrectly calculates daysSinceEpoch for negative Unix time.

  • HIVE-10209: FetchTask with VC may fail because ExecMapper.done is true.

  • HIVE-10213: MapReduce jobs using dynamic-partitioning fail on commit.

  • HIVE-10244: Vectorization : TPC-DS Q80 fails with java.lang.ClassCastException when hive.vectorized.execution.reduce.enabled is enabled.

  • HIVE-10303: HIVE-9471 broke forward compatibility of ORC files.

  • HIVE-10451: PTF deserializer fails if values are not used in reducer.

  • HIVE-10528: Hiveserver2 in HTTP mode is not applying auth_to_local rules.

  • HIVE-10569: Hive CLI gets stuck when hive.exec.parallel=true; and some exception happens during SessionState.start.

  • HIVE-10571: HiveMetaStoreClient should close existing thrift connection before its reconnect.

  • HIVE-10646: ColumnValue does not handle NULL_TYPE.

  • HIVE-10651: ORC file footer cache should be bounded.

  • HIVE-10732: Hive JDBC driver does not close operation for metadata queries.

  • HIVE-10746: Hive 1.2.0+Tez produces 1-byte FileSplits from mapred.TextInputFormat.

  • HIVE-10790: orc write on viewFS throws exception.

  • HIVE-10835: Concurrency issues in JDBC driver.

  • HIVE-10907: Hive on Tez: Classcast exception in some cases with SMB joins.

  • HIVE-10925: Non-static threadlocals in metastore code can potentially cause memory leak.

  • HIVE-10965: direct SQL for stats fails in 0-column case.

  • HIVE-10966: direct SQL for stats has a cast exception on some databases.

  • HIVE-10968: Windows: analyze json table via beeline failed throwing Class org.apache.hive.hcatalog.data.JsonSerDe not found.

  • HIVE-10971: count(*) with count(distinct) gives wrong results when hive.groupby.skewindata=true.

  • HIVE-11013: MiniTez tez_join_hash test on the branch fails with NPE (initializeOp not called?).

  • HIVE-11024: Error inserting a date value via parameter marker (PreparedStatement.setDate).

  • HIVE-11031: ORC concatenation of old files can fail while merging column statistics.

  • HIVE-11054: Read error : Partition Varchar column cannot be cast to string.

  • HIVE-11123: Fix how to confirm the RDBMS product name at Metastore.

  • HIVE-11132: Queries using join and group by produce incorrect output when hive.auto.convert.join=false and hive.optimize.reducededuplication=true.

  • HIVE-11149: Fix issue with sometimes HashMap in PerfLogger.java hangs.

  • HIVE-11172: Vectorization wrong results for aggregate query with where clause without group by.

  • HIVE-11176: Caused by: java.lang.ClassCastException: org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct cannot be cast to [Ljava.lang.Object;.

  • HIVE-11193: ConstantPropagateProcCtx should use a Set instead of a List to hold operators to be deleted.

  • HIVE-11301: thrift metastore issue when getting stats results in disconnect.

  • HIVE-11407: JDBC DatabaseMetaData.getTables with large no of tables call leads to HS2 OOM.

  • HIVE-11408: HiveServer2 is leaking ClassLoaders when add jar / temporary functions are used due to constructor caching in Hadoop ReflectionUtils.

  • HIVE-11432: Hive macro give same result for different arguments.

  • HIVE-11499: Datanucleus leaks classloaders when used using embedded metastore with HiveServer2 with UDFs.

  • HIVE-11502: Map side aggregation is extremely slow.

  • HIVE-11510: Metatool updateLocation warning on views.

  • HIVE-11517: Vectorized auto_smb_mapjoin_14.q produces different results.

  • HIVE-11583: When PTF is used over a large partitions result could be corrupted.

  • HIVE-11603: IndexOutOfBoundsException thrown when accessing a union all subquery and filtering on a column which does not exist in all underlying tables.

  • HIVE-11605: Incorrect results with bucket map join in tez.

  • HIVE-11606: Bucket map joins fail at hash table construction time.

  • HIVE-11628: DB type detection code is failing on Oracle 12.

  • HIVE-11670: Strip out password information from TezSessionState configuration.

  • HIVE-11712: Duplicate groupby keys cause ClassCastException.

  • HIVE-11720: Allow HiveServer2 to set custom http request/response header size.

  • HIVE-11724: WebHcat get jobs to order jobs on time order with latest at top.

  • HIVE-11727: Hive on Tez through Oozie: Some queries fail with fnf exception.

  • HIVE-11737: IndexOutOfBounds compiling query with duplicated groupby keys.

  • HIVE-11807: Set ORC buffer size in relation to set stripe size.

  • HIVE-11835: Type decimal(1,1) reads 0.0, 0.00, etc from text file as NULL.

  • HIVE-11839: Vectorization wrong results with filter of (CAST AS CHAR).

  • HIVE-11849: NPE in HiveHBaseTableShapshotInputFormat in query with just count(*).

  • HIVE-11901: StorageBasedAuthorizationProvider requires write permission on table for SELECT statements.

  • HIVE-11950: WebHCat status file doesn't show UTF8 character.

  • HIVE-11960: braces in join conditions are not supported.

  • HIVE-11964: RelOptHiveTable.hiveColStatsMap might contain mismatched column stats.

  • HIVE-11977: Hive should handle an external avro table with zero length files present.

  • HIVE-11988: [hive] security issue with hive and ranger for import table command.

  • HIVE-12012: select query on json table with map containing numeric values fails.

  • HIVE-12057: ORC sarg is logged too much.

  • HIVE-12076: WebHCat listing jobs after the given JobId even when templeton.jobs.listorder is set to lexicographicaldesc.

  • HIVE-12083: HIVE-10965 introduces thrift error if partNames or colNames are empty.

  • HIVE-12084: Hive queries with ORDER BY and large LIMIT fails with OutOfMemoryError Java heap space.

  • HIVE-12201: Tez settings need to be shown in set -v output when execution engine is tez.

  • HIVE-12204: Tez queries stopped running with ApplicationNotRunningException.

  • HIVE-12206: ClassNotFound Exception during query compilation with Tez and Union query and GenericUDFs.

  • HIVE-12223: Filter on Grouping__ID does not work properly.

  • HIVE-12249: Improve logging with tez.

  • HIVE-12254: Improve logging with yarn/hdfs.

  • HIVE-12261: schematool version info exit status should depend on compatibility, not equality.

  • HIVE-12262: Session log dir cannot be created in some cases.

  • HIVE-12277: Hive macro results on macro_duplicate.q different after adding ORDER BY.

  • HIVE-12340: ExecDriver.execute() unnecessarily sets METASTOREPWD to HIVE.

  • HIVE-12344: Wrong types inferred for SemiJoin generation in CBO.

  • HIVE-12354: MapJoin with double keys is slow on MR.

  • HIVE-5277: HBase handler skips rows with null valued first cells when only row key is selected.

  • HIVE-5545: HCatRecord getInteger method returns String when used on Partition columns of type INT.

  • HIVE-5631: Index creation on a skew table fails.

  • HIVE-6099: Multi insert does not work properly with distinct count.

  • HIVE-6308: COLUMNS_V2 Metastore table not populated for tables created without an explicit column list.

  • HIVE-7049: Unable to deserialize AVRO data when file schema and record schema are different and nullable.

  • HIVE-8330: HiveResultSet.findColumn() parameters are case sensitive.

  • HIVE-8512: queries with star and gby produce incorrect results.

  • HIVE-8610: Compile time skew join optimization doesn't work with auto map join.

  • HIVE-8666: hive.metastore.server.max.threads default is too high.

  • HIVE-8848: data loading from text files or text file processing doesn't handle nulls correctly.

  • HIVE-8875: hive.optimize.sort.dynamic.partition should be turned off for ACID.

  • HIVE-8895: bugs in mergejoin.

  • HIVE-8914: HDFSCleanup thread holds reference to FileSystem.

  • HIVE-8917: HIVE-5679 adds two thread safety problems.

  • HIVE-9013: Hive set command exposes metastore db password.

  • HIVE-9073: NPE when using custom windowing UDAFs.

  • HIVE-9151: Checking s against null in TezJobMonitor#getNameWithProgress() should be done earlier.

  • HIVE-9390: Enhance retry logic with regard to DB access in TxnHandler.

  • HIVE-9404: NPE in org.apache.hadoop.hive.metastore.txn.TxnHandler.determineDatabaseProduct().

  • HIVE-9474: truncate table changes permissions on the target.

  • HIVE-9484: ThriftCLIService#getDelegationToken does case sensitive comparison.

  • HIVE-9496: Slf4j warning in hive command.

  • HIVE-9566: HiveServer2 fails to start with NullPointerException.

  • HIVE-9580: Server returns incorrect result from JOIN ON VARCHAR columns.

  • HIVE-9617: UDF from_utc_timestamp throws NPE if the second argument is null.

  • HIVE-9686: HiveMetastore.logAuditEvent can be used before SASL server is started.

  • HIVE-9695: Redundant filter operator in reducer Vertex when CBO is disabled.

  • HIVE-9711: ORC Vectorization DoubleColumnVector.isRepeating=false if all entries are NaN.

  • HIVE-9720: Metastore does not properly migrate column stats when renaming a table across databases.

  • HIVE-9721: Hadoop23Shims.setFullFileStatus should check for null.

  • HIVE-9811: Hive on Tez leaks WorkMap objects.

  • HIVE-9839: HiveServer2 leaks OperationHandle on async queries which fail at compile phase.

  • HIVE-9855: Runtime skew join doesn't work when skewed data only exists in big table.

  • HIVE-9920: DROP DATABASE IF EXISTS throws exception if database does not exist.

HDP 2.2.8 provides Hive 0.14.0 and the following additional Apache patches:

  • HIVE-10062: : HiveOnTez Union followed by Multi-GB followed by Multi-insert loses data

  • HIVE-10122: Hive metastore filter-by-expression is broken for non-partition expressions

  • HIVE-10140: Window boundary is not compared correctly

  • HIVE-10183: [CBO] self-join failing in a test case

  • HIVE-10273: Union with partition tables which have no data fails with NPE

  • HIVE-10303: HIVE-9471 broke forward compatibility of ORC files

  • HIVE-10384: RetryingMetaStoreClient does not retry wrapped TTransportExceptions

  • HIVE-10450: More than one TableScan in MapWork not supported in Vectorization -- causes query to fail during vectorization

  • HIVE-10538: Fix NPE in FileSinkOperator from hashcode mismatch

  • HIVE-10578: update SQL standard authorization configuration whitelist

  • HIVE-10595: Dropping a table can cause NPEs in the compactor

  • HIVE-10607: Combination of ReducesinkDedup + TopN optimization yields incorrect result if there are multiple GBY in reducer

  • HIVE-10685: Alter table concatenate operator will cause duplicate data

  • HIVE-10698: query on view results fails with table not found error if view is created with subquery alias (CTE).

  • HIVE-10719: Hive metastore failure when alter table rename is attempted.

  • HIVE-10722: external table creation with msck in Hive can create unusable partition

  • HIVE-10808: Inner join on Null throwing Cast Exception

  • HIVE-10816: NPE in ExecDriver handleSampling when submitted via child JVM

  • HIVE-10841: [WHERE col is not null] does not work sometimes for queries with many JOIN statements

  • HIVE-10929: In Tez mode,dynamic partitioning query with union all fails at moveTask,Invalid partition key and values

  • HIVE-10963: Hive throws NPE rather than meaningful error message when window is missing

  • HIVE-10992: WebHCat should not create delegation tokens when Kerberos is not enabled

  • HIVE-11008: WebHCat GET /jobs retries on getting job details from history server is too aggressive

  • HIVE-11023: Disable directSQL if datanucleus.identifierFactory = datanucleus2

  • HIVE-11027: Hive on tez Bucket map joins fail when hashcode goes negative

  • HIVE-11028: Tez: table self join and join with another table fails with IndexOutOfBoundsException

  • HIVE-11031: ORC concatenation of old files can fail while merging column statistics

  • HIVE-11035: PPD Orc Split elimination fails because filterColumns=[-1]

  • HIVE-11095: SerDeUtils another bug ,when Text is reused

  • HIVE-11104: Select operator doesn't propagate constants appearing in expressions

  • HIVE-11112: ISO-8859-1 text output has fragments of previous longer rows appended

  • HIVE-11147: MetaTool doesn't update FS root location for partitions with space in name

  • HIVE-11172: Vectorization wrong results for aggregate query with where clause without group by

  • HIVE-11271: java.lang.IndexOutOfBoundsException when union all with if function

  • HIVE-11285: ObjectInspector for partition columns in FetchOperator in SMBJoin causes exception

  • HIVE-11303: Getting Tez LimitExceededException after dag execution on large query

  • HIVE-11356: SMB join on tez fails when one of the tables is empty

  • HIVE-11429: Increase default JDBC result set fetch size (# rows it fetches in one RPC call) to 1000 from 50

  • HIVE-11438: Join a ACID table with non-ACID table fail with MR on 1.0.0

  • HIVE-11442: Remove commons-configuration.jar from Hive distribution

  • HIVE-11493: Predicate with integer column equals double evaluates to false

  • HIVE-11613: schematool should return non zero exit status for info command, if state is inconsistent

  • HIVE-5664: Drop cascade database fails when the db has any tables with indexes

  • HIVE-6308: COLUMNS_V2 Metastore table not populated for tables created without an explicit column list.

  • HIVE-6727: Table level stats for external tables are set incorrectly

  • HIVE-7351: ANALYZE TABLE statement fails on postgres metastore

  • HIVE-7997: Potential null pointer reference in ObjectInspectorUtils#compareTypes()

  • HIVE-8099: IN operator for partition column fails when the partition column type is DATE

  • HIVE-8295: Add batch retrieve partition objects for metastore direct SQL

  • HIVE-8326: Using DbTxnManager with concurrency off results in run time error

  • HIVE-8330: HiveResultSet.findColumn() parameters are case sensitive

  • HIVE-8374: schematool fails on Postgres versions less than 9.2

  • HIVE-8448: Union All might not work due to the type conversion issue

  • HIVE-8450: Create table like does not copy over table properties

  • HIVE-8518: Compile time skew join optimization returns duplicated results

  • HIVE-8594: Wrong condition in SettableConfigUpdater#setHiveConfWhiteList()

  • HIVE-8627: Compute stats on a table from impala caused the table to be corrupted

  • HIVE-8680: Set Max Message for Binary Thrift endpoints

  • HIVE-8706: Table statistic collection on counter failed due to table name character case.

  • HIVE-8784: Querying partition does not work with JDO enabled against PostgreSQL

  • HIVE-8863: Cannot drop table with uppercase name after compute statistics for columns

  • HIVE-8869: RowSchema not updated for some ops when columns are pruned

  • HIVE-8874: Error Accessing HBase from Hive via Oozie on Kerberos 5.0.1 cluster

  • HIVE-8889: JDBC Driver ResultSet.getXXXXXX(String columnLabel) methods Broken

  • HIVE-8926: Projections that only swap input columns are identified incorrectly as identity projections

  • HIVE-9024: NullPointerException when starting WebHCat server if templeton.hive.properties is not set

  • HIVE-9060: Fix child operator references after NonBlockingOpDeDupProc

  • HIVE-9113: Explain on query failed with NPE

  • HIVE-9155: HIVE_LOCKS uses int instead of bigint hive-txn-schema-0.14.0.mssql.sql

  • HIVE-9177: Fix child operator references after NonBlockingOpDeDupProc

  • HIVE-9217: UnionProcessor misses results for multi-insert when hive.optimize.union.remove=true

  • HIVE-9228: Problem with subquery using windowing functions

  • HIVE-9301: Potential null dereference in MoveTask#createTargetPath

  • HIVE-9309: schematool fails on Postgres 8.1

  • HIVE-9322: Make null-checks consistent for MapObjectInspector subclasses.

  • HIVE-9347: Bug with max() together with rank() and grouping sets

  • HIVE-9377: UDF in_file() in WHERE predicate causes NPE.

  • HIVE-9397: SELECT max(bar) FROM foo is broken after ANALYZE ... FOR COLUMNS

  • HIVE-9437: Beeline does not add any existing HADOOP_CLASSPATH

  • HIVE-9445: Revert HIVE-5700 - enforce single date format for partition column storage

  • HIVE-9471: Bad seek in uncompressed ORC, at row-group boundary.

  • HIVE-9507: Make "LATERAL VIEW inline(expression) mytable" tolerant to nulls

  • HIVE-9509: Restore partition spec validation removed by HIVE-9445

  • HIVE-9513: NULL POINTER EXCEPTION

  • HIVE-9526: ClassCastException thrown by HiveStatement

  • HIVE-9560: When hive.stats.collect.rawdatasize=true, 'rawDataSize' for an ORC table will result in value '0' after running 'analyze table TABLE_NAME compute statistics;'

  • HIVE-9567: JSON SerDe not escaping special chars when writing char/varchar data

  • HIVE-9592: fix ArrayIndexOutOfBoundsException in date_add and date_sub initialize

  • HIVE-9619: Uninitialized read of numBitVectors in NumDistinctValueEstimator

  • HIVE-9622: Getting NPE when trying to restart HS2 when metastore is configured to use org.apache.hadoop.hive.thrift.DBTokenStore

  • HIVE-9648: Null check key provider before doing set

  • HIVE-9655: Dynamic partition table insertion error

  • HIVE-9716: Map job fails when table's LOCATION does not have scheme

  • HIVE-9720: Metastore does not properly migrate column stats when renaming a table across databases

  • HIVE-9755: Hive built-in "ngram" UDAF fails when a mapper has no matches.

  • HIVE-9936: fix potential NPE in DefaultUDAFEvaluatorResolver

  • HIVE-9950: fix rehash in CuckooSetBytes and CuckooSetLong

  • HIVE-9952: fix NPE in CorrelationUtilities

  • HIVE-9953: fix NPE in WindowingTableFunction

  • HIVE-9975: Renaming a non-existent partition should not throw out NullPointerException

  • HIVE-9984: JoinReorder's getOutputSize is exponential

HDP 2.2.6 provided Hive 0.14.0 and the following additional Apache patches:

  • HIVE-480: HDFSCleanup thread holds reference to FileSystem

  • HIVE-6468: HiveServer2 (tcp mode, with SASL layer) OOMs when getting a non-thrift message

  • HIVE-6679: HiveServer2 should support TCP Keepalive & Server Socket Timeout on blocking calls

  • HIVE-7175: Provide password file option to beeline

  • HIVE-7270: Serde info is not shown in show create table statement, but shows in the description table

  • HIVE-8295: Add batch retrieve partition objects for metastore direct SQL

  • HIVE-8485: SUMMARY-Hive metastore NPE with Oracle DB when there is empty value for string for tblproperties/serdeproperties/etc, table not usable after creation

  • HIVE-8762: HiveMetaStore.BooleanPointer should be replaced with an AtomicBoolean

  • HIVE-8791 : Hive permission inheritance throws exception S3

  • HIVE-8850: SUMMARY-[ObjectStore:: rollbackTransaction() needs to be looked into further

  • HIVE-8881: Receiving json "could not find job" error when web client tries to fetch all jobs from WebHCat but HDFS does not have the data.

  • HIVE-8888: Hive on Tez query output duplicate rows when there is explode in subqueries for joins

  • HIVE-8891: SUMMARY-[Another possible cause to NucleusObjectNotFoundException from drops/rollback

  • HIVE-8893: Implement whitelist for built-in UDFs to avoid untrusted code execution in multiuser mode

  • HIVE-8966: Delta files created by hive hcatalog streaming cannot be compacted

  • HIVE-9025: join38.q (without map join) produces incorrect result when testing with multiple reducers

  • HIVE-9038: Join tests fail on Tez

  • HIVE-9055: Tez: union all followed by group by followed by another union all gives error

  • HIVE-9106: improve the performance of null scan optimizer when several table scans share a physical path

  • HIVE-9112: Query may generate different results depending on the number of reducers

  • HIVE-9141: HiveOnTez: mix of union all, distinct, group by generates error

  • HIVE-9155: HIVE_LOCKS uses int instead of bigint hive-txn-schema-0.14.0.mssql.sql

  • HIVE-9205: Change default Tez install directory to use /tmp instead of /user and create the directory if it does not exist

  • HIVE-9234: HiveServer2 leaks FileSystem objects in FileSystem.CACHE

  • HIVE-9235: Turn off Parquet Vectorization until all data types work: DECIMAL, DATE, TIMESTAMP, CHAR, and VARCHAR

  • HIVE-9249: java.lang.ClassCastException: org.apache.hadoop.hive.serde2.io.HiveVarcharWritable cannot be cast to org.apache.hadoop.hive.common.type.HiveVarchar when joining tables

  • HIVE-9278: Cached expression feature broken in one case

  • HIVE-9316: TestSqoop tests in WebHCat test suite hardcode libdir path to hdfs

  • HIVE-9351: Running Hive Jobs with Tez cause Templeton to never report percent complete

  • HIVE-9359: SUMMARY-[Hive export OOM error when table is huge. (32TB data, 4800+ partitions)]

  • HIVE-9382 : Query got rerun with Global Limit optimization on and Fetch optimization off

  • HIVE-9390: Enhance retry logic with regard to DB access in TxnHandler

  • HIVE-9401: SimpleFetchOptimizer for limited fetches without filters

  • HIVE-9404 NPE in org.apache.hadoop.hive.metastore.txn.TxnHandler.determineDatabaseProduct()

  • HIVE-9436: SUMMARY-[RetryingMetaStoreClient does not retry JDOExceptions]

  • HIVE-9446: JDBC DatabaseMetadata.getColumns() does not work for temporary tables

  • HIVE-9473 : SQL std auth should disallow built-in udfs that allow any java methods to be called

  • HIVE-9593: ORC reader should ignore new/unknown metadata streams.

  • HIVE-9652: STDERR redirection should not use in place updates TEZ UI

  • HIVE-9665: Parallel move task optimization causes race condition

  • HIVE-9673 : Set operation handle in ATS entities for lookups

  • HIVE-9683: Client TCP keep-alive for Thrift as a workaround for THRIFT-2788

  • HIVE-9684: Incorrect disk range computation in ORC because of optional stream kind

  • HIVE-9743 Incorrect result set for vectorized left outer join

  • HIVE-9779 : ATSHook does not log the end user if doAs=false (it logs the hs2 server user)

  • HIVE-9832: Merge join followed by union and a map join in hive on tez fails.

  • HIVE-9836: Hive on Tez: fails when virtual columns are present in the join conditions (for e.g. partition columns)

  • HIVE-9841: IOException thrown by ORC should include the path of processing file

  • HIVE-9886: Tez NPE error when initialize reducer from 2 row_number function on each join side

  • HIVE-9892: HIVE schema failed to upgrade from HDP 2.0/2.1 to HDP 2.2 with schematool