HDP-2.3.4 Release Notes
Also available as:
PDF

Hive

[Important]Important

Hortonworks strongly recommends that all users running HDP 2.3.4 upgrade to HDP 2.3.4.7.

HDP 2.3.4 provides Hive 1.2.1 and the following Apache patches:

  • HIVE-10397: Fix Split Computation for Acid tables with Delta files.

  • HIVE-10592: query fails on ORC ppd on timestamp datatype on stripes with all null on the column.

  • HIVE-10752: Revert HIVE-5193.

  • HIVE-10755: Rework on HIVE-5193 to enhance the column oriented table access.

  • HIVE-10778: LLAP: Utilities::gWorkMap needs to be cleaned in HiveServer2.

  • HIVE-10807: Invalidate basic stats for insert queries if autogather=false.

  • HIVE-10980: Merge of dynamic partitions loads all data to default partition.

  • HIVE-11008: WebHCat GET /jobs retries on getting job details from history server is too aggressive.

  • HIVE-11016: MiniTez mergejoin test fails with Tez input error (issue in merge join under certain conditions.

  • HIVE-11123: Fix how to confirm the RDBMS product name at Metastore.

  • HIVE-11149: sometimes HashMap in PerfLogger.java hangs.

  • HIVE-11312: ORC format: where clause with CHAR data type not returning any rows.

  • HIVE-11371: Null pointer exception for nested table query when using ORC versus text.

  • HIVE-11372: join with between predicate comparing integer types returns no rows when ORC format used.

  • HIVE-11398: Parse wide OR and wide AND trees to flat OR/AND trees.

  • HIVE-11422: Join a ACID table with non-ACID table fail with MR.

  • HIVE-11428: Performance: Struct IN() clauses are extremely slow (~10x slower).

  • HIVE-11432: Hive macro give same result for different arguments.

  • HIVE-11448: Support vectorization of Multi-OR and Multi-AND.

  • HIVE-11461: Transform flat AND/OR into IN struct clause.

  • HIVE-11462: GenericUDFStruct should constant fold at compile time.

  • HIVE-11468: Vectorize: Struct IN() clauses.

  • HIVE-11497: Make sure --orcfiledump utility includes OrcRecordUpdate.AcidStats.

  • HIVE-11499: Hiveserver2 failing with OOM PermGen when using temporary functions due to DataNucleus caching classloaders.

  • HIVE-11510: Metatool updateLocation warning on views.

  • HIVE-11517: Vectorized auto_smb_mapjoin_14.q produces different results.

  • HIVE-11523: org.apache.hadoop.hive.ql.io.orc.FileDump should handle errors.

  • HIVE-11540: Too many delta files during Compaction OOM.

  • HIVE-11546: Projected columns read size should be scaled to split size for ORC Splits.

  • HIVE-11573: PointLookupOptimizer can be pessimistic at a low nDV.

  • HIVE-11583: When PTF is used over a large partitions result could be corrupted.

  • HIVE-11613: schematool should return non zero exit status for info command, if state is inconsistent.

  • HIVE-11634: Gen Plan Changes to support multi col in clause.

  • HIVE-11668: make sure directsql calls pre-query init when needed.

  • HIVE-11669: OrcFileDump service should support directories.

  • HIVE-11698: Add additional test for PointLookupOptimizer.

  • HIVE-11712: Duplicate groupby keys cause ClassCastException.

  • HIVE-11720: Allow customers set a custom request/response header size for hiveserver2 using http.

  • HIVE-11723: Incorrect string literal escaping.

  • HIVE-11724: WebHcat get jobs to order jobs on time order with latest at top.

  • HIVE-11737: IndexOutOfBounds compiling query with duplicated groupby keys.

  • HIVE-11745: Alter table Exchange partition with multiple partition_spec is not working.

  • HIVE-11748: HivePreparedStatement's setTimestamp() does not quote value as required.

  • HIVE-11807: Adjust compression buffer size to avoid creation of too many small stripes.

  • HIVE-11825: get_json_object(col,'$.a') is null in where clause didn’t work.

  • HIVE-11831: TXN tables in Oracle should be created with ROWDEPENDENCIES.

  • HIVE-11835: Type decimal(1,1) reads 0.0, 0.00, etc from text file as NULL.

  • HIVE-11892: UDTF run in local fetch task does not return rows forwarded during GenericUDTF.close().

  • HIVE-11901: StorageBasedAuthorizationProvider requires write permission on table for SELECT statements.

  • HIVE-11902: Abort txn cleanup thread throws SyntaxErrorException.

  • HIVE-11914: When transactions gets a heartbeat, it doesn't update the lock heartbeat.

  • HIVE-11915: BoneCP returns closed connections from the pool.

  • HIVE-11916: TxnHandler.getOpenTxnsInfo() and getOpenTxns() may produce inconsistent result.

  • HIVE-11919: Hive Union Type Mismatch.

  • HIVE-11920: ADD JAR failing with URL schemes other than file/ivy/hdfs.

  • HIVE-11934: Transaction lock retry logic results in infinite loop.

  • HIVE-11939: TxnDbUtil should turn off jdbc auto commit.

  • HIVE-11940: 'INSERT OVERWRITE' query is very slow because it creates one distcp per file to copy data from staging directory to target directory.

  • HIVE-11948: Investigate TxnHandler and CompactionTxnHandler to see where we improve concurrency.

  • HIVE-11950: WebHCat status file doesn't show UTF8 character.

  • HIVE-11960: braces in join conditions are not supported.

  • HIVE-11964: RelOptHiveTable.hiveColStatsMap might contain mismatched column stats.

  • HIVE-11975: mssql scripts contains invalid 'GO' statement.

  • HIVE-11977: Hive should handle an external avro table with zero length files present.

  • HIVE-11983: Hive streaming API uses incorrect logic to assign buckets to incoming records.

  • HIVE-11983 Hive streaming API : uses incorrect logic to assign buckets to incoming records

  • HIVE-11988: SUMMARY-[security issue with hive & ranger for import table command].

  • HIVE-11990: Humboldt: loading data inpath from a temporary table dir fails.

  • HIVE-11990: loading data inpath from a temporary table dir fails.

  • HIVE-11995: Remove repetitively setting permissions in insert/load overwrite partition.

  • HIVE-11997: Add ability to send Compaction Jobs to specific queue.

  • HIVE-11998: Improve Compaction process logging.

  • HIVE-12003: HIVE-12276: Fix messages in InvalidTable.

  • HIVE-12003: Hive Streaming API Add check to ensure table is transactional.

  • HIVE-12011: unable to create temporary table using CTAS if regular table with that name already exists.

  • HIVE-12012: select query on json table with map containing numeric values fails.

  • HIVE-12021: wrong results: HivePreFilteringRule may introduce wrong common operands.

  • HIVE-12057: ORC sarg is logged too much.

  • HIVE-12076: WebHCat listing jobs after the given JobId even when templeton.jobs.listorder is set to lexicographicaldesc.

  • HIVE-12083: HIVE-10965 introduces thrift error if partNames or colNames are empty.

  • HIVE-12084: Hive queries with ORDER BY and large LIMIT fails with OutOfMemoryError Java heap space.

  • HIVE-12156: expanding view doesn't quote reserved keyword.

  • HIVE-12179: Add option to not add spark-assembly.jar to Hive classpath.

  • HIVE-12196: NPE when converting bad timestamp value.

  • HIVE-12201: Tez settings need to be shown in set -v output when execution engine is tez.

  • HIVE-12204: Tez queries stopped running with ApplicationNotRunningException.

  • HIVE-12206: using UDF in same session after other queries fails with ClassNotFound error.

  • HIVE-12223: Filter on Grouping__ID does not work properly.

  • HIVE-12230: custom UDF configure() not called in Vectorization mode.

  • HIVE-12232: BucketingSortingReduceSinkOptimizer throws IOB exception for duplicate columns.

  • HIVE-12235: a way for admin and jdbc/odbc client to see which hiveserver2 connected in ZooKeeper discovery service.

  • HIVE-12236: Enable SimpleFetchOptimizer for more query types.

  • HIVE-12249: Improve logging with tez.

  • HIVE-12250: query on hive table on hbasestoragehandler leak zookeeper connection.

  • HIVE-12252: Streaming API HiveEndPoint can be created w/o partitionVals for partitioned table.

  • HIVE-12254: Improve logging with yarn/hdfs.

  • HIVE-12257: Enhance ORC FileDump utility to handle flush_length files.

  • HIVE-12261: schematool version info exit status should depend on compatibility, not equality.

  • HIVE-12262: Session log dir cannot be created in some cases.

  • HIVE-12266: When client exists abnormally, it doesn't release ACID locks.

  • HIVE-12273: Improve user level explain.

  • HIVE-12276: Fix messages in InvalidTable.

  • HIVE-12277: Hive macro results on macro_duplicate.q different after adding ORDER BY.

  • HIVE-12280: HiveConnection does not try other HS2 after failure for service discovery.

  • HIVE-12282: beeline update command printing in verbose mode.

  • HIVE-12295: change some logs from info to debug.

  • HIVE-12307: Streaming API TransactionBatch.close() must abort any remaining transactions in the batch.

  • HIVE-12312: Excessive logging in PPD code.

  • HIVE-12318: qtest failing due to NPE in logStats.

  • HIVE-12327: WebHCat e2e tests TestJob_1 and TestJob_2 fail.

  • HIVE-12344: Wrong types inferred for SemiJoin generation in CBO.

  • HIVE-12345: Follow up for HIVE-9013 Hidden conf vars still visible through beeline.

  • HIVE-12357: Allow user to set tez job name.

  • HIVE-12364: insert into directory query fail, using distcp when data size>hive.exec.copyfile.maxsize (default 32MB).

  • HIVE-12384: Union Operator may produce incorrect result on TEZ.

  • HIVE-12387: Bug with logging improvements in ATS.

  • HIVE-12387: Issues in Hive's use of CallerContext.

  • HIVE-12387: When using Hive on Tez or MR, no caller context shows up in YARN audit logs.

  • HIVE-12389: CompactionTxnHandler.cleanEmptyAbortedTxns() should safeguard against huge IN clauses.

  • HIVE-12396: BucketingSortingReduceSinkOptimizer may still throw IOB exception for duplicate columns.

  • HIVE-12399: Filter out NULLs in the Native Vector MapJoin operators.

  • HIVE-12418: HiveHBaseTableInputFormat.getRecordReader() causes ZooKeeper connection leak.

  • HIVE-12437: SMB join in tez fails when one of the tables is empty.

  • HIVE-12444: Global Limit optimization on ACID table without base directory may throw exception.

  • HIVE-12450: OrcFileMergeOperator does not use correct compression buffer size.

  • HIVE-12465: Hive might produce wrong results when (outer) joins are merged.

  • HIVE-12469: Bump Commons-Collections dependency from 3.2.1 to 3.2.2. to address vulnerability.

  • HIVE-12476: Oracle directSQL NPE error when fetching empty to null for serdeproperties.

  • HIVE-12498: ACID: Setting OrcRecordUpdater.OrcOptions.tableProperties() has no effect.

  • HIVE-12500: JDBC driver not overlaying params supplied via properties object when reading params from ZK.

  • HIVE-12522: Wrong FS error during Tez merge files when warehouse and scratchdir are on different FS.

  • HIVE-12523: ATS HIVE_QUERY_ID api returns different dagName than TEZ_DAG_ID api.

  • HIVE-12529: HiveTxnManager.acquireLocks() should not block forever.

  • HIVE-12529: HiveTxnManager.acquireLocks() should not block forever.

  • HIVE-12556: Have an option to kill DAG when user cancels query in HiveServer2.

  • HIVE-12563: NullPointerException with 3-way Tez merge join.

  • HIVE-12565: VectorUDAFCount.aggregateInputSelection does not handle isRepeated case.

  • HIVE-12567: Enhance TxnHandler retry logic to handle ORA-08176.

  • HIVE-12578: Hive query failing with error ClassCastException org.apache.hadoop.hive.ql.plan.ExprNodeConstantDesc cannot be cast to org.apache.hadoop.hive.ql.plan.ExprNodeColumnDesc.

  • HIVE-12583: HS2 ShutdownHookManager holds extra of Driver instance.

  • HIVE-12584: join on char cols with different length returns empty result with vectorization and tez on.

  • HIVE-12585: fix TxnHandler connection leak.

  • HIVE-5623: ORC accessing array column that's empty will fail with java out of bound exception.

  • HIVE-7723: Explain plan for complex query with lots of partitions is slow due to in-efficient collection used to find a matching ReadEntity.

  • HIVE-9013: beeline (hiveserver2 client) exposes sensitive metastore DB connection info (connection, password).

  • HIVE-9695: Redundant filter operator in reducer Vertex when CBO is disabled.

HDP 2.3.2 provided Hive 1.2.1 and the following Apache patches:

IMPROVEMENTS

  • HIVE-11037 HiveOnTez: make explain user level = true as default

BUG FIXES

  • HIVE-10140: Window boundary is not compared correctly

  • HIVE-10453: Reverted

  • HIVE-10569: Hive CLI gets stuck when hive.exec.parallel=true; and some exception happens during SessionState.start

  • HIVE-10571: HiveMetaStoreClient should close existing thrift connection before its reconnect

  • HIVE-10620: ZooKeeperHiveLock overrides equal() method but not hashcode()

  • HIVE-10646: ColumnValue does not handle NULL_TYPE

  • HIVE-10651: ORC file footer cache should be bounded

  • HIVE-10698: query on view results fails with table not found error if view is created with subquery alias (CTE).

  • HIVE-10714: Bloom filter column names specification should be case insensitive

  • HIVE-10722: external table creation with msck in Hive can create unusable partition

  • HIVE-10726: Hive JDBC setQueryTimeout should not throw exception to make it work with JMeter

  • HIVE-10731: NullPointerException in HiveParser.g

  • HIVE-10732: Hive JDBC driver does not close operation for metadata queries

  • HIVE-10771: "separatorChar" has no effect in "CREATE TABLE AS SELECT" statement

  • HIVE-10781: HadoopJobExecHelper Leaks RunningJobs

  • HIVE-10790: orc write on viewFS throws exception

  • HIVE-10793: HIVE-11587 Hybrid Grace Hash Join: Don't allocate all hash table memory upfront

  • HIVE-10802: Table join query with some constant field in select fails

  • HIVE-10808: Inner join on Null throwing Cast Exception

  • HIVE-10835: Concurrency issues in JDBC driver

  • HIVE-10880: The bucket number is not respected in insert overwrite.

  • HIVE-10925: Non-static threadlocals in metastore code can potentially cause memory leak

  • HIVE-10963: Hive throws NPE rather than meaningful error message when window is missing

  • HIVE-10972: DummyTxnManager always locks the current database in shared mode, which is incorrect.

  • HIVE-11013: MiniTez tez_join_hash test on the branch fails with NPE (initializeOp not called?)

  • HIVE-11024: Error inserting a date value via parameter marker (PreparedStatement.setDate)

  • HIVE-11029: hadoop.proxyuser.mapr.groups does not work to restrict the groups that can be impersonated

  • HIVE-11054: Read error: Partition Varchar column cannot be cast to string

  • HIVE-11079: Fix qfile tests that fail on Windows due to CR/character escape differences

  • HIVE-11087: DbTxnManager exceptions should include txnid

  • HIVE-11090: ordering issues with windows unit test runs

  • HIVE-11095: SerDeUtils another bug ,when Text is reused

  • HIVE-11102: ReaderImpl: getColumnIndicesFromNames does not work for some cases

  • HIVE-11112: ISO-8859-1 text output has fragments of previous longer rows appended

  • HIVE-11135: Fix the Beeline set and save command in order to avoid the NullPointerException

  • HIVE-11151: Calcite transitive predicate inference rule should not transitively add not null filter on non-nullable input

  • HIVE-11152: Swapping join inputs in ASTConverter

  • HIVE-11157: Hive.get(HiveConf) returns same Hive object to different user sessions

  • HIVE-11171: Join reordering algorithm might introduce projects between joins

  • HIVE-11172: Vectorization wrong results for aggregate query with where clause without group by

  • HIVE-11174: Hive does not treat floating point signed zeros as equal (-0.0 should equal 0.0 according to IEEE floating point spec)

  • HIVE-11176: Caused by: java.lang.ClassCastException: org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct cannot be cast to [Ljava.lang.Object;

  • HIVE-11193: ConstantPropagateProcCtx should use a Set instead of a List to hold operators to be deleted

  • HIVE-11198: Fix load data query file format check for partitioned tables

  • HIVE-11203: Beeline force option doesn't force execution when errors occurred in a script.

  • HIVE-11211: Reset the fields in JoinStatsRule in StatsRulesProcFactory

  • HIVE-11216: UDF GenericUDFMapKeys throws NPE when a null map value is passed in

  • HIVE-11221: In Tez mode, alter table concatenate orc files can intermittently fail with NPE

  • HIVE-11255: get_table_objects_by_name() in HiveMetaStore.java needs to retrieve table objects in multiple batches

  • HIVE-11258: The function drop_database_core() of HiveMetaStore.java may not drop all the tables

  • HIVE-11271: java.lang.IndexOutOfBoundsException when union all with if function

  • HIVE-11301: thrift metastore issue when getting stats results in disconnect

  • HIVE-11303: Getting Tez LimitExceededException after dag execution on large query

  • HIVE-11317: ACID: Improve transaction Abort logic due to timeout

  • HIVE-11320: ACID enable predicate pushdown for insert-only delta file

  • HIVE-11344: HIVE-9845 makes HCatSplit.write modify the split so that PartInfo objects are unusable after it

  • HIVE-11356: SMB join on tez fails when one of the tables is empty

  • HIVE-11357: ACID enable predicate pushdown for insert-only delta file 2

  • HIVE-11375: Broken processing of queries containing NOT (x IS NOT NULL and x 0)

  • HIVE-11407: JDBC DatabaseMetaData.getTables with large no of tables call leads to HS2 OOM

  • HIVE-11429: Increase default JDBC result set fetch size (# rows it fetches in one RPC call) to 1000 from 50

  • HIVE-11433: NPE for a multiple inner join query

  • HIVE-11442: Remove commons-configuration.jar from Hive distribution

  • HIVE-11449: HIVE-11587 "Capacity must be a power of two" error when HybridHashTableContainer memory threshold is too low

  • HIVE-11456: HCatStorer should honor mapreduce.output.basename

  • HIVE-11467: HIVE-11587 WriteBuffers rounding wbSize to next power of 2 may cause OOM

  • HIVE-11493: Predicate with integer column equals double evaluates to false

  • HIVE-11502: Map side aggregation is extremely slow

  • HIVE-11581: HiveServer2 should store connection params in ZK when using dynamic service discovery for simpler client connection string.

  • HIVE-11587: Fix memory estimates for mapjoin hashtable

  • HIVE-11592: ORC metadata section can sometimes exceed protobuf message size limit

  • HIVE-11600: Hive Parser to Support multi col in clause (x,y..) in ((..),..., ())

  • HIVE-11605: Incorrect results with bucket map join in tez.

  • HIVE-11606: Bucket map joins fail at hash table construction time

  • HIVE-11607: Export tables broken for data > 32 MB

  • HIVE-11658: Load data file format validation does not work with directories

  • HIVE-11727: (BUG-44285). Hive on Tez through Oozie: Some queries fail with fnf exception

  • HIVE-11755: Incorrect method called with Kerberos enabled in AccumuloStorageHandler

  • HIVE-11820: export tables with size of >32MB throws "java.lang.IllegalArgumentException: Skip CRC is valid only with update options"

  • HIVE-11836: ORC SARG creation throws NPE for null constants with void type

  • HIVE-11839: Vectorization wrong results with filter of (CAST AS CHAR)

  • HIVE-11849: NPE in HiveHBaseTableShapshotInputFormat in query with just count(*)

  • HIVE-11852: numRows and rawDataSize table properties are not replicated

  • HIVE-11875: JDBC Driver does not honor delegation token mechanism when readings params from ZooKeeper

  • HIVE-11897: JDO rollback can throw pointless exceptions

  • HIVE-11928: ORC footer section can also exceed protobuf message limit

  • HIVE-11936: Support SQLAnywhere as a backing DB for the hive metastore

  • HIVE-5277: HBase handler skips rows with null valued first cells when only row key is selected

  • HIVE-6727: Table level stats for external tables are set incorrectly

  • HIVE-7476: CTAS does not work properly for s3

  • HIVE-8529: HiveSessionImpl#fetchResults should not try to fetch operation log when hive.server2.logging.operation.enabled is false.

  • HIVE-9566: HiveServer2 fails to start with NullPointerException

  • HIVE-9625: Delegation tokens for HMS are not renewed

  • HIVE-9811: Hive on Tez leaks WorkMap objects

  • HIVE-9974: Sensitive data redaction: data appears in name of mapreduce job

HDP 2.3.0 provided Hive 1.2.1 and the following Apache patches:

INCOMPATIBLE CHANGES

  • HIVE-11118 Load data query should validate file formats with destination tables

NEW FEATURES

  • HIVE-10233 Hive on Tez: memory manager for grace hash join

IMPROVEMENTS

  • HIVE-11164 WebHCat should log contents of HiveConf on startup HIVE-11037 HiveOnTez: make explain user level = true as default

BUG FIXES

  • HIVE-10251 HIVE-9664 makes hive depend on ivysettings.xml (using HIVE-10251.simple.patch)

  • HIVE-10996 Aggregation / Projection over Multi-Join Inner Query producing incorrect results

  • HIVE-11028 Tez: table self join and join with another table fails with IndexOutOfBoundsException

  • HIVE-11048 Make test cbo_windowing robust

  • HIVE-11050 testCliDriver_vector_outer_join.* failures in Unit tests due to unstable data creation queries

  • HIVE-11051 Hive 1.2.0 MapJoin w/Tez - LazyBinaryArray cannot be cast to [Ljava.lang.Object;

  • HIVE-11059 hcatalog-server-extensions tests scope should depend on hive-exec

  • HIVE-11060 Make test windowing.q robust

  • HIVE-11066 Ensure tests don't share directories on FS

  • HIVE-11074 Update tests for HIVE-9302 after removing binaries

  • HIVE-11076 Explicitly set hive.cbo.enable=true for some tests

  • HIVE-11083 Make test cbo_windowing robust

  • HIVE-11104 Select operator doesn't propagate constants appearing in expressions

  • HIVE-11147 MetaTool doesn't update FS root location for partitions with space in name