Hive
HDP 2.3.4.7 provides Hive 1.2.1 and the following Apache patch:
HIVE-12766: Thread leaks from TimelineClient.
HDP 2.3.4 provided Hive 1.2.1 and the following Apache patches:
HIVE-10397: Fix Split Computation for Acid tables with Delta files.
HIVE-10592: query fails on ORC ppd on timestamp datatype on stripes with all null on the column.
HIVE-10752: Revert HIVE-5193.
HIVE-10755: Rework on HIVE-5193 to enhance the column oriented table access.
HIVE-10778: LLAP: Utilities::gWorkMap needs to be cleaned in HiveServer2.
HIVE-10807: Invalidate basic stats for insert queries if autogather=false.
HIVE-10980: Merge of dynamic partitions loads all data to default partition.
HIVE-11008: WebHCat GET /jobs retries on getting job details from history server is too aggressive.
HIVE-11016: MiniTez mergejoin test fails with Tez input error (issue in merge join under certain conditions.
HIVE-11123: Fix how to confirm the RDBMS product name at Metastore.
HIVE-11149: sometimes HashMap in PerfLogger.java hangs.
HIVE-11312: ORC format: where clause with CHAR data type not returning any rows.
HIVE-11371: Null pointer exception for nested table query when using ORC versus text.
HIVE-11372: join with between predicate comparing integer types returns no rows when ORC format used.
HIVE-11398: Parse wide OR and wide AND trees to flat OR/AND trees.
HIVE-11422: Join a ACID table with non-ACID table fail with MR.
HIVE-11428: Performance: Struct IN() clauses are extremely slow (~10x slower).
HIVE-11432: Hive macro give same result for different arguments.
HIVE-11448: Support vectorization of Multi-OR and Multi-AND.
HIVE-11461: Transform flat AND/OR into IN struct clause.
HIVE-11462: GenericUDFStruct should constant fold at compile time.
HIVE-11468: Vectorize: Struct IN() clauses.
HIVE-11497: Make sure --orcfiledump utility includes OrcRecordUpdate.AcidStats.
HIVE-11499: Hiveserver2 failing with OOM PermGen when using temporary functions due to DataNucleus caching classloaders.
HIVE-11510: Metatool updateLocation warning on views.
HIVE-11517: Vectorized auto_smb_mapjoin_14.q produces different results.
HIVE-11523: org.apache.hadoop.hive.ql.io.orc.FileDump should handle errors.
HIVE-11540: Too many delta files during Compaction OOM.
HIVE-11546: Projected columns read size should be scaled to split size for ORC Splits.
HIVE-11573: PointLookupOptimizer can be pessimistic at a low nDV.
HIVE-11583: When PTF is used over a large partitions result could be corrupted.
HIVE-11613: schematool should return non zero exit status for info command, if state is inconsistent.
HIVE-11634: Gen Plan Changes to support multi col in clause.
HIVE-11668: make sure directsql calls pre-query init when needed.
HIVE-11669: OrcFileDump service should support directories.
HIVE-11698: Add additional test for PointLookupOptimizer.
HIVE-11712: Duplicate groupby keys cause ClassCastException.
HIVE-11720: Allow customers set a custom request/response header size for hiveserver2 using http.
HIVE-11723: Incorrect string literal escaping.
HIVE-11724: WebHcat get jobs to order jobs on time order with latest at top.
HIVE-11737: IndexOutOfBounds compiling query with duplicated groupby keys.
HIVE-11745: Alter table Exchange partition with multiple partition_spec is not working.
HIVE-11748: HivePreparedStatement's setTimestamp() does not quote value as required.
HIVE-11807: Adjust compression buffer size to avoid creation of too many small stripes.
HIVE-11825: get_json_object(col,'$.a') is null in where clause didn’t work.
HIVE-11831: TXN tables in Oracle should be created with ROWDEPENDENCIES.
HIVE-11835: Type decimal(1,1) reads 0.0, 0.00, etc from text file as NULL.
HIVE-11892: UDTF run in local fetch task does not return rows forwarded during GenericUDTF.close().
HIVE-11901: StorageBasedAuthorizationProvider requires write permission on table for SELECT statements.
HIVE-11902: Abort txn cleanup thread throws SyntaxErrorException.
HIVE-11914: When transactions gets a heartbeat, it doesn't update the lock heartbeat.
HIVE-11915: BoneCP returns closed connections from the pool.
HIVE-11916: TxnHandler.getOpenTxnsInfo() and getOpenTxns() may produce inconsistent result.
HIVE-11919: Hive Union Type Mismatch.
HIVE-11920: ADD JAR failing with URL schemes other than file/ivy/hdfs.
HIVE-11934: Transaction lock retry logic results in infinite loop.
HIVE-11939: TxnDbUtil should turn off jdbc auto commit.
HIVE-11940: 'INSERT OVERWRITE' query is very slow because it creates one distcp per file to copy data from staging directory to target directory.
HIVE-11948: Investigate TxnHandler and CompactionTxnHandler to see where we improve concurrency.
HIVE-11950: WebHCat status file doesn't show UTF8 character.
HIVE-11960: braces in join conditions are not supported.
HIVE-11964: RelOptHiveTable.hiveColStatsMap might contain mismatched column stats.
HIVE-11975: mssql scripts contains invalid 'GO' statement.
HIVE-11977: Hive should handle an external avro table with zero length files present.
HIVE-11983: Hive streaming API uses incorrect logic to assign buckets to incoming records.
HIVE-11983 Hive streaming API : uses incorrect logic to assign buckets to incoming records
HIVE-11988: SUMMARY-[security issue with hive & ranger for import table command].
HIVE-11990: Humboldt: loading data inpath from a temporary table dir fails.
HIVE-11990: loading data inpath from a temporary table dir fails.
HIVE-11995: Remove repetitively setting permissions in insert/load overwrite partition.
HIVE-11997: Add ability to send Compaction Jobs to specific queue.
HIVE-11998: Improve Compaction process logging.
HIVE-12003: HIVE-12276: Fix messages in InvalidTable.
HIVE-12003: Hive Streaming API Add check to ensure table is transactional.
HIVE-12011: unable to create temporary table using CTAS if regular table with that name already exists.
HIVE-12012: select query on json table with map containing numeric values fails.
HIVE-12021: wrong results: HivePreFilteringRule may introduce wrong common operands.
HIVE-12057: ORC sarg is logged too much.
HIVE-12076: WebHCat listing jobs after the given JobId even when templeton.jobs.listorder is set to lexicographicaldesc.
HIVE-12083: HIVE-10965 introduces thrift error if partNames or colNames are empty.
HIVE-12084: Hive queries with ORDER BY and large LIMIT fails with OutOfMemoryError Java heap space.
HIVE-12156: expanding view doesn't quote reserved keyword.
HIVE-12179: Add option to not add spark-assembly.jar to Hive classpath.
HIVE-12196: NPE when converting bad timestamp value.
HIVE-12201: Tez settings need to be shown in set -v output when execution engine is tez.
HIVE-12204: Tez queries stopped running with ApplicationNotRunningException.
HIVE-12206: using UDF in same session after other queries fails with ClassNotFound error.
HIVE-12223: Filter on Grouping__ID does not work properly.
HIVE-12230: custom UDF configure() not called in Vectorization mode.
HIVE-12232: BucketingSortingReduceSinkOptimizer throws IOB exception for duplicate columns.
HIVE-12235: a way for admin and jdbc/odbc client to see which hiveserver2 connected in ZooKeeper discovery service.
HIVE-12236: Enable SimpleFetchOptimizer for more query types.
HIVE-12249: Improve logging with tez.
HIVE-12250: query on hive table on hbasestoragehandler leak zookeeper connection.
HIVE-12252: Streaming API HiveEndPoint can be created w/o partitionVals for partitioned table.
HIVE-12254: Improve logging with yarn/hdfs.
HIVE-12257: Enhance ORC FileDump utility to handle flush_length files.
HIVE-12261: schematool version info exit status should depend on compatibility, not equality.
HIVE-12262: Session log dir cannot be created in some cases.
HIVE-12266: When client exists abnormally, it doesn't release ACID locks.
HIVE-12273: Improve user level explain.
HIVE-12276: Fix messages in InvalidTable.
HIVE-12277: Hive macro results on macro_duplicate.q different after adding ORDER BY.
HIVE-12280: HiveConnection does not try other HS2 after failure for service discovery.
HIVE-12282: beeline update command printing in verbose mode.
HIVE-12295: change some logs from info to debug.
HIVE-12307: Streaming API TransactionBatch.close() must abort any remaining transactions in the batch.
HIVE-12312: Excessive logging in PPD code.
HIVE-12318: qtest failing due to NPE in logStats.
HIVE-12327: WebHCat e2e tests TestJob_1 and TestJob_2 fail.
HIVE-12344: Wrong types inferred for SemiJoin generation in CBO.
HIVE-12345: Follow up for HIVE-9013 Hidden conf vars still visible through beeline.
HIVE-12357: Allow user to set tez job name.
HIVE-12364: insert into directory query fail, using distcp when data size>hive.exec.copyfile.maxsize (default 32MB).
HIVE-12384: Union Operator may produce incorrect result on TEZ.
HIVE-12387: Bug with logging improvements in ATS.
HIVE-12387: Issues in Hive's use of CallerContext.
HIVE-12387: When using Hive on Tez or MR, no caller context shows up in YARN audit logs.
HIVE-12389: CompactionTxnHandler.cleanEmptyAbortedTxns() should safeguard against huge IN clauses.
HIVE-12396: BucketingSortingReduceSinkOptimizer may still throw IOB exception for duplicate columns.
HIVE-12399: Filter out NULLs in the Native Vector MapJoin operators.
HIVE-12418: HiveHBaseTableInputFormat.getRecordReader() causes ZooKeeper connection leak.
HIVE-12437: SMB join in tez fails when one of the tables is empty.
HIVE-12444: Global Limit optimization on ACID table without base directory may throw exception.
HIVE-12450: OrcFileMergeOperator does not use correct compression buffer size.
HIVE-12465: Hive might produce wrong results when (outer) joins are merged.
HIVE-12469: Bump Commons-Collections dependency from 3.2.1 to 3.2.2. to address vulnerability.
HIVE-12476: Oracle directSQL NPE error when fetching empty to null for serdeproperties.
HIVE-12498: ACID: Setting OrcRecordUpdater.OrcOptions.tableProperties() has no effect.
HIVE-12500: JDBC driver not overlaying params supplied via properties object when reading params from ZK.
HIVE-12522: Wrong FS error during Tez merge files when warehouse and scratchdir are on different FS.
HIVE-12523: ATS HIVE_QUERY_ID api returns different dagName than TEZ_DAG_ID api.
HIVE-12529: HiveTxnManager.acquireLocks() should not block forever.
HIVE-12529: HiveTxnManager.acquireLocks() should not block forever.
HIVE-12556: Have an option to kill DAG when user cancels query in HiveServer2.
HIVE-12563: NullPointerException with 3-way Tez merge join.
HIVE-12565: VectorUDAFCount.aggregateInputSelection does not handle isRepeated case.
HIVE-12567: Enhance TxnHandler retry logic to handle ORA-08176.
HIVE-12578: Hive query failing with error ClassCastException org.apache.hadoop.hive.ql.plan.ExprNodeConstantDesc cannot be cast to org.apache.hadoop.hive.ql.plan.ExprNodeColumnDesc.
HIVE-12583: HS2 ShutdownHookManager holds extra of Driver instance.
HIVE-12584: join on char cols with different length returns empty result with vectorization and tez on.
HIVE-12585: fix TxnHandler connection leak.
HIVE-5623: ORC accessing array column that's empty will fail with java out of bound exception.
HIVE-7723: Explain plan for complex query with lots of partitions is slow due to in-efficient collection used to find a matching ReadEntity.
HIVE-9013: beeline (hiveserver2 client) exposes sensitive metastore DB connection info (connection, password).
HIVE-9695: Redundant filter operator in reducer Vertex when CBO is disabled.
HDP 2.3.2 provided Hive 1.2.1 and the following Apache patches:
IMPROVEMENTS
HIVE-11037 HiveOnTez: make explain user level = true as default
BUG FIXES
HIVE-10140: Window boundary is not compared correctly
HIVE-10453: Reverted
HIVE-10569: Hive CLI gets stuck when hive.exec.parallel=true; and some exception happens during SessionState.start
HIVE-10571: HiveMetaStoreClient should close existing thrift connection before its reconnect
HIVE-10620: ZooKeeperHiveLock overrides equal() method but not hashcode()
HIVE-10646: ColumnValue does not handle NULL_TYPE
HIVE-10651: ORC file footer cache should be bounded
HIVE-10698: query on view results fails with table not found error if view is created with subquery alias (CTE).
HIVE-10714: Bloom filter column names specification should be case insensitive
HIVE-10722: external table creation with msck in Hive can create unusable partition
HIVE-10726: Hive JDBC setQueryTimeout should not throw exception to make it work with JMeter
HIVE-10731: NullPointerException in HiveParser.g
HIVE-10732: Hive JDBC driver does not close operation for metadata queries
HIVE-10771: "separatorChar" has no effect in "CREATE TABLE AS SELECT" statement
HIVE-10781: HadoopJobExecHelper Leaks RunningJobs
HIVE-10790: orc write on viewFS throws exception
HIVE-10793: HIVE-11587 Hybrid Grace Hash Join: Don't allocate all hash table memory upfront
HIVE-10802: Table join query with some constant field in select fails
HIVE-10808: Inner join on Null throwing Cast Exception
HIVE-10835: Concurrency issues in JDBC driver
HIVE-10880: The bucket number is not respected in insert overwrite.
HIVE-10925: Non-static threadlocals in metastore code can potentially cause memory leak
HIVE-10963: Hive throws NPE rather than meaningful error message when window is missing
HIVE-10972: DummyTxnManager always locks the current database in shared mode, which is incorrect.
HIVE-11013: MiniTez tez_join_hash test on the branch fails with NPE (initializeOp not called?)
HIVE-11024: Error inserting a date value via parameter marker (PreparedStatement.setDate)
HIVE-11029: hadoop.proxyuser.mapr.groups does not work to restrict the groups that can be impersonated
HIVE-11054: Read error: Partition Varchar column cannot be cast to string
HIVE-11079: Fix qfile tests that fail on Windows due to CR/character escape differences
HIVE-11087: DbTxnManager exceptions should include txnid
HIVE-11090: ordering issues with windows unit test runs
HIVE-11095: SerDeUtils another bug ,when Text is reused
HIVE-11102: ReaderImpl: getColumnIndicesFromNames does not work for some cases
HIVE-11112: ISO-8859-1 text output has fragments of previous longer rows appended
HIVE-11135: Fix the Beeline set and save command in order to avoid the NullPointerException
HIVE-11151: Calcite transitive predicate inference rule should not transitively add not null filter on non-nullable input
HIVE-11152: Swapping join inputs in ASTConverter
HIVE-11157: Hive.get(HiveConf) returns same Hive object to different user sessions
HIVE-11171: Join reordering algorithm might introduce projects between joins
HIVE-11172: Vectorization wrong results for aggregate query with where clause without group by
HIVE-11174: Hive does not treat floating point signed zeros as equal (-0.0 should equal 0.0 according to IEEE floating point spec)
HIVE-11176: Caused by: java.lang.ClassCastException: org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct cannot be cast to [Ljava.lang.Object;
HIVE-11193: ConstantPropagateProcCtx should use a Set instead of a List to hold operators to be deleted
HIVE-11198: Fix load data query file format check for partitioned tables
HIVE-11203: Beeline force option doesn't force execution when errors occurred in a script.
HIVE-11211: Reset the fields in JoinStatsRule in StatsRulesProcFactory
HIVE-11216: UDF GenericUDFMapKeys throws NPE when a null map value is passed in
HIVE-11221: In Tez mode, alter table concatenate orc files can intermittently fail with NPE
HIVE-11255: get_table_objects_by_name() in HiveMetaStore.java needs to retrieve table objects in multiple batches
HIVE-11258: The function drop_database_core() of HiveMetaStore.java may not drop all the tables
HIVE-11271: java.lang.IndexOutOfBoundsException when union all with if function
HIVE-11301: thrift metastore issue when getting stats results in disconnect
HIVE-11303: Getting Tez LimitExceededException after dag execution on large query
HIVE-11317: ACID: Improve transaction Abort logic due to timeout
HIVE-11320: ACID enable predicate pushdown for insert-only delta file
HIVE-11344: HIVE-9845 makes HCatSplit.write modify the split so that PartInfo objects are unusable after it
HIVE-11356: SMB join on tez fails when one of the tables is empty
HIVE-11357: ACID enable predicate pushdown for insert-only delta file 2
HIVE-11375: Broken processing of queries containing NOT (x IS NOT NULL and x 0)
HIVE-11407: JDBC DatabaseMetaData.getTables with large no of tables call leads to HS2 OOM
HIVE-11429: Increase default JDBC result set fetch size (# rows it fetches in one RPC call) to 1000 from 50
HIVE-11433: NPE for a multiple inner join query
HIVE-11442: Remove commons-configuration.jar from Hive distribution
HIVE-11449: HIVE-11587 "Capacity must be a power of two" error when HybridHashTableContainer memory threshold is too low
HIVE-11456: HCatStorer should honor mapreduce.output.basename
HIVE-11467: HIVE-11587 WriteBuffers rounding wbSize to next power of 2 may cause OOM
HIVE-11493: Predicate with integer column equals double evaluates to false
HIVE-11502: Map side aggregation is extremely slow
HIVE-11581: HiveServer2 should store connection params in ZK when using dynamic service discovery for simpler client connection string.
HIVE-11587: Fix memory estimates for mapjoin hashtable
HIVE-11592: ORC metadata section can sometimes exceed protobuf message size limit
HIVE-11600: Hive Parser to Support multi col in clause (x,y..) in ((..),..., ())
HIVE-11605: Incorrect results with bucket map join in tez.
HIVE-11606: Bucket map joins fail at hash table construction time
HIVE-11607: Export tables broken for data > 32 MB
HIVE-11658: Load data file format validation does not work with directories
HIVE-11727: (BUG-44285). Hive on Tez through Oozie: Some queries fail with fnf exception
HIVE-11755: Incorrect method called with Kerberos enabled in AccumuloStorageHandler
HIVE-11820: export tables with size of >32MB throws "java.lang.IllegalArgumentException: Skip CRC is valid only with update options"
HIVE-11836: ORC SARG creation throws NPE for null constants with void type
HIVE-11839: Vectorization wrong results with filter of (CAST AS CHAR)
HIVE-11849: NPE in HiveHBaseTableShapshotInputFormat in query with just count(*)
HIVE-11852: numRows and rawDataSize table properties are not replicated
HIVE-11875: JDBC Driver does not honor delegation token mechanism when readings params from ZooKeeper
HIVE-11897: JDO rollback can throw pointless exceptions
HIVE-11928: ORC footer section can also exceed protobuf message limit
HIVE-11936: Support SQLAnywhere as a backing DB for the hive metastore
HIVE-5277: HBase handler skips rows with null valued first cells when only row key is selected
HIVE-6727: Table level stats for external tables are set incorrectly
HIVE-7476: CTAS does not work properly for s3
HIVE-8529: HiveSessionImpl#fetchResults should not try to fetch operation log when hive.server2.logging.operation.enabled is false.
HIVE-9566: HiveServer2 fails to start with NullPointerException
HIVE-9625: Delegation tokens for HMS are not renewed
HIVE-9811: Hive on Tez leaks WorkMap objects
HIVE-9974: Sensitive data redaction: data appears in name of mapreduce job
HDP 2.3.0 provided Hive 1.2.1 and the following Apache patches:
INCOMPATIBLE CHANGES
HIVE-11118 Load data query should validate file formats with destination tables
NEW FEATURES
HIVE-10233 Hive on Tez: memory manager for grace hash join
IMPROVEMENTS
HIVE-11164 WebHCat should log contents of HiveConf on startup HIVE-11037 HiveOnTez: make explain user level = true as default
BUG FIXES
HIVE-10251 HIVE-9664 makes hive depend on ivysettings.xml (using HIVE-10251.simple.patch)
HIVE-10996 Aggregation / Projection over Multi-Join Inner Query producing incorrect results
HIVE-11028 Tez: table self join and join with another table fails with IndexOutOfBoundsException
HIVE-11048 Make test cbo_windowing robust
HIVE-11050 testCliDriver_vector_outer_join.* failures in Unit tests due to unstable data creation queries
HIVE-11051 Hive 1.2.0 MapJoin w/Tez - LazyBinaryArray cannot be cast to [Ljava.lang.Object;
HIVE-11059 hcatalog-server-extensions tests scope should depend on hive-exec
HIVE-11060 Make test windowing.q robust
HIVE-11066 Ensure tests don't share directories on FS
HIVE-11074 Update tests for HIVE-9302 after removing binaries
HIVE-11076 Explicitly set hive.cbo.enable=true for some tests
HIVE-11083 Make test cbo_windowing robust
HIVE-11104 Select operator doesn't propagate constants appearing in expressions
HIVE-11147 MetaTool doesn't update FS root location for partitions with space in name