HDP 3.1.4 Release Notes
Also available as:
PDF

Fixed Issues

Fixed issues represents selected issues that were previously logged via Hortonworks Support, but are now addressed in the current release. These issues may have been reported in previous versions within the Known Issues section; meaning they were reported by customers or identified by Hortonworks Quality Engineering team.

Incorrect-results

Bug ID Apache JIRA Apache Component Summary
BUG-121291 HIVE-21746 Hive ArrayIndexOutOfBoundsException during dynamically partitioned hash join, with CBO disabled
BUG-121285 HIVE-21862 Hive ORC ppd produces wrong result with timestamp
BUG-120240 HIVE-21509 Hive LLAP may cache corrupted column vectors and return wrong query result
BUG-120048 HIVE-21291 Hive Restore historical way of handling timestamps in Avro while keeping the new semantics at the same time
BUG-119841 HIVE-20274 Hive HiveServer2 ObjectInspectorFactory leaks for Struct and List object inspectors
BUG-119788 SPARK-23986SPARK-24495SPARK-24589SPARK-25368SPARK-26682SPARK-27160 Spark Data integrity and correctness problems with Spark 2.3.0
BUG-119025 HIVE-21460 Hive ACID: Load data followed by a select * query results in incorrect results
BUG-118705 TEZ-3982 Tez DAGAppMaster and tasks should not report negative or invalid progress

Other

Bug ID Apache JIRA Apache Component Summary
BUG-121579 BUG-121509 Atlas Atlas upgrade will likely hit out-of-memory in environments having large amount of data
BUG-121516 BUG-121509 Atlas Atlas split2: test_nested_entity_filter_search_OR_of_ORs_and_ANDs
BUG-121509 ATLAS-3033 Atlas ATLAS Hive hook fails for CTAS query involving UDFs
BUG-121414 N/A HBase Incremental and restore backup fails on AWS cluster
BUG-121375 ATLAS-3045 Atlas Notification processing to provide an option to ignore Hive dummy databases and tables
BUG-121193 ATLAS-3045, ATLAS-3045, ATLAS-3045, ATLAS-3045 Atlas Incorrect column lineage: each output column has input from *all columns* of the input table
BUG-121149 ATLAS-3333 Atlas Notification processing to provide an option to ignore Hive dummy databases and tables
BUG-121095 HBASE-21156 HBase Fix HBCK2 to work with Kerberized deployment of HBase
BUG-121018 N/A HWC Disable pruning and pushdowns for spark v2.3.x by default
BUG-120946 HIVE-21888 Hive Set hive.parquet.timestamp.skip.conversion default to true
BUG-120941 PHOENIX-5309 Phoenix No job progress after adding phoenix-hive JAR under AUX JAR PATH
BUG-120843 HBASE-22617 HBase Recovered WAL directories not getting cleaned up
BUG-120840 RANGER-2479 Ranger Change test connection preferred SQL statement for Oracle database Flavor
BUG-120817 ATLAS-3006 Atlas Option to ignore/prune metadata for temporary/staging Hive tables
BUG-120782 ATLAS-3218 Atlas Atlas user interface - entity detail page columns sorting empty value check
BUG-120734 N/A HWC Turn off smart execution by default
BUG-120717 RANGER-2475 Ranger Replacing bootstrap accordion with jquery slideToggle.
BUG-120706 HIVE-21827 Hive Multiple calls in SemanticAnalyzer do not go through getTableObjectByName method
BUG-120705 HBASE-21284 HBase Adopt less stringent compaction throughput defaults
BUG-120652 ATLAS-3034 ATLAS-3043 ATLAS-3051 ATLAS-3053 ATLAS-3055 ATLAS-3056 ATLAS-3063 ATLAS-3065 Atlas Kafka message size reduced
BUG-120588 N/A HWC Incorrect data when writes happen in parallel
BUG-120561 n/a Phoenix Upgrade Phoenix jetty 9.3
BUG-120411 ATLAS-3218 Atlas Atlas user interface - entity detail page columns sorting empty value check
BUG-120371 ATLAS-3129 Atlas Fix SSL Truststore reloader leak from AtlasBaseClient
BUG-120292 ATLAS-3129 Atlas Fix SSL Truststore reloader leak from AtlasBaseClient
BUG-120220 ATLAS-2634 ATLAS-2891 ATLAS-2975 ATLAS-3006 Atlas Improve Atlas NotificationhookConsumer performance
BUG-120082 ATLAS-3193 Atlas Hive bootstrap fails with atlas exception
BUG-120081 HIVE-21668 Hive Backport HIVE-21668 to 3.1 - Addressing vulnerabilities
BUG-120070 HIVE-21722 Hive REPL::END event not printed in the background thread pool
BUG-120055 RANGER-2423 Ranger Ranger KnoxSSO authentication in Ranger HA environment
BUG-120046 HIVE-21619 Hive Print timestamp type without precision in SQL explain extended
BUG-120033 HIVE-21717 Hive Hive bootstrap instance fails when table in source databse is updated when bootstrap is running
BUG-120032 ATLAS-3045 Atlas Hive replication fails with Atlas Exception
BUG-120022 RANGER-2419 Ranger Improve SQL script to skip statements when atlas service def is not supported
BUG-119969 HBASE-20604 HBase ProtobufLogReader#readNext can incorrectly loop to the same position in the stream until the the WAL is rolled
BUG-119898 HBASE-21688, HBASE-22200 HBase Backport fixes from separate filesystems for WAL and storefiles
BUG-119897 HBASE-20662, HBASE-21225, HBASE-21631, HBASE-21634 HBase Backport quota-related fixes
BUG-119896 HBASE-19008, HBASE-21129, HBASE-21620, HBASE-21734, HBASE-22144 HBase Backport of upstream Apache HBase issues around filters
BUG-119895 HBASE-21402, HBASE-21618, HBASE-21621, HBASE-21644, HBASE-21775, HBASE-21890, HBASE-21932, HBASE-22185 HBase Upstream Apache HBase backports
BUG-119894 HBASE-18484, HBASE-20220, HBASE-21413, HBASE-21645, HBASE-21683, HBASE-21732, HBASE-21915, HBASE-22123, HBASE-22128 HBase Backport of upstream Apache HBase operator tooling and administrative cleanliness
BUG-119893 HBASE-21738 HBase Backport of upstream Performance related fixes
BUG-119892 ATLAS-3176 ATLAS-3193 Atlas Atlas import fails if the target database is already created
BUG-119870 HIVE-21677 Hive Using strict managed tables for ACID table fixes
BUG-119848 HIVE-20291HIVE-21671 Hive Replicate streaming ingestion with transactional batch size as 1.
BUG-119819 HIVE-21651 Hive Move protobuf into hive-exec.
BUG-119809 HIVE-21232, HIVE-21294, HIVE-21564, ORC-477, , ORC-491 Hive2 Backport of upstream Hive & ORC fixes - HIVE-21232, HIVE-21294, HIVE-21564, ORC-491,ORC-477
BUG-119802 ATLAS-3177 Atlas Regression, Export : changeMarker is not set right - set to older value compared to entity's lastModifedTime
BUG-119795 ATLAS-3176 Atlas Regression , Export : Export zip doesn't contain all entities which are listed in atlas-export-order.json
BUG-119793 RANGER-2445 Ranger Import of Tagservice for zone
BUG-119792 N/A Hive External table location is not preserved at target when base dir is set as /.
BUG-119746 RANGER-2407 Ranger Update default header values sent from Ranger
BUG-119733 ATLAS-3168 Atlas fails with NPE(createTableIfNotExists exception)
BUG-119693 N/A Druid Druid Historic Nodes going down
BUG-119632 RANGER-2400, RANGER-2402 Ranger Policy name needs to be unique within security zone and service
BUG-119546 STORM-3068 Storm STORM_JAR_JVM_OPTS are not passed to storm-kafka-monitor properly
BUG-119544 STORM-3123 Storm Storm Kafka Monitor does not work with Kafka over two-way SSL
BUG-119535 RANGER-2404 Ranger Issue of delegated admin for zone and unzoned policy associated to same resource
BUG-119514 CALCITE-2929, HIVE-21342, HIVE-21544 Calcite Backport the Calcite issue where analyze compute stats for column leave behind staging dir on HDFS
BUG-119484 ATLAS-3116 Atlas Fixes the issue when Atlas is configured to use both Knox SSO and LDAP authentication. LDAP login fails, once a sign-in is made through Knox SSO.
BUG-119481 ATLAS-2842 Atlas Entities with different guids does not get referenced in array with SET cardinality
BUG-119463 ATLAS-3182 ATLAS-3188 Atlas Hive Replication fails during Atlas export ( 3.1 - 3.1 )
BUG-119374 HBASE-22128 HBase Backport Apache HBASE-22128 "Move namespace region then master crashed make deadlock"
BUG-119354 CALCITE-2976 Hive Improve materialized view rewriting coverage with disjunctive predicates
BUG-119330 RANGER-2408 Ranger Restrict Ranger User's capabilities according to their role
BUG-119255 HBASE-21915 HBase FileLink$FileLinkInputStream doesn't implement CanUnbuffer
BUG-119253 RANGER-2377 Ranger Make solr bootstrapping configurable
BUG-119251 RANGER-2392 Ranger Description of zone is not available on the UI
BUG-119241 KNOX-1848, KNOX-1851 Knox Remote Alias Discovery Service Not Working
BUG-119229 N/A HWC Hive Warehouse Connector is changing External table as Managed table in Overwrite mode
BUG-119189 RANGER-2387 Ranger Add public api v2 for security zones
BUG-119174 RANGER-2392 Ranger Create / Update zone to have provision to associate Tag based service with zone
BUG-119159 ATLAS-3054 Atlas hive_column state is not getting changed to "DELETED" after a column is replaced
BUG-119137 ATLAS-3136 Atlas Assigning a term to an invalid entity GUID throws 500, expected is 400.
BUG-119120 N/A Hive Backport HIVE-21404 MSSQL upgrade script alters the wrong column
BUG-119109 HIVE-21362 Hive HIVE-21362 - Add an input format and serde to read from protobuf files.
BUG-119078 RANGER-2384 Ranger Get All Zones API is returning response in raw format,proper response object is required.
BUG-119018 HIVE-21500 Hive Disable conversion of managed table to external and vice versa at source.
BUG-118999 CALCITE-2946 Hive RelBuilder wrongly skips creation of Aggregate that prunes columns if input produces one row at most
BUG-118890 PHOENIX-5178 Phoenix [PHOENIX-5178] - AccessDeniedException while connecting to Phoenix with read access to SYSTEM name space
BUG-118858 KNOX-1830 Knox Fix sporadic Illegal character error in TestHashicorpVaultAliasService
BUG-118833 RANGER-2399 Ranger User's listing page hits users API call twice from UI
BUG-118820 RANGER-2372 Ranger Remove non-existing URL entries from spring config file
BUG-118781 RANGER-2383 Ranger Incorrect response when trying to delete user attached to a security zone
BUG-118779 HADOOP-16104 Hadoop Common WASB tests to downgrade to skip when test a/c is namespace enabled
BUG-118772 HADOOP-16169 Hadoop Common ABFS: Bug fix for getPathProperties
BUG-118770 HADOOP-16068 Hadoop Common ABFS Authentication and Delegation Token plugins to optionally be bound to specific URI of the store
BUG-118764 HADOOP-16163 Hadoop Common NPE in setup/teardown of ITestAbfsDelegationTokens
BUG-118763 HADOOP-15813 Hadoop Common Enable more reliable SSL connection reuse
BUG-118729 KNOX-1826 Knox Replace zip4j with ShrinkWrap
BUG-118672 HADOOP-16182 Hadoop Common Update ABFS storage back-end with "close" flag when application is done writing to a file
BUG-118667 N/A Knox Let end-users customize Knox without editing start scripts
BUG-118663 RANGER-2358 Ranger Update Ranger libraries version
BUG-118644 HIVE-21435 Hive LlapBaseInputFormat should get task number from TASK_ATTEMPT_ID conf if present, while building SubmitWorkRequestProto
BUG-118633 CALCITE-2789 Calcite Upgrade jackson-databind to 2.9.8
BUG-118615 HIVE-21214 Hive Backport HIVE-21214 - HiveServer2 split 2 empty bucket tests fail due to missing data from output
BUG-118599 RANGER-2446 Ranger Include security zone details as part of admin audit for policy update
BUG-118583 RANGER-2354 Ranger Enhance Ranger UI Policy Page to have "Add Policy Condition"
BUG-118573 N/A Knox The Knox Gateway truststore should be configurable
BUG-118571 RANGER-2340 Ranger Add Policy Version to the Ranger Audit log
BUG-118530 PHOENIX-5070 Phoenix Create local index command failed with NPE
BUG-118447 N/A Hive Propagate exception from all the APIs using executeUpdate()
BUG-118444 HIVE-21403 Hive Error code for failure in case of bootstrap twice should qualify for failed admin case.
BUG-118440 RANGER-2347 Ranger Audits Tab : Zone Auditor should see audits only related to his own zones
BUG-118439 RANGER-2355 Ranger Reports page: policy listing to have column of Zone name
BUG-118428 KNOX-1787 Knox Create PasswordUtils class in gateway-util-common
BUG-118389 KNOX-1801 Knox Master secret is incorrectly assumed when a custom truststore is not specified when clientauth is enabled
BUG-118292 KNOX-1793 Knox DefaultKeystoreService should not validate the signing key on initialization
BUG-118282 KNOX-1791 Knox MasterService should be a field on GatewayServices
BUG-118252 N/A Ranger Fix Ranger vulnerabilities
BUG-118205 HIVE-21329 Hive Custom Tez runtime unordered output buffer size depending on operator pipeline
BUG-118194 KNOX-1756 Knox Knox Gateway TLS Keystore and Alias Should be Configurable
BUG-118168 RANGER-2382, RANGER-2385 Ranger Improvement to Audit page
BUG-118090 KNOX-1687 Knox Hashicorp Vault alias credential provider
BUG-118089 KNOX-1789 Knox Refactor RemoteAliasService to use service loading
BUG-118065 HBASE-21960 HBase Backport to-be-disclosed HBase security issue
BUG-118042 ATLAS-3045, ATLAS-3046, ATLAS-3048 Atlas Fixes an issue where you see duplicate tables and databases in Atlas
BUG-118035 KNOX-1802 Knox Plugins API support for ranger is missing in Knox.
BUG-117987 N/A Storm Upgrade Apache Thrift Java client library to 0.12.0 due to CVE-2018-1320
BUG-117906 HIVE-21278 Hive Fixes an ambiguity in grammar warnings at compilation time
BUG-117880 PHOENIX-4872 Phoenix Inconsistent/missing results during Phoenix CsvBulkLoadTool
BUG-117870 N/A Ranger Export policies based on zone.
BUG-117860 HIVE-21529 Hive Hive support bootstrap of ACID/MM tables on an existing policy.
BUG-117855 STORM-3046 Storm Getting a NPE leading worker to die when starting a topology.
BUG-117854 HIVE-21286 Hive Hive should support clean-up of previously bootstrapped tables when retry from different dump.
BUG-117807 HIVE-21281 Hive Repl checkpointing doesn't work when retry bootstrap load with partitions of external tables.
BUG-117758 N/A Knox Add back Livy Session API after RMP-12915
BUG-117703 HIVE-21167 Hive Bucketing: Bucketing version 1 is incorrectly partitioning data
BUG-117639 HBASE-21200 HBase "Memstore flush doesn't finish because of seekToPreviousRow() in memstore scanner." to branch-1
BUG-117634 HIVE-21188 Hive Backport HIVE-21188: Hive query on view fails with SemanticException when there is a Ranger masked policy in place
BUG-117588 RANGER-2351 Ranger Implement Import / Export of Policies by Zone
BUG-117561 ATLAS-3050 Atlas HDFS replication instance failed intermittently due to Atlas replication failure.
BUG-117559 HADOOP-16041 Hadoop Common UserAgent string for ABFS
BUG-117424 RANGER-2333, RANGER-2339 Ranger All Users should be able to view "Security Zone" tab and should get access to security zones.
BUG-117420 RANGER-2339 Ranger Hidden users and groups are getting listed in Zone administration section during zone creation.
BUG-117377 HADOOP-16036 Hadoop Common WASB: Disable jetty logging configuration announcement
BUG-117370 HADOOP-15876 Hadoop Common Use keySet().removeAll() to remove multiple keys from Map in AzureBlobFileSystemStore
BUG-117369 HADOOP-15872 Hadoop Common HADOOP-15872: REST version for ADLS Gen 2
BUG-117261 RANGER-1937 Ranger Ranger Tagsync not syncing tags in Atlas
BUG-117192 N/A Knox Knox HDP-3.0.100.0 build failure
BUG-117159 RANGER-2234 Ranger Cannot add or update a child row,a foreign key constraint fails when installing ranger-admin
BUG-117146 KNOX-1750 Knox Knox Admin UI - Unable to view descriptor service params
BUG-117134 N/A HWC HWC unittests failure - TestHive[Streaming|Server2]CredentialProvider
BUG-117116 CALCITE-2388, CALCITE-2733 Calcite Use catalog and schema from JDBC connection string to retrieve tables if specified
BUG-117115 CALCITE-2732 Calcite Upgrade Postgresql driver version
BUG-117114 CALCITE-2673 Calcite Default implementation of SqlDialect should not support pushing all functions (addendum)
BUG-117106 SPARK-26118 Spark Make Jetty's requestHeaderSize configurable in Spark
BUG-117104 KNOX-1705, KNOX-1707, KNOX-1708, KNOX-1731, KNOX-1735 Knox Knox Admin UI improvements
BUG-116782 RANGER-2411 Ranger Admin role user is able to create Zone for KMS service with REST API
BUG-116737 HIVE-21236 Hive Hive JDBC Storage Handler: TPCDS_GRP_3_88 doesn't output proper results.
BUG-116676 ATLAS-3006 Atlas Option to ignore/prune metadata for temporary/staging Hive tables
BUG-116638 KNOX-1339, KNOX-1728 Knox Allow custom parameters to be passed to dispatches
BUG-116480 RANGER-2313 Ranger Customer is getting duplicate tables and databases in Atlas
BUG-116359 HIVE-18661 Hive CachedStore: Use metastore notification log events to update cache
BUG-116358 N/A HWC SparkRHWC: test a set of user scenarios for Zeppelin, Livy, Spark Shell/Submit
BUG-116357 N/A HWC SparkR support for HWC
BUG-116346 HIVE-20772 Hive Record per-task CPU counters in LLAP
BUG-116344 HIVE-20210 Hive Simple Fetch optimizer should lead to MapReduce when filter on non-partition column and conversion is minimal
BUG-116324 KNOX-1679 Knox Admin API to add alias for a given topology
BUG-116295 HIVE-20568 Hive HWC: Append SaveMode fails with Ranger
BUG-116287 HIVE-21295 Hive Hive JDBC Storage Handler (Oracle): Few TPCDS queries producing improper results
BUG-116193 KNOX-1683 Knox Ensure truststorePass is set in KnoxSession
BUG-116145 N/A Knox Add Knox Cert to KnoxToken Service Response and KnoxSession Clients
BUG-116082 N/A Hive Hive JDBC Storage Handler : TPCDS_GRP_4_22 OOM (mysql, postgres)
BUG-116018 RANGER-2232 Ranger Implement search for Zone available in Ranger Admin on Security Zone page
BUG-116011 HIVE-21043 Hive Enable move optimization for cloud replication with strict managed tables.
BUG-115992 HIVE-19081, HIVE-20859, HIVE-20941, HIVE-20960, HIVE-21159 Hive List of Hive backports
BUG-115986 KNOX-1623 Knox Fix SecureKnoxShellTest
BUG-115974 HBASE-21551 HBase Memory leak when use scan with STREAM at server side
BUG-115967 CALCITE-2713 Hive Hive JDBC Storage Handler (Postgres): STDJOIN_66, UnionDistinct_3 and GRP_2_66 fails with "ERROR: type "pg_catalog.varchar_UTF-16LE" does not exist"
BUG-115911 RANGER-2232 Ranger Add zone name in Policy detail popup in Access Tab
BUG-115908 RANGER-2309 Ranger Partial search for group lookup on policy edit page not working properly.
BUG-115899 YARN-9114 YARN [YARN-9114] [UI2] log service redirect url should also support user name
BUG-115829 HIVE-21029 Hive Bootstrap of external tables for existing DLM clusters
BUG-115825 RANGER-2232 Ranger Add select zone option during policy creation from Ranger Admin UI
BUG-115744 N/A Pig Cherry-pick missing pig patches from HDP 2.6-maint to 3.1-maint
BUG-115724 KNOX-1364 Knox Cookie scoping path should contain the topology name
BUG-115701 STORM-2993 Storm Storm HDFS bolt throws ClosedChannelException when Time rotation policy is used
BUG-115700 HIVE-21055 Hive HDP 2.6 to 3.1 Hive onprem-onprem replication running in serial mode.
BUG-115620 HBASE-21568 HBase Permission denied exception on /mnt/hbase/cache.data on HDI environments
BUG-115596 ATLAS-2969 Atlas Hive replication failed with Atlas import failure.
BUG-115585 N/A HWC HWC: MergeBuilder
BUG-115580 CALCITE-2719 Hive Backport CALCITE-2719: In JDBC adapter for MySQL, fix cast to INTEGER and BIGINT
BUG-115443 HIVE-21013 Hive Hive JDBC Storage Handler (Oralce): select min(salary) failing with "java.io.IOException: java.io.IOException: Cannot find partitionColumn:salary in [EMP_NO, SALARY, FROM_DATE, TO_DATE]"
BUG-115436 CALCITE-2733HIVE-21383HIVE-21468 Hive Hive JDBC Storage Handler (Oracle): Seeing "java.lang.IllegalArgumentException: Multiple entries with same key: APEX_ACTIVITY_LOG=JdbcTable {APEX_ACTIVITY_LOG} and APEX_ACTIVITY_LOG=JdbcTable {APEX_ACTIVITY_LOG}"
BUG-115404 KNOX-1623 Knox Kerberos support for KnoxShell
BUG-115070 KNOX-1474, KNOX-1530, KNOX-1531, KNOX-1532, KNOX-1533, KNOX-1534, KNOX-1535 Knox Backport KNOX-1530 - Improve Gzip Compression Handling Performance
BUG-115029 RANGER-2291 Ranger Make optimized db schema script idempotent for all DB Flavors
BUG-114938 ATLAS-2995 Term Entity disassociation doesn't happen via knox proxy
BUG-114621 RANGER-2295 Ranger Set specific Ranger version in optimized schema
BUG-114459 ATLAS-2969 Atlas DLM Policy instances are failing if no atlas incremental data present
BUG-114234 RANGER-2291 Ranger Core DB may be created partially in HA setup
BUG-113835 HADOOP-15855 Hadoop Common Backport HADOOP-15855: Review hadoop credential doc, including object store details
BUG-113745 HIVE-21085 EDWS Materialized views registry starts non-external tez session
BUG-113612 YARN-8961 YARN Flow Run End Time shows 'Invalid date'
BUG-113546 N/A EDWS change service API to conform to REST approach
BUG-113270 HADOOP-15466 Hadoop Common HADOOP-15466: Correct units in adl.http.timeout
BUG-112708 N/A Hive REPL DUMP to output the included tables list in a metadata file _tables.
BUG-112707 HIVE-21764 Hive REPL DUMP should detect and bootstrap any rename table events where old table was excluded but renamed table is included.
BUG-112702 ATLAS-2921 Atlas Atlas hive-bridge import-hive.sh returns 0 in case of java error
BUG-112593 N/A Hive EDWS: store cluster status in cluster list when creating
BUG-112536 RANGER-2373 Ranger User creation POST and PUT response not showing groupIdList and groupNameList with expected data
BUG-112056 HIVE-20911 Hive Bootstrap and incremental replication for External Tables in Hive.
BUG-112038 HIVE-21763 Hive Incremental replication to allow changing include/exclude tables list in replication policy.
BUG-112034 HIVE-21762 Hive REPL DUMP to support new format for replication policy input to take included tables list.
BUG-112028 HIVE-21761 Hive Support table level replication in Hive
BUG-112015 ATLAS-2885 ATLAS-2891 Atlas Atlas replication taking longer time
BUG-111962 N/A Hive Support external table replication in Hive
BUG-111655 N/A YARN RegistryDNS: when ambari user does not have permission to execute "su root" then starting/stopping registry dns fails
BUG-111386 YARN-8803 YARN \Show flow runs in the order of recently created time in graph widgets
BUG-110789 N/A Hive EDWS: implement db storage mechanism
BUG-110358 ATLAS-3019 Atlas Export fails on setting replicatedTo value to null in export JSON.
BUG-110170 ATLAS-3019 Atlas Using transforms (replace) with both supertype and subtype fails with NPE during import
BUG-109937 ATLAS-2842 Atlas 2.6-maint - "Slow UI load and REST improvement for entities with ownedRef" and "Creating entity containing attribute of type set with duplicate values"
BUG-107745 HIVE-16812, HIVE-17231, HIVE-17917, HIVE-20635, HIVE-20664, HIVE-20694, HIVE-20730, HIVE-20738 Hive DeleteReaderValue optimization to load only the right bucket and only relevant events
BUG-95263 YARN-7824 YARN [UI2] [YARN-7824] Yarn Component Instance page should include link to container logs
BUG-92696 N/A Hive Replicate Dynamic Partition and CTAS operations on ACID tables

Performance

Bug ID Apache JIRA Apache Component Summary
BUG-121232 HIVE-21980 Hive Parsing time can be high in case of deeply nested subqueries
BUG-121157 HIVE-21934 Hive Materialized view on top of Druid not pushing everything
BUG-121142 HIVE-21912 Hive Implement BlacklistingLlapMetricsListener
BUG-121080 HADOOP-16341 Hadoop Common ShutDownHookManager: Regressed performance on Hook removals after HADOOP-15679
BUG-120873 HIVE-21232,HIVE-21332,HIVE-21509,HIVE-21686,HIVE-21768,HIVE-21805 Hive Fixes for HIVE-21805, HIVE-21332, HIVE-21686, HIVE-21509. Revert HIVE-21232
BUG-120708 HIVE-21834 Hive Avoid unnecessary calls to simplify filter conditions
BUG-120707 HIVE-21818 Hive CBO: Copying TableRelOptHiveTable has metastore traffic
BUG-120375 HIVE-21793 Hive CBO retrieves column stats even if hive.stats.fetch.column.stats is set to false
BUG-120295 HIVE-21965 Hive HiveStrictManagedMigration process can run for days!!
BUG-120275 ORC-498 Hive ReaderImpl and RecordReaderImpl open separate file handles
BUG-120047 HIVE-21685 Hive Wrong simplification in query with multiple IN clauses
BUG-119970 HIVE-21631 Hive Backport HIVE-21631 to 3.1 - Enhance metastore API to allow bulk-loading materialized views
BUG-119841 HIVE-20274 Hive HiveServer2 ObjectInspectorFactory leaks for Struct and List object inspectors
BUG-119460 HIVE-21568 Hive HiveRelOptUtil.isRowFilteringPlan should skip Project
BUG-118459 HADOOP-16126 Hadoop Common Backport to 3.1 line - ipc.Client.stop() may sleep too long to wait for all connections
BUG-116888 HADOOP-15940 Hadoop Common ABFS: For HNS account, avoid unnecessary get call when doing Rename
BUG-116750 HADOOP-15972 Hadoop Common HADOOP-15972: ABFS: update LIST_MAX_RESULTS
BUG-115948 ORC-397, ORC-435 Hive Backport ORC-397 and ORC-435 to 3.1-maint
BUG-115898 HIVE-21206 Hive Hive onprem-onprem replication is slow on HDP 3.1 cluster
BUG-115311 HIVE-20989 Hive JDBC - The GetOperationStatus + log can block query progress via sleep()
BUG-115185 MAPREDUCE-7158 MapReduce Backport MAPREDUCE-7158: Inefficient Flush Logic in JobHistory EventWriter

Potential-data-loss

Bug ID Apache JIRA Apache Component Summary
BUG-117071 HIVE-16907 Hive "INSERT INTO" overwrite old data when destination table encapsulated by backquote

Query-failure

Bug ID Apache JIRA Apache Component Summary
BUG-121291 HIVE-21746 Hive ArrayIndexOutOfBoundsException during dynamically partitioned hash join, with CBO disabled
BUG-120380 N/A HWC executeQuery() blocks when dataframe is cached and in some other scenarios as well
BUG-120161 CALCITE-3052 Hive Error while applying rule MaterializedViewAggregateRule(Project-Aggregate): ArrayIndexOutOfBoundsException
BUG-120159 CALCITE-3066 Hive RelToSqlConverter may incorrectly throw an AssertionError for some decimal literals
BUG-120041 HIVE-21600 Hive GenTezUtils.removeSemiJoinOperator may throw out of bounds exception for TS with multiple children
BUG-120040 HIVE-21613 Hive Queries with join condition having timestamp or timestamp with local time zone literal throw SemanticException
BUG-119962 HIVE-21679 Hive Replicating a CTAS event creating an MM table fails.
BUG-119171 HIVE-21539 Hive GroupBy + where clause on same column results in incorrect query rewrite
BUG-117334 HIVE-21107 Hive2 Backport HIVE-21107 to 3.1-maint - "Cannot find field" error during dynamically partitioned hash join
BUG-117260 SPARK-26379 Spark Use dummy TimeZoneId for CurrentTimestamp to avoid UnresolvedException in CurrentBatchTimestamp

Security

Bug ID Apache JIRA Apache Component Summary
BUG-120559 N/A Accumulo Update Jetty to 9.2.28.v20190418
BUG-120546 N/A Oozie Add Oozie security headers to HDP-3.1.x line
BUG-120506 KNOX-1596, KNOX-1857 Knox Fixes for KNOX-1596 and KNOX-1857
BUG-120369 N/A Oozie Eliminate Jetty from Oozie RPMs
BUG-120252 ZOOKEEPER-1045, ZOOKEEPER-1392, ZOOKEEPER-2498, ZOOKEEPER-2693, ZOOKEEPER-2726 Zookeeper Patch CVEs for Zookeeper 3.4.6
BUG-120210 N/A Spark Upgrade to jackson-databind version 2.9.8
BUG-119572 N/A Knox Upgrade bouncycastle libraries to 1.61 - knox
BUG-119509 ZOOKEEPER-1392 Zookeeper Should not allow to read ACL when not authorized to read node
BUG-119333 HIVE-21573 Hive Binary transport shall ignore principal if auth is set to delegationToken
BUG-118965 HADOOP-15994 Hadoop Common Upgrade Jackson2 to 2.9.8
BUG-118773 N/A Oozie Upgrade to jackson-databind version 2.9.8 for Oozie
BUG-118256 KNOX-1678 Knox Upgrade to jackson-databind version 2.9.8
BUG-118233 N/A Druid Upgrade to jackson-databind version 2.9.8
BUG-118198 YARN-9334 YARN YARN Service Client does not work with SPNEGO when knox is configured
BUG-116998 OOZIE-3370 Oozie Property filtering is not consistent across job submission

Stability

Bug ID Apache JIRA Apache Component Summary
BUG-121463 HIVE-22001 Hive Backport HIVE-22001 to 3.1-maint - ACID: getAcidState() should cache a recursive dir listing locally
BUG-121363 YARN-6695 YARN Race condition in RM for publishing container events vs appFinished events causes NPE
BUG-121355 HADOOP-16376 Hadoop Common ABFS: Override access() to no-op for now
BUG-121297 HADOOP-16461 Hadoop Common FileSystem cache lock parses XML within the lock
BUG-121233 HIVE-18786 Hive NPE in Hive windowing functions
BUG-121142 HIVE-21912 Hive Implement BlacklistingLlapMetricsListener
BUG-121140 HDFS-12748 HDFS NameNode memory leak when accessing webhdfs GETHOMEDIRECTORY
BUG-121076 HIVE-21815 Hive Stats in ORC file are parsed twice
BUG-121058 HIVE-21992 Hive REPL DUMP throws NPE when dumping Create Function event.
BUG-120969 HIVE-21966 Hive Llap external client - Arrow Serializer throws ArrayIndexOutOfBoundsException in some cases
BUG-120952 HIVE-21963 Hive TransactionalValidationListener.validateTableStructure should check the partition directories in the case of partitioned tables
BUG-120912 HADOOP-16350 Hadoop Common Ability to tell HDFS client not to request KMS Information from NameNode
BUG-120873 HIVE-21232, HIVE-21332,HIVE-21509,HIVE-21686,HIVE-21768,HIVE-21805 Hive Backport of HIVE-21805, HIVE-21332, HIVE-21686, HIVE-21509. Revert HIVE-21232
BUG-120841 N/A HWC close() method at session level to clean up resources
BUG-120784 ZOOKEEPER-2247 Zookeeper Zookeeper quorum goes down sporadically
BUG-120740 HIVE-21864 Hive LlapBaseInputFormat#closeAll() throws ConcurrentModificationException
BUG-120719 HDFS-14323 HDFS Distcp fails in Hadoop 3.x when 2.x source webhdfs url has special characters in hdfs file path
BUG-120689 N/A HWC Add a clean up method when spark application ends
BUG-120576 YARN-9205 YARN Backport to "HDP-3.1-maint" - When using custom resource type, application will fail to run due to the CapacityScheduler throws InvalidResourceRequestException(GREATER_THEN_MAX_ALLOCATION)
BUG-120207 CALCITE-3189, HIVE-21976 Hive Hive JDBC Storage Handler (Oracle): TPCDS queries failing with 'ORA-00907: missing right parenthesis'
BUG-120206 CALCITE-3189, HIVE-22007 Hive Hive JDBC Storage Handler (Oracle): TPCDS queries failing with 'java.sql.SQLSyntaxErrorException: ORA-00904: "TRUE": invalid identifier'
BUG-120204 CALCITE-3189, HIVE-22007 Hive Hive JDBC Storage Handler (Oracle): TPCDS queries failing with "java.sql.SQLSyntaxErrorException: ORA-00905: missing keyword"
BUG-120203 CALCITE-3189 Hive Hive JDBC Storage Handler (Oracle): TPCDS queries failing with "specified length too long for its datatype"
BUG-120017 HIVE-21678 Hive CTAS creating a Full ACID partitioned table fails because of no writeId.
BUG-120007 HIVE-21706 Hive Hive external table incremental repl dump fails if table dropped during replication in-progress
BUG-119988 HIVE-21729 Hive Hive Warehouse Connector is offsetting timestamp column by exactly one second
BUG-119841 HIVE-20274 Hive HiveServer2 ObjectInspectorFactory leaks for Struct and List object inspectors
BUG-119495 HIVE-21430 Hive INSERT into a dynamically partitioned table with hive.stats.autogather = false throws a MetaException
BUG-119391 HIVE-21602 Hive Dropping an external table created by migration case should delete the data directory.
BUG-119144 HIVE-21540 Hive Query with join condition having date literal throws SemanticException.
BUG-119143 HIVE-21564 Hive Load data into a bucketed table is ignoring partitions specs and loads data into default partition.
BUG-119124 HIVE-21497 Hive Direct SQL exception thrown by PartitionManagementTask
BUG-118996 CALCITE-2727 Hive MV rewriting bails out incorrectly when a view does not contain any table reference
BUG-118827 HIVE-21471 Hive Replicating conversion of managed to external table leaks HDFS files at target.
BUG-118682 ZOOKEEPER-1342, ZOOKEEPER-1907, ZOOKEEPER-2029, ZOOKEEPER-2186, ZOOKEEPER-602 Zookeeper Backport Hotfix request for bugs fix for issue where zookeeper node Fails to Re-join Zookeeper Quorum
BUG-118374 YARN-9336 YARN JobHistoryServer leaks CLOSE_WAIT tcp connections when using LogAggregationIndexedFileController
BUG-118335 HIVE-21376 Hive Incompatible change in Hive bucket computation
BUG-117863 HIVE-21385 Hive Hive JDBC Storage Handler (MySQL): Couple of tpcds queries get stuck in running state.
BUG-117703 HIVE-21167 Hive Bucketing: Bucketing version 1 is incorrectly partitioning data
BUG-117634 HIVE-21188 Hive Backport HIVE-21188: Hive query on view fails with SemanticException when there is a Ranger masked policy in place
BUG-117558 HADOOP-16074 Hadoop Common WASB: Update container not found error code
BUG-117555 HADOOP-16048 Hadoop Common ABFS: Fix Date format parser
BUG-117363 HADOOP-16079 Hadoop Common Token.toString faulting if any token listed can't load
BUG-117221 HIVE-21186 Hive External table REPL throwing NPE
BUG-117141 HIVE-21156 Hive SharedWorkOptimizer may preserve filter in TS incorrectly
BUG-117140 HIVE-21006 Hive Extend SharedWorkOptimizer to remove semijoins when there is a reutilization opportunity
BUG-116767 HIVE-21255 Hive Hive JDBC Storage Handler - ArrayIndexOutOfBoundsException
BUG-116760 HADOOP-16040 Hadoop Common ABFS: Bug fix for tolerateOobAppends configuration
BUG-116686 HIVE-21253 Hive Hive JDBC Storage Handler (DB2): TPCDS queries failing with CNF exception
BUG-116684 HIVE-16100 Hive2 Backport HIVE-16100 to 3.1-maint - Dynamic Sorted Partition optimizer loses sibling operators
BUG-116670 HBASE-20647 HBase HBase is stuck in initializing master state
BUG-115948 ORC-397, ORC-435 Hive Backport ORC-397 and ORC-435 to 3.1-maint
BUG-115867 HADOOP-15808 Hadoop Common Harden Token service loader use
BUG-115543 HADOOP-15975 Hadoop Common ABFS: remove timeout check for DELETE and RENAME
BUG-115480 HADOOP-15968 Hadoop Common ABFS: add try catch for UGI failure when initializing ABFS
BUG-114163 HIVE-20839 Hive2 "Cannot find field" error during dynamically partitioned hash join
BUG-112704 HADOOP-15434 Hadoop Common Backport HADOOP-15434: Upgrade to ADLS SDK that exposes current timeout
BUG-112703 HADOOP-15356 Hadoop Common Backport HADOOP-15356: Make HTTP timeout configurable in ADLS Connector

Supportability

Bug ID Apache JIRA Apache Component Summary
BUG-120079 HIVE-21061, HIVE-21398, HIVE-21499 Hive Backport Multiple Hive fixes
BUG-120035 KNOX-1827, KNOX-1828 Knox Backport Knox fixes to HDP-3.1-maint
BUG-119380 CALCITE-2858 Hive Improvements in JSON writer and reader for plans
BUG-119022 HIVE-21507 Hive Hive swallows NPE if no delegation token found
BUG-118997 CALCITE-2943 Hive Materialized view rewriting logic calls getApplicableMaterializations each time the rule is triggered
BUG-118995 CALCITE-2942 Hive Materialized view rewriting logic instantiates RelMetadataQuery each time the rule is triggered
BUG-118827 HIVE-21471 Hive Replicating conversion of managed to external table leaks HDFS files at target.
BUG-118682 ZOOKEEPER-1342, ZOOKEEPER-1907, ZOOKEEPER-2029, ZOOKEEPER-2186, ZOOKEEPER-602 Zookeeper Backport Hotfix request for bugs fix for issue where zookeeper node Fails to Re-join Zookeeper Quorum
BUG-117872 SQOOP-3420 Sqoop Invalid ERROR message initiates false alarms(Could not rename /tmp/sqoop-root/compile/xx.java to /root/./xx.java. Error: Destination '/root/./xx.java' already exists)
BUG-117557 HADOOP-16044 Hadoop Common ABFS: Better exception handling of DNS errors followup
BUG-117556 HADOOP-15662 Hadoop Common ABFS: Better exception handling of DNS errors
BUG-117118 HIVE-20822, HIVE-21046 Hive Improvements to push computation to JDBC from Calcite
BUG-116171 HADOOP-16004 Hadoop Common ABFS: Convert 404 error response in AbfsInputStream and AbfsOutPutStream to FileNotFoundException
BUG-115965 HIVE-19711 Hive Refactor HiveSchemaTool
BUG-115918 HIVE-21032 Hive Refactor HiveMetaTool

Usability

Bug ID Apache JIRA Apache Component Summary
BUG-121380 HIVE-15177 Hive Authentication with hive fails when kerberos auth type is set to fromSubject and principal contains _HOST
BUG-121351 HIVE-18735HIVE-21957 Hive Backport HIVE-18735 & HIVE-21957
BUG-121180 HIVE-19113 Hive Bucketing: Make CLUSTERED BY do CLUSTER BY if no explicit sorting is specified
BUG-121092 HIVE-21822,HIVE-21823,HIVE-21832,HIVE-21846 Hive Get the LlapDaemon metrics from TezAM
BUG-121077 HIVE-21225 Hive ACID: getAcidState() should cache a recursive dir listing locally
BUG-121075 HIVE-21863 Hive Vectorization: CASE result type casting (non-cbo path)
BUG-121074 HIVE-21742 Hive Vectorization: CASE result type casting
BUG-121043 HIVE-21958 Hive The list of table expression in the inclusion and exclusion list should be separated by '|' instead of comma.
BUG-121016 HIVE-21886 Hive Handle rename events during replace policy in table level replicatiion
BUG-120940 N/A HWC HWC: onApplicationEnd() should not cause spark application to fail
BUG-120832 HIVE-21913 Hive Backport HIVE-21913: GenericUDTFGetSplits should handle usernames in the same way as LLAP
BUG-120807 N/A Hive Backport HIVE-20707 and related patches to 3.1-maint
BUG-120777 HIVE-21837 Hive Hive exception "VoidColumnVector is not supported" for certain queries
BUG-120604 HIVE-21938HIVE-21948 Hive Hive 3 upgrade issues with large number of tables
BUG-120384 HIVE-21754 Hive Metastore schema different precision can cause comparision issues
BUG-120374 HIVE-21794 Hive Add materialized view parameters to sqlStdAuthSafeVarNameRegexes
BUG-120195 HIVE-21681 Hive Describe formatted shows incorrect information for multiple primary keys
BUG-120158 KNOX-1577 Knox Knox automatically derived dispatch whitelist doesn't seem to actually match the knox domain
BUG-120044 HIVE-21645 Hive Include CBO json plan in explain formatted
BUG-120043 HIVE-21631 Hive Enhance metastore API to allow bulk-loading materialized views
BUG-119955 OOZIE-3194 Oozie Permission issue- r+x for others under hadoop fs -ls /user/oozie/share/lib/lib_2017XXX
BUG-119906 HIVE-21538 Hive Beeline: password source though the console reader did not pass to connection param
BUG-119619 HIVE-20462 Hive "CREATE VIEW IF NOT EXISTS" fails if view already exists
BUG-119491 HIVE-20555 Hive HiveServer2: Preauthenticated subject for http transport is not retained for entire duration of http communication in some cases
BUG-119480 HIVE-20550 Hive Switch WebHCat to use beeline to submit Hive queries
BUG-119461 HIVE-21592 Hive OptimizedSql is not shown when the expression contains CONCAT
BUG-119414 HIVE-21591 Hive Backport HIVE-21591: Using triggers in non-LLAP mode should not require wm queue
BUG-119388 SQOOP-3134 Sqoop --class-name should override default Avro schema name
BUG-119375 HIVE-21587 Hive Explain formatted CBO should write row type in JSON
BUG-119257 HADOOP-15997, HADOOP-16199 Hadoop Common KMS client uses wrong UGI after HADOOP-14445 (3.1-maint)
BUG-119191 HIVE-21290 Hive Restore historical way of handling timestamps in Parquet while keeping the new semantics at the same time
BUG-119152 ZOOKEEPER-2775, ZOOKEEPER-3253 Zookeeper Zookeeper server will not rejoin quorum/ Zookeeper Lost Lock
BUG-119143 HIVE-21564 Hive Load data into a bucketed table is ignoring partitions specs and loads data into default partition.
BUG-119067 N/A Hive HWC: For SaveMode overwrite, introduce truncate option
BUG-119056 HIVE-21254 Hive Backport HIVE-21254: Pre-upgrade tool should handle exceptions and skip db/tables
BUG-118961 N/A Sqoop Modify Sqoop to use Beeline instead of Hive command
BUG-118564 HADOOP-16174 Hadoop Common Disable wildfly logs to the console
BUG-118491 OOZIE-3439 Oozie Hive2 action is not parsing application ID for TEZ from log file properly
BUG-118405 OOZIE-3440 Oozie Oozie Spark action unable to parse # character
BUG-118374 YARN-9336 YARN [STAMPY] JobHistoryServer leaks CLOSE_WAIT tcp connections when using LogAggregationIndexedFileController
BUG-118340 HIVE-21385 Hive Allow disabling pushdown of non-splittable computation to JDBC sources
BUG-118321 HIVE-21389 Hive Hive: Unable to run queries due to "Error getting policies for serviceName=cl1_hiveresponse=null"
BUG-118283 HIVE-21184 Hive Add explain and explain formatted CBO plan with cost information
BUG-118281 HIVE-18920 Hive CBO: Initialize the Janino providers ahead of 1st query
BUG-118119 HIVE-21301 Hive "show tables" command does not include views
BUG-117974 HIVE-21307 Hive Need to set GzipJSONMessageEncoder as default config for EVENT_MESSAGE_FACTORY.
BUG-117860 HIVE-21529 Hive Hive support bootstrap of ACID/MM tables on an existing policy.
BUG-117809 N/A Hive Backport HIVE-20679 to prevent large RDBMS footprint
BUG-117737 HIVE-21388 Hive Constant UDF is not pushed to JDBCStorage Handler
BUG-117715 HIVE-21261 Hive Incremental replication adds redundant COPY and MOVE tasks for external table events.
BUG-117379 HADOOP-15860 Hadoop Common ABFS: Throw IllegalArgumentException when Directory/File name ends with a period(.)
BUG-117378 HADOOP-15954 Hadoop Common ABFS: Enable owner and group conversion for MSI and login user using OAuth
BUG-117361 HDFS-13868 Hadoop Common WebHDFS: GETSNAPSHOTDIFF API NPE when param "snapshotname" is given but "oldsnapshotname" is not.
BUG-117267 SQOOP-3438 Sqoop Sqoop Import with create hcatalog table for ORC will not work in HDP 3.x as the table created would be a ACID table and transactional
BUG-116990 HIVE-21018 Hive Backport HIVE-21018 to 3.1-maint - Grouping sets size cannot be greater than 64 in HDP 3
BUG-116348 TEZ-3957 Tez Report TASK_DURATION_MILLIS as a Counter for completed tasks
BUG-115988 HIVE-16957 Hive Support CTAS for auto gather column stats
BUG-115980 HIVE-21039 Hive CURRENT_TIMESTAMP returns value in UTC time zone
BUG-115603 SQOOP-3423 Sqoop Sqoop support for HiveServer2 password based authentication
BUG-115481 HADOOP-15969 Hadoop Common ABFS: getNamespaceEnabled can fail blocking user access thru ACLs
BUG-115311 HIVE-20989 Hive JDBC - The GetOperationStatus + log can block query progress via sleep()
BUG-115298 HADOOP-15957 Hadoop Common WASB: Add asterisk wildcard support for PageBlobDirSet
BUG-114163 HIVE-20839 Hive2 "Cannot find field" error during dynamically partitioned hash join
BUG-113460 TEZ-4014 Hive EDWS: Read AM plugin descriptors from JSON
BUG-113341 HIVE-20967 Hive Handle alter events when replicate from HDP 2.6.5 to HDP 3.0
BUG-113252 N/A Hive HWC: Detect client-mode vs. cluster-mode for Kerberized cluster usage
BUG-109059 N/A Zeppelin In yarn cluster mode, zeppelin does not allow specifying additional artifacts to be uploaded via distributed cache
BUG-96263 ZEPPELIN-3340 Zeppelin Streaming support in built-in visualizations