HDP 3.1.4 Release Notes
Also available as:
PDF

HBase

This release provides HBase 2.0.2 and the listed Apache patches.

  • HBASE-22690: Deprecate / Remove OfflineMetaRepair in hbase-2+

  • HBASE-19893: restore_snapshot is broken in master branch when region splits

  • HBASE-21136: NPE in MetricsTableSourceImpl.updateFlushTime

  • HBASE-21284:Default limits for PressureAwareCompactionThroughputController are too low

  • HBASE-17525: Add space quota support to the deprecated SecureBulkLoadEndpoint

  • HBASE-15728: Add Per-Table metrics back

  • HBASE-21890: Use execute instead of submit to submit a task in RemoteProcedureDispatcher

  • HBASE-20662: Increasing space quota on a violated table does not remove SpaceViolationPolicy.DISABLE enforcement

  • HBASE-21634: Print error message when user uses unacceptable values for LIMIT while setting quotas.

  • HBASE-21225: Having RPC quota on a table/Namespace doesn't allow space quota to be removed using 'NONE'

  • HBASE-21631:list_quotas should print human readable values for LIMIT

  • HBASE-21988: Remove hadoop2.6.1-hadoop-2.6.4 as supported on branch-2.0

  • HBASE-21915: Make FileLinkInputStream implement CanUnbuffer

  • HBASE-22200: WALSplitter.hasRecoveredEdits should use same FS instance from WAL region dir

  • HBASE-21688: Address WAL filesystem issues

  • HBASE-22144:Correct MultiRowRangeFilter to work with reverse scans

  • HBASE-21734: Some optimization in FilterListWithOR

  • HBASE-21620: Problem in scan query when using more than one column prefix filter in some cases

  • HBASE-21129: Clean up duplicate codes in #equals and #hashCode methods of Filter

  • HBASE-19008: Add missing equals or hashCode method(s) to stock Filter implementations

  • HBASE-22185: RAMQueueEntry#writeToCache should freeBlock if any exception encountered instead of the IOException catch block

  • HBASE-21932: Use Runtime.getRuntime().halt to terminate regionserver when abort timeout

  • HBASE-21775: The BufferedMutator doesn't ever refresh region location cache

  • HBASE-21644: Modify table procedure runs infinitely for a table having region replication > 1

  • HBASE-21621: Reversed scan does not return expected number of rows

  • HBASE-21618: Scan with the same startRow(inclusive=true) and stopRow(inclusive=false) returns one result

  • HBASE-21402: Backport parent "HBASE-21325 Force to terminate regionserver when abort hang in somewhere"

  • HBASE-22123: REST gateway reports Insufficient permissions exceptions as 404 Not Found

  • HBASE-21732: Should call toUpperCase before using Enum.valueOf in some methods for ColumnFamilyDescriptor

  • HBASE-21683: Reset readsEnabled flag after successfully flushing the primary region

  • HBASE-21645: Perform sanity check and disallow table creation/modification with region replication < 1

  • HBASE-21413: Empty meta log doesn't get split when restart whole cluster

  • HBASE-20220: [RSGroup] Check if table exists in the cluster before moving it to the specified regionserver group

  • HBASE-18484: VerifyRep by snapshot does not work when Yarn/SourceHBase/PeerHBase located in three different HDFS clusters

  • HBASE-21738: Remove all the CSLM#size operation in our memstore because it's an quite time consuming.

  • HBASE-22128: Move namespace region then master crashed make deadlock

  • HBASE-21551: Memory leak when use scan with STREAM at server side

  • HBASE-20604: ProtobufLogReader#readNext can incorrectly loop to the same position in the stream until the the WAL is rolled

  • HBASE-21960: Ensure RESTServletContainer used by RESTServer

  • HBASE-22077: Expose sleep time as a command line argument of IntergationTestBackupRestore

  • HBASE-20734: Colocate recovered edits directory with hbase.wal.dir

  • HBASE-21568: Disable use of BlockCache for LoadIncrementalHFiles

  • HBASE-21523: Avoid extra logging when the backup system table already exists

  • HBASE-21524: Fix logging in ConnectionImplementation.isTableAvailable()

  • HBASE-21466: WALProcedureStore uses wrong FileSystem if wal.dir is not under rootdir

  • HBASE-21457: BackupUtils#getWALFilesOlderThan refers to wrong FileSystem

HDP 3.1.0 provided HBase 2.0.2 and the following Apache patches:

  • HBASE-20723: Custom hbase.wal.dir results in data loss because we write recovered edits into a different place than where the recovering region server looks for them

  • HBASE-20722: Make RegionServerTracker only depend on children changed event

  • HBASE-20668: Avoid permission change if ExportSnapshot's copy fails

  • HBASE-20657: Retrying RPC call for ModifyTableProcedure may get stuck

  • HBASE-20635: Add missing shaded-pb conversion method for UserPermission

  • HBASE-20627: Relocate RS Group pre/post hooks from RSGroupAdminServer to RSGroupAdminEndpoint

  • HBASE-20579: Include original exception in wrapped exception

  • HBASE-20579: Improve snapshot manifest copy in ExportSnapshot

  • HBASE-20547: Restore from backup will fail if done from a different file system

  • HBASE-20531: RS may throw NPE when close meta regions in shutdown procedure.

  • HBASE-20495: REST unit test fails with NoClassDefFoundError against hadoop3

  • HBASE-20196: Maintain all regions with same size in memstore flusher

  • HBASE-20136: TestKeyValue misses ClassRule and Category annotations

  • HBASE-19735: Create a client-tarball assembly

  • HBASE-19478: Utilize multi-get to speed up WAL file checking in BackupLogCleaner (Toshihiro Suzuki)

  • HBASE-18135: Implement mechanism for RegionServers to report file archival for space quotas

  • HBASE-18133: Decrease quota reaction latency by HBase

  • HBASE-17549: HBase-Spark Module: Corrected - Incorrect log at println and unwanted comment code

  • HBASE-21440: Assign procedure on the crashed server is not properly interrupted

  • HBASE-21344: Fix TestMetaShutdownHandler (addendum)

  • HBASE-21384: Procedure with holdlock=false should not be restored lock when restarts

  • HBASE-21344: hbase:meta location in ZooKeeper set to OPENING by the procedure which eventually failed but precludes Master from assigning it forever

  • HBASE-21215: Figure how to invoke hbck2; make it easy to find

  • HBASE-21372: Set hbase.assignment.maximum.attempts to Long.MAX

  • HBASE-21363: Rewrite the buildingHoldCleanupTracker method in WALProcedureStore

  • HBASE-21364: Procedure holds the lock should put to front of the queue after restart

  • HBASE-21342: FileSystem in use may get closed by other bulk load call in secure bulkLoad

  • HBASE-21349: Do not run CatalogJanitor or Nomalizer when cluster is shutting down

  • HBASE-21338: Warn if balancer is an ill-fit for cluster size

  • HBASE-21073: Redo concept of maintenance mode

  • HBASE-20937: ArrayIndexOutOfBoundsException when rolling back procedure

  • HBASE-21321: Backport HBASE-21278 to branch-2.1 and branch-2.0 ("Do not rollback successful sub procedures when rolling back a procedure")

  • HBASE-21354: Procedure may be deleted improperly during master restarts resulting in 'Corrupt'(addendum)

  • HBASE-21356: bulkLoadHFile API should ensure that rs has the source hfile's write permissionls

  • HBASE-21336: Addendum remove unused code in HBTU

  • HBASE-21355: (addendum) replace the expensive reload storefiles with reading the merge result of compacted storefiles and current storefiles

  • HBASE-21336: Simplify the implementation of WALProcedureMap

  • HBASE-21355: HStore's storeSize is calculated repeatedly which causing the confusing region split

  • HBASE-21348: Fix failing TestRegionBypass, broke by HBASE-21291

  • HBASE-21345: [hbck2] Allow version check to proceed even though master is 'initializing'.

  • HBASE-21200: Memstore flush doesn't finish because of seekToPreviousRow() in memstore scanner.

  • HBASE-21323: Should not skip force updating for a sub procedure even if it has been finished

  • HBASE-21075: Confirm that we can (rolling) upgrade from 2.0.x and 2.1.x to 2.2.x after HBASE-20881

  • HBASE-21292: IdLock.getLockEntry() may hang if interrupted

  • HBASE-21291: Add a test for bypassing stuck state-machine procedures

  • HBASE-21323: Should not skip force updating for a sub procedure even if it has been finished; ADDENDUM

  • HBASE-21288: HostingServer in UnassignProcedure is not accurate

  • HBASE-21323: Should not skip force updating for a sub procedure even if it has been finished

  • HBASE-21055: NullPointerException when balanceOverall() but server balance info is null

  • HBASE-21198: Exclude dependency on net.minidev:json-smart

  • HBASE-21310 and HBASE-21311: Addendum fix failed UTs, some UTs are not present on branch-2.1 and some are a bit different in the implementation

  • HBASE-21327: Fix minor logging issue where we don't report servername if no associated SCP

  • HBASE-12320: [canary] Cleanup of usage and add commentary

  • HBASE-21311: Split TestRestoreSnapshotFromClient

  • HBASE-21263: Mention compression algorithm along with other storefile details

  • HBASE-21242: [amv2] Miscellaneous minor log and assign procedure create improvements; ADDENDUM Fix TestHRegionInfo AND TestRegionInfoDisplay

  • HBASE-21315: The getActiveMinProcId and getActiveMaxProcId of BitSetNode are incorrect if there are no active procedure

  • HBASE-21310: Split TestCloneSnapshotFromClient

  • HBASE-21266: Not running balancer because processing dead regionservers, but empty dead rs list

  • HBASE-21290: No need to instantiate BlockCache for master which not carry table

  • HBASE-21260: The whole balancer plans might be aborted if there are more than one plans to move a same region

  • HBASE-21309: Increase the waiting timeout for TestProcedurePriority

  • HBASE-21271: [amv2] Don't throw UnsupportedOperationException when rollback called on Assign/Unassign; spiral of death

  • HBASE-21251: Refactor RegionMover

  • HBASE-21259: [amv2] Revived deadservers; recreated serverstatenode

  • HBASE-21303: [shell] clear_deadservers with no args fails

  • HBASE-21289: Remove the log "'hbase.regionserver.maxlogs' was deprecated." in AbstractFSWAL

  • HBASE-21242: [amv2] Miscellaneous minor log and assign procedure create improvements

  • HBASE-21254: Need to find a way to limit the number of proc wal files

  • HBASE-21103: nightly job should make sure cached yetus will run.

  • HBASE-21287: Allow configuring test master initialization wait time.

  • HBASE-20764: build broken when latest commit is gpg signed

  • HBASE-21271: [amv2] Don't throw UnsupportedOperationException when rollback called on Assign/Unassign; spiral of death

  • HBASE-21250: Addendum remove unused modification in hbase-server module

  • HBASE-21250: Refactor WALProcedureStore and add more comments for better understanding the implementation

  • HBASE-21223: [amv2] Remove abort_procedure from shell

  • HBASE-20506: Add doc and test for unused RetryCounter, useful-looking utility

  • HBASE-21185: - WALPrettyPrinter: Additional useful info to be printed by wal printer tool, for debugability purposes

  • HBASE-21281: Upgrade bouncycastle to latest - addendum adds test dependency

  • HBASE-21178: [BC break] : Get and Scan operation with a custom converter_class not working

  • HBASE-21282: Upgrade to lateste jetty 9.2 and 9.3 versions

  • HBASE-21281: Upgrade bouncycastle to latest

  • HBASE-21261: Add log4j.properties for hbase-rsgroup tests

  • HBASE-19275: TestSnapshotFileCache never worked properly

  • HBASE-21196: HTableMultiplexer clears the meta cache after every put operation

  • HBASE-21249: Add jitter for ProcedureUtil.getBackoffTimeMs

  • HBASE-21233: Allow the procedure implementation to skip persistence of the state after a execution

  • HBASE-21237: Use CompatRemoteProcedureResolver to dispatch open/close region requests to RS

  • HBASE-21228: Memory leak since AbstractFSWAL caches Thread object and never clean later

  • Revert "HBASE-20657 Retrying RPC call for ModifyTableProcedure may get stuck"

  • HBASE-21212: Wrong flush time when update flush metric

  • HBASE-21208: Bytes#toShort doesn't work without unsafe

  • Amend HBASE-20704: Sometimes some compacted storefiles are not archived on region close

  • HBASE-21203: TestZKMainServer#testCommandLineWorks won't pass with default 4lw whitelist

  • HBASE-21206: Scan with batch size may return incomplete cells

  • HBASE-20704: Sometimes some compacted storefiles are not archived on region close

  • HBASE-21097: Flush pressure assertion may fail in testFlushThroughputTuning

  • HBASE-21189: flaky job should gather machine stats

  • HBASE-21190: Log files and count of entries in each as we load from the MasterProcWAL store

  • HBASE-21171: Reimplement the retry backoff logic for ReopenTableRegionsProcedure

  • HBASE-21179: Fix the number of actions in responseTooSlow log

  • HBASE-21174: [REST] Failed to parse empty qualifier in TableResource#getScanResource

  • HBASE-21181: Use the same filesystem for wal archive directory and wal directory

  • HBASE-21158: Empty qualifier cell is always returned when using QualifierFilter

  • HBASE-21144: AssignmentManager.waitForAssignment is not stable

  • HBASE-21143: Update findbugs-maven-plugin to 3.0.4

  • HBASE-21001: ReplicationObserver fails to load in HBase 2.0.0

  • HBASE-20307: LoadTestTool prints too much zookeeper logging (Colin Garcia)

  • HBASE-21157: Split TableInputFormatScan to individual tests

  • HBASE-21127: TableRecordReader need to handle cursor result too

  • HBASE-20892: [UI] Start / End keys are empty on table.jsp

  • HBASE-21132: return wrong result in rest multiget

HDP 3.0.1 provided HBase HBase 2.0.0 and the following Apache patches:

  • HBASE-20723: Custom hbase.wal.dir results in data loss because we write recovered edits into a different place than where the recovering region server looks for them

  • HBASE-20722: Make RegionServerTracker only depend on children changed event

  • HBASE-20668: Avoid permission change if ExportSnapshot's copy fails

  • HBASE-20657: Retrying RPC call for ModifyTableProcedure may get stuck

  • HBASE-20635: Add missing shaded-pb conversion method for UserPermission

  • HBASE-20627: Relocate RS Group pre/post hooks from RSGroupAdminServer to RSGroupAdminEndpoint

  • HBASE-20579: Include original exception in wrapped exception

  • HBASE-20579: Improve snapshot manifest copy in ExportSnapshot

  • HBASE-20547: Restore from backup will fail if done from a different file system

  • HBASE-20531: RS may throw NPE when close meta regions in shutdown procedure.

  • HBASE-20495: REST unit test fails with NoClassDefFoundError against hadoop3

  • HBASE-20196: Maintain all regions with same size in memstore flusher

  • HBASE-20136: TestKeyValue misses ClassRule and Category annotations

  • HBASE-19735: Create a client-tarball assembly

  • HBASE-19478: Utilize multi-get to speed up WAL file checking in BackupLogCleaner (Toshihiro Suzuki)

  • HBASE-18135: Implement mechanism for RegionServers to report file archival for space quotas

  • HBASE-18133: Decrease quota reaction latency by HBase

  • HBASE-17549: HBase-Spark Module: Corrected - Incorrect log at println and unwanted comment cod