HBase

This release provides HBase 1.1.2 with no additional Apache patches.

• HBASE-13376: Improvements to Stochastic load balancer.

• HBASE-13716: Stop using Hadoop's FSConstants.

• HBASE-13848: Access InfoServer SSL passwords through Credential Provder API.

• HBASE-13947: Use MasterServices instead of Server in AssignmentManager.

• HBASE-14135: HBase Backup/Restore Phase 3: Merge backup images.

• HBASE-14473: Compute region locality in parallel.

• HBASE-14517: Show regionserver's version in master status page.

• HBASE-14606: TestSecureLoadIncrementalHFiles tests timed out in trunk build on apache.

• HBASE-15210: Undo aggressive load balancer logging at tens of lines per millisecond.

• HBASE-15515: Improve LocalityBasedCandidateGenerator in Balancer.

• HBASE-15615: Wrong sleep time when RegionServerCallable need retry.

• HBASE-16135: PeerClusterZnode under rs of removed peer may never be deleted.

• HBASE-16570: Compute region locality in parallel at startup.

• HBASE-16810: HBase Balancer throws ArrayIndexOutOfBoundsException when regionservers are in /hbase/draining znode and unloaded.

• HBASE-16852: TestDefaultCompactSelection failed on branch-1.3.

• HBASE-17387: Reduce the overhead of exception report in RegionActionResult for multi().

• HBASE-17850: Backup system repair utility.

• HBASE-17931: Assign system tables to servers with highest version.

• HBASE-18083: Make large/small file clean thread number configurable in HFileCleaner.

• HBASE-18084: Improve CleanerChore to clean from directory which consumes more disk space.

• HBASE-18164: Much faster locality cost function and candidate generator.

• HBASE-18212: In Standalone mode with local filesystem HBase logs Warning message:Failed to invoke 'unbuffer' method in class class org.apache.hadoop.fs.FSDataInputStream.

• HBASE-18808: Ineffective config check in BackupLogCleaner#getDeletableFiles().

• HBASE-19052: FixedFileTrailer should recognize CellComparatorImpl class in branch-1.x.

• HBASE-19065: HRegion#bulkLoadHFiles() should wait for concurrent Region#flush() to finish.

• HBASE-19285: Add per-table latency histograms.

• HBASE-19393: HTTP 413 FULL head while accessing HBase UI using SSL. .

• HBASE-19395: [branch-1] TestEndToEndSplitTransaction.testMasterOpsWhileSplitting fails with NPE.

• HBASE-19421: branch-1 does not compile against Hadoop 3.0.0.

• HBASE-19934: HBaseSnapshotException when read replicas is enabled and online snapshot is taken after region splitting.

• HBASE-20008: [backport] NullPointerException when restoring a snapshot after splitting a region.

HDP 2.6.4 provided HBase 1.1.2 with no additional Apache patches.

HDP 2.6.3 provided HBase 1.1.2 and the following Apache patches:

• HBASE-9393: Hbase does not closing a closed socket resulting in many CLOSE_WAIT.

• HBASE-11013: Clone Snapshots on Secure Cluster Should provide option to apply Retained User Permissions.

• HBASE-14469: Fix some comment, validation and logging around memstore lower limit configuration.

• HBASE-14584: TestNamespacesInstanceModel fails on jdk8.

• HBASE-14644: Region in transition metric is broken (Huaxiang Sun).

• HBASE-14663: HStore::close does not honor config hbase.rs.evictblocksonclose.

• HBASE-15160: Put back HFile's HDFS op latency sampling code and add metrics for monitoring (Yu Li and Enis Soztutar) HBASE-18184 Add hbase-hadoop2-compat jar as MapReduce dependency.

• HBASE-15198: RPC client not using Codec and CellBlock for puts by default.

• HBASE-15203: Reduce garbage created by path.toString() during Checksum verification.

• HBASE-15214: Valid mutate Ops fail with RPC Codec in use and region moves across.

• HBASE-15433: SnapshotManager#restoreSnapshot not update table and region count quota correctly when encountering exception (Jianwei Cui).

• HBASE-16700: Allow for coprocessor whitelisting.

• HBASE-16886: hbase-client: scanner with reversed=true and small=true gets no result.

• HBASE-17002: JMX metrics and some UI additions for space quotas.

• HBASE-17003: Documentation updates for space quotas.

• HBASE-17127: Locate region should fail fast if underlying Connection already closed.

• HBASE-17215: Separate small/large file delete threads in HFileCleaner to accelerate archived hfile cleanup speed.

• HBASE-17290: Potential loss of data for replication of bulk loaded hfiles.

• HBASE-17557: HRegionServer#reportRegionSizesForQuotas() should respond to UnsupportedOperationException.

• HBASE-17602: Reduce some quota chore periods/delays.

• HBASE-17612: [C++] Set client version info in RPC header.

• HBASE-17794: Swap "violation" for "snapshot" where appropriate.

• HBASE-17854: Use StealJobQueue in HFileCleaner.

• HBASE-17955: Various reviewboard improvements to space quota work.

• HBASE-17977: Enable the MasterSpaceQuotaObserver by default.

• HBASE-17978: Ensure superusers can circumvent actions restricted by space quota violations.

• HBASE-17981: Consolidate the space quota shell commands.

• HBASE-18024: HRegion#initializeRegionInternals should not re-create .hregioninfo file when the region directory no longer exists.

• HBASE-18061: [C++] Fix retry logic in multi-get calls (Sudeep Sunthankar).

• HBASE-18078: [C++] Harden RPC by handling various communication abnormalities.

• HBASE-18086: Create native client which creates load on selected cluster.

• HBASE-18093: Overloading the meaning of 'enabled' in Quota Manager to indicate either quota disabled or quota manager not ready is not good (Stephen Yuan Jiang).

• HBASE-18150: Do not call FSUtils.setVersion() and FSUtils.checkVersion() when using checkRootDir() to check hbase.wal.dir.

• HBASE-18168: NoSuchElementException when rolling the log (Allan Yang).

• HBASE-18204: [C++] Rpc connection close and reconnecting.

• HBASE-18338: [C++] Implement RpcTestServer (Xiaobing Zhou).

• HBASE-18371: [C++] Update folly and wangle dependencies.

• HBASE-18400: [C++] ConnectionId Equals/Hash should consider service_name (Xiaobing Zhou).

• HBASE-18407: [C++] make Configuration::Set/GetBool work for both true/false and 1/0 (Xiaobing Zhou).

• HBASE-18459: [C++] Fix Segfault in location-cache.

• HBASE-18466: [C++] Support handling exception in RpcTestServer.

• HBASE-18471: The DeleteFamily cell is skipped when StoreScanner seeks to next column.

• HBASE-18507: [C++] Support for MultiPuts in AsyncBatchRpcRetryingCaller class (Sudeep Sunthankar).

• HBASE-18536: [C++] Add fault injection infra.

• HBASE-18537: [C++] Improvements to load-client.

• HBASE-18564: [C++] Problems compiling with GCC.

• HBASE-18565: [C++] Fix deadlock in AsyncScanRetryingCaller and other RPCs.

• HBASE-18568: Correct metric of numRegions.

• HBASE-18576: [C++] Add ping for RPC test.

• HBASE-18578: [C++] Add pause for RPC test.

• HBASE-18579: Enable core dump by default for docker.

• HBASE-18603: buck complains about Build target path containing double slash.

• HBASE-18614: Setting BUCKET_CACHE_COMBINED_KEY to false disables stats on RS UI.

• HBASE-18629: Enhance ChaosMonkeyRunner with interruptibility.

• HBASE-18631: Allow ChaosMonkey properties to be specified in hbase-site.

• HBASE-18646: LogRollMasterProcedureManager: make procedure timeout, thread pool size configurable.

• HBASE-18665: ReversedScannerCallable invokes getRegionLocations incorrectly.

• HBASE-18679: Add a null check around the result of getCounters() in ITBLL.

• HBASE-18720: [C++] Update wangle dependency to pick up the new release with Apache License v2.

• HBASE-18725: [C++] Install header files as well as library.

• HBASE-18727: [C++] Don't link JNI/JVM stuff into libHbaseClient.

• HBASE-18728: [C++] The library should be named libHBaseClient, instead of libHbaseClient.

• HBASE-18743: HFiles in use by a table which has the same name and namespace with a default table cloned from snapshot may be deleted when that snapshot and default table are deleted (wenbang).

• HBASE-18757: Fix Improper bitwise & in BucketCache offset calculation.

• HBASE-18777: Create one table for subtests in multi-retry-test.

• HBASE-18802: [C++] Remove hbase-macros.h.

• HBASE-18891: BUG-89216 Upgrade netty-all dependency.

HDP 2.6.2 provides HBase 1.1.2 and the following Apache patches:

• HBASE-13706: CoprocessorClassLoader should not exempt Hive classes.

• HBASE-14425: In Secure ZooKeeper cluster superuser will not have sufficient permission if multiple values are configured in hbase.superuser.

• HBASE-14492: Increase REST server header buffer size from 8k to 64k.

• HBASE-14734: Prevent BindException when setting up MiniKdc. Port for kdc service gets selected in the constructor, but we bind to it later in MiniKdc.start()-->MiniKdc.initKDCServer() --> KdcServer.start(). In meantime, some other service can capture the port which results in BindException. The solution here is to catch the exception and retry.

• HBASE-15177: Reduce garbage created under high load.

• HBASE-15645: hbase.rpc.timeout is not used in operations of HTable.

• HBASE-16345: RpcRetryingCallerWithReadReplicas#call() should catch some RegionServer Exceptions.

• HBASE-16420: Fix source incompatibility of Table interface.

• HBASE-16488: Starting namespace and quota services in master startup asynchronizely.

• HBASE-16767: Mob compaction needs to clean up files - addendum removes createMobRefKeyValue().

• HBASE-16767: Mob compaction needs to clean up files in /hbase/mobdir/.tmp.

• HBASE-16812: Clean up the locks in MOB.

• HBASE-16841: Data loss in MOB files after cloning a snapshot and deleting that snapshot.

• HBASE-17005: Improve log message in MobFileCache.

• HBASE-17095: The ClientSimpleScanner keeps retrying if the hfile is corrupt or cannot found.

• HBASE-17151: New API to create HFile.Reader without instantiating block cache.

• HBASE-17241: Avoid compacting already compacted mob files with _del files.

• HBASE-17547: TableCatelog doesn't support multiple columns from Single Column family.

• HBASE-17922: Clean TestRegionServerHostname for hadoop3..

• HBASE-18312: Ineffective handling of FileNotFoundException in FileLink.tryOpen().

• HBASE-18358: Backport HBASE-18099 'FlushSnapshotSubprocedure should wait for concurrent Region#flush() to finish'.

HDP 2.6.1 provided HBase 1.1.2 and the following Apache patches:

• HBASE-17287: Master becomes a zombie if filesystem object closes.

• HBASE-17501: guard against NPE while reading FileTrailer and HFileBlock.

• HBASE-17779: disable_table_replication returns misleading message and does not turn off replication.

• HBASE-17785: RSGroupBasedLoadBalancer fails to assign new table regions when cloning snapshot.

• HBASE-17861: Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3.

• HBASE-18036: Data locality is not maintained after cluster restart or SSH.

• HBASE-18067: Allow default FORMATTER for shell put/get commands.

• HBASE-18075: Support non-latin table names and namespaces.

HDP 2.6.0 provided HBase 1.1.2 and the following Apache patches:

• BUG-45027:/BUG-45361 Fix hostname used with Kerberos on Dual-Homed clusters..

• HBASE-13318: RpcServer.getListenerAddress should handle when the accept channel is closed (Andrew Purtell).

• HBASE-13829: Add more ThrottleType (Guanghao Zhang).

• HBASE-14123: HBase Backup/Restore Phase 2.

• HBASE-14140: HBase Backup Admin API.

• HBASE-14148: Web UI Framable Page.

• HBASE-14172: Upgrade existing thrift binding using thrift 0.9.3 compiler (Josh Elser).

• HBASE-14258: Make region_mover.rb script case insensitive with regard to hostname.

• HBASE-14269: FuzzyRowFilter omits certain rows when multiple fuzzy key exist.

• HBASE-14283: Reverse scan doesn’t work with HFile inline index/bloom blocks.

• HBASE-14302: TableSnapshotInputFormat should not create back references when restoring snapshot.

• HBASE-14307: Incorrect use of positional read api in HFileBlock (Chris Nauroth).

• HBASE-14313: After a Connection sees ConnectionClosingException it never recovers (Elliott Clark).

• HBASE-14407: NotServingRegion: hbase region closed forever (Shuaifeng Zhou).

• HBASE-14449: Rewrite deadlock prevention for concurrent connection close.

• HBASE-14535: Integration test for rpc connection concurrency / deadlock testing (Enis Soztutar).

• HBASE-14536: Balancer & SSH interfering with each other leading to unavailability.

• HBASE-14598: ByteBufferOutputStream grows its HeapByteBuffer beyond JVM limitations (Ian Friedman).

• HBASE-14635: Fix flaky test TestSnapshotCloneIndependence.

• HBASE-14655: Narrow the scope of doAs() calls to region observer notifications for compaction.

• HBASE-14667: HBaseFsck constructors have diverged (Matteo).

• HBASE-14680: Two configs for snapshot timeout and better defaults.

• HBASE-14712: Increase MasterProcWALs clean up granularity (Matteo Bertozzi).

• HBASE-14805: status should show the master in shell (Enis).

• HBASE-14818: user_permission does not list namespace permissions (li xiang).

• HBASE-14843: TestWALProcedureStore.testLoad is flakey (Matteo Bertozz).

• HBASE-14850: C++ client implementation (TECH PREVIEW. DO NOT DEPLOY IN PROD).

• HBASE-14906: Improvements on FlushLargeStoresPolicy (Yu Li).

• HBASE-14947: WALProcedureStore improvements (Matteo Bertozzi).

• HBASE-14963: Remove Guava dependency from HBase client code (ddas).

• HBASE-15019: Replication stuck when HDFS is restarted (Matteo).

• HBASE-15100: Master WALProcs still never clean up (Matteo Bertozzi).

• HBASE-15125: HBaseFsck's adoptHdfsOrphan function creates region with wrong end key boundary (chenrongwei).

• HBASE-15136: Explore different queuing behaviors while busy.

• HBASE-15142: Procedure v2 - Basic WebUI listing the procedures (Matteo).

• HBASE-15144: Procedure v2 - Web UI displaying Store state.

• HBASE-15146: Don't block on Reader threads.

• HBASE-15221: Reload the cache on re-tried puts in HTableMultiplexer and adds a close() method to HTableMultiplexer.

• HBASE-15232: Handle region location cache mgmt in AsyncProcess for multi()'s.

• HBASE-15292: Refined ZooKeeperWatcher to prevent ZooKeeper's callback while construction (Hiroshi Ikeda).

• HBASE-15306: Make RPC call queue length dynamically configurable.

• HBASE-15377: Per-RS Get metric is time based, per-region metric is size-based (Heng Chen).

• HBASE-15378: Scanner cannot handle heartbeat message with no results (Phil Yang).

• HBASE-15465: userPermission returned by getUserPermission() for the selected namespace does not have namespace set (li xiang).

• HBASE-15518: Add per-table metrics.

• HBASE-15519: Add per-user metrics.

• HBASE-15636: hard coded wait time out value in HBaseTestingUtility#waitUntilAllRegionsAssigned might cause test failure (Stephen Yuan Jiang).

• HBASE-15752: ClassNotFoundException is encountered when custom WAL codec is not found in WALPlayer job.

• HBASE-15767: upgrade httpclient to 4.3.6.

• HBASE-15808: Reduce potential bulk load intermediate space usage and waste (Jerry He).

• HBASE-15817: Backup history should mention the type (full or incremental) of the backup.

• HBASE-15842: SnapshotInfo should display ownership information.

• HBASE-15850: Localize the configuration change in testCheckTableLocks to reduce flakiness of TestHBaseFsck test suite (Stephen Yuan Jiang).

• HBASE-15858: Some region server group shell commands don't work.

• HBASE-15861: Add support for table sets in restore operation .

• HBASE-15862: Backup - Delete- Restore does not restore deleted data.

• HBASE-15873: ACL for snapshot restore / clone is not enforced.

• HBASE-15880: RpcClientImpl#tracedWriteRequest incorrectly closes HTrace span (Mikhail).

• HBASE-15884: NPE in StoreFileScanner#skipKVsNewerThanReadpoint during reverse scan (Sergey Soldatov).

• HBASE-15899: HBase incremental restore should handle namespaces properly.

• HBASE-15904: Use comma as separator for list of tables in BackupInfo.

• HBASE-15925: compat-module maven variable not evaluated (Sean Busbey).

• HBASE-15928: hbase backup delete command does not remove backup root dir from hdfs.

• HBASE-15931: Add log for long-running tasks in AsyncProcess (Yu Li).

• HBASE-15933: NullPointerException may be thrown from SimpleRegionNormalizer#getRegionSize().

• HBASE-15953: HBase backup set list command does not list any existing sets.

• HBASE-15971: Regression: Random Read/WorkloadC slower in 1.x than 0.98 (Stack).

• HBASE-15972: hbase backup set command should not accept non-existing table.

• HBASE-16006: FileSystem should be obtained from specified path in WALInputFormat#getSplits().

• HBASE-16007: Job's Configuration should be passed to TableMapReduceUtil#addDependencyJars() in WALPlayer.

• HBASE-16012: Major compaction can't work due to obsolete scanner read point in RegionServer (Guanghao Zhang).

• HBASE-16016: Fix overflow deadline value in AssignmentManager#waitForAssignment (Stephen Yuan Jiang).

• HBASE-16023: Fastpath for the FIFO rpcscheduler.

• HBASE-16037: Backup/Restore: Make automatic mode default one (Vladimir Rodionov).

• HBASE-16045: endtime argument for VerifyReplication was incorrectly specified in usage.

• HBASE-16048: Tag InternalScanner with LimitedPrivate(HBaseInterfaceAudience.COPROC).

• HBASE-16052: Improve HBaseFsck Scalability (Ben Lau).

• HBASE-16056: Procedure v2 - fix master crash for FileNotFound (Matteo Bertozzi).

• HBASE-16059: Region normalizer fails to trigger merge action where one of the regions is empty.

• HBASE-16062: Improper error handling in WAL Reader/Writer creation (Vladimir Rodionov).

• HBASE-16064: delete backup command shows HDFS permission error when deleting the intended backup.

• HBASE-16065: hbase backup set describe command does not inform if the set does not exist (Vladimir Rodionov).

• HBASE-16088: hbase restore gives a misleading message if the table doesn't exist.

• HBASE-16132: Scan does not return all the result when regionserver is busy (binlijin).

• HBASE-16237: Blocks for hbase:meta table are not cached in L1 cache.

• HBASE-16267: Remove commons-httpclient dependency from hbase-rest module.

• HBASE-16284: Unauthorized client can shutdown the cluster.

• HBASE-16288: HFile intermediate block level indexes might recurse forever creating multi TB files.

• HBASE-16314: Retry on table snapshot failure.

• HBASE-16315: RegionSizeCalculator prints region names as binary without escapes.

• HBASE-16319: Fix TestCacheOnWrite after HBASE-16288 (Andrew Purtell).

• HBASE-16350: Undo server abort from HBASE-14968.

• HBASE-16358: Backup create command: log success/failure with backup id.

• HBASE-16359: NullPointerException in RSRpcServices.openRegion().

• HBASE-16367: Race between master and region server initialization may lead to premature server abort.

• HBASE-16377: ServerName check is ineffective in region_mover.rb.

• HBASE-16429: FSHLog: deadlock if rollWriter called when ring buffer filled with appends.

• HBASE-16604: Scanner retries on IOException can cause the scans to miss data.

• HBASE-16675: Average region size may be incorrect.

• HBASE-16678: MapReduce jobs do not update counters from ScanMetrics.

• HBASE-16721: Concurrency issue in WAL unflushed seqId tracking.

• HBASE-16750: Updating asciidoctor plugin version.

• HBASE-16824: Writer.flush() can be called on already closed streams in WAL roll.

• HBASE-16931: Setting cell's seqId to zero in compaction flow might cause RS down.

• HBASE-16985: TestClusterId failed due to wrong hbase rootdir.

• HBASE-17292: Add observer notification before bulk loaded hfile is moved to region directory.

• HBASE-5291: Add Kerberos HTTP SPNEGO authentication support to HBase web consoles.