Interface | Description |
---|---|
BulkLoadObserver |
Coprocessors implement this interface to observe and mediate bulk load operations.
|
CoprocessorService |
Coprocessor endpoints providing protobuf services should implement this
interface and return the
Service instance via CoprocessorService.getService() . |
EndpointObserver |
Coprocessors implement this interface to observe and mediate endpoint invocations
on a region.
|
MasterCoprocessorEnvironment | |
MasterObserver |
Defines coprocessor hooks for interacting with operations on the
HMaster process. |
RegionCoprocessorEnvironment | |
RegionObserver |
Coprocessors implement this interface to observe and mediate client actions
on the region.
|
RegionServerCoprocessorEnvironment | |
RegionServerObserver | |
SingletonCoprocessorService |
Coprocessor endpoints registered once per server and providing protobuf services should implement
this interface and return the
Service instance via SingletonCoprocessorService.getService() . |
WALCoprocessorEnvironment | |
WALObserver |
It's provided to have a way for coprocessors to observe, rewrite,
or skip WALEdits as they are being written to the WAL.
|
Class | Description |
---|---|
AggregateImplementation<T,S,P extends Message,Q extends Message,R extends Message> |
A concrete AggregateProtocol implementation.
|
BaseMasterAndRegionObserver | |
BaseMasterObserver | |
BaseRegionObserver |
An abstract class that implements RegionObserver.
|
BaseRegionServerObserver |
An abstract class that implements RegionServerObserver.
|
BaseRowProcessorEndpoint<S extends Message,T extends Message> |
This class demonstrates how to implement atomic read-modify-writes
using
Region.processRowsWithLocks(org.apache.hadoop.hbase.regionserver.RowProcessor<?, ?>) and Coprocessor endpoints. |
BaseWALObserver |
An abstract class that implements WALObserver.
|
ColumnAggregationEndpoint |
The aggregation implementation at a region.
|
ColumnAggregationEndpointNullResponse |
Test coprocessor endpoint that always returns
null for requests to the last region
in the table. |
ColumnAggregationEndpointWithErrors |
Test coprocessor endpoint that always throws a
DoNotRetryIOException for requests on
the last region in the table. |
ColumnInterpreter<T,S,P extends Message,Q extends Message,R extends Message> |
Defines how value for specific column is interpreted and provides utility
methods like compare, add, multiply etc for them.
|
CoprocessorHost<E extends CoprocessorEnvironment> |
Provides the common setup framework and runtime services for coprocessor
invocation from HBase services.
|
CoprocessorHost.Environment |
Encapsulation of the environment of each coprocessor
|
MultiRowMutationEndpoint |
This class demonstrates how to implement atomic multi row transactions using
HRegion.mutateRowsWithLocks(java.util.Collection, java.util.Collection)
and Coprocessor endpoints. |
ObserverContext<E extends CoprocessorEnvironment> |
Carries the execution state for a given invocation of an Observer coprocessor
(
RegionObserver , MasterObserver , or WALObserver )
method. |
ProtobufCoprocessorService |
Test implementation of a coprocessor endpoint exposing the
TestRpcServiceProtos.TestProtobufRpcProto service methods. |
SampleRegionWALObserver |
Class for testing WALObserver coprocessor.
|
SimpleRegionObserver |
A sample region observer that tests the RegionObserver interface.
|
SimpleRegionObserver.Legacy |
This implementation should trigger our legacy support because it does not directly
implement the newer API calls.
|
TestAggregateProtocol |
A test class to cover aggregate functions, that can be implemented using
Coprocessors.
|
TestBatchCoprocessorEndpoint |
TestEndpoint: test cases to verify the batch execution of coprocessor Endpoint
|
TestBigDecimalColumnInterpreter |
A test class to test BigDecimalColumnInterpreter for AggregationsProtocol
|
TestClassLoading |
Test coprocessors class loading.
|
TestCoprocessorConfiguration |
Tests for global coprocessor loading configuration
|
TestCoprocessorConfiguration.SystemCoprocessor | |
TestCoprocessorConfiguration.TableCoprocessor | |
TestCoprocessorEndpoint |
TestEndpoint: test cases to verify coprocessor Endpoint
|
TestCoprocessorInterface | |
TestCoprocessorInterface.CoprocessorII | |
TestCoprocessorInterface.CoprocessorImpl | |
TestCoprocessorStop |
Tests for master and regionserver coprocessor stop method
|
TestCoprocessorStop.FooCoprocessor | |
TestCoprocessorTableEndpoint | |
TestDoubleColumnInterpreter |
A test class to test DoubleColumnInterpreter for AggregateProtocol
|
TestHTableWrapper |
Tests class
HTableWrapper
by invoking its methods and briefly asserting the result is reasonable. |
TestMasterCoprocessorExceptionWithAbort |
Tests unhandled exceptions thrown by coprocessors running on master.
|
TestMasterCoprocessorExceptionWithAbort.BuggyMasterObserver | |
TestMasterCoprocessorExceptionWithAbort.CreateTableThread | |
TestMasterCoprocessorExceptionWithAbort.MasterTracker | |
TestMasterCoprocessorExceptionWithRemove |
Tests unhandled exceptions thrown by coprocessors running on master.
|
TestMasterCoprocessorExceptionWithRemove.BuggyMasterObserver | |
TestMasterCoprocessorExceptionWithRemove.MasterTracker | |
TestMasterObserver |
Tests invocation of the
MasterObserver
interface hooks at all appropriate times during normal HMaster operations. |
TestMasterObserver.CPMasterObserver | |
TestOpenTableInCoprocessor |
Test that a coprocessor can open a connection and write to another table, inside a hook.
|
TestOpenTableInCoprocessor.CustomThreadPoolCoprocessor |
Coprocessor that creates an HTable with a pool to write to another table
|
TestOpenTableInCoprocessor.SendToOtherTableCoprocessor |
Custom coprocessor that just copies the write to another table.
|
TestRegionObserverBypass | |
TestRegionObserverBypass.TestCoprocessor | |
TestRegionObserverInterface | |
TestRegionObserverInterface.EvenOnlyCompactor | |
TestRegionObserverScannerOpenHook | |
TestRegionObserverScannerOpenHook.CompactionCompletionNotifyingRegion | |
TestRegionObserverScannerOpenHook.EmptyRegionObsever |
Do the same logic as the
BaseRegionObserver . |
TestRegionObserverScannerOpenHook.NoDataFilter | |
TestRegionObserverScannerOpenHook.NoDataFromCompaction |
Don't allow any data to be written out in the compaction by creating a custom
StoreScanner . |
TestRegionObserverScannerOpenHook.NoDataFromFlush |
Don't allow any data in a flush by creating a custom
StoreScanner . |
TestRegionObserverScannerOpenHook.NoDataFromScan |
Don't return any data from a scan by creating a custom
StoreScanner . |
TestRegionObserverStacking | |
TestRegionObserverStacking.ObserverA | |
TestRegionObserverStacking.ObserverB | |
TestRegionObserverStacking.ObserverC | |
TestRegionServerCoprocessorEndpoint | |
TestRegionServerCoprocessorExceptionWithAbort |
Tests unhandled exceptions thrown by coprocessors running on a regionserver..
|
TestRegionServerCoprocessorExceptionWithAbort.BuggyRegionObserver | |
TestRegionServerCoprocessorExceptionWithAbort.FailedInitializationObserver | |
TestRegionServerCoprocessorExceptionWithRemove |
Tests unhandled exceptions thrown by coprocessors running on regionserver.
|
TestRegionServerCoprocessorExceptionWithRemove.BuggyRegionObserver | |
TestRegionServerObserver |
Tests invocation of the
RegionServerObserver
interface hooks at all appropriate times during normal HMaster operations. |
TestRegionServerObserver.CPRegionServerObserver | |
TestRowProcessorEndpoint |
Verifies ProcessEndpoint works.
|
TestRowProcessorEndpoint.RowProcessorEndpoint<S extends Message,T extends Message> |
This class defines two RowProcessors:
IncrementCounterProcessor and FriendsOfFriendsProcessor.
|
TestRowProcessorEndpoint.RowProcessorEndpoint.FriendsOfFriendsProcessor | |
TestRowProcessorEndpoint.RowProcessorEndpoint.IncrementCounterProcessor | |
TestRowProcessorEndpoint.RowProcessorEndpoint.RowSwapProcessor | |
TestRowProcessorEndpoint.RowProcessorEndpoint.TimeoutProcessor | |
TestWALObserver |
Tests invocation of the
MasterObserver interface hooks at
all appropriate times during normal HMaster operations. |
Enum | Description |
---|---|
RegionObserver.MutationType |
Mutation type for postMutationBeforeWAL hook
|
Exception | Description |
---|---|
CoprocessorException |
Thrown if a coprocessor encounters any exception.
|
Multiple types of coprocessors are provided to provide sufficient flexibility for potential use cases. Right now there are:
Coprocessor
interface so that coprocessor framework
can manage it internally.
Another design goal of this interface is to provide simple features for making coprocessors useful, while exposing no more internal state or control actions of the region server than necessary and not exposing them directly.
Over the lifecycle of a region, the methods of this interface are invoked when the corresponding events happen. The master transitions regions through the following states:
unassigned -> pendingOpen -> open -> pendingClose -> closed.
Coprocessors have opportunity to intercept and handle events in pendingOpen, open, and pendingClose states.
The region server is opening a region to bring it online. Coprocessors can piggyback or fail this process.
RegionObserver
interface it can
observe and mediate client actions on the region:
BaseRegionObserverCoprocessor
which
implements both Coprocessor
and RegionObserver
.
In addition, it overrides all methods with default behaviors so you don't
have to override all of them.
Here's an example of what a simple RegionObserver might look like. This
example shows how to implement access control for HBase. This
coprocessor checks user information for a given client request, e.g.,
Get/Put/Delete/Scan by injecting code at certain
RegionObserver
preXXX hooks. If the user is not allowed to access the resource, a
CoprocessorException will be thrown. And the client request will be
denied by receiving this exception.
package org.apache.hadoop.hbase.coprocessor; import java.util.List; import org.apache.hadoop.hbase.KeyValue; import org.apache.hadoop.hbase.client.Get; // Sample access-control coprocessor. It utilizes RegionObserver // and intercept preXXX() method to check user privilege for the given table // and column family. public class AccessControlCoprocessor extends BaseRegionObserverCoprocessor { // @Override public Get preGet(CoprocessorEnvironment e, Get get) throws CoprocessorException { // check permissions.. if (access_not_allowed) { throw new AccessDeniedException("User is not allowed to access."); } return get; } // override prePut(), preDelete(), etc. }
Coprocessor
and RegionObserver
provide certain hooks
for injecting user code running at each region. The user code will be triggered
by existing HTable
and HBaseAdmin
operations at
the certain hook points.
Coprocessor Endpoints allow you to define your own dynamic RPC protocol to communicate
between clients and region servers, i.e., you can create a new method, specifying custom
request parameters and return types. RPC methods exposed by coprocessor Endpoints can be
triggered by calling client side dynamic RPC functions -- HTable.coprocessorService(...)
.
To implement an Endpoint, you need to:
CoprocessorService
interface. The CoprocessorService.getService()
method should return a reference to the Endpoint's protocol buffer Service instance.
For a more detailed discussion of how to implement a coprocessor Endpoint, along with some sample
code, see the org.apache.hadoop.hbase.client.coprocessor
package documentation.
HTableDescriptor
for a newly created table.
(Currently we don't really have an on demand coprocessor loading mechanism for opened regions.)
hbase.coprocessor.region.classes
from Configuration
.
Coprocessor framework will automatically load the configured classes as
default coprocessors. The classes must be included in the classpath already.
<property> <name>hbase.coprocessor.region.classes</name> <value>org.apache.hadoop.hbase.coprocessor.AccessControlCoprocessor, org.apache.hadoop.hbase.coprocessor.ColumnAggregationProtocol</value> <description>A comma-separated list of Coprocessors that are loaded by default. For any override coprocessor method from RegionObservor or Coprocessor, these classes' implementation will be called in order. After implement your own Coprocessor, just put it in HBase's classpath and add the fully qualified class name here. </description> </property>
The first defined coprocessor will be assigned
Coprocessor.Priority.SYSTEM
as priority. And each following
coprocessor's priority will be incremented by one. Coprocessors are executed
in order according to the natural ordering of the int.
'COPROCESSOR$1' => 'hdfs://localhost:8020/hbase/coprocessors/test.jar:Test:1000' 'COPROCESSOR$2' => '/hbase/coprocessors/test2.jar:AnotherTest:1001'
<path> must point to a jar, can be on any filesystem supported by the Hadoop FileSystem object.
<class> is the coprocessor implementation class. A jar can contain more than one coprocessor implementation, but only one can be specified at a time in each table attribute.
<priority> is an integer. Coprocessors are executed in order according to the natural ordering of the int. Coprocessors can optionally abort actions. So typically one would want to put authoritative CPs (security policy implementations, perhaps) ahead of observers.
Path path = new Path(fs.getUri() + Path.SEPARATOR + "TestClassloading.jar"); // create a table that references the jar HTableDescriptor htd = new HTableDescriptor(TableName.valueOf(getClass().getTableName())); htd.addFamily(new HColumnDescriptor("test")); htd.setValue("Coprocessor$1", path.toString() + ":" + classFullName + ":" + Coprocessor.Priority.USER); HBaseAdmin admin = new HBaseAdmin(this.conf); admin.createTable(htd);Chain of RegionObservers
As described above, multiple coprocessors can be loaded at one region at the same time. In case of RegionObserver, you can have more than one RegionObservers register to one same hook point, i.e, preGet(), etc. When a region reach the hook point, the framework will invoke each registered RegionObserver by the order of assigned priority.