Choosing RegionServers to Replicate to

The master cluster RegionServers chooses potential recipients for replication and the Zookeeper watcher monitors changes in the composition of the slave clusters.

When a master cluster RegionServer initiates a replication source to a slave cluster, it first connects to the ZooKeeper ensemble of the slave using the provided cluster key. Then it scans the rs/ directory to discover all the available 'sinks' (RegionServers that are accepting incoming streams of edits to replicate) and randomly chooses a subset of them using a configured ratio which has a default value of 10 per cent. For example, if a slave cluster has 150 servers, 15 are chosen as potential recipients for edits sent by the master cluster RegionServer. Because this selection is performed by each master RegionServer, the probability that all slave RegionServers are used is very high. This method works for clusters of any size. For example, a master cluster of 10 servers replicating to a slave cluster of 5 servers with a ratio of 10 per cent causes the master cluster RegionServers to choose one server each at random.

A ZooKeeper watcher is placed on the ${zookeeper.znode.parent}/rs node of the slave cluster by each of the master cluster RegionServers. This watcher monitors changes in the composition of the slave cluster. When nodes are removed from the slave cluster, or if nodes go down or come back up, the master cluster RegionServers respond by selecting a new pool of slave Region Servers to which to replicate.