Configuring strong meta servers

The strong meta feature uses an HBase RegionServerGroup (RSGroup) to provide dedicated servers to the HBase system tables. This prevents customer table regions from being hosted on the same RegionServers as the system regions and prevents the load from these custom tables from affecting system table access and initialization.

  • You must have the COD_STRONG_META_SERVERS entitlement to use this feature.
  • This feature is only supported on AWS environments.
  • By default, meta replication is enabled and the replica number is set to 3. Therefore, ensure that each strong meta node contains at least 3 meta regions. This does not require any manual steps.
  1. Launch the CDP CLI tool.
  2. Run the following command to add the desired number of strong meta servers.
    cdp opdb update-database --environment-name envName --database-name dbName --num-desired-strong-meta-servers 3
    The above command creates 3 strong meta nodes, adds them to the cluster, and puts a region server on them.
  3. To verify if the nodes are healthy, ensure that the logs on these nodes have no errors or exceptions. If data on the cluster and balancer is enabled, these nodes must serve random regions because the balancer moves regions there.
  4. Ensure that the RSGroup feature is enabled on the COD cluster. To enable this, use the list_rsgroups command in the hbase shell.
    If the command is not enabled, the following error is displayed.
    org.apache.hadoop.hbase.exceptions.UnknownProtocolException: No registered Master Coprocessor Endpoint found for RSGroupAdminService. Has it been enabled?
    1. To enable the RSGroup feature, check whether hbase.master.loadbalancer.class property is configured. If configured, move its value to hbase.rsgroup.grouploadbalancer.class property.
    2. Set the value of hbase.master.loadbalancer.class property to org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer in the hbase-site.xml file.
    3. Set the value of hbase.coprocessor.master.classes property to org.apache.hadoop.hbase.rsgroup.RSGroupAdminEndpoint in the hbase-site.xml file.
    4. Restart the master nodes for these changes to take effect.
    5. To verify if the setup was successful, use the list_rsgroups command in the hbase shell.
  5. Create a RegionServerGroup and add the strong meta nodes to it. Ensure that the system tables are also added. Execute the following steps in an hbase shell.
    // verify that the feature works and every worker/strongmeta nodes are visible
    list_rsgroups
    //create the group (please note, that a different name can be also used)
    add_rsgroup "strongmeta"
    //verify that the group is created
    list_rsgroups
    //move the strongmeta nodes to the group (node names can be copied from the list_rsgroups output)
    move_servers_rsgroup "strongmeta",["cod--xxx-strongmeta0.cod-7216.xcu2-8y8x.dev.cldr.work:16020","cod--xxx-strongmeta1.cod-7216.xcu2-8y8x.dev.cldr.work:16020","cod--xxx-strongmeta2.cod-7216.xcu2-8y8x.dev.cldr.work:16020"]
    //verify that the nodes are added to the group
    list_rsgroups
    //move the system tables to the group
    move_tables_rsgroup "strongmeta",["hbase:meta"]
    move_tables_rsgroup "strongmeta",["hbase:ack"]
    move_tables_rsgroup "strongmeta",["hbase:namespace"]
    move_tables_rsgroup "strongmeta",["hbase:rsgroup"]
    //verify that the tables were moved
    list_rsgroups
  6. If the previous steps are successful, ensure that everything is functioning as expected using the HBase UI.
Ensure that each strong meta node now only serves system table regions.