This is the documentation for CDH 5.1.x. Documentation for other versions is available at Cloudera Documentation.

Configuring HBase Snapshots

About HBase Snapshots

In previous HBase releases, the only way to a backup or to clone a table was to use CopyTable or ExportTable, or to copy all the hfiles in HDFS after disabling the table. The disadvantages of these methods are:

  • CopyTable and ExportTable can degrade region server performance.
  • Disabling the table means no reads or writes; this is usually unacceptable.

HBase Snapshots allow you to clone a table without making data copies, and with minimal impact on Region Servers. Exporting the table to another cluster should not have any impact on the the region servers.

Use Cases

  • Recovery from user or application errors
    • Useful because it may be some time before the database administrator notices the error
        Note:

      The database administrator needs to schedule the intervals at which to take and delete snapshots. Use a script or your preferred management tool for this; it is not built into HBase.

    • The database administrator may want to save a snapshot right before a major application upgrade or change.
        Note:

      Snapshots are not primarily used for system upgrade protection because they would not roll back binaries, and would not necessarily be proof against bugs or errors in the system or the upgrade.

    • Sub-cases for recovery:
      • Rollback to previous snapshot and merge in reverted data
      • View previous snapshots and selectively merge them into production
  • Backup
    • Capture a copy of the database and store it outside HBase for disaster recovery
    • Capture previous versions of data for compliance, regulation, archiving
    • Export from snapshot on live system provides a more consistent view of HBase than CopyTable and ExportTable
  • Audit and/or report view of data at a specific time
    • Capture monthly data for compliance
    • Use for end-of-day/month/quarter reports
  • Use for Application testing
    • Test schema or application changes on like production data from snapshot and then throw away
    • For example: take a snapshot; create a new table from the snapshot content (schema plus data); manipulate the new table by changing the schema, adding and removing rows, and so on (the original table, the snapshot, and the new table remain independent of each other)
  • Offload work
    • Capture, copy, and restore data to another site
    • Export data to another cluster

Where Snapshots Are Stored

The snapshot metadata is stored in the .hbase_snapshot directory under the hbase root directory (/hbase/.hbase-snapshot). Each snapshot has its own directory that includes all the references to the hfiles, logs, and metadata needed to restore the table.

hfiles needed by the snapshot are in the traditional /hbase/data/<namespace>/<tableName>/<regionName>/<familyName>/ location if the table is still using them; otherwise they will be placed in /hbase/.archive/<namespace>/<tableName>/<regionName>/<familyName>/

Zero-copy Restore and Clone Table

From a snapshot you can create a new table (clone operation) or restore the original table. These two operations do not involve data copies; instead a link is created to point to the original hfiles.

Changes to a cloned or restored table do not affect the snapshot or (in case of a clone) the original table.

If you want to clone a table to another cluster, you need to export the snapshot to the other cluster and then execute the clone operation; see Exporting a Snapshot to Another Cluster.

Reverting to a Previous HBase Version

Snapshots don’t affect HBase backward compatibility if they are not used.

If you do use the snapshot capability, backward compatibility is affected as follows:

  • If you only take snapshots, you can still go back to a previous HBase version
  • If you have used restore or clone, you cannot go back to a previous version unless the cloned or restored tables have no links (there is no automated way to check; you would need to inspect the file system manually).

Storage Considerations

Since the hfiles are immutable, a snapshot consists of reference to the files that are in the table at the moment the snapshot is taken. No copies of the data are made during the snapshot operation, but copies may be made when a compaction or deletion is triggered. In this case, if a snapshot has a reference to the files to be removed, the files are moved to an archive folder, instead of being deleted. This allows the snapshot to be restored in full.

Because no copies are performed, multiple snapshots share the same hfiles, but in the worst case scenario, each snapshot could have different set of hfiles (tables with lots of updates, and compactions).

Configuring and Enabling Snapshots

Snapshots are on by default; to disable them, set the hbase.snapshot.enabled property in hbase-site.xml to false: 

<property>
   <name>hbase.snapshot.enabled</name>
   <value>
      false
   </value>
</property>

To enable snapshots after you have disabled them, set hbase.snapshot.enabled to true.

  Note:

If you have taken snapshots and then decide to disable snapshots, you must delete the snapshots before restarting the HBase master; the HBase master will not start if snapshots are disabled and snapshots exist.

Snapshots don’t affect HBase performance if they are not used.

Shell Commands

You can manage snapshots by using the HBase shell or the HBaseAdmin Java API.

The following table shows actions you can take from the shell:

Action

Shell command

Comments

Take a snapshot of tableX called snapshotX 

snapshot 'tableX', 'snapshotX'

Snapshots can be taken while a table is disabled, or while a table is online and serving traffic.

  • If a table is disabled (via disable <table>) an offline snapshot is taken. This snapshot is driven by the master and fully consistent with the state when the table was disabled. This is the simplest and safest method, but it involves a service interruption since the table must be disabled to take the snapshot.
  • In an online snapshot, the table remains available while the snapshot is taken, and should incur minimal noticeable performance degradation of normal read/write loads. This snapshot is coordinated by the master and run on the region servers. The current implementation - simple-flush snapshots - provides no causal consistency guarantees. Despite this shortcoming, it offers the same degree of consistency as CopyTable and overall is a huge improvement over CopyTable.

Restore snapshot snapshotX (it will replace the source table content) 

restore_snapshot ‘snapshotX’

Restoring a snapshot attempts to replace the current version of a table with another version of the table. To run this command, you must disable the target table. The restore command takes a snapshot of the table (appending a timestamp code), and then essentially clones data into the original data and removes data not in the snapshot. If the operation succeeds, the target table will be enabled. Use this capability only in an emergency; see Restrictions.

List all available snapshots 

list_snapshots

 

List all available snapshots starting with ‘mysnapshot_’ (regular expression) 

list_snapshots ‘my_snapshot_*’

 

Remove a snapshot called snapshotX 

delete_snapshot ‘snapshotX’
  

Create a new table tableY from a snapshot snapshotX 

clone_snapshot ‘snapshotX’, ‘tableY’

Cloning a snapshot creates a new read/write table that can serve the data kept at the time of the snapshot. The original table and the cloned table can be modified independently without interfering – new data written to one table will not show up on the other.

Exporting a Snapshot to Another Cluster

You can export any snapshot from one cluster to another. Exporting the snapshot copies the table's hfiles, logs, and the snapshot's metadata, from the source cluster to the destination cluster. You can specify the -copy-from option to copy from a remote cluster to the local cluster or another remote cluster. If you do not specify the -copy-from option, the hbase.rootdir in the HBase configuration is used, which means that you are exporting from the current cluster.You must specify the -copy-to option, to specify the destination cluster.
  Note: Snapshots must be enabled on the destination cluster. See Configuring and Enabling Snapshots.

The ExportSnapshot tool executes a MapReduce Job, similar to distcp, to copy files to the other cluster. It works at file-system level, so the hbase cluster can be offline.

To copy a snapshot called MySnapshot to an HBase cluster srv2 (hdfs://srv2:8082/hbase) using 16 mappers: 

hbase class org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot MySnapshot -copy-to hdfs://srv2:8082/hbase -mappers 16

To export the snapshot and change the ownership of the files during the copy: 

hbase class org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot MySnapshot -copy-to hdfs://srv2:8082/hbase -chuser MyUser -chgroup MyGroup -chmod 700 -mappers 16
You can also use the Java -D option in many tools to specify MapReduce or other configuration. properties. For example, the following command copies MY_SNAPSHOT to hdfs://cluster2/hbase using groups of 10 hfiles per mapper:
hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -Dsnapshot.export.default.map.group=10 -snapshot MY_SNAPSHOT -copy-to hdfs://cluster2/hbase
(The number of mappers is calculated as TotalNumberOfHFiles/10.)
To export from one remote cluster to another remote cluster, specify both -copy-from and -copy-to parameters. You could then reverse the direction to restore the snapshot bacvk to the first remote cluster.
hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot snapshot-test -copy-from hdfs://machine1/hbase -copy-to hdfs://machine2/my-backup
To specify a different name for the snapshot on the target cluster, use the -target option.
hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot snapshot-test -copy-from hdfs://machine1/hbase -copy-to hdfs://machine2/my-backup -target new-snapshot

Restrictions

  Warning:

Do not use merge in combination with snapshots. Merging two regions can cause data loss if snapshots or cloned tables exist for this table.

The merge is likely to corrupt the snapshot and any tables cloned from the snapshot. In addition, if the table has been restored from a snapshot, the merge may also corrupt the table. The snapshot may survive intact if the regions being merged are not in the snapshot, and clones may survive if they do not share files with the original table or snapshot. You can use the Snapinfo tool (see Information and Debugging) to check the status of the snapshot. If the status is BROKEN, the snapshot is unusable.

  • All the Masters and Region Servers must be running CDH 5.
  • If you have enabled the AccessController Coprocessor for HBase, only a global administrator can take, clone, or restore a snapshot, and these actions do not capture the ACL rights. This means that restoring a table preserves the ACL rights of the existing table, while cloning a table creates a new table that has no ACL rights until the administrator adds them.
  • Do not take, clone, or restore a snapshot during a rolling restart. Snapshots rely on the Region Servers being up; otherwise the snapshot will fail.
      Note: This restriction also applies to rolling upgrade, which can currently be done only via Cloudera Manager.

If you are using HBase Replication and you need to restore a snapshot: If you are using HBase Replication the replicas will be out of synch when you restore a snapshot. Do this only in an emergency.

  Important:

Snapshot restore is an emergency tool; you need to disable the table and table replication to get to an earlier state, and you may lose data in the process.

If you need to restore a snapshot, proceed as follows:

  1. Disable the table that is the restore target, and stop the replication
  2. Remove the table from both the master and slave clusters
  3. Restore the snapshot on the master cluster
  4. Create the table on the slave cluster and use CopyTable to initialize it.
  Note:

If this is not an emergency (for example, if you know that you have lost just a set of rows such as the rows starting with "xyz"), you can create a clone from the snapshot and create a MapReduce job to copy the data that you've lost.

In this case you don't need to stop replication or disable your main table.

Snapshot Failures

Region moves, splits, and other metadata actions that happen while a snapshot is in progress will probably cause the snapshot to fail; the software detects and rejects corrupted snapshot attempts.

Information and Debugging

You can use the SnapshotInfo tool to get information about a snapshot, including status, files, disk usage, and debugging information.

Examples:

Use the -h option to print usage instructions for the SnapshotInfo utility.

$ hbase org.apache.hadoop.hbase.snapshot.SnapshotInfo -h
Usage: bin/hbase org.apache.hadoop.hbase.snapshot.SnapshotInfo [options]
 where [options] are:
  -h|-help                Show this help and exit.
  -remote-dir             Root directory that contains the snapshots.
  -list-snapshots         List all the available snapshots and exit.
  -snapshot NAME          Snapshot to examine.
  -files                  Files and logs list.
  -stats                  Files and logs stats.
  -schema                 Describe the snapshotted table.

Use the -list-snapshots option to list all snapshots and exit. This option is new in CDH 5.1.

$ hbase org.apache.hadoop.hbase.snapshot.SnapshotInfo -list-snapshots
SNAPSHOT             | CREATION TIME        | TABLE NAME
snapshot-test        |  2014-06-24T19:02:54 | test

Use the -remote-dir option with the -list-snapshots option to list snapshots located on a remote system.

$ hbase org.apache.hadoop.hbase.snapshot.SnapshotInfo -remote-dir s3n://mybucket/mysnapshot-dir -list-snapshots
SNAPSHOT |   CREATION TIME  | TABLE NAME
snapshot-test        2014-05-01 10:30    myTable

Use the -snapshot option to print information about a specific snapshot.

$ hbase org.apache.hadoop.hbase.snapshot.SnapshotInfo -snapshot test-snapshot
Snapshot Info
----------------------------------------
   Name: test-snapshot
   Type: DISABLED
  Table: test-table
Version: 0
Created: 2012-12-30T11:21:21
**************************************************************
Use the -snapshot with the -stats options to display additional statistics about a snapshot.
$ hbase org.apache.hadoop.hbase.snapshot.SnapshotInfo -stats -snapshot snapshot-test
Snapshot Info
----------------------------------------
   Name: snapshot-test
   Type: FLUSH
  Table: test
 Format: 0
Created: 2014-06-24T19:02:54


1 HFiles (0 in archive), total size 1.0k (100.00% 1.0k shared with the source table)
Use the -schema option with the -snapshot option to display the schema of a snapshot.
$ hbase org.apache.hadoop.hbase.snapshot.SnapshotInfo  -schema -snapshot snapshot-test
Snapshot Info
----------------------------------------
   Name: snapshot-test
   Type: FLUSH
  Table: test
 Format: 0
Created: 2014-06-24T19:02:54

Table Descriptor
----------------------------------------
'test', {NAME => 'cf', DATA_BLOCK_ENCODING => 'FAST_DIFF', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0',
COMPRESSION => 'GZ', VERSIONS => '1', TTL => 'FOREVER', MIN_VERSIONS => '0', KEEP_DELETED_CELLS => 'false',
BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}

Use the -files option with the -snapshot option to list information about files contained in a snapshot.

$ hbase org.apache.hadoop.hbase.snapshot.SnapshotInfo -snapshot test-snapshot -files
Snapshot Info
----------------------------------------
   Name: test-snapshot
   Type: DISABLED
  Table: test-table
Version: 0
Created: 2012-12-30T11:21:21

Snapshot Files
----------------------------------------
   52.4k test-table/02ba3a0f8964669520cf96bb4e314c60/cf/bdf29c39da2a4f2b81889eb4f7b18107 (archive)
   52.4k test-table/02ba3a0f8964669520cf96bb4e314c60/cf/1e06029d0a2a4a709051b417aec88291 (archive)
   86.8k test-table/02ba3a0f8964669520cf96bb4e314c60/cf/506f601e14dc4c74a058be5843b99577 (archive)
   52.4k test-table/02ba3a0f8964669520cf96bb4e314c60/cf/5c7f6916ab724eacbcea218a713941c4 (archive)
  293.4k test-table/02ba3a0f8964669520cf96bb4e314c60/cf/aec5e33a6564441d9bd423e31fc93abb (archive)
   52.4k test-table/02ba3a0f8964669520cf96bb4e314c60/cf/97782b2fbf0743edaacd8fef06ba51e4 (archive)

6 HFiles (6 in archive), total size 589.7k (0.00% 0.0 shared with the source table)
0 Logs, total size 0.0
Page generated September 3, 2015.