Configuring Atomic Folder Rename
The Azure Blob Storage stores files in a flat key/value store without formal support for folders. The hadoop-azure file system layer simulates folders on top of Azure storage. By default, folder rename in the hadoop-azure file system layer is not atomic. This means that a failure during a folder rename could potentially leave some folders in the original directory and some in the new one.
Since HBase depends on atomic folder rename, a configuration setting called
fs.azure.atomic.rename.dir
can be set in core-site.xml
to specify
a comma-separated list of directories where folder rename is made atomic. If a folder rename
fails, a redo will be applied to finish. A file
<folderName>-renamePending.json
may appear temporarily and is the record of
the intention of the rename operation, to allow redo in event of a failure.
The default value of this setting is just /hbase
. To list multiple
directories, separate them with a comma. For example:
<property> <name>fs.azure.atomic.rename.dir</name> <value>/hbase,/data</value> </property>