Files and Directories
Persistence of HDFS metadata is implemented using fsimage file and edits files.
Note | |
---|---|
Do not attempt to modify metadata directories or files. Unexpected modifications can cause HDFS downtime, or even permanent data loss. This information is provided for educational purposes only. |
Persistence of HDFS metadata broadly consist of two categories of files:
- fsimage
-
Contains the complete state of the file system at a point in time. Every file system modification is assigned a unique, monotonically increasing transaction ID. An
fsimage
file represents the file system state after all modifications up to a specific transaction ID. - edits file
-
Contains a log that lists each file system change (file creation, deletion or modification) that was made after the most recent
fsimage
.
Checkpointing is the process of merging the content of the most recent
fsimage
, with all edits
applied after that
fsimage
is merged, to create a new fsimage
.
Checkpointing is triggered automatically by configuration policies or manually by
HDFS administration commands.