Ozone volume scanner
The Ozone Volume Scanner feature enables to detect any disk failures on the DataNodes. Learn how you can configure the frequency of volume scans that can detect disk failures and how to handle volume failures.
The volume scanner scans each data volume configured by
hdds.datanode.dir and each metadata volume configured by
Various events in the DataNode can trigger volume scans. Each volume scan consists of multiple checks.
Background volume scan
Datanodes scan every volume once per hour. The frequency of this check is configured with
hdds.datanode.periodic.disk.check.interval.minutes. This property defines the
minimum frequency of scans for a volume.
On-demand volume scan
Any error reading from or writing to a volume during regular datanode operation
triggers a scan of that volume. To prevent frequent scanning of the same volume, the
hdds.datanode.disk.check.min.gap configuration, which defaults to 10 minutes,
specifies the minimum time to wait between two consecutive scans of the same volume.
- Directory check
This checks that each directory configured in
dfs.container.ratis.datanode.storage.direxists and has read, write, and execute permissions by the datanode process. If this check fails, the volume is marked as failed.
- I/O checkThis checks that the underlying disk is present and functioning properly. The I/O check writes data to a small temporary file, synchronizes it to ensure it touches the hardware, reads the data, and then deletes the file. To account for intermittent errors, this check must fail multiple times before the volume is failed. The specifics of this check can be changed with the following configurations:
The size in bytes of the file to write for disk checking. During the check, the content of this file is stored in memory. The default value is 100 bytes.
The number of volume scan results in determining if the volume should be failed based on the I/O failures. The default value is 3.
The number of I/O failures that can occur out of the last
hdds.datanode.disk.check.io.test.countscans without the volume marked unhealthy. The default value is 1.
An example of using the default values:
Consider the I/O check passed on two out of the last three volume scans but failed on the latest volume scan. The volume remains healthy because one out of the last three I/O checks failed and `
hdds.datanode.disk.check.io.failures.tolerated` is set to 1. If a fourth volume scan is run and the I/O check fails, the volume is failed, because out of the last three volume scans two failed due to I/O checks.
- Time check
Both of the above checks must finish within a certain amount of time. Otherwise, the volume fails. This time limit is configured with the
hdds.datanode.disk.check.timeoutparameter. The default value is 10 minutes.
Handling Volume Failures
When a volume is marked failed, Ozone no longer uses it and triggers replication of the data from existing copies on other datanodes. After the issue on the failed volume is corrected, restart the datanode to detect the new volume.Datanodes continue to run until they have no healthy data volumes or metadata volumes remaining. If you want the datanodes to shut down after a specified number of volume failures, set
hdds.datanode.failed.metadata.volumes.toleratedto a positive number. If the set number of volume failures is crossed, the datanode shuts down automatically.