Ozone volume scanner

The Ozone Volume Scanner feature enables to detect any disk failures on the DataNodes. Learn how you can configure the frequency of volume scans that can detect disk failures and how to handle volume failures.

The volume scanner scans each data volume configured by hdds.datanode.dir and each metadata volume configured by dfs.container.ratis.datanode.storage.dir.

Various events in the DataNode can trigger volume scans. Each volume scan consists of multiple checks.

Background volume scan

Datanodes scan every volume once per hour. The frequency of this check is configured with hdds.datanode.periodic.disk.check.interval.minutes. This property defines the minimum frequency of scans for a volume.

On-demand volume scan

Any error reading from or writing to a volume during regular datanode operation triggers a scan of that volume. To prevent frequent scanning of the same volume, the hdds.datanode.disk.check.min.gap configuration, which defaults to 10 minutes, specifies the minimum time to wait between two consecutive scans of the same volume.

Directory check
This checks that each directory configured in hdds.datanode.dir and dfs.container.ratis.datanode.storage.dir exists and has read, write, and execute permissions by the datanode process. If this check fails, the volume is marked as failed.
I/O check
This checks that the underlying disk is present and functioning properly. The I/O check writes data to a small temporary file, synchronizes it to ensure it touches the hardware, reads the data, and then deletes the file. To account for intermittent errors, this check must fail multiple times before the volume is failed. The specifics of this check can be changed with the following configurations:
- hdds.datanode.disk.check.io.file.size
  The size in bytes of the file to write for disk checking. During the check, the content of this file is stored in memory. The default value is 100 bytes.
- hdds.datanode.disk.check.io.test.count
  The number of volume scan results in determining if the volume should be failed based on the I/O failures. The default value is 3.
- hdds.datanode.disk.check.io.failures.tolerated
  The number of I/O failures that can occur out of the last hdds.datanode.disk.check.io.test.count scans without the volume marked unhealthy. The default value is 1.
An example of using the default values:
Consider the I/O check passed on two out of the last three volume scans but failed on the latest volume scan. The volume remains healthy because one out of the last three I/O checks failed and `hdds.datanode.disk.check.io.failures.tolerated` is set to 1. If a fourth volume scan is run and the I/O check fails, the volume is failed, because out of the last three volume scans two failed due to I/O checks.
Time check

Both of the above checks must finish within a certain amount of time. Otherwise, the volume fails. This time limit is configured with the hdds.datanode.disk.check.timeout parameter. The default value is 10 minutes.

Handling Volume Failures

When a volume is marked failed, Ozone no longer uses it and triggers replication of the data from existing copies on other datanodes. After the issue on the failed volume is corrected, restart the datanode to detect the new volume.

Datanodes continue to run until they have no healthy data volumes or metadata volumes remaining. If you want the datanodes to shut down after a specified number of volume failures, set hdds.datanode.failed.data.volumes.tolerated or hdds.datanode.failed.metadata.volumes.tolerated to a positive number. If the set number of volume failures is crossed, the datanode shuts down automatically.