Behavioral Changes in Ozone

Cloudera Runtime 7.3.2.10000 SP1

Introduction of DiskBalancer for Ozone DataNodes

Previous behavior:

Previously, Ozone DataNodes lacked an internal mechanism to balance data across multiple disks within a single node, leading to uneven disk utilization and storage inefficiency.

New behavior:

Each DataNode now includes a DiskBalancer to balance data evenly across its internal disks. This feature is disabled by default and can be enabled by setting hdds.datanode.disk.balancer.enabled configuration property to true. For more information, see Disk Balancer.

Clarification and separation of --all and --all-status options in ozone sh snapshot lsDiff command.

Previous behavior:

Previously, the --all option was renamed to --all-status, which unintentionally exposed an underlying --all option. This caused confusion as the distinction between removing pagination limits and filtering by job status was not clearly documented or implemented, leading to inconsistent behavior in scripts.

New behavior:

Now, the options are clearly distinguished and documented:

• --all (or -a): Removes the pagination limit and lists every matching result.
• --all-status: Lists all snapshot diff jobs regardless of their current status (example, queued, in_progress, done, failed, rejected).

Improved Ozone Recon bootup time by making failed delta task reinitialization asynchronous.

Previous behavior:

Previously, during startup, Ozone Recon would perform a blocking call to reinitialize any delta tasks that had failed in a previous run. This synchronous process delayed the overall bootup time of Recon, especially if there were multiple failed tasks to process.

New behavior:

Now, the reinitialization of failed delta tasks at startup is handled as an asynchronous event. By moving these calls to a background queue, Recon can complete its bootup process more quickly without waiting for the reinitialization tasks to finish.

Reserved space added to DataNode capacity tooltip

Previous behavior:

Previously, the capacity tooltip in the DataNodes tab did not display the reserved space, making it difficult to distinguish overall filesystem capacity from the actual capacity available to Ozone.

New behavior:

Now, the DataNode capacity tooltip now explicitly displays the Reserved Space value, clearly showing the difference between total filesystem capacity and storage allocated to Ozone.

Dedicated logging for Ozone Manager deletion services and SnapDiff workers

Previous behavior:

Previously, deletion service and Snapshot Diff (SnapDiff) worker logs were routed to the main Ozone Manager log file (ozone-om.log), making it difficult to isolate and troubleshoot garbage collection or snapshot activities.

New behavior:

Deletion-related logs are now separated from ozone-om.log. All deletion service logs are consolidated into a dedicated rolling log file (om-gc.log).

Ozone snapshot rename feature disabled by default

Previous behavior:

Previously, the snapshot rename feature was enabled by default, allowing users to rename snapshots despite potential instability that could cause snapshots to become unfindable.

New behavior:

Now, the snapshot rename feature is disabled by default to prevent data integrity issues. A new configuration property, ozone.om.snapshot.rename.allowed, controls this feature and defaults to false. To enable snapshot renames, administrators must explicitly set this property to true in the Ozone Manager configuration.

Ozone snapshot metadata now records successful defragmentation timestamps

Previous behavior:

Previously, the lastDefragTime field in snapshot metadata YAML files remained unpopulated, leaving administrators without a direct way to verify when a snapshot was last defragmented.

New behavior:

The lastDefragTime field is now automatically updated whenever a snapshot database defragmentation process completes successfully, providing clear visibility into snapshot maintenance status.

Locking and new checkpoint endpoint added to Ozone Manager bootstrap process

Previous behavior:

Previously, the Ozone Manager (OM) bootstrap process did not lock snapshotted RocksDB databases. Active transactions could occur on the snapshot database during bootstrap, creating a race condition that resulted in database corruption on follower nodes and bootstrap failures.

New behavior:

Now, the OM bootstrap process uses locks to prevent race conditions and protect snapshot database integrity. Additionally, clients running Apache Ozone version 2.2.0 or later now use a new checkpoint endpoint (/v2/dbCheckpoint), while older clients continue using the legacy endpoint (/dbCheckpoint). This functionality is enabled by default and can be controlled using the ozone.om.db.checkpoint.use.inode.based.transfer configuration property.

Ozone snapshot deep cleaning enabled by default

Previous behavior:

Previously, the ozone.snapshot.deep.cleaning.enabled configuration property defaulted to false. When a snapshot was deleted, keys marked for deletion were perpetually moved to the next active snapshot in the chain rather than being purged, preventing storage reclamation and causing unreferenced data to accumulate.

New behavior:

The ozone.snapshot.deep.cleaning.enabled property now defaults to true. Deleted keys progress through the snapshot chain until they are fully purged, allowing the system to reclaim storage automatically without manual configuration.

Explicit error messages for invalid Ozone bucket and volume names

Previous behavior:

Previously, creating an Ozone bucket or volume with an invalid name (such as one containing an underscore) generated a generic error message that did not specify why validation failed.

Example,

INVALID_BUCKET_NAME Invalid bucket name: bucket_1

New behavior:

The Ozone CLI now returns a detailed error message that explicitly identifies the unsupported character causing the validation failure.

Example,

INVALID_BUCKET_NAME bucket name has an unsupported character : _

Ozone create, write, and delete operations now require Read permission

Previous behavior:

Creating, writing, or deleting an Ozone key did not require READ permission. If a user lacked access, the system returned a WRITE or DELETE permission denied error.

New behavior:

The Ozone Manager now checks permissions using the getFileStatus() API before creating, writing, or deleting. Users must have READ permission (or SELECT permission for Hive resources when Ranger RMS is enabled) for these actions to succeed. Unauthorized requests now return a READ permission denied error. This aligns with AWS S3 behaviour, preventing unauthorized users from checking whether a resource exists.

Cloudera Runtime 7.3.2.100 CHF 1

There are no behavioral changes in this release.

Cloudera Runtime 7.3.2

Migration of the ozone debug recover command to the admin om lease recover Ozone admin CLI command

The ozone debug recover command, used for lease recovery, now resides under the admin om lease recover Ozone admin CLI command.

Example,

# ozone admin om lease recover --path=ofs://<ozone.service.id>/hsyncvol/hsyncbuck/hsync/File_0.txt
Lease recovery SUCCEEDED on ofs://<ozone.service.id>/hsyncvol/hsyncbuck/hsync/File_0.txt

Changed functionality of the --all option in the ozone sh snapshot lsDiff command

Previous behavior:

Previously, the --all option was used to list snapshot diff jobs of any status.

New behavior:

Now, the --all option is used for listing limit. When the --all option is set, the listing limit is effectively set to Integer.MAX_VALUE. For the original --all behavior to list snapshot diff jobs, use the --all-status option instead.

Changed listing and deleting behavior at volume level

Previous behavior:

Previously, recursive listing and deleting a volume was supported using the ozone fs command that worked only if FSO bucket existed in volume.

New behavior:

Now, recursive listing and deleting a volume is supported only by using the ozone shell command and not the ozone fs command. The ozone shell command works for both OBS and FSO buckets inside a volume.

Example

• Recursive delete command

  # ozone fs -rm -R -skipTrash ofs://ozone1713158544/vol/

  Sample output

  24/04/15 13:12:47 WARN ozone.BasicRootedOzoneFileSystem: Recursive volume delete using ofs is not supported
  rm: Recursive volume delete using ofs is not supported. Instead use 'ozone sh volume delete -r -skipTrash -id <OM_SERVICE_ID> <Volume_URI>' command

• Recursive list command

  # ozone fs -ls -R ofs://ozone1713158544/vol/

  Sample output

  24/04/15 13:12:53 WARN ozone.BasicRootedOzoneFileSystem: Recursive root/volume list using ofs is not supported
  ls: Recursive list root/volume using ofs is not supported. Instead use 'ozone sh key list <Volume_URI>' command

Migration of the ozone debug chunkinfo API to the ozone debug replicas chunk-info API

Previous behavior:

Previously, the ozone debug chunkinfo API returned the following JSON response:

{
  "KeyLocations" : [ [ {
    "Locations" : {
      "files" : [ "<block_path>" ],
      "pipelineID" : "<pipeline_id>"
    },
    "Datanode-HostName" : "<host_name>",
    "Datanode-IP" : "<host_ip>",
    "Container-ID" : <id>,
    "Block-ID" : <blockid>
  },{},{}
]]}

New behavior:

Now, the ozone debug chunkinfo API is relocated to the ozone debug replicas chunk-info API and returns the following JSON response:

{
  "volumeName" : "<vol>",
  "bucketName" : "<buck>",
  "name" : "<key>",
  "keyLocations" : [ [ {
    "datanode" : {
      "hostname" : "<host_name>",
      "ip" : "<host_ip>",
      "uuid" : "<uuid>"
    },
    "file" : "<block_path>",
    "blockData" : {
      "blockID" : {
        "containerID" : <id>,
        "localID" : <localid>,
        "blockCommitSequenceId" : <bcsid>
      },
      "size" : <size>,
      "chunks" : [ {
        "offset" : <offset>,
        "len" : <len>,
        "checksums" : [ "<checksum>" ],
        "checksumType" : "<ctype>",
        "bytesPerChecksum" : <bpc>
      } ]
    }
  }, {}, {}
]]}

As part of this API update, the following keys are updated:

Table 1. JSON key mapping for the relocated ozone debug chunkinfo API

Key name	Key in the previous JSON	Key in the current JSON
Block path	data["KeyLocations"][0][0]["Locations"]["files"][0]	data["keyLocations"][0][0]["file"]
Container ID	"Container-ID"	blockData.blockID.containerID
Block ID	"Block-ID"	blockData.blockID.localID
Hostname	"Datanode-HostName"	datanode.hostname

The S3 Gateway web UI is moved to a new port

Previous behavior:

The S3 Gateway web UI was accessible on ports 9878 for HTTP and 9879 for HTTPS, just like S3 buckets.

New behavior:

The S3 Gateway web UI resides on new ports to avoid conflicts with S3 buckets. The new ports are 19878 for HTTP and 19879 for HTTPS.