What's New in Ozone

Cloudera Runtime 7.3.2

Cloudera Runtime 7.3.2 introduces new features of Ozone and includes all service packs and cumulative hotfixes from 7.3.1.100 through 7.3.1.706. For a comprehensive record of all updates in Cloudera Runtime 7.3.1.x, see New Features.

Cloudera Storage Optimizer

Cloudera Storage Optimizer is an intelligent data lifecycle management feature that automatically reduces storage usage by converting infrequently accessed data from replicated storage (RATIS 3×) to space-efficient erasure coding (EC). It analyzes access patterns and applies configurable policies to identify and convert cold data, reducing the storage footprint by 45% to 60% while maintaining data durability and availability. The benefit is additional usable capacity for the same licensed capacity. For more information, see Cloudera Storage Optimizer documentation.

Storage container reconciliation

Generally, container replicas are identical in nature and contain the same data. Container reconciliation is a repair mechanism that allows administrators to detect container replicas that are corrupted or have otherwise diverged from each other for any reason, and repair these replicas, making them identical again. The data within each replica is summarized as a single data checksum. When the data within the replicas diverges, the replicas will have different data checksums. After reconciling the replicas, they will each contain the same data and each replica's data checksums will match the others. This data checksum is designed to remain constant even as data is deleted from the container in the background. For more information, see Storage container reconciliation documentation.

Support for G1 Garbage Collector (G1GC) with JDK 17

When running Ozone on JDK 17, all Ozone server processes now default to the G1 Garbage Collector (G1GC). No action is required for most deployments.

Configuration Changes for the FIPS-compliant SASL Changes

Cloudera supports DIGEST-SHA, a new SASL mechanism, to replace DIGEST-MD5. DIGEST-SHA is similar to DIAGEST-MD5 except that DIGEST-SHA uses SHA256 and AES instead of MD5 and DES for message digest and encryption, respectively. As a result, DIGEST-SHA uses only FIPS-compliant algorithms. For more information, see Step 1: Prepare hosts.

Moved the port ozone.prometheus.http-port from 9094 to 9096

The port ozone.prometheus.http-port is moved from 9094 to 9096 to avoid Ozone Prometheus port colliding with HBase thrift server if they are on the same host.

Added new Ozone canaries

To configure the threshold value, the following new Ozone canaries are added along with the relevant configuration property that can be used to disable them:

• Ozone SCM Heap memory canary: ozone_scm_heap_canary_enabled
• Ozone OM Heap memory canary: ozone_om_heap_canary_enabled
• Ozone Datanode failed volumes canary: ozone_datanode_failed_volume_canary_enabled

Added role-specific Java options

Previously, Ozone Java options (JAVA_OPTS) were configured through a single shared setting. This change introduces separate Java option parameters for each Ozone service role, allowing more granular configuration.

The following role-specific parameters are now available:

• Storage Ccontainer Manager (SCM): ozone_scm_java_opts
• Ozone Manager (OM): ozone_om_java_opts
• DataNode (DN): ozone_dn_java_opts
• S3 Gateway (S3G): ozone_s3g_java_opts
• Recon: ozone_recon_java_opts

Each parameter applies only to its corresponding service and overrides the shared JAVA_OPTS configuration where applicable.

Added Knox proxy user configuration for Ozone Recon

Following new configuration properties to control Knox proxy user (impersonation) access to Ozone Recon are introduced:

• ozone.recon.http.auth.proxyuser.knox.hosts: Specifies the hosts from which Knox is allowed to impersonate users when accessing Recon.
• ozone.recon.http.auth.proxyuser.knox.groups: Specifies the user groups that Knox is allowed to impersonate when accessing Recon.

These settings restrict which hosts and user groups are permitted for Knox-based impersonation in Recon.

Added new verifier for container states

The ozone debug replicas verify command now includes a new verifier: --container-state.

ozone debug replicas verify --container-state <volume/bucket/path> -o <output.json>

This new verifier checks the container state in the Storage Container Manager (SCM) for every container in a replica. It identifies keys mapped to containers in problematic states, such as:

• Containers: DELETED or DELETING
• Replicas: DELETED, UNHEALTHY, or INVALID

The command generates a JSON file containing details on keys, blocks, and replicas, along with a pass/fail status for each check. It supports the following options:

• --all-results: Displays all verification outcomes
• -o: Specifies the output file path
• --container-cache-size: A new option that sets the number of containers stored in the in-memory cache for the --container-state check.
  The default value is 1 million containers (approximately 43MB of memory). The value must be greater than zero; otherwise, the system uses the 1 million default.

Enhanced volume health checks triggered by container scanner

When a container scanner (background or on-demand, data or metadata) marks a container as unhealthy due to corruption, the system now automatically triggers an on-demand scan of the associated volume. This enhancement helps detect and address broader volume-level issues that may exist beyond the affected container, improving overall data integrity and reliability.

New container health metrics

Starting from Cloudera Base on premises 7.3.2.0 release, Ozone Recon now exposes additional metrics for the Container Health Task to improve observability of container state across the cluster. Recon tracks the counts of:

• Missing containers
• Under-replicated containers
• Over-replicated containers
• Mis-replicated containers

Increased default Ratis segment size for Ozone Manager (OM) and Storage Container Manager (SCM)

To improve startup performance, the default values of ozone.om.ratis.segment.size and ozone.scm.ha.ratis.segment.size configurations are increased from 4 MB to 64 MB.

• The ozone.om.ratis.segment.size configuration sets the size of the raft segment used by Apache Ratis on OM.
• The ozone.scm.ha.ratis.segment.size configuration sets the size of the raft segment used by Apache Ratis on SCM.

Renaming all legacy HDFS configuration keys and the corresponding Java constants to HDDS

All legacy configuration keys previously prefixed with dfs. (used by HDFS) have been updated to use the hdds. prefix, aligning them with the HDDS component. Backward compatibility is maintained. The system will continue to recognize the old dfs. keys, but they are now deprecated. Deprecation notices and mappings are handled internally by adding DeprecationDelta in OzoneConfiguration.

Additionally, the corresponding Java constants for both the configuration key and the default value are renamed accordingly.

All deletion configurations are dynamically configurable without restart

Starting from Cloudera Base on premises 7.3.2.0 release, the following deletion configurations are dynamically reconfigurable without requiring a restart:

• For Ozone Manager: ozone.thread.number.dir.deletion
• For DataNode:
  • ozone.block.deleting.service.interval
  • ozone.block.deleting.service.timeout

Support callback on completed reconfiguration

The reconfiguration framework now supports a reconfiguration completed event. Components can now register callbacks that trigger after all properties in a task are processed. This allows the system to validate and apply interdependent settings automatically. The status and values of the reconfigured properties are accessible through ReconfigurableBase#getReconfigurationTaskStatus within the callback.

Moved and enhanced Ratis Log Parsing command in Ozone

The ozone debug ratislogparser command has been moved under the new ozone debug ratis parse subcommand. The parse subcommand now supports parsing different Ratis files, providing a more general and extensible interface.

A new optional flag --role=[om, scm, datanode] has been added to specify which schema should be used when parsing logs. By default, the schema is set to generic.

Enhanced output for ozone debug replicas chunk-info command

The ozone debug replicas chunk-info command now provides detailed information about the chunks associated with a given key. Previously, chunk-related details were only available when the --verbose flag was used. With this update, the command includes the BlockData retrieved from the getBlock call, along with additional details about each replica. The JSON output structure has also been improved for clarity.

Unified JSON output for ozone debug replicas verify command

The ozone debug replicas verify command now outputs JSON information for each key and the checks performed on it. As new verification checks are added, their results will be included in the same JSON objects. By default, the output is skipped for keys that pass all specified checks, reducing unnecessary output.

Increased Ozone Manager (OM) default snapshot limit

The default value for the Ozone Manager property ozone.om.fs.snapshot.max.limit has been increased, allowing support for up to 10,000 snapshots by default (previously 1,000). This change enables users to create and manage a significantly larger number of snapshots without requiring manual configuration changes.

Ozone Recon UI: Added new Cluster Health panel to the Grafana Dashboard

A new Cluster Health panel has been added to the existing Ozone - Overall Metrics Grafana dashboard. This panel displays the cluster growth rate, calculated dynamically using Prometheus metrics. By default, the panel shows the growth rate over the last 1 hour, but users can configure the time window as needed.

Renamed Ozone configuration property ozone.compaction.service.enabled

The configuration property ozone.compaction.service.enabled has been renamed to ozone.om.compaction.service.enabled. This change aligns with Ozone's configuration key conventions, as the property is used only by the Ozone Manager (OM).

Added bucket layout flag to freon rk command

Previously, the freon randomkeys (rk) command did not allow users to specify the bucket layout, always creating buckets with the default layout. With this improvement, a new option has been introduced to set the bucket layout through the command, providing greater flexibility.

Enhanced DataNode listing with usage-based sorting

The ozone admin datanode list command now supports two new CLI options: --most-used and --least-used. These options enable users to sort datanodes based on their usage. When used, the command output includes additional details such as used space, total capacity, and percentage of space used for each datanode. This enhancement makes it easier to identify the most and least utilized datanodes in your Ozone cluster.

Updated the Ozone SCM Container Placement policy description

The descriptions for the following Ozone configuration properties have been corrected in the ozone-default.xml to clarify that the default placement policy is SCMContainerPlacementRackAware, not SCMContainerPlacementRandom:

• ozone.scm.container.placement.impl
• ozone.scm.container.placement.ec.impl

The code fallback for an unset ozone.scm.container.placement.impl has also been updated to use SCMContainerPlacementRackAware, similar to the ozone-default.xml.

Enhanced container creation command with replication options

The ozone admin container create command has been improved to support additional replication configurations. Previously, containers could only be created with a replication factor of ONE and a replication type of STANDALONE. Administrators can now specify replication types (RATIS and EC), replication factors, and algorithms when creating containers.

ozone admin container create --replication=<replication> --replication-type=<type>