What's New in Ozone

Cloudera Runtime 7.3.2.10000 SP1

Ozone Disk Balancer

Ozone includes an intra-node Disk Balancer tool that automatically detects storage imbalances and redistributes data evenly across all disks within a single DataNode.

This feature provides the following key capabilities:

• Intra-node load balancing: Eliminates I/O hotspots and performance bottlenecks by balancing utilization across individual local volumes.
• Automated skew mitigation: Resolves disk space imbalances caused by adding new disks, replacing failed drives, or processing bulk block deletions.

For more information, see Disk Balancer.

Ozone Cluster Capacity view in Recon UI

The Cluster Capacity dashboard in Ozone Recon provides a centralized view of storage utilization across your entire cluster. Moving from high-level physical capacity to individual DataNode diagnostics, it tracks usable space, logical data distribution, pre-allocated storage, and pending deletion lifecycles to help you monitor storage health and balance capacity effectively. For more information, see Cluster Capacity.

HttpFS proxy user support in Ozone

Ozone HttpFS Gateway supports proxy user impersonation, enabling authorized service accounts such as Hive and Data Explorer to perform operations on behalf of other users. For more information, see Configuring HttpFS proxy users.

Configurable replica counts for diverged quasi-closed containers

Ozone now features configurable replica counts for diverged quasi-closed containers, enhancing data resilience.

Previously, Ozone maintained a fixed two copies for each origin of a diverged quasi-closed container. With this update, you can customize replica counts to allocate more copies to the origin with the highest Block Commit Sequence ID (BCSID). Increasing the replication factor for the healthiest, most up-to-date origin significantly reduces the risk of data loss during node failures.

The following new dynamic configuration keys are now available to manage these replica counts:

Table 1. Replica count configuration keys

Configuration key	Description	Default value
hdds.scm.replication.quasi.closed.stuck.best.origin.copies	Defines the number of copies to maintain for the origin with the highest BCSID among healthy replicas.	3
hdds.scm.replication.quasi.closed.stuck.other.origin.copies	Defines the number of copies to maintain for all other origins of the same container.	2

Enhanced container health monitoring in Storage Container Manager (SCM)

Ozone Storage Container Manager (SCM) includes a new ContainerHealthState enumeration that tracks and captures container replication statuses within container information. This enhancement enables efficient querying and monitoring of container health conditions and provides improved visibility into the replication state and overall health of containers across the cluster.

Recon container health monitoring enhancements

Ozone Recon includes an improved container health monitoring mechanism through ContainerHealthTaskV2. This update ensures that Recon uses the SCM Replication Manager as the single authoritative source of truth for container health status.

Key improvements include:

• Improved data consistency: Eliminates discrepancies between Recon and SCM by aligning Recon's health view with SCM's authoritative state.
• Enhanced tracking: Ensures containers known to SCM but not yet synchronized to Recon are accurately tracked for health issues.

Ozone Recon OM synchronization metrics for staging and queue-based processing

Ozone now includes metrics at various stages of the staging and queue-based events processing pipeline for Recon OM (Ozone Manager) synchronization. These metrics provide fine-grained observability into the Recon OM synchronization workflow, enabling operators to monitor the health and performance of the staging DB-based reprocessing of the OM DB.

Improved metadata scalability for Recon tasks

Ozone Recon now stores metadata for specific Ozone Manager (OM) background tasks in RocksDB instead of SQL Derby. This staging-based architecture improves throughput, scalability, and resource efficiency during heavy metadata synchronization.

Key Improvements include:

• Enhanced performance: Reduces overhead and locking issues during concurrent metadata processing.
• Better scalability: Handles large-scale datasets for file size distributions and table insights without SQL database bottlenecks.
• Targeted metadata migration: Applies specifically to file size counts (FILE_COUNT_BY_SIZE) and global table statistics (GLOBAL_STATS).

Accelerated replicated disk usage calculations in Recon

Recon optimizes replicated disk usage calculations by introducing a dedicated replicatedSizeOfFiles field in the Namespace Summary (NSSummary), eliminating recursive directory tree traversals.

Key Improvements include:

• Faster UI performance: Accelerates page loading times and improves responsiveness when viewing disk usage metrics in the Recon UI.
• Cluster scalability: Prevents performance bottlenecks in large clusters by eliminating heavy, recursive lookups.
• Accurate capacity tracking: Delivers precise storage metrics, including replication overhead, for effective cluster management and capacity planning.

Enhanced storage capacity visibility in DataNodes

The Storage Capacity table on the Ozone Recon DataNodes page now includes File System (FS) Reserved Space alongside existing capacity metrics.

Key Improvements include:

• Accurate capacity tracking: Displays reserved disk space directly within storage metrics to distinguish system-reserved space from total usable capacity.
• Improved cluster visibility: Helps you monitor actual available storage and prevent unexpected capacity constraints across DataNodes.

Ozone supports configurable RocksDB bottommost level compaction strategy

You can now configure the BottommostLevelCompaction strategy for RocksDB in Ozone. Previously hardcoded to kForce, this setting is now adjustable through configuration and CLI commands. This allows for better optimization of background compaction processes by choosing between kSkip, kForce, or kIfHaveCompactionFilter based on your cluster's performance needs.

Ozone volume health check toggle

Ozone introduces a configuration switch that allows you to enable or disable the RocksDB checks during volume health checks. This provides administrators with the ability to customize health check validations based on their specific environment requirements.

Added new container-to-key mapping tool

A new debugging tool allows you to quickly map containers to their associated keys. Input a list of containers to retrieve all matching keys across different layout types:

• FSO (File System Optimized)
• OBS (Object Store)
• Open (Uncommitted keys)
• MPU (Multipart Upload keys)

This simplifies key tracking and makes troubleshooting container data faster and more efficient.

Ozone Storage Container Manager (SCM) Replication Manager supports configurable container sample limits

Ozone SCM Replication Manager allows administrators to dynamically configure the number of containers sampled in each iteration. Previously, SCM hardcoded this limit to 100 containers per state per iteration. This improvement provides greater flexibility for monitoring container health and managing cluster state in large-scale deployments without requiring a service restart.

This enhancement introduces the following configuration property:

• Property name: hdds.scm.replication.container.sample.limit
• Default value: 100
• Description: Defines the maximum number of containers in each state to sample per Replication Manager iteration.
• Dynamic update: Supported. You can apply changes dynamically at runtime using the ozone admin reconfig command without restarting SCM.

NonWritableNodes metric added to Storage Container Manager

The Storage Container Manager (SCM) introduces the NonWritableNodes metric, providing administrators with a consolidated count of DataNodes currently unable to accept new write operations.

A DataNode is included in this count if it meets any of the following conditions:

• The node is not simultaneously in both HEALTHY and IN_SERVICE states (for example, nodes in ENTERING_MAINTENANCE or IN_MAINTENANCE states).
• The node has reached its maximum container allocation limit.
• The node is unable to write data to existing containers.

This metric improves cluster observability by replacing complex checks with a single data point for monitoring write availability across the cluster.

Support for deleting empty QUASI_CLOSED containers

You can now delete QUASI_CLOSED containers when all available replicas are completely empty, regardless of their current lifecycle state. To protect against data loss from lagging nodes, if a previously unseen or offline replica later reconnects with a higher Block Container Sequence ID (BCSID), the container is automatically restored to its original state. This enhancement enables clusters to safely reclaim metadata and storage space from empty, unclosed containers while guaranteeing that the system can still recover and re-replicate data if a newer replica emerges.

Ozone Container Balancer expands support for non-standard containers

The Ozone Container Balancer can now move containers that contain non-closed replicas (such as QUASI_CLOSED), provided the container meets minimum closed replica requirements. Previously, the balancer only processed containers where all replicas were strictly in a CLOSED state. This improvement optimizes data distribution and cluster balance in scenarios where certain replicas have not fully transitioned to a closed state despite the container being finalized in Storage Container Manager (SCM).

This feature is disabled by default. To enable it, set hdds.container.balancer.include.non.standard.containers configuration to true.

When enabled, the Container Balancer can process the following container states:

• CLOSED and OVER_REPLICATED containers with CLOSED replicas: Includes containers that meet the minimum required number of closed replicas based on their replication configuration (such as Ratis or Erasure Coding schemes).
• CLOSED and OVER_REPLICATED containers with QUASI_CLOSED replicas: Includes containers with QUASI_CLOSED replicas, provided the minimum required closed replicas are present.
• QUASI_CLOSED and OVER_REPLICATED containers: Includes containers where all replicas are QUASI_CLOSED and the source replica is non-empty.
• QUASI_CLOSED and HEALTHY containers: Includes containers where all replicas are QUASI_CLOSED and the source replica is non-empty.

The ozone container info command now displays Block Commitment Sequence ID (BCSID)

The ozone container info command now includes the BCSID directly in its standard console output alongside replica details. Previously accessible only in detailed JSON output, surfacing the BCSID in the default summary view improves container observability, enabling administrators to quickly verify container states and troubleshoot replica consistency without parsing complex JSON data.

Volume health counters added to ozone admin datanode list command

The ozone admin datanode list command now displays total and healthy volume counters for each DataNode directly in the cluster-wide output. By surfacing a clear healthy-to-total volume ratio (for example, 11/12), this enhancement improves cluster observability, enabling administrators to quickly identify disk degradation across the cluster and run detailed diagnostic queries like ozone admin datanode info <datanode-id> only when a volume discrepancy is found.

New latency breakdown metrics for key creation

Ozone now provides granular latency breakdown metrics for key creation operations, capturing execution time across each major stage of the creation process. By tracking individual sub-operations, these new metrics improve cluster observability, enabling administrators to quickly isolate sources of delay, pinpoint performance bottlenecks, and troubleshoot storage layer slowdowns.

Ozone SCM safemode status command supports checking all SCM nodes

Ozone enhances Storage Container Manager (SCM) safemode monitoring by adding the --all flag to the ozone admin safemode status command. This update allows you to view the safemode status of all SCM nodes in a High Availability (HA) setup simultaneously, making it easier to verify SCM state during startup and maintenance windows.

Parallel task processing for Ozone Recon

Ozone Recon now supports parallel processing for Ozone Manager (OM) RocksDB table iterations, significantly reducing database rebuild and bootstrap times on large clusters. Background operations including container-key mapping, file-size counting, and table insights now execute in parallel.

Administrators can fine-tune parallelism, thread allocation, and memory usage using four new configuration properties:

Property	Default value	Description
ozone.recon.task.reprocess.max.iterators	5	Sets the maximum number of iterators used during task reprocessing.
ozone.recon.task.reprocess.max.workers	20	Sets the maximum number of worker threads for parallel processing.
ozone.recon.task.reprocess.max.keys.in.memory	2000	Sets the maximum number of keys held in memory during reprocessing.
ozone.recon.filesizecount.flush.db.max.threshold	200000	Sets the record threshold for flushing file-size count data to the database.

Enhanced Ratis configuration flexibility through hdds.ratis. prefix

Ozone now allows administrators to set arbitrary Apache Ratis parameters directly within ozone-site.xml (or through Cloudera Manager safety valves) using the new hdds.ratis. prefix. When reading configuration files, Ozone automatically strips this prefix and passes the remaining key directly to the underlying Ratis server or client.

This enhancement simplifies administration and enables immediate, fine-grained performance tuning for the Ratis consensus layer such as RPC timeouts, retry policies, and log management without requiring Ozone code updates to expose new properties.

Example,

To configure a Ratis property such as raft.server.rpc.timeout.min, prepend hdds.ratis. to the property name in ozone-site.xml:

<property>
  <name>hdds.ratis.raft.server.rpc.timeout.min</name>
  <value>200ms</value>
</property>

Enhanced read scalability for Ozone Manager High Availability

Ozone Manager High Availability (OM HA) now supports serving read requests from follower nodes by leveraging Ratis Linearizable Reads. By distributing read operations across all OM nodes in the cluster rather than routing them exclusively through the leader, this feature significantly boosts read throughput for read-heavy workloads while freeing up leader capacity for write operations. Linearizable reads ensure that follower nodes maintain strict data consistency, delivering fully up-to-date metadata without stale read risks.

Increased default size for Ozone Manager Ratis Snapshots

The default value for ozone.om.ratis.snapshot.max.total.sst.size configuration property has been increased from 100 MB to 10 GB. This change provides sufficient capacity for larger datasets out of the box, improving Ozone Manager (OM) Ratis snapshot stability and performance in high-scale clusters while reducing the need for manual configuration tuning.

Periodic logging for Ozone SCM Safemode status

Storage Container Manager (SCM) now logs safemode rule status at regular intervals while in safemode. This provides continuous visibility into safemode progress and makes it easier to diagnose delayed safemode exits from logs.

The logging interval is configurable through the following configuration property:

• Property : hdds.scm.safemode.log.interval
• Default value : 1 minute

What's logged:

• Overall state: SCM's safemode status.
• Prechecks: Overall precheck completion status and count of validated precheck rules.
• Exit rules: Progress for each individual rule required to exit safemode.

New Grafana dashboard for Ozone SCM Safemode tracking

A new Grafana dashboard, Ozone - SCM Safemode, is now available to help you monitor the safemode status and exit progress. This is particularly useful in large clusters where safemode exit can take significant time.

The dashboard includes:

• Rule tracking: Individual charts for each safemode rule, showing target vs actual values to track progress toward exiting safemode.
• Status indicator: A visual safemode status timeline In Safemode: Displayed in red (binary value 1).
• Exited Safemode: Displayed in green (binary value 0).

Cloudera Runtime 7.3.2.100 CHF 1

Upgrade advisory for Ozone

Cloudera advises against installing or upgrading Ozone in Cloudera Runtime to 7.3.2.100 CHF. If you are currently running Ozone on Cloudera Runtime 7.3.2.0, Cloudera recommends skipping the 7.3.2.100 release and waiting to upgrade for an upcoming Cloudera Runtime 7.3.2.10000 (Service Pack 1).

Cloudera Runtime 7.3.2

Cloudera Runtime 7.3.2 introduces new features of Ozone and includes all service packs and cumulative hotfixes from 7.3.1.100 through 7.3.1.706. For a comprehensive record of all updates in Cloudera Runtime 7.3.1.x, see New Features.

Cloudera Storage Optimizer

Cloudera Storage Optimizer is an intelligent data lifecycle management feature that automatically reduces storage usage by converting infrequently accessed data from replicated storage (RATIS 3×) to space-efficient erasure coding (EC). It analyzes access patterns and applies configurable policies to identify and convert cold data, reducing the storage footprint by 45% to 60% while maintaining data durability and availability. The benefit is additional usable capacity for the same licensed capacity. For more information, see Cloudera Storage Optimizer documentation.

Storage container reconciliation

By design, container replicas are identical and contain the same data. Container reconciliation is a repair mechanism that allows administrators to detect and repair container replicas that are corrupted or have otherwise diverged from each other. The data within each replica is summarized as a single data checksum. If the replica data diverges, these checksums will differ. After reconciliation, the replicas will contain the same data with matching checksums. This data checksum remains constant even as data is deleted from the container in the background. For more information, see Storage container reconciliation documentation.

Support for G1 Garbage Collector (G1GC) with JDK 17

When running Ozone on JDK 17, all Ozone server processes now default to the G1 Garbage Collector (G1GC). No action is required for most deployments.

Configuration Changes for the FIPS-compliant SASL Changes

Cloudera supports DIGEST-SHA, a new SASL mechanism, to replace DIGEST-MD5. DIGEST-SHA uses SHA256 and AES instead of MD5 and DES for message digest and encryption, respectively. As a result, DIGEST-SHA is fully FIPS-compliant. For more information, see Step 1: Prepare hosts.

Moved the port ozone.prometheus.http-port from 9094 to 9096

The port ozone.prometheus.http-port is moved from 9094 to 9096 to avoid Ozone Prometheus port colliding with HBase thrift server if they are on the same host.

Added new Ozone canaries

To configure the threshold value, the following new Ozone canaries are added along with the relevant configuration property that can be used to disable them:

• Ozone SCM heap memory canary: ozone_scm_heap_canary_enabled
• Ozone OM heap memory canary: ozone_om_heap_canary_enabled
• Ozone Datanode failed volumes canary: ozone_datanode_failed_volume_canary_enabled

Added role-specific Java options

Previously, Ozone Java options (JAVA_OPTS) were configured through a single shared setting. This change introduces separate Java option parameters for each Ozone service role, allowing more granular configuration.

The following role-specific parameters are now available:

• Storage Container Manager (SCM): ozone_scm_java_opts
• Ozone Manager (OM): ozone_om_java_opts
• Datanode (DN): ozone_dn_java_opts
• S3 Gateway (S3G): ozone_s3g_java_opts
• Recon: ozone_recon_java_opts

Each parameter applies only to its corresponding service and overrides the shared JAVA_OPTS configuration where applicable.

Added Knox proxy user configuration for Ozone Recon

You can now restrict which hosts and groups Knox can impersonate when accessing Ozone Recon by using the following configuration properties:

• ozone.recon.http.auth.proxyuser.knox.hosts: Specifies the hosts from which Knox is allowed to impersonate users .
• ozone.recon.http.auth.proxyuser.knox.groups: Specifies the user groups that Knox is allowed to impersonate.

Added new verifier for container states

The ozone debug replicas verify command now includes the new --container-state verifier:.

ozone debug replicas verify --container-state <volume/bucket/path> -o <output.json>

This new verifier identifies keys mapped to problematic containers or replica states in SCM. It checks for the following states for every container in the replica:

• Containers: DELETED or DELETING
• Replicas: DELETED, UNHEALTHY, or INVALID

The command generates a JSON file containing details on keys, blocks, and replicas, along with a pass or fail status for each check. It supports the following options:

• --all-results: Displays all verification outcomes
• -o: Specifies the output file path
• --container-cache-size: Sets the number of containers stored in the in-memory cache for the --container-state check.
  The default value is 1 million containers (approximately 43MB of memory). The value must be greater than zero, otherwise, the system uses the 1 million default.

Enhanced volume health checks triggered by container scanner

When a container scanner (background or on-demand, data or metadata) marks a container as unhealthy due to corruption, the system now automatically triggers an on-demand scan of the associated volume. This enhancement helps detect and address broader volume-level issues that might exist beyond the affected container, improving overall data integrity and reliability.

New container health metrics

In Cloudera Base on premises 7.3.2.0 and higher releases, Ozone Recon now exposes additional metrics for the Container Health Task to improve observability of container state across the cluster. Recon tracks the number of containers in the following states:

• Missing
• Under-replicated
• Over-replicated
• Mis-replicated

Increased default Ratis segment size for Ozone Manager (OM) and Storage Container Manager (SCM)

To improve startup performance, the default values of the ozone.om.ratis.segment.size and ozone.scm.ha.ratis.segment.size configurations are increased from 4 MB to 64 MB.

• The ozone.om.ratis.segment.size configuration sets the size of the raft segment used by Apache Ratis on OM.
• The ozone.scm.ha.ratis.segment.size configuration sets the size of the raft segment used by Apache Ratis on SCM.

Renaming all legacy HDFS configuration keys and the corresponding Java constants to HDDS

All legacy configuration keys previously prefixed with dfs. (used by HDFS) are updated to use the hdds. prefix, aligning them with the HDDS component. Backward compatibility is maintained. The system will continue to recognize the old dfs. keys, but they are now deprecated. Deprecation notices and mappings are handled internally by adding DeprecationDelta in OzoneConfiguration.

Additionally, the corresponding Java constants for both the configuration key and the default value are renamed accordingly.

All deletion configurations are dynamically configurable without restart

The following deletion configurations are dynamically reconfigurable without requiring a restart:

• For Ozone Manager: ozone.thread.number.dir.deletion
• For Datanode:
  • ozone.block.deleting.service.interval
  • ozone.block.deleting.service.timeout

Support for callback on completed reconfiguration

The reconfiguration framework now supports a reconfiguration completed event. Components can now register callbacks that trigger after all properties in a task are processed. This allows the system to validate and apply interdependent settings automatically. You can access the status and values of the reconfigured properties using the ReconfigurableBase#getReconfigurationTaskStatus method within the callback.

Moved and enhanced Ratis Log Parsing command in Ozone

The ozone debug ratislogparser command is moved under the new ozone debug ratis parse subcommand. The parse subcommand now supports parsing different Ratis files, providing a more general and extensible interface.

The new --role=[om, scm, datanode] optional flag is added to specify which schema to use when parsing logs. By default, the schema is set to generic.

Enhanced output for ozone debug replicas chunk-info command

The ozone debug replicas chunk-info command now provides detailed information about the chunks associated with a given key. Previously, chunk-related details were only available when the --verbose flag was used. With this update, the command includes the BlockData information retrieved from the getBlock call, along with additional details about each replica. The JSON output structure is also improved for clarity.

Unified JSON output for ozone debug replicas verify command

The ozone debug replicas verify command now outputs JSON information for each key and the checks performed on it. As new verification checks are added, their results are included in the same JSON objects. By default, the output is skipped for keys that pass all specified checks, reducing unnecessary output.

Increased Ozone Manager (OM) default snapshot limit

The default value for the ozone.om.fs.snapshot.max.limit Ozone Manager property is increased, allowing support for up to 10,000 snapshots by default (previously 1,000). This change enables you to create and manage a significantly larger number of snapshots without requiring manual configuration changes.

Ozone Recon UI: Added new Cluster Health panel to the Grafana Dashboard

A new Cluster Health panel is added to the existing Ozone - Overall Metrics Grafana dashboard. This panel displays the cluster growth rate, calculated dynamically using Prometheus metrics. By default, the panel displays the growth rate over the last 1 hour, but you can configure the time window as required.

Renamed Ozone configuration property ozone.compaction.service.enabled

The ozone.compaction.service.enabled configuration property is renamed to ozone.om.compaction.service.enabled. This change aligns with the Ozone configuration key conventions, as the property is used only by the Ozone Manager (OM).

Added bucket layout flag to freon rk command

Previously, the freon randomkeys (rk) command did not allow you to specify the bucket layout, always creating buckets with the default layout. With this improvement, a new option is introduced to set the bucket layout using the command, providing greater flexibility.

Enhanced DataNode listing with usage-based sorting

The ozone admin datanode list command now supports the new --most-used and --least-used CLI options. These options enable you to sort Datanodes based on their usage. When used, the command output includes additional details such as used space, total capacity, and percentage of space used for each Datanode. This enhancement helps you identify the most and least used Datanodes in your Ozone cluster.

Updated the Ozone SCM container placement policy description

The descriptions for the following Ozone configuration properties are corrected in the ozone-default.xml file to clarify that the default placement policy is SCMContainerPlacementRackAware, instead of SCMContainerPlacementRandom:

• ozone.scm.container.placement.impl
• ozone.scm.container.placement.ec.impl

Additionally, if the ozone.scm.container.placement.impl property is left unset, the code fallback logic now uses the SCMContainerPlacementRackAware policy to match the default value in the ozone-default.xml file.

Enhanced container creation command with replication options

The ozone admin container create command now supports additional replication configurations. Previously, containers were limited to a replication factor of ONE and a replication type of STANDALONE. Administrators can now specify replication types (RATIS and EC), replication factors, and algorithms when creating containers using the following parameters:

ozone admin container create --replication=<replication> --replication-type=<type>