Known Issues in Ozone

Known issues and technical limitations for Ozone are addressed in Cloudera Runtime 7.3.3.

Known issues identified in Cloudera Runtime 7.3.3

CDPD-125823: ListObjectsV2 incorrectly filters CommonPrefixes when StartAfter is before matching prefixes
7.3.3
When a client calls the S3 ListObjectsV2 API with the Prefix, Delimiter, and StartAfter parameters, the Ozone S3 Gateway returns an empty CommonPrefixes list even though matching prefixes exist and sort lexicographically after the specified StartAfter value.
  • Omit StartAfter when listing objects with Delimiter and Prefix.
  • List objects without Delimiter and derive prefixes on the client side.
  • If pagination is required, use the ContinuationToken from a prior listing, or specify a StartAfter value under the same Prefix that does not trigger the issue.
Apache JIRA: HDDS-17308
CDPD-126697: CompleteMultipartUpload accepts duplicate part numbers and creates an incomplete object
7.3.3
The Ozone S3 Gateway does not reject CompleteMultipartUpload requests when the part list contains duplicate part numbers (for example, [1, 1] with the same ETag). The operation returns a success response, but the committed object reflects the bytes of only one part (for example, 5 MiB instead of 10 MiB). Additionally, HeadObject reports a PartsCount that does not match the actual ContentLength, which causes data inconsistency.
Ensure that each part number appears exactly once and in ascending order with no gaps before calling CompleteMultipartUpload.
Apache JIRA: HDDS-17554
CDPD-126757: CompleteMultipartUpload calculates incorrect ETag and PartsCount when completing with a subset of uploaded parts
7.3.3
In Ozone S3, when a user completes a multipart upload using only a subset of uploaded parts, the object metadata does not match its actual content. While the object size and payload correctly reflect only the parts listed in the completion request, HeadObject incorrectly calculates PartsCount and the multipart ETag suffix using every part uploaded during the session. For example, if a session contains parts 1, 2, and 3, but completes using only parts 1 and 3, the system reports a PartsCount of 3 and an ETag suffix of -3 instead of the expected values of 2 and -2.
None
Apache JIRA: HDDS-17572
CDPD-126906: Ozone HTTPS endpoints reject TLSv1.3 by default
7.3.3
When TLS is enabled, the Ozone S3 Gateway and the web UIs of the Ozone roles accept only TLSv1.2 by default. Clients that require TLSv1.3 fail the TLS handshake with a protocol_version error. Clients that support both TLSv1.2 and TLSv1.3 are not affected.
In Cloudera Manager, add the following configuration to the Ozone Service Advanced Configuration Snippet (Safety Valve) for ozone-site.xml configuration, and restart the Ozone service:
<property>
  <name>ozone.ssl.enabled.protocols</name>
  <value>TLSv1.2,TLSv1.3</value>
</property>
Apache JIRA: HDDS-17542
CDPD-126954: S3 CreateMultipartUpload returns 500 InternalError when the key matches an existing directory in an FSO bucket
7.3.3
In a File System Optimized (FSO) bucket, calling CreateMultipartUpload with a key that matches an existing directory (for example, dir1 when dir1/ already exists) returns an HTTP 500 InternalError. Although the Ozone Manager rejects the request because it cannot write to a directory, the S3 Gateway fails to map the error code correctly and returns a generic internal error. This behavior causes clients, such as the AWS CLI, to retry the request repeatedly.
To work around this issue, perform the following actions:
  • Do not start a multipart upload on a key that matches an existing directory.
  • Verify whether a key is a directory before starting an upload by running the ozone sh key info /volume/bucket/key command and checking whether "file": false.
  • If the target path is an existing directory, use PutObject for that path, or write the multipart object under a new key that does not conflict with an existing directory.
Apache JIRA: HDDS-17658
CDPD-126993: S3 CreateMultipartUpload succeeds under a parent path that is a file, and subsequent UploadPart calls fail with 500 InternalError
7.3.3
In a File System Optimized (FSO) bucket, if a parent path is an existing file (for example, a file named parent), a CreateMultipartUpload request for a child path (for example, parent/child) incorrectly succeeds and returns an UploadId. Every subsequent UploadPart request for that upload fails with an HTTP 500 InternalError. Although the Ozone Manager detects the file system violation (NOT_A_FILE), the S3 Gateway returns a generic 500 error instead of a 400-level client error. The multipart upload remains in progress and cannot complete, leaving aborting the upload as the only option.
To work around this issue, perform the following actions:
  • Do not start a multipart upload under a key where a parent component is an existing file.
  • Before starting an upload, verify the path by using HeadObject or the ozone sh key info command.
  • If a CreateMultipartUpload request returns an UploadId for such a path, run abort-multipart-upload to clear the incomplete upload, and ensure that all parent components in the target path are directories.
Apache JIRA: HDDS-17644
CDPD-126995: S3 ListParts with max-parts=0 or a negative value returns HTTP 500 with no S3 error body
7.3.3
When a client calls the S3 ListParts operation with max-parts=0 or a negative value (for example, -1), the S3 Gateway returns an HTTP 500 Internal Server Error instead of a standard S3 error response. This issue occurs because the S3 Gateway forwards the invalid value to the Ozone client, which throws an unhandled IllegalArgumentException ("Max Parts Should be greater than zero"). Because the response lacks a valid S3 XML error body, clients such as the AWS CLI retry the request repeatedly, which results in multiple failed requests and generic error messages.
To work around this issue, use one of the following options:
  • Specify only positive values for the max-parts parameter (typically between 1 and 1000).
  • Omit the --max-parts parameter to use the default page size, and use the part-number-marker parameter for pagination if more parts exist.
Apache JIRA: HDDS-17645
CDPD-127247: UploadPartCopy does not enforce x-amz-copy-source-if-match and x-amz-copy-source-if-none-match preconditions
7.3.3
In Ozone S3, the UploadPartCopy operation fails to enforce ETag-based preconditions (x-amz-copy-source-if-match and x-amz-copy-source-if-none-match). While date-based preconditions work correctly, Ozone S3 ignores ETag conditions. Consequently, the copy operation succeeds and writes the part even when the source ETag does not match, such as after an overwrite of the source object. The completed multipart object then contains mixed data from different versions of the source object instead of returning the expected 412 PreconditionFailed error.
None
Apache JIRA: HDDS-17649

Known Issues identified before Cloudera Runtime 7.3.3

Known issues identified before Cloudera Runtime 7.3.3 include only unresolved issues from previous releases that continue to affect the Cloudera Runtime 7.3.3 base release.

CDPD-105709: Recon UI displays inconsistent Solr health status
7.3.3 and 7.3.2.10000
The Recon Overview page incorrectly displays SOLR is HEALTHY as an alert when the service functions normally. Additionally, other pages within the Recon UI report SOLR is UNHEALTHY despite Solr remaining in a healthy state.
None
Apache JIRA: HDDS-15571
CDPD-109005: Storage Container Manager (SCM) fails to update DatanodeDetails when only port configurations change
7.3.2.10000
When a DataNode re-registers with SCM or sends heartbeats after gaining new ports, SCM fails to update the stored DatanodeDetails in the node registry. SCM triggers an update only if the IP address, hostname, or software version changes. Consequently, SCM ignores port-only updates, which leads to stale metadata and potential failures in features that rely on the new ports.
None
CDPD-109002: Ozone DataStream writes fail after enabling Ratis DataStream on existing clusters
7.3.2.10000
When Ratis DataStream is enabled on an existing cluster, open pipelines retain stale DataNode details that lack the required DataStream port information. Consequently, Ozone client writes fail when they exceed the automatic threshold because the client cannot identify the necessary port for pipeline members. This occurs because SCM does not refresh member ports for existing open pipelines after the configuration change.
None
Apache JIRA: HDDS-15799
CDPD-99722: Container data checksum reverts to a stale value after successful reconciliation
7.3.2.0 and 7.3.2.10000
In Ozone, the BackgroundContainerDataScanner service reverts the container data checksum to a stale value even after a successful reconciliation repairs corrupt chunks. This occurs because the scanner builds a checksum tree in memory while reconciliation updates the container data on disk, causing the scanner to persist an outdated tree.
The issue resolves automatically during the next container scan cycle. To resolve the issue immediately, manually trigger container reconciliation from the command line.
Apache JIRA: HDDS-14936
CDPD-105115: DataNodes fail to communicate with SCM after restart
7.3.2.10000
In a SCM High Availability (HA) cluster, a restarting or failing-over follower SCM starts its DataNode protocol server before completing its Ratis log replay. This causes SCM to process DataNode container reports against an incomplete state, which leads to dropped replica locations. Consequently, if that follower becomes the leader, affected containers appear to have missing or empty replicas, and DataNode logs record errors such as Unable to clear disk, Failed to flush DB before close, and EOFExceptions.
This issue is temporary and typically resolves during the next report cycle when SCM rebuilds the replica list.
CDPD-119561: Ozone Manager (OM) becomes unresponsive during sustained snapshot operations
7.3.2.10000
The OM leader enters a deadlock state during heavy snapshot creation and deletion loads. This occurs when the Key Deleting Service holds a snapshot database lock while waiting for a transaction that cannot complete because the flush thread is blocked. As a result, the OM stops responding to client requests and snapshot operations time out.
  • Restart the affected OM to release the locks and restore service.
  • Limit snapshot deletion and heavy write activity to reduce the risk of recurrence.
  • Increase the ozone.om.unflushed.transaction.max.count configuration to lower the probability of the deadlock (monitor for increased memory usage).
Apache JIRA: HDDS-16122
CDPD-104229: Delay in deleting snapshot YAML file by OM Snapshot Local Data Manager
7.3.2.10000
A significant delay of approximately 20 to 30 minutes occurs in the deletion of snapshot YAML files by the OM Snapshot Local Data Manager after the system processes a snapshot deletion request. This delay happens because the current design requires the inDegree, which represents the number of descendant snapshots, to reach zero before the system removes the YAML file. This behavior impacts subsequent operations, such as bootstrap test cases, that rely on the timely cleanup of these files.
None
CDPD-119610: OM leader remains in LEADER_AND_NOT_READY state during Snapshot Defragmentation Bootstrap
7.3.2.10000
The Ozone Manager (OM) leader remains in the LEADER_AND_NOT_READY state after a follower node stops for a planned bootstrap. This issue occurs due to a deadlock between the snapshot installation thread and the double-buffer flush thread on the follower node. Consequently, the cluster loses its usable quorum, and all OM client RPC requests fail with an OMLeaderNotReadyException.
Restart the affected OM follower node to break the deadlock and restore normal cluster operations.
CDPD-119172: OM follower node bootstrap fails when OM leader changes
7.3.2.10000
In an Ozone Manager (OM) High Availability (HA) setup, the bootstrap process for a follower node fails if the OM leader node changes while the bootstrap is in progress. If the original leader stops (for example, through SIGTERM), the bootstrap thread attempts to download the database snapshot from the former leader and fails with a Connection refused error. When the thread subsequently attempts to connect to the newly elected leader, it receives a 503 Service Unavailable error because the new leader is not yet ready to serve requests.
To ensure a successful bootstrap, verify that two of the three Ozone Managers (OMs) are running before bootstrapping the third node.
CDPD-102216: Toggling to Old UI from the Cluster Capacity tab results in a 404 error
7.3.2.10000
The Cluster Capacity feature is available only in the new Recon UI. If you view the Cluster Capacity page in the new UI and click Switch to Old UI, the application attempts to load the same path in the legacy interface. Because this path does not exist in the old UI, a 404 Page Not Found error appears.
Manually navigate to the main dashboard if you encounter a 404 Page Not Found error after switching to the old UI. Use the new UI to view cluster capacity information, as the legacy interface does not support this feature.
Apache JIRA: HDDS-15148
CDPD-106805: Requests to Ozone S3 Gateway through HAProxy fail with 400 Bad Request during directory creation
7.3.2.10000
When using the S3 connector to access Ozone OBS buckets through an S3 Gateway behind HAProxy, requests fail with a 400 Bad Request error. This issue occurs because zero-byte directory-marker PUT requests (which include Expect: 100-continue headers) cause the S3 Gateway to skip reading the request body and return 200 OK immediately. This leaves unconsumed bytes on the wire and desynchronizes keep-alive connections. HAProxy then misinterprets subsequent requests, causing a parsing error.
Disable keep-alive connections in the HAProxy configuration by setting option httpclose and increasing the maxconn value (for example, to 4096).
CDPD-107197: The ozone admin datanode diskbalancer status command fails when passing a DataNode ID instead of an address
7.3.2.10000
The ozone admin datanode diskbalancer status command accepts only a DataNode network address or host string. When you use a valid datanodeUuid (DataNode ID) to identify the node, the command fails with an UnknownHostException or an invalid argument error. This behavior is inconsistent with other ozone admin commands that support both UUIDs and addresses for node identification.
To check the disk balancer status, use the DataNode network address or hostname instead of its UUID. Run the ozone admin datanode list command to locate the network address.
Apache JIRA: HDDS-15690
CDPD-100716: Reading keys results in a NO_REPLICA_FOUND error after SCM follower restart and leader transfer
7.3.2.10000
When a Storage Container Manager (SCM) follower restarts, it accepts DataNode container reports before catching up with the Ratis log. Processing these reports against a stale database triggers a NotLeaderException, causing the SCM to skip updating in-memory container replica locations. If this follower is later promoted to leader, reading keys fails with a NO_REPLICA_FOUND error due to missing replica metadata.
None
Apache JIRA: HDDS-14989
CDPD-95733: Snapshot pruning tests fail due to empty compaction SST backups in Ozone in Cloudera Runtime7.3.2
7.3.2
In Ozone in Cloudera Runtime 7.3.2, snapshot pruning tests fail because the Ozone Manager (OM) RocksDB Compaction Service configuration key requires a new name. The property key changes from ozone.compaction.service.enabled (used in version 7.1.9) to ozone.om.compaction.service.enabled in version 7.3.2. If you use the old configuration key, the compaction service remains disabled by default. This causes empty compaction SST backup directories and fails the pruning operations.
Update the configuration to use the new property key. Set ozone.om.compaction.service.enabled to true to enable the compaction service in version 7.3.2.
OPSAPS-76062: Exposing "ozone.replication" in Cloudera Manager configurations and assigning a default value
7.3.2
Exposing "ozone.replication in Cloudera Manager configurations and assigning a default value to it is causing it to override bucket replication configuration as a client side configuration even when you does not mean to set the client side configurations.
None
CDPD-73792: The new snapshot could not be found after renaming the old snapshot
7.3.2
The ozone sh snapshot rename command renames snapshots. It is a feature developed by the Apache Ozone community and is included in Cloudera Base on premises 7.3.1. However, it does not work properly, and Cloudera Base on premises does not support it.
None
Apache JIRA: HDDS-11384
CDPD-63350: Force deleting a FSO bucket and its contents while running rb --force from AWS S3 API is failing
7.3.2
Force deleting a File System Optimized (FSO) bucket and its contents while running the rb --force command from the AWS S3 API might fail with an error, because the S3 client sends individual delete requests for each key in the bucket. It might delete or fail to delete individual keys or directories, depending on the availability of leaf elements. It can completely delete the bucket only when all the keys or directories are cleaned up. If some keys are not deleted, the bucket will not be deleted.
Sample error message
# aws s3 rm s3://buck-fso --recursive
delete: s3://buck-fso/dir1/
delete: s3://buck-fso/dir1/dir2/
delete: s3://buck-fso/dir3/dir4/dir5/
delete: s3://buck-fso/dir3/dir4/
delete failed: s3://buck-fso/dir3/
# Rerun the same command again
# aws s3 rm s3://buck-fso --recursive
delete: s3://buck-fso/dir1/
delete: s3://buck-fso/dir3/
# To Confirm
# ozone sh key list s3v/buck-fso
[ ]
Run the rb --force command multiple times to completely clean up the keys and directories.
Apache JIRA: HDDS-9637
CDPD-98751: Mismatched Replicas tab in the Recon UI fails to display containers with inconsistent replica checksums
7.3.2
In the Ozone Recon UI, the Mismatched Replicas tab does not update or display containers when one or more replicas have differing checksums. Instead, they are displayed in the Under-Replicated tab.
The replica can be checked in the Under-Replicated tab, or can be cross-checked against API response.
CDPD-97512: Mismatch in Open Key count between Overview page and OM DB Insight in the Recon UI
7.3.2
In the Ozone Recon UI, the Open Key count on the Summary section of the Overview page might not match the count on the OM DB Insight > Open Key tab. Users might see different Open Key values for the same cluster across these two views.
Use the OM DB Insight > Open Key tab for the most accurate Open Key count.
CDPD-97376: Container replication counts mismatch in Recon UI
7.3.2
In the Ozone Recon UI, container replication counts, including Under-Replicated, Over-Replicated, and Mis-Replicated counts, differ between the updated and the legacy Container page for the same cluster.
Refer to the legacy Container page to view accurate replication counts.
CDPD-97311: Incorrect Creation Time and Modification Time displayed on the Namespace Usage page in the Recon UI
7.3.2
In the Ozone Recon UI, the Namespace Usage page displays incorrect Creation Time and Modification Time timestamps. These values do not accurately reflect the actual creation or last modification times or dates of the namespace, resulting in inaccurate metadata information.
Retrieve the correct timestamp values directly from the API response.
CDPD-97312: Mismatch between cluster State Container count and Container Summary totals
7.3.2
The Ozone Recon UI can display discrepancies between the Storage Container Manager (SCM) container count and the Recon container summary. This occurs because Recon does not synchronize all container states such as QUASI_CLOSED leading to inconsistent totals.
No workaround within the Ozone Recon UI. Use ozone admin container report CLI command to obtain the correct container counts for all states in the Ozone cluster.
CDPD-99248: After cdh upgrade, Ozone encounters failure while running the Finalize Upgrade for SCM on role Storage Container command
7.3.2
When finalizing an Ozone upgrade for the first time from Cloudera Manager, the Finalize Upgrade for SCM on role Storage Container command might fail with the following stderr message:
"Invalid response from Storage Container Manager.
Current finalization status is: FINALIZATION_IN_PROGRESS"

This error occurs even though finalization continues to run on the Storage Container Manager (SCM).

Ignore the failure in Cloudera Manager. Use the ozone admin scm finalizationstatus command to monitor progress and wait for the process to complete on SCM.
CDPD-98892: File Size Distribution bucket size range calculation is not correct
7.3.2
If large number of buckets exists within a volume, the Ozone Recon UI might display incorrect file size distribution bucket size range calculation in the File Size Distribution chart on the Insights page.
None
Apache JIRA: HDDS-14827
CDPD-93116: Ozone client hangs intermittently when disks are full
7.3.2

This hang is caused by continuous write retries that persist until the pipeline on the Datanode closes.

The Ozone client hangs for approximately five minutes when writing data to a Datanode if a disk full exception or other Datanode error occurs. This hang is caused by continuous write retries that persist until the pipeline on the Datanode closes.
Control the request retry behavior by setting the following configurations on the client side:
Table 1. Client-side retry request configurations
Configuration Recommended value
hdds.ratis.raft.client.rpc.request.timeout 30s
hdds.ratis.client.multilinear.random.retry.policy 1s, 1
hdds.ratis.client.exponential.backoff.max.sleep 5s
hdds.ratis.client.exponential.backoff.base.sleep 1s
hdds.ratis.client.exponential.backoff.max.retries 2
Apache JIRA: HDDS-14040
CDPD-91562: test_validate_certs_configs configuration is failing with the maximum lifetime validation
7.3.2.0, 7.3.1.600
In daylight saving time zones, the autogenerated Ozone certificate duration might differ from the expected duration. This discrepancy is minor, because the default certificate duration is 365 days or five years, depending on the Ozone component.
CDPD-75954: The ozone debug ldb command and ozone auditparser fails with java.lang.UnsatisfiedLinkError
7.3.2, 7.3.1.400
For information on workaround, see Changing temporary path for Ozone services and CLI tools.
CDPD-54885: Ozone Prometheus does not work with TLS
7.3.2, 7.3.1 and its SPs and CHFs
The Prometheus service shipped by Ozone does not support TLS mode. So, Prometheus is not able to gather metrics from Ozone endpoints when TLS is enabled.
Go to Ozone > Configuration > Ozone Prometheus Endpoint Token and in the Ozone Prometheus Endpoint Token property enter any random string. This configuration generates a plaintext token in the Ozone endpoint process directory allowing Prometheus to authenticate and collect metrics despite the TLS limitation.
CDPD-56684: Keys and buckets get deleted without volume permission
7.3.2, 7.3.1 and its SPs and CHFs
When a volume deletion is initiated, the system recursively deletes all buckets and keys within the volume before attempting to delete the volume itself. Because the ACL check for volume deletion permissions occurs only in the end, all the data within the volume is deleted even without having delete permission on the volume.
CDPD-50610: Large file uploads are slow with OPEN and stream data approach
7.3.2, 7.3.1 and its SPs and CHFs
Hue file browser uses the append operation for large files. This API is not supported by Ozone in 7.1.9, therefore large file uploads can be slow or can time out in the browser.
Use native Ozone client to upload large files instead of the Hue file browser.
OPSAPS-66469: Ozone-site.xml is missing if the host does not contain HDFS roles
7.3.2, 7.3.1 and its SPs and CHFs
The client side /etc/hadoop/conf/ozone-site.xml file is not generated by Cloudera Manager if the host does not have any HDFS role. Because of this, issuing Ozone commands from that host fails because it cannot find the service name to hostname mapping. When this issue occurs, an error message is displayed: # ozone sh volume list o3://ozoneabc 23/03/06 18:46:15 WARN ha.OMProxyInfo: OzoneManager address ozoneabc:9862 for serviceID null remains unresolved for node ID null Check your ozone-site.xml file to ensure ozone manager addresses are configured properly.
Add the HDFS gateway role on that host.
CDPD-63144: hadoop.ozone.om.request.key.OMKeyRenameRequestWithFSO: Rename key failed with "failed to get parent dir" error
7.3.2 and its SPs and CHFs, 7.3.1 and its SPs and CHFs
Key rename inside the FSO bucket fails and discplays the Failed to get parent dir error. This happens when running impala workloads with ozone.
None.
CDPD-74331: Key put fails with "CompletionException: Failed to write chunk"
7.3.2 and its SPs and CHFs, 7.3.1 and its SPs and CHFs
Key put fails and displays the Failed to write chunk error when there is a volume failure during configuration.
None.
CDPD-74475: YCSB test with Hbase on Ozone degrades performance
7.3.2, 7.3.1 and its SPs and CHFs
HBase on Ozone is currently provided as a Tech Preview feature. Performance characteristics, including throughput, may not yet match those of HBase running on HDFS.
None.
CDPD-74884: Exclusive size of snapshot is always returning 0
7.3.2, 7.3.1 and its SPs and CHFs
The exclusiveSize and exclusiveReplicationSize statistics provided by the snapshot info command return a value of 0 even when the snapshot contains exclusive keys or files that do not exist in other snapshots.
None.
Apache JIRA: HDDS-11528
CDPD-75042: AWS Cli recursive delete only deletes the leaf element
7.3.2 and its SPs and CHFs, 7.3.1 and its SPs and CHFs
AWS CLI rm or delete command fails to delete all files and directories on the Ozone FSO bucket. It only deletes the leaf node.
None.
CDPD-75204: Setting file name length limit can cause NN shutdown with FSImage related error
7.3.2 and its SPs and CHFs, 7.3.1 and its SPs and CHFs
Namenode restart fails after dfs.namenode.fs-limits.max-component-length is set to a lower value and there is existing data present which exceeds the length limit.
Increase the value for the dfs.namenode.fs-limits.max-component-length parameter and restart the namenode.
CDPD-75635: Ozone write fails intermittently because SCM remains in safe mode.
7.3.2, 7.3.1 and its SPs and CHFs
Ozone write operations can fail intermittently after restarting the Storage Container Manager (SCM) leader node (or when stopping the OM leader and an SCM follower). This occurs because SCM may remain in safe mode after the restart, causing write/block allocation prechecks to fail (for example, SafeModePrecheck failed for allocateBlock).
Wait for SCM to exit safe mode automatically after it meets the required thresholds. Alternatively, manually force SCM to exit from safe mode using CLI options.