Known issues and technical limitations for Ozone are addressed in Cloudera Runtime 7.3.3.
Known issues identified in Cloudera Runtime 7.3.3
CDPD-125823: ListObjectsV2 incorrectly filters
CommonPrefixes when StartAfter is before matching prefixes
7.3.3
When a client calls the S3
ListObjectsV2 API with the Prefix,
Delimiter, and StartAfter parameters,
the Ozone S3 Gateway returns an empty CommonPrefixes list
even though matching prefixes exist and sort lexicographically after the
specified StartAfter value.
Omit StartAfter when listing objects with
Delimiter and Prefix.
List objects without Delimiter and derive prefixes
on the client side.
If pagination is required, use the
ContinuationToken from a prior listing, or
specify a StartAfter value under the same
Prefix that does not trigger the issue.
CDPD-126697: CompleteMultipartUpload accepts duplicate
part numbers and creates an incomplete object
7.3.3
The Ozone S3 Gateway does not reject
CompleteMultipartUpload requests when the part list
contains duplicate part numbers (for example, [1, 1] with
the same ETag). The operation returns a success response, but the committed
object reflects the bytes of only one part (for example, 5 MiB instead of 10
MiB). Additionally, HeadObject reports a
PartsCount that does not match the actual
ContentLength, which causes data inconsistency.
Ensure that each part number appears exactly
once and in ascending order with no gaps before calling
CompleteMultipartUpload.
CDPD-126757: CompleteMultipartUpload calculates incorrect ETag and
PartsCount when completing with a subset of uploaded parts
7.3.3
In Ozone S3, when a user completes a multipart upload
using only a subset of uploaded parts, the object metadata does not match
its actual content. While the object size and payload correctly reflect only
the parts listed in the completion request, HeadObject
incorrectly calculates PartsCount and the multipart ETag
suffix using every part uploaded during the session. For example, if a
session contains parts 1, 2, and 3, but completes using only parts 1 and 3,
the system reports a PartsCount of 3 and an ETag suffix of
-3 instead of the expected values of 2 and -2.
CDPD-126906: Ozone HTTPS endpoints reject TLSv1.3 by
default
7.3.3
When TLS is enabled, the Ozone S3 Gateway and the web
UIs of the Ozone roles accept only TLSv1.2 by default. Clients that require
TLSv1.3 fail the TLS handshake with a protocol_version
error. Clients that support both TLSv1.2 and TLSv1.3 are not affected.
In Cloudera Manager, add the
following configuration to the Ozone Service Advanced
Configuration Snippet (Safety Valve) for ozone-site.xml
configuration, and restart the Ozone
service:
CDPD-126954: S3 CreateMultipartUpload returns 500
InternalError when the key matches an existing directory in an FSO
bucket
7.3.3
In a File System Optimized (FSO) bucket, calling
CreateMultipartUpload with a key that matches an
existing directory (for example, dir1 when
dir1/ already exists) returns an HTTP 500
InternalError. Although the Ozone Manager rejects the request
because it cannot write to a directory, the S3 Gateway fails to map the
error code correctly and returns a generic internal error. This behavior
causes clients, such as the AWS CLI, to retry the request repeatedly.
To work around this issue, perform the following
actions:
Do not start a multipart upload on a key that matches an existing
directory.
Verify whether a key is a directory before starting an upload by
running the ozone sh key info /volume/bucket/key
command and checking whether "file": false.
If the target path is an existing directory, use
PutObject for that path, or write the multipart
object under a new key that does not conflict with an existing
directory.
CDPD-126993: S3 CreateMultipartUpload succeeds under a
parent path that is a file, and subsequent UploadPart calls fail with 500
InternalError
7.3.3
In a File System Optimized (FSO) bucket, if a parent
path is an existing file (for example, a file named
parent), a
CreateMultipartUpload request for a child path (for
example, parent/child) incorrectly succeeds and returns an
UploadId. Every subsequent UploadPart
request for that upload fails with an HTTP 500
InternalError. Although the Ozone Manager detects the file
system violation (NOT_A_FILE), the S3 Gateway returns a
generic 500 error instead of a 400-level client error. The multipart upload
remains in progress and cannot complete, leaving aborting the upload as the
only option.
To work around this issue, perform the following
actions:
Do not start a multipart upload under a key where a parent component
is an existing file.
Before starting an upload, verify the path by using
HeadObject or the ozone sh key
info command.
If a CreateMultipartUpload request returns an
UploadId for such a path, run
abort-multipart-upload to clear the incomplete
upload, and ensure that all parent components in the target path are
directories.
CDPD-126995: S3 ListParts with max-parts=0 or a
negative value returns HTTP 500 with no S3 error body
7.3.3
When a client calls the S3 ListParts
operation with max-parts=0 or a negative value (for
example, -1), the S3 Gateway returns an HTTP 500
Internal Server Error instead of a standard S3 error response.
This issue occurs because the S3 Gateway forwards the invalid value to the
Ozone client, which throws an unhandled
IllegalArgumentException ("Max Parts Should be greater
than zero"). Because the response lacks a valid S3 XML error body, clients
such as the AWS CLI retry the request repeatedly, which results in multiple
failed requests and generic error messages.
To work around this issue, use one of the
following options:
Specify only positive values for the max-parts
parameter (typically between 1 and 1000).
Omit the --max-parts parameter to use the default
page size, and use the part-number-marker parameter
for pagination if more parts exist.
CDPD-127247: UploadPartCopy does not enforce
x-amz-copy-source-if-match and x-amz-copy-source-if-none-match
preconditions
7.3.3
In Ozone S3, the UploadPartCopy
operation fails to enforce ETag-based preconditions
(x-amz-copy-source-if-match and
x-amz-copy-source-if-none-match). While date-based
preconditions work correctly, Ozone S3 ignores ETag conditions.
Consequently, the copy operation succeeds and writes the part even when the
source ETag does not match, such as after an overwrite of the source object.
The completed multipart object then contains mixed data from different
versions of the source object instead of returning the expected 412
PreconditionFailed error.
Known Issues identified before Cloudera Runtime 7.3.3
Known issues identified before Cloudera Runtime 7.3.3 include only
unresolved issues from previous releases that continue to affect the Cloudera Runtime 7.3.3 base release.
CDPD-105709: Recon UI displays inconsistent Solr
health status
7.3.3 and 7.3.2.10000
The Recon Overview page incorrectly displays
SOLR is HEALTHY as an alert when the service functions
normally. Additionally, other pages within the Recon UI report SOLR
is UNHEALTHY despite Solr remaining in a healthy state.
CDPD-109005: Storage Container Manager (SCM) fails to
update DatanodeDetails when only port configurations change
7.3.2.10000
When a DataNode re-registers with SCM or sends
heartbeats after gaining new ports, SCM fails to update the stored
DatanodeDetails in the node registry. SCM triggers an update only if the IP
address, hostname, or software version changes. Consequently, SCM ignores
port-only updates, which leads to stale metadata and potential failures in
features that rely on the new ports.
None
CDPD-109002: Ozone DataStream writes fail after
enabling Ratis DataStream on existing clusters
7.3.2.10000
When Ratis DataStream is enabled on an existing
cluster, open pipelines retain stale DataNode details that lack the required
DataStream port information. Consequently, Ozone client writes fail when
they exceed the automatic threshold because the client cannot identify the
necessary port for pipeline members. This occurs because SCM does not
refresh member ports for existing open pipelines after the configuration
change.
CDPD-99722: Container data checksum reverts to a stale
value after successful reconciliation
7.3.2.0 and 7.3.2.10000
In Ozone, the
BackgroundContainerDataScanner service reverts the
container data checksum to a stale value even after a successful
reconciliation repairs corrupt chunks. This occurs because the scanner
builds a checksum tree in memory while reconciliation updates the container
data on disk, causing the scanner to persist an outdated tree.
The issue resolves automatically during the next
container scan cycle. To resolve the issue immediately, manually trigger
container reconciliation from the command line.
CDPD-105115: DataNodes fail to communicate with SCM
after restart
7.3.2.10000
In a SCM High Availability (HA) cluster, a restarting
or failing-over follower SCM starts its DataNode protocol server before
completing its Ratis log replay. This causes SCM to process DataNode
container reports against an incomplete state, which leads to dropped
replica locations. Consequently, if that follower becomes the leader,
affected containers appear to have missing or empty replicas, and DataNode
logs record errors such as Unable to clear disk,
Failed to flush DB before close, and
EOFExceptions.
This issue is temporary and typically resolves
during the next report cycle when SCM rebuilds the replica list.
CDPD-119561: Ozone Manager (OM) becomes unresponsive
during sustained snapshot operations
7.3.2.10000
The OM leader enters a deadlock state during heavy
snapshot creation and deletion loads. This occurs when the Key Deleting
Service holds a snapshot database lock while waiting for a transaction that
cannot complete because the flush thread is blocked. As a result, the OM
stops responding to client requests and snapshot operations time out.
Restart the affected OM to release the locks and restore
service.
Limit snapshot deletion and heavy write activity to reduce the risk
of recurrence.
Increase the
ozone.om.unflushed.transaction.max.count
configuration to lower the probability of the deadlock (monitor for
increased memory usage).
CDPD-104229: Delay in deleting snapshot YAML file by
OM Snapshot Local Data Manager
7.3.2.10000
A significant delay of approximately 20 to 30 minutes
occurs in the deletion of snapshot YAML files by the OM Snapshot Local Data
Manager after the system processes a snapshot deletion request. This delay
happens because the current design requires the inDegree, which represents
the number of descendant snapshots, to reach zero before the system removes
the YAML file. This behavior impacts subsequent operations, such as
bootstrap test cases, that rely on the timely cleanup of these files.
None
CDPD-119610: OM leader remains in
LEADER_AND_NOT_READY state during Snapshot
Defragmentation Bootstrap
7.3.2.10000
The Ozone Manager (OM) leader remains in the
LEADER_AND_NOT_READY state after a follower node stops
for a planned bootstrap. This issue occurs due to a deadlock between the
snapshot installation thread and the double-buffer flush thread on the
follower node. Consequently, the cluster loses its usable quorum, and all OM
client RPC requests fail with an
OMLeaderNotReadyException.
Restart the affected OM follower node to break
the deadlock and restore normal cluster operations.
CDPD-119172: OM follower node bootstrap fails when OM
leader changes
7.3.2.10000
In an Ozone Manager (OM) High Availability (HA) setup,
the bootstrap process for a follower node fails if the OM leader node
changes while the bootstrap is in progress. If the original leader stops
(for example, through SIGTERM), the bootstrap thread attempts to download
the database snapshot from the former leader and fails with a
Connection refused error. When the thread subsequently attempts
to connect to the newly elected leader, it receives a 503 Service
Unavailable error because the new leader is not yet ready to
serve requests.
To ensure a successful bootstrap, verify that
two of the three Ozone Managers (OMs) are running before bootstrapping the
third node.
CDPD-102216: Toggling to Old UI from the
Cluster Capacity tab results in a 404 error
7.3.2.10000
The Cluster Capacity feature is
available only in the new Recon UI. If you view the Cluster
Capacity page in the new UI and click Switch to
Old UI, the application attempts to load the same path in
the legacy interface. Because this path does not exist in the old UI, a
404 Page Not Found error appears.
Manually navigate to the main dashboard if you
encounter a 404 Page Not Found error after switching to the
old UI. Use the new UI to view cluster capacity information, as the legacy
interface does not support this feature.
CDPD-106805: Requests to Ozone S3 Gateway through
HAProxy fail with 400 Bad Request during directory
creation
7.3.2.10000
When using the S3 connector to access Ozone OBS
buckets through an S3 Gateway behind HAProxy, requests fail with a
400 Bad Request error. This issue occurs because
zero-byte directory-marker PUT requests (which include
Expect: 100-continue headers) cause the S3 Gateway to
skip reading the request body and return 200 OK
immediately. This leaves unconsumed bytes on the wire and desynchronizes
keep-alive connections. HAProxy then misinterprets subsequent requests,
causing a parsing error.
Disable keep-alive connections in the HAProxy
configuration by setting option httpclose and increasing
the maxconn value (for example, to 4096).
CDPD-107197: The ozone admin datanode
diskbalancer status command fails when passing a DataNode ID
instead of an address
7.3.2.10000
The ozone admin datanode diskbalancer
status command accepts only a DataNode network address or host
string. When you use a valid datanodeUuid (DataNode ID) to
identify the node, the command fails with an
UnknownHostException or an invalid argument error. This
behavior is inconsistent with other ozone admin commands
that support both UUIDs and addresses for node identification.
To check the disk balancer status, use the
DataNode network address or hostname instead of its UUID. Run the
ozone admin datanode list command to locate the
network address.
CDPD-100716: Reading keys results in a
NO_REPLICA_FOUND error after SCM follower restart and
leader transfer
7.3.2.10000
When a Storage Container Manager (SCM) follower
restarts, it accepts DataNode container reports before catching up with the
Ratis log. Processing these reports against a stale database triggers a
NotLeaderException, causing the SCM to skip updating
in-memory container replica locations. If this follower is later promoted to
leader, reading keys fails with a NO_REPLICA_FOUND error
due to missing replica metadata.
CDPD-95733: Snapshot pruning tests fail due to empty
compaction SST backups in Ozone in Cloudera Runtime7.3.2
7.3.2
In Ozone in Cloudera Runtime
7.3.2, snapshot pruning tests fail because the Ozone Manager (OM) RocksDB
Compaction Service configuration key requires a new name. The property key
changes from ozone.compaction.service.enabled (used in
version 7.1.9) to ozone.om.compaction.service.enabled
in version 7.3.2. If you use the old configuration key, the compaction
service remains disabled by default. This causes empty compaction SST backup
directories and fails the pruning operations.
Update the configuration to use the new property
key. Set ozone.om.compaction.service.enabled to
true to enable the compaction service in version
7.3.2.
OPSAPS-76062: Exposing
"ozone.replication" in Cloudera Manager configurations and assigning a default value
7.3.2
Exposing "ozone.replication in
Cloudera Manager configurations and assigning a
default value to it is causing it to override bucket replication
configuration as a client side configuration even when you does not mean to
set the client side configurations.
None
CDPD-73792: The new snapshot could not be found after
renaming the old snapshot
7.3.2
The ozone sh snapshot rename
command renames snapshots. It is a feature developed by the Apache Ozone
community and is included in Cloudera Base on premises 7.3.1. However, it
does not work properly, and Cloudera Base on premises does not support
it.
CDPD-63350: Force deleting a FSO bucket and its
contents while running rb --force from AWS S3 API is
failing
7.3.2
Force deleting a File System Optimized (FSO)
bucket and its contents while running the rb --force
command from the AWS S3 API might fail with an error, because the S3 client
sends individual delete requests for each key in the bucket. It might delete
or fail to delete individual keys or directories, depending on the
availability of leaf elements. It can completely delete the bucket only when
all the keys or directories are cleaned up. If some keys are not deleted,
the bucket will not be deleted.
Sample error
message
# aws s3 rm s3://buck-fso --recursive
delete: s3://buck-fso/dir1/
delete: s3://buck-fso/dir1/dir2/
delete: s3://buck-fso/dir3/dir4/dir5/
delete: s3://buck-fso/dir3/dir4/
delete failed: s3://buck-fso/dir3/
# Rerun the same command again
# aws s3 rm s3://buck-fso --recursive
delete: s3://buck-fso/dir1/
delete: s3://buck-fso/dir3/
# To Confirm
# ozone sh key list s3v/buck-fso
[ ]
Run the rb --force command
multiple times to completely clean up the keys and directories.
CDPD-98751: Mismatched Replicas
tab in the Recon UI fails to display containers with inconsistent replica
checksums
7.3.2
In the Ozone Recon UI, the
Mismatched Replicas tab does not update or
display containers when one or more replicas have differing checksums.
Instead, they are displayed in the Under-Replicated
tab.
The replica can be checked in the
Under-Replicated tab, or can be cross-checked
against API response.
CDPD-97512: Mismatch in Open Key count between
Overview page and OM DB Insight in the Recon
UI
7.3.2
In the Ozone Recon UI, the Open
Key count on the Summary section of the
Overview page might not match the count on the OM DB Insight > Open Key tab. Users might see different Open
Key values for the same cluster across these two views.
Use the OM DB Insight > Open Key tab for the most accurate Open Key
count.
CDPD-97376: Container replication counts mismatch in
Recon UI
7.3.2
In the Ozone Recon UI, container replication
counts, including Under-Replicated,
Over-Replicated, and
Mis-Replicated counts, differ between the updated
and the legacy Container page for the same
cluster.
Refer to the legacy
Container page to view accurate replication
counts.
CDPD-97311: Incorrect Creation
Time and Modification Time displayed
on the Namespace Usage page in the Recon UI
7.3.2
In the Ozone Recon UI, the Namespace
Usage page displays incorrect Creation
Time and Modification Time
timestamps. These values do not accurately reflect the actual creation or
last modification times or dates of the namespace, resulting in inaccurate
metadata information.
Retrieve the correct timestamp values directly
from the API response.
CDPD-97312: Mismatch between cluster State Container
count and Container Summary totals
7.3.2
The Ozone Recon UI can display discrepancies
between the Storage Container Manager (SCM) container count and the Recon
container summary. This occurs because Recon does not synchronize all
container states such as QUASI_CLOSED leading to inconsistent totals.
No workaround within the Ozone Recon UI. Use
ozone admin container report CLI command to obtain
the correct container counts for all states in the Ozone cluster.
CDPD-99248: After cdh upgrade, Ozone encounters
failure while running the Finalize Upgrade for SCM on role Storage
Container command
7.3.2
When finalizing an Ozone upgrade for the first
time from Cloudera Manager, the Finalize Upgrade
for SCM on role Storage Container command might fail with the
following stderr
message:
"Invalid response from Storage Container Manager.
Current finalization status is: FINALIZATION_IN_PROGRESS"
This
error occurs even though finalization continues to run on the Storage
Container Manager (SCM).
Ignore the failure in Cloudera Manager. Use the ozone admin scm
finalizationstatus command to monitor progress and wait for
the process to complete on SCM.
CDPD-98892: File Size
Distribution bucket size range calculation is not
correct
7.3.2
If large number of buckets exists within a
volume, the Ozone Recon UI might display incorrect file size distribution
bucket size range calculation in the File Size
Distribution chart on the Insights
page.
CDPD-93116: Ozone client hangs intermittently when
disks are full
7.3.2
This hang is caused by continuous write
retries that persist until the pipeline on the Datanode closes.
The
Ozone client hangs for approximately five minutes when writing data to a
Datanode if a disk full exception or other Datanode error occurs. This hang
is caused by continuous write retries that persist until the pipeline on the
Datanode closes.
Control the request retry behavior by setting
the following configurations on the client side:
CDPD-91562:
test_validate_certs_configs configuration is failing
with the maximum lifetime validation
7.3.2.0, 7.3.1.600
In daylight saving time zones, the autogenerated Ozone
certificate duration might differ from the expected duration. This
discrepancy is minor, because the default certificate duration is 365 days
or five years, depending on the Ozone component.
CDPD-75954: The ozone debug ldb
command and ozone auditparser fails with
java.lang.UnsatisfiedLinkError
CDPD-54885: Ozone Prometheus does not work with
TLS
7.3.2, 7.3.1 and its SPs and CHFs
The Prometheus service shipped by Ozone does not
support TLS mode. So, Prometheus is not able to gather metrics from Ozone
endpoints when TLS is enabled.
Go to Ozone > Configuration > Ozone Prometheus Endpoint Token and in the Ozone Prometheus Endpoint
Token property enter any random string. This configuration
generates a plaintext token in the Ozone endpoint process directory allowing
Prometheus to authenticate and collect metrics despite the TLS limitation.
CDPD-56684: Keys and buckets get deleted without
volume permission
7.3.2, 7.3.1 and its SPs and CHFs
When a volume deletion is initiated, the system
recursively deletes all buckets and keys within the volume before attempting
to delete the volume itself. Because the ACL check for volume deletion
permissions occurs only in the end, all the data within the volume is
deleted even without having delete permission on the volume.
CDPD-50610: Large file uploads are slow with OPEN and
stream data approach
7.3.2, 7.3.1 and its SPs and CHFs
Hue file browser uses the append operation for large
files. This API is not supported by Ozone in 7.1.9, therefore large file
uploads can be slow or can time out in the browser.
Use native Ozone client to upload large files
instead of the Hue file browser.
OPSAPS-66469: Ozone-site.xml is missing if the host
does not contain HDFS roles
7.3.2, 7.3.1 and its SPs and CHFs
The client side
/etc/hadoop/conf/ozone-site.xml file is not
generated by Cloudera Manager if the host does not have
any HDFS role. Because of this, issuing Ozone commands from that host fails
because it cannot find the service name to hostname mapping. When this issue
occurs, an error message is displayed: # ozone sh volume list
o3://ozoneabc 23/03/06 18:46:15 WARN ha.OMProxyInfo: OzoneManager
address ozoneabc:9862 for serviceID null remains unresolved for node ID
null Check your ozone-site.xml file to ensure ozone manager addresses
are configured properly.
Add the HDFS gateway role on that host.
CDPD-63144: hadoop.ozone.om.request.key.OMKeyRenameRequestWithFSO: Rename
key failed with "failed to get parent dir" error
7.3.2 and its SPs and CHFs, 7.3.1 and its
SPs and CHFs
Key rename inside the FSO bucket fails and discplays the Failed to
get parent dir error. This happens when running impala
workloads with ozone.
None.
CDPD-74331: Key put fails with "CompletionException: Failed to write
chunk"
7.3.2 and its SPs and CHFs, 7.3.1 and its
SPs and CHFs
Key put fails and displays the Failed to write chunk
error when there is a volume failure during configuration.
None.
CDPD-74475: YCSB test with Hbase on Ozone degrades
performance
7.3.2, 7.3.1 and its SPs and CHFs
HBase on Ozone is currently provided as a Tech Preview
feature. Performance characteristics, including throughput, may not yet
match those of HBase running on HDFS.
None.
CDPD-74884: Exclusive size of snapshot is always returning
0
7.3.2, 7.3.1 and its SPs and CHFs
The exclusiveSize and
exclusiveReplicationSize statistics provided by the
snapshot info command return a value of
0 even when the snapshot contains exclusive keys or
files that do not exist in other snapshots.
CDPD-75042: AWS Cli recursive delete only deletes the leaf
element
7.3.2 and its SPs and CHFs, 7.3.1 and its
SPs and CHFs
AWS CLI rm or
delete command fails to delete all files and
directories on the Ozone FSO bucket. It only deletes the leaf node.
None.
CDPD-75204: Setting file name length limit can cause NN
shutdown with FSImage related error
7.3.2 and its SPs and CHFs, 7.3.1 and its
SPs and CHFs
Namenode restart fails after
dfs.namenode.fs-limits.max-component-length is set to a lower value and
there is existing data present which exceeds the length limit.
Increase the value for the
dfs.namenode.fs-limits.max-component-length
parameter and restart the namenode.
CDPD-75635: Ozone write fails intermittently because SCM
remains in safe mode.
7.3.2, 7.3.1 and its SPs and CHFs
Ozone write operations can fail intermittently after
restarting the Storage Container Manager (SCM) leader node (or when stopping
the OM leader and an SCM follower). This occurs because SCM may remain in
safe mode after the restart, causing write/block allocation prechecks to
fail (for example, SafeModePrecheck failed for allocateBlock).
Wait for SCM to exit safe mode automatically
after it meets the required thresholds. Alternatively, manually force SCM to
exit from safe mode using CLI options.