Known Issues in Cloud Connectors

Learn about the known issues in Cloud Connectors, the impact or changes to the functionality, and the workaround.

AWS SDK 2.25.53 warning about transfer manager not using CRT client

Due to the AWS SDK 2.25.53 upgrade, the following warning might be seen:

5645:2024-09-13 16:29:17,375 [setup] WARN  s3.S3TransferManager
          (LoggerAdapter.java:warn(225)) - The provided S3AsyncClient is an instance of
          MultipartS3AsyncClient, and thus multipart download feature is not enabled. To benefit
          from all features, consider using S3AsyncClient.crtBuilder().build() instead

This error message is completely harmless and should be ignored. For more information, see HADOOP-19272.

None

CDPD-76378 Uploading files to S3 takes longer than expected

Uploading files to S3 on long haul networks is delayed if the file is smaller than 64MB. As the 100-continue header is not received immediately, the files are uploaded only after several retries. The delay in uploading is caused by the fs.s3a.connection.expect.continue S3A configuration parameter. The fs.s3a.connection.expect.continue S3A configuration property controls whether or not a PUT request to the S3 store sets the "Expect: 100-continue" header, and awaits a 100 CONTINUE response before uploading any data. This allows throttling and other transient problems to be reported by the S3 store before there is an attempt by the client to upload any data.

By default, the value of the configuration property is set to true, which means that the header is set, and a 100 CONTINUE response is required.

If there is a significant delay in uploading files or if the write time of jobs increases, you can set the fs.s3a.connection.expect.continue configuration property for your service to false.