The upload uses either the PutS3Object method or the PutS3MultipartUpload method. The PutS3Object method sends the file in a single synchronous call, but it has a 5GB size limit. Larger files are sent using the PutS3MultipartUpload method. This multipart process saves state after each step so that a large upload can be resumed with minimal loss if the processor or cluster is stopped and restarted. A multipart upload consists of three steps:
For multipart uploads, the processor saves state locally tracking the upload ID and parts uploaded, which must both be provided to complete the upload. The AWS libraries select an endpoint URL based on the AWS region, but this can be overridden with the 'Endpoint Override URL' property for use with other S3-compatible endpoints. The S3 API specifies that the maximum file size for a PutS3Object upload is 5GB. It also requires that parts in a multipart upload must be at least 5MB in size, except for the last part. These limits establish the bounds for the Multipart Upload Threshold and Part Size properties.
The Object Key property value should not start with "/".
The Credentials File property allows the user to specify the path to a file containing the AWS access key and secret key. The contents of the file should be in the following format:
[default]
accessKey=<access key>
secretKey=<security key>
Make sure the credentials file is readable by the NiFi service user.
When using the Credential File property, ensure that there are no values for the Access Key and Secret Key properties. The Value column should read "No value set" for both. Note: Do not check "Set empty string" for either as the empty string is considered a set value.