PutS3Object

Description:

Puts FlowFiles to an Amazon S3 Bucket. The upload uses either the PutS3Object method or the PutS3MultipartUpload method. The PutS3Object method sends the file in a single synchronous call, but it has a 5GB size limit. Larger files are sent using the PutS3MultipartUpload method. This multipart process saves state after each step so that a large upload can be resumed with minimal loss if the processor or cluster is stopped and restarted. A multipart upload consists of three steps: 1) initiate upload, 2) upload the parts, and 3) complete the upload. For multipart uploads, the processor saves state locally tracking the upload ID and parts uploaded, which must both be provided to complete the upload. The AWS libraries select an endpoint URL based on the AWS region, but this can be overridden with the 'Endpoint Override URL' property for use with other S3-compatible endpoints. The S3 API specifies that the maximum file size for a PutS3Object upload is 5GB. It also requires that parts in a multipart upload must be at least 5MB in size, except for the last part. These limits establish the bounds for the Multipart Upload Threshold and Part Size properties.

Additional Details...

Tags:

Amazon, S3, AWS, Archive, Put

Properties:

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.

Display NameAPI NameDefault ValueAllowable ValuesDescription
Object KeyObject Key${filename}The S3 Object Key to use. This is analogous to a filename for traditional file systems.
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
BucketBucketThe S3 Bucket to interact with
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
Content TypeContent TypeSets the Content-Type HTTP header indicating the type of content stored in the associated object. The value of this header is a standard MIME type. AWS S3 Java client will attempt to determine the correct content type if one hasn't been set yet. Users are responsible for ensuring a suitable content type is set when uploading streams. If no content type is provided and cannot be determined by the filename, the default content type "application/octet-stream" will be used.
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
Content DispositionContent Disposition
  • inline
  • attachment
Sets the Content-Disposition HTTP header indicating if the content is intended to be displayed inline or should be downloaded. Possible values are 'inline' or 'attachment'. If this property is not specified, object's content-disposition will be set to filename. When 'attachment' is selected, '; filename=' plus object key are automatically appended to form final value 'attachment; filename="filename.jpg"'.
Cache ControlCache ControlSets the Cache-Control HTTP header indicating the caching directives of the associated object. Multiple directives are comma-separated.
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
Access Key IDAccess KeyNo Description Provided.
Sensitive Property: true
Supports Expression Language: true (will be evaluated using variable registry only)
Secret Access KeySecret KeyNo Description Provided.
Sensitive Property: true
Supports Expression Language: true (will be evaluated using variable registry only)
Credentials FileCredentials FilePath to a file containing AWS access key and secret key in properties file format.

This property requires exactly one file to be provided..
AWS Credentials Provider ServiceAWS Credentials Provider serviceController Service API:
AWSCredentialsProviderService
Implementations: AWSCredentialsProviderControllerService
AWSIDBrokerCloudCredentialsProviderControllerService
The Controller Service that is used to obtain AWS credentials provider
Object Tags Prefixs3-object-tags-prefixSpecifies the prefix which would be scanned against the incoming FlowFile's attributes and the matching attribute's name and value would be considered as the outgoing S3 object's Tag name and Tag value respectively. For Ex: If the incoming FlowFile carries the attributes tagS3country, tagS3PII, the tag prefix to be specified would be 'tagS3'
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
Remove Tag Prefixs3-object-remove-tags-prefixFalse
  • True
  • False
If set to 'True', the value provided for 'Object Tags Prefix' will be removed from the attribute(s) and then considered as the Tag name. For ex: If the incoming FlowFile carries the attributes tagS3country, tagS3PII and the prefix is set to 'tagS3' then the corresponding tag values would be 'country' and 'PII'
Storage ClassStorage ClassStandard
  • DeepArchive
  • Glacier
  • GlacierInstantRetrieval
  • IntelligentTiering
  • OneZoneInfrequentAccess
  • Outposts
  • ReducedRedundancy
  • Snow
  • Standard
  • StandardInfrequentAccess
No Description Provided.
RegionRegionUS West (Oregon)
  • AWS GovCloud (US) AWS Region Code : us-gov-west-1
  • AWS GovCloud (US-East) AWS Region Code : us-gov-east-1
  • US East (N. Virginia) AWS Region Code : us-east-1
  • US East (Ohio) AWS Region Code : us-east-2
  • US West (N. California) AWS Region Code : us-west-1
  • US West (Oregon) AWS Region Code : us-west-2
  • EU (Ireland) AWS Region Code : eu-west-1
  • EU (London) AWS Region Code : eu-west-2
  • EU (Paris) AWS Region Code : eu-west-3
  • EU (Frankfurt) AWS Region Code : eu-central-1
  • EU (Zurich) AWS Region Code : eu-central-2
  • EU (Stockholm) AWS Region Code : eu-north-1
  • EU (Milan) AWS Region Code : eu-south-1
  • EU (Spain) AWS Region Code : eu-south-2
  • Asia Pacific (Hong Kong) AWS Region Code : ap-east-1
  • Asia Pacific (Mumbai) AWS Region Code : ap-south-1
  • Asia Pacific (Hyderabad) AWS Region Code : ap-south-2
  • Asia Pacific (Singapore) AWS Region Code : ap-southeast-1
  • Asia Pacific (Sydney) AWS Region Code : ap-southeast-2
  • Asia Pacific (Jakarta) AWS Region Code : ap-southeast-3
  • Asia Pacific (Melbourne) AWS Region Code : ap-southeast-4
  • Asia Pacific (Tokyo) AWS Region Code : ap-northeast-1
  • Asia Pacific (Seoul) AWS Region Code : ap-northeast-2
  • Asia Pacific (Osaka) AWS Region Code : ap-northeast-3
  • South America (Sao Paulo) AWS Region Code : sa-east-1
  • China (Beijing) AWS Region Code : cn-north-1
  • China (Ningxia) AWS Region Code : cn-northwest-1
  • Canada (Central) AWS Region Code : ca-central-1
  • Middle East (UAE) AWS Region Code : me-central-1
  • Middle East (Bahrain) AWS Region Code : me-south-1
  • Africa (Cape Town) AWS Region Code : af-south-1
  • US ISO East AWS Region Code : us-iso-east-1
  • US ISOB East (Ohio) AWS Region Code : us-isob-east-1
  • US ISO West AWS Region Code : us-iso-west-1
  • Use 's3.region' Attribute Uses 's3.region' FlowFile attribute as region.
The AWS Region to connect to.
Communications TimeoutCommunications Timeout30 secsThe amount of time to wait in order to establish a connection to AWS or receive data from AWS before timing out.
Expiration Time RuleExpiration Time RuleNo Description Provided.
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
FullControl User ListFullControl User List${s3.permissions.full.users}A comma-separated list of Amazon User ID's or E-mail addresses that specifies who should have Full Control for an object
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
Read Permission User ListRead Permission User List${s3.permissions.read.users}A comma-separated list of Amazon User ID's or E-mail addresses that specifies who should have Read Access for an object
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
Write Permission User ListWrite Permission User List${s3.permissions.write.users}A comma-separated list of Amazon User ID's or E-mail addresses that specifies who should have Write Access for an object
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
Read ACL User ListRead ACL User List${s3.permissions.readacl.users}A comma-separated list of Amazon User ID's or E-mail addresses that specifies who should have permissions to read the Access Control List for an object
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
Write ACL User ListWrite ACL User List${s3.permissions.writeacl.users}A comma-separated list of Amazon User ID's or E-mail addresses that specifies who should have permissions to change the Access Control List for an object
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
OwnerOwner${s3.owner}The Amazon ID to use for the object's owner
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
Canned ACLcanned-acl${s3.permissions.cannedacl}Amazon Canned ACL for an object, one of: BucketOwnerFullControl, BucketOwnerRead, LogDeliveryWrite, AuthenticatedRead, PublicReadWrite, PublicRead, Private; will be ignored if any other ACL/permission/owner property is specified
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
SSL Context ServiceSSL Context ServiceController Service API:
SSLContextService
Implementations: StandardRestrictedSSLContextService
StandardSSLContextService
Specifies an optional SSL Context Service that, if provided, will be used to create connections
Endpoint Override URLEndpoint Override URLEndpoint URL to use instead of the AWS default including scheme, host, port, and path. The AWS libraries select an endpoint URL based on the AWS region, but this property overrides the selected endpoint URL, allowing use with other S3-compatible endpoints.
Supports Expression Language: true (will be evaluated using variable registry only)
Signer OverrideSigner OverrideDefault Signature
  • Default Signature
  • Signature Version 4
  • Signature Version 2
  • Custom Signature
The AWS S3 library uses Signature Version 4 by default but this property allows you to specify the Version 2 signer to support older S3-compatible services or even to plug in your own custom signer implementation.
Custom Signer Class Namecustom-signer-class-nameFully qualified class name of the custom signer class. The signer must implement com.amazonaws.auth.Signer interface.
Supports Expression Language: true (will be evaluated using variable registry only)

This Property is only considered if the [Signer Override] Property has a value of "Custom Signature".
Custom Signer Module Locationcustom-signer-module-locationComma-separated list of paths to files and/or directories which contain the custom signer's JAR file and its dependencies (if any).

This property expects a comma-separated list of resources. Each of the resources may be of any of the following types: directory, file.

Supports Expression Language: true (will be evaluated using variable registry only)

This Property is only considered if the [Signer Override] Property has a value of "Custom Signature".
Multipart ThresholdMultipart Threshold5 GBSpecifies the file size threshold for switch from the PutS3Object API to the PutS3MultipartUpload API. Flow files bigger than this limit will be sent using the stateful multipart process. The valid range is 50MB to 5GB.
Multipart Part SizeMultipart Part Size5 GBSpecifies the part size for use when the PutS3Multipart Upload API is used. Flow files will be broken into chunks of this size for the upload process, but the last part sent can be smaller since it is not padded. The valid range is 50MB to 5GB.
Multipart Upload AgeOff IntervalMultipart Upload AgeOff Interval60 minSpecifies the interval at which existing multipart uploads in AWS S3 will be evaluated for ageoff. When processor is triggered it will initiate the ageoff evaluation if this interval has been exceeded.
Multipart Upload Max Age ThresholdMultipart Upload Max Age Threshold7 daysSpecifies the maximum age for existing multipart uploads in AWS S3. When the ageoff process occurs, any upload older than this threshold will be aborted.
Temporary Directory Multipart States3-temporary-directory-multipart${java.io.tmpdir}Directory in which, for multipart uploads, the processor will locally save the state tracking the upload ID and parts uploaded which must both be provided to complete the upload.
Supports Expression Language: true (will be evaluated using variable registry only)
Server Side Encryptionserver-side-encryptionNone
  • None
  • AES256
Specifies the algorithm used for server side encryption.
Encryption Serviceencryption-serviceController Service API:
AmazonS3EncryptionService
Implementation: StandardS3EncryptionService
Specifies the Encryption Service Controller used to configure requests. PutS3Object: For backward compatibility, this value is ignored when 'Server Side Encryption' is set. FetchS3Object: Only needs to be configured in case of Server-side Customer Key, Client-side KMS and Client-side Customer Key encryptions.
Use Chunked Encodinguse-chunked-encodingtrue
  • true
  • false
Enables / disables chunked encoding for upload requests. Set it to false only if your endpoint does not support chunked uploading.
Use Path Style Accessuse-path-style-accessfalse
  • true
  • false
Path-style access can be enforced by setting this property to true. Set it to true if your endpoint does not support virtual-hosted-style requests, only path-style requests.
Proxy Configuration Serviceproxy-configuration-serviceController Service API:
ProxyConfigurationService
Implementation: StandardProxyConfigurationService
Specifies the Proxy Configuration Controller Service to proxy network requests. If set, it supersedes proxy settings configured per component. Supported proxies: HTTP + AuthN
Proxy HostProxy HostProxy host name or IP
Supports Expression Language: true (will be evaluated using variable registry only)
Proxy Host PortProxy Host PortProxy host port
Supports Expression Language: true (will be evaluated using variable registry only)
Proxy Usernameproxy-user-nameProxy username
Supports Expression Language: true (undefined scope)
Proxy Passwordproxy-user-passwordProxy password
Sensitive Property: true
Supports Expression Language: true (undefined scope)

Dynamic Properties:

Supports Sensitive Dynamic Properties: No

Dynamic Properties allow the user to specify both the name and value of a property.

NameValueDescription
The name of a User-Defined Metadata field to add to the S3 ObjectThe value of a User-Defined Metadata field to add to the S3 ObjectAllows user-defined metadata to be added to the S3 object as key/value pairs
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)

Relationships:

NameDescription
successFlowFiles are routed to success relationship
failureFlowFiles are routed to failure relationship

Reads Attributes:

NameDescription
filenameUses the FlowFile's filename as the filename for the S3 object

Writes Attributes:

NameDescription
s3.bucketThe S3 bucket where the Object was put in S3
s3.keyThe S3 key within where the Object was put in S3
s3.contenttypeThe S3 content type of the S3 Object that put in S3
s3.versionThe version of the S3 Object that was put to S3
s3.exceptionThe class name of the exception thrown during processor execution
s3.additionalDetailsThe S3 supplied detail from the failed operation
s3.statusCodeThe HTTP error code (if available) from the failed operation
s3.errorCodeThe S3 moniker of the failed operation
s3.errorMessageThe S3 exception message from the failed operation
s3.etagThe ETag of the S3 Object
s3.contentdispositionThe content disposition of the S3 Object that put in S3
s3.cachecontrolThe cache-control header of the S3 Object
s3.uploadIdThe uploadId used to upload the Object to S3
s3.expirationA human-readable form of the expiration date of the S3 object, if one is set
s3.sseAlgorithmThe server side encryption algorithm of the object
s3.usermetadataA human-readable form of the User Metadata of the S3 object, if any was set
s3.encryptionStrategyThe name of the encryption strategy, if any was set

State management:

This component does not store state.

Restricted:

This component is not restricted.

Input requirement:

This component requires an incoming relationship.

System Resource Considerations:

None specified.

See Also:

FetchS3Object, DeleteS3Object, ListS3