A List of S3A Configuration Properties
The following fs.s3a
configuration properties are available. To override
these default s3a settings, add your configuration to your
core-site.xml
.
<property> <name>fs.s3a.connection.maximum</name> <value>15</value> <description>Controls the maximum number of simultaneous connections to S3.</description> </property> <property> <name>fs.s3a.connection.ssl.enabled</name> <value>true</value> <description>Enables or disables SSL connections to S3.</description> </property> <property> <name>fs.s3a.endpoint</name> <description>AWS S3 endpoint to connect to. An up-to-date list is provided in the AWS Documentation: regions and endpoints. Without this property, the standard region (s3.amazonaws.com) is assumed. </description> </property> <property> <name>fs.s3a.path.style.access</name> <value>false</value> <description>Enable S3 path style access ie disabling the default virtual hosting behaviour. Useful for S3A-compliant storage providers as it removes the need to set up DNS for virtual hosting. </description> </property> <property> <name>fs.s3a.proxy.host</name> <description>Hostname of the (optional) proxy server for S3 connections.</description> </property> <property> <name>fs.s3a.proxy.port</name> <description>Proxy server port. If this property is not set but fs.s3a.proxy.host is, port 80 or 443 is assumed (consistent with the value of fs.s3a.connection.ssl.enabled).</description> </property> <property> <name>fs.s3a.proxy.username</name> <description>Username for authenticating with proxy server.</description> </property> <property> <name>fs.s3a.proxy.password</name> <description>Password for authenticating with proxy server.</description> </property> <property> <name>fs.s3a.proxy.domain</name> <description>Domain for authenticating with proxy server.</description> </property> <property> <name>fs.s3a.proxy.workstation</name> <description>Workstation for authenticating with proxy server.</description> </property> <property> <name>fs.s3a.attempts.maximum</name> <value>20</value> <description>How many times we should retry commands on transient errors.</description> </property> <property> <name>fs.s3a.connection.establish.timeout</name> <value>5000</value> <description>Socket connection setup timeout in milliseconds.</description> </property> <property> <name>fs.s3a.connection.timeout</name> <value>200000</value> <description>Socket connection timeout in milliseconds.</description> </property> <property> <name>fs.s3a.paging.maximum</name> <value>5000</value> <description>How many keys to request from S3 when doing directory listings at a time.</description> </property> <property> <name>fs.s3a.threads.max</name> <value>10</value> <description> Maximum number of concurrent active (part)uploads, which each use a thread from the threadpool.</description> </property> <property> <name>fs.s3a.socket.send.buffer</name> <value>8192</value> <description>Socket send buffer hint to amazon connector. Represented in bytes.</description> </property> <property> <name>fs.s3a.socket.recv.buffer</name> <value>8192</value> <description>Socket receive buffer hint to amazon connector. Represented in bytes.</description> </property> <property> <name>fs.s3a.threads.keepalivetime</name> <value>60</value> <description>Number of seconds a thread can be idle before being terminated.</description> </property> <property> <name>fs.s3a.max.total.tasks</name> <value>5</value> <description>Number of (part)uploads allowed to the queue before blocking additional uploads.</description> </property> <property> <name>fs.s3a.multipart.size</name> <value>100M</value> <description>How big (in bytes) to split upload or copy operations up into. A suffix from the set {K,M,G,T,P} may be used to scale the numeric value. </description> </property> <property> <name>fs.s3a.multipart.threshold</name> <value>2147483647</value> <description>How big (in bytes) to split upload or copy operations up into. This also controls the partition size in renamed files, as rename() involves copying the source file(s). A suffix from the set {K,M,G,T,P} may be used to scale the numeric value. </description> </property> <property> <name>fs.s3a.multiobjectdelete.enable</name> <value>true</value> <description>When enabled, multiple single-object delete requests are replaced by a single 'delete multiple objects'-request, reducing the number of requests. Beware: legacy S3-compatible object stores might not support this request. </description> </property> <property> <name>fs.s3a.acl.default</name> <description>Set a canned ACL for newly created and copied objects. Value may be Private, PublicRead, PublicReadWrite, AuthenticatedRead, LogDeliveryWrite, BucketOwnerRead, or BucketOwnerFullControl.</description> </property> <property> <name>fs.s3a.multipart.purge</name> <value>false</value> <description>True if you want to purge existing multipart uploads that may not have been completed/aborted correctly</description> </property> <property> <name>fs.s3a.multipart.purge.age</name> <value>86400</value> <description>Minimum age in seconds of multipart uploads to purge</description> </property> <property> <name>fs.s3a.signing-algorithm</name> <description>Override the default signing algorithm so legacy implementations can still be used</description> </property> <property> <name>fs.s3a.server-side-encryption-algorithm</name> <description>Specify a server-side encryption algorithm for s3a: file system. Unset by default, and the only other currently allowable value is AES256. </description> </property> <property> <name>fs.s3a.buffer.dir</name> <value>${hadoop.tmp.dir}/s3a</value> <description>Comma separated list of directories that will be used to buffer file uploads to. No effect if fs.s3a.fast.upload is true.</description> </property> <property> <name>fs.s3a.block.size</name> <value>32M</value> <description>Block size to use when reading files using s3a: file system. </description> </property> <property> <name>fs.s3a.user.agent.prefix</name> <value></value> <description> Sets a custom value that will be prepended to the User-Agent header sent in HTTP requests to the S3 back-end by S3AFileSystem. The User-Agent header always includes the Hadoop version number followed by a string generated by the AWS SDK. An example is "User-Agent: Hadoop 2.8.0, aws-sdk-java/1.10.6". If this optional property is set, then its value is prepended to create a customized User-Agent. For example, if this configuration property was set to "MyApp", then an example of the resulting User-Agent would be "User-Agent: MyApp, Hadoop 2.8.0, aws-sdk-java/1.10.6". </description> </property> <property> <name>fs.s3a.readahead.range</name> <value>64K</value> <description>Bytes to read ahead during a seek() before closing and re-opening the S3 HTTP connection. This option will be overridden if any call to setReadahead() is made to an open stream.</description> </property>