Many of the standard Hadoop FileSystem shell commands that interact with HDFS also can be used to interact with S3, ADLS, and WASB. They can be useful for a few specific purposes including confirming that the authentication with your cloud service works, debugging, browsing files and creating directories (as an alternative to the cloud service-specific tools), and other management operations.

When running the commands, provide a fully qualified URL. The commands use the following syntax

hadoop fs -<operation> URL 

where <operation> indicates a particular action to be performed against a directory or a file.

For example, the following command lists all files in a directory called "dir1", which resides in an Amazon S3 bucket called "bucket1":

hadoop fs -ls s3a://bucket1/dir1

Examples

Create directories and create or copy files into them:

# Create a directory
hadoop fs -mkdir s3a://bucket1/datasets/

# Upload a file from the cluster filesystem
hadoop fs -put /datasets/example.orc s3a://bucket1/datasets/

# Touch a file
hadoop fs -touchz s3a://bucket1/datasetstouched

Download and view objects:

# Copy a directory to the local filesystem
hadoop fs -copyToLocal s3a://bucket1/datasets/

# Copy a file from the object store to the local filesystem
hadoop fs -get s3a://bucket1/hello.txt /examples

# Print the object
hadoop fs -cat s3a://bucket1/hello.txt

# Print the object, unzipping it if necessary
hadoop fs -text s3a://bucket1/hello.txt

# Download log files into a local file
hadoop fs -getmerge s3a://s3a://bucket1/logs\* log.txt

Related Links

Commands That May Be Slower with S3

Operations Unsupported for S3

Deleting Objects on S3

Overwriting Objects on S3

Timestamps on S3

Security Model and Operations on S3