Using high-level (s3) commands with the AWS CLI

This topic describes how you can manage Amazon S3 buckets and objects using the aws s3 commands in the AWS CLI.

The high-level aws s3 commands simplify managing Amazon S3 objects. These commands enable you to manage the contents of Amazon S3 within itself and with local directories.

Note

When you use aws s3 commands to upload large objects to an Amazon S3 bucket, the AWS CLI automatically performs a multipart upload. You can't resume a failed upload when using these aws s3 commands.

If the multipart upload fails due to a timeout, or if you manually canceled in the AWS CLI, the AWS CLI stops the upload and cleans up any files that were created. This process can take several minutes.

If the multipart upload or cleanup process is canceled by a kill command or system failure, the created files remain in the Amazon S3 bucket. To clean up the multipart upload, use the s3api abort-multipart-upload command.

For more information, see Multipart upload overview in the Amazon Simple Storage Service Developer Guide.

Prerequisites

To run the s3 commands, you need to:

• AWS CLI installed, see Installing, updating, and uninstalling the AWS CLI for more information.

• AWS CLI configured, see Configuration basics for more information. The profile that you use must have permissions that allow the AWS operations performed by the examples.

• Understand these Amazon S3 terms:

  • Bucket – A top-level Amazon S3 folder.

  • Prefix – An Amazon S3 folder in a bucket.

  • Object – Any item that's hosted in an Amazon S3 bucket.

Create a bucket

Use the s3 mb command to make a bucket. Bucket names must be globally unique (unique across all of Amazon S3) and should be DNS compliant.

Bucket names can contain lowercase letters, numbers, hyphens, and periods. Bucket names can start and end only with a letter or number, and cannot contain a period next to a hyphen or another period.

Syntax

$ aws s3 mb <target> [--options]

s3 mb examples

The following example creates the s3://bucket-name bucket.

$ aws s3 mb s3://bucket-name

List buckets and objects

To list your buckets, folders, or objects, use the s3 ls command. Using the command without a target or options lists all buckets.

Syntax

$ aws s3 ls <target> [--options]

For a few common options to use with this command, and examples, see Frequently used options for s3 commands. For a complete list of available options, see s3 ls in the AWS CLI Command Reference.

s3 ls examples

The following example lists all of your Amazon S3 buckets.

$ aws s3 ls
2018-12-11 17:08:50 my-bucket
2018-12-14 14:55:44 my-bucket2

The following command lists all objects and prefixes in a bucket. In this example output, the prefix example/ has one file named MyFile1.txt.

$ aws s3 ls s3://bucket-name
                           PRE example/
2018-12-04 19:05:48          3 MyFile1.txt

You can filter the output to a specific prefix by including it in the command. The following command lists the objects in bucket-name/example/ (that is, objects in bucket-name filtered by the prefix example/).

$ aws s3 ls s3://bucket-name/example/
2018-12-06 18:59:32          3 MyFile1.txt

Delete buckets

To delete a bucket, use the s3 rb command.

Syntax

$ aws s3 rb <target> [--options]

s3 rb examples

The following example removes the s3://bucket-name bucket.

$ aws s3 rb s3://bucket-name

By default, the bucket must be empty for the operation to succeed. To remove a bucket that's not empty, you need to include the --force option. If you're using a versioned bucket that contains previously deleted—but retained—objects, this command does not allow you to remove the bucket. You must first remove all of the content.

The following example deletes all objects and prefixes in the bucket, and then deletes the bucket.

$ aws s3 rb s3://bucket-name --force

Delete objects

To delete objects in a bucket or your local directory, use the s3 rm command.

Syntax

$ aws s3 rm  <target> [--options]

For a few common options to use with this command, and examples, see Frequently used options for s3 commands. For a complete list of options, see s3 rm in the AWS CLI Command Reference.

s3 rm examples

The following example deletes filename.txt from s3://bucket-name/example.

$ aws s3 rm s3://bucket-name/example/filename.txt --recursive

The following example deletes all objects from s3://bucket-name/example using the --recursive option.

$ aws s3 rm s3://bucket-name/example --recursive

Move objects

Use the s3 mv command to move objects from a bucket or a local directory.

Syntax

$ aws s3 mv <source> <target> [--options]

For a few common options to use with this command, and examples, see Frequently used options for s3 commands. For a complete list of available options, see s3 mv in the AWS CLI Command Reference.

s3 mv examples

The following example moves all objects from s3://bucket-name/example to s3://my-bucket/.

$ aws s3 mv s3://bucket-name/example s3://my-bucket/

The following example moves a local file from your current working directory to the Amazon S3 bucket with the s3 cp command.

$ aws s3 mv filename.txt s3://bucket-name

The following example moves a file from your Amazon S3 bucket to your current working directory, where ./ specifies your current working directory.

$ aws s3 mv s3://bucket-name/filename.txt ./

Copy objects

Use the s3 cp command to copy objects from a bucket or a local directory.

Syntax

$ aws s3 cp <source> <target> [--options]

You can use the dash parameter for file streaming to standard input (stdin) or standard output (stdout).

Warning

If you're using PowerShell, the shell might alter the encoding of a CRLF or add a CRLF to piped input or output, or redirected output.

The s3 cp command uses the following syntax to upload a file stream from stdin to a specified bucket.

Syntax

$ aws s3 cp - <target> [--options]

The s3 cp command uses the following syntax to download an Amazon S3 file stream for stdout.

Syntax

$ aws s3 cp <target> [--options] -

For a few common options to use with this command, and examples, see Frequently used options for s3 commands. For the complete list of options, see s3 cp in the AWS CLI Command Reference.

s3 cp examples

The following example copies all objects from s3://bucket-name/example to s3://my-bucket/.

$ aws s3 cp s3://bucket-name/example s3://my-bucket/

The following example copies a local file from your current working directory to the Amazon S3 bucket with the s3 cp command.

$ aws s3 cp filename.txt s3://bucket-name

The following example copies a file from your Amazon S3 bucket to your current working directory, where ./ specifies your current working directory.

$ aws s3 cp s3://bucket-name/filename.txt ./

The following example uses echo to stream the text "hello world" to the s3://bucket-name/filename.txt file.

$ echo "hello world" | aws s3 cp - s3://bucket-name/filename.txt

The following example streams the s3://bucket-name/filename.txt file to stdout and prints the contents to the console.

$ aws s3 cp s3://bucket-name/filename.txt -
hello world

The following example streams the contents of s3://bucket-name/pre to stdout, uses the bzip2 command to compress the files, and uploads the new compressed file named key.bz2 to s3://bucket-name.

$ aws s3 cp s3://bucket-name/pre - | bzip2 --best | aws s3 cp - s3://bucket-name/key.bz2

Sync objects

The s3 sync command synchronizes the contents of a bucket and a directory, or the contents of two buckets. Typically, s3 sync copies missing or outdated files or objects between the source and target. However, you can also supply the --delete option to remove files or objects from the target that are not present in the source.

Syntax

$ aws s3 sync <source> <target> [--options]

For a few common options to use with this command, and examples, see Frequently used options for s3 commands. For a complete list of options, see s3 sync in the AWS CLI Command Reference.

s3 sync examples

The following example synchronizes the contents of an Amazon S3 prefix named path in the bucket named my-bucket with the current working directory.

s3 sync updates any files that have a size or modified time that are different from files with the same name at the destination. The output displays specific operations performed during the sync. Notice that the operation recursively synchronizes the subdirectory MySubdirectory and its contents with s3://my-bucket/path/MySubdirectory.

$ aws s3 sync . s3://my-bucket/path
upload: MySubdirectory\MyFile3.txt to s3://my-bucket/path/MySubdirectory/MyFile3.txt
upload: MyFile2.txt to s3://my-bucket/path/MyFile2.txt
upload: MyFile1.txt to s3://my-bucket/path/MyFile1.txt

The following example, which extends the previous one, shows how to use the --delete option.

// Delete local file
$ rm ./MyFile1.txt

// Attempt sync without --delete option - nothing happens
$ aws s3 sync . s3://my-bucket/path

// Sync with deletion - object is deleted from bucket
$ aws s3 sync . s3://my-bucket/path --delete
delete: s3://my-bucket/path/MyFile1.txt

// Delete object from bucket
$ aws s3 rm s3://my-bucket/path/MySubdirectory/MyFile3.txt
delete: s3://my-bucket/path/MySubdirectory/MyFile3.txt

// Sync with deletion - local file is deleted
$ aws s3 sync s3://my-bucket/path . --delete
delete: MySubdirectory\MyFile3.txt

// Sync with Infrequent Access storage class
$ aws s3 sync . s3://my-bucket/path --storage-class STANDARD_IA

When using the --delete option, the --exclude and --include options can filter files or objects to delete during an s3 sync operation. In this case, the parameter string must specify files to exclude from, or include for, deletion in the context of the target directory or bucket. The following shows an example.

Assume local directory and s3://my-bucket/path currently in sync and each contains 3 files:
MyFile1.txt
MyFile2.rtf
MyFile88.txt
'''

// Sync with delete, excluding files that match a pattern. MyFile88.txt is deleted, while remote MyFile1.txt is not.
$ aws s3 sync . s3://my-bucket/path --delete --exclude "path/MyFile?.txt"
delete: s3://my-bucket/path/MyFile88.txt
'''

// Sync with delete, excluding MyFile2.rtf - local file is NOT deleted
$ aws s3 sync s3://my-bucket/path . --delete --exclude "./MyFile2.rtf"
download: s3://my-bucket/path/MyFile1.txt to MyFile1.txt
'''

// Sync with delete, local copy of MyFile2.rtf is deleted
$ aws s3 sync s3://my-bucket/path . --delete
delete: MyFile2.rtf

Frequently used options for s3 commands

The following options are frequently used for the commands described in this topic. For a complete list of options you can use on a command, see the specific command in the AWS CLI Command Reference.

acl

s3 sync and s3 cp can use the --acl option. This enables you to set the access permissions for files copied to Amazon S3. The --acl option accepts private, public-read, and public-read-write values. For more information, see Canned ACL in the Amazon Simple Storage Service Developer Guide.

$ aws s3 sync . s3://my-bucket/path --acl public-read

exclude

When you use the s3 cp, s3 mv, s3 sync, or s3 rm command, you can filter the results by using the --exclude or --include option. The --exclude option sets rules to only exclude objects from the command, and the options apply in the order specified. This is shown in the following example.

Local directory contains 3 files:
MyFile1.txt
MyFile2.rtf
MyFile88.txt

// Exclude all .txt files, resulting in only MyFile2.rtf being copied
$ aws s3 cp . s3://my-bucket/path --exclude "*.txt"

// Exclude all .txt files but include all files with the "MyFile*.txt" format, resulting in, MyFile1.txt, MyFile2.rtf, MyFile88.txt being copied
$ aws s3 cp . s3://my-bucket/path --exclude "*.txt" --include "MyFile*.txt"

// Exclude all .txt files, but include all files with the "MyFile*.txt" format, but exclude all files with the "MyFile?.txt" format resulting in, MyFile2.rtf and MyFile88.txt being copied
$ aws s3 cp . s3://my-bucket/path --exclude "*.txt" --include "MyFile*.txt" --exclude "MyFile?.txt"

include

When you use the s3 cp, s3 mv, s3 sync, or s3 rm command, you can filter the results using the --exclude or --include option. The --include option sets rules to only include objects specified for the command, and the options apply in the order specified. This is shown in the following example.

Local directory contains 3 files:
MyFile1.txt
MyFile2.rtf
MyFile88.txt

// Include all .txt files, resulting in MyFile1.txt and MyFile88.txt being copied
$ aws s3 cp . s3://my-bucket/path --include "*.txt"

// Include all .txt files but exclude all files with the "MyFile*.txt" format, resulting in no files being copied
$ aws s3 cp . s3://my-bucket/path --include "*.txt" --exclude "MyFile*.txt"

// Include all .txt files, but exclude all files with the "MyFile*.txt" format, but include all files with the "MyFile?.txt" format resulting in MyFile1.txt being copied

$ aws s3 cp . s3://my-bucket/path --include "*.txt" --exclude "MyFile*.txt" --include "MyFile?.txt"

grant

The s3 cp, s3 mv, and s3 sync commands include a --grants option that you can use to grant permissions on the object to specified users or groups. Set the --grants option to a list of permissions using the following syntax. Replace Permission, Grantee_Type, and Grantee_ID with your own values.

Syntax

--grants Permission=Grantee_Type=Grantee_ID
         [Permission=Grantee_Type=Grantee_ID ...]

Each value contains the following elements:

• Permission – Specifies the granted permissions. Can be set to read, readacl, writeacl, or full.

• Grantee_Type – Specifies how to identify the grantee. Can be set to uri, emailaddress, or id.

• Grantee_ID – Specifies the grantee based on Grantee_Type.

  • uri – The group's URI. For more information, see Who is a grantee?

  • emailaddress – The account's email address.

  • id – The account's canonical ID.

For more information about Amazon S3 access control, see Access control.

The following example copies an object into a bucket. It grants read permissions on the object to everyone, and full permissions (read, readacl, and writeacl) to the account associated with [e-mail].

$ aws s3 cp file.txt s3://my-bucket/ --grants read=uri=http://acs.amazonaws.com/groups/global/AllUsers full=emailaddress=[e-mail]

You can also specify a nondefault storage class (REDUCED_REDUNDANCY or STANDARD_IA) for objects that you upload to Amazon S3. To do this, use the --storage-class option.

$ aws s3 cp file.txt s3://my-bucket/ --storage-class REDUCED_REDUNDANCY

recursive

When you use this option, the command is performed on all files or objects under the specified directory or prefix. The following example deletes s3://my-bucket/path and all of its contents.

$ aws s3 rm s3://my-bucket/path --recursive

Resources

AWS CLI reference:

• aws s3

• aws s3 cp

• aws s3 mb

• aws s3 mv

• aws s3 ls

• aws s3 rb

• aws s3 rm

• aws s3 sync

Service reference:

• Working with Amazon S3 buckets in the Amazon Simple Storage Service Developer Guide

• Working with Amazon S3 objects in the Amazon Simple Storage Service Developer Guide

• Listing keys hierarchically using a prefix and delimiter in the Amazon Simple Storage Service Developer Guide

• Abort multipart uploads to an S3 bucket using the AWS SDK for .NET (low-level) in the Amazon Simple Storage Service Developer Guide