Menu
AWS SDK for Go
Developer Guide

Performing Basic Amazon S3 Bucket Operations

These AWS SDK for Go examples show you how to perform the following operations on Amazon S3 buckets and bucket items:

  • List the buckets in your account

  • Create a bucket

  • List the items in a bucket

  • Upload a file to a bucket

  • Download a bucket item

  • Copy a bucket item to another bucket

  • Delete a bucket item

  • Delete all the items in a bucket

  • Restore a bucket item

  • Delete a bucket

You can download complete versions of these example files from the aws-doc-sdk-examples repository on GitHub.

Scenario

In these examples, a series of Go routines are used to perform operations on your Amazon S3 buckets. The routines use the AWS SDK for Go to perform Amazon S3 bucket operations using the following methods of the Amazon S3 client class, unless otherwise noted:

Prerequisites

List Buckets

The ListBuckets function lists the buckets in your account.

The following example lists the buckets in your account. There are no command line arguments.

Create the file s3_list_buckets.go. Add the following statements to import the Go and AWS SDK for Go packages used in the example.

Copy
import ( "github.com/aws/aws-sdk-go/aws" "github.com/aws/aws-sdk-go/aws/session" "github.com/aws/aws-sdk-go/service/s3" "fmt" "os" )

Create a function we use to display errors and exit.

Copy
func exitErrorf(msg string, args ...interface{}) { fmt.Fprintf(os.Stderr, msg+"\n", args...) os.Exit(1) }

Initialize the session that the SDK uses to load credentials from the shared credentials file ~/.aws/credentials, and create a new Amazon S3 service client.

Copy
sess, err := session.NewSession(&aws.Config{ Region: aws.String("us-west-2")}, ) // Create S3 service client svc := s3.New(sess)

Call ListBuckets. Passing nil means no filters are applied to the returned list. If an error occurs, call exitErrorf. If no error occurs, loop through the buckets, printing the name and creation date of each bucket.

Copy
result, err := svc.ListBuckets(nil) if err != nil { exitErrorf("Unable to list buckets, %v", err) } fmt.Println("Buckets:") for _, b := range result.Buckets { fmt.Printf("* %s created on %s\n", aws.StringValue(b.Name), aws.TimeValue(b.CreationDate)) }

See the complete example on GitHub.

Create a Bucket

The CreateBucket function creates a bucket in your account.

The following example creates a bucket with the name specified as a command line argument. You must specify a globally unique name for the bucket.

Create the file s3_create_bucket.go. Import the following Go and AWS SDK for Go packages.

Copy
import ( "github.com/aws/aws-sdk-go/aws" "github.com/aws/aws-sdk-go/aws/session" "github.com/aws/aws-sdk-go/service/s3" "fmt" "os" )

Create a function we use to display errors and exit.

Copy
func exitErrorf(msg string, args ...interface{}) { fmt.Fprintf(os.Stderr, msg+"\n", args...) os.Exit(1) }

The program requires one argument, the name of the bucket to create.

Copy
if len(os.Args) != 2 { exitErrorf("Bucket name missing!\nUsage: %s bucket_name", os.Args[0]) } bucket := os.Args[1]

Initialize the session that the SDK uses to load credentials from the shared credentials file ~/.aws/credentials, and create a new S3 service client.

Copy
sess, err := session.NewSession(&aws.Config{ Region: aws.String("us-west-2")}, ) // Create S3 service client svc := s3.New(sess)

Call CreateBucket, passing in the bucket name defined previously. If an error occurs, call exitErrorf. If there are no errors, wait for a notification that the bucket was created.

Copy
_, err = svc.CreateBucket(&s3.CreateBucketInput{ Bucket: aws.String(bucket), }) if err != nil { exitErrorf("Unable to create bucket %q, %v", bucket, err) } // Wait until bucket is created before finishing fmt.Printf("Waiting for bucket %q to be created...\n", bucket) err = svc.WaitUntilBucketExists(&s3.HeadBucketInput{ Bucket: aws.String(bucket), })

If the WaitUntilBucketExists call returns an error, call exitErrorf. If there are no errors, notify the user of success.

Copy
if err != nil { exitErrorf("Error occurred while waiting for bucket to be created, %v", bucket) } fmt.Printf("Bucket %q successfully created\n", bucket)

See the complete example on GitHub.

List Bucket Items

The ListObjects function lists the items in a bucket.

The following example lists the items in the bucket with the name specified as a command line argument.

Create the file s3_list_objects.go. Add the following statements to import the Go and AWS SDK for Go packages used in the example.

Copy
import ( "github.com/aws/aws-sdk-go/aws" "github.com/aws/aws-sdk-go/aws/session" "github.com/aws/aws-sdk-go/service/s3" "fmt" "os" )

Create a function we use to display errors and exit.

Copy
func exitErrorf(msg string, args ...interface{}) { fmt.Fprintf(os.Stderr, msg+"\n", args...) os.Exit(1) }

The program requires one command line argument, the name of the bucket.

Copy
if len(os.Args) != 2 { exitErrorf("Bucket name required\nUsage: %s bucket_name", os.Args[0]) } bucket := os.Args[1]

Initialize the session that the SDK uses to load credentials from the shared credentials file ~/.aws/credentials, and create a new Amazon S3 service client.

Copy
sess, err := session.NewSession(&aws.Config{ Region: aws.String("us-west-2")}, ) // Create S3 service client svc := s3.New(sess)

Call ListObjects, passing in the name of the bucket. If an error occurs, call exitErrorf. If no error occurs, loop through the items, printing the name, last modified date, size, and storage class of each item.

Copy
resp, err := svc.ListObjects(&s3.ListObjectsInput{Bucket: aws.String(bucket)}) if err != nil { exitErrorf("Unable to list items in bucket %q, %v", bucket, err) } for _, item := range resp.Contents { fmt.Println("Name: ", *item.Key) fmt.Println("Last modified:", *item.LastModified) fmt.Println("Size: ", *item.Size) fmt.Println("Storage class:", *item.StorageClass) fmt.Println("") }

See the complete example on GitHub.

Upload a File to a Bucket

The Upload function uploads an object to a bucket.

The following example uploads a file to a bucket with the names specified as command line arguments.

Create the file s3_upload_object.go. Add the following statements to import the Go and AWS SDK for Go packages used in the example.

Copy
import ( "github.com/aws/aws-sdk-go/aws" "github.com/aws/aws-sdk-go/aws/session" "github.com/aws/aws-sdk-go/service/s3/s3manager" "fmt" "os" )

Create a function we use to display errors and exit.

Copy
func exitErrorf(msg string, args ...interface{}) { fmt.Fprintf(os.Stderr, msg+"\n", args...) os.Exit(1) }

Get the bucket and file name from the command line arguments, open the file, and defer the file closing until we are done with it. If an error occurs, call exitErrorF.

Copy
if len(os.Args) != 3 { exitErrorf("bucket and file name required\nUsage: %s bucket_name filename", os.Args[0]) } bucket := os.Args[1] filename := os.Args[2] file, err := os.Open(filename) if err != nil { exitErrorf("Unable to open file %q, %v", err) } defer file.Close()

Initialize the session that the SDK uses to load credentials from the shared credentials file ~/.aws/credentials, and create a NewUploader object.

Copy
sess, err := session.NewSession(&aws.Config{ Region: aws.String("us-west-2")}, ) // Setup the S3 Upload Manager. Also see the SDK doc for the Upload Manager // for more information on configuring part size, and concurrency. // // http://docs.aws.amazon.com/sdk-for-go/api/service/s3/s3manager/#NewUploader uploader := s3manager.NewUploader(sess)

Upload the file to the bucket. If an error occurs, call exitErrorF. Otherwise, notify the user that the upload succeeded.

Copy
_, err = uploader.Upload(&s3manager.UploadInput{ Bucket: aws.String(bucket), Key: aws.String(filename), Body: file, }) if err != nil { // Print the error and exit. exitErrorf("Unable to upload %q to %q, %v", filename, bucket, err) } fmt.Printf("Successfully uploaded %q to %q\n", filename, bucket)

See the complete example on GitHub.

Download a File from a Bucket

The Download function downloads an object from a bucket.

The following example downloads an item from a bucket with the names specified as command line arguments.

Create the file s3_download_object.go. Add the following statements to import the Go and AWS SDK for Go packages used in the example.

Copy
import ( "github.com/aws/aws-sdk-go/aws" "github.com/aws/aws-sdk-go/aws/session" "github.com/aws/aws-sdk-go/service/s3" "github.com/aws/aws-sdk-go/service/s3/s3manager" "fmt" "os" )

Create a function we use to display errors and exit.

Copy
func exitErrorf(msg string, args ...interface{}) { fmt.Fprintf(os.Stderr, msg+"\n", args...) os.Exit(1) }

Get the bucket and file name from the command line arguments. If there aren't two arguments, call exitErrorf. Otherwise, create the file and defer file closing until we are done downloading. If an error occurs, call exitErrorf.

Copy
if len(os.Args) != 3 { exitErrorf("Bucket and item names required\nUsage: %s bucket_name item_name", os.Args[0]) } bucket := os.Args[1] item := os.Args[2] file, err := os.Create(item) if err != nil { exitErrorf("Unable to open file %q, %v", err) } defer file.Close()

Initialize the session that the SDK uses to load credentials from the shared credentials file ~/.aws/credentials, and create a NewDownloader object.

Copy
sess, err := session.NewSession(&aws.Config{ Region: aws.String("us-west-2")}, ) downloader := s3manager.NewDownloader(sess)
Copy
// credentials from the shared credentials file ~/.aws/credentials. sess, err := session.NewSession(&aws.Config{ Region: aws.String("us-west-2")}, ) downloader := s3manager.NewDownloader(sess)

Download the item from the bucket. If an error occurs, call exitErrorf. Otherwise, notify the user that the download succeeded.

Copy
numBytes, err := downloader.Download(file, &s3.GetObjectInput{ Bucket: aws.String(bucket), Key: aws.String(item), }) if err != nil { exitErrorf("Unable to download item %q, %v", item, err) } fmt.Println("Downloaded", file.Name(), numBytes, "bytes")

See the complete example on GitHub.

Copy an Item from one Bucket to Another

The CopyObject function copies an object from one bucket to another.

The following example copies an item from one bucket to another with the names specified as command line arguments.

Create the file s3_copy_object.go. Add the following statements to import the Go and AWS SDK for Go packages used in the example.

Copy
import ( "github.com/aws/aws-sdk-go/aws" "github.com/aws/aws-sdk-go/aws/session" "github.com/aws/aws-sdk-go/service/s3" "fmt" "os" )

Create a function we use to display errors and exit.

Copy
func exitErrorf(msg string, args ...interface{}) { fmt.Fprintf(os.Stderr, msg+"\n", args...) os.Exit(1) }

Get the names of the bucket containing the item, the item to copy, and the name of the bucket to which the item is copied. If there aren't four command line arguments, call exitErrorf.

Copy
if len(os.Args) != 4 { exitErrorf("Bucket, item, and other bucket names required\nUsage: go run s3_copy_object bucket item other-bucket") } bucket := os.Args[1] item := os.Args[2] other := os.Args[3] source := bucket + "/" + item

Initialize the session that the SDK uses to load credentials from the shared credentials file ~/.aws/credentials, and create a new Amazon S3 service client.

Copy
sess, err := session.NewSession(&aws.Config{ Region: aws.String("us-west-2")}, ) // Create S3 service client svc := s3.New(sess)

Call CopyObject, with the names of the bucket containing the item, the item to copy, and the name of the bucket to which the item is copied. If an error occurs, call exitErrorf. If no error occurs, wait for the item to be copied.

Copy
_, err = svc.CopyObject(&s3.CopyObjectInput{Bucket: aws.String(other), CopySource: aws.String(source), Key: aws.String(item)}) if err != nil { exitErrorf("Unable to copy item from bucket %q to bucket %q, %v", bucket, other, err) } // Wait to see if the item got copied err = svc.WaitUntilObjectExists(&s3.HeadObjectInput{Bucket: aws.String(other), Key: aws.String(item)})

If the WaitUntilObjectExists call returns an error, call exitErrorf. Otherwise, notify the user that the copy succeeded.

Copy
if err != nil { exitErrorf("Error occurred while waiting for item %q to be copied to bucket %q, %v", bucket, item, other, err) } fmt.Printf("Item %q successfully copied from bucket %q to bucket %q\n", item, bucket, other)

See the complete example on GitHub.

Delete an Item in a Bucket

The DeleteObject function deletes an object from a bucket.

The following example deletes an item from a bucket with the names specified as command line arguments.

Create the file s3_delete_object.go. Add the following statements to import the Go and AWS SDK for Go packages used in the example.

Copy
import ( "github.com/aws/aws-sdk-go/aws" "github.com/aws/aws-sdk-go/aws/session" "github.com/aws/aws-sdk-go/service/s3" "fmt" "os" )

Create a function we use to display errors and exit.

Copy
func exitErrorf(msg string, args ...interface{}) { fmt.Fprintf(os.Stderr, msg+"\n", args...) os.Exit(1) }

Get the name of the bucket and object to delete.

Copy
if len(os.Args) != 3 { exitErrorf("Bucket and object name required\nUsage: %s bucket_name object_name", os.Args[0]) } bucket := os.Args[1] obj := os.Args[2]

Initialize the session that the SDK uses to load credentials from the shared credentials file ~/.aws/credentials, and create a new Amazon S3 service client.

Copy
sess, err := session.NewSession(&aws.Config{ Region: aws.String("us-west-2")}, ) // Create S3 service client svc := s3.New(sess)

Call DeleteObject, passing in the names of the bucket and object to delete. If an error occurs, call exitErrorf. If no error occurs, wait until the object is deleted.

Copy
_, err = svc.DeleteObject(&s3.DeleteObjectInput{Bucket: aws.String(bucket), Key: aws.String(obj)}) if err != nil { exitErrorf("Unable to delete object %q from bucket %q, %v", obj, bucket, err) } err = svc.WaitUntilObjectNotExists(&s3.HeadObjectInput{ Bucket: aws.String(bucket), Key: aws.String(obj), })

If WaitUntilObjectNotExists returns an error, call exitErrorf. Otherwise, inform the user that the object was successfully deleted.

Copy
fmt.Printf("Object %q successfully deleted\n", obj)

See the complete example on GitHub.

Delete All the Items in a Bucket

The DeleteObjects function deletes objects from a bucket.

The following example deletes all the items from a bucket with the bucket name specified as a command line argument.

Create the file s3_delete_objects.go. Add the following statements to import the Go and AWS SDK for Go packages used in the example.

Copy
import ( "github.com/aws/aws-sdk-go/aws" "github.com/aws/aws-sdk-go/aws/session" "github.com/aws/aws-sdk-go/service/s3" "fmt" "os" )

Create a function we use to display errors and exit.

Copy
func exitErrorf(msg string, args ...interface{}) { fmt.Fprintf(os.Stderr, msg+"\n", args...) os.Exit(1) }

Get the name of the bucket.

Copy
if len(os.Args) != 2 { exitErrorf("Bucket name required\nUsage: %s BUCKET", os.Args[0]) } bucket := os.Args[1]

Initialize the session that the SDK uses to load credentials from the shared credentials file ~/.aws/credentials, and create a new Amazon S3 service client.

Copy
sess, _ := session.NewSession(&aws.Config{ Region: aws.String("us-west-2")}, ) // Create S3 service client svc := s3.New(sess)

Since buckets can have more than 1000 items and ListObjects returns up to 1000 items, set a flag to indicate when more items are available and keep track of how many items we get.

Copy
hasMoreObjects := true // Keep track of how many objects we delete totalObjects := 0

Loop through the bucket items up to 1000 at a time, creating the list of objects to delete from the list of items in the bucket. If an error occurs, call exitErrorf.

Copy
for hasMoreObjects { resp, err := svc.ListObjects(&s3.ListObjectsInput{Bucket: aws.String(bucket)}) if err != nil { exitErrorf("Unable to list items in bucket %q, %v", bucket, err) } numObjs := len(resp.Contents) totalObjects += numObjs // Create Delete object with slots for the objects to delete var items s3.Delete var objs= make([]*s3.ObjectIdentifier, numObjs) for i, o := range resp.Contents { // Add objects from command line to array objs[i] = &s3.ObjectIdentifier{Key: aws.String(*o.Key)} } // Add list of objects to delete to Delete object items.SetObjects(objs)

Call DeleteObjects, passing in the name of the bucket and the list of objects to delete. If an error occurs, call exitErrorf. Otherwise, determine whether there are more items in the bucket.

Copy
_, err = svc.DeleteObjects(&s3.DeleteObjectsInput{Bucket: &bucket, Delete: &items}) if err != nil { exitErrorf("Unable to delete objects from bucket %q, %v", bucket, err) } hasMoreObjects = *resp.IsTruncated

Once all of the items in the bucket have been deleted, inform the user of the number of objects deleted.

Copy
fmt.Println("Deleted", totalObjects, "object(s) from bucket", bucket)

See the complete example on GitHub.

Restore a Bucket Item

The RestoreObject function restores an item in a bucket.

The following example restores the items in a bucket with the names specified as command line arguments.

Create the file s3_restore_object.go. Add the following statements to import the Go and AWS SDK for Go packages used in the example.

Copy
import ( "github.com/aws/aws-sdk-go/aws" "github.com/aws/aws-sdk-go/aws/session" "github.com/aws/aws-sdk-go/service/s3" "fmt" "os" )

Create a function we use to display errors and exit.

Copy
func exitErrorf(msg string, args ...interface{}) { fmt.Fprintf(os.Stderr, msg+"\n", args...) os.Exit(1) }

The program requires two arguments, the names of the bucket and object to restore.

Copy
if len(os.Args) != 3 { exitErrorf("Bucket name and object name required\nUsage: %s bucket_name object_name", os.Args[0]) } bucket := os.Args[1] obj := os.Args[2]

Initialize the session that the SDK uses to load credentials from the shared credentials file ~/.aws/credentials, and create a new Amazon S3 service client.

Copy
sess, err := session.NewSession(&aws.Config{ Region: aws.String("us-west-2")}, ) // Create S3 service client svc := s3.New(sess)

Call RestoreObject, passing in the bucket and object names and the number of days to temporarily restore. If an error occurs, call exitErrorf. Otherwise, inform the user that the bucket should be restored in the next four hours or so.

Copy
_, err = svc.RestoreObject(&s3.RestoreObjectInput{Bucket: aws.String(bucket), Key: aws.String(obj), RestoreRequest: &s3.RestoreRequest{Days: aws.Int64(30)}}) if err != nil { exitErrorf("Could not restore %s in bucket %s, %v", obj, bucket, err) } fmt.Printf("%q should be restored to %q in about 4 hours\n", obj, bucket)

See the complete example on GitHub.

Delete a Bucket

The DeleteBucket function deletes a bucket.

The following example deletes the bucket with the name specified as a command line argument.

Create the file s3_delete_bucket.go. Import the following Go and AWS SDK for Go packages.

Copy
import ( "github.com/aws/aws-sdk-go/aws" "github.com/aws/aws-sdk-go/aws/session" "github.com/aws/aws-sdk-go/service/s3" "fmt" "os" )

Create a function we use to display errors and exit.

Copy
func exitErrorf(msg string, args ...interface{}) { fmt.Fprintf(os.Stderr, msg+"\n", args...) os.Exit(1) }

The program requires one argument, the name of the bucket to delete. If the argument is not supplied, call exitErrorf.

Copy
if len(os.Args) != 2 { exitErrorf("bucket name required\nUsage: %s bucket_name", os.Args[0]) } bucket := os.Args[1]

Initialize the session that the SDK uses to load credentials from the shared credentials file ~/.aws/credentials, and create a new S3 service client.

Copy
sess, err := session.NewSession(&aws.Config{ Region: aws.String("us-west-2")}, ) // Create S3 service client svc := s3.New(sess)

Call DeleteBucket, passing in the bucket name. If an error occurs, call exitErrorf. If there are no errors, wait for a notification that the bucket was deleted.

Copy
_, err = svc.DeleteBucket(&s3.DeleteBucketInput{ Bucket: aws.String(bucket), }) if err != nil { exitErrorf("Unable to delete bucket %q, %v", bucket, err) } // Wait until bucket is deleted before finishing fmt.Printf("Waiting for bucket %q to be deleted...\n", bucket) err = svc.WaitUntilBucketNotExists(&s3.HeadBucketInput{ Bucket: aws.String(bucket), })

If WaitUntilBucketNotExists returns an error, call exitErrorf. Otherwise, inform the user that the bucket was successfully deleted.

Copy
if err != nil { exitErrorf("Error occurred while waiting for bucket to be deleted, %v", bucket) } fmt.Printf("Bucket %q successfully deleted\n", bucket)

See the complete example on GitHub.