Working with S3 buckets on a Snowball Edge device

You can create Amazon S3 buckets on your Snowball Edge devices to store and retrieve objects on premises for applications that require local data access, local data processing, and data residency. Amazon S3 compatible storage on Snow Family devices provides a new storage class, SNOW, which uses the Amazon S3 APIs, and is designed to store data durably and redundantly across multiple Snowball Edge devices. You can use the same APIs and features on Snowball Edge buckets that you do on Amazon S3, including bucket lifecycle policies, encryption, and tagging.

Using the AWS CLI

Follow these instructions to work with Amazon S3 buckets on your device using the AWS CLI.

To set up the AWS CLI

1. Create a profile for object endpoints in ~/.aws/config.

   [profile your-profile]
   aws_access_key_id = your-access-id
   aws_secret_access_key = your-access-key
   region = snow
   ca_bundle = dev/apps/ca-certs/your-ca_bundle
   

2. Obtain a certificate from your device. For information, see the Snowball Edge Developer Guide.

3. If you installed the SDK in a virtual environment, activate it using the following command:

   source your-virtual-environment-name/bin/activate

After you set up your operations, you can access them using API calls with the AWS CLI. In the following examples, cert is the device certificate you just obtained using IAM.

Accessing object operations

aws s3api --profile your-profile list-objects-v2 --endpoint-url https://s3api-endpoint-ip

Accessing bucket operations

aws s3control --profile your-profile list-regional-buckets --account-id bucket-owner --endpoint-url https://s3ctrlapi-endpoint-ip

Using the Java SDK

Use the following example to work with Amazon S3 objects using the Java SDK.

import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
import software.amazon.awssdk.http.SdkHttpClient;
import software.amazon.awssdk.http.apache.ApacheHttpClient;
import software.amazon.awssdk.regions.Region;

import java.net.URI;

AwsBasicCredentials creds = AwsBasicCredentials.create(accessKey, secretKey); // set creds by getting Access Key and Secret Key from snowball edge
SdkHttpClient httpClient = ApacheHttpClient.builder().tlsTrustManagersProvider(trustManagersProvider).build(); // set trust managers provider with client certificate from snowball edge
String s3SnowEndpoint = "10.0.0.0"; // set s3-snow object api endpoint from describe service

S3Client s3Client = S3Client.builder().httpClient(httpClient).region(Region.of("snow")).endpointOverride(new URI(s3SnowEndpoint)).credentialsProvider(StaticCredentialsProvider.create(creds)).build();
                

Bucket ARN format

You can use the Amazon Resource Name (ARN) format listed here to identify an Amazon S3 bucket on a Snowball Edge device:

arn:partition:s3:snow:account-id:device/device-id/bucket/bucket-name

Where partition is the partition of the Region where you ordered your Snowball Edge device. device-id is the job_id if the device is a standalone Snowball Edge device, or the cluster_id if you have a Snowball Edge cluster.

Creating an S3 bucket on a Snowball Edge device

You can create Amazon S3 buckets on your Snowball Edge device to store and retrieve objects at the edge for applications that require local data access, local data processing, and data residency. Amazon S3 compatible storage on Snow Family devices provides a new storage class, SNOW, which uses Amazon S3 and is designed to store data durably and redundantly across multiple devices . You can use the same APIs and features as you do on Amazon S3 buckets, including bucket lifecycle policies, encryption, and tagging.

The following example creates an Amazon S3 bucket for a Snowball Edge device using the AWS CLI. To run this command, replace the user input placeholders with your own information.

aws s3control --profile your-profile create-bucket --bucket your-snow-bucket --endpoint-url https://s3ctrlapi-endpoint-ip

Creating and managing an object lifecycle configuration using the AWS CLI

You can use Amazon S3 Lifecycle to optimize storage capacity for Amazon S3 compatible storage on Snow Family devices. You can create lifecycle rules to expire objects as they age or are replaced by newer versions. You can create, enable, disable, or delete a lifecycle rule. For more information about Amazon S3 Lifecycle, see Managing your storage lifecycle.

Note

The AWS account that creates the bucket owns it and is the only one that can create, enable, disable, or delete a lifecycle rule.

To create and manage a lifecycle configuration for an Amazon S3 compatible storage on Snow Family devices bucket using the AWS Command Line Interface (AWS CLI), see the following examples.

PUT a lifecycle configuration on a Snowball Edge bucket

The following AWS CLI example puts a lifecycle configuration policy on a Snowball Edge bucket. This policy specifies that all objects that have the flagged prefix (myprefix) and tags expire after 10 days. To use this example, replace each user input placeholder with your own information.

First, save the lifecycle configuration policy to a JSON file. For this example, the file is named lifecycle-example.json.

{
    "Rules": [{
        "ID": "id-1",
        "Filter": {
            "And": {
                "Prefix": "myprefix",
                "Tags": [{
                        "Value": "mytagvalue1",
                        "Key": "mytagkey1"
                    },
                    {
                        "Value": "mytagvalue2",
                        "Key": "mytagkey2"
                    }
                ],
            }
        },
        "Status": "Enabled",
        "Expiration": {
            "Days": 10
        }
    }]
}

After you save the file, submit the JSON file as part of the put-bucket-lifecycle-configuration command. To use this command, replace each user input placeholder with your own information.

aws s3control put-bucket-lifecycle-configuration --bucket 
                example-snow-bucket --profile your-profile 
                --lifecycle-configuration file://lifecycle-example.json --endpoint-url https://s3ctrlapi-endpoint-ip

For more information about this command, see put-bucket-lifecycle-configuration in the AWS CLI Command Reference.

Working with S3 buckets on a Snowball Edge device

With Amazon S3 compatible storage on Snow Family devices, you can create Amazon S3 buckets on your Snowball Edge devices to store and retrieve objects on premises for applications that require local data access, local data processing, and data residency. Amazon S3 compatible storage on Snow Family devices provides a new storage class, SNOW, which uses the Amazon S3 APIs, and is designed to store data durably and redundantly across multiple Snowball Edge devices. You can use the same APIs and features on Snowball Edge buckets that you do on Amazon S3, including bucket lifecycle policies, encryption, and tagging. You can use Amazon S3 compatible storage on Snow Family devices using the AWS Command Line Interface (AWS CLI) or AWS SDKs.

Determine whether you can access an Amazon S3 compatible storage on Snow Family devices bucket

The following example uses the head-bucket command to determine if an Amazon S3 bucket exists and you have permissions to access it using the AWS CLI. To use this command, replace each user input placeholder with your own information.

aws s3api head-bucket --bucket sample-bucket --profile your-profile --endpoint-url https://s3api-endpoint-ip

Retrieve a list of buckets or regional buckets

Use the list-regional-buckets or list buckets to list Amazon S3 compatible storage on Snow Family devices buckets using the AWS CLI.

aws s3control list-regional-buckets --account-id 123456789012 --profile your-profile --endpoint-url https://s3ctrlapi-endpoint-ip

For more information about the list-regional-buckets command, see list-regional-buckets in the AWS CLI Command Reference.

 aws s3 list-buckets --account-id 123456789012 --endpoint-url https://s3api-endpoint-ip

For more information about the list-buckets command, see list-buckets in the AWS CLI Command Reference

The following SDK for Java example gets a list of buckets on Snowball Edge devices. For more information, see ListBuckets in the Amazon Simple Storage Service API Reference.

  import com.amazonaws.services.s3.model.*;
  public void listBuckets() {
    ListBucketsRequest reqListBuckets = new ListBucketsRequest()
    .withAccountId(AccountId)
    ListBucketsResult respListBuckets = s3APIClient.RegionalBuckets(reqListBuckets);
    System.out.printf("ListBuckets Response: %s%n", respListBuckets.toString());
  }               
            

The following PowerShell example gets a list of buckets on Snowball Edge devices.

   Get-S3CRegionalBucketList -AccountId 012345678910 -Endpoint "https://snowball_ip" -Region snow    

The following .NET example gets a list of buckets on Snowball Edge devices.

using Amazon.S3Control;
using Amazon.S3Control.Model;

namespace SnowTest;

internal class Program
{
    static async Task Main(string[] args)
    {
        var config = new AmazonS3ControlConfig
        {
            ServiceURL = "https://snowball_ip",
            AuthenticationRegion = "snow" // Note that this is not RegionEndpoint
        };        

        var client = new AmazonS3ControlClient(config);

        var response = await client.ListRegionalBucketsAsync(new ListRegionalBucketsRequest() 
        { 
            AccountId = "012345678910" 
        });
    }
}                
            

Get a bucket

The following example gets an Amazon S3 compatible storage on Snow Family devices bucket using the AWS CLI. To use this command, replace each user input placeholder with your own information.

aws s3control get-bucket --account-id 123456789012 --bucket DOC-EXAMPLE-BUCKET --profile your-profile --endpoint-url https://s3ctrlapi-endpoint-ip

For more information about this command, see get-bucket in the AWS CLI Command Reference.

The following Amazon S3 compatible storage on Snow Family devices example gets a bucket using the SDK for Java. For more information, see GetBucket in the Amazon Simple Storage Service API Reference.

import com.amazonaws.services.s3control.model.*;

public void getBucket(String bucketName) {

    GetBucketRequest reqGetBucket = new GetBucketRequest()
            .withBucket(bucketName)
            .withAccountId(AccountId);

    GetBucketResult respGetBucket = s3ControlClient.getBucket(reqGetBucket);
    System.out.printf("GetBucket Response: %s%n", respGetBucket.toString());
}                
            

Delete a bucket

Important

• The AWS account that creates the bucket owns it and is the only one that can delete it.

• Snow Family devices buckets must be empty before they can be deleted.

• You cannot recover a bucket after it has been deleted.

The following example deletes an Amazon S3 compatible storage on Snow Family devices bucket using the AWS CLI. To use this command, replace each user input placeholder with your own information.

aws s3control delete-bucket --account-id 123456789012 --bucket DOC-EXAMPLE-BUCKET --profile your-profile --endpoint-url https://s3ctrlapi-endpoint-ip

For more information about this command, see delete-bucket in the AWS CLI Command Reference.