Working with S3 buckets with Amazon S3 compatible storage on Snow Family devices
With Amazon S3 compatible storage on Snow Family devices, you can create Amazon S3 buckets on your Snowball Edge devices to store and
retrieve objects on premises for applications that require local data access, local data
processing, and data residency. Amazon S3 compatible storage on Snow Family devices provides a new storage class, SNOW
,
which uses the Amazon S3 APIs, and is designed to store data durably and
redundantly across multiple Snowball Edge devices. You can use the same APIs and
features on Snowball Edge buckets that you do on Amazon S3, including bucket lifecycle policies,
encryption, and tagging.
You can use Amazon S3 compatible storage on Snow Family devices using the AWS Command Line Interface (AWS CLI) or programatically through the AWS Java SDK. With the AWS CLI, you can set up an s3api or s3control endpoint and interact with it through commands. We recommend using the s3api endpoint because the same endpoint can be used for bucket and object operations.
Note
The s3api endpoint is available for version 8004 and newer of the Snowball Edge software. To find the version of the Snowball Edge software installed on a device, use the snowballEdge check-for-updates
command. To update a Snowball Edge device, see Updating software on Snowball Edge devices.
Using the AWS CLI
Follow these instructions to work with Amazon S3 buckets on your device using the AWS CLI.
To set up the AWS CLI
-
Create a profile for object endpoints in
~/.aws/config
.[profile
your-profile
] aws_access_key_id =your-access-id
aws_secret_access_key =your-access-key
region = snow ca_bundle = dev/apps/ca-certs/your-ca_bundle
-
Obtain a certificate from your device. For information, see the Snowball Edge Developer Guide.
-
If you installed the SDK in a virtual environment, activate it using the following command:
source
your-virtual-environment-name
/bin/activate
After you set up your operations, you can use the s3api SDK or s3control SDK to access S3 buckets on Snow Family devices with the AWS CLI.
Example of accessing S3 bucket using the s3api SDK
aws s3api --profile
your-profile
list-buckets --endpoint-url https://s3api-endpoint-ip
Example of accecssing S3 buckets using the s3control SDK
aws s3control --profile
your-profile
list-regional-buckets --account-idbucket-owner
--endpoint-url https://s3ctrlapi-endpoint-ip
Example of accessing S3 objects using the s3api SDK
aws s3api --profile
your-profile
list-objects-v2 --endpoint-url https://s3api-endpoint-ip
Using the Java SDK
Use the following example to work with Amazon S3 buckets and objects using the Java SDK.
import software.amazon.awssdk.services.s3.S3Client; import software.amazon.awssdk.auth.credentials.AwsBasicCredentials; import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider; import software.amazon.awssdk.http.SdkHttpClient;bg import software.amazon.awssdk.http.apache.ApacheHttpClient; import software.amazon.awssdk.regions.Region; import java.net.URI; AwsBasicCredentials creds = AwsBasicCredentials.create(accessKey, secretKey); // set creds by getting Access Key and Secret Key from snowball edge SdkHttpClient httpClient = ApacheHttpClient.builder().tlsTrustManagersProvider(trustManagersProvider).build(); // set trust managers provider with client certificate from snowball edge String s3SnowEndpoint = "10.0.0.0"; // set s3-snow object api endpoint from describe service S3Client s3Client = S3Client.builder().httpClient(httpClient).region(Region.of("snow")).endpointOverride(new URI(s3SnowEndpoint)).credentialsProvider(StaticCredentialsProvider.create(creds)).build();
Bucket ARN format
You can use the Amazon Resource Name (ARN) format listed here to identify an Amazon S3 bucket on a Snowball Edge device:
arn:
partition
:s3:snow:account-id
:device
/device-id
/bucket/bucket-name
Where partition
is the partition of the Region where
you ordered your Snowball Edge device. device-id
is the
job_id if the device is a standalone Snowball Edge device, or the
cluster_id
if you have a Snowball Edge
cluster.
Bucket location format
The bucket location format specifies the Snowball Edge device where the bucket will be createed. The bucket location has the following format:
/device-id/bucket/bucket-name
For more information, see create-bucket
Creating an S3 bucket on a Snowball Edge device
You can create Amazon S3 buckets on your Snowball Edge device to store and retrieve
objects at the edge for applications that require local data access, local data
processing, and data residency. Amazon S3 compatible storage on Snow Family devices provides a new storage class,
SNOW
, which uses Amazon S3 and is designed to store data durably and
redundantly across multiple devices . You can use the same APIs and features as you
do on Amazon S3 buckets, including bucket lifecycle policies, encryption, and
tagging.
The following example creates an Amazon S3 bucket for a Snowball Edge device using the AWS CLI. To run this command, replace the user input placeholders with your own information.
Example of creating an S3 bucket
Creating and managing an object lifecycle configuration using the AWS CLI
You can use Amazon S3 Lifecycle to optimize storage capacity for Amazon S3 compatible storage on Snow Family devices. You can create lifecycle rules to expire objects as they age or are replaced by newer versions. You can create, enable, disable, or delete a lifecycle rule. For more information about Amazon S3 Lifecycle, see Managing your storage lifecycle.
Note
The AWS account that creates the bucket owns it and is the only one that can create, enable, disable, or delete a lifecycle rule.
To create and manage a lifecycle configuration for an Amazon S3 compatible storage on Snow Family devices bucket using the AWS Command Line Interface (AWS CLI), see the following examples.
PUT a lifecycle configuration on a Snowball Edge bucket
The following AWS CLI example puts a lifecycle configuration policy on a
Snowball Edge bucket. This policy specifies that all objects that have the
flagged prefix (myprefix
) and tags expire after 10
days. To use this example, replace each user input placeholder with your own
information.
First, save the lifecycle configuration policy to a JSON file. For this
example, the file is named lifecycle-example.json
.
{ "Rules": [{ "ID": "id-1", "Filter": { "And": { "Prefix": "myprefix", "Tags": [{ "Value": "mytagvalue1", "Key": "mytagkey1" }, { "Value": "mytagvalue2", "Key": "mytagkey2" } ] } }, "Status": "Enabled", "Expiration": { "Days": 10 } }] }
After you save the file, submit the JSON file as part of the
put-bucket-lifecycle-configuration
command. To use this
command, replace each user input placeholder with your own information.
Example of put-bucket-lifecycle
command
Determining whether you can access an Amazon S3 compatible storage on Snow Family devices bucket
The following example uses the head-bucket
command to determine if an
Amazon S3 bucket exists and you have permissions to access it using the AWS CLI. To use
this command, replace each user input placeholder with your own information.
aws s3api head-bucket --bucket
sample-bucket
--profileyour-profile
--endpoint-url https://s3api-endpoint-ip
Retrieving a list of buckets or regional buckets
Use the list-regional-buckets
or list-buckets
to list
Amazon S3 compatible storage on Snow Family devices buckets using the AWS CLI.
Example of retrieving a list of buckets or regional buckets with AWS CLI
The following SDK for Java example gets a list of buckets on Snowball Edge devices. For more information, see ListBuckets in the Amazon Simple Storage Service API Reference.
import com.amazonaws.services.s3.model.*; public void listBuckets() { ListBucketsRequest reqListBuckets = new ListBucketsRequest() .withAccountId(AccountId) ListBucketsResult respListBuckets = s3APIClient.RegionalBuckets(reqListBuckets); System.out.printf("ListBuckets Response: %s%n", respListBuckets.toString()); }
The following PowerShell example gets a list of buckets on Snowball Edge devices.
Get-S3CRegionalBucketList -AccountId
012345678910
-Endpoint "https://snowball_ip
" -Region snow
The following .NET example gets a list of buckets on Snowball Edge devices.
using Amazon.S3Control; using Amazon.S3Control.Model; namespace SnowTest; internal class Program { static async Task Main(string[] args) { var config = new AmazonS3ControlConfig { ServiceURL = "https://
snowball_ip
", AuthenticationRegion = "snow" // Note that this is not RegionEndpoint }; var client = new AmazonS3ControlClient(config); var response = await client.ListRegionalBucketsAsync(new ListRegionalBucketsRequest() { AccountId = "012345678910" }); } }
Getting a bucket
The following example gets an Amazon S3 compatible storage on Snow Family devices bucket using the AWS CLI. To use this command, replace each user input placeholder with your own information.
aws s3control get-bucket --account-id
123456789012
--bucket amzn-s3-demo-bucket --profileyour-profile
--endpoint-url https://s3ctrlapi-endpoint-ip
For more information about this command, see get-bucket
The following Amazon S3 compatible storage on Snow Family devices example gets a bucket using the SDK for Java. For more information, see GetBucket in the Amazon Simple Storage Service API Reference.
import com.amazonaws.services.s3control.model.*; public void getBucket(String bucketName) { GetBucketRequest reqGetBucket = new GetBucketRequest() .withBucket(bucketName) .withAccountId(AccountId); GetBucketResult respGetBucket = s3ControlClient.getBucket(reqGetBucket); System.out.printf("GetBucket Response: %s%n", respGetBucket.toString()); }
Deleting a bucket
You can use the s3api SDK or s3control SDK to delete a bucket in Amazon S3 compatible storage on Snow Family devices.
Important
-
The AWS account that creates the bucket owns it and is the only one that can delete it.
-
Snow Family devices buckets must be empty before they can be deleted.
-
You cannot recover a bucket after it has been deleted.
The following examples delete an Amazon S3 compatible storage on Snow Family devices bucket using the AWS CLI. To use this command, replace each user input placeholder with your own information.