Consolidate Amazon S3 presigned URL generation and object downloads by using an endpoint associated with static IP addresses - AWS Prescriptive Guidance
SummaryPrerequisites and limitationsArchitectureToolsBest practicesEpicsRelated resources

Consolidate Amazon S3 presigned URL generation and object downloads by using an endpoint associated with static IP addresses

Song Jin, Eunhye Jo, and Jun Soung Lee, Amazon Web Services

Summary

This pattern simplifies access to Amazon Simple Storage Service (Amazon S3) by creating secure, custom presigned URLs for object downloads. The solution provides a single endpoint with a unique domain and static IP addresses. It's tailored for customers who require consolidation of both API and Amazon S3 endpoints under a unified domain with static IP addresses. The use case involves users following an IP and domain allowlist firewall policy, limiting API access to specific domains and IP addresses.

The architecture employs key AWS services, including AWS Global Accelerator, Amazon API Gateway, AWS Lambda, Application Load Balancer, AWS PrivateLink, and Amazon S3. This design centralizes the API for generating presigned URLs and the Amazon S3 endpoint under a single domain, linked to an accelerator with two static IP addresses. Consequently, users can effortlessly request presigned URLs and download Amazon S3 objects through a unified domain endpoint with static IP addresses.

This architecture is especially beneficial for customers with strict policies or compliance requirements, such as those in the public, medical, and finance sectors.

Prerequisites and limitations

Prerequisites

  • An active AWS account

  • A public hosted zone for your custom domain name

  • A domain imported in AWS Certificate Manager (ACM) in the AWS Region of your choice

Limitations

  • The Amazon S3 bucket name must match the domain name of the endpoint. This requirement is to ensure that the Amazon S3 endpoint can be served through the single API endpoint.

  • The custom domain name used in API Gateway should align with the domain name of the single API endpoint.

  • Some AWS services aren’t available in all AWS Regions. For Region availability, see AWS Services by Region. For specific endpoints, see Service endpoints and quotas, and choose the link for the service.

Architecture

The following diagram shows the target architecture and workflow for this pattern.

Components and workflow for presigned URL generation and object download.

The diagram illustrates the following concept and workflow:

  1. A user initiates a request to generate a presigned URL by using the custom endpoint served through AWS Global Accelerator, using the custom domain name and associated IP addresses.

  2. A Lambda function generates the presigned URL, pointing to the custom endpoint. It responds with a 301 redirect that contains the generated presigned URL. Through the redirected presigned URL, the user downloads the object automatically by using the custom endpoint served through Global Accelerator.

The components of the overall architecture for presigned URL generation and object download workflow are as follows:

  • Provisioning of static IP addresses by Global Accelerator.

  • Registration of the accelerator’s alias as an A record into the Amazon Route 53 public hosted zone with the custom domain name.

  • Creation of an Amazon S3 bucket with a bucket name that matches the registered custom domain name.

  • Creation of VPC endpoints for API Gateway and the Amazon S3 service.

  • Configuration of an internal-facing Application Load Balancer to connect to Global Accelerator.

  • Assignment of a custom domain name for API Gateway with an ACM certificate attached.

  • Deployment of a private API Gateway integrated with a Lambda function.

  • The Lambda function is equipped with an AWS Identity and Access Management (IAM) role attached (with GetObject permissions).

Tools

AWS services

  • Amazon API Gateway helps you create, publish, maintain, monitor, and secure REST, HTTP, and WebSocket APIs at any scale.

  • Application Load Balancers distribute incoming application traffic across multiple targets, such as Amazon Elastic Compute Cloud (Amazon EC2) instances, in multiple Availability Zones.

  • AWS Certificate Manager (ACM) helps you create, store, and renew public and private SSL/TLS X.509 certificates and keys that protect your AWS websites and applications.

  • AWS Cloud Development Kit (AWS CDK) is a software development framework that helps you define and provision AWS Cloud infrastructure in code.

  • AWS Global Accelerator is a global service that supports endpoints in multiple AWS Regions. You can create accelerators that direct traffic to optimal endpoints over the AWS global network. This improves the availability and performance of your internet applications that are used by a global audience.

  • AWS Identity and Access Management (IAM) helps you securely manage access to your AWS resources by controlling who is authenticated and authorized to use them.

  • AWS Lambda is a compute service that helps you run code without needing to provision or manage servers. It runs your code only when needed and scales automatically, so you pay only for the compute time that you use.

  • AWS PrivateLink helps you create unidirectional, private connections from your virtual private clouds (VPCs) to services outside of the VPC.

  • Amazon Route 53 is a highly available and scalable DNS web service.

  • Amazon Simple Storage Service (Amazon S3) is a cloud-based object storage service that helps you store, protect, and retrieve any amount of data.

Other tools

  • Terraform is an infrastructure as code (IaC) tool from HashiCorp that helps you create and manage cloud and on-premises resources.

Code repository

You can deploy this pattern by using either the AWS CDK or Terraform based on your preference. The Epics section contains instructions for both deployment methods. The code for this pattern is available in the following GitHub repositories:

Best practices

  • To enhance security in the production environment, it’s crucial to implement authorization mechanisms, such as Amazon Cognito, to restrict access to the PresignedUrl generation API.

  • Follow the principle of least privilege and grant the minimum permissions required to perform a task. For more information, see Grant least privilege and Security best practices in the IAM documentation.

Epics

TaskDescriptionSkills required

Decide on a domain name.

Decide on a public domain name for the unified Amazon S3 endpoint. The domain name is also used as the Amazon S3 bucket name.

AWS administrator, Network administrator

Create a public hosted zone.

Create a public hosted zone in Amazon Route 53. Its domain name must match the domain name that’s used in API Gateway.

AWS administrator, Network administrator

Prepare an SSL certificate.

Use AWS Certificate Manager (ACM) to request or import an SSL certificate for your web application domain.

AWS administrator, Network administrator
TaskDescriptionSkills required

Set up the Terraform development environment.

To set up the development environment, do the following:

  1. Install Terraform version 1.0 or later.

  2. Clone the GitHub s3-presignedurl-staticips-endpoint-with-terraform repository by running the following command in a terminal window:

    git clone https://github.com/aws-samples/s3-presignedurl-staticips-endpoint-with-terraform.git

AWS administrator, Cloud administrator

Modify the .tfvars and provider.tf files.

  1. Create the required .tfvars files – Create apg.tfvars files in both the 1.vpc_alb_ga and 2.apigw_s3_lambda directories. These files will contain your environment-specific variable values.

    • For 1.vpc_alb_ga/apg.tfvars, create a file with the following template:

      hcl 
 environment_name = "test"
 domain           = "your-domain.com"
 s3_bucket_prefix = "your-bucket-prefix"
 aws_region       = "ap-northeast-2" # Change to your preferred region

    • For 2.apigw_s3_lambda/apg.tfvars, create a file with the following template (you'll need to update the vpc_id after completing step 1):

      hcl 
 vpc_id           = "vpc-xxxxxxxxx"  # Update this with the VPC ID output from step 1
 environment_name = "test"
 domain           = "your-domain.com"
 s3_bucket_prefix = "your-bucket-prefix"
 aws_region       = "ap-northeast-2" # Change to your preferred region

  2. Modify the provider.tf files – In both the 1.vpc_alb_ga and 2.apigw_s3_lambda folders, edit the provider.tf files to match your local AWS configuration:

    hcl 
 provider "aws" {
  region  = "ap-northeast-2"  # Change to your preferred region
  profile = "apg"             # Change to your AWS profile name  
}

Note the following:

  • Replace the placeholder values in the .tfvars files with your actual values.

  • After completing the first Terraform deployment (1.vpc_alb_ga), update the vpc_id value in 2.apigw_s3_lambda/apg.tfvars with the VPC ID from the output.

  • Make sure that your AWS profile name in the provider.tf file matches an existing profile in your ~/.aws/credentials file.

  • The domain value should match your Route 53 hosted zone domain.

  • The s3_bucket_prefix combined with domain will form your Amazon S3 bucket name (for example, your-bucket-prefix.your-domain.com).

  • If you change aws_region, you might also need to update the hardcoded availability zones in the main.tf files to match your chosen AWS Region.

AWS administrator, Cloud administrator

Provision network resources.

To provision network resources, run the following commands:

cd ./2.vpc_alb_ga
terraform init
terraform plan --var-file=apg.tfvars
terraform apply --var-file=apg.tfvars

During the apply command’s execution, type yes when prompted.

AWS administrator, Cloud administrator

Provision API Gateway, Amazon S3, and Lambda.

To provision network resources, use the following commands:

cd ./2.apigw_s3_lambda
terraform init
terraform plan --var-file=apg.tfvars
terraform apply --var-file=apg.tfvars
AWS administrator, Cloud administrator
TaskDescriptionSkills required

Set up the AWS CDK development environment.

To set up the development environment, do the following:

  1. Install the AWS CDK.

  2. Clone the GitHub s3-presignedurl-staticips-endpoint-with-cdk repository by running the following command in a terminal window:

    git clone https://github.com/aws-samples/s3-presignedurl-staticips-endpoint-with-cdk.git

AWS administrator, Cloud administrator

Configure domain settings in the config/index.ts file.

To edit the options of the constant variable, use the following commands:

export const options = {
    certificateArn: '{arn of the acm which created before}',
    dnsAttr: {
        zoneName: '{public hosted zone name}',
        hostedZoneId: 'hosted zone Id',
    },
    domainNamePrefix: '{Prefix for the domain}',
    presignPath: 'presign',
    objectsPath: 'objects',
};

In the commands, replace each placeholder with your own information:

  • Replace '{arn of the acm which created before}' with your ACM certificate Amazon Resource Name (ARN).

  • Replace '{public hosted zone name}' with your domain name.

  • Replace 'hosted zone Id' with your Route 53 hosted zone ID.

  • Replace '{Prefix for the domain}' with the domain prefix that you want to use.

  • Optionally, modify 'presign' and 'objects' paths if needed for your specific use case.

AWS administrator, Cloud administrator

Deploy the stacks.

To deploy two stacks, one for the virtual private cloud (VPC) and another for the application, use the following command:

$ npm install 
$ cdk synth 
$ cdk deploy --all
AWS administrator, Cloud administrator
TaskDescriptionSkills required

Verify the IP addresses of the endpoint.

To verify that the domain for this pattern has static IP addresses, use the following command:

nslookup ${s3-bucket-prefix}.${domain}
Network administrator

Upload a test file that you can later download.

Upload the test file to the '/objects' folder in the Amazon S3 bucket.

AWS administrator, Cloud administrator

Invoke the API to generate a presigned URL.

To generate a presigned URL, call the URL from a browser or API client (for example, Postman) using the following format:

https://${s3-bucket-prefix}.${domain}/presign/objects/${uploaded-filename}

Replace the placeholder values in ${s3-bucket-prefix} and ${domain} with the values that you set in previous steps.

App owner

Check the result.

The expected result is that you should receive a 301 (Moved Permanently) redirect status code. This response will contain the presigned URL, which should automatically initiate the download of your test file.

Test engineer
TaskDescriptionSkills required

Destroy API Gateway, Amazon S3, and Lambda resources.

To delete resources, use the following commands:

cd ./2.apigw_s3_lambda
terraform init
terraform plan --destroy --var-file=apg.tfvars
terraform destroy --var-file=apg.tfvars
AWS administrator, Cloud administrator

Destroy network resources.

To delete network resources, use the following commands:

cd ./1.vpc_alb_ga
terraform init
terraform plan --destroy --var-file=apg.tfvars
terraform destroy --var-file=apg.tfvars
AWS administrator, Cloud administrator
TaskDescriptionSkills required

Destroy the stacks.

To destroy both the VPC and application stacks, use the following command:

$ cdk destroy --all
AWS administrator, Cloud administrator

Empty and delete the Amazon S3 buckets.

Empty and delete the object Amazon S3 bucket and the logs Amazon S3 bucket that are not deleted by default.

The Amazon S3 bucket names are ${s3-bucket-prefix}.${domain} and ${s3-bucket-prefix}.${domain}-logs.

If you prefer to use the AWS Command Line Interface (AWS CLI) for deleting the buckets, use the following commands:

$ aws s3 rm s3://${s3-bucket-prefix}.${domain} --recursive
$ aws s3 rb s3://${s3-bucket-prefix}.${domain} --force
$ aws s3 rm s3://${s3-bucket-prefix}.${domain}-logs --recursive
$ aws s3 rb s3://${s3-bucket-prefix}.${domain}-logs --force

Replace ${s3-bucket-prefix} and ${domain} with the values you set in previous steps.,/p>

AWS administrator, Cloud administrator

Related resources

AWS Blogs