Tutorial: Create a REST API as an Amazon S3 proxy in API Gateway
As an example to showcase using a REST API in API Gateway to proxy Amazon S3, this section describes how to create and
configure a REST API to expose the following Amazon S3 operations:
- Old REST API console
-
We've redesigned the API Gateway console. The old console experience will no longer be available starting December 2023.
- New REST API console
-
You might want to import the sample API as an Amazon S3 proxy, as shown in OpenAPI definitions of the
sample API as an Amazon S3
proxy. This sample contains more exposed methods. For instructions on how to import an API using
the OpenAPI definition, see Configuring a REST API using
OpenAPI.
To integrate your API Gateway API with Amazon S3, you must choose a region where both the API Gateway and Amazon S3 services are
available. For region availability, see Amazon API Gateway
Endpoints and Quotas.
Set up IAM permissions for the API to invoke Amazon S3
actions
To allow the API to invoke Amazon S3 actions, you must have the appropriate IAM policies attached to an IAM role.
To create the AWS service proxy execution role
Sign in to the AWS Management Console and open the IAM console at https://console.aws.amazon.com/iam/.
-
Choose Roles.
-
Choose Create role.
-
Choose AWS service under Select type of
trusted entity, and then select
API Gateway and select Allows API Gateway to push logs to CloudWatch Logs.
-
Choose Next, and then choose Next.
-
For Role name, enter APIGatewayS3ProxyPolicy
, and then choose Create
role.
-
In the Roles list, choose the role you just created. You
may need to scroll or use the search bar to find the role.
-
For the selected role, select the Add permissions tab.
-
Choose Attach policies from the dropdown list.
-
In the search bar, enter AmazonS3FullAccess
and choose Add permissions.
This tutorial uses a managed policy for simplicity. As a best practice, you should create your own IAM policy to grant the minimum permissions required.
-
Note the newly created Role ARN, you will use it later.
Create API resources to represent Amazon S3 resources
We will use the API's root (/
) resource as the container of an authenticated caller's Amazon S3
buckets. We will also create a Folder
and Item
resources to represent a particular Amazon S3
bucket and a particular Amazon S3 object, respectively. The folder name and object key will be specified, in the form
of path parameters as part of a request URL, by the caller.
When accessing objects whose object key includes /
or any other special character, the
character needs to be URL encoded. For example, test/test.txt
should be encoded to
test%2Ftest.txt
.
- Old REST API console
-
We've redesigned the API Gateway console. The old console experience will no longer be available starting December 2023.
To create an API resource that exposes the Amazon S3 service features
-
In the API Gateway console, create an API named MyS3. This API's root resource
(/) represents the Amazon S3 service.
-
Under the API's root resource, create a child resource named Folder and set the
required Resource Path as /{folder}.
-
For the API's Folder resource, create an Item child resource.
Set the required Resource Path as /{item}.
- New REST API console
-
To create an API resource that exposes the Amazon S3 service features
-
In the same AWS Region you created your Amazon S3 bucket, create an API named MyS3. This API's root resource
(/) represents the Amazon S3 service. In this step, you create two additional resources /{folder} and /{item}.
-
Select the API's root resource, and then choose Create resource.
Keep Proxy resource turned off.
For Resource path, select /
.
For Resource name, enter {folder}
.
Keep CORS (Cross Origin Resource Sharing) unchecked.
Choose Create resource.
Select the /{folder} resource, and then choose Create resource.
Use the previous steps to create a child resource of /{folder} named {item}
.
Your final API should look similar to the following:
Expose an API method to list the caller's Amazon S3 buckets
Getting the list of Amazon S3 buckets of the caller involves invoking the GET Service action on Amazon S3. On the API's root resource,
(/), create the GET method. Configure the GET method to integrate with the Amazon S3, as
follows.
- Old REST API console
-
We've redesigned the API Gateway console. The old console experience will no longer be available starting December 2023.
To create and initialize the API's GET /
method
-
Choose Create method on the root node (/
) from the
Actions drop-down menu at the top-right corner of the Resources
panel.
-
Choose the GET
from the drop-down list of HTTP verbs, and choose the check-mark icon to start
creating the method.
-
In the / - GET - Setup pane, choose AWS Service for
Integration type.
-
From the list, choose a region (e.g., us-west-2
) for AWS
Region.
-
From AWS Service, choose S3.
-
For AWS Subdomain, leave it blank.
-
From HTTP method, choose GET.
-
For Action Type, choose Use path override. With path override,
API Gateway forwards the client request to Amazon S3 as the corresponding Amazon S3 REST
API path-style request, in which a Amazon S3 resource is expressed by the resource path of the
s3-host-name/bucket/key
pattern. API Gateway sets the s3-host-name
and passes the client
specified bucket
and key
from the client to Amazon S3.
-
(Optional) In Path override type /.
-
Copy the previously created IAM role's ARN (from the IAM console) and paste it into
Execution role.
-
Leave any other settings as default.
-
Choose Save to finish setting up this method.
- New REST API console
-
To create and initialize the API's GET /
method
-
Select the / resource, and then choose Create method.
For method type, select GET.
For Integration type, select AWS service.
For AWS Region, select the AWS Region where you created your Amazon S3 bucket.
For AWS service, select Amazon Simple Storage Service.
Keep AWS subdomain blank.
For HTTP method, select GET.
For Action type, select Use path override. With path override,
API Gateway forwards the client request to Amazon S3 as the corresponding Amazon S3 REST
API path-style request, in which a Amazon S3 resource is expressed by the resource path of the
s3-host-name/bucket/key
pattern. API Gateway sets the s3-host-name
and passes the client
specified bucket
and key
from the client to Amazon S3.
For Path override, enter /.
For Execution role, enter the role ARN for APIGatewayS3ProxyPolicy
.
Choose Create method.
This setup integrates the frontend GET
https://your-api-host
/stage
/
request with the
backend GET https://your-s3-host
/
.
After the initial setup, you can modify these settings in the Integration Request page
of the method.
To control who can call this method of our API, we turn on the method authorization flag and set it to AWS_IAM
.
- Old REST API console
-
We've redesigned the API Gateway console. The old console experience will no longer be available starting December 2023.
To enable IAM to control access to the GET / method
-
From the Method Execution, choose Method Request.
-
Choose the pencil icon next to Authorization
-
Choose AWS_IAM
from the drop-down list.
-
Choose the check-mark icon to save the setting.
- New REST API console
To enable IAM to control access to the GET / method
-
On the Method request tab, under Method request settings, choose Edit.
-
For Authorization, from the dropdown menu, select AWS_IAM
.
-
Choose Save.
For our API to return successful responses and exceptions properly to the caller, let us declare the 200, 400
and 500 responses in Method Response. We use the default mapping for 200 responses so that
backend responses of the status code not declared here will be returned to the caller as 200 ones.
- Old REST API console
-
We've redesigned the API Gateway console. The old console experience will no longer be available starting December 2023.
To declare response types for the GET / method
-
From the Method Execution pane, choose the Method Response box.
The API Gateway declares the 200 response by default.
-
Choose Add response, enter 400
in the input text box, and
choose the check-mark to finish the declaration.
-
Repeat the above step to declare the 500 response type. The final setting is shown as follows:
- New REST API console
-
To declare response types for the GET /
method
-
On the Method response tab, under Response 200, choose Edit.
-
Choose Add header and do the following:
For Header name, enter Content-Type
.
Choose Add header.
Repeat these steps to create a Timestamp
header and a Content-Length
header.
Choose Save.
On the Method response tab, under Method responses, choose Create response.
For HTTP status code, enter 400.
You do not set any headers for this response.
Choose Save.
Repeat the following steps to create the 500 response.
You do not set any headers for this response.
Because the successful integration response from Amazon S3 returns the bucket list as an XML payload and the
default method response from API Gateway returns a JSON payload, we must map the backend Content-Type header parameter
value to the frontend counterpart. Otherwise, the client will receive application/json
for the content type when the response body is actually an XML string. The following
procedure shows how to set this up. In addition, we also want to display to the client other header parameters,
such as Date and Content-Length.
- Old REST API console
-
We've redesigned the API Gateway console. The old console experience will no longer be available starting December 2023.
To set up response header mappings for the GET / method
-
In the API Gateway console, choose Method Response. Add the
Content-Type header for the 200 response type.
-
In Integration Response, for Content-Type, type
integration.response.header.Content-Type
for the method response.
With the above header mappings, API Gateway will translate the Date
header from
the backend to the Timestamp
header for the client.
-
Still in Integration Response, choose Add integration response,
type an appropriate regular expression in the HTTP status regex text box for a remaining
method response status. Repeat until all the method response status are covered.
- New REST API console
-
To set up response header mappings for the GET / method
-
On the Integration response tab, under Default - Response, choose Edit.
For the Content-Length header, enter integration.response.header.Content-Length
for the mapping value.
For the Content-Type header, enter integration.response.header.Content-Type
for the mapping value.
For the Timestamp header, enter integration.response.header.Date
for the mapping value.
Choose Save. The result should look similar to the following:
-
On the Integration response tab, under Integration responses, choose Create response.
For HTTP status regex, enter 4\d{2}
. This maps all 4xx HTTP response status codes to the method response.
For Method response status code, select 400
.
Choose Create.
Repeat the following steps to create an integration response for the 500 method response. For HTTP status regex, enter 5\d{2}
.
As a good practice, let us test our API we have configured so far.
- Old REST API console
-
We've redesigned the API Gateway console. The old console experience will no longer be available starting December 2023.
Test the GET method on the API root resource
-
Go back to Method Execution, choose Test from the
Client box.
-
Choose Test in the GET / - Method Test pane. An example result
is shown as follows.
To use the API Gateway console to test the API as an Amazon S3 proxy, make sure that the targeted S3 bucket is from a
different region from the API's region. Otherwise, you may get a 500 Internal Server Error response. This
limitation does not apply to any deployed API.
- New REST API console
To test the GET /
method
-
Choose the Test tab. You might need to choose the right arrow button to show the tab.
-
Choose Test. The result should look like the following image:
Expose API methods to access an Amazon S3 bucket
To work with an Amazon S3 bucket, we expose methods on the /{folder} resource. Additional exposed methods are described in the Old REST console tutorial.
- Old REST API console
-
We've redesigned the API Gateway console. The old console experience will no longer be available starting December 2023.
To work with an Amazon S3 bucket, we expose the GET
, PUT
, and DELETE
methods on the /{folder} resource to list
objects in a bucket, create a new bucket, and delete an existing bucket. The instructions are similar to those
described in Expose an API method to list the caller's Amazon S3 buckets. In
the following discussions, we outline the general tasks and highlight relevant differences.
To expose GET, PUT and DELETE methods on a folder resource
-
On the /{folder} node from the Resources tree, create the
DELETE, GET and PUT methods, one at a time.
-
Set up the initial integration of each created method with its corresponding Amazon S3 endpoints. The following
screen shot illustrates this setting for the PUT /{folder}
method. For the DELETE
/{folder}
and GET /{folder}
method, replace the PUT
value of HTTP method by DELETE
and GET
, respectively.
Notice that we used the {bucket}
path parameter in the Amazon S3 endpoint URLs
to specify the bucket. We will need to map the {folder}
path parameter of the
method requests to the {bucket}
path parameter of the integration
requests.
-
To map {folder}
to {bucket}
:
-
Choose Method Execution and then Integration Request.
-
Expand URL Path Parameters and choose Add path
-
Type bucket
in the Name column and method.request.path.folder
in the Mapped from column.
Choose the check-mark icon to save the mapping.
-
In Method Request, add the Content-Type
to the
HTTP Request Headers section.
This is mostly needed for testing, when using the API Gateway console, when you must specify
application/xml
for an XML payload.
-
In Integration Request, set up a mapping for the Content-Type
header to
method.request.header.Content-Type
, following the instructions in Expose an API method to list the caller's Amazon S3 buckets.
-
To test the PUT
method, choose Test in the
Client box from Method Execution, and enter the following as input
to the testing:
-
In folder, type a bucket name,
-
For the Content-Type header, type application/xml
.
-
In Request Body, provide the bucket region as the location constraint, declared
in an XML fragment as the request payload. For example,
<CreateBucketConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<LocationConstraint>{region}
</LocationConstraint>
</CreateBucketConfiguration>
-
Repeat the preceding steps to create and configure the GET and DELETE method on the API's
/{folder} resource.
The above examples illustrate how to create a new bucket in the specified region, to view the list of objects
in the bucket, and to delete the bucket. Other Amazon S3 bucket operations allow you work with the metadata or
properties of the bucket. For example, you can set up your API to call the Amazon S3's PUT /?notification
action to set up notifications on the bucket, to call PUT /?acl to set an access control
list on the bucket, etc. The API set up is similar, except for that you must append appropriate query parameters
to the Amazon S3 endpoint URLs. At run time, you must provide the appropriate XML payload to the method request. The
same can be said about supporting the other GET and DELETE operations on a Amazon S3 bucket. For more information on
possible Amazon S3 actions on a bucket, see Amazon S3 Operations on
Buckets.
- New REST API console
To work with an Amazon S3 bucket, we expose the GET
method on the /{folder} resource to list
objects in a bucket. The instructions are similar to those
described in Expose an API method to list the caller's Amazon S3 buckets. For more methods, you can import the sample API here, OpenAPI definitions of the
sample API as an Amazon S3
proxy.
To expose the GET method on a folder resource
Select the /{folder} resource, and then choose Create method.
For method type, select GET.
For Integration type, select AWS service.
For AWS Region, select the AWS Region where you created your Amazon S3 bucket.
For AWS service, select Amazon Simple Storage Service.
Keep AWS subdomain blank.
For HTTP method, select GET.
For Action type, select Use path override.
For Path override, enter {bucket}
.
For Execution role, enter the role ARN for APIGatewayS3ProxyPolicy
.
Choose Create method.
You set the {folder}
path parameter in the Amazon S3 endpoint URL. You need to map the {folder}
path parameter of the
method request to the {bucket}
path parameter of the integration
request.
To map {folder}
to {bucket}
-
On the Integration request tab, under Integration request settings, choose Edit.
Choose URL path parameters, and then choose Add path parameter.
For Name, enter bucket
.
For Mapped from, enter method.request.path.folder
. The configuration should look similar to the following:
Choose Save.
Now, you test your API.
To test the /{folder} GET
method.
-
Choose the Test tab. You might need to choose the right arrow button to show the tab.
-
Under Path, for folder, enter the name of your bucket.
-
Choose Test.
The test result will contain a list of object in your bucket.
Expose API methods to access an Amazon S3 object in a
bucket
Amazon S3 supports GET, DELETE, HEAD, OPTIONS, POST and PUT actions to access and manage objects in a given bucket.
In this tutorial, you expose a GET
method on the {folder}/{file}
resource to get an
image from a bucket. For more applications of the {folder}/{file}
resource, see the sample API, OpenAPI definitions of the
sample API as an Amazon S3
proxy.
- Old REST API console
-
We've redesigned the API Gateway console. The old console experience will no longer be available starting December 2023.
The API setups for the PUT, GET and DELETE methods on /{folder}/{item}
are the
similar to those on /{folder}
, as prescribed in Expose API methods to access an Amazon S3 bucket. One major difference is that the object-related request path has an additional path parameter of {item}
and this path parameter must be mapped to the integration request path
parameter of {object}
.
The same is true for the GET and DELETE methods.
As an illustration, the following screen shot shows the output when testing the GET method on a {folder}/{item}
resource using the API Gateway console. The request correctly returns the
plain text of ("Welcome to README.txt") as the content of the specified file (README.txt) in the given Amazon S3 bucket
(apig-demo).
- New REST API console
-
To expose the GET method on a file resource
Select the /{item} resource, and then choose Create method.
For method type, select GET.
For Integration type, select AWS service.
For AWS Region, select the AWS Region where you created your Amazon S3 bucket.
For AWS service, select Amazon Simple Storage Service.
Keep AWS subdomain blank.
For HTTP method, select GET.
For Action type, select Use path override.
For Path override, enter {bucket}/{object}.
For Execution role, enter the role ARN for APIGatewayS3ProxyPolicy
.
Choose Create method.
You set the {bucket}
and {object}
path parameters in the Amazon S3 endpoint URL. You need
to map the {folder}
path parameter of the method request to the {bucket}
path parameter of the integration request.
To map {folder}
to {bucket}
and {item}
to {object}
-
On the Integration request tab, under Integration request settings, choose Edit.
Choose URL path parameters.
Choose Add path parameter.
For Name, enter bucket
.
For Mapped from, enter method.request.path.folder
.
Choose Add path parameter.
For Name, enter object
.
For Mapped from, enter method.request.path.item
.
Choose Save.
To test the /{folder}/{object} GET
method.
-
Choose the Test tab. You might need to choose the right arrow button to show the tab.
-
Under Path, for folder, enter the name of your bucket.
-
Under Path, for item, enter the name of an item.
-
Choose Test.
The response body will contain the contents of the item.
The request correctly returns the
plain text of ("Hello world") as the content of the specified file (test.txt) in the given Amazon S3 bucket
(DOC-EXAMPLE-BUCKET).
To download or upload binary files, which in API Gateway is considered any thing other than utf-8 encoded JSON
content, additional API settings are necessary. This is outlined as follows:
- Old REST API console
-
We've redesigned the API Gateway console. The old console experience will no longer be available starting December 2023.
To download or upload binary files from S3
-
Register the media types of the affected file to the API's binaryMediaTypes. You can do this in the
console:
-
Choose Settings for the API.
-
Under Binary Media Types, choose Add Binary Media
Type.
-
Enter the required media type, for example, image/png
.
-
Choose Save Changes to save the setting.
-
Add the Content-Type
(for upload) and/or Accept
(for download) header to the
method request to require the client to specify the required binary media type and map them to the integration
request.
-
Set Content Handling to Passthrough
in the integration request (for
upload) and in a integration response (for download). Make sure that no mapping template is defined for the
affected content type. For more information, see Integration
Passthrough Behaviors and Select VTL Mapping
Templates.
The payload size limit is 10 MB. See API Gateway quotas for configuring
and running a REST API.
Make sure that files on Amazon S3 have the correct content types added as the files' metadata. For streamable media
content, Content-Disposition:inline
may also need to be added to the metadata.
For more information about the binary support in API Gateway, see Content type conversions in
API Gateway.
- New REST API console
-
To download or upload binary files from S3
-
Register the media types of the affected file to the API's binaryMediaTypes. You can do this in the
console:
-
Choose API settings for the API.
-
Under Binary media types, choose Manage media types.
-
Choose Add binary media type, and then enter the required media type, for example, image/png
.
-
Choose Save changes to save the setting.
-
Add the Content-Type
(for upload) and/or Accept
(for download) header to the
method request to require the client to specify the required binary media type and map them to the integration
request.
-
Set Content Handling to Passthrough
in the integration request (for
upload) and in a integration response (for download). Make sure that no mapping template is defined for the
affected content type. For more information, see Integration
Passthrough Behaviors and Select VTL Mapping
Templates.
The payload size limit is 10 MB. See API Gateway quotas for configuring
and running a REST API.
Make sure that files on Amazon S3 have the correct content types added as the files' metadata. For streamable media
content, Content-Disposition:inline
may also need to be added to the metadata.
For more information about the binary support in API Gateway, see Content type conversions in
API Gateway.