Tutorial: Create a REST API as an Amazon S3 proxy in API Gateway

As an example to showcase using a REST API in API Gateway to proxy Amazon S3, this section describes how to create and configure a REST API to expose the following Amazon S3 operations:

Old REST API console

Note

We've redesigned the API Gateway console. The old console experience will no longer be available starting December 2023.

• Expose GET on the API's root resource to list all of the Amazon S3 buckets of a caller.

• Expose GET on a Folder resource to view a list of all of the objects in an Amazon S3 bucket.

• Expose PUT on a Folder resource to add a bucket to Amazon S3.

• Expose DELETE on a Folder resource to remove a bucket from Amazon S3.

• Expose GET on a Folder/Item resource to view or download an object from an Amazon S3 bucket.

• Expose PUT on a Folder/Item resource to upload an object to an Amazon S3 bucket.

• Expose HEAD on a Folder/Item resource to get object metadata in an Amazon S3 bucket.

• Expose DELETE on a Folder/Item resource to remove an object from an Amazon S3 bucket.

New REST API console

• Expose GET on the API's root resource to list all of the Amazon S3 buckets of a caller.

• Expose GET on a Folder resource to view a list of all of the objects in an Amazon S3 bucket.

• Expose GET on a Folder/Item resource to view or download an object from an Amazon S3 bucket.

You might want to import the sample API as an Amazon S3 proxy, as shown in OpenAPI definitions of the sample API as an Amazon S3 proxy. This sample contains more exposed methods. For instructions on how to import an API using the OpenAPI definition, see Configuring a REST API using OpenAPI.

Note

To integrate your API Gateway API with Amazon S3, you must choose a region where both the API Gateway and Amazon S3 services are available. For region availability, see Amazon API Gateway Endpoints and Quotas.

Topics

• Set up IAM permissions for the API to invoke Amazon S3 actions
• Create API resources to represent Amazon S3 resources
• Expose an API method to list the caller's Amazon S3 buckets
• Expose API methods to access an Amazon S3 bucket
• Expose API methods to access an Amazon S3 object in a bucket
• OpenAPI definitions of the sample API as an Amazon S3 proxy
• Call the API using a REST API client

Set up IAM permissions for the API to invoke Amazon S3 actions

To allow the API to invoke Amazon S3 actions, you must have the appropriate IAM policies attached to an IAM role.

To create the AWS service proxy execution role

1. Sign in to the AWS Management Console and open the IAM console at https://console.aws.amazon.com/iam/.

2. Choose Roles.

3. Choose Create role.

4. Choose AWS service under Select type of trusted entity, and then select API Gateway and select Allows API Gateway to push logs to CloudWatch Logs.

5. Choose Next, and then choose Next.

6. For Role name, enter APIGatewayS3ProxyPolicy, and then choose Create role.

7. In the Roles list, choose the role you just created. You may need to scroll or use the search bar to find the role.

8. For the selected role, select the Add permissions tab.

9. Choose Attach policies from the dropdown list.

10. In the search bar, enter AmazonS3FullAccess and choose Add permissions.

    Note

    This tutorial uses a managed policy for simplicity. As a best practice, you should create your own IAM policy to grant the minimum permissions required.

11. Note the newly created Role ARN, you will use it later.

Create API resources to represent Amazon S3 resources

We will use the API's root (/) resource as the container of an authenticated caller's Amazon S3 buckets. We will also create a Folder and Item resources to represent a particular Amazon S3 bucket and a particular Amazon S3 object, respectively. The folder name and object key will be specified, in the form of path parameters as part of a request URL, by the caller.

Note

When accessing objects whose object key includes / or any other special character, the character needs to be URL encoded. For example, test/test.txt should be encoded to test%2Ftest.txt.

Old REST API console

Note

We've redesigned the API Gateway console. The old console experience will no longer be available starting December 2023.

To create an API resource that exposes the Amazon S3 service features

1. In the API Gateway console, create an API named MyS3. This API's root resource (/) represents the Amazon S3 service.

2. Under the API's root resource, create a child resource named Folder and set the required Resource Path as /{folder}.

3. For the API's Folder resource, create an Item child resource. Set the required Resource Path as /{item}.

   

New REST API console

To create an API resource that exposes the Amazon S3 service features

1. In the same AWS Region you created your Amazon S3 bucket, create an API named MyS3. This API's root resource (/) represents the Amazon S3 service. In this step, you create two additional resources /{folder} and /{item}.

2. Select the API's root resource, and then choose Create resource.

3. Keep Proxy resource turned off.

4. For Resource path, select /.

5. For Resource name, enter {folder}.

6. Keep CORS (Cross Origin Resource Sharing) unchecked.

7. Choose Create resource.

8. Select the /{folder} resource, and then choose Create resource.

9. Use the previous steps to create a child resource of /{folder} named {item}.

   Your final API should look similar to the following:

   

Expose an API method to list the caller's Amazon S3 buckets

Getting the list of Amazon S3 buckets of the caller involves invoking the GET Service action on Amazon S3. On the API's root resource, (/), create the GET method. Configure the GET method to integrate with the Amazon S3, as follows.

Old REST API console

Note

We've redesigned the API Gateway console. The old console experience will no longer be available starting December 2023.

To create and initialize the API's GET / method

1. Choose Create method on the root node (/) from the Actions drop-down menu at the top-right corner of the Resources panel.

2. Choose the GET from the drop-down list of HTTP verbs, and choose the check-mark icon to start creating the method.

   

3. In the / - GET - Setup pane, choose AWS Service for Integration type.

4. From the list, choose a region (e.g., us-west-2) for AWS Region.

5. From AWS Service, choose S3.

6. For AWS Subdomain, leave it blank.

7. From HTTP method, choose GET.

8. For Action Type, choose Use path override. With path override, API Gateway forwards the client request to Amazon S3 as the corresponding Amazon S3 REST API path-style request, in which a Amazon S3 resource is expressed by the resource path of the s3-host-name/bucket/key pattern. API Gateway sets the s3-host-name and passes the client specified bucket and key from the client to Amazon S3.

9. (Optional) In Path override type /.

10. Copy the previously created IAM role's ARN (from the IAM console) and paste it into Execution role.

11. Leave any other settings as default.

12. Choose Save to finish setting up this method.

New REST API console

To create and initialize the API's GET / method

1. Select the / resource, and then choose Create method.

2. For method type, select GET.

3. For Integration type, select AWS service.

4. For AWS Region, select the AWS Region where you created your Amazon S3 bucket.

5. For AWS service, select Amazon Simple Storage Service.

6. Keep AWS subdomain blank.

7. For HTTP method, select GET.

8. For Action type, select Use path override. With path override, API Gateway forwards the client request to Amazon S3 as the corresponding Amazon S3 REST API path-style request, in which a Amazon S3 resource is expressed by the resource path of the s3-host-name/bucket/key pattern. API Gateway sets the s3-host-name and passes the client specified bucket and key from the client to Amazon S3.

9. For Path override, enter /.

10. For Execution role, enter the role ARN for APIGatewayS3ProxyPolicy.

11. Choose Create method.

This setup integrates the frontend GET https://your-api-host/stage/ request with the backend GET https://your-s3-host/.

Note

After the initial setup, you can modify these settings in the Integration Request page of the method.

To control who can call this method of our API, we turn on the method authorization flag and set it to AWS_IAM.

Old REST API console

Note

We've redesigned the API Gateway console. The old console experience will no longer be available starting December 2023.

To enable IAM to control access to the GET / method

1. From the Method Execution, choose Method Request.

2. Choose the pencil icon next to Authorization

3. Choose AWS_IAM from the drop-down list.

4. Choose the check-mark icon to save the setting.

   

New REST API console

To enable IAM to control access to the GET / method

1. On the Method request tab, under Method request settings, choose Edit.

2. For Authorization, from the dropdown menu, select AWS_IAM.

   

3. Choose Save.

For our API to return successful responses and exceptions properly to the caller, let us declare the 200, 400 and 500 responses in Method Response. We use the default mapping for 200 responses so that backend responses of the status code not declared here will be returned to the caller as 200 ones.

Old REST API console

Note

We've redesigned the API Gateway console. The old console experience will no longer be available starting December 2023.

To declare response types for the GET / method

1. From the Method Execution pane, choose the Method Response box. The API Gateway declares the 200 response by default.

2. Choose Add response, enter 400 in the input text box, and choose the check-mark to finish the declaration.

3. Repeat the above step to declare the 500 response type. The final setting is shown as follows:

   

New REST API console

To declare response types for the GET / method

1. On the Method response tab, under Response 200, choose Edit.

2. Choose Add header and do the following:

   1. For Header name, enter Content-Type.

   2. Choose Add header.

   Repeat these steps to create a Timestamp header and a Content-Length header.

3. Choose Save.

4. On the Method response tab, under Method responses, choose Create response.

5. For HTTP status code, enter 400.

   You do not set any headers for this response.

6. Choose Save.

7. Repeat the following steps to create the 500 response.

   You do not set any headers for this response.

Because the successful integration response from Amazon S3 returns the bucket list as an XML payload and the default method response from API Gateway returns a JSON payload, we must map the backend Content-Type header parameter value to the frontend counterpart. Otherwise, the client will receive application/json for the content type when the response body is actually an XML string. The following procedure shows how to set this up. In addition, we also want to display to the client other header parameters, such as Date and Content-Length.

Old REST API console

Note

We've redesigned the API Gateway console. The old console experience will no longer be available starting December 2023.

To set up response header mappings for the GET / method

1. In the API Gateway console, choose Method Response. Add the Content-Type header for the 200 response type.

   

2. In Integration Response, for Content-Type, type integration.response.header.Content-Type for the method response.

   

   With the above header mappings, API Gateway will translate the Date header from the backend to the Timestamp header for the client.

3. Still in Integration Response, choose Add integration response, type an appropriate regular expression in the HTTP status regex text box for a remaining method response status. Repeat until all the method response status are covered.

   

New REST API console

To set up response header mappings for the GET / method

1. On the Integration response tab, under Default - Response, choose Edit.

2. For the Content-Length header, enter integration.response.header.Content-Length for the mapping value.

3. For the Content-Type header, enter integration.response.header.Content-Type for the mapping value.

4. For the Timestamp header, enter integration.response.header.Date for the mapping value.

5. Choose Save. The result should look similar to the following:

   

6. On the Integration response tab, under Integration responses, choose Create response.

7. For HTTP status regex, enter 4\d{2}. This maps all 4xx HTTP response status codes to the method response.

8. For Method response status code, select 400.

9. Choose Create.

10. Repeat the following steps to create an integration response for the 500 method response. For HTTP status regex, enter 5\d{2}.

As a good practice, let us test our API we have configured so far.

Old REST API console

Note

We've redesigned the API Gateway console. The old console experience will no longer be available starting December 2023.

Test the GET method on the API root resource

1. Go back to Method Execution, choose Test from the Client box.

2. Choose Test in the GET / - Method Test pane. An example result is shown as follows.

   

Note

To use the API Gateway console to test the API as an Amazon S3 proxy, make sure that the targeted S3 bucket is from a different region from the API's region. Otherwise, you may get a 500 Internal Server Error response. This limitation does not apply to any deployed API.

New REST API console

To test the GET / method

1. Choose the Test tab. You might need to choose the right arrow button to show the tab.

2. Choose Test. The result should look like the following image:

   

Expose API methods to access an Amazon S3 bucket

To work with an Amazon S3 bucket, we expose methods on the /{folder} resource. Additional exposed methods are described in the Old REST console tutorial.

Old REST API console

Note

We've redesigned the API Gateway console. The old console experience will no longer be available starting December 2023.

To work with an Amazon S3 bucket, we expose the GET, PUT, and DELETE methods on the /{folder} resource to list objects in a bucket, create a new bucket, and delete an existing bucket. The instructions are similar to those described in Expose an API method to list the caller's Amazon S3 buckets. In the following discussions, we outline the general tasks and highlight relevant differences.

To expose GET, PUT and DELETE methods on a folder resource

1. On the /{folder} node from the Resources tree, create the DELETE, GET and PUT methods, one at a time.

2. Set up the initial integration of each created method with its corresponding Amazon S3 endpoints. The following screen shot illustrates this setting for the PUT /{folder} method. For the DELETE /{folder} and GET /{folder} method, replace the PUT value of HTTP method by DELETE and GET, respectively.

   

   Notice that we used the {bucket} path parameter in the Amazon S3 endpoint URLs to specify the bucket. We will need to map the {folder} path parameter of the method requests to the {bucket} path parameter of the integration requests.

3. To map {folder} to {bucket}:

   1. Choose Method Execution and then Integration Request.

   2. Expand URL Path Parameters and choose Add path

   3. Type bucket in the Name column and method.request.path.folder in the Mapped from column. Choose the check-mark icon to save the mapping.

      

4. In Method Request, add the Content-Type to the HTTP Request Headers section.

   

   This is mostly needed for testing, when using the API Gateway console, when you must specify application/xml for an XML payload.

5. In Integration Request, set up a mapping for the Content-Type header to method.request.header.Content-Type, following the instructions in Expose an API method to list the caller's Amazon S3 buckets.

6. To test the PUT method, choose Test in the Client box from Method Execution, and enter the following as input to the testing:

   1. In folder, type a bucket name,

   2. For the Content-Type header, type application/xml.

   3. In Request Body, provide the bucket region as the location constraint, declared in an XML fragment as the request payload. For example,

      <CreateBucketConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> 
         <LocationConstraint>{region}</LocationConstraint> 
      </CreateBucketConfiguration>

      

7. Repeat the preceding steps to create and configure the GET and DELETE method on the API's /{folder} resource.

The above examples illustrate how to create a new bucket in the specified region, to view the list of objects in the bucket, and to delete the bucket. Other Amazon S3 bucket operations allow you work with the metadata or properties of the bucket. For example, you can set up your API to call the Amazon S3's PUT /?notification action to set up notifications on the bucket, to call PUT /?acl to set an access control list on the bucket, etc. The API set up is similar, except for that you must append appropriate query parameters to the Amazon S3 endpoint URLs. At run time, you must provide the appropriate XML payload to the method request. The same can be said about supporting the other GET and DELETE operations on a Amazon S3 bucket. For more information on possible Amazon S3 actions on a bucket, see Amazon S3 Operations on Buckets.

New REST API console

To work with an Amazon S3 bucket, we expose the GET method on the /{folder} resource to list objects in a bucket. The instructions are similar to those described in Expose an API method to list the caller's Amazon S3 buckets. For more methods, you can import the sample API here, OpenAPI definitions of the sample API as an Amazon S3 proxy.

To expose the GET method on a folder resource

1. Select the /{folder} resource, and then choose Create method.

2. For method type, select GET.

3. For Integration type, select AWS service.

4. For AWS Region, select the AWS Region where you created your Amazon S3 bucket.

5. For AWS service, select Amazon Simple Storage Service.

6. Keep AWS subdomain blank.

7. For HTTP method, select GET.

8. For Action type, select Use path override.

9. For Path override, enter {bucket}.

10. For Execution role, enter the role ARN for APIGatewayS3ProxyPolicy.

11. Choose Create method.

You set the {folder} path parameter in the Amazon S3 endpoint URL. You need to map the {folder} path parameter of the method request to the {bucket} path parameter of the integration request.

To map {folder} to {bucket}

1. On the Integration request tab, under Integration request settings, choose Edit.

2. Choose URL path parameters, and then choose Add path parameter.

3. For Name, enter bucket.

4. For Mapped from, enter method.request.path.folder. The configuration should look similar to the following:

   

5. Choose Save.

Now, you test your API.

To test the /{folder} GET method.

1. Choose the Test tab. You might need to choose the right arrow button to show the tab.

2. Under Path, for folder, enter the name of your bucket.

3. Choose Test.

   The test result will contain a list of object in your bucket.

   

Expose API methods to access an Amazon S3 object in a bucket

Amazon S3 supports GET, DELETE, HEAD, OPTIONS, POST and PUT actions to access and manage objects in a given bucket. In this tutorial, you expose a GET method on the {folder}/{file} resource to get an image from a bucket. For more applications of the {folder}/{file} resource, see the sample API, OpenAPI definitions of the sample API as an Amazon S3 proxy.

Old REST API console

Note

We've redesigned the API Gateway console. The old console experience will no longer be available starting December 2023.

The API setups for the PUT, GET and DELETE methods on /{folder}/{item} are the similar to those on /{folder}, as prescribed in Expose API methods to access an Amazon S3 bucket. One major difference is that the object-related request path has an additional path parameter of {item} and this path parameter must be mapped to the integration request path parameter of {object}.

The same is true for the GET and DELETE methods.

As an illustration, the following screen shot shows the output when testing the GET method on a {folder}/{item} resource using the API Gateway console. The request correctly returns the plain text of ("Welcome to README.txt") as the content of the specified file (README.txt) in the given Amazon S3 bucket (apig-demo).

New REST API console

To expose the GET method on a file resource

1. Select the /{item} resource, and then choose Create method.

2. For method type, select GET.

3. For Integration type, select AWS service.

4. For AWS Region, select the AWS Region where you created your Amazon S3 bucket.

5. For AWS service, select Amazon Simple Storage Service.

6. Keep AWS subdomain blank.

7. For HTTP method, select GET.

8. For Action type, select Use path override.

9. For Path override, enter {bucket}/{object}.

10. For Execution role, enter the role ARN for APIGatewayS3ProxyPolicy.

11. Choose Create method.

You set the {bucket} and {object} path parameters in the Amazon S3 endpoint URL. You need to map the {folder} path parameter of the method request to the {bucket} path parameter of the integration request.

To map {folder} to {bucket} and {item} to {object}

1. On the Integration request tab, under Integration request settings, choose Edit.

2. Choose URL path parameters.

3. Choose Add path parameter.

4. For Name, enter bucket.

5. For Mapped from, enter method.request.path.folder.

6. Choose Add path parameter.

7. For Name, enter object.

8. For Mapped from, enter method.request.path.item.

9. Choose Save.

To test the /{folder}/{object} GET method.

1. Choose the Test tab. You might need to choose the right arrow button to show the tab.

2. Under Path, for folder, enter the name of your bucket.

3. Under Path, for item, enter the name of an item.

4. Choose Test.

   The response body will contain the contents of the item.

   

   The request correctly returns the plain text of ("Hello world") as the content of the specified file (test.txt) in the given Amazon S3 bucket (DOC-EXAMPLE-BUCKET).

To download or upload binary files, which in API Gateway is considered any thing other than utf-8 encoded JSON content, additional API settings are necessary. This is outlined as follows:

Old REST API console

Note

We've redesigned the API Gateway console. The old console experience will no longer be available starting December 2023.

To download or upload binary files from S3

1. Register the media types of the affected file to the API's binaryMediaTypes. You can do this in the console:

   1. Choose Settings for the API.

   2. Under Binary Media Types, choose Add Binary Media Type.

   3. Enter the required media type, for example, image/png.

   4. Choose Save Changes to save the setting.

2. Add the Content-Type (for upload) and/or Accept (for download) header to the method request to require the client to specify the required binary media type and map them to the integration request.

3. Set Content Handling to Passthrough in the integration request (for upload) and in a integration response (for download). Make sure that no mapping template is defined for the affected content type. For more information, see Integration Passthrough Behaviors and Select VTL Mapping Templates.

The payload size limit is 10 MB. See API Gateway quotas for configuring and running a REST API.

Make sure that files on Amazon S3 have the correct content types added as the files' metadata. For streamable media content, Content-Disposition:inline may also need to be added to the metadata.

For more information about the binary support in API Gateway, see Content type conversions in API Gateway.

New REST API console

To download or upload binary files from S3

1. Register the media types of the affected file to the API's binaryMediaTypes. You can do this in the console:

   1. Choose API settings for the API.

   2. Under Binary media types, choose Manage media types.

   3. Choose Add binary media type, and then enter the required media type, for example, image/png.

   4. Choose Save changes to save the setting.

2. Add the Content-Type (for upload) and/or Accept (for download) header to the method request to require the client to specify the required binary media type and map them to the integration request.

3. Set Content Handling to Passthrough in the integration request (for upload) and in a integration response (for download). Make sure that no mapping template is defined for the affected content type. For more information, see Integration Passthrough Behaviors and Select VTL Mapping Templates.

The payload size limit is 10 MB. See API Gateway quotas for configuring and running a REST API.

Make sure that files on Amazon S3 have the correct content types added as the files' metadata. For streamable media content, Content-Disposition:inline may also need to be added to the metadata.

For more information about the binary support in API Gateway, see Content type conversions in API Gateway.