Menu
Amazon API Gateway
Developer Guide

Create an API Gateway API as an Amazon Kinesis Proxy

This section describes how to create and configure an API Gateway API as an AWS proxy to access Amazon Kinesis.

For the purpose of illustration, we will create an example API to enable a client to do the following:

  1. List the user's available streams in Amazon Kinesis

  2. Create, describe, or delete a specified stream

  3. Read data records from or write data records into the specified stream

To accomplish the preceding tasks, the API exposes methods on various resources to invoke the following, respectively:

  1. The ListStreams action in Amazon Kinesis

  2. The CreateStream, DescribeStream, or DeleteStream action

  3. The GetRecords or PutRecords (including PutRecord) action in Amazon Kinesis

Specifically, we will build the API as follows:

  • Expose an HTTP GET method on the API's /streams resource and integrate the method with the ListStreams action in Amazon Kinesis to list the streams in the caller's account.

  • Expose an HTTP POST method on the API's /streams/{stream-name} resource and integrate the method with the CreateStream action in Amazon Kinesis to create a named stream in the caller's account.

  • Expose an HTTP GET method on the API's /streams/{stream-name} resource and integrate the method with the DescribeStream action in Amazon Kinesis to describe a named stream in the caller's account.

  • Expose an HTTP DELETE method on the API's /streams/{stream-name} resource and integrate the method with the DeleteStream action in Amazon Kinesis to delete a stream in the caller's account.

  • Expose an HTTP PUT method on the API's /streams/{stream-name}/record resource and integrate the method with the PutRecord action in Amazon Kinesis. This enables the client to add a single data record to the named stream.

  • Expose an HTTP PUT method on the API's /streams/{stream-name}/records resource and integrate the method with the PutRecords action in Amazon Kinesis. This enables the client to add a list of data records to the named stream.

  • Expose an HTTP GET method on the API's /streams/{stream-name}/records resource and integrate the method with the GetRecords action in Amazon Kinesis. This enables the client to list data records in the named stream, with a specified shard iterator. A shard iterator specifies the shard position from which to start reading data records sequentially.

  • Expose an HTTP GET method on the API's /streams/{stream-name}/sharditerator resource and integrate the method with the GetShardIterator action in Amazon Kinesis. This helper method must be supplied to the ListStreams action in Amazon Kinesis.

You can apply the instructions presented here to other Amazon Kinesis actions. For the complete list of the Amazon Kinesis actions, see Amazon Kinesis API Reference.

Instead of using the API Gateway console to create the sample API, you can import the sample API into API Gateway, using either the API Gateway Import API or the API Gateway Swagger Importer. For information on how to use the Import API, see Import an API. For information on how to use the API Gateway Swagger Importer, see Getting Started with the API Gateway Swagger Importer.

If you do not have an AWS account, use the following procedure to create one.

To sign up for AWS

  1. Open https://aws.amazon.com/ and choose Create an AWS Account.

  2. Follow the online instructions.

Create an IAM Role and Policy for the API to Access Amazon Kinesis

To allow the API to invoke Amazon Kinesis actions, you must have appropriate IAM policies attached to an IAM role. This section explains how to verify and to create, if necessary, the required IAM role and policies.

To enable read-only access to Amazon Kinesis, you can use the AmazonKinesisReadOnlyAccess policy that allows the Get*, List*, and Describe* actions in Amazon Kinesis to be invoked.

Copy
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "kinesis:Get*", "kinesis:List*", "kinesis:Describe*" ], "Resource": "*" } ] }

This policy is available from the IAM console and its ARN is arn:aws:iam::aws:policy/AmazonKinesisReadOnlyAccess.

To enable read-write actions in Amazon Kinesis, you can use the AmazonKinesisFullAccess policy.

Copy
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "kinesis:*", "Resource": "*" } ] }

This policy is also available from the IAM console. Its ARN is arn:aws:iam::aws:policy/AmazonKinesisFullAccess.

After you decide which IAM policy to use, attach it to a new or existing IAM role. Make sure that the API Gateway control service (apigateway.amazonaws.com) is a trusted entity of the role and is allowed to assume the execution role (sts:AssumeRole).

Copy
{ "Version": "2012-10-17", "Statement": [ { "Sid": "", "Effect": "Allow", "Principal": { "Service": "apigateway.amazonaws.com" }, "Action": "sts:AssumeRole" } ] }

If you create the execution role in the IAM console and choose the Amazon API Gateway role type, this trust policy is automatically attached.

Note the ARN of the execution role. You will need it when creating an API method and setting up its integration request.

Start to Create an API as an Amazon Kinesis Proxy

Use the following steps to create the API in the API Gateway console.

To create an API as an AWS service proxy for Amazon Kinesis

  1. In the API Gateway console, choose Create API.

  2. In API name, type KinesisProxy. Leave the default values in the other fields.

  3. For Clone from API, choose Do not clone from existing API.

  4. Type a description in Description.

  5. Choose Create API.

After the API is created, the API Gateway console displays the Resources page, which contains only the API's root (/) resource.

List Streams in Amazon Kinesis

To list streams in Amazon Kinesis, add a /streams resource to the API's root, expose a GET method on the resource, and integrate the method to the ListStreams action of Amazon Kinesis.

The following procedure describes how to list Amazon Kinesis streams by using the API Gateway console.

To list Amazon Kinesis streams by using the API Gateway console

  1. Select the API root resource. In Actions, choose Create Resource.

    In Resource Name, type Streams, leave Resource Path as the default, and choose Create Resource.

  2. Select the /Streams resource. From Actions, choose Create Method, choose GET from the list, and then choose the checkmark icon to finish creating the method.

    Note

    You can choose any of the available HTTP verbs for a method request. We use GET here, because listing streams is a READ operation.

  3. In the method's Setup pane, choose Show Advanced and then choose AWS Service Proxy.

    1. For AWS Region, choose a region (e.g., us-east-1).

    2. For AWS Service, choose Kinesis.

    3. For HTTP method, choose POST.

      Note

      For the integration request with Amazon Kinesis, you must choose the POST HTTP verb to invoke the action, although you can use any of the available HTTP verbs for the API's method request.

    4. For Action Type, choose Use action name.

    5. For Action, type ListStreams.

    6. For Execution role, type the ARN for your execution role.

    7. Choose Save to finish the initial setup of the method.

    
                                Set up the Streams:GET method for Amazon Kinesis ListStreams
                                    action.

    The initial setup of the integration request will suffice if there is no need to map data between the method and integration requests and/or between the method and integration responses. Examples discussed in this topic require data mapping, which is covered in the second half of the Integration Request pane.

  4. In the Integration Request pane, expand the HTTP Headers section:

    1. Choose Add header.

    2. In the Name column, type Content-Type.

    3. In the Mapped from column, type 'application/x-amz-json-1.1'.

    4. Choose the checkmark icon to save the setting.

  5. Expand the Body Mapping Templates section:

    1. Choose Add mapping template.

    2. For Content-Type, type application/json.

    3. Choose the checkmark icon to save the setting.

    4. Choose the pencil icon to the right of Mapping template.

    5. Choose Mapping template from the drop-down list to open the Template editor.

    6. Type {} in the template editor.

    7. Choose the checkmark icon to save the mapping template.

    The ListStreams request takes a payload of the following JSON format:

    Copy
    { "ExclusiveStartStreamName": "string", "Limit": number }

    However, the properties are optional. To use the default values, we opted for an empty JSON payload here.

    
                                Set up data mapping for the Streams:GET method for the Amazon Kinesis
                                        ListStreams action.

  6. Test the GET method on the Streams resource to invoke the ListStreams action in Amazon Kinesis:

    From the API Gateway console, select the /streams/GET entry from the Resources pane, choose the Test invocation option, and then choose Test.

    If you have already created two streams named "myStream" and "yourStream" in Amazon Kinesis, the successful test will return a 200 OK response containing the following payload:

    Copy
    { "HasMoreStreams": false, "StreamNames": [ "myStream", "yourStream" ] }

Create, Describe, and Delete a Stream in Amazon Kinesis

Creating, describing, and deleting a stream in Amazon Kinesis involves making the following Amazon Kinesis REST API requests, respectively:

Copy
POST /?Action=CreateStream HTTP/1.1 Host: kinesis.region.domain ... Content-Type: application/x-amz-json-1.1 Content-Length: PayloadSizeBytes { "ShardCount": number, "StreamName": "string" }
Copy
POST /?Action=DescribeStream HTTP/1.1 Host: kinesis.region.domain ... Content-Type: application/x-amz-json-1.1 Content-Length: PayloadSizeBytes { "ExclusiveStartShardId": "string", "Limit": number, "StreamName": "string" }
Copy
POST /?Action=DeleteStream HTTP/1.1 Host: kinesis.region.domain ... Content-Type: application/x-amz-json-1.1 Content-Length: PayloadSizeBytes { "StreamName":"string" }

We can build our API to accept the required input as a JSON payload of the method request and pass the payload through to the integration request. However, to provide more examples of data mapping between method and integration requests, and method and integration responses, we will create our API slightly differently.

We will expose the GET, POST, and Delete HTTP methods on a to-be-named Stream resource. We will use the {stream-name} path variable to hold the to-be-named stream resource and integrate these API methods with the Amazon Kinesis' DescribeStream, CreateStream, and DeleteStream actions, respectively. We require that the client pass other input data as headers, query parameters, or the payload of a method request, and we provide mapping templates to transform the data to the required integration request payload.

After the methods are created on a to-be-named stream resource, the structure of the API looks like the following:


                        Create POST, GET, and DELETE methods on a stream resource

To configure and test the GET method on a stream resource

  1. Set up the GET method to describe a named stream in Amazon Kinesis, as shown in the following.

    
                                Set up the GET-on-Stream method to invoke the
                                        DescribeStream action in Amazon Kinesis

  2. Map data from the GET method request to the integration request, as shown in the following:

    
                                Map data from the GET-on-Stream method request to
                                    integration request.

  3. Test the GET method to invoke the DescribeStream action in Amazon Kinesis:

    From the API Gateway console, select /streams/{stream-name}/GET in the Resources pane, choose Test to start testing, type the name of an existing Amazon Kinesis stream in the Path field for stream-name, and choose Test. If the test is successful, a 200 OK response is returned with a payload similar to the following:

    Copy
    { "StreamDescription": { "HasMoreShards": false, "RetentionPeriodHours": 24, "Shards": [ { "HashKeyRange": { "EndingHashKey": "68056473384187692692674921486353642290", "StartingHashKey": "0" }, "SequenceNumberRange": { "StartingSequenceNumber": "49559266461454070523309915164834022007924120923395850242" }, "ShardId": "shardId-000000000000" }, ... { "HashKeyRange": { "EndingHashKey": "340282366920938463463374607431768211455", "StartingHashKey": "272225893536750770770699685945414569164" }, "SequenceNumberRange": { "StartingSequenceNumber": "49559266461543273504104037657400164881014714369419771970" }, "ShardId": "shardId-000000000004" } ], "StreamARN": "arn:aws:kinesis:us-east-1:12345678901:stream/myStream", "StreamName": "myStream", "StreamStatus": "ACTIVE" } }

    After you deploy the API, you can make a REST request against this API method:

    Copy
    GET https://your-api-id.execute-api.region.amazonaws.com/stage/streams/myStream HTTP/1.1 Host: your-api-id.execute-api.region.amazonaws.com Content-Type: application/json Authorization: ... X-Amz-Date: 20160323T194451Z

To configure and test the POST method on a stream resource

  1. Set up the POST method on a stream resource to create the stream in Amazon Kinesis, as shown in the following:

    
                                Set up the POST method on a stream resource to invoke the
                                        CreateStream action in Amazon Kinesis.

  2. Map data from the POST method request to the integration request, as shown in the following:

    
                                Map data from a POST-on-Stream method request to the
                                    integration request.

    In this example, we use the following mapping template to set ShardCount to a fixed value of 5 if the client does not specify a value in the method request payload. Otherwise, we pass the client-supplied value to the back end.

    Copy
    { "ShardCount": #if($input.path('$.ShardCount') == '') 5 #else $input.path('$.ShardCount')", "StreamName": "$input.params('stream-name')" }

    The preceding if ( $input.path('$.ShardCount') == '' ) ... Boolean expression tests if the method request's JSON payload does not have the ShardCount property declared or if the property value is empty.

  3. Test the POST method to create a named stream in Amazon Kinesis:

    From the API Gateway console, select /streams/{stream-name}/POST in the Resources pane, choose Test to start testing, type the name of an existing Amazon Kinesis stream in Path for stream-name, and choose Test. If the test is successful, a 200 OK response is returned with no data.

    After you deploy the API, you can also make a REST API request against the POST method on a Stream resource to invoke the CreateStream action in Amazon Kinesis:

    Copy
    POST https://your-api-id.execute-api.region.amazonaws.com/stage/streams/yourStream HTTP/1.1 Host: your-api-id.execute-api.region.amazonaws.com Content-Type: application/json Authorization: ... X-Amz-Date: 20160323T194451Z { "ShardCount": 5 }

Configure and test the DELETE method on a stream resource

  1. Set up the DELETE method to invoke the DeleteStream action in Amazon Kinesis, as shown in the following.

    
                                Set up the DELETE method to invoke the
                                        DeleteStream action in Amazon Kinesis

  2. Map data from the DELETE method request to the integration request, as shown in the following:

    
                                Map data from the DELETE-on-Stream method to the integration
                                    request.

  3. Test the DELETE method to delete a named stream in Amazon Kinesis:

    From the API Gateway console, select the /streams/{stream-name}/DELETE method node in the Resources pane, choose Test to start testing, type the name of an existing Amazon Kinesis stream in Path for stream-name, and choose Test. If the test is successful, a 200 OK response is returned with no data.

    After you deploy the API, you can also make the following REST API request against the DELETE method on the Stream resource to call the DeleteStream action in Amazon Kinesis:

    Copy
    DELETE https://your-api-id.execute-api.region.amazonaws.com/stage/streams/yourStream HTTP/1.1 Host: your-api-id.execute-api.region.amazonaws.com Content-Type: application/json Authorization: ... X-Amz-Date: 20160323T194451Z {}

Get Records from and Add Records to a Stream in Amazon Kinesis

After you create a stream in Amazon Kinesis, you can add data records to the stream and read the data from the stream. Adding data records involves calling the PutRecords or PutRecord action in Amazon Kinesis. The former adds multiple records whereas the latter adds a single record to the stream.

Copy
POST /?Action=PutRecords HTTP/1.1 Host: kinesis.region.domain Authorization: AWS4-HMAC-SHA256 Credential=..., ... ... Content-Type: application/x-amz-json-1.1 Content-Length: PayloadSizeBytes { "Records": [ { "Data": blob, "ExplicitHashKey": "string", "PartitionKey": "string" } ], "StreamName": "string" }

or

Copy
POST /?Action=PutRecord HTTP/1.1 Host: kinesis.region.domain Authorization: AWS4-HMAC-SHA256 Credential=..., ... ... Content-Type: application/x-amz-json-1.1 Content-Length: PayloadSizeBytes { "Data": blob, "ExplicitHashKey": "string", "PartitionKey": "string", "SequenceNumberForOrdering": "string", "StreamName": "string" }

Here, StreamName identifies the target stream to add records. StreamName, Data, and PartitionKey are required input data. In our example, we use the default values for all of the optional input data and will not explicitly specify values for them in the input to the method request.

Reading data in Amazon Kinesis amounts to calling the GetRecords action:

Copy
POST /?Action=GetRecords HTTP/1.1 Host: kinesis.region.domain Authorization: AWS4-HMAC-SHA256 Credential=..., ... ... Content-Type: application/x-amz-json-1.1 Content-Length: PayloadSizeBytes { "ShardIterator": "string", "Limit": number }

Here, the source stream from which we are getting records is specified in the required ShardIterator value, as is shown in the following Amazon Kinesis action to obtain a shard iterator:

Copy
POST /?Action=GetShardIterator HTTP/1.1 Host: kinesis.region.domain Authorization: AWS4-HMAC-SHA256 Credential=..., ... ... Content-Type: application/x-amz-json-1.1 Content-Length: PayloadSizeBytes { "ShardId": "string", "ShardIteratorType": "string", "StartingSequenceNumber": "string", "StreamName": "string" }

For the GetRecords and PutRecords actions, we expose the GET and PUT methods, respectively, on a /records resource that is appended to a named stream resource (/{stream-name}). Similarly, we expose the PutRecord action as a PUT method on a /record resource.

Because the GetRecords action takes as input a ShardIterator value, which is obtained by calling the GetShardIterator helper action, we expose a GET helper method on a ShardIterator resource (/sharditerator).

The following figure shows the API structure of resources after the methods are created:


                        Create Records:GET|PUT|PUT|GET method for the API.

The following four procedures describe how to set up each of the methods, how to map data from the method requests to the integration requests, and how to test the methods.

To configure and test the PUT method on the record resource in the API to invoke the PutRecord action in Amazon Kinesis:

  1. Set up the PUT method, as shown in the following:

    
                                Set up the PUT method to call the PutRecord
                                    action in Amazon Kinesis.

  2. Configure data mapping for the PUT-on-Record method, as shown in the following:

    
                                Configure data mapping for the PUT
                                    /streams/{stream-name}/record method.

    The preceding mapping template assumes that the method request payload is of the following format:

    Copy
    { "Data": "some data", "PartitionKey": "some key" }

    This data can be modeled by the following JSON schema:

    Copy
    { "$schema": "http://json-schema.org/draft-04/schema#", "title": "PutRecord proxy single-record payload", "type": "object", "properties": { "Data": { "type": "string" }, "PartitionKey": { "type": "string" } } }

    You can create a model to include this schema and use the model to facilitate generating the mapping template. However, you can generate a mapping template without using any model.

  3. To test the PUT method, set the stream-name path variable to an existing stream, supply a payload of the preceding format, and then submit the method request. The successful result is a 200 OK response with a payload of the following format:

    Copy
    { "SequenceNumber": "49559409944537880850133345460169886593573102115167928386", "ShardId": "shardId-000000000004" }

To configure and test the PUT method on the records resource in the API to invoke the PutRecords action in Amazon Kinesis

  1. Set up the PUT method, as shown in the following:

    
                                Set up the PUT /streams/{stream-name}/records method for the
                                    Amazon Kinesis PutRecords action.

  2. Configure data mapping for the PUT method, as shown in the following:

    
                                Configure data mapping for the POST
                                    /streams/{stream-name}/records method.

    The preceding mapping template assumes the method request payload can be modeled by following JSON schema:

    Copy
    { "$schema": "http://json-schema.org/draft-04/schema#", "title": "PutRecords proxy payload data", "type": "object", "properties": { "records": { "type": "array", "items": { "type": "object", "properties": { "data": { "type": "string" }, "partition-key": { "type": "string" } } } } } }
  3. To test the PUT method, set the stream-name path variable to an existing stream, supply a payload as previously shown, and submit the method request. The successful result is a 200 OK response with a payload of the following format:

    Copy
    { "records": [ { "data": "some data", "partition-key": "some key" }, { "data": "some other data", "partition-key": "some key" } ] }

    The response payload will be similar to the following output:

    Copy
    { "FailedRecordCount": 0, "Records": [ { "SequenceNumber": "49559409944537880850133345460167468741933742152373764162", "ShardId": "shardId-000000000004" }, { "SequenceNumber": "49559409944537880850133345460168677667753356781548470338", "ShardId": "shardId-000000000004" } ] }

To configure and test the GET method on the ShardIterator resource in the API to invoke the GetShardIterator action in Amazon Kinesis

The GET-on-ShardIterator method is a helper method to acquire a required shard iterator before calling the GET-on-Records method.

  1. Set up the GET-on-ShardIterator method, as shown in the following:

    
                                Set up the GET /streams/{stream-name}/sharditerator
                                    method.

  2. The GetShardIterator action requires an input of a ShardId value. To pass a client-supplied ShardId value, we add a shard-id query parameter to the method request, as shown in the following:

    
                                Add the shard-id query parameter to the
                                    GET-on-ShardIterator method request.

    In the following mapping template, we add the translation of the shard-id query parameter value to the ShardId property value of the JSON payload for the GetShardIterator action in Amazon Kinesis.

  3. Configure data mapping for the GET-on-ShardIterator method:

    
                                Configure data mapping for the GET
                                    /streams/{stream-name}/sharditerator method.

  4. Using the Test option in the API Gateway console, enter an existing stream name as the stream-name Path variable value, set the shard-id Query string to an existing ShardId value (e.g., shard-000000000004), and choose Test.

    The successful response payload will be similar to the following output:

    Copy
    { "ShardIterator": "AAAAAAAAAAFYVN3VlFy..." }

    Make note of the ShardIterator value. You will need it to get records from a stream.

To configure and test the GET Method on the records resource in the API to invoke the GetRecords action in Amazon Kinesis

  1. Set up the GET method, as shown in the following:

    
                                Set up the GET /streams/{stream-name}/records
                                    method.

  2. The GetRecords action requires an input of a ShardIterator value. To pass a client-supplied ShardIterator value, we add a Shard-Iterator header parameter to the method request, as shown in the following:

    
                                Add the Shard-Iterator header parameter to the
                                    GET-on-Records method request.

    In the following mapping template, we add the mapping from the Shard-Iterator header value to the ShardIterator property value of the JSON payload for the GetRecords action in Amazon Kinesis.

  3. Configure data mapping for the GET-on-Records method:

    
                                Configure data mapping for the GET
                                    /streams/{stream-name}/records method.

  4. Using the Test option in the API Gateway console, type an existing stream name as the stream-name Path variable value, set the Shard-Iterator Header to the ShardIterator value obtained from the test run of the GET-on-ShardIterator method (above), and choose Test.

    The successful response payload will be similar to the following output:

    Copy
    { "MillisBehindLatest": 0, "NextShardIterator": "AAAAAAAAAAF...", "Records": [ ... ] }