Amazon Cognito
Developer Guide (Version Last Updated: 07/28/2016)

Amazon Cognito Streams

Amazon Cognito Streams gives developers control and insight into their data stored in Amazon Cognito. Developers can now configure an Kinesis stream to receive events as data is updated and synchronized. Amazon Cognito can push each dataset change to an Kinesis stream you own in real time.

Using Amazon Cognito Streams, you can move all of your Sync data to Kinesis, which can then be streamed to a data warehouse tool such as Amazon Redshift for further analysis. To learn more about Kinesis, see Getting Started Using Amazon Kinesis.

Configuring Streams

You can set up Amazon Cognito Streams in the Amazon Cognito console. To enable Amazon Cognito Streams in the Amazon Cognito console, you need to select the Kinesis stream to publish to and an IAM role that grants Amazon Cognito permission to put events in the selected stream.

From the console home page:

  1. Click the name of the identity pool for which you want to set up Amazon Cognito Streams. The Dashboard page for your identity pool appears.

  2. In the top-right corner of the Dashboard page, click Edit identity pool. The Edit identity pool page appears.

  3. Scroll down and click Cognito Streams to expand it.

  4. In the Stream name dropdown menu, select the name of an existing Kinesis stream. Alternatively, click Create stream to create one, entering a stream name and the number of shards. To learn about shards and for help on estimating the number of shards needed for your stream, see the Kinesis Developer Guide.

  5. In the Publish role dropdown menu, select the IAM role that grants Amazon Cognito permission to publish your stream. Click Create role to create or modify the roles associated with your identity pool in the AWS IAM Console.

  6. In the Stream status dropdown menu, select Enabled to enable the stream updates. Click Save Changes.

After you've successfully configured Amazon Cognito streams, all subsequent updates to datasets in this identity pool will be sent to the stream.

Stream Contents

Each record sent to the stream represents a single synchronization. Here is an example of a record sent to the stream:

{ "identityPoolId": "Pool Id", "identityId": "Identity Id", "dataSetName": "Dataset Name", "operation": "(replace|remove)", "kinesisSyncRecords": [ { "key": "Key", "value": "Value", "syncCount": 1, "lastModifiedDate": 1424801824343, "deviceLastModifiedDate": 1424801824343, "op": "(replace|remove)" }, ... ], "lastModifiedDate": 1424801824343, "kinesisSyncRecordsURL": "S3Url", "payloadType": "(S3Url|Inline)", "syncCount": 1 }

For updates that are larger than the Kinesis maximum payload size of 50 KB, a presigned Amazon S3 URL will be included that contains the full contents of the update.

After you have configured Amazon Cognito streams, if you delete the Kinesis stream or change the role trust permission so that it can no longer be assumed by Amazon Cognito Sync, Amazon Cognito streams will be disabled. You will need to either recreate the Kinesis stream or fix the role, and then you will need to reenable the stream.

Bulk Publishing

Once you have configured Amazon Cognito streams, you will be able to execute a bulk publish operation for the existing data in your identity pool. After you initiate a bulk publish operation, either via the console or directly via the API, Amazon Cognito will start publishing this data to the same stream that is receiving your updates.

Amazon Cognito does not guarantee uniqueness of data sent to the stream when using the bulk publish operation. You may receive the same update both as an update as well as part of a bulk publish. Keep this in mind when processing the records from your stream.

To bulk publish all of your streams, follow steps 1-6 under Configuring Streams and then click Start bulk publish. You are limited to one ongoing bulk publish operation at any given time and to one successful bulk publish request every 24 hours.