Writing App Data to Amazon Kinesis

Amazon Kinesis is a fully managed service for real-time processing of streaming data at massive scale. Amazon Kinesis Firehose is a fully managed service for delivering real-time streaming data to destinations such as Amazon Simple Storage Service (Amazon S3) and Amazon Redshift. With Firehose, you do not need to write any applications or manage any resources. You configure your data producers to send data to Firehose and it automatically delivers the data to the destination that you specified. The tutorial below explains how to integrate Amazon Kinesis and/or Amazon Kinesis Firehose with your app.

Project Setup#

Prerequisites#

You must complete all of the instructions on the Set Up the SDK for Android page before beginning this tutorial.

Initialize KinesisRecorder for Amazon Kinesis#

Pass your initialized Amazon Cognito credentials provider to the KinesisRecorder constructor:

String kinesisDirectory = "YOUR_UNIQUE_DIRECTORY";
KinesisRecorder recorder = new KinesisRecorder(
        myActivity.getDir(kinesisDirectory, 0), // An empty directory KinesisRecorder can use for storing requests
        Regions.US_WEST_2,  // Region that this Recorder should save and send requests to
        credentialsProvider); // The credentials provider to use when making requests to AWS

Initialize KinesisFirehoseRecorder for Amazon Kinesis Firehose#

Pass your initialized Amazon Cognito credentials provider to the KinesisFirehoseRecorder constructor:

String kinesisFirehoseDirectory = "YOUR_UNIQUE_DIRECTORY";
KinesisFirehoseRecorder recorder = new KinesisFirehoseRecorder(
        myActivity.getDir(kinesisFirehoseDirectory, 0), /* An empty directory KinesisFirehoseRecorder can use for storing requests */
        Regions.US_WEST_2,  /* Region that this Recorder should save and send requests to */
        credentialsProvider); /* The credentials provider to use when making requests to AWS */

Create a Kinesis Stream#

In order to use the Amazon Kinesis recorder, you must first create an Amazon Kinesis stream. You can create new streams in the Kinesis Console.

Grant Role Access to Your Amazon Kinesis Stream#

The default IAM role policy grants you access to Amazon Mobile Analytics and Amazon Cognito Sync. To use Amazon Kinesis in an application, you must allow the IAM roles associated with your Cognito Identity Pool access to your Kinesis stream. To set this policy:

  1. Navigate to the Identity and Access Management Console and choose Roles in the left-hand pane.

  2. Type your Identity Pool name into the search box. Two roles will be listed: one for unauthenticated users and one for authenticated users.

  3. Choose the role for unauthenticated users (it will have "unauth" appended to your Identity Pool name).

  4. Scroll down the web page until you see the Create Role Policy. Choose it, select Policy Generator, and then choose the Select button.

  5. Select the Allow radio button, Amazon Kinesis in the AWS Service drop-down, PutRecord under Actions, and enter the ARN to your Kinesis stream in the Amazon Resource Name (ARN) text box.

    "Resource": arn:aws:kinesis:region:account:stream/name
    "Resource": arn:aws:kinesis:us-west-2:111122223333:stream/my-stream
    
  6. Choose the Add Statement button, the Next Step button, and the Apply Policy button.

To learn more about Kinesis-specific policies, see Controlling Access to Amazon Kinesis Resources with IAM.

Grant Role Access to Your Kinesis Firehose Delivery Stream#

The default IAM role policy grants you access to Amazon Mobile Analytics and Amazon Cognito Sync. To use Kinesis Firehose in an application, you must allow the IAM roles associated with your Amazon Cognito Identity Pool access to your Kinesis Firehose delivery stream. To set this policy:

  1. Navigate to the Identity and Access Management Console and choose Roles in the left-hand pane.

  2. Type your Identity Pool name into the search box. Two roles will be listed: one for unauthenticated users and one for authenticated users.

  3. Choose the role for unauthenticated users (it will have "unauth" appended to your Identity Pool name).

  4. Scroll down the web page until you see the Create Role Policy. Choose it, select Policy Generator, and then choose the Select button.

  5. Select the Allow radio button, Amazon Kinesis in the AWS Service drop-down, PutRecord under Actions, and enter the ARN to your Kinesis stream in the Amazon Resource Name (ARN) text box.

    "Resource": arn:aws:firehose:region:account:stream/name
    "Resource": arn:aws:firehose:us-west-2:111122223333:deliverystream/my-stream
    
  6. Choose the Add Statement button, the Next Step button, and the Apply Policy button.

To learn more about Kinesis Firehose-specific policies, see Controlling Access to Amazon Kinesis Firehose.

Configure the Kinesis Service Client#

Use the KinesisRecorder class to interact with the Kinesis service. The following snippet creates an instance of the Kinesis service client:

String kinesisDirectory = "YOUR_UNIQUE_DIRECTORY";
KinesisRecorder recorder = new KinesisRecorder(
        myActivity.getDir(kinesisDirectory, 0),
        Regions.US_WEST_2,
        credentialsProvider);

YOUR_UNIQUE_DIRECTORY is a folder that should be exclusive to the Kinesis Recorder and will be used to store records. The region here should match the region you specified in the console.

Note

KinesisRecorder uses synchronous calls, so you shouldn't call KinesisRecorder methods on the main thread.

Save Records to Local Storage#

With KinesisRecorder created and configured, you can use saveRecord() to save records to local storage:

recorder.saveRecord("MyData".getBytes(),"MyStreamName");

Submit Records to Kinesis Stream#

Use the submitAllRecords synchronous method on the KinesisRecorder object to send all locally saved records to your Kinesis stream.

recorder.submitAllRecords();

To learn more about working with Amazon Kinesis, see the Amazon Kinesis Developer Resources.

To learn more about working with Amazon Kinesis Firehose, see the Amazon Kinesis Firehose Documentation.

To learn more about the Kinesis classes, see the class reference for AWSKinesisRecorder.