Amazon Kinesis Data Analytics
Developer Guide

Working with Amazon CloudWatch Logs

If an Amazon Kinesis Data Analytics application is misconfigured, it can transition to a running state during application start or update but not process any data into the in-application input stream. By adding a CloudWatch log option to the application, you can monitor for application configuration problems.

Amazon Kinesis Data Analytics can generate configuration errors under the following conditions:

  • The Kinesis Stream used for input doesn't exist.

  • The Amazon Kinesis Data Firehose delivery stream used for input doesn't exist.

  • The Amazon S3 bucket used as a reference data source doesn't exist.

  • The specified file in the reference data source in the S3 bucket doesn't exist.

  • The correct resource is not defined in the AWS Identity and Access Management (IAM) role that manages related permissions.

  • The correct permission is not defined in the IAM role that manages related permissions.

  • Kinesis Data Analytics doesn't have permission to assume the IAM role that manages related permissions.

For more information on Amazon CloudWatch, see the CloudWatch User Guide.

Adding the PutLogEvents Policy Action

Amazon Kinesis Data Analytics needs permissions to write misconfiguration errors to CloudWatch. You can add these permissions to the IAM role that Amazon Kinesis Data Analytics assumes, as described following. For more information on using an IAM role for Amazon Kinesis Data Analytics, see Granting Amazon Kinesis Data Analytics Permissions to Access Streaming and Reference Sources (Creating an IAM Role).

Trust Policy

To grant Kinesis Data Analytics permissions to assume an IAM role, you can attach the following trust policy to the role.

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "kinesisanalytics.amazonaws.com" }, "Action": "sts:AssumeRole" } ] }

Permissions Policy

To grant an application permissions to write log events to CloudWatch from an Kinesis Data Analytics resource, you can use the following IAM permissions policy.

{ "Version": "2012-10-17", "Statement": [ { "Sid": "Stmt0123456789000", "Effect": "Allow", "Action": [ "logs:PutLogEvents" ], "Resource": [ "arn:aws:logs:us-east-1:123456789012:log-group:my-log-group:log-stream:my-log-stream*" ] } ] }

Adding Configuration Error Monitoring

Use the following API actions to add a CloudWatch log option to a new or existing application or change a log option for an existing application.

Note

You can currently only add a CloudWatch log option to an application by using API actions. You can't add CloudWatch log options by using the console.

Adding a CloudWatch Log Option When Creating an Application

The following code example demonstrates how to use the CreateApplication action to add a CloudWatch log option when you create an application. For more information on Create_Application, see CreateApplication.

{ "ApplicationCode": "<The SQL code the new application will run on the input stream>", "ApplicationDescription": "<A friendly description for the new application>", "ApplicationName": "<The name for the new application>", "Inputs": [ ... ], "Outputs": [ ... ], "CloudWatchLoggingOptions": [{ "LogStreamARN": "<Amazon Resource Name (ARN) of the CloudWatch log stream to add to the new application>", "RoleARN": "<ARN of the role to use to access the log>" }] }

Adding a CloudWatch Log Option to an Existing Application

The following code example demonstrates how to use the AddApplicationCloudWatchLoggingOption action to add a CloudWatch log option to an existing application. For more information on AddApplicationCloudWatchLoggingOption, see AddApplicationCloudWatchLoggingOption.

{ "ApplicationName": "<Name of the application to add the log option to>", "CloudWatchLoggingOption": { "LogStreamARN": "<ARN of the log stream to add to the application>", "RoleARN": "<ARN of the role to use to access the log>" }, "CurrentApplicationVersionId": <Version of the application to add the log to> }

Updating an Existing CloudWatch Log Option

The following code example demonstrates how to use the UpdateApplication action to modify an existing CloudWatch log option. For more information on UpdateApplication, see UpdateApplication.

{ "ApplicationName": "<Name of the application to update the log option for>", "ApplicationUpdate": { "CloudWatchLoggingOptionUpdates": [ { "CloudWatchLoggingOptionId": "<ID of the logging option to modify>", "LogStreamARNUpdate": "<ARN of the new log stream to use>", "RoleARNUpdate": "<ARN of the new role to use to access the log stream>" } ], }, "CurrentApplicationVersionId": <ID of the application version to modify> }

Deleting a CloudWatch Log Option from an Application

The following code example demonstrates how to use the DeleteApplicationCloudWatchLoggingOption action to delete an existing CloudWatch log option. For more information on DeleteApplicationCloudWatchLoggingOption, see DeleteApplicationCloudWatchLoggingOption.

{ "ApplicationName": "<Name of application to delete log option from>", "CloudWatchLoggingOptionId": "<ID of the application log option to delete>", "CurrentApplicationVersionId": <Version of the application to delete the log option from> }

Configuration Errors

Following, you can learn details about errors that you might see in CloudWatch logs from a misconfigured application.

Error Message Format

Error messages generated by application misconfiguration appear in the following format.

{ "applicationARN": "string", "applicationVersionId": integer, "messageType": "ERROR", "message": "string", "inputId": "string", "referenceId": "string", "errorCode": "string" "messageSchemaVersion": "integer", }

The fields in an error message contain the following information:

  • applicationARN: The Amazon Resource Name (ARN) of the generating application, for example: arn:aws:kinesisanalytics:us-east-1:112233445566:application/sampleApp

  • applicationVersionId: The version of the application at the time the error was encountered. For more information, see ApplicationDetail.

  • messageType: The message type. Currently, this type can be only ERROR.

  • message: The details of the error, for example:

    There is a problem related to the configuration of your input. Please check that the resource exists, the role has the correct permissions to access the resource and that Kinesis Analytics can assume the role provided.
  • inputId: ID associated with the application input. This value is only present if this input is the cause of the error. This value is not present if referenceId is present. For more information, see DescribeApplication.

  • referenceId: ID associated with the application reference data source. This value is only present if this source is the cause of the error. This value is not present if inputId is present. For more information, see DescribeApplication.

  • errorCode: The identifier for the error. This ID is either InputError or ReferenceDataError.

  • messageSchemaVersion: A value that specifies the current message schema version, currently 1. You can check this value to see if the error message schema has been updated.

Errors

The errors that might appear in CloudWatch logs for Amazon Kinesis Data Analytics include the following.

Resource Does Not Exist

If an ARN is specified for an Kinesis input stream that doesn't exist, but the ARN is syntactically correct, an error like the following is generated.

{ "applicationARN": "arn:aws:kinesisanalytics:us-east-1:112233445566:application/sampleApp", "applicationVersionId": "5", "messageType": "ERROR", "message": "There is a problem related to the configuration of your input. Please check that the resource exists, the role has the correct permissions to access the resource and that Kinesis Analytics can assume the role provided.", "inputId":"1.1", "errorCode": "InputError", "messageSchemaVersion": "1" }

If an incorrect Amazon S3 file key is used for reference data, an error like the following is generated.

{ "applicationARN": "arn:aws:kinesisanalytics:us-east-1:112233445566:application/sampleApp", "applicationVersionId": "5", "messageType": "ERROR", "message": "There is a problem related to the configuration of your reference data. Please check that the bucket and the file exist, the role has the correct permissions to access these resources and that Kinesis Analytics can assume the role provided.", "referenceId":"1.1", "errorCode": "ReferenceDataError", "messageSchemaVersion": "1" }

Role Does Not Exist

If an ARN is specified for an IAM input role that doesn't exist, but the ARN is syntactically correct, an error like the following is generated.

{ "applicationARN": "arn:aws:kinesisanalytics:us-east-1:112233445566:application/sampleApp", "applicationVersionId": "5", "messageType": "ERROR", "message": "There is a problem related to the configuration of your input. Please check that the resource exists, the role has the correct permissions to access the resource and that Kinesis Analytics can assume the role provided.", "inputId":null, "errorCode": "InputError", "messageSchemaVersion": "1" }

Role Does Not Have Permissions to Access the Resource

If an input role is used that doesn't have permission to access the input resources, such as an Kinesis source stream, an error like the following is generated.

{ "applicationARN": "arn:aws:kinesisanalytics:us-east-1:112233445566:application/sampleApp", "applicationVersionId": "5", "messageType": "ERROR", "message": "There is a problem related to the configuration of your input. Please check that the resource exists, the role has the correct permissions to access the resource and that Kinesis Analytics can assume the role provided.", "inputId":null, "errorCode": "InputError", "messageSchemaVersion": "1" }