Amazon Kinesis Data Analytics
Developer Guide

Working with Amazon CloudWatch Logs

By adding an Amazon CloudWatch Logs option to your Amazon Kinesis Data Analytics for Java application, you can monitor for application events or configuration problems.

Note

To complete the following exercises, you need a CloudWatch log group and log stream. For information about creating a CloudWatch log group and log stream, and other CloudWatch topics, see the Amazon CloudWatch User Guide.

Adding the PutLogEvents Policy Action

Amazon Kinesis Data Analytics needs permissions to write misconfiguration errors to CloudWatch. You can add these permissions to the AWS Identity and Access Management (IAM) role that Amazon Kinesis Data Analytics assumes, as described following.

Trust Policy

To grant Kinesis Data Analytics permissions to assume an IAM role, you can attach the following trust policy to the service execution role.

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "kinesisanalytics.amazonaws.com" }, "Action": "sts:AssumeRole" } ] }

Permissions Policy

To grant permissions to an application to write log events to CloudWatch from a Kinesis Data Analytics resource, you can use the following IAM permissions policy. Provide the correct Amazon Resource Names (ARNs) for your log group and stream.

{ "Version": "2012-10-17", "Statement": [ { "Sid": "Stmt0123456789000", "Effect": "Allow", "Action": [ "logs:PutLogEvents", "logs:DescribeLogGroups", "logs:DescribeLogStreams" ], "Resource": [ "arn:aws:logs:us-east-1:123456789012:log-group:my-log-group:log-stream:my-log-stream*", "arn:aws:logs:us-east-1:093291321484:log-group:my-log-group:*", "arn:aws:logs:us-east-1:093291321484:log-group:*", ] } ] }

Adding Configuration Error Monitoring

Use the following API actions to add a CloudWatch log option to a new or existing application or change a log option for an existing application. For information about how to use a JSON file for input for an API action, see Kinesis Data Analytics API Example Code.

Adding a CloudWatch Log Option When Creating an Application

The following example demonstrates how to use the CreateApplication action to add a CloudWatch log option when you create an application. For more information, see CreateApplication.

{ "ApplicationName": "test", "ApplicationDescription": "test-application-description", "RuntimeEnvironment": "FLINK-1_6", "ServiceExecutionRole": "arn:aws:iam::123456789123:role/myrole", "ApplicationConfiguration": { "ApplicationCodeConfiguration": { "CodeContent": { "S3ContentLocation":{ "BucketARN": "arn:aws:s3:::mybucket", "FileKey": "myflink.jar" } }, "CodeContentType": "ZIPFILE" } }, "CloudWatchLoggingOptions": [{ "LogStreamARN": "<Amazon Resource Name (ARN) of the CloudWatch log stream to add to the new application>" }] }

Adding a CloudWatch Log Option to an Existing Application

The following example demonstrates how to use the AddApplicationCloudWatchLoggingOption action to add a CloudWatch log option to an existing application. For more information, see AddApplicationCloudWatchLoggingOption.

{ "ApplicationName": "<Name of the application to add the log option to>", "CloudWatchLoggingOption": { "LogStreamARN": "<ARN of the log stream to add to the application>" }, "CurrentApplicationVersionId": <Version of the application to add the log to> }

Updating an Existing CloudWatch Log Option

The following example demonstrates how to use the UpdateApplication action to modify an existing CloudWatch log option. For more information, see UpdateApplication.

{ "ApplicationName": "<Name of the application to update the log option for>", "CloudWatchLoggingOptionUpdates": [ { "CloudWatchLoggingOptionId": "<ID of the logging option to modify>", "LogStreamARNUpdate": "<ARN of the new log stream to use>" } ], "CurrentApplicationVersionId": <ID of the application version to modify> }

Deleting a CloudWatch Log Option from an Application

The following example demonstrates how to use the DeleteApplicationCloudWatchLoggingOption action to delete an existing CloudWatch log option. For more information, see DeleteApplicationCloudWatchLoggingOption.

{ "ApplicationName": "<Name of application to delete log option from>", "CloudWatchLoggingOptionId": "<ID of the application log option to delete>", "CurrentApplicationVersionId": <Version of the application to delete the log option from> }

Setting the Application Logging Level

To set the level of application logging, use the MonitoringConfiguration parameter of the CreateApplication action or the MonitoringConfigurationUpdate parameter of the UpdateApplication action.

Set the Application Logging Level when Creating an Application

The following example request for the CreateApplication action sets the application log level to ERROR:

{ "ApplicationName": "MyApplication", "ApplicationDescription": "My Application Description", "ApplicationConfiguration": { ... "FlinkApplicationConfiguration": "MonitoringConfiguration": { "ConfigurationType": "CUSTOM", "LogLevel": "ERROR" } }, "RuntimeEnvironment": "FLINK-1_6", "ServiceExecutionRole": "arn:aws:iam::123456789123:role/myrole" }

Update the Application Logging Level

The following example request for the UpdateApplication action sets the application log level to ERROR:

{ "ApplicationConfigurationUpdate": { "FlinkApplicationConfigurationUpdate": { "MonitoringConfigurationUpdate": { "ConfigurationTypeUpdate": "CUSTOM", "LogLevelUpdate": "ERROR" } } } }

Configuration Errors

The following are details about errors that you might see in CloudWatch Logs from a misconfigured application.

Error Message Format

Error messages generated by application misconfiguration appear in the following format.

{ "applicationARN": "string", "applicationVersionId": integer, "message": "string", "messageType": "ERROR", "messageSchemaVersion": "integer", "errorCode": "string" }

The fields in an error message contain the following information:

  • applicationARN: The Amazon Resource Name (ARN) of the generating application, for example: arn:aws:kinesisanalytics:us-east-1:112233445566:application/sampleApp

  • applicationVersionId: The version of the application at the time the error was encountered. For more information, see ApplicationDetail.

  • message: The details of the error, for example:

    There is a problem related to the configuration of your input. Please check that the resource exists, the role has the correct permissions to access the resource and that Kinesis Analytics can assume the role provided.
  • messageType: The message type. Currently, this type can only be ERROR.

  • messageSchemaVersion: A value that specifies the current message schema version, currently 1. You can check this value to see if the error message schema has been updated.

  • errorCode: The identifier for the error. This ID is either InputError or ReferenceDataError.

Errors

The following error might appear in CloudWatch Logs for Kinesis Data Analytics for Java Applications.

Invalid Application Code

If an application's code is not valid (such as missing a main method), an error like the following is generated.

{ "applicationARN": "arn:aws:kinesisanalytics:us-east-1:12345678912:application/myapplication-test", "applicationVersionId": 7, "message": "org.apache.flink.client.program.ProgramInvocationException: Neither a 'Main-Class', nor a 'program-class' entry was found in the jar file.", "messageType": "ERROR", "messageSchemaVersion": "1", "errorCode": "CodeError.InvalidApplicationCode" }