Amazon Kinesis Data Analytics
Developer Guide

The AWS Documentation website is getting a new look!
Try it now and let us know what you think. Switch to the new look >>

You can return to the original look by selecting English in the language selector above.

Application Scaling in Kinesis Data Analytics for Java Applications

You can configure the parallel execution of tasks and the allocation of resources for Amazon Kinesis Data Analytics for Java Applications to implement scaling. For information about how Apache Flink schedules parallel instances of tasks, see Parallel Execution in the Apache Flink documentation.

Configuring Application Parallelism and ParallelismPerKPU

You configure the parallel execution for your Kinesis Data Analytics for Java application tasks (such as reading from a source or executing an operator) using the following ParallelismConfiguration properties:

  • Parallelism — Use this property to set the default Apache Flink application parallelism. All operators, sources, and sinks execute with this parallelism unless they are overridden in the application code. The default is 1, and the default maximum is 256.

  • ParallelismPerKPU — Use this property to set the number of parallel tasks that can be scheduled per Kinesis Processing Unit (KPU) of your application. The default is 1, and the maximum is 8. For applications that have blocking operations (for example, I/O), a higher value of ParallelismPerKPU leads to full utilization of KPU resources.

Note

The limit for Paralellism is equal to ParallelismPerKPU times the limit for KPUs (which has a default of 32). The KPUs limit can be increased by requesting a limit increase. For instructions on how to request a limit increase, see "To request a limit increase" in AWS Service Limits.

For information about setting task parallelism for a specific operator, see Setting the Parallelism: Operator in the Apache Flink documentation.

Allocating Kinesis Processing Units

Kinesis Data Analytics provisions capacity as KPUs. A single KPU provides you with 1 vCPU and 4 GB of memory. For every KPU allocated, 50 GB of running application storage is also provided.

Kinesis Data Analytics calculates the KPUs that are needed to run your application using the Parallelism and ParallelismPerKPU properties, as follows:

Allocated KPUs for the application = Parallelism/ParallelismPerKPU

Kinesis Data Analytics quickly gives your applications resources in response to spikes in throughput or processing activity. It removes resources from your application gradually after the activity spike has passed. To disable the automatic allocation of resources, set the AutoScalingEnabled value to false, as described later in Updating Your Application's Parallelism.

The default limit for KPUs for your application is 32. For instructions on how to request an increase to this limit, see "To request a limit increase" in AWS Service Limits.

Automatic Scaling

Kinesis Data Analytics elastically scales your application’s parallelism to accommodate the data throughput of your source and your operator complexity for most scenarios. Kinesis Data Analytics monitors the resource (CPU) usage of your application, and elastically scales your application's parallelism. Your application scales up (increases parallelism) is typically triggered in minutes after a constant spike in CPU usage. Your application scales down (decreases parallelism) over a few hours after a drop in CPU usage. Kinesis Data Analytics will not reduce your application's CurrentParallelism value to less than your application's Parallelism setting.

You can use the AutoScalingEnabled property (part of FlinkApplicationConfiguration ) to enable or disable auto scaling behavior. Your AWS account is charged for KPUs that Kinesis Data Analytics provisions which is a function of your application's parallelism and parallelismPerKPU settings. An activity spike increases your Kinesis Data Analytics costs.

For information about pricing, see Amazon Kinesis Data Analytics pricing.

Note the following about application scaling:

  • Automatic scaling is enabled by default.

  • Your application has a default limit of 32 KPUs. For more information, see Limits.

  • When autoscaling updates application parallelism, the application experiences downtime. To avoid this downtime, do the following:

Updating Your Application's Parallelism

This section contains sample requests for API actions that set an application's parallelism. For more examples and instructions for how to use request blocks with API actions, see Kinesis Data Analytics API Example Code.

The following example request for the CreateApplication action sets parallelism when you are creating an application:

{ "ApplicationName": "string", "RuntimeEnvironment":"FLINK-1_6", "ServiceExecutionRole":"arn:aws:iam::123456789123:role/myrole", "ApplicationConfiguration": { "ApplicationCodeConfiguration":{ "CodeContent":{ "S3ContentLocation":{ "BucketARN":"arn:aws:s3:::mybucket", "FileKey":"myflink.jar", "ObjectVersion":"AbCdEfGhIjKlMnOpQrStUvWxYz12345" } }, "CodeContentType":"ZIPFILE" }, "FlinkApplicationConfiguration": { "ParallelismConfiguration": { "AutoScalingEnabled": "true", "ConfigurationType": "CUSTOM", "Parallelism": 4, "ParallelismPerKPU": 4 } } } }

The following example request for the UpdateApplication action sets parallelism for an existing application:

{ "ApplicationName": "MyApplication", "ApplicationConfigurationUpdate": { "FlinkApplicationConfigurationUpdate": { "ParallelismConfigurationUpdate": { "AutoScalingEnabledUpdate": "true", "ConfigurationType": "CUSTOM", "ParallelismPerKPUUpdate": 4, "ParallelismUpdate": 4 } } } }

The following example request for the UpdateApplication action disables parallelism for an existing application:

{ "ApplicationName": "MyApplication", "ApplicationConfigurationUpdate": { "FlinkApplicationConfigurationUpdate": { "ParallelismConfigurationUpdate": { "AutoScalingEnabledUpdate": "false" } } } }