Amazon Kinesis Data Analytics
Developer Guide

Application Scaling in Kinesis Data Analytics for Java Applications

You can configure the parallel execution of tasks and the allocation of resources for Amazon Kinesis Data Analytics for Java Applications to implement scaling. For information about how Apache Flink schedules parallel instances of tasks, see Parallel Execution in the Apache Flink documentation.

Configuring Application Parallelism and ParallelismPerKPU

You configure the parallel execution for your Kinesis Data Analytics for Java application tasks (such as reading from a source or executing an operator) using the following ParallelismConfiguration properties:

  • Parallelism — Use this property to set the default Apache Flink application parallelism. All operators, sources, and sinks execute with this parallelism unless they are overridden in the application code. The default is 1, and the default maximum is 64.

  • ParallelismPerKPU — Use this property to set the number of parallel tasks that can be scheduled per Kinesis Processing Unit (KPU) of your application. The default is 1, and the default maximum is 8. For applications that have blocking operations (for example, I/O), a higher value of ParallelismPerKPU leads to full utilization of KPU resources.

Note

The limit for Paralellism is equal to ParallelismPerKPU times the limit for KPUs (which has a default of 8). The KPUs limit can be increased by requesting a limit increase. For instructions on how to request a limit increase, see "To request a limit increase" in AWS Service Limits.

For information about setting task parallelism for a specific operator, see Setting the Parallelism: Operator in the Apache Flink documentation.

Allocating Kinesis Processing Units

Kinesis Data Analytics provisions capacity as KPUs. A single KPU provides you with 1 vCPU and 4 GB of memory. For every KPU allocated, 50 GB of running application storage is also provided.

Kinesis Data Analytics calculates the KPUs that are needed to run your application using the Parallelism and ParallelismPerKPU properties, as follows:

Allocated KPUs for the application = Parallelism/ParallelismPerKPU

Kinesis Data Analytics quickly gives your applications resources in response to spikes in throughput or processing activity. It removes resources from your application gradually after the activity spike has passed. To disable the automatic allocation of resources, set the ApplicationAutoScalingDisabled value to true, as described later in Updating Your Application's Parallelism.

The default limit for KPUs for your application is eight. For instructions on how to request an increase to this limit, see "To request a limit increase" in AWS Service Limits.

Automatic Scaling

Kinesis Data Analytics elastically scales your application’s parallelism to accommodate the data throughput of your source and your operator complexity for most scenarios. You can use the AutoScalingEnabled property (part of FlinkApplicationConfiguration ) to enable or disable auto scaling behavior. Automatic scaling is enabled by default.

Note

Your AWS account is charged for KPUs that Kinesis Data Analytics provisions which is a function of your application's parallelism and parallelismPerKPU settings. You can opt into service autoscaling functionality to optimize these costs as the service scales up when your application needs additional resources, and scales down when your application no longer needs resources. An activity spike increases your Kinesis Data Analytics costs.

For information about pricing, see Amazon Kinesis Data Analytics pricing.

Your application has a default limit of 8 KPUs. For more information, see Limits.

Updating Your Application's Parallelism

This section contains sample requests for API actions that set an application's parallelism. For more examples and instructions for how to use request blocks with API actions, see Kinesis Data Analytics API Example Code.

The following example request for the CreateApplication action sets parallelism when you are creating an application:

{ "ApplicationName": "string", ... "ApplicationConfiguration": { ... "FlinkApplicationConfiguration": { "ParallelismConfiguration": { "AutoScalingEnabled": "true", "ConfigurationType": "CUSTOM", "Parallelism": 4, "ParallelismPerKPU": 4 } } } }

The following example request for the UpdateApplication action sets parallelism for an existing application:

{ "ApplicationName": "MyApplication", "ApplicationConfigurationUpdate": { "FlinkApplicationConfigurationUpdate": { "ParallelismConfigurationUpdate": { "AutoScalingEnabledUpdate": "true", "ConfigurationType": "CUSTOM", "ParallelismPerKPUUpdate": 4, "ParallelismUpdate": 4 } } } }

The following example request for the UpdateApplication action disables parallelism for an existing application:

{ "ApplicationName": "MyApplication", "ApplicationConfigurationUpdate": { "FlinkApplicationConfigurationUpdate": { "ParallelismConfigurationUpdate": { "AutoScalingEnabledUpdate": "false" } } } }