Managing Application Backups Using Snapshots - Amazon Kinesis Data Analytics

Managing Application Backups Using Snapshots

A snapshot is the Kinesis Data Analytics implementation of an Apache Flink Savepoint. A snapshot is a user- or service-triggered, created, and managed backup of the application state. For information about Apache Flink Savepoints, see Savepoints in the Apache Flink Documentation. Using snapshots, you can restart an application from a particular snapshot of application state.

Note

For your application to restart properly with correct state data, we recommend that your application create a snapshot every 5-10 minutes.

In Kinesis Data Analytics, you manage snapshots using the following API actions:

For the per-application limit on the number of snapshots, see Limits. If your application reaches the limit on snapshots, then manually creating a snapshot fails with a LimitExceededException.

Kinesis Data Analytics never deletes snapshots. You must manually delete your snapshots using the DeleteApplicationSnapshot action.

To load a saved snapshot of application state when starting an application, use the ApplicationRestoreConfiguration parameter of the StartApplication or UpdateApplication action.

Automatic Snapshot Creation

If SnapshotsEnabled is set to true in the ApplicationSnapshotConfiguration for the application, Kinesis Data Analytics automatically creates and uses snapshots when the application is updated, scaled, or stopped to provide exactly-once processing semantics.

Note

Setting ApplicationSnapshotConfiguration::SnapshotsEnabled to false will lead to data loss during application updates.

Automatically created snapshots have the following qualities:

  • The snapshot is managed by the service, but you can see the snapshot using the ListApplicationSnapshots action. Automatically created snapshots count against your snapshot limit.

  • If your application exceeds the snapshot limit, manually created snapshots will fail, but the Kinesis Data Analytics service will still successfully create snapshots when the application is updated, scaled, or stopped. You must manually delete snapshots using the DeleteApplicationSnapshot action before creating more snapshots manually.

Restoring From a Snapshot That Contains Incompatible State Data

Because snapshots contain information about operators, restoring state data from a snapshot for an operator that has changed since the previous application version may have unexpected results. An application will fault if it attempts to restore state data from a snapshot that does not correspond to the current operator. The faulted application will be stuck in either the STOPPING or UPDATING state.

To allow an application to restore from a snapshot that contains incompatible state data, set the AllowNonRestoredState parameter of the FlinkRunConfiguration to true using the UpdateApplication action.

You will see the following behavior when an application is restored from an obsolete snapshot:

  • Operator added: If a new operator is added, the savepoint has no state data for the new operator. No fault will occur, and it is not necessary to set AllowNonRestoredState.

  • Operator deleted: If an existing operator is deleted, the savepoint has state data for the missing operator. A fault will occur unless AllowNonRestoredState is set to true.

  • Operator modified: If compatible changes are made, such as changing a parameter's type to a compatible type, the application can restore from the obsolete snapshot. For more information about restoring from snapshots, see Savepoints in the Apache Flink Documentation. An application that uses Apache Flink version 1.8 can possibly be restored from a snapshot with a different schema. An application that uses Apache Flink version 1.6 cannot be restored.

If you need to resume an application that is incompatible with existing savepoint data, we recommend that you skip restoring from the snapshot by setting the ApplicationRestoreType parameter of the StartApplication action to SKIP_RESTORE_FROM_SNAPSHOT.

For more information about how Apache Flink deals with incompatible state data, see State Schema Evolution in the Apache Flink Documentation.

Snapshot API Examples

This section includes example requests for API actions for using snapshots with an application. For information about how to use a JSON file for input for an API action, see Kinesis Data Analytics API Example Code.

Enable Snapshots for an Application

The following example request for the UpdateApplication action enables snapshots for an application:

{ "ApplicationName": "MyApplication", "CurrentApplicationVersionId": 1, "ApplicationConfigurationUpdate": { "ApplicationSnapshotConfigurationUpdate": { "SnapshotsEnabledUpdate": "true" } } }

Create a Snapshot

The following example request for the CreateApplicationSnapshot action creates a snapshot of the current application state:

{ "ApplicationName": "MyApplication", "SnapshotName": "MyCustomSnapshot" }

List Snapshots for an Application

The following example request for the ListApplicationSnapshots action lists the first 50 snapshots for the current application state:

{ "ApplicationName": "MyApplication", "Limit": 50 }

List Details for an Application Snapshot

The following example request for the DescribeApplicationSnapshot action lists details for a specific application snapshot:

{ "ApplicationName": "MyApplication", "SnapshotName": "MyCustomSnapshot" }

Delete a Snapshot

The following example request for the DeleteApplicationSnapshot action deletes a previously saved snapshot. You can get the SnapshotCreationTimestamp value using either ListApplicationSnapshots or DeleteApplicationSnapshot:

{ "ApplicationName": "MyApplication", "SnapshotName": "MyCustomSnapshot", "SnapshotCreationTimestamp": 12345678901.0, }

Restart an Application Using a Named Snapshot

The following example request for the StartApplication action starts the application using the saved state from a specific snapshot:

{ "ApplicationName": "MyApplication", "RunConfiguration": { "ApplicationRestoreConfiguration": { "ApplicationRestoreType": "RESTORE_FROM_CUSTOM_SNAPSHOT", "SnapshotName": "MyCustomSnapshot" } } }

Restart an Application Using the Most Recent Snapshot

The following example request for the StartApplication action starts the application using the most recent snapshot:

{ "ApplicationName": "MyApplication", "RunConfiguration": { "ApplicationRestoreConfiguration": { "ApplicationRestoreType": "RESTORE_FROM_LATEST_SNAPSHOT" } } }

Restart an Application Using No Snapshot

The following example request for the StartApplication action starts the application without loading application state, even if a snapshot is present:

{ "ApplicationName": "MyApplication", "RunConfiguration": { "ApplicationRestoreConfiguration": { "ApplicationRestoreType": "SKIP_RESTORE_FROM_SNAPSHOT" } } }