Amazon Kinesis Streams
Developer Guide

Integration With Producer Code

The KPL runs in a separate process, and communicates with your parent user process using IPC. This architecture is sometimes called a microservice, and is chosen for two main reasons:

1) Your user process will not crash even if the KPL crashes

Your process could have tasks unrelated to Streams, and may be able to continue operation even if the KPL crashes. It is also possible for your parent user process to restart the KPL and recover to a fully working state (this functionality is in the official wrappers).

An example is a web server that sends metrics to Streams; the server can continue serving pages even if the Streams part has stopped working. Crashing the whole server because of a bug in the KPL would therefore cause an unnecessary outage.

2) Arbitrary clients can be supported

There are always customers who use languages other than the ones officially supported. These customers should also be able to use the KPL easily.

Recommended Usage Matrix

The following usage matrix enumerates the recommended settings for different users and advises you about whether and how you should use the KPL. Keep in mind that if aggregation is enabled, de-aggregation must also be used to extract your records on the consumer side.

Producer side language

Consumer side


KCL Version Checkpoint logic Can you use the KPL? Caveats
Anything but Java * * * No N/A
Java Java Uses Java SDK directly N/A Yes If aggregation is used, you have to use the provided de-aggregation library after GetRecords calls.
Java Anything but Java Uses SDK directly N/A Yes Must disable aggregation.
Java Java 1.3.x N/A Yes Must disable aggregation
Java Java 1.4.x Calls checkpoint without any arguments Yes None
Java Java 1.4.x Calls checkpoint with an explicit sequence number Yes Either disable aggregation, or change the code to use extended sequence numbers for checkpointing
Java Anything but Java 1.3.x + Multilanguage daemon + language-specific wrapper N/A Yes Must disable aggregation