Creating and managing streams in QLDB

Amazon QLDB provides API operations to create and manage a stream of journal data from your ledger to Amazon Kinesis Data Streams. The QLDB stream captures every document revision that is committed to your journal and sends it to a Kinesis data stream.

You can use the AWS Management Console, an AWS SDK, or the AWS Command Line Interface (AWS CLI) to create a journal stream. In addition, you can also use an AWS CloudFormation template to create streams. For more information, see the AWS::QLDB::Stream resource in the AWS CloudFormation User Guide.

Stream parameters

To create a QLDB journal stream, you must provide the following configuration parameters:

Ledger name

The QLDB ledger whose journal data you want to stream to Kinesis Data Streams.

Stream name

The name that you want to assign to the QLDB journal stream. User-defined names can help identify and indicate the purpose of a stream.

Your stream name must be unique among other active streams for a given ledger. Stream names have the same naming constraints as ledger names, as defined in Quotas and limits in Amazon QLDB.

In addition to the stream name, QLDB assigns a stream ID to each QLDB stream that you create. The stream ID is unique among all streams for a given ledger, regardless of their status.

Start date and time

The date and time from which to start streaming journal data. This value can be any date and time in the past but can't be in the future.

End date and time

(Optional) The date and time that specifies when the stream ends.

If you create an indefinite stream with no end time, you must manually cancel it to end the stream. You can also cancel an active, finite stream that has not yet reached its specified end date and time.

Destination Kinesis data stream

The Kinesis Data Streams target resource to which your stream writes the data records. To learn how to create a Kinesis data stream, see Creating and updating data streams in the Amazon Kinesis Data Streams Developer Guide.

Important

• Cross-Region and cross-account streams are not supported. The specified Kinesis data stream must be in the same AWS Region and account as your ledger.

• Record aggregation in Kinesis Data Streams is enabled by default. This option lets QLDB publish multiple data records in a single Kinesis Data Streams record, increasing the number of records sent per API call.

  Record aggregation has important implications for processing records and requires de-aggregation in your stream consumer. To learn more, see KPL key concepts and Consumer de-aggregation in the Amazon Kinesis Data Streams Developer Guide.

IAM role

The IAM role that allows QLDB to assume write permissions to your Kinesis data stream. You can use the QLDB console to automatically create this role, or you can manually create it in IAM. To learn how to manually create it, see Stream permissions.

To pass a role to QLDB when requesting a journal stream, you must have permissions to perform the iam:PassRole action on the IAM role resource.

Stream ARN

Every QLDB journal stream is a subresource of a ledger and is uniquely identified by an Amazon Resource Name (ARN). The following is an example ARN of a QLDB stream with a stream ID of IiPT4brpZCqCq3f4MTHbYy for a ledger named exampleLedger.

arn:aws:qldb:us-east-1:123456789012:stream/exampleLedger/IiPT4brpZCqCq3f4MTHbYy

The following section describes how to create and cancel a QLDB stream using the AWS Management Console.

AWS Management Console

Follow these steps to create or cancel a QLDB stream using the QLDB console.

To create a stream (console)

1. Sign in to the AWS Management Console, and open the Amazon QLDB console at https://console.aws.amazon.com/qldb.

2. In the navigation pane, choose Streams.

3. Choose Create QLDB stream.

4. On the Create QLDB stream page, enter the following settings:

   • Stream name – The name that you want to assign to the QLDB stream.

   • Ledger – The ledger whose journal data you want to stream.

   • Start date and time – The inclusive timestamp in Coordinated Universal Time (UTC) from which to start streaming journal data. This timestamp defaults to the current date and time. It can't be in the future and must be earlier than the End date and time.

   • End date and time – (Optional) The exclusive timestamp (UTC) that specifies when the stream ends. If you keep this parameter blank, the stream runs indefinitely until you cancel it.

   • Destination stream – The Kinesis Data Streams target resource to which your stream writes the data records. Use the following ARN format.

     arn:aws:kinesis:aws-region:account-id:stream/kinesis-stream-name

     The following is an example.

     arn:aws:kinesis:us-east-1:123456789012:stream/stream-for-qldb

     Cross-Region and cross-account streams are not supported. The specified Kinesis data stream must be in the same AWS Region and account as your ledger.

   • Enable record aggregation in Kinesis Data Streams – (Enabled by default) Lets QLDB publish multiple data records in a single Kinesis Data Streams record, increasing the number of records sent per API call.

   • Service access – The IAM role that grants QLDB write permissions to your Kinesis data stream.

     To pass a role to QLDB when requesting a journal stream, you must have permissions to perform the iam:PassRole action on the IAM role resource.

     • Create and use a new service role – Let the console create a new role for you with the required permissions for the specified Kinesis data stream.

     • Use an existing service role – To learn how to manually create this role in IAM, see Stream permissions.

   • Tags – (Optional) Add metadata to the stream by attaching tags as key-value pairs. You can add tags to your stream to help organize and identify them. For more information, see Tagging Amazon QLDB resources.

     Choose Add tag, and then enter any key-value pairs as appropriate.

5. When the settings are as you want them, choose Create QLDB stream.

   If your request submission is successful, the console returns to the main Streams page and lists your QLDB streams with their current status.

6. After your stream is active, use Kinesis to process your stream data with a consumer application.

   Open the Kinesis Data Streams console at https://console.aws.amazon.com/kinesis/.

For information about the format of the stream data records, see QLDB stream records in Kinesis.

To learn how to handle streams that result in an error, see Handling impaired streams.

To cancel a stream (console)

You can't restart a QLDB stream after you cancel it. To resume delivery of your data to Kinesis Data Streams, you can create a new QLDB stream.

1. Open the Amazon QLDB console at https://console.aws.amazon.com/qldb.

2. In the navigation pane, choose Streams.

3. In the list of QLDB streams, select the active stream that you want to cancel.

4. Choose Cancel stream. Confirm this by entering cancel stream in the box provided.

For information about using the QLDB API with an AWS SDK or the AWS CLI to create and manage journal streams, see Developing with streams in QLDB.

Stream states

The status of a QLDB stream can be one of the following:

• ACTIVE – Is currently streaming or waiting to stream data (for an indefinite stream with no end time).

• COMPLETED – Has successfully finished streaming all journal blocks within the specified time range. This is a terminal state.

• CANCELED – Was ended by a user request before the specified end time and is no longer actively streaming data. This is a terminal state.

• IMPAIRED – Is unable to write records to Kinesis due to an error that requires your action. This is a recoverable, non-terminal state.

  If you resolve the error within one hour, the stream automatically moves to ACTIVE state. If the error remains unresolved after one hour, the stream automatically moves to FAILED state.

• FAILED – Is unable to write records to Kinesis due to an error and is in an unrecoverable, terminal state.

The following diagram illustrates how a QLDB stream resource can transition between states.

Expiration for terminal streams

Stream resources that are in a terminal state (CANCELED, COMPLETED, and FAILED) are subject to a 7-day retention period. They're automatically hard-deleted after this limit expires.

After a terminal stream is deleted, you can no longer use the QLDB console or the QLDB API to describe or list the stream resource.

Handling impaired streams

If your stream encounters an error, it moves to IMPAIRED state first. QLDB continues to retry IMPAIRED streams for up to one hour.

If you resolve the error within one hour, the stream automatically moves to ACTIVE state. If the error remains unresolved after one hour, the stream automatically moves to FAILED state.

An impaired or failed stream can have one of the following error causes:

• KINESIS_STREAM_NOT_FOUND – The destination Kinesis Data Streams resource doesn't exist. Verify that the Kinesis data stream that you provided in your QLDB stream request is correct. Then, go to Kinesis and create the data stream that you specified.

• IAM_PERMISSION_REVOKED – QLDB doesn't have enough permissions to write data records to your specified Kinesis data stream. Verify that you define a policy for your specified Kinesis data stream that grants the QLDB service (qldb.amazonaws.com) permissions to the following actions:

  • kinesis:PutRecord

  • kinesis:PutRecords

  • kinesis:DescribeStream

  • kinesis:ListShards

Monitoring impaired streams

If a stream becomes impaired, the QLDB console displays a banner that shows details about the stream and the error that it encountered. You can also use the DescribeJournalKinesisStream API operation to get a stream's status and the underlying error cause.

In addition, you can use Amazon CloudWatch to create an alarm that monitors the IsImpaired metric of a stream. For information about monitoring QLDB metrics with CloudWatch, see Amazon QLDB dimensions and metrics.