QLDB stream records in Kinesis

Important

End of support notice: Existing customers will be able to use Amazon QLDB until end of support on 07/31/2025. For more details, see Migrate an Amazon QLDB Ledger to Amazon Aurora PostgreSQL.

An Amazon QLDB stream writes three types of data records to a given Amazon Kinesis Data Streams resource: control, block summary, and revision details. All three record types are written in the binary representation of the Amazon Ion format.

Control records indicate the start and completion of your QLDB streams. Whenever a revision is committed to your journal, a QLDB stream writes all of the associated journal block data in block summary and revision details records.

The three record types are polymorphic. They all consist of a common top-level record that contains the QLDB stream ARN, the record type, and the record payload. This top-level record has the following format.

{
  qldbStreamArn: string,
  recordType: string,
  payload: {
    //control | block summary | revision details record
  }
}

The recordType field can have one of three values:

• CONTROL

• BLOCK_SUMMARY

• REVISION_DETAILS

The following sections describe the format and contents of each individual payload record.

Note

QLDB writes all stream records to Kinesis Data Streams in the binary representation of Amazon Ion. The following examples are provided in the text representation of Ion to illustrate the record contents in a readable format.

Control records

A QLDB stream writes control records to indicate its start and completion events. The following are examples of control records with sample data for each controlRecordType:

• CREATED – The first record that a QLDB stream writes to Kinesis to indicate that your newly created stream is active.

  {
    qldbStreamArn:"arn:aws:qldb:us-east-1:123456789012:stream/exampleLedger/IiPT4brpZCqCq3f4MTHbYy",
    recordType:"CONTROL",
    payload:{
      controlRecordType:"CREATED"
    }
  }

• COMPLETED – The last record that a QLDB stream writes to Kinesis to indicate that your stream has reached the specified end date and time. This record is not written if you cancel the stream.

  {
    qldbStreamArn:"arn:aws:qldb:us-east-1:123456789012:stream/exampleLedger/IiPT4brpZCqCq3f4MTHbYy",
    recordType:"CONTROL",
    payload:{
      controlRecordType:"COMPLETED"
    }
  }

Block summary records

A block summary record represents a journal block in which your document revisions are committed. A block is an object that is committed to your QLDB journal during a transaction.

The payload of a block summary record contains the block address, timestamp, and other metadata of the transaction that committed the block. It also includes summary attributes of the revisions in the block and the PartiQL statements that committed them. The following is an example of a block summary record with sample data.

Note

This block summary example is provided for informational purposes only. The hashes shown aren't real calculated hash values.

{
  qldbStreamArn:"arn:aws:qldb:us-east-1:123456789012:stream/exampleLedger/IiPT4brpZCqCq3f4MTHbYy",
  recordType:"BLOCK_SUMMARY",
  payload:{
    blockAddress:{
      strandId:"ElYL30RGoqrFCbbaQn3K6m",
      sequenceNo:60807
    },
    transactionId:"9RWohCo7My4GGkxRETAJ6M",
    blockTimestamp:2019-09-18T17:00:14.601000001Z,
    blockHash:{{6Pk9KDYJd38ci09oaHxx0D2grtgh4QBBqbDS6i9quX8=}},
    entriesHash:{{r5YoH6+NXDXxgoRzPREGAWJfn73KlZE0eTfbTxZWUDU=}},
    previousBlockHash:{{K3ti0Agk7DEponywKcQCPRYVHb5RuyxdmQFTfrloptA=}},
    entriesHashList:[
      {{pbzvz6ofJC7mD2jvgfyrY/VtR01zIZHoWy8T1VcxlGo=}},
      {{k2brC23DLMercmiOWHiURaGwHu0mQtLzdNPuviE2rcs=}},
      {{hvw1EV8k4oOkIO36kblO/+UUSFUQqCanKuDGr0aP9nQ=}},
      {{ZrLbkyzDcpJ9KWsZMZqRuKUkG/czLIJ4US+K5E31b+Q=}}
    ],
    transactionInfo:{
      statements:[
        {
          statement:"SELECT * FROM Person WHERE GovId = ?",
          startTime:2019-09-18T17:00:14.587Z,
          statementDigest:{{p4Dn0DiuYD3Xm9UQQ75YLwmoMbSfJmopOmTfMnXs26M=}}
        },
        {
          statement:"INSERT INTO Person ?",
          startTime:2019-09-18T17:00:14.594Z,
          statementDigest:{{klMLkLfa5VJqk6JUPtHkQpOsDdG4HmuUaq/VaApQflU=}}
        },
        {
          statement:"INSERT INTO VehicleRegistration ?",
          startTime:2019-09-18T17:00:14.598Z,
          statementDigest:{{B0gO9BWVNrzRYFoe7t+GVLpJ6uZcLKf5t/chkfRhspI=}}
        }
      ],
      documents:{
        '7z2OpEBgVCvCtwvx4a2JGn':{
          tableName:"Person",
          tableId:"LSkFkQvkIOjCmpTZpkfpn9",
          statements:[1]
        },
        'K0FpsSLpydLDr7hi6KUzqk':{
          tableName:"VehicleRegistration",
          tableId:"Ad3A07z0ZffC7Gpso7BXyO",
          statements:[2]
        }
      }
    },
    revisionSummaries:[
      {
        hash:{{uDthuiqSy4FwjZssyCiyFd90XoPSlIwomHBdF/OrmkE=}},
        documentId:"7z2OpEBgVCvCtwvx4a2JGn"
      },
      {
        hash:{{qJID/amu0gN3dpG5Tg0FfIFTh/U5yFkfT+g/O6k5sPM=}},
        documentId:"K0FpsSLpydLDr7hi6KUzqk"
      }
    ]
  }
}

In the revisionSummaries field, some revisions might not have a documentId. These are internal-only system revisions that don't contain user data. A QLDB stream includes these revisions in their respective block summary records because the hashes of these revisions are part of the journal's full hash chain. The full hash chain is required for cryptographic verification.

Only the revisions that do have a document ID are published in separate revision details records, as described in the following section.

Revision details records

A revision details record represents a document revision that is committed to your journal. The payload contains all of the attributes from the committed view of the revision, along with the associated table name and table ID. The following is an example of a revision record with sample data.

{
  qldbStreamArn:"arn:aws:qldb:us-east-1:123456789012:stream/exampleLedger/IiPT4brpZCqCq3f4MTHbYy",
  recordType:"REVISION_DETAILS",
  payload:{
    tableInfo:{
      tableName:"VehicleRegistration",
      tableId:"Ad3A07z0ZffC7Gpso7BXyO"
    },
    revision:{
      blockAddress:{
        strandId:"ElYL30RGoqrFCbbaQn3K6m",
        sequenceNo:60807
      },
      hash:{{qJID/amu0gN3dpG5Tg0FfIFTh/U5yFkfT+g/O6k5sPM=}},
      data:{
        VIN:"1N4AL11D75C109151",
        LicensePlateNumber:"LEWISR261LL",
        State:"WA",
        City:"Seattle",
        PendingPenaltyTicketAmount:90.25,
        ValidFromDate:2017-08-21,
        ValidToDate:2020-05-11,
        Owners:{
          PrimaryOwner:{PersonId:"7z2OpEBgVCvCtwvx4a2JGn"},
          SecondaryOwners:[]
        }
      },
      metadata:{
        id:"K0FpsSLpydLDr7hi6KUzqk",
        version:0,
        txTime:2019-09-18T17:00:14.602Z,
        txId:"9RWohCo7My4GGkxRETAJ6M"
      }
    }
  }
}

Handling duplicate and out-of-order records

QLDB streams can publish duplicate and out-of-order records to Kinesis Data Streams. So, a consumer application might need to implement its own logic to identify and handle such scenarios. The block summary and revision details records include fields that you can use for this purpose. Combined with the features of downstream services, these fields can indicate both a unique identity and a strict order for the records.

For example, consider a stream that integrates QLDB with an OpenSearch index to provide full text search capabilities over documents. In this use case, you need to avoid indexing stale (out-of-order) revisions of a document. To enforce ordering and deduplication, you can use the document ID and external version fields in OpenSearch, along with the document ID and version fields in a revision details record.

For an example of deduplication logic in a sample application that integrates QLDB with Amazon OpenSearch Service, see the GitHub repository aws-samples/amazon-qldb-streaming-amazon-opensearch-service-sample-python.