Journal export output in QLDB - Amazon Quantum Ledger Database (Amazon QLDB)
Manifest filesIon data objects

Journal export output in QLDB

An Amazon QLDB journal export job writes two manifest files in addition to the Amazon Ion data objects containing your journal blocks. These are all saved in the Amazon Simple Storage Service (Amazon S3) bucket that you provided in your export request. The following sections describe the format and contents of each output object.

Manifest files

Amazon QLDB creates two manifest files in the provided S3 bucket for each export request. The initial manifest file is created as soon as you submit the export request. The final manifest file is written after the export is complete. You can use these files to check the status of your export jobs in Amazon S3.

Initial manifest

The initial manifest indicates that your export job has started. It contains the input parameters that you passed to the request. In addition to the Amazon S3 destination and the start and end time parameters for the export, this file also contains an exportId. The exportId is a unique ID that QLDB assigns to each export job.

The file-naming convention is as follows.

s3://DOC-EXAMPLE-BUCKET/prefix/exportId.started.manifest

The following is an example of an initial manifest file and its contents. 

s3://DOC-EXAMPLE-BUCKET/journalExport/8UyXulxccYLAsbN1aon7e4.started.manifest
{
  ledgerName:"my-ledger",
  exportId:"8UyXulxccYLAsbN1aon7e4",
  inclusiveStartTime:2019-04-15T00:00:00.000Z,
  exclusiveEndTime:2019-04-15T22:00:00.000Z,
  bucket:"DOC-EXAMPLE-BUCKET",
  prefix:"journalExport",
  objectEncryptionType:"NO_ENCRYPTION"
}

Final manifest

The final manifest indicates that your export job for a particular journal strand has completed. The export job writes a separate final manifest file for each strand.

Note

In Amazon QLDB, a strand is a partition of your ledger's journal. QLDB currently supports journals with a single strand only.

The final manifest includes an ordered list of data object keys that were written during the export. The file naming convention is as follows. 

s3://DOC-EXAMPLE-BUCKET/prefix/exportId.strandId.completed.manifest

The strandId is a unique ID that QLDB assigns to the strand. The following is an example of a final manifest file and its contents. 

s3://DOC-EXAMPLE-BUCKET/journalExport/8UyXulxccYLAsbN1aon7e4.JdxjkR9bSYB5jMHWcI464T.completed.manifest
{
  keys:[
    "2019/04/15/22/JdxjkR9bSYB5jMHWcI464T.1-4.ion",
    "2019/04/15/22/JdxjkR9bSYB5jMHWcI464T.5-10.ion",
    "2019/04/15/22/JdxjkR9bSYB5jMHWcI464T.11-12.ion",
    "2019/04/15/22/JdxjkR9bSYB5jMHWcI464T.13-20.ion",
    "2019/04/15/22/JdxjkR9bSYB5jMHWcI464T.21-21.ion"
  ]
}

Ion data objects

Amazon QLDB writes journal data objects in the text representation of the Amazon Ion format (.ion) in the provided Amazon S3 bucket.

Data object names

A journal export job writes these data objects with the following naming convention. 

s3://DOC-EXAMPLE-BUCKET/prefix/yyyy/mm/dd/hh/strandId.startSn-endSn.ion

  • The output data of each export job is broken up into chunks.

  • yyyy/mm/dd/hh—The date and time when you submitted the export request. Objects that are exported within the same hour are grouped under the same Amazon S3 prefix.

  • strandId—The unique ID of the particular strand that contains the journal block being exported.

  • startSn-endSn—The sequence number range included in the object. Sequence number specifies the location of a block within a strand.

For example, suppose that you specify the following path.

s3://DOC-EXAMPLE-BUCKET/journalExport/

Your export job creates an Amazon S3 data object that looks similar to the following. 

s3://DOC-EXAMPLE-BUCKET/journalExport/2019/04/15/22/JdxjkR9bSYB5jMHWcI464T.1-5.ion

Data object contents

Each Ion data object contains journal block objects with the following format. 

{
  blockAddress: {
    strandId: String,
    sequenceNo: Int
  },
  transactionId: String,
  blockTimestamp: Datetime,
  blockHash: SHA256,
  entriesHash: SHA256,
  previousBlockHash: SHA256,
  entriesHashList: [ SHA256 ],
  transactionInfo: {
    statements: [
      {
        //PartiQL statement object
      }
    ],
    documents: {
      //document-table-statement mapping object
    }
  },
  revisions: [
    {
      //document revision object
    }
  ]
}

A block is an object that is committed to the journal during a transaction. A block contains transaction metadata along with entries that represent the document revisions that were committed in the transaction and the PartiQL statements that committed them.

The following is an example of a block with sample data. For information about the fields in a block object, see Journal contents in Amazon QLDB.

Note

This block example is for informational purposes only. The hashes shown are not real calculated hash values. 

{
  blockAddress:{
    strandId:"JdxjkR9bSYB5jMHWcI464T",
    sequenceNo:1234
  },
  transactionId:"D35qctdJRU1L1N2VhxbwSn",
  blockTimestamp:2019-10-25T17:20:21.009Z,
  blockHash:{{WYLOfZClk0lYWT3lUsSr0ONXh+Pw8MxxB+9zvTgSvlQ=}},
  entriesHash:{{xN9X96atkMvhvF3nEy6jMSVQzKjHJfz1H3bsNeg8GMA=}},
  previousBlockHash:{{IAfZ0h22ZjvcuHPSBCDy/6XNQTsqEmeY3GW0gBae8mg=}},
  entriesHashList:[
      {{F7rQIKCNn0vXVWPexilGfJn5+MCrtsSQqqVdlQxXpS4=}},
      {{C+L8gRhkzVcxt3qRJpw8w6hVEqA5A6ImGne+E7iHizo=}}
  ],
  transactionInfo:{
    statements:[
      {
        statement:"CREATE TABLE VehicleRegistration",
        startTime:2019-10-25T17:20:20.496Z,
        statementDigest:{{3jeSdejOgp6spJ8huZxDRUtp2fRXRqpOMtG43V0nXg8=}}
      },
      {
        statement:"CREATE INDEX ON VehicleRegistration (VIN)",
        startTime:2019-10-25T17:20:20.549Z,
        statementDigest:{{099D+5ZWDgA7r+aWeNUrWhc8ebBTXjgscq+mZ2dVibI=}}
      },
      {
        statement:"CREATE INDEX ON VehicleRegistration (LicensePlateNumber)",
        startTime:2019-10-25T17:20:20.560Z,
        statementDigest:{{B73tVJzVyVXicnH4n96NzU2L2JFY8e9Tjg895suWMew=}}
      },
      {
        statement:"INSERT INTO VehicleRegistration ?",
        startTime:2019-10-25T17:20:20.595Z,
        statementDigest:{{ggpon5qCXLo95K578YVhAD8ix0A0M5CcBx/W40Ey/Tk=}}
      }
    ],
    documents:{
      '8F0TPCmdNQ6JTRpiLj2TmW':{
        tableName:"VehicleRegistration",
        tableId:"BPxNiDQXCIB5l5F68KZoOz",
        statements:[3]
      }
    }
  },
  revisions:[
    {
      hash:{{FR1IWcWew0yw1TnRklo2YMF/qtwb7ohsu5FD8A4DSVg=}}
    },
    {
      blockAddress:{
        strandId:"JdxjkR9bSYB5jMHWcI464T",
        sequenceNo:1234
      },
      hash:{{t8Hj6/VC4SBitxnvBqJbOmrGytF2XAA/1c0AoSq2NQY=}},
      data:{
        VIN:"1N4AL11D75C109151",
        LicensePlateNumber:"LEWISR261LL",
        State:"WA",
        City:"Seattle",
        PendingPenaltyTicketAmount:90.25,
        ValidFromDate:2017-08-21,
        ValidToDate:2020-05-11,
        Owners:{
          PrimaryOwner:{
            PersonId:"GddsXfIYfDlKCEprOLOwYt"
          },
          SecondaryOwners:[]
        }
      },
      metadata:{
        id:"8F0TPCmdNQ6JTRpiLj2TmW",
        version:0,
        txTime:2019-10-25T17:20:20.618Z,
        txId:"D35qctdJRU1L1N2VhxbwSn"
      }
    }
  ]
}

In the revisions field, some revision objects might only contain a hash value and no other attributes. These are internal-only system revisions that don't contain user data. An export job includes these revisions in their respective blocks because the hashes of these revisions are part of the journal's full hash chain. The full hash chain is required for cryptographic verification.