Journal export output in QLDB
An Amazon QLDB journal export job writes two manifest files in addition to the Amazon Ion data objects containing your journal blocks. These are all saved in the Amazon Simple Storage Service (Amazon S3) bucket that you provided in your export request. The following sections describe the format and contents of each output object.
Topics
Manifest files
Amazon QLDB creates two manifest files in the provided S3 bucket for each export request. The initial manifest file is created as soon as you submit the export request. The final manifest file is written after the export is complete. You can use these files to check the status of your export jobs in Amazon S3.
Initial manifest
The initial manifest indicates that your export job has started. It contains
the input parameters that you passed to the request. In addition to the Amazon S3
destination and the start and end time parameters for the export, this file also
contains an exportId
. The exportId
is a unique ID that
QLDB assigns to each export job.
The file-naming convention is as follows.
s3://DOC-EXAMPLE-BUCKET
/prefix
/exportId
.started.manifest
The following is an example of an initial manifest file and its contents.
s3://DOC-EXAMPLE-BUCKET
/journalExport/8UyXulxccYLAsbN1aon7e4.started.manifest
{
ledgerName:"my-ledger",
exportId:"8UyXulxccYLAsbN1aon7e4",
inclusiveStartTime:2019-04-15T00:00:00.000Z,
exclusiveEndTime:2019-04-15T22:00:00.000Z,
bucket:"DOC-EXAMPLE-BUCKET
",
prefix:"journalExport",
objectEncryptionType:"NO_ENCRYPTION"
}
Final manifest
The final manifest indicates that your export job for a particular journal strand has completed. The export job writes a separate final manifest file for each strand.
In Amazon QLDB, a strand is a partition of your ledger's journal. QLDB currently supports journals with a single strand only.
The final manifest includes an ordered list of data object keys that were written during the export. The file naming convention is as follows.
s3://DOC-EXAMPLE-BUCKET
/prefix
/exportId
.strandId
.completed.manifest
The strandId
is a unique ID that QLDB assigns to the strand.
The following is an example of a final manifest file and its contents.
s3://DOC-EXAMPLE-BUCKET
/journalExport/8UyXulxccYLAsbN1aon7e4.JdxjkR9bSYB5jMHWcI464T.completed.manifest
{
keys:[
"2019/04/15/22/JdxjkR9bSYB5jMHWcI464T.1-4.ion",
"2019/04/15/22/JdxjkR9bSYB5jMHWcI464T.5-10.ion",
"2019/04/15/22/JdxjkR9bSYB5jMHWcI464T.11-12.ion",
"2019/04/15/22/JdxjkR9bSYB5jMHWcI464T.13-20.ion",
"2019/04/15/22/JdxjkR9bSYB5jMHWcI464T.21-21.ion"
]
}
Ion data objects
Amazon QLDB writes journal data objects in the text representation of the Amazon
Ion format (.ion
) in the provided Amazon S3 bucket.
Data object names
A journal export job writes these data objects with the following naming convention.
s3://DOC-EXAMPLE-BUCKET
/prefix
/yyyy/mm/dd/hh/strandId
.startSn
-endSn
.ion
-
The output data of each export job is broken up into chunks.
-
yyyy/mm/dd/hh
—The date and time when you submitted the export request. Objects that are exported within the same hour are grouped under the same Amazon S3 prefix. -
strandId
—The unique ID of the particular strand that contains the journal block being exported. -
startSn-endSn
—The sequence number range included in the object. Sequence number specifies the location of a block within a strand.
For example, suppose that you specify the following path.
s3://DOC-EXAMPLE-BUCKET
/journalExport/
Your export job creates an Amazon S3 data object that looks similar to the following.
s3://DOC-EXAMPLE-BUCKET
/journalExport/2019/04/15/22/JdxjkR9bSYB5jMHWcI464T.1-5.ion
Data object contents
Each Ion data object contains journal block objects with the following format.
{
blockAddress: {
strandId: String,
sequenceNo: Int
},
transactionId: String,
blockTimestamp: Datetime,
blockHash: SHA256,
entriesHash: SHA256,
previousBlockHash: SHA256,
entriesHashList: [ SHA256 ],
transactionInfo: {
statements: [
{
//PartiQL statement object
}
],
documents: {
//document-table-statement mapping object
}
},
revisions: [
{
//document revision object
}
]
}
A block is an object that is committed to the journal during a transaction. A block contains transaction metadata along with entries that represent the document revisions that were committed in the transaction and the PartiQL statements that committed them.
The following is an example of a block with sample data. For information about the fields in a block object, see Journal contents in Amazon QLDB.
This block example is for informational purposes only. The hashes shown are not real calculated hash values.
{
blockAddress:{
strandId:"JdxjkR9bSYB5jMHWcI464T",
sequenceNo:1234
},
transactionId:"D35qctdJRU1L1N2VhxbwSn",
blockTimestamp:2019-10-25T17:20:21.009Z,
blockHash:{{WYLOfZClk0lYWT3lUsSr0ONXh+Pw8MxxB+9zvTgSvlQ=}},
entriesHash:{{xN9X96atkMvhvF3nEy6jMSVQzKjHJfz1H3bsNeg8GMA=}},
previousBlockHash:{{IAfZ0h22ZjvcuHPSBCDy/6XNQTsqEmeY3GW0gBae8mg=}},
entriesHashList:[
{{F7rQIKCNn0vXVWPexilGfJn5+MCrtsSQqqVdlQxXpS4=}},
{{C+L8gRhkzVcxt3qRJpw8w6hVEqA5A6ImGne+E7iHizo=}}
],
transactionInfo:{
statements:[
{
statement:"CREATE TABLE VehicleRegistration",
startTime:2019-10-25T17:20:20.496Z,
statementDigest:{{3jeSdejOgp6spJ8huZxDRUtp2fRXRqpOMtG43V0nXg8=}}
},
{
statement:"CREATE INDEX ON VehicleRegistration (VIN)",
startTime:2019-10-25T17:20:20.549Z,
statementDigest:{{099D+5ZWDgA7r+aWeNUrWhc8ebBTXjgscq+mZ2dVibI=}}
},
{
statement:"CREATE INDEX ON VehicleRegistration (LicensePlateNumber)",
startTime:2019-10-25T17:20:20.560Z,
statementDigest:{{B73tVJzVyVXicnH4n96NzU2L2JFY8e9Tjg895suWMew=}}
},
{
statement:"INSERT INTO VehicleRegistration ?",
startTime:2019-10-25T17:20:20.595Z,
statementDigest:{{ggpon5qCXLo95K578YVhAD8ix0A0M5CcBx/W40Ey/Tk=}}
}
],
documents:{
'8F0TPCmdNQ6JTRpiLj2TmW':{
tableName:"VehicleRegistration",
tableId:"BPxNiDQXCIB5l5F68KZoOz",
statements:[3]
}
}
},
revisions:[
{
hash:{{FR1IWcWew0yw1TnRklo2YMF/qtwb7ohsu5FD8A4DSVg=}}
},
{
blockAddress:{
strandId:"JdxjkR9bSYB5jMHWcI464T",
sequenceNo:1234
},
hash:{{t8Hj6/VC4SBitxnvBqJbOmrGytF2XAA/1c0AoSq2NQY=}},
data:{
VIN:"1N4AL11D75C109151",
LicensePlateNumber:"LEWISR261LL",
State:"WA",
City:"Seattle",
PendingPenaltyTicketAmount:90.25,
ValidFromDate:2017-08-21,
ValidToDate:2020-05-11,
Owners:{
PrimaryOwner:{
PersonId:"GddsXfIYfDlKCEprOLOwYt"
},
SecondaryOwners:[]
}
},
metadata:{
id:"8F0TPCmdNQ6JTRpiLj2TmW",
version:0,
txTime:2019-10-25T17:20:20.618Z,
txId:"D35qctdJRU1L1N2VhxbwSn"
}
}
]
}
In the revisions
field, some revision objects might only contain
a hash
value and no other attributes. These are internal-only
system revisions that don't contain user data. An export job includes these
revisions in their respective blocks because the hashes of these revisions are
part of the journal's full hash chain. The full hash chain is required for
cryptographic verification.