Custom implementations of CloudTrail log file integrity validation - AWS CloudTrail

Custom implementations of CloudTrail log file integrity validation

Because CloudTrail uses industry standard, openly available cryptographic algorithms and hash functions, you can create your own tools to validate the integrity of CloudTrail log files. When log file integrity validation is enabled, CloudTrail delivers digest files to your Amazon S3 bucket. You can use these files to implement your own validation solution. For more information about digest files, see CloudTrail digest file structure.

This topic describes how digest files are signed, and then details the steps that you will need to take to implement a solution that validates the digest files and the log files that they reference.

Understanding how CloudTrail digest files are signed

CloudTrail digest files are signed with RSA digital signatures. For each digest file, CloudTrail does the following:

  1. Creates a string for data signing based on designated digest file fields (described in the next section).

  2. Gets a private key unique to the Region.

  3. Passes the SHA-256 hash of the string and the private key to the RSA signing algorithm, which produces a digital signature.

  4. Encodes the byte code of the signature into hexadecimal format.

  5. Puts the digital signature into the x-amz-meta-signature metadata property of the Amazon S3 digest file object.

Contents of the data signing string

The following CloudTrail objects are included in the string for data signing:

  • The ending timestamp of the digest file in UTC extended format (for example, 2015-05-08T07:19:37Z)

  • The current digest file S3 path

  • The hexadecimal-encoded SHA-256 hash of the current digest file

  • The hexadecimal-encoded signature of the previous digest file

The format for calculating this string and an example string are provided later in this document.

Custom validation implementation steps

When implementing a custom validation solution, you will need to validate the digest file first, and then the log files that it references.

Validate the digest file

To validate a digest file, you need its signature, the public key whose private key was used to signed it, and a data signing string that you compute.

  1. Get the digest file.

  2. Verify that the digest file has been retrieved from its original location.

  3. Get the hexadecimal-encoded signature of the digest file.

  4. Get the hexadecimal-encoded fingerprint of the public key whose private key was used to sign the digest file.

  5. Retrieve the public keys for the time range corresponding to the digest file.

  6. From among the public keys retrieved, choose the public key whose fingerprint matches the fingerprint in the digest file.

  7. Using the digest file hash and other digest file fields, recreate the data signing string used to verify the digest file signature.

  8. Validate the signature by passing in the SHA-256 hash of the string, the public key, and the signature as parameters to the RSA signature verification algorithm. If the result is true, the digest file is valid.

Validate the log files

If the digest file is valid, validate each of the log files that the digest file references.

  1. To validate the integrity of a log file, compute its SHA-256 hash value on its uncompressed content and compare the results with the hash for the log file recorded in hexadecimal in the digest. If the hashes match, the log file is valid.

  2. By using the information about the previous digest file that is included in the current digest file, validate the previous digest files and their corresponding log files in succession.

The following sections describe these steps in detail.

A. Get the digest file

The first steps are to get the most recent digest file, verify that you have retrieved it from its original location, verify its digital signature, and get the fingerprint of the public key.

  1. Using S3 Get or the AmazonS3Client class (for example), get the most recent digest file from your Amazon S3 bucket for the time range that you want to validate.

  2. Check that the S3 bucket and S3 object used to retrieve the file match the S3 bucket S3 object locations that are recorded in the digest file itself.

  3. Next, get the digital signature of the digest file from the x-amz-meta-signature metadata property of the digest file object in Amazon S3.

  4. In the digest file, get the fingerprint of the public key whose private key was used to sign the digest file from the digestPublicKeyFingerprint field.

B. Retrieve the public key for validating the digest file

To get the public key to validate the digest file, you can use either the AWS CLI or the CloudTrail API. In both cases, you specify a time range (that is, a start time and end time) for the digest files that you want to validate. One or more public keys may be returned for the time range that you specify. The returned keys may have validity time ranges that overlap.

Note

Because CloudTrail uses different private/public key pairs per Region, each digest file is signed with a private key unique to its Region. Therefore, when you validate a digest file from a particular Region, you must retrieve its public key from the same Region.

Use the AWS CLI to retrieve public keys

To retrieve public keys for digest files by using the AWS CLI, use the cloudtrail list-public-keys command. The command has the following format:

aws cloudtrail list-public-keys [--start-time <start-time>] [--end-time <end-time>]

The start-time and end-time parameters are UTC timestamps and are optional. If not specified, the current time is used, and the currently active public key or keys are returned.

Sample Response

The response will be a list of JSON objects representing the key (or keys) returned:

{ "publicKeyList": [ { "ValidityStartTime": "1436317441.0", "ValidityEndTime": "1438909441.0", "Value": "MIIBCgKCAQEAn11L2YZ9h7onug2ILi1MWyHiMRsTQjfWE+pHVRLk1QjfWhirG+lpOa8NrwQ/r7Ah5bNL6HepznOU9XTDSfmmnP97mqyc7z/upfZdS/AHhYcGaz7n6Wc/RRBU6VmiPCrAUojuSk6/GjvA8iOPFsYDuBtviXarvuLPlrT9kAd4Lb+rFfR5peEgBEkhlzc5HuWO7S0y+KunqxX6jQBnXGMtxmPBPP0FylgWGNdFtks/4YSKcgqwH0YDcawP9GGGDAeCIqPWIXDLG1jOjRRzWfCmD0iJUkz8vTsn4hq/5ZxRFE7UBAUiVcGbdnDdvVfhF9C3dQiDq3k7adQIziLT0cShgQIDAQAB", "Fingerprint": "8eba5db5bea9b640d1c96a77256fe7f2" }, { "ValidityStartTime": "1434589460.0", "ValidityEndTime": "1437181460.0", "Value": "MIIBCgKCAQEApfYL2FiZhpN74LNWVUzhR+VheYhwhYm8w0n5Gf6i95ylW5kBAWKVEmnAQG7BvS5g9SMqFDQx52fW7NWV44IvfJ2xGXT+wT+DgR6ZQ+6yxskQNqV5YcXj4Aa5Zz4jJfsYjDuO2MDTZNIzNvBNzaBJ+r2WIWAJ/Xq54kyF63B6WE38vKuDE7nSd1FqQuEoNBFLPInvgggYe2Ym1Refe2z71wNcJ2kY+q0h1BSHrSM8RWuJIw7MXwF9iQncg9jYzUlNJomozQzAG5wSRfbplcCYNY40xvGd/aAmO0m+Y+XFMrKwtLCwseHPvj843qVno6x4BJN9bpWnoPo9sdsbGoiK3QIDAQAB", "Fingerprint": "8933b39ddc64d26d8e14ffbf6566fee4" }, { "ValidityStartTime": "1434589370.0", "ValidityEndTime": "1437181370.0", "Value": "MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAqlzPJbvZJ42UdcmLfPUqXYNfOs6I8lCfao/tOs8CmzPOEdtLWugB9xoIUz78qVHdKIqxbaG4jWHfJBiOSSFBM0lt8cdVo4TnRa7oG9io5pysS6DJhBBAeXsicufsiFJR+wrUNh8RSLxL4k6G1+BhLX20tJkZ/erT97tDGBujAelqseGg3vPZbTx9SMfOLN65PdLFudLP7Gat0Z9p5jw/rjpclKfo9Bfc3heeBxWGKwBBOKnFAaN9V57pOaosCvPKmHd9bg7jsQkI9Xp22IzGLsTFJZYVA3KiTAElDMu80iFXPHEq9hKNbt9e4URFam+1utKVEiLkR2disdCmPTK0VQIDAQAB", "Fingerprint": "31e8b5433410dfb61a9dc45cc65b22ff" } ] }

Use the CloudTrail API to retrieve public keys

To retrieve public keys for digest files by using the CloudTrail API, pass in start time and end time values to the ListPublicKeys API. The ListPublicKeys API returns the public keys whose private keys were used to sign digest files within the specified time range. For each public key, the API also returns the corresponding fingerprint.

ListPublicKeys

This section describes the request parameters and response elements for the ListPublicKeys API.

Note

The encoding for the binary fields for ListPublicKeys is subject to change.

Request Parameters

Name Description
StartTime

Optionally specifies, in UTC, the start of the time range to look up public keys for CloudTrail digest files. If StartTime is not specified, the current time is used, and the current public key is returned.

Type: DateTime

EndTime

Optionally specifies, in UTC, the end of the time range to look up public keys for CloudTrail digest files. If EndTime is not specified, the current time is used.

Type: DateTime

Response Elements

PublicKeyList, an array of PublicKey objects that contains:

Name Description
Value

The DER encoded public key value in PKCS #1 format.

Type: Blob

ValidityStartTime

The starting time of validity of the public key.

Type: DateTime

ValidityEndTime

The ending time of validity of the public key.

Type: DateTime

Fingerprint

The fingerprint of the public key. The fingerprint can be used to identify the public key that you must use to validate the digest file.

Type: String

C. Choose the public key to use for validation

From among the public keys retrieved by list-public-keys or ListPublicKeys, choose the public key returned whose fingerprint matches the fingerprint recorded in the digestPublicKeyFingerprint field of the digest file. This is the public key that you will use to validate the digest file.

D. Recreate the data signing string

Now that you have the signature of the digest file and associated public key, you need to calculate the data signing string. After you have calculated the data signing string, you will have the inputs needed to verify the signature.

The data signing string has the following format:

Data_To_Sign_String = Digest_End_Timestamp_in_UTC_Extended_format + '\n' + Current_Digest_File_S3_Path + '\n' + Hex(Sha256(current-digest-file-content)) + '\n' + Previous_digest_signature_in_hex

An example Data_To_Sign_String follows.

2015-08-12T04:01:31Z S3-bucket-name/AWSLogs/111122223333/CloudTrail-Digest/us-east-2/2015/08/12/111122223333_us-east-2_CloudTrail-Digest_us-east-2_20150812T040131Z.json.gz 4ff08d7c6ecd6eb313257e839645d20363ee3784a2328a7d76b99b53cc9bcacd 6e8540b83c3ac86a0312d971a225361d28ed0af20d70c211a2d405e32abf529a8145c2966e3bb47362383a52441545ed091fb81 d4c7c09dd152b84e79099ce7a9ec35d2b264eb92eb6e090f1e5ec5d40ec8a0729c02ff57f9e30d5343a8591638f8b794972ce15bb3063a01972 98b0aee2c1c8af74ec620261529265e83a9834ebef6054979d3e9a6767dfa6fdb4ae153436c567d6ae208f988047ccfc8e5e41f7d0121e54ed66b1b904f80fb2ce304458a2a6b91685b699434b946c52589e9438f8ebe5a0d80522b2f043b3710b87d2cda43e5c1e0db921d8d540b9ad5f6d4$31b1f4a8ef2d758424329583897339493a082bb36e782143ee5464b4e3eb4ef6

After you recreate this string, you can validate the digest file.

E. Validate the digest file

Pass the SHA-256 hash of the recreated data signing string, digital signature, and public key to the RSA signature verification algorithm. If the output is true, the signature of the digest file is verified and the digest file is valid.

F. Validate the log files

After you have validated the digest file, you can validate the log files it references. The digest file contains the SHA-256 hashes of the log files. If one of the log files was modified after CloudTrail delivered it, the SHA-256 hashes will change, and the signature of digest file will not match.

The following shows how validate the log files:

  1. Do an S3 Get of the log file using the S3 location information in the digest file's logFiles.s3Bucket and logFiles.s3Object fields.

  2. If the S3 Get operation is successful, iterate through the log files listed in the digest file's logFiles array using the following steps:

    1. Retrieve the original hash of the file from the logFiles.hashValue field of the corresponding log in the digest file.

    2. Hash the uncompressed contents of the log file with the hashing algorithm specified in logFiles.hashAlgorithm.

    3. Compare the hash value that you generated with the one for the log in the digest file. If the hashes match, the log file is valid.

G. Validate additional digest and log files

In each digest file, the following fields provide the location and signature of the previous digest file:

  • previousDigestS3Bucket

  • previousDigestS3Object

  • previousDigestSignature

Use this information to visit previous digest files sequentially, validating the signature of each and the log files that they reference by using the steps in the previous sections. The only difference is that for previous digest files, you do not need to retrieve the digital signature from the digest file object's Amazon S3 metadata properties. The signature for the previous digest file is provided for you in the previousDigestSignature field.

You can go back until the starting digest file is reached, or until the chain of digest files is broken, whichever comes first.

Validating digest and log files offline

When validating digest and log files offline, you can generally follow the procedures described in the previous sections. However, you must take into account the following areas:

Handling the most recent digest file

The digital signature of the most recent (that is, "current") digest file is in the Amazon S3 metadata properties of the digest file object. In an offline scenario, the digital signature for the current digest file will not be available.

Two possible ways of handling this are:

  • Since the digital signature for the previous digest file is in the current digest file, start validating from the next-to-last digest file. With this method, the most recent digest file cannot be validated.

  • As a preliminary step, obtain the signature for the current digest file from the digest file object's metadata properties and then store it securely offline. This would allow the current digest file to be validated in addition to the previous files in the chain.

Path resolution

Fields in the downloaded digest files like s3Object and previousDigestS3Object will still be pointing to Amazon S3 online locations for log files and digest files. An offline solution must find a way to reroute these to the current path of the downloaded log and digest files.

Public keys

In order to validate offline, all of the public keys that you need for validating log files in a given time range must first be obtained online (by calling ListPublicKeys, for example) and then stored securely offline. This step must be repeated whenever you want to validate additional files outside the initial time range that you specified.

Sample validation snippet

The following sample snippet provides skeleton code for validating CloudTrail digest and log files. The skeleton code is online/offline agnostic; that is, it is up to you to decide whether to implement it with or without online connectivity to AWS. The suggested implementation uses the Java Cryptography Extension (JCE) and Bouncy Castle as a security provider.

The sample snippet shows:

  • How to create the data signing string used to validate the digest file signature.

  • How to verify the digest file signature.

  • How to verify the log file hashes.

  • A code structure for validating a chain of digest files.

import java.util.Arrays; import java.security.MessageDigest; import java.security.KeyFactory; import java.security.PublicKey; import java.security.Security; import java.security.Signature; import java.security.spec.X509EncodedKeySpec; import org.json.JSONObject; import org.bouncycastle.jce.provider.BouncyCastleProvider; import org.apache.commons.codec.binary.Hex; public class DigestFileValidator { public void validateDigestFile(String digestS3Bucket, String digestS3Object, String digestSignature) { // Using the Bouncy Castle provider as a JCE security provider - http://www.bouncycastle.org/ Security.addProvider(new BouncyCastleProvider()); // Load the digest file from S3 (using Amazon S3 Client) or from your local copy JSONObject digestFile = loadDigestFileInMemory(digestS3Bucket, digestS3Object); // Check that the digest file has been retrieved from its original location if (!digestFile.getString("digestS3Bucket").equals(digestS3Bucket) || !digestFile.getString("digestS3Object").equals(digestS3Object)) { System.err.println("Digest file has been moved from its original location."); } else { // Compute digest file hash MessageDigest messageDigest = MessageDigest.getInstance("SHA-256"); messageDigest.update(convertToByteArray(digestFile)); byte[] digestFileHash = messageDigest.digest(); messageDigest.reset(); // Compute the data to sign String dataToSign = String.format("%s%n%s/%s%n%s%n%s", digestFile.getString("digestEndTime"), digestFile.getString("digestS3Bucket"), digestFile.getString("digestS3Object"), // Constructing the S3 path of the digest file as part of the data to sign Hex.encodeHexString(digestFileHash), digestFile.getString("previousDigestSignature")); byte[] signatureContent = Hex.decodeHex(digestSignature); /* NOTE: To find the right public key to verify the signature, call CloudTrail ListPublicKey API to get a list of public keys, then match by the publicKeyFingerprint in the digest file. Also, the public key bytes returned from ListPublicKey API are DER encoded in PKCS#1 format: PublicKeyInfo ::= SEQUENCE { algorithm AlgorithmIdentifier, PublicKey BIT STRING } AlgorithmIdentifier ::= SEQUENCE { algorithm OBJECT IDENTIFIER, parameters ANY DEFINED BY algorithm OPTIONAL } */ pkcs1PublicKeyBytes = getPublicKey(digestFile.getString("digestPublicKeyFingerprint"))); // Transform the PKCS#1 formatted public key to x.509 format. RSAPublicKey rsaPublicKey = RSAPublicKey.getInstance(pkcs1PublicKeyBytes); AlgorithmIdentifier rsaEncryption = new AlgorithmIdentifier(PKCSObjectIdentifiers.rsaEncryption, null); SubjectPublicKeyInfo publicKeyInfo = new SubjectPublicKeyInfo(rsaEncryption, rsaPublicKey); // Create the PublicKey object needed for the signature validation PublicKey publicKey = KeyFactory.getInstance("RSA", "BC").generatePublic(new X509EncodedKeySpec(publicKeyInfo.getEncoded())); // Verify signature Signature signature = Signature.getInstance("SHA256withRSA", "BC"); signature.initVerify(publicKey); signature.update(dataToSign.getBytes("UTF-8")); if (signature.verify(signatureContent)) { System.out.println("Digest file signature is valid, validating log files…"); for (int i = 0; i < digestFile.getJSONArray("logFiles").length(); i++) { JSONObject logFileMetadata = digestFile.getJSONArray("logFiles").getJSONObject(i); // Compute log file hash byte[] logFileContent = loadUncompressedLogFileInMemory( logFileMetadata.getString("s3Bucket"), logFileMetadata.getString("s3Object") ); messageDigest.update(logFileContent); byte[] logFileHash = messageDigest.digest(); messageDigest.reset(); // Retrieve expected hash for the log file being processed byte[] expectedHash = Hex.decodeHex(logFileMetadata.getString("hashValue")); boolean signaturesMatch = Arrays.equals(expectedHash, logFileHash); if (!signaturesMatch) { System.err.println(String.format("Log file: %s/%s hash doesn't match.\tExpected: %s Actual: %s", logFileMetadata.getString("s3Bucket"), logFileMetadata.getString("s3Object"), Hex.encodeHexString(expectedHash), Hex.encodeHexString(logFileHash))); } else { System.out.println(String.format("Log file: %s/%s hash match", logFileMetadata.getString("s3Bucket"), logFileMetadata.getString("s3Object"))); } } } else { System.err.println("Digest signature failed validation."); } System.out.println("Digest file validation completed."); if (chainValidationIsEnabled()) { // This enables the digests' chain validation validateDigestFile( digestFile.getString("previousDigestS3Bucket"), digestFile.getString("previousDigestS3Object"), digestFile.getString("previousDigestSignature")); } } } }