Key rotation workflow

Managing the signing keys used at both token generation and validation steps is a crucial token-based access control mechanism. Any change to the keys, for instance during the key rotation step, must be carefully coordinated to avoid a transient state of inconsistency. In this state, playback API and CloudFront Functions become out-of-sync, where one of these components uses a signing key that the other does not recognize. To prevent that from happening, as part of the base module an automated key rotation pipeline is built. This pipeline is initiated the first time after the base module stack is deployed, and after that periodically according to the key rotation setting specified when launching the solution. The configuration about when the key rotation process must be initiated is saved as an EventBridge rule. Any time that workflow is initiated, the subsequent steps are controlled via AWS Step Functions workflow as depicted in the following diagram.

Key rotation workflow

A Lambda function automatically generates a new signing key with a corresponding unique UUID value, and stores it as a temporary secret in Secrets Manager. Lambda function also makes an API call to update CloudFront Function with the newly created key and UUID. Previously used key will still be available at function runtime, so that CloudFront Function code is able to validate tokens signed both by the old and new key during the transition period when the key is rotated.

The next step in the Step Functions workflow is to check every minute if the new function code, including the new key, has been deployed and that it can be used to generate new keys going forward. Once confirmed, the last Lambda function in the workflow will effectively swap the keys in Secrets Manager by moving the UUID and key value stored in the primary secret to the secondary secret, and replace the primary secret with the keys that have been created and deployed in CloudFront Functions. The next time the primary secret is retrieved by the solution’s library method, a new key is used to generate the tokens. With this staged and controlled process, at any time CloudFront Functions is able to validate all incoming requests with the tokens. After new keys are deployed in the function code, there will still be an interim period before rest of Step Functions workflow updates Secrets Manager with the new key, during which time the Playback API service relies on the older key. Because the old key and its UUID information is still present in the function code, when a token is generated using an older key by Playback API Gateway API, CloudFront Function can recognize the UUID of the used key specified in the JWT header. When the new key is eventually fetched by Playback API, it will be already confirmed that the new key has also been published on CloudFront Functions side, therefore both components can rely on the new key safely.