Deploying runtime updates
As a runtime publisher, you release each runtime update as a new runtime version. You then roll it out by updating your runtime aliases. A runtime update changes code that runs in your consumers' Lambda Event Functions. This topic explains how to roll out updates safely, monitor them, and roll them back if a problem occurs.
For an overview of how runtime updates work, see Customer-managed runtimes.
Your responsibilities as a runtime publisher
When you publish a runtime, you own its updates. Lambda doesn't patch or test customer-managed runtimes for you. Your responsibilities include the following:
-
Backward compatibility – Each update must keep working with the functions that already use the alias. Avoid changes that break existing function code or configuration.
-
Testing – Test each new runtime version before you point an alias to it.
-
Security patching – Apply patches to every component of your runtime. This includes the operating system, the language runtime, the runtime interface client, and any libraries that you include. If you build on a Lambda base image, rebuild your image to pick up new patches to the base image. For more information, see Building a runtime image from a Lambda base image.
-
Communication – Tell your consumers what changed in each update. Add release notes to each runtime version. For more information, see Update release notes. Runtime metrics don't identify consumers, and CloudTrail events in your account show which accounts use your runtime but not who to contact. We recommend that you keep a way to reach consumers, such as a registry of consumer accounts, a mailing list, or a chat channel.
How often you patch your runtime is a business decision. Base it on your security requirements and those of your consumers. To make frequent updates practical, we recommend that you automate them with a runtime update pipeline. For more information, see Building a runtime update pipeline.
Runtime update workflow
A runtime update typically follows these steps:
-
Build – Build a new runtime image. For more information, see Building runtime images.
-
Push – Push the image to an Amazon Elastic Container Registry (Amazon ECR) repository in the same AWS Region as the runtime.
-
Create a version – Create a runtime version from the image. No function uses the new version until you point an alias to it. For more information, see Runtime versions.
-
Test – Test the new version. For example, pin a test function to the new version with the Manual runtime update mode. You can also update a development alias that only your test functions use.
-
Roll out in stages – Update your aliases one at a time, with a bake period between each update. For more information, see Safe rollout mechanisms.
-
Monitor – Watch the runtime metrics for the new version and the updated alias. For more information, see Monitoring function health during a rollout.
-
Continue or roll back – If the new version is healthy, update the next alias. If you find a problem, point the alias back to the previous version. For more information, see Rolling back a runtime update.
Safe rollout mechanisms
A faulty runtime update can affect every function that uses an alias. To limit the impact, combine the following mechanisms:
- Alias-based staging
-
Create an alias for each stage of your release process, such as
dev,test, andprod. Ask consumers to use the alias that matches each of their environments. Then update each alias in turn, with a bake period between updates. A problem that appears indevdoesn't reach functions that useprod. For more information, see Alias strategies. - Region-by-Region staggering
-
Runtimes are Regional, so you update the aliases in each Region separately. Update one Region at a time, and check the runtime's health before you move to the next Region. For more information, see Deploying runtimes in multiple Regions.
- Cold-start gating
-
An alias update doesn't reach all functions at once. Functions in Auto mode pick up the new version at their next cold start. Lambda replaces execution environments routinely, so the update rolls out gradually. Environments that are already running keep the previous version until Lambda replaces them.
Functions in Function update mode pick up the new version only when their owner next deploys them. Functions in Manual mode don't change. For more information, see How alias updates reach functions.
Customer-managed runtimes don't use the two-phase rollout that Lambda uses for Lambda-managed runtimes. You control the pace of the rollout by choosing when to update each alias in each Region.
Choose a bake period that's long enough for many functions to start new execution environments with the new version. Functions in Function update mode might move to the new version days or weeks after you update the alias.
Monitoring function health during a rollout
Lambda publishes runtime metrics to Amazon CloudWatch in the account that owns the runtime. The RuntimeInvocations and RuntimeInvocationErrors metrics count invocations of all functions that use the runtime. You can view them by runtime name, runtime version, and runtime alias. For the metric reference and for instructions on viewing metrics and creating alarms, see Monitoring customer-managed runtimes.
RuntimeInvocationErrors counts invocations that fail for any of the following reasons:
-
An unhandled exception (a function error)
-
A timeout
-
A crash of the runtime or function process
-
An initialization (init) failure
Lambda doesn't count an invocation that completes as an error, even if the function returns an application-level error in its response. For example, a response body with an HTTP 500 status code doesn't count as an error. Because the metric includes errors in your consumers' own function code, compare the error rate of the old and new runtime version or alias. Don't alarm on absolute values. For more information, see Runtime metrics.
The runtime version and runtime alias dimensions help you judge a rollout in the following ways:
-
Compare versions – Compare the error rate (
RuntimeInvocationErrorsdivided byRuntimeInvocations) of the new version with the error rate of the previous version. Compare them over the same period, when traffic is similar. A higher error rate for the new version suggests a problem with the update. -
Compare aliases – Check whether errors are concentrated in the alias that you updated. For example, if the error rate rises for
devbut not forprod, the new version is the likely cause. -
See impact across all consumers – In each Region, the metrics include invocations from every account that uses the runtime. You can see the impact of an update across your whole consumer base without access to your consumers' functions. The metrics don't show function names or account IDs.
-
Track rollout progress – As Lambda replaces execution environments,
RuntimeInvocationsfor the new version rise andRuntimeInvocationsfor the previous version fall. This shows how much of your traffic uses the new version.
We recommend that you create a CloudWatch alarm on the error rate of each alias. Configure your update pipeline to stop the rollout when an alarm enters the ALARM state. For instructions on creating an alarm, see Monitoring customer-managed runtimes.
Rolling back a runtime update
To roll back an update, point the alias back to the previous runtime version. Functions in Auto mode pick up the previous version at their next cold start. You don't need to change the functions. If several aliases point to the faulty version, roll back each of them. For instructions, see Roll back a runtime update.
Functions in Function update mode don't change when you roll back an alias. They keep the runtime version that they were deployed with until their owner deploys them again. This isn't necessarily a problem. The issue in the faulty version might not affect these functions, and they might be working as expected. Tell your consumers about the rollback so that they can decide whether to redeploy.
After you roll back, fix the issue and roll forward with a new runtime version. Runtime versions are immutable, so you can't fix a version in place.
Runtime consumers can also roll back on their own. A consumer can pin a function to an earlier runtime version with Manual mode. For more information, see Using a customer-managed runtime.
Don't delete the faulty version right away. Functions in Function update or Manual mode might still use it. Check the RuntimeInvocations metric for the version before you delete it. For more information, see Deleting and restoring runtimes.
Compatibility constraints on updates
Each alias declares the capabilities and architectures that it supports. A new runtime version must support at least those capabilities and architectures. Otherwise, the alias update fails. This check stops an update from breaking functions that rely on what the alias declares.
A new version can add capabilities or architectures. For example, you can publish a multi-architecture version to add arm64 support. After you point the alias to that version, the alias gains the architecture. For more information, see Add a capability or architecture to an alias.
You can't remove a capability or architecture from an alias by pointing it to a version that drops the capability. Lambda blocks that update to protect functions that rely on the alias. To stop offering a capability, create a new alias for versions without it, and move consumers to it. For more information, see Stop offering a capability.
For more information about alias compatibility, see Alias compatibility.
Building a runtime update pipeline
We recommend that you automate runtime updates with a pipeline, for example with AWS CodePipeline and AWS CodeBuild. A pipeline makes frequent security patching practical and applies the same checks to every update. A typical pipeline has the following stages:
-
Build – Build the runtime image with the latest patches, and run unit tests.
-
Publish – Push the image to Amazon ECR, and replicate it to each Region where you publish the runtime. Then create a runtime version in each Region. To validate the request first, use the
--dry-runoption. -
Test – Pin test functions to the new version, and run integration tests. If the version supports both architectures, test a function on each architecture.
-
Update aliases in stages – Update each alias in turn, such as
dev,test, and thenprod. Update theprodalias one Region at a time. -
Bake and monitor – After each alias update, wait for a bake period. Stop the pipeline if an error rate alarm enters the
ALARMstate. -
Promote or roll back – If the alarms stay healthy, continue to the next stage. Otherwise, point the updated aliases back to the previous version.
For more information about automating updates across Regions, see Deploying runtimes in multiple Regions.
Deploy a runtime update in stages
The following example rolls out version v8.3.2 of the companyphp8 runtime. First, you create the version and test it with a pinned function. Then you update the dev, test, and prod aliases in turn.
Before you start, make sure that you have the following:
-
A runtime named
companyphp8in theus-east-1Region, in account123456789012. Itsdev,test, andprodaliases point to versionv8.3.1. -
A new runtime image, tagged
v8.3.2, in thecompanyphp8Amazon ECR repository. For more information, see Building runtime images. -
A test function named
cmr-test-fnthat uses thecompanyphp8runtime. For more information, see Using a customer-managed runtime. -
Permissions for
lambda:CreateRuntimeVersion,lambda:GetRuntimeVersion,lambda:UpdateRuntimeAlias,lambda:GetRuntimeAlias,lambda:PutRuntimeManagementConfig,lambda:AssociateRuntimeVersion, andlambda:InvokeFunction. To create the runtime version, you also needecr:BatchGetImageandecr:GetDownloadUrlForLayeron the Amazon ECR repository that contains the image. For more information, see Sharing and controlling access to customer-managed runtimes.
Deploy a runtime update in stages (AWS CLI)
First, create the new runtime version and test it with a pinned function.
To create and test a new runtime version (AWS CLI)
-
To create the runtime version, run the
create-runtime-versioncommand:aws lambda-core create-runtime-version \ --runtime-namecompanyphp8\ --namev8.3.2\ --image-uri123456789012.dkr.ecr.us-east-1.amazonaws.com/companyphp8:v8.3.2\ --release-notes "Updates PHP to 8.3.2 and applies the latest OS security patches." \ --regionus-east-1 -
To wait until the version is
ACTIVE, run the following command:aws lambda-core wait runtime-version-active \ --runtime-namecompanyphp8\ --namev8.3.2\ --regionus-east-1 -
To pin the test function to the new version, run the
put-runtime-management-configcommand:aws lambda put-runtime-management-config \ --function-namecmr-test-fn\ --update-runtime-on Manual \ --runtime-version-arnarn:aws:lambda:us-east-1:123456789012:runtime/companyphp8/version/v8.3.2\ --regionus-east-1The pin takes effect right away. Lambda updates the function, and new invocations use the pinned version.
-
To test the function, invoke it:
aws lambda invoke \ --function-namecmr-test-fn\ --regionus-east-1\response.jsonCheck that the response in
response.jsonis what you expect. Run your full set of tests against the function. -
To confirm that the function used the new version, search its logs for the
INIT_STARTline:aws logs filter-log-events \ --log-group-name /aws/lambda/cmr-test-fn\ --filter-pattern INIT_STARTThe latest
INIT_STARTline shows the runtime version, similar to the following:INIT_START Runtime Version: companyphp8:v8.3.2 Runtime Version ARN: arn:aws:lambda:us-east-1:123456789012:runtime/companyphp8/version/v8.3.2
If the version supports both x86_64 and arm64, test a function on each architecture. When you finish testing, you can return the test function to Auto mode. To do this, run put-runtime-management-config with --update-runtime-on Auto.
Next, roll out the version through your aliases.
To roll out a runtime version through aliases (AWS CLI)
-
To update the
devalias, run theupdate-runtime-aliascommand:aws lambda-core update-runtime-alias \ --runtime-namecompanyphp8\ --namedev\ --runtime-versionv8.3.2\ --regionus-east-1If the new version doesn't support all the capabilities and architectures that the alias declares, the command fails. For more information, see Compatibility constraints on updates.
If an alias has its own lifecycle, include the
--release-lifecycleoption with the complete lifecycle in eachupdate-runtime-aliascommand. Otherwise, the update clears the alias lifecycle. For more information, see Rules for release states and dates. -
To verify the update, run the
get-runtime-aliascommand:aws lambda-core get-runtime-alias \ --runtime-namecompanyphp8\ --namedev\ --query "[State,RuntimeVersion]" \ --regionus-east-1You should see the following output:
[ "ACTIVE", "v8.3.2" ] -
Monitor the
devalias for your bake period. Compare the error rate ofv8.3.2with the error rate ofv8.3.1. For instructions, see Monitoring customer-managed runtimes. -
If the
devalias is healthy, update thetestalias:aws lambda-core update-runtime-alias \ --runtime-namecompanyphp8\ --nametest\ --runtime-versionv8.3.2\ --regionus-east-1 -
Monitor the
testalias for your bake period. -
If the
testalias is healthy, update theprodalias:aws lambda-core update-runtime-alias \ --runtime-namecompanyphp8\ --nameprod\ --runtime-versionv8.3.2\ --regionus-east-1 -
Monitor the
prodalias. WatchRuntimeInvocationsforv8.3.2rise as functions move to the new version.
If you find a problem at any stage, roll back that alias, and don't update the later aliases. For instructions, see Roll back a runtime update. If you publish the runtime in more than one Region, repeat the alias updates in each Region. For more information, see Deploying runtimes in multiple Regions.
Roll back a runtime update
The following example rolls back the prod alias of the companyphp8 runtime from v8.3.2 to v8.3.1. Before you start, stop your pipeline so that it doesn't update more aliases. You need the lambda:UpdateRuntimeAlias and lambda:GetRuntimeAlias permissions.
Roll back a runtime update (console)
To roll back in the Lambda console, update the alias to point to the previous runtime version. Follow the steps in To update a runtime alias (console), and choose the previous runtime version.
Roll back a runtime update (AWS CLI)
To roll back a runtime update (AWS CLI)
-
To point the alias back to the previous version, run the
update-runtime-aliascommand:aws lambda-core update-runtime-alias \ --runtime-namecompanyphp8\ --nameprod\ --runtime-versionv8.3.1\ --regionus-east-1 -
To verify the rollback, run the
get-runtime-aliascommand:aws lambda-core get-runtime-alias \ --runtime-namecompanyphp8\ --nameprod\ --query "[State,RuntimeVersion]" \ --regionus-east-1You should see the following output:
[ "ACTIVE", "v8.3.1" ] -
Repeat the preceding steps for each other alias that points to
v8.3.2, such asdevandtest. -
Monitor
RuntimeInvocationsfor each version. As functions start new execution environments,RuntimeInvocationsforv8.3.1rise andRuntimeInvocationsforv8.3.2fall. For instructions, see Monitoring customer-managed runtimes.
To tell consumers about the problem, you can update the release notes of v8.3.2. For more information, see Update release notes. After you fix the issue, create a new version, such as v8.3.3, and roll it out. For instructions, see Deploy a runtime update in stages.