Update models in production

Deployment guardrails are a set of model deployment options in Amazon SageMaker Inference to update your machine learning models in production. Using the fully managed deployment options, you can control the switch from the current model in production to a new one. Traffic shifting modes in blue/green deployments, such as canary and linear, give you granular control over the traffic shifting process from your current model to the new one during the course of the update. There are also built-in safeguards such as auto-rollbacks that help you catch issues early and automatically take corrective action before they significantly impact production.

Deployment guardrails provide the following benefits:

• Deployment safety while updating production environments. A regressive update to a production environment can cause unplanned downtime and business impact, such as increased model latency and high error rates. Deployment guardrails help you mitigate those risks by providing best practices and built-in operational safety guardrails.

• Fully managed deployment. SageMaker takes care of setting up and orchestrating these deployments and integrates them with endpoint update mechanisms. You do not need to build and maintain orchestration, monitoring, or rollback mechanisms. You can leverage SageMaker to set up and orchestrate these deployments and focus on leveraging ML for your applications.

• Visibility. You can track the progress of your deployment through the DescribeEndpoint API or through Amazon CloudWatch Events (for supported endpoints). To learn more about events in SageMaker, see the Endpoint deployment state change section in Automating Amazon SageMaker with Amazon EventBridge. Note that if your endpoint uses any of the features in the Exclusions page, you cannot use CloudWatch Events.

Note

Deployment guardrails only apply to Asynchronous inference and Real-time inference endpoint types.

How to get started

We support two types of deployments to update models in production: blue/green deployments and rolling deployments.

• Blue/Green Deployments: You can shift traffic from your old fleet (the blue fleet) to a new fleet (green fleet) with the updates. Blue/green deployments offer multiple traffic shifting modes. A traffic shifting mode is a configuration that specifies how SageMaker routes endpoint traffic to a new fleet containing your updates. The following traffic shifting modes provide you with different levels of control over the endpoint update process:

  • All At Once Traffic Shifting shifts all of your endpoint traffic from the blue fleet to the green fleet. Once the traffic shifts to the green fleet, your pre-specified Amazon CloudWatch alarms begin monitoring the green fleet for a set amount of time (the baking period). If no alarms trip during the baking period, then SageMaker terminates the blue fleet.

  • Canary Traffic Shifting shifts one small portion of your traffic (a canary) to the green fleet and monitor it for a baking period. If the canary succeeds on the green fleet, then SageMaker shifts the rest of the traffic from the blue fleet to the green fleet before terminating the blue fleet.

  • Linear Traffic Shifting provides even more customization over the number of traffic-shifting steps and the percentage of traffic to shift for each step. While canary shifting lets you shift traffic in two steps, linear shifting extends this to n linearly spaced steps.

• Rolling Deployments: You can update your endpoint as SageMaker incrementally provisions capacity and shifts traffic to a new fleet in steps of a batch size that you specify. Instances on the new fleet are updated with the new deployment configuration, and if no CloudWatch alarms trip during the baking period, then SageMaker cleans up instances on the old fleet. This option gives you granular control over the instance count or capacity percentage shifted during each step.

You can create and manage your deployment through the UpdateEndpoint and CreateEndpoint SageMaker API and AWS Command Line Interface commands. See the individual deployment pages for more details on how to set up your deployment. Note that if your endpoint uses any of the features listed in the Exclusions page, you cannot use deployment guardrails.

To follow guided examples that shows how to use deployment guardrails, see our example Jupyter notebooks for the canary and linear traffic shifting modes.