Get started with inference routers
In this tutorial, you create an inference router that routes between endpoints in
us-west-2 and us-east-1, invoke it with the AWS CLI and the
AWS SDK for Python (Boto3), and then delete it.
Prerequisites
-
A version of the AWS CLI or an AWS SDK that supports inference routers. For more information, see Tools and Regions.
-
An IAM identity with permissions to manage the router, and to invoke the router and every destination (
sagemaker:InvokeEndpoint). For the reason, see How the router authorizes requests to destinations. The following policy is an example.
{ "Version": "2012-10-17", "Statement": [ { "Sid": "ManageAndInvokeRouter", "Effect": "Allow", "Action": [ "sagemaker:CreateInferenceRouter", "sagemaker:DescribeInferenceRouter", "sagemaker:UpdateInferenceRouter", "sagemaker:DeleteInferenceRouter", "sagemaker:InvokeInferenceRouter" ], "Resource": "arn:aws:sagemaker:us-west-2:111122223333:inference-router/my-router" }, { "Sid": "ListRouters", "Effect": "Allow", "Action": "sagemaker:ListInferenceRouters", "Resource": "*" }, { "Sid": "InvokeAndDescribeDestinations", "Effect": "Allow", "Action": [ "sagemaker:InvokeEndpoint", "sagemaker:DescribeEndpoint", "sagemaker:DescribeInferenceComponent" ], "Resource": [ "arn:aws:sagemaker:us-west-2:111122223333:endpoint/my-endpoint-usw2", "arn:aws:sagemaker:us-east-1:111122223333:endpoint/my-endpoint-use1" ] } ] }
Without DescribeEndpoint or DescribeInferenceComponent on a
destination, routing still works, but DescribeInferenceRouter shows the
destination's RoutingStatus as Unknown. If you tag the router
when you create it, you also need sagemaker:AddTags. For asynchronous
routing targets, also allow sagemaker:InvokeInferenceRouterAsync on the
router and sagemaker:InvokeEndpointAsync on every destination. For more
information, see Route asynchronous inference requests through an inference router.
Create and use an inference router
To create, invoke, update, and delete an inference router
-
Deploy the same model on an endpoint in two Regions.
This example uses
my-endpoint-usw2andmy-endpoint-use1. For instructions on deploying models to real-time endpoints, see Deploy models for real-time inference.-
Create a real-time endpoint in
us-west-2. -
Create a real-time endpoint with the same model in
us-east-1. -
Wait until both endpoints are
InService.
Both endpoints must serve the same model with the same request and response format. The router doesn't check this.
-
-
Save the following routing target as
routing-targets.json.It lists both endpoints as destinations.
[ { "Name": "my-model", "Destinations": [ { "ResourceArn": "arn:aws:sagemaker:us-west-2:111122223333:endpoint/my-endpoint-usw2" }, { "ResourceArn": "arn:aws:sagemaker:us-east-1:111122223333:endpoint/my-endpoint-use1" } ] } ] -
Create the router in
us-west-2.aws sagemaker create-inference-router \ --region us-west-2 \ --inference-router-namemy-router\ --routing-targets file://routing-targets.jsonThe response contains the router's Amazon Resource Name (ARN) in
InferenceRouterArn. -
Check the router status.
aws sagemaker describe-inference-router \ --region us-west-2 \ --inference-router-namemy-routerThe following is example output.
{ "InferenceRouterName": "my-router", "InferenceRouterArn": "arn:aws:sagemaker:us-west-2:111122223333:inference-router/my-router", "Status": "InService", "RoutingTargets": [ { "Name": "my-model", "Destinations": [ { "ResourceArn": "arn:aws:sagemaker:us-west-2:111122223333:endpoint/my-endpoint-usw2", "RoutingStatus": "Routable" }, { "ResourceArn": "arn:aws:sagemaker:us-east-1:111122223333:endpoint/my-endpoint-use1", "RoutingStatus": "Routable" } ] } ], "CreationTime": "2026-11-16T10:00:00.000Z", "LastModifiedTime": "2026-11-16T10:00:00.000Z" }Wait until
StatusisInService. If it'sFailed, seeStatusReason. For the meaning ofRoutingStatus, see How the router chooses a destination.To list the routers in a Region, run
aws sagemaker list-inference-routers --region us-west-2. -
Invoke the router.
Call
InvokeInferenceRouterwith the router name and the routing target name. The router passes the request body to the destination unchanged.aws sagemaker-inference-router-runtime invoke-inference-router \ --region us-west-2 \ --inference-router-namemy-router\ --routing-target-namemy-model\ --content-type application/json \ --body fileb://input.json\output.jsonThe model's response is written to
output.json. The command prints the destination that served the request and the routing reason.{ "ContentType": "application/json", "InvokedProductionVariant": "AllTraffic", "InvokedRoutedResourceArn": "arn:aws:sagemaker:us-west-2:111122223333:endpoint/my-endpoint-usw2", "RoutingReason": "local-preferred" }The following example uses Boto3.
import boto3 from botocore.config import Config config = Config( read_timeout=170, retries={"total_max_attempts": 2, "mode": "standard"}, ) runtime = boto3.client( "sagemaker-inference-router-runtime", region_name="us-west-2", config=config ) response = runtime.invoke_inference_router( InferenceRouterName="my-router", RoutingTargetName="my-model", ContentType="application/json", Body=b'{"inputs": "Hello"}', ) print(response["InvokedRoutedResourceArn"], response["RoutingReason"]) print(response["Body"].read())Your model has up to 60 seconds per attempt, the same limit as
InvokeEndpoint. Because the router can retry once on another destination, set your client read timeout to at least 170 seconds and limit SDK retries, as in the example. For more information, see Quotas and limits. For retry settings, see Configure client retries.You can also send
Accept,CustomAttributes, andInferenceId. The router passesCustomAttributesto the destination, and returns the destination'sCustomAttributesin the response. -
(Optional) Update the router.
UpdateInferenceRouterreplaces the complete list of routing targets. To add or remove a destination, send every routing target that you want to keep. For example, to add an endpoint inus-east-2, changerouting-targets.jsonto the following.[ { "Name": "my-model", "Destinations": [ { "ResourceArn": "arn:aws:sagemaker:us-west-2:111122223333:endpoint/my-endpoint-usw2" }, { "ResourceArn": "arn:aws:sagemaker:us-east-1:111122223333:endpoint/my-endpoint-use1" }, { "ResourceArn": "arn:aws:sagemaker:us-east-2:111122223333:endpoint/my-endpoint-use2" } ] } ]-
Add
my-endpoint-use2to theResourcelist of the caller's IAM policy. -
Update the router.
aws sagemaker update-inference-router \ --region us-west-2 \ --inference-router-namemy-router\ --routing-targets file://routing-targets.json
Changes apply to new requests within about 2 minutes.
-
-
Delete the router.
Deleting a router doesn't delete its destinations.
aws sagemaker delete-inference-router \ --region us-west-2 \ --inference-router-namemy-router -
Delete the endpoints.
ImportantEndpoints continue to incur charges until you delete them.
For instructions, see Delete Endpoints and Resources.
Move an application from InvokeEndpoint
To move an application that calls InvokeEndpoint to an inference
router, do the following:
-
Deploy the same model, with the same container and request and response format, as a destination in each Region.
-
Give callers
sagemaker:InvokeInferenceRouteron the router and keepsagemaker:InvokeEndpointon every destination. -
Change the client from
sagemaker-runtimetosagemaker-inference-router-runtime, and callInvokeInferenceRouterwithInferenceRouterNameandRoutingTargetNameinstead ofEndpointName. The body,ContentType,Accept,CustomAttributes, andInferenceIdwork as withInvokeEndpoint. -
Set the client read timeout to at least 170 seconds and at most 2 attempts. Your model still has up to 60 seconds per request.
-
Remove
TargetVariant,TargetModel,InferenceComponentName, andSessionId, and don't use response streaming; the router doesn't support them. To target an inference component, add the component as a destination. -
Handle the router's own errors:
ThrottlingException(429) when you exceed the router's request rate, andServiceUnavailableException(503). ReadInvokedRoutedResourceArnto see which destination served the request.
For asynchronous requests, use InvokeInferenceRouterAsync. The input must
be in Amazon S3 in the router's Region, and the output and failure locations come from the
routing target, not from the endpoint. For more information, see Route asynchronous inference requests through an inference router.
Next steps
For more information, see the following topics: