View a markdown version of this page

Get started with inference routers - Amazon SageMaker AI

Get started with inference routers

In this tutorial, you create an inference router that routes between endpoints in us-west-2 and us-east-1, invoke it with the AWS CLI and the AWS SDK for Python (Boto3), and then delete it.

Prerequisites

  • A version of the AWS CLI or an AWS SDK that supports inference routers. For more information, see Tools and Regions.

  • An IAM identity with permissions to manage the router, and to invoke the router and every destination (sagemaker:InvokeEndpoint). For the reason, see How the router authorizes requests to destinations. The following policy is an example.

{ "Version": "2012-10-17", "Statement": [ { "Sid": "ManageAndInvokeRouter", "Effect": "Allow", "Action": [ "sagemaker:CreateInferenceRouter", "sagemaker:DescribeInferenceRouter", "sagemaker:UpdateInferenceRouter", "sagemaker:DeleteInferenceRouter", "sagemaker:InvokeInferenceRouter" ], "Resource": "arn:aws:sagemaker:us-west-2:111122223333:inference-router/my-router" }, { "Sid": "ListRouters", "Effect": "Allow", "Action": "sagemaker:ListInferenceRouters", "Resource": "*" }, { "Sid": "InvokeAndDescribeDestinations", "Effect": "Allow", "Action": [ "sagemaker:InvokeEndpoint", "sagemaker:DescribeEndpoint", "sagemaker:DescribeInferenceComponent" ], "Resource": [ "arn:aws:sagemaker:us-west-2:111122223333:endpoint/my-endpoint-usw2", "arn:aws:sagemaker:us-east-1:111122223333:endpoint/my-endpoint-use1" ] } ] }

Without DescribeEndpoint or DescribeInferenceComponent on a destination, routing still works, but DescribeInferenceRouter shows the destination's RoutingStatus as Unknown. If you tag the router when you create it, you also need sagemaker:AddTags. For asynchronous routing targets, also allow sagemaker:InvokeInferenceRouterAsync on the router and sagemaker:InvokeEndpointAsync on every destination. For more information, see Route asynchronous inference requests through an inference router.

Create and use an inference router

To create, invoke, update, and delete an inference router
  1. Deploy the same model on an endpoint in two Regions.

    This example uses my-endpoint-usw2 and my-endpoint-use1. For instructions on deploying models to real-time endpoints, see Deploy models for real-time inference.

    1. Create a real-time endpoint in us-west-2.

    2. Create a real-time endpoint with the same model in us-east-1.

    3. Wait until both endpoints are InService.

    Both endpoints must serve the same model with the same request and response format. The router doesn't check this.

  2. Save the following routing target as routing-targets.json.

    It lists both endpoints as destinations.

    [ { "Name": "my-model", "Destinations": [ { "ResourceArn": "arn:aws:sagemaker:us-west-2:111122223333:endpoint/my-endpoint-usw2" }, { "ResourceArn": "arn:aws:sagemaker:us-east-1:111122223333:endpoint/my-endpoint-use1" } ] } ]
  3. Create the router in us-west-2.

    aws sagemaker create-inference-router \ --region us-west-2 \ --inference-router-name my-router \ --routing-targets file://routing-targets.json

    The response contains the router's Amazon Resource Name (ARN) in InferenceRouterArn.

  4. Check the router status.

    aws sagemaker describe-inference-router \ --region us-west-2 \ --inference-router-name my-router

    The following is example output.

    { "InferenceRouterName": "my-router", "InferenceRouterArn": "arn:aws:sagemaker:us-west-2:111122223333:inference-router/my-router", "Status": "InService", "RoutingTargets": [ { "Name": "my-model", "Destinations": [ { "ResourceArn": "arn:aws:sagemaker:us-west-2:111122223333:endpoint/my-endpoint-usw2", "RoutingStatus": "Routable" }, { "ResourceArn": "arn:aws:sagemaker:us-east-1:111122223333:endpoint/my-endpoint-use1", "RoutingStatus": "Routable" } ] } ], "CreationTime": "2026-11-16T10:00:00.000Z", "LastModifiedTime": "2026-11-16T10:00:00.000Z" }

    Wait until Status is InService. If it's Failed, see StatusReason. For the meaning of RoutingStatus, see How the router chooses a destination.

    To list the routers in a Region, run aws sagemaker list-inference-routers --region us-west-2.

  5. Invoke the router.

    Call InvokeInferenceRouter with the router name and the routing target name. The router passes the request body to the destination unchanged.

    aws sagemaker-inference-router-runtime invoke-inference-router \ --region us-west-2 \ --inference-router-name my-router \ --routing-target-name my-model \ --content-type application/json \ --body fileb://input.json \ output.json

    The model's response is written to output.json. The command prints the destination that served the request and the routing reason.

    { "ContentType": "application/json", "InvokedProductionVariant": "AllTraffic", "InvokedRoutedResourceArn": "arn:aws:sagemaker:us-west-2:111122223333:endpoint/my-endpoint-usw2", "RoutingReason": "local-preferred" }

    The following example uses Boto3.

    import boto3 from botocore.config import Config config = Config( read_timeout=170, retries={"total_max_attempts": 2, "mode": "standard"}, ) runtime = boto3.client( "sagemaker-inference-router-runtime", region_name="us-west-2", config=config ) response = runtime.invoke_inference_router( InferenceRouterName="my-router", RoutingTargetName="my-model", ContentType="application/json", Body=b'{"inputs": "Hello"}', ) print(response["InvokedRoutedResourceArn"], response["RoutingReason"]) print(response["Body"].read())

    Your model has up to 60 seconds per attempt, the same limit as InvokeEndpoint. Because the router can retry once on another destination, set your client read timeout to at least 170 seconds and limit SDK retries, as in the example. For more information, see Quotas and limits. For retry settings, see Configure client retries.

    You can also send Accept, CustomAttributes, and InferenceId. The router passes CustomAttributes to the destination, and returns the destination's CustomAttributes in the response.

  6. (Optional) Update the router.

    UpdateInferenceRouter replaces the complete list of routing targets. To add or remove a destination, send every routing target that you want to keep. For example, to add an endpoint in us-east-2, change routing-targets.json to the following.

    [ { "Name": "my-model", "Destinations": [ { "ResourceArn": "arn:aws:sagemaker:us-west-2:111122223333:endpoint/my-endpoint-usw2" }, { "ResourceArn": "arn:aws:sagemaker:us-east-1:111122223333:endpoint/my-endpoint-use1" }, { "ResourceArn": "arn:aws:sagemaker:us-east-2:111122223333:endpoint/my-endpoint-use2" } ] } ]
    1. Add my-endpoint-use2 to the Resource list of the caller's IAM policy.

    2. Update the router.

      aws sagemaker update-inference-router \ --region us-west-2 \ --inference-router-name my-router \ --routing-targets file://routing-targets.json

    Changes apply to new requests within about 2 minutes.

  7. Delete the router.

    Deleting a router doesn't delete its destinations.

    aws sagemaker delete-inference-router \ --region us-west-2 \ --inference-router-name my-router
  8. Delete the endpoints.

    Important

    Endpoints continue to incur charges until you delete them.

    For instructions, see Delete Endpoints and Resources.

Move an application from InvokeEndpoint

To move an application that calls InvokeEndpoint to an inference router, do the following:

  1. Deploy the same model, with the same container and request and response format, as a destination in each Region.

  2. Give callers sagemaker:InvokeInferenceRouter on the router and keep sagemaker:InvokeEndpoint on every destination.

  3. Change the client from sagemaker-runtime to sagemaker-inference-router-runtime, and call InvokeInferenceRouter with InferenceRouterName and RoutingTargetName instead of EndpointName. The body, ContentType, Accept, CustomAttributes, and InferenceId work as with InvokeEndpoint.

  4. Set the client read timeout to at least 170 seconds and at most 2 attempts. Your model still has up to 60 seconds per request.

  5. Remove TargetVariant, TargetModel, InferenceComponentName, and SessionId, and don't use response streaming; the router doesn't support them. To target an inference component, add the component as a destination.

  6. Handle the router's own errors: ThrottlingException (429) when you exceed the router's request rate, and ServiceUnavailableException (503). Read InvokedRoutedResourceArn to see which destination served the request.

For asynchronous requests, use InvokeInferenceRouterAsync. The input must be in Amazon S3 in the router's Region, and the output and failure locations come from the routing target, not from the endpoint. For more information, see Route asynchronous inference requests through an inference router.

Next steps

For more information, see the following topics: