Global Inference Gateway for Amazon SageMaker HyperPod Inference
The Amazon SageMaker HyperPod Global Inference Gateway gives you a single, managed HTTPS endpoint that sends each inference request to the best available model deployment across multiple HyperPod clusters. The clusters can be in the same AWS Region or in different Regions. For every request, the Global Inference Gateway reads the requested model, looks at the live load and health of every cluster that serves that model, and forwards the request to the cluster that is best placed to answer it. If that cluster is overloaded or unavailable, the request fails over to the next-best cluster.
Each HyperPod cluster keeps its own in-cluster Inference Gateway, which picks the model-serving pod within that cluster. The Global Inference Gateway adds a routing layer above those in-cluster gateways, called members, and picks the member for each request based on its current load (KV cache usage and queue depth), recent error rate, and Region proximity.
Use a Global Inference Gateway when you want to:
-
Serve the same model from several HyperPod clusters behind one endpoint.
-
Keep serving when a cluster, an Availability Zone, or a Region has a problem.
-
Use spare GPU capacity in other clusters or Regions when one cluster is busy.
-
Change how traffic is balanced across clusters without changing client code.
How it works
A Global Inference Gateway is made of the following parts.
- Global Inference Gateway
-
A regional AWS resource that you create with the SageMaker API. It has an ARN of the form
arn:aws:sagemaker:<region>:<account-id>:global-inference-gateway/<gateway-id>and an HTTPS endpoint (EndpointUrl) that your clients call. SageMaker AI runs and scales the routing fleet behind the endpoint for you. The fleet runs in the Region where you create the Global Inference Gateway, and routes requests to members in that Region or in other supported Regions. - Member (in-cluster Inference Gateway)
-
An in-cluster inference gateway that you have added to the Global Inference Gateway. A member is identified by three values: the HyperPod cluster ARN, the Kubernetes namespace, and the name of its
InferenceGatewayConfigresource. A Global Inference Gateway has 1 to 10 members. - Global routing
-
A setting on an
InferenceGatewayConfigresource (spec.globalRouting.enabled). When it is turned on, the in-cluster inference gateway is published privately over AWS PrivateLink so that a Global Inference Gateway can reach it without a public endpoint. It also starts sending load and health metrics for its models to the Global Inference Gateway. - Execution role
-
An IAM role in your account that the Global Inference Gateway assumes to reach your member clusters: to read the in-cluster gateway's configuration, register itself with the cluster, and read the cluster gateway's TLS certificate.
The request flow is as follows:
-
A client sends a SigV4-signed HTTPS request to the Global Inference Gateway endpoint. The request body names the model in its
modelfield. -
The Global Inference Gateway authenticates the caller, checks that the caller belongs to the gateway's owning account, and authorizes the request with IAM.
-
It finds every member that currently serves the requested model (a base model name or a LoRA adapter name), and scores each one from its latest metrics: KV cache headroom, queue depth, Region proximity, and error rate.
-
It forwards the request over a private, mutually authenticated TLS connection to the highest-scoring member.
-
The member's in-cluster inference gateway routes the request to a model-serving pod, and the response is returned to the client. Streaming responses are passed through as they are produced.
-
If the chosen member returns a throttling or availability error, the request is retried once on the next-best member.
Each member reports its load and health to the Global Inference Gateway every few seconds. The Global Inference Gateway sends traffic only to members that have reported metrics, and stops sending traffic to a member that reports it can't serve or that hasn't reported for 30 minutes. A member whose metrics are only stale keeps being scored on its last reported load.
How it relates to the in-cluster inference gateway
The Global Inference Gateway and the in-cluster inference gateway form a two-tier routing system:
| Tier 1: In-cluster Inference Gateway | Tier 2: Global Inference Gateway | |
|---|---|---|
| Question it answers | Which pod in this cluster should serve the request? | Which cluster should serve the request? |
| Where it runs | In your HyperPod cluster, installed by the HyperPod Inference add-on | Managed by SageMaker AI, in the Region where you create it |
| Signals it uses | Per-pod model server state: KV cache utilization, queue depth, prefix cache, and LoRA adapter affinity | Per-cluster model server state reported by each member (KV cache utilization, queue depth, error rate), plus Region proximity |
| How you set it up | Add-on plus an InferenceGatewayConfig (or spec.inferenceGateway on a model) |
Turn on global routing on each in-cluster gateway, then one SageMaker API create call |
| Endpoint | One endpoint per cluster | One endpoint for all member clusters, unchanged as you add or remove clusters |
Each tier is useful on its own. A cluster with only the in-cluster inference gateway gets model-aware routing within that cluster. Adding a Global Inference Gateway adds cross-cluster and cross-Region routing and failover, with no change to your models or to the in-cluster configuration other than turning on global routing. Your in-cluster gateways keep serving their own endpoints while they are members, so applications that call a cluster directly keep working.
Both tiers route on model server signals rather than generic load balancer metrics such as connection count or CPU, and neither depends on a specific model server: any server that publishes the standard metrics that the in-cluster inference gateway reads (for example vLLM or SGLang) works.
Steps to set up a Global Inference Gateway
To set up a Global Inference Gateway, complete these steps in order:
Prerequisites
Before you create a Global Inference Gateway, make sure you have the following:
-
One or more Amazon SageMaker HyperPod clusters orchestrated by Amazon EKS, in the same AWS account, in supported Regions. To create one, see Creating a SageMaker HyperPod cluster with Amazon EKS orchestration.
-
On each cluster, the Amazon SageMaker HyperPod Inference add-on, version
v3.0.0-eksbuild.1, with the Inference Gateway turned on, and anInferenceGatewayConfigthat serves your models. Global routing also uses the AWS Load Balancer Controller, which the add-on installs, and cert-manager, which the console's Quick Install adds. If you installed the add-on with the AWS CLI, install cert-manager as described in Required Add-ons. For setup, see Inference Gateway for Amazon SageMaker HyperPod Inference. -
Model servers that publish the metrics the in-cluster inference gateway reads: vLLM v0.9.2 or later, or SGLang v0.3.5.post1 or later started with
--enable-metrics. The Global Inference Gateway routes to a member only when it reports KV cache utilization and queue depth, so a member on an earlier vLLM version receives no traffic. -
The add-on configured with a TLS certificate bucket (
tlsCertificateS3Bucket). The in-cluster gateway uploads its global routing certificate to this bucket, and the Global Inference Gateway reads it to verify the cluster's identity. The AWS managed policies in this guide grant access only to buckets whose names begin withhyperpod-tls. If your bucket has a different name, add the bucket statements in Step 1a and Step 2b. -
Subnets in at least two Availability Zones of the cluster VPC tagged with
kubernetes.io/role/internal-elb=1, so that the internal Network Load Balancer for global routing spans two or more Availability Zones. -
Free IP addresses in the HyperPod cluster's subnets, in at least one Availability Zone that supports an interface endpoint for Amazon Kinesis Data Streams. The controller places the metrics endpoint there.
-
The same model name served by every member that should receive traffic for that model. The Global Inference Gateway routes on the
modelfield in the request body, so members are interchangeable only when they serve the model under the same name. -
The latest version of the AWS CLI. For instructions, see Installing or updating to the latest version of the AWS CLI.
IAM permissions for the administrator
The IAM user or role that creates and manages Global Inference Gateways needs the following permissions. Add them as an IAM policy attached to that user or role, for example as an inline policy or a customer managed policy. Replace <region>, <account-id>, and <execution-role-name> with your values. For how to attach a policy, see Adding and removing IAM identity permissions.
{ "Version": "2012-10-17", "Statement": [ { "Sid": "ManageGlobalInferenceGateways", "Effect": "Allow", "Action": [ "sagemaker:CreateGlobalInferenceGateway", "sagemaker:DescribeGlobalInferenceGateway", "sagemaker:UpdateGlobalInferenceGateway", "sagemaker:DeleteGlobalInferenceGateway", "sagemaker:AddInferenceGateway", "sagemaker:RemoveInferenceGateway", "sagemaker:ListInferenceGateways", "sagemaker:AddTags", "sagemaker:ListTags" ], "Resource": "arn:aws:sagemaker:<region>:<account-id>:global-inference-gateway/*" }, { "Sid": "ListGlobalInferenceGateways", "Effect": "Allow", "Action": "sagemaker:ListGlobalInferenceGateways", "Resource": "*" }, { "Sid": "PassExecutionRole", "Effect": "Allow", "Action": "iam:PassRole", "Resource": "arn:aws:iam::<account-id>:role/<execution-role-name>" } ] }
Plan your Global Inference Gateway
Before you start, decide how you want to set up your Global Inference Gateway. You can change members and routing weights later, but the Region and the access model are easier to get right up front.
| Decision | Options | Learn more |
|---|---|---|
| Region for the Global Inference Gateway | Any supported Region. Clients call its EndpointUrl, and SageMaker AI runs the routing fleet in that Region. Region proximity is measured from this Region, not from the client, so create it in the Region where most of your clients are. |
Supported Regions |
| Members | 1 to 10 in-cluster inference gateways, in the same account. The clusters can be in the same Region or in different Regions, but not in opt-in Regions. Each in-cluster gateway can belong to only one Global Inference Gateway. | Quotas and limits |
| Routing | The default weights, or your own. Raise RegionProximity to keep traffic in the gateway's Region, QueueDepth when response time matters most, or KVHeadroom for long prompts and responses. |
Choosing routing weights |
| How clients connect | The public EndpointUrl, or privately from your VPC through AWS PrivateLink. |
Invoke privately with AWS PrivateLink |
| Which API clients use | OpenAI-compatible paths (/v1/chat/completions, /v1/completions, /v1/embeddings), or the AWS SDK operations, which return typed errors. |
Step 4 |
| End-user authentication | If your in-cluster gateways use JWT authentication, clients pass the end user's token in a separate header. | Pass an end-user token to your clusters |
| Execution role scope | The AWS managed policy, which covers all clusters in the account and TLS buckets named hyperpod-tls*, or an inline policy scoped to specific clusters and your bucket. |
Step 2b |
Step 1: Enable global routing on each in-cluster inference gateway
Repeat this step on every HyperPod cluster that you want to add as a member.
Identify the in-cluster inference gateway to add
Each member is an InferenceGatewayConfig resource on a HyperPod cluster. You identify it by three values:
-
ClusterArn: the ARN of the HyperPod cluster. -
Namespace: the namespace of theInferenceGatewayConfig. -
Name: the name of theInferenceGatewayConfig.
The InferenceGatewayConfig routes to the model-serving pods that its schedulers select. For how to configure it, see Inference Gateway for Amazon SageMaker HyperPod Inference.
To list the in-cluster inference gateways on a cluster with their namespaces, run:
kubectl get inferencegatewayconfig --all-namespaces
If you deploy models with the HyperPod Inference Operator and set spec.inferenceGateway.enabled: true, the operator creates the InferenceGatewayConfig for you, in the model's namespace. Its name is spec.inferenceGateway.name, or inf-igw-<uuid> if you don't set one. Set the name explicitly so that it's predictable and the same on every cluster.
Confirm the in-cluster gateway serves your model
Before you turn on global routing, send a request to the in-cluster inference gateway's own endpoint and confirm that it answers for your model, as described in Invoke the gateway. If this request fails, requests through the Global Inference Gateway fail too. Resolve any issues before you continue.
1a. Grant the gateway controller the global routing permissions
When global routing is turned on, the inference gateway controller creates a VPC endpoint service in front of an internal Network Load Balancer, and uploads the gateway's certificate to your TLS bucket. After you add the in-cluster gateway to a Global Inference Gateway, the controller also creates an interface VPC endpoint and a security group for it. The metrics publisher sends metrics through this endpoint. The controller uses the same IAM role that it uses for TLS auto-issue (the role passed as inferenceGateway.serviceAccount.roleArn when you install the add-on).
Attach the AmazonSageMakerHyperPodInferenceGatewayGlobalRoutingAccess AWS managed policy to that role. Keep the AmazonSageMakerHyperPodInferenceGatewayAccess policy that the role already has for TLS auto-issue.
aws iam attach-role-policy --role-name <controller-role-name> \ --policy-arn arn:aws:iam::aws:policy/AmazonSageMakerHyperPodInferenceGatewayGlobalRoutingAccess
The policy lets the controller do the following, only in your account:
-
Endpoint service: Create and manage the VPC endpoint service in front of the gateway's Network Load Balancer, and let a Global Inference Gateway in another Region connect to it. The controller uses
ec2:DescribeRegionsto include any opt-in Regions that your account has enabled. -
Metrics endpoint: Create the interface VPC endpoint for Amazon Kinesis Data Streams, and its security group, so that the metrics publisher can send metrics. The controller can modify or delete only the endpoints and security groups that it created. Without these permissions, the controller can't create the metrics endpoint, so it doesn't start the metrics publisher, and the member doesn't receive traffic.
-
Certificate: Upload the gateway's certificate to your TLS bucket, and use your AWS KMS key through Amazon S3 if the bucket is encrypted with a customer managed key. The controller uploads the certificate every 60 to 75 minutes, so a certificate that's deleted or overwritten is restored within that time. If you change
tlsCertificateS3Bucket, the controller uses the new bucket after its next restart. -
Cluster lookup: Describe your HyperPod cluster, to find its subnets.
The policy's Amazon S3 permissions apply only to buckets whose names begin with hyperpod-tls. If your TLS bucket has a different name, also add the following statements to the controller role's permissions policy. Replace <tls-bucket> with your bucket name.
[ { "Sid": "WriteGlobalRoutingCertificate", "Effect": "Allow", "Action": ["s3:PutObject", "s3:PutObjectTagging"], "Resource": "arn:aws:s3:::<tls-bucket>/*gig/*/*-certificate-*.pem", "Condition": { "StringEquals": { "s3:RequestObjectTag/CreatedBy": "HyperPodInference", "aws:ResourceAccount": "${aws:PrincipalAccount}" }, "ForAllValues:StringEquals": { "s3:RequestObjectTagKeys": ["CreatedBy"] } } }, { "Sid": "DeleteGlobalRoutingCertificate", "Effect": "Allow", "Action": "s3:DeleteObject", "Resource": "arn:aws:s3:::<tls-bucket>/*gig/*/*-certificate-*.pem", "Condition": { "StringEquals": { "aws:ResourceAccount": "${aws:PrincipalAccount}" } } } ]
1b. Create the metrics publisher role
A metrics publisher in the cluster sends the gateway's load and health to the Global Inference Gateway. The Global Inference Gateway uses these metrics to discover which models each member serves, so it doesn't route to a member until its publisher is running. The publisher runs as the Kubernetes service account hyperpod-inference-system/inference-gateway-metrics-publisher and uses its own IAM role through IAM roles for service accounts (IRSA).
To create the metrics publisher role
-
Set the following variables, and get your cluster's OIDC issuer.
export CLUSTER=<eks-cluster-name> export REGION=<region> export ACCOUNT=<account-id> export OIDC_ID=$(aws eks describe-cluster --name $CLUSTER --region $REGION \ --query 'cluster.identity.oidc.issuer' --output text | sed 's|https://||') -
Create a trust policy that lets the publisher's service account assume the role.
cat > publisher-trust-policy.json <<EOF { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam::${ACCOUNT}:oidc-provider/${OIDC_ID}" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "${OIDC_ID}:sub": "system:serviceaccount:hyperpod-inference-system:inference-gateway-metrics-publisher", "${OIDC_ID}:aud": "sts.amazonaws.com" } } }] } EOF -
Create the role, and attach the
AmazonSageMakerHyperPodInferenceGatewayMetricsPublisherAccessAWS managed policy. The metrics stream belongs to the Global Inference Gateway and is encrypted with an AWS KMS key that the service manages. The policy allowskinesis:PutRecordsto write metrics, andkms:GenerateDataKeythrough Kinesis so that Kinesis can encrypt them.aws iam create-role --role-name AmazonSageMakerInferenceGatewayMetricsPublisherRole \ --assume-role-policy-document file://publisher-trust-policy.json aws iam attach-role-policy --role-name AmazonSageMakerInferenceGatewayMetricsPublisherRole \ --policy-arn arn:aws:iam::aws:policy/AmazonSageMakerHyperPodInferenceGatewayMetricsPublisherAccess
1c. Update the add-on
Update the HyperPod Inference add-on with the metrics publisher role.
--configuration-values replaces the add-on's whole configuration, so any value you leave out goes back to its default. Include every value the add-on already has, such as the operator execution role, the TLS bucket, KEDA, the AWS Load Balancer Controller, and the JumpStart role. If the cluster already has a TLS bucket, keep using it.
-
Read the add-on's current configuration.
aws eks describe-addon --cluster-name $CLUSTER --region $REGION \ --addon-name amazon-sagemaker-hyperpod-inference \ --query addon.configurationValues --output text -
Update the add-on with those values, plus
inferenceGateway.metricsPublisher.aws eks update-addon \ --cluster-name $CLUSTER --region $REGION \ --addon-name amazon-sagemaker-hyperpod-inference \ --addon-version v3.0.0-eksbuild.1 \ --resolve-conflicts OVERWRITE \ --configuration-values '{ "executionRoleArn": "arn:aws:iam::<account-id>:role/<operator-execution-role>", "tlsCertificateS3Bucket": "<tls-bucket>", "inferenceOperator": { "enabled": true }, "inferenceGateway": { "enabled": true, "serviceAccount": { "roleArn": "arn:aws:iam::<account-id>:role/<cert-issuer-role>" }, "metricsPublisher": { "serviceAccount": { "roleArn": "arn:aws:iam::<account-id>:role/AmazonSageMakerInferenceGatewayMetricsPublisherRole" } } }, "alb": { "enabled": true, "serviceAccount": { "create": true, "roleArn": "arn:aws:iam::<account-id>:role/<alb-irsa-role>" } } }' -
When the update finishes, confirm that the add-on is
ACTIVEwith no health issues, that the controller has rolled out, and that the in-cluster gateway still answers.aws eks describe-addon --cluster-name $CLUSTER --region $REGION \ --addon-name amazon-sagemaker-hyperpod-inference \ --query 'addon.{version:addonVersion,status:status,health:health.issues}' kubectl rollout status deploy/inference-gateway-controller -n hyperpod-inference-system --timeout=150s
1d. Turn on global routing
Set spec.globalRouting.enabled to true on the InferenceGatewayConfig that you want to add as a member.
apiVersion: inference.sagemaker.aws.amazon.com/v1alpha1 kind: InferenceGatewayConfig metadata: name: my-gateway namespace: my-namespace spec: bbr: enabled: true tls: {} globalRouting: enabled: true schedulers: - name: llama modelName: meta-llama/Llama-3.1-8B-Instruct modelSelector: matchLabels: app: llama targetPort: 8000
To turn it on for an existing configuration, run:
kubectl -n my-namespace patch inferencegatewayconfig my-gateway --type merge \ -p '{"spec":{"globalRouting":{"enabled":true}}}'
No additional access configuration is required. The controller configures the endpoint service to accept connections only from the SageMaker Global Inference Gateway service, in every supported Region that your account can use.
1e. Verify
Wait until the configuration reports Ready for global routing.
To check the global routing status, run the following command:
kubectl -n my-namespace get inferencegatewayconfig my-gateway
The output should look similar to the following:
NAME READY GLOBAL-ROUTING VPCE-SERVICE AGE my-gateway True Ready com.amazonaws.vpce.us-west-2.vpce-svc-0123456789abcdef0 5m
If GLOBAL-ROUTING doesn't reach Ready, see Global Inference Gateway troubleshooting guide.
The metrics publisher starts only after you add the in-cluster gateway to a Global Inference Gateway (Step 3). Until then, the MetricsPublisherReady condition doesn't apply.
Step 2: Create the execution role
The Global Inference Gateway uses an execution role in your account to read and update the configuration of each member cluster. It never accesses your clusters with its own identity. The role, the Global Inference Gateway, and every member cluster must be in the same AWS account.
2a. Create the role
Create a trust policy that allows the Global Inference Gateway service to assume the role only for Global Inference Gateways in your account.. The aws:SourceAccount and aws:SourceArn conditions prevent another account's Global Inference Gateway from using your role (the confused deputy problem).
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "global-inference-gateway.sagemaker.amazonaws.com" }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "aws:SourceAccount": "<account-id>" }, "ArnLike": { "aws:SourceArn": "arn:aws:sagemaker:*:<account-id>:global-inference-gateway/*" } } } ] }
After you create the Global Inference Gateway, you can restrict the role to it by replacing global-inference-gateway/* with global-inference-gateway/<gateway-id>. Keep the wildcard until then: the ID is assigned at creation, and the Global Inference Gateway assumes the role while it is being created.
2b. Attach the permissions policy
Attach the AmazonSageMakerGlobalInferenceGatewayExecutionAccess AWS managed policy to the execution role.
aws iam attach-role-policy --role-name <execution-role-name> \ --policy-arn arn:aws:iam::aws:policy/AmazonSageMakerGlobalInferenceGatewayExecutionAccess
The policy lets the role describe HyperPod clusters, call the Kubernetes API of their EKS clusters, and read the gateway certificates from your TLS bucket, only in your account. Kubernetes RBAC limits what the role can do in each cluster (see Step 2c).
Use an inline policy instead if your TLS bucket name doesn't begin with hyperpod-tls, or if you want to limit the role to specific clusters. Scope each statement to your member clusters and TLS bucket.
{ "Version": "2012-10-17", "Statement": [ { "Sid": "DescribeMemberCluster", "Effect": "Allow", "Action": "sagemaker:DescribeCluster", "Resource": "arn:aws:sagemaker:<cluster-region>:<account-id>:cluster/<cluster-id>" }, { "Sid": "MemberKubernetesApi", "Effect": "Allow", "Action": ["eks:AccessKubernetesApi", "eks:MutateViaKubernetesApi"], "Resource": "arn:aws:eks:<cluster-region>:<account-id>:cluster/<eks-cluster-name>" }, { "Sid": "ReadGatewayCertificate", "Effect": "Allow", "Action": "s3:GetObject", "Resource": "arn:aws:s3:::<tls-bucket>/gig/*", "Condition": { "StringEquals": { "s3:ExistingObjectTag/CreatedBy": "HyperPodInference" } } } ] }
2c. Grant the role access to each member's Kubernetes API
On each member's EKS cluster, create an access entry that maps the execution role to the Kubernetes group sagemaker-global-inference-gateway. The cluster's authentication mode must include API (API or API_AND_CONFIG_MAP). Don't attach any access policies. The HyperPod Inference add-on already grants this group the minimum permissions that the Global Inference Gateway needs:
-
read access to
InferenceGatewayConfigresources -
write access to the ConfigMap that records gateway membership
aws eks create-access-entry \ --cluster-name <eks-cluster-name> --region <cluster-region> \ --principal-arn arn:aws:iam::<account-id>:role/<execution-role-name> \ --type STANDARD \ --kubernetes-groups sagemaker-global-inference-gateway
If create-access-entry fails with an invalid principal error right after you create the role, wait a few seconds for the role to propagate in IAM, and then run it again.
Step 3: Create a Global Inference Gateway
Create the Global Inference Gateway with one or more members, up to 10 (see Quotas and limits). Each member is the HyperPod cluster ARN, the namespace, and the name of an InferenceGatewayConfig that has global routing turned on.
aws sagemaker create-global-inference-gateway \ --region us-west-2 \ --name my-global-gateway \ --execution-role-arn arn:aws:iam::<account-id>:role/<execution-role-name> \ --gateways '[ {"ClusterArn": "arn:aws:sagemaker:us-west-2:<account-id>:cluster/<cluster-id-1>", "Namespace": "my-namespace", "Name": "my-gateway"}, {"ClusterArn": "arn:aws:sagemaker:us-east-1:<account-id>:cluster/<cluster-id-2>", "Namespace": "my-namespace", "Name": "my-gateway"} ]'
The response returns immediately with Status set to CREATING.
{ "Arn": "arn:aws:sagemaker:us-west-2:<account-id>:global-inference-gateway/k3x9p2m7q1ab", "Id": "k3x9p2m7q1ab", "Name": "my-global-gateway", "Status": "CREATING", "ExecutionRoleArn": "arn:aws:iam::<account-id>:role/<execution-role-name>", "Gateways": [ ... ] }
Optionally, pass --routing to set your own routing weights. If you omit it, the default weights are used. For details, see Routing behavior.
Wait for the Global Inference Gateway to become active
Creation typically takes 5 to 10 minutes, and can take up to 20 minutes. To check the status, run describe-global-inference-gateway with the Global Inference Gateway's name, ID, or ARN.
aws sagemaker describe-global-inference-gateway --region us-west-2 --identifier my-global-gateway
When the Global Inference Gateway is ready, Status is ACTIVE and the response includes the EndpointUrl. The output should look similar to the following:
{ "Arn": "arn:aws:sagemaker:us-west-2:<account-id>:global-inference-gateway/k3x9p2m7q1ab", "Id": "k3x9p2m7q1ab", "Name": "my-global-gateway", "ExecutionRoleArn": "arn:aws:iam::<account-id>:role/<execution-role-name>", "Status": "ACTIVE", "EndpointUrl": "https://k3x9p2m7q1ab.<...>.us-west-2.on.aws", "VpcEndpointServiceName": "com.amazonaws.vpce.us-west-2.vpce-svc-0123456789abcdef0", "CreationTime": "2026-11-12T23:00:00Z" }
Routing is returned only if you set weights at create or update.
When Status is ACTIVE, check each member's status:
aws sagemaker list-inference-gateways --region us-west-2 --identifier my-global-gateway
{ "InferenceGatewaySummaries": [ {"ClusterArn": "arn:aws:sagemaker:us-west-2:<account-id>:cluster/<cluster-id-1>", "Namespace": "my-namespace", "Name": "my-gateway", "Status": "ACTIVE"}, {"ClusterArn": "arn:aws:sagemaker:us-east-1:<account-id>:cluster/<cluster-id-2>", "Namespace": "my-namespace", "Name": "my-gateway", "Status": "ACTIVE"} ] }
A member becomes ACTIVE only after the Global Inference Gateway has reached it over its private connection. If a member can't be reached, it is marked FAILED with a FailureReason, and the Global Inference Gateway stays ACTIVE and serves traffic from the remaining members.
After the Global Inference Gateway becomes ACTIVE, it can take up to about 2 minutes before the first request is routed, while the members' first metrics arrive. Until then, requests return ServiceUnavailableException (503).
Verify your setup
Use the following checks to confirm that each part of the setup worked. Run the kubectl checks against each member cluster, and use each cluster's Region in the aws eks check.
| Check | How to verify | Expected result |
|---|---|---|
| Each in-cluster gateway is ready for global routing (Step 1) | kubectl -n my-namespace get inferencegatewayconfig my-gateway |
READY is True, GLOBAL-ROUTING is Ready, and VPCE-SERVICE is set |
| The execution role can reach each member cluster (Step 2) | aws eks describe-access-entry --cluster-name <eks-cluster-name> --region <cluster-region> --principal-arn arn:aws:iam::<account-id>:role/<execution-role-name> |
kubernetesGroups is ["sagemaker-global-inference-gateway"] and type is STANDARD |
| The Global Inference Gateway is active (Step 3) | aws sagemaker describe-global-inference-gateway --region us-west-2 --identifier my-global-gateway |
Status is ACTIVE, and EndpointUrl is set |
| Every member is active (Step 3) | aws sagemaker list-inference-gateways --region us-west-2 --identifier my-global-gateway |
Every member's Status is ACTIVE |
| Each member's metrics publisher is running | kubectl -n my-namespace get inferencegatewayconfig my-gateway -o jsonpath='{.status.conditions[?(@.type=="MetricsPublisherReady")].status}' |
True |
If a check fails, see Global Inference Gateway troubleshooting guide.
Step 4: Invoke the Global Inference Gateway
Send requests to the EndpointUrl from describe-global-inference-gateway. Every request must be signed with SigV4 for the service name sagemaker in the Global Inference Gateway's Region, by an IAM principal in the account that owns the Global Inference Gateway.
IAM permissions for callers
Callers must be IAM principals in the account that owns the Global Inference Gateway. Requests from other accounts are denied, even if their IAM policies allow them. Grant callers the following actions on the Global Inference Gateway's ARN.
| Action | Used by |
|---|---|
sagemaker:InvokeGlobalInferenceGateway |
The InvokeGlobalInferenceGateway operation, and OpenAI-compatible requests that don't set "stream": true |
sagemaker:InvokeGlobalInferenceGatewayWithResponseStream |
The InvokeGlobalInferenceGatewayWithResponseStream operation, and OpenAI-compatible requests that set "stream": true |
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "sagemaker:InvokeGlobalInferenceGateway", "sagemaker:InvokeGlobalInferenceGatewayWithResponseStream" ], "Resource": "arn:aws:sagemaker:<region>:<account-id>:global-inference-gateway/<gateway-id>" } ] }
Option 1: OpenAI-compatible API
The Global Inference Gateway accepts OpenAI-compatible requests on the following paths and passes them to the member unchanged:
-
POST /v1/chat/completions -
POST /v1/completions -
POST /v1/embeddings
The following example uses curl's built-in SigV4 signing.
curl https://k3x9p2m7q1ab.<...>.us-west-2.on.aws/v1/chat/completions \ --aws-sigv4 "aws:amz:us-west-2:sagemaker" \ --user "$AWS_ACCESS_KEY_ID:$AWS_SECRET_ACCESS_KEY" \ -H "x-amz-security-token: $AWS_SESSION_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "model": "meta-llama/Llama-3.1-8B-Instruct", "messages": [{"role": "user", "content": "Hello"}] }'
Set "stream": true in the body to receive a streamed response. The Global Inference Gateway passes streamed chunks through as the model produces them.
On the OpenAI-compatible paths, errors produced by the Global Inference Gateway are returned with the HTTP status as {"error": "<message>", "request_id": "<id>"}. Authentication and authorization errors use {"error": {"type": "<type>", "message": "<message>", "code": "<code>"}, "request_id": "<id>"}. Error responses from the member are returned unchanged. The typed exceptions and message prefixes in Error origin and Invocation errors apply only to the InvokeGlobalInferenceGateway operations.
Option 2: AWS SDK (InvokeGlobalInferenceGateway)
The SageMaker Gateway Runtime API provides two operations:
-
InvokeGlobalInferenceGateway(POST /global-inference-gateway/{GatewayId}/invocations) returns the whole response. The response can be up to 10 MiB. -
InvokeGlobalInferenceGatewayWithResponseStream(POST /global-inference-gateway/{GatewayId}/invocations-response-stream) returns the response as a stream ofPayloadPartevents.
Both take the gateway ID, a Content-Type of application/json, and a JSON body in the chat completions format with a model field. The service sets stream in the body for you to match the operation.
Send both operations to your Global Inference Gateway's EndpointUrl: pass it as --endpoint-url in the AWS CLI or endpoint_url in the AWS SDK for Python (Boto3).
aws sagemaker-gateway-runtime invoke-global-inference-gateway \ --region us-west-2 \ --endpoint-url https://k3x9p2m7q1ab.<...>.us-west-2.on.aws \ --gateway-id k3x9p2m7q1ab \ --content-type application/json \ --body '{"model":"meta-llama/Llama-3.1-8B-Instruct","messages":[{"role":"user","content":"Hello"}]}' \ --cli-binary-format raw-in-base64-out \ response.json
import json import boto3 client = boto3.client( "sagemaker-gateway-runtime", region_name="us-west-2", endpoint_url="https://k3x9p2m7q1ab.<...>.us-west-2.on.aws", ) response = client.invoke_global_inference_gateway_with_response_stream( GatewayId="k3x9p2m7q1ab", ContentType="application/json", Body=json.dumps({ "model": "meta-llama/Llama-3.1-8B-Instruct", "messages": [{"role": "user", "content": "Hello"}], }), ) for event in response["Body"]: if "PayloadPart" in event: print(event["PayloadPart"]["Bytes"].decode(), end="") elif "InternalStreamFailure" in event: print(event) break
The PayloadPart bytes, joined in order, are the model server's OpenAI-format server-sent events: data: {…} lines with chat.completion.chunk objects, ending with data: [DONE]. A line can span two parts, so buffer the bytes and split on newlines before you parse each data: line as JSON and read choices[0].delta.content.
If the stream fails after it has started, the last event is an InternalStreamFailure (the connection to the member ended, a time limit was reached, or the member was taken out of service), and the stream ends. ModelStreamError is reserved and is not currently sent.
Both operations always send the request to the member's /v1/chat/completions path. To call /v1/completions or /v1/embeddings, use the OpenAI-compatible paths.
Pass an end-user token to your clusters
If your in-cluster inference gateways use JWT authentication (spec.auth.jwt), send the end user's token in the X-Amzn-SageMaker-Customer-Identity-Token header, and include that header in the SigV4 signature. Send the token value only, without a Bearer prefix. The Global Inference Gateway uses the Authorization header for SigV4 and removes it before forwarding, so the end-user token reaches the cluster only in X-Amzn-SageMaker-Customer-Identity-Token. The header can appear only once: a repeated header returns ValidationException (400), and a header that isn't in the signature returns AccessDeniedException (403). If the cluster rejects the token with 401 or 403, the request is not retried on another member. The SDK operations return AccessDeniedException (403) with the member's status in the message. The OpenAI-compatible paths return the member's response unchanged.
Invoke privately with AWS PrivateLink
To call the Global Inference Gateway from your VPC without going over the internet, create an interface VPC endpoint to the service named in VpcEndpointServiceName. Create the endpoint in the account that owns the Global Inference Gateway. Endpoints in other accounts can't connect.
aws ec2 create-vpc-endpoint --region us-west-2 \ --vpc-endpoint-type Interface \ --vpc-id <vpc-id> \ --service-name com.amazonaws.vpce.us-west-2.vpce-svc-0123456789abcdef0 \ --subnet-ids <subnet-id-1> <subnet-id-2> \ --security-group-ids <security-group-id>
Choose subnets in Availability Zones that the endpoint service supports. To list them, run aws ec2 describe-vpc-endpoint-services --service-names <service-name>. The endpoint's security group must allow inbound TCP 443 from every CIDR block your callers use. If your VPC has secondary CIDR blocks, which pods often use, include them, or requests time out.
Keep sending requests to EndpointUrl, signed as usual, and resolve its hostname to the endpoint inside your VPC. For example, create a Route 53 private hosted zone named after the full EndpointUrl hostname, associate it with your VPC, and add an alias record that points to the endpoint's DNS name. Don't send requests to the endpoint's own DNS name. The TLS certificate is issued for the EndpointUrl hostname, so HTTPS clients reject that connection.
After the Global Inference Gateway becomes ACTIVE, wait about a minute before you send the first request to EndpointUrl. A DNS lookup made before the name was published can be cached as not found for a short time.
Manage a Global Inference Gateway
Add a member
Add an in-cluster inference gateway from another HyperPod cluster to serve traffic through the Global Inference Gateway. A Global Inference Gateway can have up to 10 members. To add more, remove a member first or create another Global Inference Gateway.
aws sagemaker add-inference-gateway --region us-west-2 \ --identifier my-global-gateway \ --cluster-arn arn:aws:sagemaker:eu-west-1:<account-id>:cluster/<cluster-id-3> \ --namespace my-namespace \ --name my-gateway
The member starts in ADDING and moves to ACTIVE or FAILED. Check it with list-inference-gateways. Before you add a member, complete Step 1 on its cluster and add its cluster to the execution role (Step 2b and 2c). Adding a member that is already in this Global Inference Gateway succeeds and returns its current status.
Remove a member
Remove an in-cluster inference gateway so that the Global Inference Gateway stops sending it traffic.
aws sagemaker remove-inference-gateway --region us-west-2 \ --identifier my-global-gateway \ --cluster-arn arn:aws:sagemaker:eu-west-1:<account-id>:cluster/<cluster-id-3> \ --namespace my-namespace \ --name my-gateway
The member moves to REMOVING. Within about 15 seconds new requests stop going to it. Its private connection is then deleted, so requests still in flight to that member, for example long streaming responses, can be cut off. The member then disappears from list-inference-gateways. If the removal can't complete, the member shows FAILED with a FailureReason; call remove-inference-gateway again. You can't remove the last member; a Global Inference Gateway always keeps at least one. A FAILED member counts toward this, and a member that is already REMOVING doesn't. Removing an in-cluster gateway that isn't a member succeeds and does nothing.
After a member is removed, you can turn off global routing on its InferenceGatewayConfig, or add it to another Global Inference Gateway.
Remove the member before you turn off global routing, delete the InferenceGatewayConfig, or delete the last model that uses an operator-managed gateway. While a Global Inference Gateway is still connected, the in-cluster gateway can't release its endpoint service: the configuration reports GlobalRoutingDeleteBlocked and its deletion waits until the member is removed.
Change routing weights
You can update the routing weights to change how the Global Inference Gateway scores members. There are four weights, each between 0 and 1:
-
KVHeadroom(default 0.20): favors members with more free KV cache. -
QueueDepth(default 0.25): favors members with fewer waiting requests. -
RegionProximity(default 0.40): favors members in the same Region as the Global Inference Gateway. -
ErrorRate(default 0.15): favors members with fewer recent errors.
aws sagemaker update-global-inference-gateway --region us-west-2 \ --identifier my-global-gateway \ --routing '{"Weights": {"KVHeadroom": 0.3, "QueueDepth": 0.3, "RegionProximity": 0.2, "ErrorRate": 0.2}}'
The Global Inference Gateway moves to UPDATING and returns to ACTIVE when the new weights are in effect, usually within a few minutes. Requests keep being served during the update. To go back to the default weights, pass {"Weights": {}}.
If the update fails, the Global Inference Gateway returns to ACTIVE, keeps serving with its previous weights, and describe-global-inference-gateway shows a FailureReason. Send the same update again to retry. Updating to the weights already in effect returns immediately and changes nothing.
List Global Inference Gateways
To list the Global Inference Gateways in your account and Region, run list-global-inference-gateways. To list only those in a given status, add a Status filter. The valid values are CREATING, ACTIVE, UPDATING, DELETING, and FAILED.
aws sagemaker list-global-inference-gateways --region us-west-2 \ --filters Name=Status,Value=ACTIVE
Delete a Global Inference Gateway
To delete a Global Inference Gateway, use:
aws sagemaker delete-global-inference-gateway --region us-west-2 --identifier my-global-gateway
The Global Inference Gateway moves to DELETING, and SageMaker AI removes its endpoint and routing resources. When deletion finishes, its members are released and can join another Global Inference Gateway, its name can be reused, and describe-global-inference-gateway returns ResourceNotFound. Deleting doesn't change your InferenceGatewayConfig resources or your models. SageMaker AI only removes this Global Inference Gateway's entry from the gig-membership ConfigMap on each member cluster, which stops that member's metrics publisher. Calling delete again while it is DELETING has no effect; if the delete fails, call delete again to retry it.
To stop exposing an in-cluster gateway after you delete the Global Inference Gateway, turn off global routing on its InferenceGatewayConfig.
A Global Inference Gateway accepts one change at a time. While a create, update, add, remove, or delete is in progress, other change requests return ConflictException. Wait for the current change to finish, then retry.
Clean up
To stop using a Global Inference Gateway and remove the resources you created for it, follow these steps in order.
Delete the Global Inference Gateway before you turn off global routing. While a member is still connected, the in-cluster gateway can't release its endpoint service, and the configuration stays at GlobalRoutingDeleteBlocked.
-
Delete the Global Inference Gateway. Its members are released, and SageMaker AI removes the endpoint and routing resources.
aws sagemaker delete-global-inference-gateway --region us-west-2 --identifier my-global-gatewayWait until
describe-global-inference-gatewayreturnsResourceNotFound. This usually takes 7 to 8 minutes. -
On each member cluster, turn off global routing. The controller deletes the metrics publisher and its VPC endpoint, the VPC endpoint service, the internal Network Load Balancer, and the certificates it uploaded to your TLS bucket.
kubectl -n my-namespace patch inferencegatewayconfig my-gateway --type merge \ -p '{"spec":{"globalRouting":{"enabled":false}}}'Wait until the
GLOBAL-ROUTINGcolumn ofkubectl get inferencegatewayconfigis empty. -
On each member cluster, delete the execution role's access entry.
aws eks delete-access-entry --cluster-name <eks-cluster-name> --region <cluster-region> \ --principal-arn arn:aws:iam::<account-id>:role/<execution-role-name> -
Delete the execution role. If you used an inline policy in Step 2b, delete it with
aws iam delete-role-policyinstead of detaching the managed policy.aws iam detach-role-policy --role-name <execution-role-name> \ --policy-arn arn:aws:iam::aws:policy/AmazonSageMakerGlobalInferenceGatewayExecutionAccess aws iam delete-role --role-name <execution-role-name> -
Update the add-on to remove
inferenceGateway.metricsPublisher, keeping your other values as in Step 1c. Then delete the metrics publisher role.aws iam detach-role-policy --role-name AmazonSageMakerInferenceGatewayMetricsPublisherRole \ --policy-arn arn:aws:iam::aws:policy/AmazonSageMakerHyperPodInferenceGatewayMetricsPublisherAccess aws iam delete-role --role-name AmazonSageMakerInferenceGatewayMetricsPublisherRole -
Detach the global routing policy from the controller role, and remove any bucket statements that you added in Step 1a.
aws iam detach-role-policy --role-name <controller-role-name> \ --policy-arn arn:aws:iam::aws:policy/AmazonSageMakerHyperPodInferenceGatewayGlobalRoutingAccess
Keep the role itself, because the in-cluster gateway still uses it for TLS certificates. Keep your TLS bucket for the same reason.
Routing behavior
This section describes how the Global Inference Gateway decides where each request goes, how often it updates member metrics, and what happens when a member is slow, overloaded, or failing.
What the Global Inference Gateway knows about each member
Each member's in-cluster inference gateway runs a metrics publisher that reports, for every model it serves, the following signals to the Global Inference Gateway. The publisher sends a report every 5 seconds by default.
| Signal | Source in the cluster | How it is aggregated |
|---|---|---|
| Ready pods | Endpoint Picker (EPP) for the model | The highest value reported by any EPP replica |
| KV cache utilization | Model servers (vLLM, SGLang), through the EPP | Average across the model's serving pods, from 0 to 1 |
| Queue depth | Model servers, through the EPP | Average number of waiting requests per serving pod |
| Request count | In-cluster gateway (Envoy) and EPP | Total requests the member received for the model |
| Error count | In-cluster gateway (Envoy) and EPP | Server errors (5xx) plus requests the EPP rejected because it couldn't serve them |
| Served models and adapters | InferenceGatewayConfig schedulers and LoRA adapters | The base models and adapter names the member can answer |
| Component health | EPP and the global routing listener | Whether each component was reachable when the report was taken |
Request and error counts are used only when every serving pod was scraped successfully.
The Global Inference Gateway reads the latest reports every 15 seconds and computes each member's error rate over the last 5 minutes.
The Global Inference Gateway learns which models a member serves only from these reports. A member that has never sent a report receives no traffic, even if its membership status is ACTIVE. If a member's publisher stops after it has reported, the member keeps receiving traffic, scored on its last reported load, for up to 30 minutes.
Which members can receive a request
For each request, a member is a candidate only if all of the following are true:
-
It serves the model named in the request's
modelfield, either as a base model or as a LoRA adapter. -
Its membership status is
ACTIVE. -
It reports at least one ready pod for that model.
-
Its components are healthy. A member is excluded after two consecutive reports in which its Endpoint Picker or global routing listener was unreachable, and is included again when it reports healthy.
-
It has reported KV cache utilization and queue depth, unless that signal's routing weight is 0. A member that hasn't reported its error rate yet can still receive traffic.
-
It has sent metrics in the last 30 minutes.
KV cache utilization and queue depth come from your model servers' metrics. Use vLLM v0.9.2 or later, or SGLang v0.3.5.post1 or later started with --enable-metrics. Earlier versions don't publish these metrics in the form the in-cluster inference gateway reads, so the member doesn't receive traffic from the Global Inference Gateway. For details, see Inference Gateway for Amazon SageMaker HyperPod Inference. If no member serves the model, the request fails with ResourceNotFoundException (404). If members serve the model but none can take traffic right now, it fails with ServiceUnavailableException (503).
How candidates are scored
Each candidate gets a score between 0 and 1. The Global Inference Gateway sends the request to the highest score. When several candidates tie, one is picked at random, so equal members share traffic evenly.
The score is a weighted average of four signal values:
score = (wKV × KVHeadroom + wQueue × QueueDepth + wRegion × RegionProximity + wError × ErrorRate) ÷ (wKV + wQueue + wRegion + wError)
The default weights are KVHeadroom 0.20, QueueDepth 0.25, RegionProximity 0.40, and ErrorRate 0.15.
If a member hasn't reported its error rate yet, for example because it is new, that signal is left out and the remaining weights are rescaled.
Choosing routing weights
Each weight controls how much one signal counts when the Global Inference Gateway picks a member. Raise a weight to make that signal matter more.
-
KVHeadroom: A higher value favors members with more free KV cache. This helps with long prompts or long responses, which use more cache. -
QueueDepth: A higher value favors members with fewer waiting requests. This helps when response time matters most. -
RegionProximity: A higher value keeps traffic in the Global Inference Gateway's Region, and uses other Regions only when local members are busy. Set it to 0 to route on spare capacity alone, regardless of Region. -
ErrorRate: A higher value steers traffic away from members that have recently returned errors.
Set all four weights, each between 0 and 1, so that they add up to 1.0. A weight of 0 turns that signal off. Weight changes take effect without dropping requests.
RegionProximity compares each member's Region with the Region of the Global Inference Gateway, not with the Region of the client.
Metric freshness
-
New members. A member starts receiving traffic only after its first metrics arrive. Right after a Global Inference Gateway becomes
ACTIVE, or right after you add a member, it can take a few minutes before requests are routed to it. -
Stale metrics. If a member's latest report is more than 120 seconds old, the Global Inference Gateway keeps routing on the member's last known load. After 30 minutes without a report, the member stops receiving traffic until it reports again. Monitor
MetricsPublisherReadyon your clusters so that stale reports don't go unnoticed. -
Removed members. When you remove a member, it stops receiving new requests within a few minutes.
Recovery after errors
When a member that was excluded becomes healthy again, it isn't given its full share of traffic at once. It is brought back in steps of 25%, 50%, 75%, and 100% of its normal share, staying at least 5 minutes at each step. The first step starts as soon as the member reports healthy. If the member's error rate is 10% or higher while it is ramping up, it goes back to the start of the ramp and receives requests only when no other member can take them. A member that already has its full share keeps it, even if its error rate rises.If the member reports no ready pods, reports an unhealthy component, or sends no metrics for 30 minutes, it's excluded again, and it restarts the ramp when it recovers.
What happens when something fails
The following table describes how traffic behaves at each scope of failure, from a single pod to a whole Region.
| Failure | What the Global Inference Gateway does | Effect on your clients |
|---|---|---|
| A model server pod fails | Nothing. The member's in-cluster gateway stops sending requests to the pod and uses its other pods. | None, as long as the member has other ready pods for the model. |
| A model is saturated on one member | The member returns 429 or 503, and the request is retried once on the next-best member. The member's growing queue depth and KV cache usage also lower its score, so new requests shift to other members. | Requests are served by other members. If every member is saturated, clients receive ThrottlingException (429) with Retry-After. |
| A member's Endpoint Picker or global routing listener is down | The member reports the component as unhealthy, and the member is excluded from routing after two consecutive unhealthy reports. | Traffic moves to the other members that serve the model. |
| A member can't be reached (for example, its cluster or network path is down) | Each request that can't connect within 5 seconds is retried once on the next-best member. | Requests still succeed through other members, with added latency for the failed connection attempt. See the note below. |
| A member's global routing listener restarts or moves, for example during an add-on update, a node replacement, or an Availability Zone outage in the cluster's VPC | The listener runs as a single Envoy pod, so the member can't take requests from the Global Inference Gateway until the pod is running again. Requests that can't connect are retried once on the next-best member. | Brief failures or added latency for requests to that member, about 15 seconds during an add-on update. Serve each model from at least two members. |
| A member recovers | The member is brought back gradually, as described in Recovery after errors. | None. |
| All members in one Region fail | Members in other Regions keep their scores and receive the traffic. Region proximity only lowers their score; it never excludes them. | Higher latency for requests that now cross Regions. No configuration change is needed. |
| A model is served by only one member, and that member fails | There is no other member to send the request to. | Requests for that model fail until the member recovers. Serve each model from at least two members to avoid this. |
| An Availability Zone fails in the Global Inference Gateway's Region | The routing fleet runs in multiple Availability Zones, and traffic moves to the healthy ones. | Requests in flight in the failed zone, including open streams, fail and must be retried by the client. |
If a member stops sending metrics entirely, the Global Inference Gateway keeps scoring it on its last reported load, because missing metrics alone are not treated as proof that the member is down. Requests that can't reach it fail over as described in the table. To find members in this state, alarm on MetricsPublisherReady on each cluster and on FAILED members (see Recommended alarms).
Timeouts
| Limit | Value | Result when exceeded |
|---|---|---|
| First response byte from the member | 300 seconds | GatewayTimeoutException (504), not retried |
| Time between streamed chunks | 300 seconds | 504 before the first byte. After it: an InternalStreamFailure event on InvokeGlobalInferenceGatewayWithResponseStream, BadGatewayException (502) on InvokeGlobalInferenceGateway, and a truncated response on the OpenAI-compatible paths |
| Whole request, including the full stream and any retry | 15 minutes | 504 before the first byte. After it: an InternalStreamFailure event on InvokeGlobalInferenceGatewayWithResponseStream, BadGatewayException (502) on InvokeGlobalInferenceGateway, and a truncated response on the OpenAI-compatible paths |
| In-flight requests on a member that is removed or withdrawn | 10 minutes to finish | Streams still open after that end with InternalStreamFailure |
Status reference
Global Inference Gateway status
| Status | Meaning |
|---|---|
CREATING |
SageMaker AI is provisioning the endpoint and connecting to the members. |
ACTIVE |
The Global Inference Gateway is serving. Some members can still be FAILED; check list-inference-gateways. |
UPDATING |
A member is being added or removed, or new routing weights are being applied. Traffic is still served. |
DELETING |
The Global Inference Gateway and its resources are being removed. |
FAILED |
Creation or deletion failed. FailureReason explains why. If creation failed, delete the Global Inference Gateway and create it again after you fix the cause. If deletion failed, call delete-global-inference-gateway again. |
FailureReason can also be set while the status is ACTIVE. It then describes the last add, remove, or routing update that failed; the Global Inference Gateway kept serving.
Member status
The member status describes the membership, not the health of the in-cluster gateway itself.
| Status | Meaning |
|---|---|
ADDING |
The Global Inference Gateway is connecting to the in-cluster gateway and verifying that it can reach it. |
ACTIVE |
The member is connected and eligible for traffic. |
REMOVING |
The member is draining and being disconnected. |
FAILED |
The member couldn't be added, or couldn't be removed. FailureReason explains why. If it couldn't be added, fix the cause, remove the member, and add it again. If it couldn't be removed, call remove-inference-gateway again. |
Monitoring
You can monitor a Global Inference Gateway at three levels:
-
Gateway metrics in Amazon CloudWatch, which the Global Inference Gateway publishes into your account.
-
Resource status from the SageMaker API: the Global Inference Gateway's status and each member's status.
-
Member health from Kubernetes, through the
InferenceGatewayConfigconditions and the in-cluster inference gateway's own metrics.
Gateway metrics in Amazon CloudWatch
The Global Inference Gateway publishes OpenTelemetry metrics to Amazon CloudWatch in the account that owns it, in the Global Inference Gateway's Region. You don't need to turn anything on: metrics start flowing when the Global Inference Gateway starts serving.
-
Publishing interval: every 60 seconds. New data typically becomes queryable within about 6 minutes.
-
Instrumentation scope:
cloudwatch.aws/sagemaker/gateway. -
Query language: PromQL. These metrics are OpenTelemetry metrics, so they don't appear in the classic CloudWatch
ListMetricsAPI or in the classic metrics namespace view. -
Temporality: delta. Each data point is the count or distribution for its own 60-second interval, not a running total. See Query the metrics for what this means for your queries.
Metric reference
| Metric | Type | Unit | Attributes | Description |
|---|---|---|---|---|
aws.sagemaker.gateway.invocations |
Counter | {invocation} |
gateway ID, model name (when resolved), error.type (on failures) |
Every inference request that reached the Global Inference Gateway, successful or not. This is the denominator for error rates. Requests rejected before a model was resolved (for example, an oversized body, a missing model, or a failed authorization) are counted without a model name. |
aws.sagemaker.gateway.duration |
Histogram | s |
gateway ID, model name (when resolved), error.type (on failures) |
End-to-end time from when the request arrived until the response finished, including the full stream. Recorded for failed requests too, so you can see how long failures take. |
aws.sagemaker.gateway.time_to_first_byte |
Histogram | s |
gateway ID, model name | Time from when the request arrived until the Global Inference Gateway received the first part of the member's response body. For streamed responses this is close to the time to the first streamed chunk. For non-streamed responses, including every InvokeGlobalInferenceGateway call, it includes the whole generation. |
aws.sagemaker.gateway.time_to_first_token |
Histogram | s |
gateway ID, model name | Time from when the request arrived until the first chunk that contains a generated token. Recorded only for streamed responses that contain tokens. |
aws.sagemaker.gateway.input_tokens |
Counter | {token} |
gateway ID, model name | Prompt tokens, as reported by the model server in the response's usage block. |
aws.sagemaker.gateway.output_tokens |
Counter | {token} |
gateway ID, model name | Completion tokens, as reported by the model server in the response's usage block. |
aws.sagemaker.gateway.global_router.heartbeat |
Gauge | 1 |
gateway ID | Reports 1 every interval while the Global Inference Gateway's routing fleet is running. It never reports 0: missing data is the unhealthy signal. |
Attributes
Each metric includes the following attributes:
-
aws.sagemaker.gateway.id: the Global Inference Gateway's ID, for example
k3x9p2m7q1ab. Every metric has it, so you can use it to separate gateways in the same account. -
aws.sagemaker.gateway.model.name: the base model name. Requests for a LoRA adapter are reported under the base model that the adapter runs on.
-
error.type:
4xx,5xx, or_OTHER. Only failed requests have it, and only on the invocations and duration metrics. Throttled requests (429) count as4xx. A response that fails after streaming has started counts as5xx._OTHERmeans the model server returned a status code outside the standard HTTP error ranges.
Every metric also carries the resource attributes cloud.provider (aws), cloud.region, cloud.account.id, and cloud.resource_id (the Global Inference Gateway's ARN). In PromQL results these appear with an @resource. prefix, for example @resource.cloud.resource_id.
Token metrics for streamed responses Token counts are published only when the model server returns a usage block. For streamed OpenAI-compatible requests, ask for it by setting "stream_options": {"include_usage": true} in the request body. The Global Inference Gateway doesn't estimate tokens when the model server doesn't report them.
Query the metrics
Metric and attribute names contain dots, so quote them inside the PromQL selector:
{"aws.sagemaker.gateway.invocations", "aws.sagemaker.gateway.id"="k3x9p2m7q1ab"}
Because the data is delta, don't use rate() or increase() on these metrics. Those functions assume an ever-increasing counter and give wrong results here. Sum the interval values over a window instead, and divide by the window length if you need a rate. Always query a range (for example [5m]), not an instant: an instant query can miss the latest 60-second data point.
The following queries are common starting points. Replace the gateway ID.
| Question | PromQL |
|---|---|
| Requests in the last 5 minutes | sum(sum_over_time({"aws.sagemaker.gateway.invocations", "aws.sagemaker.gateway.id"="k3x9p2m7q1ab"}[5m])) |
| Requests per second, by model | sum by ("aws.sagemaker.gateway.model.name") (sum_over_time({"aws.sagemaker.gateway.invocations", "aws.sagemaker.gateway.id"="k3x9p2m7q1ab"}[5m])) / 300 |
| Server error rate | sum(sum_over_time({"aws.sagemaker.gateway.invocations", "aws.sagemaker.gateway.id"="k3x9p2m7q1ab", "error.type"="5xx"}[5m])) / sum(sum_over_time({"aws.sagemaker.gateway.invocations", "aws.sagemaker.gateway.id"="k3x9p2m7q1ab"}[5m])) |
| p99 end-to-end latency | histogram_quantile(0.99, sum by ("aws.sagemaker.gateway.id") (sum_over_time({"aws.sagemaker.gateway.duration", "aws.sagemaker.gateway.id"="k3x9p2m7q1ab"}[5m]))) |
| p50 time to first token, by model | histogram_quantile(0.5, sum by ("aws.sagemaker.gateway.model.name") (sum_over_time({"aws.sagemaker.gateway.time_to_first_token", "aws.sagemaker.gateway.id"="k3x9p2m7q1ab"}[5m]))) |
| Output tokens per minute | sum(sum_over_time({"aws.sagemaker.gateway.output_tokens", "aws.sagemaker.gateway.id"="k3x9p2m7q1ab"}[5m])) / 5 |
| Is the routing fleet reporting? | max(max_over_time({"aws.sagemaker.gateway.global_router.heartbeat", "aws.sagemaker.gateway.id"="k3x9p2m7q1ab"}[5m])) |
You can run these queries in the CloudWatch console's PromQL query editor, or call the PromQL query API directly. The API request is a SigV4-signed POST to https://monitoring.<region>.amazonaws.com/api/v1/query, signed for the service name monitoring, with credentials in the account that owns the Global Inference Gateway:
curl -s https://monitoring.us-west-2.amazonaws.com/api/v1/query \ --aws-sigv4 "aws:amz:us-west-2:monitoring" \ --user "$AWS_ACCESS_KEY_ID:$AWS_SECRET_ACCESS_KEY" \ -H "x-amz-security-token: $AWS_SESSION_TOKEN" \ --data-urlencode 'query=sum(sum_over_time({"aws.sagemaker.gateway.invocations", "aws.sagemaker.gateway.id"="k3x9p2m7q1ab"}[5m]))'
Recommended alarms
Create CloudWatch alarms on the gateway metrics to detect issues early.
-
Heartbeat: alarm when there's no data for 5 minutes, and treat missing data as breaching. This means the routing fleet isn't running.
-
Server error rate: alarm when 5xx invocations divided by all invocations go above your threshold, for example 1% over 5 minutes. This means members are failing, or the Global Inference Gateway can't reach them. To tell which, see Error origin.
-
Throttling: alarm when 4xx invocations rise along with 429 responses in your client. This means every member tried was out of capacity. Add capacity or add a member.
-
Latency: alarm when p99
durationor p50time_to_first_tokengoes above your target. This means members are queueing. Check KV cache and queue depth on the members.
Also check list-inference-gateways for FAILED members, and the MetricsPublisherReady condition on each member cluster.
Resource status
Poll describe-global-inference-gateway for the Global Inference Gateway's Status and FailureReason, and list-inference-gateways for each member's Status and FailureReason. For what each value means, see Status reference.
aws sagemaker list-inference-gateways --region us-west-2 \ --identifier my-global-gateway \ --filters Name=Status,Value=FAILED
Member health in Kubernetes
On each member cluster, the InferenceGatewayConfig reports two conditions that matter for global routing:
kubectl -n my-namespace get inferencegatewayconfig my-gateway \ -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason}: {.message}{"\n"}{end}'
-
GlobalRoutingReady: the private endpoint that the Global Inference Gateway connects to is published and has healthy targets. -
MetricsPublisherReady: the metrics publisher is sending reports to the Global Inference Gateway. While it isFalse, the Global Inference Gateway routes to the member on its last reported load, so it doesn't see load or health changes. After 30 minutes without reports, the member is removed from routing. Direct traffic to the in-cluster gateway is not affected.
The in-cluster inference gateway's own metrics (Endpoint Picker, KV cache, queue depth, model server metrics) are available as described in Observability for the Inference Gateway. They are the same signals the Global Inference Gateway uses to score members, so they are the best place to see why traffic moved between clusters.
Quotas and limits
| Resource | Limit | Notes |
|---|---|---|
| Members per Global Inference Gateway | 1–10 | A Global Inference Gateway always keeps at least one member. |
| Global Inference Gateways per member | 1 | An in-cluster inference gateway can belong to only one Global Inference Gateway at a time. |
| Models per member with global routing turned on | 10 | Global routing supports up to 10 models (schedulers) on each InferenceGatewayConfig. |
| Global Inference Gateway name | 1–63 characters | Letters, digits, and hyphens. Must start and end with a letter or digit. Unique per account and Region. Can't be exactly 12 lowercase letters or digits, because that form is reserved for Global Inference Gateway IDs. |
Global routing creates resources in each cluster's account, and these count against that account's quotas:
-
Turning on global routing creates one internal Network Load Balancer, which counts toward Network Load Balancers per Region (
L-69A177A2). If the quota is reached, theInferenceGatewayConfigstays atGlobalRoutingPending. -
Adding the gateway to a Global Inference Gateway creates one interface VPC endpoint and one security group in the cluster's VPC. These count toward Interface VPC endpoints per VPC and VPC security groups per Region.
To request an increase, use Service Quotas.
Security
-
Authentication and authorization. Every request is signed with SigV4 and authorized with IAM on the Global Inference Gateway's ARN. Only principals in the owning account can invoke it; requests from other accounts are denied even if their IAM policies allow it.
-
Encryption in transit. Clients connect over HTTPS. The Global Inference Gateway connects to each member over a private AWS PrivateLink connection, using TLS. It verifies the member's certificate, which the in-cluster gateway publishes to your TLS bucket, and presents its own client certificate. Each in-cluster gateway accepts only the client certificate of the Global Inference Gateway it belongs to, and refuses all others until it is added. You don't need to configure anything for this.
-
Certificate rotation. Each in-cluster gateway's global routing certificate is rotated automatically. Sixty days before it expires, the controller issues a new certificate and publishes it to your TLS bucket alongside the current one. Seven days after that upload, the gateway starts presenting the new certificate. The old certificate stays in the bucket for one more day, and then it's removed. If the current certificate is within one day of expiry, the gateway switches immediately. No action is needed, but don't delete objects under the
gig/prefix in your TLS bucket. -
Private member connectivity. Members don't need a public endpoint. Each member's VPC endpoint service accepts connections only from the SageMaker Global Inference Gateway service.
-
Least privilege on your clusters. The Global Inference Gateway reaches your clusters only through the execution role you provide, protected against the confused deputy problem, and is limited by Kubernetes RBAC to reading
InferenceGatewayConfigresources and updating a single membership ConfigMap. -
Metrics isolation. Each Global Inference Gateway has its own metrics stream that accepts records only from its owning account.
-
Logging. Request and response bodies, credentials, and end-user tokens are never logged.
Supported Regions
You can create a Global Inference Gateway in the following AWS Regions. Members can be HyperPod clusters in any of these Regions, in the same account as the Global Inference Gateway.
-
US East (N. Virginia)
us-east-1 -
US East (Ohio)
us-east-2 -
US West (N. California)
us-west-1 -
US West (Oregon)
us-west-2 -
South America (São Paulo)
sa-east-1 -
Europe (Frankfurt)
eu-central-1 -
Europe (Ireland)
eu-west-1 -
Europe (London)
eu-west-2 -
Europe (Stockholm)
eu-north-1 -
Asia Pacific (Tokyo)
ap-northeast-1 -
Asia Pacific (Mumbai)
ap-south-1 -
Asia Pacific (Singapore)
ap-southeast-1 -
Asia Pacific (Sydney)
ap-southeast-2
You can't add an in-cluster gateway as a member if its HyperPod cluster is in a Region that requires opt-in.
API reference
SageMaker API (control plane)
| Operation | Description |
|---|---|
CreateGlobalInferenceGateway |
Creates a Global Inference Gateway with 1–10 members. Asynchronous. Inputs: Name, ExecutionRoleArn, Gateways, optional Routing, ClientToken, Tags. |
DescribeGlobalInferenceGateway |
Returns the status, EndpointUrl, VpcEndpointServiceName, Routing, CreationTime, and FailureReason. |
ListGlobalInferenceGateways |
Lists Global Inference Gateways in the account and Region. Filter: Status. |
UpdateGlobalInferenceGateway |
Changes Routing. Asynchronous. |
DeleteGlobalInferenceGateway |
Deletes a Global Inference Gateway. Asynchronous. Idempotent. |
AddInferenceGateway |
Adds one member. Asynchronous. Idempotent. |
RemoveInferenceGateway |
Removes one member. Asynchronous. Idempotent. |
ListInferenceGateways |
Lists the members of a Global Inference Gateway, in the order they were added, with each member's status and failure reason. Filters: Status, ClusterArn. |
All operations that take an Identifier accept the Global Inference Gateway's name, ID, or ARN. Names are case-sensitive.
SageMaker Gateway Runtime API (data plane)
| Operation | Description |
|---|---|
InvokeGlobalInferenceGateway |
Sends an inference request and returns the whole response (up to 10 MiB). |
InvokeGlobalInferenceGatewayWithResponseStream |
Sends an inference request and returns the response as an event stream. |