Amazon SageMaker Gateway Runtime 2026-10-01
- Client: Aws\SageMakerGatewayRuntime\SageMakerGatewayRuntimeClient
- Service ID: sagemaker-gateway-runtime
- Version: 2026-10-01
This page describes the parameters and results for the operations of the Amazon SageMaker Gateway Runtime (2026-10-01), and shows how to use the Aws\SageMakerGatewayRuntime\SageMakerGatewayRuntimeClient object to call the described operations. This documentation is specific to the 2026-10-01 API version of the service.
Operation Summary
Each of the following operations can be created from a client using
$client->getCommand('CommandName'), where "CommandName" is the
name of one of the following operations. Note: a command is a value that
encapsulates an operation and the parameters used to create an HTTP request.
You can also create and send a command immediately using the magic methods
available on a client object: $client->commandName(/* parameters */).
You can send the command asynchronously (returning a promise) by appending the
word "Async" to the operation name: $client->commandNameAsync(/* parameters */).
- InvokeGlobalInferenceGateway ( array $params = [] )
- Buffered (non-streaming) inference.
- InvokeGlobalInferenceGatewayWithResponseStream ( array $params = [] )
- Streaming inference.
Operations
InvokeGlobalInferenceGateway
$result = $client->invokeGlobalInferenceGateway([/* ... */]); $promise = $client->invokeGlobalInferenceGatewayAsync([/* ... */]);
Buffered (non-streaming) inference. Returns the full OpenAI completion body.
Errors carry their origin in the Message prefix. Every typed error's text is in the Message member (JSON key Message), and starts with one of these prefixes:
Global Inference Gateway:-- the gateway rejected or failed the request itself: validation, authorization, throttling, routing (the gateway or model was not found, or no capacity is available), a gateway limit or time limit, or failing to reach the inference gateway (Global Inference Gateway: could not reach the inference gateway: <reason>, orGlobal Inference Gateway: timed out waiting for the inference gateway). This includes the 400 for an invalid request and the 413 for a request body that is too large.inference gateway:-- the inference gateway serving the model returned an error status. The message gives that status code and up to 256 bytes of its error message, for exampleinference gateway: 503 <message>. A BadGatewayException can instead readinference gateway: response ended unexpectedly, with no status, when the inference gateway's connection failed while it was sending the response.inference gateway (<component>):-- the same, when the inference gateway reports which of its components produced the error, for exampleinference gateway (model server): 502 <message>. The component is one of:config-- the inference gateway's configuration and proxy layer.scheduler-- the inference gateway's request scheduler.model selector-- the inference gateway's model selector.model server-- the model server.
Every error status the inference gateway returns is a typed error, chosen by that status:
- 400 and any other 4xx not listed here -- a 400 validation error.
- 401 and 403 -- AccessDeniedException (403).
- 404 -- ResourceNotFoundException.
- 408 -- GatewayTimeoutException (504).
- 429 -- ThrottlingException, with the
Retry-Afterheader. - 502, and 500, 503 or any other 5xx not listed here -- BadGatewayException (502). A 500 or 503 from the inference gateway comes after the request was dispatched, so it is not modeled as retryable, and the gateway may already have tried another inference gateway. Standard SDK retry modes may still retry a 5xx status; set max attempts to 1 if a repeated inference is unacceptable.
- 504 -- GatewayTimeoutException.
Parameter Syntax
$result = $client->invokeGlobalInferenceGateway([
'Accept' => '<string>',
'Body' => <string || resource || Psr\Http\Message\StreamInterface>, // REQUIRED
'ContentType' => '<string>', // REQUIRED
'CustomerIdentityToken' => '<string>',
'GatewayId' => '<string>', // REQUIRED
]);
Parameter Details
Members
- Accept
-
- Type: string
The MIME type you want for the response. It is passed to the model server.
- Body
-
- Required: Yes
- Type: blob (string|resource|Psr\Http\Message\StreamInterface)
The inference request body: one OpenAI JSON object, at most 10 MiB. Its content is passed to the model server unchanged, except that the request is always sent with
"stream": false: this operation returns one JSON document, so a"stream": truein the body is overridden (use InvokeGlobalInferenceGatewayWithResponseStream to stream). A body that is not one JSON object, or that repeats the top-levelmodelkey with different values, is rejected with a 400 validation error. A body larger than 10 MiB is rejected with PayloadTooLargeException (413). The gateway reads onlymodelandstream, and does not store the body. - ContentType
-
- Required: Yes
- Type: string
The MIME type of the request body. Pass
application/json(parameters such ascharset=utf-8are allowed): the body is one OpenAI JSON object. The Global Inference Gateway rejects other content types with a 415 error. - CustomerIdentityToken
-
- Type: string
Per-caller customer identity token (a JWT), forwarded unchanged to the inference gateway, which validates it. The gateway does not use it to authorize the request. Include it in the SigV4 signed headers so it cannot be altered in transit. Redacted from logs, metrics and traces.
- GatewayId
-
- Required: Yes
- Type: string
The ID of the Global Inference Gateway to invoke. IAM policies refer to the gateway by its ARN,
arn:<partition>:sagemaker:<region>:<account>:global-inference-gateway/<id>.
Result Syntax
[
'Body' => <string || resource || Psr\Http\Message\StreamInterface>,
]
Result Details
Members
- Body
-
- Type: blob (string|resource|Psr\Http\Message\StreamInterface)
The model server's OpenAI completion JSON, returned unchanged. A response larger than 10 MiB fails with BadGatewayException (502).
Errors
- GatewayTimeoutException:
504 -- The inference gateway did not respond in time. The
Messageprefix says whose time limit expired:Global Inference Gateway: timed out waiting for the inference gateway-- the gateway's limit on the wait for the first part of the response.inference gateway: <status> <message>-- the inference gateway serving the model returned a 504, or a 408 because it timed out waiting for the request to arrive. Its status code is in the message (the response status is 504), and up to 256 bytes of its error message are quoted.
The inference gateway names a component in the prefix when it reports one:
inference gateway (<component>):, where the component isconfig(the inference gateway's configuration and proxy layer),scheduler(its request scheduler),model selector(its model selector) ormodel server(the model server).Not modeled as retryable: a timeout does not prove the inference did not run, and the gateway may already have tried another inference gateway. Standard SDK retry modes may still retry a 5xx status, which could run (and bill) the inference again; set max attempts to 1 if a repeated inference is unacceptable.
- ThrottlingException:
The request was throttled before it was dispatched for inference. Retry the request with backoff, after the
Retry-Afterseconds when the response carries one. A message startinginference gatewaymeans the last inference gateway tried for the model returned a 429. The gateway may have tried others first, and they may have failed for other reasons. Up to 256 bytes of its error message are quoted.- BadGatewayException:
502 -- The gateway could not get a usable response from the inference gateway. The
Messageprefix says why:Global Inference Gateway: could not reach the inference gateway: <reason>-- the request could not be sent, or the connection closed before a response arrived (for example, the connection failed, or the inference gateway closed it after reading the request).Global Inference Gateway: timed out waiting for the inference gateway-- on the buffered operation, the inference gateway stopped sending partway through the response body and the gateway's read timeout expired. (The same text on a GatewayTimeoutException means no response arrived.)- Any other
Global Inference Gateway:message -- the gateway ended a response before it was complete, for example because it exceeded the 10 MiB limit of the buffered operation, the request exceeded its maximum duration, or the inference gateway was taken out of service. inference gateway: <status> <message>-- the inference gateway serving the model returned a 502, 500, 503 or another 5xx error. Its status code is in the message (the response status is 502), and up to 256 bytes of its error message are quoted.inference gateway: response ended unexpectedly-- the inference gateway's connection failed while it was sending the response.
The inference gateway names a component in the prefix when it reports one:
inference gateway (<component>):, where the component isconfig(the inference gateway's configuration and proxy layer),scheduler(its request scheduler),model selector(its model selector) ormodel server(the model server).Not modeled as retryable: a 502 can mean the inference already ran, and the gateway may already have tried another inference gateway. Standard SDK retry modes may still retry a 5xx status, which would run (and bill) the inference again; set max attempts to 1 if a repeated inference is unacceptable.
- PayloadTooLargeException:
The request body exceeds the 10 MiB limit. Reduce the size of the payload and send the request again.
- ServiceUnavailableException:
The service is temporarily unable to serve the request, for example because no capacity is available for the requested model. The request was not dispatched for inference. A 503 returned by the inference gateway serving the model after dispatch is reported as BadGatewayException, with the 503 in its
Message, not as this error.- InternalServerException:
An unexpected internal failure occurred.
- ResourceNotFoundException:
No gateway with this ID exists in this account and Region, or no target of the gateway serves the model named in the request body. A model that is served but has no available target right now returns ServiceUnavailableException instead. A message starting
inference gatewaymeans the inference gateway serving the model returned a 404; up to 256 bytes of its error message are quoted.- AccessDeniedException:
The caller is not authorized to invoke this gateway. Invoking requires the
sagemaker:InvokeGlobalInferenceGatewaypermission, orsagemaker:InvokeGlobalInferenceGatewayWithResponseStreamfor the streaming operation, on the gateway ARN. A message startinginference gatewaymeans the inference gateway serving the model refused the request with a 401 or 403, for example for its customer identity token. Its status code is in the message (the response status is 403), and up to 256 bytes of its error message are quoted.
InvokeGlobalInferenceGatewayWithResponseStream
$result = $client->invokeGlobalInferenceGatewayWithResponseStream([/* ... */]); $promise = $client->invokeGlobalInferenceGatewayWithResponseStreamAsync([/* ... */]);
Streaming inference. Returns an AWS event stream of PayloadPart events that carry the model server's server-sent event (SSE) bytes, so SDKs decode the stream into events for you.
The errors listed for this operation, other than ModelStreamError and InternalStreamFailure, are returned before the response starts (before the 200 status), so they end the request, not the stream. None of them can occur after the 200. ModelStreamError and InternalStreamFailure appear only as events in the stream, after the 200. Once the 200 is sent, the stream carries the response as PayloadPart events. If reading the inference gateway's response fails, the stream carries one InternalStreamFailure event and then ends. ModelStreamError is reserved and is not currently sent.
Errors carry their origin in the Message prefix. Every typed error's text is in the Message member (JSON key Message), and starts with one of these prefixes:
Global Inference Gateway:-- the gateway rejected or failed the request itself: validation, authorization, throttling, routing (the gateway or model was not found, or no capacity is available), a gateway limit or time limit, or failing to reach the inference gateway (Global Inference Gateway: could not reach the inference gateway: <reason>, orGlobal Inference Gateway: timed out waiting for the inference gateway). This includes the 400 for an invalid request and the 413 for a request body that is too large.inference gateway:-- the inference gateway serving the model returned an error status. The message gives that status code and up to 256 bytes of its error message, for exampleinference gateway: 503 <message>.inference gateway (<component>):-- the same, when the inference gateway reports which of its components produced the error, for exampleinference gateway (model server): 502 <message>. The component is one of:config-- the inference gateway's configuration and proxy layer.scheduler-- the inference gateway's request scheduler.model selector-- the inference gateway's model selector.model server-- the model server.
An InternalStreamFailure event's Message uses the same prefixes, with the same text the buffered operation uses for the same failure. Its inference gateway: message carries no status: inference gateway: response ended unexpectedly when the inference gateway's connection failed while it was sending the response.
Every error status the inference gateway returns before the 200 is a typed error, chosen by that status as for InvokeGlobalInferenceGateway:
- 400 and any other 4xx not listed here -- a 400 validation error.
- 401 and 403 -- AccessDeniedException (403).
- 404 -- ResourceNotFoundException.
- 408 -- GatewayTimeoutException (504).
- 429 -- ThrottlingException, with the
Retry-Afterheader. - 502, and 500, 503 or any other 5xx not listed here -- BadGatewayException (502). A 500 or 503 from the inference gateway comes after the request was dispatched, so it is not modeled as retryable, and the gateway may already have tried another inference gateway. Standard SDK retry modes may still retry a 5xx status; set max attempts to 1 if a repeated inference is unacceptable.
- 504 -- GatewayTimeoutException.
Parameter Syntax
$result = $client->invokeGlobalInferenceGatewayWithResponseStream([
'Accept' => '<string>',
'Body' => <string || resource || Psr\Http\Message\StreamInterface>, // REQUIRED
'ContentType' => '<string>', // REQUIRED
'CustomerIdentityToken' => '<string>',
'GatewayId' => '<string>', // REQUIRED
]);
Parameter Details
Members
- Accept
-
- Type: string
The MIME type you want for the response. It is passed to the model server.
- Body
-
- Required: Yes
- Type: blob (string|resource|Psr\Http\Message\StreamInterface)
The inference request body, as for InvokeGlobalInferenceGateway, except that the request is always sent with
"stream": true: this operation returns an event stream, so a"stream": falsein the body (or nostreamkey) is overridden. A body larger than 10 MiB is rejected with PayloadTooLargeException (413). - ContentType
-
- Required: Yes
- Type: string
The MIME type of the request body, as for InvokeGlobalInferenceGateway. Pass
application/json. - CustomerIdentityToken
-
- Type: string
The customer identity token (a JWT), as for InvokeGlobalInferenceGateway. It is passed unchanged to the inference gateway, which validates it.
- GatewayId
-
- Required: Yes
- Type: string
The ID of the Global Inference Gateway to invoke, as for InvokeGlobalInferenceGateway.
Result Syntax
[
'Body' => [ // EventParsingIterator
'InternalStreamFailure' => [
'Message' => '<string>',
],
'ModelStreamError' => [
'ErrorCode' => 'StreamBroken|ModelInvocationTimeExceeded',
'Message' => '<string>',
],
'PayloadPart' => [
'Bytes' => <string || resource || Psr\Http\Message\StreamInterface>,
],
],
]
Result Details
Members
- Body
-
- Type: EventParsingIterator supplying the following structures: PayloadPart, ModelStreamError, InternalStreamFailure
The response as an AWS event stream (application/vnd.amazon.eventstream).
Using an EventParsingIterator
To use an EventParsingIterator, you will need to loop over the events it will generate and check the top-level field to determine which type of event it is.
foreach($result['Body'] as $event) {
if (isset($event['InternalStreamFailure'])) {
// Handle the 'InternalStreamFailure' event.
} else if (isset($event['ModelStreamError'])) {
// Handle the 'ModelStreamError' event.
} else if (isset($event['PayloadPart'])) {
// Handle the 'PayloadPart' event.
}
}
Errors
- GatewayTimeoutException:
504 -- The inference gateway did not respond in time. The
Messageprefix says whose time limit expired:Global Inference Gateway: timed out waiting for the inference gateway-- the gateway's limit on the wait for the first part of the response.inference gateway: <status> <message>-- the inference gateway serving the model returned a 504, or a 408 because it timed out waiting for the request to arrive. Its status code is in the message (the response status is 504), and up to 256 bytes of its error message are quoted.
The inference gateway names a component in the prefix when it reports one:
inference gateway (<component>):, where the component isconfig(the inference gateway's configuration and proxy layer),scheduler(its request scheduler),model selector(its model selector) ormodel server(the model server).Not modeled as retryable: a timeout does not prove the inference did not run, and the gateway may already have tried another inference gateway. Standard SDK retry modes may still retry a 5xx status, which could run (and bill) the inference again; set max attempts to 1 if a repeated inference is unacceptable.
- ThrottlingException:
The request was throttled before it was dispatched for inference. Retry the request with backoff, after the
Retry-Afterseconds when the response carries one. A message startinginference gatewaymeans the last inference gateway tried for the model returned a 429. The gateway may have tried others first, and they may have failed for other reasons. Up to 256 bytes of its error message are quoted.- BadGatewayException:
502 -- The gateway could not get a usable response from the inference gateway. The
Messageprefix says why:Global Inference Gateway: could not reach the inference gateway: <reason>-- the request could not be sent, or the connection closed before a response arrived (for example, the connection failed, or the inference gateway closed it after reading the request).Global Inference Gateway: timed out waiting for the inference gateway-- on the buffered operation, the inference gateway stopped sending partway through the response body and the gateway's read timeout expired. (The same text on a GatewayTimeoutException means no response arrived.)- Any other
Global Inference Gateway:message -- the gateway ended a response before it was complete, for example because it exceeded the 10 MiB limit of the buffered operation, the request exceeded its maximum duration, or the inference gateway was taken out of service. inference gateway: <status> <message>-- the inference gateway serving the model returned a 502, 500, 503 or another 5xx error. Its status code is in the message (the response status is 502), and up to 256 bytes of its error message are quoted.inference gateway: response ended unexpectedly-- the inference gateway's connection failed while it was sending the response.
The inference gateway names a component in the prefix when it reports one:
inference gateway (<component>):, where the component isconfig(the inference gateway's configuration and proxy layer),scheduler(its request scheduler),model selector(its model selector) ormodel server(the model server).Not modeled as retryable: a 502 can mean the inference already ran, and the gateway may already have tried another inference gateway. Standard SDK retry modes may still retry a 5xx status, which would run (and bill) the inference again; set max attempts to 1 if a repeated inference is unacceptable.
- ModelStreamError:
Reserved for a model error during the stream. Not currently sent; a failure after the stream starts is reported as InternalStreamFailure.
- PayloadTooLargeException:
The request body exceeds the 10 MiB limit. Reduce the size of the payload and send the request again.
- InternalStreamFailure:
A failure after the stream started. The
Messagesays where it came from:Global Inference Gateway:when a gateway time limit expired or the inference gateway was taken out of service, orinference gateway: response ended unexpectedlywhen the inference gateway's connection failed while it was sending the response. The stream ends after this event.- ServiceUnavailableException:
The service is temporarily unable to serve the request, for example because no capacity is available for the requested model. The request was not dispatched for inference. A 503 returned by the inference gateway serving the model after dispatch is reported as BadGatewayException, with the 503 in its
Message, not as this error.- InternalServerException:
An unexpected internal failure occurred.
- ResourceNotFoundException:
No gateway with this ID exists in this account and Region, or no target of the gateway serves the model named in the request body. A model that is served but has no available target right now returns ServiceUnavailableException instead. A message starting
inference gatewaymeans the inference gateway serving the model returned a 404; up to 256 bytes of its error message are quoted.- AccessDeniedException:
The caller is not authorized to invoke this gateway. Invoking requires the
sagemaker:InvokeGlobalInferenceGatewaypermission, orsagemaker:InvokeGlobalInferenceGatewayWithResponseStreamfor the streaming operation, on the gateway ARN. A message startinginference gatewaymeans the inference gateway serving the model refused the request with a 401 or 403, for example for its customer identity token. Its status code is in the message (the response status is 403), and up to 256 bytes of its error message are quoted.
Shapes
AccessDeniedException
Description
The caller is not authorized to invoke this gateway. Invoking requires the sagemaker:InvokeGlobalInferenceGateway permission, or sagemaker:InvokeGlobalInferenceGatewayWithResponseStream for the streaming operation, on the gateway ARN. A message starting inference gateway means the inference gateway serving the model refused the request with a 401 or 403, for example for its customer identity token. Its status code is in the message (the response status is 403), and up to 256 bytes of its error message are quoted.
Members
- Message
-
- Required: Yes
- Type: string
A description of the error. It starts with a prefix such as
Global Inference Gateway:orinference gateway:that says where the error came from.
BadGatewayException
Description
502 -- The gateway could not get a usable response from the inference gateway. The Message prefix says why:
Global Inference Gateway: could not reach the inference gateway: <reason>-- the request could not be sent, or the connection closed before a response arrived (for example, the connection failed, or the inference gateway closed it after reading the request).Global Inference Gateway: timed out waiting for the inference gateway-- on the buffered operation, the inference gateway stopped sending partway through the response body and the gateway's read timeout expired. (The same text on a GatewayTimeoutException means no response arrived.)- Any other
Global Inference Gateway:message -- the gateway ended a response before it was complete, for example because it exceeded the 10 MiB limit of the buffered operation, the request exceeded its maximum duration, or the inference gateway was taken out of service. inference gateway: <status> <message>-- the inference gateway serving the model returned a 502, 500, 503 or another 5xx error. Its status code is in the message (the response status is 502), and up to 256 bytes of its error message are quoted.inference gateway: response ended unexpectedly-- the inference gateway's connection failed while it was sending the response.
The inference gateway names a component in the prefix when it reports one: inference gateway (<component>):, where the component is config (the inference gateway's configuration and proxy layer), scheduler (its request scheduler), model selector (its model selector) or model server (the model server).
Not modeled as retryable: a 502 can mean the inference already ran, and the gateway may already have tried another inference gateway. Standard SDK retry modes may still retry a 5xx status, which would run (and bill) the inference again; set max attempts to 1 if a repeated inference is unacceptable.
Members
- Message
-
- Required: Yes
- Type: string
A description of the error. It starts with a prefix such as
Global Inference Gateway:orinference gateway:that says where the error came from.
GatewayTimeoutException
Description
504 -- The inference gateway did not respond in time. The Message prefix says whose time limit expired:
Global Inference Gateway: timed out waiting for the inference gateway-- the gateway's limit on the wait for the first part of the response.inference gateway: <status> <message>-- the inference gateway serving the model returned a 504, or a 408 because it timed out waiting for the request to arrive. Its status code is in the message (the response status is 504), and up to 256 bytes of its error message are quoted.
The inference gateway names a component in the prefix when it reports one: inference gateway (<component>):, where the component is config (the inference gateway's configuration and proxy layer), scheduler (its request scheduler), model selector (its model selector) or model server (the model server).
Not modeled as retryable: a timeout does not prove the inference did not run, and the gateway may already have tried another inference gateway. Standard SDK retry modes may still retry a 5xx status, which could run (and bill) the inference again; set max attempts to 1 if a repeated inference is unacceptable.
Members
- Message
-
- Required: Yes
- Type: string
A description of the error. It starts with a prefix such as
Global Inference Gateway:orinference gateway:that says where the error came from.
InternalServerException
Description
An unexpected internal failure occurred.
Members
- Message
-
- Required: Yes
- Type: string
A description of the error. It starts with
Global Inference Gateway:, because the Global Inference Gateway returns this error itself.
InternalStreamFailure
Description
A failure after the stream started. The Message says where it came from: Global Inference Gateway: when a gateway time limit expired or the inference gateway was taken out of service, or inference gateway: response ended unexpectedly when the inference gateway's connection failed while it was sending the response. The stream ends after this event.
Members
- Message
-
- Required: Yes
- Type: string
A description of the failure. It starts with
Global Inference Gateway:orinference gateway:, which says where the failure came from.
ModelStreamError
Description
Reserved for a model error during the stream. Not currently sent; a failure after the stream starts is reported as InternalStreamFailure.
Members
- ErrorCode
-
- Type: string
The error code:
StreamBrokenif the model server's stream broke before it completed, orModelInvocationTimeExceededif the model exceeded its allowed invocation time. - Message
-
- Required: Yes
- Type: string
A description of the error.
PayloadPart
Description
A part of the streamed response.
Members
- Bytes
-
- Type: blob (string|resource|Psr\Http\Message\StreamInterface)
The bytes of this part: a chunk of the model server's server-sent event (SSE) stream.
PayloadTooLargeException
Description
The request body exceeds the 10 MiB limit. Reduce the size of the payload and send the request again.
Members
- Message
-
- Required: Yes
- Type: string
A description of the error. It starts with
Global Inference Gateway:, because the Global Inference Gateway returns this error itself.
ResourceNotFoundException
Description
No gateway with this ID exists in this account and Region, or no target of the gateway serves the model named in the request body. A model that is served but has no available target right now returns ServiceUnavailableException instead. A message starting inference gateway means the inference gateway serving the model returned a 404; up to 256 bytes of its error message are quoted.
Members
- Message
-
- Required: Yes
- Type: string
A description of the error. It starts with a prefix such as
Global Inference Gateway:orinference gateway:that says where the error came from. - ResourceId
-
- Type: string
The ID of the resource that was not found: the gateway ID, or the model name.
- ResourceType
-
- Type: string
The kind of resource that was not found: the gateway, or the model.
ResponseStream
Description
The events in the response stream. The response arrives as PayloadPart events. A failure after the stream starts arrives as one InternalStreamFailure event, and then the stream ends.
Members
- InternalStreamFailure
-
- Type: InternalStreamFailure structure
A failure after the stream started. The stream ends after this event.
- ModelStreamError
-
- Type: ModelStreamError structure
Reserved for a model error during the stream. Not currently sent.
- PayloadPart
-
- Type: PayloadPart structure
A part of the model server's response.
ServiceUnavailableException
Description
The service is temporarily unable to serve the request, for example because no capacity is available for the requested model. The request was not dispatched for inference. A 503 returned by the inference gateway serving the model after dispatch is reported as BadGatewayException, with the 503 in its Message, not as this error.
Members
- Message
-
- Required: Yes
- Type: string
A description of the error. It starts with
Global Inference Gateway:, because the Global Inference Gateway returns this error itself.
ThrottlingException
Description
The request was throttled before it was dispatched for inference. Retry the request with backoff, after the Retry-After seconds when the response carries one. A message starting inference gateway means the last inference gateway tried for the model returned a 429. The gateway may have tried others first, and they may have failed for other reasons. Up to 256 bytes of its error message are quoted.
Members
- Message
-
- Required: Yes
- Type: string
A description of the error. It starts with a prefix such as
Global Inference Gateway:orinference gateway:that says where the error came from.