SageMakerInferenceRouterRuntime / Client / invoke_inference_router

invoke_inference_router

SageMakerInferenceRouterRuntime.Client.invoke_inference_router(**kwargs)

Sends a real-time inference request to an inference router. In the request, you name the routing target, and the router sends the request to one of the routing target’s destinations. The response includes the destination that served the request and the reason that the router chose it.

The following limits apply:

  • The maximum request payload is 6 MB. A request with a larger body fails with HTTP status code 413.

  • Your model has up to 60 seconds for each attempt, the same limit as InvokeEndpoint. If your account has a higher model execution timeout for InvokeEndpoint, it doesn’t apply through an inference router. For models that take longer, use InvokeInferenceRouterAsync.

If a destination returns an error that the router can retry, such as a throttling error, before your model produces a response, the router retries the request once on another destination. A request that times out on a destination isn’t retried. Because a request can make two attempts, set your client read timeout to at least 170 seconds.

The caller needs permission to invoke the inference router and every destination of the routing target. For information about permissions, see Route requests across Regions with inference routers.

See also: AWS API Documentation

Request Syntax

response = client.invoke_inference_router(
    InferenceRouterName='string',
    RoutingTargetName='string',
    ContentType='string',
    Accept='string',
    InferenceId='string',
    PrefixAwareId='string',
    CustomAttributes='string',
    Body=b'bytes'|file
)
Parameters:
  • InferenceRouterName (string) –

    [REQUIRED]

    The name of the inference router to invoke.

  • RoutingTargetName (string) –

    [REQUIRED]

    The name of the routing target in the inference router. The routing target lists the destinations that can serve the request.

  • ContentType (string) – The MIME type of the input data in the request body. The router passes it to the destination.

  • Accept (string) – The MIME type that you want for the inference response. The router passes it to the destination.

  • InferenceId (string) – An identifier for the inference request. The router passes it to the destination.

  • PrefixAwareId (string) –

    An optional, stable identifier that serves as a routing hint for prefix-aware routing. The router passes it to the destination that serves the request. Within that destination, requests with the same prefix and the same identifier go to the same instance. If requests from different applications might have the same prompt prefix, set a different identifier for each application to differentiate their routing decisions.

    Applies only to destination endpoints configured with a RoutingStrategy of PREFIX_AWARE.

  • CustomAttributes (string) – Provides additional information about the inference request. The value is an opaque string that the router passes, unchanged, to the model container of the destination that serves the request. The value can contain up to 1,024 visible US-ASCII characters.

  • Body (bytes or seekable file-like object) –

    [REQUIRED]

    The input data for the inference, in the format that your model expects. The body can’t be empty.

Return type:

dict

Returns:

Response Syntax

{
    'Body': StreamingBody(),
    'ContentType': 'string',
    'InvokedRoutedResourceArn': 'string',
    'RoutingReason': 'string',
    'InvokedProductionVariant': 'string',
    'CustomAttributes': 'string'
}

Response Structure

  • (dict) –

    • Body (StreamingBody) –

      The inference that the model returned.

    • ContentType (string) –

      The MIME type of the inference returned in the response body.

    • InvokedRoutedResourceArn (string) –

      The Amazon Resource Name (ARN) of the endpoint or inference component that served the request.

    • RoutingReason (string) –

      The reason that the router chose the destination that served the request. Possible values:

      • local-preferred – The destination is a healthy destination in the router’s Region.

      • capacity-available – No destination in the router’s Region could take the request, and two or more destinations in other Regions could.

      • only-healthy – No destination in the router’s Region could take the request, and exactly one destination in another Region could.

      • failover – The router moved the request away from destinations in its Region because they were overloaded or returning errors. You also see this value when the router can’t assess destination health, and when no destination is healthy and the router tries one anyway.

    • InvokedProductionVariant (string) –

      The production variant of the destination endpoint that served the request.

    • CustomAttributes (string) –

      Provides additional information in the response about the inference. This is the value that the model container of the destination returned, unchanged.

Exceptions