Interface SageMakerInferenceRouterRuntimeClient
- All Superinterfaces:
AutoCloseable,AwsClient,SdkAutoCloseable,SdkClient
builder() method.
Amazon SageMaker Inference Router Runtime is the runtime API for Amazon SageMaker AI inference routers. An inference router sends each request that you make to one of the destinations of a routing target. A destination is a SageMaker AI endpoint or inference component, in your account or in another account, in the router's Amazon Web Services Region or in another Region in the same Amazon Web Services partition. The router chooses a healthy destination with capacity, and prefers destinations in its own Region.
Use InvokeInferenceRouter to get an inference in real time. Use InvokeInferenceRouterAsync
to submit a request whose input and response are stored in Amazon S3.
The inference router runtime endpoint supports HTTP/1.1 only. HTTP/2 isn't supported. If you use the Amazon Web Services SDKs or the Amazon Web Services CLI, you don't need to change anything.
-
Field Summary
FieldsModifier and TypeFieldDescriptionstatic final StringValue for looking up the service's metadata from theServiceMetadataProvider.static final String -
Method Summary
Modifier and TypeMethodDescriptionbuilder()Create a builder that can be used to configure and create aSageMakerInferenceRouterRuntimeClient.create()Create aSageMakerInferenceRouterRuntimeClientwith the region loaded from theDefaultAwsRegionProviderChainand credentials loaded from theDefaultCredentialsProvider.default InvokeInferenceRouterResponseinvokeInferenceRouter(Consumer<InvokeInferenceRouterRequest.Builder> invokeInferenceRouterRequest) Sends a real-time inference request to an inference router.default InvokeInferenceRouterResponseinvokeInferenceRouter(InvokeInferenceRouterRequest invokeInferenceRouterRequest) Sends a real-time inference request to an inference router.invokeInferenceRouterAsync(Consumer<InvokeInferenceRouterAsyncRequest.Builder> invokeInferenceRouterAsyncRequest) Submits an asynchronous inference request to an inference router.invokeInferenceRouterAsync(InvokeInferenceRouterAsyncRequest invokeInferenceRouterAsyncRequest) Submits an asynchronous inference request to an inference router.The SDK service client configuration exposes client settings to the user, e.g., ClientOverrideConfigurationstatic ServiceMetadataMethods inherited from interface software.amazon.awssdk.utils.SdkAutoCloseable
closeMethods inherited from interface software.amazon.awssdk.core.SdkClient
serviceName
-
Field Details
-
SERVICE_NAME
- See Also:
-
SERVICE_METADATA_ID
Value for looking up the service's metadata from theServiceMetadataProvider.- See Also:
-
-
Method Details
-
invokeInferenceRouter
default InvokeInferenceRouterResponse invokeInferenceRouter(InvokeInferenceRouterRequest invokeInferenceRouterRequest) throws ValidationException, AccessDeniedException, ServiceUnavailableException, ModelErrorException, ThrottlingException, RequestTimeoutException, InternalDependencyException, InternalFailureException, ModelNotReadyException, AwsServiceException, SdkClientException, SageMakerInferenceRouterRuntimeException Sends a real-time inference request to an inference router. In the request, you name the routing target, and the router sends the request to one of the routing target's destinations. The response includes the destination that served the request and the reason that the router chose it.
The following limits apply:
-
The maximum request payload is 6 MB. A request with a larger body fails with HTTP status code 413.
-
Your model has up to 60 seconds for each attempt, the same limit as
InvokeEndpoint. If your account has a higher model execution timeout forInvokeEndpoint, it doesn't apply through an inference router. For models that take longer, useInvokeInferenceRouterAsync.
If a destination returns an error that the router can retry, such as a throttling error, before your model produces a response, the router retries the request once on another destination. A request that times out on a destination isn't retried. Because a request can make two attempts, set your client read timeout to at least 170 seconds.
The caller needs permission to invoke the inference router and every destination of the routing target. For information about permissions, see Route requests across Regions with inference routers.
- Parameters:
invokeInferenceRouterRequest-- Returns:
- Result of the InvokeInferenceRouter operation returned by the service.
- See Also:
-
-
invokeInferenceRouter
default InvokeInferenceRouterResponse invokeInferenceRouter(Consumer<InvokeInferenceRouterRequest.Builder> invokeInferenceRouterRequest) throws ValidationException, AccessDeniedException, ServiceUnavailableException, ModelErrorException, ThrottlingException, RequestTimeoutException, InternalDependencyException, InternalFailureException, ModelNotReadyException, AwsServiceException, SdkClientException, SageMakerInferenceRouterRuntimeException Sends a real-time inference request to an inference router. In the request, you name the routing target, and the router sends the request to one of the routing target's destinations. The response includes the destination that served the request and the reason that the router chose it.
The following limits apply:
-
The maximum request payload is 6 MB. A request with a larger body fails with HTTP status code 413.
-
Your model has up to 60 seconds for each attempt, the same limit as
InvokeEndpoint. If your account has a higher model execution timeout forInvokeEndpoint, it doesn't apply through an inference router. For models that take longer, useInvokeInferenceRouterAsync.
If a destination returns an error that the router can retry, such as a throttling error, before your model produces a response, the router retries the request once on another destination. A request that times out on a destination isn't retried. Because a request can make two attempts, set your client read timeout to at least 170 seconds.
The caller needs permission to invoke the inference router and every destination of the routing target. For information about permissions, see Route requests across Regions with inference routers.
This is a convenience which creates an instance of the
InvokeInferenceRouterRequest.Builderavoiding the need to create one manually viaInvokeInferenceRouterRequest.builder()- Parameters:
invokeInferenceRouterRequest- AConsumerthat will call methods onInvokeInferenceRouterRequest.Builderto create a request.- Returns:
- Result of the InvokeInferenceRouter operation returned by the service.
- See Also:
-
-
invokeInferenceRouterAsync
default InvokeInferenceRouterAsyncResponse invokeInferenceRouterAsync(InvokeInferenceRouterAsyncRequest invokeInferenceRouterAsyncRequest) throws ValidationException, AccessDeniedException, ServiceUnavailableException, ThrottlingException, RequestTimeoutException, InternalFailureException, AwsServiceException, SdkClientException, SageMakerInferenceRouterRuntimeException Submits an asynchronous inference request to an inference router. Before you call this operation, upload the input data to Amazon S3 in the router's Amazon Web Services Region. The router returns as soon as it accepts the request, and sends the request to one of the destinations of the routing target. The destination reads the input from Amazon S3, and the inference response is written to the Amazon S3 output path that the routing target specifies.
If a destination doesn't start the request within the routing target's
MaxTimeInBacklogSeconds, the router moves the request to another destination. After your model starts processing the request, the request doesn't move. If no destination can take the request beforeRequestTTLSecondsruns out, or every destination has been tried, the router writes a failure object to theFailureLocationand publishes a notification to the routing target's error topic, if one is configured.For information about the permissions that the caller and each destination need, see Send asynchronous requests through an inference router.
- Parameters:
invokeInferenceRouterAsyncRequest-- Returns:
- Result of the InvokeInferenceRouterAsync operation returned by the service.
- See Also:
-
invokeInferenceRouterAsync
default InvokeInferenceRouterAsyncResponse invokeInferenceRouterAsync(Consumer<InvokeInferenceRouterAsyncRequest.Builder> invokeInferenceRouterAsyncRequest) throws ValidationException, AccessDeniedException, ServiceUnavailableException, ThrottlingException, RequestTimeoutException, InternalFailureException, AwsServiceException, SdkClientException, SageMakerInferenceRouterRuntimeException Submits an asynchronous inference request to an inference router. Before you call this operation, upload the input data to Amazon S3 in the router's Amazon Web Services Region. The router returns as soon as it accepts the request, and sends the request to one of the destinations of the routing target. The destination reads the input from Amazon S3, and the inference response is written to the Amazon S3 output path that the routing target specifies.
If a destination doesn't start the request within the routing target's
MaxTimeInBacklogSeconds, the router moves the request to another destination. After your model starts processing the request, the request doesn't move. If no destination can take the request beforeRequestTTLSecondsruns out, or every destination has been tried, the router writes a failure object to theFailureLocationand publishes a notification to the routing target's error topic, if one is configured.For information about the permissions that the caller and each destination need, see Send asynchronous requests through an inference router.
This is a convenience which creates an instance of the
InvokeInferenceRouterAsyncRequest.Builderavoiding the need to create one manually viaInvokeInferenceRouterAsyncRequest.builder()- Parameters:
invokeInferenceRouterAsyncRequest- AConsumerthat will call methods onInvokeInferenceRouterAsyncRequest.Builderto create a request.- Returns:
- Result of the InvokeInferenceRouterAsync operation returned by the service.
- See Also:
-
create
Create aSageMakerInferenceRouterRuntimeClientwith the region loaded from theDefaultAwsRegionProviderChainand credentials loaded from theDefaultCredentialsProvider. -
builder
Create a builder that can be used to configure and create aSageMakerInferenceRouterRuntimeClient. -
serviceMetadata
-
serviceClientConfiguration
Description copied from interface:SdkClientThe SDK service client configuration exposes client settings to the user, e.g., ClientOverrideConfiguration- Specified by:
serviceClientConfigurationin interfaceAwsClient- Specified by:
serviceClientConfigurationin interfaceSdkClient- Returns:
- SdkServiceClientConfiguration
-