Interface SageMakerInferenceRouterRuntimeAsyncClient
- All Superinterfaces:
AutoCloseable,AwsClient,SdkAutoCloseable,SdkClient
builder() method.The asynchronous client performs non-blocking I/O when configured with any
SdkAsyncHttpClient supported in the SDK. However, full non-blocking is not guaranteed as the async client may
perform blocking calls in some cases such as credentials retrieval and endpoint discovery as part of the async API
call.
Amazon SageMaker Inference Router Runtime is the runtime API for Amazon SageMaker AI inference routers. An inference router sends each request that you make to one of the destinations of a routing target. A destination is a SageMaker AI endpoint or inference component, in your account or in another account, in the router's Amazon Web Services Region or in another Region in the same Amazon Web Services partition. The router chooses a healthy destination with capacity, and prefers destinations in its own Region.
Use InvokeInferenceRouter to get an inference in real time. Use InvokeInferenceRouterAsync
to submit a request whose input and response are stored in Amazon S3.
The inference router runtime endpoint supports HTTP/1.1 only. HTTP/2 isn't supported. If you use the Amazon Web Services SDKs or the Amazon Web Services CLI, you don't need to change anything.
-
Field Summary
FieldsModifier and TypeFieldDescriptionstatic final StringValue for looking up the service's metadata from theServiceMetadataProvider.static final String -
Method Summary
Modifier and TypeMethodDescriptionbuilder()Create a builder that can be used to configure and create aSageMakerInferenceRouterRuntimeAsyncClient.create()Create aSageMakerInferenceRouterRuntimeAsyncClientwith the region loaded from theDefaultAwsRegionProviderChainand credentials loaded from theDefaultCredentialsProvider.invokeInferenceRouter(Consumer<InvokeInferenceRouterRequest.Builder> invokeInferenceRouterRequest) Sends a real-time inference request to an inference router.invokeInferenceRouter(InvokeInferenceRouterRequest invokeInferenceRouterRequest) Sends a real-time inference request to an inference router.invokeInferenceRouterAsync(Consumer<InvokeInferenceRouterAsyncRequest.Builder> invokeInferenceRouterAsyncRequest) Submits an asynchronous inference request to an inference router.invokeInferenceRouterAsync(InvokeInferenceRouterAsyncRequest invokeInferenceRouterAsyncRequest) Submits an asynchronous inference request to an inference router.The SDK service client configuration exposes client settings to the user, e.g., ClientOverrideConfigurationMethods inherited from interface software.amazon.awssdk.utils.SdkAutoCloseable
closeMethods inherited from interface software.amazon.awssdk.core.SdkClient
serviceName
-
Field Details
-
SERVICE_NAME
- See Also:
-
SERVICE_METADATA_ID
Value for looking up the service's metadata from theServiceMetadataProvider.- See Also:
-
-
Method Details
-
invokeInferenceRouter
default CompletableFuture<InvokeInferenceRouterResponse> invokeInferenceRouter(InvokeInferenceRouterRequest invokeInferenceRouterRequest) Sends a real-time inference request to an inference router. In the request, you name the routing target, and the router sends the request to one of the routing target's destinations. The response includes the destination that served the request and the reason that the router chose it.
The following limits apply:
-
The maximum request payload is 6 MB. A request with a larger body fails with HTTP status code 413.
-
Your model has up to 60 seconds for each attempt, the same limit as
InvokeEndpoint. If your account has a higher model execution timeout forInvokeEndpoint, it doesn't apply through an inference router. For models that take longer, useInvokeInferenceRouterAsync.
If a destination returns an error that the router can retry, such as a throttling error, before your model produces a response, the router retries the request once on another destination. A request that times out on a destination isn't retried. Because a request can make two attempts, set your client read timeout to at least 170 seconds.
The caller needs permission to invoke the inference router and every destination of the routing target. For information about permissions, see Route requests across Regions with inference routers.
- Parameters:
invokeInferenceRouterRequest-- Returns:
- A Java Future containing the result of the InvokeInferenceRouter operation returned by the service.
The CompletableFuture returned by this method can be completed exceptionally with the following exceptions. The exception returned is wrapped with CompletionException, so you need to invokeThrowable.getCause()to retrieve the underlying exception.- ValidationException The request isn't valid. Check the inference router name, the routing target name, and the request parameters, and then try again.
- AccessDeniedException You don't have permission to perform this action. Check that the caller is allowed to invoke the inference router and every destination of the routing target.
- ServiceUnavailableException The service is temporarily unavailable. For example, no destination of the routing target could serve the request, or a destination didn't respond. Retry the request with exponential backoff. The Amazon Web Services SDKs and the Amazon Web Services CLI do this by default.
- ModelErrorException The model in the destination that served the request returned an error. Check
OriginalStatusCode,OriginalMessage, and the destination's logs in Amazon CloudWatch. - ThrottlingException The request was denied because you exceeded the request rate for the inference router, or because a destination throttled the request. Retry the request with exponential backoff.
- RequestTimeoutException The inference router didn't receive the complete request within 300 seconds of accepting it, for example because of a very slow upload. Retry the request.
- InternalDependencyException The request failed because of an error in an internal dependency. Retry the request with exponential backoff.
- InternalFailureException An internal error occurred. Retry the request with exponential backoff. The Amazon Web Services SDKs and the Amazon Web Services CLI do this by default.
- ModelNotReadyException The destination isn't ready to serve the model, for example because an inference component has no copies. Retry the request later.
- SdkException Base class for all exceptions that can be thrown by the SDK (both service and client). Can be used for catch all scenarios.
- SdkClientException If any client side error occurs such as an IO related failure, failure to get credentials, etc.
- SageMakerInferenceRouterRuntimeException Base class for all service exceptions. Unknown exceptions will be thrown as an instance of this type.
- See Also:
-
-
invokeInferenceRouter
default CompletableFuture<InvokeInferenceRouterResponse> invokeInferenceRouter(Consumer<InvokeInferenceRouterRequest.Builder> invokeInferenceRouterRequest) Sends a real-time inference request to an inference router. In the request, you name the routing target, and the router sends the request to one of the routing target's destinations. The response includes the destination that served the request and the reason that the router chose it.
The following limits apply:
-
The maximum request payload is 6 MB. A request with a larger body fails with HTTP status code 413.
-
Your model has up to 60 seconds for each attempt, the same limit as
InvokeEndpoint. If your account has a higher model execution timeout forInvokeEndpoint, it doesn't apply through an inference router. For models that take longer, useInvokeInferenceRouterAsync.
If a destination returns an error that the router can retry, such as a throttling error, before your model produces a response, the router retries the request once on another destination. A request that times out on a destination isn't retried. Because a request can make two attempts, set your client read timeout to at least 170 seconds.
The caller needs permission to invoke the inference router and every destination of the routing target. For information about permissions, see Route requests across Regions with inference routers.
This is a convenience which creates an instance of the
InvokeInferenceRouterRequest.Builderavoiding the need to create one manually viaInvokeInferenceRouterRequest.builder()- Parameters:
invokeInferenceRouterRequest- AConsumerthat will call methods onInvokeInferenceRouterRequest.Builderto create a request.- Returns:
- A Java Future containing the result of the InvokeInferenceRouter operation returned by the service.
The CompletableFuture returned by this method can be completed exceptionally with the following exceptions. The exception returned is wrapped with CompletionException, so you need to invokeThrowable.getCause()to retrieve the underlying exception.- ValidationException The request isn't valid. Check the inference router name, the routing target name, and the request parameters, and then try again.
- AccessDeniedException You don't have permission to perform this action. Check that the caller is allowed to invoke the inference router and every destination of the routing target.
- ServiceUnavailableException The service is temporarily unavailable. For example, no destination of the routing target could serve the request, or a destination didn't respond. Retry the request with exponential backoff. The Amazon Web Services SDKs and the Amazon Web Services CLI do this by default.
- ModelErrorException The model in the destination that served the request returned an error. Check
OriginalStatusCode,OriginalMessage, and the destination's logs in Amazon CloudWatch. - ThrottlingException The request was denied because you exceeded the request rate for the inference router, or because a destination throttled the request. Retry the request with exponential backoff.
- RequestTimeoutException The inference router didn't receive the complete request within 300 seconds of accepting it, for example because of a very slow upload. Retry the request.
- InternalDependencyException The request failed because of an error in an internal dependency. Retry the request with exponential backoff.
- InternalFailureException An internal error occurred. Retry the request with exponential backoff. The Amazon Web Services SDKs and the Amazon Web Services CLI do this by default.
- ModelNotReadyException The destination isn't ready to serve the model, for example because an inference component has no copies. Retry the request later.
- SdkException Base class for all exceptions that can be thrown by the SDK (both service and client). Can be used for catch all scenarios.
- SdkClientException If any client side error occurs such as an IO related failure, failure to get credentials, etc.
- SageMakerInferenceRouterRuntimeException Base class for all service exceptions. Unknown exceptions will be thrown as an instance of this type.
- See Also:
-
-
invokeInferenceRouterAsync
default CompletableFuture<InvokeInferenceRouterAsyncResponse> invokeInferenceRouterAsync(InvokeInferenceRouterAsyncRequest invokeInferenceRouterAsyncRequest) Submits an asynchronous inference request to an inference router. Before you call this operation, upload the input data to Amazon S3 in the router's Amazon Web Services Region. The router returns as soon as it accepts the request, and sends the request to one of the destinations of the routing target. The destination reads the input from Amazon S3, and the inference response is written to the Amazon S3 output path that the routing target specifies.
If a destination doesn't start the request within the routing target's
MaxTimeInBacklogSeconds, the router moves the request to another destination. After your model starts processing the request, the request doesn't move. If no destination can take the request beforeRequestTTLSecondsruns out, or every destination has been tried, the router writes a failure object to theFailureLocationand publishes a notification to the routing target's error topic, if one is configured.For information about the permissions that the caller and each destination need, see Send asynchronous requests through an inference router.
- Parameters:
invokeInferenceRouterAsyncRequest-- Returns:
- A Java Future containing the result of the InvokeInferenceRouterAsync operation returned by the service.
The CompletableFuture returned by this method can be completed exceptionally with the following exceptions. The exception returned is wrapped with CompletionException, so you need to invokeThrowable.getCause()to retrieve the underlying exception.- ValidationException The request isn't valid. Check the inference router name, the routing target name, and the request parameters, and then try again.
- AccessDeniedException You don't have permission to perform this action. Check that the caller is allowed to invoke the inference router and every destination of the routing target.
- ServiceUnavailableException The service is temporarily unavailable. For example, no destination of the routing target could serve the request, or a destination didn't respond. Retry the request with exponential backoff. The Amazon Web Services SDKs and the Amazon Web Services CLI do this by default.
- ThrottlingException The request was denied because you exceeded the request rate for the inference router, or because a destination throttled the request. Retry the request with exponential backoff.
- RequestTimeoutException The inference router didn't receive the complete request within 300 seconds of accepting it, for example because of a very slow upload. Retry the request.
- InternalFailureException An internal error occurred. Retry the request with exponential backoff. The Amazon Web Services SDKs and the Amazon Web Services CLI do this by default.
- SdkException Base class for all exceptions that can be thrown by the SDK (both service and client). Can be used for catch all scenarios.
- SdkClientException If any client side error occurs such as an IO related failure, failure to get credentials, etc.
- SageMakerInferenceRouterRuntimeException Base class for all service exceptions. Unknown exceptions will be thrown as an instance of this type.
- See Also:
-
invokeInferenceRouterAsync
default CompletableFuture<InvokeInferenceRouterAsyncResponse> invokeInferenceRouterAsync(Consumer<InvokeInferenceRouterAsyncRequest.Builder> invokeInferenceRouterAsyncRequest) Submits an asynchronous inference request to an inference router. Before you call this operation, upload the input data to Amazon S3 in the router's Amazon Web Services Region. The router returns as soon as it accepts the request, and sends the request to one of the destinations of the routing target. The destination reads the input from Amazon S3, and the inference response is written to the Amazon S3 output path that the routing target specifies.
If a destination doesn't start the request within the routing target's
MaxTimeInBacklogSeconds, the router moves the request to another destination. After your model starts processing the request, the request doesn't move. If no destination can take the request beforeRequestTTLSecondsruns out, or every destination has been tried, the router writes a failure object to theFailureLocationand publishes a notification to the routing target's error topic, if one is configured.For information about the permissions that the caller and each destination need, see Send asynchronous requests through an inference router.
This is a convenience which creates an instance of the
InvokeInferenceRouterAsyncRequest.Builderavoiding the need to create one manually viaInvokeInferenceRouterAsyncRequest.builder()- Parameters:
invokeInferenceRouterAsyncRequest- AConsumerthat will call methods onInvokeInferenceRouterAsyncRequest.Builderto create a request.- Returns:
- A Java Future containing the result of the InvokeInferenceRouterAsync operation returned by the service.
The CompletableFuture returned by this method can be completed exceptionally with the following exceptions. The exception returned is wrapped with CompletionException, so you need to invokeThrowable.getCause()to retrieve the underlying exception.- ValidationException The request isn't valid. Check the inference router name, the routing target name, and the request parameters, and then try again.
- AccessDeniedException You don't have permission to perform this action. Check that the caller is allowed to invoke the inference router and every destination of the routing target.
- ServiceUnavailableException The service is temporarily unavailable. For example, no destination of the routing target could serve the request, or a destination didn't respond. Retry the request with exponential backoff. The Amazon Web Services SDKs and the Amazon Web Services CLI do this by default.
- ThrottlingException The request was denied because you exceeded the request rate for the inference router, or because a destination throttled the request. Retry the request with exponential backoff.
- RequestTimeoutException The inference router didn't receive the complete request within 300 seconds of accepting it, for example because of a very slow upload. Retry the request.
- InternalFailureException An internal error occurred. Retry the request with exponential backoff. The Amazon Web Services SDKs and the Amazon Web Services CLI do this by default.
- SdkException Base class for all exceptions that can be thrown by the SDK (both service and client). Can be used for catch all scenarios.
- SdkClientException If any client side error occurs such as an IO related failure, failure to get credentials, etc.
- SageMakerInferenceRouterRuntimeException Base class for all service exceptions. Unknown exceptions will be thrown as an instance of this type.
- See Also:
-
serviceClientConfiguration
Description copied from interface:SdkClientThe SDK service client configuration exposes client settings to the user, e.g., ClientOverrideConfiguration- Specified by:
serviceClientConfigurationin interfaceAwsClient- Specified by:
serviceClientConfigurationin interfaceSdkClient- Returns:
- SdkServiceClientConfiguration
-
create
Create aSageMakerInferenceRouterRuntimeAsyncClientwith the region loaded from theDefaultAwsRegionProviderChainand credentials loaded from theDefaultCredentialsProvider. -
builder
Create a builder that can be used to configure and create aSageMakerInferenceRouterRuntimeAsyncClient.
-