Interface SageMakerInferenceRouterRuntimeAsyncClient

All Superinterfaces:
AutoCloseable, AwsClient, SdkAutoCloseable, SdkClient

@Generated("software.amazon.awssdk:codegen") @SdkPublicApi @ThreadSafe public interface SageMakerInferenceRouterRuntimeAsyncClient extends AwsClient
Service client for accessing Amazon SageMaker Inference Router Runtime asynchronously. This can be created using the static builder() method.The asynchronous client performs non-blocking I/O when configured with any SdkAsyncHttpClient supported in the SDK. However, full non-blocking is not guaranteed as the async client may perform blocking calls in some cases such as credentials retrieval and endpoint discovery as part of the async API call.

Amazon SageMaker Inference Router Runtime is the runtime API for Amazon SageMaker AI inference routers. An inference router sends each request that you make to one of the destinations of a routing target. A destination is a SageMaker AI endpoint or inference component, in your account or in another account, in the router's Amazon Web Services Region or in another Region in the same Amazon Web Services partition. The router chooses a healthy destination with capacity, and prefers destinations in its own Region.

Use InvokeInferenceRouter to get an inference in real time. Use InvokeInferenceRouterAsync to submit a request whose input and response are stored in Amazon S3.

The inference router runtime endpoint supports HTTP/1.1 only. HTTP/2 isn't supported. If you use the Amazon Web Services SDKs or the Amazon Web Services CLI, you don't need to change anything.

  • Field Details

  • Method Details

    • invokeInferenceRouter

      default CompletableFuture<InvokeInferenceRouterResponse> invokeInferenceRouter(InvokeInferenceRouterRequest invokeInferenceRouterRequest)

      Sends a real-time inference request to an inference router. In the request, you name the routing target, and the router sends the request to one of the routing target's destinations. The response includes the destination that served the request and the reason that the router chose it.

      The following limits apply:

      • The maximum request payload is 6 MB. A request with a larger body fails with HTTP status code 413.

      • Your model has up to 60 seconds for each attempt, the same limit as InvokeEndpoint. If your account has a higher model execution timeout for InvokeEndpoint, it doesn't apply through an inference router. For models that take longer, use InvokeInferenceRouterAsync.

      If a destination returns an error that the router can retry, such as a throttling error, before your model produces a response, the router retries the request once on another destination. A request that times out on a destination isn't retried. Because a request can make two attempts, set your client read timeout to at least 170 seconds.

      The caller needs permission to invoke the inference router and every destination of the routing target. For information about permissions, see Route requests across Regions with inference routers.

      Parameters:
      invokeInferenceRouterRequest -
      Returns:
      A Java Future containing the result of the InvokeInferenceRouter operation returned by the service.
      The CompletableFuture returned by this method can be completed exceptionally with the following exceptions. The exception returned is wrapped with CompletionException, so you need to invoke Throwable.getCause() to retrieve the underlying exception.
      • ValidationException The request isn't valid. Check the inference router name, the routing target name, and the request parameters, and then try again.
      • AccessDeniedException You don't have permission to perform this action. Check that the caller is allowed to invoke the inference router and every destination of the routing target.
      • ServiceUnavailableException The service is temporarily unavailable. For example, no destination of the routing target could serve the request, or a destination didn't respond. Retry the request with exponential backoff. The Amazon Web Services SDKs and the Amazon Web Services CLI do this by default.
      • ModelErrorException The model in the destination that served the request returned an error. Check OriginalStatusCode, OriginalMessage, and the destination's logs in Amazon CloudWatch.
      • ThrottlingException The request was denied because you exceeded the request rate for the inference router, or because a destination throttled the request. Retry the request with exponential backoff.
      • RequestTimeoutException The inference router didn't receive the complete request within 300 seconds of accepting it, for example because of a very slow upload. Retry the request.
      • InternalDependencyException The request failed because of an error in an internal dependency. Retry the request with exponential backoff.
      • InternalFailureException An internal error occurred. Retry the request with exponential backoff. The Amazon Web Services SDKs and the Amazon Web Services CLI do this by default.
      • ModelNotReadyException The destination isn't ready to serve the model, for example because an inference component has no copies. Retry the request later.
      • SdkException Base class for all exceptions that can be thrown by the SDK (both service and client). Can be used for catch all scenarios.
      • SdkClientException If any client side error occurs such as an IO related failure, failure to get credentials, etc.
      • SageMakerInferenceRouterRuntimeException Base class for all service exceptions. Unknown exceptions will be thrown as an instance of this type.
      See Also:
    • invokeInferenceRouter

      default CompletableFuture<InvokeInferenceRouterResponse> invokeInferenceRouter(Consumer<InvokeInferenceRouterRequest.Builder> invokeInferenceRouterRequest)

      Sends a real-time inference request to an inference router. In the request, you name the routing target, and the router sends the request to one of the routing target's destinations. The response includes the destination that served the request and the reason that the router chose it.

      The following limits apply:

      • The maximum request payload is 6 MB. A request with a larger body fails with HTTP status code 413.

      • Your model has up to 60 seconds for each attempt, the same limit as InvokeEndpoint. If your account has a higher model execution timeout for InvokeEndpoint, it doesn't apply through an inference router. For models that take longer, use InvokeInferenceRouterAsync.

      If a destination returns an error that the router can retry, such as a throttling error, before your model produces a response, the router retries the request once on another destination. A request that times out on a destination isn't retried. Because a request can make two attempts, set your client read timeout to at least 170 seconds.

      The caller needs permission to invoke the inference router and every destination of the routing target. For information about permissions, see Route requests across Regions with inference routers.


      This is a convenience which creates an instance of the InvokeInferenceRouterRequest.Builder avoiding the need to create one manually via InvokeInferenceRouterRequest.builder()

      Parameters:
      invokeInferenceRouterRequest - A Consumer that will call methods on InvokeInferenceRouterRequest.Builder to create a request.
      Returns:
      A Java Future containing the result of the InvokeInferenceRouter operation returned by the service.
      The CompletableFuture returned by this method can be completed exceptionally with the following exceptions. The exception returned is wrapped with CompletionException, so you need to invoke Throwable.getCause() to retrieve the underlying exception.
      • ValidationException The request isn't valid. Check the inference router name, the routing target name, and the request parameters, and then try again.
      • AccessDeniedException You don't have permission to perform this action. Check that the caller is allowed to invoke the inference router and every destination of the routing target.
      • ServiceUnavailableException The service is temporarily unavailable. For example, no destination of the routing target could serve the request, or a destination didn't respond. Retry the request with exponential backoff. The Amazon Web Services SDKs and the Amazon Web Services CLI do this by default.
      • ModelErrorException The model in the destination that served the request returned an error. Check OriginalStatusCode, OriginalMessage, and the destination's logs in Amazon CloudWatch.
      • ThrottlingException The request was denied because you exceeded the request rate for the inference router, or because a destination throttled the request. Retry the request with exponential backoff.
      • RequestTimeoutException The inference router didn't receive the complete request within 300 seconds of accepting it, for example because of a very slow upload. Retry the request.
      • InternalDependencyException The request failed because of an error in an internal dependency. Retry the request with exponential backoff.
      • InternalFailureException An internal error occurred. Retry the request with exponential backoff. The Amazon Web Services SDKs and the Amazon Web Services CLI do this by default.
      • ModelNotReadyException The destination isn't ready to serve the model, for example because an inference component has no copies. Retry the request later.
      • SdkException Base class for all exceptions that can be thrown by the SDK (both service and client). Can be used for catch all scenarios.
      • SdkClientException If any client side error occurs such as an IO related failure, failure to get credentials, etc.
      • SageMakerInferenceRouterRuntimeException Base class for all service exceptions. Unknown exceptions will be thrown as an instance of this type.
      See Also:
    • invokeInferenceRouterAsync

      default CompletableFuture<InvokeInferenceRouterAsyncResponse> invokeInferenceRouterAsync(InvokeInferenceRouterAsyncRequest invokeInferenceRouterAsyncRequest)

      Submits an asynchronous inference request to an inference router. Before you call this operation, upload the input data to Amazon S3 in the router's Amazon Web Services Region. The router returns as soon as it accepts the request, and sends the request to one of the destinations of the routing target. The destination reads the input from Amazon S3, and the inference response is written to the Amazon S3 output path that the routing target specifies.

      If a destination doesn't start the request within the routing target's MaxTimeInBacklogSeconds, the router moves the request to another destination. After your model starts processing the request, the request doesn't move. If no destination can take the request before RequestTTLSeconds runs out, or every destination has been tried, the router writes a failure object to the FailureLocation and publishes a notification to the routing target's error topic, if one is configured.

      For information about the permissions that the caller and each destination need, see Send asynchronous requests through an inference router.

      Parameters:
      invokeInferenceRouterAsyncRequest -
      Returns:
      A Java Future containing the result of the InvokeInferenceRouterAsync operation returned by the service.
      The CompletableFuture returned by this method can be completed exceptionally with the following exceptions. The exception returned is wrapped with CompletionException, so you need to invoke Throwable.getCause() to retrieve the underlying exception.
      • ValidationException The request isn't valid. Check the inference router name, the routing target name, and the request parameters, and then try again.
      • AccessDeniedException You don't have permission to perform this action. Check that the caller is allowed to invoke the inference router and every destination of the routing target.
      • ServiceUnavailableException The service is temporarily unavailable. For example, no destination of the routing target could serve the request, or a destination didn't respond. Retry the request with exponential backoff. The Amazon Web Services SDKs and the Amazon Web Services CLI do this by default.
      • ThrottlingException The request was denied because you exceeded the request rate for the inference router, or because a destination throttled the request. Retry the request with exponential backoff.
      • RequestTimeoutException The inference router didn't receive the complete request within 300 seconds of accepting it, for example because of a very slow upload. Retry the request.
      • InternalFailureException An internal error occurred. Retry the request with exponential backoff. The Amazon Web Services SDKs and the Amazon Web Services CLI do this by default.
      • SdkException Base class for all exceptions that can be thrown by the SDK (both service and client). Can be used for catch all scenarios.
      • SdkClientException If any client side error occurs such as an IO related failure, failure to get credentials, etc.
      • SageMakerInferenceRouterRuntimeException Base class for all service exceptions. Unknown exceptions will be thrown as an instance of this type.
      See Also:
    • invokeInferenceRouterAsync

      default CompletableFuture<InvokeInferenceRouterAsyncResponse> invokeInferenceRouterAsync(Consumer<InvokeInferenceRouterAsyncRequest.Builder> invokeInferenceRouterAsyncRequest)

      Submits an asynchronous inference request to an inference router. Before you call this operation, upload the input data to Amazon S3 in the router's Amazon Web Services Region. The router returns as soon as it accepts the request, and sends the request to one of the destinations of the routing target. The destination reads the input from Amazon S3, and the inference response is written to the Amazon S3 output path that the routing target specifies.

      If a destination doesn't start the request within the routing target's MaxTimeInBacklogSeconds, the router moves the request to another destination. After your model starts processing the request, the request doesn't move. If no destination can take the request before RequestTTLSeconds runs out, or every destination has been tried, the router writes a failure object to the FailureLocation and publishes a notification to the routing target's error topic, if one is configured.

      For information about the permissions that the caller and each destination need, see Send asynchronous requests through an inference router.


      This is a convenience which creates an instance of the InvokeInferenceRouterAsyncRequest.Builder avoiding the need to create one manually via InvokeInferenceRouterAsyncRequest.builder()

      Parameters:
      invokeInferenceRouterAsyncRequest - A Consumer that will call methods on InvokeInferenceRouterAsyncRequest.Builder to create a request.
      Returns:
      A Java Future containing the result of the InvokeInferenceRouterAsync operation returned by the service.
      The CompletableFuture returned by this method can be completed exceptionally with the following exceptions. The exception returned is wrapped with CompletionException, so you need to invoke Throwable.getCause() to retrieve the underlying exception.
      • ValidationException The request isn't valid. Check the inference router name, the routing target name, and the request parameters, and then try again.
      • AccessDeniedException You don't have permission to perform this action. Check that the caller is allowed to invoke the inference router and every destination of the routing target.
      • ServiceUnavailableException The service is temporarily unavailable. For example, no destination of the routing target could serve the request, or a destination didn't respond. Retry the request with exponential backoff. The Amazon Web Services SDKs and the Amazon Web Services CLI do this by default.
      • ThrottlingException The request was denied because you exceeded the request rate for the inference router, or because a destination throttled the request. Retry the request with exponential backoff.
      • RequestTimeoutException The inference router didn't receive the complete request within 300 seconds of accepting it, for example because of a very slow upload. Retry the request.
      • InternalFailureException An internal error occurred. Retry the request with exponential backoff. The Amazon Web Services SDKs and the Amazon Web Services CLI do this by default.
      • SdkException Base class for all exceptions that can be thrown by the SDK (both service and client). Can be used for catch all scenarios.
      • SdkClientException If any client side error occurs such as an IO related failure, failure to get credentials, etc.
      • SageMakerInferenceRouterRuntimeException Base class for all service exceptions. Unknown exceptions will be thrown as an instance of this type.
      See Also:
    • serviceClientConfiguration

      default SageMakerInferenceRouterRuntimeServiceClientConfiguration serviceClientConfiguration()
      Description copied from interface: SdkClient
      The SDK service client configuration exposes client settings to the user, e.g., ClientOverrideConfiguration
      Specified by:
      serviceClientConfiguration in interface AwsClient
      Specified by:
      serviceClientConfiguration in interface SdkClient
      Returns:
      SdkServiceClientConfiguration
    • create

      Create a SageMakerInferenceRouterRuntimeAsyncClient with the region loaded from the DefaultAwsRegionProviderChain and credentials loaded from the DefaultCredentialsProvider.
    • builder

      Create a builder that can be used to configure and create a SageMakerInferenceRouterRuntimeAsyncClient.