Understanding intelligent prompt routing in Amazon Bedrock
Note
Intelligent prompt routing in Amazon Bedrock is in preview and is subject to change.
Amazon Bedrock intelligent prompt routing provides a single serverless endpoint for efficiently routing requests between different foundational models within the same model family. It can help you optimize for response quality and cost. They offer a comprehensive solution for managing multiple AI models through a single serverless endpoint, simplifying the process for you. Intelligent prompt routing predicts the performance of each model for each request, and dynamically routes each request to the model that it predicts is most likely to give the desired response at the lowest cost. With intelligent prompt routing, Amazon Bedrock can help you build generative AI applications by using a combination of foundational models to get better performance at a lower cost than a single foundation model.
To best utilize intelligent prompt routing, you should regularly review performance to take advantage of new models. To optimize your usage, monitor the available performance and cost metrics.
To get started with intelligent prompt routing, use the Amazon Bedrock console, AWS CLI, or AWS SDK.
During preview, you can choose to use select models in the Anthropic and Meta families.
Intelligent prompt routing offers the following benefits.
-
Helps you optimize for response quality and cost by routing prompts to different foundation models.
-
Can help improve overall performance by leveraging multiple models' strengths.
-
Simplified management without the need for complex orchestration logic.
-
Future-proof by incorporating new models as they become available.
How intelligent prompt routing works
-
Choose the model family that you want to use.
-
For each incoming request, intelligent prompt routing predicts the performance of each specified model.
-
Amazon Bedrock dynamically chooses the model that it predicts will offer the best combination of response quality and cost.
-
Amazon Bedrock sends the request to the model that you chose for processing.
-
You get back the response, which also has information about the model that Amazon Bedrock chose.
Considerations and limitations
The following are considerations and limitations for intelligent prompt routing in Amazon Bedrock.
-
During preview, you can only pick from preconfigured routers.
-
Currently, intelligent prompt routing accepts only English prompts.
-
Intelligent prompt routing can’t adjust routing decisions or responses based on application-specific performance data.
-
Intelligent prompt routing might not always provide the most optimal routing for unique or specialized use cases. How effective the routing is depends on the initial training data.