The code provided in this repository is meant to be used in a development environment. We recommend following the AWS Well-Architected Framework before you migrate any of the solutions shared here in your production environment.
You can also refer to the accompanied blog post: https://aws.amazon.com/blogs/machine-learning/multi-llm-routing-strategies-for-generative-ai-applications-on-aws/ for more information.
This repository provides sample implementations for dynamic routing of user prompts across multiple Large Language Models (LLMs). The core functionality lies in the routing logic, which accurately interprets user prompts, maps them to predefined task categories, and directs them to the most appropriate LLM for response generation. You can use these samples as a starting point to build your own custom routing solution tailored to your specific needs.
The repository contains two different approaches to implement dynamic prompt routing:
- LLM-assisted routing: Uses a classifier LLM to analyze prompts and make routing decisions
- Semantic routing: Employs semantic search to match prompts with the most appropriate LLM
For demonstration purposes, the samples use Amazon Bedrock as the LLM provider, but can be adapted to work with other LLM providers as well.
- LLM-assisted routing - Sample implementation of routing layer with the help of a classifier LLM.
- Semantic routing - Sample implementation of routing layer using semantic search.
- Reference index generation for semantic routing - Code for generating a FAISS reference index for use in the semantic routing solution.
We welcome community contributions! Please ensure your sample aligns with AWS best practices, and please update the Contents section of this README file with a link to your sample, along with a description.
See CONTRIBUTING for more information.
This library is licensed under the MIT-0 License. See the LICENSE file.
