SysPad › Examples › AI gateway (multi-provider LLM router)
AI gateway (multi-provider LLM router)
A query embedding (SageMaker) + vector search (OpenSearch) feed a router that splits evenly across OpenAI, Anthropic, and Bedrock - Bedrock reserved for traffic that must stay inside the AWS boundary.
Explicit split-edge routing across three LLM providers (not a hard-coded single vendor), and why a non-distributor router node needs an explicit edgeType override to avoid tripling the modelled spend.
- Throughput: 800 req/sec
- First to saturate: Embedding Endpoint
- Estimated cost: $4,076k/mo
How it works
- Query Router
- Routing lives in our code, not in a vendor SDK scattered through the app. Swapping a provider, or shifting the traffic mix between them, is a change here and nowhere else.
- Embedding Endpoint
- Turns the query into a vector before retrieval. GPU instances billed by the hour whether or not anyone asks a question, which makes this the floor under the monthly cost.
- Vector Search
- Retrieves the passages worth putting in the prompt. Better retrieval here means fewer input tokens downstream, so this node quietly controls the provider bill.
- Bedrock (compliance path)
- Not a third copy of the same thing: this is the path for traffic that is not allowed to leave the AWS boundary. Same models, different contract.
- Conversation Log
- Every prompt and response written down. Mostly write-only in normal operation, and the thing you will wish you had the first time a user reports a bad answer.