Route coding agent requests to the right model on Amazon Bedrock with NVIDIA NeMo Switchyard

Coding agents such as Claude Code and OpenAI Codex CLI work in tasks. A developer asks for a change, and the agent reads files, edits code, and runs tests until the work is done. One task can take a few model calls or dozens. What the developer pays for is the task, not one call. The price per token alone doesn’t tell you what a whole task costs. The total depends on how many turns a model needs, how much history each turn replays, and how often a task has to be retried or never finishes. You pay for every attempt, including attempts on tasks that never finish. So when you route a coding agent’s requests between models, judge the routing by cost per solved task: the total cost of your runs divided by the number of runs whose tests pass. NVIDIA NeMo Switchyard is an open source library that helps an AI agent choose which model handles each request. You define the models that it can use and the strategy that it follows. Its LiteLLM integration supports two strategies. Stage routing reads the recent tool activity and results in the request history, and then picks one of two candidate models. Random routing picks from the configured models, with optional weights, for traffic splits and A/B comparisons. Another option you can explore is LiteLLM, who also recently shipped a native built-in Switchyard stage router as an auto-router strategy. In this post, you use the AWS Cloud Development Kit (AWS CDK) to deploy LiteLLM, an open source gateway for large language models (LLMs), with the Switchyard routing plugin in your AWS account. LiteLLM calls the models on Amazon Bedrock . You then use the sample’s benchmark runners to measure cost per solved task for a routed model group and for single-model baseline groups, so that you can evaluate routing against your own workloads. Note: The sample installs switchyard-server 0.3.0 from crates.io and runs the published LiteLLM v1.102.0 image. NVIDIA documents switchyard-server 0.3.0 as a demo and evaluation component, and its configuration can change between releases. This sample uses the experimental Switchyard LiteLLM integration and its configuration and APIs may change between releases. This is sample code, for non-production usage. Work with your security and legal teams to meet your organizational security, regulatory, and compliance requirements before deployment. Deploy the sample in a non-production account and test it before relying on it. Solution overview Figure 1 shows the architecture of the solution. LiteLLM and the Switchyard plugin run in one container image on Amazon Elastic Container Service (Amazon ECS) on AWS Fargate , in private subnets. Figure 1: Solution architecture A request flows through these steps. The numbers match the diagram. The coding agent sends an OpenAI-compatible request with the model field set to switchyard , the LiteLLM model group that Switchyard routes, and the shared gateway key in the Authorization header. The endpoint is your hostname over HTTPS, with an AWS Certificate Manager (ACM) certificate on the load balancer, and only the IPv4 CIDR range that you allow can reach it. With a hosted zone in Amazon Route 53 , Route 53 resolves the hostname to the load balancer. AWS WAF filters the request with IP reputation, known bad input, and per-IP rate-limit rules. The Application Load Balancer forwards only /v1/chat/completions , /v1/models , and /health/liveliness to the ECS task in a private subnet, and answers every other path with 404. The task runs one container: LiteLLM with the embedded Switchyard plugin. LiteLLM validates the gateway key and calls the plugin to pick a model. Stage chooses between a capable model and a lower-cost efficient model. It reads the tool activity and results in the request history, so the agent must send the full conversation history. Random chooses from a weighted pool. Neither strategy calls a judge model. LiteLLM calls the selected model through the Amazon Bedrock Converse or ConverseStream API, with the task’s AWS Identity and Access Management (IAM) role for credentials. The call goes through an Amazon Bedrock Runtime interface endpoint in the Amazon Virtual Private Cloud (Amazon VPC) , so inference traffic stays on the AWS network. The response or token stream returns through LiteLLM and the load balancer. The x-litellm-model-name response header names the model that answered. The response body’s model field stays switchyard . LiteLLM listens on port 4000 in the container and is the load balancer’s target, and the task’s security group admits only the load balancer. The path allowlist keeps the LiteLLM admin UI, API docs, and key management routes unreachable. Amazon Elastic Container Registry (Amazon ECR) holds the gateway image. AWS Secrets Manager generates the secret behind the gateway key, and the container adds the sk- prefix when it starts. The task definition passes the profile files in LITELLM_CONFIG_TEXT and SWITCHYARD_CONFIG_TEXT , and the startup script writes them to /tmp before it starts LiteLLM. Logs go to one Amazon CloudWatch log group, / /litellm , which keeps them for 30 days, and Container Insights collects metrics. The VPC spans two Availability Zones, and one NAT gateway provides outbound access for image pulls and other AWS services. The plugin supports Stage routing, which needs exactly two candidates, and Random routing, which needs at least two and splits traffic by weight for A/B tests. Classifier-backed strategies, such as Task and Composite routing, need intermediate model calls that this integration doesn’t support. See the Switchyard routing overview for the full catalog. Prerequisites To follow this walkthrough, you need the following: A non-production AWS account with Amazon Bedrock access to Claude Opus 5.5 and NVIDIA Nemotron 3 Super 120B. The pair has been tested in US East (N. Virginia). For other AWS Regions, check Regional availability by models and tool-calling support. The AWS Command Line Interface (AWS CLI) v2 with configured credentials and an explicit AWS Region, in an account and Region that are bootstrapped for the AWS CDK . Node.js 20 or later, npm, Docker with BuildKit, and Python 3.11 or later. A hostname that you own, with either a public Route 53 hosted zone for its domain or an ACM certificate for it in the deployment Region. A short plain-HTTP test needs neither. Walkthrough You define the model group, configure Stage, deploy and test the gateway, and measure cost. Start by cloning the sample repository: git clone https://github.com/aws-samples/sample-route-coding-agent-requests-Amazon-Bedrock-nemo-switchyard.git cd sample-route-coding-agent-requests-Amazon-Bedrock-nemo-switchyard Step 1: Define the model group in LiteLLM A LiteLLM model group is one public model name with one or more deployments behind it. The sample keeps its configuration in deployment profiles under profiles/ . Each profile pairs litellm.yaml , which defines the models and gateway settings, with switchyard.toml , which controls how Switchyard picks a model. The following excerpt of profiles/stage/litellm.yaml defines the routed switchyard group, two single-model baseline groups, and the plugin registration: model_list: - model_name: switchyard litellm_params: model: bedrock/converse/global.anthropic.claude-opus-5-5 aws_region_name: os.environ/AWS_REGION - model_name: switchyard litellm_params: model: bedrock/converse/nvidia.nemotron-super-3-120b aws_region_name: os.environ/AWS_REGION # One-model baseline groups, same model strings as the routed candidates. - model_name: claude-opus-5-5 litellm_params: model: bedrock/converse/global.anthropic.claude-opus-5-5 aws_region_name: os.environ/AWS_REGION - model_name: nemotron-super-3-120b litellm_params: model: bedrock/converse/nvidia.nemotron-super-3-120b aws_region_name: os.environ/AWS_REGION # The same object selects a deployment and applies supported request changes. router_settings: plugins: - switchyard_gateway.configured.ROUTING_PLUGIN litellm_settings: callbacks: -