Nvidia’s NeMo Switchyard Routes Agent Calls Across Models, Not One Default

Nvidia has released NeMo Switchyard, an open-source Rust-based proxy and library that routes large-language-model traffic across models and providers. Its premise is a consequential break from the one-model application: an AI agent can use a different model at each step, responding to what happens while it works rather than committing every call to a single backend.

Model selection moves inside the workflow

That is designed for agent systems that perform several kinds of work in one task. A coding workflow, for example, may need stronger reasoning to understand a repository, then a less expensive model for routine changes or tool use, and an escalation path if tests produce a difficult error. Nvidia calls this a system of models: different models take different jobs inside one agentic workflow.

Switchyard can route from workflow stages and execution signals including tool results and errors. It also translates between OpenAI and Anthropic API formats, allowing an application to change those model backends without rewriting the application. The router therefore sits between an agent application and the models it calls, making the choice of model an operational decision rather than a fixed architecture choice.

A router is not the agent manager

The boundary matters. Switchyard is a routing layer, not a complete orchestration system. An orchestrator determines how a workflow proceeds: which agent acts next, when tools are called, how work is delegated and when a task is complete. Switchyard makes the narrower decision of which model receives a particular call, so it is intended to fit inside a larger agent system rather than replace one.

What the routing layer is meant to change

• Avoid sending every agent call to the same model or provider.
• Balance quality, latency and cost across a full workflow rather than judging a single call in isolation.
• Reduce dependence on one model as capabilities, pricing and infrastructure requirements change.

The cost case depends on getting escalation right

The economic appeal is straightforward but conditional. Agents can make many model calls to finish one task, and routing routine work away from a frontier model can lower the cost of that sequence. Yet a smaller model that produces invalid tool calls or needs repeated correction can erase its apparent savings; extra routing decisions can also add latency.

Nvidia says its internal benchmarks kept frontier-level accuracy while reducing task-completion costs to roughly one-third of running Anthropic’s Claude Opus 4.8 alone. The figures have not been independently audited, so they show the company’s case for routing rather than a settled performance result across customer workloads.

The infrastructure layer is becoming a competitive surface

Nvidia is entering an active layer of the stack. Google Cloud has added model routing to API Gateway for compatible requests across multiple models and providers, including Gemini. LiteLLM is adding Switchyard as a proxy plug-in, Kong is integrating it into its AI Gateway, and LangChain has tested it with Deep Agents. Those connections put routing where developers already manage model traffic and agent applications.

The unresolved work is evaluation. A routing rule can misread an edge case, and a policy that performs well now may deteriorate after a model update, a price change or a shift in tool-use performance. Companies adopting this approach will need to assess routing policies alongside individual models, and against the tasks their own agents actually perform.