Switchyard: NVIDIA’s Open Supply Routing Library

Most manufacturing AI brokers nonetheless ship each LLM name to the identical costly frontier mannequin. Classification steps, easy software calls, progress checks, and exhausting reasoning all hit the identical endpoint. The result’s pointless price and latency. NVIDIA NeMo Switchyard solves this.

It’s an open-source routing layer (proxy + library) that sits between your agent and the fashions. It decides, request by request or flip by flip, which mannequin ought to deal with the work. On this tutorial, we’ll construct a working two-model router and step by step transfer from random routing to content-aware routing. So, let’s get began.

What Precisely Does Switchyard Do?

A traditional LLM software may appear like this:

Utility
    |
    v
GPT / Claude / Native LLM

Switchyard provides a routing layer:

Utility
    |
    v
Switchyard
   /   
  v     v
Low cost   Highly effective
Mannequin   Mannequin

The applying doesn’t must know which upstream mannequin finally serves the request. Switchyard selects the precise goal and forwards the request. Let’s examine this virtually.

Step 1: Putting in Switchyard

For the CLI/server path, the venture documentation supplies a uv set up route:

uv software set up "nemo-switchyard[cli,server]"

Confirm the set up:

switchyard --version

Output:

switchyard 0.2.0
nemo-switchyard v0.2.0

Alternatively, the native Rust server will be put in instantly with Cargo:

cargo set up --locked switchyard-server

For this tutorial, we’ll route fashions by means of OpenRouter, so export your API key:

export OPENROUTER_API_KEY="your-key-here"

Don’t retailer the API key instantly within the configuration file.

Step 2: Understanding a Switchyard Configuration

Let’s begin with the best attainable setup: two fashions and random routing.

Create a YAML file named routes.random.yaml and add:

defaults:
  base_url: https://openrouter.ai/api/v1
  api_key: ${OPENROUTER_API_KEY}

routes:
  ab-test:
    kind: random_routing

    robust:
      mannequin: openai/gpt-4o

    weak:
      mannequin: openai/gpt-4o-mini

    strong_probability: 0.3
    rng_seed: 42
    fallback_target_on_evict: weak

The important thing setting is:

strong_probability: 0.3

Switchyard interprets this as roughly:

30% -> robust mannequin
70% -> weak mannequin

Random routing isn’t clever routing, however it’s helpful for A/B exams and for validating the proxy earlier than introducing a classifier. fallback_target_on_evict is required for this route kind and refers to a tier ID corresponding to robust or weak.

Step 3: Beginning the Routing Server

Begin Switchyard with:

switchyard serve 
  -c routes.random.yaml 
  --host 127.0.0.1 
  --port 4000

There is no such thing as a --dry-run choice within the examined serve CLI. Beginning the server is successfully the validation step: an invalid routing bundle fails throughout startup. You may confirm that the proxy is alive with:

curl -s http://127.0.0.1:4000/well being

Output:

{"standing":"okay"}

Step 4: Sending a Request Via the Router

Now ship an OpenAI-compatible request:

curl http://localhost:4000/v1/chat/completions 
  -H "Content material-Kind: software/json" 
  -d '{"mannequin":"ab-test","messages":[{"role":"user","content":"Explain gradient descent in simple terms."}]}'

Discover this area:

"mannequin": "ab-test"

Your consumer isn’t asking for a selected mannequin (gpt-4o or gpt-4o-mini). Switchyard chooses the precise mannequin. For this instance, the request landed on the weak tier:

"mannequin": "openai/gpt-4o-mini",
"utilization": { "prompt_tokens": 14, "completion_tokens": 247, "price": 0.0001503 }

Response:

Gradient descent is a technique utilized in optimization to seek out the minimal of a operate. Think about you are on a hilly panorama, and your purpose is to get to the bottom level within the valley. Here is the way it works, step-by-step:
1) Begin at a Random Level: You start at a random location on the hill.
2) Discover the Slope: You go searching and decide the steepness of the hill (the gradient) at your present location. This tells you which ones path is downhill.
3) Take a Step Downhill: You are taking a step within the path that goes down the steepest slope. The size of your step known as the "studying price" — in case you take small steps, you are cautious, whereas bigger steps will get you there sooner however may lead you off target.
4) Repeat: You retain repeating this course of, recalculating the slope and stepping down till you possibly can't go any decrease — that is the underside of the valley or the minimal of the operate.

In easy phrases, gradient descent is about marching down the hill step-by-step till you attain the bottom level. It is broadly utilized in machine studying to regulate fashions in order that they make higher predictions.

Step 5: Upgrading to Clever Routing

Random routing is nice for experiments, however suppose we would like this habits:

Easy request — low cost mannequin

Exhausting request — robust mannequin

Switchyard supplies a classifier route for precisely this function. The classifier estimates whether or not the weaker mannequin can remedy the duty, then applies a configured threshold. Create routes.sensible.yaml and write this configuration:

defaults:
  base_url: https://openrouter.ai/api/v1
  api_key: ${OPENROUTER_API_KEY}

routes:
  sensible:
    kind: deterministic

    classifier:
      mannequin: openai/gpt-4o-mini

    robust:
      mannequin: openai/gpt-4o

    weak:
      mannequin: openai/gpt-4o-mini

    profile: common
    session_affinity: true
    fallback_target_on_evict: weak

And begin it:

switchyard serve 
  -c routes.sensible.yaml 
  --host 127.0.0.1 
  --port 4000

Now there are three roles:

classifier
    |
    | predicts weak-model functionality
    v
+-------------------+
| Ought to weak remedy?|
+-------------------+
       /      
      /        
    sure         no
     |           |
     v           v
   weak        robust

The classifier produces a structured estimate containing a worth referred to as p_solve: an estimate of the likelihood that the weak mannequin can efficiently full the request.

Step 6: Testing the Sensible Route

Attempt a simple query:

curl http://localhost:4000/v1/chat/completions 
  -H "Content material-Kind: software/json" 
  -d '{
    "mannequin": "sensible",
    "messages": [
      {
        "role": "user",
        "content": "What is 15% of 200?"
      }
    ]
  }'

Output:

To seek out 15% of 200, you possibly can multiply 200 by 0.15:
200 × 0.15 = 30
So, 15% of 200 is 30.

Then attempt a more durable one:

curl http://localhost:4000/v1/chat/completions 
  -H "Content material-Kind: software/json" 
  -d '{
    "mannequin": "sensible",
    "max_tokens": 1500,
    "messages": [
      {
        "role": "user",
        "content": "Find the race condition in a distributed job queue where workers acquire leases using non-transactional Redis operations, then propose a failure-safe redesign."
      }
    ]
  }'

Output:

In a distributed job queue system utilizing Redis to handle and lease
jobs to staff, race situations can happen if a number of staff
try to amass a lease for a similar job concurrently utilizing
non-transactional operations. This may result in a number of staff
incorrectly believing they've efficiently acquired the lease,
leading to duplicate processing of the identical job.

### Typical Race Situation Situation
...
By incorporating these redesign components into the distributed job
queue structure, race situations will be considerably lowered
and job leases will be dealt with extra reliably and safely.

We did not hard-code the mannequin choice right here. As an alternative, the classifier determines the suitable tier for every immediate and routes the request accordingly. When you take a look at the logs, you possibly can see which mannequin was finally chosen for every request.

Step 7: Routing Coding Brokers Primarily based on Their Progress

Immediate issue isn’t the one helpful routing sign.

Contemplate a coding agent working for 30 turns. It could spend early turns exploring recordsdata, debugging failures, and reasoning about structure. Later turns might merely apply a longtime plan or make repetitive edits. Utilizing the strongest mannequin for each flip wastes inference funds. Switchyard’s stage_router is designed for this sort of multi-turn workload. It makes use of dialog and tool-result alerts to resolve whether or not a flip ought to go to a succesful or environment friendly tier.

You may create a configuration like this:

routes:
  stage:
    kind: stage_router

    robust:
      mannequin: openai/gpt-4o

    weak:
      mannequin: openai/gpt-4o-mini

    picker: efficient_first
    confidence_threshold: 0.5
    signal_recent_window: 3
    fallback_target_on_evict: weak

The concept is:

Agent flip
    |
    v
Current progress / failure alerts
    |
    v
Is additional functionality helpful now?
     / 
    /   
 weak   robust

Right here, the router seems to be for alerts related to issues corresponding to errors, repeated unproductive habits, exploration, and up to date productive adjustments. The purpose is to order the stronger mannequin for turns the place additional functionality seems helpful.

Step 8: Escalating Solely After the Weak Mannequin Struggles

One other technique is to keep away from predicting issue up entrance.

Let a budget mannequin attempt first, then escalate when proof of sustained hassle seems. The move turns into:

Request
   |
   v
Weak mannequin
   |
   v
Decide consequence
  /    
okay   struggling
 |        |
 v        v
keep    robust mannequin

Switchyard calls this escalation routing. You may create a configuration like this:

routes:
  agent:
    kind: escalation_router

    robust:
      mannequin: openai/gpt-4o

    weak:
      mannequin: openai/gpt-4o-mini

    choose:
      mannequin: openai/gpt-4o-mini
      confirmations: 2
      recent_turn_window: 28
      window_message_chars: 500

    fallback_target_on_evict: weak

That is conceptually completely different from up-front deterministic classification. Deterministic/functionality routing asks:

How troublesome does this request seem?

Escalation routing asks:

Is the weak mannequin really moving into hassle?

This makes escalation helpful for long-running agent periods the place activity issue can change over time.

Step 9: Measuring Whether or not Routing Is Truly Serving to

A router is just helpful if it improves the quality-cost trade-off. Switchyard exposes Prometheus metrics and statistics round requests, errors, latency, tokens, and routing habits. The venture additionally helps structured request telemetry and optionally available routing logs.

You may get server metrics with:

curl -s http://localhost:4000/metrics | head

and mixture JSON statistics:

curl -s http://localhost:4000/v1/stats | python3 -m json.software

For experiments, examine a minimum of three runs:

The extra helpful query isn’t whether or not the router was 85% correct, however how a lot of the robust mannequin’s high quality did routing protect, and the way a lot price and latency did it scale back? For instance:

Sturdy-only:
$20
92% activity success

Weak-only:
$5
71% activity success

Router:
$9
89% activity success

This tells you whether or not routing is economically helpful.

Remaining Ideas

As LLM programs turn into extra agentic, the query is shifting from:

Which mannequin ought to I exploit?

to:

Which mannequin ought to I exploit for this request, at this level within the workflow, beneath this price funds?

Switchyard is NVIDIA’s try to show that call into reusable infrastructure.

For a primary experiment, do not soar instantly into stage routing or advanced agent escalation.

Begin with two fashions.

Measure them independently.

Use weighted random routing to confirm your setup.

Then introduce capability-based routing and measure whether or not it preserves a lot of the robust mannequin’s high quality whereas shifting a significant share of requests to the cheaper tier.

This experiment provides you one thing way more helpful than one other LLM benchmark:

a quality-versus-cost curve on your precise workload.

And that’s finally what clever mannequin routing is making an attempt to optimize.

Kanwal Mehreen is a machine studying engineer and a technical author with a profound ardour for information science and the intersection of AI with drugs. She co-authored the e book “Maximizing Productiveness with ChatGPT”. As a Google Technology Scholar 2022 for APAC, she champions range and tutorial excellence. She’s additionally acknowledged as a Teradata Range in Tech Scholar, Mitacs Globalink Analysis Scholar, and Harvard WeCode Scholar. Kanwal is an ardent advocate for change, having based FEMCodes to empower ladies in STEM fields.