Three Jevs pick the model for 24 sub-agent tasks.
One request to a System One model decides whether a sub-agent runs on Haiku, Sonnet or Opus. I asked three Jevs, one hosted and two running on a laptop, to make that call for the same tasks. Their routes nearly match. They differ in how sure they were, and in how long an answer took on this setup. This is a gut check on a small set of tasks that Claude wrote and labelled, not a benchmark: what it does and doesn't show.
The problem: every sub-agent gets the same model
An agent harness spawns sub-agents all the time: find the callers of a function, add a flag, chase down a memory leak. Either the orchestrator spends its own tokens deciding which model each one gets, or everything runs on the most capable model. model-router makes that choice with one call, just before the spawn.
"Checkout intermittently double-charges customers. Find the cause."
Explore → lookup.
Fig. 1This run skipped steps 1 and 2 so that every task was judged by the model. If the decision model fails or times out, the router returns the config's fallback route and never throws.
Hypothesis, and what would confirm it
Hypothesis One Jev call over the text of a sub-agent task chooses a model tier well enough that a harness can move a large share of sub-agent work off its most capable model, while rarely sending a task to a model too weak for it, at a latency that is small next to starting the sub-agent.
| Check | Done when | TypeSafe | Flash 9B | 27B | Status |
|---|
Table 1Checks from the model-router README, with this run's numbers at the default bar of 0.25. Check 1 needs ≥ 100 labelled tasks from real harness sessions. This run had 24 starter tasks, so its routing numbers are not evidence for it yet.
The three decision models
Same API, same question, same 24 tasks, each asked once. One model is hosted by TypeSafe; two are Open Jev models running 4-bit on an Apple M5 Pro laptop. Flash 9B was released the day before this run.
What Jev was asked
Each request carries the task text and one Score question. The levels describe kinds of work, cheapest
first. Each maps to a model in the router's config, and Jev never sees a model name:
it can judge what a task asks for, but knows nothing about what claude-haiku-5-5 is good
at.
Which description fits the work that task asks a sub-agent to do?
How an answer becomes a route
Jev returns three probabilities that add up to 1. The obvious policy takes the biggest. The router asks something different of each route: if the task is sent here, how likely is it that it needed a more capable model? That's the sum of everything above it. It takes the cheapest route where that risk is at or under the bar. Sending a task too low costs a failed run and a retry; too high only costs money.
Fig. 2Pick an example, or any task and model. Risk for each route is the probability that the task belonged higher. The most capable route always passes, since nothing is above it. Routes come from model-router's own policy code, replayed at each bar; no new requests are made.
Result: a near tie on the scoreboard
At the default bar of 0.25, with differences shown against TypeSafe Jev.
Where the three models differ
The routes are nearly the same. The answers behind them are not: the three differ in how much probability they put on an answer, which tasks they hesitated over, and how much they depend on the routing rule.
7aHow sure each model was
Fig. 3Each dot is one task. Hover or tap a dot for the task and the full answer.
7bEvery task, side by side
Fig. 4Each bar is one model's answer, split lookup / standard / open_ended. A mark at the right gives the outcome at the bar of 0.25: ✓ as labelled, ▲ too high, ▼ too low, and ↑ when the rule moved a task above its likeliest route. "Unsure" means some model put less than 0.70 on its likeliest route.
Move the bar
The bar is arithmetic on answers already returned, so it can be tuned after the fact. Drag it and watch where each model's tasks go. For a sure model it barely matters; for an unsure one it decides the result.
Fig. 5Panels: the 24 tasks in rows by their label, each cell shaded by the route it was sent to at this bar. Charts: how many tasks go too low or too high as the bar moves from 0 to 0.60, one line per model; the vertical rule follows the slider.
Speed: one model per lane
Check 2 asks for p95 ≤ 500 ms per routing decision. Measured around each request, one at a time, after one untimed warm-up.
They are not a statement of how fast these models are. The two Open Jev figures are what one laptop measured: 4-bit MLX builds on an Apple M5 Pro running macOS, a desktop operating system and not a server, while the machine was in ordinary use with other applications running. The same weights on server hardware, at another quantisation or under another runtime would time differently.
The TypeSafe figure is a round trip over the network from that laptop to a hosted API, so it mixes their serving time with my connection. Read all three as what a routing decision cost in this setup, on this day.
Fig. 6Each dot is one request. Ticks mark p50 and p95. The three runs were spread over two hours, with one local model loaded at a time.
| Decision model | p50 | p95 | max | Check 2 |
|---|
What this does and doesn't show
This was a gut check. Read it as "the mechanism works on clear cases", not as a measured accuracy.
- The request works end to end on all three decision models.
- A Score over route descriptions separates clear cases.
- On the same laptop, a model a third of the size routed these 24 tasks as well in a third of the time, once the routing rule allowed for its doubt.
- The policy can be tuned after the fact without asking again.
- 24 tasks, asked once each. One task is 4 percentage points.
- One author wrote everything. Claude wrote the tasks, their labels and the route descriptions in one sitting, so tasks echo the wording they're judged against ("find the cause" against "a failure with an unknown cause").
- Labels are opinions. Nobody ran these tasks on the three Claude models to see which was enough. That needs a harness (check 4).
- Flash 9B's 24 of 24 has little margin. On tasks 13 to 16 it was close to an even split between two routes.
- The tasks don't separate the models. All three are at or near 24 of 24, so this says nothing about which decides best.
- How fast the models are. See the note under Speed.
- Replace the starter tasks with ≥ 100 sub-agent prompts from real harness sessions, labelled by the model that turned out to be enough.
- Wire it into a harness's spawn path and log each decision with the sub-agent's outcome.
- Try a second question for work the tiers don't order, such as "the task needs to read images".
- Compare with the orchestrator choosing the model itself: agreement, and the tokens it spends deciding.
Run details and reproducing it
| Date | 2026-10-08 |
|---|---|
| TypeSafe Jev | jev-1.13.0, pinned (what jev-latest served on the day), hosted API |
| Open Jev 27B | openjev/openjev-MLX-4bit (MLX, 4-bit, 14 GB), T=0.85 |
| Open Jev Flash 9B | openjev/OpenJev-Flash-9B-MLX-4bit (MLX, 4-bit, 5 GB), T=1.07 |
| Open Jev helper | shim.py 81a22f1b for both, with each model card's calibration settings |
| Hardware | Apple M5 Pro, 64 GB, macOS 27.2 (local models) |
| Tasks | eval/tasks.jsonl: 24, eight per route |
| Requests | One per task, sequential, plus one untimed warm-up. Rules and pinned models skipped. |
cd model-router
TYPESAFE_API_KEY=sk-... TYPESAFE_DEFAULT_MODEL=jev-1.13.0 node eval/run.mjs
JEV_BACKEND=openjev node eval/run.mjs
JEV_BACKEND=openjev OPENJEV_URL=http://127.0.0.1:3003 node eval/run.mjs
node eval/compare.mjs eval/results-*.json
Every number on this page is generated from the three recorded runs by
docs/tools/build-model-router-data.mjs, using model-router's own policy and summary code,
and checked against the summaries each run saved. Sources:
. The written report is
results-2026-10-08.md.