The maintainer runs 100% of tokens on the two most expensive Claude models. Commercial routers promise 20–98% savings; independent benchmarks say most routers can't beat simple baselines. We tested which of his real task shapes actually survive a downgrade, whether a knowledge file can change an orchestrator's delegation choices, and where routing genuinely doesn't belong. Tokens first; dollars are API-equivalent counterfactuals.
From the mined 40-day dataset (see Token Economics): every one of 3,360 requests ran on Fable 5 or Opus 4.8. Zero Sonnet, zero Haiku. And because spend concentrates in a few marathon sessions (top 5 = 69%), the routing question is not "use a small model for small sessions" — it's "which phases inside big sessions can subagents run on cheaper tiers."
Delegation has a fixed cost that gates this: tonight's live measurements put research-grade subagents at a ~16–30k-token spawn floor, median ≈58k tokens each. Routing pays when the delegated bulk clears that floor — the standing ≳30k rule survives another night of data.
Three research agents swept academic routers, commercial products, and harness-native mechanisms; claims were kept only with sources, and vendor numbers are labeled as such.
| Approach | Claimed | Independently verified |
|---|---|---|
| RouteLLM (academic) | 85% cost cut at 95% GPT-4 quality | Benchmark replications vary by dataset (45% MMLU, 35% GSM8K) |
| Commercial routers (Martian, Not Diamond, …) | 20–98% savings | RouterArena ranks Not Diamond last of 12: "frequently selects expensive models." Almost no independent validation exists anywhere. |
| Azure Model Router | "up to 60%" | The only product with independent data: ~55% real savings in one 30-day production test — plus real bugs (context capped by smallest pool model, cache degraded by model churn) |
| Open-source routers (CARROT, vLLM-SR) | — | ~35% cost cut at <2% accuracy loss (RouterArena) — beating the commercial offerings |
| Claude Code native | — | Per-subagent model: frontmatter, opusplan, /effort, experimental Advisor tool — no external router needed for this use case |
The independent benchmark's core finding — routers over-rely on the strongest model — is the maintainer's own usage pattern, automated. The failure mode of naive routing is the human default too.
Six archetype tasks distilled from real session history, identical prompts across claude-haiku-4-5 / claude-sonnet-5 / claude-opus-4-8 (scored) + Fable 5 (unscored reference), graded blind by a judge outside the scored set against pre-registered rubrics. One trial per cell — single observations, not capability claims. Survival gate = rubric points only; the design and rubrics were attacked by a clean-context adversarial reviewer before running (17 findings fixed, including de-narrating the judgment task and pinning the judge outside the arm set).
| Archetype | haiku-4-5 | sonnet-5 | opus-4-8 | Verdict (this trial) |
|---|---|---|---|---|
| T1 Mechanical transform (CSV→JSON, tricky rules) | 6/6 | 6/6 | 6/6 | Routable to Haiku — outputs value-identical |
| T2 Structured extraction (noisy log, 6 exact answers) | 6/6 | 6/6 | 5/6 | Routable to Haiku — the frontier arm was the one that erred |
| T3 Code comprehension (unfamiliar script) | 2.5/3 | 2.5/3 | 3/3 | Sonnet floor — Haiku misses control-flow side effects |
| T4 Debugging (interacting CSS causes) | 4/4* | 4/4* | 1.5/4* | Contested cell — see note; no tier verdict |
| T5 Adversarial design judgment | ~3.25/4 | 4/4 | 4/4 + bonus | Frontier — Haiku found surface flaws, missed the inference-only ones |
| T6 In-context instruction application (5 encoded gotchas) | 4/5 | 5/5† | 5/5† | Sonnet applied all 5 gotchas; Haiku missed only test-fire verification |
†T6 was re-rolled once with a corrected harness: the original sonnet/opus cells truncated at a denied tool attempt (a headless artifact, not a capability signal), so the prompt was patched to text-only — matching the pre-registered "tools disabled" design — and all four arms re-run, then blind-judged fresh. Result: sonnet 5/5, opus 5/5, haiku 4/5. Every tier applied the mechanical gotchas (python vs python3, encoding reconfigure, absolute paths, commit -F); haiku's single miss was the test-fire verification step — the habit-shaped requirement, echoing the friction study's lookup-vs-habit split. Both rounds are preserved in the raw data.
*T4: the blind judge flagged that one rubric point ("an ancestor transform breaks position:sticky") conflicts with the reference output; the point traces to the author's own knowledge base — a possible encoded overgeneralization, now filed as a distill signal. Inverting that point flips the T4 ranking; we therefore draw no routing conclusion from T4. The full mapping-decoded verdict, packets and usage are in the repo.
(1) The routing line sits lower than the maintainer's habits assume: lookup-shaped and rule-shaped work survived Haiku untouched in this trial, and the one wrong answer in extraction came from the expensive arm. (2) Verbosity is a hidden tax: for the same 6 extraction answers, Haiku emitted 453 output tokens where Opus emitted 63 — unit price is not per-task price. Route by task shape, then check output discipline.
Learned routing from an agent's own experience logs is essentially unshipped territory (two 2026 surveys — routing-side and memory-side — have zero coverage of the pattern; no framework implements it). Before proposing it as a distill feature, the cheap question: given the same 5-item task list, does an orchestrator with a routing knowledge file choose differently than one without? Per the adversarial review, success means matching E1's ground truth — not parroting the injected file — and the no-file arm's natural behavior is the headline.
Stated routing judgment was already near-perfect — yet 40 days of revealed behavior is a 100% frontier monoculture with zero delegated cheap-tier calls. The binding constraint is not knowledge; it's that the routing decision never gets asked in real sessions. This is the execution gap again (see Token Economics, friction section): prose rules don't change in-flight habits — structure does. The actionable version of "distill learns routing" is therefore less about richer rules and more about installing the decision point: subagent definitions with model: defaults, and a delegation checkpoint in heavy-session workflows.
What the literature calls "memory-augmented routing" exists as three thin papers and no products. Distill already has the full loop the papers lack: session retrospectives that notice which delegations succeeded on which tier (signal), a knowledge store with confidence/staleness metadata (memory), and a session-start monitor that injects the rules where delegation decisions happen (retrieval). The proposal that survives tonight's evidence:
model: frontmatter covers this; no external router earns its fee here./model flips don't).