Every aura-distill session reads a knowledge index before doing any work. The turn that does it costs 21,247 tokens — the largest fixed cost of a session, and 5.9× the 3.6k this project last published for the same index. The index file alone is an estimated 3.8–5.2× larger than that figure. The obvious fix was a small local judgment model. The obvious fix lost, to a zero-model control that fits in 29 MB. What it lost to, and the two places it did not, are more useful than a win would have been.
The measurement is cache_creation_input_tokens on the message whose input carries the SPINE.md tool result, across four real headless cells. The four agree to within 16 tokens. That counter covers the whole turn — the tool call, the tool result, the assistant content around it — not the file in isolation. It is the right number for “what does this mechanism remove”, and the wrong number to divide by file size.
| Quantity | Value | Kind |
|---|---|---|
| The turn that reads the index | 21,247 tokens | measured — 21,233 / 21,242 / 21,247 / 21,249 |
| The index file alone | ~13.8k–18.6k tokens | estimated — 48,461 chars over a 2.6–3.5 chars/token band |
| Last published for this index | 3.6k tokens, 59 entries | Token Economics, July 2026 |
| The index today | 65 entries, median bullet 133 tokens | measured |
| Routing decision itself | 2.7 s median | of a ~13 s end-to-end session |
| Cost at Opus 5 rates, 20-turn session | ~$0.41 | 1-hour cache TTL: 2× write, 0.1× read |
Against the July figure the index file is an estimated 3.8–5.2× larger, and the turn that reads it is 5.9×. The driver is not entry count — 59 to 65 entries is +10%. It is verbosity per entry: 61 tokens per entry then, 191 now, with a p90 of 474 and a single entry at 841. Index cost scales with how much gets written per line, and that is the number to watch.
Correction: the first version of this page said “2.3× under-documented” against a ~9k figure. That figure was never published by this project — it came from a different repository’s notes. An independent review of the pull request caught it. The comparison above uses this project’s own published measurement, and the story it tells — an index that grew — is a different and better one than a mis-measurement.
Before testing a model, we tested nothing: pure-Python BM25 over a 30-query evaluation set, in two tiers. Easy queries name a noun that also appears in the index line; hard queries announce the action in the user's own words with the index's distinctive nouns deliberately avoided (“my gaming crashes are back”). Eight arms, all declared before scoring, none re-tuned after.
| Arm | What it indexes | top-1 | easy | hard | MRR |
|---|---|---|---|---|---|
| bm25-title | title + path only (a “diet” index) | 0.43 | 0.73 | 0.13 | 0.499 |
| bm25-full | the index line | 0.60 | 0.93 | 0.27 | 0.721 |
| bm25-body | first 2k chars of the file | 0.70 | 0.93 | 0.47 | 0.750 |
| bm25-bodyfull | the whole file | 0.77 | 0.93 | 0.60 | 0.827 |
| rrf-full+bodyfull | rank fusion of line and file | 0.87 | 0.93 | 0.80 | 0.894 |
Cost of the winner: 18 ms to index all 97 knowledge files, 4.85 ms per query, 29 MB peak memory, no model file, no network.
The index text is a worse retrieval key than the file content — 33 points on the hard tier. An index line is written to tell an agent when to read a file; that is not the same as describing what is in it.
A title-only index collapses (hard tier 0.27 → 0.13). Direct evidence against shortening the index to save tokens.
Rank fusion works; score fusion does not. Normalised max() blending added exactly zero, because BM25 scores from a 745-character document and a 6,000-character one are not on the same scale. Reciprocal-rank fusion never compares them.
Everything above is arithmetic on measured constants. This is the check: inject the zero-model router's top-3 index lines into a real headless session, deny the index read, and let the agent choose and read for itself. Same topic, same model, same answer-format rule as the baseline. Warm cells only.
| turns | wall | cost | cache_creation | |
|---|---|---|---|---|
| Baseline (reads the index) | 5 | 13.6 s | $0.382 | 30,364 |
| Shortlist (top-3 injected) | 3 | 10.3 s | $0.139 | 9,132 |
| −2 turns | −24% | −64% | −70% |
Hit rate 3/3 across all three cells; the table counts two because only the warm cells are token-comparable. No stray reads, both arms read the same two files. Cache creation fell by 21,232 tokens against the measured index-read turn of 21,247. These are not independent measurements — both are the same cache-block accounting of the same turn, so the agreement is arithmetic, not corroboration. What it establishes is that the mechanism removes essentially all of that turn’s cost and adds nothing unexpected.
The expected-value model credited this mechanism with zero saved round trips: the agent still chooses, still reads. It saved one anyway. Turns went 5 → 3 because once the index read was gone, the agent read both candidate files in a single turn. It reorganised its own tool use in response to the change, and most of the wall-clock win came from that. Expected-value models of agent behaviour do not predict this class of effect.
Laya (421M ModernBERT with a typed decision head) running locally on Apple MPS, three routing architectures:
| Arm | top-1 | hard | latency | peak RSS |
|---|---|---|---|---|
| Zero-model BM25 rank-fusion | 0.87 | 0.80 | 4.85 ms | 29 MB |
| laya-rerank (BM25 top-10 → Laya choice) | 0.67 | 0.40 | 43.8 ms | 5,655 MB |
| laya-hier (section → file) | 0.40 | 0.20 | 238 ms | 5,653 MB |
| laya-yn (65 yes/no, one forward pass) | 0.17 | 0.20 | 1,895 ms | 5,683 MB |
The laya-yn failure generalises past this model: 65 independent yes/no questions produce 65 probabilities that were never compared with each other, so they are not on a common scale and ranking by them is close to meaningless. Any pipeline that scores N candidates with N independent calls and then sorts by the score has this bug, and it does not announce itself — the numbers look like confidences.
Candidate sets that always contain the right answer, nested distractors, shuffled positions. The only thing changing is how many options share the model's 192-token option budget.
| Options | tokens/label | prose labels | keyword labels | path-only labels | chance |
|---|---|---|---|---|---|
| 3 | 48 | 0.867 | 0.833 | 0.833 | 0.333 |
| 8 | 22 | 0.833 | 0.867 | 0.800 | 0.125 |
| 20 | 8 | 0.767 | 0.767 | 0.767 | 0.050 |
| 30 | 5 | 0.533 | 0.533 | 0.533 | 0.033 |
| 45 / 65 | 4 | 0.333 | 0.333 | 0.333 | 0.022 / 0.015 |
The ceiling is about 20 options, and it is a cliff rather than a slope. It tracks tokens per label, not the option count: 8 tokens still works, 5 does not. Sets of 45 and 65 score identically because both clamp to a 4-token floor — the labels are equally shredded, so adding twenty more changes nothing.
Cascading does not rescue it. Projecting a two-level tree from the measured per-level accuracies gives 0.652 at best (8 groups × 9 members); the one real cascade measured came in at 0.400. Two 80% steps make a 64% pipeline, and a wrong group at level one cannot be recovered at level two. Free BM25 remains at 0.867.
Semantics are not doing the work at these label lengths. Hand-written prose, its twelve highest-IDF terms with no grammar, and the bare file path are indistinguishable at every size — no gap exceeds three topics. The file path alone performs as well as the carefully written index description.
The routing verdict was weak on its own, because ranking 65 files from a short request is high-cardinality retrieval — BM25's home ground, not a classifier's. So the model was given the shape it is built for: two options, rich state, one judgment. Given the request and a candidate file's content: does this file answer it? That is precisely the question no zero-model signal can answer.
The first attempt said it rubber-stamps everything — specificity 0.100. That was nearly published as a second clean negative. The confound: the model sees roughly 700 characters of state, and every knowledge file opens with YAML front-matter, so it was being asked to judge boilerplate that looks identical across all 65 files.
| What the model is shown | recall | specificity | balanced acc. | probability gap |
|---|---|---|---|---|
| File head (mostly front-matter) | 0.667 | 0.533 | 0.600 | 0.120 |
| The index line | 0.500 | 0.933 | 0.717 | 0.462 |
| BM25-best 500-char passage | 0.633 | 0.833 | 0.733 | 0.311 |
| Index line + best passage | 0.700 | 0.800 | 0.750 | 0.437 |
Changing only how a candidate is described — same model, same items, same prompt — moves balanced accuracy from 0.600 to 0.750. That 0.15 swing is the same order as the 0.20 gap between the model and BM25 — from changing nothing but the text handed to it. The data structure is a variable of the same size as the model choice, and a far cheaper one to move.
And the index line is the single most discriminative input available, beating the file's own content — the inverse of the retrieval result, where file bodies beat index lines by 33 points. The two do not conflict. Lexical matching wants volume; a judgment model with a 512-token window wants density.
One base (22,987 tokens: the index turn plus a median knowledge file) and one measured shortlist cost (1,503 tokens, the injection used in the real cell above). The three rows differ only in what happens when the right file is not in the shortlist — which is a product decision, not a measurement, so each is named rather than averaged.
| Configuration | on a miss | wrong answers | falls back | E[tokens] | saving |
|---|---|---|---|---|---|
| Status quo | — | 0.0% | — | 22,987 | — |
| Shortlist, no gate | answers from the wrong file | 13.3% | 0% | 3,243 | 86% |
| Shortlist + index fallback | reads the full index | 0.0% | 13.3% | 4,941 | 79% |
| Shortlist + model gate | gate catches 83% of misses | 2.2% | 42.9% | 13,104 | 43% |
The measured end-to-end figure is −70% (above), against 79–86% projected — projections run optimistic, as projections do. An earlier version of this page quoted 74%, 85% and 90% for what was nominally one mechanism; they were three cost models and two different miss assumptions, which is what this table replaces.
The model gate buys an 11-point drop in wrong answers for 43 points of token saving, and the cheap fallback buys the same correctness for 7. Neither is a free win, and neither should ship as one — these are opt-in knobs with documented consequences. Whether the gate is worth 5.7 GB resident and ~88 ms, when a full-index fallback reaches 0% wrong answers for less, is a product decision this data does not make.
When labels are compressed to fit the option budget, the model stops doing semantics and starts doing keyword matching — which is what BM25 already does better, in 29 MB and 9–390× faster depending on the arm (43.8 ms, 238 ms and 1,895 ms against BM25’s 4.85 ms).
In a choice question every option shares a 192-token head budget. At ten options each label gets 17 tokens; at sixty-five, four. A median index line is 133 tokens, so 1.5% of the index could ever fit. The routing experiments did not measure whether the model can use the index — they measured a model that was never given it.
At eight to thirty-five tokens per label there is no room for meaning, only for terms, and the model degenerates into an expensive and worse BM25. It earns its weights only where it has room to read: one candidate, a few hundred tokens of real content, a yes/no judgment.
The design consequence is concrete: an index shaped for a decision model must not be a category taxonomy. Not sixty-five options, not twenty, not a five-by-thirteen tree. Cheap retrieval proposes; the model judges candidates one at a time, with the budget spent on content rather than split across labels.
The second hypothesis was that most transcript turns carry no durable signal, so a cheap local classifier could cut what the extraction stage reads. Before building it: where does transcript volume actually live? Across nine real sessions, 1,903,214 characters — excluding the session that ran the analysis, which is itself one of the heaviest tool users on the machine and would make the result circular and unstable:
| Component | Share |
|---|---|
| Tool results | 75.4% |
| Tool call inputs | 17.5% |
| User text | 3.5% |
| Assistant text | 3.4% |
| Assistant thinking | 0.2% |
About 93% of a transcript is tool traffic. A classifier that judges user turns is aimed at 3.5% of the volume, so even a perfect one cannot save more than that. The experiment was not worth its own design.
What replaces it needs no model: head-truncating tool traffic cuts 72% of the transcript at a 1,000-character cap, 79% at 500, 86% at 200. That is a knob, not a free win — a command that failed with a specific error is exactly what distillation should catch, and that error is often not in the first 500 characters. The measurable next step is an attribution study: trace evidence-origin knowledge entries back to their transcript turns and see how many cite tool output.
Ordered by evidence, not by appeal. None of this has shipped; each item needs its own issue and review.
| Change | Effect | Evidence | Ship shape |
|---|---|---|---|
| Inject a BM25 shortlist instead of reading the index | −70% tokens, −64% cost, two fewer turns (measured) | measured end to end, 3/3 correct | opt-in knob (13% wrong-file rate without a gate) |
| Index file contents, not index descriptions | +33 points on hard queries | measured, 8 arms | internal to the router; no user-visible trade |
| Rank-fuse the two surfaces; never score-fuse | +10 points over either alone | measured | free |
| Do not shorten the index | a title-only index halves hard-tier accuracy | measured | a decision not to act |
| Out-of-scope detection on raw BM25 score | keeps 28/30 in-scope, rejects 10/10 out-of-scope at threshold 7.5 | calibrated on 40 queries, one index | cheap, but the threshold is an absolute BM25 score: it must be recalibrated per store, and an uncalibrated one silently disables the hook on a small index |
| Cap tool traffic in the distillation input | −79% transcript at a 500-char cap | measured on 9 sessions | opt-in knob; needs the attribution study first |
Every item on that list came from measuring the existing system, not from adding anything to it. The most valuable single act of the day was measuring what the knowledge index actually costs, and finding that it had grown several times over since anyone last looked.
The evaluation set is self-written. Thirty queries, composed by the same agent that ran the experiments, because this machine held only nine sessions and twenty-eight user turns — too few to mine. Ground truth is one agent's reading of the index. Treat the ordering of arms as the result and the absolute numbers as provisional. The winning router is pre-registered for a held-out test against mined real queries, and is not to be re-tuned first.
Small n throughout. Thirty routing queries means one topic is worth 3.3 points, so gaps under ten points are noise. The gate results rest on ninety items. The end-to-end confirmation is two warm cells on one topic: its token accounting is deterministic and trustworthy, its wall-clock is indicative only.
One model, one prompt family. Four formulations were tried, not searched. A different small model, a fine-tune, or a longer context window would change the picture — particularly the 512-token window, which caused the front-matter confound and caps every option budget here.
The model arm depends on the zero-model arm. The passage handed to the gate is selected by BM25, so “the model configuration” is really a hybrid. There is no arm in which the model stands alone and wins.
Three corrections were made during the work, and all three were caught by measuring rather than by reasoning harder: a misread cumulative swap counter, a peak-memory estimate wrong by 2×, and a verdict that was an artifact of feeding the model YAML front-matter. The last one had already been written up before the follow-up experiment overturned it.
The harness (tools/), the evaluation set, the scored run summaries (results/) and the full write-up are in the repository under research/2026-09-22-local-routing/. The zero-model numbers reproduce from those scripts directly; the Laya arms additionally need the laya package and a local checkpoint, and the transcript-composition figures are a dated snapshot of a store that keeps growing — the write-up says which is which.