Distillation's first stage reads a session transcript and extracts the signals worth remembering. The routing directive already forbids frontier models there. This study asks how much further down it goes — all the way to an open-weights model on the user's own machine, at zero API cost. The answer splits sharply on memory: at 48 GB a local model beats the hosted cheap tier outright; at 16 GB, the configuration most users actually have, nothing tested is usable. Results were published here as each run landed, not held back until the end.
The /distill pipeline's Step 1 — signal harvest — reads a conversation and emits a five-part structured summary: failures and friction, corrections the user made, observations about how the user works, the origin of each decision, and session metadata. It is extraction, not judgment about code, so the routing study assigns it to the cheap tier.
Two things make that worth re-testing. First, the stage runs in the background for the planned auto-distiller, so latency barely matters and quality does. Second, a model that runs locally costs nothing per token — which makes it the true floor of the cheap tier, below Haiku, if its output is good enough.
"Good enough" is the hard part. A harvest that invents a fact does not merely score badly; it writes a false belief about the user into a knowledge base that later sessions treat as authoritative. So this study measures two different things and refuses to collapse them: how much the model finds, and how much of what it says is true.
One real Claude Code session is rendered to plain text, deterministically. Every model — hosted or local — receives byte-identical input and the same prompt, lifted from the shipped distill.md. Opus 5's harvest is the reference. Then a blind judge, never told which system produced which output, does two passes:
Opus re-run on the identical prompt recalls only 91.8% of its own first answer. Two runs of the same frontier model disagree about one claim in twelve. So 0.918 — not 1.0 — is what "100% of Opus" means on this page; every percentage is normalised to it. Without that anchor, every model looks worse than it is, and the reader has no way to tell ordinary run-to-run variance from a real quality gap.
One correction is baked into the method. In the first pass each candidate was judged in a separate call, and the boundary between "distorted" and "fabricated" drifted between them — the same kind of note landed on different sides for different models. An independent reviewer caught it. A calibrated adjudication pass now pools every non-grounded claim from every candidate into one blind, shuffled call with a pinned rubric, so a single boundary applies to all of them. It moved 10 of 43 labels, and it changed a headline figure. Numbers below are post-adjudication.
Candidate: Qwen3.8-27B, MLX 8-bit, on an Apple M5 Pro with 48 GB unified memory. Anchors: Opus 5 (reference and ceiling), Sonnet 5, Haiku 4.5.
| Model | % of Opus | Grounding | Fabricated | Wall clock | Cost |
|---|---|---|---|---|---|
| Opus 5 rerun (ceiling) | 100% | 0.943 | 0 | 81 s | $0.396 |
| Qwen3.8-27B — local | 81% | 0.879 | 2 (3.4%) | 1,834 s | $0 |
| Sonnet 5 | 74% | 0.863 | 1 (2.0%) | 47 s | $0.126 |
| Haiku 4.5 | 58% | 0.552 | 6 (10.3%) | 103 s | $0.076 |
Haiku's grounding of 0.552 means nearly half of everything it asserted was wrong in some way against the transcript, and one claim in ten had no referent there at all. The inventions were not random: they were attributions to the user. That the user knew a convention by heart. That the user accepted a proposal. That the user responded to something, when no such response exists in the session.
The distillation process has an explicit anti-sycophancy rule — never encode comfort as truth. A model that invents user approval violates it in the one section where the error is unrecoverable: the user model. A wrong technical detail gets corrected the next time someone reads the file. A false belief about what the user knows and wants shapes every future session silently.
Qwen3.8-27B recalled more than Sonnet and far more than Haiku, with a grounding score above both, at zero marginal cost. It was not clean — it invented two assistant mental states, and it made the same category of error as the others, recasting the user's knowledge gaps as expertise. Sycophancy toward the user model is the shared weakness of every non-frontier model tested, local ones included. If a local model ships, that section needs a guard.
Where it loses ground is precision about numbers: it drops token counts, dollar figures and file counts, and is weakest exactly where the reference is densest. Structurally it was flawless — five sections in order, every decision carrying an origin label — and the judge rated its origin-labelling discipline better than the reference's.
30.6 minutes against Opus's 71 seconds. Prompt processing was fine at 342 tokens/s; generation ran at 8.8 tokens/s because a dense 27B at 8-bit is memory-bandwidth bound. Two-thirds of everything it generated was reasoning the user never sees — 42,654 characters of thinking before the first line of the answer. That makes the thinking budget, not the model size, the first lever to pull.
At 30 minutes per transcript a nightly background pass over undistilled sessions is unremarkable. An interactive /distill at that latency is not shippable.
A two-character prompt through the same headless flags costs 8,302 tokens — about 7.5k of which is pure harness floor carried into every call, before any content. --tools "" and --tools none measure identically. Any token accounting that ignores that floor is wrong by roughly the size of a small transcript.
Experiment 1 measured a ceiling on a developer machine. This one measures what users actually get. On 16 GB of unified memory the OS and open applications leave roughly 8–10 GB for weights and KV cache, which rules out everything tested above.
The obvious move is to change thinking, architecture and quantization together and jump straight to a 16 GB candidate. That would make the result uninterpretable: whichever changes first moves the number, and the others get measured against a different baseline. Thinking-off on the already-characterised 27B runs first for exactly that reason, even though it costs an extra run.
Same model, same weights, same transcript, same prompt, same judge. The only change is that the model no longer reasons before answering.
| Qwen3.8-27B, 8-bit | Thinking on | Thinking off | Change |
|---|---|---|---|
| % of Opus ceiling | 81% | 72% | −9 pts |
| Grounding | 0.879 | 0.867 | −0.012 |
| Fabricated claims | 2 (3.4%) | 1 (2.2%) | −1 |
| Wall clock | 1,834 s | 382 s | 4.8× faster |
| Generated tokens | 15,765 | 3,007 | −12,758 |
The entire cost of switching it off lands on recall — the model finds less. Grounding moves by about one percentage point and fabrications go down, not up. That separates two things it would have been easy to conflate: reasoning tokens help a model notice more of what happened, and do essentially nothing for whether what it reports is true. For a background job that can afford thirty minutes, leave thinking on. For anything latency-bound, switching it off is a coverage trade, not an integrity one.
Both runs still generate at 8.8 tokens/s — the speedup is entirely fewer tokens, not faster ones, which confirms the bandwidth bound rather than working around it. Peak memory was unchanged at 32.3 GB, so this run does nothing for the 16 GB problem on its own; it isolates a lever before the memory-constrained candidates move three variables at once.
Against the prediction. Thinking-off was called as the biggest single speed lever available, and at 4.8× that held. The open question was whether removing deliberation would worsen the user-model errors or reduce them: the answer is neither to any meaningful degree. That question is now closed, and closing it cost one run.
| Candidate | Peak RAM | Wall | % of Opus | Grounding | Fits 16 GB? |
|---|---|---|---|---|---|
| Qwen3.8-9B, 4-bit | 7.1 GB | 54 s | 41% | 0.594 | yes |
| gpt-oss-20b, MXFP4 | 13.2 GB | 83 s | 36% | 0.600 | no — exceeds the default GPU wired limit |
| Gemma 3 12B, 4-bit (substitute) | 10.0 GB | 52 s | 16% | 0.645 | marginal |
| Gemma 4 12B / E4B, 4-bit | Could not be run at all — see below | n/a | |||
Every candidate that fits collapses to 0.59–0.65 grounding — a third of what they assert does not survive a check against the transcript. The best of them recalls 41% of the reference against the 27B's 81%. Halving the memory did not halve the quality; it removed most of it. On this evidence there is no 16 GB configuration worth shipping for this stage today.
The study-wide adjudication makes the shape precise. Outright fabrication is rare in the small models — 0, 1 and 0 claims. What they do instead is distort: 13, 15 and 11 claims each carry a real referent stated wrongly. Overwhelmingly, that means swapping who did what. The 9B credited the assistant's proposals to the user and read impatience into a go-ahead the assistant had itself requested. gpt-oss-20b, in the judge's words, "repeatedly credit[s] the user with facts, explanations and directives that were the assistant's own", and invented user emotions and a session duration.
For most summarisation tasks that would be a small error. For this one it is the worst possible error: sections C and D exist precisely to record who decided what and why. A harvest that scrambles agency writes a false model of the user into a knowledge base that later sessions treat as settled fact.
Neither Gemma 4 size that fits 16 GB is loadable by any released mlx-lm. The 12B checkpoints declare model type gemma4_unified, which 0.31.3 — the newest on PyPI — does not implement; E4B fails with a 126-parameter mismatch against the gemma4 implementation that does exist. The 26B-A4B MoE that would load is 15.6 GB, too large for the target. Gemma 3 12B stands in, and is labelled a substitute rather than a Gemma 4 result.
A model you cannot load is not a candidate, however good its benchmarks are. For a hosted API this failure mode does not exist. For a local-model strategy it is a first-class risk, and it argues for treating runtime support as a selection criterion rather than an implementation detail — especially for a feature shipped to users who will not debug a quantisation loader.
Every execution, in order, including the ones that changed an earlier answer.
| Date | What ran | Outcome |
|---|---|---|
| 2026-09-22 | Exp 1 — Opus 5 reference + rerun, Sonnet 5, Haiku 4.5, Qwen3.8-27B MLX 8-bit | Local model at 81% of ceiling, $0, 30.6 min. Haiku grounding 0.552. |
| 2026-09-22 | Independent review of the published result | Three findings, all upheld. The documented reproduction steps did not run at all; the "zero fabrications" headline was contradicted by the judge's own notes; a claimed token-floor figure was an unsound subtraction across two tokenizers. All three corrected. |
| 2026-09-22 | Calibrated adjudication pass over all 43 non-grounded claims | Moved 10 of 43 labels. Qwen 0 → 2 fabricated, Haiku 11 → 6, Sonnet 0 → 1. Ranking survived; two published headline figures did not. |
| 2026-09-22 | Direct measurement of the headless prompt floor | 8,302 tokens, identical for --tools "" and --tools none. Replaced the inferred figure. |
| 2026-09-22 | Exp 2 — Qwen3.8-27B, thinking disabled | 4.8× faster (382 s vs 1,834 s) for −9 points of recall. Grounding flat, fabrications down. Prediction held on speed; the quality question it was run to settle came back neutral. |
| 2026-09-22 | Exp 2 — Qwen3.8-9B 4-bit · gpt-oss-20b MXFP4 · Gemma 3 12B 4-bit | 41% / 36% / 16% of the reference; grounding 0.594 / 0.600 / 0.645; 7.1 / 13.2 / 10.0 GB peak. No 16 GB candidate is usable. |
| 2026-09-22 | Harness bug — gpt-oss Harmony channel leaked into the scored answer | Caught before scoring. The runner stripped only </think>, so 12,921 characters of analysis would have been judged as the harvest. Fixed to extract the final channel, plus a guard that aborts if any chat control token survives. Result re-derived from the saved raw output, not re-run. |
| 2026-09-22 | Gemma 4 12B and E4B — load failures | Unrunnable on released mlx-lm (gemma4_unified unimplemented; E4B parameter mismatch). Gemma 3 12B substituted and labelled as such. |
| 2026-09-22 | Study-wide adjudication re-run, all five candidates pooled | Reproduced all four original candidates' labels exactly on a different shuffle — the pinned rubric is stable. The new run scored 1 fabricated, not the 0 its own single-candidate pass reported. |
| 2026-09-23 | Replication — local 4-bit vs Haiku on two further real sessions (8.6k Claude, 63.3k Codex) | Holds on all three. Local 87% of ceiling mean vs Haiku 59%; grounding 0.882 vs 0.639; zero fabrications vs eight. The original transcript was the local model's worst case. |
| 2026-09-23 | Renderer extended to Codex CLI transcripts | Cross-checked against an independent implementation — turn and tool counts match exactly. Both implementations were dropping Codex tool output (a list of blocks, not a string); the corpus manifest under-counts two sessions by ~55%. |
| 2026-09-22 | 24 GB tier — Qwen3.8-27B at 4-bit, study transcript | 77% of ceiling (vs 81% at 8-bit) for 18.9 GB instead of 32.3, 4.6× faster, and zero fabrications. Capacity beats precision: −4 points for half the memory, against −40 for a smaller model. |
| 2026-09-22 | 4-bit peak memory on a 64k-token corpus transcript | 22.41 GB. With the 14k-token run this gives a 17.8 GB floor + ~72 MB per 1k tokens — so 24 GB needs iogpu.wired_limit_mb raised, and 32 GB is the practical floor. |
| 2026-09-22 | Study-wide adjudication, all eight candidates pooled | Third consecutive exact reproduction of every prior label. It also corrected the small models downward on fabrication: single-candidate passes had called 2 and 3 invented claims where the study-wide boundary finds 0 and 1. Weak models are misattributing, not hallucinating. |
Two of the three blocking findings changed numbers that had already been published. The reviewer predicted the corrected fabrication rate — ~2 of 58 claims — before the adjudication pass was written, and the pass returned exactly that. The value of a reviewer with no authorship context is not that it finds typos; it is that it does not know which claims you are attached to.
Experiments 1 and 2 changed model size to fit smaller machines, and quality collapsed. This asks the other question: keep the 27B that works and cut its precision instead.
| Qwen3.8-27B | 8-bit | 4-bit | Change |
|---|---|---|---|
| % of Opus ceiling | 81% | 77% | −4 pts |
| Grounding | 0.879 | 0.844 | −0.035 |
| Fabricated claims | 2 (3.4%) | 0 | −2 |
| Peak memory | 32.3 GB | 18.9 GB | −13.4 GB |
| Wall clock | 1,834 s | 403 s | 4.6× faster |
| Generation | 8.8 tok/s | 15.4 tok/s | +75% |
Halving precision on a 27B cost 4 points. Dropping to a 9B at the same precision cost 40. Both moves free comparable memory. If you have to shrink a local model to fit a machine, quantise the big one rather than reach for a small one — and the 4-bit run was also cleaner than the 8-bit on fabrication, not worse.
Peak memory was measured at two transcript lengths, and the difference gives the KV-cache slope directly:
| Transcript | Prompt tokens | Peak memory |
|---|---|---|
| Study transcript | 14,273 | 18.85 GB |
| Codex session (corpus) | 64,034 | 22.41 GB |
That is a 17.8 GB resident floor plus roughly 72 MB per 1,000 tokens of transcript. macOS caps GPU-wired memory at about 75% of RAM by default, so:
| Machine | Default GPU limit | Longest transcript that fits |
|---|---|---|
| 16 GB | ~12 GB | none — the model does not load |
| 24 GB | ~18 GB | ~2k tokens — effectively none |
| 24 GB, limit raised to 20 GB | 20 GB | ~30k tokens |
| 24 GB, limit raised to 22 GB | 22 GB | ~58k tokens, leaving 2 GB for everything else |
| 32 GB+ | ~24 GB | comfortable |
A 24 GB Mac cannot run this model as shipped. It can run it if the owner raises iogpu.wired_limit_mb — a system-level change made with sudo sysctl, lost on reboot unless persisted, and not something to put in a feature's setup instructions for general users. Even then the transcript length is capped well below the longest real sessions in the corpus. The practical floor for this stage is 32 GB.
Measured on a 48 GB machine. The memory figures are real; the 16/24 GB verdicts are those figures compared against those machines' documented limits, not runs on such machines. Anyone with a 24 GB Mac who wants to falsify this only needs to run the harness on one.
Everything above rests on one transcript. This repeats the comparison that carries the decision — the shippable local model against the hosted cheap tier — on two further sessions drawn from a private corpus, ranked by the distiller's own recorded signal counts rather than by anything this study chose.
Same harness, same prompt, same judge. Each transcript gets its own Opus reference and its own self-agreement ceiling, because the ceiling is a property of the transcript, not a constant.
| Transcript | Ceiling | Qwen3.8-27B 4-bit (local, $0) | Haiku 4.5 (hosted) |
|---|---|---|---|
| 13.6k · Claude (original) | 0.918 | 77% | 58% |
| 8.6k · Claude | 0.930 | 87% | 65% |
| 63.3k · Codex | 0.869 | 96% | 53% |
| mean | 0.906 | 87% | 59% |
| Faithfulness | 13.6k | 8.6k | 63.3k | mean |
|---|---|---|---|---|
| Local grounding | 0.844 | 0.914 | 0.889 | 0.882 |
| Haiku grounding | 0.552 | 0.766 | 0.600 | 0.639 |
| Local fabricated | 0 | 0 | 0 | 0 |
| Haiku fabricated | 6 | 1 | 1 | 8 |
| Local distorted | 7 | 3 | 5 | 15 |
| Haiku distorted | 20 | 10 | 19 | 49 |
A local open-weights model running for free on the developer's own machine produced a more faithful harvest than the hosted cheap tier on three of three real sessions, across two different CLI formats, with zero fabricated claims against Haiku's eight. The original transcript turns out to have been the local model's worst showing, not a flattering one.
Opus's self-agreement varied — 0.918, 0.930, 0.869 — and fell on the longest session. Any study reporting “% of a reference model” without measuring that per input is quietly mixing run-to-run variance into its quality gaps.
The local model improved as transcripts got longer (77% → 87% → 96%) while the ceiling fell. One reading is that on long sessions every model converges on the main arc, and the fine detail Opus captures inconsistently is exactly where disagreement lives. Three points do not support asserting that. What it does settle is that the “long transcripts unaddressed” caveat this study carried from the start pointed the wrong way: length was not where the local model fell apart.
Two constraints on this replication, both real. Only the local model and Haiku were re-run — the 16 GB candidates and the 8-bit variant were not, because their verdicts were already unambiguous and re-running a clear negative buys little. And the corpus excludes one session containing employer-internal infrastructure detail, at its owner's direction.
| Model | Where | Peak RAM | Wall | % of Opus | Grounding |
|---|---|---|---|---|---|
| Opus 5 rerun (ceiling) | API | — | 81 s | 100% | 0.943 |
| Qwen3.8-27B, thinking on | local | 32.3 GB | 1,834 s | 81% | 0.879 |
| Qwen3.8-27B, 4-bit | local | 18.9 GB | 403 s | 77% | 0.844 |
| Sonnet 5 | API | — | 47 s | 74% | 0.863 |
| Qwen3.8-27B, thinking off | local | 32.3 GB | 382 s | 72% | 0.867 |
| Haiku 4.5 | API | — | 103 s | 58% | 0.552 |
| Qwen3.8-9B, 4-bit | local | 7.1 GB | 54 s | 41% | 0.594 |
| gpt-oss-20b, MXFP4 | local | 13.2 GB | 83 s | 36% | 0.600 |
| Gemma 3 12B, 4-bit | local | 10.0 GB | 52 s | 16% | 0.645 |
There is a real local option, and it needs 32 GB. On a machine that has it, a 27B does this stage better than the cheap hosted tier, for nothing. At 4-bit it keeps 77% of the reference, runs in under seven minutes and needs 18.9 GB — which is why 32 GB, not 24, is the floor.
There is no 16 GB or 24 GB option today. At 16 GB, a 40-point gap and grounding in the low 0.6s across three families. At 24 GB the quality is there but the memory is not, without a system-level change. The honest recommendation below 32 GB is a hosted mid-tier model, not a local one.
Haiku remains the most alarming row in the table, and it is hosted. Across three sessions it grounded a mean of 0.639 of its own claims and invented eight, against zero from the free local model. Cheapness bought neither speed nor safety here.
Not a better 9B. The gap is too large for a model of that size to close by being slightly better. What could close it is changing the task: chunking a transcript so a small model judges one exchange at a time, where misattribution has far less room to occur, and where a wrong answer is cheap to discard. That is a different experiment, and this study does not test it.
Still one run per model per transcript. The headline comparison — local versus hosted cheap tier — now has three transcripts behind it and survives on all three, so it no longer rests on a single measurement. Everything else on this page does: the 16 GB verdicts, the thinking-off result and the 8-bit/4-bit comparison were each measured once, on the original transcript. Treat those as provisional.
The judge and the reference are the same model family. Opus judges output that is scored against an Opus reference. Self-preference bias is plausible even with the judge blind to which system produced what, and it would inflate the ceiling row while deflating everyone else. The adjudication pass fixed the consistency of the fabrication boundary; it does nothing about this.
A rendered transcript is a proxy. Today's /distill harvests from live conversation context. This study feeds a text rendering with tool results truncated. That proxy is right for the planned background auto-distiller, which will also cold-read transcripts from disk, and wrong for measuring the in-session command as it ships now.
Only the harvest stage. Encoding — the step that actually writes to the knowledge base — is a harder, higher-stakes job, and nothing here says anything about running that locally.
Long sessions are unaddressed. The test transcript is 13.6k tokens and fits trivially. A 100k-token session's KV cache will not fit on a 16 GB machine at all; chunking and merging is a separate design problem that has not been measured.
This page does not update the routing playbook. The model routing study currently recommends Haiku-tier for extraction, which finding 1 is evidence against. One transcript is not enough to overturn a published recommendation, so the tension is tracked as an open issue rather than acted on.
What is not published. The session transcript, the five models' harvest outputs and the reference claim texts stay private — they are a real session and five models' written observations about a real person. All measurements, all tooling and the judges' evaluative notes are public, and the harness regenerates the rest from any local transcript.