If every session wrote small "signal marks" as the conversation happened, later distillation could process cheap marks instead of full transcripts. We designed a benchmark to test it, let an adversarial reviewer attack the design before spending a token, and then let free arithmetic on 40 days of real usage answer the question the benchmark was going to ask. The feature's token case died twice — first methodologically, then numerically. This page documents both deaths, because a rejected experiment that cost nothing is a better artifact than a confident wrong answer that cost millions of tokens.
Distillation reads conversations and encodes durable knowledge. Conversations are big; distilling them wholesale risks context limits and re-reads everything to find the few signals that matter. So: inject an always-on instruction that makes the session drop a mark — one small file in the newly shipped INBOX — each time a correction, preference, surprise, or failure happens. The later distillation run consumes pre-extracted marks instead of a transcript. Intuition says: big saving.
Per the project's release rule, anything that trades tokens against something else needs proof before it ships. A benchmark was designed: replay real sessions with marking instructions injected, compare marks-based distillation against full-transcript distillation on tokens and knowledge quality, with a pre-registered approve-or-discard threshold.
Following the project's standing methodology, a clean-context agent attacked the design before any run. Verdict: redesign, not fix-then-run. Four independent fatal findings:
| # | Flaw | Why it's fatal |
|---|---|---|
| 1 | Wrong baseline | Live /distill harvests from the main context, where the conversation is already resident and cache-paid. Measuring the full arm as a fresh transcript ingest manufactures a ~90% "saving" the product never pays. |
| 2 | Missing dominant cost term | A mark written at request 5 of a 120-request session rides along as cache reads on requests 6–120. Our own token-economics study found ~85% of spend is context movement — the tax formula ignored exactly that channel. |
| 3 | Replay destroys the signal class | Replaying only user turns against a fresh agent produces a non-sequitur by turn 4 — corrections and surprises are responses to specific assistant behavior that won't recur. (Fix, had we continued: teacher-forced replay.) |
| 4 | Wrong control arm | The real shipping alternative isn't "full-transcript ingest" — it's a cheap-tier summary of the transcript generated at distill time, which costs nothing on sessions that never get distilled. The design never tested the alternative the decision was actually between. |
Every identified bias in the cost measurement pointed the same direction — toward APPROVE. That is the signature of an experiment design that will produce a confident wrong answer. It is also invisible from inside the design: it took a reviewer with zero authorship context to see it.
Accepting fatal flaw #1 relocates the feature's value. Marks are nearly worthless for manual distillation (warm context). Their honest value proposition is the planned auto-distiller: a background process that must cold-read transcripts it never saw. The question becomes an expected-value inequality:
pauto × (cold-read cost avoided) − (marking tax, carry included) > 0 ?
And that question doesn't need a benchmark. It needs arithmetic on data we already had.
The attack's own recommendation: before redesigning anything, compute the ceiling from the mined 40-day dataset (33 sessions, 3,360 requests, price-weighted with cache reads at 0.1× and output at 5×). Selected rows (mid-tier-input-token-equivalents per session; k = marks per session):
| Scenario | Marking tax m | Avoided (p-weighted) | EV / session | Sessions losing |
|---|---|---|---|---|
| k=3, batch cold read, measured p | ~5.2k | ~27k × p | +12.3k | 9 / 33 |
| k=10, batch, measured p | ~13.6k | ~27k × p | +3.8k | 18 / 33 (median negative) |
| k=20, batch, measured p | ~25.7k | ~27k × p | −8.3k | 25 / 33 |
Two corrections to our own published numbers surfaced on the way: "12 sessions manually distilled" was actually 12 invocations across 7 unique sessions; and those 7 hold 59% of transcript volume — the sessions most worth auto-distilling are the ones already distilled by hand.
A cheap-tier batch cold read of a whole transcript costs only ~27–43k equivalents per session. Against a measured ~3.58M equivalents of total session burn, the best defensible marks scenario saves ~0.34% of session cost; the most favorable assumption stack reaches ~2.2%. Marks aren't dominated by their own overhead at low volume — they're dominated by how cheap the alternative already is.
Discarded on token grounds, zero benchmark tokens spent. Under the project's pre-registered approve-or-discard rule ("significant gain or it doesn't ship"), a 0.3–2.2% ceiling is a discard — no measurement needed beyond the arithmetic.
What survives is a quality question, not a cost one: does cheap cold-read summarization miss signals that in-the-moment marking would catch? And do signals deserve to outlive transcripts that rot before any distill runs? Those belong to the auto-distiller research line, framed as a quality gate on its summarization step — not as a marking tax on every session.
Run the free arithmetic before designing the experiment. The pre-check took a fraction of the tokens the benchmark's first session would have burned, and it answered the question with higher confidence than the flawed benchmark could have.
One user, one machine, 33 sessions, one usage style (long, dense, frontier-model sessions). Transcript sizes were bounded, not directly measured (the pre-check reports low/mid/high brackets; the verdict holds across all three). Mark volume and mark size are modeled, not observed — but the conclusion is robust across a 3–20 marks/session sweep, which brackets any plausible shipping behavior. The quality question is explicitly NOT answered here: this page kills a cost claim, nothing more.
Raw computation and both design documents (the rejected v1 and the attack verdict) live on the research branch: research/2026-08-02-pre-distill-marks/.