4 studies · Core capability
Knowledge Retrieval
Does it USE knowledge correctly? A/B testing, memory degradation, confidence scaling, and operational tool reliability.
Headline: tool reliability test shows 0/5 vs 5/5 — strongest delta. Past failures become proactive prevention.
6/8 dimensions · Cognitive protection
Cognitive Bias Detection
Does it PROTECT the human? Anchoring, fatigue, loss aversion, authority, recency, and solution anchoring. Tested across engineer and PM personas.
Headline: distill poisoning discovered and solved. Specificity beats abstraction. Confidence gives permission to push back.
5/5 scenarios · Knowledge composition
Engineering vs Philosophical Frameworks
Can philosophical principles (Stoic/Pragmatist/Dialectical) complement engineering axioms? Tested on ambiguous decisions with no clear engineering answer.
Headline: conditions converge (68/67/68 of 75) — the interim “hybrid wins” signal did not survive the full run. Each scenario was won by a different condition.
7-system benchmark · User personalization
User Model: Always-On Preference Enforcement
Competitive analysis across 7 memory systems revealed user model as distill's weakest category. Root cause: preferences behind lazy-load gates. Fix: always-on inline preferences with enforcement hooks.
Headline: the winning system's advantage was placement, not architecture. Inlining 15 lines of preferences in the rules file eliminates retrieval failure.
656M tokens mined · Cost economics
The Token Economics of Remembering
Does memory pay for itself? 40 days of real transcripts priced distill's all-in overhead, measured what encoded knowledge actually prevents, and stress-tested a compressed index against held-out probes.
Headline: distill overhead is 5–6% of spend while ~85% of cost is context movement. Encoded environment gotchas stop recurring; behavioral rules don't (execution gap, quantified). Diet SPINE = trade-off knob, not free win.
3 tiers · 9 models · 3 sessions · Cost economics
Can a Local Model Do the Distilling?
Distillation's extraction stage is assigned to the cheap tier. How far down does that go — all the way to an open-weights model on the user's own machine? Blind-judged against an Opus reference, normalised to Opus's own self-agreement ceiling. Updated as runs land.
Headline: the answer splits on memory. At 48 GB a local 27B reached 81% of the ceiling at $0, beating both hosted cheap tiers on faithfulness. Replicated on three real sessions: the free local model beat hosted Haiku on every one, grounding 0.882 vs 0.639, zero fabrications against eight. Quantising to 4-bit keeps most of it for half the memory — capacity beats precision by 10× — but 32 GB is still the floor.
8 experiments · partial · Retrieval & cost
A 421M Model, and the 40 Lines of BM25 That Beat It
Can a small local judgment model replace the frontier model’s knowledge-index read? A zero-model control, three routing architectures, a relevance gate, and an option-budget sweep — plus the arithmetic that killed a distillation pre-filter before it was built.
Headline: the turn that reads the knowledge index costs 21,247 tokens — the largest fixed cost of a session, and 5.9× the 3.6k this project last published for the same index. A zero-model BM25 router beats the local model at routing and cuts 70% of those tokens end to end; the model wins only at judging one candidate at a time.
1 store in 2 snapshots + a second store’s index · 10 sessions · design under review · EN / ES
Files-Only Memory Redesign, Measured on Real Data
The files-only redesign (#75, PR #93) gives the knowledge index a byte budget, moves dated history to evidence files, archives by moving and adds an opt-in lifecycle. Its assumptions met one real store in two snapshots, a second store’s index and ten real sessions, and the design changed as a result.
Headline: the index every session reads first is a 51 KB digest, 3.2× its budget, up 65% since Aug 1 with no new entries. Moving dated history fixes 3 of 56 oversize files (splitting does the work), the lifecycle as measured would archive 2 of 23 project files, and the nested archive would have blocked migration. Review rounds six and seven adopted all four findings.
6 archetypes × 3 tiers · Cost economics
Best Model for Everything? Model Routing
Which task shapes survive a downgrade to cheaper models? Blind-judged archetype battery, an adversarially-filtered routing landscape, and a test of whether a knowledge file changes delegation choices.
Headline: lookup/rule-shaped work survived Haiku untouched — the one wrong extraction answer came from the frontier arm. Stated routing judgment is near-perfect; revealed behavior is a 100% frontier monoculture. The gap is habit, not knowledge.
0 benchmark tokens · Cost economics
The Benchmark We Refused to Run: Pre-Distill Marks
Should every session write signal marks so distillation gets cheap? An adversarial review killed the benchmark design (every bias pointed toward a false APPROVE), then free arithmetic on 40 days of real usage killed the feature's token case.
Headline: the best-case saving is 0.3–2.2% of session cost — a cheap-tier batch cold read at distill time is already too cheap to beat. Discarded without spending a benchmark token. Run the arithmetic before designing the experiment.