A mind that sleeps without changing its brain

Abstract

This is about memory: what an agent should do with the experience it accumulates while working, so that it is still useful tomorrow. Behrouz et al., Language Models Need Sleep (arXiv 2606.03979, preprint), argue that a model must periodically consolidate its in-context experience into long-term memory — and they consolidate into weights. That route is closed to anyone running a model they neither own nor retrain. We asked whether the same memory consolidation works when the only available write target is files. Two arms were built over one corpus of real, captured agent-judgment — the accumulated experience itself — and scored on a held-out slice: Arm A retrieves from an external file memory with the model frozen; Arm B folds the same experience into weights. They tie. Arm A leads by ~3 points at n=32, inside the noise of a slice that small, and given more data Arm B pulls ahead. The finding is that files are a viable memory substrate at this scale, not a better one.

Read this as:

a method release with a directional result — not a claim that files beat weights, and not a powered benchmark.

A frozen language model — one whose weights we neither own nor update — can still do the consolidation a recent preprint argues models need. Instead of writing what it learns back into parameters, it writes to an external file corpus. We put the two substrates head-to-head on real, captured agent-judgment. On a fair, balanced benchmark they tie at this scale.

predictor (held-out test, n=32)	accuracy	macro-F1	ECE
Arm A — external-file retrieval, cold-spawn (tool_uses=0)	0.688	0.683	0.057
Arm B — weights (LogReg + temperature scaling, balanced)	0.656	0.648	0.068
majority baseline (all-ENDORSED)	0.531	—	—

Read this as a directional proof-of-concept, not a powered result. The test slice is 32 rows from a 105-row corpus. Arm A edges Arm B on all three metrics, but the gap sits inside the noise of a slice this small. The claim is “tie,” not “beat.”

Consolidation is the value; the substrate is a choice

Behrouz et al., Language Models Need Sleep (arXiv 2606.03979, preprint), argue that a model must periodically consolidate its in-context experience into longer-term memory, and they do it into parameters — upward distillation plus an RL phase they call “Dreaming.”

Take the same job to a model we do not control. A frozen actor has exactly one write target left: files. It reads its own trajectory, decides what was worth keeping, and appends to a corpus it can retrieve later. Same consolidation loop, different substrate.

WAKEact → trajectory

→

SLEEPwrite what it learned

→

CORPUSfiles · retrieval

TRAINED MODELweights

→

TIE

The first test was a trap: the label, not the substrate

The first version of this benchmark (v1) returned a null. Both arms collapsed to the base rate. Under a subjective label with a 74% class skew and only 7 held-out rows, Arm A scored 2/7 = 29% with ECE 0.31 — no better than guessing the majority class. It would have been easy to read that as “files can’t consolidate.”

It was the label. We changed only the label — from the subjective, skewed target to an objective, balanced one (ENDORSED?: did the operator approve the decision) — and left both pipelines untouched. On the resulting v2 benchmark both arms moved from base-rate to above-baseline on accuracy and macro-F1 simultaneously, which neither arm could do on v1. The variable that moved the result was the label, not the substrate.

On that valid benchmark the two substrates land on each other (table above). Arm A marginally leads on all three metrics, but Arm B does not clear the pre-registered bar — beating Arm A on both macro-F1 and ECE. The honest reading is a tie. (The v1 figures are contrast only; they describe a different fixture and are never mixed into a v2 statement.)

Power cuts toward weights

The tie rests entirely on the small matched slice, and the honest tension runs the other way. Give the weights arm more data.

A ~3-point Arm A–Arm B gap at n=32 is a smoke test, not a verdict.

The parametric side also carries advantages this benchmark never measured: zero-retrieval availability (weights are active on every token — no “did the model think to grep”), generalizing compression (weights interpolate to novel-but-related inputs where a file store returns the stored item verbatim), and implicit compositionality. Our benchmark scores calibrated-label quality only. It says nothing about these.

The result is also model-relative. Arm A matched weights while making zero tool calls, because a capable actor (Opus-4.8-era) reliably reads the corpus into context. A weaker deployment target would shift the residual value of auto-injection from margin to necessity. The tie is a capability claim, not a law — and the experiment will be extended as the corpus grows.

The corpus labels itself

The scale story is a direction, not yet a result. As decisions accumulate, they connect with typed edges — confirms, supersedes, refutes — and those edges yield an objective HELD / OVERTURNED label by construction. Inter-rater agreement is κ=1.0 because the label is the graph structure: no human tagging, and it grows automatically as the corpus grows.

D-Aa decision, left standing →HELD

D-BrefutesD-A⟹D-A→OVERTURNED

D-CconfirmsD-A⟹D-A→HELD

Flag it plainly: this label is underpowered today. The corpus is roughly 92% HELD, with 9 overturns out of 115. It is imbalanced and not yet a validated benchmark — present as the scale-label direction, not as a load-bearing number.

Capture the corpus, don’t train the brain — and where training still fits

Answered from the frozen side, the counter-position to Language Models Need Sleep is not “train the model.” It is “capture the corpus, and let the corpus label itself.”

Keep that actor-specific. It holds for the capable actor we choose not to retrain; it is not “weights are useless.” Narrow inner-loop components can legitimately live in weights — Arm B is one — and internalization buys availability and generalization that this benchmark does not score.

Two payoffs fall out of the same result, and each is a separate future piece. The ENDORSED? label — predicting operator approval — is the autonomy dial. The self-labeling graph is a self-building decision graph. Named here as payoff, not expanded.