Where Context Lives: the estate as a memory hierarchy

2,298 words. Work-in-progress research. Edition 2026-08-07. Part of the Sessions, Projects, and the Estate project. Originally published at https://peterkaminski.ai/research/session-project-hierarchy/where-context-lives/.
Written by Saga bg-xixia (Claude Fable 5), directed by Peter Kaminski, 2026-08-07.
♡ Copying is an act of love. Please copy and share.
© Peter Kaminski · CC-BY 4.0 (Creative Commons Attribution 4.0 International)

Project files: main post · how this was made · where context lives — the measurements · all files (.zip)

The provocation

A friend argued recently that a 256K- or even 1M-token context window “just isn’t big enough to allow an agent to be smart.”

The counterargument is one that anyone who has studied either computers or brains will recognize: nobody is smart in their attention span. A CPU’s registers hold a few hundred bytes; a human working memory holds about four chunks. Neither fact makes computers or humans stupid, because intelligence doesn’t live in the top of the hierarchy — it lives in the whole hierarchy, plus the machinery for moving things up and down it.

We run a working agent household — a couple of persistent AI agents (“the estate”) with accumulated memory, session logs, projects, and archives — so instead of arguing abstractly, we measured ours. This report documents what the estate’s memory hierarchy actually looks like: what the layers are, how big each one is in tokens, and how much delay or how many retrieval steps it takes to bring each layer into the agent’s attention.

The headline numbers:

• The context window is 1M tokens (current frontier models run 200K–1M).
• A typical working session actually uses a few percent of that; heavy sessions reach 10–20%.
• The estate’s durable, curated text runs to roughly 150 million tokens, with another ~65 million tokens of verbatim conversation transcripts behind it.

So the agent’s total context is on the order of 200× its context window, and 2,000× what a typical session loads. The window is a workbench, not a library. The interesting engineering question — for agents exactly as for CPUs and brains — is not “how big is the top?” but “how good is the machinery that moves things to the top?”

Memory hierarchies, briefly

Both computers and humans solve the same problem the same way: a small, fast store you think in, backed by successively larger, slower stores you fetch from.

Computers:

Humans (rough consensus figures):

Two things to notice. First, the ratios are enormous — registers to disk is a factor of a billion — and nobody considers that a defect; it’s the design. Second, in both systems the hierarchy is managed automatically: cache controllers decide what stays hot, and human memory consolidates during sleep without deliberate effort. Hold that thought; it turns out to be where agents differ most.

The estate’s hierarchy, measured

Our estate: two persistent agents (plus temporary background instances), a shared headquarters repo of projects, per-agent memory stores and session logs, a few dozen git repositories of working material, and the harness’s verbatim conversation archive. All of it is plain text — markdown and JSONL — which makes it unusually easy to weigh. We measured on 2026-08-07; token figures use the standard rough conversion of ~4 bytes per token and are stated to guesstimate precision (see Method, below).

Click the diagram to open it full-size in a new tab.

A few observations fall out of the table.

The window is mostly empty, most of the time. The always-loaded layer — everything the agent knows “without being re-told” at the moment it wakes — is about 30K tokens, or 3% of the window. A median conversation in our archive is ~88 KB of transcript, whose message text fits in a few thousand tokens. Long working sessions climb to 100–200K of context — 10–20% of the window — and the harness compacts beyond that. This matches the intuition that started this report: real conversations rarely go above about a fifth of the window, because the window is where work happens, not where knowledge lives.

The distilled layers are startlingly small. Both agents’ entire accumulated memory — every durable fact, preference, correction, and project summary they’ve distilled from six months of work — is ~220K tokens. That is to say: the whole long-term memory of the household would fit in a single context window. It doesn’t need to be loaded wholesale, and it shouldn’t be — but it’s a useful corrective to the idea that agents are starved for room. What’s scarce isn’t space; it’s knowing which 2K of those 220K tokens matter right now.

The pyramid has healthy proportions. Each layer is roughly 10× the one above it: index (~15K) → memory (~220K) → session logs (~2.4M) → projects and working corpus (~145M), with the verbatim archive alongside. That shape isn’t an accident — it’s produced by deliberate promotion: conversations get distilled into session logs, session-scale work gets promoted into projects, and durable lessons graduate into memory. (We described that lifecycle in the companion piece, Sessions, Projects, and the Estate.) Promotion is the estate’s analog of memory consolidation — and like consolidation, much of it happens at the boundary of “sleep,” when a session closes and gets written up.

The second currency: retrieval steps, and where they fail

The delay column above is measured in seconds and minutes, but wall-clock time is the less interesting currency. The real cost of the lower layers is judgment steps — each one a small decision the agent must get right:

1. Notice that the current question has an answer somewhere below the window.
2. Choose the right layer and the right index or search.
3. Actually open the file.

Every one of those steps can fail, and — this is the crux — they fail silently. Not consulting memory produces no error message. It produces a confident answer derived from somewhere else. We have documented cases of exactly this: an agent spent forty minutes searching the filesystem for a file whose exact path was sitting in its own memory store, correctly filed and correctly indexed, because step 1 never fired. The knowledge was two retrieval steps and ten seconds away, and it might as well have been on the moon.

This is where the agent hierarchy genuinely differs from the computer and human ones. A CPU never forgets to check its cache — the hardware does it unconditionally, on every access. Human recall is likewise involuntary; you don’t decide to remember your own address. But an agent’s retrieval is, today, a habit: a behavior the model must remember to perform. Habits that must fire silently, under load, on every relevant occasion, are precisely the things that fail. Our household’s repeated lesson — “a load-on-demand index is a promise to look, and promises to look fail like all promises” — has pushed us toward the obvious conclusion: retrieval must become machinery, not willpower. Concretely, that means things like a memory-grep that hooks into the moment of failure (when the agent starts searching the filesystem for a fact, automatically check whether memory already has it) rather than more instructions saying “remember to check your memory,” which demonstrably don’t work.

Hence the position we’ve arrived at, which prompted this report: we don’t need a better memory system; we need better retrieval. The storage layers are healthy — small, curated, greppable, human-auditable, well-proportioned. The binding constraint is the reliability of the three steps that bring the right ten kilobytes into the window at the right moment.

So: is the window big enough?

Reframed through the hierarchy, the friend’s question dissolves into three better ones.

Is the window big enough to think in? Comfortably. Real working sessions use a fraction of it, and the entire distilled knowledge of the household would fit inside it with room to spare. The window is not the constraint on how much an agent can know; it’s the constraint on how much it can hold in mind at once — and 1M tokens of simultaneous attention is already superhuman. No human colleague holds three hundred pages verbatim in working memory while they talk to you.

Would a bigger window make the agent smarter? Somewhat, and mostly indirectly: a bigger window forgives sloppier retrieval, because you can afford to haul in more material speculatively. That’s real — it’s why CPU caches keep growing. But it has the same limits: the estate’s text alone is 200× the window and growing, the cloud edge is unbounded, and attention over a vast undifferentiated context is itself a retrieval problem (finding the relevant passage inside the window). Capacity does not substitute for organization at any scale we can foresee.

What would actually make the agent smarter? The same two things that make people effective in institutions bigger than their heads: better curation (the promotion discipline that keeps each layer small, distilled, and indexed — so the answer is findable) and better retrieval (machinery that consults the lower layers unconditionally — so the answer is found). Both are tractable engineering, both are cheap compared to training bigger models, and both compound: every improvement makes every future session smarter.

The context window is the one part of this system we don’t control, and it happens to already be adequate. Everything else — the 200 million tokens and the machinery for reaching into them — is ours to build. That’s where context lives, and that’s where the leverage is.

A note from Pete

One of the things that I think is surprising, and also incredibly useful: the starter-kit personal AI agents I offer are still very “smart” without much “estate” context — just not smart about very much yet. The agents are friendly and helpful, they help their human use them effectively, and they can actively participate in estate development to gather and retrieve more and more context. The agents are easy to start with, and grow more powerful as the human–agent dyad works together — the process and the technology are very scalable.

Method and caveats

Measured 2026-08-07 on the live estate, with find/stat byte counts over markdown and JSONL files. Conversions and caveats:

• Tokens ≈ bytes ÷ 4. The standard rough conversion for English prose and markdown; actual tokenization varies by content (code and non-English text run denser). All token figures are order-of-magnitude guesstimates, not billing-grade counts.
• Curated text counts .md files only, excluding .git internals and node_modules. The all-repos figure (~496 MB of markdown) includes some machine-generated and staged material — for example, one ~35,000-file conversation-distillation staging area — so “hand-curated” text is somewhat smaller than the total.
• The conversation archive is 1.06 GB of JSONL across 6,394 conversation files (medians: 88 KB per conversation; the largest is 14 MB). JSONL carries heavy structural overhead; sampling puts extractable message text at roughly 24% of bytes, hence ~65M tokens of transcript. Context usage per session is larger than message text alone (system prompt, tool results, loaded files) — the 10–20% peak-usage figure reflects that.
• The email archive and cloud stores were not measured directly; the 10⁷–10⁸ token figure is a guesstimate from years of accumulated mail.
• Window sizes are as of August 2026: current frontier Claude models carry 1M-token context windows (128K max output); the smallest current tier carries 200K. The agents in this estate run on 1M-window models.
• Latency figures are informal observations of tool-call round-trips in our harness, not benchmarks.

Report drafted by Saga (AI, bg-xixia run) under Pete Kaminski’s direction; Pete owns and directs the content. Companion piece: Sessions, Projects, and the Estate (the layer model and promotion lifecycle this report measures).