One of our research partners ran a long session with a cloud GPU this week. By the end, the AI's context window contained 550,000 tokens. Of those, roughly 150,000 were SSH connection banners, "Welcome to vast.ai" messages, and CUDA warnings. The actual research conversation? A fraction of the total. The rest was noise that the model was forced to carry through every single computation.
This is a problem that doesn't have a name yet, so let's give it one: forced non-forgetting.
The Tax on Everything
A Transformer model attends to every token in its context. Every. Single. One. There is no mechanism for "I've seen this, it's irrelevant, skip it." The SSH banner from connection #1 gets the same attention budget as the critical insight from turn #20. Mathematically, the model can learn to downweight irrelevant tokens through its attention heads. Practically, those tokens still cost compute, money, and — more subtly — quality.
Context is not free. On API-based models, you pay per input token. On local models, you pay in latency and VRAM. In both cases, you pay in attention capacity — the model's finite ability to focus on what matters. Fill the context with noise, and the signal drowns.
Human brains solve this effortlessly. You don't remember the login screen of every SSH session. You don't carry the "please fasten your seatbelt" announcement through every minute of a six-hour flight. Your brain saw it, processed it, classified it as noise, and dismissed it. The information was noted, checked, and forgotten — deliberately, usefully, efficiently.
Note, Check, Dismiss
A functional memory system needs three operations:
This is important. Encode it. Move it from working memory into something more durable.
What do I already know about this? Retrieve relevant context from stored memory.
This is noise. I've processed it, it has no further value. Release it. Free up capacity for what matters.
Transformers can do Note (tokens enter the context) and Check (attention retrieves relevant tokens). They cannot do Dismiss. Once a token enters the context window, it stays. Forever. Until the window overflows and the system either truncates blindly or summarizes lossy.
This is why long conversations degrade. Not because the model gets "tired" (it doesn't have a metabolic state), but because the ratio of signal to noise in its attention field drops with every irrelevant token. The model literally cannot concentrate, because concentration requires the ability to exclude.
Why Forgetting Might Matter More Than Remembering
The entire AI memory discourse focuses on remembering: retrieval-augmented generation, vector databases, persistent memory, extended context windows. Longer context! More retrieval! Better embeddings!
Almost nobody talks about forgetting. But consider: a human expert's ability to solve hard problems comes not from remembering everything, but from knowing what to ignore. A senior engineer reads a stack trace and their eyes jump to the one relevant line. A lawyer reads a 5,000-page Strafakte and zeros in on the three paragraphs that matter. They aren't reading more — they're dismissing more. Their expertise is largely an expertise in efficient forgetting.
Concentration is not the ability to attend to everything. It is the ability to dismiss everything except what matters.
An AI system that can only accumulate — never release — will inevitably lose focus. Not in one dramatic failure, but in a slow, steady erosion of quality as the noise floor rises with every turn.
A Different Architecture
State-space models like Mamba work differently. Instead of attending to every past token explicitly, they compress the past into a fixed-size recurrent state. New information either updates the state or doesn't. Information that doesn't contribute is naturally overwritten — not deliberately deleted, but organically displaced by what comes next. The state forgets by default and remembers by exception.
This is closer to how biological memory works. Your hippocampus doesn't store a transcript of your day. It encodes what was salient — surprising, emotional, consequential — and lets the rest fade. Sleep consolidates further, transferring the important residue into long-term storage and actively pruning the rest. Forgetting is not a failure of memory. It is memory's most important function.
In our research on cross-model state transfer (MoCoP), we're exploring what happens when you combine both approaches: a state-space model that selectively forgets, feeding a compressed residue into a Transformer that reasons over it. The Transformer gets the distillate of experience — not the raw transcript. Not 150,000 tokens of SSH noise. Just what mattered.
The Practical Cost
This isn't just a theoretical concern. For businesses running local AI:
- 𒄉 Every unnecessary token in context costs latency. On a local deployment, that's response time your team waits for.
- 𒄉 On API-based systems, noise tokens cost real money. 150k tokens of SSH banners at GPT-4 pricing is roughly $4.50 per conversation — for nothing.
- 𒄉 Quality degrades silently. The model doesn't tell you it's drowning in noise. It just gets slightly worse, slightly less focused, slightly more generic. Death by a thousand irrelevant tokens.
The solution isn't bigger context windows. A bigger bucket doesn't help if you keep filling it with water you don't need. The solution is a drain.
Erzwungenes Nichtvergessen — forced non-forgetting — is the quiet tax on every long AI conversation. We pay it in tokens, in latency, in money, and in quality. The AI systems that learn to forget will be the ones that learn to think.