The Story
Here’s a problem you’ve probably run into without naming it. You’ve been chatting with an AI assistant for a while, you mentioned something important twenty messages ago, and then you ask a follow-up — and it acts like that conversation never happened. The model didn’t “forget” in any human sense. It just never had a place to keep that fact.
That’s the gap a new paper from the University of Massachusetts Amherst and the University of Alberta is poking at. The work, titled “Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in LLMs” (arXiv 2510.27246), comes from a team including Mohammad Tavakoli, Alireza Salemi, Hamed Zamani, and J Ross Mitchell. And the fix they propose is almost funny in how low-tech it sounds: give the model a notepad.
Not literally a Post-it, obviously. But conceptually that’s exactly it. The framework is called “LIGHT,” and the headline idea is that instead of trying to cram an entire conversation into the model’s context window — the chunk of text it can “see” at once — you let the model keep a running scratchpad of the stuff that matters.
Let me back up and explain why this is even a question.
When people talk about an LLM’s “memory,” they usually mean the context window. Modern models brag about giant ones — a million tokens, sometimes ten million. (A token is roughly three-quarters of a word.) The pitch is: the window is so big, you’ll never run out of room. Just dump everything in.
The catch is that a big window and a good memory are not the same thing. Researchers have shown for a while now that models get worse at finding a specific fact as you bury it deeper in a long context — the “lost in the middle” effect. So a 1-million-token window doesn’t mean the model actually uses all million tokens well. It means it can technically read them. Reading and remembering are different jobs.
So the UMass and Alberta team did two things. First, they built a tougher test. Second, they built the notepad system to beat it.
The test is called “BEAM.” It’s a benchmark of 100 conversations, some short, some absurdly long — stretching from 100,000 tokens all the way to 10 million. That upper end is the point of the title: most models physically cannot fit a 10-million-token conversation in their window at all. On top of those conversations sit 2,000 hand-validated questions, sorted into ten different “memory abilities.” And those abilities are more interesting than just “find the fact.” They include things like resolving contradictions (you said one thing on Tuesday and the opposite on Friday — which holds?), event ordering, knowledge updates, tracking a user’s stated preferences, and multi-hop reasoning where the answer requires stitching two distant facts together. That’s a much closer match to how real long conversations actually behave.
Now the system itself. LIGHT borrows its structure from how cognitive scientists describe human memory, and it runs three things in parallel.
There’s “working memory” — just the last few turns of the conversation, kept verbatim. That’s your short-term recall, the stuff still fresh.
There’s “episodic memory” — an indexed archive of the whole conversation. After every turn, the system pulls out key facts and summaries, converts them into searchable vectors, and files them away. When you ask a question later, it retrieves the relevant slices. If you’ve heard the term “RAG” (retrieval-augmented generation), this is that idea, applied to the chat history itself.
And then there’s the star of the show, the “scratchpad.” After each turn, the model reasons over what was just said and writes down the salient bits — actively, in its own words. The scratchpad isn’t a raw transcript. It’s a curated set of notes the model keeps for itself. And here’s the clever part: when the scratchpad gets too long, it doesn’t just grow forever. Once it crosses 30,000 tokens, the system compresses it into a tighter 15,000-token summary. The paper explicitly frames this as mirroring how human memory consolidates — you don’t store every sensory detail of last Tuesday, you store the gist.
At answer time, the model draws on all three at once: the fresh turns, the retrieved archive, and the curated notes. A filtering step trims the scratchpad down to only the chunks relevant to the current question.
Does it work? The numbers say yes, and they get more dramatic the longer the conversation runs. At 100K tokens, LIGHT improved scores by roughly 44–49% over plain long-context models, depending on the backbone. At 1 million tokens, the gain jumped to 60–76%. And at 10 million tokens — where the baseline models simply can’t see the whole thing — LIGHT posted gains of 107% for GPT-4.1-nano and 156% for Llama-4-Maverick. The team tested it across several models, including GPT-4.1-nano, Gemini-2.0-flash, Qwen2.5-32B, and Llama-4-Maverick, so it’s not a quirk of one architecture.
One honest caveat the paper itself surfaces: averaged across all settings against the strongest baselines, the improvement is a more modest 3.5–12.7%. The eye-popping percentages live at the extreme long-conversation end. Which, frankly, is exactly where you’d want them.
The Takeaway
The thing I keep coming back to here is how stubbornly the industry has been selling the wrong fix. For two years the marketing pitch has been window size. Bigger window, bigger number, bigger memory — that’s the implied math. This paper is a quiet, well-evidenced argument that the math is wrong. A bigger window gives the model more to read; it doesn’t give it a better sense of what’s worth keeping.
That distinction connects to a thread we’ve followed before on this blog. Back in November we wrote about “Beyond Attention,” the search for alternatives to the Transformer’s self-attention mechanism, much of it motivated by how brutally expensive long context gets. And we covered Cerebras shipping a long-context coding model built to hold huge amounts of code in view at once. Both of those were, in their own way, attempts to make the window bigger or cheaper. LIGHT is the other school of thought entirely — leave the window alone, and get smarter about what flows through it. My read is that the long-context arms race and the memory-architecture school are going to converge, not compete. You’ll want an efficient window and a notepad sitting on top of it.
There’s also a reason this lands now. We’ve spent the last several months covering the shift to AI agents — Google’s “agentic era,” Anthropic picking up Vercept to push computer-using agents. Agents are the use case that breaks the old approach hardest. A chatbot session is short and disposable. An agent that works alongside you for days or weeks is, by definition, one long unbroken conversation. It needs to remember that you corrected it on Monday, that your project’s deadline moved, that you prefer one tool over another. Stuff a stateless model into that role and it fails in slow, frustrating ways. The scratchpad is less a feature for chatbots than a prerequisite for agents.
What I like most, though, is the modesty of the idea. There’s no exotic new architecture here, no retraining of the base model. LIGHT wraps around an existing LLM and orchestrates it — extract notes, compress notes, retrieve notes. That makes it the kind of advance that can show up in products fast, because it doesn’t require anyone to rebuild their foundation model. The flip side: orchestration means more moving parts, more prompts, more places for the system to misjudge what’s “salient.” A notepad is only as good as the judgment writing on it. If the model decides the wrong fact matters, it’ll confidently consolidate a mistake and carry it for the rest of the conversation.
Still, the direction feels right. The most human thing about memory was never its capacity. It’s the editing — the constant, quiet decision about what to write down and what to let go. Watching AI research arrive at the same conclusion, a notepad at a time, is the part worth paying attention to.
This article is for informational purposes only.
Photo: Domenico Loia / Unsplash
댓글 남기기