Letta FrontierMemory Shows How Agent Harnesses Shape AI Memory Performance

Oct 22, 2026 827 views
Silhouette with tangled string in the brain and two hands holding the string

The central finding from Letta’s new FrontierMemory benchmark is less about which model remembers most and more about what an agent system does with its history. Across five harnesses, scores ranged from 26.7% to 73.3% on a test designed to measure whether agents can turn prior work into better performance on later tasks. That spread suggests the software around a model can have a substantial effect on whether memory proves useful.

Letta Dream led the harness comparison at 73.3%, followed by Prime Agent at 60%. OpenAI Agents SDK scored 40%, Claude Managed Agents reached 33.3%, and Hermes Agent scored 26.7%. The results, released Wednesday, are an early snapshot: FrontierMemory 1.0 includes 15 tasks, with one attempt per task.

Testing whether experience changes the next attempt

Many memory evaluations focus on retrieving facts from a stored record. Letta argues that this misses a harder question: can an agent learn from previous work and apply that learning when the next task calls for it? Co-founder and CEO Charles Packer made the case on X, saying that recall has become a solved problem in the coding harness and bash loop era, while continual learning through an agent’s context remains an open challenge.

That distinction matters for teams building agents for repeated work. A system may retain details from a previous interaction and still fail to recognize which details are relevant later. Memory has practical value only when it changes how the agent approaches a subsequent task.

Letta co-founder Sarah Wooders framed the issue in a post on X by comparing benchmarks with workplace experience. Many coding and knowledge-work tests start each task with all relevant context already supplied. Real work often builds on what came before: a new employee needs detailed instructions, while an experienced colleague may know how to proceed with far less explanation.

Packer’s premise is that “if AI-generated memory is good, it should actually improve downstream task performance.”

FrontierMemory puts that claim into a task sequence. Agents receive transcripts from earlier work, including both successful and failed attempts. They create memories from those transcripts, then face later tasks where that prior experience can help. Letta adapted existing coding and knowledge-work evaluations for this setup, reviewed and corrected issues in the source tests, and selected 15 tasks for version 1.0.

The approach reflects a broader interest in agents that improve through accumulated experience. Developers have been combining language models with memory and feedback systems to build continually learning agents. FrontierMemory offers a way to ask whether those systems deliver better results on subsequent work, rather than simply storing more information.

Harness design changes the outcome

Letta’s comparison points to the harness as a meaningful part of the result. In one head-to-head test, Letta Dream beat OpenAI Agents SDK 60% to 40% when both used GPT-6.1 Sol to create memories and GPT-6 Sol to carry out the tasks. With Claude Opus 5.5, Dream scored 73.3%, Prime Agent reached 60%, and Claude Managed Agents scored 33.3%.

Those differences make it difficult to treat model choice as the only factor behind agent performance. The systems differ in how they review prior work and turn it into instructions or other memories for future use. Packer has pointed to this “dreaming” process as a key source of variation: the agent examines what happened, then distills it into guidance that can influence later behavior.

The results also show variation between harnesses beyond the top-ranked system. Prime Agent’s 60% score was more than twice Hermes Agent’s 26.7%. For teams evaluating agent platforms, the benchmark offers an early reason to test the complete system on repeated tasks, rather than infer its performance from the model name alone.

Memory generation itself also varied across seven tested models. Anthropic’s Claude Opus 5.5 and Z.ai’s GLM-5.3 tied at 73.3%. Kimi K3 scored 66.7%; OpenAI’s GPT-6.1 Sol and Google’s Gemini 4 Argon each reached 60%. Muse Wonder scored 46.7%, while Grok 4.7 finished at 33.3%. Packer said Opus 5.5 and GLM-5.3 were significantly better than GPT-6.1 Sol at generating useful memories.

Memory quality comes with a cost tradeoff

Higher scores came at different price points. Letta estimated GPT-6.1 Sol at $0.77 per task, compared with $2.80 for Opus 5.5 and $3.58 for GLM-5.3. The lower-cost model scored below the two leaders on memory generation, but its per-task price was substantially lower.

To examine how memory quality affects task execution, Letta used its Dream harness to generate memories with each model and then tested them on tasks run in Letta Code. The setup separates the model that prepares the memory from the model that performs the later work. In the reported example, memories created by Opus 5.5 helped GPT-6 Sol complete tasks, suggesting that teams could use one model to prepare reusable guidance and another to handle execution.

The benchmark’s examples show why a memory’s form can matter as much as its contents. For a task involving company pricing data, Opus 5.5 generated a skill that told the agent when to consult relevant information. GPT-6.1 Sol generated four separate skills, but the agent didn’t locate the one it needed. More saved material didn’t produce a better result in that case; the agent needed a useful route to the right material.

Gemini 4 Argon’s memory captured specific file sizes. That detail didn’t transfer well when a later task involved different files. Opus 5.5 instead recorded a procedure: check the usual file and email for the information. Because the instruction described where to look, it remained useful when the task details changed. This contrast highlights a practical design question for agent memory: should it preserve facts from an episode, or capture a method that can be reused across episodes?

Early results leave room for a broader test

FrontierMemory is still a small benchmark. With 15 tasks and one attempt per task, a single result can move a score by roughly 6.7 percentage points. That makes the current rankings useful as an initial comparison, but not a fine-grained measure of system quality. Letta’s current evaluation tests learning from earlier work; it doesn’t yet cover agents updating memory continuously while they work.

Packer said the next release is planned to add longer-running tasks and online memory updates. Those changes would test how agents respond as experience accumulates over a sequence, rather than relying only on a memory created before the later task begins. They could also make it easier to study how a system handles changing information and failed approaches over time.

Developers outside Letta have already shown interest in extending the comparison. After Wooders posted about the benchmark, the OpenClaw account said it would like a plugin for its platform and suggested testing memory in a multiplayer environment, where several agents share a harness. Wooders replied: “Yes definitely – would love to add OpenClaw to the harness comparison! Extending to multiplayer memory is also a very interesting future direction.”

For now, the clearest takeaway is that an agent’s stored history doesn’t speak for itself. A useful system needs to turn past work into guidance that applies to new situations, and the harness plays a visible role in that process. Larger task sets and tests of shared, continuously updated memory will show whether the early gaps persist as the benchmark expands.

Source: Amanda Caswell · thenewstack.io

Comments

Sign in to comment.
No comments yet. Be the first to comment.

Related Articles

Five AI agent harnesses took the same memory test. The re...