Agent记忆系统准确率低于RAG？研究揭秘负优化根源
Agent memory systems typically require processing hundreds of millions of tokens through operations like fact extraction, user profiling, and knowledge graph construction, yet in many cases, their answer accuracy is lower than that of direct retrieval-augmented generation (RAG) using chat history. This counterintuitive result emerges from a joint study by researchers from UIUC, Tsinghua University, and Renmin University of China, documented in the paper titled "Memory Tax: Dissecting the Net Value of Agent Memory".
To evaluate performance, the team tested four categories of memory systems—flat storage, layered lifecycle management, graph-structured storage, and training-based strategies—against traditional RAG on three benchmarks: LongMemEval-s, PersonaMem-v2, and MINT-Eval. When using the GPT-5.6 Luna model, only two of the 11 open-source memory systems (A-Mem and SimpleMem) outperformed single-round RAG, each leading by just 2 percentage points. In contrast, two-round RAG achieved 90% accuracy, surpassing all tested memory systems. Even when switching to the Qwen3.6-27B model, two-round RAG maintained its lead with 81.4% accuracy over the 11 systems. Cost analysis further highlighted that two-round RAG is more token-efficient while delivering higher accuracy, as memory systems often consumed more tokens without corresponding accuracy gains.
The superior performance of two-round RAG stems from its sophisticated retrieval pipeline, which avoids the brute-force approach of dumping entire conversations into vector databases. Instead, the system follows a structured process: first, splitting conversations by session boundaries into ~1024-character segments (no cross-session mixing, with timestamps attached), then performing parallel retrieval using both dense vectors (for semantic matching) and BM25 (for keyword matching), merging and deduplicating results, re-ranking with a specialized model (Qwen3-Reranker-0.6B), and initiating a second retrieval if the initial evidence is insufficient. This pipeline significantly boosted accuracy: in tests with Qwen3.6-27B, combining BM25 and dense vector retrieval followed by re-ranking lifted accuracy from 72.6% to 77.6%.
The underperformance of Agent Memory systems arises from their core design: processing historical data in advance without knowledge of future user questions, leading to two key issues. First, critical details are often lost during processing. For example, OpenViking retains only 39.4% of original information, such as omitting the exact $60 taxi fare to a hotel when summarizing a user’s preference for cost-effective transport, making it impossible to answer follow-up questions about potential savings from taking a train. SimpleMem mixed up the delivery date of a backpack (January 20) with the date the user started using it (January 24). MAGMA distorted a user’s request to forget about collecting antique maps, still including that detail in later recommendations. Second, even when information is retained, poor granularity can hinder retrieval: splitting details into too-small chunks (like separating $25 bike chain repair and $40 light installation) delays total cost queries, while grouping details into overly large segments (like burying a 45-minute commute time in a long user profile) makes them hard to locate.
To address these limitations, the paper proposes a key adjustment: combining memory retrieval with retrieval of the original conversation history, merging the two sets of candidates and re-ranking them before generating answers. This simple change improved accuracy across all tested systems. For instance, SimpleMem’s accuracy jumped from 73.6% to 82.8%, reducing the number of incorrect answers that occurred when relying solely on memory. This approach preserves the efficiency benefits of pre-processed memory while leveraging raw original data to fill gaps—such as retrieving exact amounts or dates that memory processing might have overlooked.
The full details of the study are available in the paper at https://arxiv.org/pdf/2610.34073.
