苹果杜克联合研发LensVLM 模型读压缩内容可按需查原文

AI agents working on long-context tasks often grapple with expanding context windows. As they process code, tool outputs, and interaction logs over extended periods, the volume of content grows rapidly. Compressing this material—whether through summaries or other methods—reduces input size but risks erasing critical details that might prove essential later. A seemingly trivial snippet today could become key evidence in the next step, and many one-way compression approaches leave models unable to retrieve the original content when needed.

Previous attempts to address this include using images to represent text, which cuts down on token counts, but this introduces clarity challenges: characters become too small for models to read, especially with high compression rates. At 5x compression, for example, a base model’s accuracy drops to just 31.3% because it can’t distinguish fine details in the compressed images.

A new approach, developed by a research team, solves this by combining text-to-image compression with selective access to original content. The method, called LensVLM, lets models first browse a full set of compressed pages, then choose specific pages to expand for deeper analysis. In seven text question-answering benchmarks, it achieved an average accuracy of 68.9% with 4.3 times effective compression—close to the 72.4% accuracy of using full uncompressed text.

This work has been accepted at a major AI conference, and a demo version of the model is publicly available for testing. The core innovation is separating the tasks of locating evidence and reading it: the model skims compressed pages to find relevant sections, then retrieves the original content for those sections to answer questions accurately.

The text-to-image compression works by converting text into images, which are then processed by vision-language models to produce visual tokens. This avoids the need to redesign text vocabularies, as compression is adjusted via font size, page dimensions, and text density. However, the trade-off is clear: higher compression means smaller characters, which are harder for models to parse.

To teach the model which pages to select, researchers used a base model, filtered questions it couldn’t answer, and trained it using demonstrations of correct page selection and expansion. They further optimized the model with reinforcement learning, boosting accuracy from 31.3% to 68.9% at 5x compression. At higher compression rates, like 15x, the base model’s accuracy falls to 18.1%, but LensVLM still reaches 52.1%—a significant improvement.

The effective compression rate accounts for both the compressed pages and the expanded content needed to answer questions. For 15x initial compression, this translates to 10.1x effective compression. In terms of speed and cost, LensVLM reduces end-to-end latency by about 30% compared to full text input, uses only 1/14th of the KV cache, and has 3.7x higher throughput, cutting GPU costs by around 73%. It’s most beneficial for long documents or scenarios where multiple queries reuse the same content.

For documents like PDFs that include tables, graphs, and layout details, the approach can be adjusted. Expanding to high-definition pages works better than extracting only text, with 41.9% accuracy at 15x compression, though this requires more input overhead. The best choice depends on whether the content relies on visual elements or pure text.

The core idea behind LensVLM is letting models access original content when needed, rather than forcing one-way compression. This ability to allocate a “reading budget”—deciding what to skim, what to expand, and when to revisit original content—could become as important as expanding context windows for long AI tasks.