多模態推論
ReToken Filters the Visual KV Cache With a Single Learnable Token, Improving Long-Video QA by 8.0 Points
Rather than feeding all image tokens into the model at once, ReToken retrieves sparse, question-relevant blocks from a prefilled visual KV cache. Its frozen-backbone approach trains only the retrieval token and fits on a single H100, although the results still come mainly from two 8B models.
ReToken proposes a very small change to address a core bottleneck in long-video and large-image-set inputs: it adds a single learnable embedding to a vision-language model and trains it as an explicit retrieval target. A video or image collection is first encoded into a reusable visual KV cache. After receiving a question, ReToken selects a small number of relevant token blocks from the cache and passes them to the frozen language model for decoding, avoiding repeatedly sending the full visual context through attention computation.
The authors note that standard attention weights may not provide a reliable signal for frame retrieval because relevant information may reside in the value space. ReToken therefore uses the learned token to read this signal and selects blocks through cross-layer voting. This design also means that a video needs to be encoded only once when answering multiple questions; each new question primarily incurs retrieval and decoding costs.
On Visual Haystacks, ReToken improves Qwen3-VL-8B by 13.4 points and InternVL3.5-8B by 12.4 points, with relative gains exceeding 20% for both. After training only on a small image-QA dataset, the method also transfers zero-shot to long videos, improving Qwen3-VL-8B by 8.0 points on LVBench. The backbone remains frozen, and both training and long-video inference fit on a single H100.
The code, MIT-licensed implementation, and two Hugging Face checkpoints are publicly available, with pinned versions of PyTorch, CUDA, FlashAttention, and Transformers. This supports reproducibility while also revealing that the current dependency stack is fairly fragile. Engineering teams should next examine end-to-end time to first answer, KV cache capacity, accuracy at different retrieval counts, and whether the method transfers to larger VLMs or different architectures. The current results do not directly demonstrate equal effectiveness for open-ended video understanding or real-time streaming.