多模態推論
ReMo Reallocates Visual Information Across Audio and Text, Cutting Omni-LLM Input Tokens by 54%
A KAIST team has proposed ReMo, a training-free method that removes visual tokens at the input stage when their content can be explained by audio, while preserving object semantics through text proxies. In the authors’ own tests, it compressed inputs to the Qwen Omni series by more than half without reducing average scores, but neither the code nor independent reproduction results have been released.
Researchers from KAIST and the University of Oxford have proposed ReMo to address sequence inflation when Omni-LLMs process video, audio, and text simultaneously. Inputs to such models can reach tens of thousands of tokens, with visual streams accounting for the majority. As sequences grow longer, the compute and memory costs of attention quickly become an inference bottleneck. ReMo does not modify model weights; instead, it reallocates cross-modal information before it enters the language model.
The method has two pathways. First, the system projects audio and visual embeddings into a shared space. If the high-level semantics of a video segment are already conveyed by sound—such as a violin being played—it removes the corresponding visual tokens. Second, for objects not covered by the audio, ReMo uses the 3.4M-parameter YOLOv8n-seg detector to rewrite object names and locations as shorter text proxies. Local details and visual information that cannot be replaced are retained. The shared space is constructed offline using 5,000 audio–image embedding pairs from VGGSound. Therefore, although the method is described as “training-free,” it still requires additional data calibration, a detector, and preprocessing.
The authors report an average 54% reduction in input tokens on Qwen2.5-Omni 3B and 7B, with normalized average scores across five benchmarks reaching 101.2% and 101.3% of the full-input baselines, respectively. On a single RTX A6000, inference time for the 3B model fell from 7.35 seconds to 4.78 seconds, while the 7B model dropped from 10.30 seconds to 5.92 seconds. Tests on the 30B Qwen3-Omni showed a similar trend. For video-based customer service, meeting understanding, and long-form audiovisual search, the results suggest that redundant cross-modal information can be eliminated outside the model. However, the slight score gains may result from noise removal and should not be interpreted as a general improvement. The current experiments are limited to Qwen Omni models, fixed benchmarks, and a single hardware configuration, and no code has yet been released. Benefits across models, streaming inputs, and real-world latency therefore remain to be verified.