多模態模型與推論
Mage-VL Selects Visual Tokens Directly from Video Encoding Data, Accelerating Streaming Multimodal Inference by Up to 3.5×
Microsoft’s Mage-VL replaces uniform video frame sampling with I/P frames, motion vectors, and residual energy, encoding only regions with significant changes. Its 4B model reportedly reduces visual tokens by more than 75% while preserving performance on static tasks, although speed and quality comparisons still come primarily from official tests.
Most video-language models first use a decoder to reconstruct complete frames, then sample them at a fixed frequency and divide them into visual patches. This pipeline discards temporal-change information already computed by the video codec and repeatedly encodes nearly static backgrounds. Microsoft’s publicly released Mage-VL instead processes streaming video in a “codec-native” manner: Mage-ViT reads sparse anchor I-frames and predicted P-frames, then uses motion vectors and residual energy to determine which 16×16 regions contain more new information before selectively converting them into visual tokens.
The paper claims that this approach can reduce visual tokens by more than 75% while preserving spatial and temporal context. Mage-ViT begins training on approximately 560 million unlabeled images and 100 million unlabeled video frames. The complete Mage-VL uses a dual-system architecture: a lightweight System 1 event gate continuously monitors the stream and activates the causal System 2 decoder only when it detects changes worth reasoning about. Compared with sending every frame to a large language model, this design more closely resembles an event-driven real-time system.
Official results show that Mage-VL-4B approaches Qwen3-VL-4B on static benchmarks, achieves higher scores in video understanding and 2D and 3D spatial reasoning, delivers up to a 3.5× improvement in end-to-end wall-clock inference speed, and outperforms the 15B Phi-4-reasoning-vision baseline. The team has released the code, model weights containing approximately 5B parameters, and an online demo. The GitHub project quickly accumulated more than 1,000 stars after release, creating the initial conditions for external evaluation of the architecture.
Engineers should next verify whether the reported speedup includes the full cost of video decoding, the tokenizer, and the primary model, and should retest it across different codecs, bitrates, camera motion patterns, and multi-stream workloads. A selector that relies on motion and residuals may also miss low-motion but semantically important details, such as slowly changing instrument readings or small distant objects. Therefore, a 75% reduction in tokens does not necessarily mean that the same compression ratio can be safely adopted across all surveillance, robotics, or real-time assistant scenarios.