多模態模型
ParVL Scales Multimodal Models in Parallel With a Shared Backbone, Allowing Separate Allocation of Vision and Language Compute
Rather than adding multiple full parameter sets, ParVL reuses the same ViT and LLM weights and distinguishes compute paths with branch-specific KV prefixes. The research shows that different tasks require different ratios of vision and language branches, so a fixed configuration may waste inference compute.
Multimodal models typically scale in one of two ways: by increasing parameter counts or by having the model generate longer sequences during inference. The former raises memory requirements, while the latter increases latency. ParVL proposes a third approach: multiple vision and language branches reuse the same InternVL3.5/Qwen3 backbone weights, with each branch receiving only a learnable KV prefix, and a lightweight aggregator combining their outputs. This increases the amount of compute actually performed without duplicating the main weights for every branch.
The team conducted full-parameter supervised fine-tuning on approximately 13 billion tokens and tested nine vision-language branch configurations on the 1B model. Public results show that the single-branch baseline achieved an equally weighted average of 49.6 across nine image benchmarks, while a configuration with four vision and four language branches reached 50.5. The larger 2B and 8B models, both using a 2:2 configuration, averaged 54.7 and 63.0, respectively. The gains are not dramatic, but the experiments reveal an important finding: there is no universally optimal allocation. OCR, mathematics, and general visual understanding tasks favor different ratios of vision to language compute.
The project has released the model implementation, DeepSpeed configurations, 11 checkpoints, training histories, and CPU tests. Evaluations cover datasets including MMMU, MathVista, ChartQA, and DocVQA. However, the training data itself has not been redistributed, some answer extraction relies on GPT-4o, and results can also be affected by hardware count, FlashAttention version, and batch composition. Engineering teams should next examine whether branch parallelism can translate into a better quality-latency curve in real-world serving systems, and whether the number of vision and language branches can be adjusted dynamically by request type instead of paying the same fixed compute cost for every workload.