多模態評測
MeViS-Audio Champion Replaces Single-Model Decisions with Candidate Mask Consensus
The winning report for MeViSv2-Audio at the eighth LSVOS breaks speech transcription, visual grounding, segmentation, and target-presence detection into a multistage pipeline, then selects a trajectory based on agreement among candidate masks. The system achieved a final score of 0.769589, although the official public leaderboard still says results are to be announced, and neither the code nor complete ablation data has been released.
MeViSv2-Audio requires a model to take an audio description of an object's motion and output the target mask for every frame of an entire video. If the described object is absent, the system must instead return empty masks. Compared with conventional text-prompted segmentation, this introduces four coupled challenges: speech recognition errors, motion semantics, cross-frame tracking, and determining that no target is present. Rather than training a single massive end-to-end model, the new champion system first uses Qwen3-ASR to convert speech into text, then employs multiple complementary grounding and segmentation models to generate candidate mask trajectories.
The key design is agreement-based selection: instead of directly trusting the highest-scoring candidate, the system calculates each trajectory's average mask agreement with the other candidates and selects the one closest to the group consensus. In effect, it estimates reliability from the intersection of different models' predictions when no ground-truth annotations are available. For queries involving concepts such as "on the left," "two," or multiple objects—which ordinary trackers can easily overlook—the pipeline also applies rules for direction, quantity, and singular-versus-plural distinctions. Finally, a video-level classifier combines vision-only scores, audiovisual scores, and scores from different queries for the same video to decide whether to output masks or an empty result.
The report lists a `J&F` score of 0.5952, a no-target accuracy of 0.7931, a target-present accuracy of 0.9205, and an overall score of 0.769589. The authors say the organizers informed them that they ranked first. The technical takeaway is that multimodal agents do not necessarily require all components to be jointly trained: candidate consensus, rule-based corrections, and an explicit rejection gate can still provide an advantage on competition data. However, the leaderboard on the official website is still marked TBD, and the paper does not provide code, inference costs, complete ablations on the number of candidate models, or cross-dataset validation. The next things to watch are whether the result can be reproduced once the leaderboard is officially finalized and the code is released, and whether the consensus mechanism remains reliable when candidate models share the same biases.