世界模型與評測
PlayWorld Has Multimodal Agents Control Nine World Models, but the Highest Long-Horizon Consistency Score Is Still Just 2.12/5
PlayWorld uses an agent that observes generated frames and dynamically adjusts control commands, preventing differences in movement magnitude across world models from making fixed trajectories incomparable. Results from 171 scenarios show that although current models can generate fluid visuals, they still struggle to preserve global spatial structure and states that continue evolving outside the field of view.
Interactive world models are often compared using identical keypress sequences. But the same command—such as “move right three times”—might make one model complete a full rotation while another turns only slightly, meaning the evaluation measures control granularity rather than world consistency. PlayWorld instead uses high-level goals as shared conditions, such as circling a statue, walking into water, or turning away and then looking back at an ongoing event. A multimodal Agent Player observes the generated frames and adjusts its actions accordingly.
Each test case still includes a manually authored base action sequence to limit path differences between models. The agent can make only five decisions—Keep, Stop, Extend, Correct, or End—and may execute no more than 40 steps. In a 25-task comparison on Genie 3 and HappyOyster, this “preset path plus minor corrections” approach changed only about 12%–15% of actions, yet achieved human preference rates of 65.6% and 67.4%, respectively, outperforming both fully fixed trajectories and complete agent-driven planning. The agent can be powered by Claude Haiku, Claude Sonnet, or Gemini 3.1 Pro, though the paper ultimately uses the lower-latency Claude Haiku.
The [PlayWorld dataset](https://huggingface.co/datasets/jocelynd/playworld-bench) contains 171 manually curated scenarios, 50 action patterns, and more than 820 questions. It evaluates over 1,400 interactive videos, each approximately 10–60 seconds long, generated by nine open or closed world models. Scoring covers geometric consistency, interaction realism, off-screen evolution, and the evolution of persistently visible states. If a trajectory never reaches the location required for a test, a gate first assigns that dimension the lowest score, preventing visually appealing videos that fail to complete the task from benefiting.
None of the nine models scores above 2.12/5 overall. Genie 3 ranks highest at 2.12, followed by HappyOyster at 1.92; the best open model, LingBot-World2, scores 1.82. Models are generally better at preserving local visuals and immediate collisions, but they may duplicate buildings when circling a landmark, restore off-screen objects to their original state when they reappear, or cause events such as cooking and painting to stall or reset. This shows that conventional measures of visual quality and frame-to-frame smoothness are insufficient proxies for a usable world state.
The [code and interface](https://github.com/kxding/PlayWorld) are publicly available, but scoring still relies on Gemini 3.1 Pro to answer sample-specific VQA rubrics, while the control agent also uses a closed model. The set of 171 scenarios remains small for rapidly evolving world models, and the underlying inference conditions are not fully consistent: five systems are operated through web interfaces, while four generate videos locally in segments. Future work should examine cross-model bias in the evaluator and whether new models reproduce the same rankings on controlled state-memory tests.