生成模型評測
Open-EA Shrinks Its Visual Generation Evaluation Agent to 3B Parameters, but Currently Offers Full Support Only for the VBench Video Workflow
Open-EA distilled multi-turn visual model evaluation trajectories into 10,042 supervised examples, moving a planning workflow that previously depended on GPT-4o to the local EA-3B model. The model, data construction pipeline, and runtime are now public, but the claim that evaluation time falls to 10% of traditional methods primarily comes from the original API-based agent and should not be interpreted as a general efficiency guarantee for EA-3B.
The newly released Open Evaluation Agent (Open-EA) aims to transform the evaluation of image and video generation models from fixed prompt sets and a single aggregate score into a multi-turn agent whose objectives can be specified in natural language. The system first breaks questions such as “How consistent is the subject?” into subdimensions, then selects targeted prompts, invokes generation models and tools such as VBench, determines what to test in the next round based on intermediate results, and ultimately produces a summary supported by observed evidence.
The main update over the ACL 2025 version of Evaluation Agent is the removal of GPT-4o as a required dependency during planning. The team used existing multi-turn rollouts to build EA-CoT-10K, whose core dataset contains 10,042 records with historical context: 7,494 tool-decision examples and 2,548 summaries spanning 15 VBench video evaluation tools. The researchers then performed full-parameter supervised fine-tuning of Qwen2.5-3B-Instruct on this dataset to produce the local EA-3B planner. The dataset also includes 986 image examples, but these were not used to train the current EA-3B model.
From an engineering perspective, the released runtime uses two OpenAI-compatible local endpoints: EA-3B handles planning, while a separate Qwen2.5-3B-Instruct instance selects prompts from a tool prompt table. The generator and VBench introduce additional workloads. Official examples typically assign one GPU to each endpoint and use a third GPU to run the video model under evaluation. By default, the endpoints bind only to `127.0.0.1`. Because they provide no authentication mechanism, anyone changing them to listen on an external interface must add their own network isolation and authentication.
The paper claims that the original Evaluation Agent can achieve comparable results using only about 10% of the evaluation time required by traditional workflows. However, Open-EA’s new experiments primarily examine policy transfer across four in-distribution and three out-of-distribution video generators, and demonstrate only “partial” cross-family generalization. The fully supported scope of the local version remains limited to VBench text-to-video evaluation; open-ended image evaluation and T2I-CompBench still use the GPT-4o path. The next steps should examine whether EA-3B’s active sampling causes it to miss low-frequency failures and whether different planners can maintain consistent rankings under a fixed generation budget.