AI 評測與開發工具
DashArena Requires Models to Provide Replayable Interaction Traces, Exposing the Gap Between Dashboards That “Display” and Those That Actually Work
DashArena evaluates model-generated interactive data dashboards across 234 open-ended tasks and uses a browser to replay the interaction workflows written by the models themselves. Adding interaction evidence raised agreement between an 8B vision judge and human preferences from 71.7% to 79.8%, yet even the best model achieved a full replay rate of only 73.3%.
Generative models can now create dashboards from data tables and natural-language requests, but a page loading successfully does not mean its filters, linked charts, or analytical workflows actually work. [DashArena](https://arxiv.org/abs/2608.10567) expands the unit of evaluation from a static screenshot to “a single HTML dashboard plus an executable interaction trace.” After completing the page, the model must also provide a JSON list of clicks, text inputs, chart-mark selections, and expected visual changes. A Playwright runner replays every step in Chromium, checking controls, text, and ECharts runtime data, then passes screenshots and execution receipts to a vision-language model for pairwise comparison.
The benchmark contains 234 tasks across 14 topic clusters. Its main leaderboard uses 120 tasks that were not included in the judge model’s training data. The researchers generated preference labels with Claude Opus 4.6, then used 510 examples with candidate order swapped to fine-tune Qwen3-VL-8B-Instruct with LoRA, producing DashJudge-8B. On 99 comparisons with a human majority opinion, it achieved 79.8% agreement, versus 70.7% for the base model. Removing the interaction traces and execution reports reduced agreement to 71.7%. Rules based only on whether a dashboard rendered, replayed successfully, or showed component changes achieved just 42.4% agreement, demonstrating that semantic quality cannot be determined by browser tests alone.
The reliability gap is even more notable from an engineering perspective. Although GPT-5.5 topped the preference leaderboard, its render rate was still only 85.8%, and its full replay rate was 73.3%. None of the evaluated models exceeded 86% and 74%, respectively. This suggests that evaluations of generative frontend systems should preserve replayable tests, execution evidence, and semantic judgments together, rather than scoring screenshots alone. [Apache ECharts](https://echarts.apache.org/en/) provides a range of interactive Canvas/SVG charts and is also the charting interface that the benchmark restricts candidates to using. Future scrutiny should focus on whether the authors fully release the tasks, runner, and weights, as well as whether self-authored traces omit failure paths that models would prefer not to reveal. At present, the average control coverage of a single trace is only 80.8%.