AI 評測與開發工具
LiveEvalBench Has Agents Actually Build and Operate Websites, Replacing Static Evaluations Based Only on Frontend Screenshot Comparisons
LiveEvalBench uses three types of agents—build, code, and browser interaction—to gather evidence, then generates checks based on each project’s actual functionality. The framework and dataset are public, but scoring still relies on LLM judges, and its cost, reproducibility, and judge bias require further quantification.
Generative frontend models can now output complete projects, but common evaluations still focus heavily on screenshot similarity, static code, or a small number of fixed tests. These methods can easily miss cases where the interface looks correct but buttons do not work, dependencies cannot be installed, routing fails, or state updates are incorrect. Conversely, implementations that differ from the reference screenshot but function correctly may also be misjudged.
LiveEvalBench, released on August 4, turns evaluation into an agent-based review workflow. A Build Engineer first builds and launches the project, a Code Engineer examines its implementation and structure, and a UI Tester operates the page in Chromium through Playwright. Rather than merely producing subjective scores, each role gathers evidence from build logs, source code, and actual interactions, which is then consolidated through a shared scoring process. The framework retains a common rubric across models while also creating implementation-oriented checks based on the functionality each project actually claims to provide, seeking to balance comparability with the diversity of frontend approaches.
The public repository provides a Python runner, configurations for four core agents, scoring templates, checkpoint-based resumption, JSONL sharding, and a results dashboard. It can connect to Anthropic, OpenAI, Google, or OpenAI-compatible endpoints. By default, each agent can execute up to 80 ReAct steps, with a per-task timeout of 5,400 seconds, and multiprocess sharding is also supported. The benchmark dataset is hosted separately on Hugging Face, making it possible to update tasks without rewriting the entire evaluation pipeline.
Its technical value lies in incorporating whether a website is genuinely usable into model comparisons, while also offering a starting point for enterprises validating AI-generated frontends. However, the paper only states that its results are highly consistent with judgments from human experts; the abstract does not provide complete agreement metrics sufficient for independent assessment. Because the adaptive checks are also generated by models, different judge models may produce different rankings. Future evaluation should examine version-pinned containers, network isolation, token usage and execution cost reporting, and ranking stability across different judge models.