AI 開發工具與評測
Claude Code 2.1.269 Adds Plugin A/B Evaluation, but Automated Scores Still Lack Quality Benchmarks
The new `claude plugin eval` command compares performance with a plugin enabled and disabled, producing JSON and HTML evaluation reports. Initial testing suggests that automatically generated criteria are better at verifying instruction following than proving that a plugin genuinely improves task quality.
Anthropic has added `claude plugin eval` to Claude Code 2.1.269, bringing plugin quality checks into the official CLI. Developers can prepare test cases, run them both with and without a plugin loaded, and obtain per-case scores, aggregate JSON, and an HTML report. This ablation design is more useful than showcasing a single successful output: when the underlying model can already complete the task, what matters is measuring the plugin’s incremental contribution—not whether the final answer merely looks reasonable.
The same release also allows `/output-style` to be switched in Remote Control, cloud, and other headless sessions, and can attach file diffs to tool results when the Bash tool handles edits. For teams maintaining internal enterprise Skills, hooks, or MCP integrations, machine-readable evaluation output provides a foundation for CI integration, regression comparisons before version upgrades, and evidence retention. However, the official release notes promise only “scored, reproducible results,” without publicly explaining evaluator calibration, repeated sampling, how model versions are pinned, or statistical uncertainty.
Public testing by evaluation researcher Hamel Husain also exposed the tool’s limitations. In a trial involving a writing-style plugin, the system generated three graders, JSONL data, and a report, but some criteria were drawn directly from the plugin’s own rules. As a result, the treatment group naturally adhered to those rules more closely than the baseline without the plugin. The results demonstrated that the plugin’s instructions were followed—not that the writing was clearer or users were more satisfied. Testers also noted that the scores and criteria lacked provenance and context, while the tool still required authors to supply real failure cases themselves—the labor-intensive error analysis at the heart of the evaluation process.
Engineering teams should therefore treat it as a regression-test runner for now, rather than a general-purpose judge of quality. A more reliable approach is to pin the CLI, the model under test, and the judge model; preserve raw traces; build a human-annotated dataset from real failure samples; and set cost limits. Tools with side effects should also use synthetic fixtures, mocked external services, and least-privilege access. It remains worth watching whether Anthropic publishes the suite schema, scoring provenance, and controls for repeated trials—and whether individual Skills can eventually be evaluated reliably through the same workflow.