AISI to add inference-compute evaluations to Evaluation Cards, publishing settings for comparisons across reports

The UK AI Security Institute (AISI) and EvalEval announced on September 22 that they are using Evaluation Cards to publicly organize evaluation results, settings, and research context. The release covers five benchmarks used in the main experiments—HealthBench, FrontierMath, Humanity’s Last Exam, SWE-Bench Pro, and Terminal-Bench 2.0—and six models, alongside two cybersecurity evaluations that use a partially different set of models. The new development is the inclusion of this data in a shared reporting framework; the related paper was already published in June. [Collaboration announcement](https://huggingface.co/blog/evaleval-aisi), [paper publication history](https://arxiv.org/abs/2606.17930).

The underlying Every Eval Ever (EEE) system uses structured records to connect models, evaluation frameworks, generation parameters, and scores, and provides converters for Inspect AI, HELM, and lm-eval-harness. Aggregate results are stored in JSON, while item-level data is stored separately in JSONL and linked through shared identifiers. Records can distinguish single-turn, multi-turn, and tool-using agent interactions. When complete item-level data is available, engineers can trace inputs, responses, and tool calls instead of relying solely on a single leaderboard score. [EEE repository and format documentation](https://github.com/evaleval/every_eval_ever).

This approach directly addresses the central issue examined in the research: inference budgets and retry rules can change scores. In the paper’s main experiments, six models were run on five benchmarks under two conditions—with and without correctness feedback—with five independent trajectories for each task and condition. The experiments increased token budgets, compacted context, and allowed repeated submissions to observe how capabilities changed as more compute was applied. [Experimental methods](https://arxiv.org/html/2606.17930v3).

The cap for each trajectory ranged from 5 million to 30 million tokens, depending on the benchmark. The count included the target model’s input, output, and reasoning tokens, but excluded tokens used by the grading model. With correctness feedback, agents could learn whether a submission had succeeded—a condition that differs from real-world services, where ground-truth answers may be unavailable. Readers therefore need to consider feedback conditions and total budgets alongside scores, rather than treating cumulative success rates as single-response accuracy. [Budget and feedback definitions](https://arxiv.org/html/2606.17930v3).

For teams building internal evaluation platforms, EEE’s value lies in its shared data interface: after converting existing records, teams can filter and compare results by model, tool configuration, and budget. This is an engineering interpretation of the public architecture. Passing format validation means only that the fields conform to the specification; it does not mean the experiment has been independently rerun, nor does it guarantee that scoring semantics are identical across frameworks. [EEE architecture](https://github.com/evaleval/every_eval_ever).

The announcement points to public reporting of existing research data, rather than a capability ranking of the latest models in September. Follow-up checks should establish whether each result includes complete item-level records, model versions, and settings, and whether the datasets, sandboxes, and graders are available. Comparisons of scores across institutions have a sound basis only when these conditions can be aligned.