AI 代理／訓練資料
CalibForge Calibrates Terminal Tasks Through Solver Disagreement, Delivering Gains of Up to 30.04 Percentage Points Across Benchmarks
CalibForge no longer treats “executable” as a sufficient condition for a valid training task. Instead, it requires candidate tasks to fall within a learnable range for the target model’s capabilities. The team released 5,431 tasks, two models, and evaluation recipes, although some cross-benchmark results are still based on a single run.
Synthetic data for terminal agents is often generated by having a model create tasks and then using test programs to confirm that they are executable. The problem is that passing validation does not mean a task has training value: tasks every model can complete are too easy, while tasks every model fails may be unsolvable or beyond current capabilities. Submitted on August 6, CalibForge turns task generation into an adversarial loop with solver feedback. Agents interact with candidate environments, and the system modifies instructions, files, containers, and validators based on complete trajectories until the difficulty falls within the target range.
The system provides two calibration methods. Multi-solver mode retains tasks for which at least one solver succeeds and at least one fails. Contrastive mode requires a designated stronger model to pass and a weaker model to fail. The public dataset contains 1,263 multi-solver tasks and 4,168 contrastive tasks across 16 domains. Each is packaged in Harbor format with a Docker environment, natural-language instructions, and executable tests. Compared with using a single model to grade tasks, this provides an observable capability boundary.
The authors trained two Mixture-of-Experts (MoE) models on successful trajectories. For the Qwen3-30B-A3B baseline, performance rose from 7.87% to 32.58% on Terminal-Bench 2.0, from 3.26% to 30.94% on SWE-bench Pro, and from 5.94% to 35.98% on Doc2Repo. Gains were substantially smaller for the stronger Qwen3.5-35B-A3B, indicating that the data’s effective range changes with the model. Engineering teams should examine whether the solvers used for calibration introduce selection bias and whether repeatedly modified tasks encourage models to learn validator shortcuts. Terminal-Bench and Doc2Repo results are reported over three runs, but SWE-bench Pro was evaluated only once. The current scores also come primarily from the authors’ public pipeline and still await independent reproduction.