代理工具
CalibForge Calibrates 5,431 Terminal Tasks Using Strong and Weak Agents; 35B Model Reaches 47.57% on Terminal-Bench 2.0
Rather than retaining every executable synthetic task, CalibForge repeatedly revises tasks until different solver agents exhibit predefined success and failure patterns. Two models fine-tuned on calibrated trajectories improved across three coding-agent benchmarks, but full training required 64 NVIDIA H20 GPUs, and some results were based on only a single run.
Synthetic training data for terminal agents is typically filtered only by whether the Docker environment can be built, the tests can run, and at least one agent can solve the task. This leaves both overly simple tasks and tasks that are effectively unlearnable. CalibForge turns task generation into an adversarial loop with a verifier: an author agent generates the instructions, initial files, Dockerfile, and tests; after solver agents attempt the task, the system revises the task and verification criteria based on their complete trajectories, with up to 50 calibration rounds per task.
The system uses two retention rules. In multi-solver mode, DeepSeek-V4-Flash, GLM-5, and Kimi K2.5 must each record at least one success and one failure. In contrastive mode, the designated DeepSeek-V4-Pro must succeed while V4-Flash fails. The former produced 1,263 tasks and the latter 4,168, spanning 16 domains including software engineering, system administration, security, scientific computing, and data processing. This “solver-relative learnability range” is more direct than manually labeling tasks as easy, medium, or hard, although difficulty also changes with the model and agent scaffold.
The researchers distilled successful trajectories from DeepSeek-V4-Pro, then performed ten epochs of full-parameter supervised fine-tuning (SFT) on Qwen3-30B-A3B-Instruct and Qwen3.5-35B-A3B. On Terminal-Bench 2.0, the former improved from 7.87% to 32.58%, while the latter rose from 39.10% to 47.57%; these results exceeded the strongest data baselines by 6.36 and 6.75 percentage points, respectively. The 30B model also improved from 3.26% to 30.94% on SWE-bench Pro and from 5.94% to 35.98% on Doc2Repo. The dataset and both model checkpoints have been released publicly, but the repository currently provides mainly the data format and evaluation entry points, not the complete task-synthesis pipeline.
The results still warrant cautious interpretation: SWE-bench Pro was evaluated only once, and the stronger 35B base model gained just 3.03 to 3.85 percentage points across the other benchmarks. Although the authors screened for contamination using text overlap, test code, and task structure, semantically similar synthetic tasks may still have slipped through. Full-parameter training used 64 NVIDIA H20 GPUs, and reproducing the complete pipeline would cost far more than downloading the data and performing conventional fine-tuning.