代理訓練與評測
Prime Intellect Unifies 365,000 Agent Tasks, Giving Evaluation and Reinforcement Learning a Shared Execution Interface
Prime Intellect has consolidated software engineering, terminal operation, and web search datasets into 23 tasksets accessible through a unified API for agent evaluation and reinforcement learning. The platform also delays injecting tests and grading materials to reduce reward hacking, but grading still shares a sandbox with the agent, so isolation remains incomplete.
Prime Intellect has released a collection of environment integrations that can be used directly for agent evaluation and reinforcement learning. It covers approximately 198,000 software engineering tasks, 28,600 terminal operation tasks, and 137,600 search tasks—more than 365,000 in total. The key development is not the creation of new datasets, but the consolidation of 23 benchmarks—each previously using its own images, testing workflows, logging formats, and graders—behind a common taskset interface.
Under the hood, `verifiers v1` divides the environment into three layers: the taskset defines the data, tools, and grading; the harness determines how the agent runs, such as through Codex or other ReAct and CLI agents; and the runtime manages local processes, Docker, or remote sandboxes. This decoupling allows researchers to swap models and agent frameworks without rewriting the task lifecycle, while the resulting traces can be passed directly to `prime-rl` for training. In an official example, GLM-4.5-Air was trained on ScaleSWE in two days using six H200 nodes.
The integration also addresses reward hacking, a common problem in agentic RL. If the agent and grading code reside in the same container, the policy may read expected answers, modify tests, or corrupt grading state. The new version therefore delays injecting test patches, expected outputs, and grading scripts until the grading stage, and revalidates some upstream data. However, Prime Intellect acknowledges that this is only a mitigation; the true structural fix is to place execution and grading in separate sandboxes.
Engineering teams should also consider another aspect of data quality: tests derived from merged pull requests may accept only the original author's implementation details, causing agents with functionally correct but different implementations to receive false-negative scores. A dataset of 365,000 tasks reduces integration costs, but it does not automatically resolve test bias, licensing differences, or benchmark contamination. The next question is whether isolated grading and agentic judging can remove this noise from the training signal.