代理與科學 AI
ScienceIDE Packages Scientific Software into Verifiable Agent Environments and Feeds Numerical Simulation Results Directly into Training
ScienceIDE combines fixed versions of scientific software, containers, physics cases, and private verifiers into shared environments for SFT, RL, and evaluation. Initial results suggest that scientific repair trajectories can improve some models and benchmarks, but only a small subset of the environments and tasks has been released, leaving limited evidence of generalization.
PhAI Labs and collaborators have released ScienceIDE, an effort to turn domain knowledge scattered across scientific codebases into executable, gradable learning environments for agents. Each task pins a specific upstream software version and runs it in a container. After an agent modifies the source code, a verifier recompiles it, runs a physics case, and compares the numerical output against reference results or invariants. Rewards therefore depend on whether the simulation has been restored to correctness, rather than whether the patch matches the authors’ solution verbatim.
The paper presents 64 environments derived from 27 scientific software packages, collectively producing 2,812 tasks and 1,076 numerical or physics checks. The vast majority involve bug fixes or reconstructing missing functions; only two are performance-optimization tasks. The public ScienceIDE-Hard benchmark contains 85 tasks spanning astrophysical fluid dynamics, plasma, ocean, and particle simulations. Fifteen model–agent combinations were each given one hour. The highest observed success rate was 67.1%, although the authors explicitly caution that the top-ranked system was run only once and that its confidence interval overlaps those of other leading systems.
The team also fine-tuned 4B, 9B, and 72B models on interaction trajectories that passed verification, while using the containerized verifier’s outcome as the reward for multi-turn RL. Notably, treating trajectories that exceeded the token or turn budget as outright failures encouraged models to shorten their responses instead of completing more difficult repairs. ScienceIDE avoided training collapse by retaining truncated trajectories in the group baseline while masking them out of the policy loss. After 30 steps, the 4B model’s mean shaped reward on two small held-out task sets rose from 0.357 to 0.857 and from 0.286 to 0.571, respectively.
The GitHub repository currently releases only 15 of the 64 environments and 30 of the 85 hard tasks; the complete task library and generation pipeline remain private. Its engineering value lies in providing a shared executable contract for scientific agents, SFT, and RL. Whether the approach can generalize beyond existing codebases, across different scientific domains, and to more open-ended discovery tasks will require independent reproduction.