AI 研究與科學代理
Fisher-R1 Trains Statistical Agents with Verifiable p-Values, Reaching 33.0% Strict pass@1 on P-Hard
P-Bench requires agents to select statistical methods, execute R code, and report p-values themselves, rather than merely checking whether the code runs. The 14B model leads the listed open models on the strict metric for difficult tasks, but the public repository currently contains only documentation.
Even scientific agents capable of writing executable code can produce precise but invalid conclusions if they choose the wrong statistical test. [P-Bench](https://arxiv.org/abs/2608.07437), introduced by a team from Stanford University and the University of Wisconsin, is designed specifically to measure this gap. Its 425 tasks use real-world data from economics, biology, and medicine. Given only a research question, a data description, and a CSV file, an agent must explore the data, choose a method, execute R code, and then submit a p-value along with a decision to reject or fail to reject the null hypothesis.
The tasks cover 17 method categories, including Cox regression, instrumental variables, mixed-effects models, and nonparametric tests. The difficult subset adds pitfalls such as outliers, heteroskedasticity, and clustered observations. The Raw metric evaluates only the direction of the conclusion, while Strict additionally requires the two-tailed z-score derived from the reported p-value to be within 0.5 of the reference answer. This makes it harder to pass by simply guessing whether a result is statistically significant.
Fisher-R1 uses Qwen2.5-Coder-7B/14B as its backbone. It is first supervised fine-tuned on 3,851 filtered trajectories and then trained with DAPO on 8,642 synthetic statistical tasks. Its reward is weighted 90% toward p-value proximity and 10% toward conclusion correctness, while also requiring trajectories to contain executable code. The 14B version achieves 33.0% pass@1 on P-Hard Strict, exceeding DeepSeek V4 Pro at 26.3% and GPT-5.4 at 30.5%. The 7B version raises its backbone's P-Easy Strict score from 36.3% to 65.7%.
One notable engineering detail is that the reward does not directly verify whether the chosen method is correct. Instead, it assumes that an inappropriate method will generally produce a different p-value. When multiple analytical strategies are equally reasonable, however, a single reference answer may still be too restrictive. In addition, at the time of verification, the [public repository](https://github.com/jmu27/FisherR1) contained only a README and did not yet provide the weights, P-Bench data, training code, or license described in the paper. The next questions are whether the full release can reproduce the reported results and how reliable the model will be in more realistic workflows involving preregistration, composite endpoints, and multiple comparisons.