科學 AI 與代理
SciDataSailor Lets Agents Explore Scientific Databases Directly, Doubling the 9B Model’s Pass@1 from 14.01 to 28.99
SciDataSailor uses Monte Carlo tree search to synthesize executable file-exploration trajectories and has released more than 2,300 supervised examples. After fine-tuning, Qwen3.5-9B made fewer trial-and-error attempts in evaluations with constrained step budgets, though some scoring still relies on an LLM judge.
Most scientific question-answering benchmarks provide models with curated text or tables, without measuring whether an agent can locate files in an unfamiliar database, infer schemas, align identifiers, and perform actual computations. SciDataSailor combines these steps into a “deep scientific data exploration” task: the agent may call Python only within a controlled sandbox and cannot use web search or external retrieval. All conclusions should be derived from file contents and program execution results.
The team uses Monte Carlo tree search to synthesize training trajectories. The system first selects exploration intents by difficulty, then adjusts the priority of initial exploration actions based on execution results and feedback. It expands high-level strategies into concrete tool actions and allocates the search budget according to branch uncertainty. Trajectories that cannot be executed, provide insufficient information, or lack evidence are filtered out. The released SciDataSailor-SFT-2K dataset actually contains 1,102 database-overview trajectories and 1,225 targeted question-answering examples spanning the life, Earth, and physical sciences.
The accompanying benchmark uses 27 raw datasets and includes 627 metadata-curation tasks and 586 scientific questions. After Qwen3.5-9B was fine-tuned on these trajectories, its Pass@1 on database-overview tasks under a 12-step limit rose from 14.01 to 28.99, while its completion rate increased from 49.76% to 96.14% and its average step count fell from 10.21 to 6.24. This suggests that trajectory supervision primarily teaches smaller models to decide more quickly which files to inspect, rather than merely adding domain knowledge.
The results also reveal the importance of evaluation budgets. When the step limit increased from 12 to 24, DeepSeek-V4-Pro’s Pass@1 on overview tasks rose from 16.43 to 52.66. Proprietary models generally find viable paths earlier, while some open models narrow the gap through additional trial and error. Engineering comparisons of scientific agents must therefore report tool-call limits, token budgets, error-recovery policies, and sandbox configurations alongside performance results.
One limitation is that metadata answers are scored by an LLM judge; only targeted question answering uses numerical tolerances and text-based rules. Although the training data has been released, the reproducibility of the full benchmark and generation pipeline still requires external validation. The 27 datasets are also insufficient to represent the access controls, corrupted files, proprietary formats, and data-governance issues encountered in real laboratories.