Back Home

AI 研究

MedChemBench Breaks Down Drug Decisions: Understanding Molecules Still Doesn’t Make Drug Selection Easy

The headline score exceeds 80%, but the detailed breakdown comes from a different model version; teams using AI to prioritize experiments should focus more on candidate selection and property prediction.

zh-Hant

Researchers who want to use AI to decide which molecule to test next need to distinguish whether a model can read molecular structures, explain evidence, or actually select candidates. On November 20, OpenAI introduced MedChemBench, which probes these abilities with 99 retrospective research questions. The official report says GPT‑6.2 Astra scored 84.5% of available points with xhigh reasoning enabled. This is the share of points earned under the scoring rubric, not the probability of successfully selecting a drug. Official announcement

Research questions that stop at the point of trade-offs

MedChemBench draws on 117 papers and patents. It provides molecules and experimental data available before historical decision points, while reserving subsequent results as reference answers and for scoring. The questions go beyond chemistry knowledge: they also ask models to judge which compounds merit further development amid competing considerations. Research paper

For example, an official question about PRMT5 inhibitors concerns a target protein studied in cancer drug research. Subsequent experiments showed that compound 23 inhibited PRMT5 inside cells at a lower concentration, while compound 24 crossed cell barriers more readily and broke down more slowly in a liver enzyme assay. To predict these properties, a model cannot treat stronger inhibition as the only advantage. Question details

Read the total score separately from drug-selection ability

In the November 9 paper, GPT‑6 Astra, with xhigh enabled, scored 95.37% on molecular structure, 91.05% on scientific explanation, 50.42% on ranking and candidate selection, and 47.02% on numerical prediction. These categories use different questions and rules, so the figures should not be treated as accuracy rates on equally difficult tasks. Breakdown of results

The scoring weights matter too: molecular structure and scientific rationale together account for an average of about 73% of the available points. Our interpretation is that a high total score needs to be understood in light of its intended use; for prioritizing experiments across candidates, the selection and prediction categories are more directly relevant. Scientific rationales are scored by a GPT‑6 Astra judge model against a rubric, so this should not be treated as validation by independent chemists. Scoring method

The model versions must also be kept distinct: the 84.5% figure belongs to GPT‑6.2 Astra, added on the announcement page; the detailed breakdown above is for GPT‑6 Astra and cannot be directly applied to the newer version. Announcement page

Measuring answers to historical questions does not validate experimental decisions

This is OpenAI’s own evaluation, and its two sources are not independent cross-validations. Testing used single-turn answers, without tools, feedback, or new experiments, and cannot rule out prior exposure to public sources. The results therefore do not yet show that the model can discover new drugs or improve laboratory decisions. Official limitations

Historical reference answers also have a limitation: the paper acknowledges that reasonable but different molecular designs may be undervalued. For research teams adopting AI, the benchmark is useful for examining structure, rationale, and final selection separately; a high score in one area does not validate the others. Discussion in the paper

Sources

  1. Can AI help scientists identify promising drug candidates?
  2. MedChemBench: Evaluating complex multimodal reasoning in medicinal chemistry