PatientAgentBench Tests Medical Agents Across 1,200 Tool-Use Dialogues, Yet the Best Model’s Triage Pass Rate Is Still Only 88%

The Amazon Science research team released PatientAgentBench on July 28 in an effort to address a gap in medical model evaluation as it moves from static question answering to agentic workflows. Instead of asking a model to answer a single medical question, the framework wraps a foundation model in a LangGraph ReAct agent, allowing it to read synthetic medical records, conduct multi-turn conversations with simulated patients, and operate 15 stateful tools for tasks including appointment scheduling, prescriptions, telehealth, and personal information management.

Each trajectory is scored by an LLM jury composed of multiple models across six dimensions: clinical safety, workflow correctness, triage quality, clinical helpfulness, task completion, and conversational quality. The team had practicing clinicians review a shared sample and reported adjacent-category agreement rates of 79% to 93% between the jury and the experts. However, this only indicates that their ratings were broadly similar; it does not eliminate bias from automated judges.

Differences among the ten models across 1,200 scenarios were driven primarily by triage performance. The weakest model had a triage pass rate of 32%, while even the strongest configuration reached only 88%. Common errors included carrying out administrative requests before completing clinical screening, trusting unverified tool outputs, and failing to provide crisis resources in emergency situations. The paper reports that frontier models still had failure rates of roughly 1% to 3% in clinical safety and workflow correctness, with the highest overall score reaching 4.25 out of 5.

The repository is publicly available and can generate synthetic scenarios on the fly from a configurable seed distribution, reducing the risk that fixed answers will be memorized. It also supports Bedrock, OpenAI-compatible, and Anthropic-compatible endpoints. Engineering teams should examine whether the evaluator models share the same model lineage as the models under test, whether synthetic patients adequately represent real-world populations and linguistic variation, and how the tool sandbox differs from actual healthcare systems in permissions, latency, and failure semantics. The authors also explicitly state that PatientAgentBench is a research benchmark, not a clinical certification or authorization for deployment.