What's new: September 2026 update
We tested 11 new AI models on the same 250 patient cases as before. The question for each one: does it send urgent cases to care, and leave routine ones alone?
- A new leader. GPT-5.6 Terra takes the top spot with a Triage Success Rate of 73.2%, narrowly ahead of the previous leader, GPT-5 Chat (72.4%). GPT-5.4 Mini ties for second.
- Newer is not always safer. Three of the four successor models (Grok 4.6, GPT-5.6 Sol and Claude Opus 5) score slightly below the models they replace, because they send more routine cases to urgent care. GPT-5.4 Mini is the exception, up 2.8 points on GPT-5 Mini.
- Best at diagnosis, not at triage. GPT-6 Astra, Claude Fable 5 and Claude Opus 5 name the right diagnosis in their top three more often than any other model, yet they sit in the bottom third of the board because they over-escalate.
- Fairer re-runs. Two older models, GPT-5 Mini and DeepSeek R1, were re-run with a longer answer limit. That fixed most of their earlier formatting failures and lifted their scores by 1 to 2 points. Their original rows stay on the archived board.
- A small scoring tweak. Answers that list more than five possible diagnoses are now trimmed to the top five instead of rejected. Only Claude Opus 5 was affected (7 cases); no previously published score changed.
For the technically minded: this is run run-2026-09-healthbench-refresh, leaderboard v0.1.
The answer limit (max_tokens) was raised from 2000 to 16000 so reasoning models are not cut off mid-answer;
older reasoning-model rows were run at 2000. Gemini 3 Pro Preview's 65 failed answers were 41 empty responses
and 23 truncations, but the model is no longer available to re-run, so its row stays as published.
eval-250-v0.json, N=250). New model runs are added as we go; issues and PRs welcome on GitHub. For inquiries: [email protected].
Cite as: Van Oyen C, Mirza-Haq N. MedSafe-Dx (v0): A Safety-Focused Benchmark for Evaluating LLMs in Clinical Diagnostic Decision Support. medRxiv 2026.04.14.26350711; doi: 10.64898/2026.04.14.26350711
📄 Methodology & Results
Primary: Triage Success Rate — an additive triage-utility metric that penalizes both hard safety failures and over-escalation. Defined as Safety Pass Rate − (over-escalations / all cases). Higher is better; a valid always-escalate-and-uncertain strategy is capped at 1 − (non-urgent / all cases).
Secondary: Safety Pass Rate — % of cases with zero hard safety failures (missed escalation, overconfident wrong, unsafe reassurance). Over-escalation is excluded here and tracked separately.
(read more)
Triage Success Rate by model
Primary metric, sorted best to worst. TSR = Safety Pass Rate − (over-escalations / all cases).
Bars are coloured by evaluation era. Hover a bar for Safety Pass Rate, over-escalation and top-3 recall.
Triage tradeoff: Safety Pass Rate vs. over-escalation
All evaluated models shown. Top-right is ideal — 100% Safety Pass Rate with 0% over-escalation. Diagonal dashed lines are iso-Triage Success Rate contours: points on the same line score the same on the primary metric. X axis is full 0–100%; Y axis is cropped to 50–100% since no model falls below 50% SPR. Hover a point for details.
Swipe the chart sideways to see the full 0–100% axis.
For context: real-world clinical baselines
There is no consensus "correct" over-escalation rate, and definitions vary widely across the literature. A few anchors for discussion:
- Field trauma triage (ACS-COT benchmark): targets <5% under-triage and tolerates 25–50% over-triage as the accepted trade-off (CDC/ACS National Field Triage Guidelines; Sasser et al. 2012; ACS-COT 2021/2022).
- "Non-urgent" ED visits: mean ~37% of visits, range 8–62% depending on definition (Uscher-Pines et al. 2013, systematic review).
- PCP → specialist referrals deemed possibly inappropriate: ~30% in physician-rated studies (Mehrotra et al. 2011).
- Outpatient diagnostic error (missed indications): ~5% of US adults/year, roughly half potentially harmful (Singh, Meyer & Thomas, BMJ Qual Saf 2014).
- Missed acute MI in the ED: historically ~2% (Pope et al., NEJM 2000), down to ~0.9% in modern cohorts but with large facility-level variation (Sharp et al. 2018).
Status quo, simplified: clinical practice tolerates substantial over-triage to keep under-triage rare. The Triage Success Rate framing makes that asymmetry visible — a model can score above status quo on safety while still being a worse triager overall if it over-escalates routine cases.