In healthcare settings the place sufferers use LLMs as a medical assistant, LLM efficiency differs between analysis and deployment. (a) Bean et al. (2025) discover a 61 share level distinction between analysis and deployment. (b) We argue this hole arises not from poorly designed benchmarks, however from implicit assumptions embedded in analysis protocols that fail to carry at deployment. (c) We suggest a taxonomy that categorizes assumptions into two varieties, process and end result, to diagnose the place the hole arises and what’s required to shut it. Closing the hole requires making assumptions express, testing which assumptions maintain, and updating analysis protocols accordingly.
Healthcare LLM benchmarks are one of many most important paradigms by which LLMs are evaluated previous to medical settings. Benchmarks present a secure goalpost that permit researchers to iterate shortly and measure progress persistently. Nevertheless, in high-stakes domains like healthcare, that very same abstraction turns into a legal responsibility. For instance, a latest examine discovered a 61 share level drop in accuracy when going from analysis to deployment (see Determine). On this setting, sufferers use LLMs as a medical assistant to raised perceive their signs, establish the underlying situation, and take applicable actions.
Furthermore, the outcomes confirmed that sufferers given entry to a extremely succesful mannequin as a medical assistant did no higher at self-diagnosis than these with none mannequin. That’s, entry to an LLM had no important impression on affected person understanding. The implication isn’t that the mannequin underperformed. Slightly, it’s that the way in which we consider is separate from what issues in deployment. For instance, throughout analysis we ask “does the mannequin get the correct reply?” whereas throughout deployment we ask “does the affected person act accurately on what the mannequin tells them?”
We argue that this hole arises due to implicit assumptions embedded in analysis that don’t maintain in the true world. That’s, the situation that the benchmarks intend to seize and the real-world situation differ resulting from implicit assumptions. This distinction in flip challenges analysis validity. Specifically, we classify assumptions into two varieties: process, which issues assumptions on dialog information, and end result, which issues assumptions over human conduct and outcomes. To deal with this, we suggest a framework referred to as BenchmarkCards that makes these assumptions express so practitioners can establish when benchmark outcomes switch to deployment.
Understanding the Analysis–Deployment Hole by Assumptions
For instance of what our framework seems like, in Determine 1 we display our place in a healthcare setting the place LLM-as-medical-assistance efficiency differs between analysis and deployment, with a 95% to 34% hole (Bean et al., 2025). Throughout analysis, the mannequin was given doctor-written, single-turn situations—one query, one reply, no follow-up—and requested to supply a analysis. Throughout deployment, sufferers interacted with the mannequin in a back-and-forth method, and success was measured by whether or not they may accurately establish their analysis afterward.
On this setting, three assumptions underlie the hole:
- Question Distribution – Analysis makes use of doctor-written queries, whereas actual sufferers produce queries which may be incomplete or imprecise.
- Interplay Sort – Analysis options single-turn interactions, whereas actual deployments contain back-and-forth dialogue.
- Choice Mediation – Analysis measures whether or not the LLM produces the proper analysis, whereas deployment measures whether or not the affected person acts on it accurately.
We word that these are broad classes of assumptions that are current throughout analysis settings, and return to those when introducing BenchmarkCards.
Stating benchmark assumptions explicitly permits us to estimate how a lot every assumption contributes to the evaluation-deployment hole — for instance, by measuring how the identical LLM performs on multi-turn interactions versus single-turn ones. Doing so in our working instance reveals that the 61 share level hole between analysis and deployment might be damaged down into 12 factors resulting from question distribution, 19 factors resulting from interplay kind, and 30 factors resulting from resolution mediation.
That final quantity displays one thing no benchmark can observe: whether or not sufferers truly observe what the mannequin tells them. Not like the primary two assumptions, which concern how the duty is structured, resolution mediation relies upon totally on human conduct. A mannequin may accurately diagnose appendicitis, but when the affected person dismisses the advice, the end result is identical as a incorrect reply. Even a superbly designed benchmark can not seize this failure mode, which suggests mannequin evaluators, deployers, and customers want a unique mind-set about assumptions altogether.
When assumptions go unspoken, the very goal of benchmark analysis —quantifying and evaluating mannequin efficiency to information deployment choices —is defeated: practitioners haven’t any option to assess whether or not benchmark outcomes maintain of their setting, or whether or not any accessible benchmark supplies dependable steering in any respect.
Closing the Hole by Benchmark Playing cards and Staged Analysis
Assumptions fall into two classes: process and end result, which defer primarily based on whether or not they are often examined with dialog information alone. For instance, assumptions on whether or not conversations are single or multi-turn are process assumptions, whereas assumptions over proxy vs medical metrics are end result assumptions
Extra usually, we will view assumptions as clustering into two varieties: process and end result. Process assumptions concern whether or not the benchmark faithfully represents the circumstances of deployment. For instance, if real-world conversations are multi-turn, does the benchmark replicate this? Consequence assumptions concern whether or not the benchmark’s analysis criterion matches what truly issues in the true world. For instance, a benchmark may measure LLM decision-making, whereas real-world efficiency will depend on what the consumer does afterward.
Critically, we word that tackling end result assumptions requires working real-world behavioral experiments. Process assumptions might be addressed by constructing benchmarks that extra intently resemble real-world conversations, however end result assumptions rely upon human conduct that no benchmark can simulate. Understanding whether or not customers act on LLM suggestions, as an illustration, requires truly observing them achieve this.
Closing the hole requires two items of data: what assumptions a benchmark makes, and whether or not these assumptions maintain in a selected deployment context. To deal with the primary level, we suggest BenchmarkCards, structured documentation that benchmark designers fill out alongside their benchmark datasets to reply questions on their analysis protocol with out anticipating any explicit downstream use (see Desk). A practitioner dealing with a deployment resolution then makes use of the playing cards to evaluate which assumptions maintain of their setting and establish which benchmarks most intently match their use case. When no present benchmark matches properly, the cardboard makes that hole seen, and alerts to the group the place new benchmarks are wanted.
A BenchmarkCard is crammed out as soon as by benchmark designers, explicitly documenting the assumptions constructed into their analysis. A practitioner then makes use of it to evaluate which assumptions maintain of their particular deployment context. The left columns doc what the benchmark assumed; the correct column reveals the place these assumptions broke down on this deployment.
As soon as assumptions are recognized, we suggest staged analysis: an iterative course of the place assumptions are examined one after the other and analysis protocols up to date accordingly. The levels are:
- Examine BenchmarkCards towards Deployment – Use BenchmarkCards to establish which assumptions maintain and which don’t.
- Acquire Knowledge for Process Assumptions – For instance, accumulate information on actual consumer interactions to seize the distinction in question distribution. This augments a pre-existing benchmark so it’s extra relevant to a real-world setting.
- Check Process Assumptions – Measure efficiency degradations and, for assumptions with massive drops, enhance the mannequin or accumulate extra focused information. As soon as process assumptions are glad, transfer to end result assumptions.
- Check Consequence Assumptions – Utilizing area experience, prioritize which end result assumptions matter most, then run behavioral research or randomized managed trials (RCTs) to check them.
A Name to Motion
Higher benchmarks are obligatory however not enough for deploying LLMs safely in healthcare. The repair requires benchmark designers to state plainly what their analysis does and doesn’t seize, practitioners to examine these assumptions towards their deployment context, and the group to construct the infrastructure that makes this customary process fairly than distinctive effort. The ask seems totally different relying on the place you sit. For AI groups contemplating deployment: take a look at assumptions earlier than you ship, not after; don’t anticipate real-world failure to let you know the place your analysis fell quick. For researchers constructing the subsequent healthcare benchmark: doc your assumptions, so future customers can choose for themselves whether or not your analysis applies to their setting. For clinicians: deal with excessive benchmark numbers as a place to begin for dialog, not a inexperienced mild.
Acknowledgements: This weblog put up relies on our paper Healthcare LLM Benchmarks Are Solely as Good as Their Express Assumptions, co-authored with Santiago Cortes-Gomez, Mateo Dulce Rubio, Fei Fang, and Bryan Wilder. Many because of Lawrence Jang, Amanda Coston, Luke Guerdan, Sang Truong, and Tori Qiu for his or her feedback on this work.
