Generative AI in Healthcare: The Deployment Record

Almost everything written about generative AI in healthcare is a catalogue. Eighteen use cases, a maturity column, a vendor per row, an ROI formula at the bottom. The catalogues are not wrong. They are just answering a question nobody has to decide: whether a generative model could draft a clinical note. It can.

The question a health system actually decides is what happens on week four. Until 2026 there was almost no published answer, because nobody had run a generative system in a live care setting with a control group and then written down what they saw. That literature now exists. It says something the catalogues do not.

What counts as generative here

Generative systems produce something new: a note, an answer, a differential, a molecule. That distinguishes them from the classification and prediction models that make up nearly all authorised medical AI, which sort or score data that already exists. The distinction is not pedantic. The entire apparatus for evaluating and regulating medical software was built for the second kind, where you can define a correct answer and count how often the model finds it. For a broader map of the 2026 evidence base for AI in healthcare, the prediction models are where the clearances are. Generative systems are where the deployments are, and the two sets barely overlap.

The adoption number everyone quotes measures intent

The figure in circulation comes from repeated surveys of healthcare leaders: 85% of respondents were exploring or had already adopted generative AI capabilities, drawn from 150 stakeholders, 29% of them C-level, fielded in the fourth quarter of 2024.

That is a real number about budgets. It is not a number about outcomes, and it is routinely quoted as though it were.

Where outcomes have been measured, they are smaller and more contradictory than the adoption figure implies. The National Academy of Medicine's volume on the subject sets them side by side: clinicians at the Permanente Medical Group reported favourable experiences and reduced documentation time with ambient note generation, while a study at Atrium Health found no overall difference in electronic health record or financial metrics, with small signals of benefit only among low-volume clinicians and heavy users. Mayo Clinic nurses saved roughly thirty seconds per message using drafted replies. At UC San Diego, physicians found the drafts relieved cognitive burden, and also read for longer and wrote longer replies.

Thirty seconds per message is a genuine result. It is not the result the catalogue implies.

The emergency department study that should change how this is bought

The most important paper on this topic in 2026 is a prospective evaluation, and it is barely cited outside methodology circles.

SHAKED is a clinical decision support system built on multiple large language models, deployed in a tertiary emergency department at Rambam Health Care Campus and evaluated to the DECIDE-AI stage 1 standard. Over four weeks, 1,138 patients were analysed across two parallel units, one using the system and one following routine rotations.

The outputs were fine. That was not the problem

Expert review rated 99 of 100 sampled outputs as clinically appropriate. No adverse events were detected. On any benchmark the catalogues use, this is a success.

Clinical adoption fell from 68% to 30%.

The decline was workload-sensitive, at an odds ratio of 0.72 per shift hour: the busier the shift got, the less the system was used. Physicians did keep reaching for it in one place, preferring it for radiology consultations at an odds ratio of 2.98. Emergency department length of stay was identical in both wings at 4.9 hours. Consultation cycle time showed a non-significant trend of 9.4 minutes shorter, at P = 0.077.

The authors' conclusion is the sentence to take away: sustained clinician engagement, rather than algorithmic accuracy, may be the key barrier to effective clinical AI use in emergency departments. They also state plainly that their findings inform randomised trial design and do not justify deployment. Both halves of that are unusual and both are worth respecting.

Read against the procurement process most health systems run, this is awkward. Vendor evaluation almost entirely tests output quality, which was not the thing that failed. The thing that failed was a clinician under pressure deciding the tool was not worth the seconds it cost, and that failure is invisible to every pilot that reports week-one usage.

Benchmarks do not predict bedside behaviour

The same gap shows up from the other direction.

Stanford Medicine published its lessons from piloting and deploying a large language model inside a large medical centre, and the headline finding is methodological: benchmark-based evaluations are insufficient for monitoring and evaluating interactions driven by clinicians, and doing it properly requires new monitoring methods. A model that scores well on a fixed question set behaves differently when real clinicians are choosing the questions, and the institution running it has to watch it continuously rather than certify it once.

Purpose-built clinical tools lost to general models

The second result is more uncomfortable for the vendor market. An independent evaluation put two specialised clinical AI products, OpenEvidence and UpToDate Expert AI, against three frontier general models across three stages: 500 MedQA questions, 500 HealthBench items, and a benchmark of 100 de-identified real clinical queries taken from physicians using a HIPAA-compliant instance in live practice, scored by 12 blinded US clinicians producing 1,800 annotations.

The frontier models outperformed the clinical tools in all three evaluations, and the clinical tools performed comparably to Google Search AI Overview on the real clinical queries. UpToDate's tool refused 19% of queries against 1 to 3% for the others. Notably, no model produced significantly more harmful content or hallucinations than any other, so the difference was usefulness rather than safety.

Buying a clinical wrapper does not automatically buy clinical quality. It may buy retrieval, provenance, an audit trail and an indemnity, all of which matter. It does not buy better answers by default, and that is now a testable claim rather than an argument.

Hallucination has a measured rate, and the rate is not fixed

The most useful number in this field comes from one of the largest manual evaluations of clinical note generation performed to date. Across 18 experimental configurations and 450 consultation transcript and note pairs, 12,999 clinician-annotated sentences yielded a 1.47% hallucination rate and a 3.45% omission rate, with 44% of hallucinations classified as major against 16.7% of omissions.

Three things in that dataset matter more than the headline rate.

Omissions are commoner, hallucinations are worse. A missing sentence is usually recoverable. An invented one is not.

The dangerous category is negation. Thirty per cent of hallucinations were negations, statements contradicting what was said in the consultation, and they clustered in the plan section, which is exactly where instructions to colleagues and patients live. A reader without the original conversation cannot tell which negation is true.

And the rate moved. Changing the prompt to require a status of "unknown" where information was missing cut major hallucinations by 75% in the same pipeline. The error rate is a property of the workflow, not of the model, which means it is something an institution can own rather than something it has to accept from a vendor.

For general clinical reasoning rather than documentation, the picture is similar in shape. A blinded randomised expert evaluation ran seven models against 685 real and simulated cases across 13 specialties, judged by 27 clinicians averaging 25.9 years of service. Between 3% and 19% of responses were rated incompetent, 40 instances of hallucination were identified, and the top-performing model after adjustment scored at a level equivalent to an early-career physician. An early-career physician is a real standard. It is also a standard that comes with supervision attached, which is the part that tends to get dropped in the retelling. This sits alongside the documented risks that apply to clinical AI generally.

Who grades the model is part of the result

One more finding should change how benchmark claims are read at all.

More than 400 physicians spanning seven specialties, varied experience levels and multiple geographic settings evaluated model-generated free-text responses to real de-identified clinical cases. Physician assessments varied substantially by clinical seniority and practice environment, producing notable shifts in relative model rankings across cohorts. The same study deployed matched AI evaluator agents configured to mirror physician characteristics: they were efficient and directionally aligned, but did not capture the nuance of human clinical judgment and could not substitute for physician-centered evaluation.

The practical consequence is short. A model ranking without a stated rater cohort is not a comparable number, and a vendor benchmark scored by AI judges is a triage tool, not evidence.

The cheapest deployment case is the one nobody is selling

There is one setting where the economics are not marginal, and it is not the one the market is built around.

A retrospective diagnostic accuracy evaluation at a tertiary public hospital in South Africa ran ten frontier multimodal models over 539 inpatient cases, with specialist panels adjudicating 300 of them as ground truth and routine ward diagnoses included as a real-world comparator. Mean diagnostic performance clustered within 15% relative variation despite a 50-fold range in cost, GPT-5 Mini came within 5% of the top model at a mean of $0.02 per case, and all ten models scored significantly higher than routine ward diagnoses on diagnosis and treatment-safety metrics.

The caveats are the authors' own and they are not small: the study is retrospective, scoring was performed by a calibrated jury of models rather than by humans throughout, and tail-risk recommendations remain a concern. But the direction is worth sitting with. In a high-volume public hospital where specialist time is the binding constraint, two cents a case is a different proposition from a per-physician monthly licence in a system that already has specialists. The affordability, robustness and governance questions matter more there than a few points of mean accuracy.

That inversion is why this evidence tends to come from outside the usual venues, and why it takes longer to reach the people writing procurement policy.

None of this is regulated as a medical device

Worth stating plainly, because it is often assumed otherwise. On 18 August 2026 the FDA issued a discussion paper on generative AI enabled medical devices and opened docket FDA-2026-N-7874, with comments due by 19 October 2026. A discussion paper is not draft guidance and binds nobody.

So every deployment described above ran under institutional governance rather than federal device authorisation. The hospital was the regulator. Anyone doing AI regulatory compliance work on a health product is currently writing the rules they will later be held to, and the dated milestones that will settle it are covered in our note on what is already scheduled.

Six questions to ask a vendor

Each one is answerable, and each one comes from a finding above.

1. What was adoption at week four, not week one? If the pilot only reports initial usage, it has not measured the thing that failed in the only prospective ED study published.
2. What happens when the shift is busy? Ask for usage disaggregated by workload. Disengagement was workload-sensitive, and a tool used only on quiet shifts has a different value than the demo suggests.
3. Who rated your benchmark, and at what seniority? Rankings shift between rater cohorts. An unattributed score is not comparable to anything.
4. What did you compare against? If the comparator is not a current frontier general model, the claim of clinical specialisation is untested.
5. What is your measured hallucination and omission rate on our note type? Not on a public benchmark. On the documents this deployment will produce, with the split between major and minor stated.
6. Who is monitoring it after go-live, and how? Certification at purchase does not survive contact with clinicians choosing their own questions.

None of these are hostile questions. A vendor with a real product has the answers, and the evidence base to check them against finally exists.

That evidence sits in three separate conversations that rarely happen in the same room: the researchers running the deployment studies, the regulators deciding what counts as proof, and the capital deciding which products get built. The summit runs Health Tech, AI Ethics and Global Financial Raise as separate tracks in the same building, across a continuous thirty-hour broadcast between Las Palmas and Bali, free online for verified researchers, students and the EX community. The reason to put them together is visible in the record above. The people measuring what generative AI does in a ward and the people deciding what it is worth have mostly not been reading each other.

FAQ

What is the difference between AI and generative AI in healthcare?

Generative systems produce new content: a clinical note, an answer to a question, a candidate molecule. The classification and prediction models that make up nearly all authorised medical AI sort or score data that already exists. The distinction matters because the evaluation and regulatory machinery was built for the second kind, where a correct answer can be defined in advance.

Is generative AI accurate enough to use in a hospital?

In the one prospective emergency department evaluation published, expert review rated 99 of 100 sampled outputs clinically appropriate and detected no adverse events, and the authors still state their findings do not justify deployment. Accuracy was not the limiting factor. Sustained use under real workload was.

How often do medical LLMs hallucinate?

One of the largest manual evaluations of clinical note generation measured 1.47% of generated sentences as hallucinations and 3.45% of source sentences as omissions, with 44% of the hallucinations rated major. The rate moved substantially with prompt and workflow changes, so it behaves as a property of the deployment rather than a fixed property of the model.

Are specialised medical AI tools better than general models?

Not on the published independent comparison. Frontier general models outperformed two purpose-built clinical tools across all three evaluation stages, and the clinical tools scored comparably to Google Search AI Overview on real clinician queries. Specialised tools may still buy provenance, audit trails and indemnity, which are separate goods.

Does generative AI in healthcare need FDA approval?

There is no binding framework yet. The FDA issued a discussion paper on generative AI enabled medical devices in August 2026 and opened a comment docket that closes on 19 October 2026. Until a rule exists, most generative deployments in US hospitals run under institutional governance rather than device authorisation.