Dangers of AI in Healthcare: The 2026 Incident Record

Most writing about the dangers of AI in healthcare describes a hospital system: an algorithm that was procured, validated, wired into the record and then found to be biased, brittle or wrong. That is a real category. It is not the one that came first in 2026.

In January, ECRI published its eighteenth annual Top 10 Health Technology Hazards, and the entry at number one was the misuse of AI chatbots in healthcare, placed above unpreparedness for a digital darkness event, substandard and falsified medical products, and seven other hazards drawn from incident investigations, reporting databases and independent device testing. Nothing in that top entry was procured by a health system. No committee approved it, no regulator cleared it as a device, and it does not appear on any model inventory, because it was never bought.

That is the shape of the problem this year. The dangers with documented harm behind them are entering healthcare through a side door.

The hazard that ranks first is not a medical device

The reason a general-purpose chatbot outranks every device on the list is structural. Large language models produce fluent, expert-sounding answers to any question put to them, and they are built to sound confident and to always supply an answer, whether or not the answer is reliable. ECRI's testing found chatbots suggesting incorrect diagnoses, recommending unnecessary testing, promoting subpar medical supplies and inventing body parts. In one test, ECRI asked whether an electrosurgical return electrode could be placed over a patient's shoulder blade. The chatbot said it was appropriate. Followed, that advice leaves the patient at risk of burns.

The exposure is the part that moves it to the top of a hazard ranking. ECRI notes that more than 40 million people a day turn to ChatGPT for health information, while these models are not regulated as medical devices even as patients and providers use them more. A device recall reaches the hospitals that bought the device. There is no equivalent mechanism here, because there is no purchase order and no installed base to notify.

The ranking has also been moving in one direction for three years. Insufficient governance of AI in medical technologies came fifth in 2024, risks from AI-enabled health technologies topped the list last year, and this year the entry narrowed to chatbots specifically. The category did not get broader as AI spread. It got more precise, which usually means the incident data got clearer.

What a documented danger actually looks like

A risk is a property of a system. A danger is what happens to a particular person on a particular day, and in this case it is in the literature with a lab value attached.

A case report in Annals of Internal Medicine: Clinical Cases describes a 60-year-old man with no psychiatric or medical history who arrived at an emergency department convinced his neighbour was poisoning him. He had read about the harms of table salt, wanted to remove chloride from his diet, consulted ChatGPT about what chloride could be replaced with, and spent three months substituting sodium bromide bought over the internet, presenting with a chloride reading of 126 mmol/L, developing paranoia and auditory and visual hallucinations within 24 hours, attempting to escape, being placed on an involuntary psychiatric hold, and spending three weeks as an inpatient before discharge with a bromide level of 1700 mg/L against a reference range of 0.9 to 7.3.

The clinically instructive part is what his doctors did next. They could not obtain his conversation logs, so they ran the query themselves. ChatGPT 3.5 returned bromide as an answer. It noted that context matters. It did not attach a health warning, and it did not ask why they wanted to know, which is precisely the question a clinician asks first. The authors draw the conclusion narrowly and correctly: it is highly unlikely that any medical expert would have named sodium bromide to a patient looking for a salt substitute, and providers now need to screen for where their patients are getting health information.

One preventable admission, three weeks long, is not a hypothetical about model bias. It is the difference this article turns on.

The same danger, wearing a hospital badge

The consumer version is easy to picture. The clinical version is the one that undermines every governance control a health system has built.

A December 2025 survey of 518 healthcare professionals by Wolters Kluwer Health found that 40% had encountered an unauthorised AI tool in their organisation, nearly 20% admitted using one, roughly one in ten had used an unauthorised tool for a direct patient care use case, and half of those who used them cited the need for faster workflows. The behaviour is not recklessness. Reporting on the same survey notes that more than half of administrators and 45% of care providers pointed to faster workflow, around a quarter of providers cited curiosity or experimentation, and about a quarter of all respondents ranked patient safety as their single biggest AI concern. People who name patient safety as their top worry are using unapproved tools anyway, because the approved ones are slower or absent.

The scale is corroborated elsewhere. Black Book Market Research found that 58% of front-line clinicians had used generic AI tools such as ChatGPT for work-related tasks at least once in the previous 30 days, with 39% using them weekly or more, and 42% of clinicians acknowledging inaccurate outputs as a risk.

Set that against how oversight is supposed to work. Every control a hospital runs, validation, monitoring, drift detection, incident reporting, post-deployment ownership, starts from a list of the systems in use. A tool that entered through a browser tab is not on that list, so none of the controls downstream of it can see the tool, and none of the harms it causes get attributed to AI in the incident record. This is the failure mode underneath a committee almost everyone has and few can enforce: the inventory is the test of whether governance is real, and shadow AI is defined by being absent from it.

Why regulation does not reach the tool doing the harm

The obvious question is why a device regulator has not intervened. The answer is that the framework is doing what it was built to do, and what it was built to do does not include this.

At its Digital Health Advisory Committee meeting on 6 November 2025, the FDA recorded that it had authorised more than 1,200 AI-enabled medical devices, that no generative AI-enabled medical device had been authorised for use in any mental health condition, that the authorised digital mental health therapeutics are cleared as adjuncts to usual care and none of them are AI-enabled, and that the meeting was explicitly not focused on broadly available generative AI platforms but on devices intended for diagnosis, cure, mitigation, treatment or prevention.

That last clause is the whole gap in a sentence. Device regulation attaches to intended use, declared by a manufacturer in a submission. A general-purpose chatbot makes no medical claim, so it is not a medical device, so it is not in scope, and the patient asking it about a salt substitute is standing outside the regulatory perimeter entirely. The agency's posture is risk-based rather than absent: general wellness products sit outside device oversight, enforcement is prioritised toward use cases with higher potential for harm, and the committee returned repeatedly to crisis escalation and to ensuring a qualified human intervenes. Those are the right mitigations for a product somebody submits. They do nothing for a product nobody submitted. An organisation can be entirely current on what AI regulatory compliance requires today and have no answer at all to a clinician pasting a chart summary into a consumer chatbot.

The dangers that are real but slower

None of this means the hospital AI dangers are imaginary. It means they belong in a different column, and they are covered properly elsewhere rather than repeated thinly here.

Automation bias, error rates that assume a review step nobody has time for, and unclear liability are harms that accumulate inside approved systems and land on identifiable people, which is why they are sorted by bearer in the risks of AI in healthcare, ranked by who carries them. The verification tax, the changing shape of the record and the trust cost are not failures at all but what a health system pays when the tool works exactly as designed. How these models behave once they leave the pilot is the generative AI deployment record, and who is obliged to answer for any of it is settled in the questions the ethics of AI in healthcare now asks.

The distinction worth keeping is temporal and it decides who should act. Those harms degrade care gradually and have an owner, however imperfect: a committee, a vendor contract, a named clinician. The chatbot hazard injures a specific person quickly and has no owner at all.

The dangers that are mostly imagined

An honest danger list has to say what is not happening, or the real items get discounted along with the invented ones.

The popular fear is autonomy: AI replacing clinicians, diagnosing without oversight, deciding treatment alone. The regulatory record points the other way. The authorised digital mental health therapeutics are adjuncts to usual care rather than replacements, no generative AI system has been cleared to treat a psychiatric condition, and the advisory committee's recurring preoccupation was building human escalation into products that have not yet been approved.

So the deployed reality is close to the inverse of the fear. Inside the hospital, AI mostly recommends and a human signs. Outside it, an unregulated consumer layer answers medical questions at enormous volume with no clinician anywhere in the loop. The autonomy people worry about already exists. It is just sitting on the patient's phone rather than in the radiology suite.

What actually reduces the danger

Four things follow directly from the evidence above, and none of them is a new committee.

Ask the patient what they consulted before they arrived. That is the case report authors' own recommendation, and it is the only control that reaches the consumer layer at all. A medication history that does not ask about internet-purchased substances taken on chatbot advice will miss the next bromism the same way this one was missed for three months.

Provide a fast sanctioned tool rather than banning the fast unsanctioned one. The survey evidence is unambiguous that speed is the driver, and a prohibition does not make the approved tool faster. It only moves the same behaviour further out of sight, which costs the organisation the one thing it still had: the chance to know it is happening.

Put consumer-grade AI on the inventory as a known exposure even though it was never procured. An inventory of what was bought describes a smaller system than the one actually operating.

And name the escalation path. The FDA committee kept returning to the same requirement for products it may one day approve, which is that a qualified human is prompted to intervene at the point of crisis. The tools already in use have no such path, so the organisation has to supply it.

The room this has to be settled in

The chatbot hazard crosses four jurisdictions and sits inside none of them. A device regulator whose authority attaches to declared intended use. A patient safety organisation with the incident data but no enforcement power. A hospital governance committee whose remit stops at what it procured. And a consumer software vendor with no healthcare mandate at all and forty million daily health questions arriving anyway. Each is behaving reasonably within its own boundary. The harm is happening in the space between them.

That gap is the structure the EX Future Summit was built around. Its nine program tracks include Government, Health Tech and AI Ethics, alongside eight hosted meet-and-greet sessions matching universities with private partners, because the regulator, the safety researcher, the clinician and the model developer currently answer this question separately and in different rooms. The summit runs 18 to 20 November 2026 as a single continuous thirty-hour broadcast between Las Palmas and Bali, twelve hours apart, with online attendance free for verified researchers, students and the EX community.

FAQ

What is the biggest danger of AI in healthcare right now?

By the 2026 hazard ranking, it is the misuse of AI chatbots, placed first out of ten by a patient safety organisation working from incident investigations and its own device testing. The ranking is notable for what it is not. It is not a diagnostic algorithm or a clinical decision support system, both of which are regulated, inventoried and monitored. It is a general-purpose tool with no medical intended use, which is exactly why it sits outside every mechanism built to catch a dangerous medical technology.

Has AI actually harmed a patient, or is this still hypothetical?

It is documented. A published case report describes a man who replaced table salt with sodium bromide for three months after consulting a chatbot about eliminating chloride from his diet, presented to an emergency department with paranoia, developed hallucinations, was placed on an involuntary psychiatric hold and spent three weeks as an inpatient with a bromide level more than two hundred times the top of the reference range. His doctors reproduced the query and received bromide as an answer with no health warning attached.

Is it safe to ask ChatGPT medical questions?

Nobody has assessed whether it is, which is the accurate answer rather than a cautious one. These models are not regulated as medical devices, so no agency has reviewed their performance for this use, and no manufacturer has claimed it. The specific failure mode identified in the published case is worth knowing: the model answered a chemistry question correctly in the abstract and never asked why the person was asking. A clinician's first move is to establish context. A model optimised to give a confident answer has no reason to.

Are AI therapy chatbots approved by the FDA?

No generative AI-enabled medical device has been authorised for use in any mental health condition. The digital mental health therapeutics that have been authorised are cleared as adjuncts to clinician-managed care, psychotherapy or medication, and none of them are AI-enabled. Products marketed as AI companions or wellness apps are generally not devices at all, which means their absence from the authorised list is not a pending approval but a different regulatory category.

Should hospitals ban staff from using consumer AI tools?

A ban addresses the symptom the survey data identifies as secondary. The stated driver is workflow speed, followed by approved tools lacking the functionality clinicians need, and neither is changed by a prohibition. What a ban reliably does change is visibility: usage that was reportable becomes usage that is concealed, at an organisation that already cannot see it on the model inventory. The alternative is not permissiveness but substitution, providing a sanctioned tool fast enough to compete with the unsanctioned one.