AI Governance in Healthcare: Who Signs Off, and When

Ask a health system whether it does AI governance in healthcare and the answer is now almost always yes. A CHIME Foundation survey of 51 healthcare organisations found that 84% had established an AI Governance Committee, with chief information officers sitting on 63% of them and chief medical information officers on 45%. Ask the second question, whether anything has to pass through that committee before it reaches a patient, and the number drops: only 59% of the same organisations had a formal, documented process requiring approval before an AI implementation.

That twenty-five-point gap between having a body and having a gate is the whole subject. A committee that cannot stop a deployment is a reading group.

The survey is small and was run by a risk vendor, so treat the exact percentages as directional. The direction is corroborated. Black Book Research found that only 29% of hospitals and health systems have implemented and enforced AI policies covering model inventory, lineage and sign-offs, with 48% still drafting them. And a survey of 233 health systems by HFMA and Eliciting Insights found 88% using AI internally against just 18% with a mature governance structure and a fully formed AI strategy, while the share of CFOs reporting some governance structure rose from 40% in 2024 to nearly 70% in 2025. Structure arrived fast. Enforcement did not follow it.

What AI governance is made of, and the part that keeps going missing

The most useful definition available comes from a scoping review published in npj Digital Medicine in May 2026: AI governance is the structures, policies and processes that ensure the responsible development, deployment and oversight of AI systems. Its value is in what the authors did next. They collected 77 AI governance frameworks written for acute care and tested each one against four components: guiding principles, an assessment method, coverage of the AI lifecycle stages, and an oversight mechanism.

Ten of the 77 had all four. Thirteen percent.

The component that was missing most often was the oversight mechanism, present in 15 frameworks, or 19.5%. Which is to say that four out of five documents published to govern healthcare AI never specify who has the power to approve or deny anything. The review is polite about this. The plainer reading is that principles are cheap to write and oversight is expensive to run, so the literature has accumulated at the cheap end. Where principles are the subject, the field is well served: the ethics of AI in healthcare is now a set of questions with named answerers, and those questions are largely settled. Governance is the harder half, because it has to say which chair the answerer sits in.

The frameworks that did include an oversight mechanism converge on the same shape: an AI-specific governance group with mandatory multidisciplinary membership across clinical, ethics, legal, IT, operations, research and consumer representation, and in the more mature cases a two-tier arrangement, a central committee owning the process with subcommittees taking specific checkpoints across the lifecycle.

Who is accountable, according to the hospitals themselves

National data says the accountability is real but distributed. The ONC data brief drawn from the American Hospital Association IT Supplement, covering 2,080 hospitals, found that 71% used predictive AI integrated with the electronic health record in 2024, up from 66% in 2023, and that 74% reported multiple entities were accountable for evaluating it, with a quarter naming four. A specific committee or task force was named by 66% and division or department leaders by 60%. IT staff came last at 41%.

ONC draws the comparison itself: this looks like the three lines of defence used to govern quantitative models in financial services, front-line operations, risk management, internal audit. That is a good sign, and it comes with a familiar failure mode. Distributed accountability only works if the handoffs are written down. Where they are not, it becomes the finding in the Black Book data that 33% of organisations say unclear internal ownership between IT, quality and safety, and compliance is what slows their governance down. Four accountable entities and no named owner is the same thing as none.

The inventory is the test of whether any of it is real

Nothing in a governance process works before the list of systems exists, and the list is where most programmes are weakest. In the CHIME survey, roughly one organisation in ten used automated product monitoring to detect AI capabilities. The rest relied on informal ad hoc discovery, cited by 51%, or on vendor release notes, also 51%. Both of those are ways of finding out about a model after somebody else decided to ship it.

The audit consequence is measurable. Only 22% of hospitals reported high confidence that they could deliver a complete, auditable AI explanation within 30 days to a regulator or payer, and that figure was 34% at large systems, 21% at community systems of three to nine facilities, and 15% at small hospitals. The stated obstacles are mostly upstream: 41% named limited explainability artifacts from vendors, things like model cards and drift reports, as their top audit barrier, and 37% said their tracking of data inputs and model versions was incomplete.

Set that against what is being spent. The median 2026 budget share for AI governance and safety was 4.2%. An organisation cannot buy an inventory, a model registry, lineage tracking, monitoring and override logs out of four percent of an AI budget, and the 30-day audit answer is exactly the artifact all of those produce. This is one of the running costs the purchase price leaves out, and it is the one most often discovered after the fact.

Approval at go-live governs a system that no longer exists

The second structural weakness is temporal. A point-in-time approval assumes the model behaves on day 300 as it did on day one, and none of the three things in the room hold still: the model gets updated, the data drifts, and the clinicians adapt to the tool.

The risks that healthcare IT leaders rank highest are the ones a launch-day review is worst at catching. Output quality and hallucinations were named as the primary data risk by 63% of CHIME respondents, and automation bias, the tendency of clinicians to over-rely on an output without independent scrutiny, was the top operational worry at 27%. Neither is visible at go-live. Both accumulate. What they cost, and to whom, is the subject of the risks of AI in healthcare sorted by who carries them, and the evidence on how these systems behave once they leave the pilot is in the generative AI deployment record.

Then there is what is arriving next. Most current deployments are recommendation-only, but 63% of the surveyed organisations planned to implement agentic AI systems, which execute workflows rather than suggest them, within twelve months. Only 8% described themselves as very confident in their ability to identify emerging AI risks. An approval granted to a tool that advises does not transfer to the same tool once it acts, and a governance process built on a spreadsheet reviewed quarterly will not notice the difference.

The correction is not a better meeting. It is treating go-live as the start of oversight, with a named person accountable for each deployed tool afterwards, thresholds agreed in advance for when it gets pulled, and a monitoring line in the budget that survives the launch.

In June 2026 the accreditor made governance an auditable object

Until this year an organisation could describe its AI governance in whatever terms it liked, because nobody outside was going to check. That changed on 1 June 2026, when Joint Commission launched its Responsible Use of AI in Healthcare certification, a voluntary programme organised around five standard areas: governance, effective data management, risk and bias reduction, monitoring and evaluating and validating safety performance, and transparency, education and training. It followed 2025 guidance developed after convening more than 20 coalitions and expert groups.

Two design decisions matter more than the announcement. The certification assesses the organisation and explicitly does not validate or certify individual AI products, which puts the burden where the survey data says the gap is. And applicants do not need to be Joint Commission accredited, so it is available to organisations outside the usual accreditation perimeter. The stated premise is scale: more than 80% of physicians already use AI in professional settings, which makes an organisational standard overdue rather than early.

Alongside it, the Coalition for Health AI published governance playbooks built with more than 100 healthcare organisations, addressing eight elements: organisational AI policy, organisational structure, organisational resources, responsible AI lifecycle management, risk and impact assessments, responsible data management and use, third party management, and education, training and feedback. Third party management earning its own element is the honest acknowledgement that most healthcare AI is bought rather than built, and that a governance programme is only as good as its vendor contracts.

None of this is law. It sits beside the statutory picture rather than inside it, and an organisation can be entirely current on what AI regulatory compliance requires today while still having no functioning internal gate. That is precisely the gap a voluntary certification is designed to occupy.

What this looks like in a hospital with no AI team

The governance burden falls hardest where the capacity is thinnest. The same ONC data shows predictive AI in use at 86% of multi-hospital system members against 37% of independent hospitals, and at 96% of large hospitals against 59% of small ones. The independent hospital is not exempt from oversight duties; it is simply later to the adoption curve and will meet them with a fraction of the staff.

Duke-Margolis convened six US health systems with established AI governance and reached a conclusion worth repeating to anyone about to copy an org chart from a larger institution: governance structures can vary significantly while remaining effective, and it is important for health systems to right-size their AI governance to their resources. Some of the working systems had governance teams of one to three people. Those assessments leaned more on developer-reported performance, IT integration, privacy and legal compliance, with local performance checked through qualitative pilots or post-implementation review rather than through a dedicated validation environment. The report is equally clear that the process is resource intensive across the board.

Joint Commission made the same judgement deliberately. Asked why the certification does not prescribe committee composition, its executives told Healthcare Dive that an early instinct toward a model org chart broke on contact with reality, since in a small organisation the chief operating officer is sometimes also the person fixing the boiler, and that requiring seven titles with seven distinct expertises to vet one ultrasound machine would put the standard out of reach of the clinic that needs it. The first governance question they suggest instead is a triage question: is this tool even applicable? A washing machine is not making a clinical decision. A CT scanner probably is.

So the scalable version of AI governance in healthcare is short. Know what you are running. Decide in advance what would count as it going wrong. Name the person who owns it after launch. Everything else is a matter of how many people you have to spread those three across.

The room the answer has to be written in

The recurring finding in all of this evidence is a composition problem. Ethics or bioethics roles appear on only 25% of governance committees, trailing legal and risk management. The oversight mechanism is the component the literature keeps omitting. Ownership sits unclear between IT, quality and compliance. None of those are technical failures. They are the result of clinical, legal, technical and policy expertise being asked, separately, to write a document that only makes sense written together, and of a regulator working on a different clock from the deployment.

That composition problem is the structure the EX Future Summit is built around. Its nine program tracks include Government, Health Tech and AI Ethics, alongside eight hosted meet-and-greet sessions matching universities with private partners, because the people who can answer the classification question, the validation question and the accountability question do not currently sit in the same building. The summit runs 18 to 20 November 2026 as a single continuous thirty-hour broadcast between Las Palmas and Bali, twelve hours apart, with online attendance free for verified researchers, students and the EX community. The practice layer built on top of the structure described here is responsible AI in healthcare.

FAQ

What is AI governance in healthcare?

It is the set of structures, policies and processes ensuring the responsible development, deployment and oversight of AI systems inside a healthcare organisation. Assessed as a framework, it resolves into four components: guiding principles, an assessment method, coverage of the AI lifecycle stages, and an oversight mechanism holding the authority to approve or deny a deployment. A document containing only the first of those is an ethics statement, not a governance framework.

Who should sit on a healthcare AI governance committee?

Health systems that run these committees draw membership from IT, clinical care, informatics, legal, privacy, ethics, compliance, human resources, patient engagement and finance, and most members need training on what AI means inside their own discipline rather than arriving with it. The composition gap worth checking against your own roster is ethics: it is represented on roughly a quarter of committees, well behind legal and risk management, which means the seat most likely to raise a question nobody else will is the one most likely to be empty.

Is AI governance in healthcare mandatory?

There is no single mandate. The Joint Commission certification launched in June 2026 is voluntary and open to organisations that are not Joint Commission accredited. Statutory duties reach healthcare AI through medical device and AI legislation rather than through any rule requiring an internal governance process, which is why an organisation can be technically compliant and still have no working gate. The practical pressure is arriving from payers, auditors and procurement instead, which is what the 30-day auditable explanation question is really measuring.

What is the difference between AI governance and AI ethics in healthcare?

Ethics decides what a good answer is. Governance decides who is required to answer, when, and on what record. The clearest evidence that they are not interchangeable is that essentially every published framework carries ethical principles, while fewer than one in five names an oversight mechanism. An organisation can hold entirely defensible ethical positions and still be unable to say who approved the model currently triaging its inpatients.

How do you start an AI governance programme from nothing?

Build the inventory first, including the AI already embedded in vendor products you did not procure as AI. Nothing downstream is possible without it, and it is the artifact most organisations lack. Then attach a written approval step to that inventory, since a body without a documented gate does not change any outcome. Then name a post-deployment owner per tool. A published framework can supply the rest of the structure, but starting with the framework and working toward the inventory is the order in which these programmes stall.