Most research sites have already made their AI decision. There is an authorized vendor, a signed agreement and a pilot running somewhere. The question that rarely gets a full answer is the harder one: who is accountable when the tool is wrong?
In most industries, a wrong answer from an AI tool costs time. In regulated work, it can cost a lot more. I work in clinical research billing compliance, where a coverage analysis decides what Medicare can be billed for and what the sponsor or the site has to absorb. Get that call wrong under NCD 310.1 and the result is not a typo. It is a potential False Claims Act problem, and FCA settlements and judgments topped $2.68 billion in fiscal year 2023 alone (DOJ, 2024).
That is the environment AI is walking into. And the tools are good. A current model can read a long protocol, pull every procedure from the schedule of events and suggest a billing designation in minutes. The risk is not that the output looks bad. The risk is that it looks RIGHT.
Governance is how you deal with “looks right.” Three things matter most.
1. Know what the tool is deciding. Before an AI system touches a regulated workflow, document what it does, what data it uses and where a human takes over. The NIST AI Risk Management Framework calls this mapping. ISO/IEC 42001 builds a management system around it. The label matters less than the habit: no undocumented AI in a regulated decision.
2. Keep a named human in the loop, and mean it. Human-in-the-loop is easy to write into a policy and easy to hollow out in practice. If an analyst approves 300 AI suggestions a day without reading them, that is not oversight. Automation bias is well documented (Parasuraman & Riley, 1997), and it gets worse as the tools get better. The reviewer needs the time, the authority, and the expertise to say no.
3. Make every output auditable. If you cannot show where a number came from (the source document, the rule applied, the version in force at the time), you cannot defend it to an auditor. This is a design requirement, not a nice-to-have. It is also why Elessar names the file and release date behind every answer it gives.
There is a vendor angle too. Most sites are not building AI. They are buying it, or using an enterprise license they already have. That means part of your compliance posture now lives in someone else’s documentation. Ask how the tool was validated, how it handles PHI, and what changes when the underlying model is updated.
None of this is an argument against AI. I use it every day. It is an argument for being honest about where responsibility sits. A model can apply a rule consistently. It cannot be answerable for the result. As Kirchschlaeger (2021) argues, responsibility remains human, and it grows as systems become more automated.
Fast and confident is not the same as defensible.
If your team is working out how AI fits into an existing compliance program, that is the work MediCove’s AI Governance Support is built for. Get in touch.
References
- Kirchschlaeger, P. G. (2021). Digital transformation and ethics: Ethical considerations on the robotization and automation of society and the economy and the use of artificial intelligence. Nomos.
- National Institute of Standards and Technology. (2023). AI Risk Management Framework (AI RMF 1.0).
- International Organization for Standardization. (2023). ISO/IEC 42001:2023 Artificial intelligence: Management system.
- Parasuraman, R., & Riley, V. (1997). Humans and automation: Use, misuse, disuse, abuse. Human Factors, 39(2), 230–253.
- U.S. Department of Justice. (2024, February 22). False Claims Act settlements and judgments exceed $2.68 billion in fiscal year 2023.