Healthcare Software with AI in It: Designing for the Audit You Will Eventually Face
Clinical software is not judged on average performance. It is judged one case at a time, in retrospect, by someone asking why the system did what it did. That reframes every architectural decision, starting with what you log.
Consumer software is evaluated in aggregate. If a recommendation engine is right most of the time, the people it fails are absorbed into a metric and the product is considered good. Clinical software does not work that way. It is evaluated one case at a time, usually after something went wrong, by a person reconstructing what the system showed and what the clinician did about it. Every architectural decision should be made with that reconstruction in mind, because it is the only evaluation that ultimately matters.
This is the practical reason AI in healthcare is harder than the model quality suggests. The difficulty is rarely getting a useful output. It is being able to say, months later, exactly which inputs produced it, which version of which model was running, what the clinician saw on screen at the time, and what they did next.
Provenance is a feature, not logging
Most systems log what happened. Clinical systems need to reproduce what was shown. Those are different requirements, and the gap between them is where retrospective reviews fail. If a summary was generated from a patient record that has since been amended, replaying the generation against the current record produces a different summary and proves nothing about the decision under review.
- The exact input snapshot, stored immutably at the moment of generation rather than reconstructed from a mutable record afterwards.
- Model and prompt version, pinned and retained, so an output can be attributed to the system that actually produced it rather than the system running today.
- The rendered output as presented, including any hedging, confidence indicator or omission the interface applied on top of the raw result.
- The human action that followed (accepted, edited, overridden, ignored), because that is the difference between a suggestion and a decision.
Assistive by construction, not by disclaimer
The line between an assistive tool and a clinical decision-making system is architectural, not legal wording. If a clinician must take an affirmative action before anything reaches a patient record, and the system cannot write to that record on its own, the boundary holds under scrutiny. If safety depends on a person reading a disclaimer, it does not.
Where AI is genuinely useful in clinical systems today
The applications that hold up are the ones that reduce administrative load rather than make clinical judgements. Drafting documentation from a consultation for a clinician to correct and sign. Summarising a long record into a briefing before an appointment, with every claim linked back to the source entry. Coding assistance where the suggestion is a shortlist and the selection is human. Triaging an inbox so urgent items surface first, while nothing is closed automatically.
What these share is that the AI output is an input to a person who was already going to do the task, and who retains the ability to do it without the system. That property is what makes the failure mode recoverable: a poor draft costs editing time, not a clinical error.
Data handling is an architecture decision, made early
Where patient data may travel, which processors may see it, what is retained by whom and for how long: these are decisions that determine which models you may use and which deployment topologies are available. Teams that defer them until a compliance review discover that the model they built against is not one they are permitted to send this data to, and the rework is structural rather than cosmetic.
The sequence that avoids this is unglamorous: establish the data boundary first, choose the model options that fit inside it, then build. It constrains the design space earlier than engineers like, and it is considerably cheaper than discovering the constraint after integration.
Legacy is the normal starting condition
Very little healthcare software is built on a clean foundation. There is an existing record system, an interoperability layer that mostly works, integrations built by people who have left, and workflows encoded in habit rather than software. Modernisation here is incremental by necessity, because the system cannot stop while it is replaced. The approach that survives contact with a clinical environment is to map dependencies honestly, migrate the highest-value component first, and run the old path in parallel until the new one has earned the trust to take over.
In clinical software, the question is never whether the model is good. It is whether you can explain a single case, a year later, to someone who is not inclined to be generous.
For clinical and product leaders evaluating a build
Ask what happens to a patient record when the model is wrong, and ask to see the trace for a single generated output end to end. A team that has built clinical software will show you an immutable input snapshot, a pinned model version and the recorded human action. A team that has not will show you an accuracy figure, which answers a question nobody in a review will ask.