On August 18, FDA asked for public feedback on how to evaluate medical devices that use generative AI. The announcement raised a question that reaches beyond medicine: what does a successful test establish when the application someone actually uses can change?
A foundation model can perform well on a test of medical knowledge. A company can then build a product around it, supply patient information, select reference materials, and decide how its answers reach a clinician. The score belongs to the model under the conditions of that test. The patient encounters the product.
That difference is the starting point for this article. Evidence of a model’s capabilities does not, by itself, establish that a particular medical application is safe and effective.
FDA has not adopted a new approach. Its discussion paper is neither guidance nor a statement of proposed regulatory expectations. It also leaves questions of legal authority unresolved. Feedback is requested by October 19, 2026.
The Application Is What the Patient Encounters
The paper explores assessing the deployed device rather than an isolated foundation model. Its concerns include convincing false outputs, limited visibility into third-party models, and changes to prompts, retrieved information, guardrails, and interfaces.
Consider a hypothetical application that summarizes a patient’s record and suggests questions for a clinician. The underlying model supplies the ability to generate text. Other software selects which parts of the record it receives. Instructions tell it what to emphasize. The screen determines whether a warning is prominent or buried beneath the summary.
An omitted medication can originate in the selection of records before the model ever sees them. A misleading recommendation can arise from an instruction that encourages certainty. A correct warning can fail to help if the clinician does not see it in time.
These are different failures. Testing medical knowledge alone would not identify all of them. A useful evaluation would have to follow the information through the application and examine what reaches the person making the clinical decision.
The Consequences Determine What Evidence Matters
FDA discusses a possible framework linking the degree and independence of the function’s activity to the severity of harm from relying on an incorrect output. It also explores benchmarking followed by clinical confirmation, proportionate to risk.
The practical distinction is easy to see. Software that helps someone prepare questions for an appointment creates different stakes from software that directs a change in medication. Even identical words can carry different consequences depending on who receives them and what happens next.
In our hypothetical example, a fluent summary is only one measure of performance. Does it preserve facts that change the treatment decision? Does it distinguish a documented diagnosis from a suspicion? Does it communicate missing information clearly enough that the clinician knows to check?
Those questions connect the test to the purpose of the product. They also explain why an impressive average score can leave an important safety question unanswered. A rare omission matters when the omitted fact is the one that should stop a treatment.
A Reviewer Needs the Ability to Recognize an Error
The paper asks how users’ clinical expertise affects risk and how reduced opportunities for human review should affect oversight of agentic devices.
For our hypothetical application, requiring a clinician to approve the summary would establish that approval occurred. It would not establish what the clinician checked.
Meaningful review would depend on practical details. Can the clinician open the underlying record? Does the screen distinguish the application’s interpretation from the source material? Is there enough time to examine a surprising recommendation? Does the reviewer have the relevant expertise?
These are questions we draw from the case, rather than requirements announced by FDA. They make human oversight something that can be examined. A signature alone cannot answer them.
Earlier Evidence Needs a Connection to Later Changes
FDA discusses repeat benchmarking, sampled clinician review, and performance monitoring after deployment. It asks how manufacturers can detect and respond to changes initiated by third-party model developers without diffusing manufacturer accountability.
Return to the hypothetical summary tool. Suppose its developer changes the instructions to produce shorter answers. The new version is faster to read. It also needs evaluation for whether compression removes clinically important qualifications.
Or suppose the model supplier changes its service. The application developer needs to know whether its earlier results still describe the product patients and clinicians now encounter. A supplier’s general assurance of improvement does not answer that application-specific question.
The oversight lesson is therefore more demanding than keeping a record of updates. Someone must identify which changes affect the basis for relying on the application, decide what to examine, and act on the result.
