A hospital’s review committee opens the documentation for a predictive tool it is considering purchasing. The tool estimates which patients are likely to deteriorate. Its intended use is described. Internal test results are supplied. Information about external validation is marked unknown.
This is a hypothetical, but the missing entry is a situation federal guidance expressly anticipates. ONC’s companion guide requires a Health IT Module to indicate when source attribute information is not available for review. ONC expects developers to exercise due diligence before determining that information is unknown, and a maintenance-of-certification requirement obliges them to review and update source attribute information on an ongoing basis.
The disclosure has done something useful: it has exposed a gap. The hospital now has to decide whether that gap must be resolved before the prediction influences care.
An Empty Answer Is Not a Failed Test
The ONC companion guide concerns the DSI certification criterion in section 170.315(b)(11). Predictive decision support interventions use models that derive relationships from training data to support decisions. The developer obligations discussed here concern interventions supplied as part of its certified health IT module.
A model card is a structured description of a model’s purpose, development, performance, and limitations. ONC specifies source-information categories and access capabilities; it does not require one particular model-card layout.
Three findings need different treatment. A test showing unacceptable performance is evidence of a problem. No available test result leaves performance unresolved. A successful test confined to a narrower population supports only the conclusions the study permits.
For the hypothetical hospital, unknown external validation does not prove the deterioration model is unsafe. It also does not establish that performance observed in development will carry over to the hospital’s patients.
The following steps describe a practical organizational response. They are not a hospital testing protocol imposed by ONC. The developer’s certification obligations and the hospital’s deployment judgment remain distinct.
Name the Missing Evidence Before Requesting More Documentation
The committee should turn the unknown entry into a specific question. Has anyone tested this model using data from a source different from its development data? Which model version was tested? Were the patients, outcome, prediction period, and operating threshold comparable to the proposed use?
A general request for validation can produce a presentation that leaves the original question unanswered. A useful request specifies the evidence sought: the testing population and setting, methods, results, limitations, and relevance to the proposed installation.
ONC’s DSI resource guide explains the source-information categories and how users can record and revise them. It provides a useful starting point for distinguishing missing development information from missing external or local evaluation.
The hospital should identify other gaps separately. Unknown training-population characteristics create uncertainty about representation. Missing local results concern performance in this setting. An absent monitoring arrangement concerns what will happen after deployment. One reassuring accuracy figure cannot resolve all three.
Find Out Whether the Evidence Is Absent or Inaccessible
The hospital should ask the supplier why the entry is unknown. Was external testing never performed? Does another company hold the results? Does the available study concern an earlier version? Are records unavailable for an older model?
These answers determine the next action. An existing report needs to be obtained and assessed. A report about another version requires an explanation of its relevance. An evaluation never performed requires a decision about whether additional testing is necessary.
The hospital should also identify who supplies the intervention. ONC distinguishes tools supplied by the certified developer from tools independently obtained by its customer. Merely displaying another company’s output through an EHR does not necessarily make the EHR developer responsible for its source information.
Contracts can make the next steps concrete: identify who supplies supporting reports, gives notice of material changes, assists investigations, and provides information needed for agreed monitoring. A promise to provide documentation is useful only if it reaches the evidence the hospital needs.
Decide Whether the Uncertainty Matters to This Use
NIST’s AI Risk Management Framework connects evaluation to the circumstances of deployment. It calls for sufficient contextual knowledge to inform whether to proceed. The framework and its companion Playbook are voluntary resources, not additional ONC certification requirements.
For the deterioration model, the committee should identify what action the score will prompt. Will it request an additional assessment, prioritize a patient for a scarce service, or support reducing observation? What happens if the prediction is wrong?
The same evidence gap has different consequences in those circumstances. A pilot that leaves care unchanged presents a different question from using the score to withhold an existing precaution.
The committee also needs to identify differences in patients and inputs. Laboratory timing, observation frequency, coding, and missing values affect what reaches the model. A study from elsewhere cannot establish equivalence simply because both institutions use electronic records.
Choose a Response That Actually Addresses the Gap
The NIST Playbook’s Measure 2.3 recommends evaluation under conditions similar to deployment and documentation of differences between the testing and operating environments. Its suggested measures include false-positive and false-negative rates and prediction latency.
For this hospital, additional evidence could come from a relevant existing study, a retrospective local evaluation, or a prospective evaluation that runs the model without exposing its predictions to clinicians making care decisions. Each answers different questions. Historical data can test predictions against recorded outcomes; it does not establish how staff will respond to alerts in practice.
Before testing, the hospital should define what would support proceeding. That includes the outcome being predicted, acceptable missed cases and false alerts, relevant patient groups, and whether warnings arrive in time to act. A small study with few relevant events should not be treated as a conclusive demonstration.
A narrower deployment is another option when evidence supports it. The organization can limit eligible populations, locations, or uses and prevent unsupported applications. If the missing evidence is essential and cannot be obtained, postponing deployment is a substantive decision.
Adding human review is useful only when the reviewer can detect or compensate for the uncertainty. A clinician who cannot recognize a model’s missed cases does not cure unknown sensitivity merely by being present in the workflow.
