On March 6, 2025, NIST finalized guidance for evaluating a privacy promise that can sound more complete than it is. A vendor says its software uses differential privacy. The customer still needs to establish what that promise protects.
NIST’s announcement identifies the problem: organizations need a more consistent way to understand vendors’ claims. The publication helps them examine the protection offered and the tradeoffs involved.
The distinction can matter before anyone compares algorithms. A promise about one record and a promise about a person’s combined contribution answer different questions. Which one did the customer buy?
The Guarantee Needs a Defined Subject
Differential privacy mathematically bounds how much an analysis’s output probabilities can change between datasets differing in a specified contribution. NIST’s SP 800-226 explains how privacy parameters, the protected unit, algorithms, and deployment assumptions determine the guarantee.
Consider a hypothetical healthcare organization evaluating a service for analyzing encounter data. Some patients have one encounter. Others have dozens. The vendor offers a guarantee covering a change to one encounter record.
The organization wants to understand protection for a patient’s history. The vendor has described protection for a different unit. Before deciding whether the service meets its needs, the organization must resolve that difference.
NIST distinguishes event-level protection from user-level protection and recommends using a unit at least as large as an individual user when possible. It warns that event-level privacy can produce surprisingly weak guarantees because it protects individual events rather than people, and calls user-level privacy the best default. A parameter alone cannot establish what is protected.
In the hypothetical, asking for a privacy score would therefore be an incomplete first step. The organization needs to identify whose information is involved, how many contributions that person can make, and which contributions the guarantee covers.
A Number Needs Its Conditions
The familiar parameter epsilon sets an upper bound on privacy loss rather than measuring it; smaller values generally mean stronger protection under comparable definitions. Repeated analyses require accounting for cumulative privacy loss.
Suppose the healthcare organization plans monthly reports, occasional exploratory queries, and an annual release for research partners. A guarantee presented for one report would leave the organization with another question: what covers the collection of outputs it intends to produce?
The purchasing decision should examine that intended use. The organization could ask the vendor to identify the covered analyses, the overall accounting approach, and what happens when the permitted budget is exhausted. Those are practical questions derived from the need to evaluate a defined guarantee.
They also make comparison more useful. Two vendors presenting the same headline number might be describing different protected contributions or release plans. The organization should establish a common basis before deciding that the offers are equivalent.
Privacy Protection Changes the Analysis
NIST explains that differential privacy generally uses calibrated randomness and that choices about noise affect usefulness. Its announcement describes interactive tools, flow charts, and sample code intended to help practitioners examine that balance. Incorrect application can compromise protection or make results less useful.
Return to the hypothetical service. The organization wants to compare outcomes across patient groups. Its evaluation should include whether the resulting estimates remain useful for that purpose, including for groups with fewer observations.
A dataset could be suitable for estimating a broad trend and unsuitable for a particular subgroup comparison. The organization would need evidence about the actual analysis it intends to perform. An overall accuracy measure would not necessarily answer that question.
NIST also addresses bias, noting that disclosure control techniques can create or magnify systemic, human, or statistical bias in results. A privacy guarantee does not establish fair outcomes. For this customer, privacy review and performance review would need to inform the same purchasing decision without treating one as proof of the other.
The Released Result and the Source Records Need Different Protections
The publication distinguishes protection for intentional releases from security for raw data. Differential privacy does not replace access controls or secure handling of source records.
In the hypothetical, a vendor might receive identifiable encounter data and publish protected statistics. The organization should examine both stages. Who receives the source records? Who can access them? What information can leave through logs, support processes, or other routes?
Those questions concern the service the organization will actually use. A guarantee attached to published statistics would not, by itself, answer questions about a separate disclosure of the source file; NIST notes that a breach of raw records nullifies the guarantee for those records.
The same care applies to implementation. The customer needs evidence that the deployed service supplies the guarantee described in the contract and technical materials. A mathematical claim about an algorithm and a statement about operating software are different evidence.
