Written by Professor Ignacio Cofone
A few years ago, UnitedHealth’s subsidiary Optum acquired a company called naviHealth and deployed its AI tool, nH Predict, to determine how long elderly patients on Medicare Advantage plans should receive post-acute care coverage. The tool predicted recovery timelines and generated coverage recommendations. The company described it as a “care-support tool.” But a STAT investigation found that managers set a target for clinical staff to keep patients’ rehab stays within 1% of the algorithm’s predictions. Patients were cut off from coverage for skilled nursing and rehabilitation that their own doctors had deemed necessary. A US Senate investigation later found that UnitedHealth’s denial rate for post-acute care more than doubled after the system went live.
For years, the company fought to limit what it had to disclose about how nH Predict worked. In March 2026, a federal court in Minnesota ordered UnitedHealth to produce internal documents: policies, training materials, employee performance records, and the workings of its internal AI review board.
This case isn’t unusual. When AI systems make consequential decisions about people, the organisations deploying them routinely resist scrutiny. They say the system is too complex, or proprietary, or that disclosure would create risks. Sometimes those claims are true. Others, they are a way of avoiding accountability for design choices that benefit the institution at the expense of the people being decided about. Distinguishing one case from the other is important.
Three kinds of opacity, only one of them technical
Opacity in AI systems comes in different forms, and two of them have little to do with system complexity.
The first kind is genuine technical complexity. Some AI systems, particularly deep learning models and large language models, learn patterns across thousands of data points in ways that resist straightforward causal explanation. Even the engineers who built them often cannot say exactly why a specific input produced a specific output. This is a real limitation. But “hard to explain fully” is different from “impossible to scrutinise at all.” Engineers can still describe what data went in, what the error rates look like, and how the system was tested. Partial explanation isn’t no explanation.
The second kind is a literacy gap. Many AI systems could be understood by someone with training in statistics or computer science, but the people affected by them, and the lawyers representing those people, lack that training. The knowledge exists; it’s just not distributed to the people who need it. This matters because it creates an imbalance: the vendor understands the system, the person being scored or sorted doesn’t, and whoever is overseeing the decision risks deferring to technical authority they cannot evaluate.
The third kind is deliberate secrecy, and it’s the most consequential. This is when information about the system is withheld by choice: a vendor invokes trade secrets, an agency cites security concerns, a developer claims disclosure would let people game the system. These are policy decisions, not technical constraints. They should be evaluated as such.
Each kind calls for a different response. Technical complexity calls for the partial disclosures that are still possible. A literacy gap calls for expert interpretation. Deliberate secrecy calls for someone to ask whether the justification for withholding actually holds up.
The usual defences are weaker than they look
When you push on deliberate secrecy, two arguments come up repeatedly: gaming and trade secrets.
The gaming argument goes like this: if people know how the system works, they will manipulate their behaviour to get a favourable outcome they don’t deserve. But this requires several things to be true at once. The criteria being disclosed must be ones that people can actually change. The cost of changing them must be low. And the change mustn’t genuinely improve the person’s standing. These conditions rarely hold together. If a credit model relies on years of payment history, improving your payment history makes you a better credit risk. When meeting the criteria requires genuine improvement, what the developer calls “gaming” is really just compliance. The gaming argument is only as strong as the specific disclosure being requested, and in most cases, meaningful information can be shared without creating a real gaming risk.
Trade secrecy claims follow a similar pattern. Vendors argue that their algorithms represent valuable intellectual property. In competitive markets for consumer products, that concern is often legitimate. But many AI systems that affect people’s lives are deployed through government contracts paid with public funds, or embedded in regulated decisions about credit, employment, or insurance. The social value of these systems depends on accuracy and fairness. And when vendors resist disclosing how their systems were validated or how errors are distributed, they prevent detection of flaws that harm the public while providing no corresponding benefit.
The deeper problem is that the people choosing opacity are often the people who benefit from it. Decision-makers adopt convenient proxies for the things they are supposed to measure. These substitutions shape what institutions end up valuing in practice, and the social costs of getting them wrong fall on affected populations, not on the institution. Secrecy makes this dynamic invisible.
Transparency isn’t binary
The most common mistake in this debate is treating transparency as all-or-nothing: either you reveal the source code or you reveal nothing.
In practice, useful transparency is graduated. Different kinds of information serve different purposes, and they can be disclosed to different audiences under different conditions. The question is always: what can be disclosed, and to whom, without compromising the things that legitimately need protection?
Start with outcome distributions: the basic breakdown of how many people were approved or denied, hired or rejected, across relevant groups. This is low-risk information. It reveals how the system behaves in practice without exposing anything confidential, and it’s extremely difficult to game. Think of outcome distributions as diagnostic: they tell you whether further investigation is needed. In the UnitedHealth case, this is the kind of data that would have raised alarms years earlier. A US Senate subcommittee found that post-acute care denial rates more than doubled after nH Predict was deployed. That pattern was sitting in the company’s own records, invisible to patients and regulators because nobody required it to be disclosed.
Then look at error profiles: how the system’s mistakes are distributed. Of the nH Predict denials that patients actually appealed, according to the lawsuit, roughly nine in ten were reversed. But fewer than one in five hundred patients appealed at all. The company knew both of these facts. Together they reveal a system that was producing errors at scale while facing almost no correction, because the people bearing the cost of those errors lacked the information and resources to challenge them. Neither the reversal rate nor the appeal rate required disclosing any proprietary code or model architecture. They required disclosing how the system was performing in practice.
For more sensitive information, phased disclosure and confidentiality protections already exist. Independent auditors can review code under non-disclosure agreements. Regulators can receive detailed validation studies. The public can receive aggregated performance reports. When the Minnesota court ordered disclosure in March 2026, it didn’t ask UnitedHealth to publish its source code. It ordered production of specific document categories to specific parties under specific conditions. This is what functional transparency looks like: not a binary switch between total secrecy and full exposure, but a set of calibrated disclosures matched to the oversight need.
What this means in practice
The nH Predict litigation offers a broader lesson for all high-stakes AI decision-making. By the time a court ordered disclosure, years had passed, patients had lost coverage they needed, and the information that would have flagged the problem had been sitting inside the company the entire time. The questions that litigation eventually forced are questions that should have been asked at procurement, at deployment, and at every review thereafter.
People who work at a company or government entity that procures AI systems can ask those questions before a court makes them. Start with the basics: what’s the system optimising for, and is that the same thing you care about? A system that predicts healthcare costs is optimising for something different from a system that predicts healthcare needs. A system that minimises false positives is making a different trade-off from one that minimises false negatives. These choices determine who benefits from the system and who bears the cost of its errors. They aren’t technical details. They are design decisions with distributional consequences. They could be specified in a procurement contract and not left to the vendor to decide privately.
Then ask for performance data. What are the error rates, and how are they distributed across the populations the system affects? How was the training data assembled, and what populations does it represent well or poorly? What validation was performed before deployment, and what ongoing monitoring is in place? None of these questions require understanding the mathematics behind the model. They require treating an AI system the way you would treat any other high-stakes institutional process: with oversight, documentation, and accountability. If a company doesn’t have any performance data to share, that’s a red flag.
If the vendor’s answer to any of these questions is “we can’t tell you,” the follow-up is: “why not?” What specifically would be compromised by this specific disclosure? “It’s proprietary” isn’t an answer to a question about error rates. “It’s too complex” isn’t an answer to a question about outcome distributions. The task isn’t to reverse-engineer a system but to ensure that the people deploying it can verify that it works as intended, and that the people affected by it have some basis for knowing whether it treated them fairly.
The black box framing flattens the reality on the ground. It suggests that opacity is an inevitable feature of complex technology. The nH Predict case tells a different story. The system wasn’t too complex to evaluate. The information needed to evaluate it wasn’t too sensitive to share. The company resisted disclosure because disclosure would have revealed what the system was actually doing. When the court ordered it, the tools it used were ordinary procedural mechanisms that courts and regulators use every day.
Algorithmic opacity isn’t a fact about technology. It’s a governance problem, and governance problems have governance solutions.
Ignacio Cofone is Professor of Law and Regulation of AI at the University of Oxford, Institute for Ethics in AI and Faculty of Law. This essay applies part of the argument in Ignacio Cofone and Katherine J. Strandburg, "Algorithmic Opacity as a Principal-Agent Problem", 15 NYU Journal of Intellectual Property Law 1 (2026).
Suggested citation: Professor Ignacio Cofone, ‘The Black Box is Usually a Choice’, (date of publication - 4 June 2026) AI Ethics at Oxford Blog; https://www.oxford-aiethics.ox.ac.uk/blog/black-box-is-usually-a-choice