Optum and the Algorithm That Used Cost as a Proxy for Need

In 2019, a team led by Ziad Obermeyer published a study in Science about an algorithm that hospitals and insurers used to decide which patients needed extra help. The tool, sold by Optum, scored patients to flag the ones who should be enrolled in high-risk care management. Algorithms of its kind shape the care of roughly 200 million Americans a year. And it was quietly steering care away from Black patients who were every bit as sick as the white patients it favored.

The algorithm did not ask who was sickest. It asked who would cost the most, and used that as a stand-in for who needed the most care. On paper this is reasonable. Sicker people usually cost more. But the link between cost and need is not the same for everyone. Black patients, for a tangle of reasons including worse access and care historically withheld, had generated lower medical costs than white patients at the same level of illness. About 1,800 dollars less a year, the study found, for people with the same chronic conditions.

So the model did its job well. It predicted cost accurately. And because it predicted cost, it concluded that Black patients, who cost less, must be healthier, and ranked them lower for the extra care they in fact needed more.

Where it failed

This is a probabilistic system, and the temptation is to hunt for bias in the data or the math. But the model was not broken. It predicted its target, cost, accurately. The failure was the target itself. Someone chose cost as a proxy for need, and nobody checked whether the proxy held across the people the model would sort.

This is a validity failure, not an accuracy one, and it is the kind the older cases in this archive did not have. A bridge either holds or it does not. A model can pass every accuracy test you give it and still be measuring the wrong thing. The break lived in the design, the decision about what to predict, and in the process that signed the model off without ever asking the validity question.

Predicting cost accurately is not the same as finding the patients who need care the most.

How it could have been caught

The fix turned out to be straightforward, and the researchers demonstrated it. Predict health, not cost. When the model was retrained to target active chronic conditions rather than dollars, the bias fell sharply. Before that, the simplest control would have surfaced it: measure the outcome across groups. Ask whether Black and white patients at the same level of illness received the same scores. That single check, run once before deployment and again in production, exposes the whole problem.

What it means for AI

Optum is the case to remember for any model that sorts people, because its lesson is the uncomfortable one. The model can be accurate, well tested, and free of any obvious bug, and still do harm, because the harm is in what it was told to optimize. Most AI systems are trained against a proxy. Engagement stands in for value, clicks for relevance, cost for need. The proxy is never quite the real thing, and the difference between the two is where the harm ends up.

The question to ask of any such system is not just how accurate it is, but accurate at what, and whether that target means the same thing for everyone it touches. Check that the proxy actually tracks the goal, and measure the results across the groups the model sorts. Do that work before the model is deciding who gets care, rather than leaving it for a study to uncover years later.