Robert Williams: When the Computer Got the Last Word
Detroit police ran a grainy still through facial recognition, got a wrong match to an old license photo, and arrested an innocent man in his driveway. The first such wrongful arrest known to be reported in the US.
A facial-recognition match was treated as an identification. No rule required corroboration before an arrest, and the officer trusted the system over the man in front of him. The control absent throughout: a match is a lead, never the sole basis for action.
In January 2020, Detroit police arrested Robert Williams in his driveway, in front of his wife and two young daughters, and held him for about thirty hours. He had done nothing. A shoplifting from a store more than a year earlier had been caught on grainy video, and an investigator ran a still from it through facial recognition software. The system returned Williams’ old driver’s license photo as a possible match. That match, with little else behind it, became the basis for his arrest.
When he was shown the surveillance image, Williams held it up next to his face and said it plainly. This is not me. He reportedly added that he hoped they did not think all Black men looked alike. By his account, an officer answered, the computer says it is you. The charges were later dropped. It is the first wrongful arrest from facial recognition known to be reported in the United States, and one of three such cases in Detroit, all of them involving Black people.
Where it failed
This is a probabilistic failure, the kind the older systems in this archive did not have. Facial recognition does not return a fact. It returns a likelihood, a best guess ranked against a database. And its accuracy is not even. The technology is least reliable on exactly the people it misidentified here, women and people of color, a gap that government testing has measured rather than assumed.
The model was only half of it, and the smaller half. The larger failure was in people and process. A probabilistic guess was treated as an identification. No rule required independent evidence before acting on a match, so a lead that should have opened an investigation closed one instead. And when the person standing in front of the officer said the match was wrong, the system was believed over the human.
A facial recognition match is a lead to check. Detroit police treated it as proof and made an arrest.
How it could have been caught
The mitigations here are not technical and they are not expensive. Write the rule that a facial recognition match is a lead and never, on its own, grounds for an arrest. Require corroborating evidence before acting on one. Tell officers plainly that the system is least accurate on the people most likely to be harmed by a mistake, so a match is met with suspicion rather than trust. Detroit eventually adopted controls close to these as part of a 2024 settlement. They read as a precise description of what would have prevented the harm. They simply arrived after it.
What it means for AI
Most AI now in production is probabilistic in just this way. It produces a ranked guess, often a good one, and fails quietly on the cases at the edges. The Williams arrest shows what happens when that guess is handed the authority of a fact, and when the human in the loop defers to the machine instead of checking it.
The agentic turn removes the one safeguard that could have saved him. An agent that acts on a match, that flags, blocks, escalates, or denies without a person able to say wait, that is not him, takes the last check and deletes it. The match has to stay a lead, with corroboration required before anyone acts on it, and a human has to keep the authority to overrule the system. That is the whole lesson of this case: what happens the moment the computer gets the last word.
← Back to the Atlas