Uber's Self-Driving Car and the Human Set Up to Fail
An Uber test car saw a woman crossing the road six seconds before it hit her, could not decide what she was, and had its emergency braking switched off. The backup was a human watching her phone.
Uber disabled the car's automatic emergency braking and made an inattentive human the only thing between the system and a person. The control absent: a working last line of defense, and an honest assumption that a lone human watching a near-perfect system will not stay alert.
On the night of March 18, 2018, in Tempe, Arizona, an Uber test vehicle operating in autonomous mode struck and killed Elaine Herzberg as she walked her bicycle across a four-lane road. It was the first pedestrian death involving a self-driving car.
The investigation found a chain of removed protections. The car’s sensors detected Herzberg about 5.6 seconds before impact. But the system could not settle on what she was, cycling between an unknown object, a vehicle, and a bicycle, and so it never predicted that she was on a path into the car. Worse, Uber had disabled the Volvo’s own factory automatic emergency braking, to keep the ride smooth, and had not built an equivalent into its own software to brake hard in an emergency. The entire last line of defense had been delegated to the safety driver behind the wheel.
The safety driver was streaming a television show on her phone. She looked up less than a second before impact.
Where it failed
This one is mixed. The perception failure was probabilistic, a system that could not classify an unusual case in time. But the decision to remove the car’s emergency braking was deterministic, a choice that guaranteed there was no automatic backstop. And the decision to rely on a single human to catch the machine’s rare, sudden failures was the deepest break of all.
That last one is worth naming, because it is the most common and the most seductive. The plan was that a human would supervise the automation and step in when it failed. But a person asked to watch a system that almost never needs them will stop watching. This is well understood, it has a name, automation complacency, and it is predictable. Uber’s process did not account for it, did not monitor whether the driver was engaged, and built a safety case on a human doing something humans reliably cannot do.
“A human reviews it” is not a control if the human has nothing to do until the half-second it all goes wrong.
How it could have been caught
Keep the automatic emergency braking on, because the whole reason it exists is the case the higher-level system gets wrong. If a human is genuinely the backstop, design for the human you have, not the one you wish for: monitor attention, keep them engaged, limit how long they sit passive, and do not pretend that watching is the same as acting. And test against the hard cases, the pedestrian who is not at a crosswalk, the object that does not fit a clean category, because those are the ones that kill.
What it means for AI
When someone answers an AI risk with do not worry, a human reviews the output, this is the accident to put on the table. That answer sounds like a control. It usually is not. A human placed at the end of an automated process that is right almost every time will rubber-stamp it, because vigilance against a rare event is not something people sustain. The human in the loop becomes a human-shaped formality.
Honesty about what the human can actually do is the whole control. If a person is the safeguard, give them a real job, the time and information to do it, and a workload that keeps them engaged. Keep the automatic protections that do not depend on attention. And treat any safety story that rests on a bored human catching a rare failure for what it is: not a control, but a way to assign blame after the fact.
← Back to the Atlas