STPA for AI: the accidents that happen when nothing breaks

Most AI risk work looks for broken parts. The model drifted, the data was biased, a vendor had an outage. Fix the part, close the finding. This is FMEA-style analysis, inherited from hardware reliability, and it works when failures actually come from broken parts.

Nancy Leveson, a safety engineer at MIT, spent her career on accidents where that assumption fails. Her framework is called STAMP, and its analysis method is STPA. The premise: safety is a control problem. Accidents happen when the controls that should constrain a system are missing, weak, or eroded, and in the worst accidents every component performs exactly as designed.

Two entries in the Failure Atlas show this. The Therac-25 bug had existed harmlessly in the previous model, and patients died only after the hardware interlock that had been catching it was removed. MCAS did what it was built to do, and the decisions that made it lethal were made in design reviews years before either crash. The components worked. People died anyway.

AI systems fail this way constantly. A model optimizes the wrong proxy while its accuracy dashboard stays green. An agent takes an action inside its permissions at exactly the wrong moment. A reviewer approves everything because the queue is long and the agent is usually right.

The four steps, applied

STPA has four steps. Here they are on a system being deployed across the market right now: an AI screening agent in recruiting. It reads applications, ranks candidates, rejects the bottom of the pool automatically, and drafts outreach to the top.

Step one: define the losses and hazards. Losses, not model metrics. Qualified candidates get filtered out systematically, the company misses hires, or it gets sued. The hazard is the system state that makes the loss possible, for example the agent rejecting on criteria nobody currently endorses.

Step two: draw the control structure. Who commands whom, and what feedback flows back up. Board, executives, an AI governance committee if one exists, talent acquisition operations, the vendor, the agent, the candidate pipeline. At each link, ask what control actions flow down and what feedback comes back. Most deployments look bad right here. Recruiters see conversion metrics but nobody reviews the rejected pool, and the governance committee gets a quarterly usage summary that says nothing about what the agent declined.

Step three: identify unsafe control actions. Leveson names four types. For this agent’s main control action, rejecting a candidate:

• Rejecting when it should not: auto-reject fires on resume patterns that correlate with a protected group.
• Failing to reject when it should: unqualified candidates flood the interviews, and someone quietly tightens thresholds without review.
• Wrong timing or order: rejections go out before the human review that policy requires.
• Applied too long: the agent keeps screening against criteria for a role that changed three weeks ago.

Step four: trace the loss scenarios. For each unsafe action, work out how it happens with no component failing. The threshold was reasonable when it was set, then the applicant mix shifted. The vendor updated the model and the release note went to an unmonitored inbox. The recruiter approval step still exists on paper, but volume turned it into a rubber stamp months ago.

Migration to accidents

That last scenario is Leveson’s most important idea. Systems migrate toward accidents. Under workload pressure the constraints erode: reviews speed up, audits shrink, the exception process becomes the normal path. Every step of the erosion is locally reasonable. The metrics stay green the whole way down because they were built to measure the happy path, and when the incident finally lands, everyone calls it a surprise.

Standard AI risk registers miss this because they assess the model at a point in time. STPA assesses the control structure over time, and asks who would notice the erosion, through what feedback, with what authority to act.

Using it

You do not need a safety engineering background. You need the four steps, an honest map of who actually controls what, and the willingness to write down the unsafe control actions your metrics would never surface. I am building the template into an interactive worksheet. Until it ships, the version I use in practice is available through the contact page.