Two kinds of wrong

A system can be wrong in two ways. Each one calls for different controls, so the distinction is worth getting right before you spend money on safeguards.

A deterministic failure fires the same way every time once its conditions are met. The eighth Knight Capital server was always going to run the old code. The malformed CrowdStrike file crashed every machine that loaded it. Because the failure repeats, you can catch it before production with testing, release verification, and staged rollouts. When a deterministic fault reaches customers anyway, the real problem is usually a missing check in the pipeline rather than the bug itself.

A probabilistic failure breaks at some rate, on some inputs. Facial recognition returns a correct match most of the time and a wrong one on the faces it handles worst. Testing cannot get this to zero because there is no single input that always fails. The control is to limit how much authority the output carries. A match is a lead, and it needs corroboration before anyone acts on it. Someone has to measure error rates on the groups most likely to be harmed, and a human has to keep the power to override the system.

Teams get into trouble when they bring deterministic habits to a probabilistic system. The model passes its tests, so they ship it as if a passing test meant correctness. A model that is 95 percent accurate is a guess that is usually right. The remaining five percent is not spread evenly. It clusters on specific kinds of cases, often the ones where a wrong answer does the most damage.

So the first question in any incident review is simple. Did this break every time, or did it break on the odds? If it broke every time, you needed a better check. If it broke on the odds, the output was carrying more authority than its accuracy justified.