Reliable and wrong

Reliability and validity get used interchangeably in AI assurance. They are different properties, and the difference is where the worst failures hide.

Reliability means the system behaves consistently. Same output for the same input, stable week to week, no drift. Validity means the thing the system measures or optimizes actually corresponds to what you care about. A bathroom scale that reads four pounds heavy every morning is perfectly reliable and always wrong.

Traditional software failures were mostly reliability failures, and our testing instincts grew up on them. Knight Capital, CrowdStrike, and Therac-25 all involved systems doing something wrong, and the fixes were about making them behave consistently and correctly. Testing catches this class of problem because there is a right answer to test against.

Validity failures look different, and the Failure Atlas keeps running into them. The Optum care-management algorithm predicted healthcare cost accurately and consistently. Every reliability test would have passed. Meanwhile it was steering care away from Black patients who were just as sick, because cost was standing in for need and the substitution did not hold across the people being scored. Meta’s feed optimized its engagement metric with great precision, and the metric had drifted a long way from the meaning it was named after. COMPAS was calibrated, which is one honest definition of a valid risk score, and it still produced double the false-positive rate for Black defendants, which fails a different honest definition. None of these systems were broken.

This is why a clean model validation report tells you less than it appears to. Validation proves the system hits its target. It rarely asks whether the target is the right one, whether the target means the same thing for every group it touches, or what accumulates in the gap between the proxy and the goal.

The questions that surface validity failures are plain ones. What is this metric a proxy for, and who decided that? If the system were perfectly accurate at its target, who could still be harmed? Does the target mean the same thing across the populations being scored? If the proxy had quietly diverged from the goal, what would we expect to see, and is anyone measuring for it?

Run both reviews and keep them separate. The reliability review tells you whether the system works. The validity review tells you whether working is enough, and the systems that pass the first while failing the second are the ones doing damage in production right now with every dashboard green.