Valid vs. Reliable Metrics: Consistent Numbers Can Still Be Wrong

Short summary

Reliability (consistency of measurement) and validity (whether you're measuring the right thing) are independent properties — a metric can be perfectly stable yet completely wrong. The most dangerous failures cluster in the reliable-but-invalid quadrant, where unwavering numbers disguise incorrectness and evade scrutiny because nothing wobbles. Optimization worsens this: when a metric becomes a target, it grows more stable while its connection to the underlying construct erodes, making systematic metric coverage across all failure modes essential.

• •Reliability and validity are independent — consistent numbers can still measure the wrong thing
• •The most expensive failures are reliable-but-invalid metrics that never wobble and thus never get audited
• •Construct coverage across all failure modes matters more than perfecting any single metric

Generated with AI, which can make mistakes.