Reliability
Reliability is another word for calibration: predicted probabilities match observed frequencies over time.
Definition
Reliability is commonly used to mean calibration. Forecasts are reliable when predicted probabilities align with observed frequencies in probability bins.
Why it matters
Reliability is core to responsible probability usage. Without it, users can be systematically misled by confident numbers that do not behave as advertised.
Common pitfalls
Overfitting: A model can look reliable in sample but fail out of sample.
Mixing domains: Reliability can differ across market types.