Proper Scoring Rule

Definition

A proper scoring rule is a way to evaluate probability forecasts such that reporting your true beliefs is optimal. It discourages gaming and encourages well calibrated predicted probabilities.

Why it matters

Proper scoring rules provide a clean standard for evaluating forecasters. They separate signal from style, and they help users improve probability thinking rather than chase narratives.

Common examples

• Brier score for binary outcomes.

• Log loss for probabilistic classification.

Common pitfalls

Using accuracy only: Accuracy ignores confidence and fails to reward well calibrated probabilities.

Comparing across different event types: Some event sets are inherently harder to forecast.