Evaluating probabilistic forecasts

The big picture

What makes a probabilistic forecast "good"?

A probabilistic forecast is not a single number but a whole distribution over what might happen. To judge it we use two ideas:

• Calibration — do the stated probabilities match reality? If you say "20% chance" on many occasions, it should happen about 20% of the time. A property that emerges from comparing forecasts to outcomes.
• Sharpness — how concentrated is the forecast? "Anything could happen" is safe but useless. A property of the forecast alone.

A proper scoring rule is a score, assigned to each (forecast, outcome) pair, that you minimise in expectation only by reporting your true beliefs. It rewards calibration and sharpness together, so it can't be gamed. Here we use the convention throughout that lower is better.

CRPS score · lower is better

Continuous Ranked Probability Score — the MAE of probabilistic forecasts

For a point forecast you would use the mean absolute error: the distance between your single guess and the truth. The CRPS generalises this to a forecast that is a distribution. Write the forecast as its cumulative distribution function (CDF) F, and the observed value as y. The CRPS compares the forecast CDF to the "perfect-hindsight" CDF — a single step from 0 to 1 at the value that actually happened:

The shaded region below is the CRPS: wherever the smooth forecast curve disagrees with the step at the observation, that squared gap is added up. A forecast that is both accurate (centred near y) and sharp (steep) leaves little gap and scores low. Drag the observation, or widen the forecast, and watch the area — and the score — change.

Breaking the CRPS into components

A single CRPS number tells you how wrong a forecast is, but not why. It splits cleanly into three additive pieces that always sum back to the total — the same decomposition scoringutils reports:

• Dispersion — the price of the forecast's own spread. Even a perfectly centred forecast pays this; it is the irreducible cost of being uncertain (and the reward for being sharp is paying less of it).
• Overprediction — extra penalty incurred because the forecast sat too high (the outcome fell below it).
• Underprediction — extra penalty because the forecast sat too low (the outcome fell above it).

Drag the controls above and watch the split: centre the forecast on y and only dispersion remains; push the observation into a tail and a directional penalty grows.

Things worth remembering

• CRPS is reported in the same units as the data (e.g. cases), so it is directly interpretable.
• For a point (deterministic) forecast the step meets the step and CRPS collapses to the absolute error — it is a true generalisation of MAE.
• An equivalent "energy" form avoids the integral: CRPS(F, y) = E|X − y| − ½ E|X − X′|, with X, X′ independent draws from F. The first term rewards accuracy; the second rewards sharpness.
• It is hard to read in isolation — its value depends on the scale of the data. Use it to compare models, horizons, or dates. (On a fast-growing epidemic, scoring on the log scale tames the scale dependence.)

PIT histogram diagnostic

Probability Integral Transform — a calibration check, not a score

Evaluate each forecast CDF F at the value that actually occurred: u = F(y) — where in the forecast distribution the outcome landed. Collected over many forecasts, these u values should be spread evenly across 0–1 if the forecasts are calibrated. Let's first build the histogram one forecast at a time, then learn to read its shape.

1 · Building the histogram — one forecast at a time

The PIT value is simply where each outcome lands on its own forecast CDF: read up from the observation y to the S-curve, then across to the axis — that height is u = F(y). The histogram is just those heights, from many forecasts, piling up. Pick a scenario, then press play and watch a fresh outcome get drawn, read off the CDF on the left, and dropped into its bin on the right.

2 · Reading the shape

Once many outcomes have piled up, the shape of the histogram tells you what is wrong. A perfectly calibrated forecaster gives a flat histogram; the classic departures are:

All four forecasters predict a Normal; the truth is also Normal. Only the shape of the predictive distribution differs.

Try it yourself

Move the forecaster's bias and spread and watch the histogram morph between the four shapes above. The truth is fixed at a standard Normal (mean 0, σ = 1); only the forecast changes. The preset buttons snap to each canonical case.

One subtlety for count data. The clean uniform result holds for continuous forecasts. For integer-valued forecasts (case counts!) the PIT is no longer exactly uniform even when calibrated, so the course uses a randomised PIT — it replaces the single step with a uniform draw across the probability jump at y — which restores uniformity under calibration.

DSS score · lower is better

Dawid–Sebastiani Score — accuracy and sharpness from just two numbers

The DSS asks only for the forecast's mean μ and standard deviation σ — no full distribution required. It is the squared standardised error plus a penalty on the spread:

Read the two terms as a tug-of-war:

• The first term is the error measured in units of the forecast's own spread. Claiming to be very sharp (small σ) and then missing is punished hard — the miss is divided by a tiny σ².
• The second term, 2 log σ, rewards being sharp (it falls as σ shrinks). That reward is what stops the score from simply preferring an infinitely wide, un-fallible forecast.

How DSS relates to CRPS

• DSS uses only the first two moments, so it is light-weight, but it is blind to shape — skewness and heavy tails are invisible to it. CRPS uses the whole distribution.
• Because of the squared standardised error, DSS is far more sensitive to outliers: a single observation deep in a sharp forecast's tail can dominate the average. CRPS is gentler and stays in the data's units.
• Both are proper. In the special case of a Normal forecast scored at its own mean, both bottom out — they just disagree on how harshly to punish the misses.

WIS score · lower is better

Weighted Interval Score — the CRPS for quantile forecasts

Forecast hubs (COVID-19, flu, RSV) don't collect a forecaster's whole distribution — they collect a handful of quantiles, e.g. the 2.5%, 25%, 50%, 75%, 97.5% points. The WIS scores exactly that. It is built from the interval score of each central prediction interval [l, u] at level (1−α):

Pay for the interval's width (sharpness), then pay a penalty if the outcome lands outside it. The WIS averages these across K intervals plus the median m, weighting each by α_k2:

The lower panel makes this concrete: the dashed line is the exact CRPS of the underlying forecast, and the WIS curve climbs toward it as the number of intervals K increases. The upper panel shows the nested intervals (the forecast "fan") and where the observation falls — intervals it escapes are drawn in red.

Same components as the CRPS

Because WIS is the finite-quantile CRPS, it carries the identical three-part decomposition — and the interval form makes each piece tangible:

• Dispersion = the summed interval widths — pure sharpness cost, paid whatever the outcome.
• Overprediction = penalties from intervals whose lower bound sat above the outcome (forecast too high).
• Underprediction = penalties from intervals whose upper bound sat below the outcome (forecast too low).

Calibration II — interval coverage diagnostic

The PIT's plain-English twin: do your intervals contain the truth as often as they claim?

The PIT histogram is one window onto calibration; interval coverage is another, often easier to explain to a non-statistician. Take the forecaster's central X% prediction interval (e.g. the 90% interval, running from the 5th to the 95th percentile). If the forecaster is calibrated, then over many forecasts the truth should fall inside that interval about X% of the time. Tally how often it actually does, for every nominal level from 0 to 100%, and plot the result.

Here every forecast is a Normal(μ_f, σ_f) and the truth is drawn from a standard Normal(0, 1). The forecaster can be biased (μ_f ≠ 0) and over- or under-dispersed (σ_f ≠ 1). Because both are Normal, the expected coverage of the central level q interval has a closed form — the solid curve — and a finite sample of n forecasts scatters around it (the dots).

Read the coverage diagram below as: x-axis = the level you claimed, y-axis = the level you achieved. The dashed 45° line is perfect calibration. An over-confident (too-sharp) forecaster makes intervals that are too narrow, so they catch the truth less often than advertised and the curve sags below the diagonal; an under-confident (too-wide) forecaster sits above it. Bias drops the whole curve, since an off-centre interval misses on the far side.

Scoring on the natural vs the log scale score

Why epidemic forecasts are often scored after a log transform

Epidemic incidence spans orders of magnitude — a wave can climb from a handful of cases to thousands and back. A proper score like the CRPS is in the units of the data, so on the natural scale a miss of 100 cases counts the same whether the truth was 50 or 5000. The few high-incidence days around the peak therefore dominate the total: a model can look excellent overall just by getting the peak roughly right, while being hopeless in the low-count growth and decline phases that matter most for situational awareness.

Scoring on the log scale — transform both forecast and observation by log(x + 1) before scoring — fixes this. Differences in log space are relative / multiplicative errors, so being a factor of two off is penalised equally whether near 10 cases or near 1000. This weights the growth, peak and decline phases more evenly and amounts, roughly, to scoring the order of magnitude (or the growth rate) rather than the raw count (Bosse et al., scoring epidemiological forecasts on transformed scales).

Below, an epidemic trajectory rises from about 1 to about 1000 and back. Two forecasters track it: A nails the peak but is sloppy at low counts, while B is consistently decent across all phases. Each bar is a single day's CRPS contribution. Flip between the natural and log scale and watch where the score "lives" — and watch the ranking flip.

Side by side