Multivariate forecast evaluation

Forecasting & evaluation of infectious disease dynamics

Scoring more than one prediction task

  • We can build forecasts for many horizons and locations at once
  • We can score them with the CRPS and summarise across dimensions
  • So far we have scored each horizon and location on its own and averaged

Caution

What are we missing when we only score one horizon and location at a time?

Marginal vs joint forecasts

Statistics refresher

  • A marginal distribution describes a single variable.
  • A joint distribution describes multiple variables and how they move together.

We can think of probabilistic forecasts in similar terms

  • Quantile forecasts are always marginal.
  • Samples can be joint across dimensions like horizon and/or location.

Marginal vs. joint predictions

Figure 1

Here the same quantitative forecasts have different meanings.

The energy score

A proper scoring rule for a vector outcome \mathbf{y}, generalising the CRPS.

\text{ES}(F, \mathbf{y}) = \mathbb{E}\lVert \mathbf{X} - \mathbf{y} \rVert - \tfrac{1}{2}\,\mathbb{E}\lVert \mathbf{X} - \mathbf{X}' \rVert

With m sample trajectories \mathcal{X}=\{\mathbf{x}_1, \dots, \mathbf{x}_m\} from the forecast, we estimate it by

\widehat{\text{ES}}(\mathcal{X},\mathbf{y}) = \frac{1}{m}\sum_{i=1}^{m}\lVert \mathbf{x}_i - \mathbf{y} \rVert - \frac{1}{2m^2}\sum_{i=1}^{m}\sum_{j=1}^{m}\lVert \mathbf{x}_i - \mathbf{x}_j \rVert

  • \mathbf{X}, \mathbf{X}' are independent draws from the forecast, \lVert\cdot\rVert is Euclidean distance
  • First term rewards accuracy, second rewards sharpness
  • In one dimension it is the CRPS
  • Needs joint samples from a modeling approach that can generate trajectories

Same marginals, different energy score

The marginal CRPS is the same for all three forecasts below, but ES are larger with less correlation.

Figure 2

Sharpness vs dependence

Figure 3

Across a row (same marginals) CRPS is identical. ES improves when horizons are correlated.

Multivariate forecasts: higher dimensions

Some (not all) models can make forecasts that are joint across horizons AND locations.

Forecasts that are joint across locations could enable decision-support for:

  • resource allocation

what is the optimal allocation of a fixed resource, like PPE or ventilators?

  • assessing multi-location peak timings

what is the likelihood that all locations peak at the same time?

Your Turn

  1. Score multi-horizon forecasts jointly with the energy score.
  2. Show that dependence changes the energy score but not the marginal CRPS.

Return to the session

References