Introduction to ensembles

Understanding, evaluating, and improving forecasts of infectious disease burden

Ensembles

  • Combine many different models’ forecasts into a single prediction

Why ensemble?

Many uncertainties, many approaches: many models

  • Layers of uncertainty

    • Model parameters
      • e.g. parameterising delay distributions
    • Model structure
      • e.g. more mechanistic or more statistical approaches

  • Want to use all available information

Why ensemble?

“Whole is greater than sum of parts”

  • Average of multiple predictions is often more performant than any individual model

    • History in weather & economic forecasting

    • Seen this in infectious disease forecasting

      • “Forecast challenges”: Ebola, dengue, flu, COVID-19…

Ensemble methods

  • Summarising across models to create single (probabilistic) prediction

    • e.g. average at each models’ probabilistic quantiles

      • Mean

      • Median - trims the outliers, so narrows the uncertainty

Ensemble methods

  • Equal or weighted combination

    • Weight models by past forecast performance

      • e.g. using forecast scores

    • Rarely better than equal average

Collaborative modelling

“Forecast hubs”

  • Crowdsourcing forecasts

    • Open source collaborative projects

    • Anyone able to contribute a forecast

  • Forecasts ensembled into a single projection

  • Also enables consistent evaluation

  • Outbreak modelling

    • Since 2013 for US influenza

    • Ebola, dengue, chikungunya, COVID-19, WNV

… European Respicast

Single model

… Multiple models

… … Multi-model ensemble

Your Turn

  1. Create unweighted and weighted ensembles using forecasts from multiple models.
  2. Evaluate the forecasts from ensembles compared to their constituent models.

Return to the session