Multi-model ensembles

Nowcasting and forecasting of infectious disease dynamics

Ensembles

  • Combine many different models’ forecasts into a single prediction

Why ensemble?

Many uncertainties, many approaches: many models

  • Layers of uncertainty

    • Model parameters
      • e.g. parameterising delay distributions
    • Model structure
      • e.g. more mechanistic or more statistical approaches

  • Want to use all available information

Why ensemble?

“Whole is greater than sum of parts”

  • Average of multiple predictions is often more performant than any individual model

    • History in weather & economic forecasting

    • Seen this in infectious disease forecasting

      • “Forecast challenges”: Ebola, dengue, flu, COVID-19…

Ensemble methods

  • Summarising across models to create single (probabilistic) prediction

    • e.g. average at each models’ probabilistic quantiles

      • Mean

      • Median - trims the outliers, so narrows the uncertainty

Ensemble methods

  • Equal or weighted combination

    • Weight models by past forecast performance

      • e.g. using forecast scores

    • Rarely better than equal average

Collaborative modelling

“Forecast hubs”

  • Crowdsourcing forecasts

    • Open source collaborative projects

    • Anyone able to contribute a forecast

  • Forecasts ensembled into a single projection

  • Also enables consistent evaluation

Infectious disease forecasting hubs have grown in popularity over the last decade

(Reich et al. 2022)

… for example, the European Respicast

Single model

… Multiple models

… … Multi-model ensemble

r fontawesome::fa("laptop-code", "white") Your Turn

  1. Create unweighted and weighted ensembles using forecasts from multiple models.
  2. Evaluate the forecasts from ensembles compared to their constituent models.

Return to the session

References

Reich, Nicholas G, Justin Lessler, Sebastian Funk, Cecile Viboud, Alessandro Vespignani, Ryan J Tibshirani, Katriona Shea, et al. 2022. “Collaborative Hubs: Making the Most of Predictive Epidemic Modeling.” Am. J. Public Health, April, e1–4. https://doi.org/10.2105/ajph.2022.306831.