Multi-model ensembles

Nowcasting and forecasting of infectious disease dynamics

Ensembles

  • Combine many different models’ forecasts into a single prediction

Why ensemble?

Many uncertainties, many approaches: many models

  • Layers of uncertainty

    • Model parameters
      • e.g. parameterising delay distributions
    • Model structure
      • e.g. more mechanistic or more statistical approaches

  • Want to use all available information

Why ensemble?

“Whole is greater than sum of parts”

  • Average of multiple predictions is often more performant than any individual model

    • History in weather & economic forecasting

    • Seen this in infectious disease forecasting

      • “Forecast challenges”: Ebola, dengue, flu, COVID-19…

Ensemble methods

  • Summarising across models to create single (probabilistic) prediction

    • e.g. average at each models’ probabilistic quantiles

      • Mean

      • Median - trims the outliers, so narrows the uncertainty

Ensemble methods

  • Equal or weighted combination

    • Weight models by past forecast performance

      • e.g. using forecast scores

    • Rarely better than equal average

Collaborative modelling

“Forecast hubs”

  • Crowdsourcing forecasts

    • Open source collaborative projects

    • Anyone able to contribute a forecast

  • Forecasts ensembled into a single projection

  • Also enables consistent evaluation

Collaborative modelling

“Forecast hubs”

  • Outbreak modelling

    • Since 2013 for US influenza

    • Ebola, dengue, chikungunya, COVID-19, WNV

… European Respicast

Single model

… Multiple models

… … Multi-model ensemble

Evaluation: European COVID-19 Hub

  • March 2021 - now

  • ~40 teams, ~50 models

  • Weekly ensemble & evaluation

Evaluation: European COVID-19 Hub

Evaluation: European COVID-19 Hub

Evaluating ensembles

  • Ensemble reduces variance

    • (+) Stable

    • (-) Can’t explore extremes

  • Dependent on components

    • (+) Ensemble typically more accurate than any individual component

    • (-) Obscures mechanisms

Ensembles: reflections

  • Comparing uncertainties

    • Which aspects of uncertainty do we want to keep (compare), or combine?

  • Collaborative modelling

    • Opportunity for exchange & evaluation

    • Consensus - at the cost of context?