Multi-model ensembles

Nowcasting and forecasting of infectious disease dynamics

Ensembles: many forecasts into one

Figure credit: Evan Ray and Nick Reich

Why ensemble?

1. Models are specialists and you want all the perspectives
   • different data sources
   • different philosophies, e.g. more mechanistic or more statistical approaches
   • different methodologies and parameterizations

2. A single “consensus forecast” is easier for decision-makers to digest

“Whole is greater than sum of parts”

Average of multiple predictions is often (not always) more performant than any individual model

• Strong evidence in weather & economic forecasting

• Recent evidence in infectious disease forecasting

  • Ebola(Funk et al. 2019)
  • dengue(Colón-González et al. 2021)
  • flu(Nicholas G. Reich et al. 2019)
  • COVID-19(Cramer et al. 2022)

Ensemble methods: how to average?

Figure credit: Howerton et al. (2023)

Ensemble methods: to weight or not?

• Weight models by past forecast performance

  • e.g. using forecast scores

• Rarely better than equal average

  • lots of uncertainty in weight estimation!
  • put a “strong prior” on equal weights, both in your mental and statistical models

Collaborative modelling “hubs”

• Projects run by research groups, public health agencies

• Participation generally open

• Standard format enables

  • data validation
  • ensemble-building
  • model evaluation
  • visualization

Hubverse

Hubs increasingly used in epi

Nicholas G. Reich et al. (2022)

… e.g., the European Respicast Hub

Single model

… Multiple models

… … Multi-model ensemble

Your Turn

1. Create unweighted and weighted ensembles using forecasts from multiple models.
2. Evaluate the forecasts from ensembles compared to their constituent models.

Return to the session

References

Colón-González, Felipe J., Leonardo Soares Bastos, Barbara Hofmann, Alison Hopkin, Quillon Harpham, Tom Crocker, Rosanna Amato, et al. 2021. “Probabilistic Seasonal Dengue Forecasting in Vietnam: A Modelling Study Using Superensembles.” PLOS Medicine 18 (3): e1003542. https://doi.org/10.1371/journal.pmed.1003542.

Cramer, Estee Y., Evan L. Ray, Velma K. Lopez, Johannes Bracher, Andrea Brennen, Alvaro J. Castro Rivadeneira, Aaron Gerding, et al. 2022. “Evaluation of Individual and Ensemble Probabilistic Forecasts of COVID-19 Mortality in the United States.” Proceedings of the National Academy of Sciences 119 (15): e2113561119. https://doi.org/10.1073/pnas.2113561119.

Funk, Sebastian, Anton Camacho, Adam J. Kucharski, Rachel Lowe, Rosalind M. Eggo, and W. John Edmunds. 2019. “Assessing the Performance of Real-Time Epidemic Forecasts: A Case Study of Ebola in the Western Area Region of Sierra Leone, 2014-15.” PLOS Computational Biology 15 (2): e1006785. https://doi.org/10.1371/journal.pcbi.1006785.

Howerton, Emily, Michael C. Runge, Tiffany L. Bogich, Rebecca K. Borchering, Hidetoshi Inamine, Justin Lessler, Luke C. Mullany, et al. 2023. “Context-Dependent Representation of Within- and Between-Model Uncertainty: Aggregating Probabilistic Predictions in Infectious Disease Epidemiology.” Journal of The Royal Society Interface 20 (198): 20220659. https://doi.org/10.1098/rsif.2022.0659.

Reich, Nicholas G, Justin Lessler, Sebastian Funk, Cecile Viboud, Alessandro Vespignani, Ryan J Tibshirani, Katriona Shea, et al. 2022. “Collaborative Hubs: Making the Most of Predictive Epidemic Modeling.” Am. J. Public Health, April, e1–4. https://doi.org/10.2105/ajph.2022.306831.

Reich, Nicholas G., Craig J. McGowan, Teresa K. Yamana, Abhinav Tushar, Evan L. Ray, Dave Osthus, Sasikiran Kandula, et al. 2019. “Accuracy of Real-Time Multi-Model Ensemble Forecasts for Seasonal Influenza in the U.S.” PLOS Computational Biology 15 (11): e1007486. https://doi.org/10.1371/journal.pcbi.1007486.