Anatomy of good forecasting models

Forecasting & evaluation of infectious disease dynamics

What is the special sauce of a good model?

  • We’ve spent a lot of time talking about baseline models, and the mechanics of generating simple models.

  • Making good forecast models is hard.

  • What are some of the clever and innovative ideas that make good models good?

What are the main obstacles to good models?

  • Predictive models either need a lot of data or very good “theory”.

    • Data: We sometimes have only on the order of 10 seasons of data. We’d like tens of thousands (or more).
    • Theory: If you could see all the eventual data, would you be able to explain why the outbreak evolved as it did? The underlying mechanics are impossibly complicated.

  • In real time, partially reported data can mislead.

  • Everyone talks up external data sources, but can they really help?

Selected good models

Dynamic Bayesian Model (Osthus et al. 2019)

🌍 Setting: 2017/18 FluSight challenge

🏆 Credentials: 4th best of 29 models

✨ Special sauce:

  • SIR models as curve-generating tool
  • hierarchical statistical discrepancy model

LNQ (R Wolfinger) & Flusion (Ray et al. 2025)

🌍 Setting:

  • LNQ: COVID-19 Forecast Hub case forecasting
  • Flusion: FluSight 2023/2024

🏆 Credentials:

  • LNQ: most accurate case forecasting 2020-2021
  • Flusion: 1st of 27 models, 2023/2024 season

✨ Special sauce:

  • gradient boosting with lots of feature engineering
  • data synthesis from multiple sources
  • ensembling several individual models

Pyrenew_HE_COVID model (US CDC CFA)

🌍 Setting: US COVID-19 Forecast Hub 2025-2026

🏆 Credentials: 1st out of 20 models1

✨ Special sauce:

  • Used semi-mechanistic renewal model.
  • Had access to internal CDC data that had 4-5 more days recent data than other models.

Google SAI (Martinson et al. 2026)

🌍 Setting: FluSight 2025/20261

🏆 Credentials: 1st out of 39 models

✨ Special sauce:

  • TreeSearch algorithm with LLM re-writing code for models
  • Ensembling multiple candidate models

KCDE (Ray et al. 2017) & Copycat (Spencer Fox)

🌍 Setting

  • KCDE: 2010-2017 FluSight Network
  • Copycat: 2024/2025 FluSight season

🏆 Credentials

  • KCDE: 3rd of 22 models
  • Copycat: 4th of 23

✨ Special sauce

  • non-parametric method of analogues
  • Copycat: matching on log growth rate
  • KCDE: matching on season week and standardized trends

Dante (Osthus and Moran 2021)

🌍 Setting: 2018/19 FluSight challenge

🏆 Credentials: 1st out of 33 models

✨ Special sauce

  • data (reporting error) + process model
  • multi-scale hierarchy across locations

Key common features of these models

Squeeze all the juice out of your data

  • Borrow information across locations. (Flusion, DBM, Dante)
  • Find other sources of surveillance data and use them for training. (Flusion, LNQ)

“Epi-informed” but not a lot of mechanism

  • The above list focused on seasonal settings, where lots of historical data are available.
  • Most models were largely “phenomenological”.

Return to the session

References

Martinson, Sarah, Michael P. Brenner, Martyna Plomecka, Brian P. Williams, Nicholas G. Reich, and Zahra Shamsi. 2026. Prospective Multi-Pathogen Disease Forecasting Using Autonomous LLM-Guided Tree Search. arXiv. https://doi.org/10.48550/arXiv.2605.16238.
Osthus, Dave, James Gattiker, Reid Priedhorsky, and Sara Y. Del Valle. 2019. “Dynamic Bayesian Influenza Forecasting in the United States with Hierarchical Discrepancy (with Discussion).” Bayesian Analysis 14 (1): 261–312. https://doi.org/10.1214/18-BA1117.
Osthus, Dave, and Kelly R. Moran. 2021. “Multiscale Influenza Forecasting.” Nature Communications 12 (1): 2991. https://doi.org/10.1038/s41467-021-23234-5.
Ray, Evan L., Krzysztof Sakrejda, Stephen A. Lauer, Michael A. Johansson, and Nicholas G. Reich. 2017. “Infectious Disease Prediction with Kernel Conditional Density Estimation.” Statistics in Medicine 36 (30): 4908–29. https://doi.org/10.1002/sim.7488.
Ray, Evan L., Yijin Wang, Russell D. Wolfinger, and Nicholas G. Reich. 2025. “Flusion: Integrating Multiple Data Sources for Accurate Influenza Predictions.” Epidemics 50 (March): 100810. https://doi.org/10.1016/j.epidem.2024.100810.