Evaluating real-world forecasts
Data analysis
Scores are often correlated, with outliers
Scores can be scale-dependent
Models don’t make the same predictions
Caution
When models have made forecasts for different subsets of tasks, it can make comparison of models complicated. For example, what if one model never forecasts for the harder prediction tasks?
Relative Skill Scores are a possible solution
Hubs accelerate scientific discovery
“Comparing the accuracy of forecasting applications is difficult because forecasting methods, forecast outcomes, and reported validation metrics varied widely.”
– Chretien et al., PLOS ONE, 2014
The US COVID-19 Forecast Hub
Launched April 2020 by the Reich Lab in collaboration with CDC. Goals were to:
- Provide decision-makers and general public with reliable information about where the pandemic is headed in the next month.
- Assess reliability of forecasts and gain insight into which modeling approaches do well.
- Create a community of infectious disease modelers underpinned by an open-science ethos.
…provided CDC with weekly forecasts
…had prominent uses
Hubverse is the tool we wish we had
Who works with a hub?
What does hubverse data look like?
Some columns define a prediction task
Others are always present
