Model importance in ensembles

Forecasting & evaluation of infectious disease dynamics

Ensembles review

  • Combining many models usually beats any single one.

But which member models actually make the ensemble better?

  • We turn to the idea of Shapley values from game theory for inspiration.

Standard scoring answers a different question

  • Standard metrics score each model on its own.
  • That measures standalone skill – not a model’s contribution to the ensemble.
  • A strong standalone model is not guaranteed to help the ensemble; a model can even hurt it (a negative contribution).(Kim et al. 2026)

Note

Model importance is to ensemble members what variable importance is to features in a random forest – but measured at the level of whole models.

The core idea

A model’s importance = how much the ensemble’s accuracy changes when that model is added or removed.

Conceptual illustration (three models). Stars = accuracy by a positively-oriented score, e.g. -WIS.

Leave-One-Model-Out (LOMO)

Compare the full ensemble to the ensemble with one model removed.

  • A = set of n models; F^{A} = full ensemble forecast.
  • F^{A^{-i}} = ensemble of every model except i.
  • Importance of i = accuracy of F^{A} - accuracy of F^{A^{-i}} (positively oriented, so >0 means model i helps).

Example with n=3: to score model 1, build the ensemble of \{2,3\} and compare it to \{1,2,3\}.

Leave-All-Subsets-of-Models-Out (LASOMO)

  • LOMO uses a single comparison (full vs full-minus-i).
  • LASOMO averages model i’s contribution over all subsets that exclude it – the full Shapley-value idea (more complete, but more expensive).

Example with n=3, scoring model 1: take every subset without model 1 — \{2\}, \{3\}, \{2,3\} — and for each compare the ensemble with model 1 added vs. without it (F^{S \cup \{1\}} vs F^{S}), then aggregate.

The modelimportance R package

  • model_importance() returns a per-task importance score for each ensemble member.
  • LOMO or LASOMO; point and probabilistic forecasts; configurable handling of missing forecasts.
  • Built on the hubverse (hubUtils, hubEnsembles, hubEvals), so it drops straight into existing hubs.
  • print(), summary(), aggregate().
model_importance(
  forecast_data, oracle_output_data,
  ensemble_fun = "simple_ensemble",
  importance_algorithm = "lomo"
)

Example: Flu MetroCast Hub 2025/26

  • Hub run by epiENGAGE Insight Net center
  • National Syndromic Surveillance Program (NSSP) data
  • Target: weekly % of ED visits due to flu
  • Locations: 64 states and Health Service Areas (HSA)
  • Models: 10 from various academic groups
  • Missing forecasts receive a 0 importance for those tasks

Example: LOMO results

Takeaways

  • An ensemble is only as good as the mix of its members – not each member’s solo skill.
  • LOMO: quick and intuitive (full vs leave-one-out). LASOMO: more like Shapley values, but computationally costlier.
  • modelimportance makes this a one-function, hubverse-native analysis.
  • Full method and a large-scale application: Kim et al. (2026).

References

Kim, Minsu, Evan L. Ray, and Nicholas G. Reich. 2026. “Beyond Forecast Leaderboards: Measuring Individual Model Importance Based on Contribution to Ensemble Accuracy.” International Journal of Forecasting 42 (3): 924–36. https://doi.org/10.1016/j.ijforecast.2025.12.006.