Selecting test-phase models for the local hub

Introduction

In this session you will continue the work in the Local hub playground session, where you started to build and fit your own models on the first few seasons of ILI data. Using results from those first few seasons of forecasting, you will now select 1-2 models to officially “submit” to the online forecast hub.

Slides

Objectives

The aims of this session are

  1. to practice building forecasting models using real data, with a focus on selecting parsimonious models for a “testing” phase, and
  2. to develop skills around using GitHub and interactions with collaborative forecasting tools.

Source file

The source file of this session is located at sessions/hub-playground-testing.qmd.

Libraries used

In this session we will use the dplyr package for data wrangling, the ggplot2 and ggtime packages for plotting, the fable package for building forecasting models, and the epidatr package for downloading epidemiological surveillance data.

Additionally, we will use some hubverse packages such as hubEvals, hubUtils, hubData, hubVis, hubEnsembles packages for building ensembles.

library("nfidd.forecasting")
library("dplyr")
library("ggplot2")
library("epidatr")
library("fable")
library("ggtime")
library("hubUtils")
library("hubEvals")
library("hubVis")
library("hubData")
library("hubEnsembles")
theme_set(theme_bw())
Tip

The best way to interact with the material is via the Visual Editor of RStudio.

A flu forecasting hub for model dev

Recall that we have built a live modeling hub for all participants of the class to use as a “sandbox” for building models and running analyses.

If you can get through this session, the reward will be seeing your forecasts alongside the other forecasts at the online dashboard for this hub.

Complete the setup from the first local hub session if you haven’t already.

Selecting models for the “testing phase”

This hub contains five seasons for which you can submit forecasts, starting with 2015/2016 and ending with 2019/2020. Our recommended procedure from the first session is to use the first two seasons as a place to do as much model fitting as you have time for. If you did not have a chance to get 2-3 models to fit and generate predictions for the first two seasons, we suggest that you return to that session and complete that task before starting here.

You could create multiple flavors of ARIMA, or any of the more complex models suggested in the first session or in the epidemiologically-motivated models. To do this in a “complete” way, you might create between 10 and 100 different variations of models.

Then, from those models, select the few that you think will do the best in the final three seasons. (We can think of the last three years as our real-time forecast “testing phase”.) This selection process is important, and tricky to think about.

Tip

In general you want to pick models that have performed well in some time-series cross-validation (similar to the analysis in the first local hub session). But, you might also want to prioritize choosing models that are simpler (i.e., have fewer parameters) and therefore are less likely to be “overfit” to the cross-validation phase.

You could also consider splitting the models into a few different classes and picking 1-2 models from each class, e.g. your best ARIMA model, your best NNETAR model, your best ensemble, etc…

Once you’ve selected the models, you can generate and submit forecasts for the test phase to your local hub.

Submitting test-phase forecasts to an online hub

Once you have selected your test-phase models, you can submit their forecasts to our online sismid-ili-forecasting-sandbox hub.

If you can get through this process, the reward will be seeing your forecasts on the online dashboard for this hub.

Process of submitting to an online hub

The best way to submit to an online hub is via a pull request initiated through GitHub.

Drag and drop

If you’re not that familiar with GitHub and you have forecasts that you’d like to upload, work with an instructor or another participant who does have access to the hub. They will be able to create a subfolder of the model-output directory of the hub for you to upload forecasts to via a Drag and Drop.

You still will need a GitHub account (sign up here if you don’t have one).

Cloning repo, submitting via PR

Caution

The following assumes that you have a GitHub account and some basic working knowledge of GitHub interactions and terminology.

  1. Fork the sismid-ili-forecasting-sandbox hub so you have a copy of this repository owned by your GitHub user account.

  2. Clone your fork of the repo to your laptop (ideally, it should live at the top level within the sismid directory with these course materials).

  3. Put whatever model-metadata and forecasts you want to submit into the correct locations in the hub.

  4. Commit these changes to your local clone and push those changes to your fork on GitHub.

  5. Open a Pull Request (PR) back to the primary repository with your changes.

  6. Once the PR is open, watch the validations, and inspect/fix any errors that appear.

  7. If your forecasts pass validations, then an instructor will merge them in.

References