We follow a weekly update cycle at the COVID-19 Forecast Hub. Every Tuesday morning, our team assembles the most recent forecasts from all teams that have submitted in the last week. We use these data to generate an ensemble forecast that synthesizes predictions of COVID-19 hospitalizations from all eligible models. These forecasts are then reviewed, analyzed, and verified by our collaborators at the US CDC. On Thursday mornings, the CDC updates their COVID-19 Forecasting webpage with the latest data from the COVID-19 Forecast Hub.
Update: As of February 20, 2023, we are no longer generating ensemble case forecasts, and as of March 6, 2023, we are no longer generating ensemble death forecasts. Reports up to March 13, 2023 include case and death forecast evaluations, but after that date they only include hospitalization forecast evaluations.
Update: We are no longer generating forecast summary reports. The last reports were generated on October 3, 2023.
Forecast Evaluations
Periodically, we evaluate the accuracy and precision of the ensemble forecast and component models over recent and historical forecasting periods. Models forecasting incident hospitalizations at a national and state level are evaluated using (adjusted relative weighted interval scores (WIS, a measure of distributional accuracy), and adjusted relative mean absolute error (MAE), and calibration scores. Scores are evaluated across weeks, locations, and targets. You can read a paper explaining these procedures in more detail, and look at the most recent monthly evaluation reports. The final report that includes case and death forecast evaluations is 2023-03-13.
Forecast Evaluation Dashboard
In collaboration with the COVID-19 Forecast Hub, the Delphi Group at Carnegie Mellon University has created a Forecast Evaluation Dashboard that compares the performance of models over time, using different outcome variables (incident cases and deaths), metrics (Weighted Interval Score, Absolute Error, and Coverage), horizons (1 through 4 week forecasts), and locations (US national and state levels). As of February 20, 2023, case forecasts are no longer being updated and as of March 6, 2023 death forecasts are no longer being updated. Learn more about the dashboard and download the code on GitHub.
Forecast Summaries
Note: As of October 3, 2023 we are no longer generating weekly forecast summary reports. Prior to this, each week, we generated a weekly report that provided top-level summary numbers from the ensemble forecast. Browse past forecast summaries.