Youssef

forecast-arena

Testing AI forecasting models against reality. Reproducible benchmarks, real world datasets and honest results.

Code on GitHub

lowest price-forecast error (MAE) of the eight
19.27

Chronos-2 on the German day-ahead price, averaged over all 24 delivery hours. The simple seasonal baseline scores 28.20.

Language
Python
Licence
MIT
Last push
16 Nov 2026
Stars
2

What it is

Real data. Real predictions. Real results. An open benchmark that tests whether time series foundation models actually beat traditional methods on real, messy operational data, with reproducible code, leakage-guarded protocols, and the failures published next to the wins.

Vendors report wins on academic benchmarks. We re-test that claim on two systems Germany runs on: Deutsche Bahn train delays and day-ahead electricity prices.

How it works

  • Rolling-origin evaluation: chronological splits, no shuffling, identical origins and inputs for every model.
  • Leakage guards by construction: models only ever receive history truncated at the origin (ADR-0001, ADR-0002).
  • Pre-registered event thresholds: disruption / negative-price / spike thresholds frozen on training data before holdout evaluation.
  • Honest metrics: MAE, RMSE, MASE vs seasonal naive, sMAPE where valid (never MAPE on prices), interval coverage, detection precision/recall, inference runtime.
  • Significance: Diebold-Mariano tests vs the best baseline with a multiple-testing caveat stated.

Results

Model MAE Skill vs naive 80%-interval coverage
Timestral 209M 1.84 +33.2% 0.78
Timestral 28M 1.87 +32.4% 0.77
Chronos-2 1.89 +31.4% 0.76
TimesFM 3.0 1.90 +31.4% 0.79
LightGBM 1.92 +30.7% ,
XGBoost 1.93 +30.2% ,

Try it

uv run --extra dashboard python dashboard/app.py   # -> http://localhost:7860