Weather, demand, and forecast uncertainty
Hyndman and Khandakar (2008) · Replication planned
Can a proposed forecasting model beat automatic ARIMA, ETS, and seasonal naive forecasts? Intervals and forecasts depend on the selected model and innovation specification.
Failure to investigate. Change seasonality or introduce a break; evaluate only at chronological origins.
Starting calculation. Start with the journal's archived R script and package. Compare current-package output separately.
Proposed extension. A break-aware combination evaluated on frozen future origins.
Software or replication gateway ↗
Source and reuse notes
Paper: JSS open-access policy; inspect the selected artifact before reuse.
Code: Selected archive license not yet inspected.
Data: Selected input-series terms not yet inspected.
DOI: 10.18637/jss.v027.i03
Torrence and Compo (1998) · Replication planned
Does an oscillation persist or appear during particular periods? Significance uses a declared background-noise model; boundary effects and multiple comparisons remain relevant.
Failure to investigate. Compare red-noise apparent cycles with the cone of influence and a prespecified search.
Starting calculation. Start with the supplied ENSO example and original routines before porting.
Proposed extension. Forecast features selected only within training histories.
Software or replication gateway ↗
Source and reuse notes
Paper: Author-hosted manuscript; general republication rights not established.
Code: MIT verified in https://raw.githubusercontent.com/ct6502/wavelets/main/LICENSE on 2026-10-09; retain notices and pin an execution commit.
Data: ENSO input terms not yet inspected.
Hyndman and Fan (2010) · New-data reconstruction planned
What uncertainty disappears when a forecast uses one weather trajectory? Weather, economic scenarios, and residual resampling must match the target horizon.
Failure to investigate. Shuffle weather hours or admit realized future temperature as though it were known.
Starting calculation. Original input access is unresolved. A vic_elec exercise would be a labelled new-data reconstruction, not reproduction of the original study.
Proposed extension. Regime-sensitive demand uncertainty with coherent weather blocks.
Software or replication gateway ↗
Source and reuse notes
Paper: Author-hosted manuscript; general republication rights not established.
Code: No original replication bundle inspected.
Data: vic_elec provenance is documented; redistribution terms and demand-unit convention must be checked before acquisition.
DOI: 10.1109/TPWRS.2009.2036017
Xu and Xie (2021) · Replication planned
How useful are prediction intervals under dependent errors? Approximate marginal coverage depends on assumptions, including the error dependence conditions; it is not exact conditional coverage for arbitrary series.
Failure to investigate. Increase dependence, change variance, and measure local coverage.
Starting calculation. Pin the maintained author code rather than assuming the supplementary archive is identical.
Proposed extension. Weather-regime calibration with a declared effective sample requirement.
Software or replication gateway ↗
Source and reuse notes
Paper: Open proceedings article; inspect the selected artifact before reuse.
Code: MIT verified in https://raw.githubusercontent.com/hamrel-cxu/EnbPI/main/LICENSE on 2026-10-09.
Data: Bundled-data terms not yet inspected.
Xu and Xie (2023) · Acquisition and replication planned
Can residual predictability improve interval calibration? Asymptotic conditional coverage is tied to consistency conditions for residual-quantile prediction.
Failure to investigate. Misspecify that predictor or shorten calibration history.
Starting calculation. Inspect and run tutorial_electric_EnbPI_SPCI.ipynb with a pinned source snapshot.
Proposed extension. Adapt residual quantiles using only break information available at the forecast origin.
Software or replication gateway ↗
Source and reuse notes
Paper: Open proceedings article; inspect the selected artifact before reuse.
Code: Explicit reuse license not established.
Data: Bundled electricity-data terms not yet inspected.
Gibbs and Candès (2024) · Acquisition and replication planned
How quickly should calibration forget earlier errors? Regret and empirical coverage-frequency claims differ from pointwise conditional coverage.
Failure to investigate. Compare abrupt and gradual variance changes, interval widths, and adaptation delays.
Starting calculation. Reproduce a simulation before the financial application and retain the original bootstrap conventions.
Proposed extension. A forecast-origin-available adaptation signal compared with unchanged DtACI and SPCI.
Software or replication gateway ↗
Source and reuse notes
Paper: Open JMLR article; inspect the selected artifact before reuse.
Code: Explicit reuse license not established.
Data: Financial-data redistribution terms not yet inspected.
Gneiting and Raftery (2007) · Computational verification planned
How can evaluation reward honest uncertainty? Scoring a full distribution and scoring an interval are distinct tasks; coverage alone does not reward sharpness.
Failure to investigate. Compare narrow under-covering intervals with very wide intervals.
Starting calculation. Implement interval score and sample CRPS independently and check analytic toy distributions; this is an original exercise, not a paper replication.
Proposed extension. Decision-relevant demand evaluation that retains proper scoring incentives.
Source and reuse notes
Paper: Author-hosted manuscript; general republication rights not established.
Code: Independent original teaching code planned.
Data: Original synthetic fixtures planned.
DOI: 10.1198/016214506000001437
Latent signals, long-run relations, and breaks
Helske (2017) · Replication planned
How should a noisy measurement change an estimate of an evolving state? The observation and transition models determine which latent quantity is estimable.
Failure to investigate. Confound state variation with measurement noise; add missing readings and outliers.
Starting calculation. Run the archived example, then compare a small Gaussian filter's likelihood and moments in three languages.
Proposed extension. An online innovation-scale adaptation with a measured false-alarm cost.
Software or replication gateway ↗
Source and reuse notes
Paper: JSS open-access policy; inspect the selected artifact before reuse.
Code: Selected package-version license not yet inspected.
Data: Example-data terms not yet inspected.
DOI: 10.18637/jss.v078.i10
Zeileis, Leisch, Hornik, and Kleiber (2002) · Replication planned
Is a changed relationship distinguishable from sampling fluctuation? Test calibration depends on the regression and disturbance assumptions; dating and monitoring answer different questions.
Failure to investigate. Use stable-model critical values after introducing dependent errors or nonstationary regressors.
Starting calculation. Separate the archived 2002 examples from functionality added in later package versions.
Proposed extension. A forecast adaptation that uses only online break evidence.
Software or replication gateway ↗
Source and reuse notes
Paper: JSS open-access policy; inspect the selected artifact before reuse.
Code: Selected package-version license not yet inspected.
Data: Example-data terms not yet inspected.
DOI: 10.18637/jss.v007.i02
Park (1992) · Method verification planned
When can trending variables have a stable long-run relation? Integration orders, stable cointegration, and long-run covariance regularity matter.
Failure to investigate. Compare unrelated random walks, then change a genuine cointegrating coefficient.
Starting calculation. Validate the documented CCR transform on a known simulated system; this is method verification, not a historical empirical-table replication.
Proposed extension. Changing long-run energy-demand relations with measurement error.
Software or replication gateway ↗
Source and reuse notes
Paper: Original article access and reuse not established; the linked source is primary software documentation citing the article.
Code: Selected arch-version license not yet inspected.
Data: Proposed fixture would be original synthetic data.
Hamilton (2018; working paper 2017) · Acquisition pending
Which apparent dynamics come from a filter? A detrending rule defines a measured object; it does not reveal a uniquely true latent cycle.
Failure to investigate. Filter a random walk and compare endpoint revisions and induced autocorrelation.
Starting calculation. Inspect the author appendix before attempting reproduction. Automated access to the NBER record returned HTTP 403 in this pass.
Proposed extension. A forecast-origin revision score reported alongside predictive accuracy.
Software or replication gateway ↗
Source and reuse notes
Paper: Article republication rights not established.
Code: Appendix license not yet inspected.
Data: Appendix input terms not yet inspected.
Ardia, Bluteau, Boudt, Catania, and Trottier (2019) · Replication planned
Does a high-volatility episode require another regime? Regime identification, innovation tails, and numerical convergence require separate checks.
Failure to investigate. Fit switching models to heavy-tailed single-regime data and retain failed fits.
Starting calculation. Run archived R materials before comparing independently checked single-regime baselines.
Proposed extension. Parametric risk versus adaptive calibration during declared variance breaks.
Software or replication gateway ↗
Source and reuse notes
Paper: JSS open-access policy; inspect the selected artifact before reuse.
Code: Selected package/archive license not yet inspected.
Data: Financial-data redistribution terms not yet inspected.
DOI: 10.18637/jss.v091.i04
Geographic dependence and policy counterfactuals
Bivand and Piras (2015) · Replication planned
Why do nominally identical spatial models differ across programs? Weights, normalization, ordering, and impact definitions are part of the specification.
Failure to investigate. Reorder observations without reordering weights or interpret a lag coefficient as the full marginal effect.
Starting calculation. Reproduce one archived county specification before comparing current R and PySAL versions and an independent Julia calculation.
Proposed extension. Propagate uncertainty about spatial exposure into estimated impacts.
Software or replication gateway ↗
Source and reuse notes
Paper: JSS open-access policy; inspect the selected artifact before reuse.
Code: Each selected software component needs its own license record.
Data: County-source and archive terms not yet inspected.
DOI: 10.18637/jss.v063.i18
Brodersen, Gallusser, Koehler, Remy, and Scott (2015) · Replication planned
When does a post-intervention forecast error identify an effect? Controls must be unaffected by treatment and their relation with the treated outcome must remain stable.
Failure to investigate. Contaminate controls with spillovers or add a coincident unrelated break.
Starting calculation. Run the official R examples and a known-effect simulation; port only a declared common model.
Proposed extension. Sensitivity to control contamination and loading drift.
Software or replication gateway ↗
Source and reuse notes
Paper: Author-hosted manuscript; general republication rights not established.
Code: Apache-2.0 verified in https://raw.githubusercontent.com/google/CausalImpact/master/LICENSE on 2026-10-09.
Data: Selected example-data terms not yet inspected.
Callaway and Sant'Anna (2021) · Replication planned
Which effects are averaged under staggered adoption? Specify group-time targets, comparison groups, parallel trends, overlap, and anticipation restrictions.
Failure to investigate. Introduce spillovers or heterogeneous dynamics and compare with a conventional two-way fixed-effects coefficient.
Starting calculation. Run the documented mpdta example. Do not invent adjacency where locations are unavailable.
Proposed extension. Exposure-sensitive comparisons with the changed target population reported.
Software or replication gateway ↗
Source and reuse notes
Paper: Public preprint; selected manuscript reuse terms not yet inspected.
Code: GPL-3 verified in https://raw.githubusercontent.com/bcallaway11/did/master/DESCRIPTION on 2026-10-09.
Data: mpdta redistribution terms not yet inspected.
DOI: 10.1016/j.jeconom.2020.12.001
Bach, Chernozhukov, Kurz, and Spindler (2022) · Method verification planned
How can flexible prediction support parameter estimation? Orthogonality and cross-fitting do not supply identification, overlap, or an appropriate dependence framework.
Failure to investigate. Randomly split dependent observations or predict accurately in a confounded design.
Starting calculation. Start with known-truth examples and identical folds before a simple independent Julia port.
Proposed extension. Cross-fitting for a precisely specified dependent sampling design, after a focused literature review.
Software or replication gateway ↗
Source and reuse notes
Paper: Open JMLR article; inspect the selected artifact before reuse.
Code: MIT reported in the preliminary source review; selected execution-commit and license readback remain pending.
Data: Selected fixture terms not yet inspected.
A replication record
Keep the paper version, software commit, package versions, input definitions, units, timestamps, transformations, and hashes with the result. Record the target before comparing estimators. A current-software example, an original-paper reproduction, and a reconstruction on another dataset answer different questions.
A failure belongs in the record too. A model that fails to converge, a forecast that lacks an available covariate, or a method that loses to its benchmark cannot quietly disappear from the comparison. Live assessments and grading keys are separate from these public research exercises.
Every laboratory begins with a model contract: the quantity of interest, information available at the decision, sampling unit, dependence and identification assumptions, uncertainty object, failure conditions, and the specific result to reproduce.