COPPERLINE

What remains predictable after a change in regime?

COPPERLINE studies forecasting when the relationship between past and future is unstable: a sensor drifts, an operating condition changes, or observations arrive late. The question is whether the uncertainty estimate remains useful as those conditions change.

The program evaluates accuracy and uncertainty together. A lower average error is insufficient if prediction intervals become overconfident precisely when conditions depart from the training record.

Training windowHeld-out future

Fold 01

Fold 02

Fold 03

Earlier observations → Later observations

Fit preprocessing and model parameters using only data available at each forecast origin. Evaluate each horizon separately.

Data and decision model

The forecast starts with what was knowable.

Each prediction is associated with an origin t, a horizon h, and the information available at t. Observation time and availability time are separate fields: an event may occur before the prediction while its report arrives afterward.

Minimum evaluation record

Field	Purpose
Observation time	When the underlying event or measurement occurred.
Availability time	When that value could first have been used by the forecaster.
Target and horizon	The quantity being predicted and the decision’s lead time.
Covariate vintage	The version of each explanatory input available at the forecast origin.
Quality state	Missingness, revision, censoring, and known instrument or process changes.

Evaluate the cost of being wrong

When underprediction and overprediction have different consequences, a central estimate does not describe the whole decision. Quantile forecasts allow the evaluation to express that asymmetry. For quantile level q, prediction ŷ, and observation y, the pinball loss is:

L_q(y, ŷ) = (q − 𝟙[y < ŷ]) (y − ŷ)

Quantile scores are assessed alongside interval coverage, width, and the performance of simple alternatives. Coverage is examined by horizon and operating regime; a satisfactory aggregate score can conceal failures during a transition.

Application context

Replenishment with an uncertain lead time.

A planning decision may depend on consumption over the next delivery window, while both consumption and the window itself vary. A point forecast can conceal the risk of a shortage; a very wide interval can make the decision uninformative.

The evaluation links the forecast to a declared rule for allocation or replenishment, then compares shortage, excess, and intervention costs across the same chronological test periods. This makes the practical value of uncertainty estimation part of the experiment.

Method

Compare persistence and seasonal baselines with statistical and learned models. Use chronological backtests, then introduce controlled drift, missingness, and observation delays.

Evaluation

Measure error by horizon and regime, interval coverage and width, and the delay between a change and its detection. Preserve a final untouched test period.

Output specification

A backtesting specification, a shift-sensitive evaluation report format, and criteria for withholding or qualifying a forecast.

Evaluation protocol and limitations +

Prevent information leakage

Use the data vintage that would actually have been available at prediction time. Revised records, future-derived features, and preprocessing fitted on the full series can all contaminate a backtest.

Examine useful uncertainty

Evaluate interval coverage alongside interval width. Intervals that contain almost every outcome by becoming arbitrarily wide are not necessarily useful for a decision.

Boundary of the study

Controlled shifts can reveal specific failure modes; they cannot represent every future change. Conclusions should remain tied to the tested data, forecast horizons, and operating conditions.

Methodological reference: Hyndman & Athanasopoulos, Time series cross-validation, Forecasting: Principles and Practice.