Playbook — forecast¶

For experiments of type: "forecast": one or many time series in long format — a series key, a timestamp, a numeric target, optional drivers — and a model that predicts the next horizon steps with intervals. The plan carries a forecast block (no config, no target_column roles): the column mapping and its config — describe_platform("plan_schema", name="forecast") lists every setting. The delivery lifecycle is get_playbook("deliver"); this playbook is what each of its steps does for a forecast.

Frame¶

What is forecast — the target column (target_col), at which grain (series_id_col: one series per store, SKU …; none for a single series), and whether a coarser level is needed too (hierarchy_cols, coarse → fine: forecasts are reconciled bottom-up).
The calendar — the spacing of the timestamps (freq: D, W-SUN, ME …) or, for irregular events, the grid to roll them up to (resample_to with resample_agg: sum for flows, mean for levels). A fit needs a real calendar. Holidays follow config.lags.holiday_country (default US): set the data's country.
Horizon — how many steps ahead the decision needs (horizon, in steps of the grid), and the season (season_length: 12 for monthly data with a yearly cycle; omitted, it is detected).
The forecast origin — the prediction moment: the last timestamp the model may see. Drivers (exog_cols) are usable only if their future values are known at that moment (exog_known_future: planned promotions, calendars) — and then the recipe must carry them for the horizon (Data).
Metric — config.metric_primary: mase (default; below 1 beats seasonal-naive), rmsse, smape (unstable near zero), mae, wape (volume-weighted); describe_platform("metric_catalog", context="forecast").
Intervals — the levels the decision uses (config.confidence_levels, default 80 and 95). A level L is the central band from the (50 − L/2)th to the (50 + L/2)th percentile, so a planning percentile p is the upper edge of level 2p − 100: a 3:1 under/over cost (critical ratio 0.75 → P75) is yhat_upper_50, P90 is yhat_upper_80. These are nominal quantile labels, not guarantees of achieved service levels. Central coverage alone cannot establish one-sided upper-bound coverage: misses need not be symmetric. Use returned one-sided evidence when available; otherwise state that the cost-optimal service level has not been verified. A target that is never negative (demand, counts) is floored at zero by default (config.non_negative: auto; on | off): points and bands never go below zero, and the backtest metrics and coverage are scored on the floored values. Do not ask the consumer to clip downstream; read _forecast_floored in the run's notes for how many values it raised.
The backend — the forecast's requirement choice (config.backend, with config.budget fast | balanced | thorough); propose it from the data's shape:
foundation (a pretrained Chronos model, zero-shot; foundation_model_name, default chronos2) — series that are short (under about three seasons), many or new, with few drivers. Its weights must be on this server: describe_platform("forecast_models") gives each checkpoint's weights — cached, download (the first run fetches it) or missing (an offline server without it: the plan refuses it; pick a cached one);
tree (one gradient-boosted model across series) — drivers or static columns carry the signal, or the person wants per-driver explanations;
stats (statistical models per series, the cheapest) — a few long, regular series without drivers;
autogluon (an AutoGluon-TimeSeries ensemble, autogluon_preset such as chronos2_ensemble or best_quality) — accuracy outranks cost. describe_platform("forecast_models") lists each backend's models. A backend change later is the person's decision, like a template. A per-series mix of backends is not supported.

Data¶

Recipe — the history in long format: one row per series and timestamp, the target and the drivers, bounded at the forecast origin (author_recipe; the prediction moment is the origin).
Known-future drivers — a driver in exog_known_future needs horizon future rows per series after the origin, with the target blank and the driver filled: the recipe UNIONs the planned values onto the history (SELECT store, month, NULL AS y, promo FROM promo_plan). With none, the tree backend fills the driver with NaN over the horizon — the plan warns, and a future date off the grid is a plan error naming it.
Roles — on the plan's forecast block: series_id_col, time_col, target_col, exog_cols (+ exog_known_future), static_cols (constant within a series), hierarchy_cols. Duplicate rows per series and timestamp are summed (aggregate_duplicates).
Leakage — prepare_data(action="leakage", params={target, time_column}); a driver that is an aggregate over the future, or a series key that encodes the outcome, is the forecast's leak.
Signal — probe_dataset(kind="forecast_signal", params={time_col, target_col, series_id_col, horizon}): does a zero-shot Chronos model beat seasonal-naive, overall and per series? It always answers; grid: "ordinal" means the timestamps had no regular spacing — re-run with resample_to before calling the signal weak. When zero-shot beats seasonal-naive clearly and the framing chose another backend, ask at the data gate: keep it, or foundation. The probe's number is chronos2_small's — not the default chronos2 a foundation run loads. A foundation run compared with it sets foundation_model_name: "chronos2_small", or the comparison names the difference in checkpoint.

Validation¶

Rolling-origin cross-validation over the latest windows of every series: config.cv_n_windows windows (from the budget when unset), config.cv_step_size apart (at least the horizon), config.cv_gap rows dropped before each window. Validation is the windows the run selected on; test is each series' final horizon rows, forecast from the same cutoff whatever the CV design — so test numbers of runs that differ only in CV design are comparable, and a change no CV window sees (future-only driver values) leaves both numbers identical. An interval level L is calibrated only with enough pooled scores — (100 + L) / (100 − L) of them, one per series and window (9 for 80 %, 39 for 95 %): the plan warns and names the cv_n_windows that would do. The band's coverage is measured on the test window (forecast_summary intervals) and on every validation window, each banded without its own residuals (get_insight(kind="forecast_coverage") — per window and pooled, with the points scored). A statistical model's own (analytic) band is counted the same way on every window, as that window's fit drew it.

The bands are calibrated on the cross-validation residuals, and on a tree run those can come from a stand-in for the deployed model: under rolling_method: "tta" (the default) the earliest window's weights score every later window, and with a budget that buys a search the windows run untuned. Each level then says calibration_basis: "proxy" with its calibration_proxies (tta, untuned): report its coverage as approximate — calibrated on proxy residuals — and cite forecast_coverage as the evidence, never "calibrated" alone. rolling_method: "refit" with budget: "fast" calibrates on the deployed model's own shape (deployed_model), at the cost of a fit per window and no search.

Baseline¶

The approved backend, the horizon and the grid. On a monthly panel of a few stores with a yearly cycle and a promotion flag:

{
  "experiment": {
    "workspace_id": "<workspace_id>", "name": "store demand baseline", "type": "forecast",
    "dataset_id": "<dataset_id>", "target": "y",
    "forecast": {
      "series_id_col": "unique_id", "time_col": "ds", "target_col": "y",
      "exog_cols": ["promo_active"], "static_cols": ["region"],
      "config": {"horizon": 6, "freq": "ME", "season_length": 12, "backend": "tree",
                 "metric_primary": "mase", "confidence_levels": [80, 95]}
    }
  },
  "hypothesis": {"lever": "baseline", "expected_effect": "The yearly cycle and promotions beat seasonal-naive (MASE below 1)."}
}

The floor first: seasonal-naive, raced inside every run. get_insight(kind="forecast_summary") says whether a non-trivial model won (beats_baseline, with its basis) and lists the weak_series whose winner does not beat it. A run whose every winner is a baseline has found no signal beyond the season.

Iterate¶

Lever (`hypothesis.lever`)	Change
`features`	add or drop a driver (`exog_cols`, `exog_known_future`, `static_cols`); a recipe column first
`recipe`	the history itself: a longer window, a cleaned series, a different rollup, the known-future rows
`split`	`cv_n_windows`, `cv_step_size`, `cv_gap` — e.g. enough windows to calibrate a level
`other`	`tree_target_transform`, `lags.*`, `holiday_country` — with the reason in `note`
`template`	the backend — within the agreed requirements and budget, respecting explicit choices
`metric`	`metric_primary` — only once the person decided it

Keep the horizon and the grid fixed while iterating: they are the framing. A hypothesis about the horizon (future driver values) cannot move validation or test: say what evidence would test it instead (the forecast chart over the horizon).

Insights¶

Interpretation checklist: choose the relevant available reads for the question and candidate:

get_insight(kind="forecast_summary") — backend, validation and test metrics, beats_baseline, weak_series, failed stages, and intervals: per level whether the band is calibrated and on what (calibration_basis: the deployed model, or a proxy), which series fall short, its test coverage and how to fix it. weak_series[].mase is the worse of the winner's finite validation and test MASE, not a test-only estimate. A series can remain flagged when validation MASE exceeds 1 even if test MASE is below 1. Use the explicit split labels in forecast_series; this warning score and test MASE need not agree.
get_insight(kind="forecast_series") — every series: its model, validation and test MASE and sMAPE, why it is weak, its band width per level — which series are hard to forecast.
get_insight(kind="forecast_coverage") — each level's coverage on every backtest window and pooled, with the points behind it: the evidence for the band the person plans with, beyond one test window.
get_insight(kind="forecast_diagnostics") — failures, preprocessing notes, seasonality strengths.
get_insight(kind="forecast_ensemble") — members and weights (autogluon, ensembles).
get_chart_data("forecast", "forecast", experiment_id) and get_chart_data("forecast", "residuals", experiment_id) for the weakest and the largest series (params={"series": …}). Use accuracy_by_distance for error by horizon, metric_over_time for window stability, model_selection for native candidate comparisons, and coverage / group_metrics when their diagnostics answer the question. Read each chart's metric and evaluation basis; a rolling-validation chart is not an independent final-test result.

The drivers, when a tree-backend model won. The run-all chain explains it (its explain stage; for a statistical or foundation winner the stage is skipped, saying why), so a driver question — does a lagged driver, a holiday or a static column carry the signal? — is answered from the explanations, not guessed from the residuals. get_capabilities lists them ready:

get_chart_data("mli", "model_importance", experiment_id) — which lags, calendar features and drivers the model leans on (exact SHAP over the lag matrix, split gain).
get_model_evaluation(experiment_id) — the regression evaluation of the explained series.
get_insight(kind="error_tree") and get_insight(kind="surrogate") — where it errs, and a readable approximation of what it does; get_insight(kind="archetype_summary") and get_insight(kind="case", params={"row_index": …}) — the regimes it sees, one forecast row's why.

Under the default tree_target_transform the boosters fit a transformed target (scaled and differenced): the explanations are in that space, and the diagnostics say so — read lag1 as the previous transformed value. A run explained at deep adds PDP and SHAP interactions.

Then the Gems table (get_playbook("iterate")): a weak series that matters to the business, an uncalibrated level the person plans with, a window whose band missed, a driver the model leans on (or ignores), a residual pattern a driver could explain.

Challenge¶

Check	Reads required
Leakage	the drivers and their `exog_known_future` in `get_run(view="settings")`: is every future value known at the origin? `get_eda(section="leakage")`
Validation design	windows that do not overlap, enough history per series (`get_insight(kind="forecast_diagnostics")`), the horizon's driver values (CV never sees them)
Stability	`get_insight(kind="forecast_series")`: how many series the baseline still wins, and whether they matter
Intervals	`get_insight(kind="forecast_summary")` `intervals`: each level the person plans with calibrated, on what (`calibration_basis` `proxy` is approximate coverage), its test coverage beside its width — a band that buys coverage by width is no finding to praise; `get_insight(kind="forecast_coverage")`: its coverage per backtest window and pooled, and on how many points
Drivers	a tree run: `get_chart_data("mli", "model_importance")` — does the model lean on a driver whose future values are not known, or on the series key?

Decide¶

There is no simulation kind. The operating point is the horizon the plan acts on and the interval level it plans with (the 80 % band for routine orders, the 95 % band for safety stock), stated with that level's test coverage, its pooled backtest coverage and the points behind it (forecast_coverage), width and calibration (approximate when its basis is proxy), and the series it is weak on. A per-series fallback (seasonal-naive for two stores) is not something a registered model serves: never offer it as an operating point — state the weak series as a limitation.

Release and operate¶

Register with register_model(action="register") and its release_record after the release approval: a forecast run is a servable model. Its operating point is documented, not applied (no apply).
Results: get_playbook("schema-forecast") (the *__forecast table, keyed on series and timestamp, one row set per interval level).
Operate: new history is a new plan (run_experiment(action="plan", parent_experiment_id=…)); drift between the training history and the new one is compare(kind="datasets").