Skip to content

Data splitting & validation

AutoSupervised.fit takes a single training frame and carves its own validation internally. Keep final test rows separate. ParleML Studio reserves an outer test before calling fit; a library caller supplies an independent test frame or uses resolve_split explicitly. This page explains validation_holdout_frac and the SplitPolicy rules for IID, grouped and time-aware validation.

In short

  • fit takes one training frame and carves its own validation; it never carves an outer test. Reserve test rows yourself, or call resolve_split before fit.
  • By default ml4 reserves 10% of the training frame (engine.validation_holdout_frac=0.1) to fit calibration, the decision threshold and the other post-fit layers.
  • Set engine.split_policy to GROUPED (with group_column) when several rows describe the same entity, or to TEMPORAL_HOLDOUT (with time_column) when you predict the future.
  • A temporal split needs engine.training_policy="chronological_train_validation" and trains one model.
  • Keep the group and time columns in the frame: ml4 reads them by name and drops them from the model matrix itself.

The single-frame model

model = AutoSupervised(cfg).fit(train_df)     # ONE frame, label column included

fit receives one DataFrame with the label column in it. Model-selection validation and the fitting set for the post-fit layers are derived from that frame. fit does not apply outer_test_frac or carve an outer test. Validation takes one of two shapes, chosen automatically by the deployed model, not by bagging:

  • Bagged out-of-fold (OOF), the cross-validation ensemble. Only when the bag itself is what ships — bagging on and engine.refit_full=False (the high_accuracy preset, for example). Every row then has a prediction from a model that never trained on it, and no data is set aside. Built-in templates pin their fold count and stack depth explicitly; accuracy-oriented templates use one stacking layer before the weighted ensemble. (One qualification, published in measures.json and explained under Bagged OOF vs a reserved holdout: a weighted-ensemble strongest model selected its blend on these same rows.)
  • Reserved holdout. Every other case, including the default. ml4 reserves a slice before fitting and uses it to fit the post-fit layers. A single deployed model (regulated, temporal, or bagging disabled) has no OOF at all — on that train/validation path ml4 may train several candidate families or HPO trials against the one reserved validation slice and then deploy the best candidate; and with refit_full on — the default — the served model is a refit over the rows the bagged children were validated against, so its OOF would not be out-of-sample for it.
cfg.engine.validation_holdout_frac = 0.1      # the default

validation_holdout_frac sizes that reserved slice, and on the shipped default settings the slice really is reserved: 10% of the training frame is withheld from the model. Only refit_full=False on a bagged run avoids it.

The reserved slice is what the decision threshold is tuned on

On an imbalanced binary problem the holdout carries roughly prevalence x 0.1 x n positives, and threshold tuning needs at least ten rows of the rarest class (delivery/fbeta.MIN_CLASS_ROWS_TO_TUNE). Below about 100 positives in the whole training frame the tuner therefore falls back to a degenerate 0.50 cut-off. Either serve the bag (engine.refit_full=False, which restores full-coverage OOF) or enlarge validation_holdout_frac.

Details: how the threshold sweep behaves on a small holdout

That per-class floor of ten is the only floor there is — there is no absolute row count below which the sweep declines. A reserved holdout of twenty rows that happens to carry ten of each class clears the check, and the sweep then returns the exact maximiser over those twenty observed probabilities: reproducible, but decided by two or three rows either side of the cut. (The served value is additionally nudged to sit between two adjacent distinct probabilities rather than on one, which removes a tie without changing the decision set; that is not shrinkage and does not make a thin sweep less thin.) Read a threshold from a small holdout as a rough position rather than a tuned value, and prefer the bag when the frame is small.

One objective is regularised for it, and only one: accuracy. Under 0-1 loss on calibrated probabilities 0.50 is the optimal cut, which makes it the accuracy sweep's theoretical answer and the only defensible thing to shrink toward. So that sweep serves 0.50 + (t - 0.50) x n / (n + 50) — 37% of the move at 30 rows, 80% at 200, 95% at 1000, 99.5% at 10 000, i.e. a real dataset gets the exact argmax and only a thin holdout is pulled back. The prior weight of 50 is AutoGluon's own absolute row floor for threshold calibration, reused as a weight rather than as a refusal: importing it as a floor would serve a hard 0.50 under an F1 objective, which on imbalanced data is frequently an all-negative prediction scoring zero.

Two things it deliberately does not do. The F-beta sweeps are not shrunk — 0.50 is not their optimum under any calibration (the F1-optimal cut on calibrated probabilities is F1*/2), so pulling them toward it would move them away from the right answer. And the accuracy shrink is skipped entirely when the model's probabilities do not straddle 0.50: ml4 ships a calibrator only when its bake-off wins, so this layer can be handed the raw output of a systematically under-confident model that never reaches 0.50, and shrinking that would walk the served cut above every score the model emits and predict one class for every row.

A threshold that could not be tuned raises a notification

The fallback raises a warning event on the run's notifications feed, titled The decision threshold was not tuned, so a served operating point nobody measured never looks like a tuned one. It names how far short the rare class fell, the degenerate value being served, and the two remedies above. The multiclass layer raises the same event for the classes whose one-vs-rest sweep refused, naming the median tuned bar the served rule divides them by instead.

Do not confuse it with the outer test fraction, split_policy.outer_test_frac (the launch wizard's "Reserved test set"). They carve different things at different times: the outer test is taken first, off the top of the whole dataset, and is never touched again until the headline score; the validation holdout is taken from what remains, and the post-fit layers — calibration, the decision threshold, the constant-baseline gate, regression intervals — are fit on it.

One isolation policy, several validation slices

Grouped bagging, fold-local encoders and the post-fit holdout use the same declared group/time rules. Their slices serve different purposes and are derived at different stages. If a grouped or temporal model-selection slice cannot be carved, the fit can fall back to AutoGluon's random split and emits Model selection ignores the split's isolation. The reserved post-fit holdout still follows the declared policy.

Reserve an outer test in library code

Use resolve_split before fit to apply outer_test_policy, outer_test_frac and any explicit outer cutoff:

import numpy as np
from parleml_ml4 import AutoSupervised
from parleml_ml4.fe.split_resolver import resolve_split

target = cfg.data.target_column
split = resolve_split(
    cfg.engine.split_policy,
    df.drop(columns=[target]), df[target],
    is_classification=cfg.ml_task.is_classification,
)
train_df = df.iloc[split.inner_index].reset_index(drop=True)
test_df = df.iloc[np.flatnonzero(split.outer_test_mask)].reset_index(drop=True)
model = AutoSupervised(cfg).fit(train_df)
report = model.evaluate(test_df.drop(columns=[target]), test_df[target])

This assumes a non-empty outer test and a configured target. The default outer_test_frac=0.2 reserves 20% when this resolver is called; the default 10% post-fit holdout is then carved from the remaining training frame.

Where the data cost is recorded

The reserved slice costs rows, and no screen in the product shows that cost. The artifact does. Every run writes a holdout block into measures.json, and it is always a dict — so "calibrated on bagged OOF" and "nothing was recorded" are never the same reading:

"holdout": { "basis": "holdout", "fraction": 0.1, "train_rows_reduced_by": 180 }
Field Path Meaning
basis always "holdout" (a reserved slice the deployed model never trained on) or "oof" (the bagged out-of-fold pass). Names the rows every post-fit layer was fitted on.
fraction holdout only The reserved slice as a share of the training frame.
train_rows_reduced_by holdout only How many rows the model did not get to train on. This is the data cost, in rows.
ensemble_weights_selected_in_sample OOF only true when a weighted-ensemble strongest model picked its blend on the same out-of-fold rows — the one qualification basis: "oof" on its own would read past.
copied_classes_excluded OOF only, when any The single-row classes the fit copied so every bag fold could see them. Each such row's out-of-fold prediction comes from a model trained on its copy, so their rows are left out of the post-fit fitting set.

fraction and train_rows_reduced_by are absent on the OOF path rather than zero: the out-of-fold pass spans every training row, so there is nothing reserved to report.

Details: bag_folds is the fold count that trained

A bagged grouped run bags over the split's own whole-group folds: ml4 hands AutoGluon the split's fold index through its validation_structure, so each bag fold validates exactly one of those folds and no group sits on both sides of one. The fold count is therefore split_policy.n_folds — engine.num_bag_folds does not apply, and the run warns when it differs — and num_bag_sets is 1, because repeating one partition would only refit the same folds.

Two rules can still train fewer folds than that: the split's fold builder caps its folds at the rarest class's row count (every fold needs one of each), and on a binary target AutoGluon merges folds until every validation fold holds the rarer class. Either way the run warns (The grouped split trains fewer bag folds than declared), and measures.json carries bag_folds, read off the fitted trainer rather than off the config. 0 is a real answer there (the run did not bag); the key is absent when the question could not be answered at all, never guessed.

SplitPolicy

SplitPolicy is the declared fold geometry. Set it on the engine section:

from parleml_ml4.contracts.split_policy import SplitPolicy, SplitStrategy

cfg.engine.split_policy = SplitPolicy(
    strategy=SplitStrategy.GROUPED,
    group_column="customer_id",
    n_folds=5,
)

The strategy picks the partitioning scheme; the other fields supply the columns and parameters that scheme needs.

The three strategies

Strategy Validation scheme Required column
IID_STRATIFIED Random stratified K-fold on the target (the default). —
GROUPED Whole groups stay on one side of the split. group_column
TEMPORAL_HOLDOUT A single out-of-time cut — train on the past, validate on the future. time_column

Fields

Field Applies to Meaning
strategy all The partitioning scheme.
group_column GROUPED The isolation key (customer / household / device).
time_column TEMPORAL_HOLDOUT The event-time axis. Parsed before it is ordered — see below.
n_folds GROUPED (default 5) Number of group-preserving CV folds — on a bagged run, exactly its bag folds.
embargo TEMPORAL_HOLDOUT Optional safety gap: purge the latest train rows before the validation period, e.g. "7D". Does not trim the outer-test boundary.
outer_test_policy all Whether an outer test is carved, not how: NONE carves none, RANDOM carves by the strategy's own rule, TEMPORAL asserts that rule is chronological.
outer_test_frac all (default 0.2) Outer-test fraction.
validation_cutoff TEMPORAL_HOLDOUT Optional explicit calibration boundary (ISO date or numeric clock). Rows after it are reserved for calibration/threshold fitting; model selection uses a separate earlier chronological slice.
outer_test_cutoff TEMPORAL_HOLDOUT An explicit validation/test boundary (ISO date or numeric clock).
allow_multilayer_stacking all False trains no stack layer. A TEMPORAL_HOLDOUT run never stacks whatever it says — see below.
seed all (default 0) The split's own seed: the K-fold shuffle and the random outer-test carve. Separate from execution.seed, which seeds the models.
Details: the time axis is parsed, not sorted as text

time_column may be a datetime column, a numeric clock (epoch seconds, a period index), or date strings. Strings are parsed with pd.to_datetime before the axis is ordered, because ranking them as text sorts lexicographically: "03/01/2024" lands before "12/01/2023", and every out-of-time cut built on that order would run backwards in time — training on the future, validating on the past, with a normal-looking score and no error. ISO-8601 (YYYY-MM-DD) sorts correctly as text either way.

Values that cannot be parsed are ranked first, so they stay on the train side of every cut — the same rule outer_test_cutoff applies, since a row with an unknown time cannot be proven to fall after a boundary. The count is logged.

Ties are never split, because splitting one would leak across the boundary the cut exists to enforce. When the requested fraction would therefore swallow every row, the carve falls back to the next distinct timestamp — the largest cut that still leaves a training set — and warns that the realised size differs from the request. Only a column with a single distinct timestamp, or none that parse at all, raises.

A tz-aware time column is normalised to UTC before any width is measured. Stripping the zone without converting keeps local wall time, which skips or repeats an hour at a daylight-saving transition — enough to under-apply an embargo, which is a leak.

A class that lands entirely in the reserved holdout

Classification only, and only on the reserved-holdout path (single-model or refit_full runs — see Bagged OOF vs a reserved holdout). It happens under temporal, whenever a class emerges late — a new product tier, a new fraud pattern, a re-coded status, exactly the case an out-of-time split is chosen for. A class confined to the latest rows cannot move to the training side without breaking the chronology that makes an out-of-time test meaningful, so ml4 refuses rather than repairing.

The other strategies repair it instead, each moving the smallest amount that stays legal. Grouped carves hand back the whole group holding the stranded class — moving a single row would split that unit across the boundary, which is the precise leak those carves exist to prevent. The stratified fallback (reached when a class has fewer than two rows, which defeats stratification) returns one row, since an IID carve has no ordering or grouping constraint to preserve. The one case that still refuses is a carve so small that handing the unit back would leave no holdout at all — with nothing on the carved side there is nothing for the post-fit layers to fit on.

When the class cannot be kept on the training side legally, ml4 raises. Every post-fit layer is fitted on that holdout, so a class the model never trained on does not merely go unscored: probability calibration fits its curve against rows the model cannot assign to their true class, the multiclass thresholds are tuned on the same evidence, and both ship. Merge or drop the class, or choose a strategy that can place it on both sides.

The outer test is treated differently — it feeds no layer, so its unseen-class rows are disclosed instead of refused, as test.unscored_rows in measures.json and a banner on the Summary tab's per-class table.

Validation recipes

The out-of-the-box scheme. Random, target-stratified K-fold — the right choice when rows are independent and exchangeable.

from parleml_ml4.contracts.split_policy import SplitPolicy, SplitStrategy

cfg.engine.split_policy = SplitPolicy(strategy=SplitStrategy.IID_STRATIFIED)

IID is also the only strategy for which conformal prediction intervals are exact; grouped and temporal splits make interval coverage approximate (and ml4 discloses this in the measures).

Keeps every row of a group (a customer, household, or device) on the same side of the split, so the model is never validated on a group it trained on — the classic leakage guard for repeated-entity data.

cfg.engine.split_policy = SplitPolicy(
    strategy=SplitStrategy.GROUPED,
    group_column="customer_id",
    n_folds=5,
)

The stacked-overfitting probe is off by default here

Stacking itself is unaffected — ml4 hands AutoGluon a group-pure fold index, so the out-of-fold columns a stacker trains on honour the isolation key. What is turned off is dynamic_stacking, the optional probe that spends a whole extra fit deciding whether to keep the stack. "auto" (the default) resolves to False on a grouped split; cfg.engine.dynamic_stacking = True runs it. The probe holds out one whole bag fold and bags its sub-fit over the others, so it never splits a group — and it needs at least three folds: below that the stack layer trains without the probe, and the run warns.

Respects chronological order. TEMPORAL_HOLDOUT makes a single out-of-time cut — train on the past, validate on the future. Use embargo to purge rows near the boundary so near-duplicate events don't leak across it.

cfg.engine.training_policy = "chronological_train_validation"
cfg.engine.split_policy = SplitPolicy(
    strategy=SplitStrategy.TEMPORAL_HOLDOUT,
    time_column="event_date",
    embargo="7D",
)

A temporal split trains one model, under the chronological policy

A temporal run trains a single model validated on a strictly-later chronological slice, so it has no out-of-fold predictions and no bagged or stacked ensemble. It pairs only with engine.training_policy="chronological_train_validation": any other policy on a temporal split is refused when the config is built or the fit resolves it, with a message naming both fields — as is the chronological policy on a split that is not temporal. You never get folds shuffled across time, and translate_config().fit_kwargs never advertises a bag or a stack the run cannot spend.

There is no rolling-origin option here

A temporal split is one out-of-time cut; ml4 does not bag a tabular run over forward-chaining folds.

Forward-chaining does still happen where it can: fold-local feature engineering rolls its origin forward whenever you declare a time_column, so a target encoder never sees a row's own future. That is driven by the time column, not by the split strategy.

For genuine rolling-origin backtesting, use the forecasting entry point (see below), which is built around it.

Leak-safe by construction

SplitPolicy validates itself on construction and rejects mismatches that would silently leak rather than ignoring them:

  • GROUPED requires group_column; the temporal strategy requires time_column.
  • group_column is honoured only by GROUPED. Setting it on another strategy is rejected, because the outer carve would isolate groups while the inner CV split a group across the boundary — a silent leak.
  • embargo and outer_test_cutoff only make sense with a time axis, so they are rejected on non-temporal strategies.
  • outer_test_policy=TEMPORAL requires a temporal strategy, so a temporal outer test can't silently carve at random.

The training policy must fit the split as well, and ml4 refuses the pairings that cannot: a temporal split needs engine.training_policy="chronological_train_validation", that policy needs a temporal split, and a regulated roster (roster.regulated) cannot use cross_validation. The config checks this when it is built and the fit again when it resolves its shape, and each refusal names the fields to change.

Encoder CV uses the same policy — there is no second set of columns

SplitPolicy governs the model validation partition and the leak-safe cross-validation used by the target encoders: the target encoders are fold-local steps, re-fit inside each fold that parleml_ml4.fe.split_resolver derives from this policy. There is no separate encoder-CV column pair on DataConfig. Set the columns here, once. See Feature engineering.

The split columns stay in the frame — and are excluded from the model matrix

group_column and time_column are read out of the training frame by name: parleml_ml4.fe.split_resolver resolves the group key and the time axis from X on every fold, outer-test carve and embargo, and raises SplitPolicy <field>=… is not a column of X when one is missing. So do not drop them before fit — a run without them does not fall back to random splitting, it fails.

They are, however, not modelled. Once the fold geometry is resolved — the group folds compiled into the bag-fold index AutoGluon bags over, the policy-carved validation slice cut — fit_predictor drops both from the training frame and from that validation slice before AutoGluon sees either (parleml_ml4.engine.fit_geometry.non_feature_split_columns), and says so at INFO:

Split columns ['customer_id'] are structural, not features: they carry this run's
grouped geometry, are declared in no feature or domain role and are read by no
recipe step, so they are dropped from the model matrix after the folds are resolved.
Details: why the split columns are dropped

The GLOBAL feature generator deliberately routes every remaining frame column through to AutoGluon: a column that stays is a feature. Left in, a group_column would become a high-cardinality identity feature whose levels are all unseen at serving time, and a time_column a raw time feature on exactly the runs whose validation is the future, where a fitted time trend extrapolates rather than generalises. Neither would inflate the validation score — the split isolates the very structure those columns carry — but both would spend capacity on a column declared for bookkeeping. Whether it happened would also depend on the warehouse's dtype: AutoGluon's type router drops a datetime64 time column on its own, while an integer one would be modelled.

The dual-role case is preserved. A split column is kept whenever the run has a second, modelling reason to read it:

  • it is declared in data.num_features / data.cat_features;
  • it is named by any data.*_domain role list, data.base_time_columns, data.key1 / key2, or the temporal.* role set (when temporal.enabled);
  • it carries a constraints.monotone_constraints pin;
  • a planned recipe step reads or emits it — the hard constraint, since a GLOBAL transformer raises input column '…' not in frame if its input disappears.

So a time_column you also put in data.time_domain still gets its calendar and days-since features, and the raw column still reaches them.

The drop applies to the model matrix, not to serving: because the columns are absent from the fitted feature list, a scoring frame that still carries them is ignored rather than rejected.

The encoders still get the key — beside the frame, not in it. The out-of-fold target encoders build an inner split inside each bag fold, and that split has to honour the same isolation key, which the drop above has removed from the fold's frame. So fit_predictor lifts the key out one step earlier (parleml_ml4.engine.fit_geometry.isolation_key_frame, before the drop), hands it to each model as a run-scoped sidecar, and the fold-local chain rejoins it by row index for the rows that fold is fitted on. Nothing about it is a feature: no model sees the column, and serving never receives it. ml4's own hyperparameter search applies the same exclusion and carries the same key, so the search fits the matrix the run will fit.

Details: how the key is carried and removed

The key travels as an AutoGluon hyperparameter, because that is the only channel into a model AutoGluon clones per bag fold — and a hyperparameter is kept, so at the end of the fit ml4 takes it back out of every persisted object (each bag child, the bag template, and predictor.pkl's fit_hyperparameters_). It is removed after the fold children and the refit_full twin have used it, and before anything is shipped: a saved model carries no row-level key, so nobody's customer ids travel with an artifact that gets copied to a registry or a serving host.

Row labels are the join, so they must be unique

ml4 does not re-label your training frame — AutoGluon stores its out-of-fold predictions under the labels you hand it, and rewriting them would mis-pair the OOF vector that fits calibration, the decision threshold and the conformal intervals. A frame with duplicate row labels and a declared isolation key is therefore refused, naming the frame, rather than silently encoding one row against another's key. Reset the index before fitting.

Forecasting: rolling-origin backtests

Time-series runs do not use SplitPolicy — forecasting model selection is a rolling-origin backtest, and every CV site (all four backends' model selection, the held-out test window, tree HPO inner folds) derives its folds from one shared grid, parleml_ml4.forecasting.splitting.CVGrid: per-series anchored windows, all-or-nothing eligibility, and an optional embargo (cv_gap). See Forecasting → Backtest geometry.

Where to go next