Troubleshooting & FAQ¶

Common messages, operational states, and how to resolve them.

In short

Each section is headed by the message or symptom you see; find yours, read the first paragraph, and open a Details block only when you need the mechanism.
A failing baseline verdict (no_signal, worse_than_constant) is a signal, not a crash: the run still finishes.
Config refusals (task vs label, training policy vs split, execution.device="cuda") stop the run before any work, and each message names the field to change.
A stale artefact is refused at load: re-fit and re-persist with the ml4 you run.
When a run misbehaves, read its events.ndjson and manifest.json first.

The baseline gate verdict¶

Every supervised run ends with a baseline honesty gate that asks: did the model actually beat the best predictor that uses none of the features — a constant (any observed class label / mean / median / class prior) or a chance-level draw from the target's own distribution — by the configured margin? Every candidate is scored and the strongest one becomes the floor; the winner is reported as strategy, and the verdict lands on model.baseline and in measures.json under baseline_gate — which is its single published home, carrying the whole BaselineReport verbatim:

`verdict`	Meaning	Passed?
`ok`	Beats the baseline by more than the margin, and a paired bootstrap of the graded rows confirms the win.	yes
`suspected_leakage`	Removes almost all of the baseline's error — implausibly good.	yes, but investigate
`no_signal`	Beats the baseline by less than the margin, or by a margin a paired bootstrap of the graded rows cannot confirm (then `low_confidence`).	no
`worse_than_constant`	Scores below the baseline.	no
`skipped`	The gate could not run (no scorable baseline), or the baseline is already perfect so there is no error left to remove.	—

A failing verdict is a signal, not a crash — the run still finishes. Usual causes and fixes:

any verdict with low_confidence: true — baseline_gate.low_confidence_reason says why. "few_rows": fewer than 30 rows were graded (baseline_gate.n_rows), so the gate is measuring the holdout as much as the model. "unconfirmed_win": the point estimate cleared the margin but the paired bootstrap's lower bound (baseline_gate.improvement_lower) did not clear zero — a win too small to tell from noise on these rows. "few_rows" wins when both apply. Either way the verdict is published but never acted on: baseline.fail_on_no_signal will not abort a low_confidence run. A 200-row dataset on the defaults leaves the gate 16 rows; see Training workflow.
no_signal — no learnable signal in the features for this target, a noisy target, or a task / metric mismatch. Add more informative features, or pick a metric that matches the problem (see Imbalanced data).
worse_than_constant — often a target / metric mismatch. Re-check ml_task and eval_metric.
suspected_leakage — a feature encodes the target (a post-outcome column, an id that maps 1:1 to the label). Audit the feature list.
skipped — there was nothing to compare against, so read it as "no verdict", not "cleared". A constant target always lands here: a correlation metric can score none of the candidates, and an error metric's constant floor is already exact. So does a metric the fitting set cannot support (AUC with one class).

Details: which rows the verdict rests on — and whether the strongest model's score is a measurement

The gate is fitted alongside calibration and the decision threshold, so it grades the fitting set: either the bagged out-of-fold pass over every training row (baseline_gate beside holdout.basis == "oof") or the reserved holdout ("holdout"). It is not one of the train / valid / test splits, and any surface that calls it "validation" is describing rows no such split contains.

strongest_basis qualifies the strongest model's own number in the same block, and it is the field to read before quoting it:

`strongest_basis`	What it means
`untuned`	No decision parameter was chosen from these rows.
`out_of_fold`	The operating point was re-selected fold-wise before scoring — an honest estimate.
`in_sample`	Too few rows to re-select, so the score is an upper bound, not a measurement.

The gate is weak on pearson / spearman

A constant has zero variance, so a correlation metric can only be floored by the shuffled draw — and one shuffle is as noisy an estimate of zero correlation as a zero-skill model is. Measured over 10 seeds, the gate catches a pure-noise strongest model about half the time (a strong model was graded ok 10/10). An ok on these two metrics is weak evidence, not a clearance.

Is there any signal at all?

Before a full fit, run the Instant Signal Check — a zero-shot probe that answers "is there learnable signal here?" in seconds.

A single model shipped, but the leaderboard shows the ensemble on top¶

Expected, and deliberate. The leaderboard is the training record — it keeps every model AutoGluon built, including the weighted ensemble, with the score that won it the top row. Which model ships is a separate decision, taken after the fit by the ensemble advantage margin: the blend has to beat the best standalone model by more than engine.ensemble_advantage_margin (default 0.005, i.e. 0.5% of that model's score) or the simpler model is deployed instead.

To confirm that is what happened, and see the arithmetic:

the run log carries ENSEMBLE_GATE_DOWNGRADE with both scores and the relative gain;
ml4_ensemble_gate.json, inside the predictor directory, carries the same verdict with a reason;
the Ensemble tab says so directly, and the composition sidecar's ensemble_gate block holds the numbers.

The most common trigger is not a close race at all: AutoGluon frequently puts all the blend weight on one base model and still ships a WeightedEnsemble_L2 wrapper around it. Its score is identical to that member's, so the relative gain is exactly 0.0 and the wrapper is removed for free.

Set engine.ensemble_advantage_margin = 0.0 to opt out and always deploy whatever AutoGluon scored highest. Raise it if your run-to-run score spread is wider than 0.5% — measure it rather than guessing, by refitting at several execution.seed values.

"engine.training_policy=… cannot run on engine.split_policy.strategy='temporal_holdout'"¶

The training policy and the split must fit together, and ml4 refuses the three pairings that cannot — when the config is built, again when the fit resolves its shape, and at launch in the product, which answers 400 with the same message:

The message says	Change
`engine.training_policy='…' cannot run on engine.split_policy.strategy='temporal_holdout'`	Set `engine.training_policy="chronological_train_validation"`: a temporal run trains one model and validates it on its latest rows.
`engine.training_policy='chronological_train_validation' needs engine.split_policy.strategy='temporal_holdout'`	Attach a temporal split (a `time_column`), or use `train_validation` / `cross_validation`.
`roster.regulated=True trains one transparent model, so it cannot use engine.training_policy='cross_validation'`	Set `train_validation` (or `chronological_train_validation` on a temporal split).

A template that uses the chronological policy may carry the default random split until a launch attaches the dataset's temporal split; only the attached split is checked against it. See Data splitting & validation.

"MaxMind GeoLite2 databases not found"¶

IP (ip_domain) features need the MaxMind GeoLite2 databases, which aren't bundled. This is non-fatal: geo / ASN features degrade to null with a warning while the stdlib classification flags still emit. To enable geo lookups, download the databases and set MAXMIND_DB_DIR (the geoip2 reader ships with ml4); describe_domains() then reports maxmind as available. To skip IP features entirely, leave ip_domain=[]. Full steps: Geography & reference data.

"N engineered column(s) shipped degraded"¶

A run-level warning, not an error: the pipeline produced those columns, but not as specified. Read the cause, because the two kinds need different responses.

A step that could not run at all leaves an inert column — a missing GeoIP database, an unreachable embedding backend, usaddress absent or a non-US country, a degenerate Gamma-Poisson vocabulary. AutoGluon drops constant columns, so the model was fitted without them rather than on dead width. Fix the cause and re-fit, or drop those columns from the config so the recipe stops promising features this environment cannot produce.

A step that ran in a reduced mode leaves the column in place, worth less than its name implies. The one to know is TargetEncode: target encoding is not out-of-fold — the slice had too few rows for a four-way inner split, or a class too rare to stratify, so that encoding is self-encoded and the model will over-trust it. Give the run more rows per fold.

The wording "group column '…' is not in the fold frame" belongs to this family, but no config can cause it: the isolation key is carried to the encoders out of band (see the split columns), so it reaches them even though the column is correctly absent from the model matrix. If you do see it, it is an ml4 defect worth reporting, not a setting to change.

Read the notice before you conclude the data is weak: a run fitted without its embeddings scores worse for a reason that has nothing to do with the dataset. See When a transformer degrades.

"Target 'y' has N classes but the task is binary_classification"¶

The declared task and the label disagree, and preflight_check refuses the run before any work is done — at launch it comes back as a 400, so the wizard says it rather than a run failing minutes later.

Two mismatches are possible and both are refused: more than two classes under binary_classification, and exactly two classes under multiclass_classification.

Details: why ml4 checks the task against the label

ml_task is declared, never inferred. It becomes AutoGluon's problem_type verbatim, and an explicit problem_type makes AutoGluon skip both its own validate_label and infer_problem_type — so nothing downstream compares the declaration back to the data. Without the check:

Message	What would happen without the check
`has N classes … but the task is binary_classification`	The run would carve its holdout, plan the recipe and pay for the whole feature-engineering and feature-selection pass, then die inside AutoGluon's label cleaner on `y does not contain exactly 2 unique values` — a bare tuple-args `AssertionError` with no hint of what was wrong.
`has exactly 2 classes … but the task is multiclass_classification`	It would train, as the identical binary model (AutoGluon's `LabelCleanerMulticlassToBinary`), and quietly change everything around it: the served rule would be the per-class one-vs-rest sweep, whose two thresholds combine into an operating point neither sweep tuned, `positive_class` would be `None`, and `measures.json` would lack its `threshold` and `brier` blocks.

Neither is a data defect — the fix is the task dropdown, and the message names the value to switch to. The usual way in is a clone or a rerun carrying a stored ml_task onto a different target: the wizard prefers the saved declaration over the fresh guess from the target's EDA stats.

If the target genuinely has more than two levels and you want a two-class model, collapse it in the dataset (or with a derived column) first — "binary on a 3-class column" is a config error, not a different model.

"The reserved holdout is the only place N class(es) appear"¶

A hard error, and deliberately so. On the reserved-holdout path (single-model or refit_full runs) every post-fit layer — probability calibration, the decision thresholds, the baseline gate — is fitted on that holdout. A class that lands entirely there is one the model never trained on, so those layers would be tuned against rows the model cannot classify, and would ship that way.

Raised only where the split structurally cannot place the class on both sides: a temporal strategy with a class that emerges late (a new product tier, a new fraud pattern, a re-coded status), or a grouped one with a class confined to a few whole groups. An IID carve repairs itself instead, so it never raises here.

Merge or drop the class, or switch to a strategy that can place it on both sides. The message names the classes.

The outer test is not gated this way — it feeds no layer, so its unseen-class rows are disclosed instead: test.unscored_rows in measures.json, and a banner on the Summary tab's per-class table explaining why the support does not sum to the held-out row count.

"N class(es) have fewer than engine.label_count_threshold=K rows and were excluded"¶

A run warning at the preflight stage, and the result of a setting you chose. engine.label_count_threshold (multiclass only) is the fewest rows a class needs, counted over the whole training frame, to be modelled at all. A class below it is excluded — together with its rows — before the holdout is carved and the recipe is planned, so the model never predicts it. Rows of an excluded class that the model is later asked to score are counted under test.unscored_rows in measures.json, the same treatment a never-seen class gets. The default of 1 keeps every class.

If the exclusion leaves fewer than three classes the run is refused instead: lower the threshold, or declare binary_classification if two classes is what you mean to model. Raising the threshold never rescues a rare class — it removes more of them — and a class with a handful of rows has no honest generalisation estimate to report either way; the remedy for one you need is more rows, or a coarser level.

Why ml4 settles the class space, not AutoGluon¶

ml4 applies one rare-class rule for every metric and tells AutoGluon to delete nothing (label_count_threshold=1 in its learner kwargs).

Details: AutoGluon's own rare-class rule

AutoGluon's own rule depends on the metric and on the fold geometry: on a bagged fit it keeps a class only with two training rows, and it copies a rare class up to that floor only for a probability metric (logloss). Left to it, a hard-label metric (accuracy, f1_macro, …) would delete a class with a single training row after the recipe had already pinned one encoder slot per class — every model would then refuse to fit and the run would train nothing.

A class left with a single training row:

Run	What happens
Bagged, no groups (the default)	The row is copied once, so every fold can see the class. The row and its copy land in different folds, so each one's out-of-fold prediction comes from a model trained on its twin. The run warns (Single-row classes were copied for bagging) and leaves those rows out of the rows calibration, the decision threshold and the baseline gate are fitted on; `holdout.copied_classes_excluded` in `measures.json` names the classes. The stacker and the weighted ensemble still learn from them, so those classes' validation scores are optimistic.
Bagged over a grouped split	Refused before anything is fitted: a copy would put one group on both sides of a fold. The message names the class — collect more rows, merge it into a coarser class, drop its rows, or use a train/validation policy.
Not bagged (train/validation, regulated, temporal)	Kept as it is: one row is AutoGluon's floor there.

"AutoGluon dropped N class(es) with too few examples to train on"¶

A tripwire. With the rule above AutoGluon deletes no class, so this warning at the ag_fit_done stage — like the model error "AutoGluon is training on N classes but the feature recipe pinned M", which says outright that it should be unreachable — means a class slipped through: please report it with the run's class counts.

If it does fire, the run still says so honestly. AutoGluon pads a deleted class back into every probability frame as an identically-zero column, so on its own it would look like a class the model scored and always got wrong; instead:

Where	What you see
Run events / log	A `warning` event naming the classes, at the `ag_fit_done` stage.
`measures.json`	`untrained_classes` at the top level (present even when the run has no outer test, and carried in the deploy bundle).
`measures.json` → `test`	`untrained_classes` again, and those rows counted under `unscored_rows` — the same treatment a never-seen class gets. Dropping the column is arithmetically free: it is zero, and the remaining columns still sum to 1.

A model you asked for is missing from the leaderboard¶

measures.json lists three different things about the roster, and reading only the first one is what makes a failed model look like one that was never requested:

Key	Meaning
`requested_models`	What the run asked for — the resolved roster, in ml4's spec vocabulary (`GBM:gbdt`, `CAT:symmetric`, …), before anything was fitted. Absent on a bundle reloaded without it, never an empty list.
`models`	The survivors: `predictor.model_names()`, what actually trained and can be served.
`model_failures`	What AutoGluon tried and could not fit, one entry per failure with `model` / `exc_type` / `model_type` / `total_time`. Absent when nothing failed.

Details: the two lists are in different vocabularies — match them by eye, not by set difference

requested_models holds ml4 specs; models and model_failures[].model hold AutoGluon's names for the fitted objects, which add the pipeline shape. A lone GBM:gbdt is LightGBM_BAG_L1. The variant is spelled out only when the roster names more than one variant of that family, in which case every one of them takes a suffix — a tuned_variants run reads LightGBM_GBDT_BAG_L1 / LightGBM_DART_BAG_L1 / LightGBM_RF_BAG_L1 (ag_args.name_suffix, so the board says which variant rather than AG's positional LightGBM_2). One requested spec can also produce several rows — one per stack level — and the WeightedEnsemble_L* combiner is a row no spec asked for.

So a spec with no corresponding row in models either failed (look for it in model_failures, whose exc_type usually names the cause — a memory ceiling, a missing optional dependency, a degenerate fold) or ran out of the run's engine.time_limit before its turn. Neither case is silent, but neither is fatal either: AutoGluon ships the models that did fit.

"execution.device='cuda' was requested but … cannot train on a GPU"¶

execution.device = "cuda" is an assertion, not a preference: when a booster library cannot actually take the device, the run aborts naming that library, why it cannot, and what to install.

execution.device='cuda' was requested but lightgbm cannot train on a GPU in this
environment: this LightGBM build has neither the OpenCL nor the CUDA GPU learner.
The default LightGBM wheel is CPU-only. Build the OpenCL backend, e.g. …
Set execution.device='auto' to fall back to CPU per library instead, or 'cpu' to
disable the GPU path entirely.

This is deliberate — a four-hour run that silently drops to CPU is a worse failure mode than a clean abort at second zero. Two ways out, depending on what you meant:

you want the GPU if there is one → set execution.device = "auto", which falls back to CPU and logs the decision once;
you meant it → install the GPU-capable build the message names.

"Incoherent config: execution.device='cuda' with hpo.provider='ml4_optuna' and a CatBoost variant"¶

A separate refusal, and not about your hardware. ml4's hyperparameter search scores its CatBoost trials on CPU-pinned fits, so a winning configuration can contain values that cannot be trained on a GPU at all (MVS bootstrap, which GPU CatBoost rejects) or that GPU CatBoost silently ignores (rsm, which would make the reported winner a lie about the fitted model). Because AutoGluon's fit-level num_gpus is a total for the run and ml4 emits no per-family override, there is no way to send just the tuned CatBoost to the CPU. Drop the CAT:* variants from roster.models, choose another hpo.provider, or use execution.device = "auto" and accept a CPU fit for this roster.

See GPU acceleration for how availability is established per library and why the decision is all-or-nothing across the roster.

"The active balancing methods require numeric transformed attributes"¶

SMOTE, ADASYN, Borderline/K-means/SVM SMOTE, NearMiss and Cluster Centroids compute distances in numeric space. Choose SMOTENC/random sampling for mixed data, or enable Use numeric attributes only to remove remaining non-numeric columns after fold-local encoding. Validation and inference then receive the same projection. See Imbalanced data & outliers.

Text embeddings are skipped¶

If embedding features don't appear, the local model's dependency is missing. Probe it before a run:

from parleml_ml4.embeddings import TextEmbeddingConfig, source_available

ok, reason = source_available(TextEmbeddingConfig())
print(ok, reason)

With text_embedding_mode="augment" the TF-IDF path still runs; the supervised embedding encoder emits inert (zero) columns rather than failing, and the run reports the degradation. A provider source fails the run instead: check that the connection exists and that a credentials resolver is installed (parleml_ml4.embeddings.set_provider_resolver). See Text & embeddings.

"Scoring stopped: the provider changed the model behind the same name"¶

The provider returns different vectors for the fingerprint sentences than when the model was trained (EmbeddingModelChanged). Every prediction on those vectors would be silently wrong, so ml4 refuses. Retrain the model on the provider's current model, or point the connection at a deployment that still serves the original one.

"Scoring stopped: … at revision …; this machine has revision(s) …"¶

The local text-embedding model's pinned revision is not on this machine and cannot be downloaded (LocalModelMismatch) — typically an offline server whose cache holds another snapshot. Download the pinned revision into the model cache (HF_HOME), or retrain.

"KNN imputation downgraded to median"¶

numeric_imputation="knn" is capped at a fixed 50 000 training rows. Above that the run fits median instead and says so in the Notifications tab — the substitution is real, so a config left on knn does not describe the model.

The cap is not a setting, and nothing raises it. Your options are to set numeric_imputation="median" so the configuration states what actually runs, or to train on fewer rows if the multivariate estimate is worth the cost.

Details: why the cap exists, and what it is not

KNNImputer keeps the whole training matrix inside the fitted object and rescans it for every scored row, so fit, serve and artefact cost all grow superlinearly (measured on a 10-column frame: 0.5 s at 500 rows, 21 s at 8,000, 97 s at 20,000).

Two things this cap is not:

A privacy control. A knn artefact embeds raw training rows at every size (~19 MB at 50,000×50), so "the scorer must not carry training data" means "don't choose knn", not "stay under the cap".
Relevant to tree models. numeric_imputation is read only by the glass-box linear path (regulated_kind of glm / scorecard / rulefit); AutoGluon's boosters fill NaN natively and never consult it, so no downgrade warning is raised there — the message below is raised instead.

See Full parameter reference.

"The imputation section is set … but this run's roster does not read it"¶

The whole imputation section — numeric_imputation, cat_imputation, the ±inf pair, and the per-column custom_num_imputations / custom_cat_imputations overrides you set on the Columns tab — rides the glass-box variants alone: regulated_kind of glm, scorecard or rulefit. Everywhere else it is inert, and that includes a regulated ebm run, which bins features natively.

The settings are not substituted, mangled or half-applied; they simply do not run. The gradient boosters route NaN natively (a learned missing direction per split), and AutoGluon's own preprocessing imputes at median for the linear and neural families. So nothing about the model is wrong — but the run's stored config records an instruction that never executed, which is why the run says so.

Two ways to resolve it: choose one of the three glass-box kinds if you want those imputers, or clear the section so the configuration describes what actually ran.

"engine.sample_weight names a weight column that is ALSO declared as a model feature"¶

A weight column has to be in the training frame but not a feature, and that is a role no other column has. This error means the column landed in data.cat_features or data.num_features as well — usually because whatever built the config classified every column that was not the target, a key or explicitly ignored as a predictor.

Remove it from the feature lists and leave engine.sample_weight pointing at it.

Details: why the double declaration is refused

There is no configuration in which the double declaration is what you meant: AutoGluon lifts the weight column out of the frame before any feature engineering runs, so as a "feature" the column burns a feature-selection slot, is mined into interaction pairs that can only materialise as all-NaN, and is planned into the GLOBAL recipe — which then indexes a column that is no longer there.

The mirror-image mistake produces a different error. Marking the column ignored so that it will not be modelled generally drops it from the frame, and AutoGluon then raises a missing-column error for the weight column it was told to extract. Neither "feature" nor "ignored" is right; see Passing a weight column from a host application for the four conditions a caller has to satisfy.

Two related refusals come from the same translator, both up front rather than deep inside a fit: engine.sample_weight="auto_weight" (AutoGluon accepts the value, nulls it, and trains completely unweighted while logging that below ml4's default verbosity), and engine.weight_evaluation=True with no weighting set (a flag AutoGluon reads only when a weight vector exists).

Anomaly detection¶

ImportError on a deep detector — autoencoder / vae / dif need pyod[torch]; suod needs pyod[suod]. Install the matching extra or choose a classical detector (ecod, iforest, hbos, …).
Low-quality verdict (result.quality["verdict"] == "low" in auto mode) — the engine isn't confident. Provide more data or constrain the detectors.
Anomaly ratio slightly off the configured contamination — both modes cut at contamination, but tied scores can move the realised share a little; the run reports the share it achieved. See Unsupervised.
A column vanished from the fitted model — a categorical in which ≥ 95 % of rows hold a distinct value is treated as a row identifier and dropped at fit (logged as a warning, listed in preprocessor.dropped_constant_columns_). Keeping one makes neighbour detectors flag every served row. If it is really a timestamp, parse it to a datetime dtype so it is encoded as a quantity instead. See Unsupervised.

Recommendation & Learning-to-Rank¶

A class was dropped — its One-vs-Rest sub-model failed the minimum validation ROC gate (min_submodel_roc). The run falls back to a unified sub-model, or excludes the class (recorded in result.excluded_classes with a reason). Lower the gate or improve features for that class.
Degenerate NDCG of 1.0 in tuning — ml4's Optuna maximises full-catalogue NDCG@k from its own ranking metrics, not the library's sampled-group NDCG, so this shouldn't surface; if a run looks too good, inspect result.best_valid_ndcg. See Ranking.

Forecasting¶

A series doesn't beat SeasonalNaive — its CV winner does not clear a naive seasonal baseline (CV MASE > 1.0, or SeasonalNaive itself won the race). It is surfaced on result.weak_series as an honest warning — never silently swapped or failed. Try a different backend (e.g. tree to exploit exogenous drivers), revisit the horizon / aggregation grain, or accept the baseline.
Foundation / AG-TS backend unavailable — install both forecasting and forecasting-autogluon extras. The pinned AutoGluon-TimeSeries 1.6.3 supports Python 3.11–3.13 within ml4. A run that never selects those backends doesn't import them; an explicit request fails fast with a pip install hint. See Forecasting.

"Why does the model serve raw probabilities?"¶

Five different things produce that state, and measures.json's calibration.outcome says which. Answer from the outcome, not from the absence of a calibration map — the whole reason the outcome is recorded is that "no calibrator shipped" was four different statements about a model.

`outcome`	What happened	What to do
`fitted`	A calibrator won its bake-off; calibrated probabilities serve.	—
`declined`	It was measured and lost, or won by less than `calibration.calibration_advantage_margin`.	Nothing is wrong. Lower the margin only if you have a reason to prefer a marginal calibrator.
`not_measured`	The fitting set was too thin to bake off honestly (see `n_rows`, `rarest_class_rows`).	More data, or a larger `engine.validation_holdout_frac` on a single-model run.
`disabled`	`calibration.apply_at_predict_time` is off. A choice, not a failure.	Switch it on and refit — or, if the run kept a decision basis, re-decide the layer without retraining.
`refit_degenerate`	The bake-off won, and the full-set refit then produced no usable map.	Try `platt` or `temperature`, which are far steadier than `isotonic` on thin data.

"Why does the Scores panel show no baseline?"¶

The constant-baseline check returned skipped, so there is no verdict to show — and a skipped gate must not render as a real number. It skips when:

baseline.enable_constant_baseline is off (a choice, and the panel is quiet rather than empty);
the fitting set has too few rows or a single class for a skill-free predictor to be scored against it at all.

The verdict lives in measures.json under baseline_gate; verdict: "skipped" is the state to look for. Do not read passed on a skipped gate — it is True, which is the one field that looks like a usable filter and inverts the meaning.

"This stage says it was skipped"¶

A skipped stage is not a failure and not a stall. ml4 emits a skipped event carrying a machine-readable reason, so an optional stage that deliberately did not run says so rather than sitting at pending forever:

Reason	Meaning
`disabled_by_config`	You switched it off. Hyperparameter search is off by default, so this is the common one.
`not_applicable_to_task`	The stage does not exist for this task — calibration on a regression run, intervals on a classification one.
`insufficient_data`	It tried and there was not enough data. Distinct from the two above, and the only one that suggests an action.
`budget_exhausted`	A size or time ceiling stopped it — e.g. a decision basis above `deployment.decision_basis_max_cells`.

"The decision layer cannot be re-decided"¶

Calibration, the decision threshold and the baseline check can normally be refitted without retraining. When they cannot, the reason is one of:

Reason	Meaning	Fix
`no_basis`	The run kept no `decision_basis/` — `deployment.persist_decision_basis` was off, or the frame was above `deployment.decision_basis_max_cells`.	Enable it (or raise the ceiling) and refit.
`no_provenance`	The pipeline was not assembled by an ml4 run, so there is no fitted revision to advance from.	—
`task_not_supported`	Regression. Its post-fit layer is the prediction interval, a different object with its own basis.	—
`strongest_mismatch`	The basis was captured from a differently trained model, so a layer refitted from it would serve an operating point derived from predictions this model never made.	Refit the model.
`custom_metric_unavailable`	The run was optimised for a `user_defined_metric` whose callable cannot be imported here. Refitting would silently grade the new operating point against the task default.	Make `user_defined_metric_ref` importable in this environment.

The last one is a refusal rather than a fallback on purpose: a refused refit is recoverable, and a refit against the wrong metric is not detectable.

"on-disk format v… but this ml4 build reads format v…"¶

A stored artefact predates the ml4 you are running. ml4 version-stamps every artefact and rejects a stale one at load rather than shimming it, because the alternative is a bare AttributeError in the middle of scoring — or, worse, a model that loads and silently scores against different geometry. There is no migration tool: re-fit and re-persist.

Error text	Artefact	Bumped because
`… holds an unsupervised pipeline in format v7.0, but this ml4 reads v8.0`	`UnsupervisedPipeline` joblib	Format `8.0` scales the numeric track before it imputes and normalises the text sub-blocks per block, so every axis of a `7.0` artefact's geometry differs; `8.0` has no `deepsvdd` detector (see Unsupervised).
`pipeline at '…/forecaster' is on-disk format v6.0 … reads format v7.0`	`FittedForecastPipeline` directory	A `6.0` bundle carries no `history_tail`, which the serve path needs to run the fit's own future-frame checks on a caller-supplied `X_future` (see Forecasting).
`FeatureRecipe is stamped 'ml4-recipe/1', but this ml4 reads 'ml4-recipe/2'`	a serialised `FeatureRecipe` (`FeatureRecipe.from_json`)	Version 2 transformers carry new fitted fields, and a calendar step names its calendar columns in `params["columns"]`. Re-plan the recipe and refit the run.
`Supervised pipeline bundle at '…/bundle' is on-disk format v3 … reads format v4`	supervised `FittedPipeline` bundle directory	The fold-local chain moved to AutoGluon's per-model feature-generator channel and the fitted transformers and baseline report carry new state (see Deployment).
`This run's metrics/measures.json was written by an older ml4 (measures_version=None; this build reads 1)`	a run's `measures.json`, read by the platform	The file has no `measures_version` stamp, or an older one: its numbers (the Brier scale, the baseline block) changed meaning. The platform answers 409 for that run's Scores, leaderboard-reference and Decision-layer views — re-run the experiment to regenerate the file.

An artefact loads only at the exact format version this build reads. A different stamp — minor versions included — or a missing one is refused before anything is unpickled: re-fit, or re-export with this ml4 version.

A re-fit is a new model, not a format conversion: re-check anything you tuned against the old artefact, such as an anomaly operating threshold.

"…no key is set: set ML4_PSEUDONYMISATION_KEY to the key this model was fitted with (fingerprint …)"¶

The model replaces identifier columns with keyed pseudonyms (HMAC-SHA256) and — by design — does not store the key. Set ML4_PSEUDONYMISATION_KEY in the environment that fits, loads or serves the model to the key it was fitted with; the message names that key's fingerprint, which you can compare with parleml_ml4.contracts.pseudonymisation.key_fingerprint(<key>). "The pseudonymisation key in reach (fingerprint …) is not the key this model was fitted with" means a different key is set. ParleML Studio manages the key for you (in its vault, config/vault.json under the data root). There is no way to recover a lost key: refit the model under a new one. See The pseudonymisation key.

Cancelling a run¶

Runs that were given an outputs_dir are cooperatively cancellable: create a file named CANCEL in that directory and the pipeline stops at the next checkpoint. A run never clears the sentinel when it starts — a CANCEL written before the run reaches its first checkpoint (a worker still importing, say) is a request for that run, and its first check honours it. The run that honours it removes it, so the next run in the same directory starts clean; a run that ends any other way leaves a sentinel it never read in place.

Every such run also writes structured events.ndjson and a manifest.json — read these first when a run behaves unexpectedly.

Details: the supervised stage stream

The supervised stages are preflight → holdout_reserved → fe_structure_decided → hyperparameter_search → ag_fit_started → ag_fit_done → post_fit_layers → registered, each a stage_start / stage_end pair, and an error raised from inside the fit ends the stage it happened in failed, with the error. It does not cover recipe planning: a library caller runs RecipePlanner().plan before the first stage, so a plan-time raise is in no stage (a run wrapped in RunTelemetry.run records it on its run_end; see the stage stream). A run whose process was killed before it could close its log — out of memory, a hard kill, a crash in native code — reads as ended, not as running: the platform, knowing its worker has exited, reports the run and every stage still open as failed or interrupted. Read the raised error, or the log, for the cause.

Resuming after a crash¶

Once AutoGluon has trained (ag_fit_done), the models are durable at path. If the post-fit layers crash afterwards, the run is recoverable — resume_pipeline rebuilds the deterministic post-fit layers from that boundary without re-training the engine.

Installation issues¶

A workload import fails — you likely need its extra, e.g. pip install "parleml-ml4[unsupervised]". Importing ml4 itself never fails for a missing optional backend.
GPU vs CPU PyTorch — from PyPI, torch is the CPU build on Windows and the CUDA build on Linux. Install it from PyTorch's CPU or CUDA index first to choose. See GPU acceleration.

FAQ¶

Do I need MaxMind to use ml4? No — only for IP (ip_domain) features, and even then the run degrades gracefully without it. Everything else works out of the box.

Does my data leave the machine? No, unless you choose a provider as the text-embedding source: the embedded text columns are then sent to that provider at fit and at every scoring call. Local models and everything else run on-device.

How do I make models small and fast to score? Use presets.real_time() — see Deployment & export and Inference.

Where is the trained model saved? Wherever you pass to save(...) / the path argument; reload it with parleml_ml4.load_pipeline(path).

Next: Training workflow · Configuration & presets