Skip to content

Supervised learning with AutoSupervised

AutoSupervised is ml4's front door for supervised learning. You give it a Ml4Config and one training frame; it plans the feature-engineering recipe, fits the model, wires the post-fit statistical layers, and hands back a served pipeline you can predict with, evaluate, and save.

In short

  • AutoSupervised(cfg).fit(train_df) takes one training frame with the target column inside it and returns a FittedPipeline.
  • fit holds out its own validation rows. Keep your test rows separate and pass them to evaluate.
  • Classification serves tuned labels from predict and calibrated probabilities from predict_proba; raw probabilities are always one keyword away.
  • Regression predictions are always in the target's original units, even when the model trained on a transformed target.
  • model.save(path) and parleml_ml4.load_pipeline(path) round-trip the whole pipeline, post-fit layers included.
import parleml_ml4
from parleml_ml4 import AutoSupervised, Ml4Config, Tasks
from parleml_ml4.contracts.metrics import EvalMetric

cfg = Ml4Config()
cfg.ml_task = Tasks.BINARY_CLASSIFICATION
cfg.eval_metric = EvalMetric.f1
cfg.data.target_column = "y"

model = AutoSupervised(cfg).fit(train_df)     # ONE frame in — label column included

preds  = model.predict(X)                     # thresholded class labels
proba  = model.predict_proba(X)               # calibrated class probabilities
report = model.evaluate(X, y)                 # fit-time measures + test scores
model.save("out/bundle")

reloaded = parleml_ml4.load_pipeline("out/bundle")

One training frame in; keep final test rows separate

fit takes a single DataFrame with the label column included, and holds out its own validation internally. There is no six-frame split to prepare; validation is either bagged out-of-fold predictions or a reserved holdout, governed by the split policy and engine.validation_holdout_frac. This is the single most important thing to internalise about the ml4 API. AutoSupervised.fit does not carve an outer test set: pass separate, untouched test rows to evaluate. ParleML Studio carves that test set before calling the library.

Fitting

model = AutoSupervised(config).fit(
    train_data,
    label=None,               # defaults to config.data.target_column
    models=None,              # defaults to config.roster.models, else ("GBM:gbdt",)
    path=None,                # where to persist the AutoGluon predictor / bundle
    events=None,              # an parleml_ml4.telemetry.EventEmitter for the stage stream
    cancel=None,              # a CancellationToken making a long fit interruptible
    mlops=None,               # opt-in MLflow logging settings
)
Keyword Meaning
label The target column. Falls back to config.data.target_column; a missing label raises before any training work.
models The roster of models to train, as StockModel specs — a gradient booster is variant-qualified ("GBM:gbdt", "GBM:dart", "GBM:rf", "GBM:extra", "GBM:arith", "CAT:symmetric", "XGB:gbtree"), a single-variant family is bare ("RF", "LR"). Each entry is one more trained model, and one more Optuna study when search is on. Falls back to config.roster.models, and finally to ("GBM:gbdt",). See Algorithms & metrics.
path Filesystem location for the trained predictor. Also the default save target.
events An parleml_ml4.telemetry.EventEmitter receiving one stage_start / stage_end pair per stage (preflight → holdout_reserved → fe_structure_decided → hyperparameter_search → ag_fit_started → ag_fit_done → post_fit_layers → registered) for a progress UI, plus the run's warning events and the per-model model_fit / model_scored progress. A raise ends the stage it happened in as failed (or cancelled). The stream starts after recipe planning — which AutoSupervised.fit does first — so a raise there emits nothing at all; see Training workflow.
cancel A CancellationToken. Touch <path>/CANCEL and the fit stops gracefully between models (and between Optuna trials when the search is on), raising CancelledError and leaving a resumable predictor. Without one the fit is uninterruptible.
mlops Opt-in MLflow logging via MLOpsRunSettings; the run id lands on the AutoSupervised instance's mlflow_run_id_, rather than the returned pipeline. See MLOps.

The reserved-holdout size is not a fit keyword: it lives on the config as engine.validation_holdout_frac (default 0.1), so a library caller and a run launched from the UI resolve it the same way. See Data splitting.

fit returns a FittedPipeline. To keep the training façade as well, use trainer = AutoSupervised(config) followed by model = trainer.fit(train_data). The pipeline is then also available as trainer.pipeline_; the façade's predict / predict_proba / evaluate / save delegate to it.

One-liner and reload helpers

parleml_ml4.fit(config, train_df) wraps the class for a single call, and parleml_ml4.load_pipeline(path) reloads a saved bundle. Both are re-exported from the top-level ml4 package.

Predicting

The fitted pipeline exposes a task-aware predict surface:

labels = model.predict(X)                       # decision-layer thresholded labels
proba  = model.predict_proba(X)                 # calibrated probability frame
raw    = model.predict_proba(X, calibrated=False)  # uncalibrated probabilities

Because ml4 owns calibration post-hoc, both raw and calibrated probabilities are always available. predict applies the tuned decision threshold (a single binary threshold, or per-class multiclass thresholds under a hard-label metric such as f1_macro; under a probability metric such as the default logloss, multiclass predict is plain argmax on the served probabilities). Which calibration map is fitted, and whether one ships at all, is covered in Probability calibration.

yhat = model.predict(X)                         # point predictions, ORIGINAL units

Regression point predictions are always returned in the original target units — if a y-transform was selected, it is inverted for you. predict_proba is undefined for regression; use predict or the interval API (see Prediction intervals). What "inverted" does and does not promise is spelled out in The regression target transform.

The regression target transform

On a regression run (target_transform.enable_target_transform_tuning, default True) ml4 may train the model on a warped target — log1p, sqrt or Yeo-Johnson — and invert every prediction back to the user's units. Two different decisions can pick that transform, and they are not equally strong evidence.

Route When it is taken What it measures
CV bake-off The ordinary case: enough rows (target_transform_tuner_min_samples, default 200), target_transform_tuner_cv_folds ≥ 2, LightGBM installed, and at least one usable screen feature. A lightweight LightGBM is CV-fit on each candidate, its held-out predictions are inverted back to original units and scored there with the run's own eval metric. A candidate ships only if it beats identity by ≥ 1%; a sub-margin win is treated as CV noise.
Skewness gate A roster that deploys only linear models (a regulated glm / scorecard, or a stock roster that is all LR), or any case the bake-off cannot run: fewer rows than min_samples, cv_folds < 2, no LightGBM, no usable screen feature. \|skew(y)\| >= 0.5 ⇒ Yeo-Johnson, otherwise nothing. Model-free: no held-out check at all. A booster fits skew natively, so the tree bake-off would veto the de-skewing a linear model genuinely needs — hence the substitution for a linear roster.

The skew gate is not a lesser version of the bake-off: it makes no out-of-sample check at all, and it logs a WARNING saying so when it ships a transform. Read a "yeo-johnson" chosen that way as "the target was skewed", not as "a competition preferred it".

Inverting is not free — the retransformation bias

Inverting a model fitted on g(y) with the plain analytic g⁻¹ does not estimate E[Y|x]. By Jensen's inequality g⁻¹(E[g(Y)|x]) ≠ E[Y|x] for any non-linear g, and for the concave transforms here (log1p, sqrt, Yeo-Johnson with λ < 1) the naive inverse is biased low — it under-predicts the large values, which on a price or duration target is where the money is. Duan's smearing estimator removes it non-parametrically, Ê[Y|x] = mean_j g⁻¹(ẑ(x) + ê_j) over the fitted residuals.

ml4 measures rather than assumes. On the deployed model's own out-of-sample pass (OOF or the reserved holdout) both inverses are scored with the run's metric, and the smeared one serves only if it wins.

Details: when smearing wins, and why it cannot change between consumers

This is the same measure-then-ship discipline as the calibration bake-off, with the same caveat that the choice is made on the rows it is measured on. Smearing usually loses on a run graded for the conditional median (MAE, a quantile): a monotone g commutes with quantiles exactly, which is also why prediction intervals are never smeared.

Whichever inverse wins, it is the one every consumer gets. predict, the runner's fitting-set inversion and the airgapped ExportedPipeline all call the same inverse_transform, so the exported artifact and the in-process model cannot serve two different numbers.

Details: what measures.json records

The target_transform block carries method, ag_score_space, reported_score_space, and two blocks that explain the transform choice:

  • selection — route (cv_bakeoff / skew_gate), held_out_checked, the target's skew, and on the bake-off route the per-candidate cv_scores and the winner's rel_gain. A missing cv_scores is the statement that no competition was run.
  • retransformation — applied, the n_rows behind the measurement, n_residuals (the stored smearing sample: one closed-form shift for log1p, at most 100 residual-bin means otherwise — every residual weighs equally), naive_score vs smeared_score under the run's metric, and naive_bias vs smeared_bias (mean prediction minus mean truth, in the user's units). So the bias is a published number even on the runs where the correction did not ship.

Every ml4-reported score — the test block, the baseline gate, interval coverage — is computed on the inverted prediction in the user's units, so whatever bias survives is inside the RMSE / MAE you read rather than hidden behind it. AutoGluon's own validation scores are the transformed-space ones; see the target_transform row of the FittedPipeline table.

Evaluating

report = model.evaluate(X_test, y_test)

evaluate returns the fit-time measures alongside reported scores on the supplied test data. It threads the run config through, so a beta or user_defined_metric run reports its real metric rather than the task-default fallback. See Inference for the measures schema.

Saving and loading

path = model.save("out/bundle")                 # or AutoSupervised(cfg, path="out/bundle")
reloaded = parleml_ml4.load_pipeline("out/bundle")

save persists the served bundle to disk; parleml_ml4.load_pipeline reconstructs the full FittedPipeline, including the post-fit layers. See Deployment for airgapped export and serving artifacts.

The FittedPipeline

fit returns a FittedPipeline — a deployed model plus the post-fit statistical layers, behind one predict surface. It bundles:

Component Role
predictor The fitted AutoGluon TabularPredictor (applies the global + fold-local FE internally).
calibration Frame-level probability calibration (binary + multiclass), fit only if it wins a bake-off.
threshold / multiclass_thresholds The tuned decision layer for the run's metric.
interval Regression prediction intervals (CQR / conformal), when enabled.
target_transform The regression y-transform — the model trains transformed, serves original units. AutoGluon's own validation scores stay in the transformed space; the deploy/leaderboard.json and deploy/ensemble_composition.json sidecars carry a score_space field saying so, and measures.json's target_transform block states both spaces (ag_score_space / reported_score_space).
baseline The baseline honesty verdict — did the model beat the best skill-free predictor, and on how many rows (n_rows)? An inconclusive verdict says why in low_confidence_reason ("few_rows" or "unconfirmed_win"); low_confidence is true whenever a reason is set.
holdout_info Transparency on the fitting set: basis is "oof" (bagged out-of-fold) or "holdout" (a reserved slice, with its fraction / train_rows_reduced_by). On the OOF path ensemble_weights_selected_in_sample says whether a weighted-ensemble strongest model chose its blend on those same rows, and copied_classes_excluded (present only when there are any) names the single-row classes the fit copied for bagging, whose rows the post-fit layers were not fitted on.
fit_rows Which rows of the training frame the fit trained on (trained) and reserved for the post-fit layers (reserved, empty on the OOF basis), by position in that frame; a position in neither was left out (an excluded rare class, or a row the temporal embargo purged). Set on the pipeline fit returns and never saved — it describes the training frame, not the model — so a loaded bundle reads None.
out_of_sample Classification only: the raw class probabilities of every row of the training frame that a model scored without having trained on it (rows, by position in that frame, with their labels, proba and classes), which is what a label audit needs. A bagged run covers every trained row by its out-of-fold prediction, plus the reserved holdout under refit_full; a run that does not bag covers the reserved holdout only. Never saved, like fit_rows.
recipe The data-driven FE plan, retained for audit.
treelite The opt-in compiled fast-inference core, when delivered.

Each layer is optional and fit only when applicable, so the same facade serves binary, multiclass, and regression tasks without task-branchy calling code.

The pipeline stages

Under the hood, fit runs three phases. Each links to a deeper guide:

  1. Plan the recipe. A RecipePlanner inspects the training data and the config to decide the concrete feature-engineering steps — which encoders fire, which interactions to try, which features to select. See Feature engineering.
  2. Fit the engine. AutoGluon trains the resolved roster under the chosen budget preset, running ml4's two feature-engineering loci (global + fold-local) and producing bagged OOF or a refit single model. See Algorithms & metrics and Constraints.
  3. Assemble the post-fit layer. From a single out-of-sample scoring pass, ml4 fits calibration, the decision threshold, regression intervals, and runs the baseline honesty gate. The first, second and fourth are one transaction (parleml_ml4.delivery.decision_layer): the threshold is fitted on whichever probabilities calibration decided should serve, and the gate grades the decision both produced. Re-deciding any of them alone would leave an operating point fitted to probabilities the model does not serve. See The training workflow, Prediction intervals, Inference, and MLOps.

Crash-after-fit recovery

Once AutoGluon has trained, the models are durable at path. If the post-fit layers crash, the run is recoverable — the deterministic post-fit setup is rebuilt from disk without re-training. The training workflow page covers this in detail.

Where to go next

Next: The training workflow · The decision layer