The decision layer

Everything ml4 fits after AutoGluon returns: probability calibration, the decision threshold, and the constant-baseline gate. They are one unit rather than three steps, and the unit can be re-decided without retraining the model.

See the calibration guide for the concepts; this page is the API.

In short

• Calibration, decision threshold and baseline gate are refitted together, never one alone.
• model.recalibrate(method=..., advantage_margin=..., apply=...) covers the common case; parleml_ml4.refit_decision_layer(model, config) takes a full DecisionLayerConfig.
• A refit works from decision_basis/ in the bundle, so it takes seconds and retrains nothing. The strongest model is never re-selected.
• A refused refit raises DecisionLayerUnavailable with a machine-readable reason.
• skill_free_floor shows the baseline score before anything is trained.

Why it is one unit

The three are not separable, and the coupling runs one way:

1. Calibration decides which probabilities serve — the fitted map's output when it wins its bake-off, the raw frame when it declines or is switched off.
2. The decision threshold is fitted on those probabilities. Re-decide calibration without refitting it and the operating point is stale by construction: a cutoff tuned at 0.31 on one probability surface, applied to another.
3. The baseline gate grades the decision those two produce, so its verdict — including no_signal, which can abort a run — is a function of the calibrated frame and the refitted threshold.

So there is no "recalibrate only" function at any layer, deliberately.

Re-deciding a layer without retraining

Each of the three is a function of one frozen frame — the strongest model's out-of-sample output on the fitting set — and nothing else. DecisionBasis is that frame, persisted into the bundle as decision_basis/, which is what makes a refit a matter of seconds rather than a retrain.

from dataclasses import replace

import parleml_ml4

model = parleml_ml4.load_pipeline("out/bundle")
fitted_config = model.decision_provenance.config

rebuilt = parleml_ml4.refit_decision_layer(
    model, replace(fitted_config, calibration_method="temperature"),
)
rebuilt.save("out/bundle")

or, for the common case, through the façade:

model = parleml_ml4.AutoSupervised(cfg)
model.fit(train_df)
model.recalibrate(method="temperature").save("out/bundle")

Given the same config, a refit reproduces the fitted layer exactly — that identity is what makes "re-decide without retraining" a statement about the same computation rather than a similar one, and it is pinned by a test rather than asserted here.

What a refit deliberately does not do

Model selection is never re-run. The leaderboard's ranking criterion is the bake-off's criterion, so a refit does break the property that selection and calibration cannot disagree about better. Re-running selection would cost exactly what a cheap refit exists to avoid, so the strongest model is frozen and the discrepancy is reported instead: provenance.selection_consistent is False from the first refit that changes behaviour, and every surface that shows a leaderboard is expected to say so.

The held-out numbers are not the criterion. A preview reports the out-of-fold bake-off numbers — the same cross-validated evidence the gate itself decides on. Choosing a calibrator by refitting and re-reading a held-out score is method selection on the test set, and it is cheap enough here to do twenty times without noticing. Choose the method for a reason — argmax stability, a policy on probability shape, a margin that suits the decision cost — and let the bake-off arbitrate within it.

When a refit is refused

DecisionLayerUnavailable carries a machine-readable reason:

reason	Meaning	The fix
no_basis	The run kept none — deployment.persist_decision_basis was off, or the frame was above deployment.decision_basis_max_cells.	Enable it (or raise the ceiling) and refit.
no_provenance	The pipeline was not assembled by an ml4 run, so there is no fitted revision to advance from.	—
task_not_supported	Regression. Its post-fit layer is the prediction interval, a different object.	—
strongest_mismatch	The basis was captured from a differently trained model.	Refit the model.
custom_metric_unavailable	The run was optimised for a metric whose scorer cannot be rebuilt here — typically a user_defined_metric whose user_defined_metric_ref does not import in this environment.	Make the callable importable.

The last one is a refusal rather than a fallback on purpose. A refit is by definition working from a loaded bundle, and the config-less scorer path falls back to the task default metric with a warning — so a cost-sensitive run would silently refit its threshold and re-grade its baseline against a metric it never optimised. A refused refit is recoverable; a refit against the wrong metric is not detectable.

Optionality

Every stage here is switchable off, and ml4 must fit, serve, save, load, predict and export with all of them off:

Off	Still works	Genuinely lost
calibration.apply_at_predict_time	Everything. The threshold is fitted on raw probabilities.	Calibration evidence. calibration_report.outcome is disabled — distinct from declined, which means it was measured and lost.
baseline.enable_constant_baseline	Everything.	The sanity verdict and the "vs baseline" lift.
deployment.persist_decision_basis	Everything except a refit.	The ability to re-decide the layer at all. The run still records which revision it is serving.

A run with calibration off but the basis kept can be calibrated later, without retraining — the basis is captured from the strongest model's out-of-sample output whatever calibration decided.

The floor, before anything is trained

The baseline gate's verdict is post-fit by necessity — it compares a strongest model. Its floor is not: what a feature-blind predictor scores is a function of the label distribution alone.

from parleml_ml4.delivery.baseline_gate import skill_free_floor

floor = skill_free_floor(y, task=Tasks.MULTICLASS_CLASSIFICATION, metric=EvalMetric.precision_macro)
# SkillFreeFloor(strategy='uniform', score=0.248, metric='precision_macro', n_rows=1000)

No model, no fit, no split — so it can be shown before a run starts, which is when it is useful. It reuses the gate's own candidate set and winner rule, so the number shown up front and the number the gate grades against afterwards cannot disagree.

skill_free_floor

skill_free_floor(
    y: ndarray,
    *,
    task: Tasks,
    metric: EvalMetric,
    seed: int = 0,
    positive_class: Any = None,
    classes: Optional[List[Any]] = None
) -> Optional[SkillFreeFloor]

The best score a feature-blind predictor achieves on y under metric.

Answerable before anything is trained, because a skill-free predictor is a function of the target's distribution alone. Show it beside a signal check and a user learns what their model has to clear before they spend twenty-eight minutes finding out.

Every candidate is scored and the strongest wins, rather than one being chosen per (task, metric) by rule: which skill-free predictor is strongest is a property of the metric, and guessing hands the model a floor that is too low. On 70/10/10/10 four-class data the best constant scores 0.17 precision_macro while a chance-level draw scores 0.26 — quote the constant and a model at 0.20 looks like it learned something.

None when no candidate is scorable at all (a single-class target under a metric that needs two, an empty frame). Refusing is the honest answer; a floor of 0 is not.

Reuses the gate's own candidate set and its own winner rule, so the number shown up front and the number the gate grades against afterwards cannot disagree — which they would within a week if this had its own copy.

SkillFreeFloor dataclass

SkillFreeFloor(
    strategy: str, score: float, metric: str, n_rows: int
)

What a predictor that ignores every feature scores on this target.

Pure static analysis of the label distribution — no model, no fit, no split. That is the point: it is knowable before a run starts, and it is the number every trained model has to beat to be worth deploying.

It is the floor half of the constant-baseline gate, separated from the verdict half. The verdict ("your model barely beats a constant") is necessarily post-fit — it compares a strongest model that does not exist yet. The floor is not, and computing it only at the end meant the one number that could have stopped a pointless run was produced after the run.

API

DecisionLayer dataclass

DecisionLayer(
    positive_class: Any = None,
    calibration: Optional[CalibrationMap] = None,
    calibration_report: Optional[CalibrationReport] = None,
    threshold: Optional[BinaryThreshold] = None,
    multiclass_thresholds: Optional[
        MulticlassThresholds
    ] = None,
    baseline: Optional[BaselineReport] = None,
    multiclass_thresholds_discarded: bool = False,
)

What one decision-layer fit decided.

Every field is optional because every layer is: calibration can be switched off or lose its bake-off, the thresholds are task-specific and the multiclass ones can be discarded in favour of argmax, and the gate is opt-out. calibration_report is the exception — there is always an account of what calibration did, including when it did nothing, because "declined" and "nobody recorded it" are different statements about a model.

as_pipeline_fields

as_pipeline_fields() -> Dict[str, Any]

This layer as the flat fields :class:~parleml_ml4.delivery.fitted_pipeline.FittedPipeline serves from — the single seam between the two shapes.

The pipeline keeps these flat rather than nesting a DecisionLayer inside it, because the grouping is a fitting-time fact and the pipeline is a serving-time object: nesting would churn every reader (predict_proba, the bundle's field sweep, MLI, export) to express something none of them care about. One method here is the whole cost of keeping both shapes right, and it is what makes a refit a dataclasses.replace rather than fifteen hand-copied assignments that drift the first time a field is added.

DecisionLayerConfig dataclass

DecisionLayerConfig(
    task: Tasks,
    metric: EvalMetric,
    beta_threshold: float,
    calibration_method: CalibrationMethod,
    calibration_advantage_margin: float,
    apply_calibration: bool,
    baseline_enabled: bool,
    baseline_margin: float,
    seed: int,
    user_defined_metric_ref: Optional[str] = None,
    user_defined_metric_direction: Optional[
        Literal["minimize", "maximize"]
    ] = None,
)

Every knob :func:~parleml_ml4.delivery.decision_layer.fit_decision_layer consults.

Frozen because the layer is fitted once from it and a later rebuild has to be able to say whether it is fitting from the same settings — a mutable config makes that unanswerable.

digest property

digest: str

Content digest of every knob the layer reads.

What makes "was this refit a behaviour change?" answerable without diffing objects: an idempotent refit leaves it unchanged, so nothing downstream has to be invalidated, and an unequal digest against the selection-time one is exactly the frozen-selection discrepancy the provenance stamps.

from_ml4_config classmethod

from_ml4_config(
    config: "Ml4Config", *, metric: EvalMetric
) -> "DecisionLayerConfig"

Project a run's config onto the fields the decision layer reads.

metric is passed rather than read off config.eval_metric because the runner has already resolved it — config.eval_metric is optional and falls back to a per-task default, and resolving it twice is how two layers come to disagree about what the run was optimised for.

build_scorer

build_scorer() -> 'Scorer'

The run's own AutoGluon Scorer, rebuilt from what was persisted.

This is the D2 fix, and the reason two metric fields live on a decision-layer config at all. resolve_ag_scorer takes an optional whole config and, when it is absent, degrades to the task-default metric with a warning — a path that exists precisely for "a loaded bundle", which is exactly what a refit is. Left on that path, every F-β or user-defined run would re-fit its threshold and re-grade its baseline against a metric it never optimised, silently.

So this rebuilds the custom scorer from the persisted fields, and raises when it cannot. A refused refit is recoverable; a refit against the wrong metric is not detectable.

from_json classmethod

from_json(payload: Dict[str, Any]) -> 'DecisionLayerConfig'

Rebuild a config from what :meth:as_json wrote.

The exact inverse, and it exists so a reader of measures.json can fit a candidate layer without loading the bundle. That is not a convenience: the alternative is deserialising an AutoGluon predictor to recover four scalars, which is the difference between a preview that runs while a user watches and one that competes with the batch-predict deserialiser.

Every field is read explicitly rather than splatted, so a payload carrying a key this build does not know is ignored rather than raising, and a MISSING key raises rather than silently defaulting — the run's own metric and seed are not values to guess at.

as_json

as_json() -> Dict[str, Any]

The config as JSON-able values — the form both the digest and measures.json use.

DecisionBasis dataclass

DecisionBasis(
    proba: ndarray,
    labels: ndarray,
    folds: ndarray,
    classes: Tuple[Any, ...],
    task: Tasks,
    positive_class: Any,
    source: BasisSource,
    fold_policy: str,
    fit_class_counts: Tuple[int, ...],
    strongest_fingerprint: str,
    ml4_version: str = "",
)

One decision layer's fitting inputs, as persisted.

Immutable on purpose: it is the record of what a fit saw, so re-fitting from it is the only operation that can be honest about producing "the same layer with a different setting".

fingerprint property

fingerprint: str

Identify this basis, so a layer can name what it was fitted from.

Metadata only — the strongest model fingerprint, the class geometry, the fold policy and the row count. It does not hash the probability matrix: the frame is the strongest model's own output on a fixed fitting set, so strongest_fingerprint already pins it, and hashing tens of megabytes on every read would make the cheap paths (a summary, a preview) pay for a guarantee they already have.

fold_pairs

fold_pairs() -> Optional[List[FoldPair]]

The bake-off folds as fit_decision_layer consumes them, or None for a shuffle.

None and "every row unassigned" are the same state on disk and mean the same thing: the run had no policy-specific geometry, so the bake-off uses its own stratified shuffle.

class_array

class_array() -> ndarray

The classes as an array of their OWN dtype.

Not dtype=object. The decoded labels feed StratifiedKFold through the calibration bake-off, and sklearn calls type_of_target on them first: an object array of integers types as 'unknown' and the splitter refuses it outright. So a run whose labels are integers — most of them — fitted perfectly at fit time and could not refit at all, on a dtype nobody chose. pd.Index is what restores the dtype the capture saw.

fit_y

fit_y() -> ndarray

The training target the gate needs, reconstructed from its histogram.

proba_frame

proba_frame() -> DataFrame

The probability frame, class-labelled as the layer expects it.

eval_y

eval_y() -> ndarray

The evaluation labels, decoded back to their original values.

save

save(directory: str | Path) -> None

Write the basis into directory (created if absent).

load classmethod

load(directory: str | Path) -> Optional['DecisionBasis']

Read a basis back, or None when the bundle carries none.

None covers both "the run opted out" and "it was above the size cap" — the distinction is recorded in basis.json for a caller that wants to explain why rebuilding is unavailable, but for loading purposes they are the same absence.

DecisionLayerProvenance dataclass

DecisionLayerProvenance(
    revision: int,
    source: Literal["fit", "rebuild"],
    created_at: str,
    ml4_version: str,
    config: DecisionLayerConfig,
    fit_config: DecisionLayerConfig,
    basis_fingerprint: Optional[str],
    selection_config_digest: str,
    history: Tuple[DecisionLayerRevision, ...] = (),
    history_dropped: int = 0,
)

What produced the decision layer a pipeline is serving.

Written at fit and replaced (never mutated) on every refit, so a pipeline and its provenance move together through dataclasses.replace like every other layer.

selection_consistent property

selection_consistent: bool

Is the strongest model still serving the layer it was selected under?

Equal at fit and after any idempotent refit; unequal the moment a refit changes behaviour. That single boolean is what a persistent caption on the leaderboard says: "strongest model selected under decision-layer revision 1; serving revision 2."

advanced

advanced(
    *,
    config: DecisionLayerConfig,
    entry: DecisionLayerRevision
) -> "DecisionLayerProvenance"

The provenance one refit later: bump the revision, append, keep everything else.

fit_config and selection_config_digest deliberately survive untouched — they are the two facts a refit must not be able to overwrite, since one is the way back and the other is the record of what selection saw.

as_json

as_json() -> Dict[str, Any]

The measures.json block — everything a reader needs without loading the bundle.

DecisionLayerRevision dataclass

DecisionLayerRevision(
    revision: int,
    created_at: str,
    source: Literal["fit", "rebuild"],
    actor: Optional[str],
    calibration_method: str,
    calibration_outcome: Optional[str],
    calibration_gain: Optional[float],
    baseline_verdict: Optional[str],
    config_digest: str,
)

One entry in the layer's history — enough to compare revisions without refitting them.

as_json

as_json() -> Dict[str, Any]

Every field here is JSON-native, so asdict is the mapping. The provenance's own as_json cannot be — it publishes two computed properties and two nested configs whose enums need naming.

DecisionLayerUnavailable

DecisionLayerUnavailable(
    reason: UnavailableReason, message: str
)

Bases: RuntimeError

This pipeline cannot re-decide its decision layer; :attr:reason says why.

Raised rather than degraded on purpose in the custom_metric_unavailable case. A refit is by definition working from a loaded bundle, and resolve_ag_scorer's config-less path falls back to the task default metric with a warning — so a cost-sensitive run whose scorer could not be rebuilt would silently refit its threshold and re-grade its baseline against a metric it was never optimised for, for precisely the runs that care most about the operating point. Refusing is the only honest outcome.

fit_decision_layer

fit_decision_layer(
    strongest_output: DataFrame | ndarray,
    eval_y: ndarray,
    fit_y: ndarray,
    *,
    config: DecisionLayerConfig,
    scorer: Scorer,
    positive_class: Any = None,
    folds: Optional[Sequence[FoldPair]] = None
) -> DecisionLayer

Fit every post-fit decision from ONE out-of-sample scoring pass.

strongest_output is the deployed model's output on the fitting set — the probability frame for classification, point predictions for regression. eval_y are that set's labels; fit_y is the training target, which only the gate reads (it derives the skill-free candidates from that support, then scores them against eval_y).

scorer is supplied rather than rebuilt here. The runner has already resolved it from the run's own config, and rebuilding it would need the whole config back — including, for the F-beta and user-defined metrics, fields that only a full Ml4Config carries.

Order is not incidental. Calibration decides which probabilities serve; the threshold is fitted on those; the gate grades the decision both produced. Reversing any pair would score a model nobody deploys.

fit_layer_from_basis

fit_layer_from_basis(
    basis: DecisionBasis, config: DecisionLayerConfig
) -> DecisionLayer

Fit a decision layer from a frozen basis alone.

Given the same config the run was fitted with, this is bit-identical to what the fit produced — which is the only thing that makes the extracted unit really the unit, and is pinned by test_a_refit_with_the_fitted_config_reproduces_the_fitted_layer. Without that identity, "re-decide without retraining" would be a claim about a different computation.

Raises rather than degrading when the run's own scorer cannot be rebuilt here; see :meth:~._config.DecisionLayerConfig.build_scorer.

refit_decision_layer

refit_decision_layer(
    pipeline: "FittedPipeline",
    config: DecisionLayerConfig,
    *,
    actor: Optional[str] = None
) -> "FittedPipeline"

A copy of pipeline serving a decision layer refitted under config.

Immutable: the input pipeline is untouched and a new one is returned through dataclasses.replace, so a caller that keeps both can compare them, and a failure part-way through leaves nothing half-modified.

actor rides into the history entry. None is honest for a library call; the product path supplies the user who asked, because a post-fit change to what a model outputs is the thing an auditor asks who authorised.

capture_basis

capture_basis(
    proba: DataFrame,
    eval_y: ndarray,
    fit_y: ndarray,
    *,
    task: Tasks,
    positive_class: Any,
    source: BasisSource,
    fold_policy: str,
    folds: Optional[Sequence[FoldPair]],
    strongest_fingerprint: str,
    max_cells: int
) -> Optional[DecisionBasis]

Freeze what a decision-layer fit saw, or None when it is too large to keep.

Returning None above max_cells rather than downsampling is deliberate: a smaller basis would answer a different bake-off, so a layer refitted from it would not be the layer a refit produces — and the whole value of keeping it is that it is.

Next: The decision layer guide · FittedPipeline