MLOps: MLflow & Unity Catalog¶

ml4 logs a fitted pipeline to MLflow as a pyfunc model on one canonical path, so a single mlflow.pyfunc.load_model serves calibrated, thresholded predictions with no ml4-specific loader. Logging is opt-in, best-effort, and hardened to load in a fresh environment or on Databricks.

In short

Pass fit(mlops=MLOpsRunSettings(...)) to log the fitted pipeline to MLflow as a pyfunc; MLflowSettings.experiment_name is the on switch.
Logging is best-effort: a logging or registry error warns and never fails a good fit.
The run carries non-default params, fit_* metrics in the metric's own units, and provenance tags; evaluate() adds test_* scores to the same run.
The pyfunc returns probabilities by default; pass params={"output": "labels"} for labels.
Register to the MLflow Model Registry with a plain name, or to Unity Catalog with a three-level name.

Optional dependency

MLflow logging needs the mlops extra: pip install "parleml-ml4[mlops]", which installs mlflow-skinny: the tracking client, the registry (Unity Catalog included) and the pyfunc wrapper, without the tracking server or UI. Install mlflow as well if you want those or mlflow models serve. MLflow is imported lazily inside the logger, so a plain install that never logs needs none of it, and a missing MLflow degrades to a skipped log, never a failed fit.

Opt-in via `fit(mlops=...)`¶

Pass an MLOpsRunSettings to fit. It bundles the per-run MLOps knobs into two sub-records, mlflow and registry. An empty MLOpsRunSettings() reproduces the "no MLOps" behaviour exactly.

from parleml_ml4 import AutoSupervised, Ml4Config
from parleml_ml4.mlops_settings import MLOpsRunSettings, MLflowSettings, ModelRegistrySettings

cfg = Ml4Config(); cfg.data.target_column = "y"

mlops = MLOpsRunSettings(
    mlflow=MLflowSettings(experiment_name="risk-q1", run_name="baseline"),
)
asv = AutoSupervised(cfg)
model = asv.fit(train_df, mlops=mlops)
asv.mlflow_run_id_        # the run id of the logging pass (None if not logged)

MLflowSettings.experiment_name is the master switch: None (the default) disables tracking entirely. Logging is best-effort: a logging or registry error is caught and warned, and never fails a good fit.

The logging pass is the registered stage of the fit's event stream. With no mlops, or MLflow disabled, nothing registers the pipeline, so that stage is recorded skipped with reason disabled_by_config. The fitted pipeline is returned in memory; persist it with save().

Sub-record	Purpose	Key fields
`MLflowSettings`	which run this fit logs to	`experiment_name`, `run_name`, `run_description`, `nested`, `custom_tags`
`ModelRegistrySettings`	where to register on success	`model_name`, `unity_catalog_model_name`, `unity_catalog_aliases`

What lands in the run besides the model¶

Params: the run's non-default config only (the same stripped diff a preset round-trip uses), flattened to dotted keys: engine.time_limit, hpo.provider, and so on. Logging the full config would bury the handful of knobs that differ between two runs.
Metrics: the fit-time headline numbers: the constant-baseline gate's strongest model and baseline scores on the run's metric, the fraction of the baseline's error removed (baseline_improvement), and the measured serving cost (infer_cost_s_per_row). Outer-test scores are not produced here, because fitting is not reporting: they come from evaluate(), which the fit does not run. They do end up in the same run; see The honest test score.

The two gate scores are logged in the metric's own units. Internally they live in AutoGluon space, where every metric is higher-is-better and a loss carries a flipped sign, so a run's RMSE is held as -0.88. That is fine while the number travels next to the field naming its metric (which is how measures.json carries it), but an MLflow metric key is the name, and its compare-runs table is a sort-by-this-column surface: a column called fit_root_mean_squared_error showing -0.88 reads as an RMSE of −0.88 and ranks two runs backwards. baseline_improvement is deliberately left alone: a fraction of error removed has no direction to correct.

Tags: the provenance that qualifies those fit_* numbers, on the one other plane a compare-runs view reads. A metric key is a bare number; these say how it was produced. They are written alongside your own custom_tags, never instead of them.

Tag	Values	What it tells you
`ml4.fitting_set`	`oof` / `holdout`	Which rows every post-fit layer (calibration, decision threshold, conformal residuals, the baseline gate) was fitted on: the bagged out-of-fold pass over every training row, or a reserved slice the deployed model never trained on.
`ml4.strongest_score_basis`	`untuned` / `out_of_fold` / `in_sample`	How the strongest model's decision was selected before `fit_<metric>` was computed. `untuned` = no decision parameter came from these rows (regression, a probability metric, a binary threshold that refused to tune, or a multiclass run serving plain argmax). `out_of_fold` = the binary operating point was re-selected fold-wise first. `in_sample` = the parameter was selected on the very rows scored, so the number is an upper bound, not a measurement: a binary set too thin to re-select fold-wise, or any multiclass run whose tuned decision layer serves (its K one-vs-rest bars and the comparison that kept them all came from these rows, and no fold-wise re-selection of that composite rule exists).
`ml4.decision_layer_revision`	`1`, `2`, …	Which revision of the decision layer (calibration + the decision threshold + the baseline gate) produced this run's numbers. `1` at fit; higher when the layer was re-decided afterwards without retraining. A re-logged model is otherwise indistinguishable from the one it replaced.
`ml4.ensemble_weights_selected_in_sample`	`true` / `false`	OOF runs only. `true` when the strongest model is AutoGluon's greedy weighted ensemble, whose blend weights were hill-climbed on the very rows the score is computed over. Absent on the holdout path, where the question cannot arise.

There is no fit_<metric>_at_tuned_threshold key: the basis is a per-run fact, not a per-run key name. A different column name would break the sort a compare-runs table exists for.

Re-deciding the layer is not a refit, and the registry must not confuse the two¶

A refit retrains the model. Re-deciding the decision layer does not: it re-fits calibration, the decision threshold and the baseline gate from the strongest model's own stored out-of-sample output, in seconds, and leaves the trained model untouched. The distinction matters to MLOps in three places:

Model selection stays frozen. The leaderboard ranked candidates on the calibration bake-off's criterion, so a re-decided layer means the strongest model was selected under one layer and serves another. Re-running selection would cost what the cheap path exists to avoid, so the discrepancy is reported instead, as decision_provenance.selection_consistent.
A registered model owns its bytes. Registration takes a model-private snapshot of the artifact, so re-deciding an experiment's layer does not reach a model registered from it. That is deliberate: a production model must not change because someone adjusted the experiment it came from.
The revision travels. ml4.decision_layer_revision is on the MLflow run, on the exported airgapped scorer, and on every derived artifact, because a number is only attributable to a model and a layer, not to a model alone.

The honest test score lands in the same run¶

fit_<metric> is measured on the rows the post-fit layers were selected on. The number a reader of the run wants is the one measured on rows nothing was fitted to, and that one only exists after evaluate(), which happens in a separate pass, after the fit's MLflow run has already been closed. evaluate() therefore writes it back:

model = asv.fit(train_df, mlops=mlops)      # logs fit_* + the provenance tags, closes the run
measures = asv.evaluate(test_X, test_y)     # adds test_* to that same run

No knob: the façade forwards the run id it recorded on mlflow_run_id_, which is set only by a fit(mlops=...) that really logged. A run id can also be passed straight to the library function, parleml_ml4.evaluation.evaluate(pipeline, X, y, mlflow_run_id="…"), which is how you attach a report to a run this process did not create.

Metric	What it is
`test_<metric>`	The deployed model on the test set, at the frozen operating point, in the metric's own units, directly comparable with `fit_<metric>`.
`test_<metric>_ci95_low` / `_high`	The bootstrap 95% interval around it (absent when `bootstrap=0`). Converted and re-sorted with the score, so a loss metric's interval does not come back inverted.
`test_rows`	Rows handed to `evaluate()`. A score without its `n` is not interpretable.
`test_unscored_rows`	Rows carrying a class the model has no column for; present only when there were some.
`test_<objective>_at_operating_point` / `test_<objective>_optimistic`	Binary only. The deployed cut vs a cut re-tuned on the test set itself. The gap is what threshold selection would have bought you in hindsight; the first number is the one to trust.
`test_calibration_brier_raw`, `test_interval_coverage_*`, …	Every other number the report produces, verbatim.

Failures here are per-key and never fatal: a rejected metric costs a warning naming the run, and the measures dict evaluate() returns is identical either way.

The pyfunc: labels vs probabilities¶

The logged model is an Ml4PyfuncModel wrapping the ml4 bundle. Classification defaults to probabilities (the contract downstream consumers expect); params={"output": "labels"} returns thresholded labels instead. Regression always returns point predictions.

import mlflow.pyfunc

served = mlflow.pyfunc.load_model("runs:/<run_id>/model")
proba  = served.predict(X)                              # default: probabilities
labels = served.predict(X, params={"output": "labels"})  # thresholded labels

Set the pseudonymisation key where the model is served

A model that pseudonymises identifier columns (card numbers, emails, phones, IPs …) scores only under the key it was fitted with, and does not store that key. Set ML4_PSEUDONYMISATION_KEY in the serving environment to the training environment's value before mlflow.pyfunc.load_model; loading refuses to start without it and names the key fingerprint the model needs. See The pseudonymisation key.

The output param is declared in the model signature, so callers keep the label-vs-proba switch even when a full input/output signature is inferred (which happens automatically when an input_example is supplied; AutoSupervised.fit supplies a small one for you).

Requirement pinning¶

A model logged on the training box must load and serve in a fresh environment. log_pipeline therefore pins a deployed-chain-only pip_requirements:

the always-present serving chain (autogluon.tabular, pandas, numpy, scikit-learn, scipy, mlflow, cloudpickle), each pinned to its installed version;
the always-present feature-engineering chain: category_encoders, kmeans1d, skrub, RapidFuzz, disposable-email-domains, phonenumbers, tldextract, user-agents. These are not a model dependency, which is why they are easy to miss. The AutoGluon learner carries ml4's global feature generator, that object holds a reference to build_global_transformer, and restoring it therefore imports parleml_ml4.fe.transformers, whose __init__ imports every transformer module. It happens inside TabularPredictor.load, on every ml4 pipeline, whatever the task, roster or recipe. The libraries that are try-guarded or imported lazily (usaddress, pycountry, pgeocode, names_dataset, holidays) are deliberately not pinned: a recipe that never uses them must not pay for them;
leaf-library pins so the strongest model can be unpickled: LightGBM for a GBM deploy, XGBoost for XGB, CatBoost for CAT, imodels for any glass-box, and interpret-core for EBM. LightGBM is also pinned when the interval layer has CQR companions (they are LGBMRegressor);
torch only when a torch-backed family (e.g. NN_TORCH, FASTAI, TabICL) is deployed, so a plain GBM deploy keeps a small serve image.

The serving Python is pinned to the training interpreter, and the ml4 release is pinned as a requirement (parleml-ml4==<version that fitted the model>) so FittedPipeline.load can import parleml_ml4 at serve time. A model never carries ml4's code, so the serving environment must be able to install that release: publish the wheels somewhere it can reach and pass the index or --find-links line in extra_pip_requirements. Logging fails when the installed ml4 has no readable version.

log_pipeline also takes extra_pip_requirements, appended verbatim, for things ml4 cannot infer, such as a private index URL or a company-internal wheel. It is a log_pipeline argument only: the AutoSupervised.fit(mlops=…) path does not expose it, so reaching it means calling parleml_ml4.delivery.mlflow_logging.log_pipeline yourself. Nothing ml4 itself needs belongs there. If a serve image is missing an ml4 dependency, that is a defect in the pins above, not something to work around per caller.

Registry vs Unity Catalog¶

Registration is driven by ModelRegistrySettings. A plain tracking run leaves the model unregistered.

Registry target	How to trigger
MLflow Model Registry	`ModelRegistrySettings(model_name="risk_model")` (a plain name)
Databricks Unity Catalog	`ModelRegistrySettings(unity_catalog_model_name="catalog.schema.model")` (a three-level name, registry URI `databricks-uc`)

mlops = MLOpsRunSettings(
    mlflow=MLflowSettings(experiment_name="risk-q1"),
    registry=ModelRegistrySettings(
        unity_catalog_model_name="main.risk.pd_model",   # 3-level → Unity Catalog
        unity_catalog_aliases=["Production"],
    ),
)
model = AutoSupervised(cfg).fit(train_df, mlops=mlops)

Under the hood, log_pipeline takes registered_model_name (the UC three-level name if given, else the plain registry model_name). A three-level name registers to Unity Catalog when the registry URI is set to databricks-uc.

Aliases¶

The first entry of unity_catalog_aliases is set as a registry alias (e.g. Production) pointing at the just-registered version. Aliasing is best-effort: a failure never fails the log, since the model is already registered.

Serving from the registry

Once registered with an alias, load by alias with mlflow.pyfunc.load_model("models:/main.risk.pd_model@Production"). The same labels-vs-proba output param applies.

Next: Deployment & export · Inference & fast scoring