Skip to content

Evaluation, trust & model cards

Three read-only layers turn a populated ProfilingResult into the numbers a model owner signs off on. They never re-fit or do I/O, and they degrade gracefully when an upstream artifact is missing.

In short

  • Pass labelled X_test / y_test (and X_valid / y_valid) to fit_profiler for held-out evaluation, or score stored predictions with evaluate_from_predictions.
  • compute_label_noise lists rows whose label a model that never trained on them disputes. A flag is a candidate for review, not a confirmed mislabel.
  • compute_trust_summary(result) grades nine tiles. Missing evidence grades yellow, never as a pass.
  • build_model_card(result) assembles a structured model card. AI-written text enters only as a labelled annotation.

Agent access

This surface is exposed to AI agents as the MCP tools get_model_evaluation (metrics, curves, per-threshold confusion) and get_insight("trust_summary" | "business_value" | …) — see the ParleML MCP docs site (parleml_mcp/docs/).

Holdout evaluation

parleml_mli.evaluation scores binary / multiclass / regression holdouts: threshold sweeps, ROC / PR / KS / lift / gains curves, calibration (Brier, ECE, reliability), and expected-cost lenses (binary τ-sweep and multiclass K×K).

The simplest path is to pass holdout frames straight to fit_profiler (data arguments — no config section required for evaluation):

from parleml_mli import ProfilingConfig, fit_profiler

config = ProfilingConfig()
result = fit_profiler(
    model=clf, X=X_train, y=y_train, target_mask=(y_train == 1),
    X_test=X_test, y_test=y_test,        # → result.model_evaluation_
    config=config,
)
ev = result.model_evaluation_

…or compute it lazily on a populated result:

from parleml_mli import build_model_evaluation_result

result.model_evaluation_ = build_model_evaluation_result(
    result,
    splits={"valid": (X_valid, y_valid), "test": (X_test, y_test)},
    X_train=X_train, y_train=y_train,
)

For bring-your-own / precomputed scores (no live model), build the bundle straight from predictions:

from parleml_mli import TaskType
from parleml_mli.evaluation import evaluate_from_predictions

ev = evaluate_from_predictions(
    task_type=TaskType.BINARY,
    splits={"test": (y_true, y_scores)},
)

A binary split's scores must be positive-class probabilities in [0, 1]; a split whose scores fall outside (a margin, a label) is refused and skipped with a warning rather than given a clipped, plausible Brier score. The proper scoring rules are ml4's own: every Brier score goes through parleml_ml4.delivery.scoring.brier_score and every log-loss through parleml_ml4.delivery.scoring.proba_log_loss, so one model shows one number wherever it is reported. Log-loss is defined on a single-class split too; ROC / PR AUC are not (None).

Calibration and the cost-matrix lenses are first-class. Every split persists only its raw (y_true, scores) sample (capped at 200k rows); the threshold sweep, ROC/PR, lift/gains and reliability curves are re-derived on demand from it via parleml_mli.evaluation.derive, so nothing heavy is cached on the joblib.

Readers that quote a single split — the Trust tiles, the model card, the narrators — all pick it with ModelEvaluationResult.preferred_split(): test, else valid, else holdout, else oof, else the first split stored.

from parleml_mli.evaluation import (
    compute_calibration,
    evaluate_business_value,
    evaluate_multiclass_cost,
)

cal = compute_calibration(y_true, y_scores)      # Brier / ECE / reliability

# Binary cost / capacity / Fβ lens over one evaluated split. It sweeps a single
# decision threshold τ; pass a ClassificationSplitEvaluation, not raw arrays.
bv = evaluate_business_value(
    ev.splits["test"],
    cost_fp=10.0, cost_fn=50.0, gain_tp=100.0,
    objective="expected_value",                  # or "fbeta" / "capacity"
)

# Multiclass analogue: a full K×K cost matrix, no threshold to sweep — the
# decision is argmin_j Σ_i P(i|x)·C[i, j]. Pass a MulticlassEvaluation.
mc = evaluate_multiclass_cost(
    mc_ev.splits["test"],
    cost_matrix=C,                               # (K, K): C[i][j] = cost(pred j | true i)
)

See the evaluation reference for every export.

Label-noise scan

compute_label_noise finds likely-mislabelled rows using confident learning (Northcutt et al., 2021): it compares each row's given label to what a model that never trained on the row confidently predicts. A row is flagged only when some other class both clears its own per-class confidence threshold and out-scores the given label — so rows the model merely finds hard (no class is confident) are left alone. Classification only; implemented directly, with no cleanlab dependency (it is AGPL-licensed).

from parleml_mli.evaluation import compute_label_noise, build_label_noise_result

# Pure core — you already have out-of-fold probabilities:
res = compute_label_noise(y_train, oof_probabilities, classes=model.classes_)

# Or let it produce OOF for a bring-your-own sklearn estimator:
res = build_label_noise_result(y_train, estimator=clf, x=X_train)  # cross_val_predict

res.suspects            # ranked review queue: row_index, given/predicted label, scores
res.estimated_noise_rate
res.per_class_noise

Pass every row you have an honest out-of-sample probability for: mislabels are not confined to any one cohort, and a model cannot dispute a label it trained on. In ParleML that is every labelled row a model scored without training on it: the training rows' out-of-fold predictions on a bagged run, the reserved validation slice and the outer test.

When those rows were scored by different models — out-of-fold rows by the fold models, the rest by a refit that is sharper on its own rows — pass threshold_rows, a mask of the rows that set each class's threshold. ParleML passes its out-of-fold rows, so every row is judged on the fold models' scale.

estimated_noise_rate is confident learning's estimate of the share of wrong labels, not n_flagged / n_total. Per given class, per_class_noise is the share of the rows the model is confident about that it assigns to another class (the calibrated confident joint's off-diagonal). A row the model is confident about no class for is evidence neither way, so it is left out rather than counted as clean. The overall rate weights those shares by how common each class is, so it is at least the flagged share. With method="self_confidence" every row counts and the two coincide.

Treat suspects as a review queue of candidates, not verdicts; some flagged rows are genuinely ambiguous cases worth a second look either way. It is a standalone artifact (no profiler stage), and parleml_mli.evaluation.plots.plot_label_noise(res) renders the per-class rate.

Trust Summary

compute_trust_summary(result) grades a model on nine traffic-light tiles (model lift, calibration, drift, fairness, error pockets, rules that hold up, archetype coverage, leakage findings, stability) using a central, tunable threshold table. These are diagnostic heuristics, not a certification or a deployment guarantee. It's a pure consumer — tiles whose artifact is missing grade "yellow" with an explanatory subtext rather than failing.

from parleml_mli import compute_trust_summary

trust = compute_trust_summary(result)
print(trust.overall_grade, trust.headline)
for tile in trust.tiles:
    print(tile.key, tile.badge, tile.value, tile.subtext)

It reads what's already on the result (model_evaluation_, explanations_.fairness, eda_.leakage_report, rules, archetypes), so populate those artifacts first for a complete grade. You can also override individual inputs:

trust = compute_trust_summary(
    result,
    fairness_report=result.explanations_.fairness,
    leakage_report=result.eda_.leakage_report,
)

The drift tile needs a genuine monitoring report. The profile's own explanations_.psi compares target class vs reference class — that is Information Value (separability, where big is good), not drift, so the tile degrades to an explanatory yellow rather than grading it. Only a PSI whose comparison is "monitoring" is graded (the model card's drift section and the narration prompts apply the same rule). Pass a train-vs-current PSI:

from parleml_mli.explanations.drift import compute_psi

drift = compute_psi(X_current, X_train,
                    binning_strategy="reference", comparison="monitoring")
trust = compute_trust_summary(result, drift_report=drift)

The rules tile grades rules on held-out rows. "Rules that hold up" (rules_hold_up) is the share of the default engine's rules (every class's, pooled, on a multiclass run) whose verdict on held-out rows is "holds up" — green from 60 %, yellow from 30 %. A rule covering too few test rows counts as not shown to hold up, so a small held-out set grades lower; the tile's details count the rules that hold up, do not, or cover too few test rows, and carry the held-out basis. When the test rows come from a temporal split the subtext adds that a failing rule may reflect drift. Rules never measured on held-out rows (profiled without X_holdout / y_holdout) are not assessed: a yellow "n/a", like any missing artefact. See Train and Test numbers.

The thresholds live in TRUST_THRESHOLDS and the tile order in TRUST_TILE_KEYS — see the trust reference. TrustSummary.schema_version (parleml_mli.trust.TRUST_SCHEMA_VERSION, 2) is bumped whenever a tile key or its meaning changes.

Model cards

build_model_card(result, ...) assembles the canonical multi-section ModelCard JSON (cover → recommendations, plus the trust summary, provenance, and a schema version). The library produces data; the backend renders DOCX / HTML / PDF.

from parleml_mli.autodoc import build_model_card

card = build_model_card(
    result,
    fitted=fitted_pipeline,     # optional ml4 FittedPipeline → "Final pipeline" section
    dataset_meta=dataset_meta,  # optional dict / row satisfying DatasetMetaLike
    trust_summary=trust,        # optional — recomputed from result if omitted
)
card.sections                   # the section payloads

Sections whose underlying artifact is missing surface as {"available": False} so the renderer can hide the row gracefully. See the autodoc reference and MODEL_CARD_SECTIONS for the section list.

AI-written text reaches the card only as an annotation the host attaches — annotations={"explain.narration": CardAnnotation(value=narration, label=...)} (and "explain.fields" for column descriptions). Every section that prints it carries the label, card.ai_written lists those sections, and without annotations the card uses its deterministic templates and says it contains no AI-written text. See AI-written narration.

Next