Ranking metrics & objectives¶

Ranking is not a workload of its own. It is the shared evaluation surface that the recommendation and Learning-to-Rank backends both emit, and that multiclass classification reuses verbatim. Every function in parleml_ml4.ranking operates on a calibrated (n_rows, n_classes) score matrix, so a One-vs-Rest recommender (many binary sub-models) and a single LTR booster produce the identical ranking.json payload. Learn the vocabulary once and it applies to every ranked workload.

In short

Every ranked workload (recommendation, Learning-to-Rank, multiclass) emits the same ranking.json payload from a (n_rows, n_classes) score matrix.
The report gives two competing views, accuracy (micro) and coverage (macro), and never picks a winner for you.
ranking_objective (accuracy / balanced / coverage) shapes the cross-product sort without retraining anything.
Recommendation and LTR grade different populations, so their ndcg@10 values are not comparable to each other.
Check report["baselines"] and the uncertainty verdict before quoting a lift.

from parleml_ml4.ranking import build_ranking_report

report = build_ranking_report(proba, y_true, objectives=("accuracy", "coverage"), k=10)

Two metric families (there is no single winner)¶

Ranking quality has two legitimate but competing views. The report emits both for every objective and lets you choose. It never silently auto-selects by NDCG, which always flatters the accuracy objective.

Accuracy / microCoverage / macro

Pool all decision occasions and reward getting the frequent products right. This is the headline view for "did we recommend the right thing".

Metric	Meaning
`hit@1`, `hit@5`, `hit@{k}`	Was a true item in the top-1 / 5 / K?
`mrr@{k}`	Mean reciprocal rank of the first hit.
`ndcg@{k}`	Position-weighted, normalised 0–1: the headline.

Average per class so a rare product counts as much as a popular one. This diagnoses popularity collapse: a model that only ever surfaces the head of the catalogue.

Metric	Meaning
macro-recall@K	Hit@K averaged per class rather than per row.
`catalog_coverage`	Expected share of the catalogue that appears in at least one top-K list.
`tail_hit@{k}`	Hit@K restricted to rare (long-tail) classes.
recommendation entropy / Gini	How evenly recommendations spread across the catalogue.

All three catalogue diagnostics are expectations over tie-breaks, not counts of one realised list.

Details: why tie-breaks are averaged

top_k resolves a tie by column order, so counting a single list would describe the draw that always prefers the lowest-indexed class. Ties here are structural, since a gate-dropped class is an exact-0.0 column. Counted on one list, a model with no preference at all would report catalog_coverage 0.5 on a 20-class catalogue at K=10, naming the alphabetically-first ten classes. Measured as an expectation it reports 1.0, which is what "no preference" means. catalog_coverage uses the exact 1 - Π(1 - P(in top-K)) per class. Entropy and Gini are computed from the expected per-class exposure counts, which sum to n_rows · K by construction.

Objectives: how the cross-product sort is shaped¶

Calibration makes a single model's probabilities trustworthy, but two independently-calibrated binary models still live on incomparable scales when their base rates differ: a 0.4 for a 1%-base-rate product is far stronger evidence than a 0.4 for a 30%-base-rate one. The comparability transforms re-express scores so the cross-product sort is fair. They shape the sort key on top of calibration. They retrain nothing, and are all off by default.

fit_comparability(proba, y_true) learns the per-class base rates and score quantiles once (persisted as a ComparabilityState); transform_for_objective then chains the relevant transforms for a named objective.

Details: where the state is fitted, precisely

The state a run persists and serves is fitted on the holdout score matrix at the end of the fit, not on the training rows. Base rates come from the labels, so they are honest; the score quantiles (used only by the percentile path) describe the holdout's score distribution. It is written to deploy/comparability.joblib and re-used unchanged by the batch re-rank and the α slider, so the operating point you tune is the one that serves.

Two paths differ deliberately. The LTR hyper-parameter search fits its own state from the training positives only (base rates, no quantiles), because every trial must be ranked through the same leak-free state. A per-trial refit would make trials incomparable to each other.

A caller who invokes transform_for_objective with no state at all gets a refusal naming what needed it, not a state derived from the matrix being scored. Such a fallback would make one entity's ranking depend on who else was in the batch: the same entity's top-1 can flip between two products purely on the composition of the request. There is nothing local a caller could honestly substitute for training base rates. The single exception is the combination that reads no base rates at all: α = 0 with no shrinkage and no percentile, which is the identity and needs no state.

Transform	What it does	Pulls toward
`apply_eb_shrinkage`	Empirical-Bayes shrinkage of each class's score toward its base rate (denoises small-sample classes).	absolute / accuracy
`apply_rank_percentile`	Ranks a score within its own class's score CDF before sorting (scale- and base-rate-invariant).	rare classes / coverage
`apply_popularity_discount`	Down-weights head classes by `α` (`OBJECTIVE_ALPHA`).	long-tail surfacing

Shrinkage and the percentile are mutually exclusive

Requesting both raises. Shrinkage is a per-class monotone map and the percentile is rank-based, so a correctly-referenced percentile of a shrunk score is identical to the percentile of the raw score. The combination can only ever be a no-op or a bug, and in practice it is a bug: the percentile looks a score up in that class's raw reference distribution, so the shrunk value would be mapped into a distribution it does not belong to. That promotes exactly the small-sample classes shrinkage exists to damp.

Pick the one that matches the problem: shrinkage to denoise small classes on the score scale, percentile to make classes comparable regardless of scale.

The three named objectives¶

ranking_objective (a RankingObjective) selects a preset α along that slider:

Objective	Emphasis
`accuracy` (default)	Get the frequent items right (`α = 0`).
`balanced`	A middle ground between head accuracy and tail coverage.
`coverage`	Surface the long tail; rare classes count fully.

Two optional knobs override the preset when set: popularity_discount (the raw α) and shrinkage_strength (the EB shrinkage weight, off when None). They are RecoConfig fields; LTRConfig has neither, and the two backends expose them differently:

	Recommendation (OvR)	Learning-to-Rank
`popularity_discount`	`RecoConfig.popularity_discount`, overridable per call via `recommend(popularity_discount=...)`	no config field; per call only: `recommend(popularity_discount=...)`
EB shrinkage	`RecoConfig.shrinkage_strength` sets the weight; `recommend(shrinkage=True)` turns it on (falling back to `50.0` when no weight is configured)	no weight knob at all: `recommend(shrinkage=True)` toggles it on at a fixed `50.0`

So on LTR both are predict-time-only: the sort can be re-shaped on a fitted pipeline without re-fitting, but the shrinkage strength is not tunable.

ranker_objective vs ranking_objective

In Learning-to-Rank these two similarly-named knobs do opposite jobs. ranker_objective is the training loss (how the booster fits, e.g. lambdarank); ranking_objective is the evaluation/sort objective (how the holdout is scored and ranked here). See Learning-to-Rank.

The report shape¶

Both backends write the same payload, so one React panel renders monolith and One-vs-Rest runs through one /ranking route:

report = build_ranking_report(proba, y_true, objectives=("accuracy", "coverage"), k=10)
# {
#   "objectives": {"accuracy": {"micro": {...}, "macro": {...}}, "coverage": {...}},
#   "objective_alpha": {"accuracy": 0.0, "coverage": 1.0},
#   "baselines": {...},
#   "uncertainty": {...},          # see "How sure is the lift?" below
#   "primary_score": {"metric": "ndcg@10", "value": ...},
#   "schema_version": 3,
# }

The primary score key is f"{primary_metric}@{k}" (default ndcg@10). Set eval_all_objectives=True (the default on RecoConfig) to score every objective in a single report.

objective_alpha records the popularity-discount α each block was actually built with, so a reader can tell a genuine trade-off curve from one operating point repeated. It is null for a block that ranked by percentile, where no α applies.

Details: a configured popularity_discount applies to the operating block only

An objective is its α, so each comparison block uses its own. Forcing a configured popularity_discount onto all of them would make the three byte-identical while they stay labelled accuracy / balanced / coverage, and the trade-off would render as a flat line.

The primary_objective block is different. It is the run's operating point, the α FittedRecoPipeline.recommend serves at, so it is measured there. Otherwise primary_score, the headline, would describe a ranker the product does not ship. objective_alpha in the payload records the α each block used, so the two are always distinguishable.

Ties are resolved by averaging the metric, not the rank¶

Tree ensembles produce exact score ties constantly, and in a One-vs-Rest run they are structural: a gate-dropped class is an exact-0.0 column for every entity.

Every metric here is therefore the expectation of that metric over the tie group: if the realised class ties with t-1 others just past b strictly-better classes, its rank is uniform on [b+1, b+t] and the metric is averaged across those positions.

Details: why not score the average rank

Scoring the tie group's average rank is the intuitive-but-wrong shortcut: substituting E[R] into a non-linear metric is not E[f(R)]. For hit@k the difference is not even conservative. A constant-score model over C classes sits at rank (C+1)/2, which clears the cutoff outright and reports a perfect hit@k = 1.000 for every catalogue with k < C ≤ 2k-1 (at the default k=10, any 11-to-19-product catalogue). rank_of_truth still returns the average rank, but it is a diagnostic only; the metrics consume rank_bounds_of_truth instead.

The evaluation protocol — which rows produced these numbers¶

Every score on this page is an average over a specific population, and the two ranking backends do not use the same one. Read this before comparing runs.

What is graded. One ranked list per evaluation unit, scored against a single realised truth per list:

	LTR	recommendation (OvR)
evaluation unit	one decision occasion (`entity`, or `(entity, bundle/day)`)	one entity (long grain) · one event row (event grain)
candidate set	the modelled catalogue, capped at `max_products` (default 500)	every modelled class
truth	the occasion's realised positive(s), one graded list each, so a multi-item basket contributes several (`n_eval_lists` ≥ `n_eval_groups`)	long: one graded list per `(entity, positive)` · event: the product taken
features	the occasion's first row, tiled across the catalogue	long: an entity row (per-candidate columns are rejected at `fit`) · event: the entity's previous event, so the graded row's own values never reach the model
holdout	`split_strategy` (default: temporal if timestamped, else entity)	whole entities, stratified by whether the entity carries a positive; newest by first appearance when timestamped

The denominators differ, and that is the trap. LTR drops holdout occasions whose realised product falls outside the capped catalogue — they never enter the mean. Recommendation keeps an entity whose truth was not modelled and scores it as a miss. Both emit a key called ndcg@10. They are therefore not comparable to each other, and LTR's is structurally the flattered one. Compare a run only against another run of the same backend, or against the popularity baseline inside its own report.

Details: other protocol decisions worth knowing

Entities with no positive in the holdout are dropped, not scored as zero. So n_rows in the report is the number of graded lists, not the number of entities.
A long-grain entity with several holdout positives contributes one truth, taken as the first in input order. The rest are counted as misses no matter how well they rank, which biases the metric downward, hardest against your most active entities.
n_unseen_truth counts graded rows whose realised class is not a column of the score matrix (unmodelled or gate-dropped). They stay in the denominator.
hit@1 / hit@5 are fixed cutoffs, emitted regardless of k; only mrr@k, ndcg@k and hit@k follow the configured k.
Macro is an unweighted mean over classes of unequal support, so a class with three eval rows counts as much as one with three thousand.
Intervals cover the micro accuracy metrics only. With a few hundred graded lists the standard error on hit@k is large. Read the bootstrap verdict (How sure is the lift?) before treating a small difference as real; coverage, macro and value metrics carry no interval.

Honest baselines¶

parleml_ml4.ranking.baselines supplies the reference points a model must beat, so a strong-looking NDCG is not mistaken for signal when "just recommend the popular things" already does as well:

GlobalPopularityBaseline: recommend the globally most-popular items to everyone.

It is scored through the same metric families as the model and lands under report["baselines"].

baselines is empty unless y_train is supplied

The baseline is fitted from the training class marginal, so build_ranking_report only produces one when it is given y_train. Without it the key is present but empty, and a headline ndcg@10 with nothing to compare against is not evidence of signal. Check report["baselines"] is non-empty before quoting a lift.

A floor already fitted on a training marginal can be passed instead (popularity=, e.g. rebuilt from persisted base rates for a scored batch); pass that or y_train, not both.

How sure is the lift?¶

Hit@2 0.418 against popularity's 0.393 is two point values; on 600 holdout customers a +0.025 edge can be noise. report["uncertainty"] (parleml_ml4.ranking.uncertainty.ranking_uncertainty) puts a paired percentile bootstrap interval on every micro accuracy metric of the served ordering (the primary objective), of each floor, and of each model-minus-floor gap (the number a decision rests on):

report = build_ranking_report(proba, y_true, k=2, y_train=y_train, groups=entity_per_row)
gap = report["uncertainty"]["baselines"]["global_pop"]["difference"]["hit@2"]
# {"estimate": 0.025, "ci_low": -0.011, "ci_high": 0.062, "verdict": "undecided"}

The unit is the entity (groups=, one label per row): one customer's ranked lists are resampled together, since they are not independent. The recommender passes the holdout entity, learning-to-rank the occasion's entity (Davison & Hinkley 1997 §3.8; Field & Welsh 2007). Without groups each row is a unit (unit: "row").
Paired: model and floor are scored on the same resamples, so the gap's interval carries only the noise in the gap (Sakai 2006; Smucker, Allan & Carterette 2007).
Percentile interval (Efron & Tibshirani 1993, ch. 13) via scipy.stats.bootstrap; 1000 resamples, 95 %, seed 0; the point estimate is the report's own number.
verdict is model_ahead / baseline_ahead when the whole interval is above / below zero and undecided when it straddles zero. n_units states the evidence.

uncertainty is None below two units. Coverage, macro and value families carry no interval.

Top-K selection at serve time¶

parleml_ml4.ranking.topk.top_k turns a score row into the ranked (class_name, score) picks a caller serves:

from parleml_ml4.ranking import top_k

picks = top_k(proba, k=5)     # top-5 class names + scores per row