Recommendation¶

Recommender is ml4's next-best-product engine. Given per-entity holdings and outcomes it produces a calibrated (n_entities, n_classes) score matrix over the shared ranking surface, then reports Hit@K / MRR@K / NDCG@K and serves top-N recommendations.

In short

Recommender trains one calibrated binary sub-model per (product, subcase) and ranks the products for each entity on the shared ranking surface.
Pick the input shape with RecoConfig.grain: long (one candidate row per entity and product, the default) or event (one row per purchase).
Features must describe the entity. fit() refuses product-identifying columns (event grain) and per-candidate columns (long grain).
Each sub-model passes a ROC quality gate or drops out. Read n_val_pos and roc_test before trusting a verdict.
The core install is enough; recommendation needs no extra of its own.

from parleml_ml4.recommendation import Recommender
from parleml_ml4.config.recommendation import RecoConfig

cfg = RecoConfig(entity_col="customer_id", target_key_col="product", label_col="bought")
model = Recommender(cfg).fit(df)
result = model.result_

No extra of its own

Recommendation drives the core ml4 AutoGluon supervised engine through a trainer seam, so it needs no dedicated optional dependency. The core install is enough.

The One-vs-Rest approach¶

Recommender does not train one monolithic multiclass model. It trains one calibrated binary sub-model per (class, subcase) and assembles their predictions into a single calibrated (n_entities, n_classes) matrix. Each column is a product's propensity. The matrix is ranked across products on the shared surface, with the same ranking.json contract the multiclass and Learning-to-Rank backends emit.

Details: every column carries its class's population prior

Each sub-model trains on a resampled population: its positives plus neg_ratio negatives per positive, most of them hard-mined lookalikes by default. Its raw probabilities therefore sit at that resampled prevalence. Undoing a uniform negative subsample kept at rate β is exact (Dal Pozzolo et al. 2015: p' = β·p / (β·p − p + 1)). A hard negative, however, is kept because it looks like a buyer, so no single β undoes hybrid mining.

Each sub-model's PriorCorrection therefore recalibrates its served probability on its realistic-eval pool (held-out entities, uniformly sampled negatives) with Platt's map, and applies Dal Pozzolo with that pool's rate. A sub-model too small to reserve that pool trains on uniform negatives instead, so its own sampling rate undoes the resampling exactly. The columns are then comparable across products: a class's mean score is its prevalence, under hybrid mining as under random.

Details: calibrated reports the served column, not what was asked for

The per-sub-model calibrator of the binary fit is gated: fit_calibration_if_beneficial runs a k-fold bake-off and serves raw probabilities when a calibrator would not beat them (and always below its rare-class floor). The recalibration above applies either way. A sub-model's calibrated scorecard flag (TrainedSubmodel.column_calibrated) is true whenever its column was recalibrated on a realistic-eval pool, and otherwise only when its own calibrator shipped.

The (n, K) matrix is always on one scale. Unless every column is calibrated, it is assembled from raw probabilities for every class (RecoEnsemble.uniformly_calibrated), each with only its training sampling rate undone. A calibrator is a monotone per-class warp, so it never changes the ranking within a class. This matrix is compared across classes, though, and there a mix of warped and unwarped columns is a scale error. Per-class calibrated probabilities remain available on each sub-model.

The fit pipeline runs these stages:

Entity / time holdout: carve a holdout of entities (time-based when a timestamp_col is present) for honest ranking evaluation.
Select modelable classes: keep products with at least min_positives positives (default 30), capped at max_classes (default 200). Drops are recorded with reasons.
Class-conditional subcases: from each entity's holdings, split a class's candidates into subcases (existing / returning / new), so an upgrade is modelled separately from a first purchase.
Per-sub-model population: assemble positives plus sampled negatives for training, and a held-out realistic-eval pool of positives plus uniformly sampled negatives.
Binary fit: delegate each sub-model to the AutoGluon trainer. The FE recipe is planned on planning_frame, not on the raw train split. The light per-sub-model budget sets the train/validation policy, so every sub-model reserves a post-fit holdout, and a target-aware planner must not read the labels of the rows its own calibration and threshold are later fitted on.
ROC quality gate: sub-models below min_submodel_roc on validation, or below chance on their held-out realistic-eval pool, are excluded. What a failing subcase falls back to depends on the class's model type (see The quality gate).
Assemble + rank: build the calibrated matrix and the ranking report. Each column is recalibrated on its sub-model's realistic-eval pool and mapped onto the class's population prior.

Input grains¶

RecoConfig.grain selects the input shape:

long (default)event

One row per (entity, target_key) candidate with a binary label_col. The caller pre-materialises the negatives.

cfg = RecoConfig(grain="long", entity_col="customer_id",
                 target_key_col="product", label_col="bought")

One row per purchase event. target_key_col is the product taken (an implicit positive) and label_col is unused. The engine derives positives and negatives per sub-model, guarding against false negatives within the sub-model's population: an entity that took the product in that state is never one of its negatives. For unified and new that is "an entity that bought it anywhere". For existing it is "an entity that bought it again", so a holder who bought something else is an existing negative. (A class-wide guard would leave every existing / returning sub-model without a single negative, because every row in those states belongs to a buyer.)

Holdings must be as of just before each event, the state the purchase was made from, as held_products / prior_products are in the reco_cross_sell_events sample. A snapshot taken after the purchase puts every first purchase into existing and leaks the answer into the routing.

cfg = RecoConfig(grain="event", entity_col="customer_id",
                 target_key_col="product", timestamp_col="ts")

Features must describe the entity, not the taken product

In this grain every row is a realised take, so a column that is an attribute of the product on that row (plan_name, a product-specific price or amount) tells the model the answer. It also cannot be supplied at serve time, where there is one row per entity and no product has been chosen yet: the run would score beautifully and be unservable.

fit() rejects such columns by name, and so does every diagnose / drift / re-evaluation surface. A column is flagged when its repeated values each map to exactly one target_key across different entities. Aggregate them to the entity level (for example the customer's historical mean) instead of dropping the information. The ParleML launch runs the same target_determined_features check on the whole frame, as fit() does, and on the feature set the run would use. It refuses the launch with the columns named, before a run is queued.

Evaluation features are taken from the entity's previous event, never from the graded event's own row. Only columns that vary within an entity are projected. A genuine entity attribute (age, tenure, segment) is known at serve time even for an entity with no history, so it keeps its own value rather than being blanked. A row with no prior event gets NA for the event-varying columns, which is what the serve path would supply for it. Ordering is by timestamp_col when one is set, so "previous" means earlier in time rather than earlier in the caller's frame.

Long grain refuses per-candidate features

In the long grain a sub-model trains on its own class's candidate rows, but every scoring path hands one row per entity to all K sub-models. A column that varies across an entity's candidate rows would be scored with another product's value for K−1 of them. Nothing downstream can see it: roc_val / roc_test are computed per class on that class's own rows, so every internal signal stays healthy while the cross-class matrix is wrong. fit rejects such columns. The ParleML launch runs the same entity_varying_features check on the feature set the run would use and refuses the launch with the columns named, before a run is queued.

What `RecoFrame` refuses¶

Validation runs when the frame is built, and the launch preflight builds one. These are therefore launch-time messages rather than failures after a multi-hour training pass.

Refused	Why it cannot be tolerated
A key that is a missing-value sentinel string (`""`, `"NULL"`, `"nan"`, `"N/A"`, `"<NA>"`, `"none"`, `"NaT"`, matched case-insensitively after trimming)	`NaN` keys are refused too, but these survive as ordinary strings and fuse every such row into one entity or one product. The phantom entity then "buys" every class, so its whole cohort is guarded out of every negative pool. Its eval features come from an unrelated customer's row, and a sentinel `target_key` that clears `min_positives` is recommended to users as a product named `""`.
Nulls in `subcase_col`	They stringify to the literal state `"nan"`, which can earn a sub-model of its own and then fail routing at eval, after the whole training pass, pointing at the wrong cause.
Contradictory candidate rows: the same `(entity, target_key)` labelled `1` and `0` (long grain)	The entity would enter that class's positive set and its negative pool at once, training the sub-model against itself. There is no defensible resolution.

Bare "na" is deliberately not treated as a sentinel: NA is Namibia's ISO code and a plausible segment or product key.

Details: repeated candidate rows and where the feature refusals run

Merely repeated (entity, target_key) rows are not refused. They are counted on n_duplicate_candidate_rows and warned once, because they inflate n_pos. That count decides whether a product clears min_positives and enters the catalogue. It also sets the class base rate behind both the popularity discount and the popularity baseline the run is declared to beat. Repeated rows are never silently dropped; that would change your data without telling you. The check is long grain only: in the event grain a repeated pair is a legitimate repeat purchase.

The two feature refusals described under the grains above (an event-grain column that identifies the row's product, a long-grain column that varies across an entity's candidate rows) are not frame validation. They depend on the feature set, so fit() raises them, on the whole frame and before it splits anything or trains a sub-model. The ParleML launch preflight runs the same two detectors (target_determined_features, entity_varying_features) the same way, so in the product they are launch-time messages too.

How the holdout is carved¶

Every split in the vertical (the ranking holdout and the per-sub-model train/valid carve) reserves whole entities, so a customer never lands on both sides. When a timestamp_col is available the entities that appeared most recently are held out; otherwise the reservation is random. The reservation is stratified by whether an entity carries a positive.

This is a cold-start test, not a backtest

The split unit is the entity, so a held-out customer contributes all of their events including their oldest, and a training customer contributes events later than most held-out ones. No time boundary separates the two sides. The question it answers is "how well does the model rank for customers it has never seen?".

Ordering is by each entity's first event, not its last. A last-event order is monotone in how many events an entity has, so it would hold out the most active customers, who also carry the most history, and flatter every metric computed on them.

Details: ordering, stratification and missing timestamps

The time an entity is ordered by is its first event over all of its rows, never the first of the rows a sub-model selected. A sub-model's positives are the rows on which the entity bought the product, so ordering by them would put each buyer at the date of its purchase. The per-sub-model holdouts would then hold the buyers who bought late in their lives (long tenure, many products already held) and train on the early ones, against an unchanged negative pool. A model that learnt "early in life ⇒ buyer" would rank the held-out buyers below the negatives. On the reco_cross_sell_events sample that ordering gives every new sub-model a validation ROC of 0.59–0.70 and a held-out ROC of 0.33–0.47, and the recommender ranks below the popularity baseline. Ordered by the entity, the two agree.

The label is an entity-level property here: the event-grain buyer guard keeps a sub-model's positives and negatives on different entities, and long grain gives an entity one row per class. An unstratified entity split is therefore an unstratified label split. With a timestamp it is a systematically skewed one, because buying an extra product both makes an entity positive and shifts its timestamps. Left unstratified, the split produces single-class validation slices, NaN ROCs, and classes dropped from the catalogue for a split artefact. Small strata still contribute at least one entity: keeping both sides non-degenerate takes precedence over honouring the fraction exactly.

Entities with no usable timestamp are treated as oldest and stay on the train side: a null date is missing information, not evidence of recency. (Sorting them last, pandas' default, would hand them the entire holdout.) When no entity has a usable timestamp at all the split degrades to random and logs. String date columns are coerced before ordering, so "2026-1-9" cannot sort after "2026-10-01".

Negative sampling¶

Configured in parleml_ml4.recommendation.negatives. Two strategies:

Knob	Default	Meaning
`negative_strategy`	`hybrid`	Capped hybrid hard-negative mining (a lookalike cloud scores candidate negatives; the hardest `hard_neg_fraction`, below the similarity cap, mix with random draws). `random` draws uniformly.
`neg_ratio`	`5.0`	Negatives sampled per positive during training.
`hard_neg_fraction`	`0.5`	Share of negatives drawn from the hardest (most buyer-like) pool.
`similarity_cap_pct`	`90.0`	PU-trap guard: candidates above this similarity percentile are excluded (they are likely unlabelled positives).

The quality gate¶

Each sub-model must clear two checks or it is dropped from the ensemble:

check	number	reason when it fails
validation ROC reaches `min_submodel_roc` (default `0.55`)	`roc_val`: the trainer's own validation split, carved from the resampled hard-negative training distribution	`below_min_roc`
held-out ROC is not below chance, once the realistic-eval pool holds at least 30 positives and 30 negatives	`roc_test`: the realistic-negative eval pool	`below_chance_on_test`

A sub-model whose validation ROC is undefined fails the gate rather than slipping through. When the cause is a single-class validation slice the reason is reported as valid_single_class rather than the generic roc_undefined, because that is a property of the split, not of the model.

Check n_val_pos before trusting a gate verdict

Three knobs compose before the gate sees anything: min_positives × (1 - submodel_holdout_fraction) × submodel_valid_fraction. At the shipped defaults that is 30 × 0.75 × 0.2 ≈ 4 validation positives. A ROC estimate on 4 positives carries a standard error near 0.13, against a threshold sitting 0.05 above no-skill, so admission is close to a coin flip for the smallest admitted classes.

The count behind each verdict is reported as n_val_pos on the sub-model scorecard, and a run warns once with a count when any sub-model was gated on fewer than ten. All three knobs are settable on RecoConfig, as is submodel_test_fraction. Raise min_positives if the catalogue can afford it.

Details: why the below-chance check exists, and the non-reversing recalibration

A model can pass the first check while ranking real data backwards. Consistently below chance is not overfitting; it is a reversed model, and a passing validation gate does not catch it. The 30-per-class floor is where the check can trust a sub-0.5 estimate. The no-skill ROC's standard error is about 0.075 at 30/30 and about 0.06 once the negatives outnumber the positives several times over (the pool keeps every eligible negative by default). A sub-model with real skill (ROC 0.6) therefore falls below 0.5 by chance only 4–9 % of the time at the floor, and far less above it. Below the floor the check abstains.

The realistic-eval pool is also where each column is recalibrated onto the class's population prior. That map is never allowed to reverse a sub-model's order: a Platt fit on a pool the sub-model ranks backwards has a negative slope, and the column would silently serve the model upside down. Such a pool gets the best fit that does not reverse (slope 0, the pool's prevalence on every row, so the cell is scored at its class prior) and a warning. The column stays calibrated, so the (n, K) matrix stays on one calibrated scale.

When a realistic-negative eval pool is reserved (the normal path), the trainer skips its own test carve and those rows go to the fit instead, about a third more training rows per sub-model. The pool's ROC is what roc_test reports, so the carve would only be scored and then overwritten.

What a failing sub-model falls back to depends on how the class was split. There is no universal unified retry, and the difference is a real modelling caveat:

class model type	is a `unified` sub-model trained?	a subcase fails ⇒
`partial_split`: some states too thin	yes, on the whole population	routes to `unified`
`split`: every state qualified	no	routes to the most-populous surviving expert
any type, all sub-models fail	—	the class is dropped, no retry

The middle row is the one to know about. A split class never trains a whole-population model, so when (say) the new-state expert fails, new entities are scored by the existing-state model, one fitted on a disjoint conditional population. That is a weaker guarantee than "falls back to unified". Check result.submodel_records (ROC, gate outcome, calibration) and manifest.json's subcase_routing to see which cells this affected.

When no sub-model survives, the class contributes no scored column. Sub-models that raised are recorded in result.failures with the exception text. The class is dropped from the scored catalogue either way.

roc_val grades; roc_test only vetoes

Each sub-model is also graded on a reserved realistic-negative eval split (the honest production distribution) when the class is big enough to reserve one. That ROC is recorded as roc_test, alongside the one-sided roc_gap (roc_val - roc_test). How good a sub-model must be is decided on the optimistic resampled distribution. roc_test refuses it only when it is below chance on enough rows, so read roc_test to judge how an admitted sub-model will generalise.

Ranking-surface outputs¶

Evaluation runs on the entity holdout through the shared surface, so the result carries the same metric families as every ranking workload: Hit@1/5/K, MRR@K, NDCG@K, plus coverage and value metrics. See Ranking for the full metric and objective vocabulary.

ranking = model.result_.ranking            # the shared ranking.json payload
print(model.result_.classes)               # modelled products
print(model.result_.optimization_metric)   # metric the binaries were tuned on

The ranking_objective knob (accuracy / balanced / coverage, default accuracy) tilts the cross-product sort along the shared α-slider. popularity_discount and shrinkage_strength override the presets when set.

ranking["uncertainty"] bootstraps the holdout's entities (an entity's graded lists move together). It gives an interval on each micro metric and on the gap to the popularity floor, with a verdict. Read it before calling a small lift a win (How sure is the lift?).

Serving¶

pipeline = model.to_pipeline()             # FittedRecoPipeline

Pass outputs_dir= to the constructor to write events.ndjson / manifest.json and persist the pipeline under <outputs_dir>/deploy/recommender/. Runs are resumable per class (resume=True), so a re-run reloads finished classes and refits only the missing ones.

Which sub-model scored a cell — and why it was recommended¶

Every (entity, product) cell is scored by one sub-model, chosen by the entity's holdings. route_cells names it, per product, with the holding in plain words:

routes = pipeline.route_cells(entities)          # {product: CellRoute}
routes["tv_package"].submodel_keys               # e.g. ["tv_package::new", ...] per row
routes["tv_package"].holding                     # "held" / "previously_held" / "not_held"
pipeline.ensemble.submodels["tv_package::new"]   # the TrainedSubmodel behind those cells

holding is None when the run has no holdings column (or routes by an explicit subcase_col). Per-recommendation reasons are built on this. The platform explains each recommended cell with the local contributions (TreeSHAP) of the sub-model that scored it, at the entity's values, and adds the holding as context: "already has it" (a repeat, extra line or upgrade) or "had it before" (a win-back). ml4 publishes the routing; the SHAP itself comes from the explanation layer, like every other attribution.

See the Recommender reference for the full API.

Next: Ranking · Recommendation example