AutoGluon's Prep options (GBM:extra, GBM:arith)

AutoGluon's extreme_quality preset trains one LightGBM it calls Prep. That model bundles two things that behave differently, and ml4 offers them apart as two roster specs. Neither is part of any preset and neither is on by default: you choose them by naming them in roster.models.

In short

• GBM:extra is a LightGBM with AutoGluon's Prep hyperparameters. As a second model beside GBM:gbdt it gained +1.72% on average (13 datasets × 3 seeds); in a wide roster it adds nothing.
• GBM:arith adds up to ~2,000 generated arithmetic columns per bag fold. It is experimental: thinly positive as a second model, negative on its own.
• Neither is on by default or in any preset. Name them in roster.models, beside GBM:gbdt, so the weighted ensemble can give a model that does not help weight 0.
• GBM:arith cannot carry monotonic constraints, is refused by the airgapped export, and trains and scores several times slower.

Spec	What it is	Status
GBM:extra	Prep's hyperparameters: extra-trees mode, 7 leaves, a low learning rate, strong regularisation. The features are exactly the other boosters'.	Measured: positive as a second model beside the default, no gain in a wide roster
GBM:arith	Prep's arithmetic feature search: the model also sees up to ~2,000 generated sums, differences, products and ratios of the run's columns.	Experimental — positive, thinly, as a second model; negative on its own. See the cautions below

cfg.roster.models = ["GBM:gbdt", "GBM:extra"]    # AutoGluon's settings, as a second model
cfg.roster.models = ["GBM:gbdt", "GBM:arith"]    # the arithmetic columns, as a second model

What is deliberately not offered. AutoGluon's whole TabPrep pipeline, and its other generators. Random-subset compression was worse than ml4 as shipped on two of the three datasets it was run on; the categorical-interaction (with target encoding) and group-by generators have not been measured on ml4's fit path at all (the one benchmark that ran the whole pipeline had only numeric datasets); and the pipeline as AutoGluon wires it removes the raw categorical columns ml4's own encoders need (see the note under GBM:arith). Each can be added as a spec if a measurement says it earns one.

Both read best beside GBM:gbdt. With two or more models ml4 fits a weighted ensemble over their out-of-fold predictions, so a model that does not help a dataset gets weight 0 and the result equals the default's. On their own they have no such safety net.

GBM:extra — AutoGluon's settings

What it trains. LightGBM with extra_trees=True, num_leaves=7, learning_rate≈0.024, min_data_in_leaf=1, lambda_l1≈0.35, lambda_l2≈1.96, row and column sampling just under 1.0, and a different seed for every bag fold. The values are AutoGluon's, copied into parleml_ml4.engine.models.autogluon_prep so that an AutoGluon upgrade cannot change what a named spec trains (a test compares them with the installed AutoGluon and fails when upstream moves).

What was measured. As a second model it gained +1.72% on average (interval [+0.52, +3.09]); replacing GBM:gbdt with it did not pass. Expand for the design and the numbers.

Details: the GBM:extra measurements

What was measured. The default roster plus GBM:extra, against the default alone, under 5-fold bagging with a weighted ensemble over the models (no stack layer and no refit_full: the harness pins those so a roster experiment differs only by its members): 13 datasets × 3 seeds, 39 paired fits, log-loss or RMSE, the mean over datasets with a 95% bootstrap interval over datasets.

	Mean improvement	95% interval	Fits that gained / unchanged / lost
default + GBM:extra vs default	+1.72%	[+0.52, +3.09]	29 / 4 / 6

Seven datasets gained on every seed; the one consistent loser was pc1 (−2.1%), where the out-of-fold scores gave the new model weight and the held-out quarter disagreed. On balance-scale, where the settings alone lose 38%, the ensemble gave the model weight 0 and matched the default. Replacing GBM:gbdt with these settings, instead of adding them, did not pass (mean −2.18%, interval [−8.94, +2.09]): use it as a second model.

What it does not change. The recipe, the column names, Explain, the exports — it is the same features under a different hyperparameter set. It honours monotonic constraints (extra-trees mode only randomises where a split falls; a test pins it). It has no search space, so under hyperparameter search it trains once at these values.

Details: the control and the wide roster

A control. Any second model might help an ensemble, so the same test was run with ml4's existing second variant, GBM:dart, in the extra slot: +0.21% (interval [−0.03, +0.47]; 13 datasets × 3 seeds), which is within noise. GBM:extra beat it head to head by +1.50% (interval [+0.18, +3.03]), so the gain belongs to these settings and is not a generic second-member effect.

In a wide roster it adds nothing. Added to the six-model diverse_ensemble roster and judged inside one fit (the weighted ensemble with and without it), GBM:extra gave +0.02% (interval [−0.26, +0.30]; 8 datasets, seed 0), which fails the pre-registered bar of +1%. The gain is for the default roster of one LightGBM; where several models already vote, these settings are one more voice among many. The cost is one more LightGBM fit; the measured paired fit-time ratio was about 1.6 on a busy machine, so it is indicative only.

GBM:arith — arithmetic feature search (experimental)

What it adds. AutoGluon's arithmetic generator builds up to 2,000 candidate columns from the columns of ml4's own frame — pairs first, then triples while the budget lasts — and its Spearman filter keeps the 2,000 most rank-correlated with the target. Both run inside each bag fold, on that fold's training rows only. Every column ml4 already builds is kept; the generated ones are added. Because the generator's operands include ml4's engineered columns (tree-guided products, cluster distances, ...), one generated column can combine more than two raw columns. On a frame wide enough to offer more than 2,000 candidates the generator samples them; that draw follows execution.seed (seed 0 is AutoGluon's own draw), so runs at different seeds sample different candidates.

Do not copy AutoGluon's wiring. AutoGluon's own Prep pipeline drops the raw categorical columns from the model's input, because its categorical generators replace them. With only the arithmetic generator nothing replaces them, and ml4's own encoders — which run after this stage — lose their inputs too. Measured: that wiring cost about 2.6% on average and 19% on one dataset. GBM:arith keeps every input.

What was measured. On its own it failed the bar for a default; as a second member it passed, thinly (+1.36%, interval [+0.07, +3.34], 12 datasets). Expand for the design and the numbers.

Details: the GBM:arith measurements

What was measured. Two designs, each against the default roster, on the 19 benchmark datasets of the harness (the second on 13 of them).

The arithmetic model on its own. The arithmetic columns added to ml4's frame of one LightGBM, against that LightGBM alone (GBM:arith as the whole roster against the default; one train/validation split per dataset; three splits, seed 0 first and then two fresh seeds). The pre-registered rule for shipping it as a default is a mean gain of at least 1%, an interval above zero, and at most a quarter of the datasets losing more than 2%. It failed on seed 0 and failed again on the fresh seeds, which is why it is off by default:

Reading	Datasets	Mean improvement	95% interval	Datasets better / worse	Losing more than 2%
seed 0	18	+1.07%	[−2.40, +4.73]	9 / 9	5
fresh seeds 1 and 2	18	−2.77%	[−8.35, +1.28]	6 / 12	7
all seeds, mean per dataset	18	−1.49%	[−4.76, +1.47]	6 / 12	7

(balance-scale, whose label is a documented product of its four attributes, gained 100% on every seed and is left out of the rule: it shows what the generator can find, not what it typically finds. The means are pulled down by breast-w, which went from +19.9% to −17.1% to −65% across its three splits; the median dataset moved by about −1.7%.)

The result depends on the dataset more than on the seed. Averaged over the three splits it won on vehicle (+14.0%; +12.3%, +23.7% and +6.1% on the three), pc1 (+5.9%), wine (+3.4%) and crab (+2.1%), and lost on heart (−8.8%), credit-g (−6.2%), adult (−4.2%), diabetes (−2.7%), bank (−2.1%), bike (−2.0%) and covtype (−1.9%). A few datasets changed sign from one split to the next (breast-w, bank, insurance): on a few hundred rows the validation slice that early stopping uses is the largest source of noise, so read one split of a small dataset as noise. A cap does not fix it (seed 0: 300 columns instead of 2,000 gave +0.30%, and heart and credit-g still lost 9–11%), and a third operand adds little (seed 0: +0.89%, interval [−0.24, +2.17]). On every one of the six datasets that gained over the three splits, at least half of the arithmetic columns' gain sat on columns with an ml4-engineered operand (a tree-guided product, a cluster distance): the structure it finds is mostly combinations of ml4's own outputs, which is something ml4's tree-guided interactions do not build.

As a second member. The shipped pipeline of ["GBM:gbdt", "GBM:arith"] under 5-fold bagging, against the default alone: 13 datasets × 3 seeds, 39 paired fits, with ml4's weighted ensemble deciding how much the new model counts. The same rule, on the 12 datasets without balance-scale, passed:

Reading	Datasets	Mean improvement	95% interval	Datasets better / worse	Losing more than 2%
all three seeds, mean per dataset	12	+1.36%	[+0.07, +3.34]	6 / 4	0
seed 0 alone	12	+1.94%	[+0.22, +4.80]		0
seed 1 alone	12	+1.09%	[−0.05, +2.67]		0
seed 2 alone	12	+1.05%	[−0.00, +2.50]		0
all three seeds, without vehicle	11	+0.50%	[−0.08, +1.15]

The gain is concentrated: vehicle +10.9%, housing +2.3%, crab +2.3% and wine +1.9%, and every other dataset within ±0.35% (balance-scale +99.95%; with it the 13 datasets average +8.94%, interval [+0.24, +24.87]). The ensemble is what makes it safe: across the 39 fits the arithmetic model got weight 0 in 14 (every time on blood and heart) and sat in the strongest model in 25, with a mean weight of 0.43, and the datasets where it lost on its own (heart, credit-g, adult, bank) cost the ensemble no more than 0.31%; no single fit lost more than 1.3%. The pass is thin: seeds 1 and 2 on their own average +1.1% with intervals that touch zero, and without vehicle the mean is +0.5%. Read it as a second member that is free where it does not help and large on a few numeric-measurement datasets, not as a general gain.

How to use it. Beside GBM:gbdt, under cross-validation, so that the ensemble's out-of-fold weights decide how much it counts: that is the design measured above. Under the single-model policy (engine.training_policy="train_validation") ml4 deploys the candidate with the best validation score, so ["GBM:gbdt", "GBM:arith"] is exactly a validation-score gate between the two. That gate was devised on seed 0, where it looked good (mean +1.63%, interval [+0.05, +3.60], no dataset losing more than 2%), and it did not hold on fresh seeds: −1.32%, interval [−6.89, +2.44], the gate opening in 16 of 36 fits. The validation slice it reads is the one early stopping already used, so the score is optimistic. Do not rely on it; prefer cross-validation for this spec.

Not yet measured. The member design inside a wide roster such as diverse_ensemble; on data much larger than the 4,000 rows of the harness; with a stack layer and refit_full (the harness pins neither, and a multi-model roster turns both on by default); and its cost on larger data. With 13 datasets the average is dominated by one of them, so a new dataset is more likely to see no change than the average gain.

Cautions

• Results swing by dataset. Read the evidence above before relying on it.
• Explain meets generated columns. They exist only inside the model, so the feature recipe does not list them. AutoGluon names each by its expression — age/balance, (a+b)*c — and ml4's name decoder reads those names: it returns every dataset column the operands are made of (importance is shared evenly among them) and labels the column "Arithmetic combination (model-generated)". One-hot levels with a hyphen and ml4's own interaction names are not mistaken for expressions; a test pins both. MLI explains the best-scoring boosted-tree model on the board, which can be this one, so expect expressions among the drivers.
• The airgapped export refuses it. The export replays the feature recipe only, so it cannot rebuild the model's input; the refusal names the way out (serve the AutoGluon bundle, FittedPipeline.save).
• It cannot carry a monotonic constraint. A product or ratio of a constrained feature with another has no direction, so the fitted function would not be monotone in it. A constrained run that names GBM:arith is refused, naming the reason, before any model is built.
• It trains slower and scores slower. On the benchmark datasets (one LightGBM, one train/validation split, a busy laptop — indicative only) the arithmetic model's fit took a median of 4.9× the default's (quartiles 1.4× and 5.8×) and it trained on 1,779 to 2,045 columns; the whole two-member 5-fold fit took a median of 15× the default's (quartiles 12× and 24×; 32 pairs that ran with at least 1.5 GB free). Every prediction rebuilds those columns.
• Wide or large data. The generator materialises its candidate columns per fold in memory; keep it to datasets of a few tens of thousands of rows, and set engine.time_limit, unless you have measured the memory.
• One random draw follows the run seed. The generator draws at random only when more than 2,000 candidate columns exist, i.e. on wide frames. ml4 adds execution.seed to AutoGluon's preset random_state, so seed 0 reproduces AutoGluon's own draw and every other seed draws fresh candidates. A fit is reproducible at a fixed seed, and a stability study across seeds varies the draw too.

The model's AutoGluon name always carries the spec (LightGBM_ARITH_BAG_L1, even when it is the only LightGBM), so the leaderboard, the ensemble board and any export say so, and the run raises one warning that repeats the caution.

How these were measured

The numbers above come from ml4's real AutoSupervised fit path, run by a benchmark harness on small public datasets subsampled to 4,000 rows. They say nothing about much larger data.

Details: the harness and the reproduction check

The measurements were made with runtime patches that gave a roster entry AutoGluon's settings or its arithmetic stage. The shipped specs were then checked against those patched variants: GBM:extra as a second model on vehicle and blood under 5-fold bagging, and GBM:arith alone on blood, credit-g and vehicle under a single split. All five pairs reproduced the patched fit's log-loss to the last digit (relative difference 0). That is why the evidence is about the shipped specs.

The runs used a frozen copy of the product code, so that concurrent changes could not move a result mid-run. A parity check on blood and vehicle gave a log-loss identical to the last digit under the frozen code and under the current fit path, at two very different amounts of free memory, so all the records are pooled.

ml4/scripts/spike_fe_variants.py fits ml4's real AutoSupervised path once per (dataset, seed, variant) on the same split; spike_fe_variants_analyze.py judges each contrast by rules fixed in code before the runs: a mean gain of at least 1%, a bootstrap interval over datasets above zero, and for the arithmetic search at most a quarter of the datasets losing more than 2%. The datasets are the small public benchmarks of the LLM-FE repository (25 CSVs), subsampled to 4,000 rows. The harness pins no stack layer and no refit_full so that a roster experiment differs only by its members; a multi-model roster turns both on by default, and the numbers above do not cover them. See the harness's module docstring for the hazards (AutoGluon's LightGBM stops early under 512 MB of free RAM, so every fit records the memory it saw).

Next: Supervised training · Stateful encoders