Unsupervised: anomaly detection & clustering¶

ml4's unsupervised workloads share one preprocessing and embedding layer but expose two independent entry points:

from parleml_ml4.unsupervised import AnomalyDetector, ClusteringDetector

Both follow the sklearn fit / predict shape, both take an optional UnsupervisedConfig for the shared preprocessor, and both persist a deployable pipeline when handed an outputs_dir.

In short

AnomalyDetector ranks unusual rows; ClusteringDetector groups similar rows. Both read your frame through one shared preprocessor.
Judge an anomaly run by result.separation, not anomaly_ratio. Judge a clustering run by verdict first, then the *_features quality indices.
Only kmeans, minibatch_kmeans, gmm and birch can score new rows; hdbscan (the default), dbscan and agglomerative cannot.
UnsupervisedPipeline.applied_overrides lists every place the run departed from your config.
A pipeline from another format version is refused on load: re-fit it.

Optional extra

PyOD (the anomaly detectors) and scikit-learn (the clustering algorithms) are core ml4 dependencies, so both detectors work without an extra. pip install parleml-ml4[unsupervised] adds UMAP and the neural text-embedding stack (parleml-ml4[embeddings]). Deep PyOD detectors (autoencoder, vae, dif) additionally need pyod[torch], and suod needs pyod[suod] — a clean ImportError with the exact pip install command is raised only when such a detector is actually requested. UMAP visualisation degrades to PCA when umap-learn is absent.

The shared preprocessor¶

The shared UnsupervisedPreprocessor mixes numeric and categorical columns into a numeric matrix; its default scaler (numeric_scaler="robust") is RobustScaler (IQR-based) so the very outliers anomaly detection hunts for cannot distort the scaling.

Columns are routed by dtype:

Column family	Encoding
Numeric	inf → NaN → `RobustScaler` → impute (`numeric_imputation`, default `knn`) (+ degenerate-IQR repair); a `<col>__missing` flag only with `missing_indicators=True`
Datetime	epoch seconds, then the numeric pipeline
Categorical, ≤ `max_levels_for_onehot` (default 20) distinct	`OneHotEncoder(handle_unknown="ignore")` + a `<col>__unseen` flag
Categorical, > `max_levels_for_onehot` distinct	frequency encoding + `<col>__rare` indicator

Four rules shape the matrix, and each one is detailed below:

Numeric columns are scaled before they are imputed, so a mostly-missing column cannot take over the distance metric.
A missing number is filled from the row's nearest neighbours; an unseen or rare category and a blank text cell raise a binary flag instead of pretending to be typical.
Datetimes become epoch seconds, and serving refuses a frame whose timezone awareness or dtype differs from the fit.
Identifier-like columns (at least 95 % distinct values) are dropped at fit.

Details: why scaling happens before imputation

RobustScaler divides by the inter-quartile range. Filling missing values first puts a growing artificial point mass at the column's median, and the IQR collapses in proportion: a column that is 50 % missing has half its rows on the median afterwards, so Q1 = Q3. Just below that line the IQR is not zero but a thin slice of the observed distribution, so the divisor is tiny — measured on a log-normal amount column, impute-then-scale reaches 1985× the variance of a healthy axis at 45–49 % missing.

Unrepaired, such a column dominates every distance the pipeline computes. annual_revenue in millions contributes ~10¹² to a squared Euclidean distance where a properly scaled neighbour contributes ~1, so KMeans/GMM/ HDBSCAN and the proximity detectors (knn, lof, cof, cblof, ocsvm) all collapse into one-dimensional methods on that single column. Nothing raises, and the validity metrics — computed on the same geometry — agree the result is good. Worse, the more missing a column is, the more it dominates.

ml4 therefore scales first. All three scalers are NaN-tolerant at fit, so the centre and spread come from the observed values alone and no missingness band has a collapsed divisor. numeric_imputation keeps its exact meaning under this order: the median is invariant under any monotone map, and the mean commutes with the affine scalers. numeric_imputation="zero" is the exception and is applied before scaling, because it asserts the missing rows are zeros — that is data, and it belongs in the statistic.

Details: the degenerate-IQR repair, for real ties

A zero IQR does not mean a column is constant. It means at least half its mass sits on one value — a zero-inflated count, a mostly-defaulted amount, or any column under numeric_imputation="zero". That is a property of the data rather than an artefact of imputation, and RobustScaler would still leave such a column in raw units.

Those columns are rescaled by their observed standard deviation, logged at WARNING, and listed on the fitted preprocessor as degenerate_scale_columns_ ({column: "std" | "constant"}). Once the IQR has collapsed, the median/IQR summary has already failed to describe the column, so robustness is moot — the spread that matters is precisely the tail the IQR cannot see. A column whose observed standard deviation is also zero is genuinely constant; it is left alone, because centring maps it to all-zeros, which is inert in every distance.

If a column shows up here, treat it as a prompt: a feature that is mostly absent or mostly defaulted is rarely one you want steering an unsupervised geometry at all.

The indicator families¶

Every encoded frame carries binary flag families, and they exist for the same reason: the absence of a value is itself information, and an unsupervised model that silently substitutes a plausible value has deleted the signal it was asked to find.

Flag	Raised when	Without the flag, the row would sit at
`<col>__missing` (numeric/datetime), opt-in	a value was absent, with `missing_indicators=True`	its filled value, with no sign it was missing
`<col>__unseen`	a categorical level was not present at fit	an all-zero one-hot block, closer to the origin than any known level
`<col>__rare`	a high-cardinality value was not present at fit	the frequency axis alone, with no sign the value is new
`<col>__missing` (text)	a text cell was blank	the zero vector, one shared point outside the cloud of rows that have text

All of them are binary, and all of them are grouped back to their source column by feature_source_map(), so the findings panel and the key-driver surface still attribute them to the attribute a human recognises rather than to an engineered axis. The text flag uses the same __missing suffix as the numeric one because it states the same fact; the two can never collide, since text columns are excluded from the structured frame.

With missing_indicators=True, a column that is 40 % missing contributes a flag that is 1 on 40 % of rows. That is real information, and it carries geometric weight. If a feature is mostly absent, the honest move is usually to drop it rather than to model it.

Missing numbers: filled from the row, not flagged by default¶

A missing number is filled from the row's nearest neighbours on the columns it has (numeric_imputation="knn", from a seeded sample of at most 5,000 donor rows), so the filled value agrees with the rest of the row. The numeric __missing flag is opt-in (missing_indicators=True): turn it on when absence is the signal, such as an unfilled field on a claim form or a reading a fraudster avoided giving.

Details: the measurement behind both defaults

On the sensor-health sample (3,000 readings; pressure and current each about 3 % missing; current tracks rotation at r 0.92), a median fill plus a __missing flag put rows with a gap and nothing else unusual in 64–84 % of an anomaly ensemble's top 25. Two mechanisms, each enough alone:

the median fill builds rows no machine produces (fast rotation, median current), which every distance detector ranks first;
a rare 0/1 axis is isolated by an isolation forest in one split whatever its scale, and moves a row a full unit away from its neighbours.

With the knn fill and no flag, gap-only rows left the top 25 entirely (0 %), genuine multi-sensor extremes made up about 80 % of the top 11, and the clustering matrix's 42 %-missing fixture stopped losing its planted groups to the flag axis.

Details: where blank text is placed

All three text encodings map an empty document to the zero vector (an empty TF-IDF row, hence an empty SVD row; an all-zero NMF loading; the embedding provider never calls the backend for a blank). Once the block is centred that becomes (0 − mean) / scale — a fixed vector at the block's own radius, shared by every blank row and occupied by no informative one. Left there, clustering would return "the rows with no text" as a cluster (which the keyword surface cannot describe: its documents are empty), and anomaly detection would rank empty above strange.

ml4 places blank rows at the typical position of the axes they lack — computed from the rows that do have text — and the flag carries the absence. What counts as blank is the shared rule the categorical track uses: a true null, whitespace only, or any of the configured cat_null_tokens ("", "NULL", "n/a", "-", "?", …).

Rows with no text still coincide in the text sub-space — they carry no information to tell them apart, exactly as rows sharing an imputed median coincide on a numeric axis. That shared point sits at the centre, not the rim, and it is labelled.

Details: how loud the text flag is allowed to be

A binary flag's variance is p(1−p), at most 0.25 — and, unlike a content axis, it gets that weight for free. On the numeric track a 42 %-missing column's __missing axis was measured taking 33.1 % of the whole matrix's variance, more than any genuine signal column. The text flags are therefore capped as a group at 0.25 in the units each text sub-block is normalised to (total variance 1 each), so the whole group can claim at most 0.25 / (k + 0.25) of the text geometry for k live sub-blocks — 20 % with TF-IDF alone, 7.7 % with TF-IDF + NMF + embeddings. Several blank columns divide that budget rather than multiplying it, and a rarely-blank column stays quiet: the cap only ever scales down. A column blank in more than half its rows is warned about at fit.

Details: why the unseen flag matters more than it looks

Under plain handle_unknown="ignore", a brand-new category encodes as all zeros. The squared distance from that row to any known level is 1, while the distance between two known levels is 2. So without the flag an unseen value is closer to every training row than two ordinary values are to each other — a novel category reads as more normal than a familiar one, which is precisely backwards for anomaly detection.

Datetime columns¶

Datetimes get an explicit branch because they are ordered quantities: the useful question is when a row sits relative to the rest, which is what epoch seconds encode. Parsing is tolerant, so a column arriving as ISO strings at serving time still lands on the same axis. datetime_columns_ lists the temporal subset of numeric_columns_.

A column is temporal if it is datetime64 or an object column whose values are datetimes — the shape pandas is forced into by a mixed-offset (DST-varying) export, and by many parse_dates CSV reads. A column of date-like strings stays categorical: sniffing text would pull ordinary id columns onto the temporal axis.

Epoch seconds is an absolute axis

Because the encoding is absolute and monotone, rows timestamped beyond the fitted range score as increasingly extreme — a detector deployed against future batches will eventually flag them all on this axis alone. The preprocessor logs which columns it encoded this way at fit.

That is a true statement about the data, but it is rarely what you want in production. If you need the recurring structure (hour-of-day, day-of-week, month) rather than absolute position, derive stationary calendar features upstream with the cyclical/time encoders and drop the raw timestamp before fitting.

Serving refuses two mismatches rather than guessing:

DatetimeTimezoneMismatchError (a ValueError) when a column's tz-awareness differs from the fit — a CSV/JSON round-trip commonly drops the offset.
DatetimeKindMismatchError (also a ValueError) when a column fitted as a datetime arrives as a bare number, such as epoch milliseconds.

Details: resolution, out-of-range sentinels and the shared epoch module

Encoding scales by the column's own datetime64 resolution, so a second-resolution column out of parquet/arrow holding a far-future instant — 9999-12-31, the usual ERP "no end date" sentinel — stays the largest value on the axis instead of wrapping into the past.

An object column of datetimes gets there by a different route, because it has no resolution of its own to read: it must go through pd.to_datetime, which materialises datetime64[ns] and therefore spans only 1677-09-21 … 2262-04-11. The sentinel would coerce to NaT, impute to the median instant, and quietly stop being distinguishable from an ordinary date — so a column that loses values that way is re-read at second resolution instead. Where that is impossible — a timezone-aware column, since numpy will not hold an offset in datetime64[s] — the loss is logged rather than absorbed, naming the count and an example. Unparseable junk still becomes NaT silently; only genuine range losses are reported, so the message stays worth reading.

Both rules live in one module (parleml_ml4._epoch) and the passive-field summary reads the same one, so the matrix and the summary cannot disagree about which columns are temporal or about where an instant sits on the axis. The module sits at the package root because it is not an unsupervised concern: the supervised velocity features read the same resolution rules.

Details: why a timezone mismatch is refused

Because the axis is absolute, the same wall clock read once as US/Eastern and once as tz-naive lands 18,000 s apart on it. transform therefore records each datetime column's tz-awareness at fit and raises DatetimeTimezoneMismatchError when the serving frame disagrees. Two different offsets are fine: both resolve to the same instants, and only the presence or absence of an offset is enforced.

It refuses rather than repairs on purpose. Repairing means guessing which zone a naive frame is in, and a wrong guess is the same silent shift with a clean conscience. Restore the offset upstream, or re-fit on data in the form it will be served.

Details: why a datetime column may not arrive as a bare number

A JSON round-trip commonly emits epoch milliseconds, and a bare number carries no unit — pd.to_datetime reads it as nanoseconds. Probed on an hourly ts fitted onto a −1.0 … 1.0 axis, the epoch-millis batch came back at −16,046.81 for every row: the whole batch displaced by one constant vector into a region no training row occupies — the same failure the identifier-drop rule exists to remove, re-entered through the numeric track. The timezone guard cannot see it, because an integer column reads "tz-naive" on both sides.

transform therefore raises DatetimeKindMismatchError when a column fitted as a datetime arrives numeric. The unit is not recoverable from the fitted column, and inferring it from the serving batch would make a row's position on the axis depend on the rows batched beside it — the same reason the clean step freezes its epoch unit at fit rather than re-voting. Neither surface guesses a unit from a serve batch. Decode the epoch at the unit you exported it in, and hand the preprocessor a datetime dtype.

Identifier-like columns¶

A high-cardinality column in which at least 95 % of the rows hold a distinct value (identifier_distinct_ratio, default 0.95) — a row id, a raw timestamp kept as text — is dropped at fit and listed in dropped_constant_columns_, with a warning. Columns above 50 % distinct but below the drop line are kept and warned about. If such a column is an id, drop it; if it is a timestamp, parse it to a datetime dtype so it is encoded as a quantity.

Details: why identifiers are dropped, and why the test is not variance

Keeping them is worse than useless. At (near-)identifier density essentially every serving value is unseen, so __rare → 1.0 and __freq → a constant: every served row is displaced by the same fixed vector in a direction no training row occupied, and neighbour-based detectors (knn, lof, ocsvm) flag the entire batch. Tree-based iforest is unaffected, which makes the failure easy to miss.

The test is deliberately not "zero variance on the frequency axis". A uniform-count column (30 categories seen twice each) is also zero-variance, yet it is a legitimate categorical whose __rare axis still flags genuinely unseen values — dropping on variance alone would discard that signal. Conversely a 97 %-distinct column has a perfectly non-degenerate frequency axis and is still dropped, because the serving-time displacement is what matters.

Anomaly detection¶

AnomalyDetector wraps PyOD. It has two modes, selected on AnomalyConfig.mode.

from parleml_ml4.unsupervised import AnomalyDetector
from parleml_ml4.config.anomaly import AnomalyConfig

det = AnomalyDetector(AnomalyConfig(mode="auto")).fit(df)
result = det.result_

print(result.n_anomalies, "of", result.n_samples, "rows flagged")
labels = result.labels             # (n_samples,) in {0, 1}
scores = result.scores             # decision scores, higher = more anomalous
top    = result.top_k_indices      # most-anomalous rows first

auto (ADEngine)single

mode="auto" (the default) hands the data to PyOD's ADEngine. It profiles the data, picks detectors from its benchmark-backed knowledge base and runs them; ml4 serves the z-score mean of every successful candidate, beside ADEngine's rank consensus and quality verdict. The configured contamination (default 0.05) is applied: ml4 writes it into each detector plan before the engine runs, which ADEngine respects — it fills its own knowledge-base default (0.1) only when the caller supplies nothing.

auto reaches 8 of the 23 detectors, and runs at most 3

On tabular data ADEngine routes from a table of 8 candidate names — ecod, hbos, iforest, copod, knn, lof, cblof, inne — not from the full catalogue, and ADEngine.plan caps the plan at 3, which ADEngine.run then executes sequentially (it is a for loop, not a parallel pool). The other 15 entries in the single-mode roster — including pca, kpca, mcd, ocsvm, gmm, kde, sos, abod, mad, cof, loda, dif, suod and the two deep models — are unreachable from auto. Use mode="single" to reach them.

cfg = AnomalyConfig(mode="auto", priority="balanced")
det = AnomalyDetector(cfg).fit(df)
print(det.explain())          # what was served + the quality report

Read result.caveats after every auto run: it states what the choice was based on and which anomalies the served detectors cannot see. priority changes the pick only on frames with at least 100 encoded columns, and only for "speed" or "accuracy".

Details: realised contamination and the served score

result.contamination reports the realised flagged rate rather than echoing the request back. In auto no single detector is picked: the labels and served scores come from ZScoreMean — the mean of every successful candidate's score, each standardised by its own training scores (detector_name="zscore_mean", a PyOD detector itself) — and ties can move the achieved rate off the requested one. Read it as "what happened", not "what I asked for". result.scoring_rule states the rule in words.

Details: what auto chose on, and the anomalies it cannot see

result.caveats opens with the basis of the choice: the candidates ADEngine shortlisted, the data shape and priority its routing rules keyed on, and the rule's own reason — no candidate is compared on a quality measure, because an unlabelled run has none. When numeric columns move together (|r| >= 0.8) a second caveat names the pairs and what the served detectors cannot see: a per-column detector (ecod, copod, hbos) cannot see a reading that is ordinary column by column but wrong together, and the distance / density / isolation detectors (knn, lof, iforest, …) weigh every column's deviation alike, so a row far along the shared trend outranks one that breaks it. None of auto's eight candidates measures distance against the correlation structure; for broken relationships use mode="single" with mcd (robust Mahalanobis distance) or pca on the numeric columns. The same caveat applies to a single run of a correlation-blind detector (parleml_ml4.unsupervised.anomaly.choice).

Details: engine_separation_ratio is not separation

quality averages three terms. Upstream publishes the first as separation, which is also the name of AnomalyResult.separation — the run's headline metric and a completely different quantity. ml4 publishes the engine's term as engine_separation_ratio, and quality carries no bare separation key; only the headline property uses the word. The engine term is clip(anomaly_mean / inlier_mean − 1, 0, 1) over the rank-normalised consensus, so it saturates at 1.0 on essentially any run with signal; the headline is an unbounded IQR-normalised gap between medians.

Details: consensus_scores when only one detector survived

AnomalyResult.consensus_scores always lives in (0, 1]. PyOD's compute_consensus short-circuits when exactly one of the planned detectors succeeded and hands back that detector's raw decision_scores_, on an unbounded scale, so ml4 applies upstream's own rankdata(s) / n to it before storing. quality["consensus_basis"] names which path produced the vector (mean_of_rank_normalised_scores or single_detector_rank_normalised), and on the single-detector path every per_detector_results[i]["consensus_spearman"] is None rather than the 1.0 a detector correlated against its own scores would trivially produce. caveats says so in prose.

Details: priority is an override, not a dial

Exactly two rules in PyOD's whole knowledge base carry a priority condition (tabular_high_dim_fast / tabular_high_dim_accurate), both additionally require n_features >= 100, and they match 'speed' and 'accuracy' only. No rule anywhere matches 'balanced' — the default — so at the default value, and at any width below 100 encoded columns whatever the value, the detector pick is decided entirely by the width/size rules and priority changes nothing.

Details: progress events

Each detector the engine evaluates streams an auto_router_trial event (phase="engine", carrying algorithm, status and the anomaly_ratio score), followed by one phase="served" summary for the served z-score mean (algorithm="zscore_mean", scored by its flagged share).

mode="single" instantiates one named detector (detector_name, default "iforest") from a curated roster of 23 estimators: ecod, copod, hbos, abod, mad, sos, kde, gmm, pca, kpca, mcd, ocsvm, lof, knn, cof, cblof, iforest, inne, loda, dif, suod, autoencoder, vae.

cfg = AnomalyConfig(mode="single", detector_name="iforest",
                    contamination=0.05, detector_kwargs={"n_estimators": 200})
det = AnomalyDetector(cfg).fit(df)

Some detectors refuse, or mislead, on ordinary data. These are properties of the estimators, not of the wrapper:

mad is univariate and pca needs a full-rank matrix; ml4 refuses both up front on frames they cannot score.
lof and cof are misled by duplicate rows; the fit warns when it sees a block of them.
autoencoder / vae need at least 16 rows.
vae (like suod) cannot reproduce its own scores.
cof cannot score a batch of n_neighbors rows or fewer.
deepsvdd is not in the roster: PyOD cannot train it. Use autoencoder.

Details: mad, pca, lof and cof on ordinary data

mad is univariate. PyOD's MAD raises on two or more columns. It is only usable on a genuinely single-numeric-column frame.
pca and mcd are fitted on a full-rank subset. A one-hot block sums to 1 on every row, so the encoded matrix is exactly rank-deficient after centring, and each __unseen axis is constant at fit. PyOD's PCA would divide by a zero explained_variance_ratio_ and score every row inf (zero anomalies, a "top 50" of rows 0–49); MCD would measure its distances in a truncated subspace. So these two detectors see a maximal set of independent columns — chosen by pivoted QR on the centred matrix — which drops one level per one-hot block (the complement of the others) and the constant axes; the run log names what was left out. A category never seen at fit then scores like the dropped reference level.
lof and cof are misled by duplicate rows. Duplicates make a block mutually nearest-neighbour, and both detectors rank that block — and its neighbourhood — above genuine outliers. Measured on 20 planted outliers beside a 40-row duplicate block, recall was 0.00 (lof) and 0.45 (cof), versus ≥0.8 for iforest / knn / cblof. One-hot encoding makes exact duplicates common, so prefer the latter three on data with repeated rows.

The fit discloses it: a run whose encoded matrix holds a block of identical rows at least as large as the neighbourhood logs a warning naming the block size and the remedies. scikit-learn detects the same condition itself, but its warning is gated on not novelty and PyOD constructs LocalOutlierFactor(novelty=True), so the upstream guard never fires on any fit ml4 performs. Raising n_neighbors above the block is the cheapest fix where you want to keep the detector — on the frame above it takes lof from 0/10 to 10/10 recovered outliers.

mode='auto' is covered too, which is the case that matters most: the engine can select a neighbourhood detector (knn on the frame above) without you having asked for one. There the remedy differs — n_neighbors belongs to ADEngine's plan, so the message points at exclude_detectors or mode='single' instead of a knob you cannot reach from there.

Details: autoencoder / vae need enough rows to fill a training batch

PyOD builds its training loader with drop_last=True, so a frame with fewer rows than batch_size yields zero batches and the fit dies with a bare UnboundLocalError from inside torch — and just above that cliff it silently trains on one batch per epoch, discarding the rest of the frame every epoch.

ml4 sizes batch_size from the row count instead (aiming for at least 8 batches per epoch, capped at PyOD's 32), which removes both failures. It still refuses any frame under 16 rows — 8 batches of 2, the smallest epoch ml4 will call a pass over the data — but that refusal is a policy, not a crash prediction: a smaller batch would let the fit run, and a neural reconstruction model estimated from a dozen rows is not a model. An explicit detector_kwargs={"batch_size": ...} still wins, and is refused if it is at least as large as the frame (there the UnboundLocalError really is what would happen) or if it is not a positive int — 0, 2.5 and True are refused, because True is an int in Python and would train on batches of 1.

Details: vae scores stochastically

PyOD's VAE routes evaluation through the same reparameterised forward pass as training, so decision_function redraws the latent sample on every call. Re-scoring the identical training frame differs from result.scores by tens of percent. The run therefore cannot be reproduced from its own record — the same class suod is in — and the caveat is carried on result.caveats. Use autoencoder when you need a deterministic reconstruction-error model.

Details: gmm chooses its component count and its covariance family

A covariance_type='full' component estimates d(d+1)/2 + d + 1 parameters, so on a wide encoded matrix a full mixture is unaffordable at any component count and sklearn's reg_covar, not the data, would decide the density. ml4 budgets 2 rows per parameter: it picks the component count from that budget (capped at 8) and drops to a diagonal covariance — logged, because a diagonal mixture cannot represent correlation within a component — when the frame cannot afford even one full component.

An explicit detector_kwargs={"covariance_type": ...} is honoured and budgeted against — the family you name is the one whose parameter count sizes the component number. An explicit n_components is honoured too, but it does not switch the covariance fallback off: at d = 60 on 2,000 rows, {"n_components": 3} gives you 3 components of a diagonal mixture, with the swap logged. Name both keys if you want both fixed. 'tied' is budgeted per-component only — its shared d(d+1)/2 block is charged to nobody, which is why it yields more components than 'full', not fewer; ml4 warns when that shared block is itself unidentified (n < 2·d(d+1)/2).

Details: cof cannot serve a batch smaller than its neighbourhood

COF.decision_function rebuilds a neighbourhood graph over the scored batch itself, and PyOD raises a bare IndexError when the batch has no more rows than n_neighbors (default 20). ml4 refuses such a batch first with a ValueError that names the minimum. A serving layer that batches by request size will hit this on small requests: score at least n_neighbors + 1 rows at a time, lower detector_kwargs={"n_neighbors": ...}, or pick a detector whose serving path uses the fitted training graph (ecod, iforest, knn, lof) for interactive scoring.

Details: why deepsvdd is not in the roster

PyOD 3.5.2 cannot train it: loss.backward() is commented out in pyod/models/deep_svdd.py, the fitted hypersphere centre is written to the inner module and read from the outer one (so it stays at the origin), and the best epoch's weights are never loaded back. It would score rows by the squared norm of a random-weight projection — deterministically, so every degeneracy guard would pass it. There is no alias: a config naming deepsvdd raises at validation. Use autoencoder for a reconstruction-error one-class model that actually trains.

Scoring new data and reason codes¶

new_labels = det.predict(X_new)                # {0, 1}
new_scores = det.decision_function(X_new)      # raw scores
norm       = det.normalized_score(X_new)       # rank surrogate in [0, 1] — NOT a probability
reasons    = det.explain_rows(X_new)           # per-row top contributing features
sweep      = det.contamination_diagnostics()   # threshold-sensitivity curve

normalized_score is not a probability

It is PyOD's min-max rescale of the decision score by the training range, clipped to [0, 1]. A row more extreme than anything seen at fit saturates at 1.0, and the value is uncalibrated — 0.9 does not mean "90 % likely anomalous". It is useful for ranking and for a colour scale, and for nothing that treats it as a probability.

AnomalyResult also carries score_summary (configurable percentiles), per_detector_results and quality (auto mode), findings (per-row explanations), and keywords_by_column (c-TF-IDF keywords contrasting the anomalous vs normal rows of each configured text column).

How good was the run? — `separation`, not `anomaly_ratio`¶

anomaly_ratio is the fraction of rows flagged. In single mode that equals the contamination you set, by construction — so ranking runs by it ranks them by a knob, not by a result.

result.separation is the honest answer:

             median(score | flagged) - median(score | unflagged)
separation = --------------------------------------------------
                       IQR(score | unflagged)

It is dimensionless and invariant to shifting or rescaling the score, so it is comparable across detectors whose raw scores are on entirely different scales — which decision_function values are, and which is why they must never be compared directly. It sits near 0 when the flagged rows look like everything else, and goes large when they genuinely stand apart, at any contamination.

It is None — never a fabricated number — when the question is undefined: no rows flagged, every row flagged, a zero unflagged IQR, or score_degeneracy already set.

There is no supervised metric here

ml4 ships no validate(y_true) helper. Any metric computed against labels on the rows the detector was fitted on is in-sample and estimates nothing. If you have ground truth, score it yourself against det.decision_function(X) with sklearn.metrics, so the choice of data is explicit and yours.

Report result.labels for the fitted run; use predict for new data. The two answer different questions and can disagree on the training rows.

Details: why result.labels and predict() can disagree

result.labels is the in-sample labelling — "which of the rows I was fitted on are anomalous?". predict(X) is the novelty labelling — "are these rows anomalous relative to what I was fitted on?" — and it is what the deployable pipeline serves.

For the neighbour-based detectors (lof, knn) the two legitimately disagree when you re-score the training rows: PyOD reads result.labels from decision_scores_ but predict from decision_function, and for those detectors a training row handed back to decision_function becomes its own nearest neighbour, looks locally dense, and reads as an inlier. Every other curated detector re-derives the same scores on both paths (to floating-point noise), so its labels match and the question never arises. This is PyOD/scikit-learn semantics, not a bug, and set_operating_threshold does not bridge it. See AnomalyResult's docstring for the full argument.

Clustering¶

ClusteringDetector wraps seven scikit-learn algorithms behind one interface. The default is hdbscan (sklearn.cluster.HDBSCAN).

Algorithm	Inductive `predict`?	k selection
`kmeans`, `minibatch_kmeans`	yes	`n_clusters` int, or `"auto"` sweep
`gmm`	yes	`gmm_n_components` int, or `"auto"` by BIC/AIC
`birch`	yes	`n_clusters` int, or `"auto"` sweep
`agglomerative`	no (transductive)	`n_clusters` int, or `"auto"` sweep
`dbscan`, `hdbscan`	no (transductive)	density params — `n_clusters` is ignored

from parleml_ml4.unsupervised import ClusteringDetector
from parleml_ml4.config.clustering import ClusteringConfig
from parleml_ml4.config.unsupervised import UnsupervisedConfig

det = ClusteringDetector(ClusteringConfig(algorithm="kmeans", n_clusters="auto")).fit(df)
res = det.result_

print(res.n_clusters, "clusters,", res.n_noise, "noise points")
print(res.silhouette_features, res.davies_bouldin_features,
      res.calinski_harabasz_features)
print(res.geometry_space)          # where the algorithm actually clustered
labels = res.labels                # -1 marks noise (density algorithms)

Which space a number lives in¶

Two spaces exist, and the field names say which is which.

Feature space — the un-reduced fused matrix, with real column names. The reported validity indices (silhouette_features, davies_bouldin_features, calinski_harabasz_features) and the per-cluster profile heatmap are always computed here. This is the space your data is actually in, so these are the numbers to quote and to compare across runs.
Clustering geometry — what the algorithm was handed, which may be a UMAP or PCA reduction (see below). geometry_space names it: "features", "umap_15", "pca_9". The auto-k sweep scores here, because that is the space in which it was choosing.

Read the reported index with three caveats

It excludes noise rows (-1), so a density algorithm is scored only on the points it actually clustered. Check n_noise alongside it.
Above silhouette_max_samples (default 10,000) the silhouette is a subsample estimate. To tell a subsample estimate from a full one, compare silhouette_sample_size against n_samples - n_noise.
On an auto-k run it is the best of N candidates, so it carries selection optimism. Compare auto_k_selected_score with auto_k_median_score before comparing a swept run against a single fit.

Details: silhouette_sample_size and the search premium

silhouette_sample_size records how many rows the metric actually scored on every run, capped or not: the scored rows after the noise mask of caveat 1, which is applied after any draw. It is None only when nothing was scored (fewer than two real clusters), which is exactly when the silhouette itself is None. Equal to n_samples - n_noise means every eligible row was scored; smaller means a draw. Neither the draw size nor the frame size would be an honest count: on a 70 %-noise run capped at 10,000 rows, about 3,000 are actually scored.

auto_k_selected_score and auto_k_median_score make the search premium legible — on a well-separated fixture the selected silhouette was 0.891 against a sweep median of 0.592.

Did the run find anything?¶

A quality score answers "how separated are the clusters it found". It does not answer "did it find any", and those are different questions — a run that collapsed to one cluster and a run whose metric is merely undefined look alike on the score. verdict answers the second one directly:

`verdict`	Meaning
`clustered`	Real structure: two or more clusters carrying enough rows to score
`single_cluster`	Everything landed in one group
`all_noise`	A density algorithm rejected every point
`degenerate`	Nothing scorable — e.g. no usable features

verdict_reason is a full human sentence and is safe to render verbatim.

metric_coverage closes the other half. The validity indices exclude noise rows, so on a density run they can describe a small minority of the data. A silhouette of 0.62 at metric_coverage == 0.14 means the 14 % of rows that were clustered are well separated — which is a much weaker claim than the number alone suggests, and often means hdbscan_min_cluster_size is too high.

What the run actually did¶

UnsupervisedPipeline.applied_overrides records every place the engine departed from the requested configuration — a skipped or clamped pre-reduction, a PCA-for-UMAP substitution, a derived dbscan_eps or birch_threshold, a clamped hdbscan_min_cluster_size, the auto-k pick and any candidates it had to drop, a subsampled silhouette, the resolved contamination, degenerate scales or scores.

It is derived from the fitted objects, not reported by the code paths that make those decisions, so a downgrade cannot forget to register itself. It lands in result.json, in report.txt under "What this run actually did", and in the MLflow params — because a run that quietly did something other than what was asked is the failure mode this whole subsystem is most prone to.

Details: library_versions, fit_frame and how entries are worded

Alongside it the artefact records library_versions (the third-party versions its pickled estimators were written by, checked at load) and fit_frame (row count, columns and a content fingerprint of the frame it was fitted on, so a read surface can refuse to join fit-time row positions into a dataset that has since changed).

Each entry is derived from what governed this algorithm, not from what was requested. The cluster count is the case that matters: DBSCAN and HDBSCAN read no n_clusters at all, so their entry says the count was discovered from density (and that an explicitly-set n_clusters was ignored); GMM's entry names gmm_n_components, the knob it actually swept; and only a run whose auto_k_score_name is set — i.e. one where candidates were really scored — is described as having had its k chosen by a sweep.

The run record: MLflow params, `result.json`, `report.txt`¶

All three carry the fields that say what the headline numbers are worth, not just the headline numbers.

Details: the honesty fields in the run record

Field	Where	Why it is there
`separation`	metric, `result.json`, report	The anomaly run's quality score: how well the ranking separates, independent of where the cut was placed
`anomaly_ratio`	param, not a metric	In `mode="single"` it is the `contamination` request read back — PyOD cuts at the `1 - contamination` quantile — so sorting runs by it compares knobs, not results. The count and the row total remain metrics
`score.<detector>.p50…p99`	metrics	Raw detector units (path length, distance, log-density, tail probability) with no normalisation between detectors — or, for an `auto` run's `score.consensus.*`, the mean of the candidates' standardised scores — so the name carries the detector and blocks a meaningless cross-run read
`score_degeneracy`, `caveats`	params, `result.json`, report	A degenerate score vector means the labelling was decided by rounding; the top-K and the findings must not then be presented as measurements
`verdict` / `verdict_reason`	params, `result.json`, report	An all-noise clustering otherwise reads like any other run
`metric_coverage`	metric, `result.json`, report	The silhouette et al. are computed on non-noise rows only; the report prints them as `0.62 (on 14% of rows)`
`birch_threshold_used`	metric, `result.json`, report	Birch's derived absolute-scale twin of `dbscan_eps_used`

Auto-k¶

n_clusters="auto" — the default — sweeps [auto_k_min, auto_k_max] (default 2..15) and picks the best k by auto_k_score — silhouette (default), davies_bouldin, or calinski_harabasz. It applies to every algorithm that takes a cluster count: KMeans, MiniBatchKMeans, Agglomerative and Birch. GMM has its own gmm_n_components="auto" sweep scored by BIC or AIC, a likelihood criterion being the natural fit for a generative model. The full sweep lands on result.auto_k_sweep, and one auto_k_winner event names the pick.

The sweep costs one fit per k. For agglomerative on large frames that is genuinely expensive (Ward linkage is O(n²) memory per fit), so narrow auto_k_max there rather than leaving a 14-point sweep to run.

A winner on the edge of the range is not a maximum. When the sweep picks auto_k_max, or an auto_k_min above the algorithm's own floor (two clusters for a silhouette, one component for a mixture), the score may still improve past the range nobody swept. result.caveats then says so — "The sweep picked k=12, the top of its range (3–12): the optimum may lie beyond it — widen the range (raise auto_k_max) or choose the count on business grounds." — and the caveat travels to the run record (caveat.<i> params, result.json, report.txt), the backend summary, the AutoDoc card and the UI. A top bound the row count clamped below auto_k_max is not flagged: there is nothing further to sweep.

Text fusion and keywords¶

Name free-text columns in text_columns on UnsupervisedConfig to embed them, reduce them, and fuse them with the structured feature block (use_text_embeddings is on by default; False turns the embedding pass off). A FusionScaler balances the reduced text block against the structured block so neither dominates the geometry.

fusion_balance balances blocks, not columns

"equal_variance" (the default) equalises each block's total variance (sqrt(Σ var)), so widening the text block does not buy it more of the geometry. Matching per-feature variance instead would cancel the column count entirely: a 50-axis TF-IDF block against 3 structured columns would take roughly 89 % of the distance mass. text_block_weight (default 1.0) multiplies the whole text block after balancing: above 1.0 the text drives the geometry, below it the structured columns do.

How the three text encodings are scaled against each other¶

text_block_standardize (default True) governs the step inside the text side, before FusionScaler sees it. Each of the three sub-blocks — TF-IDF→SVD, NMF, reduced embeddings — weighs the same, 1:1:1, however many axes each was given, while the leading axes of each block stay heavier than its trailing ones.

Details: how the sub-blocks are standardised

Each sub-block is centred per column and then divided by one scalar: its own sqrt(Σ per-feature variance), the same aggregate fusion_balance="equal_variance" uses one level up. Two properties follow, and both are the point:

Between sub-blocks — a fixed 1:1:1. Each sub-block leaves the step with total variance exactly 1, so the lexical, topical and semantic views weigh equally in the text geometry regardless of how many axes each was given. Widen tfidf_svd_components and you get a finer TF-IDF representation, not a bigger share of the distance.
Within a sub-block — the spectrum survives. One divisor rescales a block without touching the relative variances inside it, so a variance-ordered TruncatedSVD / PCA block keeps its leading axes heavier than its trailing ones.

A per-column StandardScaler would get both wrong. It would set every text axis to variance 1, which whitens the factorisation — the 50th (near-noise) SVD component would weigh exactly as much as the 1st in every Euclidean distance, the textbook noise-amplification failure of whitening. And it would make each sub-block's share of the text geometry equal to its axis count, which comes from three unrelated defaults at two different scopes (tfidf_svd_components=50 and nmf_components=10 are per column, text_embedding_umap_n_components=10 is for the whole embedding block). Measured on the shipped six-text-column support-ticket sample under stock defaults, that split would be TF-IDF 76.9 % / NMF 15.4 % / embeddings 7.7 % — the most expensive and the only semantic pass would get the smallest share. With one scalar per block it is 33.3 / 33.3 / 33.3.

The blank-text flags (<col>__missing, above) sit outside this step and are appended after the three sub-blocks, for the same reason the numeric __missing block sits outside RobustScaler: a flag pushed through a variance normaliser comes back louder than the content it annotates — inside a sub-block normaliser it would be handed the weight of all the text. They are capped as a group instead, and the whole text side (content and flags alike) is then balanced against the structured block by FusionScaler as one.

text_embedding_reducer decides whether a text run can be served

"pca" (default) is a fixed linear projection: a served row's coordinates depend only on that row. "umap" separates topics better — the BERTopic choice — but UMAP.transform derives its epoch count from the batch row count, thresholds graph edges against a batch-global maximum, and shares one rng stream across the batch, so a row's position depends on which rows were scored beside it. Measured on a 400-row frame: whole-frame versus 5 chunks disagreed on 4.3 % of rows.

A pipeline carrying a UMAP anywhere in its transform chain is therefore refused for serving (serves_rows_independently is False, and predict raises). That is the honest outcome, but it means "umap" is for transductive clustering (hdbscan / dbscan, which are not servable anyway), not for anomaly detection or a model you intend to register. "none" fuses the raw embedding block unreduced.

uns = UnsupervisedConfig(use_text_embeddings=True, text_columns=["description"])
det = ClusteringDetector(ClusteringConfig(algorithm="hdbscan"), unsupervised_config=uns).fit(df)

print(det.result_.cluster_keywords)          # {cluster_id: [distinctive terms]}

c-TF-IDF keyword extraction (keywords.py) always runs when text columns are embedded and populates cluster_keywords — the most reliable "what is this cluster" signal. A per-column, per-cluster variant lands on cluster_keywords_by_column.

Words the run ignores. text_stop_words lists words that are in the text but must not shape the groups — product names substituted into templated subjects, a signature, a site name. They are dropped from the TF-IDF encoding of tfidf_columns and from every keyword surface (fused, per column, embedding themes; on top of the English stop words those already skip).

uns = UnsupervisedConfig(tfidf_columns=["subject"], text_stop_words=["acme", "zenith"])

Details: what a stop-word entry may be

Each entry is one word as the tokenizer cuts text — two or more letters or digits, matched case-insensitively — so "Acme Router" is two entries, and a phrase, a hyphenated name or a single letter is refused at construction (it would match nothing). Neural embeddings (text_columns) still read the full text: strip such words with a recipe when they must not shape an embedding column.

Embedding needs a model this server can load

use_text_embeddings with a local source (text_embedding.source="local") needs sentence-transformers (the parleml-ml4[embeddings] extra) and the model: cached in the Hugging Face cache, or downloadable on first use while the server is online. parleml_ml4.embeddings.source_available checks both — an offline server (HF_HUB_OFFLINE) without the weights is reported unavailable instead of failing the fit at its embedding pass. ParleML refuses such a clustering launch (and its plan) with the remedies; describe_domains() reports it on the embedding domain (sentence_embedding_model).

The structured counterpart is cluster_profiles, and the fields a cluster may actually claim come from one function:

from parleml_ml4.unsupervised.clustering.reporting import distinctive_fields

for cid in sorted(c for c in det.result_.cluster_sizes if c >= 0):
    print(cid, distinctive_fields(det.result_.cluster_profiles, cid))

z_vs_global is an effect size and nothing else: it carries no sample-size term, so ranking on it directly gives a three-row cluster a confident signature built from noise. distinctive_fields applies the support floor (passive_stats.MIN_CLUSTER_SUPPORT, 10 rows, shared with the passive-field surface) and a Šidák correction for how many axes a source column contributed, then reports one axis per source. An empty list means this cluster is too small, or is the algorithm's leftovers — say so rather than falling back to a raw ranking.

The list is descriptive and carries no significance claim: there is no p-value or q-value on the profile, because any such test's null ("this cluster is a random subset of the rows") is false by construction — the labels came from an algorithm maximising this very separation on this very matrix. Rank with separation_score, never read it as a probability. See Unsupervised results for the column-by-column contract and for what z_vs_global is a z of on each kind of axis.

Two different UMAPs — only one of them is cosmetic

The Embedding2D projection produces the 2-D map for visualisation only; it never feeds any detector.

pre_reduce_umap_components is a different UMAP and it does feed the clustering geometry. It defaults to "auto", which is algorithm-aware: 15 dims for the density algorithms (hdbscan, dbscan), which genuinely degrade in high dimensions, and off for every other algorithm (including the centroid/parametric family kmeans, minibatch_kmeans, gmm, birch), which does not. Setting an explicit integer applies it to every algorithm — usually not what you want.

geometry_space on the result records what actually happened, including the substitution to PCA when umap-learn is absent or the frame is too small for UMAP's spectral init. PCA and UMAP produce materially different clusterings, so check it.

A fit writes no cluster names: the keywords, separating fields, passive fields and representative texts on the result are the evidence a host application names clusters from afterwards.

Passive field summary¶

passive_columns names columns kept out of the clustering geometry but summarised per discovered cluster — cluster on text plus a few numerics, then describe each cluster's priority / status / age mix without letting those fields steer the geometry. The result lands on result.passive_field_stats as {cluster_id: {"size": …, "fields": {column: stats}}}.

Each field is summarised by kind, and the kinds are decided by exactly the rules the preprocessor uses:

Field	Reported as
Numeric	`count` / `missing` / `mean` / `median` / `std` / `min` / `max`
Temporal	the same numeric shape, on the epoch-second axis, tagged `value_kind="datetime"` so the caller re-formats it as dates
Everything else	`distinct`, `dominant`, and a top-`passive_top_n` (default 5) value distribution

Every field also carries a score — a standardised cluster-vs-global deviation (a z-score for numerics, the largest standardised share gap for categoricals) — plus the matching global reference, so a caller can surface only the fields where a cluster actually stands out. Near-unique categoricals score 0.0: per-value shares are meaningless there.

Details: failure handling and the shared epoch rules

The whole report is best-effort and fails per column: a column that cannot be summarised is logged and left out, and the remaining fields are reported as usual. A column that produces its own statistics but no global reference is kept and scores 0.0, since there is nothing to compare it against.

The summary and the matrix describe the same columns. Both read the frame through parleml_ml4._epoch, so an object-dtype datetime column is a quantity on both surfaces, and a datetime64[s] / [ms] / [us] column reports the same instants as the [ns] one. Timezone policy is shared too — offsets are applied, never stripped, so the summary reports absolute instants like the axis does.

Scoring new data¶

Only the inductive algorithms (kmeans, minibatch_kmeans, gmm, birch) support predict(X_new); the transductive ones raise a clean NotImplementedError. predict_proba is GMM-only.

if det.is_inductive:
    assignments = det.predict(X_new)
    reasons     = det.explain_rows(X_new)   # cluster + distance + nearest other

Persisting a run¶

Pass outputs_dir= to either detector to write events.ndjson / manifest.json / ml4.log, honour a CANCEL sentinel, and persist a deployable UnsupervisedPipeline under <outputs_dir>/deploy/. det.to_pipeline() builds the same artefact in-memory. ml4.log is skipped when the host application already configured root-level logging.

A pipeline from another format version is rejected on load

UnsupervisedPipeline artefacts are version-stamped, and UnsupervisedPipeline.load() raises StaleArtifactError (a ValueError) for anything whose stamp is not the current one. There is no migration and no compatibility shim: re-fit the detector and re-persist it. Treat the re-fit as a new model and re-check the operating threshold rather than carrying the old one across.

Details: the load gates and the version history

The gate applies to the serving path too — UnsupervisedPyfunc.load_context goes through the same checked loader, so a registered model cannot slip past it.

Two further gates sit alongside it. The stamp is read before the pickled object graph is reconstructed, so a stale artefact fails with the version message rather than an AttributeError naming an internal class. And library_versions records the third-party versions the pickled estimators were written by: a differing major is refused, anything else warns.

The current stamp and what each bump changed live in one place — the version history above _FORMAT_VERSION in parleml_ml4/unsupervised/pipeline_gate.py. It is deliberately not repeated here, because a version number copied into prose goes stale the next time the format moves.

A format bump is never cosmetic. It marks a change after which an artefact from the other version would score differently. Loading such an artefact would not fail; it would quietly answer a different question — which is why the gate refuses it.

See Embeddings for the text-encoding options, and the AnomalyDetector and ClusteringDetector references for the full API.

Next: Example: anomaly detection · Example: clustering