Text & embeddings¶

ml4 turns free-text columns into model features two ways: cheap, interpretable TF-IDF → SVD topics (the default text path), and neural sentence embeddings that capture semantics TF-IDF misses. The same embedding stack also powers text-aware clustering fusion, from the same source config.

In short

List a column in data.embedding_domain to embed it, or set stateful_fe.add_text_embeddings = True to embed every text_domain column.
The vectors come from one TextEmbeddingConfig: a local sentence-transformers model (default BAAI/bge-small-en-v1.5) or a provider connection through LiteLLM.
A local model's revision is pinned at fit; a provider model is fingerprinted. Scoring stops if either does not match, rather than predict on the wrong vectors.
A provider source sends the text to the provider at fit and at every scoring call. Use a local model if the text must stay on-prem.
Embeddings are reduced to text_embedding_components (default 50) columns per text column by a frozen linear reducer.

Text columns are routed by role on DataConfig — nothing is auto-detected:

Role list	Encoder	Output
`text_domain`	TF-IDF → SVD (+ optional neural embeddings)	`<col>_SVD_*` topics
`embedding_domain`	Neural sentence embeddings (always)	`<col>_EMB_0 … _EMB_{n-1}`
`dirty_domain`	Gamma-Poisson (`skrub` GapEncoder)	high-cardinality dirty categoricals
`mt_domain` / `wt_domain`	Multi-token / weighted-token (NMF)	packed token bags

See the FE suite for the full text-encoder catalogue; this page is about the neural path.

Embed a single column¶

Route a text column to the neural path with data.embedding_domain — it is embedded even when the global switch is off:

from parleml_ml4 import Ml4Config, Tasks

cfg = Ml4Config()
cfg.ml_task = Tasks.BINARY_CLASSIFICATION
cfg.data.target_column = "escalated"
cfg.data.embedding_domain = ["description"]   # this column → neural embeddings
cfg.data.text_domain      = ["subject"]       # this column → TF-IDF topics

Each embedded column emits <col>_EMB_0 … _EMB_{n-1} — dense, numeric, tree-friendly features that decode back to their origin column through the naming grammar.

Or embed every text column globally¶

cfg.stateful_fe.add_text_embeddings = True
cfg.stateful_fe.text_embedding_mode = "augment"   # "augment" (both) | "replace" (embeddings only)
cfg.data.text_domain                = ["subject", "description"]

add_text_embeddings embeds the data.text_domain columns.
mode="augment" (default) runs both TF-IDF+SVD and embeddings on those columns (more features, more cost); mode="replace" emits embeddings only.
A per-column embedding_domain always takes the neural path regardless of this switch.

Source, model & reduction¶

from parleml_ml4.config.text_embedding import TextEmbeddingConfig

cfg.stateful_fe.text_embedding = TextEmbeddingConfig()      # a model on this machine (the default)
cfg.stateful_fe.text_embedding_reducer    = "svd"    # "svd" | "pca" | "none"
cfg.stateful_fe.text_embedding_components = 50        # reduced width per column

Where the vectors come from is one config, TextEmbeddingConfig, shared by the supervised encoder (stateful_fe.text_embedding) and clustering fusion (UnsupervisedConfig.text_embedding). It has two sources — see Where the vectors come from below.

The raw embedding is reduced by a deterministic linear reducer (TruncatedSVD or PCA — never UMAP) so the output geometry is reproducible at predict time. The reducer basis is learnt from the training embeddings and then frozen; inference rows are projected onto it (label-free, leakage-safe by construction — the step fits once at the global locus). The output width is always exactly text_embedding_components, zero-padded if the reducer yields fewer.

A local model is an optional heavy dependency; graceful when absent at fit

A local source uses sentence-transformers (the single parleml-ml4[embeddings] extra). It is import-guarded: if the package is missing at fit the encoder warns once and emits inert (all-zero) embedding columns rather than failing the run, and records the degradation. A model fitted with real vectors refuses to serve where the package is missing. A provider source never degrades: its errors fail the run — you chose it.

Details: text normalization parity

The supervised encoder feeds the reducer the raw model output (no L2 normalization) and applies strip-only input normalization (no internal-whitespace collapse), truncated at 10,000 characters. The clustering path keeps the embedder defaults (whitespace-collapsed, L2-normalized).

Where the vectors come from¶

from parleml_ml4.config.text_embedding import TextEmbeddingConfig

TextEmbeddingConfig()                                                    # local, the default model
TextEmbeddingConfig(model="sentence-transformers/all-MiniLM-L6-v2")      # local, another model
TextEmbeddingConfig(source="provider", connection="openai-embeddings")   # a provider connection

Field	Meaning
`source`	`"local"` (default): a sentence-transformers model on this machine. `"provider"`: an embedding provider's API through a named connection.
`model`	Local: a model id or a directory (`None` → `BAAI/bge-small-en-v1.5`, `DEFAULT_LOCAL_MODEL`). Provider: the model the connection serves (`None` → whatever it serves at fit).
`connection`	Provider only: the embedding-provider connection's name.
`device`	Local only: `"auto"` (CUDA, then MPS, else CPU), `"cpu"`, `"cuda"`, `"mps"`.
`batch_size`	Texts per model call or per request. Throughput only.

On this machine: the revision is pinned¶

A model id names a repository whose weights change over time. At fit, ml4 resolves the id to the revision it loads — the Hugging Face commit of the snapshot, or a sha256: digest of a model directory — and records it with the model. Scoring loads exactly that revision, from the cache or by downloading it. When this machine can produce only other revisions (offline, with another snapshot cached) scoring stops with LocalModelMismatch, naming both revisions: embedding with weights the model never saw would silently corrupt every prediction.

The weights live in the standard Hugging Face cache (HF_HOME). For air-gapped servers, pre-populate the cache with the pinned revision, or use a model directory (its content digest is the pin).

Whether this server can run a source before a run reaches its embedding pass:

from parleml_ml4.embeddings import local_model_status, source_available

source_available(TextEmbeddingConfig())          # (True, "ok") — or (False, why not)
local_model_status("BAAI/bge-small-en-v1.5")     # ("cached", "ok"), ("download", …), ("missing", …), ("not_installed", …)

A local model needs sentence-transformers and its weights: a model directory, or any revision of the model in the Hugging Face cache. A model that is not cached is fine online — download, with a reason saying it downloads on first use — and missing offline (HF_HUB_OFFLINE), with the huggingface-cli download … remedy, so an air-gapped server learns it before the fit rather than at its embedding pass. The supervised encoder then emits inert zero columns (and records the degradation); ParleML refuses a clustering launch that would embed, warns on a supervised plan, and shows the same status per model on Settings → System and in the source choice. A provider source needs LiteLLM; its connection is resolved when the source opens.

Through a provider: the fingerprint¶

Text leaves the machine on this source

A provider source sends the contents of the embedded text columns to the provider at fit and at every scoring call, and the model then depends on that connection. Use a local model if the text must stay on-prem.

A provider is reached through LiteLLM — OpenAI, Azure OpenAI, Bedrock, Vertex, Cohere, Voyage, Mistral, or any OpenAI-compatible server (TEI, vLLM, Ollama). The config names a connection, never a credential: the platform that owns the connection installs a resolver once per process, and ml4 asks it for the connection's credentials at the moment of each call. Nothing in a config, a recipe, a fitted model or a log line carries them.

from parleml_ml4.embeddings import ProviderCredentials, set_provider_resolver

def resolve(connection: str) -> ProviderCredentials:
    return ProviderCredentials(
        connection=connection, provider="openai", model="text-embedding-3-large",
        api_key=my_secret_store[connection],
    )

set_provider_resolver(resolve)

provider is one of openai, azure, bedrock, vertex_ai, cohere, voyage, mistral or openai_compatible (any server that speaks OpenAI's embeddings API at api_base); the provider backend turns it and model into LiteLLM's route. A failed call is re-raised naming the connection, the provider and the error type, with the key redacted and the provider's own exception detached — it can echo the request.

A provider can replace the model behind a name without notice. So at fit ml4 embeds three fixed sentences (parleml_ml4.embeddings.PROBE_SENTENCES) and stores the vectors with the model. Before scoring it embeds them again and compares each pair: any cosine similarity below MIN_COSINE (0.999) — or a different width — stops scoring with EmbeddingModelChanged: the provider changed the model behind the same name. Repeat calls to the same model agree far above that bound; a different model's vectors do not come near it.

Online scoring checks once per process, on the first request.
Batch scoring re-checks before every batch: call parleml_ml4.embeddings.verify_records(pipeline.text_embedding_records()).
Scoring also stops if the connection now serves a different model name (ProviderUnavailable), or cannot be resolved at all.

What the model records¶

Every fit records its sources as EmbeddingRecords, read off the fitted model:

pipeline.text_embedding_records()
# (EmbeddingRecord(source='provider', model='text-embedding-3-large', dimensions=3072,
#                  connection='openai-embeddings', provider='openai', fingerprint='3f9c…'),)
record.to_dict()
# {connection, provider, model, dimensions, fingerprint} — or {model, revision, dimensions} for a local one

FittedPipeline.text_embedding_records() and UnsupervisedPipeline.text_embedding_records() return them; they are the model's external dependencies. An artefact stamped with any other format version is refused at load with StaleArtifactError (re-train it). The supervised bundle format is 5, the unsupervised artefact format 10.0.

Every refusal subclasses parleml_ml4.embeddings.EmbeddingRefused, so a host can stop a batch or answer an online request on one type and show its message as is. A local model refuses (LocalModelMismatch) where its pinned revision cannot be loaded — and where sentence-transformers is not installed at all: a model trained on real vectors is never served zeros.

Caching¶

Vectors are memoized in a dataset-scoped sidecar cache when the run gives a cache directory (stateful_fe.text_embedding_cache_dir / UnsupervisedConfig.text_embedding_cache_dir, set by the platform), keyed by sha1(normalized text). Scoring reads and writes the same cache, so a text that was embedded for training — or for an earlier scoring call — is not sent to the model again.

Each cache file is namespaced by the model (a local model's pinned revision, or the provider connection and its routed model name) and the text preparation (normalize, collapse_whitespace, max_input_chars). A provider can replace the model behind a name, so a provider fit also checks the cache against the model: the fingerprint sentences are cached like any text, and when the vectors the cache holds for them sit below MIN_COSINE to the fresh ones, the cache is dropped before anything is read — a retrain after EmbeddingModelChanged never mixes the two models' vectors.

Details: why the namespace includes the preparation

The supervised encoder and the clustering preprocessor share one dataset-scoped cache directory on purpose — embedding the same corpus twice is the expensive thing to avoid. But they prepare text differently (the encoder pins normalize=False, collapse_whitespace=False and a 10k-char truncation). Per-row keys cannot encode normalize, so a narrower namespace would let whichever consumer ran first hand its vectors to the other — feeding un-normalized vectors into a cosine-metric reducer, or vice versa.

The cache files are plain files with no eviction; deleting them only costs a re-embed.

Text-aware clustering fusion¶

The same embeddings enrich clustering: the text block is embedded, reduced, and fused with the structured (numeric + categorical) block via a FusionScaler so neither dominates the geometry. Configure it on UnsupervisedConfig:

from parleml_ml4.config.unsupervised import UnsupervisedConfig
from parleml_ml4.config.clustering import ClusteringConfig
from parleml_ml4.config.text_embedding import TextEmbeddingConfig
from parleml_ml4.unsupervised import ClusteringDetector

uns = UnsupervisedConfig(
    use_text_embeddings=True,
    text_columns=["description"],
    text_embedding=TextEmbeddingConfig(),     # or source="provider", connection="…"
    text_embedding_reducer="pca",             # "pca" | "umap" | "none"
    text_embedding_umap_n_components=10,
    fusion_balance="equal_variance",          # "equal_variance" | "weight" | "none"
    text_block_weight=1.0,
)

det = ClusteringDetector(ClusteringConfig(algorithm="hdbscan"),
                         unsupervised_config=uns).fit(df)
print(det.result_.cluster_keywords)     # c-TF-IDF keywords per cluster (auto)

Class-based TF-IDF (c-TF-IDF) keywords are extracted automatically whenever text columns are embedded — the most reliable "what is this cluster" signal. Cluster names are not part of a fit: a host application writes them afterwards from this evidence (ParleML's clusters.names annotation). See Unsupervised for the full clustering surface.

Explaining a text column in a supervised run¶

A supervised fit mines the same c-TF-IDF signal against the target instead of against clusters, and persists it on the served pipeline:

pipeline = fit_pipeline(config, recipe, train_df)

pipeline.feature_insight.insight_columns()
# {'keywords': ['notes'], 'categorical': ['segment']}

insight = pipeline.feature_insight.keyword_insight("notes", top_n=10)
for card in insight.components:
    print(card.label, [t.text for t in card.tokens])
# yes ['superb', 'excellent build', ...]
# no  ['dreadful', 'shoddy build', ...]

Two things make this different from the per-encoder introspection you might expect:

It reads the raw text, not the encoding. A column explains identically whether it was routed to text_domain (TF-IDF), embedding_domain, mt_domain or wt_domain — one card grid, one vocabulary, no per-encoder dialect. Categorical columns (cat_features) get the parallel target-lift half of the same payload.
It is mined at fit time and shipped in the bundle. The served artefact keeps no training rows and AutoGluon owns the fold-local encoders, so nothing downstream could reconstruct it later.

Regression has no classes to contrast, so the target is cut into quantile bins and each bin becomes a card (Bin 1/4 (up to 36.3), …). Two guards keep it a display artefact rather than a cost centre: keywords are mined on a seeded sample (so a re-fit explains itself the same way), and a target with more than 20 classes is skipped — the per-group document gets too thin for c-TF-IDF to mean anything.

See FeatureInsight for the full payload.