Embeddings (`parleml_ml4.embeddings`)¶

The single, shared neural-embedding engine used by both the supervised feature-engineering encoder and clustering fusion. One source config names where the vectors come from — a model on this machine or an embedding-provider connection — and every fit records what it resolved to, which scoring then proves before it embeds. See the Text & embeddings guide for usage.

In short

TextEmbeddingConfig names the source: a local sentence-transformers model or a provider connection. Both the supervised encoder and clustering fusion use it.
open_for_fit returns an embedder and an EmbeddingRecord; open_for_scoring and verify_records prove a fitted model's source is unchanged before it embeds.
Every refusal subclasses EmbeddingRefused, so a host can catch one type.
Credentials never live in ml4: a platform installs a resolver with set_provider_resolver.

TextEmbeddingConfig¶

The source: local (a sentence-transformers model; model=None is DEFAULT_LOCAL_MODEL) or provider (a named connection, whose credentials the platform resolves at each call). Nested as StatefulFEConfig.text_embedding and UnsupervisedConfig.text_embedding.

text_embedding ¶

Where text embeddings come from — :class:TextEmbeddingConfig.

One config for every place ml4 embeds free text: the supervised :class:~parleml_ml4.fe.transformers.embedding.EmbeddingEncode (StatefulFEConfig.text_embedding) and the clustering text preprocessor (UnsupervisedConfig.text_embedding). It names the source of the vectors and nothing else; how each consumer reduces them stays on that consumer's own config.

Two sources:

local A sentence-transformers model run on this machine. model names it (None: the default, :data:DEFAULT_LOCAL_MODEL). The fit records the exact revision of the weights it used, and scoring refuses to run on any other (:mod:parleml_ml4.embeddings.sources). provider An embedding provider's API reached through a named connection — OpenAI, Azure OpenAI, Bedrock, Vertex, Cohere, Voyage, Mistral or an OpenAI-compatible server, called through LiteLLM. The config names the connection only; its credentials are resolved by the platform at the moment of each call (:func:parleml_ml4.embeddings.set_provider_resolver) and never stored. model is the model the connection serves (None: whatever it serves at fit). Text is sent to the provider at fit and at every scoring call; the fit records a fingerprint of the model, and scoring stops if the provider has changed the model behind the same name.

TextEmbeddingConfig `dataclass` ¶

TextEmbeddingConfig(
    source: EmbeddingSource = "local",
    model: Optional[str] = None,
    connection: Optional[str] = None,
    device: str = "auto",
    batch_size: int = 256,
)

Bases: StrictConfigMixin

The source of a text embedding: a local model, or a provider connection.

local_model `property` ¶

local_model: str

The local model this config runs (the default when it names none).

describe ¶

describe() -> str

One line for logs and errors: local BAAI/bge-small-en-v1.5 / provider via openai-emb.

coerce `classmethod` ¶

coerce(
    value: Union[
        "TextEmbeddingConfig", Mapping[str, Any], None
    ],
) -> "TextEmbeddingConfig"

value as a config: an instance as is, a mapping (JSON, a recipe step) built, None the default.

Opening a source: fit and scoring¶

open_for_fit turns a config into an embedder and an EmbeddingRecord; open_for_scoring opens the source a record names and proves it is the same model; verify_records re-checks a fitted model's records (batch scoring calls it before every batch).

sources ¶

Opening a text-embedding source — at fit, and again when the fitted model scores.

The two moments make different promises:

:func:open_for_fit turns a :class:~parleml_ml4.config.text_embedding.TextEmbeddingConfig into an embedder and records what it resolved to (:class:~parleml_ml4.embeddings.record.EmbeddingRecord): a local model's revision, or a provider's connection, model and fingerprint.
:func:open_for_scoring opens the source the record names and proves it is the same: the pinned local revision (:class:~parleml_ml4.embeddings.errors.LocalModelMismatch otherwise), or the provider's fingerprint (:class:~parleml_ml4.embeddings.errors.EmbeddingModelChanged), checked once per process.

:func:verify_records re-checks a fitted model's records on demand — batch scoring calls it before every batch, so a provider change is caught before the first row, not after the last.

OpenedSource `dataclass` ¶

OpenedSource(
    embedder: TextEmbedder, record: EmbeddingRecord
)

An embedder ready to use, and the record of what it embeds with.

open_for_fit ¶

open_for_fit(
    config: TextEmbeddingConfig,
    *,
    preparation: TextPreparation = TextPreparation(),
    cache_dir: Optional[str] = None
) -> OpenedSource

Open config's source for a fit and record what it resolved to.

A local source resolves the model's current revision (downloading it when needed); a provider source resolves the connection's credentials and takes the fingerprint.

open_for_scoring ¶

open_for_scoring(
    record: EmbeddingRecord,
    *,
    device: str = "auto",
    batch_size: int = 256,
    preparation: TextPreparation = TextPreparation(),
    cache_dir: Optional[str] = None
) -> TextEmbedder

Open the source record names, proven to be the one the fit used.

Raises :class:~parleml_ml4.embeddings.errors.EmbeddingRefused (a subclass naming the cause) when it is not.

verify_records ¶

verify_records(
    records: Iterable[EmbeddingRecord],
    *,
    refresh: bool = True
) -> None

Prove each record's source is still the one its model was trained on.

A local record's backend and pinned revision must be loadable here; a provider record's fingerprint is re-taken (every call with refresh=True, the batch-scoring default). Raises the first :class:~parleml_ml4.embeddings.errors.EmbeddingRefused.

EmbeddingRecord¶

What a fitted model records about each text-embedding source: a local model's pinned revision, or a provider's connection, model, dimensions and fingerprint.

record ¶

What a fitted model records about its text embeddings: :class:EmbeddingRecord.

Written at fit by every consumer that embeds, carried in the fitted state (so the model knows which vectors it was trained on), and published by the pipelines (FittedPipeline.text_embedding_records() / UnsupervisedPipeline.text_embedding_records()) so a platform can list the model's external dependencies without reading its internals.

A local record pins the model's weights: revision is the Hugging Face commit the fit loaded (or a content digest for a model directory). Scoring loads exactly that revision.
A provider record names the connection and the model, and keeps the vectors the provider returned for the fingerprint sentences (:mod:parleml_ml4.embeddings.fingerprint). Scoring re-embeds them and stops if the provider's answer moved.

EmbeddingRecord `dataclass` ¶

EmbeddingRecord(
    source: EmbeddingSource,
    model: str,
    dimensions: int,
    revision: Optional[str] = None,
    connection: Optional[str] = None,
    provider: Optional[str] = None,
    fingerprint: Optional[str] = None,
    probes: Tuple[Tuple[float, ...], ...] = (),
)

The source a model's embeddings came from, as the fit observed it.

describe ¶

describe() -> str

One plain line: OpenAI text-embedding-3-large via connection openai-embeddings.

to_dict ¶

to_dict() -> dict[str, Any]

The JSON form: every field but the probe vectors, which stay in the fitted state.

The fingerprint¶

Three fixed sentences embedded at fit and again before scoring; a cosine below MIN_COSINE stops scoring.

fingerprint ¶

The provider fingerprint: does the model behind a name still embed the way it did at fit?

A provider can replace the model behind a model name without notice — a new snapshot, a quantised deployment, another region's build. A model trained on the old vectors then scores on vectors from a different space: every prediction shifts, and nothing fails.

At fit ml4 embeds three fixed sentences (:data:PROBE_SENTENCES) and stores the vectors with the model. Before scoring it embeds them again and compares each pair by cosine similarity; any pair below :data:MIN_COSINE stops scoring with :class:~parleml_ml4.embeddings.errors.EmbeddingModelChanged. The threshold leaves room for the provider's own numeric noise (repeat calls agree to ~1e-6) and none for a different model, whose vectors for the same sentence sit far below it.

:func:check_fingerprint runs once per process for a given record (online scoring pays it on its first request); refresh=True re-checks regardless (batch scoring, before every batch).

take_probes ¶

take_probes(backend: _Embeds) -> ndarray

The backend's vectors for :data:PROBE_SENTENCES, as a (3, dim) float32 array.

fingerprint_digest ¶

fingerprint_digest(probes: ndarray) -> str

A short digest of the probe vectors, for display (comparison uses cosine, not this).

min_cosine ¶

min_cosine(stored: ndarray, fresh: ndarray) -> float

The lowest row-wise cosine similarity between two equally shaped sets of vectors.

check_fingerprint ¶

check_fingerprint(
    record: EmbeddingRecord,
    backend: _Embeds,
    *,
    refresh: bool = False
) -> None

Re-embed the probe sentences and refuse if the provider's model has changed.

Skipped when this process already verified record and refresh is false. Raises :class:EmbeddingModelChanged naming the connection and the model.

forget_verified ¶

forget_verified() -> None

Drop this process's verified fingerprints (tests; a resolver swap).

Provider credentials¶

The resolver a platform installs once per process; ml4 never stores a credential.

credentials ¶

Provider credentials: resolved by the platform at the moment of each call, never stored.

ml4 does not know where connections live. A platform installs a resolver once per process (:func:set_provider_resolver) that turns a connection name into :class:ProviderCredentials; ml4 calls it when a provider source opens — at fit, and again when a fitted model scores. The credentials live in memory for that call only: no config, recipe, fitted state or log line carries them.

ProviderCredentials `dataclass` ¶

ProviderCredentials(
    connection: str,
    provider: str,
    model: str,
    api_base: Optional[str] = None,
    api_key: Optional[str] = None,
)

Everything a provider call needs, for one connection.

set_provider_resolver ¶

set_provider_resolver(
    resolver: Optional[ProviderResolver],
) -> None

Install (or, with None, remove) the process's connection → credentials resolver.

resolve_provider_credentials ¶

resolve_provider_credentials(
    connection: str,
) -> ProviderCredentials

The credentials for connection, from the installed resolver.

Raises :class:ProviderUnavailable when no resolver is installed or the resolver cannot produce the connection (deleted, of another kind, its key missing from the vault).

Refusals¶

errors ¶

Why ml4 refuses to embed: the model a fit used is not the model scoring would use.

Every refusal subclasses :class:EmbeddingRefused, so a caller can stop a batch or an online request on one type and show its message as is — each message says what changed and what to do about it.

EmbeddingRefused ¶

Bases: RuntimeError

Embedding would not reproduce the vectors the model was trained on.

EmbeddingModelChanged ¶

Bases: EmbeddingRefused

The provider returns different vectors for the fingerprint sentences than at fit.

LocalModelMismatch ¶

Bases: EmbeddingRefused

The local model's pinned revision is not the one this machine can load.

ProviderUnavailable ¶

Bases: EmbeddingRefused

The provider connection cannot be resolved, or now serves another model.

TextEmbedder and TextPreparation¶

Cache-backed, deduplicating embedder. Blank rows map to zero vectors and never reach the backend. The cache namespace covers the model (a local revision, or the provider's routed model) and the text preparation, so the supervised encoder and the clustering preprocessor, which prepare text differently while sharing a dataset-scoped cache directory, never read each other's vectors.

embedder ¶

The cache-backed text embedder shared by trees and clustering.

Pays the expensive embedding step once per (text, model) and serves every consumer from the dataset-scoped cache — at fit, and again at scoring, so a text scored twice is embedded once. Empty/blank documents map to zero vectors and never reach the backend or the cache.

TextPreparation `dataclass` ¶

TextPreparation(
    normalize: bool = True,
    collapse_whitespace: bool = True,
    max_input_chars: int = 8000,
)

How a consumer prepares text and vectors — part of what the cached vectors mean.

normalize L2-normalises the output; collapse_whitespace folds internal whitespace runs before embedding (False: strip only); max_input_chars truncates each document (a coarse guard; token limits are model-specific). The supervised tree encoder uses (False, False, 10_000); clustering uses the defaults.

TextEmbedder ¶

TextEmbedder(
    backend: EmbeddingBackend,
    *,
    preparation: TextPreparation = TextPreparation(),
    cache_dir: Optional[str] = None
)

Embed text columns with caching, deduplication and graceful empties.

cache `property` ¶

cache: Optional[EmbeddingCache]

The dataset-scoped cache these vectors are kept in (None: no cache).

cache_namespace `property` ¶

cache_namespace: str

The cache file stem: every knob that changes the stored vectors.

The per-row key is sha1(normalized_text), which encodes neither the model nor the preparation, so both are in the file name — otherwise the tree encoder (unnormalised, strip-only) and clustering (normalised, collapsed) would read each other's vectors for the same text (UN3-2). A local model's namespace carries its pinned revision, so a changed revision never reads the old weights' vectors.

dim `property` ¶

dim: int

Output dimensionality — from the cache when populated, else the backend.

embed_texts ¶

embed_texts(raw_texts: list[object]) -> ndarray

Embed a column of raw values into a (len(raw_texts), dim) array.

Order-preserving, deduplicated and cache-backed. Blank rows are zero vectors and never hit the backend/cache.

embed_frame ¶

embed_frame(
    df: DataFrame, text_columns: list[str]
) -> dict[str, ndarray]

Embed each text column independently → {column: (n_rows, dim)}.

normalize_text ¶

normalize_text(
    value: object,
    max_chars: int,
    collapse_whitespace: bool = True,
) -> str

Coerce a cell to a clean string: NaN/None → "", strip, truncate.

When collapse_whitespace (default) internal whitespace runs are collapsed to a single space; set it False for strip-only parity.

blank_mask ¶

blank_mask(texts: list[str]) -> ndarray

Boolean mask of rows whose normalized text is empty.

Backends¶

LocalBackend loads one pinned revision of a sentence-transformers model; ProviderBackend calls a provider through LiteLLM with the credentials it was given. Backends return raw vectors; normalisation is the embedder's job. resolve_device is the single source of truth for device selection (shared with the backend's system-info probe). source_available says whether a config can run on this server — for a local source, the package and a model it can load (local_model_status: cached, download on first use while online, missing offline or for a name that is no model, or not_installed).

backends ¶

Embedding backends: a pinned local model, or a provider connection.

Each backend imports its heavy dependency only when it is built, so importing this package costs nothing. :func:source_available answers "can this source run here?" without loading a model or touching the network.

EmbeddingBackend ¶

Bases: Protocol

Encode a list of (already-normalized, non-empty) texts into vectors.

identity `instance-attribute` ¶

identity: str

Stable identifier for logs, e.g. "local:BAAI/bge-small-en-v1.5@5c38ec7c".

cache_namespace `instance-attribute` ¶

cache_namespace: str

The part of the embedding-cache file name that names these vectors' model.

dim `property` ¶

dim: int

Output dimensionality (known after the model/endpoint is resolved).

embed ¶

embed(texts: list[str]) -> ndarray

Return a (len(texts), dim) float32 array of raw (unnormalised) vectors.

l2_normalize ¶

l2_normalize(matrix: ndarray) -> ndarray

Row-wise L2 normalization that is safe for zero rows.

Zero rows are left as zeros (norm clamped to 1) rather than producing NaNs.

local_model_status ¶

local_model_status(
    model: str,
) -> tuple[LocalModelState, str]

Whether this server can load model, and why not — from the package and the cache alone.

A model directory loads as it is. A hub model id loads from the Hugging Face cache when any revision of it is there; otherwise only by downloading it, which an offline server (HF_HUB_OFFLINE) cannot do — there the fit used to fail only once it reached its embedding pass. The reason is "ok" for a cached model and says what happens otherwise.

source_available ¶

source_available(
    config: TextEmbeddingConfig,
) -> tuple[bool, str]

Whether config's source can run on this server: (ok, reason).

A local source needs sentence-transformers and its model loadable here (:func:~parleml_ml4.embeddings.backends.local.local_model_status): cached, or downloadable on first use — a reason that comes back with ok says so. A provider source needs LiteLLM; its connection's credentials are resolved when the source opens (:mod:parleml_ml4.embeddings.sources).

base ¶

Embedding backend protocol + shared numpy helpers.

A backend is the thin "encode these strings into vectors" boundary. Two implementations ship: :class:~parleml_ml4.embeddings.backends.local.LocalBackend (a pinned sentence-transformers model on this machine) and :class:~parleml_ml4.embeddings.backends.provider.ProviderBackend (a provider's API through LiteLLM). Backends return the model's raw vectors; normalisation is the embedder's job (:class:~parleml_ml4.embeddings.embedder.TextEmbedder), so it is applied identically to both.

EmbeddingBackend ¶

Bases: Protocol

Encode a list of (already-normalized, non-empty) texts into vectors.

identity `instance-attribute` ¶

identity: str

Stable identifier for logs, e.g. "local:BAAI/bge-small-en-v1.5@5c38ec7c".

cache_namespace `instance-attribute` ¶

cache_namespace: str

The part of the embedding-cache file name that names these vectors' model.

dim `property` ¶

dim: int

Output dimensionality (known after the model/endpoint is resolved).

embed ¶

embed(texts: list[str]) -> ndarray

Return a (len(texts), dim) float32 array of raw (unnormalised) vectors.

resolve_device ¶

resolve_device(requested: str = 'auto') -> str

Resolve a torch device string.

requested other than "auto"/"" is honoured verbatim; "auto" picks "cuda" then "mps" when available, else "cpu". The single source of truth for device selection (the local backend and the backend's system-info probe both use it). Torch is imported lazily so this stays cheap when no device work is needed.

l2_normalize ¶

l2_normalize(matrix: ndarray) -> ndarray

Row-wise L2 normalization that is safe for zero rows.

Zero rows are left as zeros (norm clamped to 1) rather than producing NaNs.

local ¶

The local backend: a sentence-transformers model, pinned to one revision of its weights.

A model id names a repository whose weights change over time; the same name can load different weights next month or on another machine. The fit therefore resolves the name to a revision — the Hugging Face commit of the snapshot it loads, or a content digest for a model directory — and records it (:class:~parleml_ml4.embeddings.record.EmbeddingRecord). Scoring loads exactly that revision, from the cache or by downloading it; when this machine can produce only other revisions it refuses with :class:~parleml_ml4.embeddings.errors.LocalModelMismatch instead of embedding with weights the model never saw.

sentence_transformers is imported lazily, so a stock ml4 install (without the parleml-ml4[embeddings] extra) is unaffected until a local source actually embeds.

LocalBackend ¶

LocalBackend(
    model: str,
    *,
    revision: Optional[str] = None,
    device: str = "auto",
    batch_size: int = 256
)

Encode text with one pinned revision of a sentence-transformers model.

local_backend_available ¶

local_backend_available() -> tuple[bool, str]

Whether sentence-transformers is importable, and why not.

local_model_status ¶

local_model_status(
    model: str,
) -> tuple[LocalModelState, str]

Whether this server can load model, and why not — from the package and the cache alone.

A model directory loads as it is. A hub model id loads from the Hugging Face cache when any revision of it is there; otherwise only by downloading it, which an offline server (HF_HUB_OFFLINE) cannot do — there the fit used to fail only once it reached its embedding pass. The reason is "ok" for a cached model and says what happens otherwise.

cached_revisions ¶

cached_revisions(model: str) -> list[str]

The revisions of hub model model in this machine's Hugging Face cache.

resolve_local_snapshot ¶

resolve_local_snapshot(
    model: str, revision: Optional[str] = None
) -> tuple[Path, str]

The directory holding model's weights at revision, and that revision.

revision=None (a fit) takes the current revision — a hub model's latest snapshot, downloading it if needed. A pinned revision (scoring) must be produced exactly, or :class:LocalModelMismatch is raised naming what this machine has instead.

provider ¶

The provider backend: a provider's embedding API, called through LiteLLM.

This is the one place ml4 uses LiteLLM, and it only embeds. The credentials arrive from the platform for each connection (:mod:parleml_ml4.embeddings.credentials) and are passed straight into the call — no environment variables, no files. Network retries and backoff are LiteLLM's.

A failed call is re-raised without the provider's own exception attached: a provider or HTTP error can echo the request (its URL, a key), and the error travels into run logs and job messages. What remains names the connection, the provider and the error type.

ProviderBackend ¶

ProviderBackend(
    credentials: ProviderCredentials,
    *,
    batch_size: int = 256
)

Encode text through one embedding-provider connection.

provider_backend_available ¶

provider_backend_available() -> tuple[bool, str]

Whether LiteLLM is importable, and why not.

litellm_route ¶

litellm_route(provider: str, model: str) -> str

The model name LiteLLM routes on: "<provider>/<model>".

An OpenAI-compatible server speaks OpenAI's API at its own api_base.

Next: Text & embeddings guide · Unsupervised

Embeddings (parleml_ml4.embeddings)¶

TextEmbeddingConfig¶

text_embedding ¶

TextEmbeddingConfig dataclass ¶

local_model property ¶

describe ¶

coerce classmethod ¶

Opening a source: fit and scoring¶

sources ¶

OpenedSource dataclass ¶

open_for_fit ¶

open_for_scoring ¶

verify_records ¶

EmbeddingRecord¶

record ¶

EmbeddingRecord dataclass ¶

describe ¶

to_dict ¶

The fingerprint¶

fingerprint ¶

take_probes ¶

fingerprint_digest ¶

min_cosine ¶

check_fingerprint ¶

forget_verified ¶

Provider credentials¶

credentials ¶

ProviderCredentials dataclass ¶

set_provider_resolver ¶

resolve_provider_credentials ¶

Refusals¶

errors ¶

EmbeddingRefused ¶

EmbeddingModelChanged ¶

LocalModelMismatch ¶

ProviderUnavailable ¶

TextEmbedder and TextPreparation¶

embedder ¶

TextPreparation dataclass ¶

TextEmbedder ¶

cache property ¶

cache_namespace property ¶

dim property ¶

embed_texts ¶

embed_frame ¶

normalize_text ¶

blank_mask ¶

Backends¶

backends ¶

EmbeddingBackend ¶

identity instance-attribute ¶

cache_namespace instance-attribute ¶

dim property ¶

embed ¶

l2_normalize ¶

local_model_status ¶

source_available ¶

base ¶

EmbeddingBackend ¶

identity instance-attribute ¶

cache_namespace instance-attribute ¶

dim property ¶

embed ¶

resolve_device ¶

l2_normalize ¶

local ¶

LocalBackend ¶

local_backend_available ¶

local_model_status ¶

cached_revisions ¶

resolve_local_snapshot ¶

provider ¶

ProviderBackend ¶

provider_backend_available ¶

litellm_route ¶

Embeddings (`parleml_ml4.embeddings`)¶

TextEmbeddingConfig `dataclass` ¶

local_model `property` ¶

coerce `classmethod` ¶

OpenedSource `dataclass` ¶

EmbeddingRecord `dataclass` ¶

ProviderCredentials `dataclass` ¶

TextPreparation `dataclass` ¶

cache `property` ¶

cache_namespace `property` ¶

dim `property` ¶

identity `instance-attribute` ¶

cache_namespace `instance-attribute` ¶

dim `property` ¶

identity `instance-attribute` ¶

cache_namespace `instance-attribute` ¶

dim `property` ¶