EDA & leakage

Data-level exploratory analysis and the leakage scanner. See the EDA guide and the EDAConfig parameter table.

compute_eda

compute_eda

compute_eda(
    X: DataFrame,
    *,
    y: ndarray | Series | None = None,
    config: EDAConfig | None = None,
    **kwargs: Any
) -> EDAResult

Compute every requested data-level artifact.

The duplicate-row count, its alert and the duplicate head table come from one pass over the full (un-sampled) frame, as do the checks that confirm a sample-based CONSTANT alert and an ID_LIKENESS finding; every other artifact reads the working sample.

Parameters:

Name	Type	Description	Default
X	DataFrame	Feature matrix. Not modified. Will be sub-sampled internally per config.sample_size.	required
y	ndarray | Series | None	Optional target. When given (and config.target is None), the target is attached to a copy of X under the column name from :func:_resolve_target_column_name. Rows without a target value are left out of the target-dependent artifacts (target correlations, the target tie-break of over-correlated pairs, the leakage scan's target checks) and counted, on the full frame, in a MISSING alert on the target.	None
config	EDAConfig | None	Optional pre-built :class:EDAConfig. When None, a default config is built and individual keyword arguments are copied in field-by-field.	None
**kwargs	Any	Field overrides on the active :class:EDAConfig.	{}

Returns:

Type	Description
EDAResult	Container of every successfully-computed artifact. Per-artifact failures emit a single :class:UserWarning and leave the corresponding field at its default (None / empty).

EDAConfig

EDAConfig dataclass

EDAConfig(
    sample_size: int | None = 20000,
    sample_method: str = "random",
    sample_strategy: str = "simple",
    stratify_column: str | None = None,
    sample_seed: int = 42,
    target: str | None = None,
    task_type: str | None = None,
    categorical_columns: list[str] | None = None,
    time_column: str | None = None,
    overcorrelated_threshold: float = 0.9,
    correlations_max_categorical_distinct: int = 50,
    correlation_numeric_method: str = "pearson",
    correlations_enabled: bool = True,
    n_jobs: int | None = None,
    parallelize: bool = True,
    univariate_enabled: bool = True,
    missingness_enabled: bool = True,
    outliers_enabled: bool = True,
    duplicates_enabled: bool = True,
    alerts_enabled: bool = True,
    alerts_thresholds: dict[str, Any] = dict(),
    missingness_kwargs: dict[str, Any] = (
        lambda: {"matrix_max_rows": 2000}
    )(),
    outliers_kwargs: dict[str, Any] = (
        lambda: {"iqr_factor": 1.5, "zscore_threshold": 3.5}
    )(),
    duplicates_kwargs: dict[str, Any] = (
        lambda: {"max_head_rows": 20, "subset": None}
    )(),
    profiling_config_overrides: dict[str, Any] = dict(),
    profiling_minimal_mode: bool = False,
    histogram_bins: int = 30,
    distributions_enabled: bool = True,
    spikey_bin_ratio: float = 10.0,
    spikey_min_count_share: float = 0.2,
    histogram_gap_threshold: float = 0.1,
    recommendations_enabled: bool = True,
    min_skewness_to_recommend: float = 0.5,
    skewness_improvement_required: float = 0.3,
    bivariate_enabled: bool = True,
    bivariate_min_group_size: int = 5,
    bivariate_top_n: int = 50,
    bivariate_min_numeric_unique: int = 10,
    scatter_top_n: int = 12,
    scatter_max_points: int = 5000,
    boxplot_top_n: int = 12,
    boxplot_max_outliers_per_group: int = 50,
    missingness_reorder_pattern: bool = True,
    pca_top_n_features: int = 12,
    pca_max_points: int = 5000,
    parcoords_top_n_features: int = 12,
    parcoords_max_points: int = 5000,
    leakage_enabled: bool = True,
    leakage_id_cardinality_threshold: float = 0.95,
    leakage_target_correlation_threshold: float = 0.99,
    leakage_binary_auc_threshold: float = 0.98,
    leakage_near_duplicate_threshold: float = 0.99,
    leakage_post_outcome_threshold: float = 0.5,
)

User-tunable knobs for :func:parleml_mli.eda.compute_eda.

Defaults are picked so a 100k×100 frame finishes in <30 s on a modern laptop. Tune per-artifact via the *_kwargs dicts.

sample_size class-attribute instance-attribute

sample_size: int | None = 20000

Row cap for the working sample. None profiles the full frame.

sample_method class-attribute instance-attribute

sample_method: str = 'random'

How to draw the sample: "random" or "head" (first N rows).

Ignored when :attr:sample_strategy is "stratified".

sample_strategy class-attribute instance-attribute

sample_strategy: str = 'simple'

Sampling strategy: "simple" (plain random / head per :attr:sample_method) or "stratified".

"stratified" keys the row budget off the target so an imbalanced dataset's rare class still appears in the EDA sample (proportional allocation with a one-row-per-class floor; continuous targets are binned into deciles first). Requires a target — either y passed to :func:compute_eda, :attr:target, or an explicit :attr:stratify_column. When no label is resolvable it degrades to simple random sampling with a UserWarning.

stratify_column class-attribute instance-attribute

stratify_column: str | None = None

Column to stratify on when :attr:sample_strategy="stratified". None falls back to the resolved target column.

sample_seed class-attribute instance-attribute

sample_seed: int = 42

Seed for the "random" sampler — fixed for reproducible runs.

target class-attribute instance-attribute

target: str | None = None

Name of the target column when it lives inside X; unlocks the target-correlation and leakage analyses.

task_type class-attribute instance-attribute

task_type: str | None = None

Informational hint ("classification" / "regression"). Not used for dispatch — every artifact is computed regardless.

categorical_columns class-attribute instance-attribute

categorical_columns: list[str] | None = None

Explicit categorical column list. None auto-detects by dtype.

time_column class-attribute instance-attribute

time_column: str | None = None

Target-observation-time column. When set, drives the POST_OUTCOME_DATE leakage detector: any datetime feature whose value is later than this on a high share of rows is flagged as post-outcome leakage. See parleml_mli.eda.leakage.

overcorrelated_threshold class-attribute instance-attribute

overcorrelated_threshold: float = 0.9

|correlation| at or above this raises the over-correlated alert.

The default matches common production practice (0.85–0.90). The HIGH_CORRELATION severity ladder scales relative to this threshold (see :func:parleml_mli.eda.alerts.make_high_correlation_alerts), so tightening or loosening the threshold keeps the CRITICAL / HIGH / MEDIUM tiers sensible.

correlations_max_categorical_distinct class-attribute instance-attribute

correlations_max_categorical_distinct: int = 50

Skip categorical columns with more distinct levels than this from the correlation matrix (Cramér's V blows up on high cardinality).

correlation_numeric_method class-attribute instance-attribute

correlation_numeric_method: str = 'pearson'

Numeric × numeric association method: "pearson" (linear) or "spearman" (rank / monotone). "spearman" ranks numeric columns before the kernel, so heavy-tailed / ordinal features are scored by monotone rather than linear association; Cramér's V (cat×cat) is unaffected and η² (cat×num) becomes rank-based η². Zero-variance (constant) numeric pairs are reported as NaN under either method.

correlations_enabled class-attribute instance-attribute

correlations_enabled: bool = True

Compute the numeric/categorical correlation matrix.

n_jobs class-attribute instance-attribute

n_jobs: int | None = None

Worker count for the parallel correlation kernels. None = all cores.

parallelize class-attribute instance-attribute

parallelize: bool = True

Use the multi-core correlation path; set False to force single-thread.

univariate_enabled class-attribute instance-attribute

univariate_enabled: bool = True

Compute per-column univariate statistics.

missingness_enabled class-attribute instance-attribute

missingness_enabled: bool = True

Compute the missingness matrix / pattern analysis.

outliers_enabled class-attribute instance-attribute

outliers_enabled: bool = True

Compute per-column outlier counts (IQR + z-score).

duplicates_enabled class-attribute instance-attribute

duplicates_enabled: bool = True

Keep the head table of duplicate rows (:attr:EDAResult.duplicates_df). The duplicate-row count on :attr:EDAResult.table and its DUPLICATES alert are computed either way.

alerts_enabled class-attribute instance-attribute

alerts_enabled: bool = True

Compute the consolidated data-quality alert list.

alerts_thresholds class-attribute instance-attribute

alerts_thresholds: dict[str, Any] = field(
    default_factory=dict
)

Override the mli-specific alert severity ladders.

The mli-generated alerts (HIGH_CORRELATION, SPIKEY, SKEWED, HISTOGRAM_GAP) all bucket into CRITICAL / HIGH / MEDIUM / LOW via hard-coded cut-offs. Supply any subset of the keys below to move a cut-off without forking the code; unknown keys are ignored. Defaults shown:

• high_correlation_critical_frac (0.667) — CRITICAL when |corr| ≥ threshold + frac × (1 − threshold).
• high_correlation_high_frac (0.333) — HIGH cut-off, same formula.
• spikey_high_ratio (50.0) / spikey_medium_ratio (20.0) — peak-to-mean bin-count ladder.
• skewed_high (0.6) / skewed_medium (0.4) — |robust_skewness| ladder.
• histogram_gap_high (0.30) / histogram_gap_medium (0.20) — gaps_score ladder.

The detection thresholds (what makes a column fire at all) remain the dedicated overcorrelated_threshold / spikey_bin_ratio / min_skewness_to_recommend / histogram_gap_threshold fields. Profiler-owned alerts (MISSING / IMBALANCE / …) are tuned via :attr:profiling_config_overrides instead — see the EDA MkDocs alerts reference.

missingness_kwargs class-attribute instance-attribute

missingness_kwargs: dict[str, Any] = field(
    default_factory=lambda: {"matrix_max_rows": 2000}
)

Missingness overrides — matrix_max_rows caps the rows plotted.

outliers_kwargs class-attribute instance-attribute

outliers_kwargs: dict[str, Any] = field(
    default_factory=lambda: {
        "iqr_factor": 1.5,
        "zscore_threshold": 3.5,
    }
)

Outlier thresholds — iqr_factor (Tukey fence) and zscore_threshold (3.5 = scipy-stats "extreme" cut-off).

duplicates_kwargs class-attribute instance-attribute

duplicates_kwargs: dict[str, Any] = field(
    default_factory=lambda: {
        "max_head_rows": 20,
        "subset": None,
    }
)

Duplicate-detection overrides — max_head_rows caps the head table and an optional subset of columns defines a duplicate for the head table, the count and the alert alike.

profiling_config_overrides class-attribute instance-attribute

profiling_config_overrides: dict[str, Any] = field(
    default_factory=dict
)

data_profiling (fg-data-profiling) Settings deep-merged before invocation. May hold any value — non-JSON-clean values are str()-ified for config persistence.

profiling_minimal_mode class-attribute instance-attribute

profiling_minimal_mode: bool = False

Run the profiler in its minimal (fast, fewer artifacts) mode.

histogram_bins class-attribute instance-attribute

histogram_bins: int = 30

Bins for the high-resolution numeric histogram backfilled by the distribution-enrichment helper. Replaces the profiler's 10-bin default.

distributions_enabled class-attribute instance-attribute

distributions_enabled: bool = True

Run the post-profiler distribution enrichment pass (robust skewness + spikey + gaps detectors).

spikey_bin_ratio class-attribute instance-attribute

spikey_bin_ratio: float = 10.0

A bin is "spikey" iff its count is ratio × mean(counts).

spikey_min_count_share class-attribute instance-attribute

spikey_min_count_share: float = 0.2

Spikey detector also requires the peak bin to hold this much of total mass, defending against the two-bin degenerate case.

histogram_gap_threshold class-attribute instance-attribute

histogram_gap_threshold: float = 0.1

Minimum gaps_score for a column to trip the HISTOGRAM_GAP alert (a histogram-gap heuristic, not a bimodality dip test).

recommendations_enabled class-attribute instance-attribute

recommendations_enabled: bool = True

Compute Tukey-ladder transformation recommendations.

min_skewness_to_recommend class-attribute instance-attribute

min_skewness_to_recommend: float = 0.5

Skip columns whose |robust_skewness| is below this — already near-symmetric.

skewness_improvement_required class-attribute instance-attribute

skewness_improvement_required: float = 0.3

Recommend a transform only if it reduces |skew| by at least this fraction (1 - |new|/|orig|).

bivariate_enabled class-attribute instance-attribute

bivariate_enabled: bool = True

Compute categorical × numeric variability (ANOVA + Brown-Forsythe).

bivariate_min_group_size class-attribute instance-attribute

bivariate_min_group_size: int = 5

Per-group row threshold; pairs without at least two groups passing are skipped.

bivariate_top_n class-attribute instance-attribute

bivariate_top_n: int = 50

Keep at most this many (cat × num) pairs ranked by F-statistic.

bivariate_min_numeric_unique class-attribute instance-attribute

bivariate_min_numeric_unique: int = 10

Drop auto-detected numeric columns with fewer than this many distinct non-null values from the bivariate analysis.

A boxplot is meaningful only when the numeric variable has a continuous distribution. Binary / low-cardinality numerics (encoded targets, 0/1 flags, ordinal codes) auto-detect as "numeric" by dtype but degenerate to single-tick boxes — the η²-ranked heatmap is meaningless and the per-pair boxplot is visually empty. This threshold filters them out at compute time.

When numeric_columns= is explicitly passed to :func:parleml_mli.eda.bivariate.compute_category_variability, the filter is bypassed and the user's choice is respected. Set to 0 / 1 to disable the auto-filter.

scatter_top_n class-attribute instance-attribute

scatter_top_n: int = 12

How many over-correlated pairs the correlated-pairs figure draws. 0 draws none.

scatter_max_points class-attribute instance-attribute

scatter_max_points: int = 5000

Maximum points drawn per pair in the correlated-pairs figure.

boxplot_top_n class-attribute instance-attribute

boxplot_top_n: int = 12

How many top (cat × num) pairs the category-boxplot figure offers. 0 offers none.

boxplot_max_outliers_per_group class-attribute instance-attribute

boxplot_max_outliers_per_group: int = 50

Cap on the outliers the category-boxplot figure draws per group.

missingness_reorder_pattern class-attribute instance-attribute

missingness_reorder_pattern: bool = True

Permute pattern rows / columns by the first SVD component when plotting, so co-missing blocks cluster together. Enabled by default; set False to keep the original column / row order.

pca_top_n_features class-attribute instance-attribute

pca_top_n_features: int = 12

Number of loading arrows the PCA biplot shows. The PCA itself is fit on all usable numeric columns; this only affects rendering.

pca_max_points class-attribute instance-attribute

pca_max_points: int = 5000

Maximum rows the PCA biplot projects (0: every sampled row).

parcoords_top_n_features class-attribute instance-attribute

parcoords_top_n_features: int = 12

Number of feature axes on the parallel-coordinates figure.

parcoords_max_points class-attribute instance-attribute

parcoords_max_points: int = 5000

Maximum rows the parallel-coordinates figure draws. 0 draws none.

leakage_enabled class-attribute instance-attribute

leakage_enabled: bool = True

Auto-invoke :func:parleml_mli.eda.leakage.detect_suspicious_features as the final step of :func:compute_eda. The result lands on :attr:EDAResult.leakage_report. Set to False for cheap EDA runs where the leakage scan is not needed.

leakage_id_cardinality_threshold class-attribute instance-attribute

leakage_id_cardinality_threshold: float = 0.95

Cardinality / row-count ratio above which a column is considered for ID_LIKENESS. Forwarded as id_cardinality_threshold to detect_suspicious_features.

leakage_target_correlation_threshold class-attribute instance-attribute

leakage_target_correlation_threshold: float = 0.99

|correlation| ≥ this triggers TARGET_CORRELATION_EXTREME. Deliberately stricter than :attr:overcorrelated_threshold (default 0.90) because leakage is a stricter framing than the general "overcorrelated" alert.

leakage_binary_auc_threshold class-attribute instance-attribute

leakage_binary_auc_threshold: float = 0.98

For a binary target, a numeric feature whose raw values alone rank the classes with max(ROC AUC, 1 − ROC AUC) ≥ this also triggers TARGET_CORRELATION_EXTREME — the monotone, non-linear leak (a model score, a bounded y + noise) whose |correlation| stays below :attr:leakage_target_correlation_threshold. A feature is scored only when each class has at least 30 labelled rows with a value.

leakage_near_duplicate_threshold class-attribute instance-attribute

leakage_near_duplicate_threshold: float = 0.99

Per-feature group-purity ratio that triggers NEAR_DUPLICATE_PROXY.

leakage_post_outcome_threshold class-attribute instance-attribute

leakage_post_outcome_threshold: float = 0.5

Share of rows where a datetime feature is strictly later than :attr:time_column that triggers POST_OUTCOME_DATE.

to_dict

to_dict() -> dict[str, Any]

Return a JSON-clean dict representation.

profiling_config_overrides is user-supplied and may contain Pydantic objects / enums / numpy scalars. We deep-walk it and replace anything that isn't json.dumps-clean with its str(...) repr — never raise, never silently drop. Persisted on :attr:EDAResult.config and round-trips through json.dumps in report.json.

Uses the shared :func:parleml_mli.core.jsonclean.json_clean walker (EDA-20). The previous local cleaner passed non-finite floats straight through, so a NaN in an override emitted literal NaN — invalid JSON for any strict decoder. Config values are plain settings, so a non-finite is a mistake, not an unbounded threshold: the default "null" inf-policy is right here.

EDAResult

EDAResult dataclass

EDAResult(
    table: TableStats | None = None,
    variables: dict[str, VariableStats] = dict(),
    target_correlations: DataFrame | None = None,
    correlation_matrix: DataFrame | None = None,
    overcorrelated_pairs: list[
        tuple[str, str, float]
    ] = list(),
    overcorrelated_features: list[str] = list(),
    alerts: list[AlertEntry] = list(),
    duplicates_df: DataFrame | None = None,
    outliers_df: DataFrame | None = None,
    missingness_df: DataFrame | None = None,
    missingness_pattern: DataFrame | None = None,
    transformation_recommendations_df: (
        DataFrame | None
    ) = None,
    category_variability_df: DataFrame | None = None,
    leakage_report: Any = None,
    config: dict[str, Any] = dict(),
    schema_version: int = EDA_SCHEMA_VERSION,
)

Cache of every EDA artifact computed for one run.

Every field defaults to None / empty so a partial run does not poison downstream readers. No data_profiling types ever leak into this surface — all fields are plain pandas / numpy / dict / list[dict]. The joblib bundle stays loadable on a machine without the profiler installed (compute-time dep only).

target_correlations class-attribute instance-attribute

target_correlations: DataFrame | None = None

Columns: index, correlation, raw_correlation, method. Sorted desc by correlation. Byte-compat with the legacy target_corr.json via :mod:parleml_mli.reporting.eda_serialization.

correlation_matrix class-attribute instance-attribute

correlation_matrix: DataFrame | None = None

Full p×p, indexed and columned by feature names; mixed-type via :func:parleml_mli.eda.compute_correlation_matrix.

duplicates_df class-attribute instance-attribute

duplicates_df: DataFrame | None = None

The largest duplicate-row groups of the un-sampled frame, one row per group with a leading __n_duplicates__ group-size column (see :func:parleml_mli.eda.duplicates.find_duplicate_rows). At most duplicates_kwargs.max_head_rows rows (default 20); empty when the frame has no duplicates, None when duplicates_enabled=False.

outliers_df class-attribute instance-attribute

outliers_df: DataFrame | None = None

One row per numeric column. Columns (locked contract): column, n_outliers_iqr, p_outliers_iqr, n_outliers_zscore, p_outliers_zscore, lower_iqr, upper_iqr, z_max, z_min. Sorted by p_outliers_iqr desc.

missingness_df class-attribute instance-attribute

missingness_df: DataFrame | None = None

One row per column. Columns (locked contract): column, kind, n_missing, p_missing.

missingness_pattern class-attribute instance-attribute

missingness_pattern: DataFrame | None = None

k × m boolean presence/absence matrix on a sampled set of rows (k ≤ missingness_kwargs.matrix_max_rows, default 2000), indexed by sample row number, columned by feature name.

transformation_recommendations_df class-attribute instance-attribute

transformation_recommendations_df: DataFrame | None = None

One row per numeric column with a non-trivial recommended transform. Columns: column, original_skewness, best_transform, transformed_skewness, skewness_reduction, eligible_transforms, reasons. See :func:parleml_mli.eda.compute_transformation_recommendations.

category_variability_df class-attribute instance-attribute

category_variability_df: DataFrame | None = None

One row per (cat_col, num_col) pair surviving the group-size filter. Locked columns: see :func:parleml_mli.eda.compute_category_variability.

leakage_report class-attribute instance-attribute

leakage_report: Any = None

A :class:parleml_mli.eda.leakage.LeakageReport populated by :func:parleml_mli.eda.leakage.detect_suspicious_features, auto-invoked from :func:compute_eda when EDAConfig.leakage_enabled=True.

Type-annotated as Any to avoid an import cycle (parleml_mli.eda.leakage imports :class:EDAResult); the runtime value is always either None or a LeakageReport.

plot_category_variability_heatmap

plot_category_variability_heatmap(**kw: Any) -> Figure

Overview heatmap of (cat × num) variability.

Scales to wide schemas — see :func:parleml_mli.eda.bivariate.plot_category_variability_heatmap.

to_dict

to_dict() -> dict[str, Any]

JSON-clean envelope for report.json["eda"].

Delegates to :mod:parleml_mli.reporting.eda_serialization.

from_dict classmethod

from_dict(payload: dict[str, Any]) -> 'EDAResult'

Reconstruct an :class:EDAResult from a JSON-clean payload.

Round-trips :meth:to_dict / :func:parleml_mli.reporting.eda_serialization.eda_to_dict; an artifact the envelope leaves out (it was empty or never computed) comes back None / empty.

Parameters:

Name	Type	Description	Default
payload	dict[str, Any]	The dict produced by :meth:to_dict (e.g. a report.json["eda"] envelope).	required

Returns:

Type	Description
EDAResult	A populated EDAResult. missingness_pattern is not part of the JSON envelope; on a JSON-loaded instance it is None.

Raises:

Type	Description
StaleArtifactError	payload carries a schema_version other than :data:EDA_SCHEMA_VERSION (or none).

EDA output types

The typed sub-structures stored on an EDAResult — per-variable and whole-table statistics and the consolidated data-quality alerts (with their severity / type enums) — and the PCA biplot summary the on-demand biplot draws. All are re-exported from the top level (from parleml_mli import AlertEntry).

VariableStats dataclass

VariableStats(
    name: str,
    type: VariableType,
    n: int = 0,
    n_missing: int = 0,
    p_missing: float = 0.0,
    n_distinct: int = 0,
    p_distinct: float = 0.0,
    mean: float | None = None,
    std: float | None = None,
    min: float | None = None,
    max: float | None = None,
    median: float | None = None,
    p25: float | None = None,
    p75: float | None = None,
    iqr: float | None = None,
    skewness: float | None = None,
    kurtosis: float | None = None,
    mad: float | None = None,
    n_zeros: int | None = None,
    p_zeros: float | None = None,
    n_negative: int | None = None,
    p_negative: float | None = None,
    n_infinite: int | None = None,
    top_value: Any = None,
    top_count: int | None = None,
    top_percentage: float | None = None,
    value_counts_head: dict[str, int] = dict(),
    min_datetime: str | None = None,
    max_datetime: str | None = None,
    histogram_counts: list[int] = list(),
    histogram_edges: list[float] = list(),
    robust_skewness: float | None = None,
    is_spikey: bool | None = None,
    spikey_peak_ratio: float | None = None,
    gaps_score: float | None = None,
    n_gaps: int | None = None,
)

Per-column statistics extracted from the profiler description.

Numeric vs categorical fields default to None — only the relevant subset is populated depending on the column's inferred type (type). All numeric values are plain Python float / int / None (no numpy scalars, no NaN — see :func:parleml_mli.eda._profiling_adapter.univariate_from_description).

to_dict

to_dict() -> dict[str, Any]

JSON-clean dict. Drops None/empty fields for compactness.

TableStats dataclass

TableStats(
    n_rows: int = 0,
    n_columns: int = 0,
    n_missing_cells: int = 0,
    p_missing_cells: float = 0.0,
    n_duplicate_rows: int = 0,
    p_duplicate_rows: float = 0.0,
    memory_size_bytes: int = 0,
    types_breakdown: dict[str, int] = dict(),
    n_rows_original: int | None = None,
)

Frame-wide stats (BaseDescription.table projection).

Notes

n_rows reports the sampled frame size that downstream artifacts were computed on (so per-column percentages, missingness rates, etc. line up with what the user reads). n_rows_original captures the un-sampled source size when sampling is active so the audit trail of "this report covers X rows out of a Y-row source" survives onto disk and into report.json. When sampling is off they are equal.

n_duplicate_rows counts rows that exactly repeat an earlier row (extra copies) on the un-sampled frame — compared on EDAConfig.duplicates_kwargs["subset"] when set; p_duplicate_rows divides it by n_rows_original.

n_rows_sampled property

n_rows_sampled: int

Alias for :attr:n_rows so the per-row contract is explicit on inspection (matches the v2 README).

PCASummary dataclass

PCASummary(
    scores_df: DataFrame,
    loadings_df: DataFrame,
    explained_variance_ratio: tuple[float, float],
    feature_means: dict[str, float] = dict(),
    feature_stds: dict[str, float] = dict(),
    n_dropped: int = 0,
)

The 2-component PCA :func:plot_biplot draws.

Scores are sub-sampled to EDAConfig.pca_max_points rows and the loadings keep one row per usable numeric feature.

Attributes:

Name	Type	Description
scores_df	DataFrame	Shape (n_sample, 2) with columns ["PC1", "PC2"]; the original row identity is not kept.
loadings_df	DataFrame	Shape (n_features_used, 2) indexed by feature name. Columns ["PC1", "PC2"].
explained_variance_ratio	tuple[float, float]	Two floats, one per component, summing to <= 1.0.
feature_means, feature_stds		Per-feature standardisation parameters.
n_dropped	int	Count of dropped constant / all-NaN / boolean columns (booleans are excluded to match the rest of :mod:parleml_mli.eda's numeric-subset convention).

AlertEntry dataclass

AlertEntry(
    type: AlertType,
    severity: AlertSeverity,
    column: str | None,
    description: str,
    details: dict[str, Any] = dict(),
)

One row in :attr:EDAResult.alerts.

JSON-clean: every field is a plain Python scalar / list / dict. Frozen so two AlertEntry instances with the same content hash identically (deduping in the alerts pipeline).

to_dict

to_dict() -> dict[str, Any]

JSON-clean dict for report.json["eda"]["alerts"].

AlertSeverity

Bases: str, Enum

Severity of a data-quality alert.

Ordering matches the visual mli colour palette (:func:parleml_mli.core.theme.severity_color).

AlertType

Bases: str, Enum

Data-quality alert taxonomy.

Mostly aligned with data_profiling.model.alerts.AlertType (so we can map the upstream enum 1:1), with three mli-specific additions (EMPTY, TYPE_DATE, HIGH_CARDINALITY) that cover edge cases the profiler's enum does not surface.

normalize_alert_type

normalize_alert_type(raw: Any) -> AlertType

Coerce a serialised alert-type string to an :class:AlertType.

Raises ValueError for a string that names no alert type.

detect_suspicious_features

detect_suspicious_features

detect_suspicious_features(
    df: DataFrame,
    *,
    target: str | None = None,
    time_column: str | None = None,
    eda: EDAResult | None = None,
    id_cardinality_threshold: float = 0.95,
    target_correlation_threshold: float = 0.99,
    binary_auc_threshold: float = 0.98,
    near_duplicate_threshold: float = 0.99,
    post_outcome_threshold: float = 0.5,
    high_cardinality_threshold: float = 0.95,
    full_df: DataFrame | None = None
) -> LeakageReport

Scan df for leakage / suspicious-feature patterns.

The function never raises on a malformed column — each detector is wrapped in its own try/except so a single broken column does not collapse the whole report. Per-detector failures are logged at DEBUG level and silently skipped.

Parameters:

Name	Type	Description	Default
df	DataFrame	Input frame. Not modified. When target is given it must be one of df's columns.	required
target	str | None	Optional target column name. Required for TARGET_CORRELATION_EXTREME and NEAR_DUPLICATE_PROXY. Those target checks read the rows that carry a target value — a missing target is an unlabelled row, not an extra class; every other detector reads every row.	None
time_column	str | None	Optional datetime column whose values represent target observation time. Required for POST_OUTCOME_DATE.	None
eda	EDAResult | None	Optional pre-computed :class:parleml_mli.eda.EDAResult from an earlier call to :func:parleml_mli.eda.compute_eda on the same df. When supplied, SINGLE_VALUE / HIGH_CARDINALITY / TARGET_CORRELATION_EXTREME findings are projected from its already-computed artefacts instead of re-running the underlying alert / correlation pipeline. Pass None for a stand-alone scan; the alerts-derived findings are then computed cheaply from the frame directly.	None
id_cardinality_threshold	float	Cardinality / row-count ratio above which a column is considered for ID_LIKENESS. Default 0.95.	0.95
target_correlation_threshold	float	|correlation| ≥ this triggers TARGET_CORRELATION_EXTREME. Default 0.99 (deliberately stricter than the EDAConfig.overcorrelated_threshold default of 0.90 — leakage is a stricter framing than the general "highly-correlated" alert).	0.99
binary_auc_threshold	float	For a binary target, a numeric feature whose raw values alone rank the classes with max(AUC, 1 − AUC) ≥ this also triggers TARGET_CORRELATION_EXTREME — a monotone, non-linear leak whose |r| stays below target_correlation_threshold. Default 0.98. Scored only when each class has at least 30 labelled rows with a value.	0.98
near_duplicate_threshold	float	Per-feature group-purity ratio that triggers NEAR_DUPLICATE_PROXY. Default 0.99.	0.99
post_outcome_threshold	float	Share of rows where a datetime feature is strictly later than time_column that triggers POST_OUTCOME_DATE. Default 0.5.	0.5
high_cardinality_threshold	float	Cardinality / row-count ratio above which a column is considered HIGH_CARDINALITY (used for the stand-alone path; the eda-supplied path uses the alerts pipeline's own threshold). Default 0.95.	0.95
full_df	DataFrame | None	Optional unsampled frame df was drawn from. A sample overstates how distinct a repeated key is, so ID_LIKENESS confirms its cardinality ratio on this frame when given.	None

Returns:

Type	Description
LeakageReport	A frozen, JSON-clean dataclass holding every finding the detectors produced. Empty findings list on a clean dataset.