Skip to content

Binary classification (fraud)

Profile a fraud classifier end to end: train, profile, grade trust, build a model card, save, and explain a new transaction.

In short

  • One fit_profiler call computes archetypes, rules, explanations, EDA and held-out evaluation.
  • Pass the same held-out rows as X_test (evaluation) and X_holdout (the rules' Test numbers).
  • compute_trust_summary and build_model_card read the populated result; they fit nothing.
  • result.justify explains one new row; bind result.justifier() once for many rows.

In practice the model comes from ml4

This page trains a plain LGBMClassifier to keep the focus on mli. A model built by ml4 is handed over as in Explain an ml4 model.

1. Train

import pandas as pd
from lightgbm import LGBMClassifier
from sklearn.model_selection import train_test_split

df = pd.read_parquet("transactions.parquet")
y = df.pop("is_fraud")

X_train, X_test, y_train, y_test = train_test_split(
    df, y, test_size=0.2, stratify=y, random_state=42,
)
clf = LGBMClassifier(n_estimators=400, learning_rate=0.05).fit(X_train, y_train)

2. Profile with rules, explanations, EDA and evaluation

from parleml_mli import (
    EDAConfig,
    ExplanationsConfig,
    PdpConfig,
    PermutationImportanceConfig,
    ProfilingConfig,
    ReportConfig,
    RulesConfig,
    fit_profiler,
)

config = ProfilingConfig(
    rules=RulesConfig(),
    explanations=ExplanationsConfig(
        pdp=PdpConfig(),
        permutation_importance=PermutationImportanceConfig(),
    ),
    eda=EDAConfig(),
    report=ReportConfig(output_dir="out", persist_inference_data=True),
)

result = fit_profiler(
    model=clf,
    X=X_train,
    y=y_train,
    target_mask=(y_train == 1),          # profile the fraud cohort
    X_test=X_test,
    y_test=y_test,
    X_holdout=X_test,                    # test the mined rules on rows
    y_holdout=y_test,                    # the model never saw
    config=config,
)

3. Read the archetypes & rules

# Which sub-types of fraud did the model learn?
print(result.summary_df[["cluster_id", "n_target_cases", "lift"]])

# Human-readable rules with their Train precision, Test precision and verdict
for rule in result.rules_.default_engine_summary.rules[:10]:
    test = rule.metrics_holdout["precision"] if rule.metrics_holdout else None
    print(rule.text, rule.confidence, test, rule.holds_up)

4. Grade trust & build a card

from parleml_mli import compute_trust_summary
from parleml_mli.autodoc import build_model_card

trust = compute_trust_summary(result)
for tile in trust.tiles:
    print(f"{tile.key:20s} {tile.badge}")

card = build_model_card(result, trust_summary=trust)

5. Save

result.save("out/fraud_profiling.joblib")

6. Score a single new transaction

new_row = X_test.iloc[[0]]               # a one-row DataFrame
ce = result.justify(
    x_raw=new_row,
    top_k_shap=10, top_k_neighbors=5,
    include_shap=True, include_rules=True, include_neighbors=True,
)
print(ce.prediction, ce.matched_archetype)
for d in ce.shap_drivers:
    print(d)

# Scoring many rows? Bind once and reuse:
justifier = result.justifier()
for i in range(len(X_test)):
    ce = justifier.justify(X_test.iloc[[i]])

See the decision-support guide to add ICE, LOCO and counterfactual sensitivity to the per-case payload.

Next