Skip to content

Demo: deliver a fraud model

In short

  • The whole delivery loop on 6,000 sample insurance claims, about 18 % fraudulent.
  • The agent frames the problem, writes the training table, trains, iterates, challenges and proposes an operating point. You answer each decision.
  • Promoting the model stays your click on its page.

A walk through the whole loop on the sample claims data, with Claude Code as the main agent. Claude Code comes with the extended image (Which agents can run); on the standard image, Codex runs the same loop. The claims table is a good stage: a leak hides in a prose column, a tag list mixes intake tags with investigators' tags, a text column carries signal, the labels come only from claims an investigator looked at, and time matters.

Before:

  • the backend started with PARLEML_PUBLIC_URL=http://localhost:5173 (the web app's origin in local dev), so the links the agent hands out open the app, not the API (Configuring agents);
  • a workspace with sample_datasets/tabular/insurance_claims_fraud.parquet registered as a dataset (6,000 claims, about 18 % fraudulent);

  • the standard profile and Claude Code signed in. Optionally pin the challenge to Codex (skill_agents: {ml-challenge: {adapter: codex-cli}}) for a second model's opinion.

  • Start. On the dataset, ✦ Agent → Deliver a model, with the goal in your words: "flag likely fraudulent claims when they are reported, so the investigations unit reviews them before payout; it can review about 10 %." The session profiles the data first (an EDA that outlasts a tool call is awaited, not slept on).

  • Frame. ml-frame proposes the target (fraud, positive fraudulent), the prediction moment (the claim's report date), the metric (average precision, for a rare positive) and asks what it cannot know: is a decision explained to anyone, is scoring batch or real-time, are tags assigned at intake or by investigators, how are labels made. It proposes one template for those requirements and the table's shape — for 6,000 rows, foundation_ensemble where a foundation model can run, else small_data. The framing gate asks each open question with its own answer field and the template as a choice, the proposal first.
  • Data. ml-data writes the recipe SQL — dropping adjuster_notes (written after the investigation) and the investigator-only tags — maps loss_tags as a tag list and incident_desc as prose, triages the leakage findings and runs the signal check. Confirm the triage on the Leakage tab it links, answer its open questions and approve the training table.
  • Baseline. ml-baseline trains the approved template on a time split — a job card follows it live, the experiment hangs under the step on the flow map. If a top driver (say repair_shop) might be assigned after the claim is reported, the agent asks you before it iterates.
  • Iterate. ml-iterate first sweeps the baseline's explanations — the rules that hold up on Test, archetypes, the error tree, engineered features, ALE, the surrogate, text terms, label noise — into a short table of gems. Each experiment then tests one idea with a mechanism, evidence and an expected effect, on one lever within the template, compared with the floor and the parent's noise; the candidate is explained in depth before the challenge. The experiment tree (Overview → Experiments) grows with each hypothesis on its edge.
  • Challenge. ml-challenge reviews the best candidate as a sceptic — driver timing, stability over time, label selection, the requirements — and ends Verdict: pass with conditions, each condition with its fallback.
  • Decide. ml-decide simulates operating points at the team's capacity; the release gate offers them (recommended first) and asks each condition: confirm or cannot confirm (and take its fallback).
  • Release. Commit the chosen threshold on the experiment's Calibration tab when the release gate asks, and approve the release; ml-release registers the model with its release record — the operating point it binds, your answer to each condition, the limitations — and rebuilds the AutoDoc from it. Promoting the model on its page stays your click.

Along the way: queue a message during a long training wait and watch the agent acknowledge it. Publish the challenge report to the workspace's Documents: a block verdict shows Blocked by the challenge on the experiment page (opening the report), and a later passing report clears it. The experiment page also says what each experiment was derived from, the lever and the hypothesis, and states "Registered · In production since …" once its model is in production. If a session hits a subscription limit, the chip says "Paused · provider limit, resets …"; turn on Resume automatically and it continues at the reset.

Next