Skip to content

Recovery & resume

In short

  • A session is a tree: the orchestrator and the steps it starts. Each run keeps its own state on disk.
  • A restart never loses a session. It repairs what a crash cut off and never restarts agents by itself: working sessions show Process lost until someone resumes them.
  • The page offers the one action that fits each state: answer, resume, retry or continue.
  • A provider limit pauses the session; Resume automatically continues it after the reset.

A session is a tree: the orchestrator (the root run) and its steps (child runs). Each run keeps its own state; recovery reads that state, checks whether the run's process is alive, and offers the one action that fits.

What is authoritative

File (under runs/<root_id>/) Role
run.json (one per run; steps under children/<run_id>/) The current state: status, attempt, the open question and its answer, the outcome, the failure kind, the CLI conversation id, the pending deferred operation.
events.jsonl (one per session) The write-ahead journal and audit timeline. Every change is appended (and synced) here first, then run.json is replaced atomically.
transcript.jsonl (one per run) What the agent said and did, for display. Not used for recovery.

The stored statuses are running, paused (a question is open), stopped, completed, failed and cancelled. "Lost process" (orphaned) is not stored: it is derived when a running or paused run has no live process, so a crash leaves nothing stale to clean up.

Crash repair at startup

When the backend starts, the runtime:

  1. takes runs/.runtime.lock (a second process on the same directory fails to start);
  2. removes stale import previews;
  3. for every session, cuts off a torn last line of events.jsonl (an append interrupted mid-line, never durable) and replays any events newer than run.json, so a crash between the two writes loses nothing;
  4. marks a step that never recorded its start as failed with interrupted_start;
  5. starts the auto-resume sweep and the daily pruning of expired tombstones, whose first pass runs at once.

It does not restart agents. Sessions that were working show Process lost until someone resumes them; open questions stay answerable. Shutting the backend down stops the CLI processes without cancelling anything.

What the session offers

State Process continuation Button
question open any answer_gate · decision_required Answer decision / Review decision
question answered alive none · answer_delivering — (the agent is reading it)
question answered gone resume · gate_answer_recorded Resume chain / Continue work
running alive none · host_live —
running gone resume · process_lost Resume chain / Continue work
stopped — resume · operator_stopped Resume chain / Continue work
failed gone retry · retryable_failure Retry run / Try again
failed step whose orchestrator has already finished — none · retry_blocked — (its result is final)
orchestrator completed, outcome not succeeded gone resume · blocked_outcome Continue run / Continue work
completed, cancelled — none · terminal — (a finished session offers Your turn)

The server computes this for every run (continuation on the run view); the page never guesses from colours or event order. Each button has a technical and a business label.

Resume, retry, continue and follow-up

All four are POST /api/agent/runs/{root_id}/runs/{run_id}/resume, with an optional message. The response's mode says which happened:

mode When What happens
children_then_root the orchestrator was stopped or lost its process Unfinished steps resume first, then the orchestrator, each in its saved CLI conversation.
exact_run a step is addressed Only that step resumes; its waiting orchestrator is not a blocker.
failed_run the addressed run failed (a cap, a provider limit, a login) Retry: a new attempt of that run only, with a fresh turn and time budget, in the same conversation. Its result goes to whoever was waiting for it.
continue_blocked the orchestrator finished blocked, partial or failed, or a person sends a follow-up to a finished session Continue: a new attempt of the orchestrator with its previous outcome as evidence. With a message on a successful session it is a follow-up (Your turn → Send): new work with the same context.

Resume is refused (409) when a question is still unanswered, a recovery is already running, the tree is still stopping, another run in it is alive (one work directory, one writer), or the profile changed. In the last case the page offers Continue with the current settings (force: true). A finished step cannot be reopened, and a successful session cannot be continued without a follow-up message.

Each resume or retry is a new attempt: the previous outcome is cleared and the agent must report its own, and the caps start again. If the CLI no longer has the saved conversation, the attempt fails with conversation_missing; Resume then starts a fresh conversation on the saved work directory, and the page warns about it first.

Failures

failure_kind Means Banner
provider_quota the subscription or API limit was hit Paused · provider limit, with the reset time when the CLI stated one
login_required the CLI has no usable login Paused · sign-in needed, with the command to run
runtime_limit the attempt hit its turn or wall-clock cap Attempt limit reached
conversation_missing the CLI lost the conversation The conversation was lost
missing_outcome the run ended without report_run_outcome The step ended without reporting an outcome
interrupted_start the backend stopped before the run started The start was interrupted
host_failure, execution_failure the CLI process failed, or the work did The agent host failed / The session failed

None of these is retried silently.

Provider limits and auto-resume

A provider limit (Claude's session limit, Codex's usage limit or spend cap, Gemini's quota) is not a ParleML failure. The runtime reads the reset time from the CLI's message: an absolute time with a named zone (resets 5pm (Europe/Berlin)), Codex's local try again at 11:08 PM, or a duration (retry in 3h12m). If it cannot read one confidently, the banner says the time was not stated rather than invent one.

Resume automatically when the limit resets (off by default; PUT /api/agent/runs/{root_id}/auto-resume) is offered only for a run paused on a provider limit with a stated reset. A sweep every 30 seconds resumes it 30 seconds after the reset, with a note that the resume was requested in advance. The mark belongs to that one failed attempt: any manual resume, a new failure, a cancel or a delete clears it, and a resume that fails is not repeated.

Stop, cancel, archive, delete

Routes are under /api/agent.

Action Route Effect
Stop POST /runs/{root_id}/stop Ends the CLI processes and keeps the session resumable (stopped). Jobs it launched keep running; open questions stay answerable.
Cancel POST /runs/{root_id}/cancel Final: closes open questions, settles every step, clears auto-resume. The history stays.
Archive PUT /runs/{root_id}/archive Hides a stopped or finished session from the in-flight lists; nothing is removed. Refused while it is running or waiting on a question. Resuming un-archives it.
Delete DELETE /runs/{root_id} (or POST /runs/delete for several) Removes the run tree (transcripts, uploads, events) and the work directory, leaving a tombstone. Only for a finished or cancelled session; a stopped one must be cancelled first. Refused (409 recovery_in_progress) while a resume is starting, and no resume starts while the delete runs. 403 when PARLEML_AGENTS_RUN_DELETE_ENABLED=false.

Who stopped or cancelled the session last is shown on the page (halt.json, the root's halt_kind, halted_by, halted_at) until the session is resumed; it is recorded before any process is halted. A halt is not a failure: stopped and cancelled runs carry no error, whatever the killed CLI printed on its way out, and the reason given with the request is kept in the run.stopped / run.cancelled event.

Deleting removes the conversation, not the work: experiments, recipes, models and published documents stay, and their provenance still names the session.

Deleting a workspace deletes its sessions with it. It is refused (409, with the session ids) while any of them is running, paused or stopped: cancel those first, exactly as a single session must be cancelled before its own delete.

Tombstones

A deleted session leaves tombstones/<root_id>.json: its title, skill, subject, status and outcome, its steps with their outcomes, who started it and who deleted it, and when. GET /api/agent/runs/tombstones/{root_id} returns it to anyone who may see the workspace, so a link to a deleted session still says what it was. Tombstones older than PARLEML_AGENTS_TOMBSTONE_RETENTION_DAYS (365) are removed when the backend starts and once a day after.

Next