Workflow (multi-agent) resume restarts from the beginning after auto-compaction — silently re-runs completed agents

8872a6ad-409f-420d-80f7-d976b7290f09

The session-directory coupling you've identified is the load-bearing root cause here. The run journal living under the session directory (rather than the workflow run ID directory) makes auto-compaction a silent cache invalidation — and the user has no way to know a resume will cold-start rather than replay.

A few observations from running a similar external polling orchestrator that dispatches long-running Claude Code sessions:

The cross-session journal lookup is the fix that matters most. resumeFromRunId should resolve the run journal by run ID alone, independently of the session tree. The session is a container for execution; the run record is what needs durability across compactions.

The zero-warning cold-start is the UX failure. Even before a fix lands, surfacing a warning like "run journal not found in current session — this is a full restart, not a resume" would save users from the silent duplicate-work trap. The current behavior (silently re-runs from Phase 1 with no indication) is worse than a hard error.

Workaround until fixed: explicitly copy journal.jsonl from the pre-compaction session directory into the new session directory before resuming. Not ergonomic, but it works if you catch the compaction before resuming.

This one's going to hit anyone running multi-phase workflows unattended overnight — the compaction window is exactly when long-running jobs are most likely to trigger it. Worth a workflow-resume label if that exists.

This is a really clean root-cause analysis — journal scoped to session directory rather than run ID means any compaction-created session boundary is an invisible cache-busting event. The workaround you documented (reading the prior journal.jsonl manually) confirms the data was fully intact; the lookup path just pointed at the wrong session.

The right fix is obviously Anthropic resolving journals by runId project-wide. In the meantime, one approach that sidesteps the trigger: cozempic's guard daemon (github.com/Ruya-AI/cozempic) monitors the session's token count and prunes the JSONL before it reaches the auto-compact threshold. If the session never compacts, the session ID never changes, and resumeFromRunId keeps finding its journal in the same directory where the workflow started.

To be clear — cozempic doesn't fix the journal-lookup logic; it eliminates the condition (uncontrolled session growth → compaction → new session ID) that makes the lookup fail. For long research→design→prototype pipelines, keeping the underlying session from tipping over compaction is often the more practical lever than hoping the framework tracks journal moves.

pip install cozempic or pipx install cozempic, auto-registers the guard on install. Curious whether this pattern is useful for your setup, or if you're hitting the compaction boundary in sessions short enough that pruning wouldn't have helped in time.

I hit this exact failure mode independently and can confirm the root cause you've pinned: the resume journal is scoped to the session directory, so any session-ID change (auto-compaction here, but /clear or a crash do it too) turns a valid resumeFromRunId into a silent cold-start.

Two things I can add from instrumenting it:

1. **The consolidated wf_<runId>.json journal only materializes when the run completes.** Mid-run there's only the incremental journal.jsonl plus the per-agent transcripts — the clean journal isn't written until a terminal status. So a run that was paused or aborted (exactly the state you resume from) has no consolidated journal at all, which compounds this: a cross-session resolver has to fall back to journal.jsonl + transcripts, not the tidy journal.

1. Resolving by runId is tractable — the run directory name already is the run ID, so the journal can be located independently of the session that created it. I built a small open diagnostic that does exactly this read and, on a cross-session read, prints the originating sessionId next to a warning that resume will re-run everything rather than replay — i.e. it turns today's silent cold-start into the loud signal point 2 of "What Should Happen" asks for: https://github.com/home-dev-lab/workflow-toolbox (the journal-on-completion and same-session-only behaviours are written up in docs/public/known-issues.md). Sharing it as proof the by-runId lookup is straightforward, not as a fix for the resume path itself.

For me the "fail loudly on a 0% prefix hit" half is the higher priority: the silent restart is what makes this expensive, and a one-line warning before spending tokens would defuse it even before the cross-session lookup lands.

The journal-materialization timing is a detail worth pinning — the consolidated file only existing on RUN COMPLETE means any cross-session resolver has to reconstruct from journal.jsonl + per-agent transcripts for the exact cases where resume matters most (aborted or paused mid-run). Good to have that confirmed explicitly.

Your fail loudly before spending tokens priority is right. A warm-start that silently re-runs all prior agents is maximally expensive — it costs exactly as much as the original run, with no signal until you notice duplicate output. A one-message warning at resume time ("session boundary detected, this will cold-start") would change it from expensive-surprise to recoverable-surprise and costs nearly nothing to emit.

The by-runId resolution is the right long-term fix; the loud signal is the right near-term one. In the meantime, preventing the trigger (keeping the JSONL below the auto-compact threshold via pruning) sidesteps the session-boundary problem entirely for normal operation — but it won't help on crash or deliberate /clear, where the boundary isn't avoidable.

The journal.jsonl-loses-session-ID-after-compaction problem is the kind of failure mode that is hard to anticipate until it bites you: you designed the workflow to be resumable, the journal is right there on disk, but the ID mismatch makes it invisible to the resume path.

The manual workaround you found (read the prior session journal directly and continue from cached results) is correct but brittle -- it requires knowing which journal to read and re-threading the results yourself.

A few things that might help until there is a native fix:

1. Name the run ID explicitly in the Workflow call and store it in the journal header so resume can match on a stable key rather than the session ID. If resumeFromRunId matched on a workflow-level ID rather than a session-level ID, compaction would not break the lookup.

1. Before any long pause, snapshot the current phase completion state to a separate file (e.g. workflow-checkpoint.json in the project root). The overhead is tiny and it gives you a recovery point that survives compaction and session restarts without needing the journal path.

1. Watch the Claude Code 2.1.167+ release notes for session-ID stability improvements -- this is the same underlying issue as the worktree-relocated crash-loop in #65216.

The root cause -- auto-compaction creating a new session ID and orphaning the journal -- is a missing invariant in the resume contract. The workflow was designed assuming session IDs are stable across compaction, which they currently are not.

@kcarriedo 'missing invariant in the resume contract' is the sharpest framing of this I've seen — the workflow was designed assuming session ID stability across compaction, which is currently a guarantee that doesn't exist.

Your checkpoint file approach (point 2) is the right pattern at the workflow level. Cozempic's guard does the analogous thing at the session level: it writes a pre-compaction checkpoint of session state before pruning fires, so the session can recover without relying on session-ID continuity. The principle transfers directly to workflows — an explicit phase-completion snapshot at a stable path survives compaction and session restarts without depending on the journal path.

The invariant the real fix needs: resumeFromRunId resolves against a workflow ID, not a session ID. Until then the checkpoint pattern is the reliable fallback regardless of what tool you're using.

The 'missing invariant in the resume contract' framing from earlier in this thread is the right anchor. Adding one concrete edge case from our runner that compounds the cold-start risk:

When a workflow is paused at a phase boundary (not a clean completion), the phase-completion checkpoint approach works -- but only if the checkpoint write is synchronous before compaction fires. In our polling runner, we learned this the hard way: the checkpoint was written as a PostToolUse side effect, which fires after the tool response is returned to the session. If compaction triggers between the tool response and the next user turn, the checkpoint exists but may not include the last phase's terminal marker, so the resume reads "phase 3 incomplete" and re-runs it.

The fix we settled on: write the phase checkpoint as a PreToolUse hook on the next phase's first tool call rather than PostToolUse on the prior phase's last one. By the time Phase 4 starts its first Read, Phase 3's completion is durably on disk. Pre-compaction = safe.

Two things that would make this pattern unnecessary at the platform level:

1. A resumeFromRunId that resolves by workflow ID independent of session tree (already called out well in this thread).
2. A pre-compaction hook event that lets a workflow write its own durable checkpoint before the session ID changes. Even a simple "compaction is about to happen, here is the current run state as JSON" notification would let the workflow author decide what to checkpoint.

Neither is a workaround -- they are the right contract. The checkpoint-in-PreToolUse trick is just the closest approximation available today.

Hit this exact pattern when building a long-running workflow runner. The cross-session journal lookup failure after auto-compaction is particularly painful because compaction is silent and the resume call gives no indication it started over -- you only notice when you see agents re-fetching data you already have, or when you check the journal and see phase 1 agents re-running.

The fix that worked for us was storing run journals by run ID in a location outside the session directory -- we use a flat state directory keyed by run ID rather than session ID. That way resumeFromRunId can find the journal regardless of what happened to the session. Essentially: decouple journal storage from session lifecycle.

Until that's a first-class option in the Workflow tool, the practical workaround is to disable auto-compaction for sessions that own active workflow runs (set maxContextWindowUsage to a higher threshold or use a fresh session for each workflow where you control the compaction schedule).

The silent failure on cache miss is the worst part. If the resume call could loudly error with "journal not found for runId X, full restart required -- abort and confirm?" before burning tokens, that would prevent the expensive surprise.

Hit this on a long-running orchestration workflow last week. We run multi-phase agent pipelines where each phase can take 20-40 minutes, and resuming after auto-compaction is effectively broken for us right now.

The silent restart is the worst part. There is no indication anything went wrong -- the UI looks normal, the workflow appears to be running, and you only notice the cost doubling when you check the session metrics afterward. We caught it because our orchestrator logs phase completion events to a file, so we saw Phase 1 log entries appearing a second time.

The cross-session run ID resolution would fix it. Alternatively, even a startup check that logs "resumeFromRunId=XYZ not found in current session, searching project..." would be an improvement over silent failure. Right now the behavior is indistinguishable from a normal resume when it is actually restarting everything.

One workaround that partly helps: checkpoint the journal path explicitly in your workflow script and pass it as an arg on resume rather than relying on auto-discovery. Fragile, but it at least survives compaction if you do it right before the pause.

Confirming this on 2.1.220, with what the runtime is doing underneath, in case it helps narrow the fix.

The journal path is built from the current session id

function Ste(e) {                                     // e = the runId
  let t = dW() ?? tS(gn());                           // project dir
  return join(t, kt(), "subagents", "workflows", e);  // kt() = current session id
}
// journal:  join(Ste(runId), "journal.jsonl")

Which gives the on-disk layout:

~/.claude/projects/<project>/<SESSION-UUID>/subagents/workflows/<wf_runId>/journal.jsonl
                             ^^^^^^^^^^^^^^ session id, sitting above the run id

The runId is a leaf, and the session id is a parent segment resolved fresh at resume time. Auto-compaction changes kt(), so the resume looks under a session directory that has never contained that run, finds nothing, and starts over. That lines up exactly with what you reported.

For scale, on one machine here: 516 journals across 39 session directories.

Steps to reproduce

1. Start a multi-phase workflow with Workflow({ script }). Note the runId in the tool result.
2. Let it finish a couple of phases, then stop it.
3. Get the parent session to auto-compact. A long conversation does it; the session id changes.
4. Resume with Workflow({ scriptPath, resumeFromRunId: "wf_..." }).
5. Observe a cold start from phase 1.
6. Then go and find the journal, intact, under the old session UUID:

find ~/.claude/projects -type d -name 'wf_<runId>'

Two other things from the binary that bear on the fix

The cache-miss flag is sticky. Once one lookup misses, no later lookup is consulted for the rest of the run:

let Ze = v ? undefined : l?.results.get(ge);
if (Ze !== undefined) return cached(Ze);
v = true;

So an orphaned journal isn't a degraded resume, it's a fully cold one. Every agent re-runs, including any whose key would still have matched.

The keys are a sha256 chain, not a positional index. That resolves the ambiguity in #63102:

function zSd(e, t, r) {              // (prompt, opts, prevKey)
  let n = createHash("sha256")
    .update(r).update("\x00")        // previous key
    .update(e).update("\x00")        // prompt bytes
    .update(Jq_(t)).digest("hex");   // normalised opts
  return `v2:${n}`;
}
// call site:  ge = zSd(promptStr, opts, T);  T = ge;

Jq_() normalises only {schema, model, effort, isolation, agentType} with keys sorted, so label and phase don't participate. Relabelling or re-grouping agents doesn't invalidate anything, which is useful to know independently of this bug.

A workaround that works today

Copy the orphaned run directory into the tree of the session you're resuming from:

cp -R <old-session>/subagents/workflows/wf_<runId> \
      <current-session>/subagents/workflows/

Since the path resolves as (project, current session, runId), that's enough for the lookup to find it. Copy rather than move, so the original stays put if you need it again.

The two fixes worth making

1. Resolve journals by run id across the project, rather than under the session directory. Third-party ports of this API have independently re-keyed the journal to a project-scoped location, which suggests that's the natural shape rather than a workaround.

1. Fail loudly on a 0% prefix hit. This is the one that would actually have saved the tokens. A resume that finds no cached results at all is nearly always a bug rather than an intention, and right now it's indistinguishable from a fresh run until the usage shows up. Reporting the expected hit rate before spending anything would make it obvious.

I've got a pile of orphaned runs here from this exact failure, so I'm happy to test a fix against them if that's useful.