[FEATURE] Distributed Agent Teams: coordinate Claude Code instances across machines (peer over LAN)

The cross-machine coordination gap is real and hits a different class of problem than same-machine agent teams. A few patterns people use today to approximate it:

1. Shared git repo as the coordination layer: each machine's Claude Code session commits state checkpoints to a branch, the "coordinator" machine runs a polling loop that reads other branches and routes work. Works but adds latency proportional to your commit-and-fetch cycle time (seconds to minutes).

1. Shared filesystem over NFS or SSHFS: the local mailbox at ~/.claude/teams/ can technically be placed on a network share. It works until you hit file-locking races, which happen frequently with concurrent writes from multiple machines.

1. External message bus: write a small PostToolUse hook that publishes results to Redis or a lightweight broker, and have the receiving Claude Code sessions subscribe. This works reliably but requires running and maintaining the broker.

None of these have the latency or reliability of a native LAN peer protocol. The GPU/large-RAM use case you describe -- one machine for heavy lifting, another holding the data -- is a natural fit that the current architecture can't serve cleanly.

One design question worth raising in this issue: should the peer protocol be point-to-point (machine A talks to machine B directly) or mediated (a coordinator machine routes messages to peers)? The mediated model is easier to implement and debug but adds a single point of failure. The point-to-point model is harder to implement but maps better to the "different machines with different capabilities" framing.

Disclosure: I'm building Claudeverse, which addresses some of the fleet visibility and coordination gaps around multi-session Claude Code. This issue is at the edge of that scope (cross-machine is beyond what we handle today) but worth tracking.

Distributed Agent Teams — Complete Architecture Proposal

Making the Agent Teams mailbox transport-independent, with a transactional task lifecycle, recovery, and Zero-Trust security

Nature of this proposal: this is not a new distributed-agent framework. It is an architectural evolution of the existing Agent Teams feature: keep the coordination model (task queue, mailbox, ownership) exactly as it is today, and make the transport layer pluggable so instances on different machines can cooperate. Authorship: co-designed by the practitioner running a real-world deployment (the ENS government document-processing pipeline) together with Claude. The design was derived from operational needs, using a restaurant-kitchen analogy as the reasoning tool. _Co-author: Andres Felipe._ https://claude.ai/code/artifact/efffbad4-4afc-4800-99a3-5788ec025281

---

1. Executive summary

Today Agent Teams coordinate multiple agents on the same machine via a local mailbox
(~/.claude/teams/). Real deployments run several machines on the same LAN with different
capabilities (GPU / large-RAM nodes, data-holding nodes, orchestration nodes). We need one
instance to delegate work to another machine, monitor it, and keep working — while the peer
executes and reports back.

This document proposes:

1. A pluggable Team Transport (local / shared-folder / LAN / SSH / A2A / cloud).
2. A 7-stage transactional task lifecycle (Quote → Approve → Execute → Progress →

Review → Settle → Receipt) with unique id and explicit owner.

1. 8 robustness layers (R1–R8) + a capability card + completeness extras

(time sync, versioning, fair scheduling) that make it production-grade.

Everything reuses the existing Agent Teams coordination logic; only the transport changes.

---

2. The pluggable transport (the core idea)

   Agent Teams coordination logic  (UNCHANGED)
                 │
          Team Transport (contract: send, receive, claim, ack, heartbeat)
                 │
   ┌──────────┬──────────┬────────┬───────┬───────┬────────┐
 Local     Shared      LAN      SSH     A2A    Cloud   (future)
mailbox    folder     peer
 (today)   (v1)

The programming model never changes. Whether tickets travel by paper rail, a printer, or a
screen, the kitchen works the same way. LocalMailboxTransport stays the default;
distributed transports are opt-in.

---

3. Kitchen analogy legend (the explanatory thread)

| Kitchen role | System component |
|---|---|
| Host / Maître d' | Admission + peer identity |
| Head chef / Expediter (the "pass") | Coordinator control plane (leader) |
| Stations / Cooks | Workers, each with a specialty (GPU, OCR, vision) |
| The runner / messenger | The transport carrying the ticket |
| The ticket / comanda | The task, with its lifecycle |
| Kitchen Display System (KDS) | Central observability dashboard |
| The pantry | Shared data store (network drive) |
| The closed check / POS receipt | Settlement record |
| Sentinels / kitchen cameras | Runtime monitors + anomaly detection |

---

3.1 Diagrams

Deployment overview

flowchart TB
    subgraph Control["Control plane (leader-elected + quorum)"]
        L["Head chef / Coordinator<br/>(active leader)"]
        S1["Standby chef"]
        KDS["KDS dashboard + sentinels (UEBA)"]
    end
    subgraph Transport["Team Transport (pluggable)"]
        T["Local | Shared folder | LAN | SSH | A2A | Cloud"]
    end
    subgraph Workers["Worker stations (routed by capability)"]
        W1["OCR node<br/>(Chandra 5B)"]
        W2["GPU node<br/>(vision)"]
        W3["... more nodes"]
    end
    Pantry[("Shared pantry / data<br/>network drive")]
    L -->|quote / approve / tasks| T
    T -->|claim / ack / heartbeat / results| W1
    T --> W2
    T --> W3
    W1 -->|read data / write results| Pantry
    W2 --> Pantry
    W3 --> Pantry
    W1 -->|progress / receipts| KDS
    W2 --> KDS
    W3 --> KDS
    L -. leader lease .- S1

Task lifecycle (7 stages + retry / reject / dead-letter)

stateDiagram-v2
    [*] --> Quote
    Quote --> Approve: capacity + load OK
    Quote --> [*]: declined (no capacity/capability)
    Approve --> Execute: identity OK + bound to 1 worker
    Execute --> Progress: atomic claim + lease + ACK
    Progress --> Progress: heartbeat / checkpoint
    Progress --> Execute: lease expired → watchdog reassign
    Progress --> Review: work done
    Review --> Settle: accepted
    Review --> Approve: rejected → re-queue
    Settle --> Receipt: atomic commit + release
    Receipt --> [*]: done (dedup marker)
    Progress --> Error: failure
    Error --> Execute: retry (backoff, other node)
    Error --> DeadLetter: limit reached (86 corner)
    DeadLetter --> [*]: poison / not-completed (notify)

4. The 7-stage task lifecycle

Each task has a unique id and an explicit owner, and moves through:
quote → approve → execute → progress → review → settle → receipt (or error).

4.1 Quote

• Problem: dispatching blind sends work to a machine that is overloaded, lacks the

model, or has no GPU → the task is rejected, queued forever, or crashes the node. No time
estimate, no capacity reservation.

• Solution: a capacity + load handshake. Each worker publishes its load (jobs in

flight) and its capability card; the coordinator requests a quote; the worker replies
accept + ETA or decline with reason, and a slot is reserved.

• Kitchen: the waiter asks the kitchen "can you do the risotto tonight, and how long?"

before promising the customer. "Yes, 20 min" or "86'd, out of that."

• Tech: admission/scheduling handshake; capability advertisement; capacity reservation.

4.2 Approve

• Problem: who authorizes this task to that worker? Without it, any LAN machine

could inject or steal work, or a task is double-dispatched.

• Solution: a gateway/guard validates identity (approved fingerprint) +

availability, and binds the task to a single worker.

• Kitchen: the head chef puts the ticket on the rail; the waiter cannot walk in and

cook. Only badged staff enter the kitchen.

• Tech: authentication + authorization gate; explicit approval; single-assignment.

4.3 Execute

• Problem: exactly-once ownership. If two cooks grab the same ticket → two plates

(duplication, waste, conflicting results). If nobody grabs it → abandoned.

• Solution: atomic claim + lease. The worker "calls" the ticket: moves it to

in-progress with owner=host and takes a lease with expiry. It must send an ACK
and change state. An idempotency key prevents a redelivery from re-executing.

• Kitchen: the cook shouts "I've got the risotto!" — now it's his; the expediter marks

it in progress. The label on plate+cook is a DHCP-style lease: it expires, renews, or
is released.

• Tech: atomic claim (compare-and-set / atomic rename); time-bounded lease; idempotency

key.

4.4 Progress

• Problem: visibility + liveness. The coordinator must not block waiting, but must know

the worker is alive. If the cook collapses mid-cook, the plate never comes out and the
customer waits forever (lost work).

• Solution: heartbeat + progress events (n/total, ETA) written to a **shared,

persistent board (never the dying agent's local memory) + lease renewal. If the
heartbeat stops, the lease expires and a watchdog reassigns the task to the
least-loaded worker. A checkpoint** (checklist) on the shared board lets the replacement
resume from where it stopped, not from zero.

• Kitchen: the expediter calls "how long on table 5?"; if no answer, something's wrong →

reassign. The progress is written on the shared whiteboard, like a ride-share app keeps
the trip state on its server, not on the driver's phone.

• Tech: heartbeat; dead-man's-switch lease; watchdog failover; **shared-persisted

checkpoint** for resumability.

4.5 Review

• Problem: quality gate before serving. A bad, incomplete, or conflicting result must

not reach the customer.

• Solution: the worker submits artifacts and marks review; the coordinator/inspector

validates (schema, tests, content). Reject → back to the queue.

• Kitchen: the head chef inspects the plate at the pass (smell, texture); if wrong,

"re-fire!".

• Tech: output validation gate; accept/reject with re-queue.

4.6 Settle

• Problem: deliver exactly once, close out, free resources, and resolve conflicts —

atomically. The dish isn't done until it reaches the table, the ticket is closed, and the
station is freed. A half-delivered result (network cut mid-copy) must not exist.

• Solution: ACID, ETL/parquet-style: all-or-nothing — write to a staging area, then

one atomic commit flips it to official. One dish = one cook + ACK; a reconciler
continuously enforces "1 ticket = 1 owner". Winner rule: the result from the current
valid lease holder wins; results from an expired lease are discarded. The worker
releases the ticket (like a DHCP release) and becomes free. On delivery failure
(ride-share analogy) the driver/passenger notify and the order is re-dispatched.

• Kitchen: the last 3 meters from the pass to the table — the plate is served, the ticket

leaves the rail, the station is freed, the check is closed.

• Tech: staging + atomic commit; invariant reconciliation; deterministic conflict

resolution; resource release.

4.7 Receipt

• Problem: audit, recovery, and idempotency. Without an immutable record you cannot audit

who did what, recover from a crash, or avoid re-doing completed work.

• Solution: an append-only receipt (id, owner, input hash, artifacts, timestamps,

approvals). The receipt is the LAST step and is the "closed" marker: no receipt =
order still open = safe to retry; receipt exists = do not redo (dedup).

• Kitchen: the closed check in the POS / the printed ticket: auditable, and never charged

twice.

• Tech: immutable event log; commit marker; idempotent replay.

---

5. Robustness layers (R1–R8)

| # | Layer | Problem | Solution (kitchen + tech) |
|---|---|---|---|
| R1 | Crash recovery | A cook dies mid-cook; work is lost | Shared-persisted checkpoint + watchdog reassigns; resume from the checklist, not from zero |
| R2 | Routing + backpressure | Overloading a station; sending a dish nobody can make | The guard/gateway routes by load and enforces queue limits |
| R3 | Idempotency / exactly-once | A reprinted ticket cooked twice | Receipt-based dedup (if a receipt for #123 exists, don't redo) |
| R4 | Dependencies / ordering | "OCR before classify"; courses out of order; incomplete assembly | Checklist per dish; coursing by estimated duration (fire-timing: start the 20-min dish later so both plate hot together); fan-out (split a complex dish, each cook owns a subtask and confirms); shared prep (a sauce station, reusable + cacheable). Cycles: the head chef rejects circular recipes (A→B→A) at admission (acyclic graph). Fan-in: "assemble" waits for all N receipts (barrier). Cascade on failure: dependents are held; use a backup / plan B-C (a cached sauce with a prior receipt) or remake within the retry limit; if it hits dead-letter, cancel downstream and the customer decides keep-partial or cancel |
| Capability card | Skills routing | A cook doing the wrong cuisine | Cook taxonomy (Italian cook ↔ Italian dish, Thai ↔ Thai); affinity (same dish → same cook); shared roles (sauce makers) |
| R5 | Deep security (Zero Trust / Palo Alto) | An authenticated cook can still do harm | 1 Least privilege: the sauce cook reaches only his fridge/station; permissions only toward his destination; master→node model (a node only does what the master assigns); sentinels watch movement at runtime — off-route → alert/evict (RBAC + policy enforcement + egress control, "verify at every hop"). 2 Data locality: the secret recipe is encrypted + access-controlled + masked; not copyable; held in custody by the head chef (with a vetted successor / break-glass); every plate is labeled and follows only an authorized route (DLP). 3 Encryption in transit: the plate/ticket travels covered (encrypted + signed), on an authorized route; mutual authentication blocks man-in-the-middle. 4 Microsegmentation + default-deny: the kitchen is closed/segmented (the public dining room may be open); zones are subnets/VLANs + security groups; doors use vision/fingerprint (biometrics) that allow or deny by the cook's role in that zone (default-deny); sentinels monitor entrances (no one is trapped; lateral movement is detected). 5 Impostor/anomaly: entry via id + password + MFA; the master chef knows every cook; content is validated by the inspector (Review). 5b Behavior (UEBA): if a chef makes a strange recipe, the sentinel checks who and their pattern (baseline + anomaly detection) |
| R6 | Observability (KDS) | The head chef is blind to the whole service | A central Kitchen Display: dish states, completed, available runners (transport), available cooks (capacity), same dish ordered 2+ times (duplicate/idempotency visibility), and KPIs to quantify effort (throughput, per-stage times, per-cook load, rejections). Camera-like sentinels watch state + actions and alert the head chef on anomalies (tied to R5 UEBA). End-to-end trace per order (Quote→Receipt) via state history + receipts |
| R7 | Poison / retries + pre-validation | A dish that always burns; bad input propagated | Failure taxonomy: oven (infra) fails → reschedule the same dish on another oven/node (failover; doesn't count against the poison limit); sauce/data fails → regenerate the artifact (don't propagate bad data). Validate BEFORE / during: each step validates its inputs + resource first (check the sauce before saucing; check the oven before baking) = preconditions/guards; Review validates AFTER, this validates BEFORE; on in-flight failure, redo what's possible and return to the last good state (checkpoint). Retries: max 3 (configurable) with exponential backoff (1→5→15 min). Per-dish timeout/SLA. Dead-letter ("86 corner") with an owner + full traceability (reason, who, attempts, times); labeled 🔴 poison (always fails, fix recipe/data) vs 🟡 not completed (timeout/resources, retryable). Notify the user; for complex dishes, give guidance BEFORE starting |
| R8 | Single expediter (anti split-brain) | Two head chefs calling the same rail = chaos | Leader election (Kubernetes/OpenShift/Spark style): several candidate chefs, but one leads (active) and announces it; others on standby. Leadership is a renewable lease (same as Execute); if the leader dies, re-election hands the baton to a standby. Anti split-brain: majority quorum — only the side with >50% of chefs can elect/keep a leader; the minority stops (no rogue cooking). Odd number of masters (3 or 5) |

---

6. Completeness extras

| # | Extra | Problem | Solution |
|---|---|---|---|
| E1 | Synchronized clocks | Leases/timers (Execute lease, R7 backoff, R8 baton) depend on the clock; drift expires them wrong | NTP on all kitchens; sentinels validate every clock shows the same time |
| E2 | Single recipe edition (versioning) | One cook uses the old recipe book, another the new → the "same" dish differs | Identical recipes/manuals for everyone; the head chef validates recipes; if a cook improvises / uses another version, UEBA detects different elements = version drift/tampering |
| E3 | Quota + fair priority (scheduler) | One big table monopolizes the kitchen and exhausts resources (this happened to us: we exhausted our usage quota) | Rule 0: if they don't compete for the same station → run both in parallel (calamari at the sauce station, fries at the fryer); fire-timing (R4) starts the slow one early. If they compete for the same scarce resource: weighted priority = shortest-job-first (SJF) + impact (people waiting) + aging (the complex dish gains priority the longer it waits — no starvation) + value/SLA (VIP/premium). Per-table quota: nobody hogs the kitchen. = Fair/Capacity Scheduler, YARN/K8s/Spark style |

---

7. Data model (draft)

• Task (ticket): id, type, required_capabilities[], prerequisites[],

state (quote/approve/execute/progress/review/settle/receipt/error), owner,
lease_expires_at, checklist[], estimated_min, attempts, deadline.

• Receipt: id, owner, inputs_hash, artifacts[], timestamps, approvals,

outcome.

• Approved peer: host, ip, fingerprint (SHA256), capabilities[], approved.

---

8. Why this fits Agent Teams (backward compatible)

Agent Teams already provide the task queue, mailbox, coordination, and ownership. This
proposal only replaces the transport layer and adds the lifecycle/robustness semantics
around it. The local filesystem stays the default implementation; distributed transports
(shared folder, LAN, SSH, A2A, cloud) become optional. A2A is not a competitor — it is one
possible transport implementation, giving interoperability with external agents while
preserving the Agent Teams programming model.

The value is not two Claude instances talking. It is cooperative execution across
heterogeneous machines while preserving security, explicit approvals, deterministic task
ownership, recoverability, and compatibility with the existing Agent Teams model.

---

9. Technology mapping (for technical grounding)

| Concept in this design | Established technology |
|---|---|
| Pluggable transport | Message-transport abstraction; A2A as one transport |
| Lease on Execute / leadership | Distributed lease/lock; etcd/ZooKeeper leases |
| Watchdog + reassignment | Kubernetes controller reconcile loops |
| Leader election + quorum (R8) | Kubernetes/OpenShift control plane, Spark HA, Raft/Paxos |
| Idempotency via receipt | Exactly-once processing; dedup keys |
| Settle all-or-nothing | ACID transactions / parquet-style staged commit |
| Zero-Trust security (R5) | Palo Alto: least privilege, microsegmentation, App-ID, Threat Prevention, default-deny; RBAC, mTLS, DLP, data masking, MFA, UEBA |
| Fair scheduling (E3) | YARN/Kubernetes/Spark Fair & Capacity Schedulers; SJF + aging |
| Clock sync (E1) | NTP |
| Versioning (E2) | Artifact/version pinning |

---

_End of proposal._

VOLY implements this with A2A federation — network-based agent coordination, not file-based mailbox.

A central orchestrator decomposes a task into roles (architect, developer, tester, reviewer, devops) and dispatches them to remote agent workers via Cloudflare Workers + service bindings. Each worker runs an independent claude-code executor on its own machine. Results are collected back and merged into a single report.

Transport: HTTP + async callbacks, so agents can be on separate machines with different capabilities (GPU, RAM, project context).

https://github.com/voly-codes/voly/blob/main/docs/backend/a2a.md#a2a-federation

Are you mixing claude-code with other agents (cursor, codex) or coordinating same-model instances?