[Bug] macOS: teammate spawn "fork failed: Device not configured" — xnu ptmx slot-vector race triggered by cat placeholder + respawn-pane -k (root cause for #77211)

Root cause for the recurring macOS "respawn pane failed: fork failed: Device not configured" on tmux teammate spawn (#77211 was closed by the stale bot without a diagnosis; this is the mechanism plus a deterministic repro and a backend fix). It is a kernel PTY-slot-vector race that the tmux teammate backend triggers deterministically at every 16-slot boundary.

Mechanism (xnu bsd/kern/tty_ptmx.c, apple-oss-distributions main)

Opening /dev/ptmx is two steps that are not atomic with respect to other closes:

1. ptmx_clone() (~L475–507) picks a minor number. If no free slot exists it returns pis_total — one past the end of the current vector.
2. ptmx_get_ioctl() (~L322–346) grows the vector by PTMX_GROW_VECTOR (16) only if pis_free == 0, then range-checks the minor (~L349–354) and fails with ptmx_get_ioctl failed because minor number N was out of range → ENXIO.

The vector never shrinks (ptmx_free_ioctl only bumps pis_free). So if a slot is freed between step 1 and step 2, step 1 has already chosen pis_total while step 2 sees pis_free == 1 and skips the grow → out-of-range → ENXIO. The source even has an XXX We fall off the end here comment in ptmx_clone.

Why the tmux backend hits it deterministically

TmuxBackend creates a teammate pane as: split-window -d … -- cat (placeholder occupies one PTY) → set-option -p remain-on-exit failed → respawn-pane -k -t <pane> -- <teammate cmd>. respawn-pane -k closes the placeholder's master (SIGHUP → cat exits → slot freed asynchronously) and immediately forkpty()s the replacement. When system-wide PTY usage is exactly vector_size − 1, the placeholder fills the last slot, step 1 picks pis_total, the placeholder's slot frees mid-open, step 2 declines to grow → ENXIO. Killing the failed pane returns usage to vector_size − 1, so every retry fails identically. Usage near a 16-multiple boundary is why this appears with plenty of PTY headroom (190/511 in this issue; 117/999 and 113/999 in ours) and why kern.tty.ptmx_max bumps don't help.

Evidence

• 6/6 spawn failures at 17:58–18:01 (2026-09-11) each had a kernel log line at the same millisecond: ptmx_get_ioctl failed because minor number 96 was out of range (log show --predicate 'process == "kernel" AND eventMessage CONTAINS "ptmx_get_ioctl"'). No ptmx_max limit lines.
• Reproduction (hold PTYs until usage = boundary − 1, then run the exact split/respawn sequence on a private tmux server): 6/6 fail. Same state with plain forkpty → close master → immediate openpty: 5/5 ENXIO. Close master → wait for child exit → openpty: 5/5 success. ±1 PTY: success.
• Also: a second respawn-pane on the already-failed pane crashes tmux 3.6a (server exited unexpectedly) — so retry-on-failure is not a safe workaround.

Env: Darwin 25.5.0 arm64, tmux 3.6a, Claude Code 2.1.268, teammateMode: "tmux".

Suggested fix in the backend

Any of these removes the trigger:

• Spawn the teammate command directly in split-window (no cat placeholder / no respawn-pane), setting border/title after the fact; or
• Before respawn-pane -k, kill the placeholder and wait for its pid to disappear (slot fully freed) so the new forkpty never races the free; or
• Detect ENXIO from respawn-pane and back off by briefly opening+closing one extra /dev/ptmx (forces the vector to grow) before retrying on a fresh pane.

Workaround for users: pre-grow the kernel vector once per boot (open ~512 /dev/ptmx fds and close them — 0.05 s); the vector never shrinks, so the boundary becomes unreachable at normal usage. teammateMode: "in-process" also avoids it entirely.