[FEATURE] Bidirectional Visual Streaming - Real-time Screen Share via CLI Pipes

Preflight Checklist

• [x] I have searched existing requests and this feature hasn't been requested yet
• [x] This is a single feature request (not multiple features)

Problem Statement

Summary

Enable real-time bidirectional visual streaming between users and Claude Code:

• User → Claude: Stream screen/video so Claude observes in real-time
• Claude → User: Stream visual output so Claude can show rather than just describe

This isn't about video file uploads (see related #12676) — it's about treating streams as "files that haven't reached EOF yet," enabling true real-time visual collaboration through the lightweight CLI interface.

The Collaborative Vision

The CLI is powerful precisely because it's lightweight, composable, and integrates with Unix pipelines. Extending this philosophy to visual I/O creates something no GUI-based AI assistant can match:

# User shows Claude (input stream)
ffmpeg -f x11grab -i :0 | claude --visual-in "watch me debug this"

# Claude shows user (output stream)
claude --visual-out "here's the architecture" | mpv -

# Full duplex - true pair programming
claude --screen-share "let's fix this together"

Why CLI + Streaming > GUI Screen Share

| Traditional GUI Screen Share | CLI Bidirectional Streaming |
|-----------------------------|----------------------------|
| Heavy, separate application | Lightweight, composable pipes |
| All-or-nothing experience | Granular control (resolution, fps, window) |
| Platform-specific | Cross-platform via ffmpeg/pipes |
| Can't script or automate | Fully scriptable, CI/CD integrable |
| Synchronous, blocking | Async, non-blocking, backgroundable |

The CLI's strength is composition. ffmpeg | claude | mpv is more powerful than any dedicated screen-share app because each component is replaceable, configurable, and scriptable.

Current Limitations

1. Visual input requires manual screenshot → save → reference cycle
2. "Meat delays" (human decision points) add seconds to minutes of latency
3. Claude cannot show anything visual — only describe in text
4. No way to say "watch what I do" or "let me show you"

Use Cases Enabled

User → Claude (input stream):

• Live pair programming: Claude watches your editor as you type
• Screen share debugging: Claude observes you reproduce an issue in real-time
• "Over the shoulder" mentoring: Like having an expert watching your screen

Claude → User (output stream):

• Visual annotations: Claude highlights the bug location on YOUR screen view
• Live diagrams: Claude generates architecture diagrams and streams them
• "Here's what I see": Claude shows its interpretation with markup
• Headless browser sharing: Claude shows its Playwright session

Bidirectional (the real power):

• True collaborative debugging: "Click there" → you do → "See? That's the bug"
• Interactive design: You sketch, Claude refines, you adjust, Claude annotates
• Teaching/learning: Claude demonstrates while you watch, then reverses

Proposed Solution

Core Concept: Streams are files that haven't reached EOF yet.

# Phase 1: Simplex input (MVP)
ffmpeg -f x11grab -i :0 -f mpegts pipe:1 | claude --visual-in "watch my session"

# Phase 2: Output streaming
claude --visual-out "show the data flow" | mpv --force-window -

# Phase 3: Named pipes (persistent)
mkfifo /tmp/claude-in /tmp/claude-out
claude --visual-in /tmp/claude-in --visual-out /tmp/claude-out &

# Phase 4: Full duplex
claude --screen-share "let's pair program"

Cross-Platform Support

Unix/Linux/macOS: Native pipe support, ffmpeg x11grab/avfoundation

Windows: PowerShell Core provides robust named pipe support:

# Windows named pipes work similarly
ffmpeg -f gdigrab -i desktop -f mpegts \\.\pipe\ClaudeVisualIn
claude --visual-in "\\.\pipe\ClaudeVisualIn"

Implementation Phases

1. MVP: Accept piped input, buffer until EOF, process (Unix-first)
2. Continuous Input: Process frames as they arrive, no EOF wait
3. Output Streaming: Claude generates visual output to stdout/pipe
4. Full Duplex: Bidirectional WebSocket, --screen-share single command

Alternative Solutions

• Manual screenshots: High friction, loses temporal context, unidirectional
• Video file upload (#12676): Not real-time, still unidirectional
• Text descriptions: Lossy — "the button in the upper right" vs. just pointing
• External screen share + Claude: Breaks CLI workflow, loses composition benefits

Why This Matters

The terminal is where developers live. Claude Code's power is meeting us there. But collaboration is fundamentally visual — we point, we show, we draw.

Text is lossy for visual concepts. Currently:

• User: "There's a bug in the modal animation"
• Claude: "Can you describe what you're seeing?"
• User: takes screenshot, saves, references "Here"
• Claude: "I see. Can you show me what happens when you click?"
• repeat

With bidirectional streaming:

• User: claude --screen-share "watch this bug"
• user demonstrates
• Claude: streams annotation overlay pointing at the issue
• "I see it — the z-index on line 42. Here's the fix..."

This is the missing piece of AI pair programming: shared visual context in both directions, with the lightweight composability of Unix pipes.

Related Issues

• #12676 - Video file input support (complementary but distinct — files vs. streams)
• #12644 - Screenshot clipboard paste (similar motivation, different scope)

Priority

High — This fundamentally changes how humans and AI can collaborate through a CLI.

Additional Context

Full technical specification with architecture diagrams, security considerations, and implementation details available upon request.

---

"Everything is a file" — including streams that haven't ended yet.