[FEATURE] Autonomous browser interaction for UI development iteration (canvas, drag, clipboard)

Resolved 💬 3 comments Opened Mar 24, 2026 by gabacool Closed Mar 24, 2026

Problem

Claude Code can write frontend code, deploy it, and run Playwright tests — but can't verify complex UI interactions autonomously. This forces a human into every iteration loop for UI work that involves canvas elements, mouse drag operations, or clipboard verification.

Real-world example

Fixing copy-paste in an xterm.js web terminal required 12 PRs because each attempt needed a human to: open the terminal page, click into the session, mouse-drag to select text, check clipboard content, and report back. The fix involved 4 interdependent components where removing any single piece caused silent failure (no errors). If Claude Code could perform that browser verification loop itself, it would have been 3-4 PRs fully autonomous.

What's needed beyond current Playwright MCP

The existing Playwright MCP plugin handles standard web testing (click buttons, fill forms, check text). The gap is:

  1. Canvas element interaction — xterm.js renders to canvas; Playwright can't determine text positions for targeted mouse operations
  2. Mouse drag with precise coordinates — drag-select within a canvas region
  3. Clipboard state verification — read clipboard content after an operation, especially on self-signed HTTPS contexts
  4. Compound interaction sequences — type into terminal, wait for output, drag-select output, verify clipboard, all in one verification flow

Impact

This is the difference between "AI writes code, human tests" (current) and "AI writes, tests, and iterates autonomously" (desired). For UI-heavy projects, the human-in-the-loop testing step is the primary bottleneck — often 80% of wall-clock time.

Related issues

  • #10646 — Native Browser Integration for Visual UI Testing (auto-closed)
  • #13762 — Browser/Visual Output Integration (auto-closed)
  • #30787 — Local dev server screenshot tool (open)
  • #11791 — Playwright incompatible with web sandbox proxy (open)

View original on GitHub ↗

This issue has 3 comments on GitHub. Read the full discussion on GitHub ↗