Support held-pointer browser interactions for canvas/WebGL preview QA
Feature request
Please expose low-level held-pointer controls in Claude Code Preview / browser automation so Claude can test canvas/WebGL and drag-heavy apps without relying only on app-specific JavaScript bridges.
Useful API shapes would be one of:
mouseDown(x, y),mouseMove(x, y),mouseUp(x, y)beginDrag(...),moveDrag(...),endDrag(...)- a held-drag tool that can pause before release so Claude can inspect/screenshot the UI while the pointer is still down
Problem
Claude Code Preview can interact with running apps, click, inspect, and screenshot. For DOM-backed apps this is often enough. But for canvas/WebGL apps, the meaningful UI is pixels inside a canvas rather than DOM nodes.
Some interfaces also expose their most important state while the pointer is held down: drag previews, hover previews, blocked-drop messages, range overlays, move scores, resize handles, design canvas guides, map dragging, timeline scrubbing, etc. If browser automation only supports a complete/atomic drag, Claude can observe the result after release but cannot inspect the transient held-pointer state that determines the correct release location.
Concrete example
In a Unity WebGL game, dragging a card from the hand into the world shows a real-time placement preview while the pointer is held:
- valid/blocked placement state
- range circles
- interaction highlights
- move score preview
- nearby object feedback
The game can expose a custom WebGL bridge method such as HoldCard(...), and that works for deterministic QA. But this only tests the app-specific bridge path, not the real browser pointer interaction. For manual-feel QA, Claude needs to press on the card, move while still holding, inspect the visible preview, continue moving if needed, and release intentionally.
Expected behavior
Claude Code should be able to:
- press down at a chosen coordinate or target
- move while keeping the pointer/button held down
- take a screenshot or inspect visible state before release
- continue moving while still held, if needed
- release at the chosen coordinate
This should work even when the target UI is a canvas/WebGL surface with little or no useful DOM.
Impact
This would improve Claude Code's ability to QA and iterate on:
- browser games
- Unity/WebGL previews
- drawing/canvas apps
- drag-and-drop editors
- design tools
- map/timeline/scrubber interactions
- any UI where held-pointer state is the behavior under test
This issue has 1 comment on GitHub. Read the full discussion on GitHub ↗