Expose low-level input primitives in browser tools: `key_down`/`key_up` and `mouse_down`/`mouse_move`/`mouse_up`

Summary

The browser automation tools (Claude in Chrome) currently expose input only as
self-contained, high-level actions — left_click, left_click_drag, key, hover,
each with an optional modifiers field. A modifier is pressed and released within
a single action.

This makes an entire class of web applications impossible to automate: any Canvas/SVG
editor whose interaction model depends on state held across several input events.

I hit a hard wall on a real task and had to abandon it. Details below.

What cannot be expressed today

1. Holding a modifier key across multiple actions.

Option down → click → move pointer → click → move pointer → click → Option up

Each of my actions becomes [Option down → click → Option up], so the modifier is
never held while the pointer moves. Applications that reveal a selection candidate
on modifier-hover never enter that state.

2. A drag with intermediate movement.

left_click_drag appears to dispatch only the start and end points. SVG/Canvas editors
that build a rubber-band rectangle from a stream of mousemove events see nothing
usable. In my case the same gesture was sometimes interpreted as a pan of the underlying
image, and sometimes had no effect at all — never as the rectangle-draw the app documents
for a plain drag.

3. Deterministic screenshot scaling.

Within one session and one viewport (1440x725, devicePixelRatio: 2), returned
screenshots alternated between 1440x725 and 1558x784 across calls. Since coordinates
are interpreted against the most recent screenshot, this silently shifts every
coordinate-based click by ~8%. I lost a long time to clicks that landed one table cell
away from the intended target before noticing the pattern.

Concrete failure case

A Japanese school grading system (YouMark Personal) has an editor where the teacher marks
the position of every answer field on a scanned answer sheet. The intended fast path is:
hold Option, move the pointer, and the app auto-detects the table cell under the cursor
and snaps a rectangle to it; each click commits one field and auto-advances to the next.

I needed to place 117 such rectangles. I could:

• select any field programmatically,
• create rectangles via a documented keyboard shortcut (control+shift+arrow duplication),
• read back every rectangle's exact geometry from the DOM for verification,
• and compute the exact target geometry for all 117 rectangles from the source PDF.

I could not place a single one accurately, because both routes to positioning —
drag, and Option-hover-then-click — require input state held across events.

The work was handed back to the human. Everything else about the task was automatable.

Why this is likely inexpensive

The Chrome DevTools Protocol underneath already exposes these as separate operations
(Input.dispatchKeyEvent with keyDown/keyUp, Input.dispatchMouseEvent with
mousePressed/mouseMoved/mouseReleased). This is a request to surface capability
that already exists one layer down, not to build a new subsystem.

Suggested shape

• key_down / key_up as first-class actions, so modifier state persists across

subsequent actions until explicitly released.

• mouse_down / mouse_move / mouse_up as first-class actions, so a caller can

synthesize a drag with as many intermediate points as the target app needs.

• Alternatively, a path option on left_click_drag accepting intermediate waypoints,

plus modifier state that spans the whole gesture.

• Guarantee that screenshot dimensions are stable for a given viewport, or return the

scale factor in the tool result so callers can convert reliably.

Impact

Without these, browser automation is limited to form-and-link web pages. Canvas and SVG
applications — diagram editors, design tools, map tools, annotation and grading systems,
anything with direct manipulation — remain out of reach, even when every other part of
the task is automatable and the correct output is already known.

Thanks for the work on this; it is genuinely useful software. Reporting it in the spirit
of beta feedback.