[FEATURE] Add an interactive "human-assist" mode for GUI/emulator actions in Claude Code / Claude Desktop

Preflight Checklist

• [x] I have searched existing requests and this feature hasn't been requested yet
• [x] This is a single feature request (not multiple features)

Problem Statement

When Claude is performing actions in a GUI context it doesn't have reliable visual/coordinate access to (e.g., an Android emulator, a desktop app, a browser window), it currently has to either (a) burn time and tokens inferring element positions and attempting the action itself, or (b) stop and ask the user to perform the action and report back manually. Both are worse than they need to be. This issue proposes a lightweight, opt-in "assist mode" that lets the user perform the physical action while Claude keeps driving the task.

Proposed Solution

Proposed solution

• Add a toggleable "assist mode" that can be offered at the start of any GUI-interaction-heavy task (emulator testing, desktop app automation, etc.).
• When enabled, replace autonomous GUI action attempts with a simple confirm/clarify prompt per action.
• When disabled (or declined), fall back to current autonomous behavior.
• Should apply to both Claude Code and Claude Desktop, since both can drive GUI/emulator workflows.

Expected benefit

• Saves tokens and time on tasks where Claude has to laboriously infer UI coordinates or state.
• Increases reliability, since a human confirms the action instead of Claude assuming success.
• Keeps the human in the loop without forcing them to fully take over the task or narrate results back manually.

Alternative Solutions

_No response_

Priority

Medium - Would be very helpful

Feature Category

Interactive mode (TUI)

Use Case Example

Example use case

I asked Claude Code to test an Android app in an emulator. After launching the emulator, Claude had no reliable way to identify exact button coordinates, so it spent significant time and tokens guessing, clicking, and re-checking screenshots to find the right UI element.

Instead, the flow could be:

1. After launching the emulator (or entering any GUI-driven step), Claude asks a single yes/no question: "Would you like to assist with GUI interactions for this task?"
2. If "No": Claude proceeds exactly as it does today, working out coordinates/actions on its own.
3. If "Yes": whenever Claude needs to take a GUI action, instead of attempting it blindly, it presents a small structured prompt describing the action it wants performed, e.g.:

• "Please tap the 'Login' button in the top-right corner."
• Response options: Done (confirms the action was taken) or a free-text field for "I don't understand / clarify" if the instruction is unclear.

1. Claude waits for the response, then continues the task using that confirmation as ground truth, instead of re-verifying via screenshots or guesswork.

Additional Context

The underlying capability already exists, Claude can render forms/prompts and interpret simple user responses. What's missing is a structured handoff mechanism for GUI actions specifically, so the human can act as Claude's "hands" for the parts it can't reliably do itself, without derailing the session into manual back-and-forth.