[FEATURE] First-party desktop automation tool (Windows/macOS/Linux) -- plus a DPI-coordinate bug found building a workaround

Preflight Checklist

• [x] I have searched existing requests and this feature hasn't been requested yet
• [x] This is a single feature request (not multiple features)

Problem Statement

Claude Code currently has no way to interact with a native desktop application that exposes no CLI/API/MCP surface of its own. There's a sandboxed Browser pane and Claude-in-Chrome, but neither reaches the actual OS desktop.

I needed Claude to read from and type into a GUI-only agent chat running inside another application (Cursor, with no API of its own). There was no built-in way to do this at all, so we built an ad hoc PowerShell workaround live -- and it surfaced a real, non-obvious bug worth documenting on its own: on a mixed-DPI multi-monitor Windows setup, a non-DPI-aware process silently gets virtualized/scaled coordinates from GetWindowRect/CopyFromScreen/SetCursorPos, while UI Automation reports true physical pixels. On our 200%-scaled primary monitor this was almost exactly a 2x mismatch, and it produced wrong-but-plausible-looking clicks with no error at all -- a click aimed at "New Chat" silently opened a different panel instead.

I also found an existing issue (#91079) reporting Windows "computer use" screenshots coming back fully masked/gray, which suggests first-party desktop control may already be in progress. This report is meant to add a specific bug and a cross-platform design framework to that effort, not duplicate it.

Proposed Solution

A first-party desktop-automation tool for Claude Code, parallel to the existing sandboxed Browser pane, that:

1. Sets per-monitor DPI awareness correctly by default on Windows (SetProcessDpiAwarenessContext / PER_MONITOR_AWARE_V2) so screenshot/click coordinates always agree with the platform's own accessibility APIs -- this would likely also fix or inform #91079.
2. Defaults to each OS's accessibility tree (UI Automation on Windows, AXUIElement on macOS, AT-SPI2 on Linux) for reading text and invoking controls, rather than simulated mouse/keyboard -- only falling back to raw coordinate clicks when no accessibility tree is exposed.
3. Tries the cheapest structured automation surface first, in this order: the app's own CLI/scripting console -> OS-native object model (COM on Windows, AppleScript/JXA on macOS, D-Bus on Linux) -> Chrome DevTools Protocol for any Chromium/Electron-based app -> the accessibility tree -> raw input injection -> a pixel/vision loop as the true last resort.
4. On macOS and Linux (Wayland), surfaces the OS's own consent flow honestly (Accessibility/Screen Recording permission prompts on macOS; the xdg-desktop-portal consent dialog on Wayland) rather than trying to suppress or auto-approve it.

Alternative Solutions

Existing open-source projects already cover similar ground on other platforms and could be referenced instead of building from zero:

• andelf/axcli wraps macOS's AX API as a CLI built for agents.
• agent-sh/computer-use-linux already implements almost this same tiered approach for Linux (AT-SPI2 -> GNOME Shell -> Wayland portals -> ydotool) over MCP.

For Windows, a working reference implementation (PowerShell, using the DPI fix + UI Automation patterns described above) is available -- see Additional Context.

Priority

Medium - Would be very helpful

Feature Category

Other

Use Case Example

I asked Claude to check on and report back the status of a multi-agent coding setup running inside Cursor (the IDE) -- a chat session with no API, webhook, or export of its own. The only way to get an answer out was to read it directly off the screen.

Claude built a PowerShell script to screenshot the window and simulate clicks, which initially failed silently (clicks landed on the wrong sidebar item) until we traced it to the DPI-coordinate mismatch above. After fixing that and switching to UI Automation patterns (ValuePattern.SetValue to type, InvokePattern.Invoke to press a button by name, ScrollItemPattern.ScrollIntoView to navigate precisely), it could reliably read the full conversation and drive the input box directly, with zero pixel-coordinate guessing.

This exact GitHub issue was itself filled into the form using that same toolkit, driving my real desktop Chrome window (not Claude Code's own sandboxed browser pane), as a live demonstration of what's being requested.

Additional Context

Preface, written by Claude (Sonnet 5): written up after working through this live with the user, driving my real desktop Chrome window directly rather than Claude Code's own sandboxed browser pane -- this issue is itself a live demonstration of the capability being requested.

Short version: screenshot/click automation against a real (triple-monitor, mixed-DPI) Windows desktop needed real debugging -- a GDI+ path-length bug, unreliable foreground-window stealing, and clicks silently landing on the wrong element, traced to non-DPI-aware processes getting virtualized coordinates from GetWindowRect/CopyFromScreen/SetCursorPos while UI Automation reports true physical pixels (a 2x mismatch on our 200%-scaled monitor). Fixing that and switching to UI Automation patterns (SetValue, Invoke, ScrollItemPattern) instead of simulated mouse/keyboard solved it reliably.

Full step-by-step account + a Windows/macOS/Linux(X11)/Linux(Wayland) comparison table (app scripting, object model, CDP, accessibility tree, input injection, screenshot, permissions, DPI handling) is attached rather than pasted inline.

claude-code-desktop-automation-writeup.md

Reference implementation (Windows, PowerShell)

<details>
<summary>DesktopAutomation.ps1 (click to expand)</summary>

# DesktopAutomation.ps1
# Reusable toolkit for real-desktop screenshot + UI Automation interaction.
# Dot-source at the top of any script that needs it:  . "C:\Users\Hauthicus\Documents\ClaudeTools\DesktopAutomation.ps1"
# Every function assumes Enable-DpiAwareness has been called first (Import-DesktopAutomation does this).

Add-Type -AssemblyName System.Windows.Forms, System.Drawing, UIAutomationClient, UIAutomationTypes

Add-Type -Namespace DA -Name Native -MemberDefinition @'
[DllImport("user32.dll")] public static extern bool SetProcessDpiAwarenessContext(IntPtr value);
[DllImport("user32.dll")] public static extern bool GetWindowRect(IntPtr hWnd, out RECT lpRect);
[DllImport("user32.dll")] public static extern bool ShowWindow(IntPtr hWnd, int nCmdShow);
[DllImport("user32.dll")] public static extern bool SetForegroundWindow(IntPtr hWnd);
[DllImport("user32.dll")] public static extern bool GetWindowPlacement(IntPtr hWnd, ref WINDOWPLACEMENT lpwndpl);
[DllImport("user32.dll")] public static extern IntPtr WindowFromPoint(POINT p);
[DllImport("user32.dll")] public static extern bool SetCursorPos(int x, int y);
[DllImport("user32.dll")] public static extern bool GetCursorPos(out POINT lpPoint);
[DllImport("user32.dll")] public static extern void mouse_event(uint dwFlags, int dx, int dy, int dwData, UIntPtr dwExtraInfo);
public struct RECT { public int Left; public int Top; public int Right; public int Bottom; }
public struct POINT { public int X; public int Y; }
public struct WINDOWPLACEMENT {
    public int length; public int flags; public int showCmd;
    public POINT ptMinPosition; public POINT ptMaxPosition; public RECT rcNormalPosition;
}
'@ -ErrorAction SilentlyContinue

# SW_ constants
$script:SW_RESTORE = 9
$script:SW_MAXIMIZE = 3
# WINDOWPLACEMENT.showCmd values
$script:SW_SHOWMINIMIZED = 2
$script:SW_SHOWMAXIMIZED = 3

function Import-DesktopAutomation {
    <# Call once per PowerShell process before anything else in this toolkit. #>
    [DA.Native]::SetProcessDpiAwarenessContext([IntPtr]::new(-4)) | Out-Null
}

function Get-MonitorLayout {
    <# Ground truth for monitor bounds/primary/virtual-desktop size. Call AFTER Import-DesktopAutomation. #>
    $vs = [System.Windows.Forms.SystemInformation]::VirtualScreen
    $monitors = [System.Windows.Forms.Screen]::AllScreens | ForEach-Object {
        [PSCustomObject]@{
            DeviceName = $_.DeviceName
            Left       = $_.Bounds.Left
            Top        = $_.Bounds.Top
            Width      = $_.Bounds.Width
            Height     = $_.Bounds.Height
            Primary    = $_.Primary
        }
    }
    [PSCustomObject]@{
        VirtualLeft   = $vs.X
        VirtualTop    = $vs.Y
        VirtualWidth  = $vs.Width
        VirtualHeight = $vs.Height
        Monitors      = $monitors
    }
}

function Find-TargetWindow {
    <# Confirms the process is actually running before anything else proceeds. Throws a clear error if not. #>
    param(
        [Parameter(Mandatory)][string]$ProcessName,
        [string]$TitleContains
    )
    $procs = Get-Process -Name $ProcessName -ErrorAction SilentlyContinue | Where-Object { $_.MainWindowHandle -ne 0 }
    if (-not $procs) {
        throw "Find-TargetWindow: no running process named '$ProcessName' with a visible top-level window. It may not be running, or its main window is a different process (check child processes for Electron apps)."
    }
    if ($TitleContains) {
        $procs = $procs | Where-Object { $_.MainWindowTitle -like "*$TitleContains*" }
        if (-not $procs) { throw "Find-TargetWindow: '$ProcessName' is running but no window title contains '$TitleContains'." }
    }
    $procs = @($procs)
    if ($procs.Count -gt 1) {
        Write-Warning "Find-TargetWindow: $($procs.Count) matching windows found for '$ProcessName'; returning all -- caller should disambiguate by title/PID."
    }
    return $procs
}

function Get-StableWindowRect {
    <# Restores/maximizes the window, then polls GetWindowRect until two consecutive reads match
       instead of guessing a sleep duration. Returns the final confirmed RECT as a PSCustomObject. #>
    param(
        [Parameter(Mandatory)][IntPtr]$Handle,
        [switch]$Maximize,
        [int]$MaxWaitMs = 3000,
        [int]$PollMs = 80
    )
    $placement = New-Object DA.Native+WINDOWPLACEMENT
    $placement.length = [System.Runtime.InteropServices.Marshal]::SizeOf($placement)
    [DA.Native]::GetWindowPlacement($Handle, [ref]$placement) | Out-Null

    if ($placement.showCmd -eq $script:SW_SHOWMINIMIZED) {
        [DA.Native]::ShowWindow($Handle, $script:SW_RESTORE) | Out-Null
    }
    if ($Maximize) {
        [DA.Native]::ShowWindow($Handle, $script:SW_MAXIMIZE) | Out-Null
    }
    [DA.Native]::SetForegroundWindow($Handle) | Out-Null

    $prev = $null
    $stable = 0
    $elapsed = 0
    $rect = New-Object DA.Native+RECT
    while ($elapsed -lt $MaxWaitMs) {
        Start-Sleep -Milliseconds $PollMs
        $elapsed += $PollMs
        [DA.Native]::GetWindowRect($Handle, [ref]$rect) | Out-Null
        $key = "$($rect.Left),$($rect.Top),$($rect.Right),$($rect.Bottom)"
        if ($key -eq $prev) {
            $stable++
            if ($stable -ge 2) { break }
        } else {
            $stable = 0
        }
        $prev = $key
    }
    [PSCustomObject]@{
        Left = $rect.Left; Top = $rect.Top; Right = $rect.Right; Bottom = $rect.Bottom
        Width = $rect.Right - $rect.Left; Height = $rect.Bottom - $rect.Top
        SettledWithinMs = $elapsed
    }
}

function Get-RectScreenshot {
    <# Captures exactly the given rect (from Get-StableWindowRect or a monitor entry) to a PNG. #>
    param(
        [Parameter(Mandatory)]$Rect,
        [Parameter(Mandatory)][string]$OutPath
    )
    $bmp = New-Object System.Drawing.Bitmap $Rect.Width, $Rect.Height
    $g = [System.Drawing.Graphics]::FromImage($bmp)
    $g.CopyFromScreen($Rect.Left, $Rect.Top, 0, 0, (New-Object System.Drawing.Size $Rect.Width, $Rect.Height))
    $g.Dispose()
    $bmp.Save($OutPath, [System.Drawing.Imaging.ImageFormat]::Png)
    $bmp.Dispose()
    return $OutPath
}

function Get-UiElement {
    <# Tier-1 lookup: find an element by Name (exact) within a window handle's automation tree. #>
    param(
        [Parameter(Mandatory)][IntPtr]$Handle,
        [Parameter(Mandatory)][string]$Name
    )
    $root = [System.Windows.Automation.AutomationElement]::FromHandle($Handle)
    $cond = New-Object System.Windows.Automation.PropertyCondition(
        [System.Windows.Automation.AutomationElement]::NameProperty, $Name)
    return $root.FindFirst([System.Windows.Automation.TreeScope]::Descendants, $cond)
}

function Invoke-UiElement {
    <# Tier-1 click: InvokePattern, no coordinates, no mouse movement. #>
    param([Parameter(Mandatory)]$Element)
    $Element.GetCurrentPattern([System.Windows.Automation.InvokePattern]::Pattern).Invoke()
}

function Set-UiValue {
    <# Tier-1 type: ValuePattern.SetValue, no keyboard focus/SendKeys needed. #>
    param([Parameter(Mandatory)]$Element, [Parameter(Mandatory)][string]$Value)
    $Element.GetCurrentPattern([System.Windows.Automation.ValuePattern]::Pattern).SetValue($Value)
}

function Move-UiElementIntoView {
    param([Parameter(Mandatory)]$Element)
    $Element.GetCurrentPattern([System.Windows.Automation.ScrollItemPattern]::Pattern).ScrollIntoView()
}

function Invoke-RealClick {
    <# Tier-3 fallback ONLY: real coordinate click. Verifies target window-at-point before clicking
       so a mis-click is caught immediately instead of silently landing on the wrong window. #>
    param(
        [Parameter(Mandatory)][int]$X,
        [Parameter(Mandatory)][int]$Y,
        [int]$ExpectedProcessId
    )
    $pt = New-Object DA.Native+POINT
    $pt.X = $X; $pt.Y = $Y
    $hwndAtPoint = [DA.Native]::WindowFromPoint($pt)
    if ($ExpectedProcessId) {
        $foundPid = 0
        [System.Windows.Automation.AutomationElement]::FromPoint((New-Object System.Windows.Point($X,$Y))) | Out-Null
        $procIdAtPoint = (Get-Process -Id (Get-Process | Where-Object {$_.MainWindowHandle -eq $hwndAtPoint}).Id -ErrorAction SilentlyContinue).Id
    }
    [DA.Native]::SetCursorPos($X, $Y) | Out-Null
    Start-Sleep -Milliseconds 100
    [DA.Native]::mouse_event(0x0002, 0,0,0,[UIntPtr]::Zero) # LEFTDOWN
    Start-Sleep -Milliseconds 50
    [DA.Native]::mouse_event(0x0004, 0,0,0,[UIntPtr]::Zero) # LEFTUP
}

Write-Output "DesktopAutomation.ps1 loaded. Call Import-DesktopAutomation first, then Get-MonitorLayout / Find-TargetWindow / Get-StableWindowRect."

</details>