desk-pilot

desk-pilot gives Claude Code, or any MCP client, eyes and hands on the real Windows desktop. It sees the actual screen, reads each window's controls through UI Automation, and drives the real mouse and keyboard. It has just three MCP tools on purpose, because every tool schema costs tokens in every session.

Windows only.

Install

As a Claude Code plugin, which adds the tools, a skill and a safety hook:

claude plugin marketplace add acus9345/desk-pilot

claude plugin install desk-pilot@desk-pilot

Inside an interactive claude session, the same is /plugin marketplace add acus9345/desk-pilot, then /plugin install desk-pilot@desk-pilot. Start a new session afterwards.

As a standalone MCP server for any MCP client:

cargo install desk-pilot

claude mcp add desk-pilot -- desk-pilot

The standalone server has the three tools only: no skill, and no approval hook.

Requirements: Windows 10 2004 or later, or Windows 11, and Node 22.19 or later on PATH. On first use it compiles its engine with the C# compiler that ships with .NET Framework 4.x in Windows (about 0.2 s), then reuses it. There is no Mac version.

On Windows you can also use the installer from qlinter.org (desk-pilot-setup-0.4.2.exe). It installs the Claude Code plugin for you, and needs Claude Code installed first.

Quick start

Ask Claude to do something on screen. It typically starts with:

{ "tool": "desktop_control", "action": "windows" }

{ "tool": "desktop_control", "action": "controls", "title": "Calculator" }

{ "tool": "desktop_control", "action": "click", "target": "Seven", "role": "Button" }

{ "tool": "screen_view", "window": "Calculator" }

Tools and features

screen_view

The live screen as an image with coordinate rulers. Fields: screen (primary, all or a monitor index), window (title or process), hwnd, region (x,y,w,h to zoom in), diff (only what changed), maxEdge, includeCursor, rulers, annotate (box and number the window's controls; target then takes the number), forceImage (re-send an unchanged frame).

desktop_control

One action per call. x/y are physical screen pixels; prefer target (a control's name or index) whenever a control exists.

Any step can carry expect, for example { "foreground": "Save As" }, { "exists": "OK" }, { "gone": "Loading" } or { "value": "hello" }. The step waits for the condition and fails if it never holds.

desktop_sequence

Many actions in one call, checked step by step, stopping at the first failure. Fields: steps (each with an action and desktop_control's fields), risk (low, medium or high), confirm, dryRun and capture (none, window, screen or all, for one frame at the end).

{ "tool": "desktop_sequence", "steps": [
  { "action": "click", "target": "File name" },
  { "action": "type", "text": "notes.txt" },
  { "action": "key", "key": "enter", "expect": { "gone": "Save As" } }
], "risk": "low" }

How it works

dist/server.mjs is a stdio MCP server in Node. It drives one long-lived engine process written in C# (engine/*.cs) over JSON lines. The engine keeps per-monitor DPI awareness, the UI Automation client and the Esc hotkey warm. If the engine dies it is restarted on the next call, at most 3 times in 10 minutes. After that, each request runs in its own short-lived engine, so the tools keep working, only slower.

Before clicking a named control, the engine looks it up again. A disabled control is refused, one scrolled out of view is scrolled into view, an ambiguous name returns the candidates, and a stale listing fails instead of clicking whatever moved into that slot. Input uses real mouse and keyboard events, and a click stays down for 40 ms, because canvases such as Paint's ignore instant clicks.

Frames are shrunk to 1356 px on the long side by the engine, so the coordinate line under each image is exact. An identical frame is not sent again, and the frame after a batch carries only the region that changed. End frames wait until the screen stops changing.

While Claude is acting, the screen shows it: a blue glow around the primary monitor, a pointer marker with an optional label, and a control bar at the bottom. Esc, Stop or Take over stops the batch between steps, even during a wait. None of this takes focus, and it clears itself for the moment of each screenshot.

A desktop_sequence declared medium or high risk is refused unless it carries confirm: true. In the plugin, a PreToolUse hook then turns that into a Claude Code permission prompt listing every step.

Settings

Environment variables, for example in the env block of Claude Code's settings.json:

The plugin keeps its compiled engine in the plugin data folder Claude Code provides (DESK_PILOT_DATA_DIR).

Limits and troubleshooting

• Windows only.
• desk-pilot moves your real mouse and uses your real keyboard, so don't use the computer while it acts. Press Esc to stop it.
• Windows blocks injected input into windows running as administrator, unless Claude Code also runs as administrator.
• Canvas-drawn and some Electron windows expose few or no controls; use frames there.
• Esc and the overlay are unavailable once the engine has fallen back to one-shot mode, and down is refused there.
• The overlay covers the primary monitor only, and a second monitor was not tested.
• No pen pressure and no multi-touch.
• The back and forward mouse buttons were not confirmed end to end; key: "alt+left" does the same job in most apps.
• A key held with down does not auto-repeat; key with holdMs does.
• "Node not found": install Node 22.19 or later and make sure it is on PATH, then start a new session.

Worked examples

Tool results below use desk-pilot's real format. Window handles, indexes, sizes and coordinates differ on every computer, so read the shape, not the numbers. desk-pilot drives your real mouse and keyboard, so leave the computer alone while it works.

1. Calculator: 1 + 2 =

{ "tool": "desktop_control", "action": "controls", "title": "Calculator" }

<controls window='Calculator' hwnd=0x2A0B4C shown=31 of=37>
0 [Button] 'One'
1 [Button] 'Two'
...

{ "tool": "desktop_sequence", "steps": [
  { "action": "click", "target": "One", "role": "Button" },
  { "action": "click", "target": "Plus" },
  { "action": "click", "target": "Two", "role": "Button" },
  { "action": "click", "target": "Equals", "expect": { "exists": "Display is 3" } }
] }

<batch done=4/4 ms=1180>
now: fg 'Calculator'

A clean batch reports only a count and the final state. If a step fails, the batch stops there and names it.

2. Fill a form and save through Save As

{ "tool": "desktop_sequence", "risk": "low", "steps": [
  { "action": "focus", "title": "Untitled - Notepad" },
  { "action": "type", "text": "Shopping list: milk, eggs" },
  { "action": "key", "key": "ctrl+s", "expect": { "foreground": "Save As" } },
  { "action": "click", "target": "File name:", "role": "Edit" },
  { "action": "type", "text": "list.txt" },
  { "action": "key", "key": "enter", "expect": { "gone": "Save As" } }
], "capture": "window" }

<batch done=6/6 ms=2300>
now: fg 'list.txt - Notepad'

Each expect makes its step wait for the dialog to appear or close, and fail if it never does. capture: "window" adds one frame of the front window, with only the changed area.

3. Draw in Paint

{ "tool": "screen_view", "window": "Paint", "rulers": true }

{ "tool": "desktop_control", "action": "draw", "smooth": true,
  "strokes": [[[600,400],[800,400],[800,600],[600,600],[600,400]], [[650,450],[750,550]]] }

draw ok · drew 2 strokes, 1083 px · fg 'Untitled - Paint'

Strokes move at about 1000 px/s and lift the button between strokes, because canvases such as Paint's ignore instant jumps. The frame's coordinate line tells you how image pixels map to screen pixels.

4. Type into a browser

{ "tool": "desktop_sequence", "steps": [
  { "action": "click", "target": "Email", "role": "Edit" },
  { "action": "type", "text": "ada@example.com", "expect": { "value": "ada@example.com" } },
  { "action": "click", "target": "Sign up", "role": "Button", "expect": { "exists": "Welcome" } }
] }

type reads the text back and reports verified, NOT verified or unknown. If the browser exposes few controls, take a screen_view and click by x/y instead.

5. Menus and combo boxes

{ "tool": "desktop_sequence", "steps": [
  { "action": "click", "target": "File", "role": "MenuItem" },
  { "action": "click", "target": "Page setup...", "expect": { "foreground": "Page Setup" } },
  { "action": "click", "target": "Size", "role": "ComboBox" },
  { "action": "click", "target": "A4", "role": "ListItem" },
  { "action": "click", "target": "OK", "expect": { "gone": "Page Setup" } }
] }

Open the menu and pick from it in one batch. Each click looks the control up again just before pressing it.

6. Arrange a window, including a minimised one

{ "tool": "desktop_control", "action": "window", "title": "Calculator", "show": "restore" }

{ "tool": "desktop_control", "action": "window", "title": "Calculator", "x": 100, "y": 100, "width": 400, "height": 600 }

window ok · window 'Calculator' @ 100,100 400x600

The size is the window's visible frame, without Windows' invisible resize border. desk-pilot checks the result and says so if the app snapped to a different size. A covered window must be brought forward to be used: desk-pilot sends real input, which goes to whatever is on top.

7. Read a canvas or game

{ "tool": "desktop_control", "action": "controls", "title": "My Game" }

<controls window='My Game' hwnd=0x1F0A22 shown=0 of=0>
(no UI Automation tree here; use screen_view)

{ "tool": "screen_view", "window": "My Game", "rulers": true }

Then click the coordinates you read off the image, or zoom in with region for small text.

8. A risky batch with confirm

{ "tool": "desktop_sequence", "risk": "high", "steps": [
  { "action": "click", "target": "Delete", "role": "Button" },
  { "action": "click", "target": "Yes", "expect": { "gone": "Confirm delete" } }
] }

desktop_sequence refused before touching the desktop: this batch is high risk ...

Add "confirm": true to run it. In the Claude Code plugin, a hook then shows you a permission prompt listing every step, even if the tool is on your allow list. "dryRun": true lists the plan without running it.

9. Hold a key across steps

{ "tool": "desktop_sequence", "steps": [
  { "action": "click", "target": "a.txt" },
  { "action": "down", "key": "shift" },
  { "action": "click", "target": "f.txt" },
  { "action": "up" }
] }

While something is held, every result says HOLDING. It is released by up, Esc, an interrupted turn, holdMs (30 s by default) or the engine closing.

Edge cases

• Windows running as administrator. Windows blocks injected input into them unless Claude Code also runs as administrator. The reply says Windows refused the input (another desktop, a lock screen, or an elevated window in front). Moving or resizing one is refused too.
• Classic popup menus and WinForms. desk-pilot uses the real mouse, so these work like any click. Right-click menus appear where the pointer is.
• XAML text boxes. type sends real keystrokes, so these work. Check the verified result.
• Browsers. Real input goes to the front window, so browser windows sharing one process are not a problem. Some pages expose few controls; use frames there.
• DPI and scaling. x/y are always physical screen pixels. The engine is per-monitor DPI aware, and each frame says exactly how image pixels map back to the screen.
• Several monitors. screens lists them, and screen_view takes screen: "all" or a monitor index. The Esc overlay covers only the primary monitor, and a second monitor has not been tested.
• Modal dialogs that take focus. Each result reports the window in front. Use expect: { "foreground": "..." } to wait for a dialog, and gone to wait for it to close.
• Controls that move or vanish. A named target is looked up again before each click. A stale listing fails instead of clicking whatever moved into that slot, and an ambiguous name returns the candidates. A control scrolled out of view is scrolled back into view.
• Slow apps and timeouts. expect waits up to 1.5 s by default (waitFor 5 s), and timeoutMs changes that. Each action has DESK_PILOT_TIMEOUT_MS (20 s). Past it, the engine is restarted and the action is reported as unknown, never retried.
• Esc in the middle of a batch. Esc, Stop or Take over stops the batch between steps, even during a wait. Held keys and buttons are released, and the result says the person pressed Esc and cancelled this.
• The engine crashing. It restarts on the next call, at most 3 times in 10 minutes. After that, each request runs in its own short-lived engine: slower, with no Esc, overlay or down. The next engine releases anything a killed one left held.
• A plugin update during a session. The running server notices the new version and tells you to start a new session.

FAQ

Does desk-pilot move my mouse? Yes. It uses the real mouse and keyboard, so don't use the computer while it acts. Press Esc to take over.

How is it different from deskcrew? desk-pilot is an MCP server using the real mouse and keyboard, one agent at a time. deskcrew is a command that works in the background, with a cursor per agent.

Which apps can use it? Claude Code as a plugin, or any MCP client through desk-pilot from cargo.

Why only three tools? Every tool schema costs tokens in every session, so the actions live inside three tools.

Why didn't my screenshot come back? An unchanged frame isn't sent again. Pass forceImage: true to get it anyway.

Can it read small text? Frames are 1356 px on the long side. For small text, view one window or zoom with region.

What happens if it clicks the wrong thing? Named targets are checked before each click, and a batch stops at the first failure. For anything destructive, declare risk so it needs confirm and your approval.

Does it need .NET or Visual Studio? No. It compiles its engine with the C# compiler built into Windows, once, in about 0.2 s. It needs Node 22.19 or later.

Does it work on a Mac or Linux? No, Windows only.

How do I turn off the glow and control bar? Set DESK_PILOT_TAKEOVER=0. That also turns off Esc to stop.

Common results and errors

Version and source

This page describes desk-pilot 0.4.2.

• Source: https://github.com/acus9345/desk-pilot
• crates.io: https://crates.io/crates/desk-pilot
• License: MIT

desk-pilot 讓 Claude Code 或任何 MCP 用戶端，在真正的 Windows 桌面上有眼睛也有手。它看得到實際畫面，透過 UI Automation 讀取每個視窗的控制項，並操作真正的滑鼠和鍵盤。它刻意只有三個 MCP 工具，因為每個工具的 schema 在每個工作階段都要花 token。

僅限 Windows。

安裝

作為 Claude Code 外掛，會加入工具、一個 skill 和一個安全 hook：

claude plugin marketplace add acus9345/desk-pilot

claude plugin install desk-pilot@desk-pilot

在互動式 claude 工作階段中，同樣的步驟是 /plugin marketplace add acus9345/desk-pilot，然後 /plugin install desk-pilot@desk-pilot。之後開一個新的工作階段。

作為獨立 MCP 伺服器，給任何 MCP 用戶端使用：

cargo install desk-pilot

claude mcp add desk-pilot -- desk-pilot

獨立伺服器只有三個工具：沒有 skill，也沒有核准 hook。

需求：Windows 10 2004 版或更新，或 Windows 11，以及 PATH 上的 Node 22.19 或更新版本。第一次使用時，它會用 Windows 內建 .NET Framework 4.x 的 C# 編譯器編譯引擎（約 0.2 秒），之後重複使用。沒有 Mac 版。

在 Windows 上也可以用 qlinter.org 的安裝程式（desk-pilot-setup-0.4.2.exe）。它會幫你安裝 Claude Code 外掛，需要先安裝 Claude Code。

快速開始

請 Claude 在畫面上做點事。它通常從這些開始：

{ "tool": "desktop_control", "action": "windows" }

{ "tool": "desktop_control", "action": "controls", "title": "Calculator" }

{ "tool": "desktop_control", "action": "click", "target": "Seven", "role": "Button" }

{ "tool": "screen_view", "window": "Calculator" }

工具與功能

screen_view

即時畫面的圖片，附座標尺。欄位：screen（primary、all 或螢幕編號）、window（標題或程序）、hwnd、region（x,y,w,h，放大某區）、diff（只看變化）、maxEdge、includeCursor、rulers、annotate（框出並標號視窗的控制項；之後 target 可以用編號）、forceImage（重送沒變的畫面）。

desktop_control

每次呼叫一個動作。x/y 是實體螢幕像素；只要有控制項，優先用 target（控制項的名稱或編號）。

任何步驟都可以帶 expect，例如 { "foreground": "Save As" }、{ "exists": "OK" }、{ "gone": "Loading" } 或 { "value": "hello" }。步驟會等待條件成立，一直不成立就失敗。

desktop_sequence

一次呼叫多個動作，逐步檢查，遇到第一個失敗就停。欄位：steps（每步有 action 和 desktop_control 的欄位）、risk（low、medium 或 high）、confirm、dryRun 和 capture（none、window、screen 或 all，最後拍一張）。

{ "tool": "desktop_sequence", "steps": [
  { "action": "click", "target": "File name" },
  { "action": "type", "text": "notes.txt" },
  { "action": "key", "key": "enter", "expect": { "gone": "Save As" } }
], "risk": "low" }

運作原理

dist/server.mjs 是用 Node 寫的 stdio MCP 伺服器。它透過 JSON lines 驅動一個以 C# 撰寫（engine/*.cs）、長時間執行的引擎程序。引擎讓逐螢幕 DPI 感知、UI Automation 用戶端和 Esc 熱鍵保持就緒。引擎掛掉時，下一次呼叫會重啟它，10 分鐘內最多 3 次。超過之後，每個請求都在自己的短命引擎中執行，工具仍然可用，只是較慢。

點擊具名控制項之前，引擎會重新查找一次。停用的控制項會被拒絕，捲出畫面的會捲回來，名稱不明確時會回傳候選項，過期的清單會直接失敗，而不是點到移到那個位置的東西。輸入使用真正的滑鼠和鍵盤事件，點擊會按住 40 毫秒，因為小畫家這類畫布會忽略瞬間點擊。

引擎把畫面縮到長邊 1356 像素，所以每張圖下方的座標說明是精確的。相同的畫面不會重送，批次後的畫面只帶有變化的區域。結尾畫面會等到畫面不再變化。

Claude 操作時，畫面會顯示出來：主螢幕周圍一圈藍光、一個可加標籤的指標標記，以及底部的控制列。Esc、Stop 或 Take over 會在步驟之間停止批次，即使正在等待中。這些都不會搶走焦點，而且在每次截圖的那一刻會自動隱藏。

宣告為 medium 或 high 風險的 desktop_sequence，除非帶有 confirm: true，否則會被拒絕。在外掛中，PreToolUse hook 會再把它變成列出每個步驟的 Claude Code 權限提示。

設定

環境變數，例如放在 Claude Code settings.json 的 env 區塊中：

外掛把編譯好的引擎放在 Claude Code 提供的外掛資料夾（DESK_PILOT_DATA_DIR）。

限制與疑難排解

• 僅限 Windows。
• desk-pilot 會移動你真正的滑鼠、使用你真正的鍵盤，所以它操作時不要用電腦。按 Esc 停止它。
• Windows 會擋下對以系統管理員身分執行之視窗的模擬輸入，除非 Claude Code 也以系統管理員身分執行。
• 自繪畫布和部分 Electron 視窗只露出很少或沒有控制項；那裡請用截圖。
• 引擎退回單次模式後，Esc 和覆蓋層無法使用，down 也會被拒絕。
• 覆蓋層只涵蓋主螢幕，第二個螢幕未經測試。
• 沒有觸控筆壓力，也沒有多點觸控。
• back 和 forward 滑鼠鍵沒有完整驗證過；在大多數 App 中 key: "alt+left" 效果相同。
• 用 down 按住的按鍵不會自動重複；key 加 holdMs 會。
• 「Node not found」：安裝 Node 22.19 或更新版本並確認在 PATH 上，然後開新的工作階段。

實例

以下工具結果使用 desk-pilot 真實的格式。視窗 handle、編號、大小和座標在每台電腦上都不同，請看格式，不要看數字。desk-pilot 操作你真正的滑鼠和鍵盤，所以它工作時別碰電腦。

1. 小算盤：1 + 2 =

{ "tool": "desktop_control", "action": "controls", "title": "Calculator" }

<controls window='Calculator' hwnd=0x2A0B4C shown=31 of=37>
0 [Button] 'One'
1 [Button] 'Two'
...

{ "tool": "desktop_sequence", "steps": [
  { "action": "click", "target": "One", "role": "Button" },
  { "action": "click", "target": "Plus" },
  { "action": "click", "target": "Two", "role": "Button" },
  { "action": "click", "target": "Equals", "expect": { "exists": "Display is 3" } }
] }

<batch done=4/4 ms=1180>
now: fg 'Calculator'

順利的批次只回報數量和最後狀態。某一步失敗時，批次在那裡停下並指出是哪一步。

2. 填表單並透過「另存新檔」存檔

{ "tool": "desktop_sequence", "risk": "low", "steps": [
  { "action": "focus", "title": "Untitled - Notepad" },
  { "action": "type", "text": "Shopping list: milk, eggs" },
  { "action": "key", "key": "ctrl+s", "expect": { "foreground": "Save As" } },
  { "action": "click", "target": "File name:", "role": "Edit" },
  { "action": "type", "text": "list.txt" },
  { "action": "key", "key": "enter", "expect": { "gone": "Save As" } }
], "capture": "window" }

<batch done=6/6 ms=2300>
now: fg 'list.txt - Notepad'

每個 expect 讓該步驟等待對話框出現或關閉，一直沒發生就失敗。capture: "window" 會加一張前景視窗的畫面，只含有變化的區域。

3. 在小畫家畫圖

{ "tool": "screen_view", "window": "Paint", "rulers": true }

{ "tool": "desktop_control", "action": "draw", "smooth": true,
  "strokes": [[[600,400],[800,400],[800,600],[600,600],[600,400]], [[650,450],[750,550]]] }

draw ok · drew 2 strokes, 1083 px · fg 'Untitled - Paint'

線條以約每秒 1000 像素移動，線條之間會放開按鍵，因為小畫家這類畫布會忽略瞬間跳躍。畫面的座標說明告訴你圖片像素如何對應到螢幕像素。

4. 在瀏覽器中輸入

{ "tool": "desktop_sequence", "steps": [
  { "action": "click", "target": "Email", "role": "Edit" },
  { "action": "type", "text": "ada@example.com", "expect": { "value": "ada@example.com" } },
  { "action": "click", "target": "Sign up", "role": "Button", "expect": { "exists": "Welcome" } }
] }

type 會讀回文字並回報 verified、NOT verified 或 unknown。如果瀏覽器露出的控制項很少，就用 screen_view 截圖，改用 x/y 點擊。

5. 選單與下拉選單

{ "tool": "desktop_sequence", "steps": [
  { "action": "click", "target": "File", "role": "MenuItem" },
  { "action": "click", "target": "Page setup...", "expect": { "foreground": "Page Setup" } },
  { "action": "click", "target": "Size", "role": "ComboBox" },
  { "action": "click", "target": "A4", "role": "ListItem" },
  { "action": "click", "target": "OK", "expect": { "gone": "Page Setup" } }
] }

在同一個批次中打開選單並選取。每次點擊前都會重新查找控制項。

6. 排列視窗，包括最小化的視窗

{ "tool": "desktop_control", "action": "window", "title": "Calculator", "show": "restore" }

{ "tool": "desktop_control", "action": "window", "title": "Calculator", "x": 100, "y": 100, "width": 400, "height": 600 }

window ok · window 'Calculator' @ 100,100 400x600

大小指的是視窗的可見框，不含 Windows 看不見的調整邊框。desk-pilot 會檢查結果，如果 App 自己縮放成別的大小也會說明。被蓋住的視窗必須帶到前面才能用：desk-pilot 送出的是真正的輸入，會進入最上層的東西。

7. 讀取畫布或遊戲

{ "tool": "desktop_control", "action": "controls", "title": "My Game" }

<controls window='My Game' hwnd=0x1F0A22 shown=0 of=0>
(no UI Automation tree here; use screen_view)

{ "tool": "screen_view", "window": "My Game", "rulers": true }

然後點擊從圖片讀到的座標，或用 region 放大看小字。

8. 需要 confirm 的高風險批次

{ "tool": "desktop_sequence", "risk": "high", "steps": [
  { "action": "click", "target": "Delete", "role": "Button" },
  { "action": "click", "target": "Yes", "expect": { "gone": "Confirm delete" } }
] }

desktop_sequence refused before touching the desktop: this batch is high risk ...

加上 "confirm": true 才會執行。在 Claude Code 外掛中，hook 會顯示列出每個步驟的權限提示，即使該工具在你的允許清單上。"dryRun": true 只列出計畫而不執行。

9. 跨步驟按住按鍵

{ "tool": "desktop_sequence", "steps": [
  { "action": "click", "target": "a.txt" },
  { "action": "down", "key": "shift" },
  { "action": "click", "target": "f.txt" },
  { "action": "up" }
] }

按住東西時，每個結果都會顯示 HOLDING。up、Esc、被中斷的回合、holdMs（預設 30 秒）或引擎關閉都會放開。

特殊情況

• 以系統管理員身分執行的視窗。 除非 Claude Code 也以系統管理員身分執行，否則 Windows 會擋下對它們的模擬輸入。回覆是 Windows refused the input (another desktop, a lock screen, or an elevated window in front)。移動或調整它的大小也會被拒絕。
• 傳統彈出式選單與 WinForms。 desk-pilot 使用真正的滑鼠，所以這些和一般點擊一樣可用。右鍵選單會出現在指標所在的位置。
• XAML 文字方塊。 type 送出真正的按鍵，所以可用。請查看 verified 結果。
• 瀏覽器。 真正的輸入會進入前景視窗，所以共用同一程序的瀏覽器視窗不是問題。有些網頁露出的控制項很少；那裡請用截圖。
• DPI 與縮放。 x/y 一律是實體螢幕像素。引擎支援逐螢幕 DPI，每張畫面都會說明圖片像素如何精確對應回螢幕。
• 多個螢幕。 screens 會列出它們，screen_view 接受 screen: "all" 或螢幕編號。Esc 覆蓋層只涵蓋主螢幕，第二個螢幕未經測試。
• 搶走焦點的對話框。 每個結果都會回報前景視窗。用 expect: { "foreground": "..." } 等待對話框出現，用 gone 等它關閉。
• 會移動或消失的控制項。 每次點擊前都會重新查找具名目標。過期的清單會直接失敗，而不是點到移到那個位置的東西；名稱不明確時會回傳候選項。捲出畫面的控制項會被捲回來。
• 慢的 App 與逾時。 expect 預設最多等 1.5 秒（waitFor 5 秒），可用 timeoutMs 調整。每個動作有 DESK_PILOT_TIMEOUT_MS（20 秒）的時限；超過時引擎重啟，動作回報為 unknown，絕不重試。
• 批次執行到一半按下 Esc。 Esc、Stop 或 Take over 會在步驟之間停止批次，即使正在等待中。按住的按鍵和滑鼠鍵會放開，結果顯示 the person pressed Esc and cancelled this。
• 引擎當掉。 下一次呼叫會重啟它，10 分鐘內最多 3 次。超過之後，每個請求在自己的短命引擎中執行：較慢，沒有 Esc、覆蓋層和 down。下一個引擎會放開被砍掉的引擎留下的任何按住狀態。
• 工作階段中外掛更新了。 執行中的伺服器會發現新版本，並提醒你開新的工作階段。

常見問題

desk-pilot 會移動我的滑鼠嗎？ 會。它使用真正的滑鼠和鍵盤，所以它操作時不要用電腦。按 Esc 接手。

它和 deskcrew 有什麼不同？ desk-pilot 是 MCP 伺服器，使用真正的滑鼠和鍵盤，一次一個 agent。deskcrew 是在背景工作的指令，每個 agent 有自己的游標。

哪些 App 能用它？ 作為外掛的 Claude Code，或透過 cargo 安裝的 desk-pilot 使用任何 MCP 用戶端。

為什麼只有三個工具？ 每個工具的 schema 在每個工作階段都要花 token，所以各種動作都放在三個工具裡。

為什麼我的截圖沒回來？ 沒變的畫面不會重送。傳入 forceImage: true 就一定會拿到。

它能讀小字嗎？ 畫面長邊為 1356 像素。要看小字，只看單一視窗，或用 region 放大。

點錯東西怎麼辦？ 每次點擊前都會檢查具名目標，批次在第一個失敗處停止。任何具破壞性的操作，請宣告 risk，讓它需要 confirm 和你的核准。

需要 .NET 或 Visual Studio 嗎？ 不需要。它用 Windows 內建的 C# 編譯器編譯引擎，只需一次，約 0.2 秒。它需要 Node 22.19 或更新版本。

能在 Mac 或 Linux 上用嗎？ 不能，僅限 Windows。

怎麼關掉藍光和控制列？ 設定 DESK_PILOT_TAKEOVER=0。這也會關掉用 Esc 停止的功能。

常見結果與錯誤

版本與原始碼

本頁說明 desk-pilot 0.4.2。

• 原始碼：https://github.com/acus9345/desk-pilot
• crates.io：https://crates.io/crates/desk-pilot
• 授權：MIT