Docs
Install, every command, how each tool works inside, and what to do when something goes wrong.
Publish
publish-pilotpublish-pilot is an MCP server that lets Claude, or any other agent, take a project from an empty folder to a published GitHub repo and an npm, PyPI or crates.io package. It checks your tools, handles logins, scaffolds the project, runs checks, publishes and releases. You only do the sign-in, 2FA and approval clicks.publish-pilot 是一個 MCP 伺服器，讓 Claude 或任何其他 agent 把專案從空資料夾一路帶到已發布的 GitHub repo，以及 npm、PyPI 或 crates.io 套件。它會檢查你的工具、處理登入、建立專案骨架、執行檢查、發布並建立 release。你只需要登入、2FA 和按核准。
shipkitshipkit takes a project from a local folder to a public GitHub repo and a crates.io, npm or PyPI release, safely. Its ship CLI runs every release step as a separate command that prints one JSON object, so Claude Code, a script or you can drive it one step at a time. The Claude Code plugin adds a skill that knows the order of the steps, a hook that blocks the risky ones until you say yes, and a /ship-setup command. There is no MCP server.shipkit 安全地把專案從本機資料夾帶到公開的 GitHub repo，以及 crates.io、npm 或 PyPI 的 release。它的 ship CLI 把每個發布步驟做成獨立指令，各輸出一個 JSON 物件，所以 Claude Code、指令稿或你自己都能一步一步操作。Claude Code 外掛加入一個知道步驟順序的 skill、一個在你同意前擋下高風險步驟的 hook，以及 /ship-setup 指令。沒有 MCP 伺服器。
Desktop control
desk-pilot# desk-pilot
desk-pilot gives Claude Code, or any MCP client, eyes and hands on the real Windows desktop. It sees the actual screen, reads each window's controls through UI Automation, and drives the real mouse and keyboard. It has just three MCP tools on purpose, because every tool schema costs tokens in every session.
Windows only.
## Install
As a Claude Code plugin, which adds the tools, a skill and a safety hook:
bash
claude plugin marketplace add acus9345/desk-pilot
bash
claude plugin install desk-pilot@desk-pilot
Inside an interactive claude session, the same is /plugin marketplace add acus9345/desk-pilot, then /plugin install desk-pilot@desk-pilot. Start a new session afterwards.
As a standalone MCP server for any MCP client:
bash
cargo install desk-pilot
bash
claude mcp add desk-pilot -- desk-pilot
The standalone server has the three tools only: no skill, and no approval hook.
Requirements: Windows 10 2004 or later, or Windows 11, and Node 22.19 or later on PATH. On first use it compiles its engine with the C# compiler that ships with .NET Framework 4.x in Windows (about 0.2 s), then reuses it. There is no Mac version.
On Windows you can also use the installer from qlinter.org (desk-pilot-setup-0.4.2.exe). It installs the Claude Code plugin for you, and needs Claude Code installed first.
## Quick start
Ask Claude to do something on screen. It typically starts with:
json
{ "tool": "desktop_control", "action": "windows" }
json
{ "tool": "desktop_control", "action": "controls", "title": "Calculator" }
json
{ "tool": "desktop_control", "action": "click", "target": "Seven", "role": "Button" }
json
{ "tool": "screen_view", "window": "Calculator" }
## Tools and features
### screen_view
The live screen as an image with coordinate rulers. Fields: screen (primary, all or a monitor index), window (title or process), hwnd, region (x,y,w,h to zoom in), diff (only what changed), maxEdge, includeCursor, rulers, annotate (box and number the window's controls; target then takes the number), forceImage (re-send an unchanged frame).
### desktop_control
One action per call. x/y are physical screen pixels; prefer target (a control's name or index) whenever a control exists.
| Action | What it does |
| --- | --- |
| click, doubleClick, tripleClick, rightClick, middleClick | Click a target or point. modifiers: ["ctrl"] holds keys for one click; button can be back or forward; holdMs makes a long press. |
| move | Move the pointer. |
| drag | Drag to toX/toY or onto toTarget; durationMs sets the speed. |
| draw | strokes of [x, y] points, lifting between strokes; smooth curves through them. |
| down, up | Hold a mouse button or key across steps; up alone releases everything. |
| scroll | amount notches; positive is up, and nonzero toX scrolls sideways. |
| type | Type text, then read it back (verified, NOT verified or unknown). mode is paste or keys. |
| key | Keys such as enter, ctrl+s or win+r; holdMs repeats like a held key. |
| focus | Bring a window forward. |
| waitFor | Wait for an expect condition. |
| window | Move and resize a window's visible frame, or show: minimize \| maximize \| restore. |
| launch | Start an app by Start Menu name, program, path or URL, with args and cwd. |
| clipboard | Read the clipboard, or set it with text. Reads stop at 8,000 characters. |
| windows, controls, screens | List windows; list a window's operable controls as index [Role] 'Name' @ x,y WxH (positions only with bounds: true); list monitors. |
Any step can carry expect, for example { "foreground": "Save As" }, { "exists": "OK" }, { "gone": "Loading" } or { "value": "hello" }. The step waits for the condition and fails if it never holds.
### desktop_sequence
Many actions in one call, checked step by step, stopping at the first failure. Fields: steps (each with an action and desktop_control's fields), risk (low, medium or high), confirm, dryRun and capture (none, window, screen or all, for one frame at the end).
json
{ "tool": "desktop_sequence", "steps": [
{ "action": "click", "target": "File name" },
{ "action": "type", "text": "notes.txt" },
{ "action": "key", "key": "enter", "expect": { "gone": "Save As" } }
], "risk": "low" }
## How it works
dist/server.mjs is a stdio MCP server in Node. It drives one long-lived engine process written in C# (engine/*.cs) over JSON lines. The engine keeps per-monitor DPI awareness, the UI Automation client and the Esc hotkey warm. If the engine dies it is restarted on the next call, at most 3 times in 10 minutes. After that, each request runs in its own short-lived engine, so the tools keep working, only slower.
Before clicking a named control, the engine looks it up again. A disabled control is refused, one scrolled out of view is scrolled into view, an ambiguous name returns the candidates, and a stale listing fails instead of clicking whatever moved into that slot. Input uses real mouse and keyboard events, and a click stays down for 40 ms, because canvases such as Paint's ignore instant clicks.
Frames are shrunk to 1356 px on the long side by the engine, so the coordinate line under each image is exact. An identical frame is not sent again, and the frame after a batch carries only the region that changed. End frames wait until the screen stops changing.
While Claude is acting, the screen shows it: a blue glow around the primary monitor, a pointer marker with an optional label, and a control bar at the bottom. Esc, Stop or Take over stops the batch between steps, even during a wait. None of this takes focus, and it clears itself for the moment of each screenshot.
A desktop_sequence declared medium or high risk is refused unless it carries confirm: true. In the plugin, a PreToolUse hook then turns that into a Claude Code permission prompt listing every step.
## Settings
Environment variables, for example in the env block of Claude Code's settings.json:
| Variable | Default | Meaning |
| --- | --- | --- |
| DESK_PILOT_IMAGE_MAX_EDGE | 1356 | Longest side of a delivered frame |
| DESK_PILOT_IMAGE_MAX_PIXELS | 1150000 | Pixel budget of a delivered frame |
| DESK_PILOT_MAX_SEQUENCE_STEPS | 24 | Steps per batch |
| DESK_PILOT_STEP_SETTLE_MS | 100 | Pause after each batch step |
| DESK_PILOT_CAPTURE_SETTLE_MS | 1000 | Longest wait for the screen to settle before a frame |
| DESK_PILOT_TIMEOUT_MS | 20000 | Per action; past it the engine restarts and the action is reported as unknown |
| DESK_PILOT_DEDUP_TTL_MS | 300000 | How long an unchanged frame is not re-sent |
| DESK_PILOT_TAKEOVER | on | The overlay and Esc hotkey; 0 turns them off |
| DESK_PILOT_CAPTION | English | Second line of the control bar |
| DESK_PILOT_OVERLAY_IN_FRAMES | off | 1 lets the overlay into frames |
The plugin keeps its compiled engine in the plugin data folder Claude Code provides (DESK_PILOT_DATA_DIR).
## Limits and troubleshooting
- Windows only.
- desk-pilot moves your real mouse and uses your real keyboard, so don't use the computer while it acts. Press Esc to stop it.
- Windows blocks injected input into windows running as administrator, unless Claude Code also runs as administrator.
- Canvas-drawn and some Electron windows expose few or no controls; use frames there.
- Esc and the overlay are unavailable once the engine has fallen back to one-shot mode, and down is refused there.
- The overlay covers the primary monitor only, and a second monitor was not tested.
- No pen pressure and no multi-touch.
- The back and forward mouse buttons were not confirmed end to end; key: "alt+left" does the same job in most apps.
- A key held with down does not auto-repeat; key with holdMs does.
- "Node not found": install Node 22.19 or later and make sure it is on PATH, then start a new session.
## Worked examples
Tool results below use desk-pilot's real format. Window handles, indexes, sizes and coordinates differ on every computer, so read the shape, not the numbers. desk-pilot drives your real mouse and keyboard, so leave the computer alone while it works.
### 1. Calculator: 1 + 2 =
json
{ "tool": "desktop_control", "action": "controls", "title": "Calculator" }
text
<controls window='Calculator' hwnd=0x2A0B4C shown=31 of=37>
0 [Button] 'One'
1 [Button] 'Two'
...
json
{ "tool": "desktop_sequence", "steps": [
{ "action": "click", "target": "One", "role": "Button" },
{ "action": "click", "target": "Plus" },
{ "action": "click", "target": "Two", "role": "Button" },
{ "action": "click", "target": "Equals", "expect": { "exists": "Display is 3" } }
] }
text
<batch done=4/4 ms=1180>
now: fg 'Calculator'
A clean batch reports only a count and the final state. If a step fails, the batch stops there and names it.
### 2. Fill a form and save through Save As
json
{ "tool": "desktop_sequence", "risk": "low", "steps": [
{ "action": "focus", "title": "Untitled - Notepad" },
{ "action": "type", "text": "Shopping list: milk, eggs" },
{ "action": "key", "key": "ctrl+s", "expect": { "foreground": "Save As" } },
{ "action": "click", "target": "File name:", "role": "Edit" },
{ "action": "type", "text": "list.txt" },
{ "action": "key", "key": "enter", "expect": { "gone": "Save As" } }
], "capture": "window" }
text
<batch done=6/6 ms=2300>
now: fg 'list.txt - Notepad'
Each expect makes its step wait for the dialog to appear or close, and fail if it never does. capture: "window" adds one frame of the front window, with only the changed area.
### 3. Draw in Paint
json
{ "tool": "screen_view", "window": "Paint", "rulers": true }
json
{ "tool": "desktop_control", "action": "draw", "smooth": true,
"strokes": [[[600,400],[800,400],[800,600],[600,600],[600,400]], [[650,450],[750,550]]] }
text
draw ok · drew 2 strokes, 1083 px · fg 'Untitled - Paint'
Strokes move at about 1000 px/s and lift the button between strokes, because canvases such as Paint's ignore instant jumps. The frame's coordinate line tells you how image pixels map to screen pixels.
### 4. Type into a browser
json
{ "tool": "desktop_sequence", "steps": [
{ "action": "click", "target": "Email", "role": "Edit" },
{ "action": "type", "text": "ada@example.com", "expect": { "value": "ada@example.com" } },
{ "action": "click", "target": "Sign up", "role": "Button", "expect": { "exists": "Welcome" } }
] }
type reads the text back and reports verified, NOT verified or unknown. If the browser exposes few controls, take a screen_view and click by x/y instead.
### 5. Menus and combo boxes
json
{ "tool": "desktop_sequence", "steps": [
{ "action": "click", "target": "File", "role": "MenuItem" },
{ "action": "click", "target": "Page setup...", "expect": { "foreground": "Page Setup" } },
{ "action": "click", "target": "Size", "role": "ComboBox" },
{ "action": "click", "target": "A4", "role": "ListItem" },
{ "action": "click", "target": "OK", "expect": { "gone": "Page Setup" } }
] }
Open the menu and pick from it in one batch. Each click looks the control up again just before pressing it.
### 6. Arrange a window, including a minimised one
json
{ "tool": "desktop_control", "action": "window", "title": "Calculator", "show": "restore" }
json
{ "tool": "desktop_control", "action": "window", "title": "Calculator", "x": 100, "y": 100, "width": 400, "height": 600 }
text
window ok · window 'Calculator' @ 100,100 400x600
The size is the window's visible frame, without Windows' invisible resize border. desk-pilot checks the result and says so if the app snapped to a different size. A covered window must be brought forward to be used: desk-pilot sends real input, which goes to whatever is on top.
### 7. Read a canvas or game
json
{ "tool": "desktop_control", "action": "controls", "title": "My Game" }
text
<controls window='My Game' hwnd=0x1F0A22 shown=0 of=0>
(no UI Automation tree here; use screen_view)
json
{ "tool": "screen_view", "window": "My Game", "rulers": true }
Then click the coordinates you read off the image, or zoom in with region for small text.
### 8. A risky batch with confirm
json
{ "tool": "desktop_sequence", "risk": "high", "steps": [
{ "action": "click", "target": "Delete", "role": "Button" },
{ "action": "click", "target": "Yes", "expect": { "gone": "Confirm delete" } }
] }
text
desktop_sequence refused before touching the desktop: this batch is high risk ...
Add "confirm": true to run it. In the Claude Code plugin, a hook then shows you a permission prompt listing every step, even if the tool is on your allow list. "dryRun": true lists the plan without running it.
### 9. Hold a key across steps
json
{ "tool": "desktop_sequence", "steps": [
{ "action": "click", "target": "a.txt" },
{ "action": "down", "key": "shift" },
{ "action": "click", "target": "f.txt" },
{ "action": "up" }
] }
While something is held, every result says HOLDING. It is released by up, Esc, an interrupted turn, holdMs (30 s by default) or the engine closing.
## Edge cases
- Windows running as administrator. Windows blocks injected input into them unless Claude Code also runs as administrator. The reply says Windows refused the input (another desktop, a lock screen, or an elevated window in front). Moving or resizing one is refused too.
- Classic popup menus and WinForms. desk-pilot uses the real mouse, so these work like any click. Right-click menus appear where the pointer is.
- XAML text boxes. type sends real keystrokes, so these work. Check the verified result.
- Browsers. Real input goes to the front window, so browser windows sharing one process are not a problem. Some pages expose few controls; use frames there.
- DPI and scaling. x/y are always physical screen pixels. The engine is per-monitor DPI aware, and each frame says exactly how image pixels map back to the screen.
- Several monitors. screens lists them, and screen_view takes screen: "all" or a monitor index. The Esc overlay covers only the primary monitor, and a second monitor has not been tested.
- Modal dialogs that take focus. Each result reports the window in front. Use expect: { "foreground": "..." } to wait for a dialog, and gone to wait for it to close.
- Controls that move or vanish. A named target is looked up again before each click. A stale listing fails instead of clicking whatever moved into that slot, and an ambiguous name returns the candidates. A control scrolled out of view is scrolled back into view.
- Slow apps and timeouts. expect waits up to 1.5 s by default (waitFor 5 s), and timeoutMs changes that. Each action has DESK_PILOT_TIMEOUT_MS (20 s). Past it, the engine is restarted and the action is reported as unknown, never retried.
- Esc in the middle of a batch. Esc, Stop or Take over stops the batch between steps, even during a wait. Held keys and buttons are released, and the result says the person pressed Esc and cancelled this.
- The engine crashing. It restarts on the next call, at most 3 times in 10 minutes. After that, each request runs in its own short-lived engine: slower, with no Esc, overlay or down. The next engine releases anything a killed one left held.
- A plugin update during a session. The running server notices the new version and tells you to start a new session.
## FAQ
Does desk-pilot move my mouse?
Yes. It uses the real mouse and keyboard, so don't use the computer while it acts. Press Esc to take over.
How is it different from deskcrew?
desk-pilot is an MCP server using the real mouse and keyboard, one agent at a time. deskcrew is a command that works in the background, with a cursor per agent.
Which apps can use it?
Claude Code as a plugin, or any MCP client through desk-pilot from cargo.
Why only three tools?
Every tool schema costs tokens in every session, so the actions live inside three tools.
Why didn't my screenshot come back?
An unchanged frame isn't sent again. Pass forceImage: true to get it anyway.
Can it read small text?
Frames are 1356 px on the long side. For small text, view one window or zoom with region.
What happens if it clicks the wrong thing?
Named targets are checked before each click, and a batch stops at the first failure. For anything destructive, declare risk so it needs confirm and your approval.
Does it need .NET or Visual Studio?
No. It compiles its engine with the C# compiler built into Windows, once, in about 0.2 s. It needs Node 22.19 or later.
Does it work on a Mac or Linux?
No, Windows only.
How do I turn off the glow and control bar?
Set DESK_PILOT_TAKEOVER=0. That also turns off Esc to stop.
## Common results and errors
| Result | Meaning | What to do |
| --- | --- | --- |
| <batch done=3/5 failed=1 ...> with #4 click FAILED ... | Step 4 failed, so the batch stopped. | Read the step's error, fix it and run the rest. |
| no control named '...' | Nothing with that name is on screen. | Run controls, or use role or an index. |
| ... is not the one that was listed: expected [...] | The listing is stale: a different control now sits at that index. | List controls again. |
| expectation not met within ... ms | The expect condition never held. | Raise timeoutMs, or check the step did what you meant. |
| no window titled '...' / no visible window matches '...' | The window isn't open or isn't visible. | Run windows to see the titles. |
| ... no longer exists; list windows again | The window closed. | List windows again. |
| Windows refused the input (another desktop, a lock screen, or an elevated window in front) | Input was blocked. | Unlock the screen, or close or elevate past the admin window. |
| desktop_sequence refused before touching the desktop: this batch is high risk ... | A risky batch had no confirm. | Ask the person, then add confirm: true. |
| the person pressed Esc and cancelled this | The person stopped it. | Ask before continuing. |
| unknown: ... | The outcome can't be known (engine timeout or crash). | Look with screen_view before trying again. |
| NOT verified after type | The text read back doesn't match. | Check the field. Try mode: "keys" or "paste". |
| the clipboard is locked by another application ... | Paste typing couldn't use the clipboard. | Retry with mode: "keys". |
| down is unavailable in one-shot mode ... | The engine fell back after crashes. | Start a new session. |
| step N has unknown field ... | A misspelled field in a batch step. | Fix the field name. Nothing ran. |
| desk-pilot needs the .NET Framework 4.x C# compiler (csc.exe) ... | The built-in compiler is missing. | Repair .NET Framework 4.x in Windows Features. |
## Version and source
This page describes desk-pilot 0.4.2.
- Source: https://github.com/acus9345/desk-pilot
- crates.io: https://crates.io/crates/desk-pilot
- License: MIT
desk-pilot 讓 Claude Code 或任何 MCP 用戶端，在真正的 Windows 桌面上有眼睛也有手。它看得到實際畫面，透過 UI Automation 讀取每個視窗的控制項，並操作真正的滑鼠和鍵盤。它刻意只有三個 MCP 工具，因為每個工具的 schema 在每個工作階段都要花 token。
deskcrewdeskcrew lets coding agents operate real Windows apps in the background. Each agent gets its own drawn cursor, in its own colour with its name beside it, and agents work side by side while you keep using your computer. It is one command-line program, so any agent that can run a shell command can use it, and it is built to cost as few tokens as possible.deskcrew 讓 coding agent 在背景操作真正的 Windows App。每個 agent 都有自己畫出來的游標，顏色各不相同，旁邊標著名字；agent 們並排工作，你照常使用電腦。它是單一個命令列程式，所以任何能執行 shell 指令的 agent 都能用，而且設計成盡量少花 token。
Apps
ClawdClawd is a pixel-art desktop pet for Windows. It lives on top of your taskbar, wanders about, waves, dances, writes code on a tiny laptop, naps and does backflips, and it plays ping pong, badminton and chess with you.Clawd 是 Windows 上的像素風桌面寵物。它住在工作列上方，到處走動、揮手、跳舞、在小筆電上寫程式、打盹、後空翻，還會陪你打乒乓球、羽毛球和下西洋棋。
Color PickerProject Color Picker is a small desktop colour picker with 14 colour modes, picking from anywhere on screen, and saved swatches. It's for when you don't want to open a browser every time you need a colour value.Project Color Picker 是一個小巧的桌面取色器，有 14 種色彩模式、可以從螢幕任何地方取色，還能存色票。適合不想每次需要色碼都打開瀏覽器的時候。
singbox-watchsingbox-watch is a Windows tray controller, watchdog and fail-safe toolkit for a sing-box tunnel with a static exit IP. If your computer routes all traffic through a SOCKS5 or TUN tunnel and you need to be sure it never silently falls back to the raw connection, this is that toolkit. Every feature answers a failure that actually happened.singbox-watch 是 Windows 上的系統匣控制器、監看程式和防護工具組，用於具有固定出口 IP 的 sing-box 通道。如果你的電腦把所有流量都經由 SOCKS5 或 TUN 通道送出，而你需要確定它絕不會悄悄退回原始連線，這就是你要的工具組。每個功能都對應一次真實發生過的故障。
PkgStatsPkgStats is a native desktop app (Rust and egui) that shows developer and package stats from GitHub, npm, PyPI and crates.io. Look up a user on every registry at once, or a single package or repo, with charts, and follow the ones you care about day by day.PkgStats 是一個原生桌面 App（Rust 與 egui），顯示來自 GitHub、npm、PyPI 和 crates.io 的開發者與套件統計。一次在所有套件庫查詢某位使用者，或查詢單一套件或 repo，附圖表，並逐日追蹤你關心的項目。