deskcrew
deskcrew lets coding agents operate real Windows apps in the background. Each agent gets its own drawn cursor, in its own colour with its name beside it, and agents work side by side while you keep using your computer. It is one command-line program, so any agent that can run a shell command can use it, and it is built to cost as few tokens as possible.
Windows is the tested platform. A macOS version is included but has never been run on a Mac.
Install
From crates.io (needs Rust):
cargo install deskcrew
Then make it available to your coding agents:
deskcrew setup
setup copies the program to a folder on your PATH and sets up every agent app and CLI it finds, so they use deskcrew on their own whenever a task needs the desktop, instead of built-in computer use or other desktop tools. Run it again after installing a new agent: it updates what it wrote instead of adding copies, and leaves the rest of each file alone.
| Agent (app and CLI) | What setup writes |
|---|---|
| Claude Code | skill in ~/.claude/skills, rule in ~/.claude/CLAUDE.md |
| Codex | skill in ~/.codex/skills, rule in ~/.codex/AGENTS.md |
| Gemini CLI and Antigravity | skill in ~/.gemini/skills, rule in ~/.gemini/GEMINI.md |
| GitHub Copilot CLI | skill in ~/.copilot/skills, rule in ~/.copilot/copilot-instructions.md |
| GitHub Copilot in VS Code | rule in VS Code's user prompts/deskcrew.instructions.md |
| Windsurf | skill, and rule in ~/.codeium/windsurf/memories/global_rules.md |
| OpenCode | skill, and rule in ~/.config/opencode/AGENTS.md |
| Amp, Qwen Code, Factory Droid | rule in ~/.config/amp/AGENTS.md, ~/.qwen/QWEN.md, ~/.factory/AGENTS.md |
| Kiro, Continue, Cline | rule file in ~/.kiro/steering, ~/.continue/rules, ~/Documents/Cline/Rules |
| Cursor | skill in ~/.cursor/skills. Its User Rules live in your Cursor account, so setup prints the rule to add under Customize > Rules > New |
Grok, and agents sharing ~/.agents |
skill |
The rule sits between <!-- deskcrew: start --> and <!-- deskcrew: end --> lines. For any other agent, add the rule setup prints to its own rules.
To operate Windows apps on this computer, run the deskcrew command. Start with: deskcrew help
On Windows you can use the installer from qlinter.org instead (deskcrew-setup-0.1.8.exe). It installs deskcrew.exe, and its optional "Set up deskcrew for my coding agents" task runs deskcrew setup for you. On a Mac, the same two commands work; macOS will ask for Accessibility permission, and for Screen Recording before the first picture. Agents running in the cloud, or in a sandbox without access to your desktop, cannot reach it.
Quick start
deskcrew windows
deskcrew see Calculator
w2 "Calculator" 37 controls, 1 off screen
e12 button "One"
...
deskcrew click e12
ok invoke
~ e4 "Display is 1"
Every control gets a ref such as e12, and refs stay valid across reads, so a window is read once. Every action replies with only what changed. A background service starts on first use and exits after 30 idle minutes.
To see the cursors without touching anything, run deskcrew demo --seconds 12 (Windows only).
Commands
Pick a window with -w name or -w w2 on any command; otherwise the agent's last window is used.
Reading
| Command | What it does |
|---|---|
windows [filter] |
List windows as w1, w2... |
see [window] [-f] [-a] [--find word] [--from N] |
List controls as e1, e2... Later calls show only changes. -f lists everything again, -a includes off-screen controls, --find filters, --from pages. If a large part of the window has no controls (a canvas or game), see says so, reads any text there with Windows OCR and gives its x,y. |
get e3 |
A control's full text. Password fields are refused. |
is e3 on|off|open|closed|selected|enabled|disabled |
Check a state. |
shot [window|e3] [-f] [--marks] [--rulers] [--screen [-a]] [--region x,y,w,h] [--changed] [--edge N] |
Save a PNG and print its path. Replies unchanged if it matches the last one. --marks numbers controls by ref, --rulers labels pixels, --changed crops to what changed. |
apps, screens |
Installed apps; monitors. |
status |
Version and the agents seen. |
deskcrew shot -w Paint --marks
Acting
| Command | What it does |
|---|---|
click e3 [e4 ...] | "Name" | x,y |
Click. Flags: --right, --double, --triple, --middle, --back, --forward, --with ctrl+shift. Several refs click in turn; click File Save opens a menu and picks from it. |
hover e3 | x,y |
Move the drawn cursor there and hover. |
check, uncheck, toggle, expand, collapse, clear e3 |
Set a control's state; check and uncheck leave it alone if it is already that way. |
set e3 "value" |
Replace a field's text. |
type [e3] "text" [--paste] |
Type text; --paste pastes it. |
key enter [--hold ms --real] |
Press keys. Risky shortcuts such as alt+f4 need --force. |
select e3 "Item" |
Pick from a list or combo box. |
scroll [e3] up|down [n], show e3 |
Scroll, or scroll a control into view. |
wait "text" [--gone] [--ms N] |
Wait until text shows or goes. |
drag a b |
Drag from one ref or point to another. |
draw [e3] "x,y x,y; x,y x,y" [--smooth] |
Draw strokes, separated by ;. |
down shift --real, down left x,y --mouse, up |
Hold a key or button; released by up, after 30 s, on quit and on Ctrl+Alt+Esc. |
do "click e1; set e2 hi; key enter" [--dry] [--risk low|medium|high] |
Many steps, one reply. --dry shows the plan; --risk high is refused until the person agrees and --force is added. |
launch app [args] [--front], focus, close, place x,y,w,h|min|max|restore |
Start an app (behind yours unless --front), bring a window forward, close it, or move and resize it. |
notifications [close] |
Open the notification panel and read it, then close it. |
clip, clip set "text", clip clear |
Read, set or clear the clipboard. |
resume, quit |
Continue after Ctrl+Alt+Esc; stop the service. |
cursor [off|ask|session|always] |
Who may borrow your real mouse cursor. |
Any action takes --expect "text", which fails unless the text shows afterwards.
Other flags: --agent name (who is acting), --fast (no glide), --force (allow risky keys), --real (bring the window forward briefly and use the real keyboard or a synthetic pen: a last resort), --mouse (ask to borrow your cursor: only if nothing else works).
deskcrew do "set e2 'hello'; click OK" --expect "Saved" -w Notes
When an action changes no control, the reply says whether the window's picture changed and where, so an agent rarely needs a screenshot to check its work.
How it works
deskcrew is a small client. It hands its words to a background service over a named pipe, starting the service if needed, and prints the reply. The service keeps each window's controls and refs in memory, so later reads and replies only carry changes. It runs one queue per window, so agents in different apps work at the same time, and agents sharing a window take turns.
Actions try background routes first, and each reply names the route it used. invoke, toggle, select, expand, value, range and scroll are the app's own UI Automation actions. command and message post window messages to classic controls, and to browser and Electron windows. pen is a synthetic pen for apps that draw themselves, such as Paint. keyboard is the real keyboard, used with --real. mouse is your real cursor, used only with --mouse and only if you allow it. A low-level shield blocks synthetic mouse events while accessibility requests run, because some classic controls answer by quietly clicking with the real mouse.
The drawn cursors and press outlines are layered, click-through windows kept directly above the window each agent works in, and above that window's menus and popups. Whatever covers the app covers its cursor too. Each agent is recognised from its environment or the program that started it, and gets its own colour and name. An unknown one is called "your agent".
Pictures are taken from the window itself, so it can sit behind other windows. Text in parts with no controls is read with the OCR built into Windows.
The macOS version uses the same commands and replies, and since 0.1.6 nearly the same command set. It reads through the accessibility API and presses controls with their own actions, sends clicks, drags and keys to the target app only, takes pictures with Core Graphics, reads text with Apple's Vision, and draws cursors as see-through AppKit windows. On a Mac, --real brings the app forward briefly and sends keys through the system, then gives the front back.
Compared with other tools
Measured on 2026-11-03 on the same machine, reading the same four throwaway windows three times with each tool. These are characters an agent has to read, not tokens, and only reading was measured, not acting.
| Reading a window | deskcrew 0.1.7 | agent-cu 0.1.1 | mcp-windows 1.3.28 | Windows-MCP 0.8.8 |
|---|---|---|---|---|
| Test form | 488 | 1,775 | 7,089 | 5,932 |
| Character Map | 485 | 823 | 3,760 | 4,582 |
| Calculator | 689 | 2,199 | 8,212 | 6,444 |
| File Explorer | 3,381 | 8,938 | 26,635 | 13,193 |
| Time per read after the first | 20 to 102 ms | 0.1 to 3.9 s | 27 to 200 ms | about 15 s |
| What an agent loads to learn the tool | 2,055 characters of help | a skill | 58,165 characters of tool schemas | 25,054 characters of tool schemas |
How each tool was called:
- deskcrew:
see -f. - agent-cu:
snapshot -i -c, its recommended compact mode. Its npm package has no Windows program, so it was built from the GitHub source. Its read time swung between about 70 ms and 4 s from run to run. - mcp-windows:
ui_snapshotin full mode. Its incremental mode makes repeated reads cheaper and was not measured. - Windows-MCP:
Snapshotwithout a picture. It always reads the whole desktop, other windows and taskbar included, so its figures are not for one window alone. It waits while the person is using the computer, and still took about 15 s per read after 12 idle seconds.
Files, paths and settings
| Item | Where |
|---|---|
| Cursor permission | %LOCALAPPDATA%\deskcrew\cursor.txt (lines such as ask or always Claude Code) |
| Pictures | %LOCALAPPDATA%\deskcrew\shots (Mac: ~/Library/Caches/deskcrew/shots). They can show private content. |
| Service pipe | \\.\pipe\deskcrew-<version>-<session> (Mac: a socket in $TMPDIR) |
DESKCREW_AGENT |
Names the agent, for separate cursors when one agent runs several copies. --agent "Claude 2" does the same. |
Borrowing your cursor is off by default. deskcrew cursor ask lets agents ask with an on-screen prompt: Just this once, This session, Always or No. No answer within a minute counts as no. Any setting except off must be accepted on screen. This guards against accidents, not against programs running as you.
Ctrl+Alt+Esc stops every agent (Ctrl+Option+Esc on a Mac). Reading still works, and deskcrew resume continues.
Limits and troubleshooting
- Windows is the tested platform. The macOS version has never been run on a Mac; it is only type-checked. On a Mac,
--mouseand borrowing the cursor, holding a mouse button,notificationsand the demo are not written yet, anddownholds only modifier keys (shift, ctrl, alt, cmd), which then apply to that agent's clicks and keys in that app. Linux is not supported. notifications,downand--realtake focus, and so doestype --pasteoutside a classic text box. While a--realaction runs, anything you type goes to that window.- Keyboard shortcuts, rich text boxes such as Notepad's, and typing into XAML text boxes need
--real. - WinForms ignores posted clicks while covered, so use
--realthere. A WinForms canvas needs--mouseto drag. - Browser windows of the same browser share a process and lose input when worked at the same time. Use separate apps or browser profiles for parallel agents.
- A covered browser stops repainting, so
shotcan show an old picture of it. Drawing in it still works. - Classic popup menus only answer the real mouse; without your permission they are refused.
- Apps running as administrator can be read but not driven, unless deskcrew also runs as administrator.
- Tested on one monitor at 150% scaling.
- Text read from a picture can confuse similar letters such as
Iandl. - "add --real": the action needs the window in front. "nothing changed: try --real if it should have": the app ignored the background message. "stopped by the person": someone pressed Ctrl+Alt+Esc; ask them, then run
deskcrew resume.
Worked examples
Replies below use deskcrew's real format. Refs (e12), window numbers (w2), counts and coordinates differ on every computer, so read the shape, not the numbers.
1. Calculator: 1 + 2 =
deskcrew see Calculator
w2 "Calculator" 37 controls, 1 off screen
e4 "Display is 0"
e12 button "One"
e13 button "Two"
e20 button "Plus"
e24 button "Equals"
...
deskcrew click e12 e20 e13 e24
ok 4 steps
~ e4 "Display is 3"
Four presses, one reply, and only the line that changed. The buttons are pressed through accessibility, so Calculator can stay behind your other windows.
2. Fill a form and save through a Save As dialog
deskcrew see "Untitled - Notepad"
deskcrew key ctrl+s --real
ok keyboard
+ window "Save As"
+ e40 edit "File name:"
+ e41 button "Save"
...
deskcrew do "set e40 'notes.txt'; click e41" --expect "notes.txt"
ok 3 steps
- e40-e41
~ e1 "notes.txt - Notepad"
The dialog's controls arrive as + lines, because dialogs belong to the window. --expect adds a final wait, so the reply fails if the title never shows the new name. Notepad's text area is a rich text box, so typing into it needs --real. Many apps take set instead.
3. Draw in Paint or Photon Studio
Take a shot first, so points can be given in picture pixels:
deskcrew shot -w Paint
C:\Users\you\AppData\Local\deskcrew\shots\Claude-Code-w3.png 1356x848 scale 1.42
deskcrew draw "300,300 500,300 500,500 300,500 300,300; 350,350 450,450" --smooth -w Paint
ok pen
controls unchanged; the picture changed at 420,415 290x290
Paint draws itself, so deskcrew uses a synthetic pen there. That brings Paint forward while it draws. In browser and Electron apps such as Photon Studio, the route is message: strokes go in the background, even while the window is covered. You can also draw relative to a control: draw e7 "10,10 200,10".
4. Type into a browser or Electron app
deskcrew see "Sign up"
deskcrew type e5 "Ada Lovelace"
ok message
~ e5 edit "Name" =Ada Lovelace
If Chromium drops characters, type reads the field back and sets it directly. Keep the window showing for this route. A covered browser drops some window messages, and then deskcrew falls back to accessibility, which can take focus for a moment.
5. Menus and combo boxes
deskcrew click File "Save As..." -w notepad
ok command
+ window "Save As"
...
deskcrew select e9 "Landscape"
ok select
~ e9 combobox "Orientation" =Landscape
click File Save opens a menu and picks from it in one command, which matters for modern menus: one left open between commands closes when focus returns. Classic popup menus answer only the real mouse, so they are refused unless you allow borrowing the cursor.
6. A covered or minimised window
Most routes work while the window is covered: accessibility actions, and background strokes in browsers. WinForms is the exception, and needs --real when covered. A minimised window has to be restored first. Windows may bring it forward as it does:
deskcrew place restore -w Calculator
ok
shot refuses a minimised window with that window is minimized: place restore first.
7. Read a canvas or game
deskcrew see "Crew Canvas"
w1 "Crew Canvas" 1 controls
e1 button "Paint"
no controls in 0,125 878x540; text read there (x,y for click):
"Total 42" 129,231
"Start game" 560,453
for drawings: shot --region 1642,1021,878,540
deskcrew click 560,453
see notices the large area with no controls and reads its text with Windows OCR. The x,y it gives work with click straight away, so a picture is needed only for drawings, and then only of that area.
8. Two or three agents at once
Each agent names itself, or is recognised from its program:
deskcrew --agent "Claude" do "click e12 e20 e13 e24" -w Calculator
deskcrew --agent "Cursor" draw "100,100 400,300" -w "Photon Studio"
deskcrew --agent "Codex" type e3 "meeting notes" -w Notes
Run from three agents at the same time, each gets its own coloured cursor with its name, and the three windows are worked in parallel. Agents in the same window take turns; in a browser canvas, their strokes are cut into short pieces that alternate. Windows of one browser share a process, so give parallel agents separate apps or browser profiles.
9. Check a result without a picture
deskcrew click e7 -w Paint
ok invoke
controls unchanged; the picture changed at 14,53 54x17
When no control changes, the reply still says whether the window's picture changed and where. nothing changed, on screen either means the click had no visible effect.
Edge cases
- Windows running as administrator. They can be read but not driven, because Windows blocks a normal program from sending them input. The reply is
blocked: the window runs as administrator. Run deskcrew from an elevated terminal if you must drive one. - Classic popup menus. They answer only the real mouse. deskcrew refuses with
this control only answers the real mouseunless you have allowed borrowing the cursor (deskcrew cursor ask). Try the keyboard orclick x,yfirst. - WinForms. WinForms ignores posted clicks while another window covers it, so add
--real. Its canvases don't see a posted drag as a drag, so drawing there needs--mouse. Checkboxes take focus for a moment before giving it back.click --with ctrlneeds--real, because WinForms reads the keyboard itself. - XAML text boxes (modern apps).
setworks, buttypeneeds--real. - Browsers sharing one process. Two windows of the same browser activate each other when worked at once, which loses input. Use separate browser profiles or different apps.
- DPI and scaling. All coordinates are physical screen pixels.
x,yinclickanddraware pixels in the last shot, which deskcrew maps back through the shot's scale. Tested at 150% scaling. - Several monitors.
screenslists them andshot --screen -acaptures all of them. Only one monitor has been tested. - Modal dialogs that take focus. New dialogs show in the reply as
+ window "...", and their controls join the window's list. If an app takes focus while agents work, deskcrew hands it straight back to your window. - Controls that move or vanish. Refs stay valid across reads, even when an app rebuilds its controls. A ref that is really gone gives
e12 is gone: see. Read again withseeand use the new ref. - Slow apps and timeouts. Each action waits up to 0.8 s for the window to settle before replying. Use
wait "text"(up to 60 s with--ms) or--expectfor anything slower. Chromium builds its control tree on first request, so the firstseeof a browser can take a few seconds. - Ctrl+Alt+Esc in the middle of a batch. Every agent stops between steps, typing stops mid-text, and held keys are let go. The reply says
stopped by the person with Ctrl+Alt+Escand lists what was done. Reading still works;deskcrew resumelets work continue. - The service crashing or exiting. The next command starts a new service by itself. Refs and remembered reads are lost, so run
seeagain. Held keys are released onquitand on idle exit, but if the service is killed outright a held key stays down until you press it. - Focus moving during
--realtyping. Typing stops at once, and the reply says how many characters went in.
FAQ
Does deskcrew move my mouse? No, unless you allow it. Borrowing the cursor is off by default. Even when allowed, an agent must ask, and the cursor is put back afterwards.
Can I keep working while agents work?
Yes. Background routes don't take your mouse or keyboard. Only --real, down, notifications, launch --front and some type --paste uses take focus, briefly, and they wait for a pause in your typing.
Which agents does it work with?
Any that can run a shell command on your computer. deskcrew setup sets up Claude Code, Codex, Gemini CLI and Antigravity, Copilot (CLI and VS Code), Windsurf, OpenCode, Amp, Qwen Code, Factory Droid, Kiro, Continue, Cline, Grok and Cursor, so they use deskcrew without being asked. In a test, Cursor's agent, asked only to work out 12 × 7 in Calculator, chose deskcrew by itself and answered 84.
Why is a cursor sometimes hidden? Cursors sit just above their agent's window, so whatever covers that window covers its cursor too. The work carries on.
How do I stop everything?
Press Ctrl+Alt+Esc. Then deskcrew resume continues, or deskcrew quit stops the service.
How much does it cost in tokens? Very little. A window reads in 260 to 3,375 characters in our tests, and later replies list only changes. Pictures are files the agent opens only when it needs one.
Does it work on a Mac? A Mac version is included, with nearly all the Windows commands since 0.1.6, but it has never been run on a Mac. Treat it as a first draft.
Where are screenshots saved?
In %LOCALAPPDATA%\deskcrew\shots, one per agent and window. Each new one replaces the last, and they can show private content.
Why did deskcrew refuse a shortcut?
Shortcuts that close or lock things, such as Alt+F4 or Win+R, need --force. An agent should ask you before adding it.
Does it need internet or a server? No. Everything runs on your computer, and the OCR is offline.
How do I uninstall it?
Run deskcrew quit, then cargo uninstall deskcrew, or remove it in Windows Settings if you used the installer. Delete %LOCALAPPDATA%\deskcrew, the deskcrew folder in each agent's skills folder, the text between the <!-- deskcrew: start --> and <!-- deskcrew: end --> lines in each agent's rules file, the rule files named deskcrew.md or deskcrew.instructions.md, and the rule in Cursor's User Rules.
Common replies and errors
| Reply | Meaning | What to do |
|---|---|---|
no window yet: run deskcrew windows, then deskcrew see <name> |
This agent hasn't picked a window. | Run windows, then see the one you want. |
no window matches '...' |
No window title or app contains that text. | Check deskcrew windows. |
'...' matches 3 windows: |
The name is ambiguous. | Use a window number such as -w w4. |
that window closed: run deskcrew windows |
The window is gone, or its handle was reused. | List windows again. |
e12 is gone: see |
That control no longer exists. | Run see and use the new ref. |
no control named '...': see / '...' matches 2 controls: |
No control, or several, have that name. | Use a ref from see. |
that control is disabled |
It can't be used right now. | Do what enables it first. |
no background way to do this here: add --real |
The app ignores background input for this. | Add --real, which brings the window forward briefly. |
nothing changed: try --real if it should have |
The app took the message but nothing changed. | Check the target, or try --real. |
controls unchanged; the picture changed at x,y wxh |
Something was drawn or repainted with no control change. | Usually success for drawing and canvases. |
this control only answers the real mouse: ... |
A classic control needs the physical mouse. | Try click x,y or the keyboard. Otherwise the person must allow --mouse. |
the person has not let agents borrow the cursor. ... |
--mouse is off. |
The person can run deskcrew cursor ask. |
the person said no to borrowing the cursor: find another way |
The person declined the prompt. | Use another route. |
refused: x,y on screen is not on this window |
A point falls outside the target window. | Take a new shot and recheck the coordinates. |
declared risky: ask the person, then add --force |
A risky shortcut or --risk high batch. |
Ask the person first. |
stopped by the person with Ctrl+Alt+Esc: ... |
The person stopped all agents. | Ask them, then deskcrew resume. |
stopped: focus moved away from that window, ... |
--real typing was interrupted. |
Check the text, then continue from where it stopped. |
x,y are pixels in a shot: take one first |
A point was given before any shot or OCR read. | Run shot or see first. |
blocked: the window runs as administrator |
Windows blocks input to elevated windows. | Run deskcrew elevated, or do it yourself. |
that is a password field: its text stays hidden |
get never reads passwords. |
Expected behaviour. |
timed out waiting for '...' |
wait or --expect text never appeared. |
Increase --ms, or check the action worked. |
Version and source
This page describes deskcrew 0.1.8.
- Source: https://github.com/acus9345/deskcrew
- crates.io: https://crates.io/crates/deskcrew
- License: MIT
deskcrew 讓 coding agent 在背景操作真正的 Windows App。每個 agent 都有自己畫出來的游標，顏色各不相同，旁邊標著名字；agent 們並排工作，你照常使用電腦。它是單一個命令列程式，所以任何能執行 shell 指令的 agent 都能用，而且設計成盡量少花 token。
經過測試的是 Windows 版。macOS 版已包含在內，但從未在 Mac 上執行過。
安裝
從 crates.io 安裝（需要 Rust）：
cargo install deskcrew
接著讓你的 coding agent 能用它：
deskcrew setup
setup 會把程式複製到 PATH 上的資料夾，並設定它找到的每個 agent App 與 CLI，讓它們在任務需要操作桌面時自動使用 deskcrew，而不是內建的電腦操作或其他桌面工具。安裝新的 agent 後再執行一次：它會更新自己寫入的內容而不是新增副本，檔案的其他部分保持不動。
| Agent（App 與 CLI） | setup 寫入的內容 |
|---|---|
| Claude Code | ~/.claude/skills 中的 skill，~/.claude/CLAUDE.md 中的規則 |
| Codex | ~/.codex/skills 中的 skill，~/.codex/AGENTS.md 中的規則 |
| Gemini CLI 與 Antigravity | ~/.gemini/skills 中的 skill，~/.gemini/GEMINI.md 中的規則 |
| GitHub Copilot CLI | ~/.copilot/skills 中的 skill，~/.copilot/copilot-instructions.md 中的規則 |
| VS Code 中的 GitHub Copilot | VS Code 使用者 prompts/deskcrew.instructions.md 中的規則 |
| Windsurf | skill，以及 ~/.codeium/windsurf/memories/global_rules.md 中的規則 |
| OpenCode | skill，以及 ~/.config/opencode/AGENTS.md 中的規則 |
| Amp、Qwen Code、Factory Droid | ~/.config/amp/AGENTS.md、~/.qwen/QWEN.md、~/.factory/AGENTS.md 中的規則 |
| Kiro、Continue、Cline | ~/.kiro/steering、~/.continue/rules、~/Documents/Cline/Rules 中的規則檔 |
| Cursor | ~/.cursor/skills 中的 skill。它的 User Rules 存在你的 Cursor 帳號裡，所以 setup 會印出要在 Customize > Rules > New 加入的規則 |
Grok，以及共用 ~/.agents 的 agent |
skill |
規則放在 <!-- deskcrew: start --> 和 <!-- deskcrew: end --> 兩行之間。其他 agent，請把 setup 印出的規則加入它自己的規則中。
To operate Windows apps on this computer, run the deskcrew command. Start with: deskcrew help
在 Windows 上也可以改用 qlinter.org 的安裝程式（deskcrew-setup-0.1.8.exe）。它會安裝 deskcrew.exe，可選的「Set up deskcrew for my coding agents」工作會幫你執行 deskcrew setup。在 Mac 上，同樣兩個指令可用；macOS 會要求「輔助使用」權限，第一次截圖前還會要求「螢幕錄製」權限。在雲端執行、或在無法存取你桌面的沙箱中執行的 agent 用不到它。
快速開始
deskcrew windows
deskcrew see Calculator
w2 "Calculator" 37 controls, 1 off screen
e12 button "One"
...
deskcrew click e12
ok invoke
~ e4 "Display is 1"
每個控制項都有一個 ref，例如 e12，而且 ref 在多次讀取之間保持有效，所以一個視窗只要讀一次。每個動作只回覆有變化的部分。背景服務在第一次使用時啟動，閒置 30 分鐘後結束。
想看游標而不碰任何東西，執行 deskcrew demo --seconds 12（僅限 Windows）。
指令
任何指令都可以用 -w 名稱 或 -w w2 指定視窗；否則使用該 agent 上次的視窗。
讀取
| 指令 | 作用 |
|---|---|
windows [filter] |
列出視窗，編號為 w1、w2…… |
see [window] [-f] [-a] [--find word] [--from N] |
列出控制項，編號為 e1、e2……之後的呼叫只顯示變化。-f 重新列出全部，-a 包含畫面外的控制項，--find 篩選，--from 分頁。如果視窗有一大塊沒有控制項（畫布或遊戲），see 會說明，並用 Windows OCR 讀出那裡的文字和它的 x,y。 |
get e3 |
控制項的完整文字。密碼欄位一律拒絕。 |
is e3 on|off|open|closed|selected|enabled|disabled |
檢查狀態。 |
shot [window|e3] [-f] [--marks] [--rulers] [--screen [-a]] [--region x,y,w,h] [--changed] [--edge N] |
存一張 PNG 並印出路徑。和上一張相同時回覆 unchanged。--marks 用 ref 標號控制項，--rulers 標出像素，--changed 只裁出有變化的部分。 |
apps、screens |
已安裝的 App；螢幕。 |
status |
版本與看過的 agent。 |
deskcrew shot -w Paint --marks
操作
| 指令 | 作用 |
|---|---|
click e3 [e4 ...] | "Name" | x,y |
點擊。旗標：--right、--double、--triple、--middle、--back、--forward、--with ctrl+shift。多個 ref 會依序點擊；click File Save 會打開選單並從中選取。 |
hover e3 | x,y |
把畫出的游標移過去並停留。 |
check、uncheck、toggle、expand、collapse、clear e3 |
設定控制項狀態；如果已經是那個狀態，check 和 uncheck 不會動它。 |
set e3 "value" |
取代欄位的文字。 |
type [e3] "text" [--paste] |
輸入文字；--paste 改用貼上。 |
key enter [--hold ms --real] |
按鍵。alt+f4 等有風險的快速鍵需要 --force。 |
select e3 "Item" |
從清單或下拉選單中選取。 |
scroll [e3] up|down [n]、show e3 |
捲動，或把控制項捲到看得見的地方。 |
wait "text" [--gone] [--ms N] |
等到文字出現或消失。 |
drag a b |
從一個 ref 或點拖到另一個。 |
draw [e3] "x,y x,y; x,y x,y" [--smooth] |
畫線條，線條之間用 ; 分隔。 |
down shift --real、down left x,y --mouse、up |
按住按鍵或滑鼠鍵；up、30 秒後、quit 和 Ctrl+Alt+Esc 都會放開。 |
do "click e1; set e2 hi; key enter" [--dry] [--risk low|medium|high] |
多個步驟，一次回覆。--dry 顯示計畫；--risk high 會被拒絕，直到使用者同意並加上 --force。 |
launch app [args] [--front]、focus、close、place x,y,w,h|min|max|restore |
啟動 App（除非加 --front，否則開在你的視窗後面）、把視窗帶到前面、關閉，或移動與調整大小。 |
notifications [close] |
打開通知面板並讀取，然後關閉。 |
clip、clip set "text"、clip clear |
讀取、設定或清除剪貼簿。 |
resume、quit |
在 Ctrl+Alt+Esc 之後繼續；停止服務。 |
cursor [off|ask|session|always] |
誰可以借用你真正的滑鼠游標。 |
任何動作都可以加 --expect "text"，之後沒出現那段文字就算失敗。
其他旗標：--agent name（誰在操作）、--fast（不滑行）、--force（允許有風險的按鍵）、--real（短暫把視窗帶到前面，使用真正的鍵盤或合成觸控筆：最後手段）、--mouse（要求借用你的游標：只在其他方法都不行時）。
deskcrew do "set e2 'hello'; click OK" --expect "Saved" -w Notes
當動作沒有改變任何控制項時，回覆會說明視窗的畫面有沒有變、變在哪裡，所以 agent 很少需要截圖來檢查結果。
運作原理
deskcrew 是一個小型用戶端。它透過 named pipe 把指令交給背景服務（需要時先啟動服務），然後印出回覆。服務把每個視窗的控制項和 ref 存在記憶體裡，所以之後的讀取和回覆只帶變化。每個視窗一條佇列，所以不同 App 裡的 agent 同時工作，共用同一視窗的 agent 則輪流。
動作會先嘗試背景路徑，每個回覆都會說明用了哪條路徑。invoke、toggle、select、expand、value、range 和 scroll 是 App 自己的 UI Automation 動作。command 和 message 把視窗訊息送給傳統控制項，以及瀏覽器和 Electron 視窗。pen 是合成觸控筆，用於自己繪圖的 App，例如小畫家。keyboard 是真正的鍵盤，搭配 --real 使用。mouse 是你真正的游標，只在加 --mouse 且你允許時使用。執行輔助使用請求時，一個低階防護會擋下合成的滑鼠事件，因為有些傳統控制項會悄悄用真正的滑鼠點擊來回應。
畫出的游標和按下時的外框是分層、可點穿的視窗，緊貼在每個 agent 工作的視窗上方，也在那個視窗的選單和彈出視窗上方。蓋住 App 的東西也會蓋住它的游標。每個 agent 會從它的環境或啟動它的程式被辨認出來，並得到自己的顏色和名字。無法辨認的稱為「your agent」。
截圖直接取自視窗本身，所以視窗可以在其他視窗後面。沒有控制項的區域中的文字，用 Windows 內建的 OCR 讀取。
macOS 版使用相同的指令和回覆，而且從 0.1.6 起指令也幾乎一樣齊全。它透過輔助使用 API 讀取，並用控制項自己的動作按下它們；點擊、拖曳和按鍵只送給目標 App；用 Core Graphics 截圖，用 Apple 的 Vision 讀文字，並用透明的 AppKit 視窗畫游標。在 Mac 上，--real 會短暫把 App 帶到前面、透過系統送出按鍵，然後把前景還回去。
與其他工具比較
2026-11-03 在同一台電腦上測量，每個工具都把同樣四個臨時視窗各讀三次。數字是 agent 要讀的字元數，不是 token；而且只測量讀取，沒有測量操作。
| 讀取一個視窗 | deskcrew 0.1.7 | agent-cu 0.1.1 | mcp-windows 1.3.28 | Windows-MCP 0.8.8 |
|---|---|---|---|---|
| 測試表單 | 488 | 1,775 | 7,089 | 5,932 |
| 字元對應表 | 485 | 823 | 3,760 | 4,582 |
| 小算盤 | 689 | 2,199 | 8,212 | 6,444 |
| 檔案總管 | 3,381 | 8,938 | 26,635 | 13,193 |
| 第一次之後每次讀取的時間 | 20 到 102 毫秒 | 0.1 到 3.9 秒 | 27 到 200 毫秒 | 約 15 秒 |
| agent 學會這個工具要載入的內容 | 2,055 字元的說明 | 一個 skill | 58,165 字元的工具 schema | 25,054 字元的工具 schema |
各工具的呼叫方式：
- deskcrew：
see -f。 - agent-cu：
snapshot -i -c，它建議的精簡模式。它的 npm 套件沒有 Windows 程式，所以是從 GitHub 原始碼建置的。它的讀取時間每次執行都在約 70 毫秒到 4 秒之間大幅波動。 - mcp-windows：完整模式的
ui_snapshot。它的增量模式能讓重複讀取更省，但沒有測量。 - Windows-MCP：不含圖片的
Snapshot。它總是讀取整個桌面，包括其他視窗和工作列，所以它的數字不只是單一視窗。使用者正在使用電腦時它會等待，閒置 12 秒後每次讀取仍要約 15 秒。
檔案、路徑與設定
| 項目 | 位置 |
|---|---|
| 游標權限 | %LOCALAPPDATA%\deskcrew\cursor.txt（例如 ask 或 always Claude Code 這樣的行） |
| 截圖 | %LOCALAPPDATA%\deskcrew\shots（Mac：~/Library/Caches/deskcrew/shots）。可能含有私人內容。 |
| 服務 pipe | \\.\pipe\deskcrew-<version>-<session>（Mac：$TMPDIR 中的 socket） |
DESKCREW_AGENT |
為 agent 命名，讓同一個 agent 執行多個副本時有各自的游標。--agent "Claude 2" 效果相同。 |
借用你的游標預設關閉。deskcrew cursor ask 讓 agent 透過畫面上的提示詢問：Just this once、This session、Always 或 No。一分鐘內沒回答視為拒絕。除了 off 之外的任何設定都必須在畫面上確認。這是防止意外，不是防止以你身分執行的程式。
Ctrl+Alt+Esc 會停止所有 agent（Mac 上是 Ctrl+Option+Esc）。讀取仍然可用，deskcrew resume 可以繼續。
限制與疑難排解
- 經過測試的是 Windows 版。macOS 版從未在 Mac 上執行過，只通過型別檢查。在 Mac 上，
--mouse與借用游標、按住滑鼠鍵、notifications和示範都還沒寫，down也只能按住修飾鍵（shift、ctrl、alt、cmd），之後套用到該 agent 在那個 App 中的點擊和按鍵。不支援 Linux。 notifications、down和--real會取得焦點，在傳統文字方塊以外使用type --paste也會。--real動作執行時，你輸入的任何東西都會進入那個視窗。- 鍵盤快速鍵、記事本那樣的 RTF 文字方塊，以及在 XAML 文字方塊中輸入，都需要
--real。 - WinForms 被蓋住時會忽略送來的點擊，所以那裡要用
--real。WinForms 畫布要拖曳需要--mouse。 - 同一個瀏覽器的視窗共用一個程序，同時操作時會遺失輸入。平行工作的 agent 請用不同 App 或不同瀏覽器設定檔。
- 被蓋住的瀏覽器會停止重繪，所以
shot可能拍到舊畫面。在裡面畫圖仍然有效。 - 傳統彈出式選單只接受真正的滑鼠；沒有你的允許一律拒絕。
- 以系統管理員身分執行的 App 只能讀取、不能操作，除非 deskcrew 也以系統管理員身分執行。
- 只在單一螢幕、150% 縮放下測試過。
- 從圖片讀出的文字可能混淆相似字母，例如
I和l。 - 「add --real」：這個動作需要視窗在前面。「nothing changed: try --real if it should have」：App 忽略了背景訊息。「stopped by the person」：有人按了 Ctrl+Alt+Esc；先問他們，再執行
deskcrew resume。
實例
以下回覆使用 deskcrew 真實的格式。ref（e12）、視窗編號（w2）、數量和座標在每台電腦上都不同，請看格式，不要看數字。
1. 小算盤：1 + 2 =
deskcrew see Calculator
w2 "Calculator" 37 controls, 1 off screen
e4 "Display is 0"
e12 button "One"
e13 button "Two"
e20 button "Plus"
e24 button "Equals"
...
deskcrew click e12 e20 e13 e24
ok 4 steps
~ e4 "Display is 3"
按四下、一次回覆，而且只有變化的那一行。按鈕是透過輔助使用按下的，所以小算盤可以待在你其他視窗後面。
2. 填表單並透過「另存新檔」對話框存檔
deskcrew see "Untitled - Notepad"
deskcrew key ctrl+s --real
ok keyboard
+ window "Save As"
+ e40 edit "File name:"
+ e41 button "Save"
...
deskcrew do "set e40 'notes.txt'; click e41" --expect "notes.txt"
ok 3 steps
- e40-e41
~ e1 "notes.txt - Notepad"
對話框的控制項以 + 行出現，因為對話框屬於該視窗。--expect 會在最後加一個 wait，標題沒出現新檔名就算失敗。記事本的文字區是 RTF 文字方塊，所以在裡面輸入需要 --real。很多 App 則接受 set。
3. 在小畫家或 Photon Studio 畫圖
先截一張圖，這樣就能用圖片像素指定座標：
deskcrew shot -w Paint
C:\Users\you\AppData\Local\deskcrew\shots\Claude-Code-w3.png 1356x848 scale 1.42
deskcrew draw "300,300 500,300 500,500 300,500 300,300; 350,350 450,450" --smooth -w Paint
ok pen
controls unchanged; the picture changed at 420,415 290x290
小畫家自己繪圖，所以 deskcrew 在那裡使用合成觸控筆，畫的時候會把小畫家帶到前面。在 Photon Studio 這類瀏覽器或 Electron App 中，路徑是 message：線條在背景送出，即使視窗被蓋住也一樣。你也可以相對於控制項來畫：draw e7 "10,10 200,10"。
4. 在瀏覽器或 Electron App 中輸入
deskcrew see "Sign up"
deskcrew type e5 "Ada Lovelace"
ok message
~ e5 edit "Name" =Ada Lovelace
如果 Chromium 漏掉字元，type 會讀回欄位並直接設定。走這條路徑時請讓視窗保持顯示。被蓋住的瀏覽器會丟掉部分視窗訊息，這時 deskcrew 會退回輔助使用，可能短暫取得焦點。
5. 選單與下拉選單
deskcrew click File "Save As..." -w notepad
ok command
+ window "Save As"
...
deskcrew select e9 "Landscape"
ok select
~ e9 combobox "Orientation" =Landscape
click File Save 一個指令就打開選單並選取，這對新式選單很重要：在兩個指令之間開著的選單，會在焦點回來時關掉。傳統彈出式選單只接受真正的滑鼠，所以除非你允許借用游標，否則會被拒絕。
6. 被蓋住或最小化的視窗
大部分路徑在視窗被蓋住時仍然有效：輔助使用動作，以及瀏覽器中的背景線條。WinForms 是例外，被蓋住時需要 --real。最小化的視窗必須先還原，Windows 還原時可能會把它帶到前面：
deskcrew place restore -w Calculator
ok
shot 會拒絕最小化的視窗，回覆 that window is minimized: place restore first。
7. 讀取畫布或遊戲
deskcrew see "Crew Canvas"
w1 "Crew Canvas" 1 controls
e1 button "Paint"
no controls in 0,125 878x540; text read there (x,y for click):
"Total 42" 129,231
"Start game" 560,453
for drawings: shot --region 1642,1021,878,540
deskcrew click 560,453
see 注意到一大塊沒有控制項的區域，並用 Windows OCR 讀出那裡的文字。它給的 x,y 可以直接用在 click，所以只有畫圖時才需要截圖，而且只需要那一塊。
8. 兩三個 agent 同時工作
每個 agent 自報名字，或從它的程式被辨認出來：
deskcrew --agent "Claude" do "click e12 e20 e13 e24" -w Calculator
deskcrew --agent "Cursor" draw "100,100 400,300" -w "Photon Studio"
deskcrew --agent "Codex" type e3 "meeting notes" -w Notes
由三個 agent 同時執行時，每個都有自己帶名字的彩色游標，三個視窗平行處理。同一個視窗裡的 agent 輪流；在瀏覽器畫布中，它們的線條會被切成短段交替進行。同一瀏覽器的視窗共用一個程序，所以請給平行的 agent 不同的 App 或瀏覽器設定檔。
9. 不用截圖檢查結果
deskcrew click e7 -w Paint
ok invoke
controls unchanged; the picture changed at 14,53 54x17
沒有控制項變化時，回覆仍會說明視窗畫面有沒有變、變在哪裡。nothing changed, on screen either 表示這次點擊沒有任何可見效果。
特殊情況
- 以系統管理員身分執行的視窗。 只能讀取、不能操作，因為 Windows 禁止一般程式對它們送出輸入。回覆是
blocked: the window runs as administrator。若一定要操作，請從提升權限的終端機執行 deskcrew。 - 傳統彈出式選單。 只接受真正的滑鼠。除非你允許借用游標（
deskcrew cursor ask），否則 deskcrew 會以this control only answers the real mouse拒絕。先試試鍵盤或click x,y。 - WinForms。 WinForms 被其他視窗蓋住時會忽略送來的點擊，所以加上
--real。它的畫布不會把送來的拖曳當成拖曳，所以在那裡畫圖需要--mouse。核取方塊會短暫取得焦點再還回去。click --with ctrl需要--real，因為 WinForms 自己讀鍵盤。 - XAML 文字方塊（新式 App）。
set可用，但type需要--real。 - 共用一個程序的瀏覽器。 同一個瀏覽器的兩個視窗同時操作時會互相啟用，導致遺失輸入。請用不同瀏覽器設定檔或不同 App。
- DPI 與縮放。 所有座標都是實體螢幕像素。
click和draw中的x,y是上一張截圖中的像素，deskcrew 會依截圖的比例換算回去。在 150% 縮放下測試過。 - 多個螢幕。
screens會列出它們，shot --screen -a會全部截下。只測試過單一螢幕。 - 搶走焦點的對話框。 新對話框會在回覆中以
+ window "..."出現，它的控制項會加入該視窗的清單。如果 agent 工作時有 App 搶走焦點，deskcrew 會立刻把焦點還給你的視窗。 - 會移動或消失的控制項。 即使 App 重建了控制項，ref 在多次讀取之間仍然有效。真的消失的 ref 會回覆
e12 is gone: see。用see重新讀取並使用新的 ref。 - 慢的 App 與逾時。 每個動作在回覆前最多等 0.8 秒讓視窗穩定。更慢的情況請用
wait "text"（搭配--ms最多 60 秒）或--expect。Chromium 在第一次請求時才建立控制項樹，所以對瀏覽器的第一次see可能要幾秒。 - 批次執行到一半按下 Ctrl+Alt+Esc。 所有 agent 在步驟之間停下，輸入在文字中途停止，按住的按鍵會放開。回覆是
stopped by the person with Ctrl+Alt+Esc，並列出已完成的部分。讀取仍可用；deskcrew resume讓工作繼續。 - 服務當掉或結束。 下一個指令會自動啟動新服務。ref 和記住的讀取結果會遺失，所以請再執行一次
see。按住的按鍵在quit和閒置結束時會放開，但如果服務被直接砍掉，按住的鍵會一直按著，直到你自己按一下。 --real輸入時焦點移走。 輸入會立刻停止，回覆會說明輸入了多少字元。
常見問題
deskcrew 會移動我的滑鼠嗎？ 除非你允許，否則不會。借用游標預設關閉。即使允許，agent 也必須先詢問，用完後會把游標放回原處。
agent 工作時我可以繼續用電腦嗎？
可以。背景路徑不會占用你的滑鼠或鍵盤。只有 --real、down、notifications、launch --front 和部分 type --paste 會短暫取得焦點，而且它們會等你停止打字的空檔。
它能和哪些 agent 搭配？
任何能在你電腦上執行 shell 指令的 agent。deskcrew setup 會設定 Claude Code、Codex、Gemini CLI 與 Antigravity、Copilot（CLI 與 VS Code）、Windsurf、OpenCode、Amp、Qwen Code、Factory Droid、Kiro、Continue、Cline、Grok 和 Cursor，讓它們不用被要求就使用 deskcrew。在一次測試中，Cursor 的 agent 只被要求用小算盤算出 12 × 7，就自己選擇了 deskcrew，並回答 84。
為什麼游標有時看不到？ 游標緊貼在各自 agent 的視窗上方，所以蓋住那個視窗的東西也會蓋住它的游標。工作會繼續進行。
怎麼停止一切？
按 Ctrl+Alt+Esc。之後 deskcrew resume 可以繼續，deskcrew quit 則停止服務。
它會花多少 token？ 非常少。在我們的測試中，讀一個視窗需要 260 到 3,375 個字元，之後的回覆只列出變化。截圖是檔案，agent 只在需要時才打開。
能在 Mac 上用嗎？ 已包含 Mac 版，從 0.1.6 起幾乎具備所有 Windows 指令，但從未在 Mac 上執行過。請把它當成初稿。
截圖存在哪裡？
在 %LOCALAPPDATA%\deskcrew\shots，每個 agent 和視窗一張。新的會取代舊的，而且可能含有私人內容。
為什麼 deskcrew 拒絕某個快速鍵？
會關閉或鎖定東西的快速鍵，例如 Alt+F4 或 Win+R，需要 --force。agent 應該先問你再加上它。
需要網路或伺服器嗎？ 不需要。一切都在你的電腦上執行，OCR 也是離線的。
怎麼解除安裝？
執行 deskcrew quit，然後 cargo uninstall deskcrew；如果用的是安裝程式，就在 Windows 設定中移除。再刪除 %LOCALAPPDATA%deskcrew、每個 agent 的 skills 資料夾中的 deskcrew 資料夾、每個 agent 規則檔中 <!-- deskcrew: start --> 與 <!-- deskcrew: end --> 之間的文字、名為 deskcrew.md 或 deskcrew.instructions.md 的規則檔，以及 Cursor User Rules 中的規則。
常見回覆與錯誤
| 回覆 | 意思 | 怎麼做 |
|---|---|---|
no window yet: run deskcrew windows, then deskcrew see <name> |
這個 agent 還沒選視窗。 | 執行 windows，再 see 你要的那個。 |
no window matches '...' |
沒有視窗標題或 App 含有這段文字。 | 查看 deskcrew windows。 |
'...' matches 3 windows: |
名稱不明確。 | 改用視窗編號，例如 -w w4。 |
that window closed: run deskcrew windows |
視窗不見了，或它的 handle 被重複使用。 | 重新列出視窗。 |
e12 is gone: see |
那個控制項已不存在。 | 執行 see 並使用新的 ref。 |
no control named '...': see / '...' matches 2 controls: |
沒有、或有好幾個控制項叫這個名字。 | 使用 see 給的 ref。 |
that control is disabled |
目前無法使用。 | 先做讓它啟用的事。 |
no background way to do this here: add --real |
App 在這件事上忽略背景輸入。 | 加上 --real，它會短暫把視窗帶到前面。 |
nothing changed: try --real if it should have |
App 收到訊息但沒有任何變化。 | 檢查目標，或試試 --real。 |
controls unchanged; the picture changed at x,y wxh |
有東西被畫上或重繪，但控制項沒變。 | 對畫圖和畫布來說通常代表成功。 |
this control only answers the real mouse: ... |
傳統控制項需要實體滑鼠。 | 試試 click x,y 或鍵盤。否則須由使用者允許 --mouse。 |
the person has not let agents borrow the cursor. ... |
--mouse 是關閉的。 |
使用者可以執行 deskcrew cursor ask。 |
the person said no to borrowing the cursor: find another way |
使用者拒絕了提示。 | 改用其他路徑。 |
refused: x,y on screen is not on this window |
某個點落在目標視窗外。 | 重新截圖並檢查座標。 |
declared risky: ask the person, then add --force |
有風險的快速鍵或 --risk high 批次。 |
先問使用者。 |
stopped by the person with Ctrl+Alt+Esc: ... |
使用者停止了所有 agent。 | 問他們，然後 deskcrew resume。 |
stopped: focus moved away from that window, ... |
--real 輸入被打斷。 |
檢查文字，再從停下的地方繼續。 |
x,y are pixels in a shot: take one first |
在任何截圖或 OCR 讀取之前就給了座標。 | 先執行 shot 或 see。 |
blocked: the window runs as administrator |
Windows 擋下對提升權限視窗的輸入。 | 以提升權限執行 deskcrew，或自己來。 |
that is a password field: its text stays hidden |
get 從不讀取密碼。 |
這是預期行為。 |
timed out waiting for '...' |
wait 或 --expect 的文字一直沒出現。 |
加大 --ms，或確認動作有成功。 |
版本與原始碼
本頁說明 deskcrew 0.1.8。
- 原始碼：https://github.com/acus9345/deskcrew
- crates.io：https://crates.io/crates/deskcrew
- 授權：MIT