These are generalizations from building one desktop-automation driver — a .NET 9 program (FlaUI over UIA3, plus the ModelContextProtocol SDK) that lets an AI agent read and operate a real Windows desktop over MCP and a localhost HTTP API. The project is called Deskhand. Nothing here is specific to it; every claim should hold for anyone wiring a real GUI to a model, and is checkable against the FlaUI, UI Automation, and Win32 docs.
One thread owns the automation library, and it is an STA thread
UI Automation (UIA3, and the COM world under FlaUI, WinForms, WPF interop) is apartment-threaded. If you call it from a thread pool — the default for an async web handler — you get intermittent, unreproducible COM failures: RPC_E_WRONG_THREAD, elements that resolve on one call and throw on the next, deadlocks under load. The fix is structural and you want it on day one: spin up one dedicated STA thread, give it a work queue, and marshal every automation call onto it. Handlers await a completion that the STA thread posts back. This single decision shapes the whole architecture, and retrofitting it later means touching every tool.
The kill switch and the audit log are the substrate, not features
An agent driving a real machine can delete files, send keystrokes into the wrong window, click "OK" on a dialog you never saw. Two things make such a program something you would actually leave running:
- Disarmed by default. A single
armedflag gates every state-changing action; reads are always allowed, writes refuse with a clear error until armed. The default is off. This is the difference between a bug being an inconvenience and being an incident. - An append-only log written before the action, not after. Every governed call appends a line (JSONL is enough) with timestamp, action, target, and status. When something goes wrong you need to know what the agent actually did, in order, and you cannot trust the agent's own account of it.
Build these first. Bolting governance onto a working driver means auditing every existing code path; building the driver through the gate means it is free thereafter.
Return structure before pixels, and cap what you return
An agent pays per token to read your output, and it cannot skim — the whole reply arrives at once (see reading as an agent). Two consequences:
- Prefer the semantic UI tree (control names, types, automation patterns) over screenshots. It is deterministic, it survives theme and DPI changes, and a labelled control is worth a hundred pixels of a model guessing at a button. Screenshots + OCR + template matching are the fallback for surfaces that expose no UIA — games, custom-drawn canvases, remote-desktop frames — not the default.
- A full UIA tree or a directory listing can be enormous. Truncate large output to a retrievable store and return a handle plus a summary, rather than spending the caller's context on data it may not need. Let the agent ask for more.
Match the timeout to the work, not to the client
A quick "read the clipboard" call and a "let the agent work for ten minutes" call cannot share one HTTP timeout. If the client's default is 120 seconds, a legitimate long action is aborted mid-flight and the agent sees a failure that never happened. Give long-running operations their own client (or a per-call timeout override) with a generous budget; keep the short timeout for the quick reads where a hang genuinely means trouble.
Use the right layer for enumeration
The automation library is not always the fastest or most complete source. Listing top-level windows through the Win32 EnumWindows API is complete and quick; the same list via the UIA tree can be slower and miss windows. When two layers can answer the same question, measure both — the lower-level one often wins for breadth and speed, and you reserve the rich automation layer for actually operating a control once you have found it.
The modal dialog is the real enemy of an unattended agent
Anything left running unattended will eventually be ambushed by a window it did not open: a UAC prompt, "Save changes?", an update nag, a crash reporter. It steals focus, and the agent's next keystroke lands somewhere unintended. You need a watchdog that notices new top-level windows and a rule-based dismisser — but ship it in stages, because auto-closing the wrong dialog is itself destructive: report-only first (log what appeared), then hide, and only then close by explicit rule. Never let the general case be "click the default button."
Trajectory recording is nearly free once you have an audit stream
If you already log every action and can capture the screen, you almost have training and evaluation data for free: subscribe to the audit event, snapshot the screen after each non-trivial action, and you have an ordered sequence of (observation, action, result) — a trajectory — with no new instrumentation. Reuse what the driver already produces rather than building a separate recorder.
Set-of-Mark turns any model into an agent
Rather than asking a model for pixel coordinates it is bad at, overlay numbered boxes on the screenshot and expose an act_mark(id) action. The driver owns the pixels; the model owns the choice of which thing to act on. This decouples the agent loop from any one model's coordinate precision and is the seam that lets you swap models without rewriting the loop.
One version constant, and self-update against the release tag
Keep a single version string that feeds the health endpoint, the OpenAPI document, and an update check that compares it to the latest published release. An agent (or an operator) can then ask "am I current?" and update in place, and you never have three copies of the version drifting apart.
A footgun worth naming: path translation
When a POSIX shell shim and native Windows tools share a session, their idea of a path diverges. A file written to /tmp by the shim is not where a native Windows process looks for it. Whenever two toolchains touch the same file, pin an absolute native path both agree on rather than a shell-relative one.
Related: reading as an agent · connecting over MCP · never write a file in place.