cua: Computer-Use Automation at Scale

Background

Computer-use agents have been stuck in a local maximum. Most implementations wrap a single browser instance, hardcode OS-specific automation APIs, and treat evaluation as an afterthought. The result: brittle scripts that break on minor UI changes, zero portability across Windows, macOS, and Linux, and no standard way to measure whether an agent actually succeeds at a task.

cua (trycua/cua) takes a different approach. It’s an open-source framework for computer-use automation at scale — 25,000+ stars on GitHub, written in HTML/TypeScript, designed from the ground up for cross-OS fleets. The project provides open-source drivers for Windows, macOS, and Linux, a unified protocol for agent-environment interaction, and built-in benchmarks for training, evaluation, and data generation.

The goal isn’t “make a browser click buttons.” The goal is: run 1,000 agent instances across a heterogeneous fleet, collect reproducible metrics, and feed that data back into model training. That’s the infrastructure layer the field has been missing.

Challenges

Three problems dominate computer-use automation today:

• OS fragmentation: Windows uses UI Automation, macOS uses Accessibility API, Linux uses AT-SPI/ATK. Each has different semantics, different failure modes, different latency profiles. Most frameworks pick one and ignore the rest.
• No evaluation standard: Everyone reports “success rate” differently. Is it task completion? Step accuracy? Time to finish? Without a common benchmark, you can’t compare approaches or track progress.
• Fleet management is an afterthought: Running agents at scale means handling VM provisioning, session isolation, crash recovery, log aggregation, and resource scheduling. Most projects bolt this on later — or never.

cua addresses all three. The driver layer abstracts OS differences behind a single protocol: observe (screenshot + accessibility tree), act (mouse, keyboard, scroll), and evaluate (task-specific metrics). The benchmark suite includes web navigation, form filling, file operations, and multi-step workflows — each with ground-truth labels and deterministic scoring.

The hardest part isn’t clicking a button. It’s knowing whether the click achieved anything when the DOM shifts, the selector breaks, or the app renders differently on a different OS version.

Run the benchmark locally:

git clone https://github.com/trycua/cua.git
cd cua
docker compose up -d
python -m cua.bench.run --suite web-navigation --agents 10

Results & Lessons

Early benchmark runs reveal what everyone suspects but few measure: success rates drop sharply when you move off the happy path. On clean web-navigation tasks, top agents hit 70-80% completion. Add a single popup, a dynamic layout shift, or a cross-origin iframe, and rates fall to 30-40%. Cross-OS variance adds another 15-20% gap between Linux (most stable) and Windows (most variable).

The data also shows where current models fail: long-horizon planning (10+ steps), recovery from intermediate failures, and handling ambiguous UI states. These aren’t reasoning gaps — they’re data gaps. The training corpus for computer-use tasks is tiny compared to text or code.

cua’s contribution is making that data collectible at scale. The fleet architecture spins up isolated environments, records full trajectories (observations, actions, rewards), and exports them in a format compatible with RLHF pipelines. You can run 100 parallel episodes, filter for successful trajectories, and fine-tune a model on real interaction data — not synthetic demonstrations.

Two practical takeaways for anyone building in this space:

• Invest in the evaluation harness before the agent. If you can’t measure it reproducibly across OSes, you’re guessing.
• Design for fleet operations from day one. Session isolation, crash recovery, and resource accounting are not “ops concerns” — they determine whether your data is usable.

The project is still early. The driver layer needs more coverage (mobile, remote desktop), the benchmark suite needs more task diversity, and the fleet scheduler needs better bin-packing for heterogeneous workloads. But the architecture is sound: protocol-first, evaluation-native, fleet-aware. That’s the foundation the next generation of computer-use agents will build on.