Computer Agent

Computer-Using Agent (CUA)

**[System Design](./systemdesign.md)** - Architecture, data flow, and how it works end-to-end

---

An AI that sees your screen and controls your computer — autonomously.

---

What We're Building

A fully autonomous **Computer-Using Agent** that can take any high-level goal like:

*"Open Chrome, go to Gmail, and draft an email to my boss saying I'll be late"*

...and actually do it — by seeing your screen, reasoning about what to click/type next, and executing actions in a loop until the goal is done.

No hardcoded scripts. No browser automation APIs. Just vision + reasoning + control.

This is the same class of technology being built by OpenAI (Operator), Anthropic (Claude computer use), and Google DeepMind — except this is open, runs with any LLM, and you control it.

Full Vision (Roadmap)

Screen → Vision Model → Reasoning → Action → Feedback → Repeat

Phase	Feature	Status
1	Screen capture + base64 encoding	✅ Done
2	Vision LLM understands screen content	✅ Done
3	JSON action planning (click, type, scroll, hotkey)	✅ Done
4	Action execution via pyautogui	✅ Done
5	Closed feedback loop (act → re-capture → re-plan)	✅ Done
6	Safety layer (dangerous action blocking + confirmation)	✅ Done
7	Session memory + action history logging	✅ Done
8	Multi-provider support (10 cloud APIs + local models)	✅ Done
9	Local model support (Ollama, LM Studio, llama.cpp)	✅ Done
10	UI element detection (OpenCV + optional YOLO)	✅ Done
11	Multi-step task memory across sessions	✅ Done
12	Web UI / dashboard to watch the agent live	✅ Done
13	Voice goal input ("Hey agent, do this...")	✅ Done
14	Full auto mode with rollback on failure	✅ Done

---

What We've Built (Current MVP)

Architecture

main.py              ← orchestrates the loop + all CLI flags
providers.py         ← unified interfac