AgentPilot

AgentPilot

**LLM-powered macOS computer control.** AgentPilot lets AI agents see your screen, move the mouse, type text, press hotkeys, manage windows, run shell commands, and more — all through a native macOS app with a chat interface.

Built with **Swift** (native macOS UI + system executor) and **Python** (LLM agent orchestration). Supports **Anthropic Claude**, **OpenAI GPT-4o**, and **local models via Ollama**.

   

---

Features

• 27 native macOS tools — full computer control from application management to file system operations
• Chat UI — built-in SwiftUI chat interface with real-time SSE streaming
• Multi-provider LLM — Claude, GPT-4o, or any local Ollama model
• ReAct agent loop — observe → think → act → verify, with automatic retries
• Safety first — policy engine blocks dangerous commands, kill switch (Cmd+Shift+Escape), action logging
• Long-term memory — SQLite-backed task recipes so the agent learns from past actions
• Zero cloud dependency — runs fully local with Ollama (no API keys needed)

Architecture

┌─────────────────────────────────┐
│   AgentPilot (Swift macOS App)  │
│  ┌───────────┐  ┌────────────┐  │
│  │  Chat UI  │  │  Settings  │  │
│  └─────┬─────┘  └────────────┘  │
│        │                        │
│  ┌─────▼─────────────────────┐  │
│  │   HTTP Server (:8420)     │  │
│  │   28 REST endpoints       │  │
│  └─────┬─────────────────────┘  │
│        │                        │
│  ┌─────▼─────────────────────┐  │
│  │   System Executors        │  │
│  │   CGEvent · AX API ·     │  │
│  │   Process · FileManager   │  │
│  └───────────────────────────┘  │
└─────────────────────────────────┘
          ▲
          │ HTTP
          ▼
┌───────────────