Agent Crawl

AgentCrawl

[](https://www.npmjs.com/package/agent-crawl)

**The High-Performance TypeScript Web Scraper for LLM Agents.**

AgentCrawl is built to be the "eyes" of your AI Agents. It fetches web content, strips away the noise (ads, scripts, styles), and returns clean, token-optimized Markdown ready for your LLM context window.

It features a **Hybrid Engine** that starts with extremely fast static scraping and automatically falls back to a headless browser (Playwright) only when necessary for dynamic content or authentication.

Features

• 🚀 Hybrid Engine: Instant static fetch by default, auto-switch to Headless Browser for dynamic sites.
• ⚡ Token Optimized: Returns clean Markdown, stripping 80-90% of tokens (ads, navs, footers).
• 🧠 Agent-First: Detects Main Content, removes boilerplate, and extracts semantic structure.
• 🔌 Plug-and-Play: Simple API designed for agent runtimes (scrape + crawl).
• 🛡️ Production Ready: Built-in caching, retry logic, user-agent rotation, and resource blocking.
• 🕵️ Stealth Mode: Optional best-effort browser hardening to reduce common bot-detection fingerprints.
• ✅ Predictable Errors: Non-2xx HTTP responses are surfaced as errors instead of silently parsed as success.
• 🇹 Type-Safe: 100% TypeScript with Zod validation.

Installation

npm install agent-crawl
# OR
bun add agent-crawl

CLI

AgentCrawl ships with a CLI for quick scraping and crawling from the terminal.

# Install globally
npm install -g agent-crawl

# Or use directly with npx
npx agent-crawl scrape https://example.com

Scrape a page to markdown

agent-crawl scrape https://example.com

JSON output with metadata

agent-crawl scrape https://example.com --output json

Browser mode for JS-rendered pages

agent-crawl scrape https://example.com --mode browser --stealth
agent-crawl scrape https://examp