Crawl4ai Skill

Crawl4AI Agent Skill

Scrape JavaScript-heavy sites and extract structured data via reusable CSS schemas. A portable agent skill that wraps the [Crawl4AI](https://crawl4ai.com/) CLI and Python SDK, written in the Anthropic SKILL.md format and consumable by any agent host that loads SKILL.md-format bundles (Claude Code, Codex, Cursor, OpenCode, Cline, and others).

Verified against Crawl4AI library version `0.8.9` (pinned in [`VERSION`](VERSION)).

Features

• JS-aware crawling: full headless-browser rendering with wait_until=networkidle defaults
• Schema-based extraction: derive a CSS selector schema once via LLM, apply it forever with no further LLM cost
• LLM extraction: per-request structured extraction when a schema is not worth deriving
• Content filtering: BM25 relevance filter and quality-based pruning, plain markdown or markdown-fit output
• Concurrent batch crawling: multi-URL processing with per-job concurrency caps
• Session management: persistent sessions for authenticated, multi-step flows
• CLI and SDK: both the crwl command-line tool and the crawl4ai Python SDK

Installation

Clone the repo into the skills directory your agent host loads from:

# Claude Code
git clone https://github.com/brettdavies/crawl4ai-skill.git ~/.claude/skills/crawl4ai

For other agent hosts (Codex, Cursor, OpenCode, Cline, custom agents), clone into whichever directory your host scans for SKILL.md-format bundles. Refer to your host's documentation for the skills directory location. The bundle root contains `SKILL.md`, so the skill registers automatically once the directory is on the host's skills search path.

Prerequisites

The skill calls into the Crawl4AI Python library, which must be installed in the runtime your agent uses:

pip install crawl4ai
crawl4ai-setup
crawl4ai-doctor

`crawl4ai-doctor` validates the install and confirms a headless browser is available.

Quick start

CLI:

crwl https:/