scrapai

A CLI where you describe what you want to scrape in plain English, an AI agent builds the scraper, and Scrapy runs it.

You: "Add https://bbc.co.uk to my news project"

Minutes later you have a tested, production-ready scraper stored in a database. No Python, no CSS selectors, no Scrapy knowledge. The AI agent analyzes the site, writes extraction rules, verifies quality, and saves a reusable config. Run it tomorrow or next year. Same command, no AI costs.

Built by [DiscourseLab](https://www.discourselab.ai/). Used in production across 500+ websites.

Table of Contents

• Who This Is For
• Why scrapai?
• How It Works
• Features
• Quick Start
  • Using with AI Agents
  • Migrating Existing Scrapers
• For Developers
• Architecture
• Security
• CLI Reference
• Configuration
• Limitations
• Documentation
• Contributing
• Responsible Use
• License

Who This Is For

**Good fit:**

• Teams that need to scrape many websites and don't want to write individual scrapers
• Non-technical users who can describe what they want in plain English
• Organizations where scraping is a means