Free Scraping Sop

Free Scraping SOP — Database Lead Discovery Without Paid APIs

A reusable methodology + working scripts for extracting company data from any web-accessible database **without paying for API access or subscriptions**. Built while scraping GetLatka, Crunchbase, Ashby, Greenhouse, Lever, and Wellfound for B2B outbound campaigns.

If you've ever stared at Clay's monthly bill and wondered how to replace half of it with `site:` queries and a sitemap, this is the SOP.

---

What's in this bundle

free-scraping-sop/
├── README.md                       ← you are here (setup + quickstart)
├── PLAYBOOK.md       ← full methodology, Methods 1–6, decision tree
└── scripts/
    ├── getlatka_scraper.py         ← Method 1: robots.txt → sitemap → bulk slugs
    ├── ashby_scraper.py            ← Method 2: simple single-platform Serper
    ├── greenhouse_scraper.py       ← Method 2: simple single-platform Serper
    ├── ats_scraper.py              ← Method 2: 4 ATS platforms, multi-layer queries
    ├── ats_maximizer.py            ← Method 4a: Serper discovery + platform JSON API
    ├── crunchbase_scraper.py       ← Method 5: Cloudflare-blocked → Serper-only
    ├── crunchbase_sitemap_scraper.py ← Method 5: Exa supplement (different index)
    ├── company_enricher.py         ← Method 6: domain + LinkedIn + funding via Serper
    ├── workatastartup_to_yc.py     ← Bonus: dedupe WaaS scrape vs YC baseline
    └── yc_page_enricher.py         ← Bonus: enrich YC rows from each company page

The playbook is the *why*. The scripts are the *how*. Read the README to get running, read the playbook before writing your own scraper for a new database.

---

Use as a Claude Code skill

If you use [Claude Code](https://claude.ai/code), you can drop this whole repo into your project's `.claude/skills/` folder and invoke it with `/free-scraping`. The `SKILL.md` file tells Claude how to run the scripts for you.

cp -r free-scraping-sop/ your-project/.claude/skills/free-scraping/