Well Known Bots

Well Known Bots

This repository contains a list of Well Known Bots, including robots, crawlers, validators, monitors, and spiders, in a single JSON file. Each bot is identified and provided a RegExp `pattern` to match against an HTTP `User-Agent` header. Additional metadata is available on each item.

Install

Direct download

Download the [`well-known-bots.json` file][raw-json-url] directly.

Realities

It's impossible to create a system that can detect all bots. Well-behaving bots identify themselves in a consistent manner, usually via the User-Agent patterns this project provides. It is straightforward to identify these well-behaving bots, but misbehaving bots pretend to be real clients and use various mechanisms to evade detection.

For more details, see [Non-Technical Notes in the browser-fingerprinting][non-tech-notes-url] project.

Custom bots

To block a particular bot that is not on this list, you can use an Arcjet filter. See the [Malicious traffic][arcjet-blueprint-malicious-traffic] blueprint for how to block custom bots.

Adding a New Bot

To add a new bot to the list, you need to edit the `well-known-bots.json` file and add a new entry. Follow these steps:

1. Create a new bot entry with the required fields (see structure below)
2. Add User-Agent pattern(s) that identify the bot
3. Add verification method(s) if the bot provider supports verification
4. Add example instances to validate your patterns work correctly
5. Run validation to ensure your entry is correct: node validate.js --check
6. Submit a pull request with your changes

Bot Entry Structure

Each entry in the JSON represents a specific bot or crawler and includes the following fields:

Required Fields

• id (string): A unique identifier for the bot in kebab-case (e.g., "google-crawler")
• categories (array): One or more categories the bot belongs to (see available categories)
• pattern (object)