Webis

# Webis: AI-Driven Knowledge Pipeline

[中文](README.zh.md)

[](https://www.python.org/) [](LICENSE) [](https://pypi.org/project/webis/) [](https://narwhal-lab.github.io/webis) [](https://github.com/Narwhal-Lab/Webis/actions) [](https://codecov.io/gh/Narwhal-Lab/Webis) [](https://github.com/astral-sh/ruff)

**Webis** is a comprehensive, modular framework that powers the next generation of AI applications. It connects diverse data sources (Web, SaaS, databases, etc.) to Large Language Models (LLMs) through a robust pipeline of collection, processing, and extraction.

🎯 Who is Webis for?

👥 User Type	🚀 Use Case
Researchers	Literature reviews, data collection, research analysis
Data Scientists	Training data preparation, knowledge base building
Developers	Building AI applications, integrating RAG capabilities
Business Users	Market monitoring, competitive intelligence, knowledge management
Educators	Creating educational resources, research datasets

✨ Key Features

For All Users

• 🎯 5-Minute Setup: Get started in minutes with our intuitive CLI and web interface
• 📊 Beautiful Visualizations: Interactive dashboard with charts and graphs
• 🤖 AI Assistant: Natural language interaction with your knowledge base
• 📄 Multi-format Support: PDFs, webpages, HTML, Markdown, CSV, JSON, DOCX

For Developers

• 🔌 Plugin Architecture: Everything is a plugin (sources, processors, extractors)
• **🛠️ SDK