Monitoring Llms Demo

AI Alignment Demo: Monitoring and Evaluating LLM Systems

This repository demonstrates three frameworks for monitoring Large Language Model (LLM) applications: **DeepEval**, **RAGAs**, and **DeepChecks**. These tools help ensure AI systems produce reliable, safe outputs.

**Working demo repository** tested on macOS, Linux, and Windows.

📚 Documentation

Getting Started

• SETUP_GUIDE.md - Installation and setup instructions
• ALIGNMENT_METRICS.md - Key metrics: faithfulness, toxicity, bias, fairness

Production Use

• PRODUCTION_MONITORING.md - User-facing agent monitoring
• TASK_EXECUTING_AGENTS.md - Code generation and task-executing agents
• ENTERPRISE_MONITORING.md - Organizational-level monitoring

Implementation

• IMPLEMENTING_METRICS.md - Code examples
• BEST_PRACTICES.md - Implementation guidance
• CRITICAL_THINKING.md - Evaluation framework

What is AI Alignment?

AI alignment ensures AI systems behave safely and as intended. Key properties:

• Correctness - Factually accurate outputs
• Relevancy - Responses match user queries
• Safety - Appropriate, non-harmful outputs
• Fairness - Unbiased, equitable treatment

Tools Overview

DeepEval (`deepeval_demo/`)

Comprehensive LLM evaluation framework for conversational agents, custom criteria, and multi-turn conversations.

RAGAs (`ragas_demo/`)

Specialized RAG system evaluation for retrieval quality, generation quality, and agent tool usage.

DeepChecks (`deepchecks/`)

Data and model validation for detecting data drift, quality issues, and performance degradation.

When to Use Which Tool

• DeepEval - Conversational agents, custom evaluation criteria
• RAGAs - RAG systems, retrieval-based applications, agent tool usage
• **Deep