Llm Agents Eval Tutorial

KDD 2025 Tutorial: Evaluation & Benchmarking of LLM Agents

---

📄 Abstract

The rise of LLM-based agents has opened new frontiers in AI applications, yet evaluating these agents remains a complex and underdeveloped area. This tutorial provides a systematic survey of the field of LLM agent evaluation, introducing a two-dimensional taxonomy that organizes existing work along:

• Evaluation objectives (what to evaluate): agent behavior, capabilities, reliability, safety
• Evaluation process (how to evaluate): interaction modes, datasets and benchmarks, metric computation methods, and tooling

In addition, we highlight enterprise-specific challenges, such as role-based access, the need for reliability guarantees, dynamic and long-horizon interactions, and compliance. Finally, we discuss future research directions toward holistic, more realistic, and scalable evaluation of LLM agents.

You can read the pre-print [here](https://arxiv.org/abs/2507.21504).

---

🎯 Target Audience

This tutorial is designed for applied and industry data scientists, machine learning engineers, and enterprise AI practitioners who build or deploy LLM-based agents in production systems. It is also relevant for academic researchers studying evaluation methodologies, multi-agent systems, and trustworthy language models. Participants will gain a systematic evaluation framework, practical hands-on code examples, and insights into real-world deployment challenges.

---

🗂️ Tutorial Agenda

• Introduction (5 min)
  • Motivation and tutorial goals
• Taxonomy Overview (5 min)
  • What to evaluate and how to evaluate
• Evaluation Process (25 min)
  • Interaction modes
  • Evaluation data
  • Metric computation methods
  • Evaluation tooling
  • Evaluation contexts
• Evaluation Objectives (90 min)
  • Agent Behavior
  • Agent Capabilities
  • Reliability
  • Safety & Alignment
• Enterprise-Specific Challenges (20 min)
  • Access control
  • Reliability guarantees