he-yufeng

LiteBench — Development skill for Claude Code

Development community

A pip-installable benchmark runner for LLMs and agents.

How to install LiteBench

This entry records only its repository, not the path inside it, so there is no exact command to give. Open he-yufeng/LiteBench and copy the folder into ~/.claude/skills/, or the file into ~/.claude/agents/.

What LiteBench does

A pip-installable benchmark runner for LLMs and agents. Five minutes to your first eval.

Alternatives in Development

14 Dev Runner — Prompt 14: Create Development Runner 2.2k ★
Dive — Dive is an open-source MCP Host Desktop Application that seamlessly integrates with any LLMs supporting functi 1.8k ★
Ctf 24h Fleet — 24h unattended multi-target Web CTF fleet loop runner 1.1k ★

README

LiteBench — benchmark runner for LLMs and agents

[![PyPI](https://img.shields.io/pypi/v/litebench.svg)](https://pypi.org/project/litebench/) [![Python](https://img.shields.io/pypi/pyversions/litebench.svg)](https://pypi.org/project/litebench/) [![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)

[**Quick Start**](#usage) · [**Built-in Tasks**](#built-in-tasks) · [**Custom Tasks**](#custom-tasks) · [中文](README_CN.md)

litebench run gsm8k

[中文文档](README_CN.md)

What is this?

`inspect_ai` is powerful but heavy — you write Solver and Scorer classes. `lm-evaluation-harness` is thorough but research-oriented and slow to set up. `promptfoo` tests prompts, not full agents.

**LiteBench** sits in the middle: an opinionated CLI for app developers who want to benchmark their model or agent on common tasks (HumanEval / GSM8K / MMLU / MATH / TruthfulQA / ARC) without having to write a framework first.

pip install litebench

litebench list
litebench run gsm8k -m deepseek/deepseek-chat -n 50
litebench run gsm8k -m deepseek/deepseek-chat -n 50 --resume   # continue an interrupted run
litebench run humaneval -m gpt-5 -n 20
litebench run mmlu -m claude-sonnet-4-6 --subject computer_security -n 100
litebench run gsm8k -m gpt-5 -n 20 --repeat 5   # 5 attempts per task, reports pass@1 / pass@5
litebench run math -m kimi -n 50

# Custom YAML tasks
litebench run ./my-task.yaml -m gpt-4o-mini

# Compare models
litebench runs
litebench compare  
litebench export  -o run.json

Features

6 built-in tasks — HumanEval, GSM8K, MMLU, MATH-500, TruthfulQA, ARC-Challenge.
100+ model providers via litellm — OpenAI, Anthropic, Gemini, DeepSeek, Kimi, Qwen, GLM, local Ollama, and more. Shortcuts built in: -m opus, -m kimi, `-m deepsee