he-yufeng

LiteBench — Development skill for Claude Code

Development community

A pip-installable benchmark runner for LLMs and agents.

How to install LiteBench

This entry records only its repository, not the path inside it, so there is no exact command to give. Open he-yufeng/LiteBench and copy the folder into ~/.claude/skills/, or the file into ~/.claude/agents/.

What LiteBench does

A pip-installable benchmark runner for LLMs and agents. Five minutes to your first eval.

Alternatives in Development

Clauditor — Stop Claude Code from burning through your quota in 20 minutes 367 ★
M5 Paper Buddy — 📟 Claude Code physical companion on M5Paper V1.1 e-ink 160 ★
PM Workspace Wizard — A Claude Cowork skill that builds your entire PM operating system in under 5 minutes 107 ★

README

LiteBench — benchmark runner for LLMs and agents

[![PyPI](https://img.shields.io/pypi/v/litebench.svg)](https://pypi.org/project/litebench/) [![Python](https://img.shields.io/pypi/pyversions/litebench.svg)](https://pypi.org/project/litebench/) [![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)

[**Quick Start**](#usage) · [**Built-in Tasks**](#built-in-tasks) · [**Custom Tasks**](#custom-tasks) · [中文](README_CN.md)

litebench run gsm8k

[中文文档](README_CN.md)

What is this?

`inspect_ai` is powerful but heavy — you write Solver and Scorer classes. `lm-evaluation-harness` is thorough but research-oriented and slow to set up. `promptfoo` tests prompts, not full agents.

**LiteBench** sits in the middle: an opinionated CLI for app developers who want to benchmark their model or agent on common tasks (HumanEval / GSM8K / MMLU / MATH / TruthfulQA / ARC) without having to write a framework first.

pip install litebench

litebench list
litebench run gsm8k -m deepseek/deepseek-chat -n 50
litebench run gsm8k -m deepseek/deepseek-chat -n 50 --resume   # continue an interrupted run
litebench run humaneval -m gpt-5 -n 20
litebench run mmlu -m claude-sonnet-4-6 --subject computer_security -n 100
litebench run gsm8k -m gpt-5 -n 20 --repeat 5   # 5 attempts per task, reports pass@1 / pass@5
litebench run math -m kimi -n 50

# Custom YAML tasks
litebench run ./my-task.yaml -m gpt-4o-mini

# Compare models
litebench runs
litebench compare  
litebench export  -o run.json

Features

6 built-in tasks — HumanEval, GSM8K, MMLU, MATH-500, TruthfulQA, ARC-Challenge.
100+ model providers via litellm — OpenAI, Anthropic, Gemini, DeepSeek, Kimi, Qwen, GLM, local Ollama, and more. Shortcuts built in: -m opus, -m kimi, `-m deepsee