LMBenchmark

LLM Benchmark Suite

This repository contains a comprehensive suite of benchmarks for evaluating LLM serving systems. The suite includes multiple scenarios to test different aspects of model performance.

The workload simulated in these benchmarks is a multi-round QA (question answering) task with multiple users interacting with an LLM engine concurrently. An illustration is shown below:


Available Benchmarks

1. **ShareGPT Benchmark**

   • Replays real-world conversations from ShareGPT
   • Default QPS: 1.34

2. **Short Input, Short Output (Synthetic)**

   • System prompt: 0 tokens
   • Chat history: 256 tokens
   • Answer length: 20 tokens
   • Default QPS: 15

3. **Long Input, Short Output (Synthetic)**

   • System prompt: 1000 tokens
   • Chat history: 20000 tokens
   • Answer length: 100 tokens
   • Default QPS: 0.1

Running Benchmarks

The unified script `run_benchmarks.sh` can run any combination of benchmarks with consistent configuration:

# Run all benchmarks with default QPS
./run_benchmarks.sh    all

# Run specific benchmarks with default QPS
./run_benchmarks.sh    sharegpt short-input

# Run specific benchmarks with custom QPS
./run_benchmarks.sh    sharegpt short-input 1.34 2.0 3.0

Examples

# Run all benchmarks with default QPS
./run_benchmarks.sh meta-llama/Llama-3.1-8B-Instruct http://localhost:8000 /mnt/requests/benchmark all

# Run ShareGPT and short input benchmarks with custom QPS
./run_benchmarks.sh meta-llama/Llama-3.1-8B-Instruct http://localhost:8000 /mnt/requests/benchmark sharegpt short-input 1.34 2.0 3.0

Output Files

Results are saved in CSV format with the following naming convention:

• ShareGPT: _sharegpt_output_.csv
• Short Input: `short_input_output.c