Llm Server

llm-server

Local LLM inference server using llama.cpp, optimized for MoE model offloading on a single-GPU consumer workstation.

What This Is

A production-ready setup for running **Qwen3.5-35B-A3B** (Mixture-of-Experts, ~3B active params per token) on an **RTX 5080 16GB** via llama.cpp with partial expert offloading. Achieves **~50 tok/s** generation speed at UD-Q4_K_XL quantization with only +0.9% perplexity loss vs the Q8_0 reference.

Includes a smart proxy with sampling presets and cold start, a benchmarking framework (`llm-bench`), Docker builds for Blackwell GPUs, and 12 sessions of documented optimization experiments.

Hardware

Component	Spec
GPU	NVIDIA RTX 5080 16GB GDDR7 (Blackwell, sm_120)
CPU	AMD Ryzen 9 9950X (32 threads)
RAM	128 GB DDR5-4800
CUDA	13.1, driver 590.48.01

Quick Start

1. Download the model

# Activate venv with huggingface-hub
source .venv/bin/activate
python -c "
from huggingface_hub import hf_hub_download
hf_hub_download('unsloth/Qwen3.5-35B-A3B-GGUF',
                'Qwen3.5-35B-A3B-UD-Q4_K_XL.gguf',
                local_dir='./models')
"

2. Build the Docker image

docker build \
  -f docker/Dockerfile.llama-cpp \
  --build-arg LLAMA_CPP_REF=b8322 \
  -t llm-server/llama-cpp:b8322 \
  docker/

3. Run via proxy (recommended)

The proxy handles cold start, idle shutdown, and sampling presets:

# Install proxy deps (one-time)
source .venv/bin/activate
pip install fastapi uvicorn httpx pyyaml

# Start proxy — it will auto-start the llama-server container on first request
python proxy.py

4. Query the API

# Basic request (no preset — pass-through)
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3.5",
    "messages": [{"role": "user", "content": "Hello!"}],
    "max_tokens": 256
  }'

# With sampling preset
curl "http://localhost:8080/v1/chat/comple