Vllm Turing — Development skill for Claude Code

vllm-turing

Turing (sm75) speedups for vLLM, plus the serving setup built around them: Qwen3.8-27B (hybrid GDN + attention, 262k context, MTP head, vision) served as an INT8 W8A8 checkpoint with tensor parallelism over two RTX 2080 Ti 22 GB cards (NVLink bridge), for interactive use and for agent clients over the Anthropic and OpenAI APIs.

vLLM's fast paths target Ampere and newer. On Turing, the stock vLLM falls back to slow or missing kernels: no FP8 tensor cores, no FlashAttention 2, f16 MMA with f32 accumulation at half rate on GeForce cards, and FlashInfer without its trtllm kernels. This repository replaces those paths with hand-written CUDA kernels and fixes the parts of the serving stack that break or slow down at long context on this hardware.

Results

Measured on 2× RTX 2080 Ti 22 GB with NVLink (TP2), vLLM 0.31.0, MTP with 4 draft tokens, FP8 KV cache, context 262,144 tokens:

Decode, one stream, short context	~110 tok/s
Decode, aggregate over 1 / 2 / 4 / 6 streams	114 / 150 / 272 / 374 tok/s
Decode, one stream at 120k context	70–80 tok/s (59 before the long-context verify kernel)
Prefill, 32K / 128K prompt	~3,550 / ~2,550 tok/s
KV cache pool	~280k tokens on the GPUs + 24 GB host RAM offload
Perplexity zh / code / en, W8A8 SmoothQuant	6.55 / 1.488 / 2.461

[docs/IMPROVEMENTS.md](docs/IMPROVEMENTS.md) lists every change with its measured effect.

Contents

Path
vllm_turing/	The vLLM plugin (loaded automatically through the vllm.general_plugins entry point): Python hooks and CUDA kernels (csrc/, built on first use with nvcc and loaded with ctypes)
patches/flashinfer_spec_decode.py	Patch for vLLM's FlashInfer backend: sm75 support and FULL CUDA graphs for speculative verify
monitor/	Monitoring reverse proxy in front of vLLM (dashboard, request logs, context editing for long agent sessions, Anthropic-API fixes) and log analysis tools
scripts/	Produ