Gemma4 Turboquant Bench

Gemma 4 26B + TurboQuant on a Single RTX 4090


Google's [Gemma 4 26B-A4B](https://blog.google/innovation-and-ai/technology/developers-tools/gemma-4/) (April 2026) running at full 262K context with multimodal vision on a single consumer GPU, using [TurboQuant](https://arxiv.org/abs/2504.19874) 3-bit KV cache compression (ICLR 2026). 22.3 GB of 24 GB VRAM, 129 tok/s, fully local. This repository documents the configuration, build process, and a 15-test agentic coding benchmark run through [Claude Code](https://docs.anthropic.com/en/docs/claude-code).

---

Configuration

[llama.cpp](https://github.com/ggml-org/llama.cpp) is the inference engine that loads the model into GPU memory and serves it via an HTTP API. TurboQuant KV cache compression is not yet merged into mainline llama.cpp or supported by higher-level tools like Ollama, so this setup uses a [community fork](https://github.com/TheTom/llama-cpp-turboquant) compiled from source with CUDA Flash Attention enabled. Full build instructions: [docs/build-guide.md](docs/build-guide.md)

Inference Stack

Parameter	Value
Model	Gemma 4 26B-A4B-it (model card)
Architecture	MoE — 128 experts, 8 active per forward pass
Total / active parameters	25.8B / ~4B
Weight quantization	Q5_K_M (5-bit, GGUF)
KV cache quantization	turbo3 (3-bit, TurboQuant) — selected over turbo4 for faster prefill via tensor-core MMA codepath
Context window	262,144 tokens
Inference engine	TheTom/llama-cpp-turboquant (branch feature/turboquant-kv-cache)
Flash Attention	Required for turbo KV types (-fa on)
Vision	Enabled via --mmproj multimodal projector (+1.2 GB VRAM)

Hardware

Component	Spec
GPU	NVIDIA RTX 4090 — 24,564 MiB VRAM