Delta Compress Llm

delta-compress-llm

**We applied video compression to LLM inference. The result: 10,000x less quantization error at the same storage cost.**

In video, you don't re-encode every pixel every frame. You store a keyframe, then just the differences. We do the same thing for the KV cache in LLM inference.

The Insight

During autoregressive decoding, consecutive tokens produce nearly identical KV cache values. The hidden state for "The cat sat on the **mat**" differs from "The cat sat on the **rug**" by only ~1% at most dimensions.

Standard KV cache quantization (Q4_0) compresses absolute values to 4 bits. **Delta-KV** compresses the tiny *difference* between tokens to 4 bits instead. Same bits, vastly less error.

Standard Q4_0:  value=0.5432  ->  quantize  ->  reconstruct=0.5100  |  error=0.0332
Delta Q4_0:     delta=0.0032  ->  quantize  ->  reconstruct=0.0030  |  error=0.0002
                                                                      ^^^^^^^^
                                                                      166x less error

The quantization error is proportional to the *range* of values being quantized. Deltas have 100x smaller range than absolute values, so the same 4 bits preserve 10,000x more information.

Benchmark Results

All benchmarks run on **4x AMD MI50 GPUs** (128GB HBM2 total) with ROCm 6.3.3.

Model 1: Llama 3.1 70B (Q4_K_M) — Dense Transformer

Perplexity (WikiText-2, 20 chunks, ctx=512) - lower is better

KV Cache Config	Perplexity	vs F16 Baseline	Verdict
F16 (baseline)	3.2840	-	-
Q8_0	3.2777	-0.19%	OK
Q4_0	3.4683	+5.61%	Degraded
Delta-KV (kf=16)	3.3002	+0.49%	Near-lossless
Delta-KV (kf=32)	3.2926	+0.26%	Near-lossless
Delta-KV (kf=64)	3.3027	+0.57%	Near-lossless

**Q4_0 loses 5.6% quality. Delta-KV loses only 0.26%.** Same 4-bit storage, 22x less degrada