Your KV Cache Doesn’t Have a Bit Problem. It Has a Geometry Problem.

At identical 2-bit precision, one decision about which axis you quantize along swings a benchmark score from 2.88 to 63.53. Keys and values need opposite treatment — and the reason is in the attention equation, not the hardware. Take Llama-2-13B. Group its key-value cache by a quantization group size of 32 into two bits, while leaving everything else in place — same model, same bit budget, same group sizes, same benchmarks. Depending on one single element of an implementation decision, CoQA…

Read the original article: