any time with Transformers, you already know consideration is the mind of the entire operation. It’s what lets the mannequin work out which tokens are speaking to one another, and that one mechanism is chargeable for nearly the whole lot spectacular LLMs do.
Consideration works with three parts: Question (Q), Key (Ok), and Worth (V) [1]. The dot product between Q and Ok is what tells the mannequin how a lot every token ought to concentrate on the others, and that’s basically the core of what consideration does.
Now, calling consideration the “mind” additionally means it comes with a value. Throughout inference, each time a brand new token is being predicted, the Ok and V matrices are recalculated for all of the earlier tokens too. So if 90 tokens are already there and the mannequin is predicting the 91st, it goes again and recomputes KV for all 90. Isn’t this repetitiveness a waste?
KV cache modified this. The thought is easy, as a substitute of recomputing, simply retailer the Ok and V matrices in VRAM and reuse them throughout inference. Sounds easy, proper? Which might be why each main LLM on the market has adopted it, the drop in latency is difficult to argue with.
Although KV cache got here as a silver lining for LLMs, it introduced up extra challenges. It launched further reminiscence overhead. This may not be a giant challenge for SLMs, however mega-LLMs with billions of parameters now grew to become harder to load on machines. Roughly 20-30% further VRAM is consumed by the KV cache alone. The larger limitation is that this overhead is just not static, it retains rising. This could develop as much as the mannequin measurement itself with lengthy contexts or extra concurrent customers, since every consumer will get their very own KV cache. To resolve this many researchers launched totally different approaches like Grouped-Question Consideration (GQA) [2], PagedAttention (VLLM) [3], Quantization (to 4-bit or 8-bit). Nevertheless, all of those helped with the reminiscence overhead challenge however accuracy needed to be compromised for that. There was no answer to each compress them and retain unique accuracy. Then got here TurboQuant from Google, which surprisingly manages to do each. The authors additionally show that this answer sits on the theoretical optimum, the very best for this class of drawback.
TurboQuant comes with two phases: PolarQuant and Residual Correction [4].
PolarQuant (Stage 1): Compresses the Ok and V matrices.
Residual Correction (Stage 2): Corrects the quantization error left after PolarQuant, recovering misplaced info.
Making use of each sequentially is what makes it totally different from conventional quantization. Here’s a visible breakdown:
That ought to offer you a transparent image of the pipeline of TurboQuant and the way it differs from the standard quantization we talked about. Earlier than we dive into every stage, allow us to uncover one other necessary factor: since we’re speaking about lowering the reminiscence overhead, what precisely does TurboQuant retailer in cache? and the way a lot much less reminiscence does it truly take up? Allow us to look into that visually beneath:
You may not totally grasp what Idx, QJL, and ε imply simply but, however they’ll grow to be clear as we unpack this pipeline step-by-step. For now, the desk above provides you the important concept: it exhibits precisely what TurboQuant shops in contrast with conventional quantization.
The important thing takeaway? Though each methods obtain equivalent compression charges (the additional ε scalar is negligible when you unfold it throughout the vector dimensions), TurboQuant retains accuracy on par with the unique full-precision mannequin. In reality, the official paper stories that TurboQuant delivers greater than 4.5–5x KV cache compression, that’s efficient 3.5–2.5 bits per channel, with near-zero accuracy loss in follow. That’s fairly phenomenal.
Now let’s stroll by the precise step-by-step stream of TurboQuant, the precise sequence we previewed within the diagram earlier.
Stage 1 (PolarQuant):
This entails two main operations in it, Rotation and LLoyd Max Quantization.
However why rotation within the first place? The main flaw of conventional quantization is how badly it handles outliers. To make this concrete, lets assume we’ve a 4 dimensional key vector for a token: [0.125, 0.103, 0.220, 6.030] (Outliers like this are literally fairly widespread in consideration keys). Now if we quantize them historically, the quantizer has to stretch its restricted ranges to cowl that large 6.030 spike. The outcome? One thing like [0, 0, 0, 1], nearly all the knowledge is misplaced.
Rotating the vector resolves this challenge. This “spinning” of vector in high-dimensional house (y = R*x, the place R is random orthogonal rotation matrix) removes the spike and immerses its vitality to the opposite coordinates making the vector distribution clean (Isotropic). The values are modified however the general magnitude stays the identical. After rotation, the identical instance vector may look one thing extra balanced like [1.42, -0.85, 2.31, 0.97].
This smoothed distribution for high-dimensional vectors brings us near gaussian distribution (in follow, the rotated vector is uniformly distributed on the unit sphere, as anticipated from the central restrict theorem). Because of this, every coordinate thus follows beta-like distribution over the vitality current within the vector.
the place d is head dimension
Tip (skip if you happen to’re not into the maths particulars): That is linked to a basic property in multivariate statistics, the place if X1, X2, …. Xd ~ N(0,1) are unbiased and identically distributed (i.i.d), then Xi2 ~ Chi-Squared distribution and there’s a theorem which states that:
Now rotation has led us to the purpose that we all know what’s the distribution like of coordinates. Now comes second main operation in stage 1: Lloyd Max Quantization:
The entire concept behind Lloyd-Max [5,6] is to position the quantization ranges (centroids) in precisely the suitable spots so the imply squared error is minimized. It’s mainly good clustering for 1D knowledge. Let’s simplify it with an instance. Taking the identical rotated vector as above: [1.42, -0.85, 2.31, 0.97]. Suppose we’re doing 1 bit quantization right here.
- Variety of centroids or ranges listed below are 2bits = 21 = 2.
- Allow us to take preliminary random ranges as [0.5, 1.5], their mid-point or boundary is (0.5 + 1.5)/2 ~ 1, thus the quantized values now grow to be [1.5, 0.5, 1.5, 0.5] (All values beneath 1 belong to 0.5 and above 1 belong to 1.5). That’s the thought of quantization proper? however what we discover is there’s a lot of error right here, i.e., MSE could be very excessive.
- Thus we’ve to seek out optimum ranges such that MSE is minimal and values are finest represented round them.
- That is achieved by Llyod Max quantization: since now new values are [1.5, 0.5, 1.5, 0.5], allotting two clusters:
-0.85, 0.97 –> 0.5 degree cluster,
1.42, 2.31 –> 1.5 degree cluster.
Taking their imply, 0.5 degree cluster imply ~ 0.06 and 1.5 cluster imply ~ 1.86.
So now the brand new ranges are modified from [0.5, 1.5] to [0.06, 1.86], and our new boundary now could be (0.06+1.86)/2 ~ 0.96, now values decrease than 0.96 belong to 0.06 degree and values above 0.96 belong to 1.86. This retains on reiterating till we attain some extent the place MSE doesn’t enhance.
Tip: There’s a basic statistical motive this works: the worth that minimizes squared error for any group of factors is just their imply.
However wait, operating this repetitive course of on each new vector throughout inference could be manner too sluggish, proper? Right here’s the place the rotation pays off once more. As a result of each coordinate now follows the identical recognized distribution (the Beta we noticed earlier), we don’t must compute a recent Lloyd-Max codebook for each new piece of information. As an alternative, the optimum codebook relies upon solely on two mounted parameters: the head dimension (d) and the variety of bits (b). We compute it as soon as, offline, and reuse it perpetually. A snippet of this codebook is proven beneath:
The quantized values aren’t saved in float, however within the type of indexes (idx) of ranges. Instance: if the degrees had been 8, then its listed (idx) kind is 0, 1, 2, 3, 4, 5, 6, 7. Thus needing 3 bits for storage of every worth.
Word: In TurboQuant’s Stage 1 (PolarQuant), the precise saved index (idx) makes use of b-1 bits per dimension (codebook measurement = 2{b-1}), not b bits. The additional bit per dimension comes from the QJL residual correction in Stage 2 (Similar was talked about in storage comparability diagram of this text above, hope now it’s clear) The desk above exhibits the overall Lloyd-Max setup; TurboQuant cleverly splits the finances to go away room for that correction.
These indexes are saved in cache till the token is evicted. Dequantization occurs on the fly at any time when that token’s Ok is required for consideration, idx is regarded up within the codebook to retrieve the float values for every index, and this matrix is then multiplied with the transpose of the unique rotation matrix to get again Ok̂ within the unique house. This completes the primary stage.
Subsequently, lastly we’re capable of extract residuals:
ε = Authentic Ok matrix – Ok̂ matrix [dequantized]
Stage 2 (Residual Correction):
Now that we’ve the residuals, essentially the most intriguing a part of TurboQuant follows.
Conventional quantization didn’t even look into the residuals. Nevertheless TurboQuant doesn’t discard this residual. As an alternative it asks a intelligent query, no matter info was misplaced throughout Stage 1 compression, can we extract its important traits quite than storing it totally? Consider it as asking easy sure/no questions in regards to the residual: is that this dimension leaning optimistic or unfavorable? The solutions to those sure/no questions are what Stage 2 preserves.
To do that, a random projection matrix S of form (d, d) is multiplied with the residual vector. The indicators of the ensuing values, both +1 or -1, are what truly get saved.
Signal(ε(seq_length, d) * S(d, d))
These signal projections are often known as the Quantized Johnson-Lindenstrauss (QJL) Remodel [7].
Word: The randomness of S is just not arbitrary, the Johnson-Lindenstrauss lemma ensures that random projections protect interior product construction with excessive chance.
However indicators alone solely seize route, not magnitude. So alongside QJL, the L2 norm of the residual (‖ε‖₂) can also be saved as a single scalar per vector. This scalar is what restores the magnitude throughout reconstruction.
Throughout dequantization, these saved signal bits are multiplied again with transposed S, then scaled by (√π/2)/d and the saved norm ‖ε‖₂. The authors present that with out this scaling issue, the sign-based estimation of the interior product is biased, this correction is what makes it unbiased. The precise system is proven beneath:
Lastly the 2 components from each phases are added collectively to get:
Ok̃ = Ok̂ + Ok̃QJL
A number of the final minute observations:
- Full TurboQuant pipeline summed up: Stage 1 handles the majority compression, Stage 2 hunts down what was misplaced and provides it again.
- So what truly sits in cache for every token is three issues: Idx, the signal bits QJL, and the scalar norm ‖ε‖₂. That’s the full compressed illustration.
- The authors formally show that this two-stage design reaches the theoretical optimum, which means no technique working inside the identical bit finances can do higher at preserving consideration dot merchandise.
Conclusion:
On the finish of the day, TurboQuant works as a result of it stops obsessing over good vector reconstruction and cleverly focuses on what the eye mechanism truly must see. As an alternative of combating the VRAM “tax” with extra complicated calibration, it simply makes use of a cleaner mathematical pipeline to get the job achieved.
As we hold pushing for longer context home windows, the KV cache bottleneck isn’t going away. However as this framework exhibits, we don’t essentially want extra {hardware}, we simply have to be extra intentional about how we deal with the info we have already got.
With the introduction of TurboQuant, is the chapter of KV Cache reminiscence administration lastly closed? Or is that this simply the muse for one thing much more highly effective?
Word: This breakdown represents my present understanding of the TurboQuant pipeline. Any errors in interpretation are completely my very own, and I encourage readers to discuss with the unique analysis for the complete mathematical proofs.
References:
[1] Vaswani, A., et al. (2017). Consideration Is All You Want. Advances in Neural Data Processing Techniques (NeurIPS 2017).
[2] Ainslie, J., et al. (2023). GQA: Coaching Generalized Multi-Question Transformer Fashions from Multi-Head Checkpoints. EMNLP 2023.
[3] Kwon, W., et al. (2023). Environment friendly Reminiscence Administration for Massive Language Mannequin Serving with PagedAttention. SOSP 2023.
[4] Zandieh, A., et al. (2025). TurboQuant: On-line Vector Quantization with Close to-optimal Distortion Price. arXiv:2504.19874.
[5] Lloyd, S. P. (1982). Least Squares Quantization in PCM. IEEE Transactions on Data Idea, 28(2), 129–137.
[6] Max, J. (1960). Quantizing for Minimal Distortion. IRE Transactions on Data Idea, 6(1), 7–12.
[7] Zandieh, A., et al. (2024). QJL: 1-Bit Quantized JL Remodel for KV Cache Quantization with Zero Overhead. AAAI 2025.
