Quantization Techniques for Long Context LLMs

The researchers at Google DeepMind just introduced "Matryoshka Quantization" (MatQuant), a clever new technique that could make deploying large language models much more efficient. The key insight? Rather than creating separate models for different quantization levels (int8, int4, int2), MatQuant leverages the nested "Matryoshka" structure naturally present in integer data types. Think of it like Russian nesting dolls - the int2 representation is nested within int4, which is nested within int8. Here are the major innovations: 1. Single Model, Multiple Precisions >> MatQuant trains one model that can operate at multiple precision levels (int8, int4, int2) >> You can extract lower precision models by simply slicing the most significant bits >> No need to maintain separate models for different deployment scenarios 2. Improved Low-Precision Performance >> Int2 models extracted from MatQuant are up to 10% more accurate than standard int2 quantization >> This is a huge breakthrough since int2 quantization typically severely degrades model quality >> The researchers achieved this through co-training and co-distillation across precision levels 3. Flexible Deployment >> MatQuant enables "Mix'n'Match" - using different precisions for different layers >> You can interpolate to intermediate bit-widths like int3 and int6 >> This allows fine-grained control over the accuracy vs. efficiency trade-off The results are impressive. When applied to the FFN parameters of Gemma-2 9B: >> Int8 and int4 models perform on par with individually trained baselines >> Int2 models show significant improvements (8%+ better on downstream tasks) >> Remarkably, an int2 FFN-quantized Gemma-2 9B outperforms an int8 FFN-quantized Gemma-2 2B This work represents a major step forward in model quantization, making it easier to deploy LLMs across different hardware constraints while maintaining high performance. The ability to extract multiple precision levels from a single trained model is particularly valuable for real-world applications. Looking forward to seeing how this technique gets adopted by the community and what further improvements it enables in model deployment efficiency! Let me know if you'd like me to elaborate on any aspect of the paper. I'm particularly fascinated by how they managed to improve int2 performance through the co-training approach. https://lnkd.in/g6mdmVjx

Exciting breakthrough in extreme low-bit quantization for Large Language Models! The good folks at Microsoft have developed VPTQ (Vector Post-Training Quantization), a novel approach to LLM compression. They achieved reduced model quantization perplexity by 0.01-0.34 on LLaMA-2, 0.38-0.68 on Mistral-7B, and 4.41-7.34 on LLaMA-3 over the SOTA at 2-bit. On paper, this looks extremely interesting. VPTQ (Vector Post-Training Quantization), GPTQ (Generative Pre-trained Transformer Quantization), and AWQ (Activation-Aware Weight Quantization) are all post-training quantization methods for large language models, but they differ in their approaches and performance characteristics. VPTQ uses Second-Order Optimization and Channel-Independent Second-Order Optimization to achieve extreme low-bit quantization (down to 2 bits) while maintaining competitive accuracy and inference speed. It outperforms GPTQ in terms of accuracy and compression ratio, especially at very low bit widths. GPTQ uses a one-shot weight quantization method based on approximate second-order information, achieving good results at 4 bits but struggling at lower precisions. AWQ, on the other hand, focuses on identifying and preserving critical weights during quantization, resulting in faster inference than GPTQ and sometimes better perplexity, though at the cost of slightly higher VRAM usage. Overall, VPTQ appears to offer the best balance of compression, accuracy, and speed, particularly for extreme low-bit scenarios. Key Steps for Implementing Vector Post-Training Quantization (VPTQ) for Large Language Models: 1. Formulate the quantization problem: - Use Second-Order Optimization to guide the quantization algorithm design. - Employ Channel-Independent Second-Order Optimization for granular vector quantization. 2. Initialize centroids: - Implement Hessian-Weighted Centroid Initialization. - Solve it as a Weighted K-means Clustering problem. 3. Quantize the model weights: - Iterate through each layer of the model. - For each Linear operator: a. If outlier elimination is enabled, quantize outlier weights first. b. Initialize centroids for remaining weights. c. Apply the VPTQ algorithm to quantize weights. d. If residual quantization is enabled, quantize the residual error. 4. Implement Residual Vector Quantization (optional): - Use multiple stages to further compress residual errors. - Employ separate lookup tables for each stage. 5. Apply outlier elimination (optional) 6. Perform layer-wise fine-tuning: - Fine-tune centroids and layer normalization parameters. - Use a small calibration dataset (e.g., 128 samples from C4). 7. Optimize for inference: - Implement efficient dequantization by reading centroids from codebooks. - Fuse dequantization and matrix multiplication operations where possible. VPTQ enables extreme compression of LLMs while maintaining remarkable accuracy, paving the way for more efficient deployment and inference of these powerful models.

Google solved one of the most important constraints of language models by reducing how much memory they need to run, which directly translates into freed compute capacity and lower serving cost. The paper “TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate” from Google Research focuses on a simple observation. LLMs do not rely on exact vector values during inference. They rely on inner products between vectors. If those similarities are preserved, the model behaves the same. The design follows from that. First, vectors are transformed to make compression tractable. A random rotation makes each dimension behave almost independently. That removes the need for complex, data-specific quantization schemes and allows each coordinate to be compressed separately with minimal loss. Second, the method fixes what compression breaks. Standard quantization distorts inner products, which directly affects attention. TurboQuant isolates that distortion and encodes it using a 1-bit correction signal based on a Quantized Johnson–Lindenstrauss transform. This restores unbiased similarity calculations with negligible overhead. The key architectural move is separation. One stage compresses efficiently. The other guarantees correctness of interactions between vectors. That is why the system can push compression without degrading performance. The empirical result is what matters. KV cache 𝗰𝗮𝗻 𝗯𝗲 𝗰𝗼𝗺𝗽𝗿𝗲𝘀𝘀𝗲𝗱 𝗯𝘆 𝗺𝗼𝗿𝗲 𝘁𝗵𝗮𝗻 𝟱𝘅 while maintaining the same downstream accuracy on long-context tasks. This brings compression close to the theoretical limit for this problem. Now the practical impact. Take a RAG system with 10M documents embedded at 1536 dimensions. Stored in float16, that is roughly 30 GB of vector data. With 5x to 8x compression, that drops to around 4–6 GB. The entire index fits in GPU memory. Retrieval becomes local, faster, and cheaper. On the generation side, the KV cache shrinks by the same factor. A system capped at 32k context due to memory can push toward 150k on the same hardware, or run several times more concurrent requests per GPU. In practice, a deployment that required 8 GPUs to serve long-context RAG queries can drop to 2–3 GPUs for the same throughput, or keep the hardware and scale traffic significantly. No retraining. No change to the model. Just a different way of encoding state during inference. This is a significant and imoactful breakthrough. Memory is the bottleneck in modern LLM systems. If you compress it without breaking similarity, you unlock longer context, higher throughput, and materially lower cost at the same time. Blog: https://lnkd.in/ei3Nb5Vv Paper: https://lnkd.in/eyJ4Hf9U

Can we compress LLMs to 2 bits without destroying their intelligence? The push for extreme quantization usually hits a wall around 2 bits. At that level, standard methods often see a massive drop in reasoning capabilities, making the models unusable. A new paper, "Fairy2i", proposes an interesting solution: switching from real numbers to complex numbers. Instead of retraining complex-valued models from scratch (which is expensive), the authors developed a framework to transform existing pre-trained models (like LLaMA-2) into a "complex-valued" form. Here is how it works: 1️⃣ The Complex Shift: They map the standard real-valued weights into a complex domain using a codebook of fourth roots of unity: {1, -1, i, -i}. This utilizes the 2-bit space more efficiently than the standard ternary {-1, 0, 1} approach used in binary quantization. 2️⃣ No Retraining Required: They prove a mathematical equivalence that allows them to start with a standard pre-trained checkpoint, meaning you don't lose the knowledge of the foundation model. 3️⃣ Recursive Residuals: To fix the precision loss, they use a "recursive residual" strategy. They quantize the main weights, calculate the error (residual), and then quantize that error again. The final weight is just the sum of these simple terms. The performance recovery is significant with this technique. A LLaMA-2 7B model compressed to an effective 2-bit precision achieved 62.00% average zero-shot accuracy, compared to the full precision FP16 baseline of 64.72%. For context, this outperforms widely used methods like GPTQ (3-bit) and AQLM (2-bit) on perplexity metrics. Because the weights are just {1, -1, -i, i}, inference becomes multiplication-free (mostly additions and swaps), which is a major win for efficiency on commodity hardware. Limitation: While the math is solid, maximizing the speed benefits of complex-valued arithmetic might require specialized kernel implementations on current hardware to fully realize the theoretical latency gains. Kudos to the researchers from Peking University for the amazing work. #MachineLearning #LLM #Quantization #DataScience #AI