TWLA: Achieving Ternary Weights and Low-Bit Activations for LLMs via Post-Training Quantization

Reading Notes

Personal reading notes. Original paper: arXiv:2606.13054v2.

arxiv_id: 2606.13054v2

Paper Background and Motivation

Although Large Language Models (LLMs) excel in language processing tasks, their massive parameter counts lead to high memory and computational costs, hindering deployment on resource-constrained platforms such as edge devices. The paper points out that Ternarization is a highly promising compression technique. By constraining weights to $\{-1, 0, +1\}$, it can significantly reduce model size and lower inference complexity. However, existing ternarization methods primarily face the following challenges:

1. Activation Distribution Issue: Existing methods struggle to handle heavy-tailed activation distributions. Consequently, they typically maintain high precision for activations, which fundamentally limits end-to-end inference acceleration.
2. Weight Distribution Mismatch: Empirical evidence suggests that the channel weight distribution of LLMs usually approximates a unimodal Gaussian distribution. This does not match the ternary codebook $\{-1, 0, +1\}$, leading to significant quantization errors when projecting to the ternary space.
3. High Training Cost: Although BitNet v2 demonstrated the joint deployment of ternary weights and 4-bit activations, it relies on expensive Quantization-Aware Training (QAT), with training costs exceeding $10^4$ GPU hours.

To address the above issues, the authors propose a Post-Training Quantization (PTQ) framework named TWLA (Ternarized Weights and Low-bit Activations). This framework aims to achieve 1.58-bit weight compression and 4-bit activation quantization (W1.58A4) while maintaining high accuracy, without requiring retraining.

Core Method: TWLA Framework

The TWLA framework consists of three tightly coupled modules designed to collaboratively optimize weight ternarization and low-bit activation quantization. The paper illustrates the overall workflow in Figure 3, which includes initialization, weight ternarization optimization, orthogonal rotation reshaping, and layer-aware mixed-precision allocation.

As shown in Figure 3, the overall design of TWLA follows the PTQ paradigm but introduces specific optimization strategies to accommodate the joint requirements of ternary weights and low-bit activations. The authors note that the core of TWLA lies in solving two key problems: first, how to reshape the weight distribution into a tri-modal form suitable for ternarization, and second, how to suppress activation outliers to support low-bit quantization.

Detailed Explanation of Key Modules

To achieve the above goals, the paper designs the following three core components:

1. Euclidean to Manifold Asymmetric Ternary Quantizer (E2M-ATQ)

This module aims to minimize layer output errors during the weight ternarization process. The authors propose a two-stage optimization process:

• Euclidean Initialization: First, initialization is performed based on Euclidean space.
• Manifold Relocation: Subsequently, parameters are relocated to manifold space. Through this asymmetric ternary quantizer, E2M-ATQ improves the calibrability of layer outputs, ensuring better layer output consistency even when weights are ternarized.

2. Kronecker Orthogonal Tri-modal Reshaping (KOTMS)

This is one of the core innovations of TWLA. The paper argues that a unimodal weight distribution is unsuitable for the ternary codebook, whereas a tri-modal distribution is a better match.

• Weight Reshaping: KOTMS applies an orthogonal rotation with a Kronecker structure to reshape weights into a ternary-friendly tri-modal distribution.
• Activation Outlier Suppression: Leveraging orthogonal equivalence, the same rotation matrix is shared for activation mixing. This shared orthogonal transformation statistically shrinks activation outliers and stabilizes the dynamic range, thereby facilitating low-bit activation quantization.
• Distribution Changes: The paper compares the distributions before and after applying TWLA in Figure 2. Before application, activation outliers hinder low-bit quantization, and weights follow a unimodal distribution; after application, activations are smoothed, and weights transform into a symmetric tri-modal form.

3. Inter-Layer Aware Activation Mixed Precision (ILA-AMP)

Since the quantization gains brought by the shared orthogonal transformation are inherently heterogeneous across different layers, certain layers with smaller gains may become bottlenecks for low-bit activation quantization. To address this, the authors propose ILA-AMP:

• Inter-Layer Interaction Cost: This strategy explicitly introduces the second-order interaction cost between adjacent layers into bit allocation for the first time.
• Joint Optimization: It unifies the sensitivity of each layer with the inter-layer coupling caused by distribution shifts into a quadratic surrogate objective function.
• Preventing Cascading Effects: By explicitly incorporating the inter-layer differences in activation gains caused by the shared orthogonal matrix into the optimization, ILA-AMP prevents error cascading triggered by a few weak layers, ensuring the stability of the overall system.

Experimental Results and Analysis

The paper conducted extensive experiments on multiple LLaMA series models to verify the effectiveness of TWLA under the W1.58A4 configuration.

• Accuracy Retention: As shown in Figure 1, TWLA maintains robustness on seven zero-shot tasks when both weights and activations are quantized (W1.58A4). In contrast, other methods (such as QuaRot, PB-LLM, etc.) exhibit significant performance degradation under 4-bit activation quantization. TWLA achieves a better accuracy-efficiency trade-off than weight-only quantization at the same memory cost.
• Comparison with SOTA: Experiments show that TWLA outperforms existing state-of-the-art 2-bit and sub-2-bit methods, pushing LLMs into the W1.58A4 regime in the field of PTQ.
• Inference Acceleration: By achieving end-to-end low-bit weights and activations, TWLA delivers significant inference acceleration, solving the problem where previous methods limited acceleration by retaining high-precision activations.

Summary and Outlook

In summary, the TWLA framework proposed by the authors successfully addresses the weight distribution mismatch and activation long-tail problems in LLM ternarization through the collaborative work of the E2M-ATQ, KOTMS, and ILA-AMP modules. The paper argues that TWLA not only achieves joint compression of 1.58-bit weights and 4-bit activations but also lowers the deployment threshold through a post-training quantization approach without retraining.

The main contributions of this work are:

1. Identifying key bottlenecks for LLMs under W1.58A4 (ternary quantizer calibrability, weight distribution mismatch, activation long-tail).
2. Proposing a PTQ framework without retraining that achieves joint support for ternary weights and low-bit activations.
3. Demonstrating through experiments that TWLA can outperform existing 2-bit and sub-2-bit methods under PTQ settings, providing a new solution for efficient LLM deployment.