Quantization-Aware Training: Scales, Granularity, and Calibration

Neural network quantization sits at the intersection of numerical analysis, statistical signal processing, and hardware-aware model compression. The historical arc divides cleanly into five families, walked through paragraph-by-paragraph after the SoTA leaderboard below: production INT8 baselines (2011-2018), the binary*/*ternary extreme-low-bit line (2016-2017), learnable-scale QAT (2018-2020), mixed-precision and transformer-specific QAT (2019-2021), and the LLM-PTQ era (2022-2024). Each family is defined by the granularity¹ (per-tensor vs per-channel), the placement of trainable parameters (clipping bound, step size, zero-point), and the regime of weight-vs-activation asymmetry² that drives the headline accuracy.

Two foundational surveys define the modern vocabulary. (Nagel et al. 2021) is Qualcomm AI Research's practitioner-oriented summary of the PTQ state-of-the-art circa 2021, with concrete algorithmic descriptions of CLE, AdaRound, and bias correction; (Gholami et al. 2022) is UC Berkeley's academic literature review organizing quantization into a taxonomic tree (precision type, granularity, symmetry, scheme). (Nagel et al. 2021)'s Section 4 in particular gives the "calibrate-CLE-biasCorrect-AdaRound" PTQ recipe that achieves under 1% INT8 accuracy loss on most CNN architectures, and the four-stage ablation on MobileNetV2 INT8 shows that skipping any stage costs 0.2-0.8 points top-1; an instructive sensitivity to the calibration data³ used at each step.

The training-time line was crystallized by (Jacob et al. 2018), whose integer-only inference formulation and fake-quantization graph transformation became the reference design for TensorFlow Lite and PyTorch Eager Mode. Their work rests on the straight-through estimator of (Bengio et al. 2013), which supplies a surrogate gradient for the non-differentiable rounding operator and introduces the gradient mismatch⁴ that all later QAT analyses must absorb. (Esser et al. 2020) introduced the learned step-size formulation (LSQ), showing that the learnable-scale itself can be treated as a first-class trainable parameter; the gradient-scale factor $g = 1/\sqrt{N_W q_{\max}}$ it derives is what prevents one of the dominant training-instability⁵ modes when scales co-train with weights.

The table groups the literature into five families, walked through in chronological order. The earliest production INT8 line ((Vanhoucke et al. 2011); Krishnamoorthi; (Jacob et al. 2018)) established that PTQ + per-channel weights closes most of the FP32-to-INT8 gap on well-behaved CNNs (ResNet, Inception); the residual gap on depthwise-heavy architectures (MobileNet, EfficientNet) is what motivated the QAT line, because per-tensor PTQ on MobileNet-V2 collapses to 0.1% top-1⁶.

The binary and ternary extreme line ((Courbariaux et al. 2016) BNN, (Rastegari et al. 2016) XNOR-Net, (Li et al. 2016) TWN, (Zhu et al. 2017) TTQ) showed that 1-2 bit weights preserve a substantial fraction of FP32 accuracy when paired with a learned per-tensor scale $\alpha = \|w\|_1/n$ and a first-layer-FP32, last-layer-FP32, binary-core sandwich. The XNOR-popcount kernel and the latent-weights training trick are the two technical pillars that all subsequent extreme-low-bit work inherits; the 18-point IN1k gap of XNOR-Net to FP32 ResNet-18 is the cautionary number that pure binary deployment is microcontroller-only and that 4-8 bit activations are required for production⁷.

The learnable-scale QAT family (DoReFa-Net, PACT trainable clip $\alpha$, LSQ trainable step $s$, LSQ+ trainable $(s,z)$) made the quantizer's parameters first-class trainables. LSQ passes its own FP32 baseline only on the smaller backbone: ResNet-18 reaches 71.1% at W4A4 against a 70.5% full-precision baseline, while ResNet-50 stays under it at every width, 76.7% at W4A4 and 75.8% at W3A3 against 76.9% (Esser et al. 2020). The regularization-by-noise reading survives that correction, but only where the network has capacity to spare⁸. The gradient-scale balancing $g = 1/\sqrt{N_W q_{\max}}$ is load-bearing for convergence⁹: without it, $s$ decays to zero and the quantizer collapses.

Mixed-precision allocation ((Dong et al. 2019) HAWQ, HAWQ-V2 trace, HAWQ-V3 ILP, HAQ DDPG, FracBits) attacked bit allocation¹² as a constrained optimization problem. HAWQ-V3's one-second ILP over Hessian-trace sensitivity, BOPs, and profiled-latency constraints is the most operational of these methods; FracBits' fractional-$b$ relaxation cuts the search to a single training run. The family's load-bearing limitation is scaling failure¹³: per-layer Hessian trace estimation costs hours on LLaMA-70B, which is why LLM quantization abandoned mixed-precision in favor of weight-only + outlier-handling¹⁴.

The transformer QAT line (Q8BERT W8A8, I-BERT integer-only GELU/softmax/LN, TernaryBERT W2A8 + KD) and the LLM-PTQ era (LLM.int8() outlier sidecar, GPTQ second-order Cholesky, AWQ scale-search, SmoothQuant diagonal rescale, QLoRA NF4+LoRA, SpQR sparse 3-bit, OmniQuant LWC+LET) define the modern frontier. The "quantize the matmuls, float the normalization" template established by Q8BERT in 2019 is still the universal layout for transformer QAT, with outlier-aware methods stacking on top of it for the 6.7B+ parameter regime where coordinated activation outliers¹⁵ emerge. The W4A16 weight-only regime is essentially solved (GPTQ, AWQ within 0.1 ppl of FP16); the W4A4 regime opened only with OmniQuant's block-wise reconstruction in 2024¹⁶, and the W2A16 regime is on the active research frontier.

The table separates four regimes by precision: W8A8 (production-deployable, TFLite/ONNX), W4A4 (mid-2020s frontier, edge accelerators with INT4 tensor cores), W4A16 (LLM weight-only, memory-bandwidth-bound serving), and W3-W2 (extreme weight compression with sparse outlier safety nets). Three readings. First, learnable-scale methods (LSQ, LSQ+) consistently match or exceed PTQ methods (RTN, AdaRound) on CNN backbones at 3-4 bits, but at the cost of a full retraining budget; for LLMs this cost is prohibitive and the LLM-PTQ family wins by construction. Second, KD (TernaryBERT) is the load-bearing element for sub-4-bit weight-only on small networks: 5+ GLUE points come from the teacher signal, not from the quantization recipe itself. Third, the activation precision lags the weight precision by one or two regimes at every point in the timeline: weights at 1-2 bits with activations at 32 in 2016, weights at 4 with activations at 8 in 2020, weights at 4 with activations at 16 in 2023, and only OmniQuant 2024 closing the gap to W4A4 on LLMs. This persistent weight-vs-activation asymmetry¹⁷ is the single most predictive structural feature for choosing which family to deploy.

References