Model Optimization: From FP16 to Speculative Decoding

Serving large neural networks has always collided with hardware boundaries. In the early days of deep learning, raw compute throughput was the primary ceiling. Modern large language model inference is different: memory bandwidth and key-value cache memory dominate execution latency. Moving weights from high-bandwidth memory to compute cores costs far more time and power than executing matrix multiplication.

To bypass the memory wall, optimization techniques shifted from simple numerical precision conversions to compound pipelines that alter how models store weights and generate tokens. Here is how model optimization evolved across the last eight years.

2018 – Mixed Precision and FP16 Scaling

In 2018, standard deep learning workloads ran in single-precision 32-bit floating point (FP32). When dedicated tensor cores arrived with architectures like Volta, running in half-precision 16-bit floating point (FP16) promised double the throughput and halved memory pressure.

The primary barrier was limited dynamic range. Small gradient updates and activation tensors frequently underflowed to zero, causing training instability or degradation during inference. Libraries like Apex introduced dynamic loss scaling. By scaling values prior to backward passes and unscaling them before optimizer updates, models maintained FP32 numerical stability while running at FP16 speed.

Halving the memory footprint allowed operators to double batch sizes. Yet optimization remained uniform across the model: every weight, bias, and intermediate activation kept the identical numerical bit-width.

2020 – Post-Training INT8 Quantization

By 2020, production deployments expanded beyond research clusters to cost-sensitive cloud GPUs and edge devices. Running models in 8-bit integer precision (INT8) became the standard target for convolutional networks and early transformer architectures like BERT.

Converting floating-point weights to integers required calibration datasets. Execution runtimes measured activation histograms through thousands of sample queries, calculating scale factors that minimized Kullback-Leibler divergence between the original FP32 tensors and the quantized INT8 representations.

This approach succeeded for models with predictable variance. However, scaling transformer parameters past several billion revealed a fatal vulnerability: activation outliers. Systematic feature channels began producing extreme values that destroyed INT8 dynamic range, causing output accuracy to collapse.

2022 – Outlier-Aware Weight-Only Quantization

The rise of large language models in 2022 forced a rethink of uniform quantization. Because activation outliers could not be compressed into standard INT8 ranges without clipping errors, researchers separated weight quantization from activation calculations.

Methods like SmoothQuant and Activation-aware Weight Quantization (AWQ) solved the outlier problem. Instead of quantizing activations, AWQ left activations in FP16 while quantizing weights down to 4-bit integers. Crucially, the algorithm analyzed activation magnitude to protect the top one percent of salient weight channels, leaving those sensitive channels uncompressed while quantizing the remainder.

Preserving the numerical precision of the top one percent of salient channels prevents accuracy collapse while allowing the remaining weights to execute at 4-bit precision.

This shift made it possible to load 13-billion and 70-billion parameter models onto individual developer workstations and lower-tier enterprise GPUs without catastrophic loss in reasoning quality.

2024 – Unified Optimization and Speculative Decoding

In 2024, raw weight compression hit diminishing returns. Autoregressive token generation remained memory-bound: generating a single token still required loading every weight parameter from device memory across the bus into compute registers.

Optimization strategies transitioned from static data-type reduction to dynamic generation pipelines. Speculative decoding became a central acceleration method. A small draft model generates candidate token sequences rapidly. The primary model then validates all candidate tokens in a single parallel forward pass, transforming sequential memory operations into parallel compute verification.

Concurrently, libraries like NVIDIA Model-Optimizer brought quantization, structured sparsity, and speculative draft export into unified automated pipelines. Rather than chaining disparate scripts, teams run automated optimization directly before runtime deployment:

# Example optimization pipeline using modelopt
import modelopt.torch.quantization as mtq

# Configure 4-bit AWQ with calibration data
config = mtq.INT4_AWQ_CFG
model = mtq.quantize(model, config, forward_loop=calibrate_fn)

# Export optimized model directly for TensorRT-LLM serving
mtq.export_tensorrt_llm(model, export_dir="./optimized_engine")

Where we are now

Model optimization is no longer a manual post-processing afterthought. It has transformed into an automated compilation pipeline that couples hardware architecture with algorithm design.

Modern serving stacks combine FP4 and NVFP4 numerical formats with 2:4 structured sparsity and speculative draft verifiers. Instead of hand-tuning quantization boundaries, engineers feed models into unified optimizers that profile hardware targets and generate optimized execution graphs. Memory bandwidth remains the fundamental physical constraint of machine learning, but compound optimization pipelines ensure that every byte transferred across the bus yields multiple verified tokens.