Quantization

Definition

Quantization converts some model weights, activations, or computation from higher-precision formats to lower-precision or integer representations. It can make models fit on smaller hardware and may improve throughput or latency, but effects depend on the model, method, kernels, device, context length, and workload. Reduced precision can alter quality, calibration, stability, or supported operations and must be evaluated rather than assumed.

Quantization is established, with rapidly evolving formats, kernels, hardware support, and quality-efficiency frontiers.

Why it matters

Post-training quantization modifies a trained artifact after training; quantization-aware approaches account for lower precision during adaptation or training.

A smaller file is not the whole result. Runtime memory, key-value cache, batching, prompt length, and hardware kernels also shape capacity and performance.

System anatomy

Precision format

The bit width and numerical representation used for weights, activations, or computation.

Quantization method

Calibration, grouping, scaling, and whether conversion occurs during or after training.

Runtime support

Kernels and hardware capable of executing the chosen format correctly and efficiently.

Quality evaluation

Task, safety, and calibration checks against the unquantized baseline.

Important distinctions

Parameter-efficient fine-tuning

Quantization primarily changes representation and execution; PEFT learns a behavioral adaptation with a restricted set of trainable parameters.

Model pruning

Pruning removes parameters or structure, while quantization represents values with fewer bits.

Implementation signals

• Measure quality and safety on the actual workload after conversion
• Benchmark end-to-end latency, throughput, memory, and power on target hardware
• Record format, calibration data, runtime, and conversion settings

Failure modes

• Aggregate benchmarks conceal a critical task regression
• A format is portable as a file but unsupported by the target kernel
• Memory estimates ignore runtime cache growth

Sources and further study

1. Hugging Face — Quantization overview

   Surveys lower-precision model representations and the quantization methods integrated with Transformers for reducing model memory and supporting constrained inference environments.

   Use in this library: Official guidance. Quality, memory, throughput, latency, kernel support, and hardware tradeoffs differ by model and method; target-workload evaluation remains necessary.

   guidance · guidance · Published 2026
2. ggml.org — llama.cpp

   Documents a C and C++ runtime for large-model inference across a wide range of local and cloud hardware, including GGUF models, quantization support, and an API server.

   Use in this library: First-party case study. This is implementation documentation for an inference runtime, not a complete agent, security boundary, governance system, or guarantee of model quality and performance.

   engineering · engineering · Published 2026