Every Little Bit Counts

Whether the goal is to speed up inferences, reduce energy consumption, or allow AI models to run on resource-constrained hardware platforms, model compression techniques are the go-to methods for achieving these objectives. Techniques like model distillation and pruning are frequently leveraged in these scenarios, but above all, developers rely on quantization. This process shrinks the size of a model’s parameters, such as weights and biases, to reduce the model’s overall size and computational complexity.

But as the old saying goes, you can’t get something for nothing. Reducing the precision of model weights is not magic. This approach may be guaranteed to save memory and processing time, but there are no guarantees that a quantized model will perform as well as the original. And if the model does not perform well enough to do its job, what value is there in shrinking it?

To avoid these sorts of issues, quantization needs to be approached very carefully. One option recently developed by NVIDIA is the NVFP4 data type. It is a 4-bit floating point format that was released alongside the Blackwell GPU architecture. Despite being only 4 bits in size, the unique design of NVFP4 has made it possible for it to represent a wide dynamic range of tensor values, and to significantly reduce the size and processing requirements of models without substantial reductions in performance.

NVFP4 achieves this through a dual-scaling approach that addresses one of the biggest challenges in low-bit quantization: maintaining numerical accuracy across a wide range of values. Like many other similar techniques, NVFP4 uses a basic E2M1 structure — 1 sign bit, 2 exponent bits, and 1 mantissa bit — but the real innovation is in how it scales values.

Each group of 16 values, called a micro-block, shares a dynamically calculated FP8 (E4M3) scaling factor. This fractional scaling allows NVFP4 to match the original distribution of values much more closely than previous approaches, such as MXFP4, which used coarse, power-of-two scaling over 32-value blocks. On top of this, a second, higher-precision FP32 scaling factor is applied at the tensor level to normalize data further and reduce errors.

This two-level scaling strategy means that NVFP4 can preserve more of the original model’s intelligence, even when compressed to just 4 bits. In benchmarking tests on models like DeepSeek-R1-0528, the accuracy drop from FP8 to NVFP4 was less than 1% across a wide range of tasks, and in one case, NVFP4 even outperformed FP8.

In terms of memory efficiency, NVFP4 reduces model size by about 3.5x compared to FP16 and about 1.8x compared to FP8. This savings directly translates to improved performance and scalability. Furthermore, because of the architectural advances in Blackwell GPUs and support for ultra-low precision operations, NVFP4 can help achieve up to 50x greater energy efficiency per token compared to earlier Hopper GPUs.

With tools like NVIDIA TensorRT Model Optimizer and LLM Compressor supporting the format, and prequantized models like Llama 3 and DeepSeek-R1-0528 already available, developers can start taking advantage of NVFP4 today. It is a meaningful step forward in making AI faster, smaller, and greener, without sacrificing performance.