TL;DR: bfloat16 keeps FP32’s exponent width, while float16 keeps more fractional bits. That gives bfloat16 a wider range and float16 finer precision. In training, the difference affects both small gradients and the accuracy of running totals.

bfloat16 and float16 each store a number in sixteen bits. Both appear in mixed-precision training, where different parts of a computation can use different numeric formats.

Where the sixteen bits go

A floating-point number has a sign, an exponent, and a significand. The exponent sets the scale; the significand supplies the significant digits within it. Format diagrams often call the stored fractional bits the mantissa. For normalized numbers, there is also an implicit leading bit that does not occupy storage.

The layouts below follow Google’s bfloat16 description and NVIDIA’s half-precision reference:

bfloat16 retains FP32’s sign and exponent fields while dropping the bottom sixteen fraction bits from the layout. float16 uses a narrower exponent and leaves more room for the fraction. Both fields differ between the sixteen-bit formats.

Small gradients during training

During backpropagation, gradients can become too small for float16 to represent. Rounding them to zero loses the corresponding contribution to the weight update. NVIDIA’s mixed-precision training guide describes loss scaling as a way to preserve these small gradients.

The loss is multiplied by a scale factor before backpropagation, which scales the gradients too. Before the optimizer updates the weights, the gradients are divided by that factor. The intervening calculation uses larger magnitudes, so values that would otherwise underflow can survive. Choosing the scale still matters: raising it too far can cause overflow.

bfloat16’s wider exponent reduces the need for this adjustment. Google attributes its choice for Cloud TPUs to neural networks being more sensitive to exponent width than mantissa width, and gave bfloat16 FP32’s exponent size so that underflows, overflows, and NaNs behave the same way. The post describes training without loss scaling or manual code changes through the TPU software stack’s automatic conversions. It also notes that those TPUs flush subnormal bfloat16 values to zero, so matching FP32’s exponent width does not make every tiny value behave identically.

Adding small values to a running total

Precision becomes visible even at an ordinary number such as 256. In bfloat16, the next representable value above it is 258. The ml_dtypes README demonstrates what happens when 1 is added:

from ml_dtypes import bfloat16

bfloat16(256) + bfloat16(1)
# 256

The exact answer, 257, falls halfway between those two representable values; the sum rounds back to 256. If a running total has reached that point, adding values smaller than 1 will not change it either.

The README shows this happening in a sum of random values, then keeps the accumulator in FP32 to preserve the small additions. The result is converted to bfloat16 after summation. float16’s extra fraction bits provide finer spacing at the same scale, though it too has finite precision.

This is also why a mixed-precision operation need not use the same format throughout. In Google’s description of TPU matrix multiplication, the multiplication uses bfloat16 inputs and the accumulation uses FP32. The running total gets more precision even though the multiplication uses sixteen-bit inputs.