Precision Matters in Block Scales

This is the third post in a sequence relating to the geometry of block number formats as angle preservers. In my previous post, I argued that block number formats remain direction preservers even when their block scales are quantised all the way down to powers of two, as is common in some number representations like the MX concrete formats. The main result there was that exponent-only block scaling perturbs direction by at most about $20^\circ$ , and that $20^\circ$ is not that much in high dimensions. So we ended last time with the nice result that very coarse block-scale quantisation is relatively benign.

But that doesn’t mean power-of-two scaling is optimal. If we have a fixed budget of bits for representing block scales, how should we spend them? Specifically, should we spend them on exponent range, or on significand precision?

This post argues that the answer becomes much clearer once a high-precision tensor-wide scale is introduced, which is exactly the kind of two-level scaling used in NVIDIA’s NVFP4 format. In NVFP4, 4-bit E2M1 values are combined with an FP8 E4M3 scale for each 16-value micro-block and a second-level FP32 scale for the tensor.

With such a tensor-wide scale, the block scales are relieved of their duty to try to capture the global magnitude of the tensor. Instead, we can ask a more focused task of them: reconstruct the relative amplitudes of the blocks, so that the global direction of the represented vector is preserved.

Since drafting this post, Bardia Zadeh and I have also written Direction-Preserving Number Representations, which I blogged about separately. That paper studies the related question of what directions can be obtained when each coordinate of a vector is drawn from a finite scalar alphabet. This post is about block scales rather than scalar elements, but the same product-structured geometry reappears one level higher.

I will argue in this post that once we look at the problem that way, precision in the block scales starts to matter much more. This leads to a rough rule of thumb for the relationship between block scale formats and vector lengths.

What a tensor-wide scale changes

Suppose, as per my previous posts, that each block is represented as $\hat v_b = \beta_b^\star m_b$ , where $m_b$ is a chosen mantissa vector and $\beta_b^\star$ is the ideal real-valued block scale for that mantissa direction.

Now suppose that the final represented tensor has the form

$\tilde v = \gamma \bigoplus_{b=1}^K (\tilde \beta_b m_b)$

where

$\gamma$ is the high-precision tensor-wide scale,
$\tilde \beta_b$ is the low-precision per-block scale, and
$\bigoplus$ denotes direct sum (block concatenation).

Of course, the tensor-wide scale $\gamma$ has no effect on direction at all: it multiplies the whole tensor uniformly, so it only changes magnitude. That means the tensor-wide scale can be used to absorb the global length of the vector, leaving the block scales to encode relative block amplitudes.

In other words, once a tensor-wide scale is present, the block scales stop answering the question “how large is this tensor?” and instead answer the question “how do the blocks compare with one another?”

Exact scale-only cosine factor

Let $\hat v$ denote the ideal blockwise representation obtained using the real-valued scales $\beta_b^\star$ , and let $\tilde v$ denote the represented tensor after block-scale quantisation.

Write

$x_b = \frac{\tilde \beta_b}{\beta_b^\star}.$

Then the represented block is simply

$\tilde v_b = x_b \hat v_b.$

So scale quantisation does not change any chosen block direction, it only rescales the ideal projected blocks.

Define

$\alpha_b = \frac{|\hat v_b|^2}{\sum_j |\hat v_j|^2},$

so that $\alpha_b$ is the fraction of the ideal projected energy contained in block $b$ .

Then, exactly as in the previous post, we have

$\cos(\hat v,\tilde v) = \frac{\sum_b \alpha_b x_b}{\sqrt{\sum_b \alpha_b x_b^2}}.$

This says that directional distortion from block-scale quantisation depends only on how uneven the multiplicative scale errors $x_b$ are across blocks.

If all blocks were rescaled by the same factor, direction would be unchanged.

Two jobs for two different kinds of bits

A block-scale format does two things.

First, its exponent bits determine what range of relative block scales can be represented without clipping or underflow.

Second, its significand bits determine how accurately the in-range scales are represented.

So there are really two error sources:

tail loss, from blocks whose scales fall outside range;
in-range uneven rescaling, from finite precision within range.

The interesting question is how these trade off when the total number of scale bits is fixed.

A conservative scale-only view for fixed $K$

Suppose the tensor is divided into $K$ blocks.

If we want a format-level guarantee, a natural scale-only question is:

for a given number of blocks $K$ , how should a fixed budget of scale bits be split between exponent and significand so as to control the worst-case angular loss caused by scale quantisation?

I am deliberately saying “scale-only” here because this is not a claim about the globally optimal scalar alphabet (a problem Bardia and I cover in our preprint, linked to above), nor about the full problem of choosing the mantissa vectors. It is a conservative model of the additional angular error introduced after the block directions have already been chosen.

To make this concrete, let

$e$ be the number of exponent bits,
$p$ be the significand precision in bits, and
$k=e+p$ be the total scale-field width.

Now make one simplifying assumption: the mantissa vectors used in different blocks all have the same norm. Under that assumption, projected block energy is proportional to the square of the ideal block scale. This lets us reason directly in terms of the block scales themselves.

If the exponent field is too narrow, then very small relative block scales may underflow to zero after tensor-wide normalisation. In the worst case, one block survives at unit scale and the remaining $K-1$ blocks sit just below the lower threshold and are lost.

If the smallest representable normalised scale is $\tau_e$ , then the exponent-side cosine contribution is bounded by

$\displaystyle \cos(\hat v,z)\geq \frac{1}{\sqrt{1+(K-1)\tau_e^2}}.$

Here $z$ denotes the intermediate vector obtained by zeroing the blocks that fall below the representable scale threshold.

If, on the other hand, the surviving blocks remain in range but the scale precision is limited, then the remaining error comes from uneven in-range rescaling. Suppose the multiplicative scale errors for the surviving blocks satisfy

$\displaystyle \ell_p \leq x_b \leq u_p.$

Then the interval argument from the previous post gives

$\displaystyle \cos(z,\tilde v)\geq \frac{2\sqrt{\ell_p u_p}}{\ell_p+u_p}.$

In the common symmetric relative-error model, $\ell_p=1-u$ and $u_p=1+u$ , so this becomes

$\displaystyle \cos(z,\tilde v)\geq \sqrt{1-u^2}.$

Putting the clipping and in-range effects together gives the conservative scale-only bound

$\displaystyle \cos(\hat v,\tilde v)\geq \frac{2\sqrt{\ell_p u_p}}{\ell_p+u_p}\cdot \frac{1}{\sqrt{1+(K-1)\tau_e^2}}.$

In the symmetric relative-error model, this simplifies to

$\displaystyle \cos(\hat v,\tilde v)\geq \frac{\sqrt{1-u^2}}{\sqrt{1+(K-1)\tau_e^2}}.$

This is the key scale-only design inequality.

What this says about exponent bits

The first striking feature is that exponent bits only need to control the low-energy tail.

To make clipping negligible, it is enough to ensure that

$\displaystyle (K-1)\tau_e^2 \ll 1.$

Now the dynamic range of a floating-point-like scale grows extremely quickly with exponent width. A format-dependent way to write this is

$\displaystyle \tau_e \approx 2^{-c2^e},$

where $c$ is a positive constant depending on details such as exponent bias, reserved encodings, and whether subnormals are supported.

Substituting into the clipping condition gives

$\displaystyle (K-1)2^{-2c2^e}\ll 1.$

Solving this roughly gives

$\displaystyle e \gtrsim \log_2\log_2 K+O(1).$

The important point is the growth law. The number of exponent bits needed to control relative-tail clipping grows only like $\log\log K$ .

That is a very slow growth law: once $K$ is fixed, only a small number of exponent bits is needed before clipping becomes a second-order issue for direction.

What this says about significand bits

Once clipping is under control, the remaining scale error is dominated by in-range relative precision. That is the role of the significand.

In the symmetric relative-error model, the scale-only cosine contribution is

$\displaystyle \cos(z,\tilde v)\geq \sqrt{1-u^2}.$

Equivalently, the angular contribution is at most

$\displaystyle \arcsin(u).$

If a $p$ -bit significand gives a relative scale error of the form

$\displaystyle u\approx C_{\rm round}2^{-p},$

where $C_{\rm round}$ depends on the precise rounding convention, then asking the scale field alone to contribute at most an angle $\theta$ gives the rough condition

$\displaystyle C_{\rm round}2^{-p}\lesssim \sin\theta.$

Equivalently,

$\displaystyle p\gtrsim \log_2\left(\frac{C_{\rm round}}{\sin\theta}\right).$

So once the exponent field is “good enough”, every additional scale bit is more profitably spent on significand precision than on more dynamic range. This is the central conclusion of the scale-only model.

A simple rule of thumb

The previous discussion suggests the following design rule.

Use just enough exponent bits to make clipping of important blocks negligible. Spend the rest on significand precision.

For a tensor with $K$ blocks and a scale field of width $k$ , a rough rule of thumb is

$\displaystyle e^\star \approx \left\lceil \log_2\log_2 K \right\rceil + C_{\rm format},\qquad p^\star=k-e^\star.$

Here $C_{\rm format}$ is a small format-dependent additive correction – in a real format, it depends on details such as exponent bias, special encodings, and subnormal support. This is not meant as a precise optimum, only as a rough scale-only guide. But it makes the main point quite clearly:

exponent bits become sufficient very quickly; significand bits keep helping.

What this means for modern designs

This way of looking at the problem helps explain why a format that combines

a high-precision tensor-wide scale, and
a more precise block-scale format

looks like a very sensible design.

The tensor-wide scale deals with global magnitude. That leaves the block scales free to focus on preserving the relative block amplitudes that determine global direction.

This is exactly the tradeoff that makes NVFP4 interesting. Compared with exponent-only scaling, E4M3 block scales spend some representational power on non-power-of-two precision. The second-level FP32 tensor scale then compensates for the reduced range of the more precise E4M3 block-scale format.

There is also a useful connection with Bardia’s preprint. That paper studies the scalar alphabet inside a block, and finds that for 4-bit alphabets at the NVFP4 micro-block dimension $d=16$ , E2M1 is close to an independently optimized direction-preserving alphabet. This post studies a complementary question one level higher: once those micro-blocks have scales, how should the scale alphabet itself spend its bits?

So the two messages reinforce one another:

inside a micro-block, E2M1 is a surprisingly good product-structured scalar alphabet for direction preservation;
across micro-blocks, a tensor-wide scale makes the relative block-scale problem more important, so non-power-of-two block-scale precision becomes valuable.

From the perspective of directional reconstruction, that seems like a very good bargain.

Conclusion

The previous post showed that even exponent-only block scaling preserves direction surprisingly well.

This post goes a step further. Once a tensor-wide high-precision scale is available, the main question is no longer whether coarse block scaling is robust enough. Instead, we should ask whether the block scales are making the best possible use of their bits.

From the point of view of reconstructing global direction,

exponent bits protect against clipping of low-energy tail blocks;
significand bits improve the relative amplitudes of all the important in-range blocks.

Since exponent range grows very quickly with exponent width, only a modest number of exponent bits is needed before clipping becomes a secondary issue. After that, precision matters more.

This is a scale-only argument. It assumes the block directions have already been chosen, and studies the additional directional error caused by quantising the relative block amplitudes.

Proof sketch of the conservative scale-only bound

Readers not interested in the algebra can safely skip this section.

Assume equal mantissa norms across blocks, so that ideal projected block energy is proportional to the square of the ideal block scale.

Let $\hat v$ be the ideal blockwise projected vector after tensor-wide normalisation. We model low-precision block scales in two stages.

First, let $z$ be obtained from $\hat v$ by zeroing every block whose normalised scale falls below the smallest representable positive scale $\tau_e$ . In the worst case, one block survives at scale $1$ and the remaining $K-1$ blocks sit just below $\tau_e$ and are lost. The lost projected-energy fraction is then

$\displaystyle \eta=\frac{(K-1)\tau_e^2}{1+(K-1)\tau_e^2},$

$\displaystyle \cos(\hat v,z)=\sqrt{1-\eta}=\frac{1}{\sqrt{1+(K-1)\tau_e^2}}.$

Second, form the final represented vector $\tilde v$ by applying in-range rounding to the surviving block scales. If the surviving-block multiplicative errors satisfy

$\displaystyle \ell_p\leq x_b\leq u_p,$

then the interval bound from the previous post gives

$\displaystyle \cos(z,\tilde v)\geq \frac{2\sqrt{\ell_p u_p}}{\ell_p+u_p}.$

In the symmetric relative-error model, $\ell_p=1-u$ and $u_p=1+u$ , so this becomes

$\displaystyle \cos(z,\tilde v)\geq \sqrt{1-u^2}.$

Now write $\hat v=z+r$ , where $r$ is supported only on the clipped blocks. Since $\tilde v$ is supported only on the surviving blocks, we have $r\perp \tilde v$ , and hence

$\displaystyle \cos(\hat v,\tilde v)=\frac{\langle z,\tilde v\rangle}{|\hat v|_2|\tilde v|_2}=\frac{|z|_2}{|\hat v|_2}\frac{\langle z,\tilde v\rangle}{|z|_2|\tilde v|_2}=\cos(\hat v,z)\cos(z,\tilde v).$

Therefore

$\displaystyle \cos(\hat v,\tilde v)\geq \frac{2\sqrt{\ell_p u_p}}{\ell_p+u_p}\cdot \frac{1}{\sqrt{1+(K-1)\tau_e^2}}.$

In the symmetric relative-error model, this is

$\displaystyle \cos(\hat v,\tilde v)\geq \frac{\sqrt{1-u^2}}{\sqrt{1+(K-1)\tau_e^2}}.$

Precision Matters in Block Scales

What a tensor-wide scale changes

Exact scale-only cosine factor

Two jobs for two different kinds of bits

A conservative scale-only view for fixed $K$

What this says about exponent bits

What this says about significand bits

A simple rule of thumb

What this means for modern designs

Conclusion

Proof sketch of the conservative scale-only bound

Published by George Constantinides

Leave a comment Cancel reply

What a tensor-wide scale changes

Exact scale-only cosine factor

Two jobs for two different kinds of bits

A conservative scale-only view for fixed

What this says about exponent bits

What this says about significand bits

A simple rule of thumb

What this means for modern designs

Conclusion

Proof sketch of the conservative scale-only bound

Share this:

Related

Published by George Constantinides

Leave a comment Cancel reply

A conservative scale-only view for fixed $K$