Back Home

GitHub Repo

TensorFlow Community Reports FP16 Gradient Clipping Overflow, Potentially Zeroing or Turning Finite Inputs into NaN

A TensorFlow issue report filed November 14 says intermediate calculations on FP16 gradients may overflow, causing clipping results to become zero or NaN. The report includes reproduction results on a stable release and Nightly, but maintainer confirmation, patch status, and the full scope of impact remain unclear.

Max Eddy · CC BY-SA 4.0 · Image source
zh-Hant

On November 14, the TensorFlow community filed a numerical correctness report: when computing the norm of float16 gradients directly or clipping them, intermediate operations may overflow even when both the inputs and expected results are representable in that format. The reporter says the issue reproduces on 2.21.0 and that day’s Nightly, and provides comparisons across eager execution, tf.function, and XLA. These remain the reporter’s test results.Issue report

The minimal case is a gradient vector with 4,096 elements, each equal to 4. Its L2 norm should be 256, and with a clipping threshold of 1, each element should become 0.015625. But in the report, global_norm returns infinity, clip_by_norm outputs zeros, and clip_by_global_norm outputs NaN. The reporter attributes this to the intermediate sum of squares exceeding the representable range of FP16; an FP32 comparison produces the expected result.Reproduction and root cause analysis

The official documentation provides useful context for interpreting the results: global clipping scales values in proportion to the norm of the entire set of tensors, and when the norm is infinite, the API deliberately sets the output to NaN to indicate an error. The issue to investigate, then, is how finite gradients first produce an infinite norm; looking only at the NaN after clipping may not distinguish an abnormal original gradient from overflow in an intermediate calculation.API documentation

For engineering teams, this suggests that custom training loops should check the actual data type entering the clipping operation and monitor finite values and norms both before and after clipping. The official mixed precision guide says mixed_float16 uses FP16 for layer computations and FP32 for variables to balance performance and numerical stability. So enabling mixed precision in a model alone does not establish that every gradient clipping path is affected.Mixed precision guide

As of this review, the public issue remains open, with no maintainer confirmation of the reproduction. Follow-up areas include whether norm accumulation uses a wider data type, output type compatibility, and regression testing across CPU, GPU, and XLA. The release containing an official fix remains unconfirmed.Issue status

Sources

  1. tf.float16 gradient clipping overflows on finite inputs, silently zeroing gradients or producing NaNs
  2. tf.clip_by_global_norm
  3. Mixed precision