TensorFlow Community Reports CPU Batched Matrix Multiplication Anomaly; Certain float32 Inputs May Produce Incorrect Results

On October 23, a numerical correctness report appeared on TensorFlow GitHub: when a three-dimensional float32 tensor runs tf.matmul on a CPU, batches of 2 to 16 and a reduction dimension of at least 20,000 may produce clearly incorrect outputs without an exception or warning. The reporter observed the issue with TensorFlow 2.20.0 and 2.21.0 on Ubuntu under WSL2, using an AMD Ryzen 9 7950X. It is not yet known whether it affects other processors or operating systems. Community report

The report uses NumPy float64 results as a reference and compares batched matrix multiplication with per-item 2D multiplication and a float64 path; the latter two worked as expected in the tests. After oneDNN was disabled, the combination that had produced anomalous results also returned values close to the reference. Repeated calls with the same inputs still produced different outputs, warranting further investigation, but these comparisons are not enough to confirm the underlying cause. Test code and output

This operation is within the API’s documented usage. The official documentation says tf.matmul supports tensors with two or more dimensions: the innermost two dimensions are multiplied as matrices, and the outer dimensions represent batches. Engineering teams that see anomalies in similar configurations should check the actual operation shapes and execution path, and verify the numerical results with a small reference calculation rather than relying only on whether the workflow completed successfully. In research code, incorrect intermediate values could also invalidate later stages of an algorithm and make the issue harder to locate. Matrix multiplication documentation

TensorFlow has previously explained that oneDNN optimizations may cause small numerical differences because of changes in floating-point operation order, and provided TF_ENABLE_ONEDNN_OPTS=0 as a way to disable them. The magnitude of the reported discrepancy and the variation between repeated runs require independent verification. Teams concerned about impact can set this variable before starting TensorFlow and compare results, then weigh any performance cost. As of the time of review, the issue remains open, with no maintainer-confirmed reproduction or linked fix. Next steps include testing on other CPUs, checking whether Nightly builds are affected, and watching for the eventual fix version. Official oneDNN explanation