PyTorch Community Reports Compiler Cache Correctness Issue: Repeated Backpropagation with a Retained Graph May Compute Incorrect Gradients

On October 25, a member of the PyTorch community filed issue #201970, reporting that under a particular compiler-cache state, torch.compile may produce incorrect gradients without raising an exception when backward is run repeatedly on a retained computation graph. At the time of review, the issue remained open, with no fix PR listed. The reporter said they had confirmed the behavior was reproducible but were not yet sure the diagnosis was correct. Issue report

The minimal example first compiles a squaring function and runs a standard backward pass. It then resets Dynamo, recompiles, and computes gradients twice with retain_graph=True. The report shows that the expected results of 4 and 4 instead become 4 and 8. Running in eager mode or with a cold cache produces 4 both times. The AI-assisted analysis suggests that the cache key may not distinguish intermediate buffers that can be overwritten, causing later calls to reuse unsuitable backward code. This mechanism remains an unverified hypothesis. Reproduction and analysis

The official documentation says that FXGraphCache and AOTAutogradCache can store compilation results on disk, allowing separate processes to skip recompilation when they encounter the same graph and settings. They also share the configured cache root directory. So an engineering check should include cache state and execution order; testing only the initial compilation may not be sufficient. This is a verification recommendation inferred from the documentation and report. Cache configuration documentation

Another point to consider is gradient accumulation. The official API documentation says that backward accumulates gradients in leaf nodes. In the example, .grad is cleared before each gradient calculation, so ordinary accumulation does not directly explain the reported numerical difference. The documentation also distinguishes retain_graph, which retains the original graph, from create_graph, which builds a derivative graph; tests should record these settings separately. Backward documentation

Teams using retained computation graphs with compiled training can first compare results across eager execution, a cold cache, and a reused cache. Follow-up items include an independent reproduction by maintainers, affected versions, and regression tests. Current evidence is insufficient to conclude that ordinary training workflows are broadly affected.