Ollama Community Reports ROCm Graph Initialization Running Out of Memory; Partial CPU Offloading May Still Crash

On October 15, Ollama received a report of a ROCm inference failure: a user running gemma4:26b-a4b-it-qat with Ollama 0.40.3 on a 12 GB Radeon RX 6750 XT encountered an out-of-memory error during hipGraphInstantiate on a later request, even though the memory-fit mechanism had offloaded some computation to the CPU. The inference process then terminated. This is a community report from a specific environment and cannot be generalized to all AMD deployments. Public issue

The failure is worth noting because successful model loading and an initial successful request do not establish that the service can run reliably over time. The attached configuration logs show about 2.2 GB of GPU memory remaining after fitting. After the first failure, reloading the model led to a segmentation fault after prompt processing. Lowering the batch parameters did not avoid the issue. The reporter suspects that the extra memory required for graph initialization was not adequately reserved, but that explanation remains unverified. Failure logs and configuration

AMD’s documentation explains that HIP graphs first define operations and their dependencies, then use hipGraphInstantiate() to create an executable instance. Repeated executions can then reduce kernel launch and scheduling overhead. Because the graph mechanism includes initialization and resource management steps, deployments should monitor resource needs separately during weight loading, first generation, and sustained execution. Available memory after loading alone cannot show that every execution stage is safe. This is a deployment interpretation based on the documentation and report, not a confirmed root cause. AMD HIP graph documentation

After setting GGML_CUDA_DISABLE_GRAPHS=1 in the service configuration, the reporter said tests with a long prompt of about 7,000 tokens and real prompt execution no longer crashed. However, the report of no measurable slowdown came from a different 12B model that fit entirely in GPU memory; it does not establish that disabling graphs has no performance cost for the affected model. The environment also used a GPU architecture override. Follow-up work should look for independent reproduction, peak memory use during graph initialization, and whether maintainers add extra memory reservation or a fallback to regular execution after failure. The issue remains open, with no fix branch listed. Workaround and issue status