AI 基礎設施
Azure Tests Reveal CPU–GPU Fragmentation in Agentic Workflows, While Agora Reclaims Idle Compute Without Sacrificing Tail Latency
Microsoft Azure research shows that agents repeatedly switch among model inference, tools, and orchestrators, putting the CPU back on the critical path while GPUs often sit idle. The Agora prototype can reclaim compute based on workflow patterns, but incorrectly reducing GPU allocations for parallel agents can worsen tail latency by 16×.
Agent services can no longer be treated as a series of independent LLM requests. [Research from the Microsoft Azure team](https://arxiv.org/abs/2608.04458) analyzed 24 hours of production traces and reproduced SWE-Agent, Trae, CORAL, and Owl on a server equipped with a 96-core AMD EPYC 7V12 and eight NVIDIA A100 GPUs. Each task repeatedly crosses the CPU–GPU boundary as it moves among inference, tool execution, and orchestration logic. Trae’s median host CPU utilization was only 11%, but it approached 100% when builds and tests started simultaneously. Average GPU SM activity remained below 55% across all four frameworks, showing that low average utilization does not mean there are no transient bottlenecks.
Based on these findings, the team built Agora, which dynamically harvests idle CPU cores on commodity servers, separates resources into pools for schedulers, orchestrators, and tool executors, and consolidates multiple agents onto shared inference instances. Under light load, CPU harvesting delivered 95% of the original performance for colocated workloads while slowing agents by only 2.8%. Role-aware pooling reduced tool CPU usage by as much as 46%. For Owl workloads with staggered start times, GPU memory oversubscription, state prefetching, and shared model weights reduced GPU requirements by one-third while increasing generation throughput by 82% and improving tail latency by 2.5×. [Intel’s technical materials](https://www.intel.com/content/www/us/en/content-details/916705/agentic-ai-requires-more-cpus.html) likewise note that adding host-side compute can reduce the cost of underutilized GPUs.
This is not, however, a universal prescription to provision fewer GPUs. CORAL’s agents operate in parallel; after half of their GPU capacity was removed, throughput fell by 71% and tail latency worsened by 16×. No public implementation of Agora is currently available, and the production data cannot be independently replayed. Engineering teams should first measure agent roles, bursty tool workloads, and agent overlap before deciding whether to consolidate instances or reclaim devices.