AI 代理與開發工具
Programmatic Tool Calling Matches or Beats JSON in 11 of 14 Models, With an 18.8-Percentage-Point Gap on Long-Chain Tasks
A new study compiles API schemas into typed Python stubs, allowing models to chain or invoke tools in parallel within a single inference. The method matches or outperforms native JSON for most models on a BFCL v4 subset, but it has so far been tested only with simulated tools that return their arguments, so the results do not establish that real-world API workflows would see the same benefits.
Most agent frameworks require a model to output a JSON function call at each step, receive the result, and then perform another inference. But models capable of writing code can instead generate an entire script at once. [The Bitter Lesson of Tool Calling](https://arxiv.org/abs/2608.06370), published on August 6, converts function schemas into typed Python stubs. The model calls these functions programmatically, after which the agent harness executes the code in a subprocess and captures the results. This approach, called programmatic tool calling (PTC), can directly use variables, loops, and `asyncio.gather`, avoiding an additional model turn for every link in the chain.
The study selected 309 examples across eight task categories from the [Berkeley Function Calling Leaderboard v4](https://github.com/ShishirPatil/gorilla/tree/main/berkeley-function-call-leaderboard) and tested 14 models released between November 2024 and July 2026. PTC matched or outperformed native JSON on 11 of them. GPT-5.6-Sol rose from 72.2% to 82.8%, while Terra improved from 73.5% to 84.1%—a gain of 10.6 percentage points for each. Across chained tasks requiring at least 12 steps, PTC held an overall advantage of 18.8 percentage points. In parallel fan-out tests, 13 models performed at least as well with PTC as with JSON.
The interface comparison also revealed structural limits. When using JSON, Claude Sonnet 5 began omitting items when asked to make more than 70–72 calls in the same turn; with PTC, it still enumerated all 100 calls. Code can also reduce repeated serialization at high fan-out. Across the combined inputs and outputs of 48 calls, JSON used 5,097 tokens, compared with 3,535 for PTC. At low call volumes, however, PTC was more expensive because the prompt had to include the stubs and instructions.
Not every model can switch interfaces without modification. GPT-4o, GPT-4.1, and GPT-5.4-mini rendered line breaks in multiline code as literal `\\n` sequences, causing syntax errors in the subprocess and producing scores 19.7 to 26.9 percentage points below JSON. More importantly, the BFCL stubs merely return their arguments unchanged rather than executing real APIs, and the smaller ablation experiments included only 31 to 52 questions per condition. Engineering teams adopting PTC should next retest it in restricted sandboxes with permission controls, timeouts, and dependencies on real return values, rather than relying solely on function-argument matching rates.