Oilbird Reuses Verifier Hidden States for Semantic Drafting, Achieving 4.4× Autoregressive Decoding Speed on API-Bank

Speculative decoding first uses a cheaper method to generate multiple candidate tokens, then has the original model verify them in a single pass, reducing the cost of running a large model token by token. Training-free methods typically build a continuation pool from the current request or historical requests, then use exact context-suffix matching to find reusable segments. This design avoids training a separate draft model, but is highly sensitive to changes in order IDs, timestamps, or API parameters in tool calls. Even when the correct continuation already exists in the pool, a single differing preceding token can prevent the string index from reaching it.

Oilbird reframes the problem as “insufficient addressing” rather than “insufficient candidates.” After analyzing ten benchmarks, the research team reports that, in the test with the highest concentration of tool calls, roughly half of the candidates missed by the strongest exact-matching drafter were already present in the pool. The new method retains the existing continuation pool while additionally using hidden states produced by the verifier model on confirmed tokens as semantic keys. Because these states are already computed during verification, no separate encoder pass is required. The semantic candidates are then merged into the existing lexical draft tree, allowing candidates retrieved by both indexes to be verified together.

With the same pool size and draft budget, the paper reports that integrating Oilbird with three existing drafters increased average acceptance length by 24% to 29%. On API-Bank, it achieved 4.4× the speed of standard autoregressive decoding, compared with 3.9× for the strongest training-free baseline in the authors’ evaluation framework and 2.0× for EAGLE-3. Its technical value lies in how closely it matches the repetitive structures found in agents and API services, without requiring an auxiliary model to be trained.

For now, the figures come from a single experimental framework used by the authors and cannot be translated directly into end-to-end production latency. Engineering teams should next watch for a public code release; assess the memory overhead of the semantic index and the cost of nearest-neighbor search; and determine whether the method still delivers net gains under multi-tenant cache isolation, long contexts, and low-repetition traffic.