VAKRA Replays More Than 8,000 Local APIs, Revealing Sharp Accuracy Drops Under Multi-Hop and Policy Constraints

IBM researchers have released [VAKRA](https://arxiv.org/abs/2608.12282), an attempt to address a gap in agent evaluations, which often test tool calling, RAG, multi-hop reasoning, and policy compliance separately. The benchmark transforms data from sources including BIRD-SQL into more than 8,000 executable APIs spanning 62 domains, accompanied by domain documentation. Tasks require agents to perform two to five reasoning hops, converting entities or fields obtained in one step into parameters for subsequent API calls. Some workflows must also comply with tool-use restrictions expressed in natural language.

Scoring examines more than the final answer. The system re-executes an agent’s submitted tool trajectory against local services, first checking whether the tools and returned data cover the reference answer, then verifying policy compliance, and finally using a model judge to determine whether the response is supported by the tool results. This design permits multiple valid paths while distinguishing among selecting the wrong tool, supplying incorrect parameter values, and reaching an erroneous conclusion even after retrieving the correct data. The public implementation uses Docker or Podman to run SQLite, FastAPI, ChromaDB, and MCP services. The complete dataset requires about 35GB of storage, and some tests require containers to be allocated at least 8GB of memory.

With a fixed ReAct wrapper, GPT-5.5 achieves 70.4% on Dashboard API tasks that use a single-endpoint interface, but only about 50% to 51% on two BI API interfaces that require compositional operations. Most models lose more than half their accuracy as the number of reasoning hops increases. When policies make a question unanswerable, the corresponding success rates for Claude Opus 4.7 and Gemini 3 Flash Preview fall as low as 2.4%. Trajectory analysis indicates that the main bottlenecks are entity disambiguation, cross-source alignment, and field semantics—not the generation of function-call syntax.

Engineering teams can use the environment to test whether agents can genuinely navigate heterogeneous enterprise systems, but its results should not be treated as direct measures of production reliability. Much of the data and many policies are constructed from existing research corpora and generative pipelines, while the Groundedness stage still depends on an LLM judge. Claude Opus 4.7 was also evaluated on only a subset because of cost, and the public license restricts use to non-commercial research. A key next question is whether different planners, structured policy engines, and post-failure retry strategies can narrow the multi-hop performance gap without changing the underlying model.