IOL-AI Tests Reasoning on Unreleased Linguistics Olympiad Problems: A 14B Model More Than Doubles Its Baseline Score Through Its Decoding Pipeline

The [IOL-AI Challenge paper](https://arxiv.org/abs/2608.18011), released on August 18, turns the individual-contest problems from the 2026 International Linguistics Olympiad into a hidden test set. Unlike mathematics problems, these tasks do not provide the rules upfront: models must infer the grammar, vocabulary, or numeral system of an unfamiliar language from a small amount of parallel data, then answer translation, matching, and correction questions. This makes them better suited to measuring the ability to “discover rules first, then reason.”

The open challenge received 731 submissions from 46 teams, 80.7% of which ran successfully. Each run was limited to a single T4 GPU, 30 minutes, and no internet access. Scores were calculated as the geometric mean of weighted exact match and chrF. Nine of the top 10 teams used quantized Qwen2.5-14B or Qwen3-14B models. The winning entry scored 19.79 with Qwen2.5-14B-AWQ—more than double the organizers’ 9.40 baseline for the same model.

The differences came primarily from inference pipelines rather than parameter counts. Within-team ablations showed that self-consistency voting added an average of 2.08 points, while retries or formatting corrections added 1.40 points. By contrast, chain-of-thought prompting and JSON-structured output reduced scores by 1.55 and 4.66 points, respectively. Leading solutions mostly relied on greedy decoding, allocated tokens according to the remaining time, and wrote results one problem at a time to prevent a timeout from invalidating the entire submission.

The researchers also had IOL judges anonymously evaluate five systems. Claude Opus 4.8 exceeded that year’s 70-point gold-medal threshold, performing at the level of the fourth-ranked human contestant, while Gemini 3.6 Flash reached the silver-medal threshold. The two compute-constrained open submissions ranked in the bottom 5% of contestants. Although the automated scores produced the same ranking as the human judges, they overestimated weaker systems by roughly 12 to 14 points and failed to credit correct analyses that stronger models expressed in prose.

From an engineering perspective, the notable finding is that output protocols and decoding strategies can dominate results in constrained environments. However, the test set contains few problems, so variance from stochastic decoding may be substantial. The organizers also applied post-hoc formatting corrections for frontier models, meaning small score differences should not be interpreted as stable gaps in capability.