模型訓練與程式碼 AI
OctoLong Trains Long-Context Models on Cross-Library Dependency Chains, Lifting the 8B Model’s RepoQA Score to 64.63
Instead of simply concatenating individual libraries into long documents, OctoLong recursively gathers implementations by following AST, language-server, and package dependency relationships. The research team released models ranging from 600M to 14B parameters, with experiments showing that dependency-dense data improves code retrieval, state tracking, and API usage.
Support for a 128K-token context window does not mean a model can track real dependencies scattered across multiple packages. OctoLong begins by selecting Python projects with at least 15 GitHub stars and more than five contributors. Inside containers with the required dependencies installed, it combines AST queries, the JEDI language server, and package managers to trace from seed functions to the classes they call and their implementations. The resulting 57,993 samples average roughly 107K tokens each, totaling 6.2 billion tokens. Based on LongPPL diagnostics, the authors conclude that these samples contain 3–23 times more context-sensitive key tokens than ordinary long-form text.
The team added this dataset to a long-context mid-training mixture of approximately 50 billion tokens, where it accounted for about 12%, followed by instruction tuning on roughly 10 billion tokens. Five models were built on dense Qwen3 base models, ranging from 600M to 14B parameters. Their context windows were extended from 32K to 128K by increasing the RoPE base frequency. To reduce degradation on short inputs, the team also linearly merged the original and mid-training checkpoints at a 1:9 ratio.
Under the authors’ standardized evaluation settings, OctoLong-8B scored 29.76 on LooGLE V2 Code, 65.01 on LongCodeQA, and 64.63 on RepoQA. Removing the cross-library data reduced the scores to 26.38, 62.17, and 61.58, respectively. The 14B model reached 39.88, 77.98, and 76.01 on the three benchmarks. However, OctoLong did not win across the board: the 4B model trailed the Qwen3-4B model built on the same base in general long-context tests, while the 14B model’s AA-LCR score was also lower than that of Qwen2.5-14B-1M. The dataset covers only English and Python, and contexts beyond 128K tokens—as well as cross-language calls through ABIs and FFIs—have not yet been validated. Engineering teams should next examine whether the training data and pipeline are released in full, along with reproducibility results on real-world monorepos, private packages, and multilingual build systems.