多代理與可組合泛化
Restricting Module Visibility Fosters Reusable Latent Communication; Restricted Models Lead by at Least 20 Points in 9 of 10 Paired Experiments
Four modules sharing Qwen2.5-0.5B and the same LoRA learned radically different composition strategies solely because of different attention masks. Limiting each module to its own evidence fragment made it easier to develop intermediate-value communication that transferred across examples, but the experiment still missed its preregistered overall success threshold by 0.0012.
A new study investigates a design question for modular AI: Does allowing every module to see the full input encourage models to memorize the entire problem rather than learn reusable local computations? The experiment created four “LLM cells” that shared a frozen Qwen2.5-0.5B-Instruct model, a single rank-8 LoRA, and a fixed-order communication channel using continuous vectors. Each message contained only two model-width vectors.
The task involved composing functions described in natural language over integers modulo 17. The researchers created ten pairs of models. Within each pair, the models had identical initialization bits, training order, token positions, parameter counts, and 20,000 update steps; the only difference was the attention mask. In the restricted condition, each cell could directly read only its assigned evidence fragment, while all other information had to arrive through the previous cell’s latent vector. In the global condition, every cell could see all four input segments.
Across both composition depths, the restricted model outperformed the global model by at least 20 percentage points in nine of the ten pairs. The paired median advantages at depths two and three were 0.7648 and 0.6050, respectively. When the researchers severed communication, every restricted model fell to chance-level performance. More importantly, transplanting a vector packet representing the same intermediate value from a different problem produced behavioral preservation rates of 0.94 to 1.00 at every interface across the six audited models. Substituting a counterfactual value also shifted the output toward the mathematically expected answer. Compared with a simple zero-vector ablation, this provides stronger evidence that the channel genuinely encoded interchangeable intermediate results.
Nevertheless, this remains a highly controlled synthetic task rather than a demonstration of general agent collaboration. The restricted models achieved a median accuracy of 0.6988 at depth three, falling short of the preregistered 0.70 threshold; the full experiment was therefore formally judged a failure. All earlier batches also failed to preserve general language capabilities, and one restricted-model checkpoint could not be recovered. The next question is whether these “information-ownership boundaries” can extend to different models, natural-language collaboration, tool-using agents, and real-world long-horizon workflows without sacrificing the models’ original capabilities.