Self-Play Pretraining Releases Weights at Six Scales, Learning Transferable Structure from Program Outputs

Submitted on September 24, [*Self-Play Pretraining with Zero Data*](https://arxiv.org/abs/2609.30063) proposes a pretraining method that starts from random weights: two models co-evolve and generate their own training material. The authors report that, as self-play compute increases, next-byte prediction loss on natural data such as text, images, and audio follows a fit-able scaling trend.

The process has a generator propose Brainfuck-like programs. Executing them produces byte sequences, which a learner then trains on using cross-entropy. The generator uses reinforcement learning to improve its prompts. Its reward is based on an alignment score between the learner’s gradients and recent parameter changes, and incorporates AdamW scaling. This is intended to find structures that are still useful for learning and reduce the chance that pure random noise is mistaken for difficult training material. [Method details](https://arxiv.org/html/2609.30063v1)

The authors released learner weights at six scales, ranging from about 98,000 to 24.39 million parameters. The models use a Llama-like decoder, a 256-byte vocabulary, and a 4,096-token context; checkpoints are saved every 256 rounds. This lets researchers compare how the same architecture changes over training and inspect controls using a fixed program prior and different reward designs. The release includes only seeds that completed at least 8,192 rounds, a selection criterion to keep in mind when interpreting the results. [Model card](https://huggingface.co/nourya-cohen/solomonoff-paper)

The scope of “zero data” needs to be understood precisely: natural data was not used for gradient updates, but the authors used DCLM and DNA validation losses to select hyperparameters. The main results measure byte prediction, and the scaling curve combines different models, checkpoints, and ensemble configurations. The current evidence is limited to small models, and world knowledge still has to come from real data. [Experimental setup and limitations](https://arxiv.org/html/2609.30063v1)

The released materials offer two ways to verify the results. Researchers can use the included data to redraw the figures, which requires no GPU. They can also use the evaluation script to download public corpora, prepare fixed-length sequences, and load the weights to rescore them. The scoring tool skips results that already exist, allowing runs to resume after an interruption. Redrawing the curves and rerunning model evaluations therefore differ in both required resources and strength of verification. [Reproduction guide](https://github.com/nourya-aliz/self_play_pretraining/blob/main/REPRODUCING.md)

There are still gaps for full retraining: the generator weights, optimizer state, and self-play training code have not been released, and the PCFG baseline is available only as pre-scored data. The next engineering step is to verify the evaluation results and the full training process separately, then test whether the benefits of scaling up the models justify the cost of generating training material. [Release scope](https://huggingface.co/nourya-cohen/solomonoff-paper), [baseline limitations](https://github.com/nourya-aliz/self_play_pretraining/blob/main/REPRODUCING.md)