Forking Fast Reconstructs Reasoning Branches with Change-Point Detection, Cutting the Token Budget to One-Eighth

To study when a model changes its answer during reasoning, researchers commonly use Forking Paths Analysis: they fix a baseline reasoning chain, resample many continuations from every token or sentence prefix, and then compute the distribution of final answers. This can locate the “branch points” that genuinely determine the outcome, but analyzing a single reasoning chain may require generating millions of tokens. Forking Fast starts from the premise that many sharp fluctuations in low-sample curves are merely noise caused by finite sampling, rather than evidence that the model makes a new decision at every step.

The method first uses PELT change-point detection to divide the outcome distribution into segments. Within each segment, it applies Gaussian-kernel weighting to answer counts at nearby positions, forming a Dirichlet distribution. Cross-validation selects the change-point penalty, cost function, and kernel bandwidth. Genuine sharp branches are therefore preserved, while repeated sampling information can be shared across flat regions. The authors also verify that the slope of total variation distance between independent estimates is approximately −0.49, close to the −1/2 expected from multinomial sampling noise.

The team analyzed 100 tinyMMLU questions using Llama-3-8B-Instruct and DeepSeek-R1-Distill-Llama-8B, collecting 1.77 billion tokens in total. After smoothing the low-sample data, observing once every four tokens produced an effective sample-size increase of roughly 3.3× to 5×. Combined with a sparser observation interval, this reduced the total token budget to one-eighth of the original while increasing reconstruction error only slightly. The library also provides a prefix-cached sampler, CPU reconstruction tools, data, hash lists, and an offline interactive dashboard.

This is not an inference optimization that makes models answer faster; it reduces the experimental cost of mechanistic analysis and safety auditing. The limitations are also clear: the results cover only two 8B models, five categories of multiple-choice outputs, and continuations of limited length. At the largest-magnitude branch points, smoothing may even perform slightly worse than raw counts. Credible-interval coverage is only 0.48 to 0.64, so the authors recommend using point estimates only. The key question for future work is whether the method can be extended to open-ended answers, long agent trajectories, and frontier-scale models.