The Stack v3 Opens Nearly 5 Trillion Code Tokens, Fixes Deduplication Error in Previous Version, and Embeds Source Code

Hugging Face and BigCode have released The Stack v3, rebuilding the large-scale source code corpus used to pretrain code language models. The full version was sourced from approximately 224 million GitHub repositories and 43.9 billion files, with a raw size of about 114 TB, covering 770 programming and markup languages. The ready-to-train `stack-v3-train` underwent license screening, quality filtering, PII redaction, and near-deduplication, resulting in approximately 173 million samples and 4.9 trillion tokens. The data is provided as 1,023 Snappy-compressed Parquet shards, and the official documentation supports streaming access through Hugging Face Datasets.

Compared with The Stack v2, the most practical change is that each record directly includes decoded UTF-8 source code. The previous version primarily stored Software Heritage identifiers, requiring users to retrieve the content separately. As a result, v3 makes it easier to create reproducible data snapshots. The new corpus reflects the state of default branches on GitHub through August 2025, adding roughly two years of code compared with v2, but it does not include complete Git history.

The team also disclosed that v2’s MinHash deduplication pipeline mistakenly segmented token sequences as character sequences, causing excessive deduplication. v3 uses proper token shingles and applies MinHash LSH to create clusters of near-duplicates. Users who do not accept the official language distribution or filtering rules can also rebuild the training set from the full data bucket, which retains duplicate-cluster identifiers.

This release provides a newer and more inspectable pretraining baseline for open code models, but “open” does not mean governance-free. The dataset is licensed under ODC-BY 1.0, while individual files retain their original licenses. Model developers must handle licensing obligations, residual secrets, and opt-out requests themselves. Community discussions have also noted that public repositories vary widely in quality and that the 2025 corpus may already contain AI-generated code. Researchers should next measure cross-repository benchmark contamination, deduplication recall, and the effects of different language distributions, rather than judging data quality solely by token count.