Tevatron-Elastic Tunes Depth, Tokens, and Vector Width with a Single Retrieval Checkpoint

Tevatron-Elastic aims to address model fragmentation, a common problem in production retrieval systems where separate models are trained for low-latency, low-storage, or high-quality requirements. The framework expresses three compression axes through a single `Granularity`: `layer` determines early readout or removal of upper layers, `dim` truncates Matryoshka embeddings, and `keep_ratio` together with `pool_layer` controls intermediate token merging. During training, users specify the desired operating points in a schedule; at deployment time, they select a configuration using `prune_to` or `encode_at`.

Depth and token compression reduce encoding or scoring computation, while vector width reduces index size and nearest-neighbor search costs; these are distinct forms of savings. The framework works through the hidden-state outputs, layer lists, and pooling interfaces already provided by Hugging Face models. The authors validated paths for BERT, ModernBERT, Qwen3, Llama 3, and Mistral. Their newly proposed MLTC method can also jointly train multiple token-retention ratios within the same retrieval checkpoint.

The team trained 20 checkpoints on RLHN‑680K and evaluated retrieval and BM25 top‑100 reranking on BEIR‑15. The “elasticity tax” from joint training was not always zero: at the full operating point, the ModernBERT retriever’s nDCG@10 fell from 0.476 for the dedicated model to 0.450; BERT declined by 0.013, while Qwen3‑0.6B improved by 0.008. In single-GPU measurements, the Qwen3‑0.6B reranker maintained an MRR@10 of 0.342 at layer 16 while increasing throughput by 1.75×. At layer 4, it achieved a measured 6.86× speedup, close to the 7× predicted by the FLOPs model.

The limitations are also clear: token compression provides almost no benefit for short, eight-token queries. Although all three axes can be combined simultaneously, the paper does not yet report quality results for that configuration. The experiments are also limited to specific datasets, a single GPU, and a supervised fine-tuning recipe. The main value of this release is its unified interface and 20 inspectable sets of weights, rather than a new state-of-the-art retrieval score.