LLMOps
LiteLLM 1.98 RC Adds Shadow Evaluation for Auto Router and Fixes System Prompt Cost Misclassification
The new release can copy a small sample of real requests to a candidate router, then use a blinded judge to compare responses without affecting production results. Complexity classification also no longer treats a fixed agent system prompt as a signal of each request’s technical difficulty.
LiteLLM v1.98.0-rc.1 adds pre-deployment shadow evaluation to Auto Router. Administrators can specify an API key, sampling rate, judge model, time limit, and a maximum of 2,000 turns. The system then samples successful `/v1/chat/completions` traffic and sends the same requests to a candidate router as detached tasks. The original responses are delivered as usual, while shadow responses are never served to users. The two answers are randomly assigned A/B labels before being compared by an LLM judge, reducing position bias.
The state design is more noteworthy than the feature itself. Each sample appends only one `LiteLLM_ShadowEvalAttempt` record; wins, losses, errors, judge costs, and win rates for each tier are aggregated at read time, with no cross-pod counters maintained. Job settings are immutable except for `stopped_at`, and a partial unique index limits each key to one active job. Shadow and judge calls inherit the original request’s identity and budget, but are excluded from production request counts, TPM rate limits, savings calculations, and adoption statistics. Their cost is still charged to the key.
The same release also fixes ComplexityRouter and QualityRouter. Previously, four heuristics scored the system prompt concatenated with the user’s text. A generic CLI agent prompt of roughly 1.6 KB was enough to push the single word “hi” above the default 0.15 threshold, incorrectly routing it from a low-cost Haiku-class tier to a Sonnet-class tier. The new version calculates the code, technical, simple, and multi-step signals using only the current user text.
This is still an RC. Shadow evaluation covers only single-turn Chat Completions and cannot measure how a candidate model’s response would affect later conversation turns. Requests using content masking are also skipped. Engineering teams should begin with a low sampling rate to validate judge bias, background costs, and data egress policies before allowing the router to take over production traffic.