Ollama Pre-Release Reuses Verified Models, Cutting Warm Embedding Request Latency by More Than 40%

Ollama merged PR #18947 on October 12, changing how inference routing looks up models, and included the change in v0.40.4-rc0, released that day. The optimization covers generate, chat, embed, and System One, with the goal of reducing lookup work repeated on every request after a model has already been loaded. The release is still marked as a pre-release. Release notes

The new flow consolidates manifest resolution for named models and execution backend selection into a single, validated cold lookup. When a runner already exists, it first confirms that the v2 tag still points to the same digest, then reuses the model object. Parser selection remains scoped to a single request, avoiding configuration changes when models are shared. The proposal also includes regression tests for tag changes, backend selection, concurrent unloading, and missing models. Technical proposal

In the maintainers’ published tests of subsequent warm requests to the same model, EmbeddingGemma 2’s MLX embedding response time fell from 17.0 ms to 7.52 ms, while nomic-embed-text’s GGUF response time fell from 13.8 ms to 8.11 ms. First-token latency in the cached chat case decreased by only about 4.8%. This shows that gains vary by workload and cannot be directly extrapolated to all models. Test data

For RAG engineers, this kind of fixed overhead is worth measuring: when short-text vectorization frequently calls the same model, model lookup time may account for a larger share of latency. That is an engineering inference based on the mechanics of the change. The official /api/embed supports both a single text input and an array of texts, and also provides keep_alive; comparisons across versions should therefore hold batch size and model residency conditions constant. Embedding API documentation

The published tests do not yet represent cold starts, long inputs, or high-concurrency services. Deployment teams can record client-side latency alongside the API’s total_duration, load_duration, and input token count to see whether gains come from load-related overhead; the official timing fields are measured in nanoseconds. Next, watch for inclusion in a stable release and observe real-world behavior when model tags are updated, backends are switched, and models are unloaded concurrently. Performance field documentation