Back Home

GitHub Repo

Ollama Community Proposes Multimodal Embedding Fix to Stabilize Image and Audio Input Order

On November 11, the Ollama community proposed a fix for a bug that could produce different media orderings in the same image-and-audio embedding request. The author provided a processor-level reproduction test, but the proposal has not been merged, and vector differences in real models remain unverified.

Mattruffoni · CC BY-SA 4.0 · Image source
zh-Hant

On November 11, the Ollama community submitted PR #19653, reporting that the native /api/embed endpoint may pass the same content to the model runner in different orders when a single input object contains both image and audio. The proposal is still open, with no record of an approved review, so it should not be treated as a fix in an official release.

According to the author, the issue is in parseMultimodalItem: the code iterates over a Go map and appends the decoded media data in iteration order, which can result in either “image, audio” or “audio, image.” The MLX embedding path then uses that order to construct media segments, turning the discrepancy into a difference in the model’s input sequence. The proposed fix validates the fields first, then decodes and appends them in a fixed image-then-audio order. Technical details

The author used a test double that records runner inputs and repeatedly sent requests through the actual EmbedHandler. The test covered all six permutations of the text, image, and audio fields, running each 256 times. The author reports that the ordering bug was reproducible before the fix, while all 1,536 requests and other added cases passed afterward. These results verify the data sent to the runner; they do not measure embedding vectors from a real model. Test coverage

For retrieval systems, the concern is input reproducibility. Ollama’s official documentation describes embeddings as useful for semantic search, vector databases, and RAG. If preprocessing changes the sequence seen by the model, engineers comparing index and query results have another variable to rule out. The report does not establish that existing indexes have been damaged or that search quality has declined. Embedding use cases

The scope should also be defined conservatively: the report concerns a path that combines image and audio in the same object. The official API reference currently describes inputs as text or arrays of text, so the issue should not be generalized to all embedding requests. Teams using the relevant multimodal path should watch for maintainer review, inclusion in a release, and vector and retrieval tests with a fixed model. API reference

Sources

  1. server: keep multimodal embedding inputs in a stable order — PR #19653
  2. Embeddings
  3. Generate embeddings