LLMOps 與評測工具
MLflow 3.15.2 Adds Multi-Scorer Ensembles; Immutable Dataset Versions Remain Databricks-Only
The new release can combine multiple LLM scorers using voting, averaging, or custom functions while preserving each constituent score’s rationale and source. The immutable evaluation dataset versions mentioned in the release notes currently support only Databricks; the open-source tracking backend does not yet offer equivalent functionality.
MLflow 3.15.2 brings multi-scorer composition to its GenAI evaluation interface. The new experimental `make_scorer_ensemble` can run multiple single-turn or session-level scorers, then combine their Boolean, numeric, or categorical results into a single `Feedback` object. Built-in functions include `majority_vote`, `mean`, `minimum`, `maximum`, `agg_all`, and `agg_any`; users can also supply a custom aggregation function.
This involves more than simply averaging numbers. The interface first validates the output types of the constituent scorers—for example, it rejects attempts to apply numeric averaging to yes/no categories. Single-turn and session-level scorers also cannot be mixed within the same ensemble. The aggregated result’s metadata preserves the value, rationale, and judge source of every constituent score, keeping votes on safety, relevance, or correctness traceable. When a named built-in function is used, the entire scorer ensemble can be recursively serialized and registered; a custom callable cannot complete the same serialization round-trip.
Another update is immutable evaluation dataset versions. During an evaluation run, a specific version can be resolved and its version coordinates stored in the input record. This prevents old runs from becoming impossible to reproduce after a team modifies its test cases. The capability is important for agent regression testing: if the model, prompt, judge, and dataset are not versioned together, changes in scores are difficult to attribute.
However, the brief official release notes could easily give the impression that every backend is already supported. The merged implementation is explicitly limited to Databricks evaluation datasets, and the MLflow documentation likewise states that versions cannot be supplied in non-Databricks environments. Equivalent functionality for the open-source backend remains at the proposal stage. In addition, the ensemble API is marked experimental and may change without notice. The project reports only a manual end-to-end test using three records, with no measurements of the latency, token costs, or scoring stability introduced by multiple judges. After upgrading, engineering teams should first pin the API, data backend, and tie-breaking behavior before incorporating the feature into CI quality gates.