LangChain Tests Jev Agent Evaluator, Finds Lower Score Variance Across Five Fixed Cases

LangChain published an agent evaluation experiment on September 20, comparing TypeSafe AI’s Jev with three language-model judges. The study focused on whether scores remain consistent when the same execution trace is evaluated repeatedly—a factor that affects teams’ ability to distinguish code regressions from fluctuations in the evaluator itself. [Experiment announcement](https://www.langchain.com/blog/jev-agent-evals-langsmith)

The team built a weather agent using Deep Agents and saved fixed responses, evidence, and tool calls for five cases. Each judge evaluated each case 100 times. Binary pass/fail judgments were checked against labels from a single human annotator, while continuous quality scores were used to measure variance. The 500 judgments were repeated evaluations of five fixed outputs, not 500 distinct tasks. [Experimental design](https://github.com/danielgshea/jev-as-a-judge)

The authors reported that Jev’s mean within-case variance in quality scores was 0.0000149. The corresponding variances for Sonnet 4.6, GPT-5.6 Luna, and Terra were 92, 433, and 913 times as large, respectively. All of Jev’s binary judgments matched the human labels, but its continuous scores still fluctuated, so it cannot be described as a fully deterministic judge. Average latency was 0.44 seconds per call, with a cost of approximately $0.00035; both figures apply only to this workload. [Results and limitations](https://www.langchain.com/blog/jev-agent-evals-langsmith)

Jev’s interface directly returns choices, scores, or probabilities for yes/no questions, allowing applications to combine multiple small judgments without first parsing free-form text. However, probability and quality scales must be kept distinct: a 50% probability that a condition holds does not mean an answer is of middling quality. TypeSafe’s documentation recommends separating scoring factors and then weighting them in code, which also provides a starting point for examining the scoring rules. [Typed-question documentation](https://docs.typesafe.ai/primitives)

Gaps in reproducibility remain. The repository makes the frozen cases, human labels, and analysis entry points public, but parameters such as temperature and sampling seed were not explicitly set for the language models, leaving service defaults in place. The experiment metadata also did not disclose the version of the hosted Jev service. A causal link between low variance and model architecture has therefore not been established, and consistently incorrect judgments can still produce a stable but erroneous signal. Retesting also requires tracking network latency and changes to providers’ versions. [Reproduction notes](https://github.com/danielgshea/jev-as-a-judge)

For teams building Chinese-language agents, the result offers an evaluation method: freeze execution traces first, then measure correctness and repeatability separately. Before adoption, teams should add Traditional Chinese examples, ambiguous requests, and failure cases, and establish consensus among multiple human annotators. Expanding task coverage provides a stronger test of generalization than simply increasing the number of repeated evaluations of the same case.