Rails Open-Sources lemans Evaluation Framework: File Restoration, Network Isolation, and Complete Traces Reduce Opportunities for Agent Cheating

The Ruby on Rails team has open-sourced `lemans`, the framework behind the Agents on Rails leaderboard, under the MIT License, allowing developers to build, run, and audit coding-agent evaluations. It runs on Ruby 3.4 or later, uses RubyLLM for model integration, and supports either Daytona or local Docker sandboxes. An evaluation project requires only a `bench.yml` file, an environment image, task instructions, and validators. The bundled `miniswen` is a Ruby port of mini-SWE-agent and can also be used independently as a simplified coding agent.

lemans focuses not merely on launching models, but on turning the question of whether an agent actually completed its work into inspectable artifacts. Each trial preserves the agent’s patch, an ATIF-formatted interaction trace, validation logs, cost, token usage, step count, duration, and an environment summary. Oracle and no-op agents verify, respectively, that the correct solution passes and that an empty solution fails. Before scoring, the framework restores protected surfaces—including `test/`, `bin/`, and test-environment configuration—from the initial snapshot, preventing agents from manufacturing false successes by deleting tests or rewriting test runners.

Remote sandboxes can access the internet during installation, but once the agent begins working, access is restricted to model-provider hosts, reducing opportunities to search for ready-made patches or exfiltrate data. The framework also filters known provider credentials from traces written to disk. This design arose after a model under evaluation ran `env`, nearly causing an OpenRouter key to appear in public logs.

The simultaneously published Rails leaderboard shows Qwen3.8-27B scoring 48/63, with a median runtime of about 27 minutes per run and Rails API recall of just 7.9%; GPT-5.6 Terra scores 49/63, with a median runtime of about 182 seconds. This demonstrates that task success rates cannot be interpreted independently of latency, cost, and framework knowledge. However, the current results are concentrated on Rails tasks and use the same simplified agent, so they cannot yet be generalized to overall software-engineering ability. Credential filtering also covers only secrets recognized by the framework and cannot replace least-privilege access or sanitization of external logs.