Reef Connects Online Inference, Feedback Training, and Agent Releases into a Continuous Learning Loop

Human-Agent-Society has released Reef under the Apache 2.0 license, seeking to bring model serving, interaction logging, feedback, training, candidate evaluation, and version releases—components that are typically separate—into a single continuous loop. Its core proposition is not to have the model modify itself immediately after every request, but to turn trajectories generated by online inference into versioned learning data governed by review gates, while allowing updates to both model weights and harness components such as prompts, memory, skills, tools, and orchestration rules.

Reef exposes endpoints compatible with OpenAI's `/v1/chat/completions` and Anthropic's `/v1/messages`. Each response includes an `x-reef-agent-record-id`, which applications can later use to submit scores, text, or structured feedback through `/reef/report`. The system pairs these receipts with the original interactions and handles duplicate records, session merging, and off-policy staleness. Different recipes then determine which samples qualify, when updates should be triggered, and which learning algorithms to use.

The weight-update path currently combines SGLang and Slime for serving and distributed training. After a candidate checkpoint or LoRA passes evaluation, it can be synchronized to the serving inference engines via NCCL without restarting them. The agent harness can instead be handed to Cordis, which proposes modifications based on failure trajectories. All weights, adapters, harness trees, and routing policies are treated as artifacts and stored with Git LFS. Each scenario uses an append-only release chain and compare-and-swap updates to the head, preventing older training jobs from overwriting newer versions.

This abstraction is appealing for agents that need continuous adjustment based on real-world usage data, but “passing evaluation” does not necessarily mean an update is safer. Incorrect or malicious feedback could poison the data, while fixed evaluators may reward gaming behavior. Collecting complete tool-use trajectories also raises governance concerns involving credentials and personal data. Although the project already provides a PyPI package, tutorials, tests, and multiple recipes, its performance and improvement figures are still primarily self-reported by the team. Engineering teams should next scrutinize rollback mechanisms, data deletion, online experiment isolation, and consistency across replicas during hot updates.