LLMOps
Grafana Agent Observability Reaches General Availability, Connecting Online Evaluation and Agent CI Through Open-Source SDKs
Grafana has made its internally used agent observability platform generally available and released the open-source agento11y SDK, which supports five languages and multiple agent frameworks. The platform brings conversations, tool execution, costs, OTel traces, online evaluation, and offline experiments into a unified regression workflow, although the complete product remains tied to Grafana Cloud.
Grafana announced on July 30 that Agent Observability is generally available in Grafana Cloud, aiming to extend agent monitoring beyond “tracking tokens and latency” into quality control that can be integrated into deployment workflows. Its open-source agento11y repository provides SDKs for Go, Python, JavaScript/TypeScript, Java, and .NET, along with provider interfaces for Anthropic, OpenAI, and Gemini, plus integrations with LangChain, LangGraph, OpenAI Agents, LlamaIndex, Google ADK, Strands, LiteLLM, Pydantic AI, and the Vercel AI SDK. The SDKs export traces and metrics through standard OTLP, while conversation and generation data is sent to a dedicated ingest endpoint, allowing the platform to correlate versions, costs, tool calls, and evaluation scores.
The most technically substantive part of this release is its ability to turn production incidents back into reproducible tests. Teams can apply deterministic checks such as JSON validation and regular expressions to production traffic, or use LLM-as-a-judge evaluators to assess prompt injection, PII, toxicity, and task completion. Low-scoring conversations can be added to a collection, manually annotated, and incorporated into a test suite. The SDK can then run offline experiments against the same cases to compare prompts, models, or tool versions, and serve as a CI gate in pull requests. If production scores deteriorate, the results can also be converted into Grafana metrics and alerts, linking regressions to the agent version that caused the difference.
The privacy design warrants particular scrutiny: application SDKs send complete generation messages by default but do not send tool arguments or results, while coding-agent plugins transmit only metadata by default. Even so, enterprises must independently verify their policies for prompts, user identifiers, data retention, and cross-border data transfers. Another limitation is the inherent instability of LLM judges, and Grafana has not published independent benchmarks for the GA release covering evaluation accuracy, added latency, or storage costs. Only the SDKs and plugins are open source; conversation aggregation, evaluation management, and the user interface still depend on Grafana Cloud. Engineering teams should next examine compatibility with the OTel GenAI semantic conventions, data consistency across SDKs, and whether CI evaluation scores can reliably predict regressions in production traffic.