Observability

Definition

Observability instruments model calls, context construction, tool requests, policy decisions, approvals, state transitions, verifier results, costs, latency, errors, and outcomes so people and systems can explain what happened. Observability informs decisions; it does not itself allow, deny, hold, revoke, or stop an action.

Telemetry foundations are established and agent-specific conventions are developing; causal interpretation remains difficult.

Why it matters

Observability tells you what happened. A control plane determines what is allowed to happen. Governance sets the rules; the control plane enforces them.

Useful telemetry connects technical events to identity, purpose, policy, risk, evidence, and accountable decisions.

System anatomy

Traces

Correlated model, tool, policy, verifier, and workflow spans.

Metrics

Aggregated reliability, quality, cost, latency, and intervention signals.

Events

Immutable state changes, approvals, blocks, incidents, and lifecycle actions.

Evidence links

References to exact artifacts and dispositions.

Important distinctions

Control plane

Observability provides information; a control plane evaluates and enforces governed constraints.

Evidence package

Telemetry is raw or structured runtime data; an evidence package binds relevant proof to a decision.

Implementation signals

• Correlate events to agent, owner, model, tool, policy, and release
• Minimize sensitive payload capture
• Test that alerts connect to a real intervention path

Failure modes

• Faux “live” telemetry with no attributable source
• Detailed logs leak prompts, credentials, or sensitive data
• A dashboard creates confidence without tested controls

Sources and further study

1. OpenTelemetry — GenAI observability

   Developing semantic conventions for model and agent telemetry across traces, metrics, and events.

   Use in this library: Standard or protocol. A specification defines an interface or control pattern; conformance alone does not establish authorization, security, or fitness for a use case.

   standard · standard · Published 2026-05-14
2. NIST CAISI — Challenges in monitoring deployed AI systems

   A current view of monitoring needs and limitations for deployed AI systems.

   Use in this library: Official guidance. Guidance describes recommended practice; citation does not prove that a control is implemented or effective in a particular environment.

   guidance · guidance · Published 2026-03-06
3. OpenAI — How we monitor internal coding agents for misalignment

   A current account of monitoring powerful coding agents that can act inside consequential development environments.

   Use in this library: First-party case study. This first-party account documents one organization, product, or implementation context and should not be generalized without local evidence.

   engineering · engineering · Published 2026-03-19