Gateway Guardian Observability

Gateway Guardian provides two complementary observability layers:

governed request records for audit, investigations, and replay-oriented troubleshooting
runtime telemetry for traces, logs, and metrics

In the current R1 implementation, these capabilities are split across the platform:

the gateway request path verifies requests, captures audit exchanges, records agent state CID history, emits OpenTelemetry traces, logs, and metrics, and exposes Prometheus metrics
the control plane exposes operator APIs that let users and the console inspect audit exchanges and state history

At a glance

Capability	Produced by	Stored or exported to	Primary access path	Primary use
Audit exchanges	`llm-gateway`	Gateway observability data store	`control-plane` API and console	Governance, forensics, request-level debugging
Agent state CID history	`llm-gateway` when `X-Viper-State-CID` is present	Gateway state history and blob storage	`control-plane` API and console	Reconstructing agent state at request time
OpenTelemetry traces, logs, and metrics	`llm-gateway`	OTLP-compatible backend via Collector or direct OTLP	OTLP-capable observability backend	Lifecycle tracing, diagnostics, plugin health
Prometheus metrics	`llm-gateway`	`/metrics` scrape endpoint	Prometheus and Grafana	SLIs, dashboards, and alerting

1. Audit exchanges

The gateway attempts to create an append-only audit exchange for each handled request after verification and routing logic runs.

What an audit exchange captures

Each record can include:

correlation_id, received_at, completed_at, and duration_ms
route_type:
- proxied
- local_deny
- upstream_error
decision_code and error_message
request metadata:
- HTTP method, scheme, host, path, and query
- full request headers
- request body bytes, body encoding, and byte size
response metadata:
- response status
- full response headers
- response body bytes, body encoding, and byte size
verified identity metadata:
- agent_id
- key_id
- signature profile
LLM routing metadata:
- provider
- operation
- request_model
- response_model
- stream_requested
- stream_observed
- endpoint_supported
- upstream_url
- upstream_host
request_preview and request_preview_role when a preview can be extracted from the request body

For chat-style requests, the preview logic walks backward through the messages array and extracts the last meaningful text segment it can find, truncating the preview to 150 characters. This is intended to provide quick operator context without requiring the full body to be opened first.

Durability and retention

Audit persistence is designed to be durable and retention-governed. In practice, that means:

audit capture continues independently of the synchronous request path
records are retained for a defined period and then aged out according to platform retention settings
audit availability is intended for both operational troubleshooting and governance workflows

Operator access

Most user-facing audit access is provided by the control-plane service:

GET /v1/agents/{agentId}/audit-exchanges
GET /v1/agents/{agentId}/audit-exchanges/timeseries
GET /v1/agents/{agentId}/audit-exchanges/{exchangeId}
GET /v1/agents/{agentId}/audit-exchanges/{exchangeId}/request-body
GET /v1/agents/{agentId}/audit-exchanges/{exchangeId}/response-body

These APIs support:

paginated audit exchange listing with limit and cursor
optional start and end filters using RFC3339 timestamps
timeseries buckets at 1m, 5m, 15m, 1h, 6h, or 1d
full body retrieval for request and response payloads

The console UI uses the same control-plane APIs for agent activity views and exchange inspection.

Access control and redaction

In authenticated deployments, read access to agent audit data requires either:

platform admin access, or
an active binding to the target agent

When audit exchange detail is returned through the control-plane, sensitive header values are redacted for:

Authorization
Proxy-Authorization
X-Api-Key
Cookie

Body retrieval endpoints return the stored bytes as either UTF-8 text or base64. Those bodies should be treated as sensitive operational data.

2. Agent state CID history

Gateway Guardian also records agent state attestation references so operators can answer a different question: not just what request was sent, but what state the agent was in when it sent it.

How it works

Viper attaches state attestation metadata to outbound requests with headers such as:

X-Viper-State-CID
X-Viper-Nonce
X-Viper-Timestamp
X-Viper-App-Type

When verification succeeds and a state CID is present, the gateway stores:

the current observed state CID for the agent
a historical sequence of observed state CIDs
the observation timestamp
the correlation ID
the app type captured at request time

The underlying attestation blobs are also made available to the platform so they can be retrieved later during an investigation or review.

This state data is separate from the audit exchange row itself, but the two are linked operationally through agent identity and correlation metadata.

What the state blob represents

The underlying StateAttestationData blob is a content-addressed snapshot of agent state at request time. It can include:

host information
Viper version and binary CID
session details and git state
app-specific metadata for supported agent apps
derived configuration references such as skills CIDs
derived request metadata such as provider, model, tools, and system prompt CID

This makes the state history useful for provenance, incident review, and change attribution.

Operator access

The control-plane exposes state history through:

GET /v1/agents/{agentId}/state-cids
GET /v1/agents/{agentId}/state-cids/{stateCID}/blob

The first endpoint returns the current snapshot plus recent history. The second resolves the underlying blob and returns it as UTF-8 or base64, depending on content.

3. OpenTelemetry integrations

OpenTelemetry in the current platform is implemented in the llm-gateway process. The control-plane serves stored observability data, but this R1 document refers to the gateway’s OTEL behavior across request lifecycle instrumentation and plugin runtime telemetry.

Signals and export model

The gateway initializes:

tracing
OTEL log emission
OTEL metric emission

It supports:

OTLP over http/protobuf
OTLP over http as an alias for http/protobuf
OTLP over grpc

http/protobuf is the default when no protocol is explicitly set.

The default service identity is:

service.name=guardian-llm-gateway

service.version is resolved from build information when available.

Common exporter configuration is provided through standard OpenTelemetry environment variables, including:

OTEL_TRACES_EXPORTER
OTEL_LOGS_EXPORTER
OTEL_METRICS_EXPORTER
OTEL_EXPORTER_OTLP_ENDPOINT
OTEL_EXPORTER_OTLP_PROTOCOL
OTEL_EXPORTER_OTLP_HEADERS
OTEL_RESOURCE_ATTRIBUTES

Per-signal endpoint, protocol, and header overrides may also be used when traces, logs, and metrics need to route differently.

OTEL_RESOURCE_ATTRIBUTES can add backend-specific resource dimensions such as deployment environment. The gateway always controls service.name and service.version.

Runtime behavior

The gateway’s OTEL bootstrap is fail-open:

if no exporter configuration is present for a signal, it uses a no-op provider for that signal
if OTEL_TRACES_EXPORTER=none, OTEL_LOGS_EXPORTER=none, or OTEL_METRICS_EXPORTER=none, it uses no-op providers for that signal
if exporter initialization fails, it degrades to no-op providers instead of failing the process

This means observability misconfiguration should not take the request path down.

At startup, the gateway writes process logs with OTEL bootstrap status and non-fatal warnings when exporters are missing, disabled, unsupported, or unable to initialize.

W3C propagation contract

Gateway trace propagation is backend-agnostic and uses the W3C model:

inbound traceparent, tracestate, and baggage are accepted when valid
server spans continue valid inbound lineage
malformed inbound trace context does not fail the request
outbound proxy egress always injects gateway-owned W3C trace context
stale outbound trace headers are removed before new context is injected
malformed baggage is dropped fail-open while request and trace propagation continue

When configured, async request.end plugin work also carries W3C trace propagation so post-response plugin telemetry remains connected to the originating request trace.

Gateway lifecycle traces

The primary request instrumentation name is:

guardian.gateway.lifecycle

Each proxied request produces a request-centric trace:

gateway.request
  gateway.request.prepare
  gateway.plugin.invocation     # request.start plugins, when matching plugins run
  gateway.verification
  gateway.proxy
    gateway.plugin.invocation   # proxy.pre plugins, when matching plugins run
    gateway.upstream
      gateway.plugin.invocation # proxy.post plugins, when matching plugins run

The root gateway.request span represents the full gateway request. It includes viper.state_cid when the inbound request provides X-Viper-State-CID.

Child spans represent:

request preparation and route/model extraction
signature verification and authorization decisions
plugin invocation
proxying to the configured upstream provider
upstream response handling

Lifecycle log events and attributes

The gateway emits structured OTEL lifecycle logs that mirror request progress. Every lifecycle log carries correlation_id and event.name. Logs emitted on the synchronous request path also carry event.sequence so backends can reconstruct the order of events for a request.

Stable lifecycle events include:

gateway.request.start
gateway.request.prepare.start
gateway.request.prepare.end
gateway.verification.start
gateway.verification.end
gateway.proxy.start
gateway.plugin.invocation.start
gateway.plugin.invocation.end
gateway.plugin.invocation.queued
gateway.upstream.start
gateway.upstream.end
gateway.proxy.end
gateway.request.end

Error-oriented proxy log events are also emitted from guardian.gateway.proxy for exceptional conditions such as:

gateway.proxy.request.read_error
gateway.proxy.provider_route.error
gateway.proxy.policy.error
gateway.proxy.policy.shadow_mismatch
gateway.proxy.upstream.error
gateway.proxy.agent_state.error
gateway.proxy.audit.encode.error
gateway.proxy.audit.enqueue.error

Common trace and log attributes include:

correlation_id
event.sequence
viper.state_cid
agent.id
signer.key_id
policy.decision_code
gateway.route_type
gateway.route_supported
llm.provider
llm.operation
llm.request_model
llm.response_model
llm.stream_requested
llm.stream_observed
http.method
http.status_code
gateway.duration_ms
gateway.uri
upstream.uri
verification.outcome
verification.decision_code
error

Completion severities are outcome-aware:

INFO for successful proxy responses
WARN for local denials and other 4xx outcomes
ERROR for upstream failures and 5xx outcomes

Plugin and hook observability

When gateway plugins are enabled, plugin work is represented in the same request lifecycle. Plugin invocations emit gateway.plugin.invocation spans and matching lifecycle log events.

Supported hook stages are:

request.start
proxy.pre
proxy.post
request.end

request.end hooks run asynchronously after the response path. Their queued work preserves trace propagation and continues lifecycle sequencing from the originating request. Worker invocations are emitted as gateway.plugin.invocation spans with hook.name=request.end.

Plugin observability attributes include:

hook.name
plugin.id
plugin.mode
plugin.runtime
plugin.order_index
plugin.outcome
plugin.decision_code
plugin.status_code
plugin.error_code
plugin.latency_ms
snapshot.version

Plugin outcomes include:

allow
deny
deny_ignored
error

Plugin denials are policy outcomes, not OTEL span errors. Plugin spans are marked as errors only for execution or context failures, such as plugin runtime errors, timeouts, or invalid context writes.

OTEL metrics

When gateway plugins are enabled, the gateway emits plugin runtime metrics through the guardian.llm-gateway.hooks instrumentation scope when OTEL metrics export is configured.

Plugin runtime metrics include:

Metric	Type	Description
`guardian.plugin.request_end.enqueued_total`	Counter	Async `request.end` work items enqueued
`guardian.plugin.request_end.executed_total`	Counter	Async `request.end` work items executed
`guardian.plugin.request_end.retried_total`	Counter	Async `request.end` work items retried
`guardian.plugin.request_end.dropped_total`	Counter	Async `request.end` work items dropped
`guardian.plugin.request_end.dlq_total`	Counter	Async `request.end` work items moved to the DLQ
`guardian.plugin.request_end.ack_failures_total`	Counter	Async queue acknowledgement failures
`guardian.plugin.snapshot_load_failures_total`	Counter	Plugin snapshot load failures
`guardian.plugin.snapshot_cache_fallbacks_total`	Counter	Plugin snapshot cache fallback events
`guardian.plugin.invocations_total`	Counter	Plugin invocations by hook, plugin, mode, result
`guardian.plugin.latency_ms`	Histogram	Plugin invocation latency in milliseconds
`guardian.plugin.request_end.queue_records`	Gauge	Current queued `request.end` records
`guardian.plugin.request_end.queue_bytes`	Gauge	Current queued `request.end` bytes
`guardian.plugin.active_snapshot_version`	Gauge	Active plugin snapshot version

Invocation metrics are labeled by hook, plugin ID, plugin mode, and outcome.

Payload policy

Lifecycle OTEL logs do not capture request bodies, response bodies, or raw headers. Payload inspection remains in the audit subsystem and plugin previews, where retention and redaction are controlled separately.

4. Prometheus metrics

Prometheus scraping is separate from OTEL export. The gateway exposes a dedicated metrics endpoint on a separate listener.

Current gateway-specific metrics are:

Metric	Type	Labels	Description
`gateway_http_requests_total`	Counter	`agent_id`, `route`	Total inbound gateway requests
`gateway_http_request_errors_total`	Counter	`status_code`, `agent_id`	Total `4xx` and `5xx` gateway responses

Standard Go runtime and process collectors are also registered.

OTLP metric export and Prometheus scraping are independent. Operators may bridge them in their observability backend, but the gateway exposes them through separate runtime paths.

Metrics labeling behavior

To keep label cardinality stable:

route labels use templates rather than raw request paths
the metrics endpoint does not count itself
agent_id is populated after successful identity verification
requests without a resolved agent are labeled as unknown

Examples of route labels include:

/health
/v1/agents/registrations
/v1/blobs/{cid}
/v1/agents/{agentId}/state-cids
/v1/anthropic_api/*
/v1/openai_api/*
/v1/chatgpt_backend/*

5. Recommended usage model

The observability stack is designed to be used in layers:

use audit exchanges when exact request and response evidence matters
use state CID history when operator review needs to include agent runtime state, not just payloads
use OTEL traces and logs for live debugging, request lifecycle reconstruction, correlation, and backend-specific search
use OTEL metrics for plugin runtime health, async queue visibility, and plugin latency
use Prometheus metrics for dashboards, rates, error ratios, and alerting

In practice:

audit is the governed source of truth for what the gateway handled
OTEL is the live transport for distributed telemetry
Prometheus is the lowest-friction source for gateway request health metrics

Current scope notes

OTEL in R1 covers traces, logs, and plugin runtime metrics from llm-gateway
user-facing audit and state inspection flows are primarily surfaced through control-plane and the console
audit bodies and state blobs should be handled as sensitive data and governed with the same care as agent traffic itself