Skip to content
Gateway Guardian

LLM Gateway Local Development

Status: Active local runbook Last updated: 2026-03-27

Sidecar integration reference:

  • llm-gateway/docs/sidecar-agent-registration.md

1. Prerequisites

Section titled “1. Prerequisites”
  • Go 1.23+
  • just
  • curl
  • python3
  • docker + docker compose (for local Postgres)

From repo root:

Terminal window
cd /path/to/gateway
just --justfile llm-gateway/justfile --list

2. Local Modes

Section titled “2. Local Modes”

2.1 Postgres gateway mode

Section titled “2.1 Postgres gateway mode”

Start DB:

Terminal window
just --justfile llm-gateway/justfile audit-db-up

Run gateway with registry backend and registration endpoint enabled:

Terminal window
just --justfile llm-gateway/justfile gateway-registry

Or run from local TOML runtime config:

Terminal window
just --justfile llm-gateway/justfile gateway-config

This mode enables:

  • trusted registry in Postgres
  • POST /v1/agents/registrations
  • audit writes to Postgres (default in gateway-registry)
  • Anthropic/OpenAI/ChatGPT backend endpoint routing

3. Generate signing key

Section titled “3. Generate signing key”
Terminal window
SEED=$(just --justfile llm-gateway/justfile gen-key | awk -F= '/seed_b64url/{print $2; exit}')

4. Register agent in registry mode

Section titled “4. Register agent in registry mode”

Create provisioning token from the local registration token secret:

Terminal window
TOKEN=$(just --justfile llm-gateway/justfile reg-token | awk -F= '/^token=/{print $2; exit}')

Register the agent (uses SEED public key):

Terminal window
just --justfile llm-gateway/justfile register-agent "$TOKEN" "$SEED"

5. Build signed headers and run E2E

Section titled “5. Build signed headers and run E2E”

5.1 Registry mode

Section titled “5.1 Registry mode”
Terminal window
just --justfile llm-gateway/justfile e2e-prepare-registry "$TOKEN" "$SEED"
just --justfile llm-gateway/justfile e2e-success
just --justfile llm-gateway/justfile e2e-replay
just --justfile llm-gateway/justfile e2e-missing-signature

6. Useful commands

Section titled “6. Useful commands”

Gateway health:

Terminal window
just --justfile llm-gateway/justfile gateway-health

Show migrations applied:

Terminal window
just --justfile llm-gateway/justfile audit-db-migrations

Run tests:

Terminal window
just --justfile llm-gateway/justfile test

7. OTEL Validation Workflow (OBS-01.8 + OBS-02.3)

Section titled “7. OTEL Validation Workflow (OBS-01.8 + OBS-02.3)”

Run the fast headless validation flow (collector + artifact contract check with propagation matrix assertions):

Terminal window
just --justfile llm-gateway/justfile obs-otel-validate

Expected success output includes:

  • PASS: telemetry contract validated
  • propagation scenarios validated: 9
  • artifact paths:
    • llm-gateway/observability/.artifacts/traces.jsonl
    • llm-gateway/observability/.artifacts/logs.jsonl

Run the UI-backed validation flow (Jaeger + terminal-correlated logs):

Terminal window
just --justfile llm-gateway/justfile obs-otel-validate-ui

This deterministic UI validation run:

  • generates proxy telemetry via the collector export test harness
  • defaults to 3 agents with 5 requests each (15 total)
  • validates telemetry contract artifacts
  • renders span/log correlation plus per-agent request counts in terminal
  • prints Jaeger URL (http://localhost:16686/search)

UI artifacts are written to:

  • llm-gateway/observability/.artifacts/traces.jsonl
  • llm-gateway/observability/.artifacts/logs.jsonl

Run both validation flows in sequence:

Terminal window
just --justfile llm-gateway/justfile obs-otel-validate-all

Run the fast observability contract test (topology + required fields) directly:

Terminal window
just --justfile llm-gateway/justfile obs-otel-contract-test

Run propagation contract tests directly (including upgrade-handshake propagation contract checks):

Terminal window
just --justfile llm-gateway/justfile obs-otel-propagation-contract-test

Run p95 instrumentation overhead check (delta <= 5ms by default):

Terminal window
just --justfile llm-gateway/justfile obs-otel-overhead-check

Override samples or threshold:

Terminal window
just --justfile llm-gateway/justfile obs-otel-overhead-check samples=300 max_delta_ms=4.5

Run the reproducible OBS-02.3 regression flow in one command:

Terminal window
just --justfile llm-gateway/justfile obs-otel-regression-check

This runs:

  1. obs-otel-validate (artifact + propagation contract validation)
  2. obs-otel-propagation-contract-test (Go propagation contract checks)
  3. obs-otel-overhead-check (p95 delta <= 5ms)

This command writes a report artifact at:

  • llm-gateway/observability/.artifacts/overhead-report.json

When recording OBS-02.3 evidence in an issue/PR note, include:

  • command run (obs-otel-regression-check)
  • pass/fail summary from validation output
  • overhead report summary from overhead-report.json (delta_p95_ms, max_allowed_delta_ms, pass)

Collector lifecycle helpers:

Terminal window
just --justfile llm-gateway/justfile obs-otel-collector-up
just --justfile llm-gateway/justfile obs-otel-collector-down

Compatibility alias:

Terminal window
just --justfile llm-gateway/justfile obs-otel-demo

obs-otel-demo maps to obs-otel-validate-ui.

Re-render existing UI/demo artifacts without generating new telemetry:

Terminal window
just --justfile llm-gateway/justfile obs-otel-demo-show

Override UI/demo traffic volume:

Terminal window
OBS_OTEL_DEMO_AGENT_COUNT=4 OBS_OTEL_DEMO_REQUESTS_PER_AGENT=8 \
  just --justfile llm-gateway/justfile obs-otel-validate-ui

UI stack lifecycle helpers:

Terminal window
just --justfile llm-gateway/justfile obs-otel-demo-up
just --justfile llm-gateway/justfile obs-otel-demo-down

8. Local config files

Section titled “8. Local config files”

Tracked templates:

  • llm-gateway/service/configs/runtime.example.toml
  • llm-gateway/docs/local-plugin-e2e.md

Local mutable files (gitignored):

  • llm-gateway/service/configs/runtime.local.toml

9. Troubleshooting

Section titled “9. Troubleshooting”

  • invalid config: registry-db-dsn is required when registry-backend=postgres

    • pass -registry-db-dsn, or use just --justfile llm-gateway/justfile gateway / gateway-registry

  • registration_invalid_token

    • ensure token was minted with the configured registration token secret
    • ensure token not expired

  • registration_conflict

    • agent_id conflicts with existing records, or active key ownership conflict in registry

  • sig_invalid

    • ensure registered key matches signing seed
    • ensure keyid uses did:key:z...

  • obs-otel-validate fails before telemetry validation

    • rerun with shell trace for more detail: bash -x llm-gateway/observability/run_otel_validation.sh llm-gateway
    • ensure Docker is running and ports 4317/4318 are free for the local collector
    • ensure go test ./internal/gatewayhttp -run '^TestProxyCollectorExportValidation$' can reach OBS_OTEL_COLLECTOR_ENDPOINT (default 127.0.0.1:4318)
    • if overriding collector volume env vars, ensure OBS_OTEL_COLLECTOR_AGENT_COUNT * OBS_OTEL_COLLECTOR_REQUESTS_PER_AGENT >= 9 so the propagation scenario matrix is fully exercised

  • obs-otel-validate-ui (or obs-otel-demo) fails before telemetry validation

    • rerun with shell trace for more detail: bash -x llm-gateway/observability/run_otel_demo.sh llm-gateway
    • ensure Docker is running and ports 14317/14318/16686 are free for demo collector + Jaeger
    • ensure go test ./internal/gatewayhttp -run '^TestProxyCollectorExportValidation$' can reach OBS_OTEL_DEMO_COLLECTOR_ENDPOINT (default 127.0.0.1:14318)
    • if overriding demo traffic env vars, ensure OBS_OTEL_DEMO_AGENT_COUNT * OBS_OTEL_DEMO_REQUESTS_PER_AGENT >= 9

10. Codex/OpenAI local sidecar profile

Section titled “10. Codex/OpenAI local sidecar profile”

Run the thin test sidecar in Codex profile:

Terminal window
cd llm-gateway/service
GOWORK=off go run ./scripts/test_sidecar.go \
  -listen 127.0.0.1:18082 \
  -gateway-base-url http://127.0.0.1:10000 \
  -seed-b64url "$SEED" \
  -register-token "$TOKEN" \
  -client-profile codex

Optional fallback key injection (only used when incoming Authorization is missing):

Terminal window
GOWORK=off go run ./scripts/test_sidecar.go \
  -listen 127.0.0.1:18082 \
  -gateway-base-url http://127.0.0.1:10000 \
  -seed-b64url "$SEED" \
  -register-token "$TOKEN" \
  -client-profile codex \
  -openai-api-key "$OPENAI_API_KEY"

Project-local env (example):

Terminal window
export OPENAI_BASE_URL=http://127.0.0.1:18082/v1
export OPENAI_API_KEY=sk-...

client-profile=codex rewrites OpenAI-style request paths to gateway provider-prefix routes under /v1/openai_api/v1/....