llm-gateway/docs/adr/0006-learning-system-integration.md

ADR-0006: Learning System Integration & Per-Agent Metrics

Date: 2026-04-19 Status: accepted Deciders: Rene Fichtmueller

Context

The multi-agent architecture (ADR-0005) connects heterogeneous clients (Claude Code, Codex, ChatGPT, Ollama) to a shared LLM Gateway with independent adapter layers. Each agent has different:

• Request patterns (IDE completions vs full conversations)
• Model preferences (Claude Code needs fast inference, ChatGPT clients expect GPT models)
• Success criteria (IDE: response latency + relevance, ChatGPT: token count + completion quality)
• Failure tolerance (IDE: silent fallback acceptable, ChatGPT: explicit error required)

The learning engine (Phase 2D) currently optimizes globally across all traffic. This creates a mismatch: optimizations for ChatGPT streaming may degrade IDE completions, and per-agent feedback is lost in aggregation.

Forces:

• Learning efficiency requires per-agent signal isolation (what helps Claude Code may hurt ChatGPT)
• Agents have distinct success metrics — cannot optimize for all simultaneously
• Fallback chains should be tuned per agent (IDE tolerates Ollama, ChatGPT may reject it)
• Cost attribution: multi-tenant billing requires knowing which agent consumed tokens

Decision

Extend the learning system to track per-agent metrics in parallel with global optimization:

1. Per-Agent Metric Collection

• Agent-scoped request log: gateway_request_log → agent_id + model + latency_ms + tokens_{in,out} + confidence + fallback_used
• Agent request registry: track request volume by agent and model tier (fast/medium/large)
• Agent-specific latency targets: Claude Code ≤100ms, ChatGPT ≤500ms (streaming chunk), Ollama-based adapters ≤2s

2. Agent-Scoped Learning Metrics

• Confidence evolution: Per-agent score tracks "how well does model X work for agent Y"
  • Initialized from global baseline (ADR-0003)
  • Updated on every agent request based on observed outcome (success/fallback)
  • Separate from global confidence — agent-specific signal only
• Accuracy tracking: Agent-specific success rate (model X + agent Y combination)
  • IDE: detected via code compilation success or test pass/fail
  • ChatGPT: explicit feedback via client signal (thumbs up/down in UI)
  • Ollama adapter: tracked via request completion time
• Cost per agent: Monthly token consumption × model cost + compute time
  • Agent cost reports generated on UTC 00:00 daily
  • Used for cost attribution and budgeting decisions

3. Adaptive Per-Agent Routing

• Agent-specific confidence gate (ADR-0003, threshold T) overrides global gate
  • Claude Code: T=0.65 (low latency trumps perfect accuracy)
  • ChatGPT: T=0.75 (accuracy critical, users expect quality)
  • Codex: T=0.70 (balanced)
• Per-agent fallback chain priority
  • Claude Code: Ollama → external (Mistral, Groq) if latency acceptable
  • ChatGPT: External → Ollama only if gateway unavailable
  • Codex LSP: Gateway only (no fallback)
• Agent-specific model tier selection
  • Request scoring (ADR-0002 enhanced): add agent context to dimension set
  • Dimensions now include: agent_id, context_tokens, user_language, etc.
  • Score computation per-agent lookup table (learned over time)

4. Integration with Learning Engine

• Feedback loop: agent adapter → gateway metrics → learning engine
  • Agent ID propagated in every request (header X-Agent-ID + request body)
  • Response includes agent-specific confidence and model choice rationale
• Learning job phases (30min/1h/6h/12h, ADR-0003):
  • Phase 1: Aggregate global metrics (existing)
  • Phase 2: Compute per-agent slices (new)
  • Phase 3: Update per-agent confidence scores (new)
  • Phase 4: Regenerate per-agent routing rules (new)
  • Phase 5: A/B test on 10% of traffic, measure per-agent impact
• Conflict resolution: if global and agent scores diverge
  • Agent confidence takes precedence (local signal > global)
  • Log divergence for human review (may indicate model degradation or agent change)

5. Agent Feedback Integration

• API endpoint: POST /agents/{agent-id}/feedback
  • Payload: { request_id, outcome, metadata }
  • Outcomes: success, fallback, timeout, error, user_rejected
  • Metadata: completion_quality (0-10), latency_ms, token_count
• Asynchronous feedback processing
  • Feedback ingested into agent request log (backfill for requests without explicit feedback)
  • Used to update per-agent confidence on next learning cycle
• User feedback from ChatGPT UI
  • Thumbs up/down on completion → agent feedback signal
  • Aggregated into user_satisfaction metric per model/agent pair

Alternatives Considered

Alternative 1: Global Learning Only

• Pros: Simpler implementation, unified signal, fewer moving parts
• Cons: Cannot optimize for heterogeneous agents, per-agent feedback lost, cost attribution unclear
• Why not: Agents have fundamentally different success criteria (IDE latency ≠ ChatGPT quality)

Alternative 2: Separate Learning Engines Per Agent

• Pros: Complete isolation, agent-specific optimization, no cross-agent interference
• Cons: Massive duplication, learning curves 5x longer (fewer samples per agent), no knowledge sharing
• Why not: Claude Code and ChatGPT both benefit from qwen models — throwing away cross-agent signal is wasteful

Alternative 3: Callback-Based Feedback (No Agent Context)

• Pros: Minimal changes to learning engine, compatible with existing code
• Cons: Cannot attribute feedback to specific agent, routing decisions remain global
• Why not: Feedback without agent context is noise — we would not know which agent benefited from routing change

Alternative 4: Agent Context in Request ID (Ephemeral)

• Pros: No new fields, agent context derived from request ID structure
• Cons: Fragile (if request ID format changes, tracing breaks), no standardization
• Why not: Tight coupling to request ID generation; agent metadata should be explicit

Consequences

Positive

• Per-agent cost attribution: Identify which agents are expensive (e.g., ChatGPT streaming uses 3x tokens)
• Latency SLOs per agent: Claude Code gets optimized for <100ms, ChatGPT for <500ms/chunk
• Agent-specific routing: Can prefer qwen2.5:3b for IDE, :32b for ChatGPT without global harm
• Learning efficiency: Signal isolation prevents "optimal for ChatGPT" from breaking IDE responsiveness
• Fallback diversity: Claude Code can use Ollama, ChatGPT uses external only — no one-size-fits-all risk
• Early detection of agent issues: If Claude Code confidence drops 20% in 1h, alert (possible adapter bug)

Negative

• Increased storage: Per-agent metrics = ~10x request logs compared to aggregated global (50GB → 500GB annually)
• Learning complexity: Logic for per-agent confidence updates, conflict resolution, feedback ingestion
• Operational overhead: Monthly cost reports per agent, per-agent SLO dashboards, alerting rules
• Agent coupling: Changes to agent (e.g., ChatGPT client SDK upgrade) may shift confidence — requires relearning
• Feedback dependency: Learning quality degrades if agents don't send feedback (must have fallback)

Risks

• Stale per-agent data: If ChatGPT adapter goes offline for 6h, historical confidence becomes misleading → Mitigation: decay confidence over time (10% per day)
• Contradictory scores: Global says "model X is bad", agent says "model X works great for me" → Mitigation: log divergence, human review before policy change
• Cost explosion: Per-agent metrics + request logs could 10x storage costs → Mitigation: retention policy (30 days hot, 90 days warm, 1yr cold archive)
• Privacy: Agent IDs in logs could enable tracking "which agent requested what" → Mitigation: agent_id anonymized (hash), explicit opt-out for sensitive agents

Implementation Plan

Phase 2G.4.1: Per-Agent Request Logging (Week 1)

• Add agent_id field to gateway_request_log table
• Modify client SDK / adapters to inject X-Agent-ID header
• Backfill historical requests with agent ID from source IP heuristics (fallback)
• Test with Claude Code + Codex adapters

Phase 2G.4.2: Per-Agent Confidence Scoring (Week 2)

• Create agent_confidence_scores table: (agent_id, model, score, updated_at)
• Update learning engine Phase 3 to compute per-agent slices from request log
• Implement per-agent confidence gate in router (override global gate if agent score available)
• A/B test: 10% of traffic uses per-agent routing, 90% uses global (measure impact)

Phase 2G.4.3: Per-Agent Feedback Loop (Week 2)

• Implement POST /agents/{agent-id}/feedback endpoint
• Adapter SDKs: send feedback after each completion (success/fallback/error)
• ChatGPT UI: wire feedback buttons to feedback endpoint
• Asynchronously ingest feedback into learning engine

Phase 2G.4.4: Cost Attribution & Reporting (Week 3)

• Dashboard: per-agent token consumption, monthly cost, cost per request
• Daily cost report: daily_agent_costs.csv (agent_id, tokens_in, tokens_out, cost_usd)
• Alert: if agent cost > historical avg + 2σ (detect runaway requests)

Phase 2G.4.5: Per-Agent SLO Monitoring (Week 3)

• Latency SLOs: Claude Code ≤100ms p99, ChatGPT ≤500ms p95 (streaming chunk)
• Alert: SLO breach (e.g., IDE completions suddenly >200ms) → investigate model issue
• Dashboard: per-agent latency heatmap (hourly p50/p95/p99)

Phase 2G.4.6: Documentation & Runbook (Week 4)

• ADR-0006 (this document)
• Runbook: "Agent Confidence Divergence" (what to do if global ≠ agent scores)
• Runbook: "Cost Spike Investigation" (how to debug high-cost agent)

Open Questions

1. Feedback Mechanism: Should adapters automatically send feedback, or require explicit client instrumentation?

   • Current decision: Automatic (adapters track success/fallback)
   • Open: How to detect IDE compilation success without IDE instrumentation?

2. Confidence Decay: How aggressively should per-agent confidence decay over time?

   • Current decision: 10% per day (reaches 50% confidence after ~7 days of inactivity)
   • Open: Should decay be different per agent (IDE less decay than ChatGPT)?

3. Fallback Privacy: Should fallback usage be logged per agent (privacy concern)?

   • Current decision: Yes, with anonymized agent_id
   • Open: Do sensitive agents need to opt out of logging?

4. Conflict Resolution: If global says "model X bad" but agent says "X works great", which wins?

   • Current decision: Agent wins (local > global)
   • Open: Should conflicts trigger human review before policy change?

5. Cross-Agent Learning: Can agent A learn from agent B's feedback?

   • Current decision: Yes (global learning phase pools all agent signals)
   • Open: Should some agents be "first-class" (their feedback weighs more)?

Related ADRs

• ADR-0001 — Multi-agent architecture
• ADR-0002 — Tier assignment (now per-agent)
• ADR-0003 — Confidence gate (now per-agent override)
• ADR-0005 — Agent integration protocol (feedback extension)