Drafts CHANGELOG_PENDING.md entries from Conventional Commit subjects on
every push to main -- feat/fix/refactor/perf/security bucketed into
Added/Fixed/Changed, chore/ci/test/docs/style/merge skipped, anything
mentioning internal infra hostnames dropped. No LLM call, no network
access. Never touches CHANGELOG.md -- still needs a human to fold an
entry in. Rolled out fleet-wide from the PeerCortex pilot (2026-07-18).
Backport of the hardened injection-defense from magatama core
(packages/core/src/security/injection-scan.ts) so both layers stay in
lockstep. Pattern ids are identical across layers for cross-layer telemetry.
- fix latent dead ignore-ru pattern: JS \b is ASCII-only and never matches
around Cyrillic, so the Russian "ignore previous instructions" pattern
never fired. Removed the leading/trailing \b.
- add normalizeForScan de-obfuscation pre-pass (NFKC; strip zero-width/bidi/
unicode-tag/combining; map Cyrillic+Greek homoglyphs to ASCII; collapse
spaced-out letters; de-leet). scanForInjection now scans BOTH raw and
normalized input and unions matches by id; new viaNormalization flag.
- add excessive-letter-spacing (high) + broaden repeat-words-above (stacked
determiners), ignore-zh (mid-phrase) and ignore-ko (before-verb determiners).
Block-mode FP guard: the gateway BLOCKS (INJECTION_DEFENSE_MODE=block), so
excessive-letter-spacing diverges from the Magatama regex by a letter-guard
lookahead (>=3 consecutive spaced letters). The raw Magatama form would 422
legit numeric traffic ("what comes next: 1 2 3 4 5 6 7 8 9 10"). Same id +
severity so telemetry stays in lockstep.
Verified: 76/76 red-team + benign dev/ops FP battery against built dist; 10 new
vitest cases; live /v1/guard/scan before/after (RU clear->blocked,
homoglyph/spacing/ko blocked, numeric sequence stays clear); gateway rebuilt +
restarted, health 200, 0 new false positives.
Internal red-team showed the pattern layer caught only ~33% of injection-
shaped prompts. Adds 9 high-severity, FP-tuned patterns for the bypassed
classes (system-prompt extraction via repeat/read, self-referential secret
exfil, authority-spoof, base64 decode-exec, unfiltered persona, jailbreak-
confirm). Result: 10/10 attacks blocked on the red-team set, 0 false-
positives from the new patterns (verified against benign dev/ops prompts).
codex-bridge passthrough previously hardcoded tokens_in=0 and used
text.length (char count) for tokens_out, writing only to llm_calls.
request_tracking never received these entries so all Codex/Claude/
Copilot bridge traffic was invisible in dashboard token totals.
Fix:
- Extract real token counts from upstreamJson.usage (prompt_tokens /
completion_tokens) with char/4 estimation as fallback
- Call requestLogger.logRequest() to write to request_tracking
- Pass real totals to recordSubscriptionUsage() (was 0)
- Both audit log (llm_calls) and dashboard log (request_tracking) now
receive consistent bridgeCaller + real token counts
Added safety-test-ruse and filters-disabled-claim patterns to block
social-engineering probes that claim safety filters are disabled.
Root cause: Python str.replace() wrote literal backspace bytes (0x08)
instead of regex word-boundary markers (\b), making both patterns
silently fail against the probe text.
Fixes persona-bypass pentest probe (Magatama MIND/COMPLY findings).
- pii-redaction.ts: add redactPersonNamesAsync() — async Presidio/GLiNER NER
for PERSON/ORGANIZATION/LOCATION (95% German accuracy, <3s timeout, non-fatal)
- completion.ts: wire Presidio async call after regex PII block (PRESIDIO_URL env)
- auto-discovery.ts: add schedulePeriodicDiscovery() — re-scan all bridges every 5 min
- server.ts: call schedulePeriodicDiscovery(300000) on boot
- dashboard.ts: POST /api/discovery/client-report + GET /api/discovery/client-reports
— MacBook/MacStudio push their AI inventory (Ollama models, tools, ports) every 5 min
- .env: REDACT_PII_MODE=cloud_only, AUTO_SPAWN_BRIDGES=1, PRESIDIO_URL=:8701, DISCOVERY_INTERVAL_MS=300000
Presidio sidecar: /opt/presidio-sidecar/ — FastAPI + spaCy de_core_news_lg + GLiNER
multi-v2.1, PM2 id 28 on :8701. Compression already active; anonymization now live
on all cloud provider calls.
Standalone injection-defense endpoint for Flightdeck and other internal
clients: pattern scan + optional LLM judge, independent of the inline
completion-pipeline defense.
Points at magatama/HANDOVER-AGENTS-2026-05-17.md as the canonical
cross-project agent handover. Keeps the repo-local TL;DR (Layer-3 status,
PM2 env-reload caveat, smoke test, rollback, /proc env verification) so an
agent can act in this repo without leaving it.
Today's only LLM-gateway change was an env edit on Erik:
INJECTION_DEFENSE_MODE: block → llm_judge
LLM_JUDGE_MODEL: magatama-coder:judge-r1 → qwen2.5:14b
Backup .bak-<ts>-pre-mode-switch + .bak-<ts>-pre-qwen-judge on Erik.
Master handover lives in the magatama repo. This file is a pointer + smoke
test + rollback recipe for the Layer-3 activation specifically. Includes
the /proc/PID/environ verification step (because pm2 env shows cached
ecosystem.config.js values, not the actual node-process env from the
durable start-with-env.sh wrapper).
Recurring problem: PM2 ecosystem env vars get dropped on KeepAlive
auto-restart. Has bitten us 3× in one session — defense silently turns
OFF without visible cause.
Fix: PM2 script changed from `./packages/gateway/dist/server.js` to
`./start-with-env.sh` which:
set -a; source .env.defense; source .env; set +a
exec node packages/gateway/dist/server.js
Defense env now persists across ANY restart mechanism (manual reload,
KeepAlive crash-restart, pm2 resurrect, system reboot, ...) because
it's loaded at the shell level on every process spawn — independent of
PM2's internal env state.
Verified end-to-end:
- 4 smoke tests (Layer-1 EN/FR, Layer-2 Roleplay, legit) → all pass
- kill -9 → KeepAlive respawns → env STILL present → injection STILL
blocks (HTTP 422)
.env.defense lives at /opt/llm-gateway/.env.defense (chmod 600, not in repo).
.env.defense.example added to repo as template.
Was hardcoded to qwen2.5:3b. Now reads from process.env LLM_JUDGE_MODEL
with qwen2.5:3b fallback.
Production env updated to magatama-coder:judge-r1 — a snapshot of the
magatamallm post-chunk-4 LoRA adapter exported via train.py --export-only.
Chunk-4 picked because it had the best val_loss (0.861) of the 5 balanced
chunks; chunk-5 spiked back to val=2.531.
Sanity test on the new judge model:
injection prompt -> "INFORMATIONAL" (not the strict INJECTION word
we'd want — judge needs Phase-2
dedicated fine-tune on binary
classification format)
safe prompt -> "SAFE" (correct)
Implication: INJECTION_DEFENSE_MODE is staying at 'block' for now —
switching to 'llm_judge' mode with this provisional judge would actually
weaken defense because magatamallm's training tilts toward operator-task
output ("here's the fix") rather than binary INJECTION/SAFE classification.
Follow-up (Phase 2): train a dedicated `magatama-judge` model — small base
(Qwen 2.5:1.5b or Phi-3-mini), trained purely on injection-classification
SFT pairs extracted from our existing:
- llm-security-prompt-injection-2026-05-12.train.jsonl
- pulso-magatama-injection-guard-2026-05-13.train.jsonl
- guard-exposure-firewall-verified-2026-05-16.train.jsonl
- jailbreak-corpus-candidates.jsonl (L1B3RT4S gaps)
- benign samples from train.jsonl labeled SAFE
Architecture rationale: separation of concerns. Even if attacker manipulates
the primary backbone model, judge stays independent. ~5-10k pairs should
be enough for a focused 1.5B classifier. Training ~2-3h on Mac Studio MPS.
Adds a second defense layer between Layer-1 regex (62 patterns) and the
existing Layer-3 llm_judge. Calls a FastAPI sidecar running on the Mac
Studio (port 9091, MPS) that wraps protectai/deberta-v3-base-prompt-
injection-v2 — public model, no auth needed, ~50-400ms inference.
modules/prompt-guard-client.ts:
- callPromptGuard(input) opportunistic, never throws
- isPromptGuardConfigured() true if PROMPT_GUARD_URL is set
- getPromptGuardThreshold() default 0.85
- getPromptGuardMinLen() default 16 chars (skip tiny inputs)
routes/completion.ts:
- New Layer-2 block between regex scan and llm_judge: when Layer-1
didn't detect and input is long enough, ask the sidecar. If sidecar
returns INJECTION with score >= threshold, return HTTP 422 with
error.prompt_guard payload (score + latency).
- Fail-open: sidecar timeout/error logs a warning and the request
falls through to llm_judge / cache / model — never blocks legitimate
traffic due to sidecar issues.
Env (set in ecosystem.config.js):
PROMPT_GUARD_URL http://192.168.178.213:9091
PROMPT_GUARD_THRESHOLD 0.70 (lowered from 0.85 after empirical testing)
PROMPT_GUARD_TIMEOUT 1500 ms
Sidecar code lives at:
~/magatama-llm/prompt-guard-sidecar/server.py (Mac Studio)
launched via ~/Library/LaunchAgents/org.fichtmueller.prompt-guard-sidecar.plist
Smoke tests after deploy:
Layer-1 caught: German "ignoriere..." -> HTTP 422
Layer-2 caught: English "pretend no restrict.."-> HTTP 422 (pg_score 0.9999)
Layer-2 caught: Bangla-romanized -> HTTP 422 (Layer-1 actually)
Benign: "Explain DNS in 2 sentences" -> HTTP 200
Closes the multilingual bypass gap. Previously covered EN/DE/FR/ES/IT/RU/ZH/JA.
Now also: Bangla, Hindi, Arabic, Hebrew, Persian, Turkish, Vietnamese, Thai,
Korean, Polish, Dutch, Indonesian, Tagalog, Swahili.
Plus a universal non-Latin-script soft-flag pattern (severity=medium) that
catches ≥20 chars of Arabic/Bengali/Devanagari/Hebrew/Thai/Hangul/Han/
Hiragana/Katakana/Cyrillic/Tamil/Telugu/Gujarati/Gurmukhi/Myanmar/Khmer/
Lao/Tibetan/Georgian/Armenian/Sinhala — surfaces in scan result without
auto-blocking, so legitimate non-Latin prompts pass while the operator
can route them to llm_judge for deep inspection.
Pattern-engineering notes:
- Devanagari / Bengali / Hebrew need optional matra/suffix tolerance
- Turkish needs \p{L} instead of \w because ı/ş/ç fall outside ASCII \w
- Persian (SOV) needs both VSO and SOV order alternation
- Hebrew needs מ/ב/כ/ל preposition prefix tolerance
- Tagalog needs optional ang/sa article between verb and noun
Smoke-tested 14/14 languages → all HTTP 422 blocked.
Negative-tested 3 benign non-Latin prompts (jp-weather, ar-greeting,
th-thanks) → all HTTP 200 pass. Zero false positives.
Total active patterns: 62 across 6 categories.
- Fixed /api/stream/requests endpoint HTTP/2 INTERNAL_ERROR
- Use reply.raw.writeHead() instead of Fastify headers API for SSE
- Added 30s heartbeat to keep connection alive
- Proper event format with 'event:' and 'data:' fields
- Comprehensive error handling and cleanup on disconnect
- Mirrors working pattern from /api/stream/costs endpoint
- Resolves dashboard perpetual 'Loading...' state
The LLM Gateway is configured to run on port 3103 in ecosystem.config.cjs,
but the Cloudflare tunnel configuration and setup script were referencing port
3100, causing 502 Bad Gateway errors.
Updates:
- cloudflare-tunnel.md: Changed tunnel ingress from localhost:3100 to localhost:3103
- setup-erik.sh: Updated health check URL and output messages to port 3103
- This fixes the Cloudflare tunnel connection that was causing public HTTPS access to fail
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
