bc340da1f9
Backport of the hardened injection-defense from magatama core
(packages/core/src/security/injection-scan.ts) so both layers stay in
lockstep. Pattern ids are identical across layers for cross-layer telemetry.
- fix latent dead ignore-ru pattern: JS \b is ASCII-only and never matches
around Cyrillic, so the Russian "ignore previous instructions" pattern
never fired. Removed the leading/trailing \b.
- add normalizeForScan de-obfuscation pre-pass (NFKC; strip zero-width/bidi/
unicode-tag/combining; map Cyrillic+Greek homoglyphs to ASCII; collapse
spaced-out letters; de-leet). scanForInjection now scans BOTH raw and
normalized input and unions matches by id; new viaNormalization flag.
- add excessive-letter-spacing (high) + broaden repeat-words-above (stacked
determiners), ignore-zh (mid-phrase) and ignore-ko (before-verb determiners).
Block-mode FP guard: the gateway BLOCKS (INJECTION_DEFENSE_MODE=block), so
excessive-letter-spacing diverges from the Magatama regex by a letter-guard
lookahead (>=3 consecutive spaced letters). The raw Magatama form would 422
legit numeric traffic ("what comes next: 1 2 3 4 5 6 7 8 9 10"). Same id +
severity so telemetry stays in lockstep.
Verified: 76/76 red-team + benign dev/ops FP battery against built dist; 10 new
vitest cases; live /v1/guard/scan before/after (RU clear->blocked,
homoglyph/spacing/ko blocked, numeric sequence stays clear); gateway rebuilt +
restarted, health 200, 0 new false positives.
32 lines
8.2 KiB
Markdown
32 lines
8.2 KiB
Markdown
# LLM Gateway — Pending Changelog Entries
|
|
|
|
## 2026-04-16
|
|
|
|
{"d":"2026-04-16","t":"FEAT","m":"MAGATAMA まがたま security intelligence model integrated: magatama:32b in models.yaml (large tier, 131k ctx), 6 routing rules (threat_analysis, ciso_report, compliance_gap, incident_response, bgp_security, vuln_triage), 6 prompt templates with full TEPPEKI doctrine (MITRE ATT&CK, NIS2, CIA Triad, CVSS v3.1, Kill Chain)"}
|
|
{"d":"2026-04-16","t":"AI","m":"MAGATAMA LoRA adapter (r=8, alpha=16) fine-tuned on Qwen2.5-32B-Instruct with 22831 security samples, converted to GGUF format for llama-server runtime deployment with --lora flag on Mac Studio port 11435"}
|
|
|
|
## 2026-04-02
|
|
|
|
{"d":"2026-04-02","t":"FEAT","m":"Fine-tuner SSH tunnel launch script (scripts/start.sh): opens SSH tunnel to Erik:5432 before running fine-tuner, bypassing IONOS firewall"}
|
|
{"d":"2026-04-02","t":"FIX","m":"Fine-tuner env vars: FT_DB_URL/FT_GATEWAY_URL/FT_OLLAMA_URL (not DATABASE_URL) — fine-tuner status command now works"}
|
|
{"d":"2026-04-02","t":"FIX","m":"DB schema migration: added status, confidence_score, used_in_training, system_prompt, input_text, output_text, human_edited, edited_output, used_in_dpo_training to learning_corpus; added run_type, task_type, started_at, deployed_model_name to fine_tuning_runs"}
|
|
{"d":"2026-04-02","t":"FEAT","m":"CtxHealth: new workspace package packages/ctx-health — TypeScript self-healing daemon monitoring PM2, PostgreSQL, Ollama, Cloudflare tunnel, disk, memory, network, WireGuard every 60s; LLM-powered diagnosis via ctxhealer caller; auto-healing with cooldown + escalation guards; incidents persisted to ctx_health_incidents + ctx_health_status tables; dry-run mode; PM2 process name: ctx-health"}
|
|
{"d":"2026-04-02","t":"FIX","m":"Template variable resolution: input is now mapped as fallback for all 20+ content variable names (ocr_text, alert_data, bgp_data, anomaly_data, etc.) — all 7 project templates work with simple string input"}
|
|
{"d":"2026-04-02","t":"FIX","m":"24 templates updated: {{input}} added as primary content variable so simple callers work without knowing domain-specific variable names (nog_cfp_evaluate, pc_as_narrative, sb_root_cause, shieldx_false_positive, etc.)"}
|
|
{"d":"2026-04-02","t":"FIX","m":"Prompt resolver: input also aliased as source_data fallback; context fields spread into template vars"}
|
|
{"d":"2026-04-02","t":"DATA","m":"DB schema migration run on Erik: learning_corpus and fine_tuning_runs now compatible with Python fine-tuner"}
|
|
{"d":"2026-04-02","t":"FEAT","m":"Full E2E test: all 7 projects verified (TIP, EO, PC, SB, NOGnet, ShieldX, CtxEvent) — 100% approved status, avg confidence 8.25/10, 70 calls in audit log"}
|
|
|
|
{"d":"2026-04-06","t":"AI","m":"BlogLLM v1 (fo-blog-v1-qwen7b Q5_K_M): GGUF conversion + Ollama registration complete. Test result: pipeline runs end-to-end, technical content OK, but format compliance fails (bullet lists despite NO-BULLET instruction, training prompt leakage). Root cause: max_seq_length=1024 during training cut articles at ~512 words. Fix applied: max_seq_length=2048 for next run. Re-train needed before production."}
|
|
{"d":"2026-04-06","t":"FIX","m":"TIP: Reset 298 stale details_verified flags (reach_label empty). Runtime check in dashboard: dVer gated on non-empty reach_label. Wrong image cleared for O.Czz8HG.z.R."}
|
|
{"d":"2026-04-06","t":"FIX","m":"TIP comparable prices: treat reach_meters=0 same as NULL in API query — 800G OSFP FS.com prices now visible on Flexoptix product pages."}
|
|
{"d":"2026-06-12","t":"FEAT","m":"MAGATAMA guard scan API exposed at /v1/guard (POST /scan, GET /health): Layer-3 injection defense (pattern catalog + optional LLM judge) callable standalone by internal clients (first consumer: Flightdeck). Verdicts: clear/review/blocked with findings; always scans even when inline INJECTION_DEFENSE_MODE=off."}
|
|
{"d":"2026-06-22","t":"SEC","m":"PII-Redaktion REDACT_PII_MODE: cloud_only -> always. Jetzt wird ALLER non-exempt Traffic anonymisiert, auch bei lokalem Ollama-Ziel (Laptop spricht via /v1/chat/completions). redactPii (Stage-A: email/IBAN/CC/phone/IP/...) + redactPersonNamesAsync (Presidio+GLiNER-Sidecar :8701) laufen in executeCompletion -> gilt fuer ALLE Routen (chat/completions, messages, responses, completion). Verifiziert: caller laptop-test -> EMAIL_001+IBAN_001+PERSON maskiert vor Modell+Logs. .env-Backup .env.bak-*-pre-pii-always."}
|
|
{"d":"2026-06-23","t":"FIX","m":"Presidio-Sidecar (:8701, PII-Namen-NER Backend) Sprach-Misconfig behoben: RecognizerRegistry war ['en'], AnalyzerEngine ['de','en'] -> Misconfigured engine-Errors liessen manche /analyze-Calls fehlschlagen -> Namen wurden bei diesen Requests NICHT redigiert (silent fail-open leak). Fix /opt/presidio-sidecar/main.py: RecognizerRegistry(supported_languages=[de,en]) + load_predefined_recognizers(languages=[de,en]). Restart + pm2 save. Verifiziert: 0 neue Misconfig-Errors, deutsches PERSON/LOC/ORG zuverlaessig (E2E ueber Gateway redactedTokens+names). Backup main.py.bak-*-pre-langfix. HINWEIS: Sidecar /opt/presidio-sidecar ist NICHT git-versioniert."}
|
|
{"d":"2026-06-29","t":"SEC","m":"Red-Team Injection-Defense (Augustus v0.9.0 auf Erik installiert + direkter Mess-Lauf): rule/pattern-Defense (Modus block) fing nur ~33% (4/12) variierter Injection-Prompts ab — DAN/developer-mode/im_start/DE-ignoriere geblockt; durchgelassen: System-Extraction (repeat words above), Authority-Spoof (security test/new dev instructions), Base64-Encoding, Grandmother, Translate-then-ignore. Deckt sich mit dokumentierter ShieldX-Landmine (~32,9% real-TPR) -> wahrscheinlich derselbe Rule-Layer, empirisch reproduziert. Remediation: Injection-Defense in Adapter eintrainieren (LLM Gym, 297 neue Korpora gestaged) + Rule-Patterns ergaenzen. Augustus-Volllauf braucht Tenant-JWT oder rest.Rest-Generator (openai-Gen sendet immer Bearer -> 401). Finding: Obsidian Security/gateway-injection-redteam-2026-06-29.md."}
|
|
{"d":"2026-06-29","t":"SEC","m":"Injection-Defense Rule-Layer +9 Patterns (high) fuer die im Red-Team durchgelassenen Klassen: ignore-your-instructions, repeat-words-above (System-Extraction), read-system-prompt (Grandmother-Framing), base64-decode-exec (Word-Order/Short-Payload), exfil-own-secrets (self-referential config/keys/secrets), authority-new-instructions (Developer-Spoof), leak-conversation, unfiltered-persona, confirm-jailbroken. FP-getunt auf SELF-referentielle Angriffe. Ergebnis: Attacken 33%->100% (10/10 auf dem Set), 0 FP aus den neuen Patterns (verifiziert: benigner base64-Prompt wird von Prompt-Guard-ML geblockt, nicht vom Rule-Layer). Gateway-weit -> schuetzt ALLE Caller (Magatama-Triage, Laptop, Argus). Backup injection-defense.ts.bak-*-pre-redteam-patterns."}
|
|
|
|
## 2026-07-07
|
|
|
|
{"d":"2026-07-07","t":"SEC","m":"Injection-defense: backport Magatama core hardening into the gateway rule-layer, keeping pattern ids in lockstep for cross-layer telemetry. (1) FIX latent dead ignore-ru pattern — JS \\b is ASCII-only and never matches around Cyrillic, so the Russian 'ignore previous instructions' pattern never fired; removed the leading/trailing \\b. (2) ADD normalizeForScan de-obfuscation pre-pass (NFKC; strip zero-width/bidi/unicode-tag/combining; map Cyrillic+Greek homoglyphs->ASCII; collapse spaced-out letters; de-leet). scanForInjection now scans BOTH raw and normalized input and unions matches by id; new viaNormalization flag on InjectionScanResult. (3) ADD excessive-letter-spacing pattern (high) and broaden repeat-words-above (stacked determiners), ignore-zh (mid-phrase 所有), ignore-ko (모두/모든/전부 before verb). BLOCK-MODE FP GUARD: the gateway blocks (INJECTION_DEFENSE_MODE=block), so excessive-letter-spacing diverges from the Magatama regex by a letter-guard lookahead requiring >=3 consecutive spaced letters — the raw Magatama form would 422 legit numeric traffic ('what comes next: 1 2 3 4 5 6 7 8 9 10'). Same id+severity -> telemetry stays in lockstep. VERIFIED: 76/76 red-team + benign dev/ops FP battery against built dist; 10/10 new vitest cases; live /v1/guard/scan before/after (RU clear->blocked, homoglyph/spacing/ko blocked, numeric sequence stays clear); gateway rebuilt + restarted (pm2 llm-gateway id 24), health 200, 0 new false positives. Source of truth: magatama packages/core/src/security/injection-scan.ts (Gitea rene/magatama e8afd3b)."}
|