26 Commits

Author SHA1 Message Date
Rene Fichtmueller
c3ab87b167 feat: add fo-blog-v8 training pipeline (Qwen2.5-14B, SFT+DPO)
Full v8 training pipeline for the optical networking blog model:
- train_blog_v8.py: SFT (LoRA r=64, 5 epochs) + DPO (2 epochs) on Qwen2.5-14B-Instruct
  Fixed for trl 1.2.x: SFTConfig instead of TrainingArguments, processing_class= instead
  of tokenizer=, eval_strategy= instead of deprecated evaluation_strategy=
- consolidate_v8_dataset.py: weighted merge of all data sources (820 effective SFT / 235 DPO)
- crawl_v8_sources.py: APNIC/RIPE Labs/potaroo/Cloudflare crawler with balanced div extraction
- process_v6_blogs.py: converts 101 real v6 TIP blog outputs into SFT + DPO pairs
- label_v7_quality.py: Claude-judged quality labels → v8 quality DPO pairs
- parse_real_posts.py: parses blog.fichtmueller.org Ghost CMS HTML → gold SFT records
- run_v8_pipeline.sh: autopilot (consolidate → SFT → DPO → GGUF → Ollama)
- blog-v8-training.yaml: training config reference

Dataset breakdown: 19 real posts ×3 + 196 v7-gen + 28 v6blogs ×2 + 135 external ×1.5
2026-04-19 11:44:09 +02:00
Rene Fichtmueller
79d434434f chore: MAGATAMA deployment state — download in progress, Pi-hole bypassed 2026-04-16 16:35:54 +02:00
Rene Fichtmueller
2fb0992c71 feat: add MAGATAMA まがたま security intelligence model to LLM Gateway
- Add magatama:32b to models.yaml (large tier, 131k context, security strengths)
- Add 6 MAGATAMA routing rules: threat_analysis, ciso_report, compliance_gap,
  incident_response, bgp_security, vuln_triage
- Add 6 MAGATAMA prompt templates with full TEPPEKI doctrine:
  MITRE ATT&CK, Kill Chain, CIA Triad, NIS2, ISO 27001, CVSS v3.1
- Fine-tuned on Qwen2.5-32B-Instruct with 22831 MAGATAMA security samples
  LoRA adapter: r=8, alpha=16
2026-04-16 14:31:17 +02:00
Rene Fichtmueller
132cf835b0 fix: add fixes 049-050 (OPNsense WireGuard FritzBox NAT + VLAN WAN) 2026-04-13 08:20:06 +02:00
Rene Fichtmueller
c50af63389 feat(ctx-health): add proxmox-pvestatd + opnsense-disk health checks
- Add SSH-based health check for pvestatd D-state detection on Proxmox host
  (heal via cgroup move + lock file removal + reset-failed)
- Add SSH-based disk check for OPNsense VM (threshold 75%, auto-cleanup)
- knowledge/fixes.json: add 48 training fixes including post-reboot DNS
  recovery (fix-046), cloudflared DNS-wait boot fix (fix-047), and
  vzdump load-crash scenario with recovery steps (fix-048)
2026-04-13 05:42:24 +02:00
Rene Fichtmueller
b4593b6582 feat: integrate real @shieldx/core library into gateway pipeline
Replace recursive HTTP-based ShieldX scan with direct library integration.
- 547+ rules, 50+ languages, sub-millisecond scans
- Enables: rules, entropy, indirect injection, behavioral, unicode,
  tokenizer, compressed payload detection
- Disables Ollama-dependent scanners for zero external dependency
- Response now includes threat_level, kill_chain_phase, shieldx_latency_ms
2026-04-07 09:03:02 +02:00
Rene Fichtmueller
8123343361 feat: add Flexoptix blog pipeline templates (tip_blog_angle + tip_blog_draft)
Two-stage LLM pipeline for Flexoptix-style blog generation:
- tip_blog_angle: identifies the real production situation as article angle (JSON)
- tip_blog_draft: writes continuous prose article — no headers, no bullets,
  no AI filler. Gold article included as few-shot reference. v2.0.0 with
  absolute format rules, DR4 wavelength correction (1310nm), fiber scope
  vs OPM distinction, no invented firmware versions.
2026-04-03 01:03:09 +02:00
49c673b683 feat: add ctx_morning_briefing routing rule and prompt template
Adds LLM Gateway support for CtxReport daily intelligence reports.
Uses qwen2.5:32b to generate German-language morning briefings from
infrastructure metrics collected overnight.
2026-04-03 00:44:04 +02:00
Rene Fichtmueller
e0b9fa1f53 feat: add CtxHealth self-healing daemon as new workspace package
New package @llm-gateway/ctx-health (packages/ctx-health/) — a TypeScript
infrastructure monitoring and auto-healing daemon. Monitors 8 subsystems
every 60s (PM2, PostgreSQL, Ollama, Cloudflare tunnel, disk, memory,
network, WireGuard), gets AI-powered root cause analysis via the gateway
(ctxhealer caller / ctx_health_diagnose task_type), executes healing
actions with cooldown (5min) and escalation guards (3+ failures → human
escalation), persists all incidents to ctx_health_incidents and
ctx_health_status tables. Dry-run mode via CTX_HEALTH_DRY_RUN=true.
Runs as ctx-health PM2 process on Erik server.
2026-04-03 00:16:08 +02:00
Rene Fichtmueller
a8a77e689c feat: add CtxHealth + CtxSecurity to gateway — ctxhealer:latest model, 5 routing rules, 2 templates 2026-04-03 00:14:23 +02:00
Rene Fichtmueller
9b4d1caa8a fix: routing-optimizer uses status='approved' not non-existent validation_passed column 2026-04-03 00:01:19 +02:00
Rene Fichtmueller
52697bc6fc fix: replace hardcoded Mac paths with relative paths in learning engine (routing-optimizer, prompt-optimizer, few-shot-curator) 2026-04-02 23:58:53 +02:00
Rene Fichtmueller
c3248da6c0 chore: add pending changelog entries for 2026-04-02 fixes 2026-04-02 23:52:17 +02:00
Rene Fichtmueller
719336bded fix: map input as fallback for all 20+ template content variables (ocr_text, alert_data, bgp_data, etc.) 2026-04-02 23:41:36 +02:00
Rene Fichtmueller
f1c1d107ca fix: map input to source_data fallback and spread context vars into template variables 2026-04-02 23:38:22 +02:00
Rene Fichtmueller
3bb9923255 fix: fine-tuner uses FT_DB_URL/FT_GATEWAY_URL/FT_OLLAMA_URL env vars, not DATABASE_URL 2026-04-02 23:35:27 +02:00
Rene Fichtmueller
d8deecdb32 feat: SSH tunnel launch script for fine-tuner (IONOS blocks port 5432 externally) 2026-04-02 23:28:30 +02:00
Rene Fichtmueller
499e600239 fix: fine-tuner config points to Erik DB + CF tunnel URLs
- database_url: Erik PostgreSQL (217.154.82.179:5432) with correct password
- gateway_url: https://llm-gateway.context-x.org (public CF tunnel)
- ollama_url: localhost:11434 (local Mac Studio, fine-tuner runs locally)
2026-04-02 23:23:17 +02:00
Rene Fichtmueller
0803fdb722 feat: add confidence_scorer prompt template (internal self-evaluation) 2026-04-02 23:20:31 +02:00
Rene Fichtmueller
b68d5c3fbf fix: client CompletionResponse matches actual gateway response fields
- Match field names: id, status, confidence, model, task_type, latency_ms, tokens, output
- Default URL now https://llm-gateway.context-x.org (public endpoint)
- ShieldX client uses 'shieldx' caller (not 'internal')
- tokens.in/tokens.out instead of token_count.input/output
2026-04-02 23:17:14 +02:00
Rene Fichtmueller
ac33476666 feat: add 55 prompt templates + ShieldX/LinkedIn routing rules + ban lists in Gitea
Templates (55 total, exceeds 49 target):
- TIP: transceiver_enrich, datasheet_extract, compatibility_parse, blog_generator,
  faq_answer, hype_cycle_narrative, price_anomaly, vendor_classify, product_description
- EO Global Pulse: business_card_ocr, voice_to_crm, event_prep_brief, attendee_enrich,
  meeting_suggest, lead_qualify, debrief_generate, ticket_summarize
- SwitchBlade: root_cause, alert_narrative, cve_remediation, csrd_narrative,
  transceiver_advisor, bandwidth_report, ticket_draft, firmware_assess, topology_explain
- PeerCortex: as_narrative, health_summary, rpki_explain, anomaly_hypothesis,
  peer_recommendation, incident_brief
- NOGnet: cfp_evaluate, cfp_feedback, topic_gap_analysis, meeting_match, speaker_enrich,
  sponsor_pitch, event_debrief, agenda_summary, session_intro
- ShieldX: threat_classify, pattern_describe, healing_recommend, compliance_report, false_positive
- Content: linkedin_post_de, linkedin_post_en, newsletter_dispatch_de, email_draft_de
- Internal: ban_detect, prompt_improve
- Routing rules: +55 entries for all template-based task types
- Ban lists: en.csv, de.csv, auto.csv created in Gitea (llm-banlists repo)
2026-04-02 23:14:30 +02:00
Rene Fichtmueller
c82b187548 feat: fix template resolution + add 40 routing rules for all project task types
- completion.ts now uses taskType directly for resolvePrompt (not decision.prompt_template)
  so tip_transceiver_enrich.yaml is used instead of generic_qa fallback template
- routing-rules.yaml: +40 task type entries for TIP (8), EO Pulse (8), SwitchBlade (9),
  PeerCortex (6), NOGnet (9), internal (2) — all with correct model tier assignments
- qwen2.5:3b for fast tasks (classify, short outputs)
- qwen2.5:14b for medium (most analysis tasks)
- qwen2.5:32b for large (blog posts, detailed reports, CSRD)
2026-04-02 23:11:21 +02:00
Rene Fichtmueller
2c5f7f6ebe fix: OLLAMA_URL env var takes precedence over hardcoded models.yaml URL
Gateway was reading ollama_base_url from YAML (192.168.178.169) instead of
OLLAMA_URL env var (https://ollama.fichtmueller.org). Fix getOllamaBaseUrl()
to prefer process.env['OLLAMA_URL'] and update YAML default to CF tunnel.
2026-04-02 23:05:13 +02:00
Rene Fichtmueller
773fd368e0 fix: parse DATABASE_URL in pool clients + extend Ollama health timeout to 15s
Gateway and learning DB clients now prefer DATABASE_URL connection string
over individual DB_* env vars — matches ecosystem.config.cjs convention.
Ollama health check timeout increased 5→15s for Cloudflare tunnel latency.
2026-04-02 23:03:31 +02:00
Rene Fichtmueller
4c5003f9fc feat: fix OLLAMA_URL to use Cloudflare tunnel + add 35 prompt templates
- Update OLLAMA_URL from 192.168.178.169 to https://ollama.fichtmueller.org
- Fix port from 3100 to 3103 (3100 was taken by Docker proxy on Erik)
- Fix DATABASE_URL password to llm_secure_2026
- Add GITEA_URL env var for ban list sync
- Add 35 prompt templates: TIP (10), EO Global Pulse (8), SwitchBlade (9),
  PeerCortex (3), internal (3), ShieldX (1), general (1)
2026-04-02 23:00:37 +02:00
Rene Fichtmueller
3a00ff4d33 feat: initial llm-gateway implementation
- Complete Fastify gateway with 8-stage pipeline
- Circuit breaker (opossum) per model tier
- Rate limiting per caller
- Ban list validation (EN/DE/auto-detected)
- TIP validator (SFF-8024, part numbers, wavelengths)
- Prometheus metrics
- pg-boss async queue
- PostgreSQL audit log + review queue
- 9 prompt templates (TIP, LinkedIn, ShieldX)
- Learning engine scaffolding
- Auto-learning: ban-list, few-shot, routing, prompt optimizer
2026-04-02 22:48:55 +02:00