llm-gateway

History

Rene Fichtmueller c3ab87b167 feat: add fo-blog-v8 training pipeline (Qwen2.5-14B, SFT+DPO)

Full v8 training pipeline for the optical networking blog model:
- train_blog_v8.py: SFT (LoRA r=64, 5 epochs) + DPO (2 epochs) on Qwen2.5-14B-Instruct
  Fixed for trl 1.2.x: SFTConfig instead of TrainingArguments, processing_class= instead
  of tokenizer=, eval_strategy= instead of deprecated evaluation_strategy=
- consolidate_v8_dataset.py: weighted merge of all data sources (820 effective SFT / 235 DPO)
- crawl_v8_sources.py: APNIC/RIPE Labs/potaroo/Cloudflare crawler with balanced div extraction
- process_v6_blogs.py: converts 101 real v6 TIP blog outputs into SFT + DPO pairs
- label_v7_quality.py: Claude-judged quality labels → v8 quality DPO pairs
- parse_real_posts.py: parses blog.fichtmueller.org Ghost CMS HTML → gold SFT records
- run_v8_pipeline.sh: autopilot (consolidate → SFT → DPO → GGUF → Ollama)
- blog-v8-training.yaml: training config reference

Dataset breakdown: 19 real posts ×3 + 196 v7-gen + 28 v6blogs ×2 + 135 external ×1.5

2026-04-19 11:44:09 +02:00

consolidate_v8_dataset.py

feat: add fo-blog-v8 training pipeline (Qwen2.5-14B, SFT+DPO)

2026-04-19 11:44:09 +02:00

crawl_v8_sources.py

feat: add fo-blog-v8 training pipeline (Qwen2.5-14B, SFT+DPO)

2026-04-19 11:44:09 +02:00

install_deps.sh

feat: initial llm-gateway implementation

2026-04-02 22:48:55 +02:00

label_v7_quality.py

feat: add fo-blog-v8 training pipeline (Qwen2.5-14B, SFT+DPO)