22 Commits

Author SHA1 Message Date
Rene Fichtmueller
4989c4affd fix(blog): fix claudeQueue deadlock from recursive 429 retry
The generateClaude() function was recursively calling itself inside
enqueueClaude(), creating a circular Promise dependency that permanently
deadlocked the claudeQueue. Any 429 rate-limit response would poison
the queue, blocking all future Claude API calls until server restart.

Fixes:
- Split retries into claudeApiCall() which is called from enqueueClaude
  (not re-entering the queue on retry = no circular dependency)
- Max 3 retries with increasing backoff (10s/30s/60s)
- Add resetClaudeQueue() exported function
- Add 15-minute auto-reset stall detection to enqueueClaude
- Expose resetClaudeQueue in POST /api/blog/llm/reset-queue endpoint
- Fix merge conflict markers in index.ts (duplicate scraperRouter import)
2026-04-06 02:51:28 +02:00
Rene Fichtmueller
1ac03bae0a fix: preserve user-provided title in blog generation + price floor validation
- blog/generate now uses caller title when provided; falls back to template
- Migration 027: hard price floor by speed class in verification function
  (no medians, no estimates — only real prices above minimum thresholds)
- Deleted 474 obviously wrong price observations (shipping costs scraped as prices)
2026-04-06 01:14:37 +02:00
Rene Fichtmueller
b4d9bfc9d1 fix(blog): hard story blacklist in STEP4 + LinkedIn — ban 2AM/dirty connector/lab-vs-prod stories
10 specific story patterns banned directly in draft prompt.
LinkedIn banned hooks: 'Everything looks fine', 'CRC creeping', 'Same optics same setup'.
Title Contract now injected into STEP4 as binding constraint.
2026-04-05 23:55:56 +02:00
Rene Fichtmueller
65159bd57d feat(blog): Post to Ghost + LinkedIn buttons in dashboard
- 'Post on blog.fichtmueller.org' → publishes via Ghost Admin API
- 'Post on LinkedIn' → modal with text + copy + open LinkedIn
- Ghost integration: TIP Blog Engine (JWT auth, mobiledoc format)
2026-04-05 23:33:58 +02:00
Rene Fichtmueller
6c818c79b5 feat(blog): Title Contract + Technical Sanity Check + Self-Heal + angle-aware LinkedIn generator
Pipeline now has 21 steps:
- STEP0: Title Contract binds LLM to headline promise
- STEP19: Technical Sanity Check (optical engineering accuracy)
- STEP20: Self-Heal (auto-fix technical errors preserving tone)
- STEP21: Title Contract Verification (final gate check)
- LinkedIn generator is now angle-aware (no more default Physical Layer hook)
2026-04-05 23:11:16 +02:00
Rene Fichtmueller
7c8f545c18 fix(blog): anti-repetition engine — 6 angle types, forbidden structures, existing article context injection 2026-04-05 22:47:15 +02:00
Rene Fichtmueller
957632c228 fix(blog): raise word target to 1200-1600, fix power-budget false positive in validateArticle 2026-04-05 20:49:22 +02:00
Rene Fichtmueller
15f3ff5bef fix: include linkedin_post in GET /api/blog response for SLL matching 2026-04-05 01:24:52 +02:00
Rene Fichtmueller
4eddbfbc7c feat(blog): Reduction Engine v1.0 + LaTeX/connector hard fails
- Replace STEP8b_REDUCTION with 5-pass Reduction Engine:
  Pass 1: Repetition Kill (one concept, one home)
  Pass 2: Tech Prune (LaTeX hard delete, SKU removal, formula prose replacement)
  Pass 3: Flow Rebuild (close gaps after cuts, no new content)
  Pass 4: Weight Correction (title/content alignment throughout)
  Pass 5: Humanization (rhythm variation, hedge removal, punch ending)
  Target: 700-1000 words (600-1300 range, warnings outside)

- System prompt + STEP9 QA: add hard fails for
  LaTeX formulas (\[...\], \frac{}, \text{} etc) — destroys blog flow
  DR4 connector error (DR4=MPO-12, not LC duplex; FR4=LC duplex)
  Title/content mismatch (title topic must be the spine, not just the intro)

- Gold Standard 5: market alert / pricing article template
  (correct title alignment, no LaTeX, DR4=MPO-12, ending on topic)

- WRONG PATTERNS extended with 4 new entries covering above failures

- blog.ts: step log messages updated to 11-14/14; word count
  output shows % reduction and range warning (>1300 or <600)
2026-04-04 08:57:21 +02:00
Rene Fichtmueller
7db9fad108 feat: blog engine v5 — narrative control + linkedin post + min words fix
- STEP4b_NARRATIVE_CONTROL: new pipeline step after draft; detects wrong
  narrative (technology blamed instead of processes), applies anti-FUD filter,
  reality reframe ("this becomes a problem when..."), Flexoptix voice check
- System prompt: NARRATIVE CONTROL RULE added as absolute rule #1
- Gold Standard 4: corrected "compatible vs OEM" article added as reference
- Minimum words: STEP4 raised from 1500 to 2500 words (final output was 750)
- Reduction pass: 25-35% → 15-25%, target 1500-2000 words final
- STEP_LINKEDIN_POST: generates LinkedIn post ≤2800 chars (hard limit 3000);
  stores in blog_drafts.linkedin_post + linkedin_char_count column
- Pipeline now 14 steps: v5-narrative-control
- Migration 024: linkedin_post + linkedin_char_count columns in blog_drafts
2026-04-04 08:30:27 +02:00
Rene Fichtmueller
5c9cf0e9b5 feat: blog engine v4 (reduction+style-lock passes) + flexoptix scraper fixes
Blog engine (fo-blog-pipeline.ts):
- Add STEP8b_REDUCTION: cuts article 25-35%, removes repeated concepts
- Add STEP8c_STYLE_LOCK: enforces tone consistency, fixes scope/OPM confusion,
  removes inline SKUs from article flow
- Add Gold Standard 3 to calibration (Style B troubleshooting example 2026-04-04)
- Pipeline now 12 steps (was 10), version bumped to v4-reduction-stylelock

blog.ts:
- Wire STEP8b and STEP8c into pipeline between Kill-AI-Tone and QA Check
- Update progress tracking to 12 total steps
- Update pipeline_version to 'v4-reduction-stylelock'

flexoptix-catalog.ts:
- Fix contentHash call: pass object directly, not JSON.stringify(object)

db.ts:
- price_verified=true set in content_hash early-return path (no new observation)
- image_verified=true auto-set in findOrCreateScrapedTransceiver on INSERT/UPDATE
2026-04-04 07:50:01 +02:00
Rene Fichtmueller
48cb41b27e feat: hot topics now uses market_intelligence + LLM queue reset
Hot Topics:
- SOURCE 3b: market_intelligence table (15 items, 0.6+ relevance)
  with urgency mapping per intel_type + buy signal angles
- Fix news_articles: url → source_url (correct column name)
- Fix template literals: ${year} in string literals → backticks
- Increase limit: 6 → 15 topics returned
- Lower news cluster threshold: 2 → 1 article to form topic
- More research topics per day: 2 → 3
- More evergreen topics per day: 3 → 4
- Result: 24 total topics, 15 shown (was 8 total, 6 shown)

LLM Queue:
- Add resetOllamaQueue() export + auto-reset after 15min stall
- Add getQueueDepth() for monitoring
- New endpoints: GET /api/blog/llm/status, POST /api/blog/llm/reset-queue
2026-04-02 22:23:21 +02:00
Rene Fichtmueller
292bf20348 feat: 100% verified data — no invented prices, part numbers, or designations
gatherBlogData():
- Fetches real prices from price_observations (last 30 days) per product
- Filters transceivers by speed extracted from topic keywords
- Enriches every product with verified_prices array + has_verified_price flag
- Joins DB products with vector search results (DB first — they have real prices)

contextData injection (blog.ts):
- [PRODUCT] lines: exact standard_name, form_factor, speed, reach, connector, dBm specs, Watts
- [VERIFIED PRICE] lines: real EUR/USD price, vendor, observed date, source URL
- [NO VERIFIED PRICE IN DB]: explicit tag — LLM must not invent a number
- [NO PRODUCT DATA AVAILABLE]: fallback when DB returns nothing

fo-blog-pipeline.ts system prompt:
- DATA INTEGRITY RULES block: prices/part numbers/vendors ONLY from context
- Never approximate with ~€350 or 'typically $200-600' for specific products
- Power specs only from [PRODUCT] data or REFERENCE VALUES

STEP4 context instructions:
- Explicit rules on how to use [VERIFIED PRICE] vs [NO VERIFIED PRICE]
- Invented data = HARD FAIL in QA

STEP9 QA — 3 new hard fail checks (30, 31, 32):
- Check 30: invented prices → remove or replace with flexoptix.net reference
- Check 31: invented part numbers → remove, use class name instead
- Check 32: invented vendor names → remove if not in known list
2026-04-01 17:27:55 +02:00
Rene Fichtmueller
e406262a7a fix(blog): extract article from QA, status badge ready/step X/10, calibration v6 Flexoptix balance 2026-03-31 23:52:56 +02:00
Rene Fichtmueller
08cff717a6 fix(blog): 3s delay between queued LLM pipelines to prevent nginx 429 bursts 2026-03-31 19:40:40 +02:00
Rene Fichtmueller
c7b6351e26 feat(blog): regenerate button, SEO hashtags, calibration engine v2
- POST /api/blog/:id/regenerate — re-runs full 10-step LLM pipeline on existing draft
- Regenerate button visible when quality_issues present or status=review
- SEO keywords now displayed as clickable #hashtags (copy-to-clipboard)
- fo-blog-pipeline: added PoE misuse, DR4 mislabeling, ZR/DR4 conflation as hard QA fails
- fo-blog-pipeline: 14 hard rules in system prompt (was 10)
- fo-blog-pipeline: CALIBRATION_GOLD_STANDARD + withCalibration() from 10/10 human review
- System prompt now includes gold standard example on every pipeline run
2026-03-31 16:46:25 +02:00
Rene Fichtmueller
5c7cbe0ccf feat(v0.2.6): hot topics + pipeline lock + blog delete + clean external JS
Hot Topics:
- Dynamic topics from /api/hot-topics loaded in Blog Engine tab
- 7 data sources (prices, competitors, hype cycle, news, conferences, research, evergreen)
- Urgency badges: BREAKING (red), HOT (orange), TRENDING (yellow), EMERGING (green)

Pipeline Lock:
- Only 1 generation at a time, 'Pipeline Busy' toast on double-click
- Progress bar with step names (external hot-topics.js, no inline hacks)

Blog Delete:
- DELETE /api/blog/:id endpoint
- Delete button (✕) on each blog in list
- 'Delete All Templates' button to clean up test drafts

Fix: dashboard JS extracted to external hot-topics.js to avoid sed quote hell
2026-03-31 09:54:33 +02:00
Rene Fichtmueller
98a7e12282 feat: wire 10-step FO Blog Pipeline into blog generation route
Replaces old 2-pass pipeline with full Flexoptix Style 10-step generation:
1. Topic Expansion (real scenarios + wrong assumptions)
2. Angle Selection (single strong angle + audience)
3. Outline Generation (decision-driven, no generic sections)
4. Master Draft (Flexoptix voice, 2000+ words)
5. Reality Injection (failure scenarios, operational pain)
6. Technical Deepening (specific optics, power, density)
7. Opinion Layer (clear positions, no neutrality)
8. Kill AI Tone (remove all AI fingerprints)
9. QA Check (technical accuracy verification)
10. Quality Score (1-10 auto-rating, saved as self-feedback)

Feedback loop active:
- Accumulated feedback injected into system prompt
- Auto QA scores saved to blog_feedback table
- Training data export via GET /api/blog/feedback/training-data
2026-03-31 09:16:23 +02:00
Rene Fichtmueller
73ef5766e6 feat(v0.2.1): data confidence tracking + validation + blog feedback system
- Migration 016: data_confidence column (vendor_verified/enriched_estimated/scraped_unverified)
- Migration 015: blog_feedback table with 8 quality scores + free text
- Validation script: 8 physics-based rules (wavelength↔fiber, reach plausibility, power limits)
- Blog feedback API: POST /api/blog/:id/feedback + training data export
- FO Blog Pipeline v3: 10-step Flexoptix Style prompts (Less bullshit. More engineering.)
- Auto-fix: wavelength↔fiber mismatches corrected automatically
2026-03-31 09:12:37 +02:00
Rene Fichtmueller
5a0cbed5a2 feat: dashboard v2, blog expansion, market/cable MCP tools, switch asset scrapers, scraper utilities 2026-03-30 08:07:12 +02:00
Rene Fichtmueller
c6308e93c0 feat: massive scraper expansion + hype cycle engine + lifecycle prediction
New scrapers:
- GBICS.com (BigCommerce, GBP prices, 10 categories, 78 products)
- Juniper HCT (Next.js SSR parser, 475 transceivers with specs/EOL)
- SFPcables.com (Magento store, 16 categories, 78 products)
- Fluxlight (BigCommerce, 6 pages, 118 products)
- Champion ONE (compatible vendor scraper)

Scraper fixes:
- 10Gtek: rewritten to parse HTML spec tables (152 products)
- Flexoptix: fix price extraction from Magento Hyva HTML
- Register all scrapers in CLI (--gbics, --juniper, --sfpcables, etc.)

Hype Cycle Engine enhancements:
- Data-driven enrichment from scraped vendor/price data
- Revenue lifecycle prediction (peak year, decline, revenue index)
- Regional adoption model (NA, China, APAC, Europe, RoW with lag coefficients)
- New API endpoints: /enriched, /lifecycle, /regional/:tech

DB growth: 89 → 1,168 transceivers, 0 → 416 prices, 6 vendors
Qdrant: 1,162 products embedded with nomic-embed-text

Research: Norton-Bass model, standards-to-market timelines, hype signals
2026-03-28 02:30:19 +13:00
Rene Fichtmueller
f48a809e40 feat: Phase 7 — Blog generator + scraper scheduler activation
Blog draft engine generates structured markdown from all Qdrant
collections (products, news, FAQ, troubleshooting). Supports 4
topic types: hype_cycle, comparison, new_product, tutorial.

- routes/blog.ts: POST /api/blog/generate, GET/PUT endpoints
- ecosystem.config.js: Added tip-scraper PM2 process
- Scraper scheduler (pg-boss) now running on Erik with 8 job queues
- News scraper running every 6 hours on Erik
2026-03-28 00:32:08 +13:00