Equivalences Explorer:
- GET /api/equivalences — search 63k cross-brand mappings by part number/vendor
- GET /api/equivalences/transceiver/:id — all equivalences for a specific product
- GET /api/equivalences/stats — active count, unique products, avg confidence (93.9%)
- GET /api/equivalences/top-vendors — top 20 competitor vendors by coverage
- New "Equivalences" tab in dashboard with part-number search, vendor filter,
quick-click vendor chips, and results table with confidence coloring
- Transceiver detail modal: equivalences panel (Flexoptix alternatives or competitor
products), clickable rows, confidence percentage, orange highlight for FX products
Price History Charts:
- GET /api/price-history/:id?days=90 — daily min/max/avg per source vendor (392k obs)
- Transceiver detail modal: SVG sparkline chart per vendor, legend with latest prices,
range summary — loads async without blocking the modal
LinkedIn Distribution Status:
- GET /api/blog/linkedin/history — from blog_linkedin_distribution table
- Blog tab: LinkedIn status panel showing DRY_RUN badge, posted/dry_run/skipped/failed
stats, distribution history table with URN link to live posts
MCP Server — 2 new tools:
- find_equivalences: search 63k+ verified cross-brand mappings with confidence filter
- get_price_history: 392k+ observations, daily series, per-vendor analysis, cheapest source
- fetchUrlContent() now extracts OG/meta tags (og:title, og:description,
name="description", og:site_name) as fallback content for JS-rendered SPAs
- Returns spaDetected=true when body text < 300 chars after stripping scripts
- from-url endpoint skips gatherBlogData() product injection when SPA detected,
preventing fo-blog-v10 from defaulting to optical networking domain
- additionalContext now includes SPA warning instructing LLM not to default
to optical transceiver topics unless the page is actually about that
- generated_by in pipeline UPDATE query now uses active model name instead of
hardcoded 'fo-blog-engine-v7' (reads getLlmProvider().ollamaModel)
- Dashboard shows SPA warning toast when spa_detected=true in response
- Response now includes spa_detected field for client awareness
New POST /api/blog/from-url endpoint:
- Accepts url + topic in request body
- Fetches page server-side (no CORS, 20s timeout, redirect-follow)
- Strips script/style/nav/footer/svg; extracts readable text (~5000 chars)
- Extracts page title from <title> or <h1>
- Passes extracted content as structured additional_context to the
existing 16-step FO blog pipeline (same as manual generation)
- Returns immediately; LLM pipeline runs async
- Validated: smoke test fetched flexoptix.net/en/blog/, 5040 chars,
pipeline launched with llm_enhancing=true
New "🔗 Blog aus URL generieren" panel in dashboard:
- URL input (Enter key triggers generation)
- Blog-Typ dropdown (same 8 types as manual panel)
- Button shows loading state "⏳ Fetching…" during API call
- Status line shows extracted char count after success
- Reuses pollBlogLlm() for step-by-step progress polling
- Inline status field for error display without toast spam
Speed display: fix raw Gbps decimals → formatted labels
- 1600.00G → 1.6T (≥1000 Gbps converted to T)
- 400G → 400G (clean integer, no trailing .00)
- Helper function fmtSpeed() added in dashboard JS
Lagerbestand: add stock availability per transceiver
- getFlexoptixSuggestions() extended with LEFT JOIN LATERAL on
stock_observations (latest row per transceiver)
- Returns warehouse_de_qty, warehouse_global_qty, backorder_qty,
backorder_estimated_date
- Dashboard renders color-coded badges per row:
green = DE-Lager quantity
blue = Global-Lager quantity
yellow = Zulauf with estimated delivery date if available
- Badges hidden when all quantities are null/zero (graceful fallback)
Stock dashboard (index.html):
- Replace all [DEMO]/demo badges on warehouse data with "FS.com" source labels
(data was always real scraper data, never demo in the DB)
- Update subtitle: "Scraper-Lagermengen: DEMO DATA" → "Wettbewerber-Marktdaten"
- "Recently Restocked" badge: DEMO DATA → SCRAPER DATA
Switch detail (queries.ts):
- Fix getFlexoptixSuggestions: wavelength_nm → wavelength_tx_nm,
price_verified_usd → street_price_usd (column mismatch with live schema)
- DS5000 and other OSFP switches now show all 62 Flexoptix OSFP transceivers
with direct shop links in the detail modal
Hot Topics (hot-topics.ts):
- NOG Talks + News Article clusters now fetch summary/mentioned_vendors/
mentioned_products/mentioned_standards from news_articles table
- description field builds bullet-point list per article with summaries,
key vendors/standards (vs. 3 bare titles joined with "|" before)
- buildTopicBriefing() rewritten as structured LLM document with sections:
Market Signals (bullets), Recommended Angle, Market Context (buy signal,
technologies, impact horizon), Writing Instructions (600-900 words,
actionable, opinionated, no generic summaries)
- DELETE 2133 rows from reorder_signals WHERE is_demo_data = true
- atgbics.ts: add upsertStockObservation (confidence=1, binary available
boolean from Shopify API; quantityAvailable 1/0 for in/out stock)
- optcore.ts: add upsertStockObservation (confidence=1, WooCommerce text
stock level parsed via parseStockLevel; quantityAvailable 1/0)
- Both scrapers already run every 2h on Erik scheduler
- FS.com: already captures full warehouse breakdown (DE+Global+backorder)
3x/day from Mac (02:00/10:00/18:00) at confidence=3 — no change needed
- QSFPTEK: already captures real quantities at confidence=2 — no change
- sfpcables/prolabs/wiitek: no meaningful stock signal, not modified
- sql/108: normalize form_factor across all vendors (SFP-Plus → SFP+, etc.)
and round speed_gbps for consistent matching
- sql/109: document 30→90 day matcher window change
- robots/catalog-reconcile.ts: new bulk-reconcile robot — matches all
Flexoptix products against all competitors without 30-day time limit
- scheduler.ts: register catalog:reconcile job (monthly + on-demand),
fix nightly matcher 30→90 day window, UPPER() form_factor matching,
ROUND() speed_gbps matching
Fixes: ATGBICS/NADDOD/10Gtek/ShopFiber24 had 0 Flexoptix equivalences
due to stale price_observations being filtered out. Expected coverage
improvement: 22% → 45-60% after first reconcile run.
CODEX-TASK-flexoptix-reference-matching.md — comprehensive plan to fix
zero-match gap for ATGBICS/NADDOD/10Gtek/ShopFiber24 (8.260+ products
with 0 Flexoptix equivalences).
Root cause: 30-day price_observation window excludes vendors whose
scrapers ran >30 days ago. Solution: catalog-reconcile robot (full
bulk match, no time limit), form_factor normalization (SQL 108),
30→90 day window fix in nightly matcher, on-demand API endpoint.
Expected: coverage from 22% → 45-60% after one reconcile run.
End-to-end deployment record for the 127→227 article corpus expansion:
- Gitea push (transceiver-db@f311e08)
- Magatama pool reconciliation (magatama@0e42de9)
- Erik sync via scp
- RunPod training trigger (job 0141303c, lane fo_blogllm, 500 iters)
Documents the continuous evolution plan (per-article + quarterly refresh)
and quality gates going forward.
Rewrite Python code samples to read credentials from environment
via load_credentials_from_env() helper rather than literal kwargs.
Avoids triggering pre-commit secrets scanners on 'password=' pattern
in training data while improving security guidance shown to readers.
Replace literal 'apipass'/'admin' placeholder credentials with
explicit <USER>/<PASSWORD> placeholders. Prevents false-positive
secrets scan detection in Magatama pre-commit hooks.
No real credentials were ever present — these are training
data code examples showing API connection patterns.
- index-pi.ts: removed Playwright scrapers (FS.COM, eBay enricher, switch assets)
added NADDOD (fetch-based, benefits from residential IP)
now 32 fetch-only queues safe for ARM/Pi without Chromium
- index-fs-only.ts: new dedicated FS.COM + NADDOD worker for Erik
routes through Pi SOCKS5 via PROXY_URLS=socks5://10.10.0.6:1080
Crawlee ProxyConfiguration automatically applies to Playwright crawler
- pi-scraper-setup.sh: removed inline index-pi.ts override (repo version now authoritative)
- CODEX-TASK-pi-scraper-deploy.md: full 9-step Codex spec for Pi fleet setup
covers WireGuard keypair, Erik peer config, setup script, ecosystem.config.js
- CODEX-TASK-zero-manual-review.md: deterministic equivalence matcher spec
