129 Commits

Author SHA1 Message Date
Rene Fichtmueller
f2dad45c7c fix(api): part-number ILIKE search + verified-first catalog ordering + FTS-primary product search 2026-06-04 10:14:19 +00:00
Rene Fichtmueller
adfb590ad2 feat: Transceiver Academy — full API-backed customer & employee training platform
Replaces the old LLM-training inline data module with a proper interactive
training platform for annual employee onboarding and customer education.

Content (236 KB of structured training data):
- 5 categories: Standards, Form Factors, Switches & Compatibility,
  Fiber & Infrastructure, Testing & Buying
- 22 detailed lessons with bilingual content (EN + DE)
- 74 quiz questions with explanations in both languages
- Lesson types: beginner / intermediate / advanced
- Content blocks: paragraphs, tables, callouts, code blocks, formulas, lists

API route (GET /api/training/*):
- /categories — all 5 categories with lesson/quiz counts
- /lessons?category= — lesson metadata for category
- /lessons/:id — full lesson content (sections + blocks)
- /quiz?lesson=&category= — quiz questions with bilingual answers
- /stats — aggregate stats
- Public route (no auth token required)

Dashboard (Transceiver Academy UI):
- Language toggle EN/DE (persisted in localStorage)
- Category selector tabs with lesson counts
- Lesson cards with level badge, summary, duration, completion indicator
- Full lesson viewer: renders all block types with bilingual support
- Per-lesson quiz and per-category quiz
- Question-by-question quiz engine with auto-advance, dots progress indicator
- Results screen with grade (A-F), wrong answers + explanations
- Progress tracking in localStorage, global progress bar
- Reset progress button
2026-05-14 22:31:43 +02:00
Rene Fichtmueller
e71b985c52 feat: 10 weitere Dashboard-Features (G–P)
G) SKU Bulk Pricer — POST /api/bulk-price (bis 100 Part Numbers),
   Preistabelle je Vendor, CSV-Export, Not-Found-Liste

H) Side-by-side Comparison — Checkboxes in TX-Tabelle,
   Floating Comparison-Tray (max 4 SKUs), Modal mit Specs +
   Best-Price 7d nebeneinander

I) Vendor Reliability Score — GET /api/vendors/reliability,
   Freshness(40) + Frequency(30) + Coverage(30) = 0–100,
   Progress-Bar-Badge auf Vendor-Cards

J) Price Heat Map — GET /api/price-matrix?ids=,
   Row-normalisierte Farbmatrix SKU×Vendor (grün=günstig/rot=teuer),
   Sticky SKU-Spalte, Best-Price-Spalte

K) Watchlist — localStorage-basiert (/☆ in TX-Tabelle),
   Floating Drawer, live Preis-Update via Einzelabruf

L) PDF / Print Report — window.print() + dediziertes @media print CSS
   (blendet UI-Chrome aus, behält Overview-Content)

M) Global Search Overlay — Cmd+K / Ctrl+K, durchsucht Transceivers
   + KB + News + Documents gleichzeitig, clickbare Direktlinks

N) Saved Filter Presets — localStorage tip_presets, Dropdown +
   💾-Button in TX-Filterzeile, Save/Load/Delete

O) Price Forecast — GET /api/price-forecast/:id (lineare Regression
   90d → 30d Forecast), gestricheltes Overlay auf Price-History-Chart,
   Trend-Label (rising/stable/declining)

P) Technology Radar — SVG Bull's-Eye (Adopt/Trial/Assess/Hold),
   Hype-Cycle-Phasen → Ringe gemappt, Bubbles mit Market-Signal-Score,
   Quadrant-Labels, interaktive Tooltips
2026-05-14 21:39:17 +02:00
Rene Fichtmueller
fb060ee40a feat: 6 neue Dashboard-Features (A–F)
A) Price Movers Alert
   - GET /api/procurement/price-movers?days=N&limit=N
   - CTE cur vs prior period avg, |delta_pct| >= 2%, gainers+losers
   - Procurement tab: period toggle 7d/14d/30d, stats bar, Export CSV

B) Executive Overview Pulse
   - 5 KPI cards in Overview (Buy Signals, Arbitrage Ops, Supply Alerts,
     Price Gainers, Losers) via loadProcurementPulse()
   - Top-Movers mini-table in overview card, all → Procurement tab

C) CSV Export
   - exportMoversCSV() downloads gainers+losers as CSV

D) Vendor Intelligence
   - GET /api/vendors/intelligence: per-vendor 30d stats
     (sku_count, price_obs, avg/min/max price, last_seen)
   - Top-6 banner in Vendors tab

E) Advanced Transceiver Search
   - Speed filter (1G/10G/25G/40G/100G/200G/400G/800G)
   - Fiber type filter (SMF / MMF)
   - Verified-only checkbox
   - All params forwarded to GET /api/transceivers

F) Knowledge Base Browser
   - New KB tab with full-text search (ILIKE question/answer/subcategory)
   - GET /api/kb?q=&category=&limit= (packages/api/src/routes/kb.ts)
   - Category pills, entry cards with severity badge + FF/speed tags
2026-05-14 20:54:40 +02:00
Rene Fichtmueller
bcab2b97af feat: procurement — 5 intelligence sections (A-E)
E  Buy-Now Intel    211k precomputed reorder signals surfaced,
                    filterable by form factor, signal strength bars
A  Arbitrage        59k equivalence pairs + price data, FX vs comp
                    normalized to USD, sorted by savings %
B  Switch Compat    search 429 switches → compatible transceivers
                    with prices; 58k compatibility rows
C  Supply Squeeze   4-signal detector: price momentum (30d vs 60d),
                    hype phase, AI cluster demand, stock pressure
D  Dead Stock       7,297 dead-stock SKUs matched against ascending
                    hype phases (revival candidates)

5 new API endpoints: /api/procurement/reorder-top, /arbitrage,
/switch-compat, /supply-squeeze, /dead-stock-revival
2026-05-14 18:31:07 +02:00
Rene Fichtmueller
4bd16af9a5 feat: data quality panel in Crawler Intelligence tab
GET /api/scrapers/data-quality — 4 parallel queries across 200k+
transceiver_verification_evidence rows. Returns: coverage percentages
(price 62%, image 68%, details 94%, competitor 2%), all 10 evidence
types with counts + avg confidence, 17 robot/scraper contributions,
14-day daily activity time series.

Dashboard: coverage progress bars (color-coded thresholds), evidence
type table, SVG activity sparkline, robot contributions table.
2026-05-14 16:22:25 +02:00
Rene Fichtmueller
10d13633fb feat: dynamic hype cycle + market signal engine + eBay/CapEx panels
6-source composite Market Signal Score (0-100) per transceiver technology.
New GET /api/hype-cycle/market-signals blends: Norton-Bass hype_score,
hyperscaler CapEx YoY (MSFT +68.8%, GOOG +107%, META +46.8%), price
observation activity ratio 30d vs prior 30d, AI cluster transceiver demand,
eBay secondary market sell-through velocity, internal fast-mover trend.
All 6 queries run in parallel via Promise.all().

Recommendation engine maps hype phase × capex boom × speed class →
Buy/Hold/Watch labels with tooltips. Dashboard Hype Cycle table now shows
Market Signal ● LIVE column + Recommendation column. Hyperscaler CapEx
panel + eBay panel added to hype tab. Procurement: new eBay Market section.
Sourcing Hype Cycle replaced hardcoded seed with live price observation data.
2026-05-14 16:17:52 +02:00
Rene Fichtmueller
13fe33eceb feat: procurement — Internal Demand + AI Clusters sections with real data
Two new procurement sub-tabs backed by live database tables:

📦 Internal Demand (flexoptix_internal_demand, 8,585 SKUs):
- Velocity cards: fast_mover (70 SKUs, 53k units/12M), regular, slow, dead stock
- Filterable table with demand_12m, demand_3m, trend %, form factor
- GET /api/procurement/internal-demand — summary + paginated rows

🤖 AI Clusters (ai_cluster_announcements, 396 rows last 30d):
- Live datacenter build announcements with estimated transceiver demand
- Stats: total announcements, MW sum, distinct companies, total ~transceivers
- Filter for entries with transceiver estimates; time range selector
- GET /api/procurement/ai-clusters — data + period stats

Also: replaced misleading DEMO DATA banners on Reorder Signals and ABC
Classification sections with informational notes pointing to real data.
2026-05-14 16:04:11 +02:00
Rene Fichtmueller
ea8be4aea3 feat(tip): equivalences explorer + price history charts + linkedin status + MCP tools
Equivalences Explorer:
- GET /api/equivalences — search 63k cross-brand mappings by part number/vendor
- GET /api/equivalences/transceiver/:id — all equivalences for a specific product
- GET /api/equivalences/stats — active count, unique products, avg confidence (93.9%)
- GET /api/equivalences/top-vendors — top 20 competitor vendors by coverage
- New "Equivalences" tab in dashboard with part-number search, vendor filter,
  quick-click vendor chips, and results table with confidence coloring
- Transceiver detail modal: equivalences panel (Flexoptix alternatives or competitor
  products), clickable rows, confidence percentage, orange highlight for FX products

Price History Charts:
- GET /api/price-history/:id?days=90 — daily min/max/avg per source vendor (392k obs)
- Transceiver detail modal: SVG sparkline chart per vendor, legend with latest prices,
  range summary — loads async without blocking the modal

LinkedIn Distribution Status:
- GET /api/blog/linkedin/history — from blog_linkedin_distribution table
- Blog tab: LinkedIn status panel showing DRY_RUN badge, posted/dry_run/skipped/failed
  stats, distribution history table with URN link to live posts

MCP Server — 2 new tools:
- find_equivalences: search 63k+ verified cross-brand mappings with confidence filter
- get_price_history: 392k+ observations, daily series, per-vendor analysis, cheapest source
2026-05-14 15:54:01 +02:00
Rene Fichtmueller
67310c8fe7 fix(blog): SPA-aware URL blog generation + dynamic generated_by
- fetchUrlContent() now extracts OG/meta tags (og:title, og:description,
  name="description", og:site_name) as fallback content for JS-rendered SPAs
- Returns spaDetected=true when body text < 300 chars after stripping scripts
- from-url endpoint skips gatherBlogData() product injection when SPA detected,
  preventing fo-blog-v10 from defaulting to optical networking domain
- additionalContext now includes SPA warning instructing LLM not to default
  to optical transceiver topics unless the page is actually about that
- generated_by in pipeline UPDATE query now uses active model name instead of
  hardcoded 'fo-blog-engine-v7' (reads getLlmProvider().ollamaModel)
- Dashboard shows SPA warning toast when spa_detected=true in response
- Response now includes spa_detected field for client awareness
2026-05-14 12:29:17 +02:00
Rene Fichtmueller
e0f9656684 feat: Blog Engine — generate from URL (link → BlogLLM → article)
New POST /api/blog/from-url endpoint:
- Accepts url + topic in request body
- Fetches page server-side (no CORS, 20s timeout, redirect-follow)
- Strips script/style/nav/footer/svg; extracts readable text (~5000 chars)
- Extracts page title from <title> or <h1>
- Passes extracted content as structured additional_context to the
  existing 16-step FO blog pipeline (same as manual generation)
- Returns immediately; LLM pipeline runs async
- Validated: smoke test fetched flexoptix.net/en/blog/, 5040 chars,
  pipeline launched with llm_enhancing=true

New "🔗 Blog aus URL generieren" panel in dashboard:
- URL input (Enter key triggers generation)
- Blog-Typ dropdown (same 8 types as manual panel)
- Button shows loading state " Fetching…" during API call
- Status line shows extracted char count after success
- Reuses pollBlogLlm() for step-by-step progress polling
- Inline status field for error display without toast spam
2026-05-14 00:55:35 +02:00
Rene Fichtmueller
9b8b03e783 feat: Flexoptix section — speed formatting + Lagerbestand display
Speed display: fix raw Gbps decimals → formatted labels
- 1600.00G → 1.6T (≥1000 Gbps converted to T)
- 400G → 400G (clean integer, no trailing .00)
- Helper function fmtSpeed() added in dashboard JS

Lagerbestand: add stock availability per transceiver
- getFlexoptixSuggestions() extended with LEFT JOIN LATERAL on
  stock_observations (latest row per transceiver)
- Returns warehouse_de_qty, warehouse_global_qty, backorder_qty,
  backorder_estimated_date
- Dashboard renders color-coded badges per row:
    green  = DE-Lager quantity
    blue   = Global-Lager quantity
    yellow = Zulauf with estimated delivery date if available
- Badges hidden when all quantities are null/zero (graceful fallback)
2026-05-14 00:52:21 +02:00
Rene Fichtmueller
de179c4c7c fix: remove DEMO labels from real stock data; fix switch Flexoptix suggestions; enrich Hot Topics LLM context
Stock dashboard (index.html):
- Replace all [DEMO]/demo badges on warehouse data with "FS.com" source labels
  (data was always real scraper data, never demo in the DB)
- Update subtitle: "Scraper-Lagermengen: DEMO DATA" → "Wettbewerber-Marktdaten"
- "Recently Restocked" badge: DEMO DATA → SCRAPER DATA

Switch detail (queries.ts):
- Fix getFlexoptixSuggestions: wavelength_nm → wavelength_tx_nm,
  price_verified_usd → street_price_usd (column mismatch with live schema)
- DS5000 and other OSFP switches now show all 62 Flexoptix OSFP transceivers
  with direct shop links in the detail modal

Hot Topics (hot-topics.ts):
- NOG Talks + News Article clusters now fetch summary/mentioned_vendors/
  mentioned_products/mentioned_standards from news_articles table
- description field builds bullet-point list per article with summaries,
  key vendors/standards (vs. 3 bare titles joined with "|" before)
- buildTopicBriefing() rewritten as structured LLM document with sections:
  Market Signals (bullets), Recommended Angle, Market Context (buy signal,
  technologies, impact horizon), Writing Instructions (600-900 words,
  actionable, opinionated, no generic summaries)
2026-05-14 00:33:45 +02:00
Rene Fichtmueller
0d7a92e749 feat: Abverkauf velocity engine — sql/118 + analyzer + API endpoints
- sql/118-stock-velocity.sql: new stock_velocity (UPSERT per tx×vendor)
  and stock_velocity_events tables with TimescaleDB-compatible indexes
- stock-velocity-analyzer.ts: computes sell-through from stock_observations
  time-series; detects sold/zulauf/data_gap events, trims top-10% outliers,
  predicts stockout date, assigns high/medium/low/insufficient confidence
- scheduler.ts: analyze:stock:velocity job at 04:30/12:30/20:30 UTC
- stock.ts: GET /api/stock/velocity (paginated, filterable by vendor/confidence/
  stockout_days) + GET /api/stock/velocity/:id (per-product with event history)
- First run: 208 products, 979 sell events, 2811 Zulauf events written
2026-05-14 00:24:58 +02:00
Rene Fichtmueller
048bf0dcf2 feat: add Codex task for Flexoptix reference matching overhaul
CODEX-TASK-flexoptix-reference-matching.md — comprehensive plan to fix
zero-match gap for ATGBICS/NADDOD/10Gtek/ShopFiber24 (8.260+ products
with 0 Flexoptix equivalences).

Root cause: 30-day price_observation window excludes vendors whose
scrapers ran >30 days ago. Solution: catalog-reconcile robot (full
bulk match, no time limit), form_factor normalization (SQL 108),
30→90 day window fix in nightly matcher, on-demand API endpoint.

Expected: coverage from 22% → 45-60% after one reconcile run.
2026-05-13 16:51:53 +02:00
Rene Fichtmueller
a0657ee565 fix: filter TIP hot topics quality 2026-05-10 15:54:38 +02:00
Rene Fichtmueller
5eb1b07183 fix: close stale TIP manual review queue 2026-05-10 10:23:07 +02:00
Rene Fichtmueller
cf0e471fa4 feat: close TIP research resolution states 2026-05-10 10:13:09 +02:00
Rene Fichtmueller
10af2ca244 fix: generated_by tag — v6-length-fix → v7 2026-05-10 09:55:39 +02:00
Rene Fichtmueller
b58f7cee41 feat: resolve OEM price status and part details 2026-05-10 01:16:49 +02:00
Rene Fichtmueller
650de6ba9a feat: add verification evidence state model 2026-05-09 23:06:21 +02:00
Rene Fichtmueller
1af4f090f7 fix: harden TIP verification cleanup 2026-05-09 22:16:29 +02:00
Rene Fichtmueller
a43e572946 fix: advance TIP product verification robots 2026-05-09 20:19:19 +02:00
Rene Fichtmueller
fc18b00157 fix: verify copper cable semantics 2026-05-09 16:55:50 +02:00
Rene Fichtmueller
61acccf5df fix: require strict comparable transceiver evidence 2026-05-09 16:02:49 +02:00
Rene Fichtmueller
43b7250180 fix: automate equivalence research review queue 2026-05-09 07:48:11 +02:00
Rene Fichtmueller
a1a525b332 chore: sync API routes, dashboard hot-topics, MCP server, scraper package, scripts 2026-05-06 23:39:04 +02:00
Rene Fichtmueller
270bd12382 feat(dashboard): clickable LLM model selector — switch blog engine at runtime
- client.ts: BLOG_LLM_PROVIDER/OLLAMA_LLM_MODEL as mutable state (setLlmProvider/
  getLlmProvider). Reads blog-llm-settings.json on startup for persistence.
  All generate()/checkHealth()/chat() calls use dynamic provider() + llmModel()
  — no restart required for switches.

- blog.ts: POST /api/blog/llm/switch endpoint — validates provider, calls
  setLlmProvider(), writes settings file, returns previous+active state.

- index.html: all 4 model cards now clickable (cursor:pointer, hover fade).
  switchBlogLlm(provider, model) — disables cards during switch, shows
  green/red feedback toast, auto-refreshes status. SSH instruction removed.
2026-04-29 01:15:45 +02:00
Rene Fichtmueller
b5decc517f fix: hard-cap blog generation to 800-1100 words
LLM_OPTS.maxTokens 8192 → 1600, LLM_REFINE 6144 → 1800,
Step 4 master draft 8192 → 1600. Added explicit word-count
constraint to STEP4_MASTER_DRAFT prompt: HARD LIMIT 800-1100 words.

Root cause: no token ceiling → fo-blog-v6 produced 4000-5000w articles.
Generated-by label updated to fo-blog-engine-v6-length-fix.
2026-04-28 22:45:48 +02:00
Rene Fichtmueller
d7144731e0 feat(scraper): add 100+ OEM seed scrapers + tip-llm-guided inference layer
New OEM transceiver seed scrapers (94 cron-scheduled, 24/7):
- Media/Broadcast: Evertz, Grass Valley, Haivision, Viasat
- Asian Optical: FiberHome, Oplink, Accelink, Hisense Broadband
- Optical Mfrs: Lumentum, II-VI/Coherent, Source Photonics, O-Net,
  InnoLight, AOI, Sumitomo Electric, NeoPhotonics
- Industrial: GE Grid, Schweitzer, Moxa Industrial, Cisco IE,
  Phoenix Contact, Beckhoff, Omron, ABB, Siemens, Schneider, Rockwell, Belden
- Enterprise/DC: Arista, Pica8, Pluribus, DriveNets, Cisco (Meraki/Catalyst/Nexus/ASR)
- Cloud: AWS, Azure, Google Cloud, Meta
- Storage: NetApp, Pure Storage, HPE Storage, IBM Storage, Dell Storage, Hitachi Vantara
- 5G/RAN: Samsung Networks, Nokia AirScale, Ericsson RAN, Mavenir
- Security: Check Point, Barracuda, Fortinet, Palo Alto
- Telecom Optical: ADVA, PacketLight, FiberHome, Accelink, Hisense

API: tip-llm-guided inference layer (strict schema + repair-retry + safe fallback)
- POST /api/tip-llm/infer|research-plan|extract|finding|health
- Hard JSON schema enforcement, create_finding=false on empty evidence
- Confidence gate (>= 0.4), validation with consistency check

Build: added incremental=true to scraper tsconfig (OOM prevention)
Scheduler: 87 → 94 registered workers
2026-04-27 00:00:14 +02:00
Rene Fichtmueller
d5be0ba43c fix: catch non-zero exit in local train, return JSON instead of 500
- sh -lc replaced with bash to avoid dash/profile.d syntax errors
- runCommand errors now caught in local provider path
- stdout/stderr extracted from error object and returned as JSON
- No more HTTP 500 on script failure
2026-04-25 23:24:04 +02:00
Rene Fichtmueller
bfb43809a8 feat: standards audit + form factors reference
- Migration 100: adds `description` column to standards (plain-language
  DE·EN for non-technical colleagues), fills all 63 standards incl.
  complete 200G tier (SR4/DR4/FR4/LR4/ER4/CR4), copper DAC variants,
  PON family (GPON/XG-PON1/NG-PON2/25G-PON), 1.6T emerging standard

- Migration 101: new form_factors table — 20 entries covering SFP family
  (SFP→SFP112), QSFP family (QSFP+→QSFP-DD800), OSFP family (OSFP→OSFP224),
  CFP family, legacy XFP/CXP with full_name, speed, channels, status,
  supersedes chain, and bilingual plain-language descriptions

- GET /api/form-factors — new endpoint, returns all form factors with
  transceiver_count join
- GET /api/form-factors/:name — single form factor detail

Dashboard Standards tab:
  - DB description shown as subtitle in standards table rows
  - Full DE + EN description in standard detail panel
  - New Form Factors grid section with status badges, speed, channel info,
    family color coding, supersedes chain
  - openFormFactorDetail() panel with full specs + transceiver link
  - Search extended to match description + notes fields
2026-04-25 20:58:45 +02:00
Rene Fichtmueller
ff4bc34930 fix: remove IP restriction from internal-demand (Cloudflare tunnel breaks req.ip)
JWT auth + RLS is the real security boundary. IP check was blocking legitimate
dashboard users accessing via Cloudflare tunnel (X-Forwarded-For = real user IP,
not 127.0.0.1). Added /api/internal/demand/stock-analysis: demand × live stock
combined analysis with reorder_urgency, coverage_days_de, momentum_ratio.
Dashboard: new Demand × Live Stock Analyse panel with critical/low/ok/overstock chips.
2026-04-25 20:41:35 +02:00
Rene Fichtmueller
f162e03978 feat: Flexoptix internal demand intelligence + real forecast calibration
- Migration 099: flexoptix_internal_demand table with RLS + v_demand_by_speed view
- Import script: AES-256-CBC decrypt → parse 8585 SKUs → upsert with velocity class
- 279 SKUs cross-referenced to transceiver catalog; 1288 with real demand data
- New /api/internal/demand/* routes (by-speed, velocity, hype-weights, forecast-input)
  — protected by JWT auth + localhost/LAN IP restriction middleware
- Forecast engine calibrated with real Flexoptix run-rates (demand_calibrated flag)
- Dashboard: real Flexoptix Sales Velocity panel replaces DEMO DATA in Warehouse tab
  with momentum indicators, velocity class breakdown, trend arrows
- Security: data stays on private server; RLS enforces is_internal=TRUE at DB layer
2026-04-25 17:44:20 +02:00
Rene Fichtmueller
8c625ff1d2 feat(training): add TIP selflearning workflows 2026-04-25 12:21:56 +02:00
Rene Fichtmueller
9e6be570a3 feat: more switch image coverage + system health metrics + Erik monitor
switch-image-fetcher:
- Add Fortinet URL builder (11 FortiSwitch models)
- Add Quanta Cloud Technology, Allied Telesis, Ufispace, Netgear URL builders
- Fix alcatel-lucent-enterprise slug missing from URL_BUILDERS dispatcher
- Fix NVIDIA builder to skip ConnectX/BlueField adapters (not switches)
- Add aruba slug alias for hpe-aruba

health endpoint:
- Add system metrics: CPU load (1/5/15m), memory usage, disk usage
- Add load_status indicator (ok/busy/overloaded)
- Expose process RSS memory
- Used by external monitors

scripts/monitor-erik.sh:
- Cron-ready health check script for Claudi (.82) and Raspberry Pis
- Checks TIP API health endpoint (load, memory, disk, DB latency)
- Checks PM2 process state via SSH (errored/stopped detection)
- ntfy.sh push notifications (set NTFY_TOPIC env var)
- Includes systemd service + timer unit comments for auto-install
2026-04-21 00:31:43 +02:00
Rene Fichtmueller
a2492d833b feat: Flexoptix order section per switch + reject generic/logo images 2026-04-20 23:31:36 +02:00
Rene Fichtmueller
c0cd0dc1ca feat: compatibility panel — verification_method, competitor prices, spec-match collapsible
- API: getCompatibleTransceivers() returns verification_method, orders vendor_compat first
- Dashboard: Flexoptix section splits vendor-tested vs spec-match (collapsed)
- Dashboard: Competitor section shows vendor-tested with prices, spec-match as chips
2026-04-20 22:52:49 +02:00
Rene Fichtmueller
0fb4850dfa fix: price-comparison SKU lookup — wrong column refs (so.stock_level, search_url_template) 2026-04-19 00:12:18 +02:00
Rene Fichtmueller
446ac667b0 feat: side-by-side competitor comparison + fix 1.6T speed_gbps
- Fix OSFP-DR8-1.6T-FL and OSFP-2FR4-1.6T-FL: speed_gbps was 200, now 1600
  → FS.com 1.6T products now correctly match as comparables for Flexoptix O.1316T.C.05.M
- API: extend comparable price query to return comp_form_factor, comp_speed_gbps,
  comp_reach_meters, comp_reach_label, comp_fiber_type, comp_wavelengths
- Dashboard: replace plain comparable price row with side-by-side spec comparison card
  showing Flexoptix vs. competitor: Form Factor, Speed, Reach, Fiber, Wavelengths
  with color coding (green=match, orange=mismatch) and savings badge (−45% günstiger)
2026-04-18 21:51:41 +02:00
Rene Fichtmueller
62d97a783c feat: add claude-code LLM provider + update dashboard to fo-blog-v5
- client.ts: add claude-code provider routing BLOG_LLM_PROVIDER=claude-code
  to claude-bridge (flat-rate, no API billing via Claude Code subscription)
- checkHealth() now pings /health on claude-bridge for real availability check
- Default OLLAMA_LLM_MODEL changed from qwen2.5:14b to fo-blog-v5
- Dashboard: add claude-code card (EMPFOHLEN), rename fo-blog-v3 → fo-blog-v5
- loadBlogLLMStatus() handles all 3 providers: claude-code/anthropic/ollama
- Grid expanded from 3 to 4 columns to accommodate new card
- ecosystem.config.js + .env on Erik: OLLAMA_LLM_MODEL=fo-blog-v5 confirmed
2026-04-18 20:45:14 +02:00
Rene Fichtmueller
1c8dec52c9 feat: Price Comparison dashboard + Eoptolink OEM scraper
- Add public /api/price-comparison API (summary, top-50, per-SKU detail)
  — no auth required, 3 Express routes, DISTINCT ON latest-price logic
- Add '💲 Price Comparison' dashboard tab: stat cards, form-factor
  breakdown, top-50 SKU table (clickable rows → SKU detail), per-vendor
  price + stock + spread% lookup panel
- Add Eoptolink OEM catalog scraper (93 product-solution pages,
  part-number regex EOLO-*/EOLQ-* etc., no prices, seeds transceivers
  table as manufacturer entries)
- Register scrape:catalog:eoptolink in scheduler: schedule every 4h
  (40 */4 * * *), lazy-import worker, added to known-jobs array
2026-04-18 01:02:08 +02:00
Rene Fichtmueller
e9fcda2811 feat: wire finder.ts + switch-docs + Ollama LLM tools to MCP server
MCP Server (packages/mcp-server/src/index.ts):
- Register registerSwitchDocTools (switch-docs.ts) — switch documentation lookup
- Register finderTools dynamically (finder.ts) — find_flexoptix_for_switch, get_competitor_alerts
- Add analyze_market_with_llm tool: qwen2.5:14b via Ollama, enriched with live hype cycle + pricing + news
- Add generate_blog_post tool: fo-blog-v5 (fine-tuned) with qwen2.5:14b fallback, enriched with live pricing data
- OLLAMA_BASE_URL env var (default: https://ollama.fichtmueller.org)

Also includes scraper improvements (ascentoptics, atgbics, gbics, skylane, ebay-enricher),
API route updates (blog, blog-sll, health, hot-topics, transceivers, queries),
and dashboard hot-topics refresh.
2026-04-18 00:21:58 +02:00
Rene Fichtmueller
b88a6e28cf feat: /api/hype-cycle/analysis endpoint — DB-backed Bass-fitted results from hype_cycle_analysis table 2026-04-18 00:11:08 +02:00
Rene Fichtmueller
861243ea3f feat: stock confidence badges, multi-vendor price comparison, expanded Cisco TMG + Juniper HCT
Stock API & Dashboard:
- /api/stock/summary: vendor_breakdown adds avg_confidence, currencies, conf_per_warehouse/aggregated/boolean
- /api/stock/summary: new price_comparison endpoint (multi-vendor SKUs, min/max/avg price)
- /api/stock/summary: totals adds multi_vendor_skus count
- Dashboard: 6th stat card (Multi-Vendor SKUs), confidence badge column (🟢 L3 / 🟡 L2 /  L1)
- Dashboard: price comparison table with vendor-by-vendor price breakdown
- Dashboard: subtitle updated to include QSFPTEK + NADDOD
- Dashboard: top sellers link to product URLs

Cisco TMG improvements:
- Added 5 new platform families: 8000 Series, NCS5500, NCS540, NCS560, NCS1000
- Per-device query strategy: iterates all switch model IDs from family filter
  instead of getting only 1 switch per family → 58 switches per N9300 run
- Graceful error handling per device with rate limiting (1s between requests)

Juniper HCT: ran manually → 475 Juniper-brand transceivers seeded
2026-04-17 23:33:31 +02:00
Rene Fichtmueller
5b35b2b8be feat(scraper+api): warehouse stock data pipeline — FS.com v2, SmartOptics v2, Stock API
Scraper changes:
- fs-com.ts v2: Playwright stealth patches + www.fs.com/de/ URL fix (de.fs.com DNS NXDOMAIN).
  Extracts DE-Lager, Global-Lager, Nachlieferung, units_sold, compatible_brands, price_net.
  Mac-side runner (run-fs-scraper-mac.sh) via SSH tunnel for residential IP access.
  Fast-fail connectivity check on datacenter IPs that are blocked by Cloudflare.
- smartoptics.ts v2: WooCommerce REST API fallback + 8 catalog categories + relative URL fix.
  Was finding only 8 products, now discovers 18+ with multi-category crawl.

DB layer:
- db.ts: add upsertStockObservation() — writes 10 new stock_observations columns
  (warehouse_de_qty, warehouse_global_qty, backorder_qty, units_sold, compatible_brands,
  price_net, product_url, delivery dates) with dedup check.

API:
- routes/stock.ts: GET /api/stock, /api/stock/summary, /api/stock/:id
  Warehouse breakdowns per transceiver/vendor with top-sellers and vendor summary.
- routes/review.ts: equivalence review queue (approve/reject/bulk-approve).
- index.ts: register /api/stock and /api/review routes.

Dashboard:
- index.html: 🏭 Stock tab with stat cards (DE-Lager, Global-Lager, Nachlieferung totals),
  top-sellers table, vendor breakdown, recently-restocked events, part-number lookup.

SQL migrations:
- 034: blog-review-tag, 035: price-observations is_anomalous, 036: transceiver-equivalences.
2026-04-17 10:45:59 +02:00
Rene Fichtmueller
cddc92c9d2 feat: TIP audit fixes — Qdrant init, switches columns, verification fix, crawler live status, demo data badges
- Migration 032: add system_type, is_linecard, chassis_model, slot_type, flexbox_* to switches table
- Migration 032: fix compute_transceiver_verification() to count seed data as details_verified (100% now)
- Migration 032: add is_demo_data flag to reorder_signals, abc_classification, market_intelligence, stock_snapshots
- Cisco 8000: insert 8812, 8818, 8800-LC-36FH, 8800-LC-48H with correct vendor slug 'cisco'
- API: add /api/scrapers/jobs endpoint exposing pg-boss job queue (active/recent/queues)
- Dashboard: live job queue panel in Crawler Intelligence tab (active jobs + recent 4h completions)
- Dashboard: DEMO DATA badge now uses is_demo_data column (was checking wrong field is_demo)
- Blog engine: configured fo-blog-v3-qwen7b fine-tuned model via tip-api ecosystem.config.js
- Qdrant: all 6 collections created, seeded (2135 products, 29 FAQs, 39 news, 20 troubleshooting)
2026-04-09 20:29:46 +02:00
Rene Fichtmueller
cf75eee8ad feat: linecard system support, Cisco 8000 accuracy, price anomaly detection
API/finder:
- Add modular chassis support: sibling linecards fetched when is_linecard=true
- Add chassis linecards when system_type=modular
- Extend switch response: system_type, is_linecard, chassis_model, slot_type,
  flexbox_compat_mode, flexbox_notes, description, switching_capacity_tbps,
  total_ports, category, lifecycle_status, features, use_cases, linecards[]

API/transceivers:
- Filter price_observations with COALESCE(is_anomalous, false) = false
  (direct prices + comparable market prices)

Scraper/db:
- Add PRICE_BOUNDS map (per form-factor min/max USD sanity bounds)
- Add isPriceAnomalous() — marks DB price_observations as is_anomalous=true
- Add competitor_verified flag: set true when valid competitor price stored
- upsertPriceObservation: skip prices outside sanity bounds, set competitor_verified

Scraper/hash:
- contentHash() now accepts Record<string,unknown> | string (union type)
  to support both structured objects and legacy string callers

Scrapers (skylane, tscom, wiitek):
- Fix contentHash() call signature: pass objects not JSON.stringify strings
- Fix wiitek: remove invalid 'name' param, fix t.id → transceiverId

Migrations:
- Add is_anomalous, competitor_verified, competitor_verified_at,
  image_primary columns
- Recreate sync_fully_verified trigger to include competitor_verified
- Add is_linecard, chassis_model, system_type, slot_type,
  flexbox_compat_mode, flexbox_notes to switches table
2026-04-09 09:06:22 +02:00
Rene Fichtmueller
51af249361 Merge remote-tracking branch 'github/main'
# Conflicts:
#	packages/api/src/llm/fo-blog-pipeline.ts
#	packages/api/src/routes/blog.ts
#	packages/scraper/src/scheduler.ts
#	packages/scraper/src/scrapers/fs-com.ts
#	packages/scraper/src/scrapers/gbics.ts
2026-04-06 18:03:36 +02:00
Rene Fichtmueller
3928755c60 fix: correct verified badge, comparable pricing, and clickable product images
- Reset details_verified=false for 298 products where reach_label is empty (DB migration)
- Runtime check in dashboard: dVer requires non-empty reach_label regardless of DB flag
- comparable price query: treat reach_meters=0 same as NULL so 800G OSFP products
  find FS.com equivalent prices (was blocked by reach_meters=0 != NULL shortcircuit)
- Product image area now fully clickable with vendor link overlay when product_page_url exists
- Clear wrong image for O.Czz8HG.z.R (was showing unrelated OSFP product image)
2026-04-06 10:24:39 +02:00