rene/transceiver-db
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
293 Commits 5 Branches 9 Tags
Commit Graph

293 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Rene Fichtmueller
6be2c131d3 fix: add PID lock to run-fs-scraper-mac.sh — prevent simultaneous instances 
Adds /tmp/tip-fs-scraper.lock PID file to prevent launchd from running
a second instance while the previous one is still active (e.g. 2am
schedule fires, runs past 10am when launchd fires again). Without this,
concurrent instances caused rmSync(storage-fs-phase1) in one instance
to delete the Crawlee storage dir while the other was still using it,
resulting in ENOENT crashes.
 2026-04-18 02:43:28 +02:00
Rene Fichtmueller
681fd5ced6 chore: gitignore all storage-* Crawlee dirs + local credentials 2026-04-18 02:40:34 +02:00
Rene Fichtmueller
c5e7e7d7f6 fix: remove POSTGRES_PASSWORD export from run-fs-scraper-mac.sh — sourced from ~/.tip/.env only 2026-04-18 02:37:42 +02:00
Rene Fichtmueller
fc3224be76 fix: remove hardcoded POSTGRES_PASSWORD from run-fs-scraper-mac.sh — use ~/.tip/.env 2026-04-18 02:37:05 +02:00
Rene Fichtmueller
696127366c chore: changelog — Playwright headless shell fix, withIsolatedStorage race fix, FS.com launchd fix 2026-04-18 02:35:55 +02:00
Rene Fichtmueller
419af4a24e fix: remove all withIsolatedStorage wrappers, add makeCrawleeConfig to remaining Crawlee scrapers 
- scheduler.ts: remove withIsolatedStorage from ALL scrapers (atgbics,
  optcore, ufispace, edgecore, ebay-*, market-intel, community-issues,
  cisco, juniper, sonic, 10gtek, prolabs, switch-assets, fs)
  eliminates global CRAWLEE_STORAGE_DIR race condition entirely
- fs-com.ts: replace purgeDefaultStorages() with rmSync on isolated
  storage dirs (fs-phase1, fs-phase2); pass makeCrawleeConfig to both
  PlaywrightCrawler instances
- switch-assets-crawler.ts: add makeCrawleeConfig('switch-assets')
- switch-assets-playwright.ts: add makeCrawleeConfig('switch-assets-playwright')
- naddod.ts: restore clean error logging (remove debug instrumentation)
 2026-04-18 02:19:53 +02:00
Rene Fichtmueller
d9e5331161 debug: widen NADDOD error slice to 300 chars, add pre-insert logging 2026-04-18 02:00:03 +02:00
Rene Fichtmueller
24481b09e6 fix: eBay enricher Crawlee isolation + ephemeral queues 
- Add makeCrawleeConfig isolation to CheerioCrawler instances
- Switch from named persistent RequestQueue to ephemeral null queues:
  named queues retain 'handled' state and skip all URLs on re-runs,
  causing 0 observations on every run after the first.
- Applies to both enrichSwitchFromEbay and enrichTransceiversFromEbay.
 2026-04-18 01:42:08 +02:00
Rene Fichtmueller
35a02057f4 chore: sync-to-erik now includes scraper rebuild + daemon restart; changelog 2026-04-18 01:37:12 +02:00
Rene Fichtmueller
c7d7456de9 fix: instance-level Crawlee storage isolation + eBay vendor type 
- Add utils/crawlee-config.ts: makeCrawleeConfig(name) returns a
  Crawlee Configuration with isolated localDataDirectory per scraper.
  Uses storageClientOptions (not global CRAWLEE_STORAGE_DIR) so
  concurrent pg-boss workers in the same process don't race on
  the shared env var.

- Apply makeCrawleeConfig to all 6 Crawlee-based scrapers:
  optcore (PlaywrightCrawler), atgbics (PlaywrightCrawler),
  community-issues (CheerioCrawler + RequestQueue),
  edgecore (CheerioCrawler), ufispace (CheerioCrawler),
  market-intelligence (CheerioCrawler).

- scheduler.ts: add withIsolatedStorage for optcore and market-intel
  workers (was missing, caused storage-fs path bleed from fs scraper).

- ebay-enricher.ts: fix vendor type 'marketplace' -> 'reseller' to
  satisfy vendors_type_check constraint
  ['manufacturer','distributor','oem','reseller','compatible'].
 2026-04-18 01:35:57 +02:00
Rene Fichtmueller
4b751a771b fix: NADDOD stockLevel 'unknown' → 'on_request' — invalid value for price_observations check constraint 2026-04-18 01:21:31 +02:00
Rene Fichtmueller
2b770aa1a9 chore: cleanup — rename digikey→mouser, remove orphan files, gitignore Crawlee artifacts 
- Rename scrapers/digikey.ts → scrapers/mouser.ts: export scrapeMouser()
  (file was Mouser API implementation mislabeled from task origin)
- Fix scheduler.ts mouser-oem worker: import scrapeMouser from ./scrapers/mouser
- Delete switch-seed-smb.ts (unreferenced, no CLI flag, no scheduler job)
- Add storage/, storage-fs/, .crawlee/ to .gitignore (Crawlee runtime artifacts)
 2026-04-18 01:09:10 +02:00
Rene Fichtmueller
1c8dec52c9 feat: Price Comparison dashboard + Eoptolink OEM scraper 
- Add public /api/price-comparison API (summary, top-50, per-SKU detail)
  — no auth required, 3 Express routes, DISTINCT ON latest-price logic
- Add '💲 Price Comparison' dashboard tab: stat cards, form-factor
  breakdown, top-50 SKU table (clickable rows → SKU detail), per-vendor
  price + stock + spread% lookup panel
- Add Eoptolink OEM catalog scraper (93 product-solution pages,
  part-number regex EOLO-*/EOLQ-* etc., no prices, seeds transceivers
  table as manufacturer entries)
- Register scrape:catalog:eoptolink in scheduler: schedule every 4h
  (40 */4 * * *), lazy-import worker, added to known-jobs array
 2026-04-18 01:02:08 +02:00
Rene Fichtmueller
63b4a65c28 chore: changelog entry — MCP Server v0.2.0 (finder + switch-docs + Ollama LLM tools) 2026-04-18 00:29:42 +02:00
Rene Fichtmueller
e9fcda2811 feat: wire finder.ts + switch-docs + Ollama LLM tools to MCP server 
MCP Server (packages/mcp-server/src/index.ts):
- Register registerSwitchDocTools (switch-docs.ts) — switch documentation lookup
- Register finderTools dynamically (finder.ts) — find_flexoptix_for_switch, get_competitor_alerts
- Add analyze_market_with_llm tool: qwen2.5:14b via Ollama, enriched with live hype cycle + pricing + news
- Add generate_blog_post tool: fo-blog-v5 (fine-tuned) with qwen2.5:14b fallback, enriched with live pricing data
- OLLAMA_BASE_URL env var (default: https://ollama.fichtmueller.org)

Also includes scraper improvements (ascentoptics, atgbics, gbics, skylane, ebay-enricher),
API route updates (blog, blog-sll, health, hot-topics, transceivers, queries),
and dashboard hot-topics refresh.
 2026-04-18 00:21:58 +02:00
Rene Fichtmueller
b88a6e28cf feat: /api/hype-cycle/analysis endpoint — DB-backed Bass-fitted results from hype_cycle_analysis table 2026-04-18 00:11:08 +02:00
Rene Fichtmueller
9d3019d0c0 feat: Norton-Bass Hype Cycle Engine — market_metrics seed + Bass fitting + Gartner phase detection 2026-04-18 00:09:08 +02:00
Rene Fichtmueller
75cea9fe90 feat: Mouser Electronics API scraper for OEM reference prices (Juniper/Cisco/Arista PIDs) 2026-04-18 00:04:35 +02:00
Rene Fichtmueller
5c5841d4d1 chore: prune 242 irrelevant vendors from DB (348→106), add changelog entry 2026-04-17 23:58:13 +02:00
Rene Fichtmueller
60736bd5df chore: changelog entries 2026-04-17 stock dashboard + Cisco TMG + Juniper HCT 2026-04-17 23:33:49 +02:00
Rene Fichtmueller
861243ea3f feat: stock confidence badges, multi-vendor price comparison, expanded Cisco TMG + Juniper HCT 
Stock API & Dashboard:
- /api/stock/summary: vendor_breakdown adds avg_confidence, currencies, conf_per_warehouse/aggregated/boolean
- /api/stock/summary: new price_comparison endpoint (multi-vendor SKUs, min/max/avg price)
- /api/stock/summary: totals adds multi_vendor_skus count
- Dashboard: 6th stat card (Multi-Vendor SKUs), confidence badge column (🟢 L3 / 🟡 L2 / ⚪ L1)
- Dashboard: price comparison table with vendor-by-vendor price breakdown
- Dashboard: subtitle updated to include QSFPTEK + NADDOD
- Dashboard: top sellers link to product URLs

Cisco TMG improvements:
- Added 5 new platform families: 8000 Series, NCS5500, NCS540, NCS560, NCS1000
- Per-device query strategy: iterates all switch model IDs from family filter
  instead of getting only 1 switch per family → 58 switches per N9300 run
- Graceful error handling per device with rate limiting (1s between requests)

Juniper HCT: ran manually → 475 Juniper-brand transceivers seeded
 2026-04-17 23:33:31 +02:00
Rene Fichtmueller
5393f73c17 feat: stock quality schema + QSFPTEK/NADDOD v2 scrapers with real-time stock counts 
- Migration 028 (retroactive): document warehouse columns added to stock_observations
- Migration 037: composite indexes for DISTINCT ON (transceiver_id, source_vendor_id) queries
- Migration 038: add stock_confidence (1/2/3), price_currency, price_includes_tax,
  stock_vendor_ts to stock_observations + TRUNCATE test-run data

db.ts: upsertStockObservation now accepts stockConfidence, priceCurrency,
priceIncludesTax, stockVendorTs; delta detection includes quantity_available

fs-com.ts: passes stockConfidence=3 + priceCurrency=EUR + priceIncludesTax=false

qsfptek.ts v2: Phase 1 API listing + Phase 2 detail-page stock extraction
- Parses 'X in real-time stock, DATE' from product detail pages
- Writes stock_observations with confidence=2 + stockVendorTs
- Up to 500 detail pages/run at 2s rate limit

naddod.ts v2: complete rewrite from WooCommerce to Astro sitemap-based
- Discovers products via /sitemaps/products.xml (600+ products)
- URL format: /products/XXXXX.html
- Extracts 'In Stock: X' exact counts from SSR HTML
- Writes both price + stock observations (confidence 1 or 2)
 2026-04-17 22:54:40 +02:00
Rene Fichtmueller
5b35b2b8be feat(scraper+api): warehouse stock data pipeline — FS.com v2, SmartOptics v2, Stock API 
Scraper changes:
- fs-com.ts v2: Playwright stealth patches + www.fs.com/de/ URL fix (de.fs.com DNS NXDOMAIN).
  Extracts DE-Lager, Global-Lager, Nachlieferung, units_sold, compatible_brands, price_net.
  Mac-side runner (run-fs-scraper-mac.sh) via SSH tunnel for residential IP access.
  Fast-fail connectivity check on datacenter IPs that are blocked by Cloudflare.
- smartoptics.ts v2: WooCommerce REST API fallback + 8 catalog categories + relative URL fix.
  Was finding only 8 products, now discovers 18+ with multi-category crawl.

DB layer:
- db.ts: add upsertStockObservation() — writes 10 new stock_observations columns
  (warehouse_de_qty, warehouse_global_qty, backorder_qty, units_sold, compatible_brands,
  price_net, product_url, delivery dates) with dedup check.

API:
- routes/stock.ts: GET /api/stock, /api/stock/summary, /api/stock/:id
  Warehouse breakdowns per transceiver/vendor with top-sellers and vendor summary.
- routes/review.ts: equivalence review queue (approve/reject/bulk-approve).
- index.ts: register /api/stock and /api/review routes.

Dashboard:
- index.html: 🏭 Stock tab with stat cards (DE-Lager, Global-Lager, Nachlieferung totals),
  top-sellers table, vendor breakdown, recently-restocked events, part-number lookup.

SQL migrations:
- 034: blog-review-tag, 035: price-observations is_anomalous, 036: transceiver-equivalences.
 2026-04-17 10:45:59 +02:00
Rene Fichtmueller
662cd1f90b fix(scraper): FiberMall URL schema + price parser + Flexoptix EUR comma bug 
FiberMall:
- Correct /store-XXXXX-name.htm category URLs (was /c/xxx/ → HTTP 404)
- Parser: split on new_proList_mainListLi, price from data-price on
  currency_price span — fix 0.00 false-match from SKU variant items
- Also scrape SKU brand variant links from .sku_item divs
- Result: 3,410 prices now in DB (was 0)

Flexoptix:
- Fix extractPrice regex for EUR thousand-separator format
  (2,921.60 EUR was parsed as 2 EUR)
- Add OSFP224 / 1.6T search queries (4 new, form factor was missing)
- Fix O.138HG2.C.05 stale price 3009.60→2921.60 EUR

Schema: competitor_verified + competitor_verified_at columns
added via ALTER TABLE (were referenced in code but missing in DB)

CHANGELOG: added 6 entries for 2026-04-12
 2026-04-12 04:26:35 +02:00
Rene Fichtmueller
cdb8ef6e61 feat(scraper): add FiberMall/Vcelink/OpticsBay scrapers, fix QSFPTEK API migration 
- New scrapers: fibermall.ts (WooCommerce), vcelink.ts (Shopify), opticsbay.ts (WooCommerce)
- QSFPTEK rewritten to use /mall/commodity/list API (old OpenCart /c/*.html paths gone 404)
  - New: attribute-based filtering by data rate (1G/10G/25G/40G/100G/200G/400G/800G)
  - Scrapes HTML fragments, extracts US$ prices and product URLs
- scheduler.ts: +3 queues/schedules/workers (fibermall, vcelink, opticsbay) → 61 total workers
- index-pi.ts: Pi fleet picks up all 3 new scrapers
 2026-04-11 19:13:36 +02:00
Rene Fichtmueller
8905382a49 chore: add changelog entries 2026-04-05 through 2026-04-09 2026-04-11 18:50:17 +02:00
Rene Fichtmueller
ef30da18d1 chore: update changelog for 2026-04-11 session 2026-04-11 13:57:31 +02:00
Rene Fichtmueller
148d2e1000 fix(scraper): set CRAWLEE_PURGE_ON_START=1 in withIsolatedStorage 
Crawlee's SessionPool throws 'Could not find SDK_SESSION_POOL_STATE.json'
when initializing against a freshly-created isolated storage dir.
Setting CRAWLEE_PURGE_ON_START=1 tells Crawlee to start fresh instead
of trying to load non-existent session state — fixes FS.com and ATGBICS
crashes at the start of every 2h cycle after the dirs were cleaned up.
 2026-04-11 07:27:24 +02:00
Rene Fichtmueller
45c48755e4 feat(scraper): add NADDOD/QSFPTEK/AddOn to scheduler, fix pre-existing TS build errors 
- Register scrape:pricing:naddod (48 */2), qsfptek (52 */2), addon (55 */2) in pg-boss
- Add boss.work() handlers for all three (fetch-based, run on Erik)
- Fix findOrCreateScrapedTransceiver callers: remove invalid `name`/`url` params,
  fix `t.id` → `t` (function already returns string ID)
- Fix ebay-enricher: remove invalid `extractType` option, use extraction.standard_name
  instead of non-existent `.description`, fix cheerio type incompatibility
- Fix community-issues: description → summary, publishedDate → published_at
- Startup zombie cleanup already deployed (index.ts) — no changes needed
- ProLabs rewritten to fetch-based catalog scraper (no Playwright, bypasses WAF)
 2026-04-11 03:17:33 +02:00
Rene Fichtmueller
6febb9c88e refactor(prolabs): replace Playwright+Firefox with fetch-based catalog scraper 
ProLabs uses B2B quote model - prices require reseller account and are
not shown publicly (schema.org always shows price=0.00). Fighting
CloudFront WAF with Firefox automation is pointless.

New approach:
- Sitemap-driven: downloads all 14 sitemaps to collect product URLs
- fetch-based: curl-compatible HTTP requests bypass CloudFront TLS detection
- catalog-only: writes part numbers + specs to transceivers table
- Rate-limited: 300ms between requests (~3 req/sec)
- No proxy needed: Pi nodes no longer consumed for ProLabs
 2026-04-11 02:57:13 +02:00
Rene Fichtmueller
7af5b32b3f ui: redesign LLM panel for light theme readability 
Replace hard-coded purple/green colors with theme CSS variables.
Dark code blocks (#1e1e1e bg), orange accent for active borders/badges,
dark green for status text, amber for warnings — all readable on white.
 2026-04-09 21:20:43 +02:00
Rene Fichtmueller
bf626f9de6 fix: route Pi-destined scrapers exclusively to Pi worker fleet 
Remove boss.work() registrations for lightweight fetch/cheerio scrapers
from Erik's scheduler. Pis are now the SOLE consumers of these queues:
fluxlight, gbics, optcore, champion-one, sfpcables, blueoptics, fiber24,
tscom, skylane, ascentoptics, gaotek, smartoptics, hubersuhner, news,
market-intel.
 2026-04-09 20:50:57 +02:00
Rene Fichtmueller
c898f52bbe feat: add LLM model selector panel to Blog Engine tab 
Shows active model (fo-blog-v3-qwen7b / claude-sonnet-4-6 / qwen2.5:14b),
live status from /api/blog/llm/status, ratings, config instructions,
and highlights which model is currently active.
 2026-04-09 20:42:03 +02:00
Rene Fichtmueller
3d00a4a00a feat: 800G standards deep enrichment + Pi Starlink proxy-agent support 
- Migration 033: comprehensive technical update for all 4x 800G standards
  - 800GBASE-SR8: full optical specs (Tx OMA 2.3 dBm, Rx sens. -4.6 dBm, KP4 FEC,
    MPO-16 APC, CMIS 5.2, ≤16W, 60m OM3/100m OM4, VCSEL 850nm, 53.125 GBd PAM4)
  - 800GBASE-DR8: 500m SMF, EML 1310nm, 8x parallel fiber, MPO-12, -9dBm sensitivity
  - 800GBASE-LR4: 2km CWDM4 WDM (1270/1290/1310/1330nm), 4x 106.25 GBd PAM4, LC duplex
  - 800G-ZR (OIF-800ZR-01.0): DP-16QAM 96 GBd, 1000km EDFA, SD-FEC, 20-24W, DCO license
- Pi scraper: add optional SOCKS5 proxy via dante-server on WireGuard IP
  - Enables Starlink bandwidth contribution (PROXY_AGENT=1 flag)
  - Scraper routes selected jobs through Pi SOCKS5 for different IP range
 2026-04-09 20:34:27 +02:00
Rene Fichtmueller
cddc92c9d2 feat: TIP audit fixes — Qdrant init, switches columns, verification fix, crawler live status, demo data badges 
- Migration 032: add system_type, is_linecard, chassis_model, slot_type, flexbox_* to switches table
- Migration 032: fix compute_transceiver_verification() to count seed data as details_verified (100% now)
- Migration 032: add is_demo_data flag to reorder_signals, abc_classification, market_intelligence, stock_snapshots
- Cisco 8000: insert 8812, 8818, 8800-LC-36FH, 8800-LC-48H with correct vendor slug 'cisco'
- API: add /api/scrapers/jobs endpoint exposing pg-boss job queue (active/recent/queues)
- Dashboard: live job queue panel in Crawler Intelligence tab (active jobs + recent 4h completions)
- Dashboard: DEMO DATA badge now uses is_demo_data column (was checking wrong field is_demo)
- Blog engine: configured fo-blog-v3-qwen7b fine-tuned model via tip-api ecosystem.config.js
- Qdrant: all 6 collections created, seeded (2135 products, 29 FAQs, 39 news, 20 troubleshooting)
 2026-04-09 20:29:46 +02:00
Rene Fichtmueller
7d005ba1f3 data: add Cisco 8000 accuracy migration (031) 
Documents all Cisco 8000 research findings as idempotent SQL migration:
- 8201-32FH: Silicon One Q200 (not Q100), full description, features, use_cases
- 8608: modular chassis type, corrected description
- NEW: 8812 (12-slot), 8818 (18-slot) modular chassis
- NEW: 8800-LC-36FH (36×400G QSFP-DD), 8800-LC-48H (48×100G QSFP28) line cards

flexbox_notes covers:
- IOS XR EEPROM/PID check behavior (TX laser disabled on unknown PID)
- transceiver permit pid all (per-interface, NOT global)
- DCO license required per 100G for ZR/ZR+ coherent optics (all vendors)
- No RTU license for grey non-coherent optics
- TAC refusal + PFM alarm on override
- IOS XR upgrade risk for 3rd-party optics
- DOM monitoring may show incorrect values
- Flexoptix FlexBox Cisco Systems mode = native PID, no override needed
 2026-04-09 09:09:23 +02:00
Rene Fichtmueller
cf75eee8ad feat: linecard system support, Cisco 8000 accuracy, price anomaly detection 
API/finder:
- Add modular chassis support: sibling linecards fetched when is_linecard=true
- Add chassis linecards when system_type=modular
- Extend switch response: system_type, is_linecard, chassis_model, slot_type,
  flexbox_compat_mode, flexbox_notes, description, switching_capacity_tbps,
  total_ports, category, lifecycle_status, features, use_cases, linecards[]

API/transceivers:
- Filter price_observations with COALESCE(is_anomalous, false) = false
  (direct prices + comparable market prices)

Scraper/db:
- Add PRICE_BOUNDS map (per form-factor min/max USD sanity bounds)
- Add isPriceAnomalous() — marks DB price_observations as is_anomalous=true
- Add competitor_verified flag: set true when valid competitor price stored
- upsertPriceObservation: skip prices outside sanity bounds, set competitor_verified

Scraper/hash:
- contentHash() now accepts Record<string,unknown> | string (union type)
  to support both structured objects and legacy string callers

Scrapers (skylane, tscom, wiitek):
- Fix contentHash() call signature: pass objects not JSON.stringify strings
- Fix wiitek: remove invalid 'name' param, fix t.id → transceiverId

Migrations:
- Add is_anomalous, competitor_verified, competitor_verified_at,
  image_primary columns
- Recreate sync_fully_verified trigger to include competitor_verified
- Add is_linecard, chassis_model, system_type, slot_type,
  flexbox_compat_mode, flexbox_notes to switches table
 2026-04-09 09:06:22 +02:00
Rene Fichtmueller
240e7f46f2 feat(scraper): add SOCKS5 proxy rotation for fs-com, atgbics, gbics scrapers 
Routes requests through CT130/131/132 proxy pool (192.168.178.77/76/74:1080)
when PROXY_URLS env var is set. Uses ProxyConfiguration from crawlee for
PlaywrightCrawler scrapers and socks-proxy-agent for fetch-based scrapers.
 2026-04-08 08:17:49 +02:00
Rene Fichtmueller
772ce2074d feat: add blog training articles 056-100 for fo-blog-v3 fine-tuning 
45 expert articles covering: Cisco/Juniper/Arista optic compatibility mechanics,
100G/400G/800G optics selection, DWDM/ROADM/WSS architecture, fiber standards,
coherent pluggables, AI cluster optics, carrier timing, EEPROM programming,
market pricing 2026, hyperscale procurement, transceiver failure analysis, and more.
 2026-04-07 08:59:16 +02:00
Rene Fichtmueller
12a19639fd fix: seed script accepts category as fallback for missing type field 2026-04-07 01:24:15 +02:00
Rene Fichtmueller
0572ab5a71 feat: add blog training articles 041-055 for fo-blog-v2 fine-tuning 
15 expert articles covering: CPO/silicon photonics 2026, 800G OSFP vs QSFP-DD,
400ZR/OpenZR+/ZR+ comparison, laser safety, OSNR/link budget, counterfeit detection,
DOM deep dive, 400G DR4/FR4/LR4, WDM primer, temp grades, spine-leaf strategy,
proactive replacement, OEM lock-in, OM3/4/5, lifecycle management.
 2026-04-07 01:08:27 +02:00
Rene Fichtmueller
99fca6b531 feat(training): add blog-031 through blog-040 — 10 expert articles 
Topics: CWDM4/PSM4, MSA compliance, DAC/AOC TCO, grey vs DWDM,
ESD damage, tunable DWDM, FEC deep-dive, CPO hype cycle,
CMIS 4.0, vendor evaluation. Ø 1,180 words each.
 2026-04-06 18:15:46 +02:00
Rene Fichtmueller
51af249361 Merge remote-tracking branch 'github/main' 
# Conflicts:
#	packages/api/src/llm/fo-blog-pipeline.ts
#	packages/api/src/routes/blog.ts
#	packages/scraper/src/scheduler.ts
#	packages/scraper/src/scrapers/fs-com.ts
#	packages/scraper/src/scrapers/gbics.ts
 2026-04-06 18:03:36 +02:00
Rene Fichtmueller
285a91b945 feat(training): add blog-016 through blog-030 — 15 expert training articles 
Adds 15 Sonnet-quality blog articles for fo-blog-v1 fine-tuning:
tutorials, comparisons, tech deep-dives covering 400G/800G topics.
Also adds seed-blog-training-data.py script for learning_corpus import.
 2026-04-06 17:59:14 +02:00
Rene Fichtmueller
bf06993b63 ui: show creation time (HH:MM) alongside date in blog list 2026-04-06 17:07:05 +02:00
Rene Fichtmueller
3928755c60 fix: correct verified badge, comparable pricing, and clickable product images 
- Reset details_verified=false for 298 products where reach_label is empty (DB migration)
- Runtime check in dashboard: dVer requires non-empty reach_label regardless of DB flag
- comparable price query: treat reach_meters=0 same as NULL so 800G OSFP products
  find FS.com equivalent prices (was blocked by reach_meters=0 != NULL shortcircuit)
- Product image area now fully clickable with vendor link overlay when product_page_url exists
- Clear wrong image for O.Czz8HG.z.R (was showing unrelated OSFP product image)
 2026-04-06 10:24:39 +02:00
Rene Fichtmueller
8f060d0159 feat(training): add blog-014 new_product and blog-015 competitor_analysis 
Completes training data coverage for all 8 blog types:
market_alert(2), comparison(1), technology_deep_dive(4), tutorial(3),
hype_cycle(1), buying_guide(1), migration_guide(1), new_product(1),
competitor_analysis(1) — 15 gold-standard articles total
 2026-04-06 04:16:00 +02:00
Rene Fichtmueller
4acf293690 fix(llm): checkHealth uses key presence check, not live API call 
Live Anthropic API call during health check causes 429 when the pipeline
is actively running, blocking all subsequent regenerate requests.
 2026-04-06 04:07:21 +02:00
Rene Fichtmueller
f7bdee9583 feat: add 2 more gold-standard blog training articles (13 total) 
- blog-012: technology_deep_dive — coherent vs direct-detect decision framework
- blog-013: market_alert — transceiver price cycle, when to buy

Training set now covers: market_alert(2), comparison(1), technology_deep_dive(4),
tutorial(3), hype_cycle(1), buying_guide(1), migration_guide(1) — 13 total
 2026-04-06 03:09:55 +02:00
Rene Fichtmueller
de05bbbec8 docs: update training data README to reflect 11 articles 2026-04-06 02:55:34 +02:00