Rene Fichtmueller
c7697308f6
feat: NOG conference talks scraper + hot topics integration
...
NOG Talks Scraper (packages/scraper/src/scrapers/nog-talks.ts):
- Crawls DENOG (15-17), NANOG (91-93), RIPE (87-89), ENOG, NLNOG, Euro-IX
- Relevance scoring: optical keywords (+3pts each), network keywords (+1pt)
Only talks scoring ≥2 stored, high-relevance (≥6) also to market_intelligence
- CtxEvent cross-DB bridge: when ctxmeet DB has ConferenceTalk rows,
pulls directly via dblink (same Postgres instance, no network hop)
- Runs weekly Monday 06:00 UTC (pg-boss schedule)
- Output: news_articles (source='NOG Talks: EVENT') + market_intelligence
Hot Topics (packages/api/src/routes/hot-topics.ts):
- SOURCE 3c: NOG talk clusters displayed as conference topics in hot list
Grouped by event (DENOG15, NANOG93...) with speaker + abstract preview
Filtered: source LIKE 'NOG Talks:%' AND relevance > 0.4 AND < 6 months
- Limit raised to 20 topics (was 15)
- Added nog_talks to sources metadata
Scheduler & Pi fleet:
- scrape:nog-talks queue registered in scheduler.ts + index-pi.ts
- Weekly cron: Monday 06:00 UTC (every Pi can handle it independently)
- First job triggered immediately
2026-04-02 22:38:00 +02:00
Rene Fichtmueller
6ccaa03932
feat: add index-pi.ts with all 44 workers for Pi fleet scraper nodes
...
Complete Pi scraper entry point covering all pricing, catalog, compat,
intelligence and prediction signal scrapers. Includes 5 new form-factor
coverage scrapers (comms-express, router-switch, multimode-inc,
optictransceiver, wiitek). Erik runs only API+DB, all scraping on Pis.
2026-04-02 09:34:05 +02:00
