5 Commits

Author SHA1 Message Date
Rene Fichtmueller
ab059c2fd1 fix(community-issues): scrapeTransceiverCompatIssues falls back to ports_config when no compat entries 2026-04-20 23:00:00 +02:00
Rene Fichtmueller
4bf5c95824 feat: Flexoptix compatibility scraper + transceiver issue scanner
- Add flexoptix-compat.ts: maps switch models to compatible Flexoptix transceivers
  via search API (vendor_compat) with form-factor fallback (spec_match)
  Scheduled daily at 09:00 UTC as scrape:compat:flexoptix
- Enhance community-issues.ts: add vendor advisory sources (Cisco Field Notices,
  Juniper KB, SONiC GitHub Issues) + new scrapeTransceiverCompatIssues() that
  searches for switch+transceiver combination problems specifically
- Scheduler: 59 schedules, 78 workers
2026-04-20 22:50:57 +02:00
Rene Fichtmueller
c7d7456de9 fix: instance-level Crawlee storage isolation + eBay vendor type
- Add utils/crawlee-config.ts: makeCrawleeConfig(name) returns a
  Crawlee Configuration with isolated localDataDirectory per scraper.
  Uses storageClientOptions (not global CRAWLEE_STORAGE_DIR) so
  concurrent pg-boss workers in the same process don't race on
  the shared env var.

- Apply makeCrawleeConfig to all 6 Crawlee-based scrapers:
  optcore (PlaywrightCrawler), atgbics (PlaywrightCrawler),
  community-issues (CheerioCrawler + RequestQueue),
  edgecore (CheerioCrawler), ufispace (CheerioCrawler),
  market-intelligence (CheerioCrawler).

- scheduler.ts: add withIsolatedStorage for optcore and market-intel
  workers (was missing, caused storage-fs path bleed from fs scraper).

- ebay-enricher.ts: fix vendor type 'marketplace' -> 'reseller' to
  satisfy vendors_type_check constraint
  ['manufacturer','distributor','oem','reseller','compatible'].
2026-04-18 01:35:57 +02:00
Rene Fichtmueller
45c48755e4 feat(scraper): add NADDOD/QSFPTEK/AddOn to scheduler, fix pre-existing TS build errors
- Register scrape:pricing:naddod (48 */2), qsfptek (52 */2), addon (55 */2) in pg-boss
- Add boss.work() handlers for all three (fetch-based, run on Erik)
- Fix findOrCreateScrapedTransceiver callers: remove invalid `name`/`url` params,
  fix `t.id` → `t` (function already returns string ID)
- Fix ebay-enricher: remove invalid `extractType` option, use extraction.standard_name
  instead of non-existent `.description`, fix cheerio type incompatibility
- Fix community-issues: description → summary, publishedDate → published_at
- Startup zombie cleanup already deployed (index.ts) — no changes needed
- ProLabs rewritten to fetch-based catalog scraper (no Playwright, bypasses WAF)
2026-04-11 03:17:33 +02:00
Rene Fichtmueller
4020ec77d9 feat: product intelligence layer — eBay enricher, community issues, datasheets+manuals API
- Migration 020: product_issues table, condition/marketplace on price_observations, features JSONB
- eBay enricher: switch features/description/refurb prices + transceiver condition pricing
- Community issues scraper: Reddit/ServeTheHome/Arista/Cisco community bug reports
- 7 pre-seeded issues (DCS-7800R3, SG350, QFX5120, CRS326, USW-Pro etc.)
- API: /switches/:id/issues + /switches/:id/documents endpoints
- Dashboard switch modal: features from DB, description, eBay refurb price, issues+docs async
- Datasheet finder for Arista/Cisco/Juniper/HPE vendor pages
- Scheduler: 4 new jobs (ebay enrichment nightly, community issues weekly)
2026-04-01 22:46:27 +02:00