2 Commits

Author	SHA1	Message	Date
Rene Fichtmueller	1026787318	feat: add proxy network, image backfill, and scraper improvements ... - Add TIP Proxy Network (packages/proxy-agent): SOCKS5 proxy agent for residential IP bypass of CloudFront WAF blocks - Add /api/proxy/* routes: node registration, heartbeat, load balancing - Add image extraction to Flexoptix catalog scraper (GraphQL small_image) - Add image extraction to Optcore scraper (Playwright gallery img) - Fix Fluxlight price scraping (BigCommerce HTML structure: data-product-price-without-tax) - Add SmartOptics scraper (8 DWDM/coherent products, og:image extraction) - Fix findOrCreateScrapedTransceiver to update image_url for existing records - Add image backfill script (backfill-images.ts): 178 Flexoptix images added - Fix DB connection pool: max 5, idleTimeoutMillis 10s (was unlimited, caused >100 connections) - Add proxy.ts utility for scraper proxy rotation	2026-04-03 21:13:03 +02:00
Rene Fichtmueller	072978f1a4	feat: 24/7 scraping fleet — 8 new vendors + continuous schedule + Pi setup ... New scrapers (8): - BlueOptics (EUR, every 4h) - ShopFiber24 (EUR, every 4h) - T&S Communication (USD, every 4h) - SmartOptics (catalog, every 8h) - HUBER+SUHNER (catalog, every 8h) - Skylane Optics (USD, every 4h) - AscentOptics (USD, every 4h) - GAO Tek (USD, every 4h) Scheduler: nightly window → 24/7 continuous (42 jobs total) - Playwright scrapers: every 8h (FS.com, 10Gtek, ATGBICS, ProLabs) - Fetch/Cheerio: every 4h (11 lightweight vendors) - Flexoptix catalog: every 2h (primary price source) - eBay enrichment: every 6h - Compatibility matrices: every 12h - Compute jobs: every 4h Pi fleet: scripts/pi-scraper-setup.sh for one-command Pi node setup	2026-04-02 01:09:05 +02:00