Rene Fichtmueller
|
ae0bda9e06
|
feat: proxy network — geo-lookup, uptime tracking, dedup fix
- IP geo-lookup via ip-api.com on register/heartbeat (country_code, city)
- heartbeat_count column + uptime_pct computation on every heartbeat
- Deduplication: register returns existing token for same IP+port
- Heartbeat no longer overwrites registered IP (prevents IPv6 churn conflicts)
- Migration 023: heartbeat_count column + backfill existing nodes
|
2026-04-04 08:15:32 +02:00 |
|
Rene Fichtmueller
|
1026787318
|
feat: add proxy network, image backfill, and scraper improvements
- Add TIP Proxy Network (packages/proxy-agent): SOCKS5 proxy agent
for residential IP bypass of CloudFront WAF blocks
- Add /api/proxy/* routes: node registration, heartbeat, load balancing
- Add image extraction to Flexoptix catalog scraper (GraphQL small_image)
- Add image extraction to Optcore scraper (Playwright gallery img)
- Fix Fluxlight price scraping (BigCommerce HTML structure: data-product-price-without-tax)
- Add SmartOptics scraper (8 DWDM/coherent products, og:image extraction)
- Fix findOrCreateScrapedTransceiver to update image_url for existing records
- Add image backfill script (backfill-images.ts): 178 Flexoptix images added
- Fix DB connection pool: max 5, idleTimeoutMillis 10s (was unlimited, caused >100 connections)
- Add proxy.ts utility for scraper proxy rotation
|
2026-04-03 21:13:03 +02:00 |
|