Rene Fichtmueller
f8809d999f
feat(scraper+api): warehouse stock data pipeline — FS.com v2, SmartOptics v2, Stock API
...
Scraper changes:
- fs-com.ts v2: Playwright stealth patches + www.fs.com/de/ URL fix (de.fs.com DNS NXDOMAIN).
Extracts DE-Lager, Global-Lager, Nachlieferung, units_sold, compatible_brands, price_net.
Mac-side runner (run-fs-scraper-mac.sh) via SSH tunnel for residential IP access.
Fast-fail connectivity check on datacenter IPs that are blocked by Cloudflare.
- smartoptics.ts v2: WooCommerce REST API fallback + 8 catalog categories + relative URL fix.
Was finding only 8 products, now discovers 18+ with multi-category crawl.
DB layer:
- db.ts: add upsertStockObservation() — writes 10 new stock_observations columns
(warehouse_de_qty, warehouse_global_qty, backorder_qty, units_sold, compatible_brands,
price_net, product_url, delivery dates) with dedup check.
API:
- routes/stock.ts: GET /api/stock, /api/stock/summary, /api/stock/:id
Warehouse breakdowns per transceiver/vendor with top-sellers and vendor summary.
- routes/review.ts: equivalence review queue (approve/reject/bulk-approve).
- index.ts: register /api/stock and /api/review routes.
Dashboard:
- index.html: 🏭 Stock tab with stat cards (DE-Lager, Global-Lager, Nachlieferung totals),
top-sellers table, vendor breakdown, recently-restocked events, part-number lookup.
SQL migrations:
- 034: blog-review-tag, 035: price-observations is_anomalous, 036: transceiver-equivalences.
2026-04-17 10:45:59 +02:00
Rene Fichtmueller
edc9311d7b
feat: add proxy network, image backfill, and scraper improvements
...
- Add TIP Proxy Network (packages/proxy-agent): SOCKS5 proxy agent
for residential IP bypass of CloudFront WAF blocks
- Add /api/proxy/* routes: node registration, heartbeat, load balancing
- Add image extraction to Flexoptix catalog scraper (GraphQL small_image)
- Add image extraction to Optcore scraper (Playwright gallery img)
- Fix Fluxlight price scraping (BigCommerce HTML structure: data-product-price-without-tax)
- Add SmartOptics scraper (8 DWDM/coherent products, og:image extraction)
- Fix findOrCreateScrapedTransceiver to update image_url for existing records
- Add image backfill script (backfill-images.ts): 178 Flexoptix images added
- Fix DB connection pool: max 5, idleTimeoutMillis 10s (was unlimited, caused >100 connections)
- Add proxy.ts utility for scraper proxy rotation
2026-04-03 21:13:03 +02:00
Rene Fichtmueller
53a135310e
feat: 24/7 scraping fleet — 8 new vendors + continuous schedule + Pi setup
...
New scrapers (8):
- BlueOptics (EUR, every 4h)
- ShopFiber24 (EUR, every 4h)
- T&S Communication (USD, every 4h)
- SmartOptics (catalog, every 8h)
- HUBER+SUHNER (catalog, every 8h)
- Skylane Optics (USD, every 4h)
- AscentOptics (USD, every 4h)
- GAO Tek (USD, every 4h)
Scheduler: nightly window → 24/7 continuous (42 jobs total)
- Playwright scrapers: every 8h (FS.com, 10Gtek, ATGBICS, ProLabs)
- Fetch/Cheerio: every 4h (11 lightweight vendors)
- Flexoptix catalog: every 2h (primary price source)
- eBay enrichment: every 6h
- Compatibility matrices: every 12h
- Compute jobs: every 4h
Pi fleet: scripts/pi-scraper-setup.sh for one-command Pi node setup
2026-04-02 01:09:05 +02:00
