Rene Fichtmueller
5393f73c17
feat: stock quality schema + QSFPTEK/NADDOD v2 scrapers with real-time stock counts
...
- Migration 028 (retroactive): document warehouse columns added to stock_observations
- Migration 037: composite indexes for DISTINCT ON (transceiver_id, source_vendor_id) queries
- Migration 038: add stock_confidence (1/2/3), price_currency, price_includes_tax,
stock_vendor_ts to stock_observations + TRUNCATE test-run data
db.ts: upsertStockObservation now accepts stockConfidence, priceCurrency,
priceIncludesTax, stockVendorTs; delta detection includes quantity_available
fs-com.ts: passes stockConfidence=3 + priceCurrency=EUR + priceIncludesTax=false
qsfptek.ts v2: Phase 1 API listing + Phase 2 detail-page stock extraction
- Parses 'X in real-time stock, DATE' from product detail pages
- Writes stock_observations with confidence=2 + stockVendorTs
- Up to 500 detail pages/run at 2s rate limit
naddod.ts v2: complete rewrite from WooCommerce to Astro sitemap-based
- Discovers products via /sitemaps/products.xml (600+ products)
- URL format: /products/XXXXX.html
- Extracts 'In Stock: X' exact counts from SSR HTML
- Writes both price + stock observations (confidence 1 or 2)
2026-04-17 22:54:40 +02:00
Rene Fichtmueller
cdb8ef6e61
feat(scraper): add FiberMall/Vcelink/OpticsBay scrapers, fix QSFPTEK API migration
...
- New scrapers: fibermall.ts (WooCommerce), vcelink.ts (Shopify), opticsbay.ts (WooCommerce)
- QSFPTEK rewritten to use /mall/commodity/list API (old OpenCart /c/*.html paths gone 404)
- New: attribute-based filtering by data rate (1G/10G/25G/40G/100G/200G/400G/800G)
- Scrapes HTML fragments, extracts US$ prices and product URLs
- scheduler.ts: +3 queues/schedules/workers (fibermall, vcelink, opticsbay) → 61 total workers
- index-pi.ts: Pi fleet picks up all 3 new scrapers
2026-04-11 19:13:36 +02:00
Rene Fichtmueller
2348238888
feat: add NADDOD, QSFPTEK, and AddOn Networks scrapers
...
Three new fetch-based price scrapers for compatible optics vendors:
- NADDOD: WooCommerce, USD, ~800+ SKUs
- QSFPTEK: Custom PHP shop, USD, ~1000+ SKUs
- AddOn Networks: Magento/custom, USD, ~2500 SKUs
All registered in scheduler (8-12h intervals) and index.ts --flags.
Build: 0 TypeScript errors.
2026-03-30 21:20:23 +02:00
