CHANGELOG_PENDING.md

@@ -3,11 +3,9 @@
 Format: `{"d":"YYYY-MM-DD","t":"TYPE","m":"Description"}`
 Types: FEAT · FIX · UI · DATA · AI · INFRA
 
-{"d":"2026-04-18","t":"FIX","m":"PM2 SKIP_FS_SCRAPER env not picked up by tip-scraper-daemon: pm2 restart --update-env did not apply new ecosystem.config.js vars because PM2 loaded from its saved dump. Fixed: pm2 delete + pm2 start ecosystem.config.js --only tip-scraper-daemon + pm2 save. Daemon restarted fresh (ID 83, 0 restarts) with SKIP_FS_SCRAPER=true now confirmed live. FS.com job now correctly skips on Erik instead of failing with ENOENT."}
 {"d":"2026-04-18","t":"FIX","m":"FS.com Mac scraper: suppress Crawlee post-run ENOENT unhandledRejection — Crawlee's FileSystemStorage fires a final _isTaskReadyFunction call after run() resolves, reading a request .json that was already processed/cleaned-up. This ENOENT triggered process.exit(1) before Phase 2 completed, causing 7 days of missing FS.com price data. Fixed: targeted unhandledRejection handler in require.main block swallows ENOENT from request_queues paths while re-raising real errors."}
 {"d":"2026-04-18","t":"FIX","m":"FS.com Mac scraper: PID lock (/tmp/tip-fs-scraper.lock) added to run-fs-scraper-mac.sh — prevents concurrent instances when launchd 2am fire overlaps with a still-running earlier run. Previous concurrent instances caused rmSync(storage-fs-phase1) race (one instance deletes the storage dir while another is using it), crashing Phase 2."}
 {"d":"2026-04-18","t":"FIX","m":"Scraper health monitor: tiered alerts replacing false-positive 6h threshold. Old: fired every 3h for any vendor with 0 new prices (including stable prices). New: 🔴 CRITICAL (last price >7 days), 🟡 WARNING (last price 48h-7 days), ✅ STABLE (0 new prices but last price ≤48h — content hash dedup, scraper running OK). Shows pg-boss job state+time for faster root-cause."}
-{"d":"2026-04-18","t":"FIX","m":"Daemon stability: global unhandledRejection handler in scheduler index.ts — Crawlee post-run lock-file ENOENT (request_queues path) was crashing the daemon process (process.exit(1)) which killed all active pg-boss jobs and triggered PM2 restart loops. Fix: swallow ENOENT from request_queues paths at the scheduler level; re-raise all other rejections. Also: FS.com scheduler worker now skips (SKIP_FS_SCRAPER=true env var) on Erik where Cloudflare WAF blocks datacenter IPs; Mac launchd handles FS.com scraping. Created missing Crawlee storage dirs: storage-fs-phase1, storage-fs-phase2, storage-ebay-transceivers, storage-fs. Health monitor pg-boss lookup extended from 12h → 26h; added completedMap; vendors with recent job completion + historical prices classified as STABLE (not CRITICAL) — eliminates ATGBICS/Fluxlight false-positive alerts."}
 {"d":"2026-04-18","t":"FIX","m":"Playwright chromium_headless_shell-1217 installed on Erik (/opt/playwright-browsers/): ATGBICS and FS.com PlaywrightCrawler were throwing BrowserLaunchError on every run since Crawlee browser-pool requires chromium_headless_shell binary, not regular chromium. Fixed by: PLAYWRIGHT_BROWSERS_PATH=/opt/playwright-browsers npx playwright install chromium --with-deps on Erik."}
 {"d":"2026-04-18","t":"FIX","m":"Crawlee withIsolatedStorage global env-var race condition eliminated: scheduler.ts removed all withIsolatedStorage() wrappers (were mutating process.env.CRAWLEE_STORAGE_DIR globally, causing concurrent scrapers to pick up wrong storage dirs). All Crawlee scrapers now use makeCrawleeConfig(name) instance-level Configuration. fs-com.ts migrated to fs-phase1/fs-phase2 storage names with rmSync cleanup before each phase. switch-assets-crawler.ts and switch-assets-playwright.ts now pass makeCrawleeConfig. Fixed: ATGBICS, community-issues, market-intel, FS.com."}
 {"d":"2026-04-18","t":"FIX","m":"FS.com Mac launchd scraper (org.tip.fs-scraper): was failing with exit code 126 (Operation not permitted) because macOS TCC blocks launchd agents from accessing ~/Desktop/Claude Code/ path. Fixed: script moved to ~/.tip/run-fs-scraper-mac.sh, plist WorkingDirectory changed to ~/.tip. FS.com is now scraping from residential Mac IP again."}

packages/scraper/src/scrapers/fs-com.ts

@@ -354,20 +354,7 @@ async function collectProductUrls(
     },
   }, makeCrawleeConfig("fs-phase1"));
 
-  try {
+  await crawler.run(listingRequests);
-    await crawler.run(listingRequests);
-  } catch (err) {
-    const msg = err instanceof Error ? err.message : String(err);
-    if (msg.includes("ENOENT") && msg.includes("request_queues")) {
-      // Benign Crawlee post-run lock-file race: _isTaskReadyFunction reads a
-      // request .json that was already cleaned up after the crawl finished.
-      // Crawlee catches + re-throws it internally, which rejects crawler.run().
-      // Safe to ignore — all pages were already processed.
-      console.warn("[Phase 1] Crawlee post-run ENOENT (benign, ignoring)");
-    } else {
-      throw err;
-    }
-  }
   console.log(`[Phase 1] ${products.size} unique products across ${CATEGORY_URLS.length} categories`);
   return products;
 }
@@ -629,16 +616,7 @@ async function scrapeProductDetails(
     },
   }, makeCrawleeConfig("fs-phase2"));
 
-  try {
+  await crawler.run(requests);
-    await crawler.run(requests);
-  } catch (err) {
-    const msg = err instanceof Error ? err.message : String(err);
-    if (msg.includes("ENOENT") && msg.includes("request_queues")) {
-      console.warn("[Phase 2] Crawlee post-run ENOENT (benign, ignoring)");
-    } else {
-      throw err;
-    }
-  }
   return details;
 }