rene/transceiver-db
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: skip Optcore on Erik — Cloudflare blocks datacenter IP

Browse Source 
optcore.net blocks Erik's IP (82.165.222.127) via Cloudflare WAF.
WP REST API returns HTML block page instead of JSON → 0 product URLs
→ 0 scraped pages every run. Add SKIP_OPTCORE_SCRAPER guard matching
the existing SKIP_FS_SCRAPER pattern. Set in ecosystem.config.js on
Erik. Residential IP (Mac launchd) would be needed to use this scraper.
This commit is contained in:
Rene Fichtmueller 2026-04-18 05:41:56 +02:00
parent e11e351f5e
commit 48adcd3fc9
1 changed files with 7 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
packages/scraper/src/scheduler.ts
View File

@ -366,6 +366,13 @@ export async function registerWorkers(boss: PgBoss): Promise<void> {
});
await boss.work("scrape:pricing:optcore", async () => {
// Optcore.net WP REST API is blocked by Cloudflare WAF for Erik's datacenter
// IP (82.165.222.127). Set SKIP_OPTCORE_SCRAPER=true to suppress the wasted
// run. A residential IP (Mac launchd) would be needed to scrape Optcore.
if (process.env["SKIP_OPTCORE_SCRAPER"] === "true") {
console.log(`[${new Date().toISOString()}] Optcore pricing: SKIPPED (SKIP_OPTCORE_SCRAPER=true)`);
return;
}
console.log(`[${new Date().toISOString()}] Running: Optcore pricing`);
const { scrapeOptcore } = await import("./scrapers/optcore");
await scrapeOptcore();