Crawlee Evaluation and FS.com URL Discovery

Date: 2026-05-09

Question

Operator asked with highest priority whether these repositories help TIP:

https://github.com/apify/crawlee
https://github.com/apify/crawlee-python
https://github.com/hiteshchoudhary/crawlee-project

Evaluation

apify/crawlee helps directly, but TIP already uses it in the TypeScript scraper stack. The priority is to harden our current usage rather than introduce a new crawler framework.

Best immediate Crawlee practices for TIP:

keep per-vendor bounded runs
use stable uniqueKey/target IDs so retries do not create duplicate rows
keep Crawlee storage directories isolated per vendor/run class
record no-text and max-retry URLs as a separate retry class
use AutoscaledPool telemetry as a safety signal
keep Erik at low concurrency and move heavier work to Pi/Proxmox workers

apify/crawlee-python is useful for future isolated worker experiments on Pi/Proxmox, especially where Python extraction libraries help. It should not replace the current TypeScript crawler core today.

hiteshchoudhary/crawlee-project is a small community/demo app, not a production building block for TIP.

Code

Changed:

packages/scraper/src/scrapers/fs-com.ts

Added:

FS_URL_DISCOVERY_ONLY=1
target row propagation with targetTransceiverId
image verification for target rows
H1/part/spec deterministic detail verification when FS.com lacks a spec table

Live Runs

URL discovery pilot:

target 20
scraped 19
failed 0
no-url rows: 76 -> 57

Full URL discovery:

target 56
scraped 55
failed 1
failed URL: https://www.fs.com/de/products/229461.html
no-url rows: 57 -> 2

DB reconciliation:

target 57
scraped 55
failed 0
new prices 41
stock observations 40
specs verified 55

Build:

pnpm -C packages/scraper build passed on Erik

FS.com Final State

total rows: 383
price verified: 379
image verified: 374
details verified: 373
price+image+details: 373
fully verified: 205
missing URL: 2
missing image URL: 9
missing reach label: 4
missing fiber type: 9
HTML product-like rows: 373
HTML product-like complete: 371
no-url rows: 2
category rows: 4

Remaining no-url rows:

Change
FS-229461

TIP health after run:

status: healthy
load status: ok
memory used: 13%
global image verified: 10711
global details verified: 9929
global fully verified: 8526

Training Pool

Pushed:

4d9a11c crawl: add fscom url discovery learning record

Do not claim FS.com is 100% complete yet. Remaining work:

classify Change
retry or classify FS-229461
classify 4 category rows
close 9 image/fiber gaps
then move to next high-value competitor with the same bounded Crawlee pattern

2.8 KiB Raw Blame History