rene/transceiver-db
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
442 Commits 5 Branches 9 Tags
Commit Graph

72 Commits

Author SHA1 Message Date
Rene Fichtmueller
0d7a92e749 feat: Abverkauf velocity engine — sql/118 + analyzer + API endpoints 
- sql/118-stock-velocity.sql: new stock_velocity (UPSERT per tx×vendor)
  and stock_velocity_events tables with TimescaleDB-compatible indexes
- stock-velocity-analyzer.ts: computes sell-through from stock_observations
  time-series; detects sold/zulauf/data_gap events, trims top-10% outliers,
  predicts stockout date, assigns high/medium/low/insufficient confidence
- scheduler.ts: analyze:stock:velocity job at 04:30/12:30/20:30 UTC
- stock.ts: GET /api/stock/velocity (paginated, filterable by vendor/confidence/
  stockout_days) + GET /api/stock/velocity/:id (per-product with event history)
- First run: 208 products, 979 sell events, 2811 Zulauf events written
 2026-05-14 00:24:58 +02:00
Rene Fichtmueller
db6b97186a feat: OPN+spec equivalence matchers, 400G pricing, TIP_LLM training data 
- Add OPN-based equivalence matcher robot (7,245 manufacturer-confirmed matches, confidence=1.0)
- Add spec-based equivalence matcher robot (683 matches, confidence=0.85)
  - Matches by form_factor + speed_gbps + reach_tier + wavelength ±10nm
  - Safety cap: skip FX products matching >30 competitors (too generic)
  - Daily schedule: 04:30 UTC via pg-boss
- SQL migrations 116 (OPN) + 117 (spec) with tip_extract_wavelength_nm() + tip_reach_tier() helpers
- Fix tenGtek.ts: add 3 missing 400G categories (QSFP-DD, QSFP112) — closes pricing gap
- Generate tip-llm-pricing-v1.jsonl: 80 DB-grounded QA pairs (pricing, equivalences, 400G)
- Rebuild TIP_LLM training pool: 11,999 pairs (+127 vs prev), deployed to Erik
- FX product equivalence coverage: 88.1% (959/1089)
 2026-05-13 21:33:19 +02:00
Rene Fichtmueller
2f85571784 feat: Flexoptix full product detail sync (sql/115 + detail-enricher robot) 
Pulls complete per-SKU specifications and compatibility data from the
Flexoptix API (specifications=1&compatibilities=1) and writes structured
fields to the transceivers table for datasheet generation.

SQL migration 115:
- Adds fx_specifications JSONB (raw spec blob for datasheet gen)
- Adds fx_compatibilities JSONB (full OEM compatibility matrix)
- Adds compliance_code, laser_type, receiver_type, supported_protocols[]
- Adds extinction_ratio_db, cdr_support, inbuilt_fec, detail_synced_at
- GIN index on fx_compatibilities for vendor/OPN queries

flexoptix-detail-enricher.ts:
- Per-SKU API calls with rate-limiting (600ms/call, 100 SKUs/run)
- Parses all spec labels → structured fields (power, budget, tx/rx dBm,
  modulation, wavelengths, temp range, DOM, laser type, receiver type)
- Strips :Sx variant suffixes before API queries (self-configure SKUs)
- COALESCE writes — never overwrites existing data, only fills gaps
- Tracks detail_synced_at, retries stale entries after 7 days

flexoptix-api-sync.ts:
- Also stores image_url and product_page_url during bulk sync

scheduler.ts:
- Registers enrich:flexoptix-details daily at 03:00 UTC

Results after initial run:
- 791/968 FX products (81.7%) fully enriched
- 26.0 avg compatibility entries per product (OEM vendor + OPN)
- 25.7 avg spec fields per product
- DFB(483), EML(148), FP(72), VCSEL(44) laser type distribution
 2026-05-13 18:49:28 +02:00
Rene Fichtmueller
d1bde66e39 feat: deterministic equivalence matcher + full wavelength/connector enrichment 
Replace confidence-based matcher with deterministic 6-field exact match:
- form_factor (exact), speed_gbps (±0.1G), fiber_type (exact),
  reach (±10%), wavelength_tx (±5nm), connector_type (exact)
- Complete products → confidence=1.0, never creates pending records
- Incomplete products → enhanced confidence ≥0.85, still auto_approved
- PENDING CREATED: 0 (by design, permanent)

Migrations:
- sql/113: Connector type inference from IEEE lookup + form-factor rules
  (970→479 missing connector for FX products)
- sql/114: Extend IEEE lookup with 400G/800G/1.6T OSFP/QSFP-DD standards,
  wavelength fallback (SMF→1310nm, MMF→850nm), clear pending queue to 0

Enrichment results (before→after):
- FX fully complete: 50 → 555 / 1,089 (+505)
- Total fully complete: ~3,600 → 15,431 / 18,133 (+11,800)
- FX coverage: 54.7% → 55.8% (608/1,089 matched)
- Deterministic matches: 0 → 44,596 (confidence=1.0)
- Wavelength-mismatched records rejected: 521
- Pending queue: 42 → 0 (permanent)

New match stats:
- 55,743 new deterministic auto_approved matches
- 521 legacy wavelength-mismatch records rejected
- Total active: 53,447 auto_approved + 1,987 approved
 2026-05-13 17:59:08 +02:00
Rene Fichtmueller
9979b79434 feat: wavelength/connector enrichment schema + enricher robot 
- sql/110: add wavelength_tx_nm, wavelength_rx_nm, connector_type,
  data_completeness, enrichment_needed columns + trigger
- sql/111: IEEE/MSA standards wavelength lookup table (SFP→OSFP)
- sql/112: migrate existing wavelengths TEXT → integer columns
- robots/wavelength-enricher.ts: fills missing wavelengths from IEEE
  lookup (deterministic) then product-name regex, runs every 4h
- scheduler: register enrich:wavelength job (4h schedule)

Fixes over-broad matching where 1G SFPs match 500+ competitors
due to missing wavelength discrimination.
 2026-05-13 17:35:42 +02:00
Rene Fichtmueller
98b241f462 feat: implement Flexoptix reference matching overhaul 
- sql/108: normalize form_factor across all vendors (SFP-Plus → SFP+, etc.)
  and round speed_gbps for consistent matching
- sql/109: document 30→90 day matcher window change
- robots/catalog-reconcile.ts: new bulk-reconcile robot — matches all
  Flexoptix products against all competitors without 30-day time limit
- scheduler.ts: register catalog:reconcile job (monthly + on-demand),
  fix nightly matcher 30→90 day window, UPPER() form_factor matching,
  ROUND() speed_gbps matching

Fixes: ATGBICS/NADDOD/10Gtek/ShopFiber24 had 0 Flexoptix equivalences
due to stale price_observations being filtered out. Expected coverage
improvement: 22% → 45-60% after first reconcile run.
 2026-05-13 16:55:45 +02:00
Rene Fichtmueller
a20094755d feat(scraper): Flexoptix REST API sync robot + scheduler integration 
Replaces the GraphQL/search-based Flexoptix scraper with a proper
Magento 2 REST API integration that delivers authoritative SKUs,
prices, stock levels and compatibility data.

New files:
- packages/scraper/src/robots/flexoptix-api-sync.ts
  Self-contained robot: auth → paginated fetch → normalize → DB write.
  Reads FLEXOPTIX_API_BASE_URL / _USERNAME / _PASSWORD from env.
  Returns { fetched, normalized, skipped, priceWrites, stockWrites }.
  No file intermediary — in-memory pipeline.

- scripts/import-flexoptix-catalog.ts
  One-shot CLI importer for the Pulso-generated JSONL (Codex handover).

- docs/FLEXOPTIX_CATALOG_IMPORT.md
  Runbook for manual import + per-SKU specifications enrichment.

Scheduler changes:
- Added sync:flexoptix-catalog queue + work() handler
- Scheduled every 2h at 0 */2 * * * (same cadence as legacy job)
- scrape:pricing:flexoptix kept as legacy GraphQL fallback

Also includes Codex-generated additions from this sprint:
- audiocodes-oem scraper, seed-batch35/36/37, db.ts improvements,
  sql/102 verification reconcile, README + package.json updates
 2026-05-13 16:36:33 +02:00
Rene Fichtmueller
5eb1b07183 fix: close stale TIP manual review queue 2026-05-10 10:23:07 +02:00
Rene Fichtmueller
0d4bcb6924 fix: preserve explicit competitor states in reconcile 2026-05-10 00:17:26 +02:00
Rene Fichtmueller
635a102932 feat: close open competitor research states 2026-05-10 00:03:42 +02:00
Rene Fichtmueller
650de6ba9a feat: add verification evidence state model 2026-05-09 23:06:21 +02:00
Rene Fichtmueller
1af4f090f7 fix: harden TIP verification cleanup 2026-05-09 22:16:29 +02:00
Rene Fichtmueller
a43e572946 fix: advance TIP product verification robots 2026-05-09 20:19:19 +02:00
Rene Fichtmueller
43b7250180 fix: automate equivalence research review queue 2026-05-09 07:48:11 +02:00
Rene Fichtmueller
199f36be48 fix(scraper): auto-create pg-boss queues before scheduling + worker/schedule order 
- scheduler: patch boss.schedule() to call createQueue() first (idempotent),
  fixing FK constraint errors after DB reset — no need to touch 277 call sites
- index: registerWorkers() before registerSchedules() since boss.work() must
  register handlers before schedules fire
- dashboard: fix switchBlogLlm() to use api() helper (adds Bearer auth token)
  instead of raw fetch() which was returning 401 Unauthorized
 2026-04-29 16:14:25 +02:00
Rene Fichtmueller
39a63e0401 fix(scheduler): vendor discovery crawlers daily 24/7 (not weekly) 2026-04-28 23:59:00 +02:00
Rene Fichtmueller
297dc46f2b feat(crawler-llm): intelligent vendor discovery pipeline + TIPLLM training data 
- spec-validator.ts: physical plausibility checks (form factor↔speed matrix,
  wavelength↔fiber consistency, IEEE standard cross-check, reach limits).
  Outputs tier (high/medium/low/rejected) + confidence_delta for LLM scores.

- training-data-writer.ts: converts validated crawler extractions to SFT JSONL
  training pairs (spec_qa / crawl_reasoning / validation / discovery types).
  Auto-commits and pushes to Gitea tip-training-data repo in batches of 50.

- vendor-discovery-crawler.ts: PlaywrightCrawler pipeline — catalog URL →
  LLM extraction (scrapeWithLLM) → spec validation → DB persist +
  Gitea SFT training pairs. 8 vendor configs registered
  (Cisco/Juniper/Arista/FS.com/Flexoptix/Nokia/Huawei/II-VI).

- scheduler.ts: 8 weekly discover:vendor:* jobs added (Sun 20:00–Mon 10:00 UTC).
  Total registered jobs: 102.

- Gitea repo created: gitea.context-x.org/rene/tip-training-data
 2026-04-28 23:46:34 +02:00
Rene Fichtmueller
2466cc5d82 feat(scraper): batch 37 OEM seeds — Extreme (Legacy), Nortel, 3Com, Avaya 
Added 4 legacy OEM transceiver catalog seed scrapers (72 PIDs total):
- extreme-legacy-oem.ts: 18 PIDs — Summit/BlackDiamond 10052H/10318/10325 family, Legacy
- nortel-legacy-oem.ts: 18 PIDs — Passport/BayStack AA1419xxx + XFP, incl. GBIC, Legacy
- 3com-legacy-oem.ts: 18 PIDs — Switch 5500/7750 3C17770/3CSFP9x + XFP/GBIC, Legacy
- avaya-legacy-oem.ts: 18 PIDs — ERS/VSP AA1419xxx + 700480xxx QSFP28, Legacy

Scheduler: wired at 05:30/05:45/06:00/06:15 UTC. All 72 PIDs seeded clean.
 2026-04-28 23:31:13 +02:00
Rene Fichtmueller
e684d3d1c3 feat(scraper): batch 36 OEM seeds — EnGenius, Palo Alto Networks, Brocade, Foundry Networks 
Added 4 new OEM transceiver catalog seed scrapers (72 PIDs total):
- engenius-oem.ts: 18 PIDs — ECS switch series 1G–100G SFP/SFP+/SFP28/QSFP28 + DAC/AOC
- paloalto-networks-oem.ts: 18 PIDs — PA-3200/5200/7000/5450 NGFW SFP/SFP+/SFP28/QSFP28 + DAC
- brocade-legacy-oem.ts: 18 PIDs — ICX/FCX/VDX/MLX E1MG/10G-SFPP family, market_status=Legacy
- foundry-networks-oem.ts: 18 PIDs — FastIron/NetIron FDR- series incl. XFP, market_status=Legacy

Scheduler: wired at 04:30/04:45/05:00/05:15 UTC. All 72 PIDs seeded clean.
 2026-04-28 23:28:10 +02:00
Rene Fichtmueller
e9b8cb95db feat(scraper): batch 35 OEM seeds — Sierra Wireless, Senao, EMCORE, Reflex Photonics 
Added 4 new OEM transceiver catalog seed scrapers (75 PIDs total):
- sierra-wireless-oem.ts: 18 PIDs — RV55/RV50X/LX60 SFP/SFP+/QSFP+ incl. Industrial -40~85°C
- senao-oem.ts: 20 PIDs — EnGenius ECS switches 1G–100G SFP/SFP+/SFP28/QSFP28 + DAC
- emcore-oem.ts: 20 PIDs — ORION coherent ZR/ZR+/CFP2-DCO 400G + MIL-grade avionics
- reflex-photonics-oem.ts: 17 PIDs — LightABLE MIL-STD-810H + RAD-HARD space-grade

Scheduler: wired at 03:30/03:45/04:00/04:15 UTC. All 75 PIDs seeded to TIP DB.
 2026-04-28 23:24:53 +02:00
Rene Fichtmueller
32d3ded169 feat: add Finisar, Acacia, Inphi OEM scrapers (batch 34) 
- finisar-oem: 17 PIDs (FTLX/FTLC historical BoM series, 1G-100G, widely referenced)
- acacia-oem: 14 PIDs (AC400/AC1200 coherent CFP2-DCO/QSFP-DD/OSFP up to 1.2T)
- inphi-oem: 13 PIDs (ColorZ/COLORZ-II DWDM QSFP28/QSFP-DD + 800G OSFP)
- scheduler: wired all 3 at 02:45/03:00/03:15 UTC
 2026-04-28 23:14:06 +02:00
Rene Fichtmueller
1023b24fd0 feat: add Black Box, Radiflow, DragonWave, Teledyne LeCroy OEM scrapers (batch 33) 
- black-box-oem: 19 PIDs (enterprise LAN SFP/SFP+/SFP28/QSFP28 + BiDi + DAC)
- radiflow-oem: 17 PIDs (OT/ICS security, 100M-100G incl. substation BiDi, category=Industrial)
- dragonwave-oem: 17 PIDs (microwave backhaul fiber uplinks 100M-100G, market_status=Legacy)
- teledyne-lecroy-oem: 18 PIDs (T&M oscilloscopes/analyzers SFP+-QSFP-DD up to 400G ZR)
- scheduler: wired all 4 at 01:45/02:00/02:15/02:30 UTC
 2026-04-28 23:07:26 +02:00
Rene Fichtmueller
7f59f445b6 feat: add Cambium Networks, Tektronix, Clearfield, Lanner OEM scrapers (batch 32) 
- cambium-networks-oem: 18 PIDs (cnMatrix/PTP820 1G-100G + BiDi + DAC)
- tektronix-oem: 19 PIDs (T&M SFP/SFP+/SFP28/QSFP28/QSFP-DD up to 400G ZR coherent)
- clearfield-oem: 16 PIDs (FTTP/FTTx GPON/XGS-PON OLT+ONT + 1G-100G backhaul, heavy Telecom)
- lanner-oem: 20 PIDs (NFVI/uCPE 1G-100G + BiDi + DAC stack)
- scheduler: wired all 4 at 00:45/01:00/01:15/01:30 UTC
 2026-04-28 23:04:08 +02:00
Rene Fichtmueller
22788db26b feat: add Rohde & Schwarz, L3Harris, Zhone OEM scrapers (batch 31) 
- rohde-schwarz-oem: 19 PIDs (T&M optical modules, SFP/SFP+/SFP28/QSFP28/QSFP-DD up to 400G ZR coherent)
- l3harris-oem: 18 PIDs (MIL-grade ruggedized SFP/SFP+/SFP28/QSFP+/QSFP28, category=Industrial)
- zhone-oem: 18 PIDs (GPON/XGS-PON/EPON OLT+ONT plus 1G-100G uplinks, heavy Telecom set)
- scheduler: wired all 3 at 00:00/00:15/00:30 UTC with workers
 2026-04-28 22:57:23 +02:00
Rene Fichtmueller
ab6888fec8 feat: add OEM seed scrapers batch 29-30 (8 vendors, 147 PIDs) 
Adds scrapers for:
- AudioCodes (12 PIDs) — SBC/media gateway transceivers
- Anritsu (19 PIDs) — T&M platform optical modules
- NETSCOUT (19 PIDs) — nGenius probe + InfiniStream optics
- Curtiss-Wright (19 PIDs) — MIL-grade ruggedized transceivers
- ECI Telecom (18 PIDs) — DWDM/OTN/SONET carrier platform
- UTStarcom (17 PIDs) — GPON/XGS-PON/EPON broadband access
- Turbolink (23 PIDs) — Taiwanese OEM transceiver manufacturer
- Chelsio (20 PIDs) — iWARP RDMA NIC optical modules

Scheduler: 8 new cron slots 22:00-23:45 UTC daily.
DB: 12,937 → 13,084 transceivers, 181 → 189 vendors.
 2026-04-27 00:44:18 +02:00
Rene Fichtmueller
d7144731e0 feat(scraper): add 100+ OEM seed scrapers + tip-llm-guided inference layer 
New OEM transceiver seed scrapers (94 cron-scheduled, 24/7):
- Media/Broadcast: Evertz, Grass Valley, Haivision, Viasat
- Asian Optical: FiberHome, Oplink, Accelink, Hisense Broadband
- Optical Mfrs: Lumentum, II-VI/Coherent, Source Photonics, O-Net,
  InnoLight, AOI, Sumitomo Electric, NeoPhotonics
- Industrial: GE Grid, Schweitzer, Moxa Industrial, Cisco IE,
  Phoenix Contact, Beckhoff, Omron, ABB, Siemens, Schneider, Rockwell, Belden
- Enterprise/DC: Arista, Pica8, Pluribus, DriveNets, Cisco (Meraki/Catalyst/Nexus/ASR)
- Cloud: AWS, Azure, Google Cloud, Meta
- Storage: NetApp, Pure Storage, HPE Storage, IBM Storage, Dell Storage, Hitachi Vantara
- 5G/RAN: Samsung Networks, Nokia AirScale, Ericsson RAN, Mavenir
- Security: Check Point, Barracuda, Fortinet, Palo Alto
- Telecom Optical: ADVA, PacketLight, FiberHome, Accelink, Hisense

API: tip-llm-guided inference layer (strict schema + repair-retry + safe fallback)
- POST /api/tip-llm/infer|research-plan|extract|finding|health
- Hard JSON schema enforcement, create_finding=false on empty evidence
- Confidence gate (>= 0.4), validation with consistency check

Build: added incremental=true to scraper tsconfig (OOM prevention)
Scheduler: 87 → 94 registered workers
 2026-04-27 00:00:14 +02:00
Rene Fichtmueller
4479429b29 feat: Brocade/RUCKUS ICX OEM seed (E1MG/E10G/E40G/E100G series, 29 PIDs) 2026-04-26 20:15:54 +02:00
Rene Fichtmueller
ad295a2b4b feat: NVIDIA/Mellanox OEM seed (Spectrum + LinkX portfolio) 
36 PIDs: MFM/MMA/MCP LinkX series covering SFP/SFP+/SFP28/QSFP+/
QSFP28/QSFP56/QSFP-DD/OSFP + LinkX DAC 25G-400G. Includes
SN2000-SN5600 Spectrum switch transceivers. Scheduler: 05:45 daily.
 2026-04-26 19:20:12 +02:00
Rene Fichtmueller
b3a2eff776 feat: Dell EMC + Extreme Networks OEM transceiver seeds 
Dell EMC (34 PIDs): PowerSwitch OS10 + legacy Force10 naming,
1G-400G QSFP-DD + DAC. Maps to Dell Technologies vendor.

Extreme Networks (33 PIDs): Summit/ExtremeSwitching 10xxH part
numbers, 1G-400G QSFP-DD + DAC. Scheduler: 05:15 + 05:30 daily.
 2026-04-26 19:18:31 +02:00
Rene Fichtmueller
1c1b8e1e9d feat: Huawei OEM transceiver seed + scheduler 
Add 43 Huawei OEM PIDs covering CloudEngine/NetEngine platform:
SFP-GE/SFP+-10G/SFP28-25G/QSFP+-40G/QSFP28-100G/QSFP56-200G/
QSFP-DD-400G/OSFP-800G + DAC. Includes BOM alias codes.
Scheduler: daily 05:00.
 2026-04-26 19:16:16 +02:00
Rene Fichtmueller
b4c8b9b625 feat: Nokia/Alcatel-Lucent OEM seed + scheduler 
Add 41 Nokia OEM transceiver PIDs: SFP-1G/SFP-10G/SFP-25G/QSFP+-40G/
QSFP28-100G/QSFPDD-200G+400G + DWDM ZR/ZR+ + DAC. Includes legacy
3HExxxxxxxx alternate part numbers in notes field.
Scheduler: daily 04:45.
 2026-04-26 19:14:27 +02:00
Rene Fichtmueller
51cee266f5 feat: HPE/Aruba OEM seed + Cisco TMG upsert fix 
Add 43 HPE/Aruba OEM transceiver PIDs (J/JL/JH/R series — 1G through
400G QSFP-DD + DAC/AOC). Scheduler: daily 04:30.

Cisco TMG scraper: fixed market_status/temp_range constraint violations,
switched to always-upsert pattern. Result: 423 switches, 22476 Cisco
OEM transceivers, 22476 compat entries written to DB.

Update CHANGELOG_PENDING with all session data changes.
 2026-04-26 19:12:27 +02:00
Rene Fichtmueller
c9a50ad551 feat: Juniper OEM seed scraper + BlueOptics HTTP/1.1 fix 
Add 59 Juniper OEM transceiver PIDs (SFP/SFP+/SFP28/QSFP+/QSFP28/
QSFP56/QSFP-DD/OSFP + DAC/AOC) to seed the transceivers table.
Register scrape:catalog:juniper-oem in scheduler (daily 04:15).

Fix BlueOptics scraper: force HTTP/1.1 via Node.js https.get() to
bypass server bug where HTTP/2 returns empty response body. Also
update catalog path from /transceivers/ to /Transceivers_1.
 2026-04-26 19:08:09 +02:00
Rene Fichtmueller
cc85d3d0f8 feat: Cisco OEM + Arista OEM transceiver catalog scrapers 
- cisco-tmg.ts: upsert Cisco OEM transceivers from TMG API instead of
  SELECT-only. Parsers for formFactor/speed/reach/fiberType/tempRange.
  Fixes market_status ('EOL') + temp_range ('COM'/'IND') check constraints.
- arista-oem.ts: seed scraper for 69 Arista OEM PIDs (1G→800G,
  SFP/SFP28/QSFP+/QSFP28/QSFP-DD/OSFP/QSFP-DD800) with full specs.
- scheduler.ts: daily arista-oem seed at 04:00 UTC
 2026-04-26 19:00:21 +02:00
Rene Fichtmueller
ba998f4c01 fix: vendor_compat 0%→100%, price denorm, wiitek disabled, price-denorm scheduler 
- Migration 094: images for 12 Cisco 8K MPA + A9K-8HG-FLEX + ASR-9000V models
- Migration 095: price denorm refresh (EUR 679→1376, USD 166→835 with 180d window)
- Migration 096: bulk vendor_compat by form_factor — all 9013 transceivers now
  have OEM compatibility patterns (was 0/9013 because all slugs are scraped-*)
- wiitek.ts: disable dead scraper (wiitek.com unreachable since 2026-04, EAI_AGAIN)
- scheduler.ts: add compute:price-denorm job (daily 05:30 UTC) to keep
  street_price_usd/price_verified_eur fresh without manual migration runs
- seed-from-npm.ts: ON CONFLICT now also updates vendor_compat (was only updated_at)
 2026-04-25 08:55:21 +02:00
Rene Fichtmueller
18a9e1346e feat: Playwright image scraper for bot-blocked vendors (Arista/Dell/Edgecore/Fortinet/Extreme) 2026-04-21 06:16:05 +02:00
Rene Fichtmueller
823b64bd24 perf: load-aware scraper guard + higher rate limits + /tmp crawlee storage 2026-04-20 23:35:02 +02:00
Rene Fichtmueller
4bf5c95824 feat: Flexoptix compatibility scraper + transceiver issue scanner 
- Add flexoptix-compat.ts: maps switch models to compatible Flexoptix transceivers
  via search API (vendor_compat) with form-factor fallback (spec_match)
  Scheduled daily at 09:00 UTC as scrape:compat:flexoptix
- Enhance community-issues.ts: add vendor advisory sources (Cisco Field Notices,
  Juniper KB, SONiC GitHub Issues) + new scrapeTransceiverCompatIssues() that
  searches for switch+transceiver combination problems specifically
- Scheduler: 59 schedules, 78 workers
 2026-04-20 22:50:57 +02:00
Rene Fichtmueller
a0a7a97d83 feat: switch image fetcher + og:image scheduler job + dashboard thumbnail column 
- Add switch-image-fetcher.ts: og:image-based image discovery for all 86 seeded switches
  (covers Cisco, Arista, Juniper, NVIDIA, Edgecore, Celestica, Asterfusion, Dell,
   HPE/Aruba, Huawei, Nokia, Extreme, MikroTik, Ubiquiti, FS.COM, Supermicro)
- Wire fetchSwitchImages() into scheduler as scrape:images:switches (daily 08:30 UTC)
- Dashboard: add 48px thumbnail column to switch table (lazy img with gear icon fallback)
 2026-04-20 22:44:08 +02:00
Rene Fichtmueller
48adcd3fc9 fix: skip Optcore on Erik — Cloudflare blocks datacenter IP 
optcore.net blocks Erik's IP (82.165.222.127) via Cloudflare WAF.
WP REST API returns HTML block page instead of JSON → 0 product URLs
→ 0 scraped pages every run. Add SKIP_OPTCORE_SCRAPER guard matching
the existing SKIP_FS_SCRAPER pattern. Set in ecosystem.config.js on
Erik. Residential IP (Mac launchd) would be needed to use this scraper.
 2026-04-18 05:41:56 +02:00
Rene Fichtmueller
93d825dc04 fix: daemon stability + health monitor accuracy 
- Add global unhandledRejection handler in scheduler daemon to swallow
  Crawlee's benign post-run ENOENT lock-file races (prevents process.exit(1))
- Add SKIP_FS_SCRAPER env var: skip FS.com worker on Erik where Cloudflare
  WAF blocks datacenter IPs (Mac launchd handles FS.com from residential IP)
- Remove FS.COM from health monitor EXPECTED_VENDORS (skipped on Erik)
- Health monitor: extend pg-boss lookup from 12h → 26h, add completed-job
  map; if job ran OK in last 26h + vendor has historical prices → mark
  STABLE instead of CRITICAL (fixes ATGBICS/Fluxlight hash-dedup false positives)
- Install Playwright Chromium on Erik (fixes ATGBICS BrowserLaunchError)
- Create missing Crawlee storage dirs on Erik (storage-fs-phase1/2,
  storage-ebay-transceivers) to prevent ENOENT on first Crawlee run
 2026-04-18 03:16:59 +02:00
Rene Fichtmueller
24ff9822ac fix: improve scraper health monitor — tiered alerts, suppress stable-price false positives 
Previous logic fired an alert whenever prices_6h=0, even when prices
were genuinely stable (content hash dedup prevents duplicate inserts).
This caused Flexoptix, ATGBICS and others to trigger alerts every 3h
despite their scrapers running successfully.

New logic:
  🔴 CRITICAL: last price > 7 days (genuine failure)
  🟡 WARNING:  last price 48h–7 days (possibly stale)
  ✅ STABLE:   last price ≤48h, 0 new (prices unchanged, scraper OK)

Also shows pg-boss job state/time alongside each vendor for faster
root-cause diagnosis. Trimmed EXPECTED_VENDORS to vendors with actual
scraper implementations (removed never-scraped placeholders).
 2026-04-18 02:54:28 +02:00
Rene Fichtmueller
419af4a24e fix: remove all withIsolatedStorage wrappers, add makeCrawleeConfig to remaining Crawlee scrapers 
- scheduler.ts: remove withIsolatedStorage from ALL scrapers (atgbics,
  optcore, ufispace, edgecore, ebay-*, market-intel, community-issues,
  cisco, juniper, sonic, 10gtek, prolabs, switch-assets, fs)
  eliminates global CRAWLEE_STORAGE_DIR race condition entirely
- fs-com.ts: replace purgeDefaultStorages() with rmSync on isolated
  storage dirs (fs-phase1, fs-phase2); pass makeCrawleeConfig to both
  PlaywrightCrawler instances
- switch-assets-crawler.ts: add makeCrawleeConfig('switch-assets')
- switch-assets-playwright.ts: add makeCrawleeConfig('switch-assets-playwright')
- naddod.ts: restore clean error logging (remove debug instrumentation)
 2026-04-18 02:19:53 +02:00
Rene Fichtmueller
c7d7456de9 fix: instance-level Crawlee storage isolation + eBay vendor type 
- Add utils/crawlee-config.ts: makeCrawleeConfig(name) returns a
  Crawlee Configuration with isolated localDataDirectory per scraper.
  Uses storageClientOptions (not global CRAWLEE_STORAGE_DIR) so
  concurrent pg-boss workers in the same process don't race on
  the shared env var.

- Apply makeCrawleeConfig to all 6 Crawlee-based scrapers:
  optcore (PlaywrightCrawler), atgbics (PlaywrightCrawler),
  community-issues (CheerioCrawler + RequestQueue),
  edgecore (CheerioCrawler), ufispace (CheerioCrawler),
  market-intelligence (CheerioCrawler).

- scheduler.ts: add withIsolatedStorage for optcore and market-intel
  workers (was missing, caused storage-fs path bleed from fs scraper).

- ebay-enricher.ts: fix vendor type 'marketplace' -> 'reseller' to
  satisfy vendors_type_check constraint
  ['manufacturer','distributor','oem','reseller','compatible'].
 2026-04-18 01:35:57 +02:00
Rene Fichtmueller
2b770aa1a9 chore: cleanup — rename digikey→mouser, remove orphan files, gitignore Crawlee artifacts 
- Rename scrapers/digikey.ts → scrapers/mouser.ts: export scrapeMouser()
  (file was Mouser API implementation mislabeled from task origin)
- Fix scheduler.ts mouser-oem worker: import scrapeMouser from ./scrapers/mouser
- Delete switch-seed-smb.ts (unreferenced, no CLI flag, no scheduler job)
- Add storage/, storage-fs/, .crawlee/ to .gitignore (Crawlee runtime artifacts)
 2026-04-18 01:09:10 +02:00
Rene Fichtmueller
1c8dec52c9 feat: Price Comparison dashboard + Eoptolink OEM scraper 
- Add public /api/price-comparison API (summary, top-50, per-SKU detail)
  — no auth required, 3 Express routes, DISTINCT ON latest-price logic
- Add '💲 Price Comparison' dashboard tab: stat cards, form-factor
  breakdown, top-50 SKU table (clickable rows → SKU detail), per-vendor
  price + stock + spread% lookup panel
- Add Eoptolink OEM catalog scraper (93 product-solution pages,
  part-number regex EOLO-*/EOLQ-* etc., no prices, seeds transceivers
  table as manufacturer entries)
- Register scrape:catalog:eoptolink in scheduler: schedule every 4h
  (40 */4 * * *), lazy-import worker, added to known-jobs array
 2026-04-18 01:02:08 +02:00
Rene Fichtmueller
9d3019d0c0 feat: Norton-Bass Hype Cycle Engine — market_metrics seed + Bass fitting + Gartner phase detection 2026-04-18 00:09:08 +02:00
Rene Fichtmueller
75cea9fe90 feat: Mouser Electronics API scraper for OEM reference prices (Juniper/Cisco/Arista PIDs) 2026-04-18 00:04:35 +02:00
Rene Fichtmueller
cdb8ef6e61 feat(scraper): add FiberMall/Vcelink/OpticsBay scrapers, fix QSFPTEK API migration 
- New scrapers: fibermall.ts (WooCommerce), vcelink.ts (Shopify), opticsbay.ts (WooCommerce)
- QSFPTEK rewritten to use /mall/commodity/list API (old OpenCart /c/*.html paths gone 404)
  - New: attribute-based filtering by data rate (1G/10G/25G/40G/100G/200G/400G/800G)
  - Scrapes HTML fragments, extracts US$ prices and product URLs
- scheduler.ts: +3 queues/schedules/workers (fibermall, vcelink, opticsbay) → 61 total workers
- index-pi.ts: Pi fleet picks up all 3 new scrapers
 2026-04-11 19:13:36 +02:00
Rene Fichtmueller
148d2e1000 fix(scraper): set CRAWLEE_PURGE_ON_START=1 in withIsolatedStorage 
Crawlee's SessionPool throws 'Could not find SDK_SESSION_POOL_STATE.json'
when initializing against a freshly-created isolated storage dir.
Setting CRAWLEE_PURGE_ON_START=1 tells Crawlee to start fresh instead
of trying to load non-existent session state — fixes FS.com and ATGBICS
crashes at the start of every 2h cycle after the dirs were cleaned up.
 2026-04-11 07:27:24 +02:00