rene/transceiver-db
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
260 Commits 5 Branches 9 Tags
Commit Graph

260 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Rene Fichtmueller
356add2e49 feat(data): image migrations 075-077 — Cisco NCS 9300 SE1, QCT, QNAP, Sophos, Barracuda, Peplink, Westermo 
Migration 075: Cisco Nexus 9300 SE1 (7 models) — poster-image + support CDN
Migration 076: QuantaMesh (3), QNAP QSW-M5216-1T, Sophos CS210-48FP (5 models)
Migration 077: Barracuda F900, Peplink SD Switch 24-Port, Westermo Lynx 5612 (3 models)

Coverage: 428 → 443 images (63.8% → 66.0%)
 2026-04-21 14:49:29 +02:00
Rene Fichtmueller
ba5ab9c502 chore: redact internal IP from CHANGELOG before public push 2026-04-21 14:31:07 +02:00
Rene Fichtmueller
5d7a7a9876 data: add switch images migrations 073-074 (Cisco ASR9k/NCS540 + multi-vendor) 
Migration 073 — Cisco ASR 9000 + NCS 540 Series (18 entries):
  ASR-9001/9901/9902/9903 + FC fabric-card variants
  A9KV-V2-AC/DC-A/DC-E satellite shelf
  N540-28Z4C, N540-24Z8Q2C, N540-12Z20G, N540-ACC, N540X-16Z4G8Q2C
  Sources: networktigers.com Shopify CDN, tempestns.com WP CDN
  Coverage: 400 → 418 (62.3%)

Migration 074 — Extreme / Ruijie / Ruckus / ZTE / Edgecore (10 entries):
  Extreme SLX 9740-40C, X695-48Y-8C, 5520-48T
  Ruijie RG-S6920-4C, RG-S5760C-24SFP/8GT8XS-X
  Ruckus ICX 7150-48PF, ICX 7550-48ZP
  ZTE ZXR10 5960-56PM-H, ZXR10 9908
  Edgecore AS7712-32X
  Coverage: 418 → 428 (63.8%)

All URLs verified HTTP 200. A/D power variants share chassis images.
 2026-04-21 14:30:42 +02:00
Rene Fichtmueller
4e927acf55 data: switch image coverage 065-072 — Cisco/Juniper/Arista/NVIDIA/Huawei/Nokia/Dell/Extreme/HPE/Ubiquiti/Supermicro/Celestica/Asterfusion/FS.com/Edgecore 
Migrations 065-072: 72 verified image URLs across all 17 tier-1 vendors.
- 065: Cisco 8000/Catalyst/Nexus/NCS (14) — cisco.com/c/dam doc CDN
- 066: Juniper EX/MX/QFX (10) — juniper.net/content/dam image library
- 067: Arista remaining 7000-series (11) — arista.com QSG front-panel PNGs
- 068: NVIDIA Mellanox SN-series (5) — docscontent.nvidia.com dims4 CDN
- 069: Huawei CloudEngine/NE40E (5) + Nokia IXR-D3L/7750 SR-14s (2)
- 070: Dell PowerSwitch ON-series (5) + Extreme Networks 8720/X465 (2)
- 071: HPE Aruba CX 6300M/8100/8360 (3) + Ubiquiti USW (3) + Supermicro (2)
- 072: Celestica DS3000/4000/5000 (3) + Asterfusion CX-N (3) + FS.com (2) + Edgecore (2)

All URLs verified HTTP 200 (2026-04-21). 5 models skipped (no public image found):
Arista 7280R3A-48D5, 750-36Y; NVIDIA SN3750-SX; Nokia 7750 SR-1 (hotlink-protected), 7220 IXR-H4.
 2026-04-21 11:34:38 +02:00
Rene Fichtmueller
382de40d5d data: Avaya/NetApp/Keysight/A10/Evertz/RAD/Ekinops/DrayTek/Fujitsu/Broadcom/Calix/Citrix images (migration 064) 2026-04-21 10:18:51 +02:00
Rene Fichtmueller
3cd6d44ea5 data: Sophos XGS 6500 image + Zyxel XS3800-28 URL fix (migration 063) 2026-04-21 10:16:51 +02:00
Rene Fichtmueller
b7e8097a57 data: Cambium/Gigamon/SonicWall/Planet/PaloAlto/Westermo/Zyxel/Synology/TRENDnet/Waystream/Kemp/LANCOM images (migrations 061-062) 2026-04-21 10:10:32 +02:00
Rene Fichtmueller
906619e602 docs: update CHANGELOG_PENDING with migrations 055-060 2026-04-21 09:57:06 +02:00
Rene Fichtmueller
d3eb53ad1b data: HPE Aruba CX + Extreme Networks images 0%→100% (migration 060) 2026-04-21 09:56:11 +02:00
Rene Fichtmueller
6827f6081f data: Arista/Edgecore/Fortinet/Dell/Huawei/D-Link/Netgear/Check Point/Ruckus images (migrations 057-059) 2026-04-21 09:55:22 +02:00
Rene Fichtmueller
3b42f0aa90 data: ALE/H3C/Hirschmann/Ciena/Netberg images 0%→100% (migration 056) 2026-04-21 09:50:46 +02:00
Rene Fichtmueller
b9ab664a56 data: MikroTik CRS/CCR CDN images + NVIDIA ConnectX-7 (migration 055) 2026-04-21 09:48:30 +02:00
Rene Fichtmueller
e2061c6c4d data: direct image injection for Nokia, F5, Delta Networks, Siemens, TP-Link 
Migration 051: TP-Link TL-SG3452XP + TL-SX3016F via static.tp-link.com CDN
Migration 052: Nokia 6/6 — 7220 IXR-D3L/H4 (docs.nokia.com graphics),
  7250 IXR-10 + 7750 SR-1 (tempestns.com), SR-14s (telecomcauliffe.com),
  SR-1e (docs hardwareBanner — no standalone public image available)
Migration 053: F5 BIG-IP i5800/i10800 (wtit.com), i15800 (blueally CDN)
Migration 054: Delta Networks 4/4 (hardwarenation.com + manualslib),
  Siemens SCALANCE 4/4 — X-200/X-300/X-500 via images.sw.cdn.siemens.com

All 14 URLs verified HTTP 200 with correct image content-type (2026-04-21).
CHANGELOG_PENDING.md updated for all 4 migrations.
 2026-04-21 08:42:14 +02:00
Rene Fichtmueller
466cda285b data: image coverage improvements — QCT, Allied Telesis + CHANGELOG update 
- sql/046: QCT QuantaMesh T3048-LY8 direct image injection
- sql/050: Allied Telesis 3/3 (AT-x530-28GSX, AT-x530L-52GPX, AT-x950-28XSQ)
  via alliedtelesis.com Drupal static files CDN (og:image, all 200 PNG)
- CHANGELOG: document filter pattern fixes + all 6 new vendor migrations
  (Moxa/UfiSpace/Brocade/NVIDIA/Allied Telesis — 19 new images total)
 2026-04-21 08:11:57 +02:00
Rene Fichtmueller
bbc6f560dd fix: add image filter patterns and direct URL migrations for 6 vendors 
- switch-image-playwright.ts + switch-image-fetcher.ts: add filter patterns
  for /webimage-404/ (Netgear 404 hero), /Brand/ + /cybersecurity.png/
  (Moxa brand marketing images not product photos)
- sql/047: Moxa 4/4 models — CDN getattachment paths (hotlink-protected,
  Referer: moxa.com required; R2 proxy needed for production display)
- sql/048: UfiSpace 6/6 models — ufispace.com/image/<hash>/ direct PNGs;
  Brocade G720+G730 — broadcom.com og:image; ICX 7850-48FS — CommScope/Ruckus
  vistancenetworks.com ImageServer (rand param is cache-bust only, not auth)
- sql/049: NVIDIA SN-series 6/6 — docscontent.nvidia.com (SN2201/3700/4700)
  and S3 direct (SN5400/5600); SN3750-SX via uvation reseller CDN
 2026-04-21 07:57:55 +02:00
Rene Fichtmueller
b65e4452db fix: add error-graphic, icon-library, illustration filters to GENERIC_IMAGE_PATTERNS 
- /404[-_]error/i, /error[-_]graphic/i — Broadcom 404-ERROR-GRAPHIC.png
- /\/icon[-_]library\//i — D-Link navigation/icon-library path images
- /[-_]illustration[._]/i — Arista Cloud-Legacy_Illustration and similar diagrams
- Nokia banner, Huawei marketing, banners/ path patterns (Playwright scraper)
- Cookie consent patterns synced to switch-image-fetcher.ts (was only in Playwright)
 2026-04-21 07:38:01 +02:00
Rene Fichtmueller
f4afe14af4 feat: add 12 new vendor URL builders to Playwright image scraper 
- Nokia, Huawei, Ciena, Moxa, D-Link, Alcatel-Lucent Enterprise,
  Asterfusion, Brocade: passthrough builders (use stored product_page_url)
- NVIDIA Networking: SN-series URL builder (sn5600 → /ethernet-switching/sn5600/)
- Netgear: lowercase model slug builder for /business/wired/switches/fully-managed/
- UfiSpace: hardcoded sitemap-verified URL map (all 6 S9xxx models)
- QCT: hardcoded URL map for T3048-LY8 and T7032-IX1
- Add Nokia banner / Huawei marketing image patterns to GENERIC_IMAGE_PATTERNS
 2026-04-21 07:24:11 +02:00
Rene Fichtmueller
6a4b4700cb data: inject Edgecore product images directly (Playwright blocked by 403) 
Edgecore blocks headless browsers (Playwright 403) but serves og:image via
plain HTTP. 5 models resolved via direct curl extraction:
- DCS204, DCS510, DCS810, EPS203 → edge-core.com/product/<slug>/
- Minipack2 → minipack-as8000-open-modular-platform product page
AS7535/AS7726/AS7946/AS9516 not on Edgecore's public WooCommerce site.
 2026-04-21 07:07:24 +02:00
Rene Fichtmueller
8f36eff956 fix(scraper): filter OneTrust/cookie-consent images + skip in img fallback 
cdn.cookielaw.org logos appear as the largest DOM image on Dell/Extreme
product pages when the cookie consent overlay is present. Added to both
GENERIC_IMAGE_PATTERNS (isGenericImage filter) and img fallback skipPattern
so the next-largest actual product image can be found.
 2026-04-21 06:45:41 +02:00
Rene Fichtmueller
d67fbe31da fix(scraper): fall through to img fallback when og:image is generic/logo 
Previously: if og:image existed (even as a Dell logo URL), page.evaluate() returned
early and the img fallback was never tried. Now: meta tags are extracted first, then
isGenericImage() is checked in Node.js, and the img fallback runs if meta image is null
or generic. This allows vendors like Dell (og:image = logo) to still get product images
via the DOM fallback.
 2026-04-21 06:36:12 +02:00
Rene Fichtmueller
09d3a60b7c fix(scraper): fix Edgecore/Extreme URL builders, broaden img fallback, fix ENOENT 
- buildEdgecoreUrl: /product/<slug>/ (WooCommerce, no .html) with EDGECORE_SLUG_MAP
  for AS7712-32X→as7712-32x-ec, Minipack2→minipack-as8000-open-modular-platform
- buildFortinetUrl: returns null (all pages redirect to generic, no usable og:image)
- buildExtremeUrl: direct product URL (extremenetworks.com/product/<slug>)
- img fallback: remove strict 'product/switch/router/hardware' path requirement;
  now takes largest image >=200x150px excluding flags/icons/spinners — isGenericImage()
  filters hero/banner/logo afterward
- ENOENT fix: unique per-run Crawlee storage dir (timestamp suffix) prevents
  stale request-queue file contamination between back-to-back vendor runs
 2026-04-21 06:33:32 +02:00
Rene Fichtmueller
87b9416592 fix(scraper): fix Arista series-level URL builder + bypass Crawlee URL deduplication 
- buildAristaUrl() now extracts series prefix (7060X5-32QS → 7060x5-series)
  instead of individual model URLs that lack og:image
- Strip trailing sub-variant 'A' so R3A → R3 series page
- Add uniqueKey: row.id to each request — prevents Crawlee from deduplicating
  models that share the same series URL (e.g. 7060x5-series)
- For Arista: always prefer fresh builder URL over stored product_page_url
  so stale individual-model URLs don't override correct series pages
 2026-04-21 06:22:41 +02:00
Rene Fichtmueller
18a9e1346e feat: Playwright image scraper for bot-blocked vendors (Arista/Dell/Edgecore/Fortinet/Extreme) 2026-04-21 06:16:05 +02:00
Rene Fichtmueller
653824f23b fix: Cisco line card URL mapping (8800/84/86 → 8000 family page, skip ASR9K logo-only) 2026-04-21 00:49:32 +02:00
Rene Fichtmueller
c9333ab5ea fix: MikroTik hardcoded slug map for + models (crs305/312/317/326) 2026-04-21 00:45:41 +02:00
Rene Fichtmueller
9618a4f0e0 fix: Cisco 8000 builder URL + MikroTik lowercase + new vendor builders 
URL builder fixes:
- Cisco 8000: update to new /site/us/en/ URL scheme (family page, not per-model)
- MikroTik: fix to lowercase+underscore format (was uppercase, caused 404)
- Fortinet: set to null — JS-rendered pages, all redirect to generic page
- Alcatel-Lucent Enterprise slug added to dispatcher (was missing, caused 0 hits)
- Add Quanta, Allied Telesis, Ufispace, Netgear URL builders
- NVIDIA: skip ConnectX/BlueField non-switch models

Migration 044:
- Clear 35 wrong NCS-5500 URLs from Cisco 8000-series models
- Pre-set correct 8000-series family URL for 21 models without images
 2026-04-21 00:41:31 +02:00
Rene Fichtmueller
f340482752 fix: monitor-erik.sh — correct Erik SSH target + fix awk header skip 2026-04-21 00:34:28 +02:00
Rene Fichtmueller
9e6be570a3 feat: more switch image coverage + system health metrics + Erik monitor 
switch-image-fetcher:
- Add Fortinet URL builder (11 FortiSwitch models)
- Add Quanta Cloud Technology, Allied Telesis, Ufispace, Netgear URL builders
- Fix alcatel-lucent-enterprise slug missing from URL_BUILDERS dispatcher
- Fix NVIDIA builder to skip ConnectX/BlueField adapters (not switches)
- Add aruba slug alias for hpe-aruba

health endpoint:
- Add system metrics: CPU load (1/5/15m), memory usage, disk usage
- Add load_status indicator (ok/busy/overloaded)
- Expose process RSS memory
- Used by external monitors

scripts/monitor-erik.sh:
- Cron-ready health check script for Claudi (.82) and Raspberry Pis
- Checks TIP API health endpoint (load, memory, disk, DB latency)
- Checks PM2 process state via SSH (errored/stopped detection)
- ntfy.sh push notifications (set NTFY_TOPIC env var)
- Includes systemd service + timer unit comments for auto-install
 2026-04-21 00:31:43 +02:00
Rene Fichtmueller
823b64bd24 perf: load-aware scraper guard + higher rate limits + /tmp crawlee storage 2026-04-20 23:35:02 +02:00
Rene Fichtmueller
a2492d833b feat: Flexoptix order section per switch + reject generic/logo images 2026-04-20 23:31:36 +02:00
Rene Fichtmueller
0a60d821fb fix: expand compatibility.verification_method CHECK to include vendor_compat + spec_match 2026-04-20 23:23:23 +02:00
Rene Fichtmueller
4f277df703 fix(migration-041): use 'manual' doc_type instead of config_guide (CHECK constraint) 2026-04-20 23:17:41 +02:00
Rene Fichtmueller
dfadc8eb4e fix: add certifications column to switches table (migration 039a) 2026-04-20 23:16:52 +02:00
Rene Fichtmueller
5737ae0362 ui: update Finder quick examples to use actual seeded switch models 
Replace placeholder models (93180YC-FX3, 7280R3A, QFX5120 — not in DB) with
the real seeded switches: N9K-C9364C, 93600CD-GX, 7060CX2-32S, QFX5130-32CD, SN5600
 2026-04-20 23:00:36 +02:00
Rene Fichtmueller
ab059c2fd1 fix(community-issues): scrapeTransceiverCompatIssues falls back to ports_config when no compat entries 2026-04-20 23:00:00 +02:00
Rene Fichtmueller
8cf19e9b78 data: migration 041 — seed switch datasheets (Cisco, Arista, Juniper, NVIDIA) 
Seeds known-good official datasheet PDFs and config guide URLs for the 24 initial
seeded switches. Directly visible in switch detail panel → Datasheets & Manuals.
 2026-04-20 22:58:18 +02:00
Rene Fichtmueller
21f5250353 feat: switch facts — migration 040 seeds power/weight/certifications + dashboard shows them 
- Migration 040: seeds rack_units, typical_power_w, max_power_w, weight_kg, certifications
  for 23 initial switches (Cisco Nexus/Catalyst, Arista, Juniper, NVIDIA, Edgecore/Celestica/Asterfusion)
- Dashboard: Specifications section now shows Typical Power, Weight, Certifications (colored pills)
 2026-04-20 22:56:53 +02:00
Rene Fichtmueller
307a5ea38a chore: update changelog for 2026-04-20 switch image + compat features 2026-04-20 22:53:24 +02:00
Rene Fichtmueller
c0cd0dc1ca feat: compatibility panel — verification_method, competitor prices, spec-match collapsible 
- API: getCompatibleTransceivers() returns verification_method, orders vendor_compat first
- Dashboard: Flexoptix section splits vendor-tested vs spec-match (collapsed)
- Dashboard: Competitor section shows vendor-tested with prices, spec-match as chips
 2026-04-20 22:52:49 +02:00
Rene Fichtmueller
4bf5c95824 feat: Flexoptix compatibility scraper + transceiver issue scanner 
- Add flexoptix-compat.ts: maps switch models to compatible Flexoptix transceivers
  via search API (vendor_compat) with form-factor fallback (spec_match)
  Scheduled daily at 09:00 UTC as scrape:compat:flexoptix
- Enhance community-issues.ts: add vendor advisory sources (Cisco Field Notices,
  Juniper KB, SONiC GitHub Issues) + new scrapeTransceiverCompatIssues() that
  searches for switch+transceiver combination problems specifically
- Scheduler: 59 schedules, 78 workers
 2026-04-20 22:50:57 +02:00
Rene Fichtmueller
a0a7a97d83 feat: switch image fetcher + og:image scheduler job + dashboard thumbnail column 
- Add switch-image-fetcher.ts: og:image-based image discovery for all 86 seeded switches
  (covers Cisco, Arista, Juniper, NVIDIA, Edgecore, Celestica, Asterfusion, Dell,
   HPE/Aruba, Huawei, Nokia, Extreme, MikroTik, Ubiquiti, FS.COM, Supermicro)
- Wire fetchSwitchImages() into scheduler as scrape:images:switches (daily 08:30 UTC)
- Dashboard: add 48px thumbnail column to switch table (lazy img with gear icon fallback)
 2026-04-20 22:44:08 +02:00
Rene Fichtmueller
aa91798e8d fix(vcelink): resolve TS 5.9 narrowing quirk with explicit cast in dead code 
price?: number narrowing via typeof/!== undefined does not work for
arithmetic comparisons in TypeScript 5.9 dead code paths; use 'as number'
cast to keep the dead code compilable while the early-return guard above
prevents runtime execution entirely.
 2026-04-20 22:18:13 +02:00
Rene Fichtmueller
1aba912a15 fix(scrapers): fix ATGBics theme migration, NADDOD URL, disable VCELink 
- ATGBics: update HTML parser from old card--product theme to new
  card__info theme (Shopify template changed April 2026); name now
  extracted from href link text instead of aria-label
- NADDOD: correct ensureVendor shop URL from /collections/transceivers
  (404) to /collection/optical-transceivers
- VCELink: disable scraper — site pivoted from optical transceivers to
  audio/video/cable products; all collection URLs return 404
 2026-04-20 22:11:24 +02:00
Rene Fichtmueller
ca943f1f86 ui: comprehensive DEMO/MODELL tagging across all dashboard sections with synthetic data 
- Stock tab nav: ⚠DEMO badge
- Stock section subtitle: clarify prices=real vs. Lager/Verkauf=DEMO
- Stat cards: DE-Lager, Global-Lager, Nachlieferung labels tagged [DEMO]
- Recently Restocked header: DEMO DATA badge
- Stock detail lookup: [demo] inline on all warehouse/units_sold fields
- Top Sellers: already tagged (previous commit)
- Procurement > Reorder Signals: DEMO DATA banner (based on synthetic ABC data)
- Procurement > ABC Classification: DEMO DATA banner
- Hype Cycle: MODELL badge on header (Norton-Bass = mathematical estimate)
- Hype Cycle table: Adoption/Peak/To Plateau columns tagged [M] = Modell
- Hype Cycle legend: explains [M] vs real data
- Market Intelligence + Lifecycle Events: no tag (real scraped data)
 2026-04-20 21:52:10 +02:00
Rene Fichtmueller
9f3cd46f9c ui: mark Top Sellers widget data as DEMO (synthetic seed data, not real sales) 2026-04-20 21:44:33 +02:00
Rene Fichtmueller
0fb4850dfa fix: price-comparison SKU lookup — wrong column refs (so.stock_level, search_url_template) 2026-04-19 00:12:18 +02:00
Rene Fichtmueller
b0ed54f386 feat: register fiber24 + fibermall in index, move atgbics to fetch-only section 2026-04-18 22:50:52 +02:00
Rene Fichtmueller
cb5a587d7e feat: rewrite ATGBICS scraper — static HTML, correct collection handles, GBP cookie 
- Replaces Playwright with pure fetch() — static HTML has prices
- Correct collection handles (compatible-transceivers-sfpp-10g etc.)
- Cookie: cart_currency=GBP forces GBP pricing from any geo-IP
- Handles 35+ pages per category × 24 products = 840+ SFP+ products
- No IP-blocking with static HTML (Playwright was the trigger)
- Adds scripts/run-atgbics-mac.sh for Mac-side runner if needed
 2026-04-18 22:48:29 +02:00
Rene Fichtmueller
785a6731ab fix: fiber24 stockLevel on_request (was unknown — violated DB constraint) 2026-04-18 22:26:45 +02:00
Rene Fichtmueller
d4ad9f4641 fix: ShopFiber24 sitemap-based scraping + Fibermall image extraction 
ShopFiber24 (fiber24.ts):
- Complete rewrite: was using JS-rendered catalog (all prices = 0)
- New strategy: fetch sitemap_0.xml.gz → 310 product DE-URLs
- Each product page has Schema.org microdata: itemprop=price, sku, image
- Extracts: price (minPrice), SKU, image_url, name, specs
- Rate: 1 req/1.5s, no Playwright needed

FiberMall (fibermall.ts):
- Add imageUrl to Product interface
- Extract first fibermall.com/photo/*.jpg from product listing card
- Write image_url to transceivers table (has_image=true) on upsert
- SKU variants share parent product image
- 304 FiberMall transceivers will get images on next scraper run
 2026-04-18 22:20:57 +02:00