NADDOD uses LD+JSON for pricing (Astro/Shopify structure):
{"offers":{"price":"731.00","priceCurrency":"USD",...}}
Old regex (/US$\s*.../) never matched → all 132 price obs were lucky
text matches, not systematic. Now: parse all ld+json blocks first,
fall back to regex.
Also broaden sitemap URL regex to capture new-style URLs without .html:
/products/nvidia-networking/102612 (was being missed)
FS.com changed their HTML structure; compound class names are gone.
Current layout (verified 2026-05-06):
<div class="no_tax">5,10 € ohne MwSt.</div> ← B2B net price (preferred)
<div class="price">6,07 €</div> ← gross fallback
<div class="standard_price">6,07 €</div> ← gross fallback
Old selectors ([class*='price-value'] etc.) matched nothing → all prices
stored as €? null. New .no_tax first gives us the correct net/B2B price.
Previously always sliced first 600 URLs from sitemap, missing 6700+ products.
Now stores offset in naddod-cursor.json, advances by 600 per run with wrap-around.
Full sitemap coverage in ~13 runs (26h). Also adds TIP_STORAGE_DIR env support.
Static HTML collection pages return wrong results (all redirect to same 9 products).
Switch to /collections/{handle}/products.json?limit=250&page=N API which is:
- Reliable JSON (no HTML parsing)
- Correct per-collection product lists
- Clean pagination (stop at < limit results)
- Covers 11 key transceiver collections (1G, 10G, 25G, 40G, 100G, 400G)
- upsertPriceObservation: insert new observation if last one is >7 days old,
even when price (content_hash) hasn't changed — keeps timeseries data fresh
- ATGBICS: detect Shopify catalog wrap-around by tracking per-category seen URLs;
stop pagination when all products on a page were already seen in a prior page
- ATGBICS: improve hasNextPage to match &page=N anchored in href params
- scheduler: patch boss.schedule() to call createQueue() first (idempotent),
fixing FK constraint errors after DB reset — no need to touch 277 call sites
- index: registerWorkers() before registerSchedules() since boss.work() must
register handlers before schedules fire
- dashboard: fix switchBlogLlm() to use api() helper (adds Bearer auth token)
instead of raw fetch() which was returning 401 Unauthorized
Add 59 Juniper OEM transceiver PIDs (SFP/SFP+/SFP28/QSFP+/QSFP28/
QSFP56/QSFP-DD/OSFP + DAC/AOC) to seed the transceivers table.
Register scrape:catalog:juniper-oem in scheduler (daily 04:15).
Fix BlueOptics scraper: force HTTP/1.1 via Node.js https.get() to
bypass server bug where HTTP/2 returns empty response body. Also
update catalog path from /transceivers/ to /Transceivers_1.
- cisco-tmg.ts: upsert Cisco OEM transceivers from TMG API instead of
SELECT-only. Parsers for formFactor/speed/reach/fiberType/tempRange.
Fixes market_status ('EOL') + temp_range ('COM'/'IND') check constraints.
- arista-oem.ts: seed scraper for 69 Arista OEM PIDs (1G→800G,
SFP/SFP28/QSFP+/QSFP28/QSFP-DD/OSFP/QSFP-DD800) with full specs.
- scheduler.ts: daily arista-oem seed at 04:00 UTC
- sh -lc replaced with bash to avoid dash/profile.d syntax errors
- runCommand errors now caught in local provider path
- stdout/stderr extracted from error object and returned as JSON
- No more HTTP 500 on script failure
- Add ids to Local Train buttons (sl-local-btn-tip, sl-local-btn-blog)
- Add _slLocalReady flag updated after status load
- _updateLocalTrainButtons() enables/disables buttons based on local.ready
- startSelflearningTrain() guards local provider with early check + helpful message
- Status banner shows checkmarks for RunPod/HF/Local config state
- API call: +&vendor=Flexoptix filter
- Section renamed to 'Flexoptix-Lösungen' with favicon
- Each row shows 'Shop ↗' link directly to flexoptix.net search
- 'Show all' button labeled 'Alle X Flexoptix-{FF}-Produkte'
- Primary button: flexoptix.net with form factor filter
- Empty state: friendly message when no Flexoptix products yet
Standards tab now has two sub-tabs:
- Standards: existing table (default)
- Formfaktoren: full-page grid with search/family/status filters
Form factor detail panel (openFormFactorDetail) rebuilt as async:
- Mini inline hype cycle curve SVG with dot at exact position
- Buy signal per form factor (BUY_NOW / CONSIDER / WAIT / HOLD / AVOID)
- Typed use-case bullets per form factor (20 mapped)
- Top-10 transceiver mini-list (live fetch from /api/transceivers?form_factor=)
- Clicking a transceiver row opens its full detail panel
- Supersedes chain as clickable badges (navigate to that form factor)
- flexoptix.net search link
FF_HYPE data: curated hype cycle positions for all 20 form factors
FF_USE_CASES: tailored use case lists per form factor
filterFormFactors: now also filters by search text + status dropdown
- Migration 100: adds `description` column to standards (plain-language
DE·EN for non-technical colleagues), fills all 63 standards incl.
complete 200G tier (SR4/DR4/FR4/LR4/ER4/CR4), copper DAC variants,
PON family (GPON/XG-PON1/NG-PON2/25G-PON), 1.6T emerging standard
- Migration 101: new form_factors table — 20 entries covering SFP family
(SFP→SFP112), QSFP family (QSFP+→QSFP-DD800), OSFP family (OSFP→OSFP224),
CFP family, legacy XFP/CXP with full_name, speed, channels, status,
supersedes chain, and bilingual plain-language descriptions
- GET /api/form-factors — new endpoint, returns all form factors with
transceiver_count join
- GET /api/form-factors/:name — single form factor detail
Dashboard Standards tab:
- DB description shown as subtitle in standards table rows
- Full DE + EN description in standard detail panel
- New Form Factors grid section with status badges, speed, channel info,
family color coding, supersedes chain
- openFormFactorDetail() panel with full specs + transceiver link
- Search extended to match description + notes fields
JWT auth + RLS is the real security boundary. IP check was blocking legitimate
dashboard users accessing via Cloudflare tunnel (X-Forwarded-For = real user IP,
not 127.0.0.1). Added /api/internal/demand/stock-analysis: demand × live stock
combined analysis with reorder_urgency, coverage_days_de, momentum_ratio.
Dashboard: new Demand × Live Stock Analyse panel with critical/low/ok/overstock chips.
- Migration 099: flexoptix_internal_demand table with RLS + v_demand_by_speed view
- Import script: AES-256-CBC decrypt → parse 8585 SKUs → upsert with velocity class
- 279 SKUs cross-referenced to transceiver catalog; 1288 with real demand data
- New /api/internal/demand/* routes (by-speed, velocity, hype-weights, forecast-input)
— protected by JWT auth + localhost/LAN IP restriction middleware
- Forecast engine calibrated with real Flexoptix run-rates (demand_calibrated flag)
- Dashboard: real Flexoptix Sales Velocity panel replaces DEMO DATA in Warehouse tab
with momentum indicators, velocity class breakdown, trend arrows
- Security: data stays on private server; RLS enforces is_internal=TRUE at DB layer
cdn.cookielaw.org logos appear as the largest DOM image on Dell/Extreme
product pages when the cookie consent overlay is present. Added to both
GENERIC_IMAGE_PATTERNS (isGenericImage filter) and img fallback skipPattern
so the next-largest actual product image can be found.
