reorder_signals grew to 4.49M rows / 1.19GB — the compute job INSERTed a fresh
row per transceiver every 4h run but never deleted old ones (24h TTL filtered
them at read time via DISTINCT ON + expires_at, but they were never purged).
4.37M rows were already expired dead weight.
Fix: DELETE existing rows for a transceiver before inserting its new signal, so
the table holds exactly one (latest) row per transceiver. Cleaned up to 18,175
rows / 4.5MB (99.6% reclaimed, VACUUM FULL). Backup: reorder_signals_keep_bak_20260606.
Verified: re-running compute:reorder-signals keeps count stable at 18,175.
Root cause of the persistent sync:flexoptix-catalog HTTP 401: line 397 used
'?? null' which only coerces null/undefined. With FLEXOPTIX_API_TOKEN='' (empty
string set in .env), token stayed '' and line 485's 'token ?? getBearerToken()'
returned '' instead of performing the username/password login — sending an empty
'Bearer ' header that the products endpoint rejected with 401.
Fix: '|| null' coerces empty string to null so the bearer-login fallback fires.
Verified: sync now completes (username/password -> customer token -> products 200,
3 products/price/stock writes on limit=50). Credentials were correct all along.
The Research Robot panel showed only an LLM assessment (info, no action). Now:
API (research-robot.ts):
- GET enriches response with recommendations[] computed from live pgboss state:
classifies each persistently-failing job (auth/401, network, no-handler, other)
into severity + concrete advice + offered actions.
- POST /action {action,job}: dispatch (enqueue one run), pause (remove from
schedule with backup to research_robot_paused_schedules), resume (restore).
All validated against the pgboss.queue whitelist.
Dashboard:
- Renders each recommendation as a card with severity colour, cause, last error,
and action buttons (Jetzt auslösen / Pausieren / Fortsetzen / Token-Anleitung).
- Verified: sync:flexoptix-catalog -> critical auth (HTTP 401), offers
token-help + pause. Dispatch/pause/resume roundtrip tested green.
The 30d-vs-60d price momentum aggregated AVG/median across whatever SKUs
happened to be in a speed/form-factor bucket each period. New expensive SKUs
entering the catalog (NVIDIA switches at 30k USD, AOC cables) faked huge jumps
— 400G OSFP showed +151% when matched-SKU reality was 0%.
Now: compute per-transceiver median price in each period, keep only SKUs present
in BOTH periods (>=2 obs each), report the median of per-SKU pct deltas. Also
excludes non-transceiver form factors, AOC/DAC cables, switch SKUs, price>15k,
and anomalous observations. Result: 400G OSFP +151%->0%, signals 21->8, and the
ones that remain (NVIDIA MFA7U10 +84% same-SKU) are genuine price moves.
Adds parseWarehouseStock() to decode the HTML-entity-encoded warehouse_stock JSON
(us/nl/sg/cn per-region array). When the static page has warehouse data, writes:
warehouse_de_qty ← nl (EU-closest warehouse)
warehouse_global_qty ← sum(us+nl+sg+cn), or falls back to quantity_available
stock_confidence ← 3 (L3) when warehouse breakdown available, else 2
Note: per-warehouse quantities require JS execution to populate (API-loaded);
static HTML has [0,0] placeholders. The fallback ensures NADDOD global totals
appear in the competitor-by-tech dashboard comparison.
Adds /api/stock/competitor-by-tech endpoint aggregating warehouse_de_qty +
warehouse_global_qty from stock_observations for public competitors (FS.COM
etc.) per technology class. Dashboard velocity table gets two new columns
FS.COM DE + FS.COM Global with traffic-light coloring vs. monthly demand.
Clicking any signal card opens a modal with a 180-day SVG line chart
per source vendor (multi-line, colour-coded), x-axis dates, y-axis price,
current best price summary. Uses existing /api/price-history/:id endpoint.
No external chart library — pure inline SVG.
Group by part_number instead of transceiver_id (eliminates OEM duplicate rows).
Use PERCENTILE_CONT median instead of AVG to reduce single-outlier impact.
Add CV-filter (stddev/avg <= 0.35 over 2x window) to exclude high-variance
sources like Mouser quantity-tier pricing that produces artificial swings.
Blog LLM client probes BLOG_OLLAMA_URL (primary, WireGuard tunnel to Mac
Studio loopback Ollama) and falls back to BLOG_OLLAMA_URL_FALLBACK
(Cloudflare tunnel) when the primary transport is unreachable. Re-probed
at startup and every 60s; prefers primary when available. Both tunnels
terminate on the Mac loopback over independent transports, so the blog
keeps reaching fo-blog regardless of which transport drops.
Blog auto-discovery + generation now use BLOG_OLLAMA_URL (-> Mac Studio
192.168.178.213:11434 over the Erik<->home WireGuard tunnel), falling
back to OLLAMA_URL. Search/embeddings stay on Erik-local OLLAMA_URL
(nomic-embed-text). Fixes blog model not-found after OLLAMA_URL was
repointed to Erik-local for the search fix.
- sql/108: normalize form_factor across all vendors (SFP-Plus → SFP+, etc.)
and round speed_gbps for consistent matching
- sql/109: document 30→90 day matcher window change
- robots/catalog-reconcile.ts: new bulk-reconcile robot — matches all
Flexoptix products against all competitors without 30-day time limit
- scheduler.ts: register catalog:reconcile job (monthly + on-demand),
fix nightly matcher 30→90 day window, UPPER() form_factor matching,
ROUND() speed_gbps matching
Fixes: ATGBICS/NADDOD/10Gtek/ShopFiber24 had 0 Flexoptix equivalences
due to stale price_observations being filtered out. Expected coverage
improvement: 22% → 45-60% after first reconcile run.
CODEX-TASK-flexoptix-reference-matching.md — comprehensive plan to fix
zero-match gap for ATGBICS/NADDOD/10Gtek/ShopFiber24 (8.260+ products
with 0 Flexoptix equivalences).
Root cause: 30-day price_observation window excludes vendors whose
scrapers ran >30 days ago. Solution: catalog-reconcile robot (full
bulk match, no time limit), form_factor normalization (SQL 108),
30→90 day window fix in nightly matcher, on-demand API endpoint.
Expected: coverage from 22% → 45-60% after one reconcile run.
- index-pi.ts: removed Playwright scrapers (FS.COM, eBay enricher, switch assets)
added NADDOD (fetch-based, benefits from residential IP)
now 32 fetch-only queues safe for ARM/Pi without Chromium
- index-fs-only.ts: new dedicated FS.COM + NADDOD worker for Erik
routes through Pi SOCKS5 via PROXY_URLS=socks5://10.10.0.6:1080
Crawlee ProxyConfiguration automatically applies to Playwright crawler
- pi-scraper-setup.sh: removed inline index-pi.ts override (repo version now authoritative)
- CODEX-TASK-pi-scraper-deploy.md: full 9-step Codex spec for Pi fleet setup
covers WireGuard keypair, Erik peer config, setup script, ecosystem.config.js
- CODEX-TASK-zero-manual-review.md: deterministic equivalence matcher spec
