catalog-reconcile.ts inserted new pending equivalences with
re_research_due_at always NULL. The daily maintenance:re-research-equivalences
job only evaluates rows where re_research_due_at IS NOT NULL AND <= NOW() -- so
the job that was built to auto-approve/reject the queue never saw fresh
pending rows. Backlog reached 289,681 rows before anyone noticed (found live
2026-07-13, one-time triage in sql/139: 41,963 approved, 247,718 rejected,
57,683 of those flagged for re-research once missing specs get enriched).
Fix: pending inserts now get re_research_due_at=NOW() so the job drains them
going forward. Also: reject branch now distinguishes data-gap rejects
(missing wavelength/fiber/reach evidence -> re-checked in 21 days once a
scraper fills the gap) from confirmed-mismatch/no-price/low-confidence
rejects (permanent, re_research_due_at stays NULL -- nothing changes on a
re-check against the same stored specs).
Add parseStandardName(title) to derive standard from product name
(SR, LR, ER, DR4, FR4, ZR+, etc.) since the V2 API has no protocol field.
Apply to all future syncs. Companion SQL backfill added 1,072 more
standard_names (total 2,281 / 29.8% FX coverage, up from 15.8%).
Phase 1 — DB: adds dwdm_channel, cwdm_channel, cdr_type, protocol_family
columns to transceivers (migration 134). Backfill from Flexoptix notes:
1,209 standard_names (+1,579%), 1,177 cdr_types, 2,498 wdm_types,
7,918 protocol_families.
Phase 2 — API sync: flexoptix-api-sync.ts now writes standard_name,
wdm_type, cdr_type, cdr_support, protocol_family on every sync.
Phase 3 — Scoring: catalog-reconcile.ts adds hard rejects for
protocol_family/temp_range/wdm_type mismatches, -25pt standard_name
mismatch penalty, CDR type scoring, and requires fx.standard_name to
be set before auto_approve — closes the 77k false-positive gap.
Parse Flexoptix API compat entries (compatible_to_vendor + original_part_number)
and batch-upsert into flexoptix_product_map during each catalog sync.
- Fix extractCompatibility() to recognise compatibleToVendor key from API
- Add batchUpsertProductMap(): 500-row batch INSERTs with ON CONFLICT update
- Add writeProductMapEntries() per product, called after price/stock writes
- Track mapWrites in FlexoptixSyncResult and log in completion message
First sync: 9254 upserts → 2903 unique rows, 95 vendors, 343 Flexoptix SKUs.
Example: CWDM-SFP10G-1330 (Cisco) → P.1696.10.D.I (EUR 150.77, SFP+ 10G).
- Add two-phase sync: bulk catalog fetch then targeted batch enrichment
(50 SKUs/call, ~10 extra calls) to retrieve self_configure_price field
which is absent from paginated responses
- Write self_configure_price as flexoptix-selfconfigure marketplace rows
in price_observations (343/346 products enriched on first run)
- Add list_price support (flexoptix-msrp marketplace) for future use
- Use API-provided type field (Transceiver/Cables/Accessories) in
categoryFor() instead of inferring from title alone
- Add productType field to CatalogProduct interface
Two bugs blocked all Flexoptix stock_observations:
1. db.ts: compatible_brands null param needs ::text[] cast; price_currency needs
::bpchar cast in stock_observations INSERT
2. flexoptix-api-sync.ts: UPDATE transceivers used $1 in WHERE IS NOT NULL clause
without explicit type cast — PostgreSQL cant infer type of untyped null params
in WHERE context. Fixed with $1::text / $2::text casts.
Result: 346 stock observations written on first run after fix.
upsertStockObservation dedup guard was preventing writes for unchanged
stock data (e.g. ATGBICS all-OOS for 55d), causing v_scraper_health to
mark sources as dead even when scrapers run normally. Skip dedup when
last observation is >7 days old so health view stays current.
- escalation-sink: add NTFY_TOKEN Bearer auth header support
- self-heal-guards: pass max_load from GuardConfig to isLoadAcceptable
- control-loop: include MAX_LOAD in guardConfig so DQO_MAX_LOAD env is respected
Close the missing feedback loop. Detection (v_scraper_health, v_data_quality_alerts,
check_switch_quality) was accurate but inert; dispatch was blind fixed-cadence
(boss.send() is never called from any health path). tip-dqo is a separate, read-mostly
control plane that closes DETECT -> PRIORITIZE -> DISPATCH -> VERIFY -> RECHECK against
observation-table freshness (never pg-boss job state), with DB-enforced guardrails.
- sql/129: source_registry (signal->queue map + per-source SLA + guardrail contract +
global kill-switch), self_heal_dispatch append-only ledger, scraper_run_log,
acknowledged_exceptions, v_dqo_targets ranked view.
- orchestrator/: control-loop, self-heal-guards (pure, unit-tested), escalation-sink
(ntfy + no-op fallback), index entrypoint. Reuses the verification-robots dispatch
idiom (extracted shared dispatchQueues) and the quality views as-is.
- Guardrails: kill-switch, plan-mode default, per-source cooldown/budget/backoff/
circuit-breaker, load-guard, erik-safe profile cap, delta-gated escalation. Root-cause
routing escalates code_fix/blocked (e.g. ATGBICS silent stock-write break) to a human
instead of looping retries; only staleness is auto-dispatched.
- security: remove hardcoded 'tip_dev_2026' DB password fallback across 6 sites ->
fail-closed requireDbPassword() / dbConnectionString().
SELF_HEAL_PLAN_ONLY defaults true (dispatches nothing; writes plan rows for review).
Verified against the live DB: detects ATGBICS-stock dead/54d -> escalate, Flexoptix-stock
stale/30d -> plan-dispatch; guards + ledger correct; 8/8 guard unit tests pass.
Root cause: the form-factor fallback in flexoptix-compat matched transceivers by
form factor only, ignoring speed, so a 400G OSFP port matched every OSFP module
incl. 1.6T (5686+ impossible rows across 266 switches).
- flexoptix-compat: fallback now filters transceiver speed <= switch max port
speed (derived from ports_config + stored max_speed_gbps). Prevents regeneration.
- sql/125: evidence-based effective-ceiling (stored/ports/curated) views;
flag 5811 impossible spec_match rows status=incompatible (curated rows never
touched, understated switches never false-flagged); backfill max_speed_gbps for
327 switches from ports_config; re-derive speed for 306 speed<=0 transceivers
(form-factor capped, high-confidence only). v_switch_quality_alerts is the task
list; v_switch_understated_ports surfaces switches needing datasheet enrichment.
FS.com renders per-warehouse availability as a bare word-date on each line
("2 Stk. im DE-Lager, 8. Juli 2026 Lieferbar"). The old regex only matched a
"Lieferung:"/"erwartet:" prefix or numeric dd.mm.yyyy, so warehouse_de/global
delivery_date + backorder_estimated_date were 0% populated across 85k rows, and
lead_time_days was never written at all.
- new fs-com-parse.ts: pure, unit-tested parsers (13 node:test cases).
- extractDeliveryDate finds the bare word/numeric date per warehouse, bounded to
its own line so a date never bleeds to a neighbouring warehouse.
- computeLeadTimeDays = min(delivery_date - today), floored at 0; threaded into
upsertStockObservation (now writes lead_time_days) and upsertPriceObservation.
- parseGermanQty: K/M abbreviations now treat '.' as a decimal ("4.8K"=4800,
"1.4M"=1.4M) instead of stripping it (was "1.4M"->14,000,000 — the outlier).
- units_sold sanity gate at ingest (FS_MAX_UNITS_SOLD, default 2,000,000).
- sql/122: purge impossible historical units_sold, re-document columns as collected.
sql/119: product_type enum (module|dac|aoc|aec|switch|nic|accessory) computed by
trigger trg_transceivers_biu from part_number/category/reach on every insert/update.
Backfilled 41008 rows (36284 module, 2791 dac, 1835 aoc, 29 aec, 27 accessory,
26 switch, 16 nic). Cables and switches/NICs no longer counted as modules;
BASE-T/RJ45 copper modules protected from cable demotion; AOC/DAC keyword beats
the optical-token guard. Consumers filter product_type=module.
sql/120: conservative form_factor recovery. Corrects only provably-wrong rows
(form_factors.max_speed < speed_gbps) via an explicit title token; 87 fixed
(e.g. 400G cables mislabelled SFP+ to QSFP-DD), 947 opaque rows left untouched
(never guessed). Scraped original preserved in new column form_factor_raw.
sql/121: document lead_time_days as not-collected (0 percent populated on all
time-series tables) so no consumer fabricates a default.
scraper/utils/db.ts: note that product_type/form_factor are trigger-maintained.
nominalSpeedFromSpecs missed '1.6T Ethernet' protocol entries (only matched 'G'),
returning 800 for twin-port 1.6T parts instead of 1600. Added Terabit handling +
guarded against 'Nx 800G' aggregate partial-matches. After re-deriving all 805
catalog-confirmed FX parts from their stored datasheet specs, DB now matches the
Flexoptix datasheet 100%: Form Factor 802/802, Speed 801/801, 0 deviations.
Root cause of unreliable speeds: the bulk Flexoptix products API returns NO speed
or form-factor field, so the catalog sync guessed them by parsing the product
NAME (e.g. 'FO-109010-CWDM' -> 100000G). Cryptic FX codes like 'O.164HG2.2.C:Sx'
are unparseable, producing garbage.
The detail enricher already pulls per-SKU specifications=1 (rate-limited) but only
wrote secondary fields. Now it also derives:
form_factor <- structured 'Form Factor' spec (authoritative datasheet value)
speed_gbps <- highest Ethernet rate in 'Supported Protocols', fallback to the
'Bandwidth' line-rate upper bound mapped to nominal
Both OVERWRITE the corrupt bulk values (COALESCE(spec, existing)). Never derived
from the product name. Verified: 100/100 freshly-enriched FX parts now have
physically-consistent form_factor/speed (0 contradictions), incl. uncrackable
codes correctly resolved to OSFP/800G, QSFP-DD800/800G etc.
reorder_signals grew to 4.49M rows / 1.19GB — the compute job INSERTed a fresh
row per transceiver every 4h run but never deleted old ones (24h TTL filtered
them at read time via DISTINCT ON + expires_at, but they were never purged).
4.37M rows were already expired dead weight.
Fix: DELETE existing rows for a transceiver before inserting its new signal, so
the table holds exactly one (latest) row per transceiver. Cleaned up to 18,175
rows / 4.5MB (99.6% reclaimed, VACUUM FULL). Backup: reorder_signals_keep_bak_20260606.
Verified: re-running compute:reorder-signals keeps count stable at 18,175.
Root cause of the persistent sync:flexoptix-catalog HTTP 401: line 397 used
'?? null' which only coerces null/undefined. With FLEXOPTIX_API_TOKEN='' (empty
string set in .env), token stayed '' and line 485's 'token ?? getBearerToken()'
returned '' instead of performing the username/password login — sending an empty
'Bearer ' header that the products endpoint rejected with 401.
Fix: '|| null' coerces empty string to null so the bearer-login fallback fires.
Verified: sync now completes (username/password -> customer token -> products 200,
3 products/price/stock writes on limit=50). Credentials were correct all along.
Adds parseWarehouseStock() to decode the HTML-entity-encoded warehouse_stock JSON
(us/nl/sg/cn per-region array). When the static page has warehouse data, writes:
warehouse_de_qty ← nl (EU-closest warehouse)
warehouse_global_qty ← sum(us+nl+sg+cn), or falls back to quantity_available
stock_confidence ← 3 (L3) when warehouse breakdown available, else 2
Note: per-warehouse quantities require JS execution to populate (API-loaded);
static HTML has [0,0] placeholders. The fallback ensures NADDOD global totals
appear in the competitor-by-tech dashboard comparison.
- DELETE 2133 rows from reorder_signals WHERE is_demo_data = true
- atgbics.ts: add upsertStockObservation (confidence=1, binary available
boolean from Shopify API; quantityAvailable 1/0 for in/out stock)
- optcore.ts: add upsertStockObservation (confidence=1, WooCommerce text
stock level parsed via parseStockLevel; quantityAvailable 1/0)
- Both scrapers already run every 2h on Erik scheduler
- FS.com: already captures full warehouse breakdown (DE+Global+backorder)
3x/day from Mac (02:00/10:00/18:00) at confidence=3 — no change needed
- QSFPTEK: already captures real quantities at confidence=2 — no change
- sfpcables/prolabs/wiitek: no meaningful stock signal, not modified
- sql/108: normalize form_factor across all vendors (SFP-Plus → SFP+, etc.)
and round speed_gbps for consistent matching
- sql/109: document 30→90 day matcher window change
- robots/catalog-reconcile.ts: new bulk-reconcile robot — matches all
Flexoptix products against all competitors without 30-day time limit
- scheduler.ts: register catalog:reconcile job (monthly + on-demand),
fix nightly matcher 30→90 day window, UPPER() form_factor matching,
ROUND() speed_gbps matching
Fixes: ATGBICS/NADDOD/10Gtek/ShopFiber24 had 0 Flexoptix equivalences
due to stale price_observations being filtered out. Expected coverage
improvement: 22% → 45-60% after first reconcile run.
- index-pi.ts: removed Playwright scrapers (FS.COM, eBay enricher, switch assets)
added NADDOD (fetch-based, benefits from residential IP)
now 32 fetch-only queues safe for ARM/Pi without Chromium
- index-fs-only.ts: new dedicated FS.COM + NADDOD worker for Erik
routes through Pi SOCKS5 via PROXY_URLS=socks5://10.10.0.6:1080
Crawlee ProxyConfiguration automatically applies to Playwright crawler
- pi-scraper-setup.sh: removed inline index-pi.ts override (repo version now authoritative)
- CODEX-TASK-pi-scraper-deploy.md: full 9-step Codex spec for Pi fleet setup
covers WireGuard keypair, Erik peer config, setup script, ecosystem.config.js
- CODEX-TASK-zero-manual-review.md: deterministic equivalence matcher spec
