One-time companion to commit e7299ac7 (spec-matcher.ts now inserts pending,
not auto_approved). Re-scores the pre-existing spec-basis auto_approved rows
against the rigorous evaluateEquivalenceResearch() criteria: 972 (67%)
genuinely hold up and stay auto_approved, 483 (33%) get demoted -- some
permanently (technical mismatch / no recent competitor price), some flagged
for re-research in 21 days (missing wavelength/fiber/reach evidence, not a
wrong match). Backup in spec_matcher_reclassify_backup_20260717.
Set-based port of scheduler.ts evaluateEquivalenceResearch(), applied once
live to every pending row with re_research_due_at IS NULL (the root cause
this pairs with a code fix for, previous commit). Result: 41,963 approved,
187,599 rejected as technical mismatch, 57,683 rejected as insufficient
evidence (re-research in 21 days), 2,436 rejected for no recent competitor
price. Backup of pre-triage state in
equivalence_pending_triage_backup_20260713.
126 only guards writes to compatibility; a later transceivers.speed_gbps update
(e.g. daily enrich-modulation-fec.sh self-heal) never re-validated existing
compatible spec_match rows. Found live 2026-07-12: 189 rows had drifted
physically-impossible again. One-time remediation (189 rows -> incompatible,
backed up to compat_speed_drift_backup_20260712) + AFTER UPDATE OF speed_gbps
trigger on transceivers closes the gap for good. check_switch_quality() passes.
Root cause of the equivalence review-queue backlog: spec-matcher read
transceivers.wavelengths (text), but the enricher wrote wavelength_tx_nm
(int) — 27,908 rows had the wavelength in the wrong column, and tx_nm
itself defaulted incorrectly to 1310nm in ~22% of disagreements with the
authoritative part-derived value.
sql/131: backfill + sync trigger from wavelength_tx_nm into wavelengths.
sql/132: correct authority (bare-number wavelengths, not tx_nm) + format
normalization + valid promotion on the full spec-matcher gate.
sql/133: one-directional reach-tier substitution (higher reach may satisfy
a lower-reach requirement, never the reverse; ZR/coherent excluded — real
link-budget/receiver-overload risk, not a simple tier comparison). Tagged
as its own status (compatible_superset), not merged into auto_approved.
- Replace plaintext tip (transceiver_db) and llm_gateway DB passwords with
env-var references guarded by fail-fast checks, across 15 scripts + sql runbook.
Covers quoted exports, unquoted psql calls, python env dicts and
env-or-hardcoded-default fallbacks.
- Untrack training-data/*.jsonl (~48MB, kept on disk) and gitignore *.jsonl.
- Add .security-scan-allowlist for legit false positives: vendor name flexoptix,
own homelab infra, sibling-project codenames in the sync journal, env-resolved
DB URIs, example placeholders, ML tokenizer tokens.
- Pre-push leak scanner goes from 397 to 0 findings, green without --no-verify.
Follow-up (not in this commit): rotate the leaked values (llm_gateway first,
tip fleet-wide coordinated); git history still holds the old passwords.
The Adapter (sql/129) seeded all vendor sources as erik-safe, but TIP data is
fetched by the 3 residential Pi 5 workers (tashi/portia/ester), all now
Playwright-capable — not Erik (datacenter IP, blocked by FS.com/ATGBICS).
14 sources -> pi-fetch (fixes gbics/sfpcables never-dispatching under erik-safe).
Config only; loop stays in Plan-Mode. 10gtek stays proxmox-heavy.
Close the missing feedback loop. Detection (v_scraper_health, v_data_quality_alerts,
check_switch_quality) was accurate but inert; dispatch was blind fixed-cadence
(boss.send() is never called from any health path). tip-dqo is a separate, read-mostly
control plane that closes DETECT -> PRIORITIZE -> DISPATCH -> VERIFY -> RECHECK against
observation-table freshness (never pg-boss job state), with DB-enforced guardrails.
- sql/129: source_registry (signal->queue map + per-source SLA + guardrail contract +
global kill-switch), self_heal_dispatch append-only ledger, scraper_run_log,
acknowledged_exceptions, v_dqo_targets ranked view.
- orchestrator/: control-loop, self-heal-guards (pure, unit-tested), escalation-sink
(ntfy + no-op fallback), index entrypoint. Reuses the verification-robots dispatch
idiom (extracted shared dispatchQueues) and the quality views as-is.
- Guardrails: kill-switch, plan-mode default, per-source cooldown/budget/backoff/
circuit-breaker, load-guard, erik-safe profile cap, delta-gated escalation. Root-cause
routing escalates code_fix/blocked (e.g. ATGBICS silent stock-write break) to a human
instead of looping retries; only staleness is auto-dispatched.
- security: remove hardcoded 'tip_dev_2026' DB password fallback across 6 sites ->
fail-closed requireDbPassword() / dbConnectionString().
SELF_HEAL_PLAN_ONLY defaults true (dispatches nothing; writes plan rows for review).
Verified against the live DB: detects ATGBICS-stock dead/54d -> escalate, Flexoptix-stock
stale/30d -> plan-dispatch; guards + ledger correct; 8/8 guard unit tests pass.
- fn_strip_double_url_prefix + one-time fix for 436 rows whose scraped URLs had a
duplicated host prefix (https://host.comhttps://host.com/x) -> unfetchable.
Ingest guard trg_transceiver_url_normalize prevents recurrence.
- v_datasheet_enrichment_worklist: pipeline input listing every switch/transceiver
needing a field filled from source, with its best source URL + type.
One article (The Next Platform) had published_at 2026-07-15 while scraped
2026-06-19 -- an article cannot be published after it was scraped. One-time
clamp to the scrape date + BEFORE INSERT/UPDATE trigger to prevent recurrence.
- trg_compat_speed_guard: BEFORE INSERT/UPDATE on compatibility forces
algorithmic (spec_match/community) rows whose transceiver is faster than the
switch max port speed to status=incompatible. Curated rows never touched.
Makes the "no physically-impossible compatibility" invariant structural, so
it survives a scraper regression.
- check_switch_quality(): on-demand proof; CI gate is
`select bool_and(pass) from check_switch_quality() where enforced` (must be
true). Data-source gaps (coding coverage, speed<=0, datasheet) tracked as
enforced=false so they do not fail the gate before the data exists.
Root cause: the form-factor fallback in flexoptix-compat matched transceivers by
form factor only, ignoring speed, so a 400G OSFP port matched every OSFP module
incl. 1.6T (5686+ impossible rows across 266 switches).
- flexoptix-compat: fallback now filters transceiver speed <= switch max port
speed (derived from ports_config + stored max_speed_gbps). Prevents regeneration.
- sql/125: evidence-based effective-ceiling (stored/ports/curated) views;
flag 5811 impossible spec_match rows status=incompatible (curated rows never
touched, understated switches never false-flagged); backfill max_speed_gbps for
327 switches from ports_config; re-derive speed for 306 speed<=0 transceivers
(form-factor capped, high-confidence only). v_switch_quality_alerts is the task
list; v_switch_understated_ports surfaces switches needing datasheet enrichment.
Reading packages/scraper showed QSFPTEK/ATGBICS populate quantity_available
100%; they simply lack FS.com-only units_sold/warehouse-split. The 123 alert
keyed on those FS.com fields and false-positived QSFPTEK. Rekey to
quantity_available OR warehouse_qty so the alert list drains to empty; surface
pct_qty_available/pct_vendor_ts in v_stock_completeness.
FS.com renders per-warehouse availability as a bare word-date on each line
("2 Stk. im DE-Lager, 8. Juli 2026 Lieferbar"). The old regex only matched a
"Lieferung:"/"erwartet:" prefix or numeric dd.mm.yyyy, so warehouse_de/global
delivery_date + backorder_estimated_date were 0% populated across 85k rows, and
lead_time_days was never written at all.
- new fs-com-parse.ts: pure, unit-tested parsers (13 node:test cases).
- extractDeliveryDate finds the bare word/numeric date per warehouse, bounded to
its own line so a date never bleeds to a neighbouring warehouse.
- computeLeadTimeDays = min(delivery_date - today), floored at 0; threaded into
upsertStockObservation (now writes lead_time_days) and upsertPriceObservation.
- parseGermanQty: K/M abbreviations now treat '.' as a decimal ("4.8K"=4800,
"1.4M"=1.4M) instead of stripping it (was "1.4M"->14,000,000 — the outlier).
- units_sold sanity gate at ingest (FS_MAX_UNITS_SOLD, default 2,000,000).
- sql/122: purge impossible historical units_sold, re-document columns as collected.
sql/119: product_type enum (module|dac|aoc|aec|switch|nic|accessory) computed by
trigger trg_transceivers_biu from part_number/category/reach on every insert/update.
Backfilled 41008 rows (36284 module, 2791 dac, 1835 aoc, 29 aec, 27 accessory,
26 switch, 16 nic). Cables and switches/NICs no longer counted as modules;
BASE-T/RJ45 copper modules protected from cable demotion; AOC/DAC keyword beats
the optical-token guard. Consumers filter product_type=module.
sql/120: conservative form_factor recovery. Corrects only provably-wrong rows
(form_factors.max_speed < speed_gbps) via an explicit title token; 87 fixed
(e.g. 400G cables mislabelled SFP+ to QSFP-DD), 947 opaque rows left untouched
(never guessed). Scraped original preserved in new column form_factor_raw.
sql/121: document lead_time_days as not-collected (0 percent populated on all
time-series tables) so no consumer fabricates a default.
scraper/utils/db.ts: note that product_type/form_factor are trigger-maintained.
- sql/108: normalize form_factor across all vendors (SFP-Plus → SFP+, etc.)
and round speed_gbps for consistent matching
- sql/109: document 30→90 day matcher window change
- robots/catalog-reconcile.ts: new bulk-reconcile robot — matches all
Flexoptix products against all competitors without 30-day time limit
- scheduler.ts: register catalog:reconcile job (monthly + on-demand),
fix nightly matcher 30→90 day window, UPPER() form_factor matching,
ROUND() speed_gbps matching
Fixes: ATGBICS/NADDOD/10Gtek/ShopFiber24 had 0 Flexoptix equivalences
due to stale price_observations being filtered out. Expected coverage
improvement: 22% → 45-60% after first reconcile run.
- Migration 100: adds `description` column to standards (plain-language
DE·EN for non-technical colleagues), fills all 63 standards incl.
complete 200G tier (SR4/DR4/FR4/LR4/ER4/CR4), copper DAC variants,
PON family (GPON/XG-PON1/NG-PON2/25G-PON), 1.6T emerging standard
- Migration 101: new form_factors table — 20 entries covering SFP family
(SFP→SFP112), QSFP family (QSFP+→QSFP-DD800), OSFP family (OSFP→OSFP224),
CFP family, legacy XFP/CXP with full_name, speed, channels, status,
supersedes chain, and bilingual plain-language descriptions
- GET /api/form-factors — new endpoint, returns all form factors with
transceiver_count join
- GET /api/form-factors/:name — single form factor detail
Dashboard Standards tab:
- DB description shown as subtitle in standards table rows
- Full DE + EN description in standard detail panel
- New Form Factors grid section with status badges, speed, channel info,
family color coding, supersedes chain
- openFormFactorDetail() panel with full specs + transceiver link
- Search extended to match description + notes fields
- Migration 099: flexoptix_internal_demand table with RLS + v_demand_by_speed view
- Import script: AES-256-CBC decrypt → parse 8585 SKUs → upsert with velocity class
- 279 SKUs cross-referenced to transceiver catalog; 1288 with real demand data
- New /api/internal/demand/* routes (by-speed, velocity, hype-weights, forecast-input)
— protected by JWT auth + localhost/LAN IP restriction middleware
- Forecast engine calibrated with real Flexoptix run-rates (demand_calibrated flag)
- Dashboard: real Flexoptix Sales Velocity panel replaces DEMO DATA in Warehouse tab
with momentum indicators, velocity class breakdown, trend arrows
- Security: data stays on private server; RLS enforces is_internal=TRUE at DB layer
7 models: AS7312-54X, AS7312-54XS, AS7326-56X, AS7716-32X, AS7816-64X,
AS9716-32D, AS7512-32X. All from SONiC HCL Accton/Edgecore vendor.
Sources: edge-core.com official WP CDN + stordis.com + epsglobal.com.
Also adds scripts/apply-pending-migrations.sh for Erik DB catchup.
A9K-4HG-FLEX-FC, A9K-8X100G-LB-TR, A9K-4X100GE base: all share identical
hardware with previously-covered -TR/-SE siblings. FC/SE/TR are Cisco
software license suffixes on the same physical line card. Coverage: 595 → 598.
