SCRAPERS list used 'flexoptix-catalog' as DB lookup key but vendors.slug
is 'flexoptix' — no match → 0 records shown.
Fix: added dbSlug override field to SCRAPERS entries; lookup now uses
dbSlug || name so flexoptix-catalog/vendors/supported all map to
the correct 'flexoptix' slug in sourceMap.
Blog engine (fo-blog-pipeline.ts):
- Add STEP8b_REDUCTION: cuts article 25-35%, removes repeated concepts
- Add STEP8c_STYLE_LOCK: enforces tone consistency, fixes scope/OPM confusion,
removes inline SKUs from article flow
- Add Gold Standard 3 to calibration (Style B troubleshooting example 2026-04-04)
- Pipeline now 12 steps (was 10), version bumped to v4-reduction-stylelock
blog.ts:
- Wire STEP8b and STEP8c into pipeline between Kill-AI-Tone and QA Check
- Update progress tracking to 12 total steps
- Update pipeline_version to 'v4-reduction-stylelock'
flexoptix-catalog.ts:
- Fix contentHash call: pass object directly, not JSON.stringify(object)
db.ts:
- price_verified=true set in content_hash early-return path (no new observation)
- image_verified=true auto-set in findOrCreateScrapedTransceiver on INSERT/UPDATE
- findOrCreateScrapedTransceiver now sets image_verified=true when writing image_url
- upsertPriceObservation now sets price_verified=true on the transceiver after inserting price
- Both INSERT and UPDATE paths covered for image_verified sync
- Eliminates need for manual backfill after scraper runs
PostgreSQL max_connections was being exceeded (100/100).
- Limit pg-boss internal pool to 4 connections
- Added idle_in_transaction_session_timeout=30s to PostgreSQL config
- Already raised max_connections to 300 (container config)
System now stable at ~98/300 connections
Dead code leftover from STEP4_MASTER_DRAFT rewrite was sitting outside
any template literal, causing compilation failure. Removed duplicate
CONTEXT DATA RULES block and orphaned `{{OUTLINE}}`/`{{CONTEXT_DATA}}`
placeholders that were not wrapped in a string.
Core changes:
- HARD RULES rewritten: zero tolerance for ## headers, #### Scenario: patterns, bullet sections
- Gold article added as reference standard in STEP4_MASTER_DRAFT
- MANDATORY SECTIONS removed — replaced with continuous prose requirement
- STEP3_OUTLINE: now a flow plan (3-4 beats), not a section list
- STEP5_REALITY_INJECTION: no longer adds 'What Breaks' sections — injects into prose
- STEP9_QA_CHECK: format violations now primary HARD FAIL, above content checks
- DR4 wavelength fix: 1310nm = 0.35 dB/km (not 1550nm = 0.22 dB/km)
- Scope description fix: visual inspection tool ≠ loss measurement device
- Invented firmware version numbers now explicit HARD FAIL
token variable was undefined in loadCrawlerStatus() scope (only declared
inside IIFE auth guard, not globally). All API calls silently failed with
401. Fix: read token from localStorage at start of function, consistent
with getAuthHeaders() pattern used in all other load functions.
Replace static CSS ::after tooltips with JS-powered smart tooltips.
Tooltips now detect available space above/below and flip accordingly,
and clamp horizontally to viewport bounds. Hide on scroll.
Complete Pi scraper entry point covering all pricing, catalog, compat,
intelligence and prediction signal scrapers. Includes 5 new form-factor
coverage scrapers (comms-express, router-switch, multimode-inc,
optictransceiver, wiitek). Erik runs only API+DB, all scraping on Pis.
Add Comms-Express, Router-Switch.com, Multimode Inc, OpticTransceiver.com,
and Wiitek scrapers covering CFP2-DCO, CFP4, OSFP224, QSFP112, CXP, GBIC,
XENPAK, CSFP, SFP-DD, SFP56, QSFP56 and other previously-uncovered form
factors. Each scheduled every 8h. Worker registrations added to scheduler.
Also export db alias in utils/db.ts to fix eBay enricher + community scrapers
crashing with 'Cannot read properties of undefined (reading query)'.
- POST /api/auth/login: HMAC-SHA256 signed 7-day token, password from DASHBOARD_PASSWORD env
- GET /api/auth/verify: stateless token validation
- requireAuth middleware applied to all /api/* routes (except /api/health + /api/auth)
- /dashboard/login.html: dark TIP-themed login page with show/hide password toggle
- index.html: auth guard redirect to login + Authorization header on all api() calls
- No secrets in code — password stored in .env only
- downloadDocuments(): fetches PDFs from product_documents and documents tables
using curl, organises into switches/ transceivers/ whitepapers/ other/ subdirs
- Integrated into runNightlyNasSync() — runs after JSON exports
- rsync incremental — only new/changed files transferred
- NAS dir structure: /volume1/tip-data/datasheets/{switches,transceivers,whitepapers,other}
- max-filesize 50MB guard per file
- utils/logger.ts: minimal console-based logger (debug/info/warn/error)
used by community-issues and ebay-enricher scrapers
- scripts/pi-scraper-setup.sh: step 7 adds optional WireGuard setup
(pass WG_PRIVKEY + WG_ADDR env vars) — connects Pi to Erik for DB access
auto-detects dead ethernet and routes WG traffic via working interface
New scrapers (8):
- BlueOptics (EUR, every 4h)
- ShopFiber24 (EUR, every 4h)
- T&S Communication (USD, every 4h)
- SmartOptics (catalog, every 8h)
- HUBER+SUHNER (catalog, every 8h)
- Skylane Optics (USD, every 4h)
- AscentOptics (USD, every 4h)
- GAO Tek (USD, every 4h)
Scheduler: nightly window → 24/7 continuous (42 jobs total)
- Playwright scrapers: every 8h (FS.com, 10Gtek, ATGBICS, ProLabs)
- Fetch/Cheerio: every 4h (11 lightweight vendors)
- Flexoptix catalog: every 2h (primary price source)
- eBay enrichment: every 6h
- Compatibility matrices: every 12h
- Compute jobs: every 4h
Pi fleet: scripts/pi-scraper-setup.sh for one-command Pi node setup
Remove orphan schedules (addon/naddod/qsfptek) that had no registered workers.
Pre-create request_queues/default, datasets/default, key_value_stores/default
before each scraper run to avoid ENOENT when Crawlee tries to write lock files.
Previously missing from scheduler:
- Champion ONE, Fluxlight, GBICs, SFPCables pricing
- Juniper HCT, SONiC HCL, Ufispace, Edgecore compatibility
- Flexoptix supported vendors
- Switch assets enrichment
Full nightly sequence now covers every scraper in the fleet.
All jobs staggered with 15-30 min gaps to respect vendor rate limits.
- Rename nav tab and sub-nav from 'Procurement Intel' to 'Procurement Intelligence'
- Add data-tip tooltips to all 8 ABC table column headers
- Add title attributes to signal badges, ABC class badges, supply risk, stock/price/lead trend spans, signal strength bar
- Add hover descriptions to Market Intelligence type icons, buy signal badges, technology tags, impact horizon, source
- Add hover descriptions to Lifecycle Events type icons, buy signal badges, impact level, effective date
- Tooltips explain business meaning of every data point (e.g. ABC classification formula, demand score composition, supply risk levels)
- api() helper now parses JSON body on non-2xx responses so error.suggestion
is available in catch blocks
- runFinder() catch shows 'Switch not found' + suggestion instead of 'Error: HTTP 404'
- finder.ts: normalized search (removes hyphens/spaces) + token-based fallback
so 'sg350-28' → 'SG350-28', 'N9K-C93180' → Nexus 93180, etc.
- CHANGELOG_PENDING.md: 26 entries from v0.1.0 to today in JSON-line format
- GET /api/changelog: parses and serves entries as JSON array
- Overview tab: changelog card with type badges (FEAT/FIX/UI/DATA/AI/INFRA),
dates, show recent/all toggle
- isGarbageName(): detects scraped-slugs, 'All Optical Transceivers', 'Compatible NNGbps...',
generic form-factor descriptions with no real SKU
- Panel title priority: real standard_name → part_number → description → constructed from specs
- Details warning shown when details_verified = false (amber banner)
- sql/018: marks garbage entries as data_confidence='garbage' for future DELETE
- API: also returns comparable_prices from technically equivalent products
(same form_factor + speed_gbps + reach ±25%, different vendor, last 30 days)
- Dashboard: direct prices shown first, then separator + comparable products
- Comparable entries show vendor + exact part number scraped from their site
- Verified badge = real URL + observed within 7 days (strict)
- 100% VERIFIED bar: checkmarks now rgba(255,255,255,0.92) instead of #2d6a4f (was invisible on green bg)
- Pricing: only show prices with real URL; no MSRP/estimated fallback
- Verified badge only if observed within 7 days; older prices shown without badge
- Temperature Range: COM→'0-70°C (COM)', IND→'-40-85°C (IND)'
- GET /api/transceivers/🆔 returns competitor_prices[] from price_observations
- Detail view: verification summary bar (★ 100% VERIFIED / partial)
- Detail view: Current Prices section with vendor, price, verified badge, date, link
- Detail view: tag tooltips on vendor/category/market_status chips
- List view: new Verified column with 100% stamp or price check
- Optical Budget: TX Power Min/Max labels clarified
- Panel close button: high-contrast rgba background, 38px, bold ×, shadow
- Detail view: full product name (SKU + description) shown above image
- List view: NAME column shows part_number on line 1, descriptive name line 2
- txDescName() helper builds name from description field or constructs from specs
DB (017-verification-tags.sql):
- New columns: price_verified, price_verified_eur, price_verified_url, price_verified_at
- New columns: image_verified, details_verified, fully_verified, fully_verified_at
- compute_transceiver_verification(uuid): per-product verification logic
• price_verified: real scraped URL + price > 0 + observed in last 30 days
• image_verified: R2 stored OR image_url from known vendor CDNs (flexoptix.net, fs.com, etc.), no placeholder
• details_verified: product_page_url + all core fields (form_factor, speed, reach, fiber_type, part_number) populated
• fully_verified: all three true simultaneously
- recompute_all_verification(): bulk recompute, returns stats
- Initial run: 3575 price_verified, 1173 image_verified, 1380 details_verified, 258 fully_verified
- Indexes on price_verified, fully_verified for fast filtering
- v_verified_products view
API finder.ts:
- SELECT now includes all verification fields
- Response maps: price_verified, price_verified_eur, price_verified_url, image_verified, details_verified, fully_verified
API health.ts:
- verification block: counts + coverage percentages in /api/health
Dashboard Finder:
- 'Verified Price': green checkmark ✓ next to price, tooltip explains source
- '100% Verified' stamp: dark green gradient badge top of card, card gets green border
- 'price source ↗' link to original scraped URL
- Summary bar: 'X × 100% Verified · Y with verified prices'
- New 'Finder' tab between Switches and Blog Engine
- Search by switch model (free text), filter by speed
- Quick-access buttons: Nexus 93180, Nexus 9332D, Arista 7280R3A, Juniper QFX5120
- Results grouped by speed class (400G QSFP-DD, 100G QSFP28, etc.)
- Shows: part number, vendor, reach, fiber type, connector, verified price, stock status
- Flexoptix products highlighted with orange left border + FLEXOPTIX badge
- Buy link → flexoptix.net for each result
- Uses existing /api/finder endpoint with 33,993 compatibility entries
