Complete Pi scraper entry point covering all pricing, catalog, compat,
intelligence and prediction signal scrapers. Includes 5 new form-factor
coverage scrapers (comms-express, router-switch, multimode-inc,
optictransceiver, wiitek). Erik runs only API+DB, all scraping on Pis.
Add Comms-Express, Router-Switch.com, Multimode Inc, OpticTransceiver.com,
and Wiitek scrapers covering CFP2-DCO, CFP4, OSFP224, QSFP112, CXP, GBIC,
XENPAK, CSFP, SFP-DD, SFP56, QSFP56 and other previously-uncovered form
factors. Each scheduled every 8h. Worker registrations added to scheduler.
Also export db alias in utils/db.ts to fix eBay enricher + community scrapers
crashing with 'Cannot read properties of undefined (reading query)'.
- POST /api/auth/login: HMAC-SHA256 signed 7-day token, password from DASHBOARD_PASSWORD env
- GET /api/auth/verify: stateless token validation
- requireAuth middleware applied to all /api/* routes (except /api/health + /api/auth)
- /dashboard/login.html: dark TIP-themed login page with show/hide password toggle
- index.html: auth guard redirect to login + Authorization header on all api() calls
- No secrets in code — password stored in .env only
- downloadDocuments(): fetches PDFs from product_documents and documents tables
using curl, organises into switches/ transceivers/ whitepapers/ other/ subdirs
- Integrated into runNightlyNasSync() — runs after JSON exports
- rsync incremental — only new/changed files transferred
- NAS dir structure: /volume1/tip-data/datasheets/{switches,transceivers,whitepapers,other}
- max-filesize 50MB guard per file
- utils/logger.ts: minimal console-based logger (debug/info/warn/error)
used by community-issues and ebay-enricher scrapers
- scripts/pi-scraper-setup.sh: step 7 adds optional WireGuard setup
(pass WG_PRIVKEY + WG_ADDR env vars) — connects Pi to Erik for DB access
auto-detects dead ethernet and routes WG traffic via working interface
New scrapers (8):
- BlueOptics (EUR, every 4h)
- ShopFiber24 (EUR, every 4h)
- T&S Communication (USD, every 4h)
- SmartOptics (catalog, every 8h)
- HUBER+SUHNER (catalog, every 8h)
- Skylane Optics (USD, every 4h)
- AscentOptics (USD, every 4h)
- GAO Tek (USD, every 4h)
Scheduler: nightly window → 24/7 continuous (42 jobs total)
- Playwright scrapers: every 8h (FS.com, 10Gtek, ATGBICS, ProLabs)
- Fetch/Cheerio: every 4h (11 lightweight vendors)
- Flexoptix catalog: every 2h (primary price source)
- eBay enrichment: every 6h
- Compatibility matrices: every 12h
- Compute jobs: every 4h
Pi fleet: scripts/pi-scraper-setup.sh for one-command Pi node setup
Remove orphan schedules (addon/naddod/qsfptek) that had no registered workers.
Pre-create request_queues/default, datasets/default, key_value_stores/default
before each scraper run to avoid ENOENT when Crawlee tries to write lock files.
Previously missing from scheduler:
- Champion ONE, Fluxlight, GBICs, SFPCables pricing
- Juniper HCT, SONiC HCL, Ufispace, Edgecore compatibility
- Flexoptix supported vendors
- Switch assets enrichment
Full nightly sequence now covers every scraper in the fleet.
All jobs staggered with 15-30 min gaps to respect vendor rate limits.
- Rename nav tab and sub-nav from 'Procurement Intel' to 'Procurement Intelligence'
- Add data-tip tooltips to all 8 ABC table column headers
- Add title attributes to signal badges, ABC class badges, supply risk, stock/price/lead trend spans, signal strength bar
- Add hover descriptions to Market Intelligence type icons, buy signal badges, technology tags, impact horizon, source
- Add hover descriptions to Lifecycle Events type icons, buy signal badges, impact level, effective date
- Tooltips explain business meaning of every data point (e.g. ABC classification formula, demand score composition, supply risk levels)
- api() helper now parses JSON body on non-2xx responses so error.suggestion
is available in catch blocks
- runFinder() catch shows 'Switch not found' + suggestion instead of 'Error: HTTP 404'
- finder.ts: normalized search (removes hyphens/spaces) + token-based fallback
so 'sg350-28' → 'SG350-28', 'N9K-C93180' → Nexus 93180, etc.
- CHANGELOG_PENDING.md: 26 entries from v0.1.0 to today in JSON-line format
- GET /api/changelog: parses and serves entries as JSON array
- Overview tab: changelog card with type badges (FEAT/FIX/UI/DATA/AI/INFRA),
dates, show recent/all toggle
- isGarbageName(): detects scraped-slugs, 'All Optical Transceivers', 'Compatible NNGbps...',
generic form-factor descriptions with no real SKU
- Panel title priority: real standard_name → part_number → description → constructed from specs
- Details warning shown when details_verified = false (amber banner)
- sql/018: marks garbage entries as data_confidence='garbage' for future DELETE
- API: also returns comparable_prices from technically equivalent products
(same form_factor + speed_gbps + reach ±25%, different vendor, last 30 days)
- Dashboard: direct prices shown first, then separator + comparable products
- Comparable entries show vendor + exact part number scraped from their site
- Verified badge = real URL + observed within 7 days (strict)
- 100% VERIFIED bar: checkmarks now rgba(255,255,255,0.92) instead of #2d6a4f (was invisible on green bg)
- Pricing: only show prices with real URL; no MSRP/estimated fallback
- Verified badge only if observed within 7 days; older prices shown without badge
- Temperature Range: COM→'0-70°C (COM)', IND→'-40-85°C (IND)'
- GET /api/transceivers/🆔 returns competitor_prices[] from price_observations
- Detail view: verification summary bar (★ 100% VERIFIED / partial)
- Detail view: Current Prices section with vendor, price, verified badge, date, link
- Detail view: tag tooltips on vendor/category/market_status chips
- List view: new Verified column with 100% stamp or price check
- Optical Budget: TX Power Min/Max labels clarified
- Panel close button: high-contrast rgba background, 38px, bold ×, shadow
- Detail view: full product name (SKU + description) shown above image
- List view: NAME column shows part_number on line 1, descriptive name line 2
- txDescName() helper builds name from description field or constructs from specs
DB (017-verification-tags.sql):
- New columns: price_verified, price_verified_eur, price_verified_url, price_verified_at
- New columns: image_verified, details_verified, fully_verified, fully_verified_at
- compute_transceiver_verification(uuid): per-product verification logic
• price_verified: real scraped URL + price > 0 + observed in last 30 days
• image_verified: R2 stored OR image_url from known vendor CDNs (flexoptix.net, fs.com, etc.), no placeholder
• details_verified: product_page_url + all core fields (form_factor, speed, reach, fiber_type, part_number) populated
• fully_verified: all three true simultaneously
- recompute_all_verification(): bulk recompute, returns stats
- Initial run: 3575 price_verified, 1173 image_verified, 1380 details_verified, 258 fully_verified
- Indexes on price_verified, fully_verified for fast filtering
- v_verified_products view
API finder.ts:
- SELECT now includes all verification fields
- Response maps: price_verified, price_verified_eur, price_verified_url, image_verified, details_verified, fully_verified
API health.ts:
- verification block: counts + coverage percentages in /api/health
Dashboard Finder:
- 'Verified Price': green checkmark ✓ next to price, tooltip explains source
- '100% Verified' stamp: dark green gradient badge top of card, card gets green border
- 'price source ↗' link to original scraped URL
- Summary bar: 'X × 100% Verified · Y with verified prices'
- New 'Finder' tab between Switches and Blog Engine
- Search by switch model (free text), filter by speed
- Quick-access buttons: Nexus 93180, Nexus 9332D, Arista 7280R3A, Juniper QFX5120
- Results grouped by speed class (400G QSFP-DD, 100G QSFP28, etc.)
- Shows: part number, vendor, reach, fiber type, connector, verified price, stock status
- Flexoptix products highlighted with orange left border + FLEXOPTIX badge
- Buy link → flexoptix.net for each result
- Uses existing /api/finder endpoint with 33,993 compatibility entries
gatherBlogData():
- Fetches real prices from price_observations (last 30 days) per product
- Filters transceivers by speed extracted from topic keywords
- Enriches every product with verified_prices array + has_verified_price flag
- Joins DB products with vector search results (DB first — they have real prices)
contextData injection (blog.ts):
- [PRODUCT] lines: exact standard_name, form_factor, speed, reach, connector, dBm specs, Watts
- [VERIFIED PRICE] lines: real EUR/USD price, vendor, observed date, source URL
- [NO VERIFIED PRICE IN DB]: explicit tag — LLM must not invent a number
- [NO PRODUCT DATA AVAILABLE]: fallback when DB returns nothing
fo-blog-pipeline.ts system prompt:
- DATA INTEGRITY RULES block: prices/part numbers/vendors ONLY from context
- Never approximate with ~€350 or 'typically $200-600' for specific products
- Power specs only from [PRODUCT] data or REFERENCE VALUES
STEP4 context instructions:
- Explicit rules on how to use [VERIFIED PRICE] vs [NO VERIFIED PRICE]
- Invented data = HARD FAIL in QA
STEP9 QA — 3 new hard fail checks (30, 31, 32):
- Check 30: invented prices → remove or replace with flexoptix.net reference
- Check 31: invented part numbers → remove, use class name instead
- Check 32: invented vendor names → remove if not in known list
- Expanded research pool to 9 topics (was 3), evergreen to 12 (was 3)
- Conference topics: added Photonics West, CIOE, NFOEC follow-up, year-end review
- Standards topics: 3 rotating variants (IEEE tracker, SFF-8024 registry, OIF CEI-112G)
- seededShuffle(): day-of-year as seed — stable within the day, different every day
- API response adds refreshes_at (next midnight UTC) for frontend countdown
- Dashboard subtitle shows 'rotates daily · next refresh in Xh'
- Hot topic cards now pass full title + angle into generateBlog() correctly
- Fix SR4/DR4 fiber count: both use 8 fibers (4TX+4RX), difference is MMF vs SMF
- Fix power per port: 400G=10-15W/port, 800G=15-25W/port (not 1kW/port)
- Fix pricing context: always distinguish OEM ($1-5K) vs compatible ($200-600)
- Add HARD RULES 15-21: fiber count, power, pricing, no markdown headers,
max 6 sections, no repeated topics, flow over format
- Add QA CALIBRATION FAILS 16-21: same rules enforced at QA step
- Add fiber/power reference tables with correct values
- Strip markdown (##/###/####/**) from all output — plain text only
- Add Style B gold example (10/10 validated prose article)
- Update STEP5 reality injection with correct SR4→DR4 description
- Update STEP8 kill-AI-tone to strip markdown headers + merge duplicates
- CALIBRATION_GOLD_STANDARD now covers two validated styles: A (structured) and B (prose)
- Style B: no headers, no bullets, 1-3 sentence paragraphs, reframe ending
- STEP8_KILL_AI_TONE: prose conversion option for over-structured articles
- STEP4_MASTER_DRAFT: explicit style choice instruction (A vs B based on angle)
- Gold standard includes exact prose rhythm patterns from 10/10 human-reviewed article
- Wrong patterns expanded: symmetric sections, checklist endings, transition clichés
- POST /api/blog/:id/regenerate — re-runs full 10-step LLM pipeline on existing draft
- Regenerate button visible when quality_issues present or status=review
- SEO keywords now displayed as clickable #hashtags (copy-to-clipboard)
- fo-blog-pipeline: added PoE misuse, DR4 mislabeling, ZR/DR4 conflation as hard QA fails
- fo-blog-pipeline: 14 hard rules in system prompt (was 10)
- fo-blog-pipeline: CALIBRATION_GOLD_STANDARD + withCalibration() from 10/10 human review
- System prompt now includes gold standard example on every pipeline run
Hot Topics:
- Dynamic topics from /api/hot-topics loaded in Blog Engine tab
- 7 data sources (prices, competitors, hype cycle, news, conferences, research, evergreen)
- Urgency badges: BREAKING (red), HOT (orange), TRENDING (yellow), EMERGING (green)
Pipeline Lock:
- Only 1 generation at a time, 'Pipeline Busy' toast on double-click
- Progress bar with step names (external hot-topics.js, no inline hacks)
Blog Delete:
- DELETE /api/blog/:id endpoint
- Delete button (✕) on each blog in list
- 'Delete All Templates' button to clean up test drafts
Fix: dashboard JS extracted to external hot-topics.js to avoid sed quote hell
Hot Topics Engine (GET /api/hot-topics):
- 7 data sources: price movements, competitor alerts, hype cycle transitions,
news articles, conference calendar, research trends, evergreen topics
- Auto-discovers BREAKING/HOT/TRENDING/EMERGING topics
- Dashboard loads topics dynamically with urgency badges and source labels
- Click any topic → generates blog with that angle
Pipeline Lock (critical UX fix):
- Only 1 blog generation at a time (blogPipelineRunning flag)
- 'Pipeline Busy' toast if user clicks while generating
- Lock released on completion, timeout, or error
Dashboard:
- Static 3 cards replaced with dynamic hot topics grid
- 'Refresh Topics' button
- Topics show urgency color (red=breaking, orange=hot, yellow=trending, green=emerging)
- Auto-loads when Blog Engine tab opens
When you click Generate:
- Dark overlay with orange progress bar shows pipeline status
- Live step counter: 'Step 3/10: Outline Generation — decision-driven structure'
- Percentage updates every 15 seconds via API polling
- When done: shows word count + QA score, auto-opens the article
- No more silent template dump — user sees the entire pipeline working
Blog engine was falling back to template because qwen2.5:14b is on Mac Studio (.213),
not on Erik (localhost). Fixed ecosystem.config.js to use 192.168.178.213:11434.
This was the root cause why the 10-step pipeline never executed.
System prompt: 10 HARD RULES (non-negotiable, article fails QA without them)
- Mandatory WHAT BREAKS IN PRODUCTION section (2+ specific failures with symptoms/cause/fix)
- Mandatory HIDDEN COSTS section (cleaning, troubleshooting time, cabling redesign, training)
- Mandatory WHEN NOT TO USE section for every recommendation
- Absolute statement rule: NEVER without conditions/context
- Cabling reality: MPO polarity, SR4→DR4 migration, cleaning requirements
- Brutal hook requirement: not 'If you're still...' but 'You're about to sign a PO. Stop.'
- Minimum 2500 words (was 2000)
Step 5 (Reality Injection): Now checks for ALL mandatory sections and adds if missing
Step 9 (QA Check): Hard fail checks — article is NOT publishable without production failures + hidden costs
Feedback source: Human expert review scoring 7.5/10, targeting 9.5+
