# TIP Changelog

Format: `{"d":"YYYY-MM-DD","t":"TYPE","m":"Description"}`
Types: FEAT · FIX · UI · DATA · AI · INFRA

{"d":"2026-04-18","t":"FIX","m":"PM2 SKIP_FS_SCRAPER env not picked up by tip-scraper-daemon: pm2 restart --update-env did not apply new ecosystem.config.js vars because PM2 loaded from its saved dump. Fixed: pm2 delete + pm2 start ecosystem.config.js --only tip-scraper-daemon + pm2 save. Daemon restarted fresh (ID 83, 0 restarts) with SKIP_FS_SCRAPER=true now confirmed live. FS.com job now correctly skips on Erik instead of failing with ENOENT."}
{"d":"2026-04-18","t":"FIX","m":"FS.com Mac scraper: suppress Crawlee post-run ENOENT unhandledRejection — Crawlee's FileSystemStorage fires a final _isTaskReadyFunction call after run() resolves, reading a request .json that was already processed/cleaned-up. This ENOENT triggered process.exit(1) before Phase 2 completed, causing 7 days of missing FS.com price data. Fixed: targeted unhandledRejection handler in require.main block swallows ENOENT from request_queues paths while re-raising real errors."}
{"d":"2026-04-18","t":"FIX","m":"FS.com Mac scraper: PID lock (/tmp/tip-fs-scraper.lock) added to run-fs-scraper-mac.sh — prevents concurrent instances when launchd 2am fire overlaps with a still-running earlier run. Previous concurrent instances caused rmSync(storage-fs-phase1) race (one instance deletes the storage dir while another is using it), crashing Phase 2."}
{"d":"2026-04-18","t":"FIX","m":"Scraper health monitor: tiered alerts replacing false-positive 6h threshold. Old: fired every 3h for any vendor with 0 new prices (including stable prices). New: 🔴 CRITICAL (last price >7 days), 🟡 WARNING (last price 48h-7 days), ✅ STABLE (0 new prices but last price ≤48h — content hash dedup, scraper running OK). Shows pg-boss job state+time for faster root-cause."}
{"d":"2026-04-18","t":"FIX","m":"Daemon stability: global unhandledRejection handler in scheduler index.ts — Crawlee post-run lock-file ENOENT (request_queues path) was crashing the daemon process (process.exit(1)) which killed all active pg-boss jobs and triggered PM2 restart loops. Fix: swallow ENOENT from request_queues paths at the scheduler level; re-raise all other rejections. Also: FS.com scheduler worker now skips (SKIP_FS_SCRAPER=true env var) on Erik where Cloudflare WAF blocks datacenter IPs; Mac launchd handles FS.com scraping. Created missing Crawlee storage dirs: storage-fs-phase1, storage-fs-phase2, storage-ebay-transceivers, storage-fs. Health monitor pg-boss lookup extended from 12h → 26h; added completedMap; vendors with recent job completion + historical prices classified as STABLE (not CRITICAL) — eliminates ATGBICS/Fluxlight false-positive alerts."}
{"d":"2026-04-18","t":"FIX","m":"Playwright chromium_headless_shell-1217 installed on Erik (/opt/playwright-browsers/): ATGBICS and FS.com PlaywrightCrawler were throwing BrowserLaunchError on every run since Crawlee browser-pool requires chromium_headless_shell binary, not regular chromium. Fixed by: PLAYWRIGHT_BROWSERS_PATH=/opt/playwright-browsers npx playwright install chromium --with-deps on Erik."}
{"d":"2026-04-18","t":"FIX","m":"Crawlee withIsolatedStorage global env-var race condition eliminated: scheduler.ts removed all withIsolatedStorage() wrappers (were mutating process.env.CRAWLEE_STORAGE_DIR globally, causing concurrent scrapers to pick up wrong storage dirs). All Crawlee scrapers now use makeCrawleeConfig(name) instance-level Configuration. fs-com.ts migrated to fs-phase1/fs-phase2 storage names with rmSync cleanup before each phase. switch-assets-crawler.ts and switch-assets-playwright.ts now pass makeCrawleeConfig. Fixed: ATGBICS, community-issues, market-intel, FS.com."}
{"d":"2026-04-18","t":"FIX","m":"FS.com Mac launchd scraper (org.tip.fs-scraper): was failing with exit code 126 (Operation not permitted) because macOS TCC blocks launchd agents from accessing ~/Desktop/Claude Code/ path. Fixed: script moved to ~/.tip/run-fs-scraper-mac.sh, plist WorkingDirectory changed to ~/.tip. FS.com is now scraping from residential Mac IP again."}
{"d":"2026-04-18","t":"FIX","m":"NADDOD stockLevel 'unknown' -> 'on_request': invalid value for price_observations_stock_level_check constraint — was causing all NADDOD price insertions to fail."}
{"d":"2026-04-18","t":"FIX","m":"Crawlee makeCrawleeConfig: clear request_queues/default before each run — Crawlee FileSystemStorage marks URLs as HANDLED (state=4, orderNo=null) after processing. With purgeOnStart=false these entries persisted, so next crawler.run(startUrls) deduplicated all startUrls → requestsTotal=0 → immediate finish with 0 scraped pages. Fix: rmSync(request_queues/default) at start of makeCrawleeConfig(). Safe: session pool lives in key_value_stores/, not request_queues/. ATGBICS confirmed fixed: now scrapes 6 categories, 78 products, 33 unique with prices."}
{"d":"2026-04-18","t":"FIX","m":"Optcore scraper: add SKIP_OPTCORE_SCRAPER guard — optcore.net Cloudflare WAF blocks Erik IP (82.165.222.127). WP REST API returns 403/HTML block page, catch handler returns 0 URLs → 0 products every run. Set SKIP_OPTCORE_SCRAPER=true in ecosystem.config.js. Pattern mirrors SKIP_FS_SCRAPER. Residential IP (Mac launchd) required for Optcore."}
{"d":"2026-04-18","t":"FIX","m":"10Gtek scraper: was finding 152 products but 0 prices — 10gtek.com main site only shows technical spec tables (no prices). Rewrote scraper to target sfpcables.com (10Gtek's own retail store, same company) which exposes Magento product listings with Model: <part> + US$X.XX prices. Added Magento loop-detection via seen-part dedup (stops pagination when all products on a page were already seen). XFP title-after-pipe fallback for part number extraction. Removed QSFP-DD (not on sfpcables.com). Result: 50 products, 49 prices on first live run. Health monitor CRITICAL alert resolved."}
{"d":"2026-04-18","t":"FEAT","m":"Price Comparison Dashboard: public /api/price-comparison (summary, list top-50 SKUs by vendor coverage, per-SKU detail). Express Router, no auth required. New '💲 Price Comparison' dashboard tab with stat cards, form-factor breakdown table, top-50 SKU table (clickable rows), and SKU detail lookup with per-vendor prices + stock + spread %."}
{"d":"2026-04-18","t":"DATA","m":"Eoptolink OEM catalog scraper: harvests 93 product-solution pages from eoptolink.com, extracts part numbers (EOLO-*/EOLQ-* format), seeds transceivers table as manufacturer=Eoptolink entries with form_factor/speed/fiber/category. No prices (B2B OEM). Scheduled every 4h (40 */4 * * *)."}
{"d":"2026-04-18","t":"FIX","m":"stock_observations repopulated after TRUNCATE: storage-fs/request_queues/default/ directory re-created on Erik; NADDOD scraper manual-triggered; 4+ prices confirmed written within 20s."}
{"d":"2026-04-17","t":"FEAT","m":"MCP Server v0.2.0: wired finder.ts (find_flexoptix_for_switch, get_competitor_alerts), switch-docs (get_switch_docs, get_switch_image), analyze_market_with_llm (qwen2.5:14b via Ollama, enriched with live hype cycle + pricing + news), generate_blog_post (fo-blog-v5 fine-tuned model with qwen2.5:14b fallback + live pricing enrichment). OLLAMA_BASE_URL env var for Ollama endpoint."}
{"d":"2026-04-17","t":"UI","m":"Stock dashboard: 6th stat card (Multi-Vendor SKUs), confidence quality badge column in vendor breakdown (🟢 L3 per-warehouse / 🟡 L2 aggregated / ⚪ L1 boolean), new Multi-Vendor Price Comparison table with min/max/avg per SKU. Subtitle updated to mention QSFPTEK + NADDOD."}
{"d":"2026-04-17","t":"FEAT","m":"/api/stock/summary enhanced: vendor_breakdown adds avg_confidence + currencies + confidence breakdown (conf_per_warehouse/aggregated/boolean); new price_comparison endpoint (top 50 SKUs tracked by 2+ vendors with price spread); totals adds multi_vendor_skus count."}
{"d":"2026-04-17","t":"DATA","m":"Cisco TMG expanded to 17 platform families (+5 new: 8000 Series, NCS5500, NCS540, NCS560, NCS1000). Per-device query strategy replaces family-level search: iterates all switch IDs from filter → 58 switches per N9300 vs 1 before. 856 compat entries / 174 switches after re-run."}
{"d":"2026-04-17","t":"DATA","m":"Juniper HCT scraper run: 475 Juniper-brand transceivers seeded into transceivers table (form factor, speed, reach, fiber type from apps.juniper.net/hct). No prices (OEM). Scheduled to run at 6:15 + 18:15 daily."}
{"d":"2026-04-17","t":"DATA","m":"Competitor research: QSFPTEK shows real-time aggregated stock count (e.g. '5507 in real-time stock, 17 Apr 2026') + USD prices; NADDOD shows exact per-product counts ('In Stock: 543') via Astro SSR. Both scraped publicly, no login required. Flexoptix confirmed exact Lagerbestand + EUR prices. FS.com: EUR prices yes, exact counts no."}
{"d":"2026-04-17","t":"DATA","m":"stock_observations selective cleanup + schema upgrade: TRUNCATE stock_observations (186 FS.com test-run rows cleared, will repopulate on next launchd run). Added 4 new quality columns via migration 038: stock_confidence (1=boolean/2=aggregated/3=per-warehouse), price_currency CHAR(3), price_includes_tax BOOLEAN, stock_vendor_ts TIMESTAMPTZ."}
{"d":"2026-04-17","t":"FEAT","m":"Migration 028 retroactively committed to repo (028-stock-observations-warehouse-columns.sql) — documents the 10 warehouse columns applied directly to Erik DB. Guards with IF NOT EXISTS for safe re-application."}
{"d":"2026-04-17","t":"FEAT","m":"upsertStockObservation upgraded: new optional params stockConfidence (1|2|3), priceCurrency (ISO 4217), priceIncludesTax (boolean), stockVendorTs (timestamptz). FS.com now writes stockConfidence=3+priceCurrency=EUR+priceIncludesTax=false. Delta detection now also checks quantity_available changes."}
{"d":"2026-04-17","t":"FEAT","m":"QSFPTEK scraper v2: Phase 1 uses existing /mall/commodity/list API for product catalog (880+ products from sitemap). Phase 2 fetches /en/product/XXXXX.html detail pages to extract 'X in real-time stock, DATE' — writes stock_observations with stockConfidence=2 + stockVendorTs. Up to 500 detail pages per run at 2s rate limit."}
{"d":"2026-04-17","t":"FEAT","m":"NADDOD scraper v2: complete rewrite — migrated from WooCommerce category scraping to Astro sitemap-based discovery (/sitemaps/products.xml, /products/XXXXX.html). Extracts 'In Stock: X' exact counts from server-rendered HTML. Writes both price_observations (USD) and stock_observations (stockConfidence=1 or 2 depending on data visibility)."}
{"d":"2026-04-17","t":"DATA","m":"FS.com first warehouse data load: 268 products scraped, 186 stock_observations written — DE-Lager 128,428 units, Global-Lager 156,052 units, Backorder 37,495, 53.4M units sold total. Top seller: SFP-10GSR-85 with 14M units sold."}
{"d":"2026-04-17","t":"FIX","m":"upsertStockObservation: skip condition now includes backorder_qty — backorder-only products (DE=0 GL=0 BO>0) like coherent ZR/ZRH were silently dropped instead of being recorded"}
{"d":"2026-04-17","t":"FIX","m":"FS.com price extraction: broad fallback regex now only accepts prices >€100 to reject FS.com's €79 'Preis auf Anfrage' placeholder — prevents fake price observations on 1G/10G/25G/40G/100G transceivers"}
{"d":"2026-04-17","t":"UI","m":"Dashboard: stock observations count in header stats bar + warehouse stock summary card in Overview tab (hidden until stock_observations populated); both driven by /api/health stock block"}
{"d":"2026-04-17","t":"FEAT","m":"Health API: /api/health now includes stock block — total_observations, transceivers_with_stock, vendors_with_stock, total_de_qty, total_global_qty, last_observation_at from stock_observations"}
{"d":"2026-04-17","t":"INFRA","m":"FS.com Mac-side runner: launchd plist at 02:00/10:00/18:00 + run-fs-scraper-mac.sh via SSH tunnel to Erik DB port 5433 — residential IP required, datacenter IP blocked by FS.com Cloudflare WAF"}
{"d":"2026-04-17","t":"FEAT","m":"Stock API: GET /api/stock, /api/stock/summary, /api/stock/:id — warehouse breakdowns (DE-Lager, Global-Lager, Nachlieferung, units_sold) per transceiver/vendor"}
{"d":"2026-04-17","t":"DATA","m":"upsertStockObservation() in db.ts — writes 10 new stock_observations columns (warehouse_de_qty, warehouse_global_qty, backorder_qty, units_sold, compatible_brands, price_net, product_url, delivery dates)"}
{"d":"2026-04-17","t":"DATA","m":"FS.com scraper v2: Playwright-based, extracts DE-Lager + Global-Lager + Nachlieferung + Verkauft counts, German number/date parsing, 120-URL pre-queue, 12-category crawl, 12h dedup window"}
{"d":"2026-04-17","t":"FIX","m":"SmartOptics scraper v2: WooCommerce REST API fallback + 8 catalog categories + relative URL regex fix — was finding only 8 products, now discovers full catalog"}

---

{"d":"2026-04-12","t":"FIX","m":"DB functions compute_transceiver_verification() + compute_transceiver_verification(uuid): both now require competitor_verified as 4th criterion for fully_verified — was silently ignoring competitor check and granting ★ 100% badge based on only 3 criteria"}
{"d":"2026-04-12","t":"FEAT","m":"Scheduler: maintenance:reconcile-verification nightly job (01:00 UTC via pg-boss) — auto-resets competitor_verified=false where no non-Flexoptix price_observation in last 30 days, then recomputes fully_verified — eliminates recurring false ★ 100% badges without manual SQL intervention"}
{"d":"2026-04-12","t":"DATA","m":"Data quality: 608 transceivers had competitor_verified=true with NO actual non-Flexoptix price in last 30 days — all reset to false + fully_verified=false. ★ 100% badge now only shows when genuinely earned. Triggered by user catching false badges on 1.6T OSFP products."}
{"d":"2026-04-12","t":"FIX","m":"ATGBICS + FS.COM scrapers: PlaywrightCrawler useSessionPool=false added — eliminates SDK_SESSION_POOL_STATE.json crash on every run; withIsolatedStorage now pre-seeds empty session state file as belt-and-suspenders"}
{"d":"2026-04-12","t":"FIX","m":"Skylane scraper: pagination now breaks on zero NEW unique product URLs (was looping all 10 pages because Algolia returns same content regardless of ?page=N)"}
{"d":"2026-04-12","t":"FIX","m":"AscentOptics scraper fully rewritten: uses /product-list?is_render=1&category_id=CID JSON API (was hitting 404 on old /catalog/ URLs); hardcoded category IDs for 14 transceiver form factors; no prices (OEM Get Quote model)"}
{"d":"2026-04-12","t":"UI","m":"Dashboard transceiver table: VERIFIED column now shows all 4 individual criteria per row (✓/— P=Price, I=Image, D=Details, C=Competitor) in green/red — ★ 100% badge only when all 4 met; uses competitor_verified DB column"}
{"d":"2026-04-12","t":"FIX","m":"Data quality: 59 anomalous price observations deleted (FS.COM accessories EUR 1-18 misidentified as OSFP/QSFP-DD/QSFP28; ATGBICS QSFP-DD sub-$60) — 49 transceivers competitor_verified degraded to false, 1 fully_verified badge removed"}
{"d":"2026-04-12","t":"FIX","m":"upsertPriceObservation: hard floor $1.50 USD added before form-factor bounds check — catches accessories/cables misidentified as transceivers when form_factor defaults to SFP with loose [2,3000] bounds"}
{"d":"2026-04-12","t":"FIX","m":"GBICS scraper: attribute order changed on site — regex updated from aria-label→href→data-event-type to dual-pass href+aria-label (both orders), data-event-type no longer required; prices now correctly extracted"}
{"d":"2026-04-12","t":"FIX","m":"Scheduler: 11 missing boss.work() handlers added for lightweight scrapers (fluxlight, gbics, optcore, champion-one, sfpcables, blueoptics, fiber24, tscom, skylane, ascentoptics, gaotek) — jobs were queued by cron but never consumed; scrapers stale 24-48h"}
{"d":"2026-04-12","t":"FIX","m":"withIsolatedStorage: removed rmSync cleanup of Crawlee storage dir — dir deletion caused SDK_SESSION_POOL_STATE.json not found crash on every Playwright scraper restart (ATGBICS/FS.COM failed every 2h cycle)"}
{"d":"2026-04-12","t":"FEAT","m":"Scheduler: monitor:scraper-health job added (every 3h via pg-boss) — checks price_observations per vendor in last 6h, logs SCRAPER HEALTH ALERT to pm2 stderr for any vendor with 0 new prices"}
{"d":"2026-04-12","t":"FIX","m":"Health check vendor names corrected: SFPCables→SFPcables, Fiber24→ShopFiber24, T&S Com→T&S Communication to match actual vendor table values"}
{"d":"2026-04-12","t":"FIX","m":"FiberMall scraper: URL schema corrected — wrong /c/1g-sfp-transceiver/ paths (HTTP 404) replaced with actual /store-XXXXX-name.htm category URLs discovered via homepage navigation scrape"}
{"d":"2026-04-12","t":"FIX","m":"FiberMall parser: product card split on new_proList_mainListLi (Vue.js SSR), price extracted from <span class=currency_price data-price=X.XX> — fixed false-match on data-price=0.00 from SKU variant items that appears before real price in each card"}
{"d":"2026-04-12","t":"FIX","m":"FiberMall: also scrapes SKU brand variants from .sku_item divs within each product group (Cisco/Arista/Juniper compatible versions listed per product)"}
{"d":"2026-04-12","t":"FIX","m":"Flexoptix price parsing: EUR text regex /([\d.]+)\s*EUR/ matched only digits before thousand separator (2,921.60 EUR → 2 EUR) — fixed to /([\d,]+\.?\d*)\s*EUR/ with comma strip; affects all Flexoptix prices >999 EUR"}
{"d":"2026-04-12","t":"FIX","m":"Flexoptix catalog: O.138HG2.C.05 (1.6T OSFP224 2x DR4) price corrected 3009.60→2921.60 EUR (stale since 2026-04-09, Flexoptix.net shows FLEXBOX price 2921.60 via data-price-amount attribute)"}
{"d":"2026-04-12","t":"FEAT","m":"Flexoptix catalog: 4 new search queries added — OSFP224 1.6T, OSFP224, 1.6T DR4, 1.6T transceiver — covers new 1.6T form factor previously missing entirely from catalog scraper"}
{"d":"2026-04-12","t":"FIX","m":"Schema: competitor_verified + competitor_verified_at columns added to transceivers table (ALTER TABLE) — were referenced in db.ts upsertPriceObservation but not in schema, causing price writes to fail silently for all competitor vendors (FiberMall, QSFPTEK etc.)"}
{"d":"2026-04-11","t":"FEAT","m":"Scraper coverage expansion: 3 new scrapers added — FiberMall (fibermall.com, USD), Vcelink (vcelink.com, USD, Shopify), OpticsBay (opticsbay.com, USD, WooCommerce) — all wired into scheduler and Pi fleet"}
{"d":"2026-04-11","t":"FIX","m":"QSFPTEK scraper fully rewritten: site migrated from OpenCart to custom Java/Spring+Vue — old /c/*.html paths 404, now uses /mall/commodity/list API with attribute-based data rate filtering; 8 attribute IDs for 1G/10G/25G/40G/100G/200G/400G/800G"}
{"d":"2026-04-11","t":"INFRA","m":"Scheduler: 61 workers total, 53 cron schedules — FiberMall/Vcelink/OpticsBay added at :03, :07, :57 past even hours"}
{"d":"2026-04-09","t":"FEAT","m":"Price anomaly detection: PRICE_BOUNDS per form-factor in db.ts upsertPriceObservation — prices outside [min,max] USD range silently rejected to prevent garbage data (e.g. SFP+ [4, 5000], OSFP224 [200, 60000])"}
{"d":"2026-04-09","t":"UI","m":"Dashboard: LLM panel redesigned for light theme readability; LLM model selector added to Blog Engine tab"}
{"d":"2026-04-09","t":"INFRA","m":"Pi Starlink proxy-agent: scraper routes selected lightweight scrapers exclusively to Pi worker fleet via SOCKS5 — no Playwright traffic on Pi nodes"}
{"d":"2026-04-09","t":"DATA","m":"800G standards deep enrichment: migration 033 — IEEE 802.3df, OIF 800G IA, 800G MSA, OSFP MSA, QSFP-DD800 MSA with links, status, timeline"}
{"d":"2026-04-09","t":"FEAT","m":"Linecard system support: switches can have linecard slots; Cisco 8000 accuracy migration (031) with correct port count and linecard data"}
{"d":"2026-04-09","t":"FIX","m":"Qdrant init, switch column verification, crawler live status, demo data badges — stability audit fixes"}
{"d":"2026-04-08","t":"FEAT","m":"Scraper: SOCKS5 proxy rotation for FS.com, ATGBICS, GBICS via Pi fleet nodes — residential IPs for CloudFront WAF bypass"}
{"d":"2026-04-07","t":"AI","m":"Blog fine-tuning: 100 gold-standard training articles added (blog-001 to blog-100) for fo-blog-v2/v3 fine-tuning dataset"}
{"d":"2026-04-06","t":"FIX","m":"Scraper: FS.com switched to de.fs.com for EUR prices as primary source; parsePrice hardened (requires currency symbol, uses largest number)"}
{"d":"2026-04-06","t":"FIX","m":"Scraper: bot User-Agents replaced with Chrome UA; dead domain scrapers disabled"}
{"d":"2026-04-06","t":"FIX","m":"Blog: Claude API calls serialized via queue to prevent 429 rate-limit spam; claudeQueue deadlock from recursive 429 retry fixed"}
{"d":"2026-04-06","t":"FEAT","m":"Blog: Anthropic Claude provider added to LLM client — claude-bridge on Erik used as flat-rate backend"}
{"d":"2026-04-06","t":"DATA","m":"Migrations 026+027: price observation cleanup and FS.com EUR currency fix"}
{"d":"2026-04-06","t":"FIX","m":"Dashboard: verified badge logic corrected; comparable pricing shown properly; product images clickable"}
{"d":"2026-04-06","t":"AI","m":"Blog training: 13 gold-standard articles added to BlogLLM training set"}
{"d":"2026-04-05","t":"FEAT","m":"Blog Engine: AEM/APM pipeline steps + SLL context builder + LinkedIn v2 prompts; blog routes mounted (blogSllRouter + scraperRouter)"}
{"d":"2026-04-05","t":"FEAT","m":"Blog: Title Contract + Technical Sanity Check + Self-Heal + angle-aware LinkedIn generator; anti-repetition engine with 6 angle types and forbidden structures"}
{"d":"2026-04-05","t":"FEAT","m":"Blog: Post to Ghost + Post to LinkedIn buttons in dashboard"}
{"d":"2026-04-05","t":"FIX","m":"Blog: hard story blacklist in STEP4 + LinkedIn (2AM/dirty connector/lab-vs-prod stories banned); word target 1200–1600; power-budget false positive fix"}
{"d":"2026-04-05","t":"FEAT","m":"Dashboard: data verification status section added to Overview tab"}
{"d":"2026-04-05","t":"FEAT","m":"Scraper: all pricing scrapers unified to 2h 24/7 cycle — full competitor coverage with no scheduling gaps; 4th verification criterion (Competitor) added"}
{"d":"2026-04-11","t":"FIX","m":"Scraper: CRAWLEE_PURGE_ON_START=1 set in withIsolatedStorage — fixes FS.com + ATGBICS crash on startup (SDK_SESSION_POOL_STATE.json not found in fresh isolated storage dir)"}
{"d":"2026-04-11","t":"FEAT","m":"Scraper: NADDOD, QSFPTEK, AddOn Networks added to pg-boss scheduler (every 2h, slots :48/:52/:55) — 24 pricing queues total, 58 workers"}
{"d":"2026-04-11","t":"FIX","m":"Scraper: ProLabs rewritten from PlaywrightCrawler (blocked by CloudFront WAF TLS fingerprinting) to fetch-based sitemap scraper — catalog-only (B2B quote model, no public prices)"}
{"d":"2026-04-11","t":"FIX","m":"Scraper: startup zombie cleanup in index.ts — on daemon restart, active pg-boss jobs older than 5 min are marked failed to allow re-queueing at next cron tick"}
{"d":"2026-04-11","t":"FIX","m":"Scraper: pre-existing TypeScript build errors fixed (findOrCreateScrapedTransceiver: removed invalid name/url/extractType params; ebay-enricher cheerio type mismatch; community-issues description→summary, publishedDate→published_at)"}
{"d":"2026-04-04","t":"AI","m":"Blog Engine v5: STEP8b replaced with Reduction Engine v1.0 (5-pass: Repetition Kill → Tech Prune → Flow Rebuild → Weight Correction → Humanization); target 700-1000 words; LaTeX hard delete in Pass 2; title/content alignment in Pass 4; word count range enforcement 600-1300 with warnings"}
{"d":"2026-04-04","t":"DATA","m":"Blog calibration: Gold Standard 5 added (market alert / pricing article — 2026-04-04; title matches body throughout; no LaTeX; DR4 = MPO-12 not LC duplex; ending lands on title topic not generic close)"}
{"d":"2026-04-04","t":"AI","m":"Blog Engine v5: system prompt + STEP9 QA hardened with LaTeX hard fail (\\[...\\] destroys flow), DR4 connector hard fail (DR4=MPO-12, FR4=LC duplex), title/content alignment check (12d); WRONG PATTERNS extended with 4 new entries"}
{"d":"2026-04-04","t":"AI","m":"Blog Engine v5: STEP4b Narrative Control (4-correction pass after draft — root cause assignment, anti-FUD filter, reality reframe, Flexoptix voice check); minimum words 1500→2500; reduction pass 25-35%→15-25%; pipeline now 14 steps, version v5-narrative-control"}
{"d":"2026-04-04","t":"AI","m":"Blog Engine v5: STEP_LINKEDIN_POST — generates LinkedIn post ≤2800 chars from final article (hook + 3-5 insights + CTA + hashtags); stored in blog_drafts.linkedin_post + linkedin_char_count; hard truncation at 2800 if LLM exceeds limit"}
{"d":"2026-04-04","t":"DATA","m":"Blog calibration: Gold Standard 4 added (compatible vs OEM narrative correction — 2026-04-04; optic = not root problem, exposes existing issues; correct Flexoptix framing: validation responsibility shifts to operator)"}
{"d":"2026-04-04","t":"DATA","m":"Migration 024: linkedin_post + linkedin_char_count columns in blog_drafts"}
{"d":"2026-04-04","t":"FIX","m":"Proxy Network: IP geo-lookup via ip-api.com on register/heartbeat (country_code + city now populated); heartbeat_count column + uptime_pct computed per heartbeat (was always 0.00); dedup fix — register returns existing token for same IP+port; heartbeat no longer overwrites registered IP (prevented IPv6 churn conflicts)"}
{"d":"2026-04-04","t":"DATA","m":"Proxy Network: migration 023 — heartbeat_count column added, existing node uptime_pct backfilled, duplicate registration from same IPv6 removed (4 nodes → 3)"}
{"d":"2026-04-04","t":"AI","m":"Blog Engine v4: STEP8b Reduction Pass (25-35% content cut, removes repeated concepts) + STEP8c Style Lock (tone consistency, scope/OPM fix, no inline SKUs) — pipeline now 12 steps, version v4-reduction-stylelock"}
{"d":"2026-04-04","t":"DATA","m":"Blog calibration: Gold Standard 3 added (Style B troubleshooting — 2026-04-04 field feedback, flowing narrative, zero sections, failure as behavior not scenario)"}
{"d":"2026-04-04","t":"FIX","m":"Flexoptix scraper: contentHash call fixed (was passing JSON.stringify string, now passes object directly)"}
{"d":"2026-04-03","t":"FEAT","m":"TIP Proxy Network (packages/proxy-agent): SOCKS5 residential proxy for CloudFront WAF bypass — node registration, heartbeat, load balancing with uptime+latency scoring"}
{"d":"2026-04-03","t":"FEAT","m":"proxy-agent CLI package (@tip/proxy-agent): tip-agent start/status/stop, configurable bandwidth cap, 30s heartbeat, graceful shutdown"}
{"d":"2026-04-03","t":"FIX","m":"DB utils: price_verified=true now set in content_hash early-return path (no new observation); image_verified=true auto-set on INSERT and on image_url update in findOrCreateScrapedTransceiver"}
{"d":"2026-04-03","t":"FIX","m":"pg-boss pool: max connections reduced to 4 + idle_in_transaction_session_timeout=30s — fixed PostgreSQL max_connections exceeded (100/100)"}
{"d":"2026-04-03","t":"DATA","m":"Image backfill: 178 Flexoptix images added via GraphQL small_image — Optcore images via Playwright gallery — findOrCreateScrapedTransceiver now updates image_url for existing records"}
{"d":"2026-04-03","t":"FEAT","m":"SmartOptics scraper: DWDM/coherent product catalog, og:image extraction, 8 products with form factor + reach detection"}
{"d":"2026-04-03","t":"FIX","m":"Fluxlight scraper: price extraction fixed for BigCommerce HTML (data-product-price-without-tax attribute)"}
{"d":"2026-04-03","t":"AI","m":"Blog Engine v3: STEP4 prose requirement (zero tolerance for ## headers, #### Scenario: patterns, bullet sections) — STEP3 outline as flow plan (3-4 beats) — STEP9 format violations as primary hard fail"}
{"d":"2026-04-03","t":"FIX","m":"Blog engine: DR4 wavelength corrected to 1310nm=0.35dB/km; scope description fixed (visual tool, not loss measurement device)"}
{"d":"2026-04-03","t":"FIX","m":"Blog engine: orphaned floating text in fo-blog-pipeline.ts removed (dead code outside template literal causing TypeScript build failure)"}
{"d":"2026-04-03","t":"FEAT","m":"NOG Talks scraper: DENOG/NANOG/RIPE/ENOG/NLNOG/Euro-IX conference talks — relevance scoring, optical keyword detection, weekly pg-boss job, CtxEvent cross-DB bridge via dblink"}
{"d":"2026-04-03","t":"FEAT","m":"Hot Topics v2: market_intelligence as SOURCE 3b (0.6+ relevance, urgency mapping per intel_type), NOG Talks as SOURCE 3c (grouped by event with speaker+abstract), limit 20 topics"}
{"d":"2026-04-03","t":"FIX","m":"Flexoptix scraper: 1G SFP coverage fixed (added SFP LX/SX/ZX queries); SKU suffix stripping (:Sx → base SKU); pagination cap removed (200-product limit was blocking full catalog); Phase 1→2 URL enrichment"}
{"d":"2026-04-03","t":"FEAT","m":"Prediction intelligence fully_verified trigger: PostgreSQL trigger trg_sync_fully_verified auto-computes fully_verified = price_verified AND image_verified AND details_verified — mass backfill: 258 → 3615 badges"}
{"d":"2026-04-02","t":"FEAT","m":"Auth: password-protected login page — HMAC-SHA256 signed token, requireAuth middleware on all API routes, dark TIP-themed login page"}
{"d":"2026-04-02","t":"INFRA","m":"Raspberry Pi fleet: 3x Pi nodes running 24/7 as lightweight scraper workers via WireGuard VPN, pg-boss multi-node queue sharing"}
{"d":"2026-04-02","t":"INFRA","m":"WireGuard VPN: Pi fleet tunnel for secure PostgreSQL access to production DB"}
{"d":"2026-04-02","t":"FEAT","m":"Prediction Intelligence System (migration 022): 7 new tables — hyperscaler_capex, distributor_lead_times, github_tech_signals, marketplace_velocity, ai_cluster_announcements, standards_activity, forecast_signals"}
{"d":"2026-04-02","t":"FEAT","m":"SEC EDGAR scraper: XBRL API, quarterly CapEx for Amazon/Microsoft/Alphabet/Meta — DC-share estimate + YoY growth"}
{"d":"2026-04-02","t":"FEAT","m":"GitHub Signals scraper: weekly repo_count/commit/stars for 400G/800G/ZR/CMIS/CPO/silicon-photonics tech adoption tracking"}
{"d":"2026-04-02","t":"FEAT","m":"eBay Velocity scraper: sold/active listing counts + avg price for 9 transceiver search terms — every 12h"}
{"d":"2026-04-02","t":"FEAT","m":"AI Cluster Announcements scraper: 6 RSS feeds (DataCenterKnowledge, DC Dynamics, Blocks&Files, Next Platform, ServeTheHome) — extracts company, MW, network speed, estimated transceivers"}
{"d":"2026-04-02","t":"FEAT","m":"Distributor Lead Times scraper: Mouser, Digi-Key, RS Components — in_stock, stock_qty, lead_time_weeks, price — daily"}
{"d":"2026-04-02","t":"FEAT","m":"Standards Tracker: IEEE 802.3 project table, OIF hot topics, IETF Datatracker API — tracks in-progress/ballot/published status — weekly"}
{"d":"2026-04-02","t":"FEAT","m":"Forecast Engine: weighted demand index (0-100) from 6 signal types — capex 0.30, ai_clusters 0.25, ebay_velocity 0.20, lead_times 0.15, github 0.06, standards 0.04 — 3/9/12/18 month horizons for 5 technologies"}
{"d":"2026-04-02","t":"FEAT","m":"NAS sync: datasheet/manual download — PDFs from product_documents organized into switches/transceivers/whitepapers/other"}
{"d":"2026-04-02","t":"INFRA","m":"Scheduler: 50 total pg-boss jobs — 8 new prediction/forecast jobs with cron schedules"}
{"d":"2026-04-03","t":"FEAT","m":"TIP Proxy Network: residential proxy pool — contributor nodes donate bandwidth, SOCKS5 server (Node.js net only), register/heartbeat/next/rotate/stats API, round-robin routing with uptime+latency scoring"}
{"d":"2026-04-03","t":"FEAT","m":"@tip/proxy-agent: standalone CLI package — tip-agent start/status/stop, configurable bandwidth cap, 30s heartbeat, graceful shutdown"}
{"d":"2026-04-03","t":"UI","m":"Dashboard Network tab: node stats, join-the-network card with token generator, install command box, country breakdown table"}
{"d":"2026-04-03","t":"INFRA","m":"Mac Studio home node: tip-agent running on 192.168.178.213:1081, PROXY_URL=socks5://192.168.178.213:1081 set in PM2 env for scraper+api, ProLabs WAF bypass now active"}
{"d":"2026-04-01","t":"FEAT","m":"Product Intelligence Layer (migration 020): product_issues table (forum/community bugs), condition+marketplace on price_observations, features JSONB on switches+transceivers"}
{"d":"2026-04-01","t":"FEAT","m":"eBay Enricher: scrapes eBay.de for switch/transceiver listings — extracts features, description, refurbished prices, images — nightly via pg-boss"}
{"d":"2026-04-01","t":"FEAT","m":"Community Issues Scraper: extracts known bugs/incompatibilities from Reddit, ServeTheHome, Arista Community, Cisco Community, NetworkEngineering SE"}
{"d":"2026-04-01","t":"DATA","m":"7 pre-seeded community issues: Arista QSFP28 EOS compatibility, Cisco SFP DOM bug, Juniper QFX5120 config tip, SG350 SFP speed limit, MikroTik CRS326 QoS, DCS-7800R3 QSA, UniFi third-party warning"}
{"d":"2026-04-01","t":"FEAT","m":"API: GET /api/switches/:id/issues — known community issues with severity, tags, source links; GET /api/switches/:id/documents — official datasheets+manuals"}
{"d":"2026-04-01","t":"UI","m":"Switch detail modal: shows features array from DB, description, eBay refurbished price, known issues with severity color coding, datasheets with download links"}
{"d":"2026-04-01","t":"FEAT","m":"Datasheet Finder: discovers and links official PDF datasheets/manuals from vendor sites (Arista, Cisco, Juniper, HPE) for existing switches"}
{"d":"2026-04-01","t":"DATA","m":"SMB/campus switch seed: 26 models across Cisco SG/CBS 350/550/CBS350, HPE Aruba 1820/2530/2930F, Ubiquiti UniFi Pro/Aggregation, MikroTik CRS326/354/504, Netgear M4300/M4500, Zyxel XGS"}
{"d":"2026-04-01","t":"FIX","m":"forecast.ts: fixed fiveYearProjection accessor (hype.forecast.fiveYearProjection[n] instead of hype.forecast[n])"}
{"d":"2026-04-01","t":"FEAT","m":"Procurement Intelligence Engine: stock_snapshots, abc_classification, reorder_signals, product_lifecycle_events, market_intelligence tables"}
{"d":"2026-04-01","t":"FEAT","m":"Crawler LLM: Ollama-based two-stage extractor (page type detection + structured product extraction) with vendor profiles for 7 vendors"}
{"d":"2026-04-01","t":"FEAT","m":"ABC classification: dynamic A/B/C turnover scoring from price observations, compatibility breadth, vendor count — computed daily"}
{"d":"2026-04-01","t":"FEAT","m":"Reorder signals: buy_now/wait/hold/monitor with signal strength and reasons — computed daily from stock trends, price trends, lead times"}
{"d":"2026-04-01","t":"DATA","m":"Market intelligence seeded: OFC 2026, AWS CapEx $105B, Azure CapEx $80B+, Coherent 400G ZR+ lead times 16-20w, EU TED €2.1B tenders, ECOC 2026, IEEE 802.3df"}
{"d":"2026-04-01","t":"DATA","m":"Lifecycle events seeded: Cisco SFP-10G-LR EOL 2026-06-30, Juniper SFPP-10GE-ER EOL 2026-09-01, 400ZR ratified, 800G MSA draft"}
{"d":"2026-04-01","t":"UI","m":"Procurement Intel tab: Reorder Signals, ABC Classes table, Market Intel cards, Lifecycle Events — live on dashboard"}
{"d":"2026-04-01","t":"FEAT","m":"Market intelligence scraper: OFC/ECOC, IEEE 802.3, EU TED, Farnell/Mouser lead times, LightReading, FierceTelecom — weekly via pg-boss"}
{"d":"2026-04-01","t":"FIX","m":"Dashboard: garbage product names (scraped-*, All Optical Transceivers) no longer shown as product titles — isGarbageName() filter"}
{"d":"2026-04-01","t":"FIX","m":"Dashboard: competitor comparable prices shown as inline tooltip (ⓘ) instead of block element breaking price row layout"}
{"d":"2026-04-01","t":"UI","m":"Dashboard: 100% VERIFIED badge with white-on-green sub-items (Price ✓, Image ✓, Details ✓) — explicit === true checks, no false positives"}
{"d":"2026-04-01","t":"UI","m":"Dashboard list view: SKU + descriptive name on two lines, Verified column with ★ 100% badge"}
{"d":"2026-04-01","t":"UI","m":"Dashboard detail view: manufacturer product name above image, temperature range decoded (COM → 0–70°C), close button visible on light background"}
{"d":"2026-04-01","t":"DATA","m":"Migration 018: garbage data cleanup — marks scraped-* and category-page scrapes as data_confidence=garbage"}
{"d":"2026-04-01","t":"FEAT","m":"Migration 017: verification tags — price_verified, image_verified, details_verified, fully_verified columns + compute_transceiver_verification() function"}
{"d":"2026-03-31","t":"DATA","m":"Migration 016: data_confidence scoring (garbage/low/medium/high)"}
{"d":"2026-03-31","t":"FEAT","m":"Migration 013: v0.2.0 Sales Intelligence tables — competitor_alerts, price_changes, generated_datasheets, sales_forecasts, blog_posts_v2"}
{"d":"2026-03-31","t":"FEAT","m":"Transport planner route: GET /api/transport — city-pair fiber route recommendations with switch and transceiver BOM"}
{"d":"2026-03-31","t":"FEAT","m":"Blog Engine v2: market_alert, migration_guide, competitor_analysis, buying_guide types with data enrichment pipeline"}
{"d":"2026-03-31","t":"FEAT","m":"Competitor alerts route: GET /api/competitor-alerts — price changes, new products, stock events with acknowledge workflow"}
{"d":"2026-03-30","t":"FEAT","m":"Switch→Flexoptix Finder: GET /api/finder — enter switch model, get matching Flexoptix transceivers with prices and shop links"}
{"d":"2026-03-30","t":"FEAT","m":"MCP Server: 12 tools including find_transceiver, get_compatibility, get_hype_cycle, generate_blog, plan_transport"}
{"d":"2026-03-30","t":"FEAT","m":"Norton-Bass Hype Cycle engine: multigenerational diffusion model for 15 transceiver technologies with adoption curves"}
{"d":"2026-03-30","t":"DATA","m":"440 switches seeded including Cisco, Arista, Juniper, Edgecore, Mellanox, whitebox OCP switches"}
{"d":"2026-03-30","t":"DATA","m":"33,993 compatibility entries — transceiver↔switch compatibility matrix"}
{"d":"2026-03-30","t":"FEAT","m":"Price monitoring: 23 scrapers, 60+ data sources, pg-boss scheduler — permanent monitoring"}
{"d":"2026-03-30","t":"FEAT","m":"Qdrant vector DB integration: hybrid full-text + semantic search across products, FAQ, datasheets, news"}
{"d":"2026-03-30","t":"INFRA","m":"Stack deployed: PostgreSQL 17 + TimescaleDB, Qdrant, Cloudflare R2 for images, PM2"}
{"d":"2026-03-30","t":"DATA","m":"v0.1.0: 5,018 transceivers, 351 vendors seeded from 23 initial scrapers"}
{"d":"2026-04-17","t":"DATA","m":"Vendor cleanup: pruned 242 irrelevant OEM/manufacturer vendors with no transceiver or switch data — 348→106 vendors"}
{"d":"2026-04-18","t":"FEAT","m":"Mouser Electronics API scraper: OEM reference prices for Juniper/Cisco/Arista PIDs — scheduled daily 03:00, MOUSER_API_KEY env var required"}
{"d":"2026-04-18","t":"FEAT","m":"Hype Cycle Engine: Norton-Bass diffusion model fitted to 6 tech generations (10G/100G/400G-QSFP-DD/800G-OSFP/400G-ZR/1.6T). Bass params via grid search, Gartner phase detection, ASP log-linear projection. Seeded market_metrics + hype_cycle_analysis table. Scheduled daily 04:30. API: GET /api/hype-cycle/analysis"}
{"d":"2026-04-18","t":"DATA","m":"migration 039: hype_cycle_analysis table (Bass p/q/M params, phase, score, projected share 1y/3y, ASP current + decline %). market_metrics CHECK extended with hype_score type"}