94 lines
16 KiB
Markdown
94 lines
16 KiB
Markdown
# TIP Changelog
|
||
|
||
Format: `{"d":"YYYY-MM-DD","t":"TYPE","m":"Description"}`
|
||
Types: FEAT · FIX · UI · DATA · AI · INFRA
|
||
|
||
---
|
||
|
||
{"d":"2026-04-11","t":"FIX","m":"Scraper: CRAWLEE_PURGE_ON_START=1 set in withIsolatedStorage — fixes FS.com + ATGBICS crash on startup (SDK_SESSION_POOL_STATE.json not found in fresh isolated storage dir)"}
|
||
{"d":"2026-04-11","t":"FEAT","m":"Scraper: NADDOD, QSFPTEK, AddOn Networks added to pg-boss scheduler (every 2h, slots :48/:52/:55) — 24 pricing queues total, 58 workers"}
|
||
{"d":"2026-04-11","t":"FIX","m":"Scraper: ProLabs rewritten from PlaywrightCrawler (blocked by CloudFront WAF TLS fingerprinting) to fetch-based sitemap scraper — catalog-only (B2B quote model, no public prices)"}
|
||
{"d":"2026-04-11","t":"FIX","m":"Scraper: startup zombie cleanup in index.ts — on daemon restart, active pg-boss jobs older than 5 min are marked failed to allow re-queueing at next cron tick"}
|
||
{"d":"2026-04-11","t":"FIX","m":"Scraper: pre-existing TypeScript build errors fixed (findOrCreateScrapedTransceiver: removed invalid name/url/extractType params; ebay-enricher cheerio type mismatch; community-issues description→summary, publishedDate→published_at)"}
|
||
{"d":"2026-04-04","t":"AI","m":"Blog Engine v5: STEP8b replaced with Reduction Engine v1.0 (5-pass: Repetition Kill → Tech Prune → Flow Rebuild → Weight Correction → Humanization); target 700-1000 words; LaTeX hard delete in Pass 2; title/content alignment in Pass 4; word count range enforcement 600-1300 with warnings"}
|
||
{"d":"2026-04-04","t":"DATA","m":"Blog calibration: Gold Standard 5 added (market alert / pricing article — 2026-04-04; title matches body throughout; no LaTeX; DR4 = MPO-12 not LC duplex; ending lands on title topic not generic close)"}
|
||
{"d":"2026-04-04","t":"AI","m":"Blog Engine v5: system prompt + STEP9 QA hardened with LaTeX hard fail (\\[...\\] destroys flow), DR4 connector hard fail (DR4=MPO-12, FR4=LC duplex), title/content alignment check (12d); WRONG PATTERNS extended with 4 new entries"}
|
||
{"d":"2026-04-04","t":"AI","m":"Blog Engine v5: STEP4b Narrative Control (4-correction pass after draft — root cause assignment, anti-FUD filter, reality reframe, Flexoptix voice check); minimum words 1500→2500; reduction pass 25-35%→15-25%; pipeline now 14 steps, version v5-narrative-control"}
|
||
{"d":"2026-04-04","t":"AI","m":"Blog Engine v5: STEP_LINKEDIN_POST — generates LinkedIn post ≤2800 chars from final article (hook + 3-5 insights + CTA + hashtags); stored in blog_drafts.linkedin_post + linkedin_char_count; hard truncation at 2800 if LLM exceeds limit"}
|
||
{"d":"2026-04-04","t":"DATA","m":"Blog calibration: Gold Standard 4 added (compatible vs OEM narrative correction — 2026-04-04; optic = not root problem, exposes existing issues; correct Flexoptix framing: validation responsibility shifts to operator)"}
|
||
{"d":"2026-04-04","t":"DATA","m":"Migration 024: linkedin_post + linkedin_char_count columns in blog_drafts"}
|
||
{"d":"2026-04-04","t":"FIX","m":"Proxy Network: IP geo-lookup via ip-api.com on register/heartbeat (country_code + city now populated); heartbeat_count column + uptime_pct computed per heartbeat (was always 0.00); dedup fix — register returns existing token for same IP+port; heartbeat no longer overwrites registered IP (prevented IPv6 churn conflicts)"}
|
||
{"d":"2026-04-04","t":"DATA","m":"Proxy Network: migration 023 — heartbeat_count column added, existing node uptime_pct backfilled, duplicate registration from same IPv6 removed (4 nodes → 3)"}
|
||
{"d":"2026-04-04","t":"AI","m":"Blog Engine v4: STEP8b Reduction Pass (25-35% content cut, removes repeated concepts) + STEP8c Style Lock (tone consistency, scope/OPM fix, no inline SKUs) — pipeline now 12 steps, version v4-reduction-stylelock"}
|
||
{"d":"2026-04-04","t":"DATA","m":"Blog calibration: Gold Standard 3 added (Style B troubleshooting — 2026-04-04 field feedback, flowing narrative, zero sections, failure as behavior not scenario)"}
|
||
{"d":"2026-04-04","t":"FIX","m":"Flexoptix scraper: contentHash call fixed (was passing JSON.stringify string, now passes object directly)"}
|
||
{"d":"2026-04-03","t":"FEAT","m":"TIP Proxy Network (packages/proxy-agent): SOCKS5 residential proxy for CloudFront WAF bypass — node registration, heartbeat, load balancing with uptime+latency scoring"}
|
||
{"d":"2026-04-03","t":"FEAT","m":"proxy-agent CLI package (@tip/proxy-agent): tip-agent start/status/stop, configurable bandwidth cap, 30s heartbeat, graceful shutdown"}
|
||
{"d":"2026-04-03","t":"FIX","m":"DB utils: price_verified=true now set in content_hash early-return path (no new observation); image_verified=true auto-set on INSERT and on image_url update in findOrCreateScrapedTransceiver"}
|
||
{"d":"2026-04-03","t":"FIX","m":"pg-boss pool: max connections reduced to 4 + idle_in_transaction_session_timeout=30s — fixed PostgreSQL max_connections exceeded (100/100)"}
|
||
{"d":"2026-04-03","t":"DATA","m":"Image backfill: 178 Flexoptix images added via GraphQL small_image — Optcore images via Playwright gallery — findOrCreateScrapedTransceiver now updates image_url for existing records"}
|
||
{"d":"2026-04-03","t":"FEAT","m":"SmartOptics scraper: DWDM/coherent product catalog, og:image extraction, 8 products with form factor + reach detection"}
|
||
{"d":"2026-04-03","t":"FIX","m":"Fluxlight scraper: price extraction fixed for BigCommerce HTML (data-product-price-without-tax attribute)"}
|
||
{"d":"2026-04-03","t":"AI","m":"Blog Engine v3: STEP4 prose requirement (zero tolerance for ## headers, #### Scenario: patterns, bullet sections) — STEP3 outline as flow plan (3-4 beats) — STEP9 format violations as primary hard fail"}
|
||
{"d":"2026-04-03","t":"FIX","m":"Blog engine: DR4 wavelength corrected to 1310nm=0.35dB/km; scope description fixed (visual tool, not loss measurement device)"}
|
||
{"d":"2026-04-03","t":"FIX","m":"Blog engine: orphaned floating text in fo-blog-pipeline.ts removed (dead code outside template literal causing TypeScript build failure)"}
|
||
{"d":"2026-04-03","t":"FEAT","m":"NOG Talks scraper: DENOG/NANOG/RIPE/ENOG/NLNOG/Euro-IX conference talks — relevance scoring, optical keyword detection, weekly pg-boss job, CtxEvent cross-DB bridge via dblink"}
|
||
{"d":"2026-04-03","t":"FEAT","m":"Hot Topics v2: market_intelligence as SOURCE 3b (0.6+ relevance, urgency mapping per intel_type), NOG Talks as SOURCE 3c (grouped by event with speaker+abstract), limit 20 topics"}
|
||
{"d":"2026-04-03","t":"FIX","m":"Flexoptix scraper: 1G SFP coverage fixed (added SFP LX/SX/ZX queries); SKU suffix stripping (:Sx → base SKU); pagination cap removed (200-product limit was blocking full catalog); Phase 1→2 URL enrichment"}
|
||
{"d":"2026-04-03","t":"FEAT","m":"Prediction intelligence fully_verified trigger: PostgreSQL trigger trg_sync_fully_verified auto-computes fully_verified = price_verified AND image_verified AND details_verified — mass backfill: 258 → 3615 badges"}
|
||
{"d":"2026-04-02","t":"FEAT","m":"Auth: password-protected login page — HMAC-SHA256 signed token, requireAuth middleware on all API routes, dark TIP-themed login page"}
|
||
{"d":"2026-04-02","t":"INFRA","m":"Raspberry Pi fleet: 3x Pi nodes running 24/7 as lightweight scraper workers via WireGuard VPN, pg-boss multi-node queue sharing"}
|
||
{"d":"2026-04-02","t":"INFRA","m":"WireGuard VPN: Pi fleet tunnel for secure PostgreSQL access to production DB"}
|
||
{"d":"2026-04-02","t":"FEAT","m":"Prediction Intelligence System (migration 022): 7 new tables — hyperscaler_capex, distributor_lead_times, github_tech_signals, marketplace_velocity, ai_cluster_announcements, standards_activity, forecast_signals"}
|
||
{"d":"2026-04-02","t":"FEAT","m":"SEC EDGAR scraper: XBRL API, quarterly CapEx for Amazon/Microsoft/Alphabet/Meta — DC-share estimate + YoY growth"}
|
||
{"d":"2026-04-02","t":"FEAT","m":"GitHub Signals scraper: weekly repo_count/commit/stars for 400G/800G/ZR/CMIS/CPO/silicon-photonics tech adoption tracking"}
|
||
{"d":"2026-04-02","t":"FEAT","m":"eBay Velocity scraper: sold/active listing counts + avg price for 9 transceiver search terms — every 12h"}
|
||
{"d":"2026-04-02","t":"FEAT","m":"AI Cluster Announcements scraper: 6 RSS feeds (DataCenterKnowledge, DC Dynamics, Blocks&Files, Next Platform, ServeTheHome) — extracts company, MW, network speed, estimated transceivers"}
|
||
{"d":"2026-04-02","t":"FEAT","m":"Distributor Lead Times scraper: Mouser, Digi-Key, RS Components — in_stock, stock_qty, lead_time_weeks, price — daily"}
|
||
{"d":"2026-04-02","t":"FEAT","m":"Standards Tracker: IEEE 802.3 project table, OIF hot topics, IETF Datatracker API — tracks in-progress/ballot/published status — weekly"}
|
||
{"d":"2026-04-02","t":"FEAT","m":"Forecast Engine: weighted demand index (0-100) from 6 signal types — capex 0.30, ai_clusters 0.25, ebay_velocity 0.20, lead_times 0.15, github 0.06, standards 0.04 — 3/9/12/18 month horizons for 5 technologies"}
|
||
{"d":"2026-04-02","t":"FEAT","m":"NAS sync: datasheet/manual download — PDFs from product_documents organized into switches/transceivers/whitepapers/other"}
|
||
{"d":"2026-04-02","t":"INFRA","m":"Scheduler: 50 total pg-boss jobs — 8 new prediction/forecast jobs with cron schedules"}
|
||
{"d":"2026-04-03","t":"FEAT","m":"TIP Proxy Network: residential proxy pool — contributor nodes donate bandwidth, SOCKS5 server (Node.js net only), register/heartbeat/next/rotate/stats API, round-robin routing with uptime+latency scoring"}
|
||
{"d":"2026-04-03","t":"FEAT","m":"@tip/proxy-agent: standalone CLI package — tip-agent start/status/stop, configurable bandwidth cap, 30s heartbeat, graceful shutdown"}
|
||
{"d":"2026-04-03","t":"UI","m":"Dashboard Network tab: node stats, join-the-network card with token generator, install command box, country breakdown table"}
|
||
{"d":"2026-04-03","t":"INFRA","m":"Mac Studio home node: tip-agent running on 192.168.178.213:1081, PROXY_URL=socks5://192.168.178.213:1081 set in PM2 env for scraper+api, ProLabs WAF bypass now active"}
|
||
{"d":"2026-04-01","t":"FEAT","m":"Product Intelligence Layer (migration 020): product_issues table (forum/community bugs), condition+marketplace on price_observations, features JSONB on switches+transceivers"}
|
||
{"d":"2026-04-01","t":"FEAT","m":"eBay Enricher: scrapes eBay.de for switch/transceiver listings — extracts features, description, refurbished prices, images — nightly via pg-boss"}
|
||
{"d":"2026-04-01","t":"FEAT","m":"Community Issues Scraper: extracts known bugs/incompatibilities from Reddit, ServeTheHome, Arista Community, Cisco Community, NetworkEngineering SE"}
|
||
{"d":"2026-04-01","t":"DATA","m":"7 pre-seeded community issues: Arista QSFP28 EOS compatibility, Cisco SFP DOM bug, Juniper QFX5120 config tip, SG350 SFP speed limit, MikroTik CRS326 QoS, DCS-7800R3 QSA, UniFi third-party warning"}
|
||
{"d":"2026-04-01","t":"FEAT","m":"API: GET /api/switches/:id/issues — known community issues with severity, tags, source links; GET /api/switches/:id/documents — official datasheets+manuals"}
|
||
{"d":"2026-04-01","t":"UI","m":"Switch detail modal: shows features array from DB, description, eBay refurbished price, known issues with severity color coding, datasheets with download links"}
|
||
{"d":"2026-04-01","t":"FEAT","m":"Datasheet Finder: discovers and links official PDF datasheets/manuals from vendor sites (Arista, Cisco, Juniper, HPE) for existing switches"}
|
||
{"d":"2026-04-01","t":"DATA","m":"SMB/campus switch seed: 26 models across Cisco SG/CBS 350/550/CBS350, HPE Aruba 1820/2530/2930F, Ubiquiti UniFi Pro/Aggregation, MikroTik CRS326/354/504, Netgear M4300/M4500, Zyxel XGS"}
|
||
{"d":"2026-04-01","t":"FIX","m":"forecast.ts: fixed fiveYearProjection accessor (hype.forecast.fiveYearProjection[n] instead of hype.forecast[n])"}
|
||
{"d":"2026-04-01","t":"FEAT","m":"Procurement Intelligence Engine: stock_snapshots, abc_classification, reorder_signals, product_lifecycle_events, market_intelligence tables"}
|
||
{"d":"2026-04-01","t":"FEAT","m":"Crawler LLM: Ollama-based two-stage extractor (page type detection + structured product extraction) with vendor profiles for 7 vendors"}
|
||
{"d":"2026-04-01","t":"FEAT","m":"ABC classification: dynamic A/B/C turnover scoring from price observations, compatibility breadth, vendor count — computed daily"}
|
||
{"d":"2026-04-01","t":"FEAT","m":"Reorder signals: buy_now/wait/hold/monitor with signal strength and reasons — computed daily from stock trends, price trends, lead times"}
|
||
{"d":"2026-04-01","t":"DATA","m":"Market intelligence seeded: OFC 2026, AWS CapEx $105B, Azure CapEx $80B+, Coherent 400G ZR+ lead times 16-20w, EU TED €2.1B tenders, ECOC 2026, IEEE 802.3df"}
|
||
{"d":"2026-04-01","t":"DATA","m":"Lifecycle events seeded: Cisco SFP-10G-LR EOL 2026-06-30, Juniper SFPP-10GE-ER EOL 2026-09-01, 400ZR ratified, 800G MSA draft"}
|
||
{"d":"2026-04-01","t":"UI","m":"Procurement Intel tab: Reorder Signals, ABC Classes table, Market Intel cards, Lifecycle Events — live on dashboard"}
|
||
{"d":"2026-04-01","t":"FEAT","m":"Market intelligence scraper: OFC/ECOC, IEEE 802.3, EU TED, Farnell/Mouser lead times, LightReading, FierceTelecom — weekly via pg-boss"}
|
||
{"d":"2026-04-01","t":"FIX","m":"Dashboard: garbage product names (scraped-*, All Optical Transceivers) no longer shown as product titles — isGarbageName() filter"}
|
||
{"d":"2026-04-01","t":"FIX","m":"Dashboard: competitor comparable prices shown as inline tooltip (ⓘ) instead of block element breaking price row layout"}
|
||
{"d":"2026-04-01","t":"UI","m":"Dashboard: 100% VERIFIED badge with white-on-green sub-items (Price ✓, Image ✓, Details ✓) — explicit === true checks, no false positives"}
|
||
{"d":"2026-04-01","t":"UI","m":"Dashboard list view: SKU + descriptive name on two lines, Verified column with ★ 100% badge"}
|
||
{"d":"2026-04-01","t":"UI","m":"Dashboard detail view: manufacturer product name above image, temperature range decoded (COM → 0–70°C), close button visible on light background"}
|
||
{"d":"2026-04-01","t":"DATA","m":"Migration 018: garbage data cleanup — marks scraped-* and category-page scrapes as data_confidence=garbage"}
|
||
{"d":"2026-04-01","t":"FEAT","m":"Migration 017: verification tags — price_verified, image_verified, details_verified, fully_verified columns + compute_transceiver_verification() function"}
|
||
{"d":"2026-03-31","t":"DATA","m":"Migration 016: data_confidence scoring (garbage/low/medium/high)"}
|
||
{"d":"2026-03-31","t":"FEAT","m":"Migration 013: v0.2.0 Sales Intelligence tables — competitor_alerts, price_changes, generated_datasheets, sales_forecasts, blog_posts_v2"}
|
||
{"d":"2026-03-31","t":"FEAT","m":"Transport planner route: GET /api/transport — city-pair fiber route recommendations with switch and transceiver BOM"}
|
||
{"d":"2026-03-31","t":"FEAT","m":"Blog Engine v2: market_alert, migration_guide, competitor_analysis, buying_guide types with data enrichment pipeline"}
|
||
{"d":"2026-03-31","t":"FEAT","m":"Competitor alerts route: GET /api/competitor-alerts — price changes, new products, stock events with acknowledge workflow"}
|
||
{"d":"2026-03-30","t":"FEAT","m":"Switch→Flexoptix Finder: GET /api/finder — enter switch model, get matching Flexoptix transceivers with prices and shop links"}
|
||
{"d":"2026-03-30","t":"FEAT","m":"MCP Server: 12 tools including find_transceiver, get_compatibility, get_hype_cycle, generate_blog, plan_transport"}
|
||
{"d":"2026-03-30","t":"FEAT","m":"Norton-Bass Hype Cycle engine: multigenerational diffusion model for 15 transceiver technologies with adoption curves"}
|
||
{"d":"2026-03-30","t":"DATA","m":"440 switches seeded including Cisco, Arista, Juniper, Edgecore, Mellanox, whitebox OCP switches"}
|
||
{"d":"2026-03-30","t":"DATA","m":"33,993 compatibility entries — transceiver↔switch compatibility matrix"}
|
||
{"d":"2026-03-30","t":"FEAT","m":"Price monitoring: 23 scrapers, 60+ data sources, pg-boss scheduler — permanent monitoring"}
|
||
{"d":"2026-03-30","t":"FEAT","m":"Qdrant vector DB integration: hybrid full-text + semantic search across products, FAQ, datasheets, news"}
|
||
{"d":"2026-03-30","t":"INFRA","m":"Stack deployed: PostgreSQL 17 + TimescaleDB, Qdrant, Cloudflare R2 for images, PM2"}
|
||
{"d":"2026-03-30","t":"DATA","m":"v0.1.0: 5,018 transceivers, 351 vendors seeded from 23 initial scrapers"}
|