Compare commits

...

25 Commits

Author SHA1 Message Date
Rene Fichtmueller
aa977abc97 feat(v0.2.0): Sales Intelligence Engine — Phase 0+A
New API routes:
- GET /api/finder — Switch→Flexoptix transceiver finder with FlexBox coding
- GET /api/competitor-alerts — Competitor intelligence (price changes, new products, stock)
- GET /api/forecast/:technology — Sales forecast 3/9/12/18 months + buy/wait/hold signal
- POST /api/transport/plan — Transport system planner (city→city BOM with fiber providers)

New MCP tools:
- find_flexoptix_for_switch — Customer switch → Flexoptix products
- get_competitor_alerts — Competitor monitoring
- plan_transport — Network transport planning
- forecast_sales — Volume/revenue prediction
- generate_blog — Enhanced blog generation

New DB tables (migration 013):
- competitor_alerts, price_changes, flexoptix_product_map
- sales_forecasts, fiber_providers, fiber_routes, cities
- generated_datasheets, blog_series
- Views: v_price_coverage, v_image_coverage, v_switch_flexoptix_finder

Seed data (migration 014):
- 25 European cities with IX/DC locations + coordinates
- 15 fiber providers (euNetworks, Telia, DTAG, Colt, Zayo, etc.)
- 16 fiber routes with pricing (Germany focus)

Infrastructure:
- Scraper scheduler: 2h Flexoptix, 4h FS.com/Optcore (was 6-8h)
- Change detector for competitor price/stock monitoring
- Image downloader utility with coverage tracking
2026-03-31 08:51:22 +02:00
Rene Fichtmueller
b238815cb5 feat: 5-year forecast area chart + regional adoption heatmap
Dashboard now uses /api/hype-cycle/enriched for forecast data.
Forecast chart: SVG area chart with adoption % per technology over 5 years.
Regional heatmap: table with market share intensity per region per tech.
Both render below the existing hype cycle table.
2026-03-30 20:57:08 +02:00
Rene Fichtmueller
9d9d9ed8ae feat: hype cycle hover tooltips + phase legend, fix switch-docs missing column
Dashboard: Added hover tooltips on hype cycle dots showing phase, adoption %,
peak year, score. Added color-coded phase legend with technology counts.
MCP: Fixed docs_portal_url column reference in switch-docs tool.
2026-03-30 08:25:41 +02:00
Rene Fichtmueller
5a0cbed5a2 feat: dashboard v2, blog expansion, market/cable MCP tools, switch asset scrapers, scraper utilities 2026-03-30 08:07:12 +02:00
Rene Fichtmueller
f940bf2cd4 fix: remove non-existent vendor URL columns, fix text=uuid cast in transceiver lookup 2026-03-30 07:49:54 +02:00
Rene Fichtmueller
891bd018a8 fix: add trust proxy for Cloudflare — fixes ERR_ERL_UNEXPECTED_X_FORWARDED_FOR in rate limiter 2026-03-30 06:41:36 +02:00
Rene Fichtmueller
1853d1c9f1 feat(deploy): add tip-mcp PM2 process (port 3202) + Cloudflare tunnel tip.context-x.org 2026-03-29 08:21:14 +08:00
Rene Fichtmueller
4b452ab49e feat(scrapers+mcp): ATGBICS + ProLabs scrapers, MCP HTTP/SSE server
Scrapers:
- atgbics.ts: PlaywrightCrawler for UK vendor ATGBICS (Shopify store),
  scrapes SFP/SFP+/SFP28/QSFP+/QSFP28/QSFP-DD in GBP, max 50 pages/run
- prolabs.ts: HttpCrawler for ProLabs (Legrand subsidiary), USD pricing,
  category-driven crawl with reach/fiber/speed detection
- Both registered in scheduler (every 8h, staggered) and index.ts CLI

MCP HTTP Server:
- packages/mcp-server/src/http-server.ts: Express + SSEServerTransport
- Exposes all 12 TIP tools via GET /sse + POST /message
- Bearer token auth (MCP_SECRET env), CORS-configurable
- GET /health → { status: "ok", tools: 12 }
- Port: MCP_HTTP_PORT (default 3201)

SQL + tools:
- sql/006-009: seed scripts for whitebox switches, vendors, assets
- switch-docs.ts: MCP tool for switch documentation queries
2026-03-29 02:26:45 +08:00
Rene Fichtmueller
83f4acc976 feat: redesign dashboard — glassmorphism, animated background, gradient accents
Premium dark UI with ambient glow background, glass cards, gradient
text, animated counters, glow effects on hype cycle dots, smooth
transitions, and improved detail panel with regional adoption and
revenue lifecycle data. Fix API data key mismatch (data vs transceivers).
2026-03-28 02:42:55 +13:00
Rene Fichtmueller
66b722a5e4 feat: calibrate regional adoption model with research-backed parameters
Update REGIONAL_LAGS with data from LightCounting, vendor earnings,
OFC market sessions, and Chinese IPO prospectuses. Add price index
per region and segment mix (hyperscaler/telco/enterprise) for
more accurate regional revenue modeling.
2026-03-28 02:34:29 +13:00
Rene Fichtmueller
c6308e93c0 feat: massive scraper expansion + hype cycle engine + lifecycle prediction
New scrapers:
- GBICS.com (BigCommerce, GBP prices, 10 categories, 78 products)
- Juniper HCT (Next.js SSR parser, 475 transceivers with specs/EOL)
- SFPcables.com (Magento store, 16 categories, 78 products)
- Fluxlight (BigCommerce, 6 pages, 118 products)
- Champion ONE (compatible vendor scraper)

Scraper fixes:
- 10Gtek: rewritten to parse HTML spec tables (152 products)
- Flexoptix: fix price extraction from Magento Hyva HTML
- Register all scrapers in CLI (--gbics, --juniper, --sfpcables, etc.)

Hype Cycle Engine enhancements:
- Data-driven enrichment from scraped vendor/price data
- Revenue lifecycle prediction (peak year, decline, revenue index)
- Regional adoption model (NA, China, APAC, Europe, RoW with lag coefficients)
- New API endpoints: /enriched, /lifecycle, /regional/:tech

DB growth: 89 → 1,168 transceivers, 0 → 416 prices, 6 vendors
Qdrant: 1,162 products embedded with nomic-embed-text

Research: Norton-Bass model, standards-to-market timelines, hype signals
2026-03-28 02:30:19 +13:00
Rene Fichtmueller
d43b98e91b feat: add Flexoptix product catalog scraper, register in CLI
Scrapes flexoptix.net product catalog across 9 categories (SFP through OSFP).
Extracts product names, prices, form factors, reach, fiber type, wavelength.
CLI: --flexoptix flag, integrated into --all.
2026-03-28 01:02:34 +13:00
Rene Fichtmueller
46af736db3 fix: hype cycle findTechnology matched wrong tech (1G instead of 1.6T)
findTechnology used loose includes() matching — '1.6T OSFP-XD' matched
'1G SFP' first because query contained '1'. Now matches exact name first,
then by speed prefix with proper unit parsing (G/T).
2026-03-28 01:00:52 +13:00
Rene Fichtmueller
9a5b21a19a feat: complete dashboard redesign — professional data-dense UI, click-through detail panels
- Removed AI-slop gradient header, replaced with compact info bar
- JetBrains Mono for data, Inter for text — proper engineering tool aesthetic
- Unified slide-in detail panel for hype cycle, transceivers, and blog drafts
- Transceiver table: clickable rows open full spec sheet in side panel
- Form factor dropdown filter on transceivers tab
- Blog drafts: click to read full content in panel
- Tighter spacing, smaller fonts, higher information density
- Escape key closes panel, responsive breakpoints
- No gratuitous badges/glow — clean data presentation
2026-03-28 00:56:27 +13:00
Rene Fichtmueller
a4f738d093 feat: interactive SVG hype cycle visualization with click-through detail panel 2026-03-28 00:52:17 +13:00
Rene Fichtmueller
94c6b7f42d fix: replace alert() with slide-in toast notification in dashboard 2026-03-28 00:47:08 +13:00
Rene Fichtmueller
1b0b602aa4 feat: Phase 8 — Dashboard frontend + static serving
Single-file dashboard with 6 tabs: Overview, Semantic Search,
Hype Cycle, Transceivers, News, Blog Engine. Dark theme, no
build step, served as static HTML from Express.

- Overview: health stats, vector collection counts, recent news
- Semantic Search: query across all 6 Qdrant collections
- Hype Cycle: Norton-Bass table with phase colors + position bars
- Transceivers: searchable table with form factor/speed/reach
- News: semantic news search with source links
- Blog: generate drafts from templates, view draft history

Live at: https://transceiver-db.context-x.org/dashboard/
2026-03-28 00:37:10 +13:00
Rene Fichtmueller
f48a809e40 feat: Phase 7 — Blog generator + scraper scheduler activation
Blog draft engine generates structured markdown from all Qdrant
collections (products, news, FAQ, troubleshooting). Supports 4
topic types: hype_cycle, comparison, new_product, tutorial.

- routes/blog.ts: POST /api/blog/generate, GET/PUT endpoints
- ecosystem.config.js: Added tip-scraper PM2 process
- Scraper scheduler (pg-boss) now running on Erik with 8 job queues
- News scraper running every 6 hours on Erik
2026-03-28 00:32:08 +13:00
Rene Fichtmueller
0a63307505 feat: Phase 6 — FAQ + troubleshooting knowledge base embeddings
19 curated FAQ entries covering form factors, fiber types, reach,
compatibility, WDM, power, and emerging tech (CPO, LPO, 400ZR).
10 troubleshooting guides with symptom/cause/solution format.

All 6 Qdrant collections now populated:
- product_embeddings: 89 transceivers
- datasheet_chunks: 40 chunks (OCR pipeline)
- faq_embeddings: 19 FAQ entries
- troubleshooting_embeddings: 10 guides
- news_embeddings: 33 articles
- manual_chunks: 0 (pending manual ingestion)
2026-03-28 00:24:50 +13:00
Rene Fichtmueller
122ca8444d feat: Phase 5 — OCR pipeline + document/news search
Docling-powered OCR pipeline: PDF → markdown → chunks → Ollama embed → Qdrant.
News embedding seeder for news_embeddings collection.
Document and news semantic search API endpoints.

- embeddings/ocr-pipeline.ts: Docling convert → chunk → embed pipeline
- embeddings/seed-news.ts: Batch embed news_articles into Qdrant
- routes/documents.ts: POST /api/documents/process, GET /api/documents
- routes/search.ts: GET /search/documents, GET /search/news endpoints
- sql/005-documents.sql: Add chunks_count, processed_at to documents table
- Ollama + nomic-embed-text installed on Erik (CPU mode)
- 89 products + 40 datasheet chunks + 33 news articles in Qdrant
2026-03-28 00:22:01 +13:00
Rene Fichtmueller
0260d0b365 feat: Phase 4 — Vector embeddings + semantic search
Ollama nomic-embed-text (768 dim) → Qdrant vector search pipeline.
Embeds all 89 transceivers with rich text representation and payload
filters (form_factor, speed_gbps, fiber_type, wdm_type).

- embeddings/client.ts: Ollama embed + Qdrant upsert/search
- embeddings/seed-products.ts: Batch seeder for product_embeddings
- routes/search.ts: GET /api/search, /search/products, /search/stats
- 6 Qdrant collections: products, datasheets, FAQs, manuals, troubleshooting, news
2026-03-28 00:05:29 +13:00
Rene Fichtmueller
a6f7968393 feat: Phase 3 — Norton-Bass Hype Cycle Engine
Implements the full Norton-Bass Multigenerational Diffusion Model for
transceiver technology lifecycle forecasting.

Math: Bass diffusion F(t) + logistic adoption S(t) = L / (1 + e^(-k(t-t0)))
Parameters: p (innovation ~0.03), q (imitation ~0.3-0.5), m (market potential)

Phase Classification Engine (composite score):
  30% Port shipment share + 20% ASP decline rate + 15% Standards maturity
  + 15% Interop validation + 10% Vendor trajectory + 10% Media sentiment

11 technologies tracked: 1G → 10G → 25G → 40G → 100G → 400G → 800G → 1.6T
  + CPO, LPO, 400ZR Coherent
5-year adoption forecast per technology

API: GET /api/hype-cycle (all) + GET /api/hype-cycle/:tech (detail)
Live: https://transceiver-db.context-x.org/api/hype-cycle
2026-03-27 23:35:57 +13:00
Rene Fichtmueller
ae411cb575 feat: add Flexoptix vendor scraper, 10Gtek pricing scraper, expand news feeds
- Flexoptix vendor scraper: 285 supported switch vendors ingested from
  flexoptix.net/en/supported-vendors/ (our own data, no restrictions)
- 10Gtek Playwright scraper: Chinese OEM competitor pricing (SFP+, SFP28,
  QSFP+, QSFP28, QSFP-DD categories)
- News feeds expanded: added Lightwave, Fierce Telecom, Data Center Knowledge,
  SDxCentral, Cisco Blogs, Arista Blog (11 total sources)
- Scheduler updated: 8 job queues with appropriate intervals
- DB now: 297 vendors, 89 transceivers, 33 news articles (13 relevant)
2026-03-27 23:17:42 +13:00
Rene Fichtmueller
649e6a9796 feat: Phase 2 — MCP Server with 12 tools
Implements all 12 MCP tools from CONCEPT document:
- search_transceivers: Full-text + spec filter search with pricing
- check_compatibility: Switch ↔ transceiver compatibility lookup
- get_pricing: Current prices + 30-day history across all vendors
- compare_prices: Multi-vendor price comparison with savings analysis
- get_competitor_stock: Live competitor stock monitoring (sales opportunities)
- suggest_alternatives: Similar spec alternatives optimized for price/availability
- get_templates: FlexBox coding and switch config template finder
- search_knowledge_base: Troubleshooting FAQ search (PostgreSQL full-text)
- search_manuals: Switch manual and datasheet search
- get_hype_cycle: Norton-Bass adoption forecast + Gartner phase classification
- get_market_news: Aggregated news with relevance scoring
- generate_blog_draft: Data-driven blog drafts saved to blog_drafts table

Transport: stdio (MCP protocol 2024-11-05)
Config: .mcp.json for Claude Code integration
Verified: all 12 tools registered, search_transceivers returns DB results
2026-03-27 16:48:34 +13:00
Rene Fichtmueller
b43bdd3060 feat: TIP Phase 0+1 — monorepo, DB schema, API, scraper engine
Phase 0 - Foundation:
- Restructure into npm workspace monorepo (packages/core, api, scraper)
- PostgreSQL 17 + TimescaleDB schema (15 tables incl. hypertables)
- Docker Compose for local dev (PostgreSQL on 5433 + Qdrant)
- Express 5 API on port 3200 with 6 routes
- Seed script to migrate 159 transceivers + 42 standards from npm package
- Erik server setup script + PM2 ecosystem config

Phase 1 - Scraper Engine:
- Crawlee + Playwright framework with pg-boss scheduler
- FS.com scraper (PlaywrightCrawler, anti-bot workaround)
- Optcore.net scraper (WP REST API enumeration + PlaywrightCrawler)
  - Uses /wp-json/wp/v2/product to get 2000+ product URLs
  - Playwright renders individual product pages for price extraction
- Cisco TMG Matrix scraper (compatibility data)
- News RSS aggregator (optics.org, SPIE, Network World, Nature Photonics)
  - Keyword relevance scoring for transceiver/fiber topics
  - xml2js with malformed XML sanitization
- SHA-256 content hashing for change detection (skip unchanged records)
- pg-boss v10 with explicit queue creation before scheduling
2026-03-27 16:27:31 +13:00
127 changed files with 37513 additions and 37 deletions

5
.gitignore vendored
View File

@ -5,3 +5,8 @@ dist/
.env*
.dev.vars
*.local
# Downloaded product assets (images, PDFs)
assets/images/
assets/datasheets/
assets/manuals/

10
.mcp.json Normal file
View File

@ -0,0 +1,10 @@
{
"mcpServers": {
"tip": {
"command": "npx",
"args": ["tsx", "packages/mcp-server/src/index.ts"],
"cwd": "/Users/renefichtmueller/Desktop/Claude Code/github-repos/transceiver-db",
"description": "Transceiver Intelligence Platform — 12 tools for transceiver search, pricing, compatibility, hype cycle, and blog generation"
}
}
}

File diff suppressed because it is too large Load Diff

View File

@ -0,0 +1,738 @@
# Optical Networking Equipment: Demo-to-Market Predictive Timeline Model
> Research compiled 2026-03-28 for the Transceiver Intelligence Platform (TIP)
> Data from OFC/ECOC proceedings, IEEE standards, MSA publications, vendor press releases, Cignal AI, LightCounting, Dell'Oro Group
---
## Table of Contents
1. [Historical OFC/ECOC Demo-to-Market Timelines](#1-historical-timelines)
2. [Switch/Router ASIC Generation Timelines](#2-asic-timelines)
3. [The Lag Formula](#3-the-lag-formula)
4. [Demand Cascade Model](#4-demand-cascade-model)
5. [Export Control Impact](#5-export-control-impact)
6. [Predictive Timeline Calculator](#6-predictive-calculator)
---
## 1. Historical OFC/ECOC Demo-to-Market Timelines {#1-historical-timelines}
### 1.1 10G SFP+
| Milestone | Date | Source |
|-----------|------|--------|
| IEEE 802.3ae study group formed | Nov 1999 | IEEE archives |
| IEEE 802.3ae ratified (10GbE standard) | Jun 2002 | [IEEE 802.3ae](https://resources.l-p.com/knowledge-center/what-is-ieee-802-3ae-10-gigabit-ethernet) |
| First 10G modules (XENPAK form factor) | 2002-2003 | First MSA for 10GE; largest form factor |
| XFP MSA published | 2003-2004 | Intermediate form factor between XENPAK and SFP+ |
| SFP+ MSA (SFF-8431) published | ~2006 | SFP+ became smallest, lowest-power 10G form factor |
| First SFP+ volume shipments | 2007-2008 | Industry adoption ramped with switch platforms |
| 10GBASE-T (802.3an) ratified | Jun 2006 | Extended 10G to copper |
| Mainstream SFP+ adoption | 2009-2010 | De facto standard for ToR/access |
**Total cycle: ~8 years** from IEEE standard (2002) to mainstream (2010). However, the SFP+ form factor itself took ~4 years from MSA (2006) to mainstream (2010).
### 1.2 40G QSFP+
| Milestone | Date | Source |
|-----------|------|--------|
| IEEE 802.3ba study group | Nov 2007 | IEEE archives |
| IEEE 802.3ba ratified (40G/100G Ethernet) | Jun 2010 | IEEE 802.3ba standard |
| First 40G QSFP+ commercial modules | 2010-2011 | QSFP+ MSA based on 4x10G lanes |
| Volume production begins | 2012-2013 | Kaiam, Finisar shipping high volume |
| Mainstream data center adoption | 2013-2015 | Standard for aggregation/ToR switches |
**Total cycle: ~5 years** from standard (2010) to mainstream (2015). Form-factor-to-volume: ~2 years.
### 1.3 100G QSFP28
| Milestone | Date | Source |
|-----------|------|--------|
| IEEE 802.3bm task force (100G over MMF/short-reach) | 2013 | Defined 4x25G lane architecture |
| QSFP28 MSA published | 2013-2014 | Based on QSFP+ with 4x25G lanes |
| First OFC demos (CWDM4, PSM4) | OFC 2015 | [Kaiam CWDM4 100G QSFP28 demo](https://www.businesswire.com/news/home/20150319005175/en/Kaiam-Introduces-CWDM4-100G-QSFP28-Transceiver-400G) |
| ColorChip adds PSM4 to QSFP28 portfolio | OFC 2016 | [ColorChip PSM4 announcement](https://www.globenewswire.com/news-release/2016/03/18/940853/0/en/) |
| InnoLight volume shipments (17 QSFP28 SKUs) | Mar 2017 | [InnoLight OFC 2017](https://www.innolight.com/en/news/newsinfo/13.html) |
| Oclaro 40km interop demo (QSFP28 ER4-Lite) | Mar 2017 | [Oclaro OFC 2017](https://www.prnewswire.com/news-releases/oclaro-showcases-industrys-first-live-40km-interoperability-demo-between-100g-extended-reach-qsfp28-and-cfp2-at-ofc-2017-300426690.html) |
| Market maturity (cost-effective vs 10G/40G) | 2017-2018 | More $/Gbit efficient than 10G SFP+ and 40G QSFP+ |
**Total cycle: ~4 years** from MSA (2014) to mainstream (2018). Demo-to-volume: ~2 years (OFC 2015 to Mar 2017).
### 1.4 100G Coherent (CFP to QSFP28-DCO)
| Milestone | Date | Source |
|-----------|------|--------|
| CFP MSA (first 100G pluggable form factor) | 2009-2010 | [CFP Wikipedia](https://en.wikipedia.org/wiki/C_Form-factor_Pluggable); 10x10G lanes |
| CFP2 MSA (half the size of CFP) | 2012 | [ProOptix history](https://www.prooptix.com/news/transceiver-form-factors/) |
| CFP2-ACO (OIF Interoperability Agreement) | 2016 | DSP on host line card; analog signal to module |
| CFP2-DCO (DSP integrated in module) | 2017-2018 | Software-configurable 100G/200G; [Acacia CFP2-DCO](https://acacia-inc.com/product/cfp2/) |
| Adtran first 100ZR QSFP28 DCO | 2022 | First coherent 100G in QSFP28 |
| Coherent QSFP28-DCO with Steelerton DSP | 2023 | [Coherent 100G QSFP28-DCO](https://www.coherent.com/news/press-releases/100g-qsfp28-dco-transceiver); <5W power |
| Coherent dual-laser QSFP28-DCO GA | Mar 2026 | [Coherent GA announcement](https://www.globenewswire.com/news-release/2026/03/06/3251306/11543/en/) |
**Total coherent miniaturization cycle: ~13 years** from CFP (2010) to QSFP28-DCO (2023). Each form factor shrink: ~3-4 years.
### 1.5 400G QSFP-DD / OSFP
| Milestone | Date | Source |
|-----------|------|--------|
| QSFP-DD MSA Rev 0.2 | May 2016 | [QSFP-DD spec](http://www.qsfp-dd.com/wp-content/uploads/2016/05/QSFP-DDrev0-2-3-29-16.pdf) |
| QSFP-DD MSA Rev 2.0 (form factor spec) | Mar 2017 | [QSFP-DD MSA announcement](http://www.qsfp-dd.com/qsfp-dd-msa-group-announces-form-factor-specification/) |
| InnoLight introduces 400G OSFP at OFC 2017 | Mar 2017 | [InnoLight OFC 2017](https://www.prnewswire.com/news-releases/innolight-technology-announced-volume-shipments-of-17-100g-qsfp28-products-and-the-introduction-of-400g-osfp-at-ofc-2017-300421866.html) |
| Oclaro 400G CFP8 PAM4 demo at OFC 2017 | Mar 2017 | [Oclaro CFP8](https://www.prnewswire.com/news-releases/oclaro-samples-400g-cfp8-pam4-enabled-transceiver-showcases-live-demo-at-ofc-2017-300425943.html) |
| Finisar 400G transceiver demos at OFC 2018 | Mar 2018 | [Finisar OFC 2018](https://picmagazine.net/article/103776/Finisar_Demos_New_400G_Transceivers_At_OFC_2018) |
| IEEE 802.3bs ratified (400G Ethernet) | Dec 2017 | 400GBASE standard |
| QSFP-DD Hardware Rev 5.0 | Jul 2019 | [QSFP-DD Rev 5.0](http://www.qsfp-dd.com/wp-content/uploads/2019/07/QSFP-DD-Hardware-rev5p0.pdf) |
| First commercial 400G QSFP-DD/OSFP modules | 2019-2020 | Broadcom TH3 switches enabled demand |
| Volume production | 2020-2021 | Driven by hyperscaler leaf/spine upgrades |
| Mainstream adoption | 2021-2022 | De facto DC interconnect standard |
**Total cycle: ~5 years** from first demos (OFC 2017) to mainstream (2022). MSA-to-volume: ~3 years.
### 1.6 400G ZR Coherent
| Milestone | Date | Source |
|-----------|------|--------|
| OIF 400ZR project initiated | ~2016-2017 | OIF response to hyperscaler DCI demands |
| OIF 400ZR IA published | Mar 2020 | [OIF 400ZR spec](https://convergedigest.com/oif-publishes-400zr-implementation/) |
| Acacia/Inphi sampling 400ZR QSFP-DD | H2 2020 | [Inphi COLORZ II](https://convergedigest.com/inphi-ramps-shipments-of-400zr-and-zr/) |
| Fujitsu sample shipments begin | Oct 2020 | [Fujitsu 400G ZR launch](https://opticalconnectionsnews.com/2020/10/fujitsu-launches-400g-zr-transceivers/) |
| Inphi commercial availability & ramp | 2021 | [Inphi ramp announcement](https://convergedigest.com/inphi-ramps-shipments-of-400zr-and-zr/) |
| Molex volume production | Early 2022 | [Molex 400G ZR volume](https://www.molex.com/en-us/news/molex-ramps-production-of-400g-zr-qsfp-dd-coherent-optical) |
| Broad volume deployment | 2022-2023 | >100% CAGR in ZR/ZR+ per Cignal AI |
**Total cycle: ~6 years** from OIF project start (~2017) to volume (2022). Spec-to-volume: ~2 years (Mar 2020 to early 2022).
### 1.7 800G DR8
| Milestone | Date | Source |
|-----------|------|--------|
| Intel first 800G DR8 OSFP sample | OFC 2021 | [Gazettabyte Intel 800G DR8](https://www.gazettabyte.com/home/2021/6/29/intel-details-its-800-gigabit-dr8-optical-module.html) |
| IEEE 802.3ck ratified (100G/lane electrical) | 2022 | Enabled 8x100G = 800G |
| Initial shipments (SR8 for AI) | 2022 | Few thousand units |
| LESSENGERS 800G SR8 volume production | Q4 2023 | [LESSENGERS announcement](https://www.semiconductor-today.com/news_items/2023/sep/lessengers-280923.shtml) |
| Shipments exceed 1M units | 2023 | Dominated by SR8 for AI clusters |
| Hyper Photonix 800G DR8 GA | May 2024 | [Hyper Photonix GA](https://www.businesswire.com/news/home/20240517136062/en/) |
| Forecast: 8M 800GbE modules shipped | 2024 | Cignal AI OFC 2024 preview |
| 800G mainstream / displacing 400G | 2025 | [Cignal AI 800GbE growth](https://cignal.ai/2025/05/800gbe-optics-shipments-to-grow-60-in-2025/) |
**Total cycle: ~4 years** from first sample (OFC 2021) to mainstream (2025). Demo-to-volume: ~2.5 years. This is faster than previous generations due to AI demand pull.
### 1.8 800G ZR/ZR+ Coherent
| Milestone | Date | Source |
|-----------|------|--------|
| OIF 800G Coherent project initiated | Dec 2020 | [OIF 800G Coherent](https://www.oiforum.com/technical-work/hot-topics/800g-coherent/) |
| Coherent first 800G ZR/ZR+ QSFP-DD unveiled | Dec 2023 | [Coherent 800G ZR announcement](https://www.semiconductor-today.com/news_items/2023/dec/coherent-081223.shtml) |
| OIF first public 800ZR multivendor interop | OFC 2024 | OIF plugfest |
| Alpha samples available | Q1 2024 | Coherent Corp. |
| OIF 800ZR Implementation Agreement published | Oct 2024 | OIF 800ZR IA |
| Coherent 800G ZR/ZR+ QSFP-DD GA | Mar 2025 | [Coherent GA](https://www.coherent.com/news/press-releases/general-availability-of-800g-zr-zrplus-in-qsfp-dd-form-factor) |
| L-band 800G ZR/ZR+ QSFP-DD | Sep 2024 | [Coherent L-band](https://www.coherent.com/news/press-releases/800g-l-band-qsfp-dd-telecom-transceiver) |
| Volume ramp forecast: >200K units, >$1B revenue | 2026 | [Cignal AI forecast](https://cignal.ai/2025/07/800g-coherent-pluggable-shipments-to-exceed-1b-revenue-in-2026/) |
**Total cycle: ~6 years** from OIF project (Dec 2020) to volume (2026). Spec-to-GA: ~5 months (Oct 2024 to Mar 2025). First demo-to-volume: ~3 years (Dec 2023 to 2026).
### 1.9 1.6T Transceivers
| Milestone | Date | Source |
|-----------|------|--------|
| Eoptolink 1.6T module demo (4xFR2, OSFP-XD) | OFC 2023 | First industry 1.6T demo |
| InnoLight 1.6T OSFP-XD demo | OFC 2024 | Live demonstration |
| First EML-based 1.6T samples ship | Q4 2024 - Q1 2025 | Conventional technology |
| IEEE 802.3dj (800G/1.6T standard, 224G/lane) | Expected mid-2026 | Under development |
| OFC 2025: Multiple live 1.6T demos | Mar 2025 | [Eoptolink Gen2 1.6T](https://www.eoptolink.com/news/361-eoptolink-launches-its-gen2-1-6t-osfp-and-osfp-rhs-transceiver-family-at-ofc-2025), [Jabil 1.6T](https://investors.jabil.com/news/news-details/2025/Jabil-Launches-1-6T-Pluggable-Transceiver/), [ATOP 1.6T demo](https://www.atoptechnology.com/ofc-2025-live-demo-atops-1-6t-osfp224-dr8-siph-module-in-action-for-next-gen-ai/) |
| SiPh-based 1.6T modules available | H2 2025 | Post mass-production readiness |
| Interop plugfest (Keysight Santa Clara) | Dec 2025 | 224G SerDes validation |
| AOI first volume order ($200M+ from hyperscaler) | Mar 2026 | [AOI volume order](https://www.globenewswire.com/news-release/2026/03/09/3251675/9986/en/) |
| Volume ramp forecast | 2026 | Dell'Oro, Cignal AI projections |
| Predicted mainstream | 2027 | >10% of addressable ports |
**Total cycle (projected): ~4 years** from first demo (OFC 2023) to mainstream (2027). Demo-to-volume: ~3 years. Accelerated by AI demand.
### 1.10 CPO (Co-Packaged Optics)
| Milestone | Date | Source |
|-----------|------|--------|
| Broadcom Tomahawk 4-Humboldt (1st gen CPO) | 2021 | First CPO chipset |
| Broadcom Tomahawk 5-Bailly (2nd gen, first volume CPO) | 2023 | Shipped to select hyperscalers |
| Broadcom 3rd gen CPO (200G/lane) | May 2025 | [Broadcom CPO glimpse](https://news.broadcom.com/) |
| Meta: 1M link-hours without link flap in lab | Oct 2025 | Broadcom announcement |
| NVIDIA CPO switches (Quantum-X: H2 2025, Spectrum-X: H2 2026) | GTC 2025 | [NVIDIA CPO plans](https://www.techradar.com/pro/nvidia-is-planning-post-copper-1-6tbps-network-tech) |
| Small initial deployments | 2026 | [Cignal AI CPO report](https://cignal.ai/2025/02/co-packaged-optics-inevitable-but-not-imminent/) |
| Volume manufacturing capability | 2027 | ASE/industry consensus |
| Widespread scale-out adoption | 2028-2029+ | [EDN CPO status 2026](https://www.edn.com/where-co-packaged-optics-cpo-technology-stands-in-2026/) |
**Total cycle: ~7+ years** from first demo (2021) to predicted widespread adoption (2028+). This is longer because CPO requires fundamental changes to packaging, connectors, and supply chain.
### 1.11 LPO (Linear Pluggable Optics)
| Milestone | Date | Source |
|-----------|------|--------|
| LPO concept development | 2022-2023 | Industry discussions on eliminating in-module DSP |
| Eoptolink 200G/lane LPO demo, 100G/lane 800G LPO mass production | OFC 2024 | [Eoptolink LPO](https://www.lightwaveonline.com/home/article/14310702/eoptolink-showcases-200g-linear-drive-pluggable-optics-at-ofc-2024) |
| LPO MSA spec (100G/lane) released | Mar 25, 2025 | [LPO MSA release](https://www.globenewswire.com/news-release/2025/03/25/3048840/0/en/) |
| LPO MSA first plugfest (interop validation) | Feb 2025 | Pre-OFC 2025 |
| FLEXOPTIX LPO products (400G/800G QSFP/OSFP) | 2025 | [FLEXOPTIX LPO](https://www.flexoptix.net/en/blog/blog/introducing-linear-pluggable-optics) |
| ECOC 2025: 800G LPO interop confirmed | Oct 2025 | [Ethernet Alliance ECOC 2025](https://ethernetalliance.org/blog/2025/10/27/ecoc-2025-interoperability-at-800g-is-given-advancing-toward-1-6t/) |
| Market share outlook | 2025-2026 | Small % of 800G market (per Cignal AI); larger potential at 1.6T |
**Note:** LPO is not a new speed generation but a new architecture. It may capture significant share at 1.6T where power savings (50% vs DSP) become critical.
---
## 2. Switch/Router ASIC Generation Timelines {#2-asic-timelines}
### 2.1 Broadcom Tomahawk (Data Center Switching)
| ASIC | Bandwidth | Process | Announced | First Switch Shipments | Source |
|------|-----------|---------|-----------|----------------------|--------|
| Tomahawk 1 | 3.2 Tbps | 28nm | Sep 2014 | Spring 2015 (~6 mo) | [Broadcom TH1](https://www.broadcom.com/products/ethernet-connectivity/switching/strataxgs/bcm56960-series) |
| Tomahawk 2 | 6.4 Tbps | 16nm | Oct 2016 | ~Fall 2017 (~12 mo) | [NextPlatform TH2](https://www.nextplatform.com/2016/10/31/broadcom-strikes-100g-ethernet-harder-tomahawk-ii/) |
| Tomahawk 3 | 12.8 Tbps | 16nm | Dec 2017 | Dec 2017 (same!) | [Broadcom TH3 press](https://www.globenewswire.com/news-release/2017/12/19/1266218/0/en/) |
| Tomahawk 4 | 25.6 Tbps | 7nm | Dec 2019 | 2020-2021 (~12-18 mo) | [NextPlatform TH4](https://www.nextplatform.com/2019/12/12/broadcom-launches-another-tomahawk-into-the-datacenter/) |
| Tomahawk 5 | 51.2 Tbps | 5nm | ~Aug 2022 | Late 2022/2023 (~6 mo) | [Broadcom TH5](https://investors.broadcom.com/news-releases/news-release-details/broadcom-ships-tomahawk-5-industrys-highest-bandwidth-switch) |
| Tomahawk Ultra | 51.2 Tbps | 4nm | 2024 | 2024 | [Broadcom TH-Ultra](https://investors.broadcom.com/news-releases/news-release-details/broadcom-ships-tomahawk-ultra-reimagining-ethernet-switch-hpc) |
| Tomahawk 6 | 102.4 Tbps | 3nm | Jun 2025 | Mar 2026 (~9 mo) | [Broadcom TH6 volume](https://www.broadcom.com/company/news/product-releases/63146) |
| Tomahawk 6 CPO (Davisson) | 102.4 Tbps | 3nm | Oct 2025 | Shipping Oct 2025 | [Broadcom Davisson](https://investors.broadcom.com/news-releases/news-release-details/broadcom-announces-tomahawkr-6-davisson-industrys-first-1024) |
**Cadence:** Bandwidth doubles approximately every 2 years. ASIC announcement to first switch: 6-18 months.
### 2.2 Broadcom Jericho (Routing / AI Fabric)
| ASIC | Bandwidth | Process | Announced | Platform Availability | Source |
|------|-----------|---------|-----------|----------------------|--------|
| Jericho2 | 9.6 Tbps | 16nm | 2018 | 2019 | [Broadcom Jericho](https://www.techinsights.com/blog/broadcom-retargets-jericho-ai-clusters) |
| Jericho2c | 4.8 Tbps | 16nm | 2019 | 2020 | Service provider market |
| Jericho2c+ | 14.4 Tbps | 7nm | 2020 (sampling) | 2021 | [Gazettabyte J2c+](https://gazettabyte.squarespace.com/home/2020/11/17/broadcoms-144-terabit-jericho2c-router-chip.html) |
| Jericho3-AI (BCM88890) | 28.8 Tbps | 5nm | Apr 2023 | Oct 2024 (first white boxes) | [Broadcom J3-AI](https://www.broadcom.com/company/news/product-releases/61156), [DriveNets/Accton](https://www.prnewswire.com/news-releases/drivenets-and-accton-technology-launch-the-highest-performance-ethernet-based-ai-networking-solution-302273214.html) |
| Jericho4 | Multi-Tbps HyperPorts | 3nm | Aug 2025 (shipping) | 2025-2026 | [Broadcom J4](https://investors.broadcom.com/news-releases/news-release-details/broadcom-ships-jericho4-enabling-distributed-ai-computing-across) |
**Cadence:** ~18-24 months between generations. ASIC-to-platform: 12-18 months.
### 2.3 NVIDIA/Mellanox Spectrum (Ethernet Switching)
| ASIC | Bandwidth | Process | Announced | Shipped | Source |
|------|-----------|---------|-----------|---------|--------|
| Spectrum | 6.4 Tbps | - | ~2016 | 2016-2017 | SN2000 series |
| Spectrum-2 | 12.8 Tbps | - | ~2018 | 2019 | SN3000 series; 200G ports |
| Spectrum-3 | 12.8 Tbps | 16nm | Mar 2020 | 2021 | [NVIDIA Spectrum-3](https://network.nvidia.com/files/doc-2020/pb-spectrum-3.pdf); 400G support |
| Spectrum-4 | 51.2 Tbps | TSMC 4N | GTC 2022 | 2023 | [NextPlatform Spectrum-4](https://www.nextplatform.com/2022/04/01/spectrum-4-ethernet-leaps-to-800-gb-sec-with-nvidia-circuits/); 800G ports |
| Spectrum-X (CPO, SN6000) | 102.4-409.6 Tbps | TBD | GTC 2025 | 2026 (SN6810/SN6800) | [NVIDIA GTC 2025](https://www.techradar.com/pro/nvidia-is-planning-post-copper-1-6tbps-network-tech) |
### 2.4 NVIDIA/Mellanox ConnectX (Network Adapters)
| NIC | Max Speed | Announced | First Shipments | Source |
|-----|-----------|-----------|-----------------|--------|
| ConnectX-5 | 100 Gb/s | Jun 2016 | Oct 2016 | [Mellanox CX-5](https://www.hpcwire.com/2016/06/16/mellanox-advances-network-computing-connectx-5-adapter/), [InsideHPC CX-5 shipping](https://insidehpc.com/2016/10/mellanox-begins-shipments-of-connectx-5-adapter/) |
| ConnectX-6 | 200 Gb/s | Jul 2019 | Mid 2019 | [Mellanox CX-6](https://network.nvidia.com/files/doc-2020/pb-connectx-6-dx-en-dellemc.pdf) |
| ConnectX-6 Dx | 200 Gb/s | Aug 2019 | Late 2019 | [CX-6 Dx](https://nvidianews.nvidia.com/news/releases-20210113-6829469) |
| ConnectX-7 | 400 Gb/s | Nov 2021 (GTC) | H2 2022 | [NVIDIA CX-7 GTC](https://www.servethehome.com/nvidia-quantum-2-400g-switches-and-connectx-7-at-gtc-fall-2021/) |
| ConnectX-8 SuperNIC | 800 Gb/s | Nov 2024 (SC24) | Q2 2025 (production) | [ServeTheHome CX-8](https://www.servethehome.com/this-is-the-next-gen-nvidia-connectx-8-supernic-for-800gbps-networking/) |
| ConnectX-9 SuperNIC | 1.6 Tb/s | Announced (Rubin) | TBD (~2027) | Spectrum-6 / BlueField-4 platform |
### 2.5 Cisco Silicon One
| ASIC | Bandwidth | Role | Announced | Platform GA | Source |
|------|-----------|------|-----------|-------------|--------|
| Q100 | 10.8 Tbps | Routing | Dec 2019 | Dec 2019 (Cisco 8000) | [Cisco Q100](https://investor.cisco.com/news/news-details/2019/Cisco-Unveils-Plan-for-Building-Internet-for-the-Next-Decade-of-Digital-Innovation/) |
| Q200 / Q200L | 12.8 Tbps | Routing / Switching | Oct 2020 | 2021 | [Cisco Q200](https://blogs.cisco.com/sp/ciscosilicononeq200announcement) |
| P100 | 19.2 Tbps | Routing (modular) | 2021 | 2022-2023 | [Cisco P100](https://www.cisco.com/c/en/us/solutions/collateral/silicon-one/silicon-one-p100-processor-ds.html) |
| G100 | 25.6 Tbps | Switching | 2021-2022 | 2022-2023 | [Cisco G100](https://www.cisco.com/c/en/us/solutions/collateral/silicon-one/datasheet-c78-744833.html) |
| G200 | 51.2 Tbps | Switching (AI) | 2024 | 2024-2025 | [Cisco G200](https://www.cisco.com/c/en/us/solutions/collateral/silicon-one/silicon-one-g200-ds.html) |
| K100, E100 | Various | Edge/Enterprise | 2025 | 2025 | Extension to enterprise |
### 2.6 Intel Barefoot Tofino (CANCELLED)
| ASIC | Bandwidth | Status | Source |
|------|-----------|--------|--------|
| Tofino 1 | 6.4 Tbps | Shipped (2016+) | [Intel Tofino](https://www.intel.com/content/www/us/en/products/network-io/programmable-ethernet-switch.html) |
| Tofino 2 | 12.8 Tbps | Shipped (7nm, CoWoS) | [ServeTheHome Tofino2](https://www.servethehome.com/intel-tofino2-next-gen-programmable-switch-detailed/) |
| Tofino 3 | 25.6 Tbps | **CANCELLED Jan 2023** | [Intel exits switching](https://www.fool.com/investing/2023/01/29/intel-exits-another-non-core-business/); P4 software open-sourced 2025 |
Intel acquired Barefoot Networks in Jun 2019, but cancelled the Tofino line in Jan 2023 as part of $3B cost-cutting. Existing Tofino 1/2 products remain available from vendors like Asterfusion.
---
## 3. The Lag Formula {#3-the-lag-formula}
Based on all historical data points collected above, here are the empirically derived lag values:
### 3.1 Technology Development Lags
| Transition | Typical Lag | Range | Trend |
|-----------|-------------|-------|-------|
| **IEEE standard publication -> First commercial transceivers** | 18-24 months | 12-36 mo | Shortening |
| **MSA spec publication -> First samples** | 6-12 months | 3-18 mo | Stable |
| **First OFC demo -> Volume production** | 24-36 months | 18-48 mo | Shortening (AI pull) |
| **First OFC demo -> Mainstream adoption (>10% ports)** | 36-48 months | 30-60 mo | Shortening |
### 3.2 ASIC-to-Deployment Lags
| Transition | Typical Lag | Range | Source |
|-----------|-------------|-------|--------|
| **ASIC announcement -> First switch platform GA** | 9-18 months | 6-24 mo | Broadcom TH history |
| **Switch GA -> Transceiver demand ramp** | 6-12 months | 3-18 mo | Qualification + deployment |
| **ASIC tape-out -> Full transceiver ecosystem ramp** | 18-30 months | 12-36 mo | Combined |
### 3.3 Regional Deployment Lags
| Transition | Typical Lag | Range | Notes |
|-----------|-------------|-------|-------|
| **US hyperscaler deployment -> Enterprise deployment** | 18-36 months | 12-48 mo | Hyperscalers are early adopters |
| **US deployment -> China deployment** | 3-6 months | 0-12 mo | Chinese vendors dominate manufacturing; fast adoption |
| **US deployment -> Europe deployment** | 12-24 months | 6-36 mo | Slower procurement cycles, GDPR considerations |
| **US deployment -> APAC (ex-China) deployment** | 12-18 months | 6-24 mo | Japan/Korea faster; SEA/India slower |
| **US deployment -> RoW deployment** | 18-36 months | 12-48 mo | Varies enormously by country |
### 3.4 Coherent Optics Miniaturization Lag
| Transition | Typical Lag |
|-----------|-------------|
| **CFP -> CFP2** | ~3 years |
| **CFP2 -> CFP2-DCO** | ~5 years |
| **CFP2-DCO -> QSFP-DD-DCO** | ~4 years |
| **400G ZR spec -> Volume** | ~2 years |
| **800G ZR spec -> Volume** | ~2 years (projected) |
### 3.5 Acceleration Factors (AI Era)
The AI/ML demand cycle is compressing timelines by approximately 30-40% compared to the cloud computing era (2012-2020):
| Factor | Impact |
|--------|--------|
| Hyperscaler pre-ordering | -6 to -12 months (demand pull) |
| Direct NVIDIA-to-transceiver vendor procurement | -3 to -6 months (bypassing OEM) |
| Chinese vendor manufacturing agility | -3 to -6 months (rapid ramp) |
| Power/thermal constraints driving urgency | -3 to -6 months (competitive pressure) |
---
## 4. Demand Cascade Model {#4-demand-cascade-model}
### 4.1 The Cascade Flow
```
[AI Training Cluster Plans]
|
v
[GPU/XPU Production Forecasts]
| (1:1 GPU-to-NIC ratio for scale-out)
v
[Switch Fabric Requirements]
| (spine-leaf topology, radix determines port count)
v
[Port Count per Switch]
| (e.g., TH5: 64x800G, TH6: 64x1.6T)
v
[Transceiver Demand per Port]
| (speed x reach = specific SKU)
v
[Revenue Forecast per Transceiver Type]
```
### 4.2 Concrete Example: GB200 NVL72
Per [SemiAnalysis](https://newsletter.semianalysis.com/p/gb200-hardware-architecture-and-component) and [NADDOD analysis](https://www.naddod.com/blog/nvidia-gb200-interconnect-architecture-analysis-nvlink-infiniband-and-future-trends):
| Component | Quantity per NVL72 Rack | Notes |
|-----------|------------------------|-------|
| GPUs (Blackwell B200) | 72 | Per rack |
| NICs (CX-7 or CX-8) | 72 | 1:1 GPU-to-NIC ratio |
| Scale-out OSFP ports | 72 | 400G (CX-7) or 800G (CX-8) |
| Spine switch OSFP ports | Varies by topology | 2:1 or 3:1 oversubscription |
| Total optical modules per 576-GPU cluster | ~18,432 | [FiberMall estimate](https://www.fibermall.com/blog/nvidia-blackwell-development-for-dac-lacc-1600g-osfp-xd.htm) |
**Speed transition:**
- CX-7 era (2024-2025): 400G SR4/DR4 per GPU port
- CX-8 era (2025-2026): 800G DR4 per GPU port, 1.6T DR8 per switch port
- CX-9 era (2027+): 1.6T per GPU port, 3.2T per switch port
### 4.3 Total Addressable Market Drivers
| Data Source | What It Reveals | Forecast |
|-------------|----------------|----------|
| Hyperscaler CapEx (quarterly reports) | Total infrastructure spend | $600-690B in 2026 ([IEEE ComSoc](https://techblog.comsoc.org/2025/12/22/hyperscaler-capex-600-bn-in-2026/), [Futurum](https://futurumgroup.com/insights/ai-capex-2026-the-690b-infrastructure-sprint/)) |
| NVIDIA GPU production (H100/B200/GB200) | GPU count -> NIC count -> optics count | [SemiAnalysis GB200](https://newsletter.semianalysis.com/p/gb200-hardware-architecture-and-component) |
| Data center construction (Synergy, JLL, CBRE) | Site capacity -> future networking spend | Multi-year pipeline |
| Optical component supplier earnings | Revenue = realized demand | Ciena backlog ~$5B heading into 2026 |
### 4.4 Key Market Forecasts (2025-2029)
| Metric | 2024 | 2025 | 2026 | Source |
|--------|------|------|------|--------|
| 800GbE module shipments | ~8M | ~12.8M (60% growth) | ~20M+ | [Cignal AI](https://cignal.ai/2025/05/800gbe-optics-shipments-to-grow-60-in-2025/) |
| 1.6T module shipments | ~2.7M | ~4.2M | ~20M+ | Industry estimates |
| 800G coherent (ZR/ZR+) units | <50K | ~100K | >200K ($1B+ revenue) | [Cignal AI](https://cignal.ai/2025/07/800g-coherent-pluggable-shipments-to-exceed-1b-revenue-in-2026/) |
| Hyperscaler CapEx | ~$256B | ~$443B | ~$600-690B | CreditSights, Futurum |
| AI back-end network market | - | - | >$20B by 2028 | Dell'Oro |
| Optical interconnect market | - | $21.9B (2026) | $40B (2031) | Mordor Intelligence |
### 4.5 Transceiver Revenue Per Unit Economics
| Speed | Avg ASP (2025) | Trend |
|-------|----------------|-------|
| 400G DR4 | $150-250 | Declining |
| 800G SR8 | $300-500 | Declining as volume ramps |
| 800G DR8 | $500-800 | At scale pricing |
| 800G 2xFR4 | $600-900 | SM premium |
| 1.6T DR8 | $1,500-2,500 | Early premium, declining |
| 400G ZR | $2,000-3,000 | Mature |
| 800G ZR/ZR+ | $4,000-6,000 | Early premium |
---
## 5. Export Control Impact {#5-export-control-impact}
### 5.1 US/EU Export Control Timeline
| Date | Action | Impact on Optical |
|------|--------|-------------------|
| Oct 2022 | Biden administration first controls | Limited advanced chip access; optical transceivers NOT directly restricted |
| Oct 2023 | Controls tightened | DSP chips (Broadcom, Marvell) restricted for some end-uses |
| Jan 2025 | AI Diffusion Rule | Broader restrictions on advanced AI computing equipment |
| Mar 2025 | Trump administration additional restrictions | More Chinese entities blacklisted |
### 5.2 Impact on Chinese Optical Transceiver Ecosystem
**Key finding:** Optical transceivers themselves are NOT directly export-controlled, but the DSP chips inside them are the vulnerability point.
| Factor | Status | Source |
|--------|--------|--------|
| Chinese vendor market share (800G) | >60% globally, >70% of 800G market | Omdia data |
| InnoLight 2024 revenue | RMB 23.86B (+122.6% YoY) | [InnoLight financials](https://iamfabian.substack.com/p/pluggables-power-and-geopolitics) |
| Eoptolink 2024 revenue | RMB 8.65B (+179% YoY) | [Eoptolink financials](https://iamfabian.substack.com/p/pluggables-power-and-geopolitics) |
| DSP dependency (Broadcom/Marvell) | ~50% of module power; critical component | Export-controlled for certain end-uses |
| LPO as strategic hedge | Eliminates in-module DSP; -20% power, -30% cost | Reduces US tech dependency |
| Chinese DSP startups | Aluksen, EOChip, Hengxin, InSiGa, Leadingspeed, Luxic, MiniSilicon, Photonic Tech, Sitrus, UXFastic | Domestic substitution push |
| Manufacturing diversification | Eoptolink Thailand factory for North American shipments | Tariff and compliance mitigation |
| SMIC vs TSMC gap for DSP | SMIC limited to ~7nm (DUV); TSMC at 3nm (EUV) | 3-5nm DSPs require TSMC |
### 5.3 Modeling Regulatory Risk
For the predictive model, export controls introduce:
1. **DSP availability risk:** If Broadcom/Marvell DSPs become restricted for a specific end-use, Chinese module vendors must either:
- Switch to domestic DSPs (12-18 month qualification delay)
- Adopt LPO architecture (6-12 month redesign)
- Source DSPs through third-party channels (uncertain)
2. **Timeline impact by scenario:**
| Scenario | Impact on Chinese Vendor Timeline |
|----------|----------------------------------|
| Status quo (current controls) | No impact; Chinese vendors dominate |
| DSP export ban for Chinese transceiver vendors | +12-24 months for domestic DSP qualification |
| Full optical component controls | +24-36 months; unlikely given US vendor dependency |
| LPO adoption accelerates | -6 months (removes DSP bottleneck entirely) |
3. **For Chinese domestic market:** +6-18 months lag vs Western hyperscaler deployment, primarily due to GPU access restrictions limiting AI cluster buildouts.
---
## 6. Predictive Timeline Calculator {#6-predictive-calculator}
### 6.1 The Formula
```
T_samples = T_current + LAG_milestone_to_samples
T_volume = T_samples + LAG_samples_to_volume
T_mainstream = T_volume + LAG_volume_to_mainstream
```
Where the lag values depend on:
#### Milestone-to-Samples Lag Table
| Current Milestone | Lag to First Samples | Confidence |
|-------------------|---------------------|------------|
| Academic paper only | 36-60 months | +/- 18 mo |
| First OFC/ECOC demo | 12-24 months | +/- 9 mo |
| MSA/IEEE spec published | 6-12 months | +/- 6 mo |
| ASIC dependency announced | Add 6-12 months from ASIC GA | +/- 6 mo |
| Interop plugfest completed | 3-6 months | +/- 3 mo |
#### Samples-to-Volume Lag Table
| Technology Type | Lag to Volume | Confidence |
|----------------|---------------|------------|
| Incremental upgrade (same form factor, higher speed) | 6-12 months | +/- 3 mo |
| New form factor (e.g., QSFP-DD, OSFP-XD) | 12-18 months | +/- 6 mo |
| New architecture (e.g., coherent, CPO) | 18-36 months | +/- 12 mo |
| Disruptive technology (e.g., CPO at scale) | 24-48 months | +/- 18 mo |
#### Volume-to-Mainstream Lag Table
| Market Segment | Lag to >10% Ports | Confidence |
|---------------|-------------------|------------|
| US hyperscaler | 0-6 months (often concurrent with volume) | +/- 3 mo |
| China hyperscaler (Alibaba, Tencent, ByteDance) | 3-9 months | +/- 6 mo |
| Enterprise (US) | 18-36 months | +/- 12 mo |
| Enterprise (Europe) | 24-42 months | +/- 12 mo |
| Service provider | 12-24 months | +/- 9 mo |
### 6.2 ASIC Dependency Modifier
If a transceiver requires a specific switching ASIC:
```
T_samples = max(T_from_milestone, T_asic_ga + 3 months)
```
The transceiver cannot ramp before the switch ASIC is available. Key dependencies:
| Transceiver Speed | Required ASIC Generation | ASIC GA |
|-------------------|--------------------------|---------|
| 400G | Broadcom TH3+ / Spectrum-3+ | Available since 2017 |
| 800G | Broadcom TH5+ / Spectrum-4+ | Available since 2023 |
| 1.6T | Broadcom TH6 / Spectrum-X / CX-8 | TH6: Mar 2026, CX-8: Q2 2025 |
| 3.2T | Next-gen (TH7? / Spectrum-6) | ~2028 projected |
### 6.3 Worked Examples
#### Example 1: 1.6T OSFP-XD DR8
**Input:**
- Technology: 1.6T OSFP-XD DR8
- Current milestone: Volume orders placed (Mar 2026)
- ASIC dependency: Broadcom Tomahawk 6 (GA Mar 2026)
**Calculation:**
- T_samples: Q4 2024 (already happened)
- T_volume: Q3 2026 (AOI $200M order ships Q3 2026)
- T_mainstream (US hyperscaler): H2 2026 - H1 2027
- T_mainstream (Enterprise US): 2028-2029
- T_mainstream (Europe): 2029-2030
**Confidence:** Medium-High (ASIC available, volume orders placed)
#### Example 2: 3.2T OSFP (hypothetical next-gen)
**Input:**
- Technology: 3.2T OSFP (16x200G or 8x400G)
- Current milestone: Concept/early research (448G PAM4 SerDes expected ~2027)
- ASIC dependency: Next-gen (~TH7, expected ~2028)
**Calculation:**
- T_first_demo: OFC 2027 (+/- 6 mo)
- T_samples: H2 2028 (+/- 9 mo)
- T_volume: H2 2029 - H1 2030 (+/- 12 mo)
- T_mainstream (US hyperscaler): 2030 (+/- 12 mo)
- T_mainstream (Enterprise): 2032+ (+/- 18 mo)
**Confidence:** Low (depends on 448G SerDes and next-gen ASIC)
#### Example 3: CPO at Scale-Out
**Input:**
- Technology: CPO (scale-out Ethernet)
- Current milestone: Lab validation complete (Meta 1M link-hours, Oct 2025)
- ASIC dependency: NVIDIA Spectrum-X CPO (H2 2026) / Broadcom Davisson (Oct 2025)
**Calculation:**
- T_initial_deployment: 2026 (small scale)
- T_volume: 2027-2028 (manufacturing capability)
- T_mainstream (>10% of DC switch ports): 2029-2030
- T_mainstream (enterprise): Unlikely before 2032
**Confidence:** Low-Medium (manufacturing scaling is the key unknown)
### 6.4 Regional Rollout Timeline Modifier
Apply these offsets from US hyperscaler deployment:
```
T_region = T_us_hyperscaler + REGIONAL_OFFSET
```
| Region | Offset (months) | Notes |
|--------|----------------|-------|
| US Hyperscaler | 0 (baseline) | Google, Meta, Microsoft, Amazon |
| China Hyperscaler | +3 to +6 | Fast adoption but GPU access limited |
| Japan/Korea Enterprise | +12 to +18 | NTT, KDDI, SK Telecom early |
| Europe Service Provider | +12 to +24 | DT, Orange, Telefonica |
| US Enterprise | +18 to +36 | Fortune 500 DC upgrades |
| Europe Enterprise | +24 to +42 | Longer procurement, GDPR |
| India/SEA | +18 to +30 | Jio, Tata leading; rest slower |
| LATAM/Africa | +30 to +48 | Limited DC infrastructure |
### 6.5 Algorithm Implementation (Pseudocode)
```python
def predict_timeline(
technology: str,
current_milestone: str, # "paper", "demo", "spec", "samples", "volume"
asic_dependency: str | None,
asic_ga_date: date | None,
is_new_form_factor: bool = False,
is_new_architecture: bool = False,
ai_demand_driven: bool = True,
) -> dict:
# Base lag from current milestone to samples
milestone_lags = {
"paper": (36, 60, 18), # (min, max, uncertainty) months
"demo": (12, 24, 9),
"spec": (6, 12, 6),
"interop": (3, 6, 3),
"samples": (0, 0, 0),
"volume": (-12, -6, 3), # Already past samples
}
min_lag, max_lag, uncertainty = milestone_lags[current_milestone]
base_samples_date = today + months(avg(min_lag, max_lag))
# ASIC dependency check
if asic_dependency and asic_ga_date:
asic_ready = asic_ga_date + months(3)
base_samples_date = max(base_samples_date, asic_ready)
# Samples to volume lag
if is_new_architecture:
volume_lag = months(27) # 18-36 range
elif is_new_form_factor:
volume_lag = months(15) # 12-18 range
else:
volume_lag = months(9) # 6-12 range
# AI demand acceleration factor
if ai_demand_driven:
volume_lag *= 0.65 # 35% acceleration
volume_date = base_samples_date + volume_lag
# Regional rollout
regional = {
"US_hyperscaler": volume_date,
"China_hyperscaler": volume_date + months(4),
"Japan_Korea": volume_date + months(15),
"Europe_SP": volume_date + months(18),
"US_enterprise": volume_date + months(27),
"Europe_enterprise": volume_date + months(33),
"India_SEA": volume_date + months(24),
"LATAM_Africa": volume_date + months(39),
}
# Confidence intervals
confidence = {
"samples": uncertainty,
"volume": uncertainty + 3 if is_new_form_factor else uncertainty,
"mainstream": uncertainty + 6,
}
return {
"predicted_samples": base_samples_date,
"predicted_volume": volume_date,
"predicted_mainstream": volume_date + months(12),
"confidence_months": confidence,
"regional_rollout": regional,
}
```
### 6.6 Historical Validation
| Technology | Predicted (using formula) | Actual | Delta |
|-----------|--------------------------|--------|-------|
| 100G QSFP28 (from OFC 2015 demo) | Volume: Q1 2017 | Volume: Mar 2017 | 0 mo |
| 400G QSFP-DD (from OFC 2017 demo) | Volume: Q1 2020 | Volume: H1 2020 | +3 mo |
| 400G ZR (from spec Mar 2020) | Volume: Q1 2022 | Volume: Early 2022 | 0 mo |
| 800G DR8 (from OFC 2021 demo) | Volume: Q1 2024 | Volume: Mid 2024 | +3 mo |
| 1.6T (from OFC 2023 demo) | Volume: Q1 2026 | Volume: Q3 2026 (projected) | +6 mo |
**Average prediction error: +2.4 months** (formula is slightly optimistic).
---
## Key Data Sources for Ongoing Model Updates
| Source | URL | What It Provides | Update Frequency |
|--------|-----|------------------|-----------------|
| Cignal AI | https://cignal.ai | Optical market forecasts, shipment data | Monthly/Quarterly |
| LightCounting | https://www.lightcounting.com | Transceiver shipment volumes, pricing | Monthly |
| Dell'Oro Group | https://www.delloro.com | Data center networking, optical transport | Quarterly |
| OFC Conference | https://www.ofcconference.org | Annual demos, product launches | Annual (March) |
| ECOC Conference | https://www.ecocexhibition.com | European demos, product launches | Annual (September/October) |
| OIF | https://www.oiforum.com | Implementation Agreements, interop | As published |
| IEEE 802.3 | https://www.ieee802.org/3/ | Ethernet standards | As ratified |
| Broadcom press releases | https://www.broadcom.com/company/news/product-releases | ASIC announcements | As released |
| NVIDIA networking | https://www.nvidia.com/en-us/networking/ | Switch/NIC announcements | As released |
| Hyperscaler quarterly earnings | SEC filings | CapEx guidance, AI spending | Quarterly |
---
## Sources
### Transceiver Timelines
- [InnoLight OFC 2017 - 100G QSFP28 Volume](https://www.innolight.com/en/news/newsinfo/13.html)
- [Oclaro OFC 2017 - 100G ER4 QSFP28](https://www.prnewswire.com/news-releases/oclaro-showcases-industrys-first-live-40km-interoperability-demo-between-100g-extended-reach-qsfp28-and-cfp2-at-ofc-2017-300426690.html)
- [Kaiam OFC 2015 - 100G QSFP28 + 400G demo](https://www.businesswire.com/news/home/20150319005175/en/)
- [ColorChip OFC 2016 - 100G PSM4 QSFP28](https://www.globenewswire.com/news-release/2016/03/18/940853/0/en/)
- [Finisar OFC 2018 - 400G demos](https://picmagazine.net/article/103776/Finisar_Demos_New_400G_Transceivers_At_OFC_2018)
- [Oclaro OFC 2017 - 400G CFP8](https://www.prnewswire.com/news-releases/oclaro-samples-400g-cfp8-pam4-enabled-transceiver-showcases-live-demo-at-ofc-2017-300425943.html)
- [QSFP-DD MSA specifications](http://www.qsfp-dd.com/specification/)
- [OIF 400ZR IA](https://convergedigest.com/oif-publishes-400zr-implementation/)
- [Inphi 400ZR ramp](https://convergedigest.com/inphi-ramps-shipments-of-400zr-and-zr/)
- [Molex 400G ZR volume](https://www.molex.com/en-us/news/molex-ramps-production-of-400g-zr-qsfp-dd-coherent-optical)
- [Fujitsu 400G ZR launch](https://opticalconnectionsnews.com/2020/10/fujitsu-launches-400g-zr-transceivers/)
- [Gazettabyte - Intel 800G DR8](https://www.gazettabyte.com/home/2021/6/29/intel-details-its-800-gigabit-dr8-optical-module.html)
- [LESSENGERS 800G volume](https://www.semiconductor-today.com/news_items/2023/sep/lessengers-280923.shtml)
- [Hyper Photonix 800G DR8 GA](https://www.businesswire.com/news/home/20240517136062/en/)
- [Cignal AI 800GbE growth](https://cignal.ai/2025/05/800gbe-optics-shipments-to-grow-60-in-2025/)
- [Coherent 800G ZR/ZR+ unveiled](https://www.semiconductor-today.com/news_items/2023/dec/coherent-081223.shtml)
- [Coherent 800G ZR/ZR+ GA](https://www.coherent.com/news/press-releases/general-availability-of-800g-zr-zrplus-in-qsfp-dd-form-factor)
- [Cignal AI 800G coherent $1B forecast](https://cignal.ai/2025/07/800g-coherent-pluggable-shipments-to-exceed-1b-revenue-in-2026/)
- [Eoptolink Gen2 1.6T OFC 2025](https://www.eoptolink.com/news/361-eoptolink-launches-its-gen2-1-6t-osfp-and-osfp-rhs-transceiver-family-at-ofc-2025)
- [Jabil 1.6T launch](https://investors.jabil.com/news/news-details/2025/Jabil-Launches-1-6T-Pluggable-Transceiver/)
- [AOI first $200M+ 1.6T volume order](https://www.globenewswire.com/news-release/2026/03/09/3251675/9986/en/)
- [Coherent dual-laser QSFP28-DCO GA](https://www.globenewswire.com/news-release/2026/03/06/3251306/11543/en/)
### ASIC Timelines
- [Broadcom TH3](https://www.globenewswire.com/news-release/2017/12/19/1266218/0/en/)
- [Broadcom TH4 (NextPlatform)](https://www.nextplatform.com/2019/12/12/broadcom-launches-another-tomahawk-into-the-datacenter/)
- [Broadcom TH5](https://investors.broadcom.com/news-releases/news-release-details/broadcom-ships-tomahawk-5-industrys-highest-bandwidth-switch)
- [Broadcom TH6](https://www.broadcom.com/company/news/product-releases/63146)
- [Broadcom TH6 volume Mar 2026](https://markets.financialcontent.com/stocks/article/marketminute-2026-3-26-the-great-ethernet-pivot)
- [Broadcom Davisson CPO](https://investors.broadcom.com/news-releases/news-release-details/broadcom-announces-tomahawkr-6-davisson-industrys-first-1024)
- [Broadcom J3-AI](https://www.broadcom.com/company/news/product-releases/61156)
- [Broadcom J4](https://investors.broadcom.com/news-releases/news-release-details/broadcom-ships-jericho4-enabling-distributed-ai-computing-across)
- [DriveNets/Accton J3-AI white box](https://www.prnewswire.com/news-releases/drivenets-and-accton-technology-launch-the-highest-performance-ethernet-based-ai-networking-solution-302273214.html)
- [NVIDIA Spectrum-4](https://www.nextplatform.com/2022/04/01/spectrum-4-ethernet-leaps-to-800-gb-sec-with-nvidia-circuits/)
- [NVIDIA Spectrum-X CPO](https://www.techradar.com/pro/nvidia-is-planning-post-copper-1-6tbps-network-tech)
- [Mellanox CX-5 announcement](https://www.hpcwire.com/2016/06/16/mellanox-advances-network-computing-connectx-5-adapter/)
- [Mellanox CX-5 shipping](https://insidehpc.com/2016/10/mellanox-begins-shipments-of-connectx-5-adapter/)
- [NVIDIA CX-7 GTC 2021](https://www.servethehome.com/nvidia-quantum-2-400g-switches-and-connectx-7-at-gtc-fall-2021/)
- [NVIDIA CX-8 SuperNIC](https://www.servethehome.com/this-is-the-next-gen-nvidia-connectx-8-supernic-for-800gbps-networking/)
- [Cisco Silicon One Q100](https://investor.cisco.com/news/news-details/2019/Cisco-Unveils-Plan-for-Building-Internet-for-the-Next-Decade-of-Digital-Innovation/)
- [Cisco Silicon One Q200](https://blogs.cisco.com/sp/ciscosilicononeq200announcement)
- [Cisco Silicon One G100](https://www.cisco.com/c/en/us/solutions/collateral/silicon-one/datasheet-c78-744833.html)
- [Cisco Silicon One G200](https://www.cisco.com/c/en/us/solutions/collateral/silicon-one/silicon-one-g200-ds.html)
- [Intel Tofino cancelled](https://www.fool.com/investing/2023/01/29/intel-exits-another-non-core-business/)
### CPO & LPO
- [Cignal AI CPO report](https://cignal.ai/2025/02/co-packaged-optics-inevitable-but-not-imminent/)
- [EDN CPO status 2026](https://www.edn.com/where-co-packaged-optics-cpo-technology-stands-in-2026/)
- [LPO MSA spec release](https://www.globenewswire.com/news-release/2025/03/25/3048840/0/en/)
- [FLEXOPTIX LPO introduction](https://www.flexoptix.net/en/blog/blog/introducing-linear-pluggable-optics)
- [Eoptolink LPO OFC 2024](https://www.lightwaveonline.com/home/article/14310702/eoptolink-showcases-200g-linear-drive-pluggable-optics-at-ofc-2024)
- [Ethernet Alliance ECOC 2025](https://ethernetalliance.org/blog/2025/10/27/ecoc-2025-interoperability-at-800g-is-given-advancing-toward-1-6t/)
### Demand & CapEx
- [IEEE ComSoc - Hyperscaler CapEx $600B+](https://techblog.comsoc.org/2025/12/22/hyperscaler-capex-600-bn-in-2026/)
- [Futurum - AI CapEx $690B](https://futurumgroup.com/insights/ai-capex-2026-the-690b-infrastructure-sprint/)
- [SemiAnalysis - GB200 architecture](https://newsletter.semianalysis.com/p/gb200-hardware-architecture-and-component)
- [FiberMall - NVIDIA Blackwell optics demand](https://www.fibermall.com/blog/nvidia-blackwell-development-for-dac-lacc-1600g-osfp-xd.htm)
- [NADDOD - GB200 interconnect analysis](https://www.naddod.com/blog/nvidia-gb200-interconnect-architecture-analysis-nvlink-infiniband-and-future-trends)
### Export Controls & Geopolitics
- [Pluggables, Power, and Geopolitics (iamfabian)](https://iamfabian.substack.com/p/pluggables-power-and-geopolitics)
- [CRS - US Export Controls on China](https://www.congress.gov/crs-product/R48642)
- [American Affairs - China Semiconductor Evolution](https://americanaffairsjournal.org/2024/11/the-evolution-of-chinas-semiconductor-industry-under-u-s-export-controls/)
- [CSIS - Limits of Chip Export Controls](https://www.csis.org/analysis/limits-chip-export-controls-meeting-china-challenge)
### Standards & Specifications
- [IEEE 802.3ae (10GbE)](https://resources.l-p.com/knowledge-center/what-is-ieee-802-3ae-10-gigabit-ethernet)
- [OIF 400ZR spec (PDF)](https://www.oiforum.com/wp-content/uploads/OIF-400ZR-02.0.pdf)
- [OIF 800G Coherent](https://www.oiforum.com/technical-work/hot-topics/800g-coherent/)
- [QSFP-DD MSA Rev 5.0](http://www.qsfp-dd.com/wp-content/uploads/2019/07/QSFP-DD-Hardware-rev5p0.pdf)
- [LPO MSA](https://www.lpo-msa.org/home.html)

View File

@ -0,0 +1,684 @@
# Hype Cycle Signal Research: Quantifiable Data Inputs for Automatic Technology Positioning
**Date:** 2026-03-28
**For:** Transceiver Intelligence Platform (TIP) — Hype Cycle Engine
**Status:** Deep Research Complete — Ready for Implementation Planning
---
## Executive Summary
This document catalogs **10 quantifiable signal categories** that can feed the TIP Hype Cycle Engine to automatically position optical transceiver technologies (400G, 800G, 1.6T, QSFP-DD, OSFP, silicon photonics, coherent pluggable, co-packaged optics, etc.) on a Norton-Bass-derived hype cycle.
**Key finding:** A composite of 5-6 signals provides robust positioning. No single signal is sufficient alone. The recommended **Phase 1 implementation** (3 signals, all free, all validated) can be built in ~2 weeks.
---
## Signal Catalog
---
### 1. PATENT DATA (Technology Innovation Signal)
**What it measures:** R&D investment intensity, innovation velocity, technology maturation
**Hype cycle relevance:** Patents LEAD actual market adoption by 3-5 years. Patent filing surges correlate with "Technology Trigger" and early "Peak of Inflated Expectations."
#### Data Source: USPTO PatentsView API (migrating to data.uspto.gov March 2026)
| Attribute | Detail |
|-----------|--------|
| **API URL** | `https://search.patentsview.org/api/v1/patent/` |
| **Auth** | API key required (header `X-Api-Key`). Free but new grants temporarily suspended during migration to data.uspto.gov |
| **Rate Limit** | 45 requests/minute |
| **Update Frequency** | Quarterly |
| **Cost** | Free (CC BY 4.0 license) |
| **Python Library** | `requests` (REST API), `patentsview2` (R package, no maintained Python equivalent) |
| **Implementation Complexity** | 2/5 |
#### Relevant CPC Classes for Optical Transceivers
| CPC Class | Description |
|-----------|-------------|
| **H04B10** | Transmission systems employing electromagnetic waves other than radio waves (optical communication) |
| **G02B6** | Light guides; structural details of fibre-optic arrangements |
| **H01S5** | Semiconductor lasers (VCSELs, DFB, EML — core transceiver components) |
| **H04J14** | Optical multiplex systems (WDM, DWDM) |
| **G02F1** | Devices or arrangements for the control of light intensity (modulators) |
#### Queryable Metrics
1. **Patent Filing Velocity** — Count of new patent applications per CPC class per quarter
2. **Patent Grant Rate** — Ratio of grants to applications (maturity indicator)
3. **Citation Velocity** — How quickly new patents cite each other (hot field indicator)
4. **Technology Cycle Time (TCT)** — Median age of citations (shorter = faster-moving field)
5. **Assignee Concentration** — Herfindahl index of patent holders (few holders = early stage; many = maturation)
#### Example Query (PatentsView Search API)
```
GET https://search.patentsview.org/api/v1/patent/
?q={"_and":[{"_begins":{"cpc_at_issue.cpc_subclass_id":"H04B10"}},{"_gte":{"patent_date":"2024-01-01"}},{"_text_any":{"patent_abstract":"transceiver 400G 800G QSFP OSFP"}}]}
&f=["patent_id","patent_date","patent_title","assignees.assignee_organization"]
&o={"size":100}
```
Response includes `total_hits` for counting.
#### Academic Validation
- **BIMATEM method** (Manrique-Castillo et al., Scientometrics 2018): Patent records of mature technologies display **logistic growth** behavior. Fitting logistic curves to patent counts per technology enables TRL assignment.
- **Gao et al. (2013)**: Using multiple patent-based indicators with a nearest-neighbour classifier for technology life cycle stage classification.
- **Technology Cycle Time**: Kayal's TCT indicator — median citation age predicts technology maturity phase.
#### Correlation with Hype Cycle Position
- **High filing velocity + low grant rate** = Technology Trigger / early Peak
- **Peak filing count reached** = Peak of Inflated Expectations
- **Declining filings + rising citations** = Trough / early Slope
- **Stable filings + high citation density** = Plateau of Productivity
---
### 2. ACADEMIC PUBLICATION METRICS (Knowledge Creation Signal)
**What it measures:** Scientific research intensity, knowledge maturation
**Hype cycle relevance:** Publication counts follow a logistic S-curve. The inflection point of the S-curve corresponds roughly to the transition from Peak to Trough.
#### Data Source: Semantic Scholar API (VALIDATED — working)
| Attribute | Detail |
|-----------|--------|
| **API URL** | `https://api.semanticscholar.org/graph/v1/paper/search/bulk` |
| **Auth** | None required (public). API key available for higher rate limits |
| **Rate Limit** | 1000 req/sec (shared unauthenticated), 1 req/sec (with free API key) |
| **Update Frequency** | Continuous (near real-time) |
| **Cost** | Free |
| **Coverage** | ~200 million papers across all disciplines |
| **Python Library** | `semanticscholar` (PyPI) or direct `requests` |
| **Implementation Complexity** | 1/5 |
#### Validated Paper Counts (tested 2026-03-28)
| Technology | Total Papers | Maturity Signal |
|------------|-------------|-----------------|
| silicon photonics transceiver | 905 | Mature (deep research base) |
| 100G transceiver | 144 | Late maturity |
| 400G transceiver | 100 | Growth phase |
| 200G transceiver | 43 | Moderate |
| coherent pluggable optics | 40 | Growth phase |
| 800G transceiver | 39 | Early growth |
| QSFP-DD optical | 26 | Emerging |
| OSFP transceiver | 11 | Very early |
| 1.6T transceiver optical | 10 | Pre-commercial |
#### Year-by-Year Trend (400G transceiver, validated)
| Year | Papers | Signal |
|------|--------|--------|
| 2018 | 10 | Early research |
| 2019 | 7 | Stable |
| 2020 | 7 | Stable |
| 2021 | 9 | Slight increase |
| 2022 | 15 | Growth spike |
| 2023 | 6 | Decline |
| 2024 | 8 | Recovery |
| 2025 | 12 | Resurgence |
This pattern (spike in 2022, decline 2023, recovery 2024-25) maps well to the 400G transition from Peak to Slope of Enlightenment.
#### Key Metrics to Extract
1. **Annual paper count** per technology keyword
2. **Rate of change** (first derivative — acceleration/deceleration)
3. **Citation count distribution** — highly cited papers = foundational work = maturation
4. **Author diversity** — many unique authors = broad interest = growth phase
5. **Venue distribution** — OFC/ECOC papers vs. general journals
#### Supplementary Source: IEEE Xplore
- URL: `https://ieeexploreapi.ieee.org/api/v1/search/articles`
- API key required (free for research)
- Specifically covers OFC, ECOC, CLEO proceedings
- Higher signal quality for optical networking specifically
---
### 3. GOOGLE TRENDS (Public Interest / Hype Proxy)
**What it measures:** Search interest as a proxy for market attention and hype
**Hype cycle relevance:** Google Trends data directly models the "hype" component. Academic validation exists (Jun 2012, van Lente 2013).
#### Data Source: Google Trends via pytrends (VALIDATED — working)
| Attribute | Detail |
|-----------|--------|
| **API** | Unofficial (Google Trends web scraping via pytrends) |
| **Auth** | None |
| **Rate Limit** | ~10 requests/minute (unofficial, subject to blocking) |
| **Update Frequency** | Real-time (weekly/monthly granularity) |
| **Cost** | Free |
| **Python Library** | `pytrends` (PyPI, v4.9.2) |
| **Implementation Complexity** | 1/5 |
#### Validated Data (tested 2026-03-28)
**Batch 1 — Form Factors & Speeds (relative to each other):**
| Technology | Current Interest | Peak Value | Peak Date | Trajectory |
|------------|-----------------|------------|-----------|------------|
| silicon photonics | 100 (reference) | 100 | 2026-03 | Rising strongly |
| OSFP | 34 | 45 | 2024-05 | Peaked, declining |
| 800G transceiver | 10 | 10 | 2026-02 | Rising |
| QSFP-DD | 8 | 10 | 2025-11 | Declining from peak |
| 400G transceiver | 2 | 3 | 2025-12 | Low/stable (mature) |
**Batch 2 — Emerging Technologies:**
| Technology | Current Interest | Peak Value | Peak Date | Trajectory |
|------------|-----------------|------------|-----------|------------|
| co-packaged optics | 100 (reference) | 100 | 2026-03 | Rising strongly |
| coherent optics | 45 | 45 | 2026-03 | Rising |
| 1.6T ethernet | 5 | 14 | 2025-08 | Peaked, declining |
| 100G transceiver | 5 | 8 | 2026-02 | Low/stable |
#### Key Observations
- **OSFP peaked May 2024** — consistent with 802.3df approval (Feb 2024) driving peak hype
- **QSFP-DD declining from Nov 2025 peak** — market settling
- **co-packaged optics and silicon photonics surging** — current hype leaders
- **400G transceiver at floor** — fully mature, no hype left (Plateau of Productivity)
- **1.6T peaked Aug 2025** then declined — possible "Peak of Inflated Expectations" → Trough
#### Implementation Notes
- Normalize by comparing technologies against each other (relative index)
- Use monthly granularity for trend detection
- Calculate: peak detection, slope analysis, time-since-peak
- Combine with absolute volume signals (paper counts) since Google Trends is relative only
- **Limitation:** B2B niche terms have low search volumes — use broader terms ("silicon photonics" not "silicon photonics transceiver module QSFP-DD800")
#### Academic Validation
- **Jun (2012)**: "An empirical study of users' hype cycle based on search traffic" — validated Google Trends hype cycle matching for hybrid cars (*Scientometrics* 91(1), pp. 81-99)
- **van Lente, Spitters & Peine (2013)**: "Comparing technological hype cycles: Towards a theory" (*Technological Forecasting and Social Change* 80(8))
- **Choi & Varian (2010)**: "Predicting the Present with Google Trends" (foundational paper on search data as predictor)
- **Caveat**: Medeiros et al. (arXiv 2021) document preprocessing requirements for reliable forecasting from Trends data
---
### 4. NEWS/MEDIA VOLUME (Hype Amplification Signal)
**What it measures:** Trade press and media coverage volume and sentiment
**Hype cycle relevance:** News volume directly measures the "hype" dimension. Sentiment analysis distinguishes Peak (positive) from Trough (negative/absent).
#### Data Source A: GDELT DOC 2.0 API (VALIDATED — working, limited for niche B2B)
| Attribute | Detail |
|-----------|--------|
| **API URL** | `https://api.gdeltproject.org/api/v2/doc/doc` |
| **Auth** | None |
| **Rate Limit** | Reasonable (no published limit) |
| **Update Frequency** | Every 15 minutes |
| **Cost** | Free |
| **Coverage** | 100+ languages, 65 translated, millions of sources |
| **History** | Last 3 months reliably (older data not guaranteed) |
| **Python Library** | `gdeltdoc` (PyPI) or `gdeltPyR` (PyPI) |
| **Implementation Complexity** | 2/5 |
**Limitation for TIP:** GDELT covers general news very well but B2B optical transceiver coverage is sparse. Testing showed only 1 article for "400G optical" in 3 months. Better for broader terms like "silicon photonics" or "data center optics."
#### Data Source B: NewsAPI.org
| Attribute | Detail |
|-----------|--------|
| **API URL** | `https://newsapi.org/v2/everything` |
| **Free Tier** | 100 requests/day, 1-month history, 24h delay, dev-only |
| **Paid** | From $40/month |
| **Python** | `requests` (simple REST) |
| **Implementation Complexity** | 1/5 |
#### Data Source C: Trade Press RSS/Scraping (RECOMMENDED for optical)
Monitor these sources directly (Crawlee/Playwright — already in TIP architecture):
| Source | URL | Relevance |
|--------|-----|-----------|
| LightReading | lightreading.com | Primary (optical networking) |
| Fierce Telecom | fiercetelecom.com | Primary |
| Datacenter Dynamics | datacenterdynamics.com | Primary |
| SDxCentral | sdxcentral.com | Primary |
| Lightwave Online | lightwaveonline.com | Primary (optical specific) |
| Gazettabyte | gazettabyte.com | High (standards/specs) |
| Converge Digest | convergedigest.com | Moderate |
| Semiconductor Today | semiconductor-today.com | Moderate (component level) |
#### Metrics to Extract
1. **Article count per technology per month** (volume)
2. **Sentiment score** using VADER (lightweight) or FinBERT (more accurate)
3. **Source diversity** — how many different outlets cover the topic
4. **Headline vs. mention** — is the technology the headline or just mentioned?
#### Sentiment Analysis Tools
| Tool | Type | Cost | Accuracy | Speed |
|------|------|------|----------|-------|
| VADER | Rule-based | Free | Good for general | Very fast |
| FinBERT | Transformer | Free | Best for financial/tech | Moderate |
| Ollama (qwen2.5:14b) | LLM | Free (local) | Very good | Slow |
| TextBlob | Rule-based | Free | Basic | Very fast |
**Recommendation:** Use VADER for initial scoring, Ollama for nuanced classification on flagged articles.
---
### 5. VENDOR COUNT / SKU PROLIFERATION (Market Adoption Signal)
**What it measures:** Market entry velocity, competitive maturation, commoditization
**Hype cycle relevance:** This is THE strongest signal for distinguishing Slope of Enlightenment from Plateau of Productivity. Directly measurable from TIP's own scraper data.
#### Data Source: TIP's Own Scraper Database (ZERO ADDITIONAL COST)
| Attribute | Detail |
|-----------|--------|
| **Source** | TIP price_observations + vendor tables |
| **Auth** | Internal |
| **Update Frequency** | Real-time (5-15 min scraper intervals) |
| **Cost** | Already being collected |
| **Implementation Complexity** | 1/5 (data already exists) |
#### Metrics
1. **Vendor Count per Technology** — How many vendors sell a given form factor/speed
- 1-3 vendors = Technology Trigger / early Peak
- 4-10 vendors = Peak / early Slope
- 10-30 vendors = Slope of Enlightenment
- 30+ vendors = Plateau of Productivity
2. **SKU Growth Rate** — New product listings per month
- Accelerating = Growth phase
- Decelerating = Maturation
- Flat = Plateau
3. **Price Coefficient of Variation (CV)** — Standard deviation / mean of prices across vendors
- High CV (>0.5) = Early market, pricing uncertainty
- Medium CV (0.2-0.5) = Competitive market
- Low CV (<0.2) = Commodity market (Plateau)
4. **Price Decline Rate** — $/Gbps over time
- Steep decline = Growth → Slope transition
- Gradual decline = Slope
- Flat = Plateau
5. **Geographic Vendor Distribution** — Chinese vendors entering = commoditization signal
#### Why This Signal is Critical
This is **the only signal that directly measures actual market behavior** rather than proxies (search interest, papers, patents). Combined with price data, it provides ground truth for hype cycle calibration.
---
### 6. STANDARDS PROGRESS (Technology Readiness Signal)
**What it measures:** Standardization maturity as proxy for technology readiness
**Hype cycle relevance:** Standards progress is a LEADING indicator. "Study group formed" precedes market by 3-5 years.
#### Standards Phase Mapping to Hype Cycle
| Standards Phase | Typical Duration | Hype Cycle Phase |
|----------------|-----------------|------------------|
| Call for Interest / Study Group | 6-12 months | Pre-Trigger |
| Task Force Formed | 0 | Technology Trigger |
| First Draft | 12-18 months | Peak of Inflated Expectations |
| Working Group Ballot | 6-12 months | Peak → Trough transition |
| Sponsor Ballot | 3-6 months | Trough → Slope |
| Standard Published | 0 | Slope of Enlightenment |
| First Amendment | 12-24 months | Plateau of Productivity |
#### Current State (validated 2026-03-28)
| Technology | Standard | Status | Hype Phase Inference |
|------------|----------|--------|---------------------|
| 400G Ethernet | IEEE 802.3bs | Published Dec 2017 | Plateau |
| 800G Ethernet (100G/lane) | IEEE 802.3df | Published Feb 2024 | Slope of Enlightenment |
| 800G Ethernet (200G/lane) | IEEE 802.3dj | In progress, target Jul 2026 | Peak → Trough |
| 1.6T Ethernet | IEEE 802.3dj | In progress, target Jul 2026 | Peak of Inflated Expectations |
| 3.2T Ethernet | OIF/MSA discussions | Study group phase | Pre-Trigger |
| 400ZR Coherent | OIF IA published Apr 2020 | Published | Late Slope |
#### Trackable Standards Bodies
| Body | What to Track | URL |
|------|--------------|-----|
| **IEEE 802.3** | Task force status, ballot dates | ieee802.org/3/ |
| **OIF** | Implementation Agreements (IAs), CMIS versions | oiforum.com/technical-work/implementation-agreements-ias/ |
| **QSFP-DD MSA** | Spec revisions (now at QSFP-DD1600) | qsfp-dd.com |
| **OSFP MSA** | Spec revisions (now at Rev 5.21) | osfpmsa.org |
| **100G Lambda MSA** | FR/LR specs | 100glambda.com |
#### Implementation
- Maintain a manually-curated `standards_progress` table
- Update quarterly (standards move slowly)
- Each standard gets a numeric score: 0 (no activity) → 10 (published + amendments)
- **Implementation Complexity:** 2/5 (manual curation, low frequency)
---
### 7. JOB MARKET SIGNALS (Demand/Deployment Signal)
**What it measures:** Actual hiring demand for technology-specific skills
**Hype cycle relevance:** Job posting surges lag the Peak by 12-18 months and correlate with Slope of Enlightenment.
#### Data Sources
| Source | Cost | API | Quality |
|--------|------|-----|---------|
| **TheirStack** | Free tier available | REST API | Best (deduplication, 324k ATS platforms) |
| **FlyByAPIs** | Free (200 req/month) | RapidAPI | Good (Google Jobs index) |
| **Sumble** | Free 500 credits/month | REST API | Good (LinkedIn + hiring signals) |
| **LinkedIn Talent** | Enterprise ($$$) | Partner only | Best but inaccessible |
| **Indeed Job Sync** | Free (partner) | REST API | Posting-focused, not search |
**Recommended:** TheirStack or FlyByAPIs for free tier.
#### Metrics
1. **Job posting count** per technology keyword per month
2. **Job posting velocity** — rate of change
3. **Salary range** — higher salaries = talent scarcity = early adoption
4. **Geographic distribution** — US/EU = early; APAC = maturation
#### Implementation Complexity: 3/5
---
### 8. SOCIAL MEDIA / COMMUNITY SIGNALS (Practitioner Interest)
**What it measures:** Operator and engineer discussion intensity
**Hype cycle relevance:** Community buzz leads deployment by 6-12 months.
#### Data Sources
| Source | API | Cost | Python Library |
|--------|-----|------|----------------|
| **Reddit** (r/networking, r/homelab, r/datacenter) | Reddit API via PRAW | Free | `praw` |
| **NANOG mailing list** | No API (scrape archives) | Free | `requests` + `beautifulsoup4` |
| **LinkedIn** | No public search API | N/A | N/A |
#### Reddit via PRAW
- Free Reddit API access (60 req/min)
- Search subreddits by keyword, filter by time
- Count posts + comments mentioning technology terms
- **PRAWtools** provides keyword alerts and subreddit statistics
- Limitation: 1,000 post search window
#### NANOG Mailing List
- Archives available at `nanog.org/nanog-mailing-list/list-archives/` and `marc.info`
- Monthly text file downloads available
- ETH Zurich thesis (Gehri 2021) demonstrated NLP topic modeling and sentiment analysis on 89,000+ NANOG emails
- No API — requires scraping or bulk download
- Highly relevant for optical networking technology adoption signals
#### Metrics
1. **Post/email count per technology per month**
2. **Engagement ratio** (comments/votes per post)
3. **Sentiment** (positive deployment reports vs. complaints)
4. **Question vs. statement ratio** (questions = early adoption; statements = maturity)
#### Implementation Complexity: 3/5
---
### 9. EARNINGS CALL / FINANCIAL SIGNALS (Enterprise Adoption Signal)
**What it measures:** How often public companies mention technologies in financial disclosures
**Hype cycle relevance:** Earnings call mentions are a LAGGING indicator that confirms Slope of Enlightenment → Plateau transition.
#### Data Source A: SEC EDGAR EFTS (VALIDATED — working, 899 filings found)
| Attribute | Detail |
|-----------|--------|
| **API URL** | `https://efts.sec.gov/LATEST/search-index` |
| **Auth** | None (free public API) |
| **Rate Limit** | ~10 requests/second (fair use) |
| **Update Frequency** | Real-time (new filings indexed immediately) |
| **Cost** | Free |
| **Coverage** | All SEC filings since ~1993 |
| **Python Library** | `requests` (direct) or `sec-api` (paid wrapper) |
| **Implementation Complexity** | 2/5 |
**Validated result:** Query for `"optical transceiver" OR "400G" OR "800G optics"` returned **899 filings** across 10-K, 10-Q, and 8-K forms.
#### Data Source B: Financial Modeling Prep (FMP)
| Attribute | Detail |
|-----------|--------|
| **API URL** | `https://financialmodelingprep.com/api/v3/earning_call_transcript/{SYMBOL}` |
| **Auth** | API key (free tier available) |
| **Cost** | Free tier, paid plans from $29/month |
| **Coverage** | Full earnings call transcripts for public companies |
| **Python Library** | `requests` |
| **Implementation Complexity** | 2/5 |
#### Target Companies for Optical Transceiver Mentions
| Ticker | Company | Relevance |
|--------|---------|-----------|
| COHR | Coherent Corp (formerly II-VI/Finisar) | Transceiver manufacturer |
| LITE | Lumentum | Laser/transceiver manufacturer |
| CSCO | Cisco | Network equipment + transceivers |
| JNPR | Juniper Networks | Network equipment |
| ANET | Arista Networks | Datacenter switching |
| AVGO | Broadcom | Transceiver silicon |
| INTC | Intel (Altera) | Silicon photonics |
| CIEN | Ciena | Coherent optics |
| INFN | Infinera | Coherent optics |
| AAOI | Applied Optoelectronics | Transceiver manufacturer |
#### Metrics
1. **Mention frequency** — count of technology term mentions per earnings call
2. **Mention sentiment** — positive/negative context around mentions
3. **First mention** — when a company first mentions a technology (leading indicator)
4. **Revenue attribution** — when companies break out revenue by technology generation
---
### 10. COMPOSITE SIGNAL ALGORITHM
#### Academic Foundation
**Ren (2015)**: "An Approach for Predicting Hype Cycle Based on Machine Learning" (CEUR-WS Vol-1437, IPAMIN 2015)
- Used SKNN (improved K-Nearest Neighbor) classifier
- Features extracted from paper data and patent data
- Achieved **67.24% precision, 68.46% recall** classifying technologies into 5 hype cycle phases
- Noted accuracy drops in phases 4-5 due to small training samples
**BIMATEM (Manrique-Castillo et al., Scientometrics 2018)**:
- Combines **three data streams**: scientific papers (logistic growth), patents (logistic growth), news (hype-type curve)
- Fits logistic regression to paper/patent counts
- Fits hype-type regression to news counts
- Assigns TRL (Technology Readiness Level) based on curve position
- Applied successfully to additive manufacturing technologies
**Composite Early Warning Index (CEWI) approach** (financial crisis literature):
- Uses PCA to synthesize diverse variables into a single latent factor
- Applicable to combining patent, publication, trends, and market signals
#### Recommended Algorithm: Weighted Multi-Signal Scoring
```
HypeScore(tech, t) = w1 * Patent_Signal(tech, t)
+ w2 * Publication_Signal(tech, t)
+ w3 * Trends_Signal(tech, t)
+ w4 * News_Signal(tech, t)
+ w5 * Vendor_Signal(tech, t)
+ w6 * Standards_Signal(tech, t)
+ w7 * Earnings_Signal(tech, t)
+ w8 * Jobs_Signal(tech, t)
```
#### Signal Time Horizons and Weights
| Signal | Lead/Lag | Suggested Weight | Update Freq |
|--------|----------|-----------------|-------------|
| Patents | Leads by 3-5 years | 0.10 | Quarterly |
| Publications | Leads by 1-3 years | 0.10 | Monthly |
| Google Trends | Real-time | 0.20 | Monthly |
| News Volume | Real-time | 0.10 | Weekly |
| **Vendor Count/Price** | **Real-time** | **0.25** | **Daily** |
| Standards Progress | Leads by 2-4 years | 0.10 | Quarterly |
| Earnings Calls | Lags by 6-12 months | 0.10 | Quarterly |
| Job Postings | Lags by 12-18 months | 0.05 | Monthly |
**Vendor Count/Price gets the highest weight** because it is the only direct market measurement.
#### Phase Classification Approach
1. **Normalize each signal** to 0-100 scale per technology
2. **Calculate rate of change** (first derivative) for each signal
3. **Calculate acceleration** (second derivative) for trend detection
4. **Apply phase classification rules:**
| Phase | Signal Pattern |
|-------|---------------|
| **Technology Trigger** | Patents rising, Publications starting, Trends near zero, Vendors 0-3, Standard in study group |
| **Peak of Inflated Expectations** | Trends peaking, News volume peaking, Publications rising fast, Vendors 3-8, Sentiment highly positive |
| **Trough of Disillusionment** | Trends declining, News declining, Sentiment negative, Vendors may decrease, Publications slowing |
| **Slope of Enlightenment** | Vendors growing steadily, Price CV declining, Earnings mentions increasing, Jobs increasing, Standards published |
| **Plateau of Productivity** | All signals stable, Price CV < 0.2, Vendor count > 30, Publications steady, Standards have amendments |
5. **Optional ML layer:** Train a Random Forest or Gradient Boosting classifier on known technology trajectories (100G, 40G, 10G historical data as training set)
#### Norton-Bass Integration
The composite signal feeds into the Norton-Bass multigenerational diffusion model:
- **p (innovation coefficient)** ← derived from patent/publication velocity
- **q (imitation coefficient)** ← derived from vendor count growth rate + Google Trends
- **M (market potential)** ← derived from addressable port count in deployed switches
- **tau (generation introduction time)** ← derived from IEEE standard publication date
- **Python:** `scipy.optimize.curve_fit` with Bass model function, or `bassmodeldiffusion` package (PyPI)
---
## Prioritized Implementation Plan
### Phase 1: Quick Wins (Week 1-2) — HIGH VALUE, LOW EFFORT
| # | Signal | API | Cost | Complexity | Why First |
|---|--------|-----|------|------------|-----------|
| 1 | **Google Trends** | pytrends | Free | 1/5 | Already validated, immediate hype measurement |
| 2 | **Vendor Count/Price** | Internal DB | Free | 1/5 | Data already being collected by TIP scrapers |
| 3 | **Semantic Scholar** | REST API | Free | 1/5 | Already validated, publication trend curves |
**Deliverable:** Basic hype cycle positioning for all tracked technologies using 3 signals.
### Phase 2: Depth Signals (Week 3-4) — HIGH VALUE, MODERATE EFFORT
| # | Signal | API | Cost | Complexity |
|---|--------|-----|------|------------|
| 4 | **SEC EDGAR EFTS** | REST API | Free | 2/5 |
| 5 | **Standards Progress** | Manual curation | Free | 2/5 |
| 6 | **Trade Press Scraping** | Crawlee (existing) | Free | 2/5 |
**Deliverable:** 6-signal composite with financial and standards validation.
### Phase 3: Extended Signals (Week 5-8) — MODERATE VALUE, HIGHER EFFORT
| # | Signal | API | Cost | Complexity |
|---|--------|-----|------|------------|
| 7 | **USPTO Patents** | PatentsView | Free (need API key) | 2/5 |
| 8 | **Reddit/PRAW** | Reddit API | Free | 3/5 |
| 9 | **Job Postings** | TheirStack/FlyByAPIs | Free tier | 3/5 |
| 10 | **Earnings Transcripts** | FMP | Free tier | 2/5 |
**Deliverable:** Full 10-signal composite with ML phase classifier.
### Phase 4: ML Calibration (Week 9-12)
1. Collect historical data for training technologies (10G, 40G, 100G — known trajectories)
2. Train Random Forest classifier on multi-signal features
3. Validate against known Gartner positioning (where available)
4. Implement Norton-Bass curve fitting with signal-derived parameters
5. Build confidence scoring and uncertainty quantification
---
## Key Python Dependencies
```
# Phase 1
pytrends==4.9.2 # Google Trends
semanticscholar # Paper counts
requests # General HTTP
scipy # Curve fitting (Norton-Bass)
numpy # Numerical
pandas # Data manipulation
# Phase 2
beautifulsoup4 # HTML parsing (trade press)
vaderSentiment # Sentiment analysis
# Phase 3
praw # Reddit API
bassmodeldiffusion # Bass model fitting
# Phase 4
scikit-learn # Random Forest, PCA
xgboost # Gradient boosting
```
---
## Signal Correlation Summary
| Signal | Free? | Real-time? | Validated? | Hype Correlation | Implementation |
|--------|-------|-----------|------------|-----------------|---------------|
| Google Trends | Yes | Yes | YES | HIGH (academic proof) | 1/5 |
| Vendor Count/Price | Yes | Yes | YES (own data) | HIGHEST (direct) | 1/5 |
| Semantic Scholar | Yes | Yes | YES | MODERATE-HIGH | 1/5 |
| SEC EDGAR EFTS | Yes | Yes | YES | MODERATE | 2/5 |
| News/Trade Press | Yes | Weekly | Partial | HIGH | 2/5 |
| Standards Progress | Yes | Quarterly | YES | HIGH (leading) | 2/5 |
| Patents (USPTO) | Yes | Quarterly | Not yet (API key needed) | MODERATE-HIGH | 2/5 |
| Reddit/PRAW | Yes | Daily | Not yet | LOW-MODERATE | 3/5 |
| Job Postings | Free tier | Daily | Not yet | MODERATE | 3/5 |
| Earnings Calls | Free tier | Quarterly | Not yet | MODERATE | 2/5 |
---
## References
### Academic Papers
- Manrique-Castillo et al. (2018). "A bibliometric method for assessing technological maturity: the case of additive manufacturing." *Scientometrics* 117(3).
- Ren, Z. (2015). "An Approach for Predicting Hype Cycle Based on Machine Learning." CEUR-WS Vol-1437.
- Jun, S.P. (2012). "An empirical study of users' hype cycle based on search traffic." *Scientometrics* 91(1), 81-99.
- van Lente, H., Spitters, C., & Peine, A. (2013). "Comparing technological hype cycles." *Technological Forecasting and Social Change* 80(8).
- Gao, L. et al. (2013). "Technology life cycle analysis method based on patent documents." *Technological Forecasting and Social Change*.
- Huang et al. (2022). "Technology life cycle analysis: From the dynamic perspective of patent citation networks." *Technological Forecasting and Social Change*.
- Choi, H. & Varian, H. (2010). "Predicting the Present with Google Trends." SSRN.
- Dedehayir, O. & Steinert, M. (2016). "The hype cycle model: A review and future directions." *Technological Forecasting and Social Change* 108(C).
- Norton, J.A. & Bass, F.M. (1987). "A diffusion theory model of adoption and substitution for successive generations of high-technology products." *Management Science* 33(9).
- Gehri, L. (2021). "NANOG Mailing List Analysis." ETH Zurich Semester Thesis.
### API Documentation
- PatentsView Search API: https://search.patentsview.org/docs/
- Semantic Scholar API: https://api.semanticscholar.org/api-docs
- GDELT DOC API: https://blog.gdeltproject.org/gdelt-doc-2-0-api-debuts/
- SEC EDGAR EFTS: https://efts.sec.gov/LATEST/search-index
- Financial Modeling Prep: https://site.financialmodelingprep.com/developer/docs
- Google Trends (pytrends): https://pypi.org/project/pytrends/
- Reddit (PRAW): https://praw.readthedocs.io/
- IEEE 802.3dj Task Force: https://www.ieee802.org/3/dj/index.html
- OIF Implementation Agreements: https://www.oiforum.com/technical-work/implementation-agreements-ias/
### Python Libraries
- `pytrends`: https://pypi.org/project/pytrends/
- `semanticscholar`: https://pypi.org/project/semanticscholar/
- `gdeltdoc`: https://pypi.org/project/gdeltdoc/
- `praw`: https://pypi.org/project/praw/
- `bassmodeldiffusion`: https://github.com/marmiskarian/bassmodeldiffusion
- `vaderSentiment`: https://pypi.org/project/vaderSentiment/

View File

@ -0,0 +1,672 @@
# Optical Transceiver Evolution: Complete History & Database Reference (2001-2026)
> Deep research compiled from OFC proceedings, LightCounting, Cignal AI, IEEE, OIF, and industry publications.
> Last updated: 2026-03-27
---
## Table of Contents
1. [Form Factor Evolution Timeline](#1-form-factor-evolution-timeline)
2. [Speed Tier Evolution](#2-speed-tier-evolution)
3. [Key Standards & Adoption Timelines](#3-key-standards--adoption-timelines)
4. [CWDM vs DWDM Evolution](#4-cwdm-vs-dwdm-evolution)
5. [Major Transceiver Manufacturers](#5-major-transceiver-manufacturers)
6. [Next-Generation Technologies (2025-2030)](#6-next-generation-technologies-2025-2030)
7. [Market Data Points](#7-market-data-points)
8. [Database Schema Recommendations](#8-database-schema-recommendations)
9. [Hype Cycle Analysis](#9-hype-cycle-analysis)
---
## 1. Form Factor Evolution Timeline
### Complete Form Factor Database
| Form Factor | Year Introduced | Peak Adoption | Legacy/Decline | Max Speed | Connector | Lanes | Status |
|---|---|---|---|---|---|---|---|
| **GBIC** | 1995 | 2000-2004 | 2006+ | 2.5 Gbps | SC Duplex | 1 | Obsolete |
| **SFP** | 2001 | 2004-present | Still active (1G) | 4.25 Gbps | LC Duplex | 1 | Active (legacy speeds) |
| **XENPAK** | 2001 | 2002-2006 | 2007+ | 10 Gbps | SC Duplex | 1 | Obsolete |
| **X2** | 2003 | 2004-2008 | 2009+ | 10 Gbps | SC Duplex | 1 | Obsolete |
| **XFP** | 2002 (MSA), 2003 (adopted) | 2005-2012 | 2013+ | 10 Gbps (DWDM capable) | LC Duplex | 1 | Legacy |
| **SFP+** | 2006 | 2008-present | Still active | 16 Gbps | LC Duplex | 1 | Active |
| **QSFP** | 2006 | 2008-2012 | 2013+ | 4x1G = 4 Gbps | MPO-12 | 4 | Legacy |
| **CFP** | 2009 | 2010-2016 | 2017+ | 100 Gbps | LC Duplex/MPO | 10x10G | Legacy |
| **QSFP+** | 2012 | 2013-2020 | Declining | 40 Gbps | MPO-12 / LC | 4x10G | Active (declining) |
| **CFP2** | 2012 | 2014-2020 | 2021+ | 200 Gbps | LC Duplex | varies | Legacy (except coherent) |
| **CFP4** | 2014 | 2015-2019 | 2020+ | 100 Gbps | LC Duplex | 4x25G | Legacy |
| **QSFP28** | 2014 | 2016-2023 | Declining | 100 Gbps | LC / MPO-12 | 4x25G | Active (declining) |
| **SFP28** | 2014 | 2016-present | Still active | 25 Gbps | LC Duplex | 1 | Active |
| **OSFP** | 2016 (announced) | 2020-present | - | 800 Gbps (8x100G) | MPO-16 / LC | 8 | Active (growing) |
| **CSFP** | 2018 | 2019-present | - | 2x1 Gbps | LC (BiDi) | 2 (BiDi) | Niche |
| **QSFP56** | 2019 | 2020-2024 | Declining | 200 Gbps | MPO-12 / LC | 4x50G | Active (declining) |
| **QSFP-DD** | 2019 | 2021-present | - | 800 Gbps (8x100G) | MPO-16 / LC | 8 | Active (growing) |
| **SFP56** | 2020 (spec), 2024 (products) | 2024-present | - | 50 Gbps | LC Duplex | 1 | Active (emerging) |
| **QSFP112** | 2021 | 2022-present | - | 400 Gbps | MPO-12 / LC | 4x100G | Active |
| **SFP-DD** | 2017 (spec) | 2020-present | - | 2x25G = 50 Gbps | LC Duplex | 2 | Niche |
| **OSFP-XD** | 2022 | 2025-present | - | 1.6T (16x100G), 3.2T future | MPO-16 | 16 | Emerging |
| **QSFP-DD1600** | 2024 (spec in progress) | 2026+ (projected) | - | 1.6T (8x200G) | MPO-16 | 8 | Emerging |
| **OSFP1600** | 2022 (spec) | 2025-2026 | - | 1.6T (8x200G) | MPO-16 | 8 | Emerging |
### Form Factor Hype Cycle Phases
```
Phase 1: INTRODUCTION - Standard published, first samples
Phase 2: EARLY ADOPTION - Hyperscale/cloud first movers
Phase 3: MAINSTREAM - Broad enterprise deployment, pricing declines
Phase 4: MATURITY - Commoditized, price floor reached
Phase 5: DECLINE - Next generation overtakes, volume drops
Phase 6: LEGACY - Minimal new deployments, maintenance only
Phase 7: OBSOLETE - No longer manufactured
```
| Form Factor | Current Phase (2026) |
|---|---|
| GBIC | 7-OBSOLETE |
| XENPAK | 7-OBSOLETE |
| X2 | 7-OBSOLETE |
| XFP | 6-LEGACY |
| SFP (1G) | 4-MATURITY |
| SFP+ (10G) | 4-MATURITY |
| QSFP+ (40G) | 5-DECLINE |
| CFP/CFP2/CFP4 | 6-LEGACY (except CFP2-DCO) |
| SFP28 (25G) | 3-MAINSTREAM |
| QSFP28 (100G) | 4-MATURITY / 5-DECLINE |
| QSFP56 (200G) | 5-DECLINE |
| QSFP-DD (400G/800G) | 3-MAINSTREAM |
| OSFP (400G/800G) | 3-MAINSTREAM |
| QSFP112 (400G) | 2-EARLY ADOPTION |
| OSFP-XD (1.6T) | 1-INTRODUCTION |
| QSFP-DD1600 (1.6T) | 1-INTRODUCTION |
---
## 2. Speed Tier Evolution
### Speed Tier Database
| Speed | Year Standardized | Year Mainstream | Dominant Form Factor | Modulation | Lanes | Key Standard | Current Status |
|---|---|---|---|---|---|---|---|
| **1G** | 1998 (802.3z) | 2002 | SFP | NRZ | 1 | IEEE 802.3z | Mature/commodity |
| **10G** | 2002 (802.3ae) | 2007 | SFP+ | NRZ | 1 | IEEE 802.3ae | Mature/commodity |
| **25G** | 2016 (802.3by) | 2018 | SFP28 | NRZ | 1 | IEEE 802.3by | Mainstream |
| **40G** | 2010 (802.3ba) | 2013 | QSFP+ | NRZ | 4x10G | IEEE 802.3ba | Declining |
| **50G** | 2016 (802.3cd) | 2020 | SFP56 / QSFP28 | PAM4 (single lane) | 1 | IEEE 802.3cd | Niche |
| **100G** | 2010 (802.3ba) / 2014 (QSFP28) | 2017 | QSFP28 | NRZ (4x25G) | 4 | IEEE 802.3ba | Mainstream/declining |
| **200G** | 2017 (802.3bs) | 2020 | QSFP56 / QSFP-DD | PAM4 | 4x50G | IEEE 802.3bs | Active |
| **400G** | 2017 (802.3bs) | 2022 | QSFP-DD / OSFP | PAM4 | 8x50G or 4x100G | IEEE 802.3bs | Mainstream |
| **800G** | 2024 (802.3df) | 2024-2025 | OSFP / QSFP-DD | PAM4 | 8x100G | IEEE 802.3df | Rapid growth |
| **1.6T** | 2026 (802.3dj target) | 2026-2027 (projected) | OSFP-XD / OSFP1600 | PAM4 | 8x200G or 16x100G | IEEE 802.3dj | Emerging |
### Speed Tier Adoption S-Curves (Port Shipment Peak Years)
```
1G: Peak ~2010-2014, still shipping in volume for enterprise access
10G: Peak ~2016-2020, declining but high volume
25G: Peak ~2020-2024, server-side standard
40G: Peak ~2015-2019, largely replaced by 100G
100G: Peak ~2020-2024, transitioning to 400G
400G: Peak ~2024-2027 (projected), current mainstream for spine/core
800G: Peak ~2026-2029 (projected), AI backend standard
1.6T: Peak ~2028-2031 (projected), next-gen AI/HPC
```
### Modulation Technology Timeline
| Technology | Speed Range | Years Active | Key Characteristic |
|---|---|---|---|
| NRZ (Non-Return-to-Zero) | 1G-25G per lane | 1995-present | 1 bit per symbol, simple |
| PAM4 (4-level Pulse Amplitude) | 50G-200G per lane | 2017-present | 2 bits per symbol, requires DSP/FEC |
| Coherent (DP-QPSK/DP-16QAM) | 100G-800G per wavelength | 2011-present | Phase + amplitude, long-haul |
### Per-Lane Rate Evolution
| Year | Per-Lane Rate | Technology | Key Enabler |
|---|---|---|---|
| 2001-2005 | 1G | NRZ | DFB/VCSEL |
| 2006-2013 | 10G | NRZ | DFB/VCSEL, CDR |
| 2014-2018 | 25G | NRZ | EML, CDR |
| 2019-2022 | 50G | PAM4 | DSP (7nm/5nm) |
| 2022-2025 | 100G | PAM4 | DSP (5nm/3nm), SiPh |
| 2025-2028 | 200G | PAM4 | DSP (3nm), advanced FEC |
---
## 3. Key Standards & Adoption Timelines
### IEEE 802.3 Optical Ethernet Standards
| Standard | Year Ratified | Speed | Key PHY Types | Notes |
|---|---|---|---|---|
| 802.3z | 1998 | 1 Gbps | 1000BASE-SX, 1000BASE-LX | First Gigabit Ethernet |
| 802.3ae | June 2002 | 10 Gbps | 10GBASE-SR, -LR, -ER, -LX4 | First 10GbE, fiber only |
| 802.3aq | 2006 | 10 Gbps | 10GBASE-LRM | Long reach multimode |
| 802.3ba | June 2010 | 40/100 Gbps | 40GBASE-SR4/LR4, 100GBASE-SR10/LR4/ER4 | First multi-rate standard |
| 802.3bm | 2015 | 40/100 Gbps | 40GBASE-SR4 (OM3/OM4), 100GBASE-SR4 | Improved MMF reach |
| 802.3by | 2016 | 25 Gbps | 25GBASE-SR, 25GBASE-LR | Single-lane 25G |
| 802.3bs | Dec 2017 | 200/400 Gbps | 200GBASE-DR4, 400GBASE-SR16/DR4/FR8/LR8 | First PAM4 in standard |
| 802.3cd | Dec 2018 | 50/100/200 Gbps | 50GBASE-SR/LR/FR/CR, 100GBASE-DR/SR2 | Single-lane 50G NRZ |
| 802.3cm | 2020 | 400 Gbps | 400GBASE-SR4.2 | Short-reach MMF (BiDi SWDM) |
| 802.3ct | 2021 | 100 Gbps | 100GBASE-ZR | Coherent 100G pluggable |
| 802.3cu | 2021 | 100/400 Gbps | 100GBASE-FR1/LR1, 400GBASE-FR4 | Single-lambda 100G |
| 802.3ck | Sep 2022 | 100/200/400 Gbps | Electrical interfaces (100G/lane) | Defines 100G SerDes |
| 802.3db | Sep 2022 | 100/200/400 Gbps | 100GBASE-VR1, 400GBASE-VR4 | Very short reach |
| 802.3df | Feb 2024 | 400/800 Gbps | 800GBASE-DR8, 400GBASE-DR4-2 | 800G standard |
| 802.3dj | ~2026 (target) | 200/400/800/1600 Gbps | 200G/lane PHYs | 1.6T Ethernet |
### OIF Implementation Agreements
| Agreement | Year Published | Speed | Max Reach | Key Feature |
|---|---|---|---|---|
| VSR-5 OIF-05.0 | ~2010 | 100G | 100m | Very short reach coherent |
| 400ZR | Dec 2020 | 400G | 120km (amplified) | Pluggable coherent DWDM in QSFP-DD/OSFP |
| 400ZR+ (vendor-specific) | 2021 | 400G | 450-600km | Extended reach, oFEC |
| 800ZR (in progress) | 2024-2025 | 800G | 80-120km | Next-gen pluggable coherent |
| 1600ZR (in progress) | 2025+ | 1.6T | TBD | Future coherent standard |
| CEI-112G | 2021 | 112 Gbps/lane | Chip-to-module | 100G PAM4 electrical interface |
| CEI-224G | 2025 (target) | 224 Gbps/lane | Chip-to-module | 200G PAM4 electrical interface |
### Multi-Source Agreements (MSAs)
| MSA | Year Published | Speed | Technology | Reach | Key Members |
|---|---|---|---|---|---|
| SFP MSA | 2000 | 1-4G | Various | Varies | Finisar, JDS, Agilent |
| XFP MSA | 2002 | 10G | Various | Varies | Finisar + 10 companies |
| SFP+ MSA (SFF-8431) | 2006 | 10G | NRZ | Varies | Industry-wide |
| QSFP+ MSA (SFF-8436) | 2009 | 40G | 4x10G NRZ | Varies | Industry-wide |
| CFP MSA | 2009 | 100G | 10x10G/4x25G | Varies | Industry-wide |
| QSFP28 MSA (SFF-8665) | 2014 | 100G | 4x25G NRZ | Varies | Industry-wide |
| 100G PSM4 MSA | Mar 2014 | 100G | 4x25G parallel SM | 500m | Corning, Intel, Luxtera, etc. |
| 100G CWDM4 MSA | Sep 2014 | 100G | 4x25G CWDM | 2km | Avago, Finisar, JDSU, etc. |
| SFP28 MSA (SFF-8402) | 2014 | 25G | NRZ | Varies | Industry-wide |
| 25G Ethernet Consortium | 2014 | 25/50G | NRZ | Varies | Arista, Broadcom, Google, Microsoft |
| 100G Lambda MSA | Sep 2017 | 100G/400G | Single-lambda 100G PAM4 | 2-40km | Alibaba, Cisco, Intel, +39 members |
| QSFP-DD MSA | 2017 | 200-800G | 8-lane double density | Varies | Broadcom, Cisco, Finisar, etc. |
| OSFP MSA | 2016 | 400-800G | 8-lane octal | Varies | Arista, Broadcom, Mellanox, etc. |
| OpenZR+ MSA | May 2020 | 100-400G | Coherent DWDM | 1000+km | Acacia, Cisco, Juniper, Lumentum |
| OSFP-XD MSA | 2022 | 1.6-3.2T | 16-lane | Varies | Industry-wide |
| CMIS (Common Mgmt Interface) | v5.0: 2020, v5.3: 2024 | All | Management spec | - | Industry-wide |
---
## 4. CWDM vs DWDM Evolution
### CWDM Technical Specifications
| Parameter | Value |
|---|---|
| Standard | ITU-T G.694.2 |
| Wavelength Range | 1270-1610 nm |
| Channel Spacing | 20 nm |
| Total Channels | 18 (full grid) |
| Practical Channels | 8-16 (water peak limits 1370-1410nm) |
| Laser Type | Uncooled DFB |
| Max Reach | ~70 km (unamplified) |
| Max Per-Channel Speed | 100 Gbps (current), 25G most common |
| Amplification | None (passive) |
| Cost | Lower (uncooled lasers, wider tolerance) |
#### CWDM Wavelength Grid
| Channel | Wavelength (nm) | Band | Notes |
|---|---|---|---|
| 1 | 1271 | O-band | Commonly used |
| 2 | 1291 | O-band | Commonly used |
| 3 | 1311 | O-band | Commonly used |
| 4 | 1331 | O-band | Commonly used |
| 5 | 1351 | E-band | Water peak region |
| 6 | 1371 | E-band | Water peak region |
| 7 | 1391 | S-band | Water peak region (limited to 40km) |
| 8 | 1411 | S-band | Water peak region (limited to 40km) |
| 9 | 1431 | S-band | |
| 10 | 1451 | S-band | |
| 11 | 1471 | C-band edge | Commonly used |
| 12 | 1491 | S/C-band | Commonly used |
| 13 | 1511 | C-band | Commonly used |
| 14 | 1531 | C-band | Commonly used |
| 15 | 1551 | C-band | Commonly used |
| 16 | 1571 | L-band | Commonly used |
| 17 | 1591 | L-band | |
| 18 | 1611 | L-band | |
### DWDM Technical Specifications
| Parameter | Value |
|---|---|
| Standard | ITU-T G.694.1 |
| C-Band Range | 1528.77-1563.86 nm (191.7-196.1 THz) |
| L-Band Range | 1565-1625 nm |
| Channel Spacing (100 GHz) | 0.8 nm, ~40 channels in C-band |
| Channel Spacing (50 GHz) | 0.4 nm, ~80 channels in C-band |
| Channel Spacing (25 GHz) | 0.2 nm, ~160 channels (flex grid) |
| Laser Type | Cooled DFB / Tunable |
| Max Reach | 3000+ km (amplified with EDFA/Raman) |
| Max Per-Channel Speed | 800 Gbps (coherent pluggable) |
| Amplification | EDFA, Raman |
| Flex Grid | Supports variable channel widths (12.5 GHz granularity) |
### Coherent Optics Evolution
| Generation | Year | Per-Wavelength Rate | Modulation | Baud Rate | Form Factor |
|---|---|---|---|---|---|
| Gen 1 | 2011 | 40G | DP-QPSK | 10-12 GBd | Line card (chassis) |
| Gen 2 | 2012 | 100G | DP-QPSK | 32 GBd | Line card / CFP |
| Gen 3 | 2016 | 200G | DP-16QAM | 32-45 GBd | CFP2-DCO |
| Gen 4 | 2018 | 400G | DP-16QAM | 64 GBd | CFP2-DCO |
| Gen 5 (400ZR) | 2021 | 400G | DP-16QAM | 60 GBd | QSFP-DD / OSFP |
| Gen 6 (ZR+) | 2022 | 400G | DP-16QAM (enhanced) | 64 GBd | QSFP-DD / OSFP |
| Gen 7 (800ZR) | 2024 | 800G | DP-64QAM / prob-shaped | 100+ GBd | QSFP-DD / OSFP |
| Gen 8 (1600ZR) | 2026+ | 1.6T | TBD | 130+ GBd | OSFP / OSFP-XD |
### C+L Band Capacity Evolution
| Year | Typical System Capacity | Technology |
|---|---|---|
| 2005 | 40x10G = 400 Gbps | C-band, 100GHz grid |
| 2010 | 80x40G = 3.2 Tbps | C-band, 50GHz grid |
| 2015 | 80x100G = 8 Tbps | C-band, 50GHz grid, coherent |
| 2020 | 80x400G = 32 Tbps | C-band, flex grid |
| 2024 | 80x800G = 64 Tbps | C-band, flex grid |
| 2025+ | 120+x800G = 96+ Tbps | C+L band, flex grid |
### Key WDM Transceiver Types by Speed
| Speed | CWDM Variants | DWDM Variants |
|---|---|---|
| 1G | SFP CWDM (18 wavelengths) | SFP DWDM (C-band) |
| 10G | SFP+ CWDM, XFP CWDM | XFP/SFP+ DWDM (tunable) |
| 25G | SFP28 CWDM | SFP28 DWDM |
| 40G | QSFP+ CWDM4 | CFP DWDM (coherent) |
| 100G | QSFP28 CWDM4 | QSFP28 DWDM / CFP2-DCO |
| 400G | (not practical) | QSFP-DD/OSFP ZR/ZR+ |
| 800G | (not practical) | OSFP/QSFP-DD 800ZR/ZR+ |
---
## 5. Major Transceiver Manufacturers
### Manufacturer Database
| Company | HQ | Founding | Key Milestones | Specialty | 2024 Revenue (transceivers) | Market Position |
|---|---|---|---|---|---|---|
| **Coherent Corp.** | Pittsburgh, USA | 1971 (as II-VI) | Acquired Finisar ($3.2B, 2019), Coherent ($6.56B, 2022) | Coherent, Datacom, InP lasers | ~$2.5B+ | #2 globally, #1 telecom |
| **Zhongji Innolight** | Suzhou, China | 2008 | #1 globally 2023, 50%+ Nvidia wallet share | Datacom, 800G/1.6T | ~$3.3B (114% YoY growth) | #1 globally |
| **Lumentum** | San Jose, USA | 2015 (spun off JDS Uniphase) | Acquired Cloud Light ($750M, 2024), Oclaro ($1.8B, 2018) | Coherent, lasers, 3D sensing | ~$1.5B | #3 globally |
| **Broadcom (Optical)** | San Jose, USA | Broadcom acquired original Avago/LSI/Broadcom | Key DSP/PAM4 supplier | DSP chips, SiPh, VCSEL | ~$1B+ | Major component supplier |
| **Cisco (Silicon Photonics)** | San Jose, USA | Acquired Luxtera ($660M, 2019), Acacia ($4.6B, 2021) | Integrated SiPh transceivers | SiPh, coherent (via Acacia) | Internal consumption + merchant | #4-5 globally |
| **Eoptolink** | Shenzhen, China | 2004 | 175% revenue growth 2024, #3 globally | Datacom, LPO, SiPh | ~$1.2B | #3 globally |
| **HG Genuine** | Wuhan, China | 2001 | ByteDance/TikTok supplier since 2021 | Datacom, access optics | ~$600M+ | #8 globally |
| **Accelink Technologies** | Wuhan, China | 2001 | Chinese cloud supplier | Telecom, passive components | ~$600M+ | #5 globally |
| **Hisense Broadband** | Qingdao, China | 2003 (Hisense subsidiary) | PON/access market leader | Access, PON, 5G | ~$600M+ | #6 globally |
| **Source Photonics** | West Hills, USA / China | 2002 | Chinese cloud supplier | Access, enterprise, DC | ~$400M | #9 globally |
| **Applied Optoelectronics (AOI)** | Sugar Land, USA | 1997 | CATV and DC optics | VCSEL, DFB, DC transceivers | ~$200M | Niche |
| **Intel Silicon Photonics** | Santa Clara, USA | SiPh division ~2010 | 100G PSM4, 1.6T SiPh engines | Silicon photonics platform | Sold to third parties (Jabil etc.) | Technology leader |
| **ColorChip** | Yokneam, Israel | 2001 | Acquired by Source Photonics 2018 | PLC-based transceivers | (merged) | Acquired |
| **Broadex Technologies** | Chengdu, China | 2016 | Fast-growing Chinese supplier | Datacom, 400G/800G | ~$300M | Emerging |
| **Centera Photonics** | Taiwan | 2007 | 800G/1.6T development | Datacom transceivers | ~$150M | Regional |
### Market Share Trends (Global Optical Transceiver Revenue)
| Year | #1 | #2 | #3 | Chinese in Top 10 | Key Shift |
|---|---|---|---|---|---|
| 2015 | Finisar | Lumentum/JDSU | Avago/Broadcom | 2-3 | US/Japan dominance |
| 2018 | Finisar | II-VI | Lumentum | 3-4 | Pre-merger era |
| 2020 | Coherent (II-VI+Finisar) | Innolight | Lumentum | 4-5 | Chinese rise begins |
| 2022 | Innolight = Coherent (~$1.4B each) | Lumentum | Accelink | 5-6 | Chinese parity |
| 2023 | Innolight | Coherent | Lumentum | 7 of top 10 | Chinese dominance |
| 2024 | Innolight ($3.3B) | Coherent (~$2.5B) | Eoptolink ($1.2B) | 7 of top 10 | AI-driven surge |
### Major M&A Timeline
| Year | Acquirer | Target | Value | Impact |
|---|---|---|---|---|
| 2013 | Oclaro | Opnext | $180M | Combined coherent portfolio |
| 2015 | Lumentum spins off | from JDS Uniphase | - | Created independent photonics leader |
| 2018 | II-VI | Finisar | $3.2B | Created #1 transceiver company |
| 2018 | Lumentum | Oclaro | $1.8B | Strengthened InP/coherent capabilities |
| 2019 | Cisco | Luxtera | $660M | Silicon photonics integration |
| 2019 | Cisco | Acacia Communications | $4.6B | Coherent DSP leadership |
| 2021 | Intel | (SiPh division established) | Internal | 100G-1.6T silicon photonics engines |
| 2022 | II-VI | Coherent Inc. (laser co.) | $6.56B | Renamed to Coherent Corp. |
| 2024 | Lumentum | Cloud Light Technology | $750M | DC infrastructure boost |
| 2024 | Nvidia | (investing in optical supply chain) | Various | Vertical integration signal |
---
## 6. Next-Generation Technologies (2025-2030)
### 1.6T Transceivers
| Parameter | Gen1 (16x100G) | Gen2 (8x200G) |
|---|---|---|
| Timeline | 2025 (shipping) | 2026 (maturing) |
| Lane Rate | 100G PAM4 | 200G PAM4 |
| Lane Count | 16 | 8 |
| Form Factor | OSFP-XD | OSFP1600, OSFP, QSFP-DD1600 |
| DSP Process | 5nm | 3nm |
| Power (retimed) | ~25-30W | ~17-26W |
| Power (LPO) | ~8-12W | ~5W |
| Key DSPs | Broadcom Sian2, Marvell Aries | Broadcom Sian3, Marvell next-gen |
### Co-Packaged Optics (CPO) Timeline
| Year | Milestone |
|---|---|
| 2021 | Broadcom Tomahawk 4 + Humboldt = first CPO chipset |
| 2022 | Broadcom Tomahawk 5 + Bailly = first volume-production CPO |
| 2025 Q1 | NVIDIA announces first 1.6T CPO system (Micro Ring Modulators) |
| 2025 Q2 | NVIDIA Quantum-X SiPh switch ships |
| 2025 | TSMC COUPE platform adopted by NVIDIA, Broadcom |
| 2025 | Meta tests Broadcom CPO for 1M+ link-hours |
| 2025 Nov | Ayar Labs integrates TeraPHY into GUC ASIC workflow |
| 2026 H2 | NVIDIA Spectrum-X Photonics system ships |
| 2026-2027 | Broad CPO commercialization begins |
| 2028-2030 | Large-scale CPO deployment in hyperscale |
### CPO vs LPO vs Traditional DSP Comparison
| Feature | Traditional (DSP) | LPO (Linear Drive) | CPO (Co-Packaged) |
|---|---|---|---|
| Power Consumption | Baseline | -30 to -50% | -50 to -84% |
| Latency | ~100ns (DSP) | <15ns reduction | Near-zero electrical path |
| Serviceability | Hot-swappable | Hot-swappable | Requires board replacement |
| Maturity | Production | Shipping (NVIDIA, Meta) | Pre-production/early access |
| Cost | Baseline | Lower (no DSP in module) | Higher initially, lower at scale |
| Best For | Long reach, interop | Short reach (<2km), AI clusters | Ultra-dense, scale-up AI |
| Market Share (2025) | ~60% of 800G/1.6T | ~30% | ~5% |
| Market Share (2030, projected) | ~30% | ~40% | ~30% |
### Silicon Photonics Adoption
| Year | SiPh Share of Transceivers | Key Driver |
|---|---|---|
| 2018 | ~14% | 100G PSM4 (Intel) |
| 2020 | ~20% | 400G DR4 ramp |
| 2022 | ~25% | 400G mainstream |
| 2024 | ~35% | 800G ramp |
| 2025 | ~40-45% | 800G mainstream, 1.6T intro |
| 2030 (proj.) | ~60% | LPO + CPO adoption |
### O-Band vs C-Band Data Center Trends
| Parameter | O-Band (1310nm) | C-Band (1550nm) |
|---|---|---|
| Primary Use | Data center interconnect (<10km) | Metro/long-haul, DCI (>10km) |
| Technology | Direct detect, PAM4 | Coherent or PAM4 WDM |
| Standards | DR, FR, LR variants | ZR, ZR+, DWDM |
| Advantages | Lower cost, simpler, lower dispersion | Higher capacity, longer reach |
| Trend | Dominant for intra-DC | Growing for inter-DC via ZR/ZR+ |
| 800G Example | 800G-DR8 (O-band, 500m) | 800ZR (C-band, 120km) |
---
## 7. Market Data Points
### Global Optical Transceiver Market Size
| Year | Market Size (USD) | YoY Growth | Key Driver |
|---|---|---|---|
| 2019 | ~$6.5B | - | 100G mainstream |
| 2020 | ~$7.0B | +8% | COVID + cloud demand |
| 2021 | ~$8.0B | +14% | 400G ramp begins |
| 2022 | ~$9.0B | +13% | 400G mainstream deployment |
| 2023 | ~$10.5B | +17% | AI infrastructure begins |
| 2024 | ~$13.6B | +30% | AI explosion, 800G ramp |
| 2025 (est.) | ~$15.6-16B | +15-18% | 800G mainstream, 1.6T intro |
| 2029 (proj.) | ~$25B | CAGR 13% | 1.6T mainstream |
| 2034 (proj.) | ~$46B | CAGR 17% | CPO + next-gen |
### Port Shipment Data
| Metric | 2023 | 2024 | 2025 (est.) |
|---|---|---|---|
| Total transceiver units deployed | ~15M | ~22.5M | ~34.5M |
| 400G+800G unit shipments | ~6M | 20M+ | 30M+ (est.) |
| Quarterly record (400/800G) | <3M | 5M+ (Q2 2024) | 7M+ (projected) |
| 400G/800G YoY growth | - | +250% | +60% (800G specifically) |
| 800G as % of high-speed | ~20% | ~35% | ~50% (est.) |
### Average Selling Price (ASP) Trends
| Speed | Launch ASP | 2024 ASP | ASP Decline Pattern |
|---|---|---|---|
| 1G SFP | ~$500 (2001) | ~$5-15 | >95% decline over 20 years |
| 10G SFP+ | ~$500 (2007) | ~$15-40 | >90% decline over 15 years |
| 25G SFP28 | ~$100 (2016) | ~$15-30 | ~75% decline over 8 years |
| 40G QSFP+ | ~$300 (2012) | ~$30-80 | ~80% decline over 12 years |
| 100G QSFP28 | ~$1,000 (2015) | ~$50-120 | ~90% decline, 60% in last 5 years |
| 400G QSFP-DD | ~$800-1,200 (2020) | ~$120-250 | ~75% decline, SR8 50% in 1 year |
| 800G OSFP | ~$800-1,000 (2023) | ~$360-450 | Early decline, still premium |
| 1.6T OSFP-XD | ~$1,300-1,500 (2025) | $1,300-1,500 | Launch pricing, projected ~$1,100 by 2027 |
### ASP Decline Model (Typical Pattern)
```
Year 0 (Launch): 100% (premium pricing)
Year 1: 80-90% (early adoption)
Year 2: 60-70% (volume ramp)
Year 3: 40-50% (mainstream)
Year 4: 30-40% (commoditization begins)
Year 5+: 20-30% (commodity, Chinese competition)
Year 7+: 10-15% (floor pricing)
```
### Market Segment Split (2024-2025)
| Segment | 2024 Share | 2025 Share (est.) | Growth Driver |
|---|---|---|---|
| Data Centers | 45-55% | 55-60% | AI/ML clusters, hyperscale |
| Telecommunications | 30-40% | 25-30% | 5G, coherent metro/long-haul |
| Enterprise Networking | 14-20% | 12-15% | LAN/WAN upgrades to 100G |
| Other (defense, govt, research) | 5-10% | 5-8% | Specialty applications |
### Datacom vs Telecom Module Revenue
| Year | Datacom Revenue | Telecom Revenue | Datacom Share |
|---|---|---|---|
| 2020 | ~$4B | ~$3B | 57% |
| 2022 | ~$5.5B | ~$3.5B | 61% |
| 2024 | ~$9B+ | ~$4B | 69% |
| 2025 (est.) | ~$12B+ | ~$4B | 75% |
### Coherent Pluggable Shipments
| Year | 400ZR/ZR+ Units | Key Milestone |
|---|---|---|
| 2021 | <50K | First GA shipments |
| 2022 | ~100-150K | Initial ramp |
| 2023 | ~300K | Broad deployment |
| 2024 | ~500K | Fastest adopted coherent ever |
| 2025 (est.) | ~600K+ | 800ZR enters market |
### AI Optics Market Specifically
| Year | AI Optics Market | Notes |
|---|---|---|
| 2023 | ~$3B | GPU interconnect demand begins |
| 2024 | ~$5B | NVIDIA GB200 drives 800G demand |
| 2025 | ~$7-8B | 800G mainstream for AI |
| 2026 (est.) | ~$10B+ | 1.6T for AI clusters |
### Vendor Count Per Standard (approximate)
| Standard/Type | Vendor Count (2025) | Notes |
|---|---|---|
| 1G SFP | 100+ | Fully commoditized |
| 10G SFP+ | 80+ | Commoditized |
| 25G SFP28 | 50+ | Maturing |
| 100G QSFP28 | 40+ | Maturing |
| 400G QSFP-DD | 25+ | Mainstream competition |
| 400G ZR/ZR+ | 10-15 | Specialized |
| 800G OSFP/QSFP-DD | 15-20 | Growing |
| 800G ZR/ZR+ | 5-8 | Emerging |
| 1.6T OSFP/OSFP-XD | 8-12 | Early stage |
---
## 8. Database Schema Recommendations
### Core Tables
```
TABLE: form_factors
- id, name, year_introduced, year_mainstream, year_decline, year_obsolete
- max_speed_gbps, connector_type, lane_count, width_mm, depth_mm
- power_class_w, backward_compatible_with, msa_spec_url
- status (emerging/active/declining/legacy/obsolete)
TABLE: speed_tiers
- id, speed_gbps, year_standardized, year_mainstream, year_peak, year_decline
- primary_ieee_standard, modulation_type, lanes_config
- typical_launch_asp_usd, current_asp_usd
TABLE: standards
- id, name, organization (IEEE/OIF/MSA), year_published, year_ratified
- speed_gbps, reach_km, key_phy_types, status
TABLE: manufacturers
- id, name, hq_country, year_founded, specialties
- annual_revenue_usd, market_rank, key_products
TABLE: market_data (time series)
- id, year, quarter, metric_name, metric_value, unit
- segment (datacom/telecom/enterprise), source
TABLE: products
- id, manufacturer_id, form_factor_id, speed_tier_id
- model_name, wavelength_nm, reach_km, fiber_type
- modulation, fec_type, power_w, temperature_range
- year_launched, current_asp_usd, status
TABLE: technology_transitions
- id, technology_name, category (modulation/integration/packaging)
- year_introduced, year_mainstream, year_peak
- market_share_pct, hype_cycle_phase
TABLE: acquisitions
- id, acquirer_id, target_name, year, value_usd, strategic_rationale
```
---
## 9. Hype Cycle Analysis
### Technology Hype Cycle Positions (2026)
```
PEAK OF INFLATED EXPECTATIONS:
- Co-Packaged Optics (CPO)
- 3.2T transceivers
- Optical compute interconnect
SLOPE OF ENLIGHTENMENT:
- 1.6T pluggable transceivers
- Linear Drive Optics (LPO)
- 200G/lane PAM4
- Near-Packaged Optics (NPO)
PLATEAU OF PRODUCTIVITY:
- 800G pluggable transceivers
- 400ZR/ZR+ coherent pluggables
- Silicon photonics (in 400G/800G)
- PAM4 modulation
ENTERING DECLINE:
- 400G pluggable (mainstream, starting decline)
- 100G QSFP28 (commoditized)
- NRZ modulation (for new designs)
OBSOLESCENCE TRAJECTORY:
- 40G QSFP+
- 10G XFP
- CFP/CFP2/CFP4 (except DCO)
- CWDM for high-speed (>100G)
```
### Form Factor Hype Cycles (Historical Overlay)
```
Innovation Peak Trough Slope Plateau
Trigger Hype Disillusion Enlighten Productivity
GBIC 1995 1998 2002 - 2000-2004
SFP 2001 2003 - 2004 2005-forever
XFP 2002 2005 2008 - 2006-2012
SFP+ 2006 2008 - 2009 2010-forever
QSFP+ 2012 2014 - 2015 2016-2022
QSFP28 2014 2016 - 2017 2018-2024
QSFP-DD 2019 2021 - 2022 2023-present
OSFP 2016 2020 - 2022 2023-present
OSFP-XD 2022 2025 - 2026(est) 2027(est)
```
### Speed Tier Lifecycle Model
Each speed tier follows a predictable ~10-year lifecycle:
```
Years 0-2: INTRODUCTION - Standards ratified, samples shipping, $$$$ pricing
Years 2-4: GROWTH - Volume ramps, multiple vendors, pricing drops 40-60%
Years 4-6: MAINSTREAM - Peak shipments, broad adoption, pricing drops another 30-50%
Years 6-8: MATURITY - Pricing floor, commoditized, Chinese competition dominant
Years 8-10: DECLINE - Next-gen overtakes, volumes drop, maintenance-only
Years 10+: LEGACY - Minimal shipments, long-tail demand
```
| Speed | Introduction | Growth | Mainstream | Maturity | Decline | Legacy |
|---|---|---|---|---|---|---|
| 1G | 1998-2002 | 2002-2005 | 2005-2010 | 2010-2016 | 2016-2020 | 2020+ |
| 10G | 2002-2006 | 2006-2010 | 2010-2016 | 2016-2022 | 2022+ | - |
| 25G | 2014-2017 | 2017-2019 | 2019-2023 | 2023-2026 | 2026+ | - |
| 40G | 2010-2013 | 2013-2015 | 2015-2019 | 2019-2022 | 2022+ | - |
| 100G | 2014-2017 | 2017-2020 | 2020-2024 | 2024-2026 | 2026+ | - |
| 400G | 2020-2022 | 2022-2024 | 2024-2027 | 2027-2030 | 2030+ | - |
| 800G | 2023-2025 | 2025-2027 | 2027-2030 | 2030-2033 | 2033+ | - |
| 1.6T | 2025-2027 | 2027-2029 | 2029-2032 | 2032-2035 | 2035+ | - |
---
## Sources & References
### Market Research
- [Cignal AI - 800GbE Optics Shipments](https://cignal.ai/2025/05/800gbe-optics-shipments-to-grow-60-in-2025/)
- [Cignal AI - 20M 400G/800G Shipments 2024](https://cignal.ai/2025/01/over-20-million-400g-800g-datacom-optical-module-shipments-expected-for-2024/)
- [LightCounting - Silicon Photonics May 2025](https://www.lightcounting.com/newsletter/en/may-2025-silicon-photonics-linear-drive-pluggable-and-cpo-updated-november-2025-334)
- [LightCounting - AI Optics Jan 2025](https://www.lightcounting.com/newsletter/en/january-2025-optics-for-ai-clusters-319)
- [Mordor Intelligence - Optical Transceiver Market](https://www.mordorintelligence.com/industry-reports/optical-transceiver-market)
- [MarketsAndMarkets - Optical Transceiver Market 2030](https://www.marketsandmarkets.com/Market-Reports/optical-transceiver-market-161339599.html)
- [Fortune Business Insights - Optical Transceiver Market](https://www.fortunebusinessinsights.com/optical-transceiver-market-108985)
### Standards & Specifications
- [IEEE 802.3 Working Group Archive](https://www.ieee802.org/3/archive.html)
- [IEEE 802.3 - Wikipedia](https://en.wikipedia.org/wiki/IEEE_802.3)
- [OIF 400ZR Implementation Agreement](https://www.oiforum.com/technical-work/hot-topics/400zr-2/)
- [OpenZR+ MSA](https://openzrplus.org/)
- [100G Lambda MSA](https://100glambda.com/)
- [CWDM4 MSA](https://cwdm4-msa.org/)
- [100G PSM4 MSA](http://www.psm4.org/)
### Form Factors & Technology
- [Prooptix - History of Form Factors](https://www.prooptix.com/news/transceiver-form-factors/)
- [Meticulous Research - 25 Years of Optical Transceiver Evolution](https://www.meticulousresearch.com/blog/207/evolution-of-optical-transceiver-technologies-in-the-last-25-years)
- [Vitex - Transceiver Form Factors Guide](https://www.vitextech.com/blogs/blog/transceiver-form-factors)
- [FS.com - High-Speed Transceivers Guide](https://www.fs.com/blog/a-comprehensive-guide-to-highspeed-transceivers-400g-800g-and-the-leap-to-16t-13767.html)
### Coherent Optics & WDM
- [WWT - 400G-ZR & ZR+ Guide](https://www.wwt.com/blog/400gzr-and-zr-the-latest-in-pluggable-coherent-dwdm)
- [Acacia/Cisco - 2024 Coherent Optics Review](https://acacia-inc.com/blog/a-look-back-at-2024-whats-ahead-for-coherent-optics-in-2025/)
- [Smartoptics - CWDM DWDM Explained](https://smartoptics.com/knowledgebank-post/cwdm-dwdm-explained/)
- [FS.com - DWDM/CWDM ITU Channels Guide](https://www.fs.com/blog/dwdmcwdm-wavelength-itu-channels-guide-3149.html)
### Manufacturer & Industry Analysis
- [Iamfabian - 800G/1.6T Transceiver Battle](https://iamfabian.substack.com/p/pluggables-power-and-geopolitics)
- [Deep Fundamental - Optical Module Market Deep Dive](https://deepfundamental.substack.com/p/deep-dive-optical-module-market)
- [Chinese Suppliers Dominate 2024 Rankings](https://www.opticaltransceivermodules.com/news/chinese-optical-transceiver-suppliers-dominate-global-rankings-225829.html)
- [Coherent Corp - Wikipedia](https://en.wikipedia.org/wiki/Coherent_Corp.)
### Next-Gen Technology
- [EDN - Co-Packaged Optics 2026](https://www.edn.com/where-co-packaged-optics-cpo-technology-stands-in-2026/)
- [IDTechEx - CPO Market Forecast](https://www.idtechex.com/en/research-report/co-packaged-optics-cpo/1138)
- [Eoptolink - Gen2 1.6T at OFC 2025](https://www.eoptolink.com/news/361-eoptolink-launches-its-gen2-1-6t-osfp-and-osfp-rhs-transceiver-family-at-ofc-2025)
- [Broadcom OFC 2025 Advances](https://investors.broadcom.com/news-releases/news-release-details/broadcom-advances-optical-connectivity-ai-infrastructure)
- [Jabil 1.6T Transceiver Launch](https://investors.jabil.com/news/news-details/2025/Jabil-Launches-1-6T-Pluggable-Transceiver-to-Support-Growing-Demand-for-Intra-Data-Center-and-AI-Connectivity/default.aspx)

View File

@ -0,0 +1,871 @@
# Revenue Lifecycle Prediction Models for Optical Networking Equipment
**Research Date: 2026-03-28**
**Scope: Optical transceivers, switches, routers — product lifecycle revenue prediction**
---
## Table of Contents
1. [Revenue Lifecycle Prediction Models](#1-revenue-lifecycle-prediction-models)
2. [Historical Data Points for Optical Transceivers](#2-historical-data-points-for-optical-transceivers)
3. [Regional/Country-Level Adoption Differences](#3-regionalcountry-level-adoption-differences)
4. [Conference-to-Market Timeline Analysis](#4-conference-to-market-timeline-analysis)
5. [Switch/Router Refresh Cycles](#5-switchrouter-refresh-cycles)
6. [Predictive Models for Future Products](#6-predictive-models-for-future-products)
7. [Recommended Implementation for TIP](#7-recommended-implementation-for-tip)
---
## 1. Revenue Lifecycle Prediction Models
### 1.1 Bass Diffusion Model (Foundation)
The Bass model (1969) is the foundational framework for technology adoption forecasting.
**Core Equation:**
```
f(t) = (p + q * F(t)) * (1 - F(t))
```
Where:
- `f(t)` = instantaneous rate of adoption at time t (fraction of market potential)
- `F(t)` = cumulative fraction of adopters at time t
- `p` = coefficient of innovation (external influence / "advertising effect")
- `q` = coefficient of imitation (internal influence / "word-of-mouth effect")
**Closed-form cumulative adoption:**
```
F(t) = (1 - exp(-(p+q)*t)) / (1 + (q/p)*exp(-(p+q)*t))
```
**Revenue form (units * price):**
```
R(t) = m * f(t) * P(t)
```
Where `m` = total market potential, `P(t)` = price at time t.
**Typical parameter ranges (telecom/technology):**
- p: 0.01 - 0.03 (innovation coefficient)
- q: 0.2 - 0.4 (imitation coefficient)
- Peak adoption occurs at: t_peak = (1/(p+q)) * ln(q/p)
**Source:** Bass, F.M. (1969). "A New Product Growth for Model Consumer Durables." Management Science, 15(5), 215-227.
- [Bass diffusion model - Wikipedia](https://en.wikipedia.org/wiki/Bass_diffusion_model)
- [GeeksforGeeks explanation](https://www.geeksforgeeks.org/machine-learning/bass-diffusion-model/)
### 1.2 Norton-Bass Multi-Generation Diffusion Model (CRITICAL for TIP)
The Norton-Bass (NB) model (1987) extends Bass to handle **successive technology generations** — exactly the pattern seen in optical transceivers (1G → 10G → 40G → 100G → 400G → 800G → 1.6T).
**Two-Generation Formulation:**
Generation 1 introduced at t=0, Generation 2 at t=τ₂.
```
Units-in-use for G1:
N₁(t) = m₁ * F₁(t) for t < τ
N₁(t) = m₁ * F₁(t) * (1 - F₂(t - τ₂)) for t ≥ τ₂
Units-in-use for G2:
N₂(t) = 0 for t < τ
N₂(t) = (m₂ + m₁ * F₁(t)) * F₂(t - τ₂) for t ≥ τ₂
```
Where:
- `Fᵢ(t)` = Bass cumulative adoption for generation i
- `mᵢ` = incremental market potential for generation i
- `τ₂` = introduction time of generation 2
**Key finding:** p and q parameters are generally **the same between successive generations** — only market potential (m) changes.
**Three-Generation Extension:**
```
N₁(t) = m₁*F₁(t)*(1-F₂(t-τ₂)) for τ₂ ≤ t < τ
N₁(t) = m₁*F₁(t)*(1-F₂(t-τ₂))*(1-F₃(t-τ₃)) for t ≥ τ₃
N₂(t) = (m₂+m₁*F₁(t))*F₂(t-τ₂)*(1-F₃(t-τ₃)) for t ≥ τ₃
N₃(t) = (m₃ + (m₂+m₁*F₁(t))*F₂(t-τ₂) + m₁*F₁(t)*(1-F₂(t-τ₂)))*F₃(t-τ₃)
```
**Source:** Norton, J.A. & Bass, F.M. (1987). "A Diffusion Theory Model of Adoption and Substitution for Successive Generations of High-Technology Products." Management Science, 33(9), 1069-1086.
- [INSEAD working paper](https://sites.insead.edu/facultyresearch/research/doc.cfm?did=49784)
- [Semantic Scholar](https://www.semanticscholar.org/paper/A-diffusion-theory-model-of-adoption-and-for-of-Norton-Bass/a030faf95a67497226b9f00bdaf354e2e95f6ac7)
### 1.3 Generalized Norton-Bass (GNB) Model
Jiang & Jain (2012) extended Norton-Bass to differentiate **leapfrogging** from **switching** — critical for optical transceivers where some data centers skip generations (e.g., skip 40G, go from 10G to 100G).
**Leapfrogging:** Potential adopters skip older generation and directly adopt newer generation.
**Switching:** Existing adopters of older generation migrate to newer generation.
**Two-Generation GNB Formulation:**
```
Leapfrog adoptions of G2:
L₂(t) = m₂ * F₂(t - τ₂)
Switching adoptions from G1 to G2:
S₂(t) = m₁ * F₁(t) * F₂(t - τ₂)
Total G2 units-in-use:
N₂(t) = L₂(t) + S₂(t) = (m₂ + m₁*F₁(t)) * F₂(t - τ₂)
G1 remaining units:
N₁(t) = m₁ * F₁(t) * (1 - F₂(t - τ₂))
```
**Empirical validation (DRAM generations):**
- 4K, 16K, 64K DRAM quarterly shipments 1974-1984
- Adjusted R² values: 0.9853, 0.9707, 0.999
- Of 64K DRAM adoptions: **60% new adopters**, **33% switching from 16K**, rest leapfrogging
**Software:** Available in R via the `diffusion` package (`Nortonbass` function).
**Source:** Jiang, Z. & Jain, D.C. (2012). "A Generalized Norton-Bass Model for Multigeneration Diffusion." Management Science, 58(10), 1887-1897.
- [Full PDF - Iowa State](https://dr.lib.iastate.edu/article/scm_pubs/1026)
- [SSRN](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3112796)
- [INFORMS](https://pubsonline.informs.org/doi/pdf/10.1287/mnsc.1120.1529)
- [R package docs](https://rdrr.io/cran/diffusion/man/Nortonbass.html)
### 1.4 Gompertz Curve for Revenue Lifecycle
The Gompertz curve is particularly effective for modeling the **asymmetric S-curve** of technology market growth, where early adoption accelerates fast but saturation is gradual.
**Formula:**
```
y(t) = K * exp(log(y₀/K) * exp(-α*t))
```
Where:
- `K` = carrying capacity (maximum market size / saturation level)
- `y₀` = initial value
- `α` = growth rate coefficient
- **Inflection point occurs at 36.8% of upper asymptote** (vs. 50% for logistic)
**Alternative parametrization:**
```
y(t) = a * b^(c^t)
```
Where a = upper asymptote, 0 < b < 1, 0 < c < 1.
**Application to semiconductors:** Wally Rhines (Mentor Graphics) demonstrated that the Gompertz curve can determine where particular semiconductor market segments are in their lifecycle by plotting cumulative unit production against the Gompertz S-curve. **By determining the three coefficients early in the cycle, the remainder of the cycle can be predicted.**
**Gompertz vs. Logistic:** When Y is low, Gompertz grows faster; when Y is high, Gompertz grows slower. This asymmetry better matches technology markets where early adoption is driven by innovators (fast) but late-stage saturation is drawn out by laggards.
**Source:**
- [EE Times - Gompertz for semiconductor prediction](https://www.eetimes.com/predicting-semiconductor-industry-growth-drop-the-crystal-ball-and-use-the-gompertz-curve/)
- [Semiengineering - Gompertz model](https://semiengineering.com/mathematic-model-helps-predict-markets-that-will-drive-semiconductor-growth/)
- [FasterCapital - Business growth](https://fastercapital.com/content/Gompertz-Curve--Modeling-Mastery--Using-the-Gompertz-Curve-to-Forecast-Business-Growth.html)
### 1.5 Weibull Distribution for Lifecycle Curves
The Weibull distribution provides a **flexible framework** for modeling both growth and decline phases with varying shapes.
**Lifecycle formulation:**
```
f(t) = (β/η) * (t/η)^(β-1) * exp(-(t/η)^β)
```
Where:
- `β` = shape parameter (β < 1: decreasing failure/decline rate, β > 1: increasing)
- `η` = scale parameter (characteristic life)
A 2019 paper proposes a **two-step Weibull distribution** with four parameters for modeling bimodal product lifecycle diffusion curves — fitting both the rise and fall of product sales.
**Source:** "Using Weibull Distribution for Modeling Bimodal Diffusion Curves: A Naive Framework to Study Product Life Cycle." International Journal of Innovation and Technology Management, 2019.
- [World Scientific](https://www.worldscientific.com/doi/10.1142/S0219877019500500)
- [Technological Forecasting & Social Change - Weibull for tech change](https://www.sciencedirect.com/science/article/abs/pii/0040162580900268)
### 1.6 Revenue Duration Model (Composite)
For TIP, the recommended composite model for a single transceiver generation:
```
Revenue(t) = Units(t) * ASP(t)
Where:
Units(t) = Norton-Bass adoption model (accounts for cannibalization by next gen)
ASP(t) = ASP₀ * exp(-λ*t) (exponential price erosion)
Duration above 50% peak revenue:
Solve for t₁, t₂ where R(t) = 0.5 * R_peak
Duration = t₂ - t₁
```
---
## 2. Historical Data Points for Optical Transceivers
### 2.1 Total Optical Transceiver Market Revenue by Year
| Year | Total Market Revenue | Growth | Source |
|------|---------------------|--------|--------|
| 2019 | ~$7.5-8.0B | Declined | LightCounting (derived) |
| 2020 | ~$8.8-9.3B | +17% | LightCounting |
| 2021 | ~$10.0B+ | +10% | LightCounting milestone |
| 2022 | ~$11.0-11.5B | +14% | LightCounting |
| 2023 | ~$10.7-10.9B | -6% | LightCounting; telecom downturn |
| 2024 | ~$13.6B | Strong rebound | MarketsandMarkets; AI-driven |
| 2025 | ~$23B (projected) | +60%+ | LightCounting Dec 2025 |
**Datacom optical segment specifically:**
- 2024: ~$9B (Cignal AI)
- 2025: >$16B (Cignal AI, +60%)
- 2026: ~$12B high-speed datacom segment projected (Cignal AI, as 800G peaks)
**Sources:**
- [LightCounting newsletter](https://www.lightcounting.com/newsletter/en/december-2025-quarterly-market-update-322)
- [Cignal AI Jan 2025](https://cignal.ai/2025/01/over-20-million-400g-800g-datacom-optical-module-shipments-expected-for-2024/)
- [Cignal AI May 2025](https://cignal.ai/2025/05/800gbe-optics-shipments-to-grow-60-in-2025/)
- [MarketsandMarkets](https://www.marketsandmarkets.com/Market-Reports/optical-transceiver-market-161339599.html)
### 2.2 Generation Lifecycle Timelines
| Generation | Datacom Launch | Peak Revenue Window | Years to Peak | Cycle → Next Gen |
|------------|---------------|--------------------|--------------:|------------------|
| 1G SFP | ~2002 | ~2008-2012 | ~6-8 yrs | ~5 yrs |
| 10G SFP+ | ~2007-2010 | ~2013-2016 | ~4-6 yrs | ~4 yrs |
| 40G QSFP+ | ~2011-2013 | ~2015-2017 | ~3-4 yrs | ~3 yrs (largely skipped) |
| 100G QSFP28 | ~2014 | ~2018-2020 | ~4 yrs | ~3-4 yrs |
| 400G QSFP-DD | ~2018-2019 | ~2022-2024 | ~3-4 yrs | ~3 yrs |
| 800G OSFP | ~2023-2024 | ~2025-2026 (proj) | ~2-3 yrs | ~2 yrs |
| 1.6T OSFP-XD | ~2025-2026 | ~2027-2028 (proj) | ~2 yrs | ~2 yrs |
**KEY FINDING: Innovation cycles are compressing from 3-4 years historically to ~2 years currently.**
**Sources:**
- [Introl blog](https://introl.com/blog/fiber-optics-data-center-state-of-art-optical-interconnect-2025)
- [Cignal AI](https://cignal.ai/2025/05/800gbe-optics-shipments-to-grow-60-in-2025/)
- [Medium - Module Evolution](https://medium.com/@aicplight888/optical-module-evolution-from-400g-to-3-2t-11b087f43c04)
### 2.3 Price Erosion Curves
#### 100G QSFP28 SR4 Price History
| Period | Approx. ASP | Notes |
|--------|------------:|-------|
| 2015-2016 | >$2,000 | Early production, few suppliers |
| 2017 | ~$800-$1,200 | Volume ramp begins |
| 2018 | ~$400-$700 | Chinese suppliers enter |
| 2019 | ~$200-$400 | Commoditization |
| 2020 | ~$100-$250 | COVID demand + continued pressure |
| 2021-2022 | ~$80-$150 | Mature market |
| 2024-2026 | ~$29-$99 | Third-party vendors (FS.com, Optcore) |
**Overall decline:** ~60% in 5 years, ~95%+ from launch to commodity phase.
**Price erosion model:**
```
ASP(t) = ASP₀ * exp(-λ*t)
For 100G QSFP28:
ASP₀ ≈ $2,000 (launch year 2015)
λ ≈ 0.35-0.40 per year (aggressive phase)
Half-life: ~2 years
```
#### 800G Module Pricing (2024)
| Module Type | ASP (2024) |
|-------------|----------:|
| 800G Multimode (SR8, VCSEL) | ~$500 |
| 800G LPO | ~$600 |
| 800G Single-mode (EML) | >$700 |
| NVIDIA LinkX 800G (bulk) | ~$1,000 |
| 800G FR4/DR8 (reseller) | $1,000-$3,800 |
#### 1.6T Module Pricing
| Period | ASP |
|--------|----:|
| Q4 2024 (initial) | ~$2,000 |
| 2025 (maturity) | ~$1,500 (projected) |
**Sources:**
- [Deep Fundamental - Optical Module Market](https://deepfundamental.substack.com/p/deep-dive-optical-module-market)
- [Approved Networks](https://approvednetworks.com/blog/a-look-ahead-2024-optical-transceiver-market-trends/)
- [FS.com](https://www.fs.com/c/100g-qsfp28-sfp-dd-1159)
### 2.4 Shipment Volumes
| Year | 400G+800G Units | 800G Alone | 1.6T |
|------|----------------:|----------:|-----:|
| 2022 | ~5M (est.) | Early | — |
| 2023 | ~8M (est.) | Ramp | — |
| 2024 | >20M | ~10M | ~300K (Q4) |
| 2025 | — | 12-15M (proj) | 2-6M (proj) |
**GPU-to-module ratio:** 1 H100 = 2.5x 800G modules (training); 1 B200 = 2.5x 1.6T modules.
**Sources:**
- [Cignal AI](https://cignal.ai/2025/01/over-20-million-400g-800g-datacom-optical-module-shipments-expected-for-2024/)
- [Deep Fundamental](https://deepfundamental.substack.com/p/deep-dive-optical-module-market)
### 2.5 400G ZR Coherent Timeline (Case Study)
| Milestone | Date | Volume |
|-----------|------|--------|
| OIF 400ZR spec finalized | ~2019-2020 | — |
| First commercial shipments | Late 2021 | >60,000 units |
| OFC 2022 demos / volume ramp | 2022 | ~190,000 units |
| Mass deployment (hyperscale + telco) | 2023-2024 | Bulk of WDM bandwidth |
| 800G ZR GA announced | March 2025 | Next gen arriving |
**Timeline: Spec → first shipment: ~18-24 months. First shipment → volume: ~12 months. Total spec → volume: ~30-36 months.**
**Sources:**
- [FiberMall](https://www.fibermall.com/blog/400g-zr-sell-well-800g-transceiver-standardized.htm)
- [Coherent 800G ZR announcement](https://www.globenewswire.com/news-release/2025/03/28/3051358/11543/en/Coherent-Announces-General-Availability-of-800G-ZR-ZR-QSFP-DD-Transceiver.html)
- [PrecisionOT](https://www.precisionot.com/400gzr_systems_engineering/)
---
## 3. Regional/Country-Level Adoption Differences
### 3.1 Adoption Tier Framework
Based on research findings, optical transceiver adoption follows a tiered geographic pattern:
| Tier | Region | Adoption Lag | Primary Drivers |
|------|--------|-------------|-----------------|
| **Tier 1** | US Hyperscalers (Google, Meta, Amazon, MS) | Reference (0 months) | AI training, scale-out DC |
| **Tier 1B** | Chinese Hyperscalers (Alibaba, Tencent, ByteDance) | 6-12 months | Domestic manufacturing, export controls |
| **Tier 2** | Japan/Korea (NTT, SK Telecom) | 12-18 months | Early coherent, methodical qualification |
| **Tier 3** | European Telcos (DT, Orange, Telefonica) | 24-36 months | Regulatory, longer procurement cycles |
| **Tier 4** | India/SEA/LATAM | 36-60 months | Infrastructure buildout, cost sensitivity |
### 3.2 US Hyperscalers (Tier 1)
- **Lead adopters** for every generation — first to deploy at scale.
- Google's hyperscale DCs have deployed optical circuit switching at massive scale.
- NVIDIA/Meta/Google driving LPO adoption: >40% of short-reach 800G links by late 2025.
- NVIDIA's bulk 800G LinkX price: ~$1,000/transceiver at 100K+ volumes.
- 92% of 2025 hyperscale DC contracts specify OSFP-XD for 1.6T.
**Source:** [Hector Weyl blog](https://www.hectorweyl.com/blogs/blog/the-ai-driven-revolution-in-optical-networking-powering-the-next-era-of-high-speed-energy-efficient-connectivity)
### 3.3 Chinese Market (Tier 1B)
- **Manufacturing dominance:** Chinese manufacturers (Innolight, Eoptolink, Accelink) hold ~60% of merchant 800G market share.
- Innolight: ~40% global 800G share; >50% of NVIDIA procurement.
- Eoptolink: ~20% of NVIDIA's 800G LPO orders.
- **Critical vulnerability:** Chinese vendors remain dependent on US silicon — 5nm/3nm DSPs sourced almost exclusively from Broadcom and Marvell.
- Current export restrictions target compute chips, NOT networking signal processors — but this could change.
- Tencent was first deployer of Broadcom Humboldt CPO (2021).
- Accelink upgraded 1.6T OSFP224 at OFC 2025; Eoptolink launched Gen2 1.6T at OFC 2025.
- Asia-Pacific holds 30% of optical interconnect market share (fastest-growing region).
**Source:** [Substack - Pluggables, Power, and Geopolitics](https://iamfabian.substack.com/p/pluggables-power-and-geopolitics)
### 3.4 Europe (Tier 3)
- European presence focuses on **equipment vendors** (Ciena, Nokia) rather than hyperscale deployments.
- Ciena active in hyper-rail photonics, 1600ZR/ZR+ pluggables (acquired Nubis Communications).
- European telcos typically 2-3 years behind hyperscalers in adopting new transceiver generations.
- Regulatory and procurement cycle overhead extends adoption timelines.
### 3.5 Bass Model with Geographic Heterogeneity
Academic research confirms that Bass model parameters vary significantly across countries:
**Key findings:**
- Multi-country diffusion modeling helps overcome the "data hunger" problem — use earlier-adopting countries' data to predict later-adopting ones.
- BRIC mobile adoption study: India's `q` value was much higher than other BRIC countries.
- European broadband study: Bass model parameters for OECD countries showed peak adoption has already passed.
- 3G mobile across 35 countries: NLMIXED approach with pooled multi-country data.
**Recommended approach for TIP:**
```
For each region r:
F_r(t) = Bass(p_r, q_r, m_r, t - lag_r)
Where lag_r = geographic adoption lag (months):
US Hyperscaler: lag = 0
China Hyperscaler: lag = 6-12
Japan/Korea: lag = 12-18
Europe Telco: lag = 24-36
India/SEA/LATAM: lag = 36-60
And p_r, q_r may be adjusted per region:
Hyperscalers: higher p (innovation-driven), lower q
Telcos: lower p, higher q (imitation-driven)
Emerging: lower p, lower q, much higher m (larger potential)
```
**Sources:**
- [ScienceDirect - Heterogeneity in diffusion](https://www.sciencedirect.com/science/article/abs/pii/S0040162514000870)
- [ScienceDirect - Broadband diffusion Europe](https://www.sciencedirect.com/science/article/abs/pii/S004016251100134X)
- [Academia.edu - Bass model BRIC](https://www.academia.edu/11437115/Diffusion_of_mobile_communications_Application_of_bass_diffusion_model_to_BRIC_countries)
- [Tandfonline - Agent-based Bass](https://www.tandfonline.com/doi/full/10.1080/13873954.2024.2350244)
---
## 4. Conference-to-Market Timeline Analysis
### 4.1 Standards Pipeline
The typical pipeline from concept to product:
```
OIF electrical interface → IEEE formal standard → MSA form factor spec → Product GA
Typical timing:
OIF spec → IEEE ratification: 12-18 months
MSA spec → first product samples: 6-12 months
First samples → GA shipping: 6-12 months
GA → volume production: 6-12 months
TOTAL: OIF spec → volume production: 30-48 months
```
### 4.2 Historical Conference-to-Market Timelines
#### 400G ZR
| Event | Date |
|-------|------|
| OIF 400ZR spec finalized | ~2020 |
| First commercial shipments | Q4 2021 |
| OFC 2022 demos / ramp | 2022 |
| Volume deployment | 2022-2023 |
| **Spec → volume: ~24-30 months** | |
#### 800G
| Event | Date |
|-------|------|
| 800G Pluggable MSA founded | Sept 2019 |
| MSA PSM8 spec (first 800G pluggable) | 2020 |
| OSFP 800G spec released | June 2021 |
| First shipments | 2023 |
| Volume production | 2024 |
| **MSA founding → volume: ~5 years; Spec → volume: ~3-4 years** | |
#### 1.6T
| Event | Date |
|-------|------|
| OFC 2025 demos (multiple vendors) | April 2025 |
| OFC 2026 demos (400G/lambda DR4) | March 2026 |
| IEEE 802.3dj 200G/lane expected | Mid 2026 |
| Sampling | Late 2025 |
| Production ramp (projected) | Late 2026 |
| Volume deployment | 2027 |
| **Demo → volume: ~24 months** | |
#### 3.2T
| Event | Date |
|-------|------|
| Coherent demos at OFC 2026 | March 2026 |
| Expected arrival | ~2026-2027 (samples) |
| **LightCounting added 3.2T to forecast** | **July 2024** |
### 4.3 Conference-to-Market Formula for TIP
```
T_volume = T_demo + Pipeline_Lag
Where Pipeline_Lag depends on technology maturity:
Incremental (same platform, higher speed):
Pipeline_Lag = 18-24 months
New platform (new form factor, new SerDes):
Pipeline_Lag = 30-36 months
Paradigm shift (CPO, new physics):
Pipeline_Lag = 48-60 months
```
**Key signals to monitor:**
1. OIF electrical interface spec release → 30-48 months to volume
2. MSA spec release → 24-36 months to volume
3. IEEE standard ratification → 12-24 months to volume (spec often trails products)
4. Multiple vendors demoing at OFC/ECOC → 18-24 months to volume
5. LightCounting adding category to forecast → 24-30 months to volume
**Sources:**
- [LPO MSA](https://www.lpo-msa.org/news/lpo-msa-announces-release-of-specification-for-linear-pluggable-optica)
- [IEEE 802.3](https://en.wikipedia.org/wiki/IEEE_802.3)
- [FS.com MSA intro](https://community.fs.com/article/how-much-do-you-know-about-msa-standard.html)
- [Eoptolink OFC 2026](https://www.prnewswire.com/news-releases/eoptolink-demos-imdd-400g-per-lambda-based-1-6t-dr4-optical-transceiver-solution-at-ofc-2026--302712390.html)
- [EDN - OFC 2025 1.6T innovations](https://www.edn.com/ofc-2025-unveils-1-6t-networking-innovations/)
- [Coherent 1.6T at OFC 2025](https://www.globenewswire.com/news-release/2025/04/01/3053470/11543/en/Coherent-Demonstrates-1-6T-Optical-Transceivers-Based-on-200G-VCSELs.html)
---
## 5. Switch/Router Refresh Cycles
### 5.1 Broadcom Tomahawk ASIC Timeline (Sets Industry Cadence)
| Gen | Year | Bandwidth | Process | Key Optics |
|-----|------|-----------|---------|------------|
| TH1 | 2014 | 3.2 Tb/s | 28nm | 10G/25G |
| TH2 | 2016 | 6.4 Tb/s | 16nm | 25G/50G |
| TH3 | 2017-18 | 12.8 Tb/s | 16nm | 50G/100G |
| TH4 | 2019-20 | 25.6 Tb/s | 7nm | 100G/400G |
| TH5 | 2022 | 51.2 Tb/s | 5nm | 400G/800G |
| TH6 | 2025 | 102.4 Tb/s | 3nm | 800G/1.6T |
| TH7 | ~2027 | 204.8 Tb/s | (planned) | 1.6T/3.2T |
| TH8 | ~2029 | 409.6 Tb/s | (planned) | 3.2T+ |
**Cadence: Bandwidth doubles every ~2 years.** A single TH5 replaces 48 TH1 switches (95% power reduction).
**CRITICAL:** Pluggable optics consume ~50% of system power and >50% of system cost.
**Sources:**
- [Broadcom TH5](https://investors.broadcom.com/news-releases/news-release-details/broadcom-ships-tomahawk-5-industrys-highest-bandwidth-switch)
- [Broadcom TH6 launch](https://www.broadcom.com/company/news/product-releases/63146)
- [TechInsights - TH5](https://www.techinsights.com/blog/tomahawk-5-switches-512tbps)
- [NextPlatform - TH6 102.4T](https://www.nextplatform.com/2025/06/03/the-ai-datacenter-is-ravenous-for-102-4-tb-sec-ethernet/)
- [ServeTheHome - TH6](https://www.servethehome.com/broadcom-tomahawk-6-launched-for-1-6tbe-generation/)
- [NADDOD - TH6](https://www.naddod.com/blog/broadcom-tomahawk-6-102-4-t-ethernet-switch-chip-for-ai-fabrics)
### 5.2 Cisco Nexus Refresh Cycle
| Platform | Generation | Release | Optics Support |
|----------|-----------|---------|----------------|
| Nexus 9364C | Cloud Scale | ~2018-2019 | 100G/400G |
| Nexus 9364D-GX2A | Current gen | May 2022 | 400G |
| Nexus 9364C-H1 | Updated | April 2024 | 400G |
| Nexus 9364E variants | Next gen | Feb 2025 | 800G |
| Nexus 9364C (EOL) | — | EOS Aug 2023 | Support ends Jan 2029 |
**Refresh cycle: ~2-3 years per platform generation.**
**Source:** [Cisco Nexus 9000 series](https://www.cisco.com/c/en/us/support/switches/nexus-9000-series-switches/series.html)
### 5.3 Arista Refresh Cycle
| Platform | ASIC | Timeline |
|----------|------|----------|
| 7800R3 | Jericho 2 | Prior gen |
| 7800R4 | Jericho 3-AI/3+ | Shipping 2024-2025 |
The 7800R4 supports 1,152x 400G or 576x 800G ports. Existing 7800R3 systems can be upgraded with R4 fabric modules.
**Source:** [Arista 7800R4](https://www.arista.com/en/products/7800r4-series)
### 5.4 NVIDIA Networking
- **Spectrum-X** switches with **ConnectX-7** NICs: current generation for AI clusters.
- ConnectX-8 / Spectrum-4 expected to follow standard ~2-year NVIDIA cadence.
- **Quantum-X800**: 144 ports of 800G CPO (unveiled 2025).
- Each GPU requires **6 pluggable transceivers** consuming 30W each.
- 100K GPU cluster = ~200K transceivers (100K scale-up + 100K scale-out).
- Scaling to 1M GPUs would consume ~180MW in optics alone.
**Source:** [NVIDIA LinkX](https://www.nvidia.com/en-us/networking/interconnect/)
### 5.5 ASIC-to-Transceiver Demand Formula
```
Transceiver_Demand_Surge = f(ASIC_GA + Switch_GA_Lag + Qualification_Lag)
Where:
ASIC_GA: Broadcom ships to OEMs
Switch_GA_Lag: OEM builds switch (+6-12 months)
Qualification_Lag: Customer qualifies transceiver (+3-6 months)
Total: ASIC ship → transceiver demand surge: 9-18 months
Demand magnitude:
Per TH5 switch: 64x 800G transceivers = 64 modules
Per TH6 switch: 64x 1.6T or 128x 800G transceivers
```
---
## 6. Predictive Models for Future Products
### 6.1 3.2T Transceivers
**Signals to watch:**
- Coherent demoed 3.2T pluggable technologies at OFC 2026
- LightCounting added 3.2T to forecasts in July 2024
- IEEE 802.3 expected to start 400G/lane standardization work post-802.3dj
- Broadcom TH7 (204.8T) roadmapped for ~2027
**Predicted timeline:**
- Samples: 2027
- GA: 2028
- Volume: 2029
### 6.2 CPO (Co-Packaged Optics)
**Market forecasts:**
| Source | 2025 | 2026 | 2030+ |
|--------|-----:|-----:|------:|
| Precedence Research | $95M | $124M | $1,055M (2034) |
| Mordor Intelligence | $121M | $165M | $764M (2031) |
| IDTechEx | — | — | $20B+ (2036) |
| LightCounting | — | — | LPO+CPO >$10B (2026) |
**Key milestones:**
- Broadcom Humboldt (1st gen CPO): Jan 2021 (Tencent deployed)
- Broadcom Bailly (TH5 CPO, 51.2T): 2024 — 50K+ shipped in 2025
- Broadcom Davisson (TH6 CPO, 102.4T): 2025 announced
- NVIDIA Quantum-X800: 144x 800G CPO, shipping H2 2025
- IEEE 802.3 CPO at 800G/1.6T ratification: expected late 2027
- **Large-scale CPO deployments: 2028-2030** (Yole Group)
**Impact on pluggable revenue:**
- Pluggables remain majority of DC optical links through the decade (LightCounting).
- CPO captures scale-up (GPU-to-GPU) first; pluggables retain scale-out (DC-to-DC).
- CPO for scale-up is the "killer application."
**Sources:**
- [Precedence Research](https://www.precedenceresearch.com/co-packaged-optics-market)
- [IDTechEx](https://www.idtechex.com/en/research-report/co-packaged-optics-cpo/1138)
- [EDN - CPO in 2026](https://www.edn.com/where-co-packaged-optics-cpo-technology-stands-in-2026/)
- [Lightwaveonline](https://www.lightwaveonline.com/home/article/55265639/ai-fuels-optical-transceiver-and-lpo-cpo-demand)
- [Broadcom CPO](https://investors.broadcom.com/news-releases/news-release-details/broadcom-delivers-industrys-first-512-tbps-co-packaged-optics)
### 6.3 LPO (Linear Pluggable Optics)
**Adoption timeline:**
- 2024: ~few hundred 800G LPO units (NVIDIA primary customer)
- 2025: 1-2M units; >40% of short-reach 800G links in AI DCs by late 2025
- 2027: >8M 1.6T LPO ports expected
- LPO MSA 100G/lane spec finalized: March 2025
- CAGR >35% through 2033
**Power advantage:** 1.6T LPO = ~10W vs. conventional 1.6T = 30W+
**Source:**
- [LPO MSA](https://www.lpo-msa.org/news/lpo-msa-announces-release-of-specification-for-linear-pluggable-optica)
- [Gigalight - LPO & CPO](https://www.gigalight.com/news-events/insights-8540.html)
### 6.4 Silicon Photonics vs. InP Market Share Evolution
| Year | SiPh Share | InP/GaAs Share |
|------|----------:|---------------:|
| 2022 | 24% | 76% |
| 2025 | 30% | 70% |
| 2028 | 44% (projected) | 56% |
| 2030 | 60% (projected) | 40% |
**Driver:** LPO and CPO designs overwhelmingly use SiPh platforms. All LPO/CPO devices (except VCSELs) will be SiPh-based.
**InP retains strategic importance** for: coherent transceivers, high-performance lasers, and vertical integration (Coherent, Lumentum).
**Source:**
- [LightCounting SiPh report](https://www.lightcounting.com/newsletter/en/may-2025-silicon-photonics-linear-drive-pluggable-and-cpo-updated-november-2025-334)
- [EE Times](https://www.eetimes.com/ai-demand-reshapes-optical-connectivity-and-photonics-roadmaps/)
---
## 7. Recommended Implementation for TIP
### 7.1 Core Model: Multi-Generation Norton-Bass with Price Erosion
```typescript
interface TransceiverGeneration {
name: string; // e.g., "100G QSFP28"
speed_gbps: number; // 100, 400, 800, 1600
launch_year: number; // datacom first commercial ship
market_potential_m: number; // total addressable units (millions)
p: number; // innovation coefficient (0.01-0.03)
q: number; // imitation coefficient (0.2-0.4)
asp_launch: number; // ASP at launch ($)
price_decay_lambda: number; // exponential decay rate
form_factor: string; // SFP+, QSFP28, QSFP-DD, OSFP, OSFP-XD
}
// Revenue model for generation i at time t
function generationRevenue(gen: TransceiverGeneration, t: number, nextGen?: TransceiverGeneration): number {
const F_t = bassCumulativeAdoption(gen.p, gen.q, t - gen.launch_year);
// Cannibalization by next generation
let cannibalization = 0;
if (nextGen && t >= nextGen.launch_year) {
const F_next = bassCumulativeAdoption(nextGen.p, nextGen.q, t - nextGen.launch_year);
cannibalization = F_next;
}
const units_in_use = gen.market_potential_m * F_t * (1 - cannibalization);
const asp = gen.asp_launch * Math.exp(-gen.price_decay_lambda * (t - gen.launch_year));
return units_in_use * asp;
}
// Bass cumulative adoption
function bassCumulativeAdoption(p: number, q: number, t: number): number {
if (t < 0) return 0;
return (1 - Math.exp(-(p + q) * t)) / (1 + (q / p) * Math.exp(-(p + q) * t));
}
```
### 7.2 Calibrated Parameters for Known Generations
| Generation | m (M units) | p | q | ASP₀ ($) | λ (decay/yr) | Launch |
|-----------|----------:|----:|----:|--------:|----------:|------:|
| 10G SFP+ | 500 | 0.015 | 0.30 | 500 | 0.25 | 2008 |
| 40G QSFP+ | 100 | 0.010 | 0.25 | 800 | 0.30 | 2012 |
| 100G QSFP28 | 400 | 0.020 | 0.35 | 2000 | 0.38 | 2015 |
| 400G QSFP-DD | 300 | 0.025 | 0.35 | 1500 | 0.35 | 2019 |
| 800G OSFP | 250 | 0.030 | 0.40 | 700 | 0.30 | 2024 |
| 1.6T OSFP-XD | 200 | 0.035 | 0.40 | 2000 | 0.35 | 2026 |
*Note: These are initial estimates to be calibrated against LightCounting/Cignal AI data. Parameters should be fitted using nonlinear least squares on observed shipment data.*
### 7.3 Geographic Revenue Multiplier
```typescript
interface RegionConfig {
name: string;
adoption_lag_months: number;
market_share_pct: number;
p_multiplier: number; // adjust innovation coefficient
q_multiplier: number; // adjust imitation coefficient
}
const REGIONS: RegionConfig[] = [
{ name: "US Hyperscaler", adoption_lag_months: 0, market_share_pct: 35, p_multiplier: 1.5, q_multiplier: 0.8 },
{ name: "China Hyperscaler", adoption_lag_months: 9, market_share_pct: 25, p_multiplier: 1.2, q_multiplier: 1.0 },
{ name: "Japan/Korea", adoption_lag_months: 15, market_share_pct: 10, p_multiplier: 1.0, q_multiplier: 1.1 },
{ name: "Europe Telco", adoption_lag_months: 30, market_share_pct: 15, p_multiplier: 0.7, q_multiplier: 1.2 },
{ name: "India/SEA/LATAM", adoption_lag_months: 48, market_share_pct: 15, p_multiplier: 0.5, q_multiplier: 0.6 },
];
```
### 7.4 Conference Signal Pipeline Tracker
```typescript
interface TechnologySignal {
technology: string;
signal_type: "OIF_SPEC" | "IEEE_STANDARD" | "MSA_SPEC" | "OFC_DEMO" | "ECOC_DEMO" | "LC_FORECAST_ADD" | "FIRST_SHIP" | "VOLUME";
date: Date;
predicted_volume_date: Date; // computed
confidence: number; // 0-1
}
// Pipeline lag by signal type (months to volume production)
const SIGNAL_TO_VOLUME_LAG: Record<string, number> = {
"OIF_SPEC": 36, // 30-42 months
"IEEE_STANDARD": 18, // 12-24 months
"MSA_SPEC": 30, // 24-36 months
"OFC_DEMO": 21, // 18-24 months (multiple vendor demos)
"ECOC_DEMO": 24, // 18-30 months
"LC_FORECAST_ADD": 27, // 24-30 months
"FIRST_SHIP": 12, // 9-15 months
};
```
### 7.5 ASIC Demand Correlation Model
```
Transceiver_Revenue(t) = Σ [Switch_Shipments(ASIC_gen, t - lag) * Ports_Per_Switch * ASP(speed, t)]
Where:
ASIC generations: TH4→TH5→TH6→TH7
lag = 9-18 months (ASIC ship → transceiver surge)
Ports_Per_Switch: 64 (TH5), 64-128 (TH6)
Monitor: Broadcom ASIC announcements as leading indicator
→ OEM switch GA as confirming signal
→ Transceiver qualification as demand signal
```
### 7.6 Key Metrics Dashboard for TIP
For each transceiver generation, TIP should compute and display:
1. **Lifecycle Stage:** {Pre-launch | Ramp | Growth | Peak | Decline | EOL}
2. **Time to Peak Revenue:** Derived from Norton-Bass fit
3. **Current ASP vs. Launch ASP:** Price erosion percentage
4. **Revenue Duration >50% Peak:** How many quarters remaining above half-peak
5. **Cannibalization Index:** What % of market potential is being captured by next gen
6. **Geographic Heatmap:** Adoption stage by region
7. **Leading Indicators:** Conference demos, spec milestones, ASIC launches
### 7.7 Data Sources for Calibration
| Source | Data Type | Access | Cost |
|--------|-----------|--------|------|
| LightCounting | Revenue, shipments, ASP by speed | Subscription | $$$ |
| Cignal AI | Datacom revenue, component market | Subscription | $$$ |
| Dell'Oro | Ethernet switch/router market | Subscription | $$$ |
| Yole Group | SiPh, CPO market forecasts | Reports | $$ |
| IDTechEx | CPO market forecasts | Reports | $$ |
| Broadcom press releases | ASIC launch dates | Free | $0 |
| OFC/ECOC proceedings | Demo tracking | Conference fee | $ |
| IEEE 802.3 minutes | Standards timeline | Free | $0 |
| Company earnings calls | Revenue by segment, guidance | Free (SEC filings) | $0 |
| Innolight/Coherent 10-K | Supplier revenue, growth rates | Free (SEC/CSRC) | $0 |
---
## Appendix A: Key Reference Papers
1. Bass, F.M. (1969). "A New Product Growth for Model Consumer Durables." Management Science.
2. Norton, J.A. & Bass, F.M. (1987). "A Diffusion Theory Model of Adoption and Substitution for Successive Generations of High-Technology Products." Management Science, 33(9).
3. Jiang, Z. & Jain, D.C. (2012). "A Generalized Norton-Bass Model for Multigeneration Diffusion." Management Science, 58(10), 1887-1897.
4. Meade, N. & Islam, T. (2006). "Modelling and forecasting the diffusion of innovation - A 25-year review." International Journal of Forecasting.
5. Tsai, B.H. (2013). "Predicting semiconductor industry growth." Technological Forecasting and Social Change. (Gompertz curve application)
6. Jaafari, A. (2019). "Using Weibull Distribution for Modeling Bimodal Diffusion Curves." Int. J. Innovation and Technology Management.
## Appendix B: All Sources Used
- [Bass diffusion model - Wikipedia](https://en.wikipedia.org/wiki/Bass_diffusion_model)
- [IEEE Xplore - Technology forecasting using Bass model](https://ieeexplore.ieee.org/document/5339534/)
- [GNB Model - INSEAD](https://sites.insead.edu/facultyresearch/research/doc.cfm?did=49784)
- [GNB Model - INFORMS](https://pubsonline.informs.org/doi/pdf/10.1287/mnsc.1120.1529)
- [GNB Model - SSRN](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3112796)
- [GNB Model - Iowa State](https://dr.lib.iastate.edu/article/scm_pubs/1026)
- [R diffusion package](https://rdrr.io/cran/diffusion/man/Nortonbass.html)
- [Heterogeneity in diffusion - ScienceDirect](https://www.sciencedirect.com/science/article/abs/pii/S0040162514000870)
- [Bass model broadband Europe - ScienceDirect](https://www.sciencedirect.com/science/article/abs/pii/S004016251100134X)
- [Bass model BRIC - Academia.edu](https://www.academia.edu/11437115/Diffusion_of_mobile_communications_Application_of_bass_diffusion_model_to_BRIC_countries)
- [Agent-based Bass - Tandfonline](https://www.tandfonline.com/doi/full/10.1080/13873954.2024.2350244)
- [Gompertz for semiconductors - EE Times](https://www.eetimes.com/predicting-semiconductor-industry-growth-drop-the-crystal-ball-and-use-the-gompertz-curve/)
- [Gompertz for semiconductors - Semiengineering](https://semiengineering.com/mathematic-model-helps-predict-markets-that-will-drive-semiconductor-growth/)
- [Weibull for bimodal PLC - World Scientific](https://www.worldscientific.com/doi/10.1142/S0219877019500500)
- [Weibull for tech change - ScienceDirect](https://www.sciencedirect.com/science/article/abs/pii/0040162580900268)
- [MarketsandMarkets - Optical Transceiver](https://www.marketsandmarkets.com/Market-Reports/optical-transceiver-market-161339599.html)
- [Cignal AI - 800G shipments 2025](https://cignal.ai/2025/05/800gbe-optics-shipments-to-grow-60-in-2025/)
- [Cignal AI - 20M 400G/800G 2024](https://cignal.ai/2025/01/over-20-million-400g-800g-datacom-optical-module-shipments-expected-for-2024/)
- [LightCounting - Sales of 800G](https://www.lightcounting.com/newsletter/en/june-2025-quarterly-market-update-332)
- [LightCounting - $23B in 2025](https://www.lightcounting.com/newsletter/en/december-2025-quarterly-market-update-322)
- [LightCounting - Ethernet optics 2024](https://www.lightcounting.com/newsletter/en/september-2024-ethernet-optics-296)
- [LightCounting - Market forecast](https://www.lightcounting.com/newsletter/en/april-2024-market-forecast-289)
- [Coherent - 800G ZR GA](https://www.globenewswire.com/news-release/2025/03/28/3051358/11543/en/Coherent-Announces-General-Availability-of-800G-ZR-ZR-QSFP-DD-Transceiver.html)
- [Coherent - 1.6T VCSELs](https://www.globenewswire.com/news-release/2025/04/01/3053470/11543/en/Coherent-Demonstrates-1-6T-Optical-Transceivers-Based-on-200G-VCSELs.html)
- [Coherent - 3.2T at OFC 2026](https://www.stocktitan.net/news/COHR/coherent-demonstrates-technologies-for-next-generation-pluggable-02zn8msgvh1f.html)
- [Eoptolink - 1.6T DR4 OFC 2026](https://www.prnewswire.com/news-releases/eoptolink-demos-imdd-400g-per-lambda-based-1-6t-dr4-optical-transceiver-solution-at-ofc-2026--302712390.html)
- [PrecisionOT - 400G ZR](https://www.precisionot.com/400gzr_systems_engineering/)
- [Deep Fundamental - Module Market](https://deepfundamental.substack.com/p/deep-dive-optical-module-market)
- [Pluggables Power Geopolitics - Substack](https://iamfabian.substack.com/p/pluggables-power-and-geopolitics)
- [Broadcom TH5](https://investors.broadcom.com/news-releases/news-release-details/broadcom-ships-tomahawk-5-industrys-highest-bandwidth-switch)
- [Broadcom TH6](https://www.broadcom.com/company/news/product-releases/63146)
- [Broadcom TH4](https://investors.broadcom.com/news-releases/news-release-details/broadcom-ships-tomahawk-4-industrys-highest-bandwidth-ethernet)
- [Broadcom CPO](https://investors.broadcom.com/news-releases/news-release-details/broadcom-delivers-industrys-first-512-tbps-co-packaged-optics)
- [TechInsights - TH5](https://www.techinsights.com/blog/tomahawk-5-switches-512tbps)
- [NextPlatform - TH6](https://www.nextplatform.com/2025/06/03/the-ai-datacenter-is-ravenous-for-102-4-tb-sec-ethernet/)
- [NextPlatform - CPO](https://www.nextplatform.com/2025/10/17/the-third-time-will-be-the-charm-for-broadcom-switch-co-packaged-optics/)
- [ServeTheHome - TH6](https://www.servethehome.com/broadcom-tomahawk-6-launched-for-1-6tbe-generation/)
- [Arista 7800R4](https://www.arista.com/en/products/7800r4-series)
- [Cisco Nexus 9000](https://www.cisco.com/c/en/us/support/switches/nexus-9000-series-switches/series.html)
- [NVIDIA LinkX](https://www.nvidia.com/en-us/networking/interconnect/)
- [Precedence Research - CPO](https://www.precedenceresearch.com/co-packaged-optics-market)
- [IDTechEx - CPO](https://www.idtechex.com/en/research-report/co-packaged-optics-cpo/1138)
- [EDN - CPO 2026](https://www.edn.com/where-co-packaged-optics-cpo-technology-stands-in-2026/)
- [Lightwaveonline - LPO CPO](https://www.lightwaveonline.com/home/article/55265639/ai-fuels-optical-transceiver-and-lpo-cpo-demand)
- [LPO MSA](https://www.lpo-msa.org/news/lpo-msa-announces-release-of-specification-for-linear-pluggable-optica)
- [LightCounting - SiPh](https://www.lightcounting.com/newsletter/en/may-2025-silicon-photonics-linear-drive-pluggable-and-cpo-updated-november-2025-334)
- [EE Times - AI reshapes photonics](https://www.eetimes.com/ai-demand-reshapes-optical-connectivity-and-photonics-roadmaps/)
- [Nature Communications - SiPh roadmap](https://www.nature.com/articles/s41467-024-44750-0)
- [AIM Photonics - Commercialization](https://www.aimphotonics.com/news/from-breakthrough-to-market-enabling-the-commercialization-of-photonic-technologies)
- [IEEE 802.3 - Wikipedia](https://en.wikipedia.org/wiki/IEEE_802.3)
- [FS.com - MSA standards](https://community.fs.com/article/how-much-do-you-know-about-msa-standard.html)
- [Hector Weyl - AI optical networking](https://www.hectorweyl.com/blogs/blog/the-ai-driven-revolution-in-optical-networking-powering-the-next-era-of-high-speed-energy-efficient-connectivity)

View File

@ -0,0 +1,665 @@
# Standards-to-Market Timeline Database: Optical Transceivers
**Date:** 2026-03-28
**For:** Transceiver Intelligence Platform (TIP) — Hype Cycle Engine & Predictive Model
**Sources:** IEEE archives, OFC/ECOC proceedings, LightCounting, Cignal AI, Dell'Oro Group, Gazettabyte, vendor press releases, SemiAnalysis
---
## Table of Contents
1. [IEEE Standard Ratification Dates](#1-ieee-standard-ratification-dates)
2. [Standard → First Product → Mainstream Timeline per Generation](#2-generation-timelines)
3. [Price Decline to Mainstream Levels](#3-price-decline-curves)
4. [OFC/ECOC Demo → Product → Mainstream Pipeline](#4-conference-pipeline)
5. [ASIC/SerDes Availability as Leading Indicators](#5-asic-leading-indicators)
6. [Broadcom, Marvell, Intel ASIC Roadmaps](#6-asic-roadmaps)
7. [Current Status: 800G and 1.6T](#7-current-status)
8. [Consolidated Timeline Database](#8-timeline-database)
9. [Prediction Methodology](#9-prediction-methodology)
---
## 1. IEEE Standard Ratification Dates {#1-ieee-standard-ratification-dates}
### Core Ethernet Physical Layer Standards
| Standard | Speed | Study Group | Task Force | Ratified | Key PHY Types |
|----------|-------|-------------|------------|----------|---------------|
| **802.3z** | 1 Gbps | — | — | **Jun 1998** | 1000BASE-SX, 1000BASE-LX |
| **802.3ae** | 10 Gbps | Nov 1999 | Mar 2000 | **Jun 2002** | 10GBASE-SR, -LR, -ER |
| **802.3ba** | 40/100 Gbps | Nov 2007 | Dec 2008 | **Jun 2010** | 40GBASE-SR4/LR4, 100GBASE-SR10/LR4 |
| **802.3bm** | 40/100 Gbps | — | — | **Feb 2015** | 100GBASE-SR4 (improved MMF) |
| **802.3by** | 25 Gbps | — | — | **Jun 2016** | 25GBASE-SR, 25GBASE-LR |
| **802.3bs** | 200/400 Gbps | Nov 2013 | May 2014 | **Dec 2017** | 200GBASE-DR4, 400GBASE-DR4/FR8/LR8 |
| **802.3cd** | 50/100/200 Gbps | — | — | **Dec 2018** | 50GBASE-SR/FR/LR (single-lane 50G) |
| **802.3ck** | 100/200/400 Gbps | — | — | **Sep 2022** | 100G/lane electrical SerDes |
| **802.3df** | 400/800 Gbps | — | — | **Feb 2024** | 800GBASE-DR8, 400GBASE-DR4-2 |
| **802.3dj** | 200/400/800/1600 Gbps | Nov 2022 | — | **Sep 2026 (target)** | 200G/lane, 1.6TbE (D2.2 WG ballot Sep 2025) |
### OIF Implementation Agreements
| Agreement | Published | Speed | Reach | Significance |
|-----------|-----------|-------|-------|-------------|
| 400ZR | Mar 2020 | 400G | 120km | First pluggable coherent DWDM standard |
| OpenZR+ MSA | May 2020 | 100-400G | 1000+km | Extended coherent reach |
| CEI-112G | 2021 | 112 Gbps/lane | Chip-to-module | Enabled 100G PAM4 interfaces |
| 800ZR | Oct 2024 | 800G | 80-120km | Next-gen pluggable coherent |
| CEI-224G | 2025 (target) | 224 Gbps/lane | Chip-to-module | Enables 200G PAM4 interfaces |
| 1600ZR | 2027+ (projected) | 1.6T | TBD | Future coherent standard |
---
## 2. Standard → First Product → Mainstream Timeline per Generation {#2-generation-timelines}
### 2.1 Complete Generation Timeline Database
#### 1G Ethernet (802.3z)
| Milestone | Date | Lag from Prior |
|-----------|------|---------------|
| IEEE 802.3z ratified | Jun 1998 | — |
| First GBIC modules | 1998-1999 | ~6-12 months |
| SFP MSA published | 2000 | +2 years |
| SFP volume shipments | 2001-2002 | +3-4 years |
| Mainstream enterprise adoption | 2002-2004 | +4-6 years |
| Commodity pricing (<$20) | 2006+ | +8 years |
| **Standard-to-mainstream: ~5 years** | | |
#### 10G Ethernet (802.3ae)
| Milestone | Date | Lag from Prior |
|-----------|------|---------------|
| Study group formed | Nov 1999 | — |
| IEEE 802.3ae ratified | Jun 2002 | +31 months |
| First XENPAK modules ship | 2002-2003 | ~6 months from standard |
| XFP MSA published | 2003-2004 | +12-18 months |
| SFP+ MSA (SFF-8431) published | ~2006 | +4 years from standard |
| First SFP+ volume shipments | 2007-2008 | +5-6 years from standard |
| Mainstream SFP+ adoption | 2009-2010 | +7-8 years from standard |
| Commodity pricing (<$30 for SR) | 2014+ | +12 years from standard |
| **Standard-to-mainstream: ~8 years** (but SFP+ MSA-to-mainstream: ~4 years) | | |
#### 40G Ethernet (802.3ba — 40G portion)
| Milestone | Date | Lag from Prior |
|-----------|------|---------------|
| Study group formed | Nov 2007 | — |
| IEEE 802.3ba ratified | Jun 2010 | +31 months |
| First 40G QSFP+ commercial modules | 2010-2011 | ~6-12 months |
| Volume production begins | 2012-2013 | +2-3 years |
| Mainstream data center adoption | 2013-2015 | +3-5 years |
| Price decline begins (Chinese vendors) | 2015-2016 | +5-6 years |
| **Standard-to-mainstream: ~5 years** | | |
| **Note:** 40G was partially skipped; many went 10G→100G | | |
#### 100G Ethernet (802.3ba — 100G portion, then 802.3bm/QSFP28)
| Milestone | Date | Lag from Prior |
|-----------|------|---------------|
| IEEE 802.3ba ratified (100G) | Jun 2010 | — |
| First CFP 100G modules | 2010-2011 | ~6-12 months |
| QSFP28 MSA published | 2013-2014 | +3-4 years |
| First OFC demos (CWDM4/PSM4 QSFP28) | OFC 2015 | +5 years from standard |
| InnoLight volume QSFP28 shipments | Mar 2017 | +7 years from 802.3ba |
| Market maturity (cost parity with 10G $/Gbps) | 2017-2018 | +7-8 years from 802.3ba |
| Commodity pricing (<$100 SR4) | 2021-2022 | +11-12 years from 802.3ba |
| Ultra-commodity (<$30 from third-party) | 2024-2026 | +14-16 years |
| **QSFP28 MSA-to-mainstream: ~4 years** | | |
#### 200/400G Ethernet (802.3bs)
| Milestone | Date | Lag from Prior |
|-----------|------|---------------|
| Study group formed | Nov 2013 | — |
| IEEE 802.3bs ratified | Dec 2017 | +49 months |
| QSFP-DD MSA Rev 2.0 | Mar 2017 | (preceded standard!) |
| InnoLight 400G OSFP intro at OFC 2017 | Mar 2017 | (preceded standard!) |
| First commercial 400G QSFP-DD/OSFP | 2019-2020 | +2 years from standard |
| Volume production | 2020-2021 | +3-4 years |
| Mainstream DC adoption (>10% ports) | 2021-2022 | +4-5 years |
| Price decline accelerates | 2023-2024 | +6-7 years |
| 400G SR8 prices -50% in one year | End 2023 | +6 years |
| 400G now "mainstream" per Nokia | 2025-2026 | +8 years |
| **Standard-to-mainstream: ~4-5 years** | | |
| **First OFC demo-to-mainstream: ~5 years** | | |
#### 800G Ethernet (802.3ck + 802.3df)
| Milestone | Date | Lag from Prior |
|-----------|------|---------------|
| 802.3ck ratified (100G/lane electrical) | Sep 2022 | Enabler standard |
| Intel first 800G DR8 OSFP sample | OFC 2021 | Pre-standard demo |
| Initial SR8 shipments for AI | 2022 | Pre-802.3df |
| LESSENGERS 800G SR8 volume production | Q4 2023 | Pre-802.3df |
| IEEE 802.3df ratified (800G standard) | Feb 2024 | — |
| Hyper Photonix 800G DR8 GA | May 2024 | +3 months post-standard |
| 800G shipments exceed 1M units | 2023 | Pre-standard |
| Cignal AI: 8M 800GbE modules forecast | 2024 | ~simultaneous with standard |
| 800G surpasses 400G in shipments (first time) | Q4 2023 | Pre-standard |
| 800G mainstream / displacing 400G | 2025 | +1 year post-standard |
| Cignal AI: 12.8M units (60% growth) | 2025 | +1 year |
| **Standard-to-mainstream: ~1 year** (but products shipped pre-standard) | | |
| **First demo-to-mainstream: ~4 years** (OFC 2021 → 2025) | | |
| **KEY INSIGHT:** AI demand pulled 800G deployment ahead of standard ratification | | |
#### 1.6T Ethernet (802.3dj — in progress)
| Milestone | Date | Lag from Prior |
|-----------|------|---------------|
| 802.3dj task force (split from 802.3df) | Nov 2022 | — |
| Eoptolink 1.6T module demo (OSFP-XD) | OFC 2023 | +5 months from TF |
| InnoLight 1.6T OSFP-XD demo | OFC 2024 | +17 months |
| First EML-based 1.6T samples ship | Q4 2024 - Q1 2025 | +25-27 months |
| OFC 2025: Multiple live 1.6T demos | Mar 2025 | +28 months |
| Keysight 224G SerDes interop plugfest | Dec 2025 | +37 months |
| AOI first volume order ($200M+) | Mar 2026 | +40 months |
| OFC 2026: Live multi-vendor 1.6T interop | Mar 2026 | +40 months |
| Broadcom Tomahawk 6 volume (enables 1.6T ports) | Mar 2026 | +40 months |
| IEEE 802.3dj ratification (target) | Sep 2026 | +46 months |
| Dell'Oro: First year of volume 1.6T switch deployment | 2026 | +48 months |
| Volume ramp forecast | H2 2026 | Pre-standard |
| Predicted mainstream (>10% addressable ports) | 2027 | ~+6 months post-standard |
| **PATTERN: Products shipping ~6 months BEFORE standard ratification** | | |
| **First demo-to-volume: ~3 years** (OFC 2023 → H2 2026) | | |
---
## 3. Price Decline to Mainstream Levels {#3-price-decline-curves}
### Price Erosion Model
```
ASP(t) = ASP₀ * exp(-λ*t)
Where:
ASP₀ = launch price
λ = annual price erosion rate
t = years since launch
```
### Historical Price Decline Data
| Generation | Launch ASP | Year 1 | Year 2 | Year 3 | Year 5 | Year 8+ | λ (per year) | Half-life |
|------------|-----------|--------|--------|--------|--------|---------|-------------|-----------|
| **10G SFP+ SR** | ~$500 (2008) | $350 | $200 | $120 | $50 | $15-25 | 0.35-0.40 | ~2 years |
| **40G QSFP+ SR4** | ~$400 (2011) | $300 | $200 | $120 | $50 | $20 | 0.30-0.35 | ~2.2 years |
| **100G QSFP28 SR4** | ~$2,000 (2015) | $1,000 | $500 | $250 | $100 | $30-50 | 0.35-0.40 | ~2 years |
| **400G QSFP-DD DR4** | ~$1,500 (2020) | $800 | $400 | $200 | $150 | — | 0.40-0.45 | ~1.8 years |
| **400G SR8** | ~$600 (2022) | $400 | $200 | — | — | — | 0.50 (aggressive) | ~1.4 years |
| **800G SR8** | ~$800 (2023) | $500 | $300-500 | — | — | — | 0.25-0.30 (early) | ~2.5 years |
| **800G DR8** | ~$2,000 (2024) | $800-1,200 | $500-800 | — | — | — | 0.35 (projected) | ~2 years |
| **1.6T DR8** | ~$2,500 (2025) | $1,500 | — | — | — | — | 0.40 (projected) | ~1.8 years |
### Price Milestone Definitions
| Level | Definition | Typical Timing |
|-------|-----------|---------------|
| **Launch premium** | First 12 months, <5 vendors | ASP |
| **Early volume** | 5-15 vendors, hyperscale deployment | ASP₀ * 0.4-0.6 (Year 2-3) |
| **Mainstream** | 15-30 vendors, enterprise deployment | ASP₀ * 0.1-0.2 (Year 4-6) |
| **Commodity** | 30+ vendors, third-party compatible | ASP₀ * 0.02-0.05 (Year 7+) |
### Key Price Observations (2025-2026)
| Module | Current ASP (2025-2026) | Status |
|--------|------------------------|--------|
| 100G QSFP28 SR4 | $29-$99 | Ultra-commodity |
| 400G DR4 | $150-$250 | Late mainstream, declining |
| 400G SR8 | <$200 | Commodity (50% decline in 2023) |
| 800G SR8 | $300-$500 | Early mainstream |
| 800G DR8 | $500-$800 | Mainstream ramp |
| 800G 2xFR4 | $600-$900 | Premium |
| 800G ZR/ZR+ | $4,000-$6,000 | Early premium |
| 1.6T DR8 | $1,500-$2,500 | Launch premium |
| 400G ZR | $2,000-$3,000 | Mature premium |
### Cost-per-Gbps Trend
| Year | Best $/Gbps (short-reach datacom) | Generation |
|------|-----------------------------------|-----------|
| 2015 | $20/Gbps | 100G QSFP28 launch |
| 2018 | $2-4/Gbps | 100G mainstream |
| 2020 | $3-4/Gbps | 400G launch |
| 2022 | $0.50-1.00/Gbps | 400G mainstream (SiPh) |
| 2024 | $0.50/Gbps | 400G SiPh commodity |
| 2025 | $0.40-0.60/Gbps | 800G early mainstream |
| 2026 (proj.) | $0.30-0.50/Gbps | 800G mainstream |
| 2027 (proj.) | $1.00-1.50/Gbps → declining | 1.6T early volume |
---
## 4. OFC/ECOC Demo → Product → Mainstream Pipeline {#4-conference-pipeline}
### Historical Conference-to-Market Timelines
| Technology | First OFC/ECOC Demo | First Commercial Product | Volume Production | Mainstream Adoption | Demo→Volume | Demo→Mainstream |
|-----------|---------------------|-------------------------|-------------------|---------------------|-------------|-----------------|
| 10G SFP+ | OFC 2006 | 2007-2008 | 2008-2009 | 2009-2010 | 2-3 years | 3-4 years |
| 40G QSFP+ | OFC 2009 | 2010-2011 | 2012-2013 | 2013-2015 | 3-4 years | 4-6 years |
| 100G QSFP28 | OFC 2015 | 2016 | 2017 | 2017-2018 | 2 years | 2-3 years |
| 100G CFP-DCO | OFC 2010 | 2011 | 2012 | 2013-2014 | 2 years | 3-4 years |
| 400G QSFP-DD | OFC 2017 | 2019-2020 | 2020-2021 | 2021-2022 | 3-4 years | 4-5 years |
| 400G ZR | OFC 2019 | H2 2020 | 2021-2022 | 2022-2023 | 2-3 years | 3-4 years |
| 800G DR8 | OFC 2021 | 2022-2023 | 2023-2024 | 2025 | 2-3 years | ~4 years |
| 800G ZR/ZR+ | ECOC 2023 | Q1 2024 (alpha) | 2025 (GA) | 2026 (projected) | 2-3 years | ~3 years |
| 1.6T OSFP-XD | OFC 2023 | Q4 2024 | H2 2026 (projected) | 2027 (projected) | 3 years | ~4 years |
| CPO | OFC 2021 | 2023 (select) | 2027 (projected) | 2029+ (projected) | 6+ years | 8+ years |
### Observed Trend: Acceleration
| Era | Average Demo→Mainstream | Driver |
|-----|------------------------|--------|
| Pre-cloud (2002-2010) | 5-8 years | Enterprise procurement cycles |
| Cloud era (2010-2020) | 3-5 years | Hyperscale demand, Chinese manufacturing |
| AI era (2020-2026) | 2-4 years | NVIDIA demand pull, pre-ordering, LPO |
### OFC/ECOC Signal Taxonomy
| Conference Signal | Meaning | Timeline Implication |
|------------------|---------|---------------------|
| Paper-only presentation | Early research | 3-5 years to product |
| Live demo (single vendor) | Working prototype | 2-3 years to volume |
| Multi-vendor interop demo | Ecosystem ready | 12-18 months to volume |
| Plugfest results announced | Qualification stage | 6-12 months to volume |
| Volume shipping announcement | Production | Already available |
---
## 5. ASIC/SerDes Availability as Leading Indicators {#5-asic-leading-indicators}
### The ASIC Dependency Chain
```
SerDes IP → DSP ASIC tape-out → DSP sampling → Module design-in →
Module qualification → Switch ASIC GA → Switch platform GA →
Transceiver demand ramp → Volume deployment
```
### SerDes Generation Timeline
| SerDes Rate | OIF Spec | First Silicon | Volume Availability | Enabled Speeds |
|-------------|----------|---------------|--------------------:|----------------|
| 25G NRZ | CEI-25G (~2010) | 2011-2012 | 2013-2014 | 100G (4x25G) |
| 56G PAM4 | CEI-56G (~2015) | 2016-2017 | 2018-2019 | 200G (4x50G), 400G (8x50G) |
| 112G PAM4 | CEI-112G (2021) | 2020-2021 | 2022-2023 | 400G (4x100G), 800G (8x100G) |
| 224G PAM4 | CEI-224G (2025 target) | 2024 (sampling) | 2025-2026 | 800G (4x200G), 1.6T (8x200G) |
| 448G PAM4 | TBD (~2028) | ~2027 (projected) | ~2029 (projected) | 1.6T (4x400G), 3.2T (8x400G) |
### ASIC-to-Transceiver Lag (Empirical)
| Transition | Typical Lag | Range | Evidence |
|-----------|-------------|-------|----------|
| **Switch ASIC announcement → First switch GA** | 9-18 months | 6-24 months | Broadcom TH series history |
| **Switch GA → Transceiver demand ramp** | 6-12 months | 3-18 months | Qualification + deployment |
| **DSP ASIC sampling → Module qualification** | 6-9 months | 3-12 months | Design-in cycle |
| **DSP ASIC GA → Module volume production** | 3-6 months | 1-9 months | Shortening with pre-qualification |
| **Complete: ASIC tape-out → Transceiver ecosystem ramp** | 18-30 months | 12-36 months | Combined pipeline |
### The "ASIC Gate" — No Transceiver Ramps Without Switch Support
| Transceiver Speed | Required Switch ASIC | ASIC GA | Transceiver Volume Ramp |
|-------------------|---------------------|---------|------------------------|
| 100G QSFP28 | Broadcom TH1 (3.2T, 32x100G) | Spring 2015 | 2016-2017 |
| 400G QSFP-DD | Broadcom TH3 (12.8T, 32x400G) | Dec 2017 | 2019-2020 |
| 800G OSFP | Broadcom TH5 (51.2T, 64x800G) | Late 2022 | 2023-2024 |
| 1.6T OSFP-XD | Broadcom TH6 (102.4T, 64x1.6T) | Mar 2026 | H2 2026 (projected) |
| 3.2T (future) | TH7 (projected ~204.8T) | ~2028 | ~2029-2030 |
---
## 6. Broadcom, Marvell, Intel ASIC Roadmaps {#6-asic-roadmaps}
### 6.1 Broadcom Switch ASICs (Tomahawk Series)
| ASIC | Bandwidth | Process | Announced | Switch GA | SerDes | Optical Ports |
|------|-----------|---------|-----------|-----------|--------|---------------|
| TH1 | 3.2 Tbps | 28nm | Sep 2014 | Spring 2015 | 25G NRZ | 32x100G |
| TH2 | 6.4 Tbps | 16nm | Oct 2016 | Fall 2017 | 25G NRZ | 64x100G |
| TH3 | 12.8 Tbps | 16nm | Dec 2017 | Dec 2017 | 50G PAM4 | 32x400G |
| TH4 | 25.6 Tbps | 7nm | Dec 2019 | 2020-2021 | 50G PAM4 | 64x400G |
| TH5 | 51.2 Tbps | 5nm | Aug 2022 | Late 2022 | 112G PAM4 | 64x800G |
| TH-Ultra | 51.2 Tbps | 4nm | 2024 | 2024 | 112G PAM4 | 64x800G (AI-optimized) |
| **TH6** | **102.4 Tbps** | **3nm** | **Jun 2025** | **Mar 2026** | **224G PAM4** | **64x1.6T** |
| TH6 Davisson (CPO) | 102.4 Tbps | 3nm | Oct 2025 | Oct 2025 | 224G PAM4 | CPO integrated |
**Cadence:** Bandwidth doubles every ~2 years. Announcement-to-GA: 6-18 months.
### 6.2 Broadcom Optical DSP Roadmap (Sian Family)
| DSP | Process | Speed | Power (1.6T) | Announced | Status (Mar 2026) |
|-----|---------|-------|-------------|-----------|-------------------|
| **Sian** (BCM85822) | 5nm | 200G/lane optical | ~30W | ECOC 2023 (Oct 2023) | Production |
| **Sian2** | 5nm | 200G/lane elec+optical | ~28W | 2024 | Production |
| **Sian2M** | 5nm | 200G/lane MMF | <25W (SR8) | 2024 | Production |
| **Sian3** | 3nm | 200G/lane SMF | <23W | 2025 | Sampling, production Q3 2025 |
| **Taurus** (BCM83640) | 3nm | **400G/lane** | TBD | **Mar 2026** | Announced (first 400G/lane DSP) |
**Key insight:** Taurus (400G/lane) enables future 1.6T in 4-lane and 3.2T in 8-lane configurations. This is the bridge to the 3.2T generation.
### 6.3 Marvell Optical DSP Roadmap
| DSP | Process | Speed | Status (Mar 2026) | Key Feature |
|-----|---------|-------|--------------------|-------------|
| **Orion** | 7nm | 400G/800G | Production (legacy) | Widely deployed |
| **Nova** (MV-CD432) | 5nm | 1.6T (100G elec/200G opt) | GA (Mar 2024) | First 200G/lane 1.6T DSP |
| **Nova 2** | — | 1.6T (200G elec+optical) | Sampling Q2 2024 | Full 200G/lane end-to-end |
| **Ara** | 3nm | 1.6T / 800G | **Mass volume shipping (2025)** | Industry's first 3nm optical DSP |
| **Ara T** | 3nm | 1.6T (transmit-retimed) | **Announced Mar 2026** | Power-optimized for LRO |
| **Ara X** | 3nm | 1.6T (reliability) | **Announced Mar 2026** | Advanced link reliability |
| **Petra** | 3nm | Gearbox (8x100G→4x200G) | **Announced Mar 2026** | Bridge chip |
| **Aquila M** | 3nm | O-band coherent-lite | **Announced Mar 2026** | Integrated MACsec |
| **Electra** | **2nm** | 1.6T ZR/ZR+ coherent | **Sampling H2 2026** | Industry-first 2nm coherent DSP |
| **Libra** | 2nm | 800G ZR/ZR+ coherent | **Sampling H2 2026** | Next-gen coherent |
**Key insight:** Marvell Ara (3nm) is already in mass volume. Marvell is 6-12 months ahead of Broadcom on 1.6T DSP availability, but Broadcom counters with the Taurus 400G/lane roadmap.
### 6.4 Intel Silicon Photonics
| Product | Speed | Status | Significance |
|---------|-------|--------|-------------|
| Intel SiPh 100G PSM4 | 100G | Production (since ~2016) | Pioneered SiPh transceivers |
| Intel 800G DR8 OSFP (first sample) | 800G | OFC 2021 demo | First 800G DR8 in the industry |
| Intel SiPh engines (sold to Jabil, ATOP) | 100G-1.6T | Active | Platform licensing model |
| Intel Tofino 3 (switching ASIC) | — | **CANCELLED Jan 2023** | Intel exited switching ASICs |
**Key insight:** Intel's role has shifted from integrated products to SiPh engine licensing. Jabil's 1.6T module (OFC 2025) uses Intel SiPh technology.
### 6.5 Other Key ASIC Players
| Company | Products | Role | Status |
|---------|----------|------|--------|
| **Semtech** | GN8234 redriver, GN1834D TIA, GN187N1 driver | Analog components for LPO/FRO | Live demos at OFC 2026 |
| **Synopsys** | 224G SerDes IP | IP licensing to ASIC makers | Leading IP provider |
| **Credo** | HiWire active cables, line card DSPs | Active cable/retimer market | Shipping 112G, developing 224G |
| **MediaTek** | 224G SerDes (for Google TPU v8e) | Custom ASIC SerDes | Broke into Google ecosystem |
| **NVIDIA** | ConnectX-8/9 NICs, NVLink SerDes | Network adapter ASICs | CX-8 (800G) production Q2 2025 |
### 6.6 ASIC Predictive Signal Summary
| Signal | What It Predicts | Lead Time |
|--------|-----------------|-----------|
| SerDes IP announcement | New speed tier feasibility | 3-5 years before volume |
| DSP ASIC tape-out | Module design starts | 18-24 months before volume |
| DSP sampling to module vendors | Module prototypes in 6 months | 12-18 months before volume |
| Switch ASIC GA | Port demand imminent | 6-12 months before transceiver ramp |
| NIC ASIC GA (ConnectX-N) | Server-side demand confirmed | 3-6 months before optics ramp |
| Multi-vendor plugfest success | Ecosystem validated | 6-12 months before mainstream |
---
## 7. Current Status: 800G and 1.6T {#7-current-status}
### 7.1 800G Status (March 2026)
| Metric | Value | Source |
|--------|-------|--------|
| **Phase** | Late Slope of Enlightenment / early Plateau | Hype cycle analysis |
| IEEE standard | 802.3df ratified Feb 2024 | IEEE |
| Units shipped (2024) | ~8-10M | Cignal AI |
| Units forecast (2025) | ~12.8M (+60% YoY) | Cignal AI |
| Units forecast (2026) | ~20M+ | Industry estimates |
| ASP trend | $300-800 depending on reach | Declining |
| Vendor count | 30+ active vendors | Market data |
| Form factors | QSFP-DD800, OSFP | Both mature |
| DSP ecosystem | Broadcom Sian family, Marvell Orion/Ara | Fully available |
| Switch support | TH5, TH-Ultra, Spectrum-4, Silicon One G200 | Multiple platforms |
| 800G ZR/ZR+ units (2026 forecast) | >200K, >$1B revenue | Cignal AI |
| **Assessment: 800G is mainstream for AI backend and rapidly commoditizing for datacom** | | |
### 7.2 1.6T Status (March 2026)
| Metric | Value | Source |
|--------|-------|--------|
| **Phase** | Peak of Inflated Expectations / early Slope | Hype cycle analysis |
| IEEE standard | 802.3dj D2.2 (WG ballot), target Sep 2026 | IEEE |
| Units shipped (2025) | <1M (select NVIDIA/hyperscale) | Industry estimates |
| First volume orders | AOI $200M+ (Mar 2026) | Press release |
| Dell'Oro forecast | First year of volume 1.6T switches in 2026 | Dell'Oro Group |
| Dell'Oro forecast | >5M ports within 1-2 years of first shipments | Dell'Oro Group |
| ASP | $1,500-$2,500 (DR8) | Market data |
| Vendor count | 10-15 with demos/samples | Growing rapidly |
| Form factors | OSFP-XD (16x100G), OSFP1600 (8x200G), QSFP-DD1600 | Gen1 → Gen2 transition |
| DSP ecosystem | Marvell Ara (mass volume), Broadcom Sian2/3, Semtech | Available |
| Switch support | Broadcom TH6 (GA Mar 2026), NVIDIA Spectrum-X | Just becoming available |
| NIC support | NVIDIA ConnectX-8 (production Q2 2025) | Available |
| OFC 2026 demos | Multi-vendor live interop (FRO, LRO, LPO) | Ecosystem validated |
| 224G SerDes plugfest | Dec 2025 at Keysight | Passed |
| **Assessment: 1.6T transitioning from demos to volume. H2 2026 = inflection point.** | | |
### 7.3 Future: 3.2T and Beyond
| Metric | Value |
|--------|-------|
| **Phase** | Technology Trigger / Pre-commercial |
| First demos | Semtech showed 3.2T ACC at OFC 2026 (448G/channel) |
| Standard | No IEEE task force yet; OIF/MSA discussions |
| ASIC dependency | 448G SerDes (~2027-2028), next-gen switch ASIC (~TH7, 2028) |
| Projected first samples | 2027-2028 |
| Projected volume | 2029-2030 |
| Projected mainstream | 2030-2031 |
| CPO relevance | At 3.2T, CPO may capture 15-30% of market |
---
## 8. Consolidated Timeline Database {#8-timeline-database}
### Master Timeline: All Generations
| Gen | Standard | Ratified | First Demo | First Product | Volume | Mainstream | Commodity | Standard→Mainstream | Demo→Mainstream |
|-----|----------|----------|------------|---------------|--------|------------|-----------|--------------------:|----------------:|
| 1G | 802.3z | 1998 | ~1997 | 1998 | 2001 | 2002-2004 | 2006 | **5 yrs** | 6 yrs |
| 10G | 802.3ae | Jun 2002 | OFC 2001 | 2002 (XENPAK) | 2007 (SFP+) | 2009-2010 | 2014 | **8 yrs** | 9 yrs |
| 25G | 802.3by | Jun 2016 | OFC 2015 | 2016 | 2018 | 2019-2020 | 2023 | **4 yrs** | 5 yrs |
| 40G | 802.3ba | Jun 2010 | OFC 2009 | 2010-2011 | 2012-2013 | 2013-2015 | 2017 | **5 yrs** | 6 yrs |
| 100G | 802.3ba/bm | Jun 2010 | OFC 2010 | 2011 (CFP) | 2017 (QSFP28) | 2017-2018 | 2022 | **8 yrs** | 8 yrs |
| 200G | 802.3bs | Dec 2017 | OFC 2018 | 2019 | 2020-2021 | 2020-2021 | — | **3 yrs** | 3 yrs |
| 400G | 802.3bs | Dec 2017 | OFC 2017 | 2019-2020 | 2020-2021 | 2021-2022 | 2025-2026 | **4-5 yrs** | 5 yrs |
| 800G | 802.3df | Feb 2024 | OFC 2021 | 2022 | 2023-2024 | 2025 | — | **1 yr** | 4 yrs |
| 1.6T | 802.3dj | Sep 2026* | OFC 2023 | Q4 2024 | H2 2026* | 2027* | — | **1 yr*** | 4 yrs* |
| 3.2T | TBD | ~2029* | OFC 2026 | ~2028* | ~2029-2030* | ~2030-2031* | — | **~1-2 yrs*** | ~4-5 yrs* |
*Projected values
### Key Finding: Cycle Compression
| Era | Standard→Mainstream | Demo→Mainstream | Primary Driver |
|-----|--------------------:|----------------:|---------------|
| **1998-2010 (Enterprise)** | 5-8 years | 6-9 years | Slow enterprise procurement, single-vendor qualification |
| **2010-2020 (Cloud)** | 3-5 years | 3-5 years | Hyperscale demand, Chinese manufacturing capacity |
| **2020-2026 (AI)** | 1-2 years | 3-4 years | AI demand pull, pre-standard deployment, NVIDIA procurement |
| **Trend** | Converging to ~1 year | Stable at ~4 years | Products now ship before standards ratify |
### The "Pre-Standard Deployment" Phenomenon
Starting with 800G, products began shipping **before** standards were ratified. This is driven by:
1. **MSA specs substitute for IEEE** — QSFP-DD and OSFP MSAs provide sufficient interop specs
2. **Hyperscaler procurement power** — Single-vendor qualification bypasses multi-vendor standard need
3. **AI urgency** — GPU cluster buildout cannot wait for IEEE consensus
4. **SerDes maturity** — OIF CEI specs provide electrical interface standardization independently
This means **IEEE standard ratification is becoming a lagging indicator**, not a leading one. The leading indicators are:
1. Switch ASIC availability (e.g., TH6 GA for 1.6T)
2. DSP ASIC availability (e.g., Marvell Ara mass volume for 1.6T)
3. NIC availability (e.g., ConnectX-8 for 800G)
4. Multi-vendor plugfest success
5. First hyperscaler volume order
---
## 9. Prediction Methodology {#9-prediction-methodology}
### 9.1 The TIP Predictive Timeline Formula
For any new transceiver technology, estimate deployment milestones using:
```
T_volume = max(T_switch_asic_ga, T_dsp_ga, T_plugfest) + OFFSET_volume
T_mainstream = T_volume + OFFSET_mainstream(segment)
T_commodity = T_mainstream + OFFSET_commodity
T_standard = T_volume +/- 6 months (no longer gates deployment)
```
#### Offset Tables
**Volume Offset (from ASIC/ecosystem readiness):**
| Technology Type | OFFSET_volume | Confidence |
|----------------|---------------|------------|
| Incremental speed (same form factor) | 3-6 months | +/- 3 mo |
| New form factor | 6-12 months | +/- 6 mo |
| New modulation scheme | 12-18 months | +/- 9 mo |
| New architecture (CPO) | 18-36 months | +/- 12 mo |
**Mainstream Offset (from volume, by segment):**
| Segment | OFFSET_mainstream | Confidence |
|---------|------------------|------------|
| US hyperscaler | 0-6 months | +/- 3 mo |
| China hyperscaler | 6-12 months | +/- 6 mo |
| Japan/Korea telco | 12-18 months | +/- 6 mo |
| Enterprise (US) | 18-36 months | +/- 12 mo |
| European telco | 24-36 months | +/- 12 mo |
| India/SEA/LATAM | 36-60 months | +/- 18 mo |
**Commodity Offset (from mainstream):**
| Speed Class | OFFSET_commodity | Driver |
|------------|-----------------|--------|
| 100G and below | 3-5 years | Many Chinese vendors, SiPh |
| 400G | 3-4 years | Aggressive price erosion |
| 800G | 3-4 years (projected) | AI volume drives fast commoditization |
| 1.6T | 3-4 years (projected) | Following 800G pattern |
### 9.2 Leading Indicator Scoring System
Score each indicator 0-10 to predict how close a technology is to volume deployment:
| Indicator | Score 0 | Score 5 | Score 10 |
|-----------|---------|---------|----------|
| Switch ASIC | Not announced | Sampling | GA and shipping |
| Optical DSP | Concept only | Sampling to vendors | Mass volume |
| NIC support | No plans | Roadmap announced | Production |
| IEEE standard | No study group | Task force active | Published |
| MSA spec | No spec | Draft published | Rev 3.0+ |
| OFC/ECOC demos | Paper only | Single-vendor demo | Multi-vendor interop |
| Plugfest | None | Planned | Completed successfully |
| Volume orders | None | LOIs/pre-orders | $100M+ orders placed |
| Vendor count | 0-2 | 5-10 | 15+ |
| Price trend | Launch premium | Early decline | Aggressive decline |
**Interpretation:**
- Score 0-25: 3+ years from volume
- Score 25-50: 18-36 months from volume
- Score 50-75: 6-18 months from volume
- Score 75-100: Volume imminent or achieved
### 9.3 Current Scores (March 2026)
| Technology | Switch ASIC | DSP | NIC | IEEE | MSA | Demo | Plugfest | Orders | Vendors | Price | **Total** | **Assessment** |
|-----------|:-----------:|:---:|:---:|:----:|:---:|:----:|:--------:|:------:|:-------:|:-----:|:---------:|:--------------|
| **800G** | 10 | 10 | 10 | 10 | 10 | 10 | 10 | 10 | 10 | 8 | **98** | Mainstream |
| **1.6T** | 9 | 9 | 8 | 7 | 8 | 9 | 9 | 8 | 6 | 3 | **76** | Volume imminent |
| **800G ZR** | 10 | 9 | 10 | 10 | 10 | 10 | 10 | 8 | 5 | 3 | **85** | Early mainstream |
| **1.6T ZR** | 5 | 4 | 5 | 2 | 3 | 3 | 2 | 2 | 2 | 1 | **29** | 2-3 years out |
| **3.2T** | 2 | 2 | 1 | 0 | 1 | 3 | 0 | 0 | 1 | 0 | **10** | 4+ years out |
| **CPO (scale-out)** | 7 | 6 | 5 | 3 | 5 | 7 | 5 | 4 | 4 | 2 | **48** | 2-3 years from volume |
### 9.4 Applying the Model: 1.6T Deployment Prediction
**Inputs (March 2026):**
- Switch ASIC: Broadcom TH6 GA Mar 2026 ✓
- DSP: Marvell Ara mass volume ✓, Broadcom Sian3 production Q3 2025 ✓
- NIC: NVIDIA ConnectX-8 production Q2 2025 ✓
- Multi-vendor plugfest: Dec 2025 at Keysight ✓
- First volume order: AOI $200M+ Mar 2026 ✓
- IEEE 802.3dj: Target Sep 2026 (not yet, but MSAs ready)
**Calculation:**
```
T_switch_asic_ga = Mar 2026
T_dsp_ga = Q1 2025 (Marvell Ara)
T_plugfest = Dec 2025
max(all) = Mar 2026
T_volume = Mar 2026 + 3 months = ~Q3 2026
T_mainstream(US hyperscaler) = Q3 2026 + 3 months = ~Q4 2026 / Q1 2027
T_mainstream(China) = Q3 2026 + 9 months = ~Q2 2027
T_mainstream(Enterprise US) = Q3 2026 + 24 months = ~Q3 2028
T_mainstream(Europe) = Q3 2026 + 30 months = ~Q1 2029
T_commodity = Q1 2027 + 3.5 years = ~H2 2030
```
**Confidence: Medium-High** (all ASIC dependencies met, ecosystem validated, volume orders placed)
### 9.5 Norton-Bass Integration
The timeline database feeds Norton-Bass model parameters:
| Parameter | Derivation | Source Signal |
|-----------|-----------|---------------|
| **tau (introduction time)** | T_volume from formula above | ASIC GA + offset |
| **p (innovation coefficient)** | 0.01-0.03 (typical for B2B tech) | Patent/publication velocity |
| **q (imitation coefficient)** | 0.20-0.40 (varies by segment) | Vendor count growth rate + Google Trends |
| **m (market potential)** | Total addressable ports | Switch ASIC ports × hyperscaler CapEx forecast |
| **Price function P(t)** | ASP₀ * exp(-λ*t) | Historical price erosion rates per generation |
### 9.6 Validation Against Historical Generations
| Generation | Model Predicted Mainstream | Actual Mainstream | Error |
|-----------|--------------------------|-------------------|-------|
| 40G QSFP+ | 2014 (TH1 2015 - 1yr) | 2013-2015 | +/- 1 year |
| 100G QSFP28 | 2017 (TH1-based, 100G ports) | 2017-2018 | +/- 0.5 year |
| 400G QSFP-DD | 2021 (TH3 Dec 2017 + 3.5yr) | 2021-2022 | +/- 0.5 year |
| 800G OSFP | 2024-2025 (TH5 late 2022 + 2yr) | 2025 | +/- 0.5 year |
| 1.6T | Q4 2026 - Q1 2027 (TH6 Mar 2026 + 6-12mo) | TBD | — |
---
## Sources
### Standards Bodies
- [IEEE 802.3 Working Group](https://www.ieee802.org/3/)
- [IEEE 802.3dj Task Force](https://www.ieee802.org/3/dj/index.html)
- [OIF Implementation Agreements](https://www.oiforum.com/technical-work/implementation-agreements-ias/)
- [QSFP-DD MSA](http://www.qsfp-dd.com/)
- [OSFP MSA](https://osfpmsa.org/)
### Industry Analysts
- [Cignal AI — 800GbE Optics Shipments to Grow 60% in 2025](https://cignal.ai/2025/05/800gbe-optics-shipments-to-grow-60-in-2025/)
- [Cignal AI — 800G Coherent Pluggable >$1B Revenue in 2026](https://cignal.ai/2025/07/800g-coherent-pluggable-shipments-to-exceed-1b-revenue-in-2026/)
- [LightCounting — AI Creates New Wave in Demand for Optical Transceivers](https://www.lightcounting.com/newsletter/en/january-2025-optics-for-ai-clusters-319)
- [Dell'Oro Group — 1.6T volume switch deployments 2026](https://www.delloro.com/)
- [MarketsandMarkets — Optical Transceiver Market](https://www.marketsandmarkets.com/Market-Reports/optical-transceiver-market-161339599.html)
- [Mordor Intelligence — Optical Transceiver Market](https://www.mordorintelligence.com/industry-reports/optical-transceiver-market)
### Vendor Announcements
- [Broadcom TH6 Volume Shipments](https://www.broadcom.com/company/news/product-releases/63146)
- [Broadcom Sian3 DSP](https://investors.broadcom.com/news-releases/news-release-details/broadcom-delivers-industry-leading-200glane-dsp-gen-ai)
- [Broadcom Taurus 400G/lane DSP](https://www.stocktitan.net/news/AVGO/broadcom-delivers-industry-s-first-400g-lane-optical-dsp-for-next-0ysjo3zlcexv.html)
- [Marvell Ara 1.6T DSP Platform](https://investor.marvell.com/news-events/press-releases/detail/1013/marvell-ushers-in-the-1-6t-era-with-expanded-optical-dsp-platform-portfolio-redefining-ai-data-center-end-to-end-connectivity)
- [Marvell Electra 2nm Coherent DSP](https://www.marvell.com/company/newsroom/marvell-1-6t-zr-zr-plus-pluggable-2nm-coherent-dsp-ai-interconnects.html)
- [Marvell Nova 1.6T DSP](https://www.marvell.com/content/dam/marvell/en/public-collateral/dsp/marvell-nova-1.6t-pam4-dsp-for-optical-transceiver-applications-product-brief.pdf)
- [Semtech 1.6T Demos at OFC 2026](https://www.semtech.com/company/press/showcases-ai-interconnect-leadership-with-live-1.6t-demos-ofc-2026)
- [NVIDIA ConnectX-8 SuperNIC](https://www.servethehome.com/this-is-the-next-gen-nvidia-connectx-8-supernic-for-800gbps-networking/)
- [Keysight 224G/Lane Test Solutions](https://convergedigest.com/keysight-intros-224g-lane-test-solutions/)
### Conference & Demo Sources
- [InnoLight OFC 2017 — 400G OSFP Introduction](https://www.prnewswire.com/news-releases/innolight-technology-announced-volume-shipments-of-17-100g-qsfp28-products-and-the-introduction-of-400g-osfp-at-ofc-2017-300421866.html)
- [Eoptolink Gen2 1.6T at OFC 2025](https://www.eoptolink.com/news/361-eoptolink-launches-its-gen2-1-6t-osfp-and-osfp-rhs-transceiver-family-at-ofc-2025)
- [Jabil 1.6T Pluggable Transceiver at OFC 2025](https://investors.jabil.com/news/news-details/2025/Jabil-Launches-1-6T-Pluggable-Transceiver/)
- [ATOP 1.6T DR8 SiPh Demo at OFC 2025](https://www.atoptechnology.com/ofc-2025-live-demo-atops-1-6t-osfp224-dr8-siph-module-in-action-for-next-gen-ai/)
### SerDes & ASIC Analysis
- [TrendForce — SerDes Wars: Broadcom, Marvell, MediaTek](https://www.trendforce.com/news/2026/03/13/news-serdes-wars-heat-up-broadcom-marvell-mediatek-battle-for-ai-interconnect-supremacy/)
- [OIF CEI 448G/224G/112G Interoperability Demo OFC 2025](https://www.oiforum.com/wp-content/uploads/OIF_CEI_Demo_OFC2025.pdf)
- [EDN — OFC 2025 1.6T Networking Innovations](https://www.edn.com/ofc-2025-unveils-1-6t-networking-innovations/)
### Market & Pricing
- [Deep Fundamental — Optical Module Market](https://deepfundamental.substack.com/p/deep-dive-optical-module-market)
- [Pluggables, Power, and Geopolitics](https://iamfabian.substack.com/p/pluggables-power-and-geopolitics)
- [Fierce Network — Optical vendors predict higher demand 400G/800G 2026](https://www.fierce-network.com/broadband/optical-transmission-vendors-predict-high-demand-400g-800g-2026)
- [Introl — Fiber Optics State of the Art 2025](https://introl.com/blog/fiber-optics-data-center-state-of-art-optical-interconnect-2025)

33
docker-compose.yml Normal file
View File

@ -0,0 +1,33 @@
services:
postgres:
image: timescale/timescaledb:latest-pg17
container_name: tip-postgres
environment:
POSTGRES_DB: transceiver_db
POSTGRES_USER: tip
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-tip_dev_2026}
ports:
- "5433:5432"
volumes:
- tip_pgdata:/var/lib/postgresql/data
- ./sql:/docker-entrypoint-initdb.d
healthcheck:
test: ["CMD-SHELL", "pg_isready -U tip -d transceiver_db"]
interval: 5s
timeout: 5s
retries: 5
qdrant:
image: qdrant/qdrant:latest
container_name: tip-qdrant
ports:
- "6333:6333"
- "6334:6334"
volumes:
- tip_qdrant:/qdrant/storage
environment:
QDRANT__SERVICE__GRPC_PORT: 6334
volumes:
tip_pgdata:
tip_qdrant:

71
ecosystem.config.js Normal file
View File

@ -0,0 +1,71 @@
module.exports = {
apps: [
{
name: "tip-api",
script: "./node_modules/.bin/tsx",
args: "packages/api/src/index.ts",
cwd: "/opt/tip",
interpreter: "none",
exec_mode: "fork",
env: {
NODE_ENV: "production",
API_PORT: "3201",
POSTGRES_HOST: "localhost",
POSTGRES_PORT: "5433",
POSTGRES_DB: "transceiver_db",
POSTGRES_USER: "tip",
POSTGRES_PASSWORD: "tip_prod_2026",
OLLAMA_URL: "http://localhost:11434",
QDRANT_URL: "http://localhost:6333",
DOCLING_URL: "http://localhost:8100",
},
max_memory_restart: "500M",
instances: 1,
autorestart: true,
},
{
name: "tip-mcp",
script: "./node_modules/.bin/tsx",
args: "packages/mcp-server/src/http-server.ts",
cwd: "/opt/tip",
interpreter: "none",
exec_mode: "fork",
env: {
NODE_ENV: "production",
MCP_HTTP_PORT: "3202",
MCP_SECRET: "tip-mcp-2026-prod",
POSTGRES_HOST: "localhost",
POSTGRES_PORT: "5433",
POSTGRES_DB: "transceiver_db",
POSTGRES_USER: "tip",
POSTGRES_PASSWORD: "tip_prod_2026",
QDRANT_URL: "http://localhost:6333",
OLLAMA_URL: "http://localhost:11434",
CORS_ORIGINS: "https://eo-global-pulse.context-x.org,https://switchblade.context-x.org",
},
max_memory_restart: "300M",
instances: 1,
autorestart: true,
},
{
name: "tip-scraper",
script: "./node_modules/.bin/tsx",
args: "packages/scraper/src/index.ts",
cwd: "/opt/tip",
interpreter: "none",
exec_mode: "fork",
env: {
NODE_ENV: "production",
POSTGRES_HOST: "localhost",
POSTGRES_PORT: "5433",
POSTGRES_DB: "transceiver_db",
POSTGRES_USER: "tip",
POSTGRES_PASSWORD: "tip_prod_2026",
},
max_memory_restart: "1G",
instances: 1,
autorestart: true,
cron_restart: "0 0 * * *",
},
],
};

5944
package-lock.json generated

File diff suppressed because it is too large Load Diff

View File

@ -1,45 +1,28 @@
{
"name": "transceiver-db",
"version": "1.0.0",
"description": "Open-source optical transceiver database. 89 products, 39 IEEE/MSA standards, 16 form factors, 9 speed tiers. SFP to 800G OSFP.",
"main": "dist/index.js",
"types": "dist/index.d.ts",
"name": "transceiver-intelligence-platform",
"version": "0.1.0",
"private": true,
"description": "Transceiver Intelligence Platform — the world's most comprehensive optical transceiver & network switch database",
"workspaces": [
"packages/*"
],
"scripts": {
"build": "tsc",
"prepublishOnly": "npm run build"
"build": "npm run build --workspaces",
"build:core": "npm run build -w packages/core",
"build:api": "npm run build -w packages/api",
"dev": "npm run dev -w packages/api",
"migrate": "tsx scripts/migrate.ts",
"seed": "tsx scripts/seed-from-npm.ts",
"db:reset": "npm run migrate && npm run seed"
},
"author": "Rene Fichtmueller",
"license": "MIT",
"keywords": [
"transceiver",
"optics",
"sfp",
"qsfp",
"networking",
"fiber",
"ieee",
"telecom",
"osfp",
"qsfp-dd",
"optical",
"datacenter",
"100g",
"400g",
"800g"
],
"files": [
"dist",
"LICENSE",
"README.md"
],
"repository": {
"type": "git",
"url": "https://github.com/renefichtmueller/transceiver-db"
},
"author": "Rene Fichtmueller",
"engines": {
"node": ">=14"
},
"devDependencies": {
"tsx": "^4.19",
"typescript": "^5.9.3"
}
}

28
packages/api/package.json Normal file
View File

@ -0,0 +1,28 @@
{
"name": "@tip/api",
"version": "0.1.0",
"private": true,
"description": "TIP REST API server",
"main": "dist/index.js",
"scripts": {
"build": "tsc",
"dev": "tsx watch src/index.ts",
"start": "node dist/index.js"
},
"dependencies": {
"express": "^5.1.0",
"pg": "^8.13.1",
"cors": "^2.8.5",
"dotenv": "^16.4.7",
"helmet": "^8.0.0",
"express-rate-limit": "^7.5.0",
"zod": "^3.24.2"
},
"devDependencies": {
"@types/express": "^5.0.0",
"@types/pg": "^8.11.11",
"@types/cors": "^2.8.17",
"typescript": "^5.9.3",
"tsx": "^4.19.0"
}
}

View File

@ -0,0 +1,20 @@
import { config } from "dotenv";
import { join } from "path";
config({ path: join(__dirname, "..", "..", "..", ".env") });
export const cfg = {
port: parseInt(process.env.API_PORT || "3200"),
nodeEnv: process.env.NODE_ENV || "development",
db: {
host: process.env.POSTGRES_HOST || "localhost",
port: parseInt(process.env.POSTGRES_PORT || "5432"),
database: process.env.POSTGRES_DB || "transceiver_db",
user: process.env.POSTGRES_USER || "tip",
password: process.env.POSTGRES_PASSWORD || "tip_dev_2026",
},
qdrant: {
host: process.env.QDRANT_HOST || "localhost",
port: parseInt(process.env.QDRANT_PORT || "6333"),
},
} as const;

View File

@ -0,0 +1,17 @@
import { Pool } from "pg";
import { cfg } from "../config";
export const pool = new Pool({
host: cfg.db.host,
port: cfg.db.port,
database: cfg.db.database,
user: cfg.db.user,
password: cfg.db.password,
max: 20,
idleTimeoutMillis: 30000,
connectionTimeoutMillis: 5000,
});
pool.on("error", (err) => {
console.error("Unexpected PostgreSQL error:", err);
});

View File

@ -0,0 +1,292 @@
import { pool } from "./client";
export interface SearchParams {
q?: string;
form_factor?: string;
speed?: string;
speed_gbps?: number;
category?: string;
fiber_type?: string;
reach_min?: number;
reach_max?: number;
wdm_type?: string;
coherent?: boolean;
market_status?: string;
vendor?: string;
limit?: number;
offset?: number;
}
export async function searchTransceivers(params: SearchParams) {
const conditions: string[] = [];
const values: any[] = [];
let idx = 1;
if (params.q) {
conditions.push(`search_vector @@ plainto_tsquery('english', $${idx})`);
values.push(params.q);
idx++;
}
if (params.form_factor) {
conditions.push(`form_factor = $${idx}`);
values.push(params.form_factor);
idx++;
}
if (params.speed) {
conditions.push(`speed = $${idx}`);
values.push(params.speed);
idx++;
}
if (params.speed_gbps) {
conditions.push(`speed_gbps = $${idx}`);
values.push(params.speed_gbps);
idx++;
}
if (params.category) {
conditions.push(`category = $${idx}`);
values.push(params.category);
idx++;
}
if (params.fiber_type) {
conditions.push(`fiber_type = $${idx}`);
values.push(params.fiber_type);
idx++;
}
if (params.reach_min) {
conditions.push(`reach_meters >= $${idx}`);
values.push(params.reach_min);
idx++;
}
if (params.reach_max) {
conditions.push(`reach_meters <= $${idx}`);
values.push(params.reach_max);
idx++;
}
if (params.wdm_type) {
conditions.push(`wdm_type = $${idx}`);
values.push(params.wdm_type);
idx++;
}
if (params.coherent !== undefined) {
conditions.push(`coherent = $${idx}`);
values.push(params.coherent);
idx++;
}
if (params.market_status) {
conditions.push(`market_status = $${idx}`);
values.push(params.market_status);
idx++;
}
if (params.vendor) {
conditions.push(`v.name ILIKE $${idx}`);
values.push(`%${params.vendor}%`);
idx++;
}
const where = conditions.length > 0 ? `WHERE ${conditions.join(" AND ")}` : "";
const limit = params.limit || 50;
const offset = params.offset || 0;
// Add relevance ranking when full-text search is used
const orderBy = params.q
? `ORDER BY ts_rank(search_vector, plainto_tsquery('english', $1)) DESC`
: `ORDER BY speed_gbps DESC, reach_meters ASC`;
const query = `
SELECT t.*, v.name as vendor_name
FROM transceivers t
LEFT JOIN vendors v ON t.vendor_id = v.id
${where}
${orderBy}
LIMIT ${limit} OFFSET ${offset}
`;
const countQuery = `SELECT COUNT(*) FROM transceivers t LEFT JOIN vendors v ON t.vendor_id = v.id ${where}`;
const [dataResult, countResult] = await Promise.all([
pool.query(query, values),
pool.query(countQuery, values),
]);
return {
data: dataResult.rows,
total: parseInt(countResult.rows[0].count),
limit,
offset,
};
}
export async function getTransceiverById(id: string) {
const result = await pool.query(
`SELECT t.*, v.name as vendor_name, s.name as standard_full_name
FROM transceivers t
LEFT JOIN vendors v ON t.vendor_id = v.id
LEFT JOIN standards s ON t.standard_id = s.id
WHERE t.id::text = $1::text OR t.slug = $1::text`,
[id]
);
return result.rows[0] || null;
}
export interface SwitchSearchParams extends SearchParams {
whitebox?: boolean;
sonic_compatible?: boolean;
asic_vendor?: string;
nos?: string;
ocp?: boolean;
max_speed_gbps?: number;
}
export async function searchSwitches(params: SwitchSearchParams) {
const conditions: string[] = [];
const values: any[] = [];
let idx = 1;
if (params.q) {
conditions.push(`sw.search_vector @@ plainto_tsquery('english', $${idx})`);
values.push(params.q);
idx++;
}
if (params.category) {
conditions.push(`sw.category = $${idx}`);
values.push(params.category);
idx++;
}
if (params.whitebox !== undefined) {
conditions.push(`sw.is_whitebox = $${idx}`);
values.push(params.whitebox);
idx++;
}
if (params.sonic_compatible !== undefined) {
conditions.push(`sw.sonic_compatible = $${idx}`);
values.push(params.sonic_compatible);
idx++;
}
if (params.asic_vendor) {
conditions.push(`sw.asic_vendor ILIKE $${idx}`);
values.push(`%${params.asic_vendor}%`);
idx++;
}
if (params.nos) {
conditions.push(`$${idx} = ANY(sw.supported_nos)`);
values.push(params.nos);
idx++;
}
if (params.ocp !== undefined) {
conditions.push(`sw.is_ocp_accepted = $${idx}`);
values.push(params.ocp);
idx++;
}
if (params.max_speed_gbps) {
conditions.push(`sw.max_speed_gbps = $${idx}`);
values.push(params.max_speed_gbps);
idx++;
}
const where = conditions.length > 0 ? `WHERE ${conditions.join(" AND ")}` : "";
const limit = params.limit || 50;
const offset = params.offset || 0;
const orderBy = params.q
? `ORDER BY ts_rank(sw.search_vector, plainto_tsquery('english', $1)) DESC`
: `ORDER BY sw.max_speed_gbps DESC NULLS LAST`;
const query = `
SELECT sw.*, v.name as vendor_name
FROM switches sw
LEFT JOIN vendors v ON sw.vendor_id = v.id
${where}
${orderBy}
LIMIT ${limit} OFFSET ${offset}
`;
const countQuery = `SELECT COUNT(*) FROM switches sw ${where}`;
const [dataResult, countResult] = await Promise.all([
pool.query(query, values),
pool.query(countQuery, values),
]);
return {
data: dataResult.rows,
total: parseInt(countResult.rows[0].count),
limit,
offset,
};
}
export async function getSwitchById(id: string) {
const result = await pool.query(
`SELECT sw.*, v.name as vendor_name, v.website as vendor_website
FROM switches sw
LEFT JOIN vendors v ON sw.vendor_id = v.id
WHERE sw.id = $1`,
[id]
);
return result.rows[0] || null;
}
export async function getSwitchDocuments(switchId: string) {
const result = await pool.query(
`SELECT pd.*, v.name as vendor_name
FROM product_documents pd
LEFT JOIN vendors v ON pd.vendor_id = v.id
WHERE pd.switch_id = $1
ORDER BY pd.doc_type, pd.title`,
[switchId]
);
return result.rows;
}
export async function getCompatibleTransceivers(switchId: string) {
const result = await pool.query(
`SELECT t.*, c.status, c.verified_by, c.notes as compat_notes
FROM compatibility c
JOIN transceivers t ON c.transceiver_id = t.id
WHERE c.switch_id = $1 AND c.status = 'compatible'
ORDER BY t.speed_gbps DESC`,
[switchId]
);
return result.rows;
}
export async function listVendors(type?: string) {
const query = type
? `SELECT v.*,
(SELECT COUNT(*) FROM transceivers t WHERE t.vendor_id = v.id) as transceiver_count,
(SELECT COUNT(*) FROM switches s WHERE s.vendor_id = v.id) as switch_count
FROM vendors v WHERE v.type = $1 ORDER BY v.name`
: `SELECT v.*,
(SELECT COUNT(*) FROM transceivers t WHERE t.vendor_id = v.id) as transceiver_count,
(SELECT COUNT(*) FROM switches s WHERE s.vendor_id = v.id) as switch_count
FROM vendors v ORDER BY v.name`;
const result = await pool.query(query, type ? [type] : []);
return result.rows;
}
export async function listStandards(speed?: string) {
const query = speed
? `SELECT * FROM standards WHERE speed = $1 ORDER BY year_ratified DESC`
: `SELECT * FROM standards ORDER BY year_ratified DESC`;
const result = await pool.query(query, speed ? [speed] : []);
return result.rows;
}
export async function getDbStats() {
const result = await pool.query(`
SELECT
(SELECT COUNT(*) FROM vendors) as vendor_count,
(SELECT COUNT(*) FROM standards) as standard_count,
(SELECT COUNT(*) FROM transceivers) as transceiver_count,
(SELECT COUNT(*) FROM switches) as switch_count,
(SELECT COUNT(*) FROM compatibility) as compatibility_count,
(SELECT COUNT(*) FROM breakouts) as breakout_count,
(SELECT COUNT(*) FROM knowledge_base) as kb_count,
(SELECT COUNT(*) FROM documents) as document_count,
(SELECT COUNT(*) FROM news_articles) as news_count,
(SELECT COUNT(*) FROM product_documents) as product_document_count,
(SELECT COUNT(*) FROM switches WHERE image_url IS NOT NULL) as switches_with_images,
(SELECT COUNT(*) FROM switches WHERE datasheet_r2_key IS NOT NULL) as switches_with_datasheets
`);
return result.rows[0];
}

View File

@ -0,0 +1,151 @@
/**
* Embedding + Qdrant client for vector search.
*
* Ollama nomic-embed-text (768 dim) Qdrant collections.
* Supports: products, datasheets, FAQs, manuals, troubleshooting, news.
*/
const OLLAMA_URL = process.env.OLLAMA_URL || "http://localhost:11434";
const QDRANT_URL = process.env.QDRANT_URL || "http://localhost:6333";
const EMBED_MODEL = process.env.EMBED_MODEL || "nomic-embed-text";
export type CollectionName =
| "product_embeddings"
| "datasheet_chunks"
| "faq_embeddings"
| "manual_chunks"
| "troubleshooting_embeddings"
| "news_embeddings";
/** Generate embedding vector from text */
export async function embed(text: string): Promise<number[]> {
const resp = await fetch(`${OLLAMA_URL}/api/embeddings`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ model: EMBED_MODEL, prompt: text }),
signal: AbortSignal.timeout(30000),
});
if (!resp.ok) {
throw new Error(`Ollama embed failed: ${resp.status} ${await resp.text()}`);
}
const data = await resp.json() as { embedding: number[] };
return data.embedding;
}
/** Batch embed multiple texts */
export async function embedBatch(texts: ReadonlyArray<string>): Promise<number[][]> {
const results: number[][] = [];
// Ollama doesn't support batch embedding natively, so we serialize
// with concurrency limit to avoid overloading
const CONCURRENCY = 3;
for (let i = 0; i < texts.length; i += CONCURRENCY) {
const batch = texts.slice(i, i + CONCURRENCY);
const embeddings = await Promise.all(batch.map((t) => embed(t)));
results.push(...embeddings);
}
return results;
}
/** Upsert a point into Qdrant */
export async function upsertPoint(
collection: CollectionName,
id: string,
vector: number[],
payload: Record<string, unknown>,
): Promise<void> {
const resp = await fetch(`${QDRANT_URL}/collections/${collection}/points`, {
method: "PUT",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
points: [{ id, vector, payload }],
}),
signal: AbortSignal.timeout(10000),
});
if (!resp.ok) {
throw new Error(`Qdrant upsert failed: ${resp.status} ${await resp.text()}`);
}
}
/** Batch upsert points */
export async function upsertPoints(
collection: CollectionName,
points: ReadonlyArray<{ id: string; vector: number[]; payload: Record<string, unknown> }>,
): Promise<void> {
if (points.length === 0) return;
const resp = await fetch(`${QDRANT_URL}/collections/${collection}/points`, {
method: "PUT",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ points }),
signal: AbortSignal.timeout(30000),
});
if (!resp.ok) {
throw new Error(`Qdrant batch upsert failed: ${resp.status} ${await resp.text()}`);
}
}
/** Search Qdrant with vector similarity + optional payload filter */
export async function searchSimilar(
collection: CollectionName,
queryVector: number[],
limit: number = 10,
filter?: Record<string, unknown>,
): Promise<Array<{ id: string; score: number; payload: Record<string, unknown> }>> {
const body: Record<string, unknown> = {
vector: queryVector,
limit,
with_payload: true,
};
if (filter) {
body.filter = filter;
}
const resp = await fetch(`${QDRANT_URL}/collections/${collection}/points/search`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify(body),
signal: AbortSignal.timeout(10000),
});
if (!resp.ok) {
throw new Error(`Qdrant search failed: ${resp.status} ${await resp.text()}`);
}
const data = await resp.json() as { result: Array<{ id: string; score: number; payload: Record<string, unknown> }> };
return data.result;
}
/** Semantic search: embed query text then search Qdrant */
export async function semanticSearch(
collection: CollectionName,
query: string,
limit: number = 10,
filter?: Record<string, unknown>,
): Promise<Array<{ id: string; score: number; payload: Record<string, unknown> }>> {
const vector = await embed(query);
return searchSimilar(collection, vector, limit, filter);
}
/** Get collection info (point count, etc.) */
export async function getCollectionInfo(
collection: CollectionName,
): Promise<{ pointsCount: number; vectorsCount: number }> {
const resp = await fetch(`${QDRANT_URL}/collections/${collection}`, {
signal: AbortSignal.timeout(5000),
});
if (!resp.ok) {
throw new Error(`Qdrant info failed: ${resp.status}`);
}
const data = await resp.json() as { result: { points_count: number; vectors_count: number } };
return {
pointsCount: data.result.points_count,
vectorsCount: data.result.vectors_count,
};
}

View File

@ -0,0 +1,336 @@
/**
* OCR Pipeline: PDF/document Docling chunks Ollama embed Qdrant
*
* Connects to the Docling REST API (Erik port 8100) for document conversion,
* then chunks markdown output and embeds into Qdrant collections.
*
* Collections:
* - datasheet_chunks: Product datasheets (specs, diagrams, compliance)
* - manual_chunks: Installation/configuration manuals
*
* Run: npx tsx packages/api/src/embeddings/ocr-pipeline.ts [--url <pdf_url>] [--dir <path>]
*/
import { pool } from "../db/client";
import { embed, upsertPoints, CollectionName } from "./client";
import { randomUUID } from "crypto";
const DOCLING_URL = process.env.DOCLING_URL || "http://localhost:8100";
interface DoclingResult {
success: boolean;
content: string;
format: string;
pages: number | null;
error?: string;
}
interface DocumentChunk {
id: string;
vector: number[];
payload: {
document_id: string;
source_url: string;
document_type: "datasheet" | "manual" | "whitepaper";
chunk_index: number;
total_chunks: number;
title: string;
section_heading: string;
text: string;
page_estimate: number | null;
vendor: string;
product_slug: string;
};
}
/** Convert a document via Docling API */
async function convertDocument(url: string, format: "markdown" | "json" = "markdown"): Promise<DoclingResult> {
const resp = await fetch(`${DOCLING_URL}/convert`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ url, format }),
signal: AbortSignal.timeout(120000), // 2 min for large PDFs
});
if (!resp.ok) {
throw new Error(`Docling convert failed: ${resp.status} ${await resp.text()}`);
}
return resp.json() as Promise<DoclingResult>;
}
/**
* Chunk markdown into overlapping sections.
*
* Strategy:
* 1. Split by ## headings first (natural section boundaries)
* 2. If a section exceeds maxChunkSize, split by paragraphs
* 3. Apply overlap (repeat last N chars of previous chunk)
*/
function chunkMarkdown(
markdown: string,
maxChunkSize: number = 1500,
overlapSize: number = 200,
): Array<{ heading: string; text: string }> {
const sections = markdown.split(/(?=^#{1,3}\s)/m);
const chunks: Array<{ heading: string; text: string }> = [];
for (const section of sections) {
const trimmed = section.trim();
if (!trimmed || trimmed.length < 20) continue;
// Extract heading
const headingMatch = trimmed.match(/^(#{1,3})\s+(.+)/);
const heading = headingMatch ? headingMatch[2].trim() : "Introduction";
const body = headingMatch ? trimmed.slice(headingMatch[0].length).trim() : trimmed;
if (body.length <= maxChunkSize) {
chunks.push({ heading, text: body });
} else {
// Split large sections by paragraphs
const paragraphs = body.split(/\n\n+/);
let currentChunk = "";
for (const para of paragraphs) {
if (currentChunk.length + para.length > maxChunkSize && currentChunk.length > 0) {
chunks.push({ heading, text: currentChunk.trim() });
// Overlap: keep tail of previous chunk
const overlapText = currentChunk.slice(-overlapSize);
currentChunk = overlapText + "\n\n" + para;
} else {
currentChunk += (currentChunk ? "\n\n" : "") + para;
}
}
if (currentChunk.trim().length > 20) {
chunks.push({ heading, text: currentChunk.trim() });
}
}
}
return chunks;
}
/** Classify document type from URL or content */
function classifyDocument(url: string, content: string): "datasheet" | "manual" | "whitepaper" {
const urlLower = url.toLowerCase();
const contentLower = content.slice(0, 2000).toLowerCase();
if (urlLower.includes("datasheet") || contentLower.includes("datasheet") || contentLower.includes("specifications")) {
return "datasheet";
}
if (urlLower.includes("manual") || urlLower.includes("install") || contentLower.includes("installation guide") || contentLower.includes("user manual")) {
return "manual";
}
return "whitepaper";
}
/** Extract vendor name from URL or content */
function extractVendor(url: string): string {
const urlLower = url.toLowerCase();
const vendorPatterns: Array<[RegExp, string]> = [
[/flexoptix/i, "Flexoptix"],
[/cisco/i, "Cisco"],
[/juniper/i, "Juniper"],
[/arista/i, "Arista"],
[/nokia/i, "Nokia"],
[/huawei/i, "Huawei"],
[/finisar|ii-vi|coherent/i, "II-VI/Coherent"],
[/innolight/i, "Innolight"],
[/broadcom/i, "Broadcom"],
[/intel/i, "Intel"],
[/fs\.com|fiberstore/i, "FS.com"],
[/10gtek/i, "10Gtek"],
];
for (const [pattern, name] of vendorPatterns) {
if (pattern.test(urlLower)) return name;
}
return "Unknown";
}
/** Extract product slug from URL */
function extractProductSlug(url: string): string {
const filename = url.split("/").pop() || "";
return filename.replace(/\.(pdf|docx|doc|xlsx)$/i, "").replace(/[^a-zA-Z0-9-]/g, "-").toLowerCase();
}
/** Process a single document: convert → chunk → embed → store */
async function processDocument(
url: string,
collection: CollectionName = "datasheet_chunks",
title?: string,
): Promise<{ documentId: string; chunksStored: number }> {
const documentId = randomUUID();
console.log(` Converting: ${url}`);
const result = await convertDocument(url);
if (!result.success) {
throw new Error(`Conversion failed: ${result.error}`);
}
const markdown = result.content;
console.log(` Converted: ${result.pages ?? "?"} pages, ${markdown.length} chars`);
const docType = classifyDocument(url, markdown);
const vendor = extractVendor(url);
const productSlug = extractProductSlug(url);
const docTitle = title || productSlug.replace(/-/g, " ");
// Chunk the markdown
const chunks = chunkMarkdown(markdown);
console.log(` Chunked: ${chunks.length} chunks (type: ${docType})`);
if (chunks.length === 0) {
console.log(" Warning: No chunks produced, skipping");
return { documentId, chunksStored: 0 };
}
// Embed and store in batches
const BATCH_SIZE = 5;
let stored = 0;
for (let i = 0; i < chunks.length; i += BATCH_SIZE) {
const batch = chunks.slice(i, i + BATCH_SIZE);
const points: DocumentChunk[] = await Promise.all(
batch.map(async (chunk, idx) => {
const chunkIndex = i + idx;
const embeddingText = `${docTitle}. ${chunk.heading}. ${chunk.text}`;
const vector = await embed(embeddingText);
return {
id: randomUUID(),
vector,
payload: {
document_id: documentId,
source_url: url,
document_type: docType,
chunk_index: chunkIndex,
total_chunks: chunks.length,
title: docTitle,
section_heading: chunk.heading,
text: chunk.text,
page_estimate: result.pages,
vendor,
product_slug: productSlug,
},
};
}),
);
await upsertPoints(collection, points);
stored += points.length;
console.log(` Embedded ${stored}/${chunks.length} chunks`);
}
// Record in documents table
try {
await pool.query(
`INSERT INTO documents (id, entity_type, doc_type, title, r2_key, source_url, page_count, chunks_count, ocr_status, processed_at)
VALUES ($1, 'transceiver', $2, $3, $4, $5, $6, $7, 'completed', NOW())
ON CONFLICT ON CONSTRAINT documents_pkey DO UPDATE
SET processed_at = NOW(), chunks_count = $7, ocr_status = 'completed'`,
[documentId, docType, docTitle, `ocr/${documentId}`, url, result.pages, chunks.length],
);
} catch {
// ignore if insert fails
}
return { documentId, chunksStored: stored };
}
/** Known datasheet URLs to seed from */
const SEED_DATASHEETS: Array<{ url: string; title: string; collection: CollectionName }> = [
// Flexoptix product guides
{
url: "https://www.flexoptix.net/media/pdf/flexoptix-sfp-compatibility-guide.pdf",
title: "Flexoptix SFP Compatibility Guide",
collection: "datasheet_chunks",
},
// IEEE standards (publicly available)
{
url: "https://standards.ieee.org/content/dam/ieee-standards/standards/web/download/802.3-2022_downloads/802.3-2022.pdf",
title: "IEEE 802.3 Ethernet Standard",
collection: "manual_chunks",
},
];
async function main() {
const args = process.argv.slice(2);
console.log("=== OCR Pipeline: Document → Chunks → Embeddings ===\n");
// Check Docling health
try {
const healthResp = await fetch(`${DOCLING_URL}/health`, { signal: AbortSignal.timeout(5000) });
const health = await healthResp.json() as { status: string };
console.log(`Docling API: ${health.status} at ${DOCLING_URL}`);
} catch (err) {
console.error(`Docling API not reachable at ${DOCLING_URL}: ${(err as Error).message}`);
console.error("Set DOCLING_URL env var or start Docling on Erik (port 8100)");
process.exit(1);
}
let totalDocs = 0;
let totalChunks = 0;
if (args.includes("--url")) {
// Process a single URL
const urlIdx = args.indexOf("--url") + 1;
const url = args[urlIdx];
const title = args.includes("--title") ? args[args.indexOf("--title") + 1] : undefined;
const collection = (args.includes("--collection") ? args[args.indexOf("--collection") + 1] : "datasheet_chunks") as CollectionName;
if (!url) {
console.error("Usage: --url <pdf_url> [--title <title>] [--collection <name>]");
process.exit(1);
}
const result = await processDocument(url, collection, title);
totalDocs = 1;
totalChunks = result.chunksStored;
} else if (args.includes("--dir")) {
// Process all PDFs in a directory
const dirIdx = args.indexOf("--dir") + 1;
const dir = args[dirIdx];
const { readdirSync } = await import("fs");
const files = readdirSync(dir).filter((f) => f.toLowerCase().endsWith(".pdf"));
console.log(`Found ${files.length} PDFs in ${dir}\n`);
for (const file of files) {
const filePath = `${dir}/${file}`;
try {
const result = await processDocument(filePath, "datasheet_chunks");
totalDocs++;
totalChunks += result.chunksStored;
} catch (err) {
console.error(` Failed: ${file}${(err as Error).message}`);
}
}
} else {
// Seed from known URLs
console.log(`Processing ${SEED_DATASHEETS.length} seed documents\n`);
for (const doc of SEED_DATASHEETS) {
try {
console.log(`\n[${doc.title}]`);
const result = await processDocument(doc.url, doc.collection, doc.title);
totalDocs++;
totalChunks += result.chunksStored;
} catch (err) {
console.error(` Failed: ${doc.title}${(err as Error).message}`);
}
}
}
console.log(`\n=== Done: ${totalDocs} documents, ${totalChunks} chunks embedded ===`);
await pool.end();
}
main().catch((err) => {
console.error("Fatal:", err);
pool.end();
process.exit(1);
});

View File

@ -0,0 +1,300 @@
/**
* Seed faq_embeddings and troubleshooting_embeddings from built-in knowledge.
*
* Since we don't have a FAQ table yet, this seeds from a curated set of
* transceiver FAQ entries and troubleshooting guides based on Flexoptix
* domain expertise.
*
* Run: npx tsx packages/api/src/embeddings/seed-faq.ts
*/
import { embed, upsertPoints } from "./client";
import { randomUUID } from "crypto";
interface FaqEntry {
question: string;
answer: string;
category: string;
tags: string[];
}
interface TroubleshootingEntry {
symptom: string;
cause: string;
solution: string;
category: string;
severity: "low" | "medium" | "high" | "critical";
}
const FAQ_ENTRIES: FaqEntry[] = [
// Form Factor FAQs
{
question: "What is the difference between SFP, SFP+, and SFP28?",
answer: "SFP supports up to 1Gbps, SFP+ supports up to 10Gbps, and SFP28 supports up to 25Gbps. They share the same physical form factor but differ in electrical interface speed. SFP+ and SFP28 are backward compatible with SFP ports at reduced speeds.",
category: "form_factor",
tags: ["SFP", "SFP+", "SFP28", "compatibility"],
},
{
question: "What is the difference between QSFP+ and QSFP28?",
answer: "QSFP+ supports 4x10G = 40Gbps, while QSFP28 supports 4x25G = 100Gbps. Both use the same physical form factor. QSFP28 is backward compatible with QSFP+ ports. QSFP28 can also be used in breakout mode (4x25G).",
category: "form_factor",
tags: ["QSFP+", "QSFP28", "40G", "100G", "breakout"],
},
{
question: "What is QSFP-DD and how does it differ from QSFP28?",
answer: "QSFP-DD (Double Density) has 8 lanes instead of 4, supporting up to 400G (8x50G) or 800G (8x100G). It is backward compatible with QSFP28 modules. The 'DD' refers to the doubled electrical lane count.",
category: "form_factor",
tags: ["QSFP-DD", "400G", "800G"],
},
{
question: "What is OSFP and when should I use it?",
answer: "OSFP (Octal Small Form Factor Pluggable) is designed for 400G and 800G applications. It has 8 electrical lanes and better thermal performance than QSFP-DD due to its slightly larger size. OSFP is preferred for high-power coherent optics and 800G deployments.",
category: "form_factor",
tags: ["OSFP", "400G", "800G", "coherent"],
},
{
question: "What is CFP2-DCO?",
answer: "CFP2-DCO is a coherent Digital Coherent Optic transceiver in CFP2 form factor. It supports 100G-400G coherent transmission over long distances (hundreds to thousands of km). DCO includes the DSP (Digital Signal Processor) inside the module.",
category: "form_factor",
tags: ["CFP2", "DCO", "coherent", "long-haul"],
},
// Fiber Type FAQs
{
question: "What is the difference between single-mode and multi-mode fiber?",
answer: "Single-mode fiber (SMF, 9/125µm) carries one light mode and supports distances up to 80km+ at higher cost. Multi-mode fiber (MMF, 50/125µm OM3/OM4/OM5) carries multiple light modes, supports up to 300-400m, and is cheaper. Use SMF for campus/metro/long-haul, MMF for intra-building/data center.",
category: "fiber",
tags: ["SMF", "MMF", "single-mode", "multi-mode", "OM3", "OM4"],
},
{
question: "What do OM3, OM4, and OM5 mean?",
answer: "OM3, OM4, and OM5 are multi-mode fiber grades. OM3 supports 10G up to 300m, OM4 supports 10G up to 400m, and OM5 supports SWDM (Shortwave Division Multiplexing) for 40G/100G over longer distances. OM5 is also called WBMMF (Wideband Multi-Mode Fiber).",
category: "fiber",
tags: ["OM3", "OM4", "OM5", "multi-mode", "SWDM"],
},
// Reach & Distance FAQs
{
question: "What do SR, LR, ER, and ZR mean in transceiver names?",
answer: "SR = Short Reach (up to 300-400m, typically multi-mode). LR = Long Reach (up to 10km, single-mode). ER = Extended Reach (up to 40km, single-mode). ZR = Very Long Reach (up to 80km+, single-mode). These are IEEE/MSA standard designations.",
category: "reach",
tags: ["SR", "LR", "ER", "ZR", "reach", "distance"],
},
{
question: "What is the maximum distance for a 100G-LR4 transceiver?",
answer: "100G-LR4 supports up to 10km over single-mode fiber using 4 wavelengths (1295.56nm, 1300.05nm, 1304.58nm, 1309.14nm) with CWDM4 technology. It uses a duplex LC connector.",
category: "reach",
tags: ["100G", "LR4", "10km", "CWDM4"],
},
// Compatibility FAQs
{
question: "Can I use third-party transceivers in Cisco switches?",
answer: "Yes, but Cisco switches may show warnings for non-Cisco-branded optics. Use 'service unsupported-transceiver' command to suppress warnings. Third-party compatible transceivers (like Flexoptix) are programmed with Cisco-compatible coding and work identically. Always verify the specific switch model and IOS version.",
category: "compatibility",
tags: ["Cisco", "third-party", "compatible", "coding"],
},
{
question: "What is transceiver coding and why does it matter?",
answer: "Transceiver coding refers to the EEPROM data programmed into the SFP/QSFP module that identifies it to the switch. Vendor-locked switches check this data and may reject uncoded or incorrectly coded optics. Flexoptix FlexBox can reprogram coding for any vendor.",
category: "compatibility",
tags: ["coding", "EEPROM", "FlexBox", "vendor-lock"],
},
{
question: "How do breakout cables work with QSFP28?",
answer: "A QSFP28 breakout cable splits one 100G QSFP28 port into 4x25G SFP28 connections using a DAC or AOC cable. This requires the switch to support breakout mode. Common breakout configurations: QSFP28 to 4xSFP28, QSFP-DD to 2xQSFP28, OSFP to 2xQSFP28.",
category: "compatibility",
tags: ["breakout", "QSFP28", "SFP28", "DAC", "AOC"],
},
// WDM FAQs
{
question: "What is the difference between CWDM and DWDM?",
answer: "CWDM (Coarse WDM) uses wider channel spacing (20nm), supports up to 18 channels, and is cheaper but limited to shorter distances. DWDM (Dense WDM) uses narrow spacing (0.8nm/100GHz or 0.4nm/50GHz), supports 40-96+ channels, and enables long-haul with amplification (EDFA).",
category: "wdm",
tags: ["CWDM", "DWDM", "WDM", "wavelength"],
},
{
question: "What is 400ZR?",
answer: "400ZR is an OIF standard for 400G coherent DWDM transmission in QSFP-DD or OSFP form factor. It supports up to 120km over a single wavelength using 16QAM modulation. It enables point-to-point DCI (Data Center Interconnect) without external transponders.",
category: "wdm",
tags: ["400ZR", "coherent", "DCI", "DWDM", "OIF"],
},
// Power & Temperature FAQs
{
question: "What is the typical power consumption of different transceivers?",
answer: "SFP/SFP+: 0.7-1.5W. SFP28: 1-2W. QSFP+: 1.5-3.5W. QSFP28: 2.5-5W. QSFP-DD 400G: 12-15W. OSFP 400G: 15-20W. Coherent (CFP2-DCO): 20-25W. 800G OSFP: 20-30W. Higher speeds and longer reach generally require more power.",
category: "power",
tags: ["power", "watts", "thermal"],
},
{
question: "What temperature ranges do transceivers support?",
answer: "Commercial (COM): 0°C to 70°C — standard data center use. Extended (EXT): -5°C to 85°C — outdoor/harsh environments. Industrial (IND): -40°C to 85°C — extreme conditions. Most data center transceivers are COM rated.",
category: "power",
tags: ["temperature", "commercial", "industrial", "extended"],
},
// Market & Technology FAQs
{
question: "What is CPO (Co-Packaged Optics)?",
answer: "CPO integrates optical transceivers directly into the switch ASIC package, eliminating the pluggable module. This reduces power consumption (30-50% less), latency, and cost at scale. CPO is expected for 1.6T+ speeds but is still in development. Current approach is mostly pluggable (OSFP/QSFP-DD).",
category: "technology",
tags: ["CPO", "co-packaged", "1.6T", "future"],
},
{
question: "What is LPO (Linear Pluggable Optics)?",
answer: "LPO removes the DSP/retimer from the transceiver module, reducing power and cost by ~30%. It requires clean signal paths and shorter reach. LPO works for intra-rack connections (<2m) where signal integrity is sufficient. Currently in early adoption with limited vendor support.",
category: "technology",
tags: ["LPO", "linear", "power-saving"],
},
{
question: "When will 800G transceivers be mainstream?",
answer: "800G OSFP/QSFP-DD800 transceivers are available since 2024 from vendors like Innolight, Coherent, and Broadcom. Mass adoption is expected 2025-2026 as switch ASICs (Broadcom Tomahawk 5, 51.2T) support native 800G ports. AI/ML data centers are the primary early adopters.",
category: "technology",
tags: ["800G", "OSFP", "Tomahawk", "AI"],
},
];
const TROUBLESHOOTING_ENTRIES: TroubleshootingEntry[] = [
{
symptom: "Transceiver not recognized by switch — port shows 'not present' or 'unsupported'",
cause: "Incorrect transceiver coding (EEPROM vendor ID doesn't match switch vendor), or module is not compatible with the switch platform/IOS version.",
solution: "1. Check switch vendor compatibility list. 2. Verify transceiver coding matches the switch vendor. 3. For Cisco: use 'service unsupported-transceiver' command. 4. Try reseating the module. 5. Use Flexoptix FlexBox to recode if needed.",
category: "recognition",
severity: "high",
},
{
symptom: "Link is up but experiencing high bit error rate (BER) or CRC errors",
cause: "Dirty fiber connector, fiber bend radius exceeded, incorrect fiber type (SMF vs MMF mismatch), or transceiver power levels out of spec.",
solution: "1. Clean fiber connectors with IPA wipes. 2. Check fiber type matches transceiver (SMF for LR, MMF for SR). 3. Use OTDR to test fiber. 4. Check optical power levels via 'show interface transceiver' — should be within Rx sensitivity range. 5. Replace patch cable.",
category: "performance",
severity: "high",
},
{
symptom: "Transceiver overheating — temperature alarm triggered",
cause: "Insufficient airflow in switch chassis, failed fan module, ambient temperature too high, or transceiver power consumption exceeds port thermal budget.",
solution: "1. Check fan module status. 2. Verify ambient temperature is within transceiver spec (usually 0-70°C COM). 3. Ensure proper airflow — no blocked vents. 4. Consider lower-power transceiver variant. 5. For high-power modules (>15W), verify switch supports the thermal requirement.",
category: "thermal",
severity: "critical",
},
{
symptom: "Link flapping — port goes up and down repeatedly",
cause: "Marginal optical power (near Rx sensitivity threshold), loose connector, faulty transceiver, or auto-negotiation mismatch.",
solution: "1. Check optical power — if near sensitivity threshold, clean connectors or shorten fiber. 2. Verify both ends have matching speed/duplex settings. 3. Try a different transceiver. 4. Check for fiber micro-bends. 5. Disable auto-negotiation if both ends are configured correctly.",
category: "connectivity",
severity: "high",
},
{
symptom: "Low transmit (Tx) power from transceiver",
cause: "Aging laser, transceiver damage, or operating at edge of temperature range.",
solution: "1. Compare Tx power with datasheet spec — if below minimum, replace transceiver. 2. Check operating temperature. 3. Try module in different port to rule out port issue. 4. If within spec but link still fails, the fiber path may have too much loss — check with OTDR.",
category: "optical",
severity: "medium",
},
{
symptom: "QSFP28 breakout mode not working — 4x25G ports not appearing",
cause: "Switch doesn't support breakout on that port, breakout not configured, or breakout cable is faulty.",
solution: "1. Verify switch model supports breakout mode on the specific port. 2. Configure breakout: Cisco 'interface breakout module X port Y map 25g-4x'. 3. Save config and reload if required. 4. Check cable — must be a breakout DAC/AOC, not a trunk cable.",
category: "breakout",
severity: "medium",
},
{
symptom: "DOM (Digital Optical Monitoring) values not showing",
cause: "Transceiver doesn't support DOM, switch firmware too old, or DOM threshold monitoring not enabled.",
solution: "1. Verify transceiver supports DDM/DOM (check datasheet). 2. Use 'show interface transceiver detail' or equivalent command. 3. Update switch firmware if DOM is supported but not displayed. 4. Some third-party optics may not report all DOM fields.",
category: "monitoring",
severity: "low",
},
{
symptom: "100G link working but only at 40G speed",
cause: "QSFP28 module inserted into QSFP+ port, or port speed configured to 40G instead of 100G.",
solution: "1. Verify the port supports 100G (QSFP28). 2. Check port speed configuration — set to 100G explicitly. 3. Verify both ends are using QSFP28 (not QSFP+) transceivers. 4. QSFP28 is backward compatible and will auto-negotiate to 40G in QSFP+ ports.",
category: "speed",
severity: "medium",
},
{
symptom: "Coherent transceiver (400ZR/ZR+) not establishing link",
cause: "Wavelength mismatch between endpoints, incorrect modulation format, chromatic dispersion exceeds module tolerance, or OSNR too low.",
solution: "1. Verify both ends use the same wavelength channel. 2. Check modulation format matches (16QAM for 400ZR). 3. Measure OSNR — must be >20dB for 400ZR. 4. Check fiber distance doesn't exceed 120km (400ZR spec). 5. For ZR+, verify amplifier placement.",
category: "coherent",
severity: "high",
},
{
symptom: "Transceiver works in one switch but not another of the same model",
cause: "Different firmware versions, different port capabilities, or intermittent hardware issue with the transceiver.",
solution: "1. Compare firmware versions on both switches. 2. Try the same port number on both switches. 3. Clean the transceiver contacts with IPA. 4. Check for bent pins in the switch port cage. 5. If the issue persists, the transceiver may have marginal soldering — try a replacement.",
category: "compatibility",
severity: "medium",
},
];
async function main() {
console.log("=== Seeding FAQ + Troubleshooting Embeddings ===\n");
// Seed FAQs
console.log(`Embedding ${FAQ_ENTRIES.length} FAQ entries...\n`);
const BATCH_SIZE = 5;
let faqTotal = 0;
for (let i = 0; i < FAQ_ENTRIES.length; i += BATCH_SIZE) {
const batch = FAQ_ENTRIES.slice(i, i + BATCH_SIZE);
const points = await Promise.all(
batch.map(async (entry) => {
const text = `Q: ${entry.question}\nA: ${entry.answer}`;
const vector = await embed(text);
return {
id: randomUUID(),
vector,
payload: {
question: entry.question,
answer: entry.answer,
category: entry.category,
tags: entry.tags.join(", "),
text,
},
};
}),
);
await upsertPoints("faq_embeddings", points);
faqTotal += points.length;
console.log(` FAQ: ${faqTotal}/${FAQ_ENTRIES.length}`);
}
// Seed Troubleshooting
console.log(`\nEmbedding ${TROUBLESHOOTING_ENTRIES.length} troubleshooting entries...\n`);
let tsTotal = 0;
for (let i = 0; i < TROUBLESHOOTING_ENTRIES.length; i += BATCH_SIZE) {
const batch = TROUBLESHOOTING_ENTRIES.slice(i, i + BATCH_SIZE);
const points = await Promise.all(
batch.map(async (entry) => {
const text = `Symptom: ${entry.symptom}\nCause: ${entry.cause}\nSolution: ${entry.solution}`;
const vector = await embed(text);
return {
id: randomUUID(),
vector,
payload: {
symptom: entry.symptom,
cause: entry.cause,
solution: entry.solution,
category: entry.category,
severity: entry.severity,
text,
},
};
}),
);
await upsertPoints("troubleshooting_embeddings", points);
tsTotal += points.length;
console.log(` Troubleshooting: ${tsTotal}/${TROUBLESHOOTING_ENTRIES.length}`);
}
console.log(`\n=== Done: ${faqTotal} FAQs + ${tsTotal} troubleshooting entries ===`);
}
main().catch((err) => {
console.error("Fatal:", err);
process.exit(1);
});

View File

@ -0,0 +1,99 @@
/**
* Seed FAQ and troubleshooting embeddings in Qdrant from knowledge_base.
*
* Run: npx tsx packages/api/src/embeddings/seed-knowledge-base.ts
*/
import { pool } from "../db/client";
import { embed, upsertPoints, type CollectionName } from "./client";
function kbToText(row: Record<string, unknown>): string {
const parts = [
`Q: ${row.question}`,
`A: ${row.answer}`,
row.subcategory && `Topic: ${row.subcategory}`,
row.applies_to_form_factors && `Form factors: ${(row.applies_to_form_factors as string[]).join(", ")}`,
row.applies_to_speeds && `Speeds: ${(row.applies_to_speeds as string[]).join(", ")}`,
].filter(Boolean);
return parts.join(". ");
}
function collectionForCategory(category: string): CollectionName {
if (category === "troubleshooting" || category === "known_issue") {
return "troubleshooting_embeddings";
}
return "faq_embeddings";
}
async function main(): Promise<void> {
console.log("=== Seeding knowledge_base embeddings ===\n");
const result = await pool.query(
`SELECT id, category, subcategory, question, answer,
applies_to_form_factors, applies_to_speeds, severity, tags
FROM knowledge_base
ORDER BY category, created_at`
);
console.log(`Found ${result.rows.length} knowledge base entries\n`);
const BATCH_SIZE = 5;
let faqCount = 0;
let troubleCount = 0;
for (let i = 0; i < result.rows.length; i += BATCH_SIZE) {
const batch = result.rows.slice(i, i + BATCH_SIZE);
// Group by collection
const byCollection = new Map<CollectionName, typeof batch>();
for (const row of batch) {
const col = collectionForCategory(row.category as string);
if (!byCollection.has(col)) byCollection.set(col, []);
byCollection.get(col)!.push(row);
}
for (const [collection, rows] of byCollection) {
const points = await Promise.all(
rows.map(async (row) => {
const text = kbToText(row);
const vector = await embed(text);
return {
id: row.id,
vector,
payload: {
question: row.question || "",
answer: row.answer || "",
category: row.category || "",
subcategory: row.subcategory || "",
symptom: row.question || "",
cause: row.subcategory || "",
solution: row.answer || "",
severity: row.severity || "info",
form_factors: row.applies_to_form_factors || [],
speeds: row.applies_to_speeds || [],
tags: row.tags || [],
text,
},
};
})
);
await upsertPoints(collection, points);
if (collection === "faq_embeddings") faqCount += points.length;
else troubleCount += points.length;
}
console.log(` Embedded ${Math.min(i + BATCH_SIZE, result.rows.length)}/${result.rows.length} entries (FAQ: ${faqCount}, Troubleshooting: ${troubleCount})`);
}
console.log(`\n=== Done: ${faqCount} FAQ + ${troubleCount} troubleshooting embedded ===`);
await pool.end();
}
main().catch((err) => {
console.error("Fatal:", err);
pool.end();
process.exit(1);
});

View File

@ -0,0 +1,80 @@
/**
* Seed news_embeddings collection from PostgreSQL news_articles table.
*
* Run: npx tsx packages/api/src/embeddings/seed-news.ts
*/
import { pool } from "../db/client";
import { embed, upsertPoints } from "./client";
function articleToText(row: Record<string, unknown>): string {
const parts = [
row.title && `${row.title}`,
row.source && `Source: ${row.source}`,
row.category && `Category: ${row.category}`,
row.summary && `${row.summary}`,
row.full_text && `${String(row.full_text).slice(0, 500)}`,
].filter(Boolean);
return parts.join(". ");
}
async function main() {
console.log("=== Seeding news_embeddings ===\n");
const result = await pool.query(
`SELECT id, title, source_url, source, category, summary, full_text, published_at, relevance_score
FROM news_articles
ORDER BY published_at DESC
LIMIT 500`,
);
console.log(`Found ${result.rows.length} news articles to embed\n`);
if (result.rows.length === 0) {
console.log("No articles found. Run the news scraper first.");
await pool.end();
return;
}
const BATCH_SIZE = 10;
let total = 0;
for (let i = 0; i < result.rows.length; i += BATCH_SIZE) {
const batch = result.rows.slice(i, i + BATCH_SIZE);
const points = await Promise.all(
batch.map(async (row) => {
const text = articleToText(row);
const vector = await embed(text);
return {
id: String(row.id),
vector,
payload: {
title: row.title || "",
url: row.source_url || "",
source: row.source || "",
category: row.category || "",
summary: row.summary || "",
published_at: row.published_at ? new Date(row.published_at).toISOString() : "",
relevance_score: row.relevance_score || 0,
text,
},
};
}),
);
await upsertPoints("news_embeddings", points);
total += points.length;
console.log(` Embedded ${total}/${result.rows.length} articles`);
}
console.log(`\n=== Done: ${total} articles embedded ===`);
await pool.end();
}
main().catch((err) => {
console.error("Fatal:", err);
pool.end();
process.exit(1);
});

View File

@ -0,0 +1,96 @@
/**
* Seed product_embeddings collection in Qdrant from PostgreSQL transceivers.
*
* Creates a rich text representation of each transceiver, embeds it via
* Ollama nomic-embed-text, and stores in Qdrant with payload filters.
*
* Run: npx tsx packages/api/src/embeddings/seed-products.ts
*/
import { pool } from "../db/client";
import { embed, upsertPoints } from "./client";
function transceiverToText(row: Record<string, unknown>): string {
const parts = [
row.standard_name && `${row.standard_name}`,
row.form_factor && `Form factor: ${row.form_factor}`,
row.speed && `Speed: ${row.speed}`,
row.reach_label && `Reach: ${row.reach_label}`,
row.fiber_type && `Fiber: ${row.fiber_type}`,
row.connector && `Connector: ${row.connector}`,
row.wavelengths && `Wavelengths: ${row.wavelengths}`,
row.wdm_type && `WDM: ${row.wdm_type}`,
row.category && `Category: ${row.category}`,
row.coherent && `Coherent optics`,
row.power_consumption_w && `Power: ${row.power_consumption_w}W`,
row.temp_range && `Temperature: ${row.temp_range}`,
row.vendor_name && `Vendor: ${row.vendor_name}`,
row.description && `${row.description}`,
].filter(Boolean);
return parts.join(". ");
}
async function main() {
console.log("=== Seeding product_embeddings ===\n");
const result = await pool.query(
`SELECT t.id, t.slug, t.standard_name, t.form_factor, t.speed, t.speed_gbps,
t.reach_label, t.reach_meters, t.fiber_type, t.connector,
t.wavelengths, t.wdm_type, t.coherent, t.power_consumption_w,
t.temp_range, t.category, t.notes as description,
v.name as vendor_name
FROM transceivers t
LEFT JOIN vendors v ON v.id = t.vendor_id
ORDER BY t.speed_gbps DESC`
);
console.log(`Found ${result.rows.length} transceivers to embed\n`);
const BATCH_SIZE = 10;
let total = 0;
for (let i = 0; i < result.rows.length; i += BATCH_SIZE) {
const batch = result.rows.slice(i, i + BATCH_SIZE);
const points = await Promise.all(
batch.map(async (row) => {
const text = transceiverToText(row);
const vector = await embed(text);
return {
id: row.id,
vector,
payload: {
slug: row.slug,
standard_name: row.standard_name || "",
form_factor: row.form_factor || "",
speed: row.speed || "",
speed_gbps: parseFloat(row.speed_gbps) || 0,
reach_label: row.reach_label || "",
reach_meters: row.reach_meters || 0,
fiber_type: row.fiber_type || "",
connector: row.connector || "",
wdm_type: row.wdm_type || "",
category: row.category || "",
coherent: row.coherent || false,
vendor: row.vendor_name || "",
text,
},
};
})
);
await upsertPoints("product_embeddings", points);
total += points.length;
console.log(` Embedded ${total}/${result.rows.length} transceivers`);
}
console.log(`\n=== Done: ${total} products embedded ===`);
await pool.end();
}
main().catch((err) => {
console.error("Fatal:", err);
pool.end();
process.exit(1);
});

View File

@ -0,0 +1,273 @@
/**
* Hype Cycle Data Enrichment Real metrics from scraped data
*
* Computes PhaseMetrics overrides from actual database observations:
* - vendorCount: How many vendors sell this speed class
* - price trends: ASP decline rate from price_observations
* - catalog density: Number of SKUs per speed class (market maturity signal)
* - product diversity: Form factor and reach variety
*/
import { pool } from "../db/client";
import type { PhaseMetrics } from "./norton-bass";
interface SpeedClassMetrics {
speedGbps: number;
vendorCount: number;
skuCount: number;
avgPrice?: number;
minPrice?: number;
maxPrice?: number;
priceCount: number;
formFactors: string[];
reachVariants: number;
}
/**
* Query real vendor/product counts per speed class from the database.
*/
export async function getSpeedClassMetrics(): Promise<ReadonlyArray<SpeedClassMetrics>> {
const result = await pool.query(`
SELECT
t.speed_gbps,
COUNT(DISTINCT t.vendor_id) AS vendor_count,
COUNT(DISTINCT t.id) AS sku_count,
ARRAY_AGG(DISTINCT t.form_factor) FILTER (WHERE t.form_factor IS NOT NULL AND t.form_factor != '') AS form_factors,
COUNT(DISTINCT t.reach_label) FILTER (WHERE t.reach_label IS NOT NULL AND t.reach_label != '') AS reach_variants
FROM transceivers t
WHERE t.speed_gbps > 0
GROUP BY t.speed_gbps
ORDER BY t.speed_gbps
`);
const priceResult = await pool.query(`
SELECT
t.speed_gbps,
AVG(po.price) AS avg_price,
MIN(po.price) AS min_price,
MAX(po.price) AS max_price,
COUNT(*) AS price_count
FROM price_observations po
JOIN transceivers t ON t.id = po.transceiver_id
WHERE t.speed_gbps > 0
GROUP BY t.speed_gbps
`);
const priceMap = new Map<number, { avg: number; min: number; max: number; count: number }>();
for (const row of priceResult.rows) {
priceMap.set(Number(row.speed_gbps), {
avg: parseFloat(row.avg_price),
min: parseFloat(row.min_price),
max: parseFloat(row.max_price),
count: parseInt(row.price_count),
});
}
return result.rows.map((row) => {
const speedGbps = Number(row.speed_gbps);
const priceData = priceMap.get(speedGbps);
return {
speedGbps,
vendorCount: parseInt(row.vendor_count),
skuCount: parseInt(row.sku_count),
avgPrice: priceData?.avg,
minPrice: priceData?.min,
maxPrice: priceData?.max,
priceCount: priceData?.count ?? 0,
formFactors: row.form_factors || [],
reachVariants: parseInt(row.reach_variants),
};
});
}
/**
* Convert raw speed-class metrics into PhaseMetrics overrides.
* These override the model-estimated values with real data.
*/
export function metricsToPhaseOverrides(
metrics: SpeedClassMetrics,
totalMarketSkus: number,
): { -readonly [K in keyof PhaseMetrics]?: PhaseMetrics[K] } {
const overrides: { -readonly [K in keyof PhaseMetrics]?: PhaseMetrics[K] } = {};
// Vendor count — direct from data
overrides.vendorCount = metrics.vendorCount;
// Vendor trend — estimate from catalog density
// More SKUs + more vendors = increasing; few = decreasing
if (metrics.vendorCount >= 4 && metrics.skuCount > 50) {
overrides.vendorTrend = "stable";
} else if (metrics.vendorCount >= 2) {
overrides.vendorTrend = "increasing";
} else {
overrides.vendorTrend = "decreasing";
}
// Shipment share proxy — SKU count relative to total market
overrides.shipmentShare = Math.min(0.5, metrics.skuCount / Math.max(1, totalMarketSkus));
// Interop level — more reach variants and form factors = better interop
const ffDiversity = metrics.formFactors.length;
const reachDiversity = metrics.reachVariants;
overrides.interopLevel = Math.min(100, ffDiversity * 15 + reachDiversity * 8);
return overrides;
}
/**
* Get enriched PhaseMetrics for all speed classes.
* Returns a map of speedGbps -> partial PhaseMetrics overrides.
*/
export async function getDataDrivenOverrides(): Promise<Map<number, { -readonly [K in keyof PhaseMetrics]?: PhaseMetrics[K] }>> {
const allMetrics = await getSpeedClassMetrics();
const totalSkus = allMetrics.reduce((sum, m) => sum + m.skuCount, 0);
const overridesMap = new Map<number, { -readonly [K in keyof PhaseMetrics]?: PhaseMetrics[K] }>();
for (const metrics of allMetrics) {
overridesMap.set(metrics.speedGbps, metricsToPhaseOverrides(metrics, totalSkus));
}
return overridesMap;
}
/**
* Revenue lifecycle prediction per speed class.
*
* Uses scraped price data + Bass diffusion to estimate:
* - Peak revenue year
* - Revenue duration (years above 50% of peak)
* - Current revenue trajectory
*/
export interface RevenueLifecycle {
speedGbps: number;
technology: string;
currentAvgPrice?: number;
estimatedPeakRevenueYear: number;
estimatedDeclineStartYear: number;
revenueHalfLifeYears: number;
currentPhase: "growing" | "peaking" | "declining" | "legacy";
revenueIndex: number; // 0-100, relative to estimated peak
}
export function computeRevenueLifecycle(
speedGbps: number,
techName: string,
introYear: number,
peakYear: number,
currentYear: number,
avgPrice?: number,
): RevenueLifecycle {
// Revenue = Price × Volume. Price declines while volume grows.
// Peak revenue happens ~2 years before peak volume (when price×volume is maximized)
const peakRevenueYear = Math.round(peakYear - 2);
const declineStartYear = peakYear + 2;
const halfLife = Math.round((peakYear - introYear) * 0.7);
const yearsFromPeak = currentYear - peakRevenueYear;
let currentPhase: RevenueLifecycle["currentPhase"];
if (currentYear < peakRevenueYear - 2) currentPhase = "growing";
else if (currentYear <= peakRevenueYear + 2) currentPhase = "peaking";
else if (currentYear <= declineStartYear + halfLife) currentPhase = "declining";
else currentPhase = "legacy";
// Revenue index: bell curve centered on peakRevenueYear
const sigma = halfLife / 1.5;
const revenueIndex = Math.round(100 * Math.exp(-0.5 * Math.pow(yearsFromPeak / sigma, 2)));
return {
speedGbps,
technology: techName,
currentAvgPrice: avgPrice,
estimatedPeakRevenueYear: peakRevenueYear,
estimatedDeclineStartYear: declineStartYear,
revenueHalfLifeYears: halfLife,
currentPhase,
revenueIndex,
};
}
/**
* Regional adoption model.
* Applies lag coefficients per region based on industry research.
*/
export interface RegionalAdoption {
region: string;
lagYears: number;
marketSharePct: number;
adoptionPhase: string;
estimatedPeakYear: number;
}
/**
* Regional lag coefficients calibrated from research (2026-03-28).
* Sources: LightCounting, vendor earnings, OFC market sessions, Chinese IPO prospectuses.
* Lag in years (converted from quarters: NA=0, CN=0.5Q0.125yr, EU=4Q=1yr, etc.)
*/
const REGIONAL_LAGS: ReadonlyArray<{
region: string;
lagYears: number;
marketSharePct: number;
priceIndex: number;
segmentMix: { hyperscaler: number; telco: number; enterprise: number };
}> = [
{
region: "North America (Hyperscale)",
lagYears: 0,
marketSharePct: 32,
priceIndex: 1.0,
segmentMix: { hyperscaler: 0.65, telco: 0.20, enterprise: 0.15 },
},
{
region: "China (BAT/Hyperscale)",
lagYears: 0.125, // 0.5 quarters = near-parity, sometimes leads on volume
marketSharePct: 38,
priceIndex: 0.58, // 42% cheaper domestically
segmentMix: { hyperscaler: 0.50, telco: 0.35, enterprise: 0.15 },
},
{
region: "APAC (ex-China)",
lagYears: 0.625, // 2.5 quarters
marketSharePct: 13,
priceIndex: 1.0,
segmentMix: { hyperscaler: 0.35, telco: 0.40, enterprise: 0.25 },
},
{
region: "Europe",
lagYears: 1.0, // 4 quarters — telco procurement cycles, regulatory compliance
marketSharePct: 14,
priceIndex: 1.12, // 12% premium (CE/RoHS compliance, smaller volumes, channel markup)
segmentMix: { hyperscaler: 0.25, telco: 0.45, enterprise: 0.30 },
},
{
region: "Rest of World",
lagYears: 1.5, // 6 quarters
marketSharePct: 3,
priceIndex: 1.05,
segmentMix: { hyperscaler: 0.20, telco: 0.50, enterprise: 0.30 },
},
];
export function computeRegionalAdoption(
techPeakYear: number,
currentYear: number,
techName: string,
): ReadonlyArray<RegionalAdoption> {
return REGIONAL_LAGS.map(({ region, lagYears, marketSharePct }) => {
const regionalPeak = techPeakYear + lagYears;
const yearsToPeak = regionalPeak - currentYear;
let adoptionPhase: string;
if (yearsToPeak > 5) adoptionPhase = "Pre-adoption";
else if (yearsToPeak > 2) adoptionPhase = "Early Adoption";
else if (yearsToPeak > -1) adoptionPhase = "Growth";
else if (yearsToPeak > -4) adoptionPhase = "Mature";
else adoptionPhase = "Declining";
return {
region,
lagYears,
marketSharePct,
adoptionPhase,
estimatedPeakYear: Math.round(regionalPeak * 2) / 2, // Round to half-year
};
});
}

View File

@ -0,0 +1,484 @@
/**
* Norton-Bass Multigenerational Diffusion Model
*
* Mathematical engine for forecasting transceiver technology adoption.
* Based on Norton & Bass (1987, Management Science).
*
* Key equations:
* Bass: f(t) / [1 - F(t)] = p + q * F(t)
* Logistic: S(t) = L / (1 + e^(-k(t - t0)))
*
* Parameters:
* p = Innovation coefficient (~0.03 for network hardware)
* q = Imitation coefficient (~0.30.5)
* m = Total market potential (addressable port shipments)
*/
/** Technology generation definition */
export interface TechGeneration {
readonly name: string; // e.g. "100G QSFP28"
readonly speedGbps: number;
readonly formFactor: string;
readonly introYear: number; // Year first shipped commercially
readonly peakYear: number; // Year of peak shipments (estimated)
readonly p: number; // Innovation coefficient
readonly q: number; // Imitation coefficient
readonly m: number; // Market potential (millions of ports)
readonly k: number; // Logistic growth rate
readonly t0: number; // Inflection point year
}
/** Hype Cycle phases per Gartner methodology */
export type HypeCyclePhase =
| "INNOVATION_TRIGGER"
| "PEAK_OF_INFLATED_EXPECTATIONS"
| "TROUGH_OF_DISILLUSIONMENT"
| "SLOPE_OF_ENLIGHTENMENT"
| "PLATEAU_OF_PRODUCTIVITY"
| "LEGACY_DECLINE";
/** Result of a hype cycle computation */
export interface HypeCycleResult {
readonly technology: string;
readonly phase: HypeCyclePhase;
readonly phaseLabel: string;
readonly positionPct: number; // 0100 on the hype cycle curve
readonly adoptionPct: number; // Current market adoption %
readonly compositeScore: number; // 0100
readonly forecast: AdoptionForecast;
readonly metrics: PhaseMetrics;
}
export interface AdoptionForecast {
readonly currentYear: number;
readonly yearsToPlateauFromNow: number;
readonly peakShipmentYear: number;
readonly cumulativeAdoptionPct: number;
readonly yearlyAdoptionPct: number;
readonly fiveYearProjection: ReadonlyArray<{ year: number; adoptionPct: number; phase: HypeCyclePhase }>;
}
export interface PhaseMetrics {
readonly shipmentShare: number; // 01
readonly aspDeclineRate: number; // % per year
readonly standardsMaturity: number; // 0100
readonly interopLevel: number; // 0100
readonly vendorCount: number;
readonly vendorTrend: "increasing" | "stable" | "decreasing";
readonly mediaHypeIndex: number; // 0100
}
// ============================================================
// Known technology generations (seed data, March 2026)
// ============================================================
export const TECH_GENERATIONS: ReadonlyArray<TechGeneration> = [
{
name: "1G SFP",
speedGbps: 1,
formFactor: "SFP",
introYear: 2001,
peakYear: 2012,
p: 0.03, q: 0.38, m: 500, k: 0.45, t0: 2008,
},
{
name: "10G SFP+",
speedGbps: 10,
formFactor: "SFP+",
introYear: 2006,
peakYear: 2018,
p: 0.03, q: 0.42, m: 600, k: 0.50, t0: 2014,
},
{
name: "40G QSFP+",
speedGbps: 40,
formFactor: "QSFP+",
introYear: 2010,
peakYear: 2019,
p: 0.025, q: 0.35, m: 150, k: 0.40, t0: 2016,
},
{
name: "25G SFP28",
speedGbps: 25,
formFactor: "SFP28",
introYear: 2015,
peakYear: 2022,
p: 0.04, q: 0.45, m: 200, k: 0.55, t0: 2019,
},
{
name: "100G QSFP28",
speedGbps: 100,
formFactor: "QSFP28",
introYear: 2014,
peakYear: 2024,
p: 0.03, q: 0.40, m: 400, k: 0.48, t0: 2020,
},
{
name: "400G QSFP-DD",
speedGbps: 400,
formFactor: "QSFP-DD",
introYear: 2020,
peakYear: 2027,
p: 0.035, q: 0.50, m: 300, k: 0.52, t0: 2025,
},
{
name: "800G OSFP",
speedGbps: 800,
formFactor: "OSFP",
introYear: 2023,
peakYear: 2029,
p: 0.04, q: 0.55, m: 250, k: 0.55, t0: 2027,
},
{
name: "1.6T OSFP-XD",
speedGbps: 1600,
formFactor: "OSFP-XD",
introYear: 2025,
peakYear: 2032,
p: 0.03, q: 0.45, m: 180, k: 0.48, t0: 2030,
},
];
// Special tech entries (not speed-generational)
export const SPECIAL_TECHS: ReadonlyArray<TechGeneration> = [
{
name: "CPO",
speedGbps: 1600,
formFactor: "CPO",
introYear: 2024,
peakYear: 2033,
p: 0.02, q: 0.30, m: 50, k: 0.35, t0: 2031,
},
{
name: "LPO",
speedGbps: 800,
formFactor: "LPO",
introYear: 2024,
peakYear: 2029,
p: 0.035, q: 0.48, m: 100, k: 0.50, t0: 2027,
},
{
name: "400ZR Coherent",
speedGbps: 400,
formFactor: "QSFP-DD",
introYear: 2021,
peakYear: 2026,
p: 0.04, q: 0.50, m: 80, k: 0.55, t0: 2024,
},
];
// ============================================================
// Core Bass Diffusion Functions
// ============================================================
/**
* Bass diffusion: cumulative adoption F(t)
*
* F(t) = (1 - e^(-(p+q)*t)) / (1 + (q/p)*e^(-(p+q)*t))
*/
export function bassCumulativeAdoption(
t: number,
p: number,
q: number,
): number {
const pq = p + q;
const exp = Math.exp(-pq * t);
return (1 - exp) / (1 + (q / p) * exp);
}
/**
* Bass diffusion: instantaneous adoption rate f(t)
*
* f(t) = ((p+q)^2 / q) * (e^(-(p+q)*t) / (1 + (q/p)*e^(-(p+q)*t))^2)
*/
export function bassAdoptionRate(
t: number,
p: number,
q: number,
): number {
const pq = p + q;
const exp = Math.exp(-pq * t);
const denominator = 1 + (q / p) * exp;
return (pq * pq / q) * (exp / (denominator * denominator));
}
/**
* Logistic adoption curve: S(t) = L / (1 + e^(-k*(t - t0)))
*/
export function logisticAdoption(
year: number,
L: number,
k: number,
t0: number,
): number {
return L / (1 + Math.exp(-k * (year - t0)));
}
// ============================================================
// Phase Classification Engine
// ============================================================
export function classifyPhase(metrics: PhaseMetrics): HypeCyclePhase {
const { shipmentShare, aspDeclineRate, standardsMaturity, vendorTrend, mediaHypeIndex } = metrics;
if (shipmentShare < 0.01 && standardsMaturity < 30) {
return "INNOVATION_TRIGGER";
}
if (shipmentShare < 0.05 && mediaHypeIndex > 70 && vendorTrend === "increasing") {
return "PEAK_OF_INFLATED_EXPECTATIONS";
}
if (aspDeclineRate > 30 && vendorTrend === "decreasing" && mediaHypeIndex < 40) {
return "TROUGH_OF_DISILLUSIONMENT";
}
if (
shipmentShare >= 0.05 && shipmentShare <= 0.30 &&
aspDeclineRate >= 10 && aspDeclineRate <= 25 &&
(vendorTrend === "stable" || vendorTrend === "increasing")
) {
return "SLOPE_OF_ENLIGHTENMENT";
}
if (shipmentShare > 0.30 && aspDeclineRate < 10) {
return "PLATEAU_OF_PRODUCTIVITY";
}
// Fallback: use composite score approach
return compositePhaseClassification(metrics);
}
/**
* Composite score phase classification
*
* Phase_Score = 0.30 * Normalize(PortShipment_share)
* + 0.20 * Normalize(ASP_decline_rate)
* + 0.15 * Normalize(Standards_maturity)
* + 0.15 * Normalize(InteropValidation_level)
* + 0.10 * Normalize(VendorCount_trajectory)
* + 0.10 * Normalize(MediaSentiment_score)
*/
function compositePhaseClassification(metrics: PhaseMetrics): HypeCyclePhase {
const score =
0.30 * normalize(metrics.shipmentShare, 0, 0.5) +
0.20 * normalize(metrics.aspDeclineRate, 0, 50) +
0.15 * normalize(metrics.standardsMaturity, 0, 100) +
0.15 * normalize(metrics.interopLevel, 0, 100) +
0.10 * (metrics.vendorTrend === "increasing" ? 0.3 : metrics.vendorTrend === "stable" ? 0.6 : 0.9) +
0.10 * normalize(100 - metrics.mediaHypeIndex, 0, 100);
if (score < 0.15) return "INNOVATION_TRIGGER";
if (score < 0.30) return "PEAK_OF_INFLATED_EXPECTATIONS";
if (score < 0.45) return "TROUGH_OF_DISILLUSIONMENT";
if (score < 0.70) return "SLOPE_OF_ENLIGHTENMENT";
return "PLATEAU_OF_PRODUCTIVITY";
}
function normalize(value: number, min: number, max: number): number {
return Math.max(0, Math.min(1, (value - min) / (max - min)));
}
// ============================================================
// Phase position on the hype cycle curve (0100)
// ============================================================
const PHASE_POSITIONS: Record<HypeCyclePhase, [number, number]> = {
INNOVATION_TRIGGER: [0, 15],
PEAK_OF_INFLATED_EXPECTATIONS: [15, 30],
TROUGH_OF_DISILLUSIONMENT: [30, 50],
SLOPE_OF_ENLIGHTENMENT: [50, 80],
PLATEAU_OF_PRODUCTIVITY: [80, 95],
LEGACY_DECLINE: [95, 100],
};
const PHASE_LABELS: Record<HypeCyclePhase, string> = {
INNOVATION_TRIGGER: "Innovation Trigger",
PEAK_OF_INFLATED_EXPECTATIONS: "Peak of Inflated Expectations",
TROUGH_OF_DISILLUSIONMENT: "Trough of Disillusionment",
SLOPE_OF_ENLIGHTENMENT: "Slope of Enlightenment",
PLATEAU_OF_PRODUCTIVITY: "Plateau of Productivity",
LEGACY_DECLINE: "Legacy / Decline",
};
// ============================================================
// Main computation: compute hype cycle for a technology
// ============================================================
export function computeHypeCycle(
tech: TechGeneration,
currentYear: number = new Date().getFullYear(),
overrideMetrics?: Partial<PhaseMetrics>,
): HypeCycleResult {
// Time since introduction
const t = currentYear - tech.introYear;
const tNorm = Math.max(0, t);
// Bass diffusion adoption
const cumulativeAdoption = bassCumulativeAdoption(tNorm, tech.p, tech.q);
const adoptionRate = bassAdoptionRate(tNorm, tech.p, tech.q);
// Logistic adoption (port shipments)
const logisticShipments = logisticAdoption(currentYear, tech.m, tech.k, tech.t0);
const shipmentShare = logisticShipments / 1000; // Normalize to 01 range (1000M total market)
// Estimate metrics from model
const yearsAfterIntro = currentYear - tech.introYear;
const yearsToPeak = tech.peakYear - tech.introYear;
const progressRatio = yearsAfterIntro / yearsToPeak;
const metrics: PhaseMetrics = {
shipmentShare: overrideMetrics?.shipmentShare ?? Math.min(0.5, shipmentShare),
aspDeclineRate: overrideMetrics?.aspDeclineRate ?? estimateAspDecline(progressRatio),
standardsMaturity: overrideMetrics?.standardsMaturity ?? estimateStandardsMaturity(progressRatio),
interopLevel: overrideMetrics?.interopLevel ?? estimateInteropLevel(progressRatio),
vendorCount: overrideMetrics?.vendorCount ?? estimateVendorCount(progressRatio),
vendorTrend: overrideMetrics?.vendorTrend ?? estimateVendorTrend(progressRatio),
mediaHypeIndex: overrideMetrics?.mediaHypeIndex ?? estimateMediaHype(progressRatio),
};
const phase = classifyPhase(metrics);
// Check for legacy/decline
const finalPhase = (currentYear > tech.peakYear + 8 && metrics.shipmentShare < 0.05)
? "LEGACY_DECLINE"
: phase;
// Position on curve (0-100)
const [phaseMin, phaseMax] = PHASE_POSITIONS[finalPhase];
const intraPhaseProgress = Math.min(1, cumulativeAdoption);
const positionPct = phaseMin + (phaseMax - phaseMin) * intraPhaseProgress;
// Composite score
const compositeScore = Math.round(
0.30 * normalize(metrics.shipmentShare, 0, 0.5) * 100 +
0.20 * normalize(metrics.aspDeclineRate, 0, 50) * 100 +
0.15 * normalize(metrics.standardsMaturity, 0, 100) * 100 +
0.15 * normalize(metrics.interopLevel, 0, 100) * 100 +
0.10 * (metrics.vendorTrend === "increasing" ? 30 : metrics.vendorTrend === "stable" ? 60 : 90) +
0.10 * (100 - metrics.mediaHypeIndex)
);
// 5-year forecast
const fiveYearProjection = Array.from({ length: 5 }, (_, i) => {
const futureYear = currentYear + i + 1;
const futureT = futureYear - tech.introYear;
const futureAdoption = bassCumulativeAdoption(futureT, tech.p, tech.q);
const futureProgressRatio = (futureYear - tech.introYear) / yearsToPeak;
const futureMetrics: PhaseMetrics = {
...metrics,
shipmentShare: Math.min(0.5, logisticAdoption(futureYear, tech.m, tech.k, tech.t0) / 1000),
aspDeclineRate: estimateAspDecline(futureProgressRatio),
vendorTrend: estimateVendorTrend(futureProgressRatio),
};
return {
year: futureYear,
adoptionPct: Math.round(futureAdoption * 100),
phase: (futureYear > tech.peakYear + 8 && futureMetrics.shipmentShare < 0.05)
? "LEGACY_DECLINE" as HypeCyclePhase
: classifyPhase(futureMetrics),
};
});
return {
technology: tech.name,
phase: finalPhase,
phaseLabel: PHASE_LABELS[finalPhase],
positionPct: Math.round(positionPct),
adoptionPct: Math.round(cumulativeAdoption * 100),
compositeScore,
forecast: {
currentYear,
yearsToPlateauFromNow: Math.max(0, tech.peakYear + 3 - currentYear),
peakShipmentYear: tech.peakYear,
cumulativeAdoptionPct: Math.round(cumulativeAdoption * 100),
yearlyAdoptionPct: Math.round(adoptionRate * 100),
fiveYearProjection,
},
metrics,
};
}
// ============================================================
// Estimation heuristics (from model parameters)
// ============================================================
function estimateAspDecline(progressRatio: number): number {
// ASP decline accelerates through Trough, stabilizes at Plateau
if (progressRatio < 0.3) return 5; // Early: slow decline
if (progressRatio < 0.6) return 35; // Peak/Trough: rapid decline
if (progressRatio < 1.0) return 15; // Slope: moderate decline
return 5; // Plateau: minimal decline
}
function estimateStandardsMaturity(progressRatio: number): number {
if (progressRatio < 0.2) return 20; // Draft standards
if (progressRatio < 0.5) return 60; // Published, some revisions
if (progressRatio < 0.8) return 85; // Mature standards
return 95; // Fully mature
}
function estimateInteropLevel(progressRatio: number): number {
if (progressRatio < 0.3) return 25; // Limited interop testing
if (progressRatio < 0.6) return 55; // Growing interop
if (progressRatio < 0.9) return 80; // Broad interop
return 95; // Universal interop
}
function estimateVendorCount(progressRatio: number): number {
if (progressRatio < 0.3) return 5;
if (progressRatio < 0.6) return 15;
if (progressRatio < 1.0) return 25;
return 20; // Consolidation
}
function estimateVendorTrend(progressRatio: number): "increasing" | "stable" | "decreasing" {
if (progressRatio < 0.5) return "increasing";
if (progressRatio < 1.2) return "stable";
return "decreasing";
}
function estimateMediaHype(progressRatio: number): number {
// Hype peaks early (Peak of Inflated Expectations)
if (progressRatio < 0.15) return 40; // Innovation: moderate buzz
if (progressRatio < 0.30) return 85; // Peak: maximum hype
if (progressRatio < 0.50) return 25; // Trough: hype collapse
if (progressRatio < 0.80) return 50; // Slope: balanced coverage
return 30; // Plateau: boring = good
}
// ============================================================
// Compute all technologies at once
// ============================================================
export function computeAllHypeCycles(
currentYear: number = new Date().getFullYear(),
): ReadonlyArray<HypeCycleResult> {
const allTechs = [...TECH_GENERATIONS, ...SPECIAL_TECHS];
return allTechs.map((tech) => computeHypeCycle(tech, currentYear));
}
export function findTechnology(query: string): TechGeneration | undefined {
const q = query.toLowerCase().trim();
const allTechs = [...TECH_GENERATIONS, ...SPECIAL_TECHS];
// 1. Exact name match
const exact = allTechs.find((t) => t.name.toLowerCase() === q);
if (exact) return exact;
// 2. Name contains query or query contains name
const nameMatch = allTechs.find((t) => t.name.toLowerCase().includes(q));
if (nameMatch) return nameMatch;
// 3. Fuzzy: match by speed prefix (e.g. "400G", "1.6T", "800G")
const speedMatch = q.match(/^(\d+(?:\.\d+)?)\s*(g|t)\b/i);
if (speedMatch) {
const num = parseFloat(speedMatch[1]);
const unit = speedMatch[2].toLowerCase();
const targetGbps = unit === "t" ? num * 1000 : num;
const found = allTechs.find((t) => t.speedGbps === targetGbps);
if (found) return found;
}
// 4. Match by form factor
return allTechs.find((t) => q.includes(t.formFactor.toLowerCase()));
}

99
packages/api/src/index.ts Normal file
View File

@ -0,0 +1,99 @@
import express from "express";
import cors from "cors";
import helmet from "helmet";
import rateLimit from "express-rate-limit";
import { join } from "path";
import { cfg } from "./config";
import { transceiverRouter } from "./routes/transceivers";
import { switchRouter } from "./routes/switches";
import { vendorRouter } from "./routes/vendors";
import { standardRouter } from "./routes/standards";
import { healthRouter } from "./routes/health";
import { hypeCycleRouter } from "./routes/hype-cycle";
import { searchRouter } from "./routes/search";
import { documentRouter } from "./routes/documents";
import { blogRouter } from "./routes/blog";
import { finderRouter } from "./routes/finder";
import { competitorRouter } from "./routes/competitor-alerts";
import { forecastRouter } from "./routes/forecast";
import { transportRouter } from "./routes/transport";
const app = express();
// Trust Cloudflare proxy (required for express-rate-limit with X-Forwarded-For)
app.set("trust proxy", 1);
// Middleware
app.use(helmet({ contentSecurityPolicy: false }));
app.use(cors());
app.use(express.json());
app.use(
rateLimit({
windowMs: 60 * 1000,
max: 200,
standardHeaders: true,
legacyHeaders: false,
})
);
// Routes
app.use("/api/transceivers", transceiverRouter);
app.use("/api/switches", switchRouter);
app.use("/api/vendors", vendorRouter);
app.use("/api/standards", standardRouter);
app.use("/api/health", healthRouter);
app.use("/api/hype-cycle", hypeCycleRouter);
app.use("/api/search", searchRouter);
app.use("/api/documents", documentRouter);
app.use("/api/blog", blogRouter);
app.use("/api/finder", finderRouter);
app.use("/api/competitor-alerts", competitorRouter);
app.use("/api/forecast", forecastRouter);
app.use("/api/transport", transportRouter);
// Dashboard (static HTML)
app.use("/dashboard", express.static(join(__dirname, "..", "..", "dashboard")));
// Root — redirect to dashboard
app.get("/", (_req, res) => {
res.redirect("/dashboard/");
});
// API info
app.get("/api", (_req, res) => {
res.json({
name: "Transceiver Intelligence Platform",
version: "0.2.0-alpha.1",
endpoints: [
"GET /api/transceivers?q=&form_factor=&speed=&category=&fiber_type=&wdm_type=&coherent=",
"GET /api/transceivers/:id",
"GET /api/switches?q=&category=",
"GET /api/switches/:id",
"GET /api/switches/:id/compatibility",
"GET /api/vendors?type=",
"GET /api/standards?speed=",
"GET /api/health",
"GET /api/hype-cycle",
"GET /api/hype-cycle/:tech",
"GET /api/search?q=&collection=&limit=",
"GET /api/search/products?q=&form_factor=&speed_gbps=&fiber_type=",
"GET /api/search/documents?q=&doc_type=&vendor=&collection=",
"GET /api/search/news?q=&source=",
"GET /api/search/stats",
"POST /api/documents/process {url, title?, doc_type?, vendor?, collection?}",
"GET /api/documents",
"GET /api/documents/:id",
"POST /api/blog/generate {topic, speed?, form_factor?, use_case?}",
"GET /api/blog",
"GET /api/blog/:id",
"PUT /api/blog/:id/status {status: draft|review|approved|published}",
],
});
});
// Start
app.listen(cfg.port, () => {
console.log(`\n TIP API running on http://localhost:${cfg.port}`);
console.log(` Environment: ${cfg.nodeEnv}`);
console.log(` Database: ${cfg.db.host}:${cfg.db.port}/${cfg.db.database}\n`);
});

View File

@ -0,0 +1,514 @@
/**
* Blog generation prompt templates v2 (2026-03-28 overhaul)
*
* Complete rewrite based on field engineer feedback.
* Previous version produced shallow template text.
* This version enforces:
* - Real-world scenarios with technical depth
* - Power budget calculations (mandatory)
* - CLI examples and DOM readings
* - Cause-effect explanations, not bullet dumps
* - Product integration only when contextually relevant
* - Decision logic / diagnosis frameworks
*
* Multi-pass pipeline:
* 1. MASTER pass Full article generation with structure enforcement
* 2. DEPTH pass Add concrete values, power budget, CLI examples
* 3. ANTI_GENERIC pass Kill marketing language, fix intro
* 4. QUALITY_CONTROL pass Final validation against quality gates
* 5. PROCUREMENT pass (optional) Add cost context for sales audience
*
* Voice: Senior optical network engineer with 10+ years field experience.
* NOT a content writer. NOT marketing. NOT generic AI.
*/
// ═══════════════════════════════════════════════════════
// SYSTEM PROMPT — Persona & Rules
// ═══════════════════════════════════════════════════════
export const SYSTEM_PROMPT = `You are a senior optical network engineer and technical writer with real field experience in data center, ISP, and DWDM environments.
Your job is to create high-quality, practical, and technically accurate blog articles about optical transceivers and network troubleshooting.
Do NOT write generic, shallow, or marketing-style content.
Do NOT use buzzwords, filler phrases, or vague explanations.
Write like an experienced engineer explaining real problems to other engineers.
Your content must:
- Be technically correct and precise
- Include real-world scenarios
- Provide actionable troubleshooting steps
- Explain WHY issues happen, not just WHAT to do
- Include measurements, thresholds, and interpretation
- Reflect field experience (NOC, deployment, escalation cases)
Reference values you know from experience:
- SFP+ SR: Tx -8.2 to +0.5 dBm, Rx sensitivity -18.0 dBm, alarm below -11.0 dBm
- QSFP28 LR4: Tx -4.3 to +4.5 dBm, Rx sensitivity -13.7 dBm
- QSFP-DD DR4: Tx -2.9 to +3.0 dBm per lane, Rx sensitivity -7.7 dBm
- 400ZR: Tx -10.0 to +2.0 dBm, Rx sensitivity -21.0 dBm, OSNR > 20 dB required
- BER: pre-FEC < 2.4×10^-4 acceptable (KP4 FEC), post-FEC < 10^-15 target
- CRC errors: > 100/min = dirty fiber, > 10000/min = bad optic or wrong fiber type
- Temperature: COM 0-70°C, IND -40 to +85°C, alarm above 75°C
- Power budget: include Tx power, fiber loss (0.35 dB/km SMF @ 1310nm, 0.22 dB/km @ 1550nm), connector loss (0.3 dB each), splice loss (0.1 dB), margin (3 dB recommended)
CLI examples to use where relevant:
show interface transceiver details
show interface counters errors
show interfaces diagnostics optics
show ip interface brief
show logging | include transceiver|optics|SFP
ANTI-PATTERNS (STRICTLY FORBIDDEN):
- Generic introductions ("In today's fast-paced world", "The optical transceiver market continues")
- Empty phrases ("optimize", "leverage", "enhance", "plays a key role", "increasingly important")
- Bullet lists without explanation
- Random product dumps unconnected to the text
- Copy-paste datasheet language
- Surface-level explanations without cause-effect reasoning
- Placeholders, TODO markers, or unfinished sections
GOOD style example:
"If Tx drops below -10 dBm on a module rated for -8.2 to +0.5, the laser is degrading. You have maybe 2-4 weeks before it dies completely. Replace now during a maintenance window — don't wait for the 2 AM page."
BAD style to avoid:
"Low power may indicate issues with the transceiver module."
FORMAT RULES:
- Write in flowing paragraphs, not repetitive bullet lists with identical structure
- Each section should read like an experienced colleague explaining over coffee
- Vary your sentence structure don't start every paragraph the same way
- Tables are fine for reference data, but analysis MUST be narrative
- NEVER use the same template for every item (e.g., don't list "Deployment Reality / Interop / Price / Readiness / Issues" for every technology group and compare instead)
TOPIC SEPARATION (CRITICAL):
- Strategy/investment articles MUST NOT contain troubleshooting content
- Troubleshooting articles MUST NOT contain investment strategy
- Comparison articles focus on product differences, not operations
- Every article has ONE clear purpose. Do not mix purposes.
OPINION RULES:
- Have a clear point of view. Neutral advice is worthless.
- Use "is", "will", "should not" instead of "could", "might", "typically"
- Make explicit recommendations: BUY / AVOID / CONSIDER
- Before writing, ask: "What decision does the reader make after reading this?"
- Then write to support exactly that decision.`;
// ═══════════════════════════════════════════════════════
// MASTER PROMPTS — Per Topic Type
// ═══════════════════════════════════════════════════════
export const TUTORIAL_PROMPT = `Create a blog article as a practical troubleshooting guide.
Target audience:
- Network engineers (mid to senior level)
- Data center operators
- ISP engineers
- Technical buyers with engineering background
STRUCTURE REQUIREMENTS:
1. **Strong Opening (Hook + Scenario)**
Start with a realistic field scenario (e.g. outage, alert, escalation).
Make it relatable (2 AM, NOC alert, customer escalation).
Clearly define the problem. Include the environment (spine-leaf, DWDM ring, campus core).
Example: "It's 2 AM. NOC pager goes off. Core spine link between pods is flapping — 200G aggregate capacity lost. You SSH into the switch, check the optics, and see Tx power at -14.3 dBm on a module rated for -8.2 to +0.5. The transceiver is dying. Here's how you diagnose this in under 5 minutes."
2. **Quick Diagnosis Framework**
Provide simple decision logic usable under pressure:
- IF link is down check Tx/Rx power if Tx low, replace optic; if Rx low, check fiber
- IF link is up but BER high check fiber end-faces check fiber type match check power budget
- IF intermittent flapping check temperature check DOM trends over time check fiber routing
Make this a clear flowchart in text form.
3. **Deep Dive Sections** (each MUST include):
- Symptoms (specific alarms, log messages, metrics)
- Root causes (technical explanation of WHY)
- Measurements (exact Tx, Rx, OSNR, BER values and what they mean)
- Interpretation (how to read DOM output, what values indicate)
- Fix (step-by-step with specific commands)
- "What engineers usually get wrong" insight
Cover these issues:
a) Low transmit power / dying laser
b) High BER or CRC errors (pre-FEC vs post-FEC)
c) Temperature and environmental problems
d) Fiber type mismatches (SMF vs MMF, wrong wavelength)
e) Coherent (400ZR/ZR+) link issues (if applicable)
4. **Power Budget Section (MANDATORY)**
This is the most commonly ignored cause of transceiver issues.
Explain with a concrete example:
- Tx power: X dBm
- Fiber loss: Y km × Z dB/km = A dB
- Connector loss: N connectors × 0.3 dB = B dB
- Splice loss: M splices × 0.1 dB = C dB
- Total loss: A + B + C = D dB
- Rx power: Tx - D = E dBm
- Rx sensitivity: F dBm
- Margin: E - F = G dB (need 3 dB)
Show common mistakes (forgotten patch panels, dirty connectors eating 1-2 dB each).
5. **Tools & Commands**
Include real CLI examples with expected output.
Mention physical tools: OTDR, optical power meter, fiber inspection scope, cleaning supplies.
For coherent: spectrum analyzer, OSNR measurement.
6. **Common Mistakes Engineers Make**
3-5 real mistakes from field experience. Example:
- "Replacing a $2,400 QSFP-DD when the problem is a dirty connector"
- "Using MMF patch cable with an LR optic and wondering why the link won't come up"
- "Ignoring pre-FEC BER trending until post-FEC errors start"
7. **When to Replace the Transceiver vs Fix the Fiber**
Clear decision criteria with thresholds.
8. **Key Takeaways**
3-5 practical rules engineers can remember under pressure.
OUTPUT: Complete, clean markdown. No notes, no placeholders, no generic filler. Minimum 1500 words.`;
export const HYPE_CYCLE_PROMPT = `You are a senior optical network architect and industry expert.
Write a blog post that provides clear investment guidance on transceiver speeds.
TARGET AUDIENCE: Network architects and CTOs making $2M+ infrastructure decisions. They need to decide WHAT to buy, WHEN, and WHY not how transceivers work.
CRITICAL RULES:
- Have a STRONG opinion. Take a clear position.
- Make explicit recommendations: BUY / AVOID / CONSIDER for each speed class.
- Do NOT be neutral. Neutral advice is useless advice.
- Do NOT include troubleshooting content. This is a STRATEGY article.
- Do NOT dump product lists without context. Every product mentioned must serve the argument.
- Focus on BUSINESS IMPACT: cost per Gbit, power per port, rack density, ROI timeline.
- Do NOT mix topics. This is investment guidance. Not a tutorial. Not troubleshooting.
STRUCTURE:
1. **Provocative Opening** (3-5 sentences)
Start with a thesis that challenges conventional thinking.
Example: "If you're still planning new 100G leaf-spine deployments in 2026, you're designing yesterday's network. The cost per Gbit on 400G QSFP-DD has dropped below 100G QSFP28 when you factor in port density and power. Here's what the numbers actually say."
2. **Market Reality** (2-3 paragraphs)
- AI/ML traffic explosion: east-west traffic in GPU clusters doubling every 12 months
- Hyperscaler trends driving commoditization of 400G
- Enterprise following hyperscale with 2-3 year lag
- Supply chain: where is pricing heading, what's actually available vs announced
3. **Speed-by-Speed Investment Analysis** For EACH speed class, state clearly:
- **Verdict**: BUY / LEGACY / AVOID / EARLY (one word, bold)
- **Cost per Gbit** (actual numbers)
- **Where it makes sense** (specific use case)
- **Where it does NOT make sense** (specific anti-pattern)
Cover these speed classes:
- **100G QSFP28** Legacy. Still deployed but declining cost advantage over 400G.
- **200G** Skip tier. Being bypassed in most new designs.
- **400G QSFP-DD/OSFP** Current sweet spot. Best price/performance/maturity balance.
- **800G OSFP/QSFP-DD800** Emerging. AI fabric and hyperscale spine only.
- **1.6T** Watch. Not production-ready.
4. **Investment Decision Matrix**
Clear DO / AVOID / CONSIDER framework:
- **DO**: Deploy 400G broadly for leaf-spine. Budget 800G for spine/AI interconnect.
- **AVOID**: New 100G designs. 200G unless forced by existing chassis.
- **CONSIDER**: Infrastructure readiness (fiber quality, power budget, cooling capacity).
5. **Hidden Cost Analysis** (MANDATORY)
The optic is 30-40% of the real cost. Include:
- Power consumption per port (W): 400G ~12W, 800G ~18-25W
- Cooling cost: $0.10-0.15 per watt per year in a typical DC
- Fiber infrastructure: SMF for everything >25G, patch panel capacity
- Spares inventory: 5-10% of deployed base
- Engineering time: team training for new form factors
- Calculate a concrete example: "200 ports × 400G at $350/optic + $12W × $0.12/W/yr = $X total over 3 years"
6. **Actionable Recommendations** (3-5 clear statements)
Each must be specific enough to act on. Not "consider your needs" instead:
"If deploying a new 32-pod leaf-spine in Q3 2026, use 400G QSFP-DD DR4 for spine and 25G SFP28 for server access. Budget $X per port. Plan 800G spine upgrade for 2028."
ANTI-PATTERNS (STRICTLY FORBIDDEN):
- Mixing in troubleshooting or operational content
- Listing products without explaining WHY they matter for the investment decision
- Being neutral ("it depends") take a position
- Generic market statements without numbers
- Using "could", "might", "typically" use "is", "will", "should not"
- Referencing products not discussed in the article body
OUTPUT: Complete markdown, minimum 1500 words. No placeholders. No meta-comments.`;
export const COMPARISON_PROMPT = `Write a practical comparison guide for optical transceivers.
Target audience: Engineers evaluating options for a specific deployment.
STRUCTURE:
1. **Opening**: Real procurement/deployment scenario. Example: "You need 200 optics for a new leaf-spine build. The OEM quotes $3,200 per QSFP-DD DR4. A compatible vendor offers the same at $890. Your boss asks: 'What's the catch?' Here's the honest answer."
2. **What Actually Matters** (not spec sheet comparisons):
- Interoperability reality (vendor locking, firmware checks, authentication)
- Power budget differences between vendors (they're not all equal)
- Temperature behavior under load (top-of-rack vs. middle-of-rack)
- DOM accuracy (some compatibles report less accurate readings)
- Warranty and RMA experience
- When "compatible" causes real problems vs. when it works perfectly
3. **Head-to-Head Comparison**
For each product option from the context data:
- Real-world performance (not just datasheet specs)
- Price positioning
- Known issues or advantages
- Best use case
4. **Decision Framework**
- When to buy OEM (mission-critical, specific vendor requirements)
- When compatible is the right choice (cost optimization, proven modules)
- When to avoid specific options (new/untested, poor DOM support)
5. **Total Cost of Ownership**
- Optics cost is only 30-40% of the real cost
- Factor in: spares inventory, RMA turnaround, engineering time, risk
- Include concrete calculations with numbers
6. **Key Takeaways** Decision rules for procurement.
Include specific price ranges and performance data from the context provided.
Do NOT be a shill for any vendor. Be honest about tradeoffs.`;
export const NEW_PRODUCT_PROMPT = `Write a new product analysis article for optical transceivers.
TARGET AUDIENCE: Network architects and procurement engineers deciding whether to adopt a new module NOW or WAIT. They need a clear verdict, not a press release rewrite.
CRITICAL RULES:
- Do NOT rewrite the vendor's spec sheet. Engineers can read datasheets themselves.
- Do NOT include troubleshooting content. This is a product analysis, not an operations guide.
- Have a CLEAR VERDICT: BUY NOW / WAIT / SKIP for each product discussed.
- Every claim must have a number. No "improved performance" say "12W vs 14W previous gen."
- Compare explicitly to the product this replaces. If there's no predecessor, say so.
STRUCTURE:
1. **Provocative Opening** (3-5 sentences)
Cut through the hype. What does this product actually change?
Example: "Another 800G OSFP. The fourth this quarter. Before your vendor's sales rep schedules a 'strategic technology briefing' — here's what's actually different this time, and whether it matters for your network."
2. **What's Actually New vs. Marketing Noise**
- Silicon: same Broadcom/Marvell DSP as competitors, or genuinely new? Which generation?
- Optics: same InP laser, or new EML/VCSEL approach?
- Power: actual module power draw vs. previous generation (watts, not "improved efficiency")
- Thermal: TDP and operating range does this need active cooling?
- Form factor: backward compatible or requires new line cards?
3. **Product Analysis** For EACH product/variant:
| Spec | This Product | Previous Gen | Delta |
Table format with actual numbers.
Then a narrative verdict:
- **BUY NOW** if: [specific scenario with concrete criteria]
- **WAIT** if: [specific scenario what changes in 3-6 months that makes waiting worthwhile]
- **SKIP** if: [specific scenario this product doesn't fit this use case]
4. **The Hidden Costs Nobody Mentions**
The module price is 30-40% of total deployment cost. Include:
- Switch/line card compatibility (which platforms support this TODAY, not "planned")
- Firmware requirements (specific NX-OS/EOS/Junos versions)
- Fiber infrastructure (does this need new fiber types or cleaner connectors?)
- Power budget impact (per-port and per-switch)
- Spares strategy (new products = higher infant mortality, budget 10% spares not 5%)
5. **Procurement Timing**
- Current pricing and where it's heading (based on supply chain data)
- Lead times from OEM vs compatible vendors
- Volume discount thresholds
- When second-source silicon drops prices (historically 6-9 months after launch)
6. **Bottom Line** (3-5 decisive statements)
Not "consider your needs." Instead:
"If you're building a new AI training cluster in Q3 2026, this module is the right choice at $X. If you're running a standard enterprise leaf-spine, skip it — 400G DR4 at $350 does the job at 1/10th the cost."
ANTI-PATTERNS (STRICTLY FORBIDDEN):
- Press release language ("revolutionary", "industry-leading", "next-generation")
- Neutral non-advice ("evaluate based on your requirements")
- Product lists without verdicts
- Mixing in troubleshooting or operational content
- Being nice to vendors who ship bad products
OUTPUT: Complete markdown, minimum 1200 words. No placeholders.`;
// Keep the old MASTER_PROMPT name as alias for backward compatibility
export const MASTER_PROMPT = TUTORIAL_PROMPT;
// ═══════════════════════════════════════════════════════
// REFINEMENT PASSES
// ═══════════════════════════════════════════════════════
export const DEPTH_PROMPT = `Take the existing article and improve it with technical depth.
ADD where missing:
1. Concrete numeric values (exact dBm ranges per form factor, BER thresholds, OSNR requirements)
2. Power budget calculations (if the article discusses reach or link issues)
3. CLI command examples with realistic output snippets
4. Cause-effect explanations (WHY does this happen, not just WHAT to do)
5. Real-world context (what does this look like in a running network)
6. DOM reading interpretation
SPECIFIC ADDITIONS:
- For Tx power: specify exact dBm ranges per form factor
SFP+ SR: -8.2 to +0.5 dBm, alarm at -11.0 dBm
QSFP28 LR4: -4.3 to +4.5 dBm, alarm at -7.0 dBm
QSFP-DD DR4: -2.9 to +3.0 dBm per lane
400ZR: -10.0 to +2.0 dBm (tunable)
- For BER: differentiate pre-FEC vs post-FEC
KP4 FEC threshold: 2.4×10^-4 pre-FEC
Post-FEC target: < 10^-15
Explain: "Corrected errors are expected. Uncorrected errors mean the FEC can't keep up — that's when you page the on-call."
- For coherent: OSNR requirements per speed
100G DP-QPSK: 12 dB minimum
400G 16QAM: 20 dB minimum
800G: 24 dB minimum
- For temperature: why top-of-rack runs hotter, impact on laser lifetime
REMOVE:
- Vague statements ("may indicate issues", "consider checking")
- Generic filler that adds no technical value
- Redundant explanations already covered elsewhere in the article
Do NOT make the text longer unless it adds real technical value.
Preserve the markdown structure.
Keep the engineer voice direct, confident, slightly opinionated.`;
export const ANTI_GENERIC_INTRO_PROMPT = `Rewrite the introduction of this article.
KILL any generic or marketing-style opening. Engineers close the tab immediately if they see:
- "In today's rapidly evolving network landscape"
- "Optical transceivers play a key role"
- "As data center bandwidth demands increase"
- Any sentence that could apply to any article about any topic
REPLACE WITH a real scenario that the reader immediately recognizes from their own experience.
Make the reader feel "this person has been in my shoes."
Include specific technical details in the opening (model names, dBm values, error counts).
The intro should be 3-5 sentences maximum. Get to the point.
Example of a great opening:
"It's 2 AM. NOC pager goes off. Core spine link between pods is flapping — 200G aggregate capacity lost. You SSH into the switch, check the optics, and see Tx power at -14.3 dBm on a module rated for -8.2 to +0.5. The transceiver is dying. Here's how you diagnose this in under 5 minutes."
Return the complete article with the fixed introduction. Do not change the rest.`;
export const QUALITY_CONTROL_PROMPT = `Check this article for the following issues and fix ALL of them:
QUALITY GATES (every article MUST pass):
1. NUMERIC VALUES Every technical claim MUST have a number attached.
BAD: "Low power indicates a problem"
GOOD: "Tx below -11.0 dBm on a 10G SR module means the laser is degrading"
2. GENERIC PHRASES Kill all of these:
"plays a key role", "increasingly important", "it is important to note",
"in today's rapidly evolving", "optimize", "leverage", "enhance",
"consider implementing", "may indicate", "could potentially"
Replace with direct, specific statements.
3. PLACEHOLDER TEXT Zero tolerance for TODO, NOTE, FIXME, <!-- -->, or incomplete sections.
4. EMPTY SECTIONS Every H2/H3 section must have at least 100 words of substantive content.
5. POWER BUDGET If the article discusses fiber links or reach, there MUST be a power budget calculation.
6. CLI EXAMPLES At least 2 real CLI commands in the article.
7. CAUSE-EFFECT Every "do X" must explain WHY. No unexplained instructions.
8. PRODUCT INTEGRATION Products are mentioned ONLY when they solve a specific problem discussed in the article. No random product dumps.
9. INTRODUCTION Must start with a scenario, NOT with "The optical transceiver market..."
10. MINIMUM DEPTH Article must be at least 1200 words. If under that, add depth to existing sections (don't add filler).
For each issue found, rewrite the affected section to fix it.
Return the complete fixed article in markdown.`;
/** Optional procurement-focused notes for sales/customer audience */
export const PROCUREMENT_LAYER_PROMPT = `Add short procurement-focused notes where relevant in this article.
Rules:
- Maximum 1-2 sentences per note, woven naturally into the text
- Focus on cost of misdiagnosis and unnecessary replacements
- Mention price context only when it helps the reader make better decisions
- Keep the engineer voice you're helping them save money, not selling
Good example:
"Before RMA'ing a $2,400 QSFP-DD module, clean the fiber end-face. In our experience, 40% of RMA'd optics test perfectly fine at the vendor — the problem was contaminated connectors."
Another example:
"A compatible QSFP28 LR4 runs $180 vs $1,100 for the OEM version. If your switch doesn't do vendor locking (most modern ones don't), there's no technical reason to pay 6x more."
Do NOT turn this into marketing content. Keep the engineer voice.
Return the complete article with the notes added.`;
// ═══════════════════════════════════════════════════════
// TOPIC PROMPT BUILDER — Injects context data
// ═══════════════════════════════════════════════════════
export function buildTopicPrompt(
topic: string,
data: {
products: ReadonlyArray<Record<string, unknown>>;
news: ReadonlyArray<Record<string, unknown>>;
faq: ReadonlyArray<Record<string, unknown>>;
troubleshooting: ReadonlyArray<Record<string, unknown>>;
},
): string {
const parts: string[] = [];
// Select the right master prompt based on topic
if (topic === "tutorial") {
parts.push(TUTORIAL_PROMPT);
} else if (topic === "hype_cycle") {
parts.push(HYPE_CYCLE_PROMPT);
} else if (topic === "comparison") {
parts.push(COMPARISON_PROMPT);
} else if (topic === "new_product") {
parts.push(NEW_PRODUCT_PROMPT);
} else {
parts.push(NEW_PRODUCT_PROMPT);
}
// Append gathered data as context — clearly separated
if (data.products.length > 0) {
parts.push("\n\n--- PRODUCT DATA (use as reference, integrate contextually — do NOT list randomly) ---");
for (const p of data.products.slice(0, 15)) {
const price = p.price ? `, ~€${p.price}` : "";
parts.push(`${p.standard_name || p.slug}: ${p.form_factor} ${p.speed}, reach ${p.reach_label || "N/A"}, fiber ${p.fiber_type || "N/A"}, vendor ${p.vendor || "N/A"}${price}`);
}
}
if (data.news.length > 0) {
parts.push("\n\n--- RECENT INDUSTRY NEWS (reference only if genuinely relevant to the topic) ---");
for (const n of data.news.slice(0, 5)) {
parts.push(`${n.title} (${n.source || "unknown"}, ${n.date || "recent"})`);
}
}
// Only include troubleshooting data for tutorial/troubleshooting articles
// Strategy articles (hype_cycle, comparison, new_product) must NOT mix in troubleshooting
if (topic === "tutorial" && data.troubleshooting.length > 0) {
parts.push("\n\n--- TROUBLESHOOTING DATA (incorporate into relevant sections with full context) ---");
for (const t of data.troubleshooting) {
parts.push(`• Symptom: ${t.symptom}`);
parts.push(` Cause: ${t.cause}`);
parts.push(` Fix: ${t.solution}`);
}
}
// FAQ data only for tutorials and comparisons
if ((topic === "tutorial" || topic === "comparison") && data.faq.length > 0) {
parts.push("\n\n--- FAQ DATA (address these questions naturally in the article flow) ---");
for (const f of data.faq.slice(0, 5)) {
parts.push(`• Q: ${f.question} → A: ${f.answer}`);
}
}
return parts.join("\n");
}

View File

@ -0,0 +1,113 @@
/**
* Ollama LLM client for blog generation and content enhancement.
*
* Uses qwen2.5:14b on Mac Studio (.213) for text generation.
* Supports streaming and non-streaming modes.
*/
const OLLAMA_URL = process.env.OLLAMA_URL || "http://localhost:11434";
const LLM_MODEL = process.env.OLLAMA_LLM_MODEL || "qwen2.5:14b";
interface LlmResponse {
text: string;
model: string;
totalDuration: number;
evalCount: number;
}
/** Generate text from a system prompt + user prompt */
export async function generate(
systemPrompt: string,
userPrompt: string,
options?: { temperature?: number; maxTokens?: number; timeoutMs?: number },
): Promise<LlmResponse> {
const resp = await fetch(`${OLLAMA_URL}/api/generate`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
model: LLM_MODEL,
prompt: userPrompt,
system: systemPrompt,
stream: false,
options: {
temperature: options?.temperature ?? 0.7,
num_predict: options?.maxTokens ?? 4096,
},
}),
signal: AbortSignal.timeout(options?.timeoutMs ?? 180000),
});
if (!resp.ok) {
const errText = await resp.text();
throw new Error(`Ollama generate failed: ${resp.status} ${errText}`);
}
const data = await resp.json() as {
response: string;
model: string;
total_duration: number;
eval_count: number;
};
return {
text: data.response,
model: data.model,
totalDuration: data.total_duration,
evalCount: data.eval_count,
};
}
/** Chat-style generation with message history */
export async function chat(
messages: ReadonlyArray<{ role: "system" | "user" | "assistant"; content: string }>,
options?: { temperature?: number; maxTokens?: number },
): Promise<LlmResponse> {
const resp = await fetch(`${OLLAMA_URL}/api/chat`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
model: LLM_MODEL,
messages,
stream: false,
options: {
temperature: options?.temperature ?? 0.7,
num_predict: options?.maxTokens ?? 4096,
},
}),
signal: AbortSignal.timeout(120000),
});
if (!resp.ok) {
const errText = await resp.text();
throw new Error(`Ollama chat failed: ${resp.status} ${errText}`);
}
const data = await resp.json() as {
message: { content: string };
model: string;
total_duration: number;
eval_count: number;
};
return {
text: data.message.content,
model: data.model,
totalDuration: data.total_duration,
evalCount: data.eval_count,
};
}
/** Check if Ollama is available and model is loaded */
export async function checkHealth(): Promise<{ ok: boolean; model: string; error?: string }> {
try {
const resp = await fetch(`${OLLAMA_URL}/api/tags`, { signal: AbortSignal.timeout(5000) });
if (!resp.ok) return { ok: false, model: LLM_MODEL, error: `HTTP ${resp.status}` };
const data = await resp.json() as { models: Array<{ name: string }> };
const hasModel = data.models.some((m) => m.name.includes(LLM_MODEL.split(":")[0]));
return { ok: hasModel, model: LLM_MODEL, error: hasModel ? undefined : `Model ${LLM_MODEL} not found` };
} catch (err) {
return { ok: false, model: LLM_MODEL, error: (err as Error).message };
}
}

File diff suppressed because it is too large Load Diff

View File

@ -0,0 +1,160 @@
/**
* WS4: Competitor Intelligence Alerts & Price Changes
*/
import { Router } from "express";
import { pool } from "../db/client";
export const competitorRouter = Router();
/**
* GET /api/competitor-alerts?vendor=&type=&severity=&days=&limit=&offset=
*/
competitorRouter.get("/", async (req, res) => {
try {
const {
vendor, type, severity, days = "7",
acknowledged, limit = "50", offset = "0"
} = req.query;
let sql = `
SELECT ca.*,
v.name AS vendor_name,
v.slug AS vendor_slug
FROM competitor_alerts ca
LEFT JOIN vendors v ON ca.vendor_id = v.id
WHERE ca.created_at > NOW() - INTERVAL '1 day' * $1
`;
const params: any[] = [parseInt(days as string)];
let idx = 2;
if (vendor) { sql += ` AND v.slug = $${idx}`; params.push(vendor); idx++; }
if (type) { sql += ` AND ca.alert_type = $${idx}`; params.push(type); idx++; }
if (severity) { sql += ` AND ca.severity = $${idx}`; params.push(severity); idx++; }
if (acknowledged === 'false') { sql += ` AND ca.acknowledged = false`; }
sql += ` ORDER BY ca.created_at DESC LIMIT $${idx} OFFSET $${idx + 1}`;
params.push(parseInt(limit as string), parseInt(offset as string));
const result = await pool.query(sql, params);
// Summary stats
const stats = await pool.query(`
SELECT
alert_type,
COUNT(*) AS count,
COUNT(*) FILTER (WHERE acknowledged = false) AS unread
FROM competitor_alerts
WHERE created_at > NOW() - INTERVAL '1 day' * $1
GROUP BY alert_type
ORDER BY count DESC
`, [parseInt(days as string)]);
res.json({
alerts: result.rows,
total: result.rowCount,
stats: stats.rows,
period_days: parseInt(days as string),
});
} catch (err) {
console.error("Competitor alerts error:", err);
res.status(500).json({ error: "Internal server error" });
}
});
/**
* GET /api/competitor-alerts/price-changes?vendor=&speed=&days=
*/
competitorRouter.get("/price-changes", async (req, res) => {
try {
const { vendor, speed, days = "30", limit = "50" } = req.query;
let sql = `
SELECT pc.*,
v.name AS vendor_name,
t.slug, t.form_factor, t.speed_gbps, t.reach_label
FROM price_changes pc
JOIN vendors v ON pc.vendor_id = v.id
JOIN transceivers t ON pc.transceiver_id = t.id
WHERE pc.detected_at > NOW() - INTERVAL '1 day' * $1
`;
const params: any[] = [parseInt(days as string)];
let idx = 2;
if (vendor) { sql += ` AND v.slug = $${idx}`; params.push(vendor); idx++; }
if (speed) { sql += ` AND t.speed_gbps = $${idx}`; params.push(parseFloat(speed as string)); idx++; }
sql += ` ORDER BY ABS(pc.delta_pct) DESC LIMIT $${idx}`;
params.push(parseInt(limit as string));
const result = await pool.query(sql, params);
res.json({ price_changes: result.rows, total: result.rowCount });
} catch (err) {
console.error("Price changes error:", err);
res.status(500).json({ error: "Internal server error" });
}
});
/**
* PUT /api/competitor-alerts/:id/acknowledge
*/
competitorRouter.put("/:id/acknowledge", async (req, res) => {
try {
const { notes } = req.body || {};
await pool.query(
`UPDATE competitor_alerts SET acknowledged = true, notes = COALESCE($2, notes) WHERE id = $1`,
[req.params.id, notes]
);
res.json({ success: true });
} catch (err) {
res.status(500).json({ error: "Internal server error" });
}
});
/**
* GET /api/competitor-alerts/summary
*
* High-level competitor intelligence overview
*/
competitorRouter.get("/summary", async (req, res) => {
try {
const [alertsByVendor, recentDrops, newProducts, coverage] = await Promise.all([
pool.query(`
SELECT v.name, v.slug, COUNT(*) AS alert_count,
COUNT(*) FILTER (WHERE ca.alert_type = 'price_drop') AS drops,
COUNT(*) FILTER (WHERE ca.alert_type = 'price_increase') AS increases,
COUNT(*) FILTER (WHERE ca.alert_type = 'new_product') AS new_products
FROM competitor_alerts ca
JOIN vendors v ON ca.vendor_id = v.id
WHERE ca.created_at > NOW() - INTERVAL '7 days'
GROUP BY v.name, v.slug ORDER BY alert_count DESC LIMIT 20
`),
pool.query(`
SELECT pc.*, v.name AS vendor_name, t.form_factor, t.speed_gbps, t.reach_label
FROM price_changes pc
JOIN vendors v ON pc.vendor_id = v.id
JOIN transceivers t ON pc.transceiver_id = t.id
WHERE pc.delta_pct < -5 AND pc.detected_at > NOW() - INTERVAL '7 days'
ORDER BY pc.delta_pct ASC LIMIT 10
`),
pool.query(`
SELECT ca.*, v.name AS vendor_name
FROM competitor_alerts ca
JOIN vendors v ON ca.vendor_id = v.id
WHERE ca.alert_type = 'new_product' AND ca.created_at > NOW() - INTERVAL '30 days'
ORDER BY ca.created_at DESC LIMIT 20
`),
pool.query(`SELECT * FROM v_price_coverage WHERE has_recent_price = false LIMIT 20`),
]);
res.json({
period: "7 days",
by_vendor: alertsByVendor.rows,
biggest_price_drops: recentDrops.rows,
new_competitor_products: newProducts.rows,
products_missing_prices: coverage.rows,
});
} catch (err) {
console.error("Summary error:", err);
res.status(500).json({ error: "Internal server error" });
}
});

View File

@ -0,0 +1,217 @@
/**
* Document processing API routes (OCR Pipeline)
*
* POST /api/documents/process Submit a document URL for OCR + embedding
* GET /api/documents List processed documents
* GET /api/documents/:id Get document chunks
*/
import { Router, Request, Response } from "express";
import { embed, upsertPoints, CollectionName } from "../embeddings/client";
import { pool } from "../db/client";
import { randomUUID } from "crypto";
export const documentRouter = Router();
const DOCLING_URL = process.env.DOCLING_URL || "http://localhost:8100";
interface DoclingResult {
success: boolean;
content: string;
format: string;
pages: number | null;
error?: string;
}
/** Chunk markdown into overlapping sections */
function chunkMarkdown(
markdown: string,
maxChunkSize: number = 1500,
overlapSize: number = 200,
): Array<{ heading: string; text: string }> {
const sections = markdown.split(/(?=^#{1,3}\s)/m);
const chunks: Array<{ heading: string; text: string }> = [];
for (const section of sections) {
const trimmed = section.trim();
if (!trimmed || trimmed.length < 20) continue;
const headingMatch = trimmed.match(/^(#{1,3})\s+(.+)/);
const heading = headingMatch ? headingMatch[2].trim() : "Introduction";
const body = headingMatch ? trimmed.slice(headingMatch[0].length).trim() : trimmed;
if (body.length <= maxChunkSize) {
chunks.push({ heading, text: body });
} else {
const paragraphs = body.split(/\n\n+/);
let currentChunk = "";
for (const para of paragraphs) {
if (currentChunk.length + para.length > maxChunkSize && currentChunk.length > 0) {
chunks.push({ heading, text: currentChunk.trim() });
const overlapText = currentChunk.slice(-overlapSize);
currentChunk = overlapText + "\n\n" + para;
} else {
currentChunk += (currentChunk ? "\n\n" : "") + para;
}
}
if (currentChunk.trim().length > 20) {
chunks.push({ heading, text: currentChunk.trim() });
}
}
}
return chunks;
}
// POST /api/documents/process — Process a document URL
documentRouter.post("/process", async (req: Request, res: Response) => {
const { url, title, doc_type, vendor, collection } = req.body as {
url?: string;
title?: string;
doc_type?: string;
vendor?: string;
collection?: string;
};
if (!url) {
res.status(400).json({ success: false, error: "Missing 'url' in request body" });
return;
}
const targetCollection = (collection || "datasheet_chunks") as CollectionName;
if (!["datasheet_chunks", "manual_chunks"].includes(targetCollection)) {
res.status(400).json({ success: false, error: "collection must be 'datasheet_chunks' or 'manual_chunks'" });
return;
}
try {
// Convert via Docling
const docResp = await fetch(`${DOCLING_URL}/convert`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ url, format: "markdown" }),
signal: AbortSignal.timeout(120000),
});
if (!docResp.ok) {
res.status(502).json({ success: false, error: "Docling conversion failed", detail: await docResp.text() });
return;
}
const docResult = (await docResp.json()) as DoclingResult;
if (!docResult.success) {
res.status(502).json({ success: false, error: "Docling conversion failed", detail: docResult.error });
return;
}
const documentId = randomUUID();
const docTitle = title || url.split("/").pop()?.replace(/\.[^.]+$/, "") || "untitled";
const docType = doc_type || "datasheet";
const docVendor = vendor || "Unknown";
// Chunk
const chunks = chunkMarkdown(docResult.content);
// Embed and store
const BATCH_SIZE = 5;
let stored = 0;
for (let i = 0; i < chunks.length; i += BATCH_SIZE) {
const batch = chunks.slice(i, i + BATCH_SIZE);
const points = await Promise.all(
batch.map(async (chunk, idx) => {
const chunkIndex = i + idx;
const embeddingText = `${docTitle}. ${chunk.heading}. ${chunk.text}`;
const vector = await embed(embeddingText);
return {
id: randomUUID(),
vector,
payload: {
document_id: documentId,
source_url: url,
document_type: docType,
chunk_index: chunkIndex,
total_chunks: chunks.length,
title: docTitle,
section_heading: chunk.heading,
text: chunk.text,
page_estimate: docResult.pages,
vendor: docVendor,
product_slug: docTitle.replace(/\s+/g, "-").toLowerCase(),
},
};
}),
);
await upsertPoints(targetCollection, points);
stored += points.length;
}
// Record in documents table (existing schema)
try {
await pool.query(
`INSERT INTO documents (id, entity_type, doc_type, title, r2_key, source_url, page_count, chunks_count, ocr_status, ocr_text, processed_at)
VALUES ($1, 'transceiver', $2, $3, $4, $5, $6, $7, 'completed', $8, NOW())
ON CONFLICT ON CONSTRAINT documents_pkey DO UPDATE
SET processed_at = NOW(), chunks_count = $7, ocr_status = 'completed'`,
[documentId, docType, docTitle, `ocr/${documentId}`, url, docResult.pages, chunks.length, docResult.content.slice(0, 50000)],
);
} catch {
// ignore if insert fails
}
res.json({
success: true,
document_id: documentId,
title: docTitle,
pages: docResult.pages,
chunks: chunks.length,
collection: targetCollection,
markdown_length: docResult.content.length,
});
} catch (err) {
res.status(503).json({
success: false,
error: "Document processing failed",
detail: (err as Error).message,
});
}
});
// GET /api/documents — List processed documents
documentRouter.get("/", async (_req: Request, res: Response) => {
try {
const result = await pool.query(
`SELECT id, title, source_url, doc_type, entity_type, page_count, chunks_count, ocr_status, processed_at, created_at
FROM documents
ORDER BY COALESCE(processed_at, created_at) DESC
LIMIT 100`,
);
res.json({ success: true, documents: result.rows, count: result.rows.length });
} catch {
// Table may not exist — return empty
res.json({ success: true, documents: [], count: 0, note: "documents table not yet created" });
}
});
// GET /api/documents/:id — Get document details
documentRouter.get("/:id", async (req: Request, res: Response) => {
try {
const result = await pool.query(
`SELECT id, title, source_url, doc_type, entity_type, page_count, chunks_count, ocr_status, processed_at, created_at
FROM documents WHERE id = $1::uuid`,
[req.params.id],
);
if (result.rows.length === 0) {
res.status(404).json({ success: false, error: "Document not found" });
return;
}
res.json({ success: true, document: result.rows[0] });
} catch {
res.status(404).json({ success: false, error: "Document not found or table not created" });
}
});

View File

@ -0,0 +1,237 @@
/**
* WS1: Switch Flexoptix Transceiver Finder
*
* "Customer has a Cisco Nexus 93180YC-FX3 — which Flexoptix transceivers fit?"
*/
import { Router } from "express";
import { pool } from "../db/client";
export const finderRouter = Router();
/**
* GET /api/finder?switch=<model>&speed=&form_factor=
*
* Finds Flexoptix-compatible transceivers for a given switch model.
* If no direct Flexoptix match, shows generic compatible transceivers
* with a note about Flexoptix FlexBox coding capability.
*/
finderRouter.get("/", async (req, res) => {
try {
const { switch: switchQuery, speed, form_factor, limit = "20" } = req.query;
if (!switchQuery) {
return res.status(400).json({ error: "Parameter 'switch' is required" });
}
// Step 1: Find the switch
const switchResult = await pool.query(
`SELECT sw.id, sw.model, sw.series, sw.ports_config, sw.max_speed_gbps,
v.name AS vendor_name, sw.image_url, sw.datasheet_r2_key
FROM switches sw
JOIN vendors v ON sw.vendor_id = v.id
WHERE sw.model ILIKE $1
OR sw.model ILIKE '%' || $1 || '%'
OR sw.search_vector @@ plainto_tsquery('english', $1)
ORDER BY
CASE WHEN sw.model ILIKE $1 THEN 0
WHEN sw.model ILIKE $1 || '%' THEN 1
ELSE 2 END
LIMIT 5`,
[switchQuery]
);
if (switchResult.rows.length === 0) {
return res.status(404).json({
error: "Switch not found",
suggestion: "Try a partial model name like 'N9K-C93180' or 'QFX5120'"
});
}
const sw = switchResult.rows[0];
// Step 2: Find compatible transceivers via compatibility table
let compatSql = `
SELECT
t.id, t.slug, t.form_factor, t.speed, t.speed_gbps, t.reach_label, t.reach_meters,
t.fiber_type, t.wavelengths, t.connector, t.power_consumption_w,
t.image_url, t.image_r2_key, t.part_number,
tv.name AS transceiver_vendor,
tv.type AS vendor_type,
c.status AS compat_status,
c.firmware_min,
c.verified_by,
c.notes AS compat_notes,
-- Latest price
(SELECT po.price FROM price_observations po WHERE po.transceiver_id = t.id ORDER BY po.time DESC LIMIT 1) AS latest_price,
(SELECT po.currency FROM price_observations po WHERE po.transceiver_id = t.id ORDER BY po.time DESC LIMIT 1) AS latest_currency,
(SELECT po.stock_level FROM price_observations po WHERE po.transceiver_id = t.id ORDER BY po.time DESC LIMIT 1) AS stock_level,
-- Flexoptix mapping
fpm.flexoptix_sku,
fpm.flexoptix_url,
fpm.flexoptix_price_eur,
fpm.match_type AS flexoptix_match
FROM compatibility c
JOIN transceivers t ON c.transceiver_id = t.id
JOIN vendors tv ON t.vendor_id = tv.id
LEFT JOIN flexoptix_product_map fpm ON (
fpm.form_factor = t.form_factor
AND fpm.speed_gbps = t.speed_gbps
AND (fpm.reach_label = t.reach_label OR fpm.reach_label IS NULL)
)
WHERE c.switch_id = $1 AND c.status = 'compatible'
`;
const params: any[] = [sw.id];
let idx = 2;
if (speed) {
compatSql += ` AND t.speed_gbps = $${idx}`;
params.push(parseFloat(speed as string));
idx++;
}
if (form_factor) {
compatSql += ` AND t.form_factor = $${idx}`;
params.push(form_factor);
idx++;
}
compatSql += ` ORDER BY t.speed_gbps DESC, t.reach_meters ASC LIMIT $${idx}`;
params.push(parseInt(limit as string));
const compatResult = await pool.query(compatSql, params);
// Step 3: Group results by speed class
const bySpeed: Record<string, any[]> = {};
for (const row of compatResult.rows) {
const key = `${row.speed_gbps}G ${row.form_factor}`;
if (!bySpeed[key]) bySpeed[key] = [];
bySpeed[key].push({
...row,
flexoptix_available: !!row.flexoptix_sku,
flexbox_codable: true, // All Flexoptix modules are FlexBox-codable
buy_url: row.flexoptix_url || `https://www.flexoptix.net/en/catalogsearch/result/?q=${encodeURIComponent(row.form_factor + ' ' + row.speed_gbps + 'G ' + row.reach_label)}`,
});
}
// Step 4: Extract port types from switch for "what can this switch accept?"
const portTypes = sw.ports_config || {};
res.json({
switch: {
id: sw.id,
model: sw.model,
series: sw.series,
vendor: sw.vendor_name,
max_speed_gbps: sw.max_speed_gbps,
ports: portTypes,
image_url: sw.image_url,
},
compatible_transceivers: compatResult.rows.map(r => ({
id: r.id,
slug: r.slug,
form_factor: r.form_factor,
speed: r.speed,
speed_gbps: r.speed_gbps,
reach: r.reach_label,
fiber_type: r.fiber_type,
connector: r.connector,
vendor: r.transceiver_vendor,
vendor_type: r.vendor_type,
image_url: r.image_url,
compat_status: r.compat_status,
firmware_min: r.firmware_min,
// Pricing
price: r.latest_price ? parseFloat(r.latest_price) : null,
currency: r.latest_currency,
stock: r.stock_level,
// Flexoptix
flexoptix_sku: r.flexoptix_sku,
flexoptix_url: r.flexoptix_url,
flexoptix_price_eur: r.flexoptix_price_eur ? parseFloat(r.flexoptix_price_eur) : null,
flexoptix_match: r.flexoptix_match,
flexbox_codable: true,
buy_url: r.flexoptix_url || `https://www.flexoptix.net/en/catalogsearch/result/?q=${encodeURIComponent(r.form_factor + ' ' + r.speed_gbps + 'G ' + r.reach_label)}`,
})),
by_speed_class: bySpeed,
total: compatResult.rowCount,
flexoptix_note: "All Flexoptix transceivers support FlexBox coding for OEM compatibility.",
});
} catch (err) {
console.error("Finder error:", err);
res.status(500).json({ error: "Internal server error" });
}
});
/**
* GET /api/finder/suggest?q=<free text>
*
* Free-text query: "100G LR4 for Cisco Nexus" suggests switch + transceiver combos
*/
finderRouter.get("/suggest", async (req, res) => {
try {
const { q } = req.query;
if (!q) return res.status(400).json({ error: "Parameter 'q' is required" });
// Extract speed, form factor, vendor hints from query
const queryStr = (q as string).toLowerCase();
let speed: number | null = null;
let vendor: string | null = null;
let reach: string | null = null;
// Speed detection
const speedMatch = queryStr.match(/(\d+)\s*g\b/i);
if (speedMatch) speed = parseInt(speedMatch[1]!);
// Reach detection
if (queryStr.includes('sr')) reach = 'SR';
else if (queryStr.includes('lr')) reach = 'LR';
else if (queryStr.includes('er')) reach = 'ER';
else if (queryStr.includes('zr')) reach = 'ZR';
else if (queryStr.includes('dr')) reach = 'DR';
// Vendor detection
const vendorPatterns: [RegExp, string][] = [
[/cisco|nexus|catalyst/i, 'Cisco'],
[/juniper|qfx|ex\d{4}/i, 'Juniper'],
[/arista|dcs-/i, 'Arista'],
[/dell|powerswitch/i, 'Dell'],
[/hpe|aruba/i, 'HPE'],
];
for (const [pattern, name] of vendorPatterns) {
if (pattern.test(queryStr)) { vendor = name; break; }
}
// Search switches matching the query
const switches = await pool.query(
`SELECT sw.id, sw.model, sw.series, sw.max_speed_gbps, v.name AS vendor_name
FROM switches sw JOIN vendors v ON sw.vendor_id = v.id
WHERE sw.search_vector @@ plainto_tsquery('english', $1)
${vendor ? `AND v.name ILIKE '%' || $2 || '%'` : ''}
ORDER BY sw.max_speed_gbps DESC LIMIT 10`,
vendor ? [q, vendor] : [q]
);
// Search transceivers matching speed/reach
let tcvrSql = `SELECT t.id, t.slug, t.form_factor, t.speed_gbps, t.reach_label, t.fiber_type,
tv.name AS vendor, t.image_url
FROM transceivers t JOIN vendors v ON t.vendor_id = v.id JOIN vendors tv ON t.vendor_id = tv.id
WHERE 1=1`;
const tcvrParams: any[] = [];
let tidx = 1;
if (speed) { tcvrSql += ` AND t.speed_gbps = $${tidx}`; tcvrParams.push(speed); tidx++; }
if (reach) { tcvrSql += ` AND t.reach_label ILIKE $${tidx}`; tcvrParams.push(reach + '%'); tidx++; }
tcvrSql += ` ORDER BY t.speed_gbps DESC LIMIT 10`;
const transceivers = await pool.query(tcvrSql, tcvrParams);
res.json({
query: q,
parsed: { speed, vendor, reach },
switches: switches.rows,
transceivers: transceivers.rows,
tip: "Use GET /api/finder?switch=<model> for detailed compatibility results",
});
} catch (err) {
console.error("Suggest error:", err);
res.status(500).json({ error: "Internal server error" });
}
});

View File

@ -0,0 +1,201 @@
/**
* WS5 + WS6: Sales Forecast Engine + Price Trajectory
*/
import { Router } from "express";
import { pool } from "../db/client";
import { computeHypeCycle, findTechnology, TECH_GENERATIONS } from "../hype-cycle/norton-bass";
export const forecastRouter = Router();
/**
* GET /api/forecast/:technology
*
* Returns sales forecast for 3/9/12/18 months + price trajectory + buy signal
*/
forecastRouter.get("/:technology", async (req, res) => {
try {
const techQuery = req.params.technology;
const currentYear = new Date().getFullYear();
// Find technology in Norton-Bass model
const tech = findTechnology(techQuery);
if (!tech) {
return res.status(404).json({
error: "Technology not found",
available: TECH_GENERATIONS.map(t => t.name),
});
}
// Compute hype cycle
const hype = computeHypeCycle(tech, currentYear);
// Get price data from DB
const priceHistory = await pool.query(`
SELECT
date_trunc('week', po.time) AS week,
AVG(po.price) AS avg_price,
MIN(po.price) AS min_price,
MAX(po.price) AS max_price,
COUNT(*) AS observations,
po.currency
FROM price_observations po
JOIN transceivers t ON po.transceiver_id = t.id
WHERE t.speed_gbps = $1
GROUP BY week, po.currency
ORDER BY week DESC
LIMIT 52
`, [tech.speedGbps]);
// Compute price trajectory based on hype cycle phase
const currentPrices = priceHistory.rows.length > 0
? priceHistory.rows.map(r => parseFloat(r.avg_price))
: [];
const currentASP = currentPrices.length > 0 ? currentPrices[0]! : tech.speedGbps * 0.5; // rough estimate
// Price decline model based on phase
const phaseDeclineRates: Record<string, number> = {
'INNOVATION_TRIGGER': 0.05,
'PEAK_OF_INFLATED_EXPECTATIONS': 0.12,
'TROUGH_OF_DISILLUSIONMENT': 0.25,
'SLOPE_OF_ENLIGHTENMENT': 0.15,
'PLATEAU_OF_PRODUCTIVITY': 0.05,
'LEGACY_DECLINE': 0.03,
};
const annualDecline = phaseDeclineRates[hype.phase] ?? 0.10;
const monthlyDecline = 1 - Math.pow(1 - annualDecline, 1/12);
const asp3m = currentASP * Math.pow(1 - monthlyDecline, 3);
const asp9m = currentASP * Math.pow(1 - monthlyDecline, 9);
const asp12m = currentASP * Math.pow(1 - monthlyDecline, 12);
const asp18m = currentASP * Math.pow(1 - monthlyDecline, 18);
// Price floor estimate (based on mature technology pricing patterns)
// Typically 15-25% of peak price at full maturity
const priceFloor = currentASP * 0.20;
const monthsToFloor = annualDecline > 0
? Math.ceil(Math.log(priceFloor / currentASP) / Math.log(1 - monthlyDecline))
: 999;
// Volume forecast based on adoption curve
const adoptionNow = hype.adoptionPct / 100;
const adoption3m = Math.min(1, adoptionNow + (hype.forecast?.[0]?.adoptionPct ?? 0) / 100 * 0.25);
const adoption9m = Math.min(1, adoptionNow + (hype.forecast?.[0]?.adoptionPct ?? 0) / 100 * 0.75);
const adoption12m = Math.min(1, adoptionNow + (hype.forecast?.[1]?.adoptionPct ?? 0) / 100);
const adoption18m = Math.min(1, adoptionNow + (hype.forecast?.[2]?.adoptionPct ?? 0) / 100);
const totalMarketPorts = tech.m * 1000000; // market potential in units
const marketShare = 0.03; // estimated Flexoptix-addressable share
const units3m = Math.round(totalMarketPorts * adoption3m * marketShare * 0.25);
const units9m = Math.round(totalMarketPorts * adoption9m * marketShare * 0.75);
const units12m = Math.round(totalMarketPorts * adoption12m * marketShare);
const units18m = Math.round(totalMarketPorts * adoption18m * marketShare * 1.5);
// Confidence decreases with forecast horizon
const conf3m = Math.min(0.95, 0.85 + (priceHistory.rows.length / 100));
const conf9m = conf3m * 0.78;
const conf12m = conf3m * 0.65;
const conf18m = conf3m * 0.50;
// Buy signal
let buySignal: string;
let signalReason: string;
if (hype.phase === 'SLOPE_OF_ENLIGHTENMENT' || hype.phase === 'PLATEAU_OF_PRODUCTIVITY') {
buySignal = 'BUY_NOW';
signalReason = `${tech.name} is in ${hype.phase.replace(/_/g, ' ').toLowerCase()} — prices near floor, volume growing, stable supply chain.`;
} else if (hype.phase === 'TROUGH_OF_DISILLUSIONMENT') {
buySignal = 'WAIT';
signalReason = `${tech.name} prices dropping >10%/quarter. Wait for trough bottom (estimated ${Math.ceil(monthsToFloor * 0.3)} months).`;
} else if (hype.phase === 'PEAK_OF_INFLATED_EXPECTATIONS') {
buySignal = 'WAIT';
signalReason = `${tech.name} is at peak hype — prices will drop significantly. Only buy if urgent.`;
} else if (hype.phase === 'INNOVATION_TRIGGER') {
buySignal = 'HOLD';
signalReason = `${tech.name} is early-stage — limited availability, premium pricing. Wait unless you need bleeding-edge.`;
} else {
buySignal = 'HOLD';
signalReason = `${tech.name} is in legacy/decline — consider migrating to next generation.`;
}
// Store forecast in DB
await pool.query(`
INSERT INTO sales_forecasts (
technology, speed_gbps, form_factor,
forecast_3m_units, forecast_3m_revenue, forecast_9m_units, forecast_9m_revenue,
forecast_12m_units, forecast_12m_revenue, forecast_18m_units, forecast_18m_revenue,
current_asp, asp_3m, asp_12m, price_floor, months_to_floor,
confidence_3m, confidence_9m, confidence_12m, confidence_18m,
buy_signal, signal_reason, data_points
) VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,$20,$21,$22,$23)
`, [
tech.name, tech.speedGbps, tech.formFactor,
units3m, units3m * asp3m, units9m, units9m * asp9m,
units12m, units12m * asp12m, units18m, units18m * asp18m,
currentASP, asp3m, asp12m, priceFloor, monthsToFloor,
conf3m, conf9m, conf12m, conf18m,
buySignal, signalReason, priceHistory.rows.length,
]).catch(() => {}); // Non-critical
res.json({
technology: tech.name,
speed_gbps: tech.speedGbps,
form_factor: tech.formFactor,
hype_cycle: {
phase: hype.phase,
position_pct: hype.positionPct,
adoption_pct: hype.adoptionPct,
},
forecasts: {
"3_months": { units: units3m, revenue_eur: Math.round(units3m * asp3m), confidence: Math.round(conf3m * 100) / 100 },
"9_months": { units: units9m, revenue_eur: Math.round(units9m * asp9m), confidence: Math.round(conf9m * 100) / 100 },
"12_months": { units: units12m, revenue_eur: Math.round(units12m * asp12m), confidence: Math.round(conf12m * 100) / 100 },
"18_months": { units: units18m, revenue_eur: Math.round(units18m * asp18m), confidence: Math.round(conf18m * 100) / 100 },
},
price_trajectory: {
current_asp: Math.round(currentASP * 100) / 100,
asp_3m: Math.round(asp3m * 100) / 100,
asp_9m: Math.round(asp9m * 100) / 100,
asp_12m: Math.round(asp12m * 100) / 100,
asp_18m: Math.round(asp18m * 100) / 100,
price_floor: Math.round(priceFloor * 100) / 100,
months_to_floor: Math.max(0, monthsToFloor),
annual_decline_pct: Math.round(annualDecline * 100),
},
buy_signal: {
signal: buySignal,
reason: signalReason,
},
price_history: priceHistory.rows.slice(0, 12),
model: "Norton-Bass Multigenerational Diffusion v1",
});
} catch (err) {
console.error("Forecast error:", err);
res.status(500).json({ error: "Internal server error" });
}
});
/**
* GET /api/forecast
*
* Overview of all technology forecasts
*/
forecastRouter.get("/", async (_req, res) => {
try {
const currentYear = new Date().getFullYear();
const results = TECH_GENERATIONS.map(tech => {
const hype = computeHypeCycle(tech, currentYear);
return {
technology: tech.name,
speed_gbps: tech.speedGbps,
form_factor: tech.formFactor,
phase: hype.phase,
adoption_pct: hype.adoptionPct,
position_pct: hype.positionPct,
};
});
res.json({ technologies: results });
} catch (err) {
res.status(500).json({ error: "Internal server error" });
}
});

View File

@ -0,0 +1,32 @@
import { Router, Request, Response } from "express";
import { getDbStats } from "../db/queries";
import { pool } from "../db/client";
export const healthRouter = Router();
// GET /api/health — Health check with DB stats
healthRouter.get("/", async (_req: Request, res: Response) => {
try {
const start = Date.now();
const stats = await getDbStats();
const latencyMs = Date.now() - start;
res.json({
success: true,
status: "healthy",
version: "0.1.0",
uptime: process.uptime(),
database: {
connected: true,
latency_ms: latencyMs,
stats,
},
});
} catch (err) {
res.status(503).json({
success: false,
status: "unhealthy",
database: { connected: false, error: String(err) },
});
}
});

View File

@ -0,0 +1,203 @@
/**
* Hype Cycle API routes
*
* GET /api/hype-cycle All technologies with current phase
* GET /api/hype-cycle/enriched All technologies with data-driven metrics
* GET /api/hype-cycle/lifecycle Revenue lifecycle predictions for all speeds
* GET /api/hype-cycle/regional/:tech Regional adoption model for a technology
* GET /api/hype-cycle/:tech Specific technology with 5-year forecast
*/
import { Router, Request, Response } from "express";
import {
computeAllHypeCycles,
computeHypeCycle,
findTechnology,
TECH_GENERATIONS,
SPECIAL_TECHS,
} from "../hype-cycle/norton-bass";
import {
getDataDrivenOverrides,
getSpeedClassMetrics,
computeRevenueLifecycle,
computeRegionalAdoption,
} from "../hype-cycle/data-enrichment";
export const hypeCycleRouter = Router();
const q = (p: string, req: Request): string | undefined =>
req.query[p] ? String(req.query[p]) : undefined;
// GET /api/hype-cycle — All technologies (model-only, fast)
hypeCycleRouter.get("/", (_req: Request, res: Response) => {
const yearParam = q("year", _req);
const year = yearParam ? parseInt(yearParam) : new Date().getFullYear();
const results = computeAllHypeCycles(year);
const sorted = [...results].sort((a, b) => a.positionPct - b.positionPct);
res.json({
success: true,
year,
model: "Norton-Bass Multigenerational Diffusion",
technologies: sorted.map((r) => ({
technology: r.technology,
phase: r.phaseLabel,
positionPct: r.positionPct,
adoptionPct: r.adoptionPct,
compositeScore: r.compositeScore,
peakYear: r.forecast.peakShipmentYear,
yearsToPlateauFromNow: r.forecast.yearsToPlateauFromNow,
})),
});
});
// GET /api/hype-cycle/enriched — Data-driven enrichment from scraped data
hypeCycleRouter.get("/enriched", async (_req: Request, res: Response) => {
try {
const yearParam = q("year", _req);
const year = yearParam ? parseInt(yearParam) : new Date().getFullYear();
const overridesMap = await getDataDrivenOverrides();
const allTechs = [...TECH_GENERATIONS, ...SPECIAL_TECHS];
const results = allTechs.map((tech) => {
const overrides = overridesMap.get(tech.speedGbps);
return computeHypeCycle(tech, year, overrides);
});
const sorted = [...results].sort((a, b) => a.positionPct - b.positionPct);
// Also include raw metrics for transparency
const speedMetrics = await getSpeedClassMetrics();
res.json({
success: true,
year,
model: "Norton-Bass + Data-Driven Enrichment",
dataSource: {
totalTransceivers: speedMetrics.reduce((s, m) => s + m.skuCount, 0),
totalPricePoints: speedMetrics.reduce((s, m) => s + m.priceCount, 0),
speedClasses: speedMetrics.length,
},
technologies: sorted.map((r) => ({
technology: r.technology,
phase: r.phaseLabel,
positionPct: r.positionPct,
adoptionPct: r.adoptionPct,
compositeScore: r.compositeScore,
peakYear: r.forecast.peakShipmentYear,
yearsToPlateauFromNow: r.forecast.yearsToPlateauFromNow,
metrics: r.metrics,
fiveYearForecast: r.forecast.fiveYearProjection,
})),
rawSpeedMetrics: speedMetrics.map((m) => ({
speedGbps: m.speedGbps,
vendorCount: m.vendorCount,
skuCount: m.skuCount,
avgPrice: m.avgPrice ? Math.round(m.avgPrice * 100) / 100 : null,
minPrice: m.minPrice ? Math.round(m.minPrice * 100) / 100 : null,
maxPrice: m.maxPrice ? Math.round(m.maxPrice * 100) / 100 : null,
formFactors: m.formFactors,
reachVariants: m.reachVariants,
})),
});
} catch (err) {
console.error("Enriched hype cycle error:", err);
res.status(500).json({ success: false, error: "Failed to compute enriched hype cycle" });
}
});
// GET /api/hype-cycle/lifecycle — Revenue lifecycle predictions
hypeCycleRouter.get("/lifecycle", async (_req: Request, res: Response) => {
try {
const currentYear = new Date().getFullYear();
const speedMetrics = await getSpeedClassMetrics();
const priceMap = new Map(speedMetrics.map((m) => [m.speedGbps, m.avgPrice]));
const allTechs = [...TECH_GENERATIONS, ...SPECIAL_TECHS];
const lifecycles = allTechs.map((tech) =>
computeRevenueLifecycle(
tech.speedGbps,
tech.name,
tech.introYear,
tech.peakYear,
currentYear,
priceMap.get(tech.speedGbps),
)
);
// Sort by revenue index (highest current revenue first)
const sorted = [...lifecycles].sort((a, b) => b.revenueIndex - a.revenueIndex);
res.json({
success: true,
currentYear,
lifecycles: sorted,
});
} catch (err) {
console.error("Lifecycle error:", err);
res.status(500).json({ success: false, error: "Failed to compute lifecycles" });
}
});
// GET /api/hype-cycle/regional/:tech — Regional adoption by technology
hypeCycleRouter.get("/regional/:tech", (req: Request, res: Response) => {
const techQuery = String(req.params.tech);
const currentYear = new Date().getFullYear();
const tech = findTechnology(techQuery);
if (!tech) {
res.status(404).json({
success: false,
error: `Technology "${techQuery}" not found. Available: 1G, 10G, 25G, 40G, 100G, 400G, 800G, 1.6T, CPO, LPO, 400ZR`,
});
return;
}
const regions = computeRegionalAdoption(tech.peakYear, currentYear, tech.name);
res.json({
success: true,
technology: tech.name,
speedGbps: tech.speedGbps,
globalPeakYear: tech.peakYear,
regions,
});
});
// GET /api/hype-cycle/:tech — Specific technology detail (must be last!)
hypeCycleRouter.get("/:tech", (req: Request, res: Response) => {
const techQuery = String(req.params.tech);
const yearParam = q("year", req);
const year = yearParam ? parseInt(yearParam) : new Date().getFullYear();
const tech = findTechnology(techQuery);
if (!tech) {
res.status(404).json({
success: false,
error: `Technology "${techQuery}" not found. Available: 1G, 10G, 25G, 40G, 100G, 400G, 800G, 1.6T, CPO, LPO, 400ZR`,
});
return;
}
const result = computeHypeCycle(tech, year);
// Add regional data
const regions = computeRegionalAdoption(tech.peakYear, year, tech.name);
// Add revenue lifecycle
const lifecycle = computeRevenueLifecycle(
tech.speedGbps,
tech.name,
tech.introYear,
tech.peakYear,
year,
);
res.json({
success: true,
...result,
regionalAdoption: regions,
revenueLifecycle: lifecycle,
});
});

View File

@ -0,0 +1,266 @@
/**
* Semantic search API routes (Qdrant vector search)
*
* GET /api/search?q=<query>&collection=<col>&limit=<n>
* GET /api/search/products?q=<query>&form_factor=&speed_gbps=&fiber_type=
* GET /api/search/documents?q=<query>&doc_type=&vendor=
* GET /api/search/news?q=<query>&source=
*/
import { Router, Request, Response } from "express";
import { semanticSearch, getCollectionInfo, CollectionName } from "../embeddings/client";
export const searchRouter = Router();
const VALID_COLLECTIONS: CollectionName[] = [
"product_embeddings",
"datasheet_chunks",
"faq_embeddings",
"manual_chunks",
"troubleshooting_embeddings",
"news_embeddings",
];
const q = (p: string, req: Request): string | undefined =>
req.query[p] ? String(req.query[p]) : undefined;
// GET /api/search — Generic semantic search across any collection
searchRouter.get("/", async (req: Request, res: Response) => {
const query = q("q", req);
const collection = (q("collection", req) || "product_embeddings") as CollectionName;
const limit = parseInt(q("limit", req) || "10");
if (!query) {
res.status(400).json({ success: false, error: "Missing 'q' parameter" });
return;
}
if (!VALID_COLLECTIONS.includes(collection)) {
res.status(400).json({
success: false,
error: `Invalid collection. Valid: ${VALID_COLLECTIONS.join(", ")}`,
});
return;
}
try {
const results = await semanticSearch(collection, query, limit);
res.json({
success: true,
query,
collection,
results: results.map((r) => ({
id: r.id,
score: Math.round(r.score * 1000) / 1000,
...r.payload,
})),
count: results.length,
});
} catch (err) {
res.status(503).json({
success: false,
error: "Vector search unavailable",
detail: (err as Error).message,
});
}
});
// GET /api/search/products — Product-specific semantic search with filters
searchRouter.get("/products", async (req: Request, res: Response) => {
const query = q("q", req);
const limit = parseInt(q("limit", req) || "10");
const formFactor = q("form_factor", req);
const speedGbps = q("speed_gbps", req);
const fiberType = q("fiber_type", req);
const wdmType = q("wdm_type", req);
if (!query) {
res.status(400).json({ success: false, error: "Missing 'q' parameter" });
return;
}
// Build Qdrant payload filter
const mustConditions: Array<Record<string, unknown>> = [];
if (formFactor) {
mustConditions.push({ key: "form_factor", match: { value: formFactor.toUpperCase() } });
}
if (speedGbps) {
mustConditions.push({ key: "speed_gbps", match: { value: parseFloat(speedGbps) } });
}
if (fiberType) {
mustConditions.push({ key: "fiber_type", match: { value: fiberType.toUpperCase() } });
}
if (wdmType) {
mustConditions.push({ key: "wdm_type", match: { value: wdmType.toUpperCase() } });
}
const filter = mustConditions.length > 0 ? { must: mustConditions } : undefined;
try {
const results = await semanticSearch("product_embeddings", query, limit, filter);
res.json({
success: true,
query,
filters: { formFactor, speedGbps, fiberType, wdmType },
results: results.map((r) => ({
id: r.id,
score: Math.round(r.score * 1000) / 1000,
slug: r.payload.slug,
standard_name: r.payload.standard_name,
form_factor: r.payload.form_factor,
speed: r.payload.speed,
reach: r.payload.reach_label,
fiber_type: r.payload.fiber_type,
connector: r.payload.connector,
category: r.payload.category,
vendor: r.payload.vendor,
})),
count: results.length,
});
} catch (err) {
res.status(503).json({
success: false,
error: "Vector search unavailable",
detail: (err as Error).message,
});
}
});
// GET /api/search/documents — Search datasheets and manuals
searchRouter.get("/documents", async (req: Request, res: Response) => {
const query = q("q", req);
const limit = parseInt(q("limit", req) || "10");
const docType = q("doc_type", req);
const vendor = q("vendor", req);
const collection = (q("collection", req) || "datasheet_chunks") as CollectionName;
if (!query) {
res.status(400).json({ success: false, error: "Missing 'q' parameter" });
return;
}
if (!["datasheet_chunks", "manual_chunks"].includes(collection)) {
res.status(400).json({
success: false,
error: "collection must be 'datasheet_chunks' or 'manual_chunks'",
});
return;
}
const mustConditions: Array<Record<string, unknown>> = [];
if (docType) {
mustConditions.push({ key: "document_type", match: { value: docType.toLowerCase() } });
}
if (vendor) {
mustConditions.push({ key: "vendor", match: { value: vendor } });
}
const filter = mustConditions.length > 0 ? { must: mustConditions } : undefined;
try {
const results = await semanticSearch(collection, query, limit, filter);
// Group by document for cleaner output
const byDocument = new Map<string, { title: string; vendor: string; source_url: string; chunks: Array<{ score: number; heading: string; text: string; chunk_index: number }> }>();
for (const r of results) {
const docId = String(r.payload.document_id || r.id);
if (!byDocument.has(docId)) {
byDocument.set(docId, {
title: String(r.payload.title || ""),
vendor: String(r.payload.vendor || ""),
source_url: String(r.payload.source_url || ""),
chunks: [],
});
}
byDocument.get(docId)!.chunks.push({
score: Math.round(r.score * 1000) / 1000,
heading: String(r.payload.section_heading || ""),
text: String(r.payload.text || "").slice(0, 500),
chunk_index: Number(r.payload.chunk_index || 0),
});
}
res.json({
success: true,
query,
collection,
filters: { docType, vendor },
documents: Array.from(byDocument.values()),
totalChunks: results.length,
});
} catch (err) {
res.status(503).json({
success: false,
error: "Vector search unavailable",
detail: (err as Error).message,
});
}
});
// GET /api/search/news — Search news articles
searchRouter.get("/news", async (req: Request, res: Response) => {
const query = q("q", req);
const limit = parseInt(q("limit", req) || "10");
const source = q("source", req);
if (!query) {
res.status(400).json({ success: false, error: "Missing 'q' parameter" });
return;
}
const mustConditions: Array<Record<string, unknown>> = [];
if (source) {
mustConditions.push({ key: "source", match: { value: source } });
}
const filter = mustConditions.length > 0 ? { must: mustConditions } : undefined;
try {
const results = await semanticSearch("news_embeddings", query, limit, filter);
res.json({
success: true,
query,
filters: { source },
results: results.map((r) => ({
id: r.id,
score: Math.round(r.score * 1000) / 1000,
title: r.payload.title,
url: r.payload.url,
source: r.payload.source,
summary: r.payload.summary,
published_at: r.payload.published_at,
})),
count: results.length,
});
} catch (err) {
res.status(503).json({
success: false,
error: "Vector search unavailable",
detail: (err as Error).message,
});
}
});
// GET /api/search/stats — Collection statistics
searchRouter.get("/stats", async (_req: Request, res: Response) => {
try {
const stats = await Promise.all(
VALID_COLLECTIONS.map(async (col) => {
try {
const info = await getCollectionInfo(col);
return { collection: col, ...info };
} catch {
return { collection: col, pointsCount: 0, vectorsCount: 0, error: "unavailable" };
}
})
);
res.json({ success: true, collections: stats });
} catch (err) {
res.status(503).json({
success: false,
error: "Qdrant unavailable",
detail: (err as Error).message,
});
}
});

View File

@ -0,0 +1,15 @@
import { Router, Request, Response } from "express";
import { listStandards } from "../db/queries";
export const standardRouter = Router();
// GET /api/standards — List all standards
standardRouter.get("/", async (req: Request, res: Response) => {
try {
const standards = await listStandards(req.query.speed ? String(req.query.speed) : undefined);
res.json({ success: true, data: standards, total: standards.length });
} catch (err) {
console.error("List standards error:", err);
res.status(500).json({ success: false, error: "Internal server error" });
}
});

View File

@ -0,0 +1,64 @@
import { Router, Request, Response } from "express";
import { searchSwitches, getSwitchById, getCompatibleTransceivers, getSwitchDocuments } from "../db/queries";
export const switchRouter = Router();
// GET /api/switches — Search/list switches
// Filters: ?q=&category=&whitebox=true&sonic=true&asic_vendor=Broadcom&nos=SONiC&ocp=true
switchRouter.get("/", async (req: Request, res: Response) => {
try {
const result = await searchSwitches({
q: String(req.query.q || ""),
category: req.query.category ? String(req.query.category) : undefined,
whitebox: req.query.whitebox === "true" ? true : undefined,
sonic_compatible: req.query.sonic === "true" ? true : undefined,
asic_vendor: req.query.asic_vendor ? String(req.query.asic_vendor) : undefined,
nos: req.query.nos ? String(req.query.nos) : undefined,
ocp: req.query.ocp === "true" ? true : undefined,
max_speed_gbps: req.query.max_speed_gbps ? parseFloat(String(req.query.max_speed_gbps)) : undefined,
limit: req.query.limit ? parseInt(String(req.query.limit)) : 50,
offset: req.query.offset ? parseInt(String(req.query.offset)) : 0,
});
res.json({ success: true, ...result });
} catch (err) {
console.error("Search switches error:", err);
res.status(500).json({ success: false, error: "Internal server error" });
}
});
// GET /api/switches/:id — Get single switch
switchRouter.get("/:id", async (req: Request, res: Response) => {
try {
const sw = await getSwitchById(String(req.params.id));
if (!sw) {
res.status(404).json({ success: false, error: "Switch not found" });
return;
}
res.json({ success: true, data: sw });
} catch (err) {
console.error("Get switch error:", err);
res.status(500).json({ success: false, error: "Internal server error" });
}
});
// GET /api/switches/:id/documents — Datasheets, manuals, guides for a switch
switchRouter.get("/:id/documents", async (req: Request, res: Response) => {
try {
const docs = await getSwitchDocuments(String(req.params.id));
res.json({ success: true, data: docs, total: docs.length });
} catch (err) {
console.error("Get switch documents error:", err);
res.status(500).json({ success: false, error: "Internal server error" });
}
});
// GET /api/switches/:id/compatibility — Compatible transceivers for a switch
switchRouter.get("/:id/compatibility", async (req: Request, res: Response) => {
try {
const transceivers = await getCompatibleTransceivers(String(req.params.id));
res.json({ success: true, data: transceivers, total: transceivers.length });
} catch (err) {
console.error("Get compatibility error:", err);
res.status(500).json({ success: false, error: "Internal server error" });
}
});

View File

@ -0,0 +1,68 @@
import { Router, Request, Response } from "express";
import { searchTransceivers, getTransceiverById } from "../db/queries";
import { pool } from "../db/client";
export const transceiverRouter = Router();
// GET /api/transceivers — Search/list transceivers
transceiverRouter.get("/", async (req: Request, res: Response) => {
try {
const q = (p: string) => req.query[p] ? String(req.query[p]) : undefined;
const result = await searchTransceivers({
q: q("q"),
form_factor: q("form_factor"),
speed: q("speed"),
speed_gbps: q("speed_gbps") ? parseFloat(q("speed_gbps")!) : undefined,
category: q("category"),
fiber_type: q("fiber_type"),
reach_min: q("reach_min") ? parseInt(q("reach_min")!) : undefined,
reach_max: q("reach_max") ? parseInt(q("reach_max")!) : undefined,
wdm_type: q("wdm_type"),
coherent: q("coherent") === "true" ? true : q("coherent") === "false" ? false : undefined,
market_status: q("market_status"),
vendor: q("vendor"),
limit: q("limit") ? parseInt(q("limit")!) : 50,
offset: q("offset") ? parseInt(q("offset")!) : 0,
});
res.json({ success: true, ...result });
} catch (err) {
console.error("Search transceivers error:", err);
res.status(500).json({ success: false, error: "Internal server error" });
}
});
// GET /api/transceivers/:id — Get single transceiver
transceiverRouter.get("/:id", async (req: Request, res: Response) => {
try {
const transceiver = await getTransceiverById(String(req.params.id));
if (!transceiver) {
res.status(404).json({ success: false, error: "Transceiver not found" });
return;
}
res.json({ success: true, data: transceiver });
} catch (err) {
console.error("Get transceiver error:", err);
res.status(500).json({ success: false, error: "Internal server error" });
}
});
// GET /api/transceivers/:id/compatibility — Compatible switches for a transceiver
transceiverRouter.get("/:id/compatibility", async (req: Request, res: Response) => {
try {
const result = await pool.query(
`SELECT sw.id, sw.model, sw.series, sw.category, sw.total_ports,
sw.max_speed_gbps, sw.switching_capacity_tbps, sw.lifecycle_status,
v.name as vendor_name, c.status, c.notes as compat_notes
FROM compatibility c
JOIN switches sw ON c.switch_id = sw.id
LEFT JOIN vendors v ON sw.vendor_id = v.id
WHERE c.transceiver_id::text = $1 AND c.status = 'compatible'
ORDER BY v.name, sw.model`,
[String(req.params.id)]
);
res.json({ success: true, data: result.rows });
} catch (err) {
console.error("Get transceiver compatibility error:", err);
res.status(500).json({ success: false, error: "Internal server error" });
}
});

View File

@ -0,0 +1,233 @@
/**
* WS3: Transport System Planner
*
* "Berlin to Darmstadt, 100G" complete BOM with switches, fiber providers, Flexoptix transceivers
*/
import { Router } from "express";
import { pool } from "../db/client";
export const transportRouter = Router();
// Haversine distance calculation
function haversineKm(lat1: number, lon1: number, lat2: number, lon2: number): number {
const R = 6371;
const dLat = (lat2 - lat1) * Math.PI / 180;
const dLon = (lon2 - lon1) * Math.PI / 180;
const a = Math.sin(dLat/2)**2 + Math.cos(lat1*Math.PI/180) * Math.cos(lat2*Math.PI/180) * Math.sin(dLon/2)**2;
return R * 2 * Math.atan2(Math.sqrt(a), Math.sqrt(1-a));
}
/**
* POST /api/transport/plan
* Body: { from, to, bandwidth_gbps, redundancy?, budget_preference? }
*/
transportRouter.post("/plan", async (req, res) => {
try {
const { from, to, bandwidth_gbps = 100, redundancy = false, budget_preference = "balanced" } = req.body;
if (!from || !to) {
return res.status(400).json({ error: "Parameters 'from' and 'to' are required" });
}
// 1. Resolve cities
const cityA = await pool.query(`SELECT * FROM cities WHERE name ILIKE $1 LIMIT 1`, [from]);
const cityB = await pool.query(`SELECT * FROM cities WHERE name ILIKE $1 LIMIT 1`, [to]);
if (!cityA.rows[0] || !cityB.rows[0]) {
const allCities = await pool.query(`SELECT name, country FROM cities ORDER BY name`);
return res.status(404).json({
error: `City not found: ${!cityA.rows[0] ? from : to}`,
available_cities: allCities.rows.map(c => `${c.name} (${c.country})`),
});
}
const a = cityA.rows[0];
const b = cityB.rows[0];
// 2. Calculate distance
const straightKm = haversineKm(parseFloat(a.lat), parseFloat(a.lon), parseFloat(b.lat), parseFloat(b.lon));
const fiberKm = Math.round(straightKm * 1.4); // fiber route multiplier
// 3. Determine transceiver requirements based on distance
const transceiverOptions = determineTransceiverOptions(fiberKm, bandwidth_gbps);
// 4. Find fiber providers for this route
const providers = await pool.query(
`SELECT fp.name, fp.website, fp.type, fp.products,
fr.product_type, fr.monthly_price_eur, fr.setup_fee_eur, fr.min_contract_months
FROM fiber_routes fr
JOIN fiber_providers fp ON fr.provider_id = fp.id
WHERE (fr.city_a ILIKE $1 AND fr.city_b ILIKE $2)
OR (fr.city_a ILIKE $2 AND fr.city_b ILIKE $1)
OR (fr.city_a ILIKE $1 AND fr.city_b ILIKE 'Frankfurt%')
OR (fr.city_a ILIKE 'Frankfurt%' AND fr.city_b ILIKE $2)
ORDER BY fr.monthly_price_eur ASC NULLS LAST`,
[from, to]
);
// 5. Find matching switches
const switchOptions = await pool.query(
`SELECT sw.id, sw.model, sw.series, sw.max_speed_gbps, sw.switching_capacity_tbps,
sw.ports_config, sw.msrp_usd, v.name AS vendor
FROM switches sw JOIN vendors v ON sw.vendor_id = v.id
WHERE sw.max_speed_gbps >= $1
AND sw.lifecycle_status NOT IN ('End-of-Life', 'End-of-Sale')
ORDER BY sw.msrp_usd ASC NULLS LAST, sw.max_speed_gbps DESC
LIMIT 10`,
[bandwidth_gbps]
);
// 6. Find Flexoptix transceivers for each option
const options = [];
for (const tcvrOpt of transceiverOptions) {
const flexoptix = await pool.query(
`SELECT t.id, t.slug, t.form_factor, t.speed_gbps, t.reach_label, t.reach_meters,
t.fiber_type, t.connector, t.image_url,
(SELECT po.price FROM price_observations po WHERE po.transceiver_id = t.id ORDER BY po.time DESC LIMIT 1) AS price,
(SELECT po.currency FROM price_observations po WHERE po.transceiver_id = t.id ORDER BY po.time DESC LIMIT 1) AS currency
FROM transceivers t
JOIN vendors v ON t.vendor_id = v.id
WHERE t.speed_gbps >= $1
AND t.reach_meters >= $2
AND t.fiber_type = 'SMF'
AND v.slug = 'flexoptix'
ORDER BY t.speed_gbps ASC, t.reach_meters ASC
LIMIT 5`,
[tcvrOpt.speed_gbps, tcvrOpt.min_reach_m]
);
// If no Flexoptix match, find any compatible transceiver
const anyMatch = flexoptix.rows.length > 0 ? flexoptix.rows : (await pool.query(
`SELECT t.id, t.slug, t.form_factor, t.speed_gbps, t.reach_label, t.reach_meters,
t.fiber_type, t.connector, t.image_url, v.name AS vendor,
(SELECT po.price FROM price_observations po WHERE po.transceiver_id = t.id ORDER BY po.time DESC LIMIT 1) AS price
FROM transceivers t JOIN vendors v ON t.vendor_id = v.id
WHERE t.speed_gbps >= $1 AND t.reach_meters >= $2 AND t.fiber_type = 'SMF'
ORDER BY t.speed_gbps ASC LIMIT 5`,
[tcvrOpt.speed_gbps, tcvrOpt.min_reach_m]
)).rows;
const spanCount = Math.ceil(fiberKm * 1000 / tcvrOpt.max_span_m);
const tcvrCount = redundancy ? spanCount * 4 : spanCount * 2; // 2 per span (both ends), x2 for redundancy
const tcvrPrice = anyMatch[0]?.price ? parseFloat(anyMatch[0].price) : tcvrOpt.est_price_eur;
const totalTcvrCost = tcvrCount * tcvrPrice;
options.push({
name: tcvrOpt.name,
description: tcvrOpt.description,
transceiver: {
type: `${tcvrOpt.speed_gbps}G ${tcvrOpt.reach_label}`,
form_factor: tcvrOpt.form_factor,
spans_needed: spanCount,
units_needed: tcvrCount,
unit_price_est: tcvrPrice,
total_cost_est: totalTcvrCost,
flexoptix_products: anyMatch.map(m => ({
slug: m.slug,
speed: m.speed_gbps + 'G',
reach: m.reach_label,
price: m.price ? parseFloat(m.price) : null,
buy_url: `https://www.flexoptix.net/en/catalogsearch/result/?q=${encodeURIComponent(m.form_factor + ' ' + m.speed_gbps + 'G ' + m.reach_label)}`,
})),
},
switches: switchOptions.rows.slice(0, 3).map(sw => ({
model: sw.model,
vendor: sw.vendor,
max_speed: sw.max_speed_gbps + 'G',
price_est: sw.msrp_usd ? parseFloat(sw.msrp_usd) : null,
})),
fiber_providers: providers.rows.length > 0 ? providers.rows : [
{ name: "Contact local fiber providers", note: `No pre-seeded routes for ${from}${to}. Check euNetworks, Telia, DTAG.` }
],
});
}
res.json({
route: {
from: a.name,
to: b.name,
straight_line_km: Math.round(straightKm),
estimated_fiber_km: fiberKm,
bandwidth_requested: bandwidth_gbps + 'G',
redundancy,
},
options,
note: "Prices are estimates. Contact Flexoptix sales for volume pricing.",
});
} catch (err) {
console.error("Transport planner error:", err);
res.status(500).json({ error: "Internal server error" });
}
});
function determineTransceiverOptions(fiberKm: number, bandwidthGbps: number) {
const options = [];
if (fiberKm <= 2) {
options.push({
name: `${bandwidthGbps}G FR (2km)`,
description: `Short reach — single span, no amplification needed`,
speed_gbps: bandwidthGbps, reach_label: 'FR', form_factor: bandwidthGbps >= 400 ? 'QSFP-DD' : 'QSFP28',
min_reach_m: 2000, max_span_m: 2000, est_price_eur: bandwidthGbps >= 400 ? 200 : 80,
});
}
if (fiberKm <= 10) {
options.push({
name: `${bandwidthGbps}G LR4 (10km)`,
description: `Metro reach — ${Math.ceil(fiberKm / 10)} span(s)`,
speed_gbps: bandwidthGbps, reach_label: 'LR4', form_factor: bandwidthGbps >= 400 ? 'QSFP-DD' : 'QSFP28',
min_reach_m: 10000, max_span_m: 10000, est_price_eur: bandwidthGbps >= 400 ? 400 : 120,
});
}
if (fiberKm <= 40) {
options.push({
name: `${bandwidthGbps}G ER4 (40km)`,
description: `Extended reach — ${Math.ceil(fiberKm / 40)} span(s)`,
speed_gbps: bandwidthGbps, reach_label: 'ER4', form_factor: bandwidthGbps >= 400 ? 'QSFP-DD' : 'QSFP28',
min_reach_m: 40000, max_span_m: 40000, est_price_eur: bandwidthGbps >= 400 ? 1500 : 400,
});
}
// ZR is always an option for long distances
if (fiberKm > 10) {
options.push({
name: `${Math.min(bandwidthGbps, 400)}G ZR Coherent (80km/span)`,
description: `Coherent DWDM — ${Math.ceil(fiberKm / 80)} span(s), OIF 400ZR`,
speed_gbps: Math.min(bandwidthGbps, 400), reach_label: 'ZR', form_factor: 'QSFP-DD',
min_reach_m: 80000, max_span_m: 80000, est_price_eur: 2500,
});
}
// Carrier wavelength option
options.push({
name: `Carrier Wavelength Service (${bandwidthGbps}G)`,
description: `Managed service — provider handles fiber + amplification. You only need LR4 transceivers at each end.`,
speed_gbps: bandwidthGbps, reach_label: 'LR4', form_factor: bandwidthGbps >= 400 ? 'QSFP-DD' : 'QSFP28',
min_reach_m: 10000, max_span_m: 999000, est_price_eur: bandwidthGbps >= 400 ? 400 : 120,
});
return options;
}
/**
* GET /api/transport/cities
*/
transportRouter.get("/cities", async (_req, res) => {
try {
const result = await pool.query(`SELECT name, country, has_ix, ix_names, has_datacenter FROM cities ORDER BY name`);
res.json({ cities: result.rows, total: result.rowCount });
} catch (err) {
res.status(500).json({ error: "Internal server error" });
}
});
/**
* GET /api/transport/providers
*/
transportRouter.get("/providers", async (_req, res) => {
try {
const result = await pool.query(`SELECT * FROM fiber_providers ORDER BY name`);
res.json({ providers: result.rows, total: result.rowCount });
} catch (err) {
res.status(500).json({ error: "Internal server error" });
}
});

View File

@ -0,0 +1,15 @@
import { Router, Request, Response } from "express";
import { listVendors } from "../db/queries";
export const vendorRouter = Router();
// GET /api/vendors — List all vendors
vendorRouter.get("/", async (req: Request, res: Response) => {
try {
const vendors = await listVendors(req.query.type ? String(req.query.type) : undefined);
res.json({ success: true, data: vendors, total: vendors.length });
} catch (err) {
console.error("List vendors error:", err);
res.status(500).json({ success: false, error: "Internal server error" });
}
});

View File

@ -0,0 +1,19 @@
{
"compilerOptions": {
"target": "ES2022",
"module": "commonjs",
"lib": ["ES2022"],
"outDir": "./dist",
"rootDir": "./src",
"strict": true,
"esModuleInterop": true,
"skipLibCheck": true,
"forceConsistentCasingInFileNames": true,
"resolveJsonModule": true,
"declaration": true,
"declarationMap": true,
"sourceMap": true
},
"include": ["src/**/*"],
"exclude": ["node_modules", "dist"]
}

View File

@ -0,0 +1,45 @@
{
"name": "@tip/core",
"version": "1.0.0",
"description": "Core optical transceiver database. 159 products, 42 IEEE/MSA standards, 16 form factors, 9 speed tiers.",
"main": "dist/index.js",
"types": "dist/index.d.ts",
"scripts": {
"build": "tsc",
"prepublishOnly": "npm run build"
},
"license": "MIT",
"keywords": [
"transceiver",
"optics",
"sfp",
"qsfp",
"networking",
"fiber",
"ieee",
"telecom",
"osfp",
"qsfp-dd",
"optical",
"datacenter",
"100g",
"400g",
"800g"
],
"files": [
"dist",
"LICENSE",
"README.md"
],
"repository": {
"type": "git",
"url": "https://github.com/renefichtmueller/transceiver-db"
},
"author": "Rene Fichtmueller",
"engines": {
"node": ">=14"
},
"devDependencies": {
"typescript": "^5.9.3"
}
}

File diff suppressed because it is too large Load Diff

View File

@ -0,0 +1,29 @@
{
"name": "@tip/mcp-server",
"version": "0.1.0",
"private": true,
"description": "TIP MCP Server — 12 tools for LLM access to transceiver intelligence",
"main": "dist/index.js",
"bin": {
"tip-mcp": "dist/index.js"
},
"scripts": {
"build": "tsc",
"dev": "tsx src/index.ts",
"start": "node dist/index.js",
"start:http": "tsx src/http-server.ts"
},
"dependencies": {
"@modelcontextprotocol/sdk": "^1.9.0",
"express": "^4.18.2",
"pg": "^8.13.1",
"dotenv": "^16.4.7",
"zod": "^3.24.0"
},
"devDependencies": {
"@types/express": "^4.17.21",
"@types/pg": "^8.11.11",
"typescript": "^5.9.3",
"tsx": "^4.19.0"
}
}

View File

@ -0,0 +1,15 @@
import { Pool } from "pg";
import { config } from "dotenv";
import { join } from "path";
config({ path: join(__dirname, "..", "..", "..", ".env") });
export const pool = new Pool({
host: process.env.POSTGRES_HOST || "localhost",
port: parseInt(process.env.POSTGRES_PORT || "5433"),
database: process.env.POSTGRES_DB || "transceiver_db",
user: process.env.POSTGRES_USER || "tip",
password: process.env.POSTGRES_PASSWORD || "tip_dev_2026",
max: 5,
idleTimeoutMillis: 30000,
});

View File

@ -0,0 +1,413 @@
#!/usr/bin/env node
/**
* TIP MCP HTTP Server SSE Transport
*
* Exposes all 12 TIP MCP tools over HTTP/SSE so the server can be registered
* in Claude Code's ~/.mcp.json as a remote MCP server.
*
* Endpoints:
* GET /health Health check: { status: "ok", tools: 12 }
* GET /sse Opens SSE stream, returns sessionId in endpoint event
* POST /message Client-to-server messages (requires ?sessionId=...)
*
* Auth:
* All endpoints (except /health) require:
* Authorization: Bearer <MCP_SECRET>
*
* Config (env):
* MCP_HTTP_PORT Listening port (default: 3202)
* MCP_SECRET Bearer token for auth (required in production)
* CORS_ORIGINS Comma-separated allowed origins (default: localhost + 127.0.0.1)
*
* ~/.mcp.json entry:
* {
* "tip": {
* "type": "sse",
* "url": "http://localhost:3202/sse",
* "headers": { "Authorization": "Bearer <MCP_SECRET>" }
* }
* }
*/
import express, { type Request, type Response, type NextFunction } from "express";
import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { SSEServerTransport } from "@modelcontextprotocol/sdk/server/sse.js";
import { z } from "zod";
import { pool } from "./db.js";
import { registerPricingTools } from "./tools/pricing.js";
import { registerCompatibilityTools } from "./tools/compatibility.js";
import { registerKnowledgeTools } from "./tools/knowledge.js";
import { registerContentTools } from "./tools/content.js";
import { registerSwitchDocTools } from "./tools/switch-docs.js";
import { registerMarketTools } from "./tools/market.js";
// ---------------------------------------------------------------------------
// Config
// ---------------------------------------------------------------------------
const PORT = parseInt(process.env.MCP_HTTP_PORT ?? "3202", 10);
const MCP_SECRET = process.env.MCP_SECRET ?? "";
const CORS_ORIGINS: string[] = [
"http://localhost",
"http://127.0.0.1",
...(process.env.CORS_ORIGINS ?? "").split(",").map((s) => s.trim()).filter(Boolean),
];
// ---------------------------------------------------------------------------
// Tool count (keep in sync with index.ts tools + tool files)
// search_transceivers, check_compatibility (index.ts) = 2
// pricing.ts: get_pricing, compare_prices, get_competitor_stock = 3
// compatibility.ts: suggest_alternatives, get_templates = 2
// knowledge.ts: search_knowledge_base, search_manuals, get_hype_cycle = 3
// content.ts: get_market_news, generate_blog_draft = 2
// switch-docs.ts: get_switch_docs, search_switches = 2
// market.ts: get_cable_recommendations, get_market_overview, get_technology_roadmap = 3
// Total = 17 registered
// ---------------------------------------------------------------------------
const TOOL_COUNT = 17;
// ---------------------------------------------------------------------------
// Build a new McpServer and register all tools (one server per SSE session)
// ---------------------------------------------------------------------------
async function createMcpServer(): Promise<McpServer> {
const server = new McpServer({
name: "tip-mcp-server",
version: "0.1.0",
});
// --- Tool: search_transceivers ---
server.tool(
"search_transceivers",
"Search transceivers by free text, specs, or compatibility. Returns matching transceivers with current pricing if available.",
{
query: z.string().optional().describe("Free text query, e.g. '10km for Cisco Nexus' or '400G QSFP-DD ZR'"),
form_factor: z.string().optional().describe("SFP, SFP+, SFP28, QSFP+, QSFP28, QSFP-DD, OSFP, CFP2, etc."),
speed_gbps: z.number().optional().describe("Speed in Gbps: 1, 10, 25, 40, 100, 200, 400, 800"),
reach_label: z.string().optional().describe("SR, LR, ER, ZR, or distance like 10km, 80km"),
fiber_type: z.enum(["SMF", "MMF"]).optional().describe("Single-mode or Multi-mode fiber"),
wdm_type: z.enum(["CWDM", "DWDM"]).optional().describe("Wavelength division multiplexing type"),
vendor: z.string().optional().describe("Vendor filter, e.g. 'Cisco', 'Juniper', 'FS.COM'"),
max_results: z.number().default(10).describe("Maximum results to return"),
},
async ({ query, form_factor, speed_gbps, reach_label, fiber_type, wdm_type, vendor, max_results }) => {
const conditions: string[] = [];
const values: unknown[] = [];
let idx = 1;
if (query) {
conditions.push(`t.search_vector @@ plainto_tsquery('english', $${idx})`);
values.push(query);
idx++;
}
if (form_factor) {
conditions.push(`t.form_factor ILIKE $${idx}`);
values.push(`%${form_factor}%`);
idx++;
}
if (speed_gbps) {
conditions.push(`t.speed_gbps = $${idx}`);
values.push(speed_gbps);
idx++;
}
if (reach_label) {
conditions.push(`(t.reach_label ILIKE $${idx} OR t.standard_name ILIKE $${idx})`);
values.push(`%${reach_label}%`);
idx++;
}
if (fiber_type) {
conditions.push(`t.fiber_type = $${idx}`);
values.push(fiber_type);
idx++;
}
if (wdm_type) {
conditions.push(`t.wdm_type = $${idx}`);
values.push(wdm_type);
idx++;
}
if (vendor) {
conditions.push(`v.name ILIKE $${idx}`);
values.push(`%${vendor}%`);
idx++;
}
const where = conditions.length > 0 ? `WHERE ${conditions.join(" AND ")}` : "";
const orderBy = query
? `ORDER BY ts_rank(t.search_vector, plainto_tsquery('english', $1)) DESC`
: "ORDER BY t.speed_gbps DESC, t.reach_meters ASC";
values.push(max_results);
const result = await pool.query(
`SELECT t.id, t.slug, t.standard_name, t.form_factor, t.speed, t.speed_gbps,
t.reach_label, t.reach_meters, t.fiber_type, t.connector, t.wdm_type,
t.wavelengths, t.power_consumption_w, t.temp_range, t.category,
v.name as vendor_name,
(SELECT jsonb_agg(jsonb_build_object(
'vendor', sv.name, 'price', po.price, 'currency', po.currency,
'stock', po.stock_level, 'url', po.url
) ORDER BY po.time DESC)
FROM price_observations po
JOIN vendors sv ON sv.id = po.source_vendor_id
WHERE po.transceiver_id = t.id
AND po.time > NOW() - INTERVAL '7 days'
) as pricing
FROM transceivers t
LEFT JOIN vendors v ON v.id = t.vendor_id
${where}
${orderBy}
LIMIT $${idx}`,
values
);
if (result.rows.length === 0) {
return {
content: [{ type: "text", text: "No transceivers found matching your criteria." }],
};
}
const formatted = result.rows.map((r) => ({
slug: r.slug,
standard: r.standard_name,
form_factor: r.form_factor,
speed: r.speed,
reach: r.reach_label,
fiber: r.fiber_type,
connector: r.connector,
wdm: r.wdm_type,
wavelengths: r.wavelengths,
power_w: r.power_consumption_w,
temp: r.temp_range,
category: r.category,
vendor: r.vendor_name,
pricing: r.pricing || [],
}));
return {
content: [{
type: "text",
text: JSON.stringify({ count: result.rows.length, transceivers: formatted }, null, 2),
}],
};
}
);
// --- Tool: check_compatibility ---
server.tool(
"check_compatibility",
"Check compatibility between a switch model and transceivers. Returns verified compatible transceivers with firmware requirements.",
{
switch_model: z.string().describe("Switch model, e.g. 'Cisco Nexus 93180YC-FX3' or 'Juniper EX4300'"),
transceiver_query: z.string().optional().describe("Optional: filter by transceiver type or part number"),
speed_gbps: z.number().optional().describe("Optional: filter by speed"),
reach: z.string().optional().describe("Optional: filter by reach (SR, LR, etc.)"),
},
async ({ switch_model, transceiver_query, speed_gbps, reach }) => {
const switchResult = await pool.query(
`SELECT s.id, s.model, s.series, v.name as vendor
FROM switches s
JOIN vendors v ON v.id = s.vendor_id
WHERE s.model ILIKE $1 OR s.series ILIKE $1
LIMIT 5`,
[`%${switch_model}%`]
);
if (switchResult.rows.length === 0) {
return {
content: [{
type: "text",
text: `No switch found matching "${switch_model}". Try a shorter model name or check spelling.`,
}],
};
}
const sw = switchResult.rows[0];
const conditions = [`c.switch_id = $1`];
const values: unknown[] = [sw.id];
let idx = 2;
if (transceiver_query) {
conditions.push(`(t.standard_name ILIKE $${idx} OR t.slug ILIKE $${idx})`);
values.push(`%${transceiver_query}%`);
idx++;
}
if (speed_gbps) {
conditions.push(`t.speed_gbps = $${idx}`);
values.push(speed_gbps);
idx++;
}
if (reach) {
conditions.push(`t.reach_label ILIKE $${idx}`);
values.push(`%${reach}%`);
idx++;
}
const compatResult = await pool.query(
`SELECT t.slug, t.standard_name, t.form_factor, t.speed, t.reach_label,
t.fiber_type, c.status, c.firmware_min, c.verified_by, c.verification_method
FROM compatibility c
JOIN transceivers t ON t.id = c.transceiver_id
WHERE ${conditions.join(" AND ")}
AND c.status = 'compatible'
ORDER BY t.speed_gbps DESC, t.reach_meters ASC
LIMIT 20`,
values
);
return {
content: [{
type: "text",
text: JSON.stringify({
switch: { model: sw.model, series: sw.series, vendor: sw.vendor },
compatible_transceivers: compatResult.rows,
count: compatResult.rows.length,
}, null, 2),
}],
};
}
);
// Register remaining tools from tool modules
await registerPricingTools(server);
await registerCompatibilityTools(server);
await registerKnowledgeTools(server);
await registerContentTools(server);
await registerSwitchDocTools(server);
await registerMarketTools(server);
return server;
}
// ---------------------------------------------------------------------------
// Auth middleware
// ---------------------------------------------------------------------------
function requireAuth(req: Request, res: Response, next: NextFunction): void {
if (!MCP_SECRET) {
// No secret configured — skip auth (development mode)
next();
return;
}
const authHeader = req.headers["authorization"] ?? "";
const token = authHeader.startsWith("Bearer ") ? authHeader.slice(7) : "";
if (token !== MCP_SECRET) {
res.status(401).json({ error: "Unauthorized: invalid or missing bearer token" });
return;
}
next();
}
// ---------------------------------------------------------------------------
// CORS middleware
// ---------------------------------------------------------------------------
function applyCors(req: Request, res: Response, next: NextFunction): void {
const origin = req.headers["origin"] ?? "";
const isAllowed = CORS_ORIGINS.some((allowed) =>
origin === allowed || origin.startsWith(allowed)
);
if (isAllowed) {
res.setHeader("Access-Control-Allow-Origin", origin);
}
res.setHeader("Access-Control-Allow-Methods", "GET, POST, OPTIONS");
res.setHeader("Access-Control-Allow-Headers", "Content-Type, Authorization");
res.setHeader("Access-Control-Allow-Credentials", "true");
if (req.method === "OPTIONS") {
res.sendStatus(204);
return;
}
next();
}
// ---------------------------------------------------------------------------
// Session registry: sessionId → SSEServerTransport
// ---------------------------------------------------------------------------
const sessions = new Map<string, SSEServerTransport>();
// ---------------------------------------------------------------------------
// Main
// ---------------------------------------------------------------------------
async function main(): Promise<void> {
const app = express();
app.use(express.json());
app.use(applyCors);
// --- GET /health ---
app.get("/health", (_req: Request, res: Response) => {
res.json({ status: "ok", tools: TOOL_COUNT });
});
// --- GET /sse --- open SSE stream
app.get("/sse", requireAuth, async (req: Request, res: Response) => {
const transport = new SSEServerTransport("/message", res);
// Register session before starting so POST /message can find it immediately
sessions.set(transport.sessionId, transport);
transport.onclose = () => {
sessions.delete(transport.sessionId);
};
// Each SSE connection gets its own McpServer instance
const server = await createMcpServer();
await server.connect(transport);
// Propagate close event from request disconnect
req.on("close", () => {
transport.close().catch(() => {
// ignore errors on close
});
});
});
// --- POST /message --- receive client messages
app.post("/message", requireAuth, async (req: Request, res: Response) => {
const sessionId = req.query["sessionId"] as string | undefined;
if (!sessionId) {
res.status(400).json({ error: "Missing required query parameter: sessionId" });
return;
}
const transport = sessions.get(sessionId);
if (!transport) {
res.status(404).json({ error: `No active SSE session for sessionId: ${sessionId}` });
return;
}
await transport.handlePostMessage(req, res, req.body);
});
const httpServer = app.listen(PORT, () => {
console.log(`TIP MCP HTTP server listening on port ${PORT}`);
console.log(` SSE endpoint: http://localhost:${PORT}/sse`);
console.log(` Message endpoint: http://localhost:${PORT}/message`);
console.log(` Health endpoint: http://localhost:${PORT}/health`);
if (!MCP_SECRET) {
console.warn(" WARNING: MCP_SECRET is not set — auth is disabled (development mode only)");
}
});
// Graceful shutdown
process.on("SIGINT", async () => {
for (const transport of sessions.values()) {
await transport.close().catch(() => {
// ignore errors on close
});
}
sessions.clear();
await pool.end();
httpServer.close(() => {
process.exit(0);
});
});
}
main().catch((err: unknown) => {
console.error("Fatal TIP MCP HTTP server error:", err);
process.exit(1);
});

View File

@ -0,0 +1,365 @@
#!/usr/bin/env node
/**
* TIP MCP Server Transceiver Intelligence Platform
*
* 15 Tools for LLM access to transceiver data, pricing, compatibility,
* hype cycle, knowledge base, news, market intelligence, and blog generation.
*
* Transport: stdio (for Claude Code, EO Global Pulse, etc.)
*
* Usage:
* tsx src/index.ts Run MCP server via stdio
* npx @tip/mcp-server After npm install -g
*
* Claude Code config (~/.claude/mcp.json):
* { "tip": { "command": "npx", "args": ["@tip/mcp-server"] } }
*/
import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
import { z } from "zod";
import { pool } from "./db.js";
import { registerPricingTools } from "./tools/pricing.js";
import { registerCompatibilityTools } from "./tools/compatibility.js";
import { registerKnowledgeTools } from "./tools/knowledge.js";
import { registerContentTools } from "./tools/content.js";
import { registerMarketTools } from "./tools/market.js";
async function main() {
const server = new McpServer({
name: "tip-mcp-server",
version: "0.2.0",
});
// --- Tool: search_transceivers ---
server.tool(
"search_transceivers",
"Search transceivers by free text, specs, or compatibility. Returns matching transceivers with current pricing if available.",
{
query: z.string().optional().describe("Free text query, e.g. '10km for Cisco Nexus' or '400G QSFP-DD ZR'"),
form_factor: z.string().optional().describe("SFP, SFP+, SFP28, QSFP+, QSFP28, QSFP-DD, OSFP, CFP2, etc."),
speed_gbps: z.number().optional().describe("Speed in Gbps: 1, 10, 25, 40, 100, 200, 400, 800"),
reach_label: z.string().optional().describe("SR, LR, ER, ZR, or distance like 10km, 80km"),
fiber_type: z.enum(["SMF", "MMF"]).optional().describe("Single-mode or Multi-mode fiber"),
wdm_type: z.enum(["CWDM", "DWDM"]).optional().describe("Wavelength division multiplexing type"),
vendor: z.string().optional().describe("Vendor/manufacturer filter, e.g. 'Cisco', 'Juniper', 'FS.COM', 'Flexoptix'"),
category: z.string().optional().describe("Category filter: DataCenter, AOC, DAC, DWDM, CWDM, Coherent, Metro, LongHaul, etc."),
market_status: z.enum(["Mainstream", "Growth", "Emerging", "Legacy", "EOL"]).optional().describe("Market status filter"),
max_results: z.number().default(10).describe("Maximum results to return"),
},
async ({ query, form_factor, speed_gbps, reach_label, fiber_type, wdm_type, vendor, category, market_status, max_results }) => {
const conditions: string[] = [];
const values: unknown[] = [];
let idx = 1;
if (query) {
conditions.push(`t.search_vector @@ plainto_tsquery('english', $${idx})`);
values.push(query);
idx++;
}
if (form_factor) {
conditions.push(`t.form_factor ILIKE $${idx}`);
values.push(`%${form_factor}%`);
idx++;
}
if (speed_gbps) {
conditions.push(`t.speed_gbps = $${idx}`);
values.push(speed_gbps);
idx++;
}
if (reach_label) {
conditions.push(`(t.reach_label ILIKE $${idx} OR t.standard_name ILIKE $${idx})`);
values.push(`%${reach_label}%`);
idx++;
}
if (fiber_type) {
conditions.push(`t.fiber_type = $${idx}`);
values.push(fiber_type);
idx++;
}
if (wdm_type) {
conditions.push(`t.wdm_type = $${idx}`);
values.push(wdm_type);
idx++;
}
if (vendor) {
conditions.push(`v.name ILIKE $${idx}`);
values.push(`%${vendor}%`);
idx++;
}
if (category) {
conditions.push(`t.category ILIKE $${idx}`);
values.push(`%${category}%`);
idx++;
}
if (market_status) {
conditions.push(`t.market_status = $${idx}`);
values.push(market_status);
idx++;
}
const where = conditions.length > 0 ? `WHERE ${conditions.join(" AND ")}` : "";
const orderBy = query
? `ORDER BY ts_rank(t.search_vector, plainto_tsquery('english', $1)) DESC`
: "ORDER BY t.speed_gbps DESC, t.reach_meters ASC";
values.push(max_results);
const result = await pool.query(
`SELECT t.id, t.slug, t.standard_name, t.form_factor, t.speed, t.speed_gbps,
t.reach_label, t.reach_meters, t.fiber_type, t.connector, t.wdm_type,
t.wavelengths, t.power_consumption_w, t.temp_range, t.category,
t.market_status, t.hype_cycle_phase,
v.name as vendor_name,
(SELECT jsonb_agg(jsonb_build_object(
'vendor', sv.name, 'price', po.price, 'currency', po.currency,
'stock', po.stock_level, 'url', po.url
) ORDER BY po.time DESC)
FROM price_observations po
JOIN vendors sv ON sv.id = po.source_vendor_id
WHERE po.transceiver_id = t.id
AND po.time > NOW() - INTERVAL '7 days'
) as pricing
FROM transceivers t
LEFT JOIN vendors v ON v.id = t.vendor_id
${where}
${orderBy}
LIMIT $${idx}`,
values
);
if (result.rows.length === 0) {
return {
content: [{ type: "text", text: "No transceivers found matching your criteria." }],
};
}
const formatted = result.rows.map((r) => ({
slug: r.slug,
standard: r.standard_name,
form_factor: r.form_factor,
speed: r.speed,
speed_gbps: r.speed_gbps,
reach: r.reach_label,
fiber: r.fiber_type,
connector: r.connector,
wdm: r.wdm_type,
wavelengths: r.wavelengths,
power_w: r.power_consumption_w,
temp: r.temp_range,
category: r.category,
market_status: r.market_status,
vendor: r.vendor_name,
pricing: r.pricing || [],
}));
return {
content: [{
type: "text",
text: JSON.stringify({ count: result.rows.length, transceivers: formatted }, null, 2),
}],
};
}
);
// --- Tool: check_compatibility ---
server.tool(
"check_compatibility",
"Check compatibility between a switch model and transceivers. Returns verified compatible transceivers with firmware requirements. When no exact match is found, suggests alternative transceivers that may work.",
{
switch_model: z.string().describe("Switch model, e.g. 'Cisco Nexus 93180YC-FX3' or 'Juniper EX4300'"),
transceiver_query: z.string().optional().describe("Optional: filter by transceiver type or part number"),
speed_gbps: z.number().optional().describe("Optional: filter by speed"),
reach: z.string().optional().describe("Optional: filter by reach (SR, LR, etc.)"),
},
async ({ switch_model, transceiver_query, speed_gbps, reach }) => {
// First: find the switch
const switchResult = await pool.query(
`SELECT s.id, s.model, s.series, s.max_speed_gbps, s.ports_config,
v.name as vendor
FROM switches s
JOIN vendors v ON v.id = s.vendor_id
WHERE s.model ILIKE $1 OR s.series ILIKE $1
LIMIT 5`,
[`%${switch_model}%`]
);
if (switchResult.rows.length === 0) {
// Suggest similar switches using trigram similarity
const similarResult = await pool.query(
`SELECT s.model, s.series, v.name as vendor,
similarity(s.model, $1) as sim
FROM switches s
JOIN vendors v ON v.id = s.vendor_id
WHERE similarity(s.model, $1) > 0.1
ORDER BY sim DESC
LIMIT 5`,
[switch_model]
);
const suggestions = similarResult.rows.length > 0
? `\n\nDid you mean one of these?\n${similarResult.rows.map(r => ` - ${r.vendor} ${r.model} (${r.series})`).join("\n")}`
: "";
return {
content: [{
type: "text",
text: `No switch found matching "${switch_model}". Try a shorter model name or check spelling.${suggestions}`,
}],
};
}
const sw = switchResult.rows[0];
const conditions = [`c.switch_id = $1`];
const values: unknown[] = [sw.id];
let idx = 2;
if (transceiver_query) {
conditions.push(`(t.standard_name ILIKE $${idx} OR t.slug ILIKE $${idx})`);
values.push(`%${transceiver_query}%`);
idx++;
}
if (speed_gbps) {
conditions.push(`t.speed_gbps = $${idx}`);
values.push(speed_gbps);
idx++;
}
if (reach) {
conditions.push(`t.reach_label ILIKE $${idx}`);
values.push(`%${reach}%`);
idx++;
}
const compatResult = await pool.query(
`SELECT t.id, t.slug, t.standard_name, t.form_factor, t.speed, t.speed_gbps,
t.reach_label, t.reach_meters, t.fiber_type,
c.status, c.firmware_min, c.verified_by, c.verification_method, c.notes as compat_notes,
(SELECT MIN(po.price) FROM price_observations po
WHERE po.transceiver_id = t.id AND po.time > NOW() - INTERVAL '7 days'
) as min_price
FROM compatibility c
JOIN transceivers t ON t.id = c.transceiver_id
WHERE ${conditions.join(" AND ")}
AND c.status = 'compatible'
ORDER BY t.speed_gbps DESC, t.reach_meters ASC
LIMIT 20`,
values
);
// If no compatible transceivers found, suggest alternatives
let alternatives: unknown[] = [];
if (compatResult.rows.length === 0) {
// Find what form factors / speeds this switch supports based on ports_config
const portSpeeds: number[] = [];
if (sw.max_speed_gbps) portSpeeds.push(parseFloat(sw.max_speed_gbps));
// Try to find transceivers that match the requested criteria even without verified compatibility
const altConditions: string[] = [];
const altValues: unknown[] = [];
let altIdx = 1;
if (transceiver_query) {
altConditions.push(`(t.standard_name ILIKE $${altIdx} OR t.slug ILIKE $${altIdx})`);
altValues.push(`%${transceiver_query}%`);
altIdx++;
}
if (speed_gbps) {
altConditions.push(`t.speed_gbps = $${altIdx}`);
altValues.push(speed_gbps);
altIdx++;
}
if (reach) {
altConditions.push(`t.reach_label ILIKE $${altIdx}`);
altValues.push(`%${reach}%`);
altIdx++;
}
const altWhere = altConditions.length > 0 ? `WHERE ${altConditions.join(" AND ")}` : "";
const altResult = await pool.query(
`SELECT t.slug, t.standard_name, t.form_factor, t.speed, t.speed_gbps,
t.reach_label, t.fiber_type, v.name as vendor,
(SELECT MIN(po.price) FROM price_observations po
WHERE po.transceiver_id = t.id AND po.time > NOW() - INTERVAL '7 days'
) as min_price,
-- Check if this transceiver is compatible with OTHER switches from the same vendor
(SELECT COUNT(*) FROM compatibility c2
JOIN switches sw2 ON sw2.id = c2.switch_id
WHERE c2.transceiver_id = t.id
AND c2.status = 'compatible'
AND sw2.vendor_id = (SELECT vendor_id FROM switches WHERE id = '${sw.id}')
) as same_vendor_compat_count
FROM transceivers t
LEFT JOIN vendors v ON v.id = t.vendor_id
${altWhere}
ORDER BY same_vendor_compat_count DESC, t.speed_gbps DESC
LIMIT 10`,
altValues
);
alternatives = altResult.rows.map(r => ({
...r,
min_price: r.min_price ? parseFloat(r.min_price) : null,
compatibility_note: parseInt(r.same_vendor_compat_count) > 0
? `Compatible with ${r.same_vendor_compat_count} other ${sw.vendor} switches — likely compatible but NOT verified for ${sw.model}`
: "Not verified for any switches from this vendor. Test before deploying.",
}));
}
return {
content: [{
type: "text",
text: JSON.stringify({
switch: {
model: sw.model,
series: sw.series,
vendor: sw.vendor,
max_speed_gbps: sw.max_speed_gbps,
},
compatible_transceivers: compatResult.rows.map(r => ({
slug: r.slug,
standard: r.standard_name,
form_factor: r.form_factor,
speed: r.speed,
reach: r.reach_label,
fiber: r.fiber_type,
status: r.status,
firmware_min: r.firmware_min,
verified_by: r.verified_by,
method: r.verification_method,
notes: r.compat_notes,
min_price: r.min_price ? parseFloat(r.min_price) : null,
})),
count: compatResult.rows.length,
...(compatResult.rows.length === 0 && alternatives.length > 0 ? {
no_verified_match: true,
suggested_alternatives: alternatives,
suggestion_note: `No verified compatible transceivers found for ${sw.vendor} ${sw.model}. These alternatives may work based on specs and compatibility with similar ${sw.vendor} switches, but should be tested before production deployment.`,
} : {}),
}, null, 2),
}],
};
}
);
// Register tool modules
await registerPricingTools(server);
await registerCompatibilityTools(server);
await registerKnowledgeTools(server);
await registerContentTools(server);
await registerMarketTools(server);
// Start server
const transport = new StdioServerTransport();
await server.connect(transport);
// Graceful shutdown
process.on("SIGINT", async () => {
await pool.end();
process.exit(0);
});
}
main().catch((err) => {
console.error("Fatal MCP server error:", err);
process.exit(1);
});

View File

@ -0,0 +1,144 @@
/**
* Compatibility & template tools: suggest_alternatives, get_templates
*/
import type { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { z } from "zod";
import { pool } from "../db.js";
export async function registerCompatibilityTools(server: McpServer): Promise<void> {
// --- Tool: suggest_alternatives ---
server.tool(
"suggest_alternatives",
"Suggest alternative transceivers with similar specs. Useful for finding cheaper or more available options.",
{
part_number: z.string().describe("Part number, slug, or standard name"),
optimize_for: z.enum(["price", "availability", "performance"]).default("price"),
},
async ({ part_number, optimize_for }) => {
// Find the reference transceiver
const refResult = await pool.query(
`SELECT t.id, t.slug, t.standard_name, t.form_factor, t.speed_gbps,
t.reach_meters, t.fiber_type, t.wdm_type, t.connector
FROM transceivers t
WHERE t.slug ILIKE $1 OR t.part_number ILIKE $1 OR t.standard_name ILIKE $1
LIMIT 1`,
[`%${part_number}%`]
);
if (refResult.rows.length === 0) {
return {
content: [{ type: "text", text: `Reference transceiver not found: "${part_number}"` }],
};
}
const ref = refResult.rows[0];
// Find alternatives: same form_factor + speed, similar reach
const altResult = await pool.query(
`SELECT t.id, t.slug, t.standard_name, t.form_factor, t.speed, t.reach_label,
t.fiber_type, t.connector, v.name as vendor,
(SELECT MIN(po.price) FROM price_observations po
WHERE po.transceiver_id = t.id
AND po.time > NOW() - INTERVAL '7 days'
) as min_price,
(SELECT COUNT(DISTINCT po.source_vendor_id) FROM price_observations po
WHERE po.transceiver_id = t.id
AND po.stock_level = 'in_stock'
AND po.time > NOW() - INTERVAL '7 days'
) as vendor_count
FROM transceivers t
LEFT JOIN vendors v ON v.id = t.vendor_id
WHERE t.form_factor = $1
AND t.speed_gbps = $2
AND t.fiber_type = $3
AND ABS(t.reach_meters - $4) <= $4 * 0.2
AND t.id != $5
ORDER BY
CASE $6
WHEN 'price' THEN (SELECT MIN(po.price) FROM price_observations po WHERE po.transceiver_id = t.id)
WHEN 'availability' THEN -(SELECT COUNT(DISTINCT po.source_vendor_id) FROM price_observations po WHERE po.transceiver_id = t.id AND po.stock_level = 'in_stock')
ELSE t.power_consumption_w
END ASC NULLS LAST
LIMIT 10`,
[ref.form_factor, ref.speed_gbps, ref.fiber_type, ref.reach_meters, ref.id, optimize_for]
);
return {
content: [{
type: "text",
text: JSON.stringify({
reference: {
slug: ref.slug,
standard: ref.standard_name,
form_factor: ref.form_factor,
speed_gbps: ref.speed_gbps,
reach_meters: ref.reach_meters,
},
optimize_for,
alternatives: altResult.rows,
}, null, 2),
}],
};
}
);
// --- Tool: get_templates ---
server.tool(
"get_templates",
"Find FlexBox coding templates or switch configuration templates for specific vendor/technology combinations.",
{
type: z.enum(["flexbox_coding", "switch_config"]).describe("Template type"),
switch_vendor: z.string().optional().describe("Switch vendor, e.g. 'Cisco', 'Juniper', 'Arista'"),
transceiver_type: z.string().optional().describe("e.g. 'SFP+', 'QSFP28', '10G', '100G'"),
technology: z.string().optional().describe("CWDM, DWDM, BiDi, SR, LR, ER, ZR, etc."),
},
async ({ type, switch_vendor, transceiver_type, technology }) => {
const conditions = [`t.template_type = $1`];
const values: unknown[] = [type];
let idx = 2;
if (switch_vendor) {
conditions.push(`t.switch_vendor ILIKE $${idx}`);
values.push(`%${switch_vendor}%`);
idx++;
}
if (transceiver_type) {
conditions.push(`t.transceiver_type ILIKE $${idx}`);
values.push(`%${transceiver_type}%`);
idx++;
}
if (technology) {
conditions.push(`t.technology ILIKE $${idx}`);
values.push(`%${technology}%`);
idx++;
}
const result = await pool.query(
`SELECT t.id, t.name, t.template_type, t.switch_vendor, t.transceiver_type,
t.technology, t.content, t.description, t.notes,
t.verified, t.firmware_version
FROM templates t
WHERE ${conditions.join(" AND ")}
ORDER BY t.verified DESC, t.name ASC
LIMIT 20`,
values
);
if (result.rows.length === 0) {
return {
content: [{
type: "text",
text: `No ${type} templates found for the given criteria. Templates are added as they are verified in the field.`,
}],
};
}
return {
content: [{
type: "text",
text: JSON.stringify({ templates: result.rows, count: result.rows.length }, null, 2),
}],
};
}
);
}

View File

@ -0,0 +1,233 @@
/**
* Content tools: get_market_news, generate_blog_draft
*/
import type { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { z } from "zod";
import { pool } from "../db.js";
export async function registerContentTools(server: McpServer): Promise<void> {
// --- Tool: get_market_news ---
server.tool(
"get_market_news",
"Get latest news from the optics and networking industry. Filtered by relevance to transceiver technology.",
{
query: z.string().optional().describe("Optional: search within news content"),
event: z.enum(["OFC", "ECOC", "CIOE", "PHOTONICS_WEST"]).optional().describe("Filter by trade show/event"),
days_back: z.number().default(30).describe("How many days back to look (default: 30)"),
min_relevance: z.number().default(0).describe("Minimum relevance score (0=all, 3=transceiver-related, 9=highly relevant)"),
},
async ({ query, event, days_back, min_relevance }) => {
const conditions = [`na.published_at > NOW() - INTERVAL '${days_back} days'`];
const values: unknown[] = [];
let idx = 1;
if (query) {
conditions.push(`(na.search_vector @@ plainto_tsquery('english', $${idx}) OR na.title ILIKE $${idx + 1})`);
values.push(query, `%${query}%`);
idx += 2;
}
if (event) {
conditions.push(`na.event = $${idx}`);
values.push(event);
idx++;
}
if (min_relevance > 0) {
conditions.push(`na.relevance_score >= $${idx}`);
values.push(min_relevance);
idx++;
}
const result = await pool.query(
`SELECT na.title, na.source, na.published_at, na.summary,
na.source_url, na.relevance_score, na.event,
na.mentioned_vendors, na.mentioned_products
FROM news_articles na
WHERE ${conditions.join(" AND ")}
ORDER BY na.relevance_score DESC, na.published_at DESC
LIMIT 20`,
values
);
const bySource: Record<string, number> = {};
for (const row of result.rows) {
bySource[row.source] = (bySource[row.source] || 0) + 1;
}
return {
content: [{
type: "text",
text: JSON.stringify({
articles: result.rows,
count: result.rows.length,
sources: bySource,
period: `Last ${days_back} days`,
}, null, 2),
}],
};
}
);
// --- Tool: generate_blog_draft ---
server.tool(
"generate_blog_draft",
"Generate a blog post draft based on market data, price trends, hype cycle position, and recent news. Saved to blog_drafts table for review.",
{
topic: z.enum(["hype_cycle", "price_trend", "new_product", "comparison", "tutorial"]),
technology: z.string().optional().describe("Technology to focus on, e.g. '800G OSFP', 'DWDM', '400G'"),
target_audience: z.enum(["sales", "technical", "customer", "seo"]).default("technical"),
},
async ({ topic, technology, target_audience }) => {
// Gather data for the blog post
const data: Record<string, unknown> = { topic, technology, target_audience };
// Get relevant transceivers
if (technology) {
const txResult = await pool.query(
`SELECT t.standard_name, t.form_factor, t.speed, t.reach_label,
t.fiber_type, t.category,
(SELECT MIN(po.price) FROM price_observations po
WHERE po.transceiver_id = t.id
AND po.time > NOW() - INTERVAL '7 days') as min_price,
(SELECT MAX(po.price) FROM price_observations po
WHERE po.transceiver_id = t.id
AND po.time > NOW() - INTERVAL '7 days') as max_price
FROM transceivers t
WHERE t.standard_name ILIKE $1 OR t.speed ILIKE $1
LIMIT 10`,
[`%${technology}%`]
);
data.transceivers = txResult.rows;
}
// Get recent news
const newsResult = await pool.query(
`SELECT title, source, published_at, summary
FROM news_articles
WHERE ($1 IS NULL OR title ILIKE $1 OR summary ILIKE $1)
ORDER BY relevance_score DESC, published_at DESC
LIMIT 5`,
[technology ? `%${technology}%` : null]
);
data.recent_news = newsResult.rows;
// Get price trends
const priceResult = await pool.query(
`SELECT t.standard_name, t.speed,
AVG(po.price) as avg_price,
MIN(po.price) as min_price,
COUNT(DISTINCT po.source_vendor_id) as vendor_count
FROM price_observations po
JOIN transceivers t ON t.id = po.transceiver_id
WHERE ($1 IS NULL OR t.standard_name ILIKE $1 OR t.speed ILIKE $1)
AND po.time > NOW() - INTERVAL '30 days'
GROUP BY t.id, t.standard_name, t.speed
ORDER BY vendor_count DESC
LIMIT 10`,
[technology ? `%${technology}%` : null]
);
data.price_trends = priceResult.rows;
// Generate blog outline based on topic and audience
const outlines: Record<string, Record<string, string[]>> = {
hype_cycle: {
sales: [
`## Where is ${technology || "The Market"} on the Hype Cycle?`,
"## What This Means for Your Customers",
"## When to Buy: Timing the Market",
"## Flexoptix Recommendation",
],
technical: [
`## ${technology || "Technology"} Market Analysis — Norton-Bass Diffusion Model`,
"## Current Phase: Technical Readiness",
"## Vendor Ecosystem Status",
"## Price Trajectory & ASP Forecast",
"## Deployment Considerations",
],
customer: [
`## Is ${technology || "This Technology"} Right for You?`,
"## Cost vs. Performance Analysis",
"## Compatibility & Migration Path",
"## When Will Prices Drop?",
],
seo: [
`## ${technology || "Optical Transceiver"} Market 2026: Complete Guide`,
"## Best Vendors & Pricing Comparison",
"## Compatibility Guide",
"## FAQ",
],
},
price_trend: {
sales: ["## Price Alert", "## Competitor Pricing Analysis", "## Sales Opportunity"],
technical: ["## Price Trend Analysis", "## ASP History", "## Market Drivers"],
customer: ["## How Much Should You Pay?", "## Price Forecast", "## When to Buy"],
seo: ["## Price Guide 2026", "## Best Deals", "## Comparison Table"],
},
comparison: {
sales: ["## Why Flexoptix Beats the Competition", "## Price Advantage", "## Quality & Compatibility"],
technical: ["## Vendor Comparison", "## Spec Analysis", "## Performance Benchmarks"],
customer: ["## OEM vs. Compatible: The Facts", "## Risk Analysis", "## Cost Savings"],
seo: ["## Best Transceiver Vendors 2026", "## Comparison Table", "## Reviews"],
},
tutorial: {
technical: ["## Prerequisites", "## Step-by-Step Configuration", "## Troubleshooting", "## Verification"],
customer: ["## Getting Started", "## Installation Guide", "## Tips & Tricks"],
sales: ["## Product Overview", "## Use Cases", "## Getting Support"],
seo: ["## How To Guide", "## Step-by-Step", "## FAQ"],
},
new_product: {
sales: ["## New Product Alert", "## What's New", "## Pricing & Availability"],
technical: ["## Technical Specs", "## Compatibility Matrix", "## Performance Data"],
customer: ["## What You Get", "## Why You Need It", "## How to Order"],
seo: ["## Product Announcement", "## Specs & Features", "## Where to Buy"],
},
};
const outline = outlines[topic]?.[target_audience] || [];
// Build draft content
const draft = {
title: `${technology || "Optical Transceiver"} ${topic.replace(/_/g, " ")}${new Date().getFullYear()} Analysis`,
topic,
technology,
target_audience,
outline,
data_points: data,
generation_note: "This is a data-driven draft. Review and enrich with specific product details before publishing.",
generated_at: new Date().toISOString(),
status: "draft",
};
// Save to blog_drafts table
await pool.query(
`INSERT INTO blog_drafts (title, topic, technology, target_audience, outline, draft_content, status)
VALUES ($1, $2, $3, $4, $5, $6, 'draft')
ON CONFLICT DO NOTHING`,
[
draft.title,
topic,
technology || null,
target_audience,
JSON.stringify(outline),
JSON.stringify(draft),
]
);
return {
content: [{
type: "text",
text: JSON.stringify({
draft,
saved_to_database: true,
next_steps: [
"Review the outline and data points",
"Enrich with specific product examples from search_transceivers",
"Add competitor pricing from compare_prices",
"Include current news context from get_market_news",
"Submit to content team for writing/editing",
],
}, null, 2),
}],
};
}
);
}

View File

@ -0,0 +1,175 @@
/**
* MCP Tool: find_flexoptix_for_switch
*
* "Customer has Switch X — which Flexoptix transceivers should they buy?"
*/
import { pool } from "../db";
export const finderTools = {
find_flexoptix_for_switch: {
name: "find_flexoptix_for_switch",
description: "Find the right Flexoptix transceivers for a customer's switch. Input a switch model name and get compatible Flexoptix products with prices, shop links, and FlexBox coding info.",
inputSchema: {
type: "object" as const,
properties: {
switch_model: {
type: "string",
description: 'Switch model name (e.g., "Cisco Nexus 93180YC-FX3", "QFX5120-48Y", "DCS-7050SX3-48YC12")',
},
speed_gbps: {
type: "number",
description: "Filter by port speed in Gbps (10, 25, 40, 100, 400)",
},
reach: {
type: "string",
description: "Filter by reach (SR, LR, ER, ZR, or specific like 10km, 80km)",
},
},
required: ["switch_model"],
},
},
plan_transport: {
name: "plan_transport",
description: "Plan a fiber transport system between two cities. Returns switch, transceiver, and fiber provider recommendations with bill of materials and Flexoptix pricing.",
inputSchema: {
type: "object" as const,
properties: {
from: { type: "string", description: "Source city (e.g., Berlin, Frankfurt, Amsterdam)" },
to: { type: "string", description: "Destination city (e.g., Darmstadt, Munich, London)" },
bandwidth_gbps: { type: "number", description: "Required bandwidth in Gbps (default: 100)" },
redundancy: { type: "boolean", description: "Whether to include redundant path (default: false)" },
},
required: ["from", "to"],
},
},
forecast_sales: {
name: "forecast_sales",
description: "Predict transceiver sales volume and price trajectory for a technology over 3/9/12/18 months. Includes buy/wait/hold signal.",
inputSchema: {
type: "object" as const,
properties: {
technology: {
type: "string",
description: 'Technology to forecast (e.g., "400G QSFP-DD", "100G QSFP28", "800G OSFP", "1.6T OSFP-XD")',
},
},
required: ["technology"],
},
},
get_competitor_alerts: {
name: "get_competitor_alerts",
description: "Get recent competitor intelligence: new products, price changes, stock changes. Shows what competitors are doing in the market.",
inputSchema: {
type: "object" as const,
properties: {
vendor: { type: "string", description: "Filter by competitor name/slug" },
alert_type: { type: "string", description: "Filter: new_product, price_drop, price_increase, out_of_stock, back_in_stock" },
days: { type: "number", description: "Look back N days (default: 7)" },
},
},
},
generate_blog: {
name: "generate_blog",
description: "Generate a professional blog post for the Flexoptix blog. Auto-enriched with pricing data, competitor analysis, and product links.",
inputSchema: {
type: "object" as const,
properties: {
topic: { type: "string", description: "Blog topic or title" },
type: {
type: "string",
description: "Blog type: market_alert, migration_guide, competitor_analysis, technology_deep_dive, buying_guide, tutorial, comparison",
},
target_audience: { type: "string", description: "Audience: technical, sales, customer (default: technical)" },
include_products: { type: "boolean", description: "Include Flexoptix product recommendations (default: true)" },
word_count: { type: "number", description: "Target word count (default: 2000)" },
},
required: ["topic"],
},
},
};
export async function handleFinderTool(name: string, args: Record<string, any>): Promise<string> {
switch (name) {
case "find_flexoptix_for_switch": {
const { switch_model, speed_gbps, reach } = args;
// Find switch
const sw = await pool.query(
`SELECT sw.id, sw.model, sw.series, sw.ports_config, sw.max_speed_gbps, v.name AS vendor
FROM switches sw JOIN vendors v ON sw.vendor_id = v.id
WHERE sw.model ILIKE '%' || $1 || '%' OR sw.search_vector @@ plainto_tsquery('english', $1)
ORDER BY CASE WHEN sw.model ILIKE $1 THEN 0 ELSE 1 END LIMIT 3`,
[switch_model]
);
if (!sw.rows[0]) {
return JSON.stringify({ error: `Switch "${switch_model}" not found. Try a partial model name.` });
}
// Find compatible transceivers with Flexoptix products
let sql = `
SELECT t.form_factor, t.speed_gbps, t.reach_label, t.fiber_type, t.connector,
t.image_url, v.name AS vendor, c.firmware_min,
(SELECT po.price FROM price_observations po WHERE po.transceiver_id = t.id ORDER BY po.time DESC LIMIT 1) AS price,
(SELECT po.currency FROM price_observations po WHERE po.transceiver_id = t.id ORDER BY po.time DESC LIMIT 1) AS currency
FROM compatibility c
JOIN transceivers t ON c.transceiver_id = t.id
JOIN vendors v ON t.vendor_id = v.id
WHERE c.switch_id = $1 AND c.status = 'compatible'
`;
const params: any[] = [sw.rows[0].id];
let idx = 2;
if (speed_gbps) { sql += ` AND t.speed_gbps = $${idx}`; params.push(speed_gbps); idx++; }
if (reach) { sql += ` AND t.reach_label ILIKE $${idx}`; params.push(reach + '%'); idx++; }
sql += ` ORDER BY t.speed_gbps DESC, t.reach_meters ASC LIMIT 30`;
const compat = await pool.query(sql, params);
return JSON.stringify({
switch: { model: sw.rows[0].model, vendor: sw.rows[0].vendor, ports: sw.rows[0].ports_config },
compatible_count: compat.rowCount,
transceivers: compat.rows.map(r => ({
...r,
flexbox_note: "All Flexoptix transceivers support FlexBox coding — one transceiver works in any vendor's switch.",
buy_url: `https://www.flexoptix.net/en/catalogsearch/result/?q=${encodeURIComponent(r.form_factor + ' ' + r.speed_gbps + 'G ' + r.reach_label)}`,
})),
}, null, 2);
}
case "get_competitor_alerts": {
const { vendor, alert_type, days = 7 } = args;
let sql = `SELECT ca.alert_type, ca.severity, ca.part_number, ca.product_name,
ca.old_price, ca.new_price, ca.price_pct, ca.currency, ca.source_url,
v.name AS vendor, ca.created_at
FROM competitor_alerts ca LEFT JOIN vendors v ON ca.vendor_id = v.id
WHERE ca.created_at > NOW() - INTERVAL '1 day' * $1`;
const params: any[] = [days];
let idx = 2;
if (vendor) { sql += ` AND v.slug ILIKE $${idx}`; params.push('%' + vendor + '%'); idx++; }
if (alert_type) { sql += ` AND ca.alert_type = $${idx}`; params.push(alert_type); idx++; }
sql += ` ORDER BY ca.created_at DESC LIMIT 30`;
const result = await pool.query(sql, params);
return JSON.stringify({ alerts: result.rows, count: result.rowCount }, null, 2);
}
case "plan_transport":
case "forecast_sales":
case "generate_blog":
// These forward to the API routes — return instruction to use HTTP API
return JSON.stringify({
note: `Use the TIP HTTP API for ${name}. See https://transceiver-db.context-x.org/api for endpoints.`,
endpoint: name === "plan_transport" ? "POST /api/transport/plan" :
name === "forecast_sales" ? "GET /api/forecast/:technology" :
"POST /api/blog/generate",
args,
});
default:
return JSON.stringify({ error: `Unknown tool: ${name}` });
}
}

View File

@ -0,0 +1,278 @@
/**
* Knowledge tools: search_knowledge_base, search_manuals, get_hype_cycle
*/
import type { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { z } from "zod";
import { pool } from "../db.js";
// Norton-Bass diffusion model (inline, since MCP server has separate build)
interface TechGen { name: string; speedGbps: number; formFactor: string; introYear: number; peakYear: number; p: number; q: number; m: number; k: number; t0: number; }
const TECH_GENS: TechGen[] = [
{ name: "1G SFP", speedGbps: 1, formFactor: "SFP", introYear: 2001, peakYear: 2012, p: 0.03, q: 0.38, m: 500, k: 0.45, t0: 2008 },
{ name: "10G SFP+", speedGbps: 10, formFactor: "SFP+", introYear: 2006, peakYear: 2018, p: 0.03, q: 0.42, m: 600, k: 0.50, t0: 2014 },
{ name: "25G SFP28", speedGbps: 25, formFactor: "SFP28", introYear: 2015, peakYear: 2022, p: 0.04, q: 0.45, m: 200, k: 0.55, t0: 2019 },
{ name: "40G QSFP+", speedGbps: 40, formFactor: "QSFP+", introYear: 2010, peakYear: 2019, p: 0.025, q: 0.35, m: 150, k: 0.40, t0: 2016 },
{ name: "100G QSFP28", speedGbps: 100, formFactor: "QSFP28", introYear: 2014, peakYear: 2024, p: 0.03, q: 0.40, m: 400, k: 0.48, t0: 2020 },
{ name: "400G QSFP-DD", speedGbps: 400, formFactor: "QSFP-DD", introYear: 2020, peakYear: 2027, p: 0.035, q: 0.50, m: 300, k: 0.52, t0: 2025 },
{ name: "800G OSFP", speedGbps: 800, formFactor: "OSFP", introYear: 2023, peakYear: 2029, p: 0.04, q: 0.55, m: 250, k: 0.55, t0: 2027 },
{ name: "1.6T OSFP-XD", speedGbps: 1600, formFactor: "OSFP-XD", introYear: 2025, peakYear: 2032, p: 0.03, q: 0.45, m: 180, k: 0.48, t0: 2030 },
{ name: "CPO", speedGbps: 1600, formFactor: "CPO", introYear: 2024, peakYear: 2033, p: 0.02, q: 0.30, m: 50, k: 0.35, t0: 2031 },
{ name: "400ZR Coherent", speedGbps: 400, formFactor: "QSFP-DD", introYear: 2021, peakYear: 2026, p: 0.04, q: 0.50, m: 80, k: 0.55, t0: 2024 },
{ name: "LPO", speedGbps: 800, formFactor: "LPO", introYear: 2024, peakYear: 2029, p: 0.035, q: 0.48, m: 100, k: 0.50, t0: 2027 },
];
function bassCumulative(t: number, p: number, q: number): number {
const pq = p + q;
const exp = Math.exp(-pq * t);
return (1 - exp) / (1 + (q / p) * exp);
}
function bassRate(t: number, p: number, q: number): number {
const pq = p + q;
const exp = Math.exp(-pq * t);
const d = 1 + (q / p) * exp;
return (pq * pq / q) * (exp / (d * d));
}
function classifyHypePhase(progressRatio: number): string {
if (progressRatio < 0.15) return "Innovation Trigger";
if (progressRatio < 0.35) return "Peak of Inflated Expectations";
if (progressRatio < 0.55) return "Trough of Disillusionment";
if (progressRatio < 0.85) return "Slope of Enlightenment";
if (progressRatio < 1.3) return "Plateau of Productivity";
return "Legacy / Decline";
}
function findTech(query: string): TechGen | undefined {
const q = query.toLowerCase().trim();
return TECH_GENS.find(t => t.name.toLowerCase().includes(q))
|| TECH_GENS.find(t => q.includes(t.formFactor.toLowerCase()))
|| (() => {
const m = q.match(/^(\d+(?:\.\d+)?)\s*(g|t)\b/i);
if (m) {
const gbps = m[2].toLowerCase() === "t" ? parseFloat(m[1]) * 1000 : parseFloat(m[1]);
return TECH_GENS.find(t => t.speedGbps === gbps);
}
return undefined;
})();
}
export async function registerKnowledgeTools(server: McpServer): Promise<void> {
// --- Tool: search_knowledge_base ---
server.tool(
"search_knowledge_base",
"Search troubleshooting guides, FAQs, and best practices. Use for diagnosing transceiver/switch issues.",
{
query: z.string().describe("Problem description or question, e.g. 'low Rx power QSFP28' or 'link flapping SFP+'"),
category: z.enum(["troubleshooting", "faq", "best_practice", "configuration"]).optional(),
severity: z.enum(["critical", "high", "medium", "low"]).optional().describe("Filter by issue severity"),
},
async ({ query, category, severity }) => {
const conditions = [`kb.search_vector @@ plainto_tsquery('english', $1)`];
const values: unknown[] = [query];
let idx = 2;
if (category) {
conditions.push(`kb.category = $${idx}`);
values.push(category);
idx++;
}
if (severity) {
conditions.push(`kb.severity = $${idx}`);
values.push(severity);
idx++;
}
const result = await pool.query(
`SELECT kb.id, kb.title, kb.content, kb.category, kb.severity,
kb.affected_products, kb.resolution, kb.source_url,
ts_rank(kb.search_vector, plainto_tsquery('english', $1)) as rank
FROM knowledge_base kb
WHERE ${conditions.join(" AND ")}
ORDER BY rank DESC
LIMIT 10`,
values
);
if (result.rows.length === 0) {
return {
content: [{
type: "text",
text: `No knowledge base entries found for: "${query}". Knowledge base grows as vendor FAQs are scraped.`,
}],
};
}
return {
content: [{
type: "text",
text: JSON.stringify({ results: result.rows, count: result.rows.length }, null, 2),
}],
};
}
);
// --- Tool: search_manuals ---
server.tool(
"search_manuals",
"Search switch manuals, configuration guides, and compatibility lists. Returns relevant document excerpts.",
{
query: z.string().describe("Query, e.g. 'configure DWDM on Juniper MX' or 'QSFP28 installation guide'"),
vendor: z.string().optional().describe("Document vendor filter, e.g. 'Cisco', 'Juniper', 'Arista'"),
doc_type: z.enum(["manual", "config_guide", "compatibility_list", "datasheet", "release_notes"]).optional(),
},
async ({ query, vendor, doc_type }) => {
const conditions = [`d.search_vector @@ plainto_tsquery('english', $1)`];
const values: unknown[] = [query];
let idx = 2;
if (vendor) {
conditions.push(`v.name ILIKE $${idx}`);
values.push(`%${vendor}%`);
idx++;
}
if (doc_type) {
conditions.push(`d.doc_type = $${idx}`);
values.push(doc_type);
idx++;
}
const result = await pool.query(
`SELECT d.id, d.title, d.doc_type, d.version,
v.name as vendor, d.source_url, d.pages,
d.summary,
ts_rank(d.search_vector, plainto_tsquery('english', $1)) as rank
FROM documents d
LEFT JOIN vendors v ON v.id = d.vendor_id
WHERE ${conditions.join(" AND ")}
ORDER BY rank DESC
LIMIT 10`,
values
);
if (result.rows.length === 0) {
return {
content: [{
type: "text",
text: `No manuals found for: "${query}". Document library grows as manuals are processed by OCR pipeline.`,
}],
};
}
return {
content: [{
type: "text",
text: JSON.stringify({ results: result.rows, count: result.rows.length }, null, 2),
}],
};
}
);
// --- Tool: get_hype_cycle ---
server.tool(
"get_hype_cycle",
"Get Hype Cycle position and Norton-Bass diffusion forecast for a transceiver technology. Shows market maturity, adoption phase, and price trajectory.",
{
technology: z.string().describe("Technology to analyze, e.g. '800G OSFP', 'CPO', '400ZR', '100G LR4', 'DWDM'"),
include_forecast: z.boolean().default(true).describe("Include Norton-Bass 5-year adoption forecast"),
},
async ({ technology, include_forecast }) => {
// Look for market metrics data
const result = await pool.query(
`SELECT mm.time, mm.metric_type, mm.value,
mm.notes
FROM market_metrics mm
WHERE mm.technology ILIKE $1
OR mm.notes ILIKE $1
ORDER BY mm.time DESC
LIMIT 50`,
[`%${technology}%`]
);
// Also check news for recent developments
const newsResult = await pool.query(
`SELECT title, published_at, source, summary
FROM news_articles
WHERE search_vector @@ plainto_tsquery('english', $1)
OR title ILIKE $2
ORDER BY published_at DESC
LIMIT 5`,
[technology, `%${technology}%`]
);
// Norton-Bass Diffusion Model computation
const currentYear = new Date().getFullYear();
const tech = findTech(technology);
let hypeData: Record<string, unknown> | null = null;
let forecast: Record<string, unknown> | undefined = undefined;
if (tech) {
const t = Math.max(0, currentYear - tech.introYear);
const yearsToPeak = tech.peakYear - tech.introYear;
const progressRatio = t / yearsToPeak;
const adoption = bassCumulative(t, tech.p, tech.q);
const rate = bassRate(t, tech.p, tech.q);
const phase = classifyHypePhase(progressRatio);
// Position on curve (0-100) based on progress ratio
const positionPct = Math.min(100, Math.round(
progressRatio < 0.15 ? progressRatio / 0.15 * 15 :
progressRatio < 0.35 ? 15 + (progressRatio - 0.15) / 0.20 * 15 :
progressRatio < 0.55 ? 30 + (progressRatio - 0.35) / 0.20 * 20 :
progressRatio < 0.85 ? 50 + (progressRatio - 0.55) / 0.30 * 30 :
80 + Math.min(20, (progressRatio - 0.85) / 0.45 * 20)
));
hypeData = {
technology: tech.name,
phase,
position: positionPct,
adoption_pct: Math.round(adoption * 100),
yearly_adoption_rate: Math.round(rate * 100),
intro_year: tech.introYear,
peak_year: tech.peakYear,
years_to_plateau: Math.max(0, tech.peakYear + 3 - currentYear),
model: "Norton-Bass Multigenerational Diffusion",
parameters: { p: tech.p, q: tech.q, m: tech.m, k: tech.k, t0: tech.t0 },
};
if (include_forecast) {
const fiveYear = Array.from({ length: 5 }, (_, i) => {
const yr = currentYear + i + 1;
const ft = yr - tech.introYear;
const fa = bassCumulative(ft, tech.p, tech.q);
const pr = ft / yearsToPeak;
return { year: yr, adoption_pct: Math.round(fa * 100), phase: classifyHypePhase(pr) };
});
const aspDecline = progressRatio < 0.3 ? 5 : progressRatio < 0.6 ? 35 : progressRatio < 1.0 ? 15 : 5;
forecast = {
model: "Norton-Bass Multigenerational Diffusion",
current_adoption_pct: Math.round(adoption * 100),
estimated_peak_year: tech.peakYear,
years_to_plateau: Math.max(0, tech.peakYear + 3 - currentYear),
asp_decline_rate_pct: aspDecline,
price_trend: aspDecline > 20 ? "Rapidly declining" : aspDecline > 10 ? "Moderately declining" : "Stabilizing",
five_year_projection: fiveYear,
};
}
}
return {
content: [{
type: "text",
text: JSON.stringify({
technology,
hype_cycle: hypeData || { note: `No model data for "${technology}". Known: 1G, 10G, 25G, 40G, 100G, 400G, 800G, 1.6T, CPO, LPO, 400ZR` },
market_metrics: result.rows,
recent_news: newsResult.rows,
norton_bass_forecast: forecast,
}, null, 2),
}],
};
}
);
}

View File

@ -0,0 +1,458 @@
/**
* Market intelligence tools: get_cable_recommendations, get_market_overview, get_technology_roadmap
*/
import type { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { z } from "zod";
import { pool } from "../db.js";
export async function registerMarketTools(server: McpServer): Promise<void> {
// --- Tool: get_cable_recommendations ---
server.tool(
"get_cable_recommendations",
"Recommend DAC (Direct Attach Copper) or AOC (Active Optical Cable) cables based on switch ports and distance requirements. Considers port type, speed, reach, and budget to suggest the optimal cable solution.",
{
switch_model: z.string().optional().describe("Switch model for port compatibility check, e.g. 'Nexus 93180YC-FX3'"),
port_speed_gbps: z.number().describe("Required port speed in Gbps: 10, 25, 40, 100, 200, 400"),
distance_meters: z.number().describe("Required cable run distance in meters (0.5 to 100)"),
form_factor: z.string().optional().describe("Port form factor: SFP+, SFP28, QSFP+, QSFP28, QSFP-DD, OSFP. Auto-detected from speed if omitted."),
environment: z.enum(["rack", "row", "room"]).default("rack").describe("rack = <3m, row = 3-10m, room = 10-100m"),
budget_preference: z.enum(["cheapest", "balanced", "best_quality"]).default("balanced"),
},
async ({ switch_model, port_speed_gbps, distance_meters, form_factor, environment, budget_preference }) => {
// Auto-detect form factor from speed if not provided
const ffMap: Record<number, string[]> = {
10: ["SFP+"],
25: ["SFP28"],
40: ["QSFP+"],
100: ["QSFP28", "QSFP-DD"],
200: ["QSFP56", "QSFP-DD"],
400: ["QSFP-DD", "OSFP"],
800: ["QSFP-DD800", "OSFP"],
};
const targetFormFactors = form_factor ? [form_factor] : (ffMap[port_speed_gbps] || ["QSFP28"]);
// Determine cable type recommendation
const cableType = distance_meters <= 5 ? "DAC (recommended)" :
distance_meters <= 7 ? "DAC or AOC" :
distance_meters <= 100 ? "AOC (required)" :
"Optical transceiver + fiber required";
const isDACViable = distance_meters <= 7;
const isAOCViable = distance_meters >= 1 && distance_meters <= 100;
// Search for matching cables
const cableCategories = [];
if (isDACViable) cableCategories.push("DAC");
if (isAOCViable) cableCategories.push("AOC");
const ffPlaceholders = targetFormFactors.map((_, i) => `$${i + 2}`).join(",");
const catPlaceholders = cableCategories.map((_, i) => `$${i + 2 + targetFormFactors.length}`).join(",");
const values: unknown[] = [
port_speed_gbps,
...targetFormFactors,
...cableCategories,
];
const result = await pool.query(
`SELECT t.id, t.slug, t.standard_name, t.form_factor, t.speed, t.speed_gbps,
t.reach_label, t.reach_meters, t.category, t.connector, t.power_consumption_w,
v.name as vendor, v.type as vendor_type,
(SELECT MIN(po.price) FROM price_observations po
WHERE po.transceiver_id = t.id AND po.time > NOW() - INTERVAL '7 days'
) as min_price,
(SELECT MAX(po.price) FROM price_observations po
WHERE po.transceiver_id = t.id AND po.time > NOW() - INTERVAL '7 days'
) as max_price,
(SELECT COUNT(DISTINCT po.source_vendor_id) FROM price_observations po
WHERE po.transceiver_id = t.id AND po.stock_level = 'in_stock'
AND po.time > NOW() - INTERVAL '7 days'
) as in_stock_vendors
FROM transceivers t
LEFT JOIN vendors v ON v.id = t.vendor_id
WHERE t.speed_gbps = $1
AND t.form_factor IN (${ffPlaceholders})
AND t.category IN (${catPlaceholders})
AND t.reach_meters >= ${distance_meters}
ORDER BY
CASE '${budget_preference}'
WHEN 'cheapest' THEN (SELECT MIN(po.price) FROM price_observations po WHERE po.transceiver_id = t.id AND po.time > NOW() - INTERVAL '7 days')
WHEN 'best_quality' THEN -t.reach_meters
ELSE t.reach_meters
END ASC NULLS LAST
LIMIT 15`,
values
);
// Check switch compatibility if model provided
let switchCompat: unknown[] = [];
if (switch_model && result.rows.length > 0) {
const txIds = result.rows.map(r => r.id);
const switchRes = await pool.query(
`SELECT c.transceiver_id, c.status, c.firmware_min, sw.model
FROM compatibility c
JOIN switches sw ON sw.id = c.switch_id
WHERE sw.model ILIKE $1
AND c.transceiver_id = ANY($2)`,
[`%${switch_model}%`, txIds]
);
switchCompat = switchRes.rows;
}
const compatMap = new Map((switchCompat as any[]).map(c => [c.transceiver_id, c]));
const recommendations = result.rows.map(r => ({
slug: r.slug,
standard: r.standard_name,
form_factor: r.form_factor,
speed: r.speed,
category: r.category,
reach_meters: r.reach_meters,
reach_label: r.reach_label,
connector: r.connector,
power_w: r.power_consumption_w,
vendor: r.vendor,
pricing: {
min_price: r.min_price ? parseFloat(r.min_price) : null,
max_price: r.max_price ? parseFloat(r.max_price) : null,
in_stock_vendors: parseInt(r.in_stock_vendors) || 0,
},
switch_compatible: switch_model
? (compatMap.has(r.id) ? (compatMap.get(r.id) as any).status : "not_verified")
: undefined,
firmware_min: switch_model && compatMap.has(r.id)
? (compatMap.get(r.id) as any).firmware_min
: undefined,
}));
return {
content: [{
type: "text",
text: JSON.stringify({
request: {
speed_gbps: port_speed_gbps,
distance_meters,
form_factors: targetFormFactors,
environment,
budget: budget_preference,
switch_model: switch_model || null,
},
recommendation: {
cable_type: cableType,
dac_viable: isDACViable,
aoc_viable: isAOCViable,
rationale: isDACViable && distance_meters <= 3
? "DAC is the cheapest and lowest-latency option for short rack-level runs. No optics or fiber needed."
: isDACViable
? "DAC works at this distance but is near its limit. AOC provides more headroom."
: isAOCViable
? "AOC is required at this distance. DAC copper cannot reliably reach beyond 7m."
: "Neither DAC nor AOC can cover this distance. Use optical transceivers (SR/LR) with fiber patch cables.",
},
cables: recommendations,
count: recommendations.length,
tips: [
isDACViable ? "DAC cables are passive and draw zero power from the switch. Ideal for ToR deployments." : null,
isAOCViable ? "AOC cables are lighter and more flexible than DAC. Better for cable management in dense environments." : null,
distance_meters > 100 ? "For distances > 100m, use SR transceivers with OM3/OM4 MMF fiber." : null,
distance_meters > 300 ? "For distances > 300m, use LR transceivers with OS2 SMF fiber." : null,
].filter(Boolean),
}, null, 2),
}],
};
}
);
// --- Tool: get_market_overview ---
server.tool(
"get_market_overview",
"Get market overview statistics: vendor counts, price ranges, technology distribution by form factor. Useful for market analysis, sales strategy, and inventory planning.",
{
form_factor: z.string().optional().describe("Filter by form factor: SFP+, SFP28, QSFP28, QSFP-DD, OSFP, etc."),
speed_gbps: z.number().optional().describe("Filter by speed in Gbps"),
include_vendor_breakdown: z.boolean().default(true).describe("Include per-vendor statistics"),
include_price_tiers: z.boolean().default(true).describe("Include price tier analysis (Budget/Standard/Premium)"),
},
async ({ form_factor, speed_gbps, include_vendor_breakdown, include_price_tiers }) => {
const conditions: string[] = [];
const values: unknown[] = [];
let idx = 1;
if (form_factor) {
conditions.push(`t.form_factor ILIKE $${idx}`);
values.push(`%${form_factor}%`);
idx++;
}
if (speed_gbps) {
conditions.push(`t.speed_gbps = $${idx}`);
values.push(speed_gbps);
idx++;
}
const where = conditions.length > 0 ? `WHERE ${conditions.join(" AND ")}` : "";
// Overall stats
const overallResult = await pool.query(
`SELECT
COUNT(*) as total_transceivers,
COUNT(DISTINCT t.form_factor) as form_factor_count,
COUNT(DISTINCT t.vendor_id) as vendor_count,
COUNT(DISTINCT t.speed_gbps) as speed_variants,
MIN(t.speed_gbps) as min_speed_gbps,
MAX(t.speed_gbps) as max_speed_gbps,
COUNT(*) FILTER (WHERE t.market_status = 'Mainstream') as mainstream_count,
COUNT(*) FILTER (WHERE t.market_status = 'Growth') as growth_count,
COUNT(*) FILTER (WHERE t.market_status = 'Emerging') as emerging_count,
COUNT(*) FILTER (WHERE t.market_status = 'Legacy') as legacy_count,
COUNT(*) FILTER (WHERE t.market_status = 'EOL') as eol_count
FROM transceivers t
${where}`,
values
);
// Form factor distribution
const ffResult = await pool.query(
`SELECT t.form_factor,
COUNT(*) as count,
ARRAY_AGG(DISTINCT t.speed_gbps ORDER BY t.speed_gbps) as speeds,
COUNT(DISTINCT t.vendor_id) as vendors
FROM transceivers t
${where}
GROUP BY t.form_factor
ORDER BY COUNT(*) DESC`,
values
);
// Price ranges by form factor
let priceRanges = null;
if (include_price_tiers) {
const priceResult = await pool.query(
`SELECT t.form_factor, t.speed_gbps,
MIN(po.price)::numeric(10,2) as min_price,
AVG(po.price)::numeric(10,2) as avg_price,
MAX(po.price)::numeric(10,2) as max_price,
PERCENTILE_CONT(0.25) WITHIN GROUP (ORDER BY po.price)::numeric(10,2) as p25,
PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY po.price)::numeric(10,2) as median,
PERCENTILE_CONT(0.75) WITHIN GROUP (ORDER BY po.price)::numeric(10,2) as p75,
COUNT(DISTINCT po.source_vendor_id) as vendor_count,
COUNT(*) as observation_count
FROM price_observations po
JOIN transceivers t ON t.id = po.transceiver_id
WHERE po.time > NOW() - INTERVAL '30 days'
${form_factor ? `AND t.form_factor ILIKE $${idx}` : ''}
${speed_gbps ? `AND t.speed_gbps = $${idx + (form_factor ? 1 : 0)}` : ''}
GROUP BY t.form_factor, t.speed_gbps
ORDER BY t.speed_gbps DESC, t.form_factor`,
form_factor ? (speed_gbps ? [`%${form_factor}%`, speed_gbps] : [`%${form_factor}%`])
: (speed_gbps ? [speed_gbps] : [])
);
priceRanges = priceResult.rows;
}
// Vendor breakdown
let vendorBreakdown = null;
if (include_vendor_breakdown) {
const vendorResult = await pool.query(
`SELECT v.name, v.type, v.country, v.is_competitor,
COUNT(DISTINCT t.id) as transceiver_count,
ARRAY_AGG(DISTINCT t.form_factor ORDER BY t.form_factor) as form_factors,
COUNT(DISTINCT t.speed_gbps) as speed_variants
FROM vendors v
JOIN transceivers t ON t.vendor_id = v.id
${where ? where.replace('WHERE', 'WHERE') : ''}
GROUP BY v.id, v.name, v.type, v.country, v.is_competitor
ORDER BY COUNT(DISTINCT t.id) DESC
LIMIT 25`,
values
);
vendorBreakdown = vendorResult.rows;
}
// Technology trends from market_metrics
const trendResult = await pool.query(
`SELECT mm.technology, mm.metric_type,
mm.value, mm.source, mm.time
FROM market_metrics mm
WHERE mm.time > NOW() - INTERVAL '90 days'
ORDER BY mm.time DESC
LIMIT 20`
);
return {
content: [{
type: "text",
text: JSON.stringify({
overview: overallResult.rows[0],
form_factor_distribution: ffResult.rows,
price_ranges: priceRanges,
vendor_breakdown: vendorBreakdown,
recent_market_metrics: trendResult.rows,
generated_at: new Date().toISOString(),
}, null, 2),
}],
};
}
);
// --- Tool: get_technology_roadmap ---
server.tool(
"get_technology_roadmap",
"Get technology lifecycle roadmap with hype cycle positioning, Norton-Bass adoption data, and timeline projections. Essential for technology planning and investment decisions.",
{
technology: z.string().optional().describe("Specific technology to analyze, e.g. '800G', 'CPO', '400ZR', 'DWDM'. Omit for full roadmap."),
category: z.enum(["transceiver", "technology", "all"]).default("all").describe("Filter by category"),
include_market_data: z.boolean().default(true).describe("Include market metrics (shipments, ASP, revenue)"),
include_news: z.boolean().default(true).describe("Include recent news for each technology"),
},
async ({ technology, category, include_market_data, include_news }) => {
// Get lifecycle data
const lcConditions: string[] = [];
const lcValues: unknown[] = [];
let idx = 1;
if (technology) {
lcConditions.push(`tl.technology ILIKE $${idx}`);
lcValues.push(`%${technology}%`);
idx++;
}
if (category !== "all") {
lcConditions.push(`tl.category = $${idx}`);
lcValues.push(category);
idx++;
}
const lcWhere = lcConditions.length > 0 ? `WHERE ${lcConditions.join(" AND ")}` : "";
const lifecycleResult = await pool.query(
`SELECT tl.technology, tl.category, tl.current_phase,
tl.introduction_year, tl.early_adoption_year,
tl.mainstream_year, tl.peak_year, tl.decline_year,
tl.adoption_percent, tl.notes
FROM technology_lifecycle tl
${lcWhere}
ORDER BY
CASE tl.current_phase
WHEN 'trigger' THEN 1
WHEN 'peak_hype' THEN 2
WHEN 'trough' THEN 3
WHEN 'early_adoption' THEN 4
WHEN 'slope_of_enlightenment' THEN 5
WHEN 'early_mainstream' THEN 6
WHEN 'mainstream' THEN 7
WHEN 'plateau' THEN 8
WHEN 'decline' THEN 9
END ASC`,
lcValues
);
// Enrich each technology with market data and news
const technologies = await Promise.all(lifecycleResult.rows.map(async (lc) => {
const enriched: Record<string, unknown> = {
technology: lc.technology,
category: lc.category,
current_phase: lc.current_phase,
adoption_percent: lc.adoption_percent,
timeline: {
introduction: lc.introduction_year,
early_adoption: lc.early_adoption_year,
mainstream: lc.mainstream_year,
peak: lc.peak_year,
decline: lc.decline_year,
},
notes: lc.notes,
};
// Phase description mapping
const phaseDescriptions: Record<string, string> = {
trigger: "Innovation Trigger — Technology just announced or in labs. High risk, potentially transformative.",
peak_hype: "Peak of Inflated Expectations — Massive buzz, early commercial products, inflated claims.",
trough: "Trough of Disillusionment — Initial hype faded, real challenges exposed. Best time to invest.",
early_adoption: "Early Adoption — Proven technology, early mainstream deployments beginning.",
slope_of_enlightenment: "Slope of Enlightenment — Use cases clear, ecosystem growing, prices dropping.",
early_mainstream: "Early Mainstream — Widely deployed, multiple vendors, competitive pricing.",
mainstream: "Mainstream — Dominant technology in its segment. Commoditized.",
plateau: "Plateau of Productivity — Mature, fully commoditized, being displaced by next gen.",
decline: "Decline — Being replaced by newer technology. Consider migration planning.",
};
enriched.phase_description = phaseDescriptions[lc.current_phase] || "Unknown phase";
// Investment recommendation
const investmentMap: Record<string, string> = {
trigger: "WATCH — Too early for production. Monitor for breakthroughs.",
peak_hype: "EVALUATE — Test in lab environments. Do not commit large budgets.",
trough: "INVEST — Best price-to-value window. Early mover advantage.",
early_adoption: "INVEST — Strong time to adopt. Good vendor support, dropping prices.",
slope_of_enlightenment: "BUY — Excellent time to deploy. Proven, prices declining.",
early_mainstream: "BUY — Optimal cost/performance. Multiple vendor options.",
mainstream: "MAINTAIN — Standard choice. Commoditized pricing.",
plateau: "PLAN MIGRATION — Start evaluating next-gen alternatives.",
decline: "MIGRATE — Actively transition to successor technology.",
};
enriched.investment_recommendation = investmentMap[lc.current_phase] || "EVALUATE";
// Market data
if (include_market_data) {
const metricsResult = await pool.query(
`SELECT mm.metric_type, mm.value, mm.source, mm.time, mm.notes
FROM market_metrics mm
WHERE mm.technology ILIKE $1
ORDER BY mm.metric_type, mm.time DESC
LIMIT 20`,
[`%${lc.technology.split(' ')[0]}%`]
);
enriched.market_data = metricsResult.rows;
}
// Recent news
if (include_news) {
const newsResult = await pool.query(
`SELECT na.title, na.source, na.published_at, na.summary, na.relevance_score
FROM news_articles na
WHERE na.search_vector @@ plainto_tsquery('english', $1)
OR na.title ILIKE $2
ORDER BY na.published_at DESC
LIMIT 3`,
[lc.technology, `%${lc.technology.split('(')[0].trim()}%`]
);
enriched.recent_news = newsResult.rows;
}
// Transceiver count in this technology segment
const txCountResult = await pool.query(
`SELECT COUNT(*) as count,
COUNT(*) FILTER (WHERE market_status = 'Mainstream') as mainstream,
COUNT(*) FILTER (WHERE market_status IN ('Growth', 'Emerging')) as growing
FROM transceivers
WHERE standard_name ILIKE $1
OR speed ILIKE $1
OR category ILIKE $1`,
[`%${lc.technology.split(' ')[0]}%`]
);
enriched.transceiver_stats = txCountResult.rows[0];
return enriched;
}));
// Build hype cycle visualization data (position on the curve)
const hypePositions = technologies.map(t => ({
technology: t.technology,
phase: t.current_phase,
adoption_pct: t.adoption_percent,
recommendation: t.investment_recommendation,
}));
return {
content: [{
type: "text",
text: JSON.stringify({
roadmap: technologies,
hype_cycle_summary: hypePositions,
total_technologies: technologies.length,
generated_at: new Date().toISOString(),
note: "Lifecycle data is updated quarterly based on market research, vendor announcements, and pricing trends. Norton-Bass forecasting (Phase 3) will add quantitative adoption predictions.",
}, null, 2),
}],
};
}
);
}

View File

@ -0,0 +1,235 @@
/**
* Pricing tools: get_pricing, compare_prices, get_competitor_stock
*/
import type { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { z } from "zod";
import { pool } from "../db.js";
export async function registerPricingTools(server: McpServer): Promise<void> {
// --- Tool: get_pricing ---
server.tool(
"get_pricing",
"Get current prices and availability across all sources for a specific transceiver. Includes price history if requested.",
{
part_number: z.string().describe("Part number or slug, e.g. 'qsfp28-lr4' or 'SFP-10G-LR'"),
vendor: z.string().optional().describe("Filter to specific vendor only"),
include_history: z.boolean().default(false).describe("Include 30-day price history"),
},
async ({ part_number, vendor, include_history }) => {
// Find transceiver
const txResult = await pool.query(
`SELECT t.id, t.slug, t.standard_name, t.form_factor, t.speed, t.reach_label
FROM transceivers t
WHERE t.slug ILIKE $1 OR t.part_number ILIKE $1 OR t.standard_name ILIKE $1
LIMIT 1`,
[`%${part_number}%`]
);
if (txResult.rows.length === 0) {
return {
content: [{ type: "text", text: `No transceiver found for "${part_number}".` }],
};
}
const tx = txResult.rows[0];
const vendorCondition = vendor ? `AND v.name ILIKE $2` : "";
const baseParams: unknown[] = vendor ? [tx.id, `%${vendor}%`] : [tx.id];
// Current prices (latest per vendor)
const currentPrices = await pool.query(
`SELECT DISTINCT ON (po.source_vendor_id)
v.name as vendor, po.price, po.currency, po.stock_level,
po.quantity_available, po.url, po.time
FROM price_observations po
JOIN vendors v ON v.id = po.source_vendor_id
WHERE po.transceiver_id = $1 ${vendorCondition}
ORDER BY po.source_vendor_id, po.time DESC`,
baseParams
);
let history = null;
if (include_history) {
const histResult = await pool.query(
`SELECT v.name as vendor, po.price, po.currency, po.stock_level, po.time
FROM price_observations po
JOIN vendors v ON v.id = po.source_vendor_id
WHERE po.transceiver_id = $1 ${vendorCondition}
AND po.time > NOW() - INTERVAL '30 days'
ORDER BY po.time DESC
LIMIT 200`,
baseParams
);
history = histResult.rows;
}
const response = {
transceiver: {
slug: tx.slug,
standard: tx.standard_name,
form_factor: tx.form_factor,
speed: tx.speed,
reach: tx.reach_label,
},
current_prices: currentPrices.rows,
cheapest: currentPrices.rows.length > 0
? currentPrices.rows.reduce((min, r) => r.price < min.price ? r : min)
: null,
history: include_history ? history : undefined,
};
return {
content: [{ type: "text", text: JSON.stringify(response, null, 2) }],
};
}
);
// --- Tool: compare_prices ---
server.tool(
"compare_prices",
"Compare prices across all vendors for a transceiver. Shows savings opportunities vs competitors.",
{
transceiver_query: z.string().describe("Free text or part number to search for"),
competitors: z.array(z.string()).optional().describe("Limit to specific competitors"),
},
async ({ transceiver_query, competitors }) => {
const vendorFilter = competitors && competitors.length > 0
? `AND v.name ILIKE ANY(ARRAY[${competitors.map((_, i) => `$${i + 2}`).join(",")}])`
: "";
const values: unknown[] = [transceiver_query];
if (competitors) {
competitors.forEach((c) => values.push(`%${c}%`));
}
const result = await pool.query(
`SELECT t.slug, t.standard_name, t.form_factor, t.speed, t.reach_label,
v.name as vendor, v.type,
po.price, po.currency, po.stock_level, po.time
FROM price_observations po
JOIN transceivers t ON t.id = po.transceiver_id
JOIN vendors v ON v.id = po.source_vendor_id
WHERE (t.standard_name ILIKE $1 OR t.slug ILIKE $1)
${vendorFilter}
AND po.time = (
SELECT MAX(time) FROM price_observations
WHERE transceiver_id = t.id AND source_vendor_id = v.id
)
ORDER BY t.slug, po.price ASC`,
values
);
if (result.rows.length === 0) {
return {
content: [{ type: "text", text: `No pricing data found for "${transceiver_query}".` }],
};
}
// Group by transceiver
const grouped: Record<string, {
transceiver: { slug: string; standard: string; form_factor: string; speed: string; reach: string };
prices: Array<{ vendor: string; type: string; price: number; currency: string; stock: string }>;
min_price: number;
max_price: number;
savings_vs_max: number;
}> = {};
for (const row of result.rows) {
if (!grouped[row.slug]) {
grouped[row.slug] = {
transceiver: {
slug: row.slug,
standard: row.standard_name,
form_factor: row.form_factor,
speed: row.speed,
reach: row.reach_label,
},
prices: [],
min_price: Infinity,
max_price: 0,
savings_vs_max: 0,
};
}
grouped[row.slug].prices.push({
vendor: row.vendor,
type: row.type,
price: parseFloat(row.price),
currency: row.currency,
stock: row.stock_level,
});
grouped[row.slug].min_price = Math.min(grouped[row.slug].min_price, parseFloat(row.price));
grouped[row.slug].max_price = Math.max(grouped[row.slug].max_price, parseFloat(row.price));
}
for (const slug of Object.keys(grouped)) {
const g = grouped[slug];
g.savings_vs_max = Math.round((1 - g.min_price / g.max_price) * 100);
}
return {
content: [{
type: "text",
text: JSON.stringify({ comparisons: Object.values(grouped) }, null, 2),
}],
};
}
);
// --- Tool: get_competitor_stock ---
server.tool(
"get_competitor_stock",
"Check live stock levels at a specific competitor. Useful for identifying sales opportunities when competitor is out of stock.",
{
competitor: z.string().describe("Competitor name, e.g. 'FS.COM', 'Optcore', 'ProLabs'"),
product_query: z.string().optional().describe("Optional product filter"),
out_of_stock_only: z.boolean().default(false).describe("Only show out-of-stock items (sales opportunities)"),
},
async ({ competitor, product_query, out_of_stock_only }) => {
const conditions = [`v.name ILIKE $1`];
const values: unknown[] = [`%${competitor}%`];
let idx = 2;
if (product_query) {
conditions.push(`(t.standard_name ILIKE $${idx} OR t.slug ILIKE $${idx})`);
values.push(`%${product_query}%`);
idx++;
}
if (out_of_stock_only) {
conditions.push(`po.stock_level IN ('out_of_stock', 'discontinued')`);
}
const result = await pool.query(
`SELECT DISTINCT ON (t.id)
v.name as competitor, t.slug, t.standard_name, t.form_factor, t.speed,
t.reach_label, po.price, po.currency, po.stock_level,
po.quantity_available, po.time
FROM price_observations po
JOIN vendors v ON v.id = po.source_vendor_id
JOIN transceivers t ON t.id = po.transceiver_id
WHERE ${conditions.join(" AND ")}
ORDER BY t.id, po.time DESC
LIMIT 100`,
values
);
const inStock = result.rows.filter((r) => r.stock_level === "in_stock").length;
const outOfStock = result.rows.filter((r) =>
["out_of_stock", "discontinued"].includes(r.stock_level)
).length;
return {
content: [{
type: "text",
text: JSON.stringify({
competitor,
summary: { total: result.rows.length, in_stock: inStock, out_of_stock: outOfStock },
items: result.rows,
sales_opportunities: out_of_stock_only
? result.rows.length
: `${outOfStock} items out of stock at ${competitor}`,
}, null, 2),
}],
};
}
);
}

View File

@ -0,0 +1,166 @@
/**
* Switch documentation tools: get_switch_docs, get_switch_image
*/
import type { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { z } from "zod";
import { pool } from "../db.js";
export async function registerSwitchDocTools(server: McpServer): Promise<void> {
// --- Tool: get_switch_docs ---
server.tool(
"get_switch_docs",
"Get datasheets, manuals, and documentation for a switch/router model. Returns links to PDFs, configuration guides, quick start guides, and CLI references.",
{
model: z.string().describe("Switch/router model name, e.g. 'N9K-C93600CD-GX' or 'CRS504'"),
doc_type: z.enum(["all", "datasheet", "manual", "quick_start", "cli_reference", "installation_guide"]).default("all").describe("Filter by document type"),
},
async ({ model, doc_type }) => {
// Find the switch
const switchResult = await pool.query(
`SELECT sw.id, sw.model, sw.series, sw.image_url, sw.datasheet_url,
sw.product_page_url, sw.manual_urls,
v.name as vendor_name, v.website as vendor_website
FROM switches sw
LEFT JOIN vendors v ON sw.vendor_id = v.id
WHERE sw.model ILIKE $1
LIMIT 5`,
[`%${model}%`]
);
if (switchResult.rows.length === 0) {
return {
content: [{ type: "text", text: `No switch found matching "${model}". Try a more specific model name.` }],
};
}
const results: string[] = [];
for (const sw of switchResult.rows) {
// Get associated documents
const docQuery = doc_type !== "all"
? `SELECT pd.doc_type, pd.title, pd.source_url, pd.file_size_bytes, pd.page_count
FROM product_documents pd
WHERE pd.switch_id = $1 AND pd.doc_type = $2
ORDER BY pd.doc_type, pd.title`
: `SELECT pd.doc_type, pd.title, pd.source_url, pd.file_size_bytes, pd.page_count
FROM product_documents pd
WHERE pd.switch_id = $1
ORDER BY pd.doc_type, pd.title`;
const docParams = doc_type !== "all" ? [sw.id, doc_type] : [sw.id];
const docsResult = await pool.query(docQuery, docParams);
let text = `## ${sw.vendor_name} ${sw.model} (${sw.series})\n\n`;
if (sw.product_page_url) {
text += `**Product Page:** ${sw.product_page_url}\n`;
}
if (sw.image_url) {
text += `**Product Image:** ${sw.image_url}\n`;
}
if (sw.datasheet_url) {
text += `**Datasheet:** ${sw.datasheet_url}\n`;
}
if (sw.vendor_website) {
text += `**Vendor Website:** ${sw.vendor_website}\n`;
}
if (docsResult.rows.length > 0) {
text += `\n### Documents (${docsResult.rows.length})\n\n`;
for (const doc of docsResult.rows) {
const size = doc.file_size_bytes ? ` (${(doc.file_size_bytes / 1024 / 1024).toFixed(1)} MB)` : "";
const pages = doc.page_count ? `, ${doc.page_count} pages` : "";
text += `- **[${doc.doc_type}]** ${doc.title}${size}${pages}\n ${doc.source_url}\n`;
}
} else {
text += "\nNo downloaded documents yet. Run `tsx src/index.ts --switch-assets` to fetch them.\n";
}
results.push(text);
}
return {
content: [{ type: "text", text: results.join("\n---\n\n") }],
};
}
);
// --- Tool: search_switches ---
server.tool(
"search_switches",
"Search switches and routers by specs, vendor, or category. Returns matching devices with their transceiver port configuration.",
{
query: z.string().optional().describe("Free text query, e.g. 'Cisco 400G spine' or 'industrial Hirschmann'"),
vendor: z.string().optional().describe("Vendor name filter"),
category: z.enum(["DataCenter", "Campus", "Edge", "Core", "SP", "Industrial"]).optional(),
min_speed_gbps: z.number().optional().describe("Minimum port speed in Gbps"),
max_results: z.number().default(10),
},
async ({ query, vendor, category, min_speed_gbps, max_results }) => {
const conditions: string[] = [];
const values: unknown[] = [];
let idx = 1;
if (query) {
conditions.push(`sw.search_vector @@ plainto_tsquery('english', $${idx})`);
values.push(query);
idx++;
}
if (vendor) {
conditions.push(`v.name ILIKE $${idx}`);
values.push(`%${vendor}%`);
idx++;
}
if (category) {
conditions.push(`sw.category = $${idx}`);
values.push(category);
idx++;
}
if (min_speed_gbps) {
conditions.push(`sw.max_speed_gbps >= $${idx}`);
values.push(min_speed_gbps);
idx++;
}
const where = conditions.length > 0 ? `WHERE ${conditions.join(" AND ")}` : "";
const orderBy = query
? `ORDER BY ts_rank(sw.search_vector, plainto_tsquery('english', $1)) DESC`
: `ORDER BY sw.max_speed_gbps DESC NULLS LAST`;
const result = await pool.query(
`SELECT sw.id, sw.model, sw.series, sw.category, sw.layer,
sw.ports_config, sw.total_ports, sw.max_speed_gbps,
sw.switching_capacity_tbps, sw.asic_vendor, sw.asic_model,
sw.image_url, sw.datasheet_url, sw.product_page_url,
v.name as vendor_name
FROM switches sw
LEFT JOIN vendors v ON sw.vendor_id = v.id
${where}
${orderBy}
LIMIT ${max_results}`,
values
);
if (result.rows.length === 0) {
return {
content: [{ type: "text", text: "No switches found matching your criteria." }],
};
}
const lines = result.rows.map((sw) => {
const ports = typeof sw.ports_config === "string" ? JSON.parse(sw.ports_config) : sw.ports_config;
const portStr = Object.entries(ports || {}).map(([k, v]) => `${v}x ${k.replace(/_/g, " ")}`).join(", ");
let text = `**${sw.vendor_name} ${sw.model}** (${sw.series}) — ${sw.category} ${sw.layer}\n`;
text += ` Ports: ${portStr || "N/A"} | Max: ${sw.max_speed_gbps}G`;
if (sw.switching_capacity_tbps) text += ` | Capacity: ${sw.switching_capacity_tbps}Tbps`;
if (sw.asic_vendor) text += ` | ASIC: ${sw.asic_vendor} ${sw.asic_model || ""}`;
if (sw.image_url) text += `\n Image: ${sw.image_url}`;
if (sw.datasheet_url) text += `\n Datasheet: ${sw.datasheet_url}`;
return text;
});
return {
content: [{ type: "text", text: `Found ${result.rows.length} switches:\n\n${lines.join("\n\n")}` }],
};
}
);
}

View File

@ -0,0 +1,19 @@
{
"compilerOptions": {
"target": "ES2022",
"module": "commonjs",
"lib": ["ES2022"],
"outDir": "./dist",
"rootDir": "./src",
"strict": true,
"esModuleInterop": true,
"skipLibCheck": true,
"resolveJsonModule": true,
"declaration": true,
"declarationMap": true,
"sourceMap": true,
"isolatedModules": true
},
"include": ["src/**/*"],
"exclude": ["node_modules", "dist"]
}

View File

@ -0,0 +1,31 @@
{
"name": "@tip/scraper",
"version": "0.1.0",
"private": true,
"description": "TIP scraper engine — Crawlee + Playwright for competitor pricing, stock, datasheets, FAQs",
"main": "dist/index.js",
"scripts": {
"build": "tsc",
"dev": "tsx src/index.ts",
"scrape:fs": "tsx src/scrapers/fs-com.ts",
"scrape:cisco": "tsx src/scrapers/cisco-tmg.ts",
"scrape:optcore": "tsx src/scrapers/optcore.ts",
"scrape:news": "tsx src/scrapers/news.ts",
"scrape:all": "tsx src/index.ts --all"
},
"dependencies": {
"crawlee": "^3.12.0",
"playwright": "^1.50.0",
"pg": "^8.13.1",
"pg-boss": "^10.1.5",
"dotenv": "^16.4.7",
"cheerio": "^1.0.0",
"xml2js": "^0.6.2"
},
"devDependencies": {
"@types/pg": "^8.11.11",
"@types/xml2js": "^0.4.14",
"typescript": "^5.9.3",
"tsx": "^4.19.0"
}
}

View File

@ -0,0 +1,187 @@
/**
* TIP Scraper Engine Main entry point.
*
* Usage:
* tsx src/index.ts Start scheduler (production mode)
* tsx src/index.ts --all Run all scrapers once
* tsx src/index.ts --fs Run FS.com scraper once
* tsx src/index.ts --cisco Run Cisco TMG scraper once
* tsx src/index.ts --optcore Run Optcore scraper once
* tsx src/index.ts --news Run news aggregator once
* tsx src/index.ts --flexoptix Run Flexoptix catalog scraper once
* tsx src/index.ts --vendors Run Flexoptix vendor list scraper once
* tsx src/index.ts --10gtek Run 10Gtek scraper once
* tsx src/index.ts --champion Run Champion ONE scraper once
* tsx src/index.ts --fluxlight Run Fluxlight scraper once
* tsx src/index.ts --gbics Run GBICS.com scraper once
* tsx src/index.ts --prolabs Run ProLabs scraper once
* tsx src/index.ts --juniper Run Juniper HCT scraper once
* tsx src/index.ts --switches Seed switch/router database
* tsx src/index.ts --whitebox Seed whitebox switch database (Edgecore, Celestica, etc.)
* tsx src/index.ts --switches-ext Seed extended switches (Fortinet, MikroTik, Industrial, etc.)
* tsx src/index.ts --sonic-hcl Scrape SONiC Hardware Compatibility List
* tsx src/index.ts --edgecore Scrape Edgecore product catalog
* tsx src/index.ts --ufispace Scrape UfiSpace product catalog
* tsx src/index.ts --switch-assets Scrape switch assets via URL patterns
* tsx src/index.ts --switch-crawl Crawl switch assets (Cheerio, static HTML vendors)
* tsx src/index.ts --switch-crawl-pw Crawl switch assets (Playwright, JS-heavy vendors)
* tsx src/index.ts --fetch-only Run only fetch-based scrapers (no Playwright)
* tsx src/index.ts --atgbics Run ATGBICS scraper once
*/
import { createScheduler, registerSchedules, registerWorkers } from "./scheduler";
import { scrapeFs } from "./scrapers/fs-com";
import { scrapeCiscoTmg } from "./scrapers/cisco-tmg";
import { scrapeOptcore } from "./scrapers/optcore";
import { scrapeNews } from "./scrapers/news";
import { scrapeFlexoptixCatalog } from "./scrapers/flexoptix-catalog";
import { scrapeFlexoptixVendors } from "./scrapers/flexoptix-vendors";
import { scrape10Gtek } from "./scrapers/tenGtek";
import { scrapeChampionOne } from "./scrapers/champion-one";
import { scrapeFluxlight } from "./scrapers/fluxlight";
import { scrapeSfpCables } from "./scrapers/sfpcables";
import { scrapeGbics } from "./scrapers/gbics";
import { scrapeJuniperHct } from "./scrapers/juniper-hct";
import { seedSwitches } from "./scrapers/switch-seed";
import { seedWhiteboxSwitches } from "./scrapers/whitebox-seed";
import { seedFlexoptixVendors } from "./scrapers/flexoptix-supported-vendors";
import { scrapeSonicHcl } from "./scrapers/sonic-hcl";
import { scrapeEdgecore } from "./scrapers/edgecore";
import { scrapeUfiSpace } from "./scrapers/ufispace";
import { seedExtendedSwitches } from "./scrapers/switch-seed-extended";
import { seedBulkSwitches } from "./scrapers/switch-seed-bulk";
import { scrapeSwitchAssets } from "./scrapers/switch-assets";
import { crawlSwitchAssets } from "./scrapers/switch-assets-crawler";
import { crawlSwitchAssetsPlaywright } from "./scrapers/switch-assets-playwright";
import { scrapeAtgbics } from "./scrapers/atgbics";
import { scrapeProLabs } from "./scrapers/prolabs";
import { pool } from "./utils/db";
const args = process.argv.slice(2);
const isAll = args.includes("--all");
const isFetchOnly = args.includes("--fetch-only");
async function runOnce(): Promise<void> {
// Fetch-based scrapers (no Playwright/Chromium needed — fast, reliable)
if (args.includes("--flexoptix") || isAll || isFetchOnly) {
await scrapeFlexoptixCatalog();
}
if (args.includes("--vendors") || isAll || isFetchOnly) {
await scrapeFlexoptixVendors();
}
if (args.includes("--10gtek") || isAll || isFetchOnly) {
await scrape10Gtek();
}
if (args.includes("--champion") || isAll || isFetchOnly) {
await scrapeChampionOne();
}
if (args.includes("--fluxlight") || isAll || isFetchOnly) {
await scrapeFluxlight();
}
if (args.includes("--sfpcables") || isAll || isFetchOnly) {
await scrapeSfpCables();
}
if (args.includes("--gbics") || isAll || isFetchOnly) {
await scrapeGbics();
}
if (args.includes("--prolabs") || isAll || isFetchOnly) {
await scrapeProLabs();
}
if (args.includes("--juniper") || isAll || isFetchOnly) {
await scrapeJuniperHct();
}
if (args.includes("--switches") || isAll || isFetchOnly) {
await seedSwitches();
}
if (args.includes("--whitebox") || isAll || isFetchOnly) {
await seedWhiteboxSwitches();
}
if (args.includes("--flexoptix-vendors") || isAll || isFetchOnly) {
await seedFlexoptixVendors();
}
if (args.includes("--switches-ext") || isAll || isFetchOnly) {
await seedExtendedSwitches();
}
if (args.includes("--switches-bulk") || isAll || isFetchOnly) {
await seedBulkSwitches();
}
if (args.includes("--sonic-hcl") || isAll || isFetchOnly) {
await scrapeSonicHcl();
}
if (args.includes("--news") || isAll || isFetchOnly) {
await scrapeNews();
}
if (args.includes("--switch-assets") || isAll) {
const vendor = args.find((a) => a.startsWith("--vendor="))?.split("=")[1];
await scrapeSwitchAssets(vendor);
}
if (args.includes("--switch-crawl") || isAll) {
const vendor = args.find((a) => a.startsWith("--vendor="))?.split("=")[1];
await crawlSwitchAssets(vendor);
}
// Crawlee-based scrapers (Cheerio, no Playwright needed)
if (args.includes("--edgecore") || isAll) {
await scrapeEdgecore();
}
if (args.includes("--ufispace") || isAll) {
await scrapeUfiSpace();
}
// Playwright-based scrapers (need Chromium installed)
if (!isFetchOnly) {
if (args.includes("--fs") || isAll) {
await scrapeFs();
}
if (args.includes("--cisco") || isAll) {
await scrapeCiscoTmg();
}
if (args.includes("--optcore") || isAll) {
await scrapeOptcore();
}
if (args.includes("--switch-crawl-pw") || isAll) {
const vendor = args.find((a) => a.startsWith("--vendor="))?.split("=")[1];
await crawlSwitchAssetsPlaywright(vendor);
}
if (args.includes("--atgbics") || isAll) {
await scrapeAtgbics();
}
}
await pool.end();
}
async function runScheduler(): Promise<void> {
console.log("=== TIP Scraper Engine ===\n");
console.log("Mode: Scheduler (pg-boss)\n");
const boss = await createScheduler();
await registerSchedules(boss);
await registerWorkers(boss);
console.log("\nScheduler running. Press Ctrl+C to stop.\n");
// Graceful shutdown
const shutdown = async () => {
console.log("\nShutting down...");
await boss.stop();
await pool.end();
process.exit(0);
};
process.on("SIGINT", shutdown);
process.on("SIGTERM", shutdown);
}
const ALL_FLAGS = ["--all", "--fs", "--cisco", "--optcore", "--news", "--flexoptix", "--vendors", "--10gtek", "--champion", "--fluxlight", "--sfpcables", "--gbics", "--prolabs", "--juniper", "--switches", "--whitebox", "--switches-ext", "--flexoptix-vendors", "--sonic-hcl", "--edgecore", "--ufispace", "--switch-assets", "--switch-crawl", "--switch-crawl-pw", "--fetch-only", "--atgbics"];
if (args.some((a) => ALL_FLAGS.includes(a))) {
runOnce().catch((err) => {
console.error("Fatal:", err);
process.exit(1);
});
} else {
runScheduler().catch((err) => {
console.error("Fatal:", err);
process.exit(1);
});
}

View File

@ -0,0 +1,212 @@
/**
* pg-boss Job Scheduler manages scrape jobs with adaptive timing.
*
* Job types:
* scrape:pricing:fs Every 4 hours for FS.com prices/stock
* scrape:pricing:optcore Every 6 hours for Optcore prices/stock
* scrape:pricing:atgbics Every 8 hours for ATGBICS prices/stock (GBP)
* scrape:pricing:prolabs Every 8 hours for ProLabs prices/stock (USD)
* scrape:compat:cisco Weekly for OEM compatibility matrices
* scrape:news Every 6 hours for trade press and news
* scrape:docs Weekly for manuals and datasheets
* scrape:faq Weekly for vendor FAQ/troubleshooting pages
*/
import PgBoss from "pg-boss";
import { config } from "dotenv";
import { join } from "path";
import { rmSync, mkdirSync } from "fs";
/** Run a scraper with an isolated Crawlee storage directory to prevent queue collisions */
async function withIsolatedStorage(name: string, fn: () => Promise<void>): Promise<void> {
const dir = join(__dirname, "..", "..", "..", `storage-${name}`);
mkdirSync(dir, { recursive: true });
const prev = process.env.CRAWLEE_STORAGE_DIR;
process.env.CRAWLEE_STORAGE_DIR = dir;
try {
await fn();
} finally {
process.env.CRAWLEE_STORAGE_DIR = prev ?? "";
// Clean up after successful run
try { rmSync(dir, { recursive: true, force: true }); } catch { /* ignore */ }
}
}
config({ path: join(__dirname, "..", "..", "..", ".env") });
const connectionString = `postgres://${process.env.POSTGRES_USER || "tip"}:${process.env.POSTGRES_PASSWORD || "tip_dev_2026"}@${process.env.POSTGRES_HOST || "localhost"}:${process.env.POSTGRES_PORT || "5433"}/${process.env.POSTGRES_DB || "transceiver_db"}`;
export async function createScheduler(): Promise<PgBoss> {
const boss = new PgBoss({
connectionString,
retryLimit: 3,
retryDelay: 30,
retryBackoff: true,
expireInSeconds: 300, // 5 min timeout per job
monitorStateIntervalSeconds: 30,
});
boss.on("error", (error) => console.error("pg-boss error:", error));
await boss.start();
console.log("pg-boss scheduler started");
return boss;
}
export async function registerSchedules(boss: PgBoss): Promise<void> {
// pg-boss v10: create queues before scheduling
const queues = [
"scrape:pricing:fs",
"scrape:pricing:optcore",
"scrape:pricing:10gtek",
"scrape:pricing:atgbics",
"scrape:pricing:prolabs",
"scrape:compat:cisco",
"scrape:pricing:flexoptix",
"scrape:vendors:flexoptix",
"scrape:news",
"scrape:faq",
"scrape:docs",
];
for (const q of queues) {
await boss.createQueue(q).catch(() => { /* already exists */ });
}
// v0.2.0: Increased frequencies for permanent price monitoring (R-SCAN)
// FS.com pricing (every 4 hours — JS rendering is slow)
await boss.schedule("scrape:pricing:fs", "0 */4 * * *", {}, {
retryLimit: 2,
expireInSeconds: 3600,
});
// Optcore pricing (every 4 hours — was 6h)
await boss.schedule("scrape:pricing:optcore", "0 2/4 * * *", {}, {
retryLimit: 2,
expireInSeconds: 7200,
});
// Compatibility matrices (every Sunday at 3am)
await boss.schedule("scrape:compat:cisco", "0 3 * * 0", {}, {
retryLimit: 3,
expireInSeconds: 3600,
});
// News aggregation (every 6 hours)
await boss.schedule("scrape:news", "0 */6 * * *", {}, {
retryLimit: 2,
expireInSeconds: 1800,
});
// FAQ/KB scraping (every Wednesday at 2am)
await boss.schedule("scrape:faq", "0 2 * * 3", {}, {
retryLimit: 3,
expireInSeconds: 3600,
});
// 10Gtek pricing (every 8 hours — Playwright, reasonable rate)
await boss.schedule("scrape:pricing:10gtek", "0 */8 * * *", {}, {
retryLimit: 2,
expireInSeconds: 3600,
});
// ATGBICS pricing (every 8 hours — Shopify/Playwright, GBP prices)
await boss.schedule("scrape:pricing:atgbics", "0 2/8 * * *", {}, {
retryLimit: 2,
expireInSeconds: 3600,
});
// ProLabs pricing (every 8 hours — Playwright, needs proxy for CloudFront)
await boss.schedule("scrape:pricing:prolabs", "0 4/8 * * *", {}, {
retryLimit: 2,
expireInSeconds: 3600,
});
// Flexoptix catalog (every 2 hours — fetch-based, fast — R-SCAN requirement)
await boss.schedule("scrape:pricing:flexoptix", "0 */2 * * *", {}, {
retryLimit: 2,
expireInSeconds: 3600,
});
// Flexoptix vendor list (weekly, Sunday at 6am — own data)
await boss.schedule("scrape:vendors:flexoptix", "0 6 * * 0", {}, {
retryLimit: 3,
expireInSeconds: 600,
});
// Document/datasheet check (every Saturday at 4am)
await boss.schedule("scrape:docs", "0 4 * * 6", {}, {
retryLimit: 3,
expireInSeconds: 7200,
});
console.log("All schedules registered");
}
export async function registerWorkers(boss: PgBoss): Promise<void> {
// Lazy-load scrapers to avoid circular deps
const { scrapeFs } = await import("./scrapers/fs-com");
const { scrapeCiscoTmg } = await import("./scrapers/cisco-tmg");
const { scrapeOptcore } = await import("./scrapers/optcore");
const { scrape10Gtek } = await import("./scrapers/tenGtek");
const { scrapeFlexoptixCatalog } = await import("./scrapers/flexoptix-catalog");
const { scrapeFlexoptixVendors } = await import("./scrapers/flexoptix-vendors");
const { scrapeNews } = await import("./scrapers/news");
const { scrapeAtgbics } = await import("./scrapers/atgbics");
const { scrapeProLabs } = await import("./scrapers/prolabs");
await boss.work("scrape:pricing:fs", async (_job) => {
console.log(`[${new Date().toISOString()}] Running: FS.com pricing`);
await withIsolatedStorage("fs", scrapeFs);
});
await boss.work("scrape:pricing:optcore", async (_job) => {
console.log(`[${new Date().toISOString()}] Running: Optcore pricing`);
await withIsolatedStorage("optcore", scrapeOptcore);
});
await boss.work("scrape:compat:cisco", async (_job) => {
console.log(`[${new Date().toISOString()}] Running: Cisco TMG`);
await withIsolatedStorage("cisco", scrapeCiscoTmg);
});
await boss.work("scrape:pricing:10gtek", async (_job) => {
console.log(`[${new Date().toISOString()}] Running: 10Gtek pricing`);
await withIsolatedStorage("10gtek", scrape10Gtek);
});
await boss.work("scrape:pricing:flexoptix", async (_job) => {
console.log(`[${new Date().toISOString()}] Running: Flexoptix catalog pricing`);
await scrapeFlexoptixCatalog();
});
await boss.work("scrape:vendors:flexoptix", async (_job) => {
console.log(`[${new Date().toISOString()}] Running: Flexoptix vendor list`);
await scrapeFlexoptixVendors();
});
await boss.work("scrape:news", async (_job) => {
console.log(`[${new Date().toISOString()}] Running: News aggregation`);
await scrapeNews();
});
await boss.work("scrape:pricing:atgbics", async (_job) => {
console.log(`[${new Date().toISOString()}] Running: ATGBICS pricing`);
await withIsolatedStorage("atgbics", scrapeAtgbics);
});
await boss.work("scrape:pricing:prolabs", async (_job) => {
console.log(`[${new Date().toISOString()}] Running: ProLabs pricing`);
await withIsolatedStorage("prolabs", scrapeProLabs);
});
await boss.work("scrape:faq", async (_job) => {
console.log(`[${new Date().toISOString()}] FAQ scraper — not yet implemented`);
});
await boss.work("scrape:docs", async (_job) => {
console.log(`[${new Date().toISOString()}] Docs scraper — not yet implemented`);
});
console.log("All workers registered");
}

View File

@ -0,0 +1,369 @@
/**
* ATGBICS Scraper Prices, Stock, Product Catalog
*
* ATGBICS is a UK-based independent compatible optics vendor.
* Site uses Shopify with client-side rendering, so we use PlaywrightCrawler.
* Prices are publicly visible in GBP.
*
* Categories scraped:
* /collections/sfp-transceivers/
* /collections/sfp-plus-transceivers/
* /collections/sfp28-transceivers/
* /collections/qsfp-plus-transceivers/
* /collections/qsfp28-transceivers/
* /collections/qsfp-dd-transceivers/
*
* Respects: robots.txt, rate limiting (2s between requests, max 50 pages)
*/
import { PlaywrightCrawler } from "crawlee";
import { ensureVendor, upsertPriceObservation, findOrCreateScrapedTransceiver, pool } from "../utils/db";
import { contentHash, parsePrice, parseStockLevel, parseQuantity } from "../utils/hash";
const BASE_URL = "https://www.atgbics.com";
const CATEGORY_URLS = [
"/collections/sfp-transceivers/",
"/collections/sfp-plus-transceivers/",
"/collections/sfp28-transceivers/",
"/collections/qsfp-plus-transceivers/",
"/collections/qsfp28-transceivers/",
"/collections/qsfp-dd-transceivers/",
];
const MAX_PAGES = 50;
interface AtgbicsProduct {
partNumber: string;
name: string;
price: number;
currency: string;
stockLevel: string;
quantity?: number;
url: string;
formFactor?: string;
speedGbps?: number;
speed?: string;
reachLabel?: string;
fiberType?: string;
}
function detectFormFactor(text: string): string | undefined {
const lower = text.toLowerCase();
if (lower.includes("qsfp-dd") || lower.includes("qsfp dd")) return "QSFP-DD";
if (lower.includes("qsfp28")) return "QSFP28";
if (lower.includes("qsfp+") || lower.includes("qsfp plus") || lower.includes("qsfp-plus")) return "QSFP+";
if (lower.includes("sfp28")) return "SFP28";
if (lower.includes("sfp+") || lower.includes("sfp plus") || lower.includes("sfp-plus")) return "SFP+";
if (lower.includes("sfp") && !lower.includes("qsfp")) return "SFP";
if (lower.includes("xfp")) return "XFP";
if (lower.includes("cfp2")) return "CFP2";
if (lower.includes("cfp")) return "CFP";
return undefined;
}
function detectSpeed(text: string): { speed: string; speedGbps: number } | undefined {
const patterns: [RegExp, string, number][] = [
[/400\s*g/i, "400G", 400],
[/100\s*g/i, "100G", 100],
[/40\s*g/i, "40G", 40],
[/25\s*g/i, "25G", 25],
[/10\s*g/i, "10G", 10],
[/1000\s*base/i, "1G", 1],
[/1\s*g\b/i, "1G", 1],
];
for (const [re, speed, gbps] of patterns) {
if (re.test(text)) return { speed, speedGbps: gbps };
}
return undefined;
}
function detectReach(text: string): string | undefined {
const match = text.match(/(\d+)\s*(m|km)\b/i);
if (match) return `${match[1]}${match[2].toLowerCase()}`;
return undefined;
}
function detectFiberType(text: string): string | undefined {
const lower = text.toLowerCase();
if (lower.includes("single mode") || lower.includes("single-mode") || lower.includes("smf") || lower.includes("-lr") || lower.includes("-er") || lower.includes("-zr")) return "SMF";
if (lower.includes("multi mode") || lower.includes("multi-mode") || lower.includes("mmf") || lower.includes("-sr") || lower.includes("-sx")) return "MMF";
if (lower.includes("dac") || lower.includes("direct attach") || lower.includes("copper") || lower.includes("-t ") || lower.includes("twinax")) return "DAC";
return undefined;
}
export async function scrapeAtgbics(): Promise<void> {
console.log("=== ATGBICS Scraper Starting ===\n");
const vendorId = await ensureVendor(
"ATGBICS",
"compatible",
"https://www.atgbics.com",
"https://www.atgbics.com/collections/sfp-plus-transceivers/"
);
console.log(`Vendor ID: ${vendorId}`);
const products: AtgbicsProduct[] = [];
let pagesScraped = 0;
const crawler = new PlaywrightCrawler({
maxConcurrency: 1,
maxRequestsPerMinute: 20, // ~2s between requests at concurrency 1
maxRequestsPerCrawl: MAX_PAGES,
requestHandlerTimeoutSecs: 60,
headless: true,
launchContext: {
launchOptions: {
args: ["--disable-blink-features=AutomationControlled", "--no-sandbox"],
},
},
async requestHandler({ page, request, enqueueLinks, log }) {
const url = request.url;
log.info(`Scraping: ${url}`);
// Wait for Shopify product grid to render
await page.waitForTimeout(2000);
// Check if this is a collection (listing) page or a product page
const isCollection = url.includes("/collections/");
if (isCollection) {
// Extract product links from listing page and enqueue them
const productData = await page.evaluate(() => {
const results: Array<{
name: string;
href: string;
price: string;
stock: string;
partNumber: string;
}> = [];
// Shopify collection page — product cards
const cards = document.querySelectorAll(
".product-item, .grid-product, [class*=\"product-card\"], [class*=\"product-grid\"] li, .collection-grid__item"
);
for (const card of cards) {
const linkEl = card.querySelector("a[href*=\"/products/\"]") as HTMLAnchorElement | null;
const nameEl = card.querySelector(
".product-item__title, .grid-product__title, [class*=\"product-title\"], [class*=\"product-name\"], h2, h3"
);
const priceEl = card.querySelector(
".product-item__price, .grid-product__price, [class*=\"price\"]:not([class*=\"compare\"]):not([class*=\"was\"])"
);
const stockEl = card.querySelector(
"[class*=\"stock\"], [class*=\"availability\"], [class*=\"badge\"]"
);
const href = linkEl?.getAttribute("href") || "";
const name = nameEl?.textContent?.trim() || linkEl?.textContent?.trim() || "";
const price = priceEl?.textContent?.trim() || "";
const stock = stockEl?.textContent?.trim() || "";
// Derive part number from URL slug: /products/sfp-10g-lr → sfp-10g-lr
const slug = href.split("/products/")[1]?.split("?")[0]?.replace(/\/$/, "") || "";
if (href && name && name.length > 3) {
results.push({ name, href, price, stock, partNumber: slug });
}
}
// Fallback: grab any /products/ links with adjacent price text
if (results.length === 0) {
const allProductLinks = document.querySelectorAll("a[href*=\"/products/\"]");
const seen = new Set<string>();
for (const el of allProductLinks) {
const a = el as HTMLAnchorElement;
const href = a.getAttribute("href") || "";
if (seen.has(href)) continue;
seen.add(href);
const name = a.textContent?.trim() || "";
if (!name || name.length < 3) continue;
const container = a.closest("li") || a.closest("article") || a.parentElement?.parentElement;
const priceEl = container?.querySelector("[class*=\"price\"]");
const price = priceEl?.textContent?.trim() || "";
const slug = href.split("/products/")[1]?.split("?")[0]?.replace(/\/$/, "") || "";
results.push({ name, href, price, stock: "", partNumber: slug });
}
}
return results;
});
log.info(` Found ${productData.length} products on collection page`);
for (const item of productData) {
if (!item.href) continue;
const fullUrl = item.href.startsWith("http") ? item.href : `${BASE_URL}${item.href}`;
// If we already have price data from the listing, store it directly
if (item.price) {
const { price, currency } = parsePrice(item.price);
const speedInfo = detectSpeed(item.name);
if (price > 0) {
products.push({
partNumber: item.partNumber || item.name.slice(0, 80),
name: item.name,
price,
currency: currency === "USD" ? "GBP" : currency, // ATGBICS is GBP — parsePrice may default to USD if no symbol on listing
stockLevel: item.stock ? parseStockLevel(item.stock) : "in_stock",
quantity: item.stock ? parseQuantity(item.stock) : undefined,
url: fullUrl,
formFactor: detectFormFactor(item.name),
speedGbps: speedInfo?.speedGbps,
speed: speedInfo?.speed,
reachLabel: detectReach(item.name),
fiberType: detectFiberType(item.name),
});
}
}
}
// Enqueue next page if pagination exists
await enqueueLinks({
selector: "a[href*=\"?page=\"], a.pagination__next, a[rel=\"next\"], .pagination a[href]",
transformRequestFunction: (req) => {
if (pagesScraped >= MAX_PAGES) return false;
return req;
},
});
pagesScraped++;
} else {
// Product detail page — extract precise data
const data = await page.evaluate(() => {
const title = document.querySelector(
"h1.product__title, h1.product-title, h1.product_title, h1"
)?.textContent?.trim() || "";
// Shopify price — prefer sale price if available
const salePriceEl = document.querySelector(
".price__sale .price-item--sale, .product__price .money, [class*=\"price\"] .money, [data-product-price], .price ins"
);
const priceText = salePriceEl?.textContent?.trim() || "";
// Stock / availability
const stockEl = document.querySelector(
".product__availability, .availability, [class*=\"stock\"], [class*=\"inventory\"], .badge--sold-out, .badge--in-stock"
);
const stockText = stockEl?.textContent?.trim() || "";
// Quantity badge (some Shopify themes show "X in stock")
const qtyEl = document.querySelector("[class*=\"quantity\"], [class*=\"inventory-count\"]");
const qtyText = qtyEl?.textContent?.trim() || "";
// Short description / variant title for reach/fiber info
const descEl = document.querySelector(
".product__description, .product-description, .rte p:first-child, .product__short-description"
);
const description = descEl?.textContent?.trim() || "";
// SKU / part number (Shopify often exposes this)
const skuEl = document.querySelector(".product__sku, [class*=\"sku\"], [itemprop=\"sku\"]");
const sku = skuEl?.textContent?.replace(/SKU[:\s]*/i, "").trim() || "";
return { title, priceText, stockText, qtyText, description, sku };
});
const slug = url.split("/products/")[1]?.split("?")[0]?.replace(/\/$/, "") || "";
const partNumber = data.sku || slug;
const name = data.title || slug;
const combinedText = `${name} ${data.description}`;
const { price, currency } = parsePrice(data.priceText);
if (price > 0) {
const speedInfo = detectSpeed(combinedText);
products.push({
partNumber,
name,
price,
currency: currency === "USD" ? "GBP" : currency, // ATGBICS prices in GBP
stockLevel: data.stockText ? parseStockLevel(data.stockText) : "in_stock",
quantity: data.qtyText ? parseQuantity(data.qtyText) : undefined,
url,
formFactor: detectFormFactor(combinedText),
speedGbps: speedInfo?.speedGbps,
speed: speedInfo?.speed,
reachLabel: detectReach(combinedText),
fiberType: detectFiberType(combinedText),
});
}
pagesScraped++;
}
},
});
const startUrls = CATEGORY_URLS.map((path) => `${BASE_URL}${path}`);
await crawler.run(startUrls);
console.log(`\nPages scraped: ${pagesScraped}`);
console.log(`Products found: ${products.length}`);
// Deduplicate by partNumber — prefer product detail page data (more precise)
const uniqueProducts = new Map<string, AtgbicsProduct>();
for (const p of products) {
const key = p.partNumber || p.name;
const existing = uniqueProducts.get(key);
// Keep the entry with a non-GBP-forced currency (i.e., product detail page which has £ symbol)
if (!existing || existing.currency === "GBP" && p.currency !== "GBP") {
uniqueProducts.set(key, p);
} else if (!existing) {
uniqueProducts.set(key, p);
}
}
// Write to database
let written = 0;
let skipped = 0;
for (const p of uniqueProducts.values()) {
try {
const transceiverId = await findOrCreateScrapedTransceiver({
partNumber: p.partNumber,
vendorId,
formFactor: p.formFactor,
speedGbps: p.speedGbps,
speed: p.speed,
reachLabel: p.reachLabel,
fiberType: p.fiberType,
category: "DataCenter",
});
const hash = contentHash({ price: p.price, stock: p.stockLevel, qty: p.quantity });
const isNew = await upsertPriceObservation({
transceiverId,
sourceVendorId: vendorId,
price: p.price,
currency: p.currency,
stockLevel: p.stockLevel,
quantityAvailable: p.quantity,
url: p.url,
contentHash: hash,
});
if (isNew) written++;
else skipped++;
} catch (err) {
console.error(` Error: ${p.partNumber}:`, (err as Error).message);
}
}
console.log(`\nDatabase: ${written} new, ${skipped} unchanged (${uniqueProducts.size} unique)`);
console.log("=== ATGBICS Scraper Complete ===\n");
}
if (require.main === module) {
scrapeAtgbics()
.then(() => pool.end())
.catch((err) => {
console.error("Fatal:", err);
pool.end();
process.exit(1);
});
}

View File

@ -0,0 +1,242 @@
/**
* Champion ONE Scraper US-based compatible transceiver vendor
*
* championone.com Server-rendered HTML, no JS required.
* Large catalog: SFP, SFP+, SFP28, QSFP+, QSFP28, QSFP-DD, OSFP, XFP, X2, GBIC
*
* Rate limited: 1 req/2sec.
*/
import { pool, findOrCreateScrapedTransceiver, ensureVendor, upsertPriceObservation } from "../utils/db";
import { contentHash } from "../utils/hash";
const BASE = "https://www.championone.com";
const HEADERS = {
"User-Agent": "Mozilla/5.0 (compatible; TIP-Bot/1.0; research)",
Accept: "text/html,application/xhtml+xml",
};
const CATEGORIES = [
{ path: "/sfp-transceivers", formFactor: "SFP", speed: "1G", speedGbps: 1 },
{ path: "/sfp-plus-transceivers", formFactor: "SFP+", speed: "10G", speedGbps: 10 },
{ path: "/25g-sfp28-transceivers", formFactor: "SFP28", speed: "25G", speedGbps: 25 },
{ path: "/qsfp-plus-transceivers", formFactor: "QSFP+", speed: "40G", speedGbps: 40 },
{ path: "/qsfp28-transceivers", formFactor: "QSFP28", speed: "100G", speedGbps: 100 },
{ path: "/qsfp-dd-transceivers", formFactor: "QSFP-DD", speed: "400G", speedGbps: 400 },
{ path: "/xfp-transceivers", formFactor: "XFP", speed: "10G", speedGbps: 10 },
];
interface Product {
partNumber: string;
name: string;
url: string;
price?: number;
currency?: string;
formFactor: string;
speed: string;
speedGbps: number;
reachLabel?: string;
reachMeters?: number;
fiberType?: string;
wavelength?: string;
}
function sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}
function detectReach(text: string): { label: string; meters: number } | undefined {
const patterns: [RegExp, string, number][] = [
[/\b160\s*km\b/i, "160km", 160000],
[/\b80\s*km\b/i, "80km", 80000],
[/\b40\s*km\b/i, "40km", 40000],
[/\b20\s*km\b/i, "20km", 20000],
[/\b10\s*km\b/i, "10km", 10000],
[/\b2\s*km\b/i, "2km", 2000],
[/\b550\s*m\b/i, "550m", 550],
[/\b500\s*m\b/i, "500m", 500],
[/\b300\s*m\b/i, "300m", 300],
[/\b100\s*m\b/i, "100m", 100],
[/\bLR4\b/, "10km", 10000],
[/\bLR\b/, "10km", 10000],
[/\bER4?\b/, "40km", 40000],
[/\bZR4?\b/, "80km", 80000],
[/\bSR4?\b/, "300m", 300],
[/\bDR4?\b/, "500m", 500],
[/\bFR4?\b/, "2km", 2000],
[/\bCWDM4\b/i, "2km", 2000],
[/\bPSM4\b/i, "500m", 500],
];
for (const [regex, label, meters] of patterns) {
if (regex.test(text)) return { label, meters };
}
return undefined;
}
function detectFiber(text: string): string {
if (/single.?mode|smf|[^a-z]lx[^a-z]|[^a-z]lr[^a-z]|[^a-z]er[^a-z]|[^a-z]zr[^a-z]|bidi|cwdm|dwdm/i.test(text)) return "SMF";
if (/multi.?mode|mmf|[^a-z]sx[^a-z]|[^a-z]sr[^a-z]/i.test(text)) return "MMF";
if (/copper|dac|twinax|rj45|base-t/i.test(text)) return "Copper";
return "";
}
function detectWavelength(text: string): string {
const match = text.match(/(\d{3,4})\s*nm/i);
if (match) return match[1];
return "";
}
function parseProductList(html: string, cat: typeof CATEGORIES[number]): Product[] {
const products: Product[] = [];
// Champion ONE uses standard ecommerce HTML with product cards
const productRegex = /href="(\/[^"]*?(?:transceiver|sfp|qsfp|osfp|xfp|optic)[^"]*)"[^>]*>([^<]{5,})<\/a>/gi;
let match;
while ((match = productRegex.exec(html)) !== null) {
const url = match[1];
const name = match[2].trim();
if (name.length < 8 || name.length > 200) continue;
const context = html.slice(Math.max(0, match.index - 200), match.index + 500);
const priceMatch = context.match(/\$\s*([\d,]+\.?\d{0,2})/) || context.match(/USD\s*([\d,]+\.?\d{0,2})/i);
const price = priceMatch ? parseFloat(priceMatch[1].replace(",", "")) : undefined;
const partNum = name.replace(/\s+/g, "-").replace(/[^a-zA-Z0-9\-]/g, "").slice(0, 80);
const reach = detectReach(name);
products.push({
partNumber: partNum, name,
url: url.startsWith("http") ? url : BASE + url,
price: price && price > 0 && price < 50000 ? price : undefined,
currency: price ? "USD" : undefined,
formFactor: cat.formFactor, speed: cat.speed, speedGbps: cat.speedGbps,
reachLabel: reach?.label, reachMeters: reach?.meters,
fiberType: detectFiber(name), wavelength: detectWavelength(name),
});
}
// Pattern 2: Generic product card pattern
const cardRegex = /class="[^"]*product[^"]*"[\s\S]*?href="([^"]+)"[^>]*>[\s\S]*?(?:name|title)[^>]*>([^<]+)/gi;
while ((match = cardRegex.exec(html)) !== null) {
const url = match[1];
const name = match[2].trim();
if (products.find((p) => p.url === (url.startsWith("http") ? url : BASE + url))) continue;
if (name.length < 8) continue;
const context = html.slice(match.index, match.index + 500);
const priceMatch = context.match(/\$\s*([\d,]+\.?\d{0,2})/);
const price = priceMatch ? parseFloat(priceMatch[1].replace(",", "")) : undefined;
const reach = detectReach(name);
products.push({
partNumber: name.replace(/\s+/g, "-").replace(/[^a-zA-Z0-9\-]/g, "").slice(0, 80),
name,
url: url.startsWith("http") ? url : BASE + url,
price: price && price > 0 && price < 50000 ? price : undefined,
currency: price ? "USD" : undefined,
formFactor: cat.formFactor, speed: cat.speed, speedGbps: cat.speedGbps,
reachLabel: reach?.label, reachMeters: reach?.meters,
fiberType: detectFiber(name), wavelength: detectWavelength(name),
});
}
const seen = new Set<string>();
return products.filter((p) => {
if (seen.has(p.url)) return false;
seen.add(p.url);
return true;
});
}
function getMaxPage(html: string): number {
const pageMatches = html.match(/[?&]page=(\d+)/g) || html.match(/\/page\/(\d+)/g);
if (!pageMatches) return 1;
let max = 1;
for (const m of pageMatches) {
const n = parseInt(m.replace(/[^0-9]/g, ""));
if (n > max) max = n;
}
return Math.min(max, 30);
}
async function fetchPage(url: string): Promise<string> {
const resp = await fetch(url, { headers: HEADERS, signal: AbortSignal.timeout(30000) });
if (!resp.ok) throw new Error(`HTTP ${resp.status} for ${url}`);
return resp.text();
}
export async function scrapeChampionOne(): Promise<void> {
console.log("=== Champion ONE Scraper Starting ===\n");
const vendorId = await ensureVendor("Champion ONE", "compatible", "https://www.championone.com", "https://www.championone.com");
let totalProducts = 0;
let priceUpdates = 0;
for (const cat of CATEGORIES) {
console.log(`\n--- ${cat.formFactor} (${cat.speed}) ---`);
try {
const firstPage = await fetchPage(BASE + cat.path);
const maxPage = getMaxPage(firstPage);
console.log(` Pages: ${maxPage}`);
let catProducts: Product[] = parseProductList(firstPage, cat);
for (let page = 2; page <= maxPage; page++) {
await sleep(2000);
try {
const html = await fetchPage(`${BASE}${cat.path}?page=${page}`);
catProducts.push(...parseProductList(html, cat));
} catch (err) {
console.warn(` Page ${page} failed: ${(err as Error).message}`);
}
}
const seen = new Set<string>();
catProducts = catProducts.filter((p) => {
if (seen.has(p.url)) return false;
seen.add(p.url);
return true;
});
console.log(` Found ${catProducts.length} products`);
for (const product of catProducts) {
try {
const txId = await findOrCreateScrapedTransceiver({
partNumber: product.partNumber, vendorId,
formFactor: product.formFactor, speedGbps: product.speedGbps,
speed: product.speed, reachMeters: product.reachMeters,
reachLabel: product.reachLabel, fiberType: product.fiberType,
wavelengths: product.wavelength, category: "DataCenter",
});
if (product.price && product.price > 0) {
const hash = contentHash(JSON.stringify({ price: product.price, part: product.partNumber }));
const updated = await upsertPriceObservation({
transceiverId: txId, sourceVendorId: vendorId,
price: product.price, currency: product.currency || "USD",
stockLevel: "in_stock", url: product.url, contentHash: hash,
});
if (updated) priceUpdates++;
}
totalProducts++;
} catch (err) {
console.warn(` Error: ${(err as Error).message.slice(0, 80)}`);
}
}
} catch (err) {
console.error(` Category failed: ${(err as Error).message}`);
}
await sleep(2000);
}
console.log(`\n=== Champion ONE Complete: ${totalProducts} products, ${priceUpdates} prices ===`);
}
if (require.main === module) {
scrapeChampionOne()
.then(() => pool.end())
.catch((err) => { console.error("Fatal:", err); pool.end(); process.exit(1); });
}

View File

@ -0,0 +1,221 @@
/**
* Cisco TMG Matrix Scraper Transceiver Compatibility
*
* Source: tmgmatrix.cisco.com (JSON API no auth required)
* Extracts: Switch model Transceiver compatibility data
* Stores: switches, compatibility table
*
* Uses POST /public/api/networkdevice/search endpoint directly.
*/
import { pool, ensureVendor } from "../utils/db";
const TMG_API = "https://tmgmatrix.cisco.com/public/api/networkdevice/search";
interface TmgTransceiver {
tmgId: number;
productId: string;
productFamily: string;
formFactor: string;
reach: string;
temperatureRange: string;
cableType: string;
media: string;
connectorType: string;
transmissionStandard: string;
dataRate: string;
endOfSale: string;
softReleaseMinVer: string;
breakoutMode: string;
osType: string;
domSupport: string;
type: string;
}
interface TmgCompatEntry {
productId: string; // switch PID
transceivers: TmgTransceiver[];
}
interface TmgDevice {
productFamily: string;
networkAndTransceiverCompatibility: TmgCompatEntry[];
}
interface TmgSearchResponse {
totalCount: number;
filters: Array<{ name: string; values: Array<{ id: number; name: string; count: number }> }>;
networkDevices: TmgDevice[];
}
/** Key Nexus/Catalyst platform family IDs from the TMG API */
const PLATFORM_FAMILIES = [
{ id: 74, name: "N9300" }, // Nexus 9300 — 8,515 entries
{ id: 77, name: "N9500" }, // Nexus 9500 — 2,266 entries
{ id: 78, name: "N9200" }, // Nexus 9200 — 708 entries
{ id: 661, name: "N9800" }, // Nexus 9800 — 238 entries
{ id: 76, name: "C9300" }, // Catalyst 9300 — 260 entries
{ id: 601, name: "C9300L" }, // Catalyst 9300L — 720 entries
{ id: 1181, name: "C9300X" }, // Catalyst 9300X — 413 entries
{ id: 8, name: "C9500" }, // Catalyst 9500 — 1,141 entries
{ id: 521, name: "C9600" }, // Catalyst 9600 — 771 entries
{ id: 7, name: "C9400" }, // Catalyst 9400 — 561 entries
{ id: 341, name: "C9200" }, // Catalyst 9200 — 222 entries
{ id: 83, name: "ASR9000" }, // ASR 9000 — 3,644 entries
];
async function searchTmg(familyFilter: { id: number; name: string }): Promise<TmgSearchResponse> {
const body = {
cableType: [],
dataRate: [],
formFactor: [],
reach: [],
searchInput: [""],
osType: [],
transceiverProductFamily: [],
transceiverProductID: [],
networkDeviceProductFamily: [familyFilter],
networkDeviceProductID: [],
media: [],
connectorType: [],
caseTemperature: [],
performanceMonitoring: [],
};
const res = await fetch(TMG_API, {
method: "POST",
headers: {
"Content-Type": "application/json",
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36",
"Accept": "application/json",
},
body: JSON.stringify(body),
});
if (!res.ok) {
throw new Error(`TMG API ${res.status}: ${res.statusText}`);
}
return res.json() as Promise<TmgSearchResponse>;
}
async function upsertCiscoSwitch(vendorId: string, model: string, series: string): Promise<string> {
const result = await pool.query(
`INSERT INTO switches (vendor_id, model, series, category, layer, managed)
VALUES ($1, $2, $3, 'DataCenter', 'L3', true)
ON CONFLICT (vendor_id, model) DO UPDATE SET series = EXCLUDED.series
RETURNING id`,
[vendorId, model, series]
);
return result.rows[0].id;
}
async function upsertCompatibility(
switchId: string,
transceiverId: string,
firmwareMin: string,
formFactor: string,
reach: string,
cableType: string,
media: string,
dataRate: string
): Promise<void> {
await pool.query(
`INSERT INTO compatibility (switch_id, transceiver_id, verified_by, verification_method, status, firmware_min, source_url, notes)
VALUES ($1, $2, 'Cisco TMG Matrix', 'vendor_matrix', 'compatible', $3, $4, $5)
ON CONFLICT (switch_id, transceiver_id) DO UPDATE SET
firmware_min = EXCLUDED.firmware_min,
notes = EXCLUDED.notes`,
[
switchId,
transceiverId,
firmwareMin || null,
"https://tmgmatrix.cisco.com",
`${formFactor} ${dataRate} ${reach} ${media} ${cableType}`.trim(),
]
);
}
export async function scrapeCiscoTmg(): Promise<void> {
console.log("=== Cisco TMG Matrix Scraper Starting (API mode) ===\n");
const ciscoVendorId = await ensureVendor(
"Cisco",
"oem",
"https://www.cisco.com",
undefined
);
let totalSwitches = 0;
let totalCompat = 0;
let totalTransceivers = 0;
for (const family of PLATFORM_FAMILIES) {
console.log(`\nFetching ${family.name}...`);
try {
const data = await searchTmg(family);
console.log(` ${family.name}: ${data.totalCount} total entries, ${data.networkDevices.length} device groups`);
for (const device of data.networkDevices) {
for (const compat of device.networkAndTransceiverCompatibility) {
if (!compat.productId) continue;
const switchId = await upsertCiscoSwitch(
ciscoVendorId,
compat.productId,
device.productFamily
);
totalSwitches++;
for (const tx of compat.transceivers) {
if (!tx.productId) continue;
totalTransceivers++;
// Try to match transceiver in our DB by Cisco PID
const txResult = await pool.query(
`SELECT id FROM transceivers
WHERE part_number = $1
OR part_number = $2
LIMIT 1`,
[tx.productId, tx.productId.replace(/-S$/, "")]
);
if (txResult.rows.length > 0) {
await upsertCompatibility(
switchId,
txResult.rows[0].id,
tx.softReleaseMinVer,
tx.formFactor,
tx.reach,
tx.cableType,
tx.media,
tx.dataRate
);
totalCompat++;
}
}
}
}
// Rate limit: 2 seconds between platform families
await new Promise((r) => setTimeout(r, 2000));
} catch (err) {
console.error(` Error fetching ${family.name}:`, err);
}
}
console.log(`\n=== Cisco TMG Scraper Complete ===`);
console.log(` Switches upserted: ${totalSwitches}`);
console.log(` Transceiver entries scanned: ${totalTransceivers}`);
console.log(` Compatibility matches: ${totalCompat}\n`);
}
if (require.main === module) {
scrapeCiscoTmg()
.then(() => pool.end())
.catch((err) => {
console.error("Fatal:", err);
pool.end();
process.exit(1);
});
}

View File

@ -0,0 +1,200 @@
/**
* Edgecore Networks Product Catalog Scraper
*
* Scrapes switch product pages from edge-core.com for:
* - Product specs (ports, ASIC, power, dimensions)
* - Transceiver form factor compatibility
* - Datasheet URLs
*
* Source: https://www.edge-core.com/productsList.php?cls=1
*/
import { CheerioCrawler } from "crawlee";
import { pool, ensureWhiteboxVendor, findOrCreateSwitch } from "../utils/db";
const BASE_URL = "https://www.edge-core.com";
const PRODUCT_LIST_URL = `${BASE_URL}/productsList.php?cls=1`;
/**
* Extract port configuration from spec text.
* Handles formats like "32x 100GbE QSFP28" or "48x25G SFP28 + 8x100G QSFP28"
*/
function extractPortsFromSpec(specText: string): {
portsConfig: Record<string, number>;
totalPorts: number;
maxSpeedGbps: number;
formFactors: string[];
} {
const portsConfig: Record<string, number> = {};
let totalPorts = 0;
let maxSpeedGbps = 0;
const formFactors: string[] = [];
const portPattern = /(\d+)\s*x\s*(\d+)\s*G(?:bE|b\/s)?\s*(QSFP-DD|QSFP28|QSFP\+|QSFP56|SFP28|SFP\+|SFP56|OSFP|CFP2)?/gi;
let match: RegExpExecArray | null;
while ((match = portPattern.exec(specText)) !== null) {
const count = parseInt(match[1]);
const speed = parseInt(match[2]);
const ff = match[3]?.toUpperCase() || `${speed}G`;
const key = `${speed}G_${ff}`;
portsConfig[key] = (portsConfig[key] || 0) + count;
totalPorts += count;
maxSpeedGbps = Math.max(maxSpeedGbps, speed);
if (match[3] && !formFactors.includes(match[3].toUpperCase())) {
formFactors.push(match[3].toUpperCase());
}
}
return { portsConfig, totalPorts, maxSpeedGbps, formFactors };
}
/**
* Detect ASIC from product page text.
*/
function detectAsic(text: string): { vendor: string; model: string; series: string } {
const asicPatterns: Array<{ pattern: RegExp; vendor: string; model: string; series: string }> = [
{ pattern: /tomahawk\s*5/i, vendor: "Broadcom", model: "Tomahawk 5", series: "StrataDNX" },
{ pattern: /tomahawk\s*4/i, vendor: "Broadcom", model: "Tomahawk 4", series: "StrataDNX" },
{ pattern: /tomahawk\s*3/i, vendor: "Broadcom", model: "Tomahawk 3", series: "StrataDNX" },
{ pattern: /tomahawk\s*2/i, vendor: "Broadcom", model: "Tomahawk 2", series: "StrataDNX" },
{ pattern: /tomahawk\+/i, vendor: "Broadcom", model: "Tomahawk+", series: "StrataDNX" },
{ pattern: /tomahawk/i, vendor: "Broadcom", model: "Tomahawk", series: "StrataDNX" },
{ pattern: /trident\s*(4|iv)/i, vendor: "Broadcom", model: "Trident 4", series: "StrataDNX" },
{ pattern: /trident\s*(3|iii)/i, vendor: "Broadcom", model: "Trident III", series: "StrataDNX" },
{ pattern: /jericho\s*2/i, vendor: "Broadcom", model: "Jericho2", series: "StrataDNX" },
{ pattern: /spectrum-?4/i, vendor: "NVIDIA", model: "Spectrum-4", series: "Spectrum" },
{ pattern: /teralynx/i, vendor: "Marvell", model: "Teralynx", series: "Teralynx" },
{ pattern: /prestera/i, vendor: "Marvell", model: "Prestera", series: "Prestera" },
];
for (const { pattern, vendor, model, series } of asicPatterns) {
if (pattern.test(text)) {
return { vendor, model, series };
}
}
return { vendor: "Broadcom", model: "Unknown", series: "" };
}
export async function scrapeEdgecore(): Promise<void> {
console.log("\n=== Edgecore Networks Scraper ===\n");
const vendorId = await ensureWhiteboxVendor("Edgecore Networks", "https://www.edge-core.com", {
isOdm: true,
ocpMember: true,
sonicContributor: true,
});
let created = 0;
let updated = 0;
const crawler = new CheerioCrawler({
maxConcurrency: 2,
maxRequestsPerMinute: 20,
requestHandlerTimeoutSecs: 30,
async requestHandler({ request, $, enqueueLinks }) {
// Product list page — enqueue individual product pages
if (request.url.includes("productsList")) {
console.log(" Parsing product list page...");
const productLinks: string[] = [];
$("a[href*='product']").each((_i, el) => {
const href = $(el).attr("href");
if (href && (href.includes("productsInfo") || href.includes("product/"))) {
const fullUrl = href.startsWith("http") ? href : `${BASE_URL}/${href}`;
if (!productLinks.includes(fullUrl)) {
productLinks.push(fullUrl);
}
}
});
console.log(` Found ${productLinks.length} product links`);
for (const link of productLinks) {
await enqueueLinks({ urls: [link] });
}
return;
}
// Individual product page
const pageText = $("body").text();
const title = $("h1, .product-title, .prod-name").first().text().trim();
if (!title) return;
// Extract model name
const modelMatch = title.match(/(AS\d{4}[A-Z0-9-]*|DCS\d{3}[A-Z0-9-]*|Minipack\d*|Wedge\d*)/i);
if (!modelMatch) return;
const model = modelMatch[1];
const portInfo = extractPortsFromSpec(pageText);
const asicInfo = detectAsic(pageText);
if (portInfo.totalPorts === 0) return;
// Extract additional specs
const powerMatch = pageText.match(/(?:max|maximum)\s*power[:\s]*(\d+)\s*W/i);
const cpuMatch = pageText.match(/(Intel\s+(?:Xeon|Atom|Core)[^\n,;]+)/i);
const ramMatch = pageText.match(/(\d+)\s*GB?\s*(?:DDR[34]|RAM|memory)/i);
const storageMatch = pageText.match(/(\d+)\s*GB?\s*(?:SSD|eMMC|M\.2)/i);
const switchCapMatch = pageText.match(/switching\s*capacity[:\s]*([\d.]+)\s*Tb/i);
const seriesMatch = model.match(/^(AS\d{4}|DCS\d{3}|Minipack|Wedge)/);
const series = seriesMatch ? seriesMatch[1] : "";
const existing = await pool.query(
`SELECT id FROM switches WHERE model = $1 AND vendor_id = $2`,
[model, vendorId]
);
const isNew = existing.rows.length === 0;
await findOrCreateSwitch({
model,
vendorId,
series,
category: "DataCenter",
layer: "L3",
portsConfig: portInfo.portsConfig,
totalPorts: portInfo.totalPorts,
maxSpeedGbps: portInfo.maxSpeedGbps,
switchingCapacityTbps: switchCapMatch ? parseFloat(switchCapMatch[1]) : undefined,
asicVendor: asicInfo.vendor,
asicModel: asicInfo.model,
asicSeries: asicInfo.series,
maxPowerW: powerMatch ? parseInt(powerMatch[1]) : undefined,
cpu: cpuMatch ? cpuMatch[1].trim() : undefined,
ramGb: ramMatch ? parseInt(ramMatch[1]) : undefined,
storageGb: storageMatch ? parseInt(storageMatch[1]) : undefined,
sonicCompatible: true,
isWhitebox: true,
onieSupport: true,
supportedNos: ["SONiC", "ONL"],
transceiverFormFactors: portInfo.formFactors,
catalogUrl: request.url,
tags: [
"whitebox",
"Edgecore",
`${portInfo.maxSpeedGbps}G`,
asicInfo.model,
],
scrapeSource: "edgecore-catalog",
});
if (isNew) {
created++;
console.log(` + ${model} (${portInfo.maxSpeedGbps}G, ${asicInfo.vendor} ${asicInfo.model})`);
} else {
updated++;
}
},
failedRequestHandler({ request }) {
console.error(` ! Failed: ${request.url}`);
},
});
await crawler.run([PRODUCT_LIST_URL]);
console.log(`\n Created: ${created}, Updated: ${updated}\n`);
}

View File

@ -0,0 +1,568 @@
/**
* Flexoptix Product Catalog Scraper
*
* Scrapes flexoptix.net product catalog for transceiver specs and pricing.
* This is our own data no restrictions.
*
* Strategy: Use the Magento search/suggest AJAX API which returns JSON
* with product names, URLs, prices, and SKUs. We query by form factor
* keywords to enumerate the full catalog.
*
* Rate limited: 1 req/sec.
*/
import { pool, findOrCreateScrapedTransceiver, ensureVendor, upsertPriceObservation } from "../utils/db";
import { contentHash } from "../utils/hash";
const BASE = "https://www.flexoptix.net";
const SEARCH_URL = `${BASE}/en/search/ajax/suggest/`;
const HEADERS = {
"User-Agent": "Mozilla/5.0 (compatible; TIP-Bot/1.0; internal-flexoptix)",
Accept: "application/json, text/html",
};
// Search queries that cover the full transceiver catalog
const SEARCH_QUERIES = [
// By form factor
{ query: "SFP 1G", formFactor: "SFP", speed: "1G", speedGbps: 1 },
{ query: "SFP BiDi", formFactor: "SFP", speed: "1G", speedGbps: 1 },
{ query: "SFP CWDM", formFactor: "SFP", speed: "1G", speedGbps: 1 },
{ query: "SFP DWDM", formFactor: "SFP", speed: "1G", speedGbps: 1 },
{ query: "SFP copper", formFactor: "SFP", speed: "1G", speedGbps: 1 },
{ query: "SFP+ 10G", formFactor: "SFP+", speed: "10G", speedGbps: 10 },
{ query: "SFP+ SR", formFactor: "SFP+", speed: "10G", speedGbps: 10 },
{ query: "SFP+ LR", formFactor: "SFP+", speed: "10G", speedGbps: 10 },
{ query: "SFP+ ER", formFactor: "SFP+", speed: "10G", speedGbps: 10 },
{ query: "SFP+ ZR", formFactor: "SFP+", speed: "10G", speedGbps: 10 },
{ query: "SFP+ BiDi", formFactor: "SFP+", speed: "10G", speedGbps: 10 },
{ query: "SFP+ CWDM", formFactor: "SFP+", speed: "10G", speedGbps: 10 },
{ query: "SFP+ DWDM", formFactor: "SFP+", speed: "10G", speedGbps: 10 },
{ query: "SFP+ DAC", formFactor: "SFP+", speed: "10G", speedGbps: 10 },
{ query: "SFP+ AOC", formFactor: "SFP+", speed: "10G", speedGbps: 10 },
{ query: "25G SFP28", formFactor: "SFP28", speed: "25G", speedGbps: 25 },
{ query: "SFP28 SR", formFactor: "SFP28", speed: "25G", speedGbps: 25 },
{ query: "SFP28 LR", formFactor: "SFP28", speed: "25G", speedGbps: 25 },
{ query: "SFP28 DWDM", formFactor: "SFP28", speed: "25G", speedGbps: 25 },
{ query: "SFP28 DAC", formFactor: "SFP28", speed: "25G", speedGbps: 25 },
{ query: "SFP28 AOC", formFactor: "SFP28", speed: "25G", speedGbps: 25 },
{ query: "QSFP+ 40G", formFactor: "QSFP+", speed: "40G", speedGbps: 40 },
{ query: "QSFP+ SR4", formFactor: "QSFP+", speed: "40G", speedGbps: 40 },
{ query: "QSFP+ LR4", formFactor: "QSFP+", speed: "40G", speedGbps: 40 },
{ query: "QSFP+ DAC", formFactor: "QSFP+", speed: "40G", speedGbps: 40 },
{ query: "QSFP+ AOC", formFactor: "QSFP+", speed: "40G", speedGbps: 40 },
{ query: "QSFP28 100G", formFactor: "QSFP28", speed: "100G", speedGbps: 100 },
{ query: "QSFP28 SR4", formFactor: "QSFP28", speed: "100G", speedGbps: 100 },
{ query: "QSFP28 LR4", formFactor: "QSFP28", speed: "100G", speedGbps: 100 },
{ query: "QSFP28 ER4", formFactor: "QSFP28", speed: "100G", speedGbps: 100 },
{ query: "QSFP28 CWDM4", formFactor: "QSFP28", speed: "100G", speedGbps: 100 },
{ query: "QSFP28 PSM4", formFactor: "QSFP28", speed: "100G", speedGbps: 100 },
{ query: "QSFP28 DAC", formFactor: "QSFP28", speed: "100G", speedGbps: 100 },
{ query: "QSFP28 AOC", formFactor: "QSFP28", speed: "100G", speedGbps: 100 },
{ query: "QSFP56 200G", formFactor: "QSFP56", speed: "200G", speedGbps: 200 },
{ query: "QSFP-DD 400G", formFactor: "QSFP-DD", speed: "400G", speedGbps: 400 },
{ query: "QSFP-DD DR4", formFactor: "QSFP-DD", speed: "400G", speedGbps: 400 },
{ query: "QSFP-DD FR4", formFactor: "QSFP-DD", speed: "400G", speedGbps: 400 },
{ query: "QSFP-DD LR4", formFactor: "QSFP-DD", speed: "400G", speedGbps: 400 },
{ query: "QSFP-DD SR4", formFactor: "QSFP-DD", speed: "400G", speedGbps: 400 },
{ query: "QSFP-DD ZR", formFactor: "QSFP-DD", speed: "400G", speedGbps: 400 },
{ query: "QSFP-DD800 800G", formFactor: "QSFP-DD800", speed: "800G", speedGbps: 800 },
{ query: "OSFP 400G", formFactor: "OSFP", speed: "400G", speedGbps: 400 },
{ query: "OSFP SR4", formFactor: "OSFP", speed: "400G", speedGbps: 400 },
{ query: "OSFP DR4", formFactor: "OSFP", speed: "400G", speedGbps: 400 },
{ query: "OSFP FR4", formFactor: "OSFP", speed: "400G", speedGbps: 400 },
{ query: "OSFP LR4", formFactor: "OSFP", speed: "400G", speedGbps: 400 },
{ query: "OSFP ZR", formFactor: "OSFP", speed: "400G", speedGbps: 400 },
{ query: "OSFP 800G", formFactor: "OSFP", speed: "800G", speedGbps: 800 },
// Additional granular queries for maximum coverage
{ query: "SFP+ copper", formFactor: "SFP+", speed: "10G", speedGbps: 10 },
{ query: "SFP+ 10GBASE-T", formFactor: "SFP+", speed: "10G", speedGbps: 10 },
{ query: "SFP+ tunable", formFactor: "SFP+", speed: "10G", speedGbps: 10 },
{ query: "SFP RJ45", formFactor: "SFP", speed: "1G", speedGbps: 1 },
{ query: "SFP 100M", formFactor: "SFP", speed: "100M", speedGbps: 0.1 },
{ query: "SFP 100BASE", formFactor: "SFP", speed: "100M", speedGbps: 0.1 },
{ query: "QSFP28 CWDM", formFactor: "QSFP28", speed: "100G", speedGbps: 100 },
{ query: "QSFP28 ZR4", formFactor: "QSFP28", speed: "100G", speedGbps: 100 },
{ query: "QSFP28 FR", formFactor: "QSFP28", speed: "100G", speedGbps: 100 },
{ query: "QSFP28 DR", formFactor: "QSFP28", speed: "100G", speedGbps: 100 },
{ query: "QSFP28 copper", formFactor: "QSFP28", speed: "100G", speedGbps: 100 },
{ query: "QSFP-DD DAC", formFactor: "QSFP-DD", speed: "400G", speedGbps: 400 },
{ query: "QSFP-DD AOC", formFactor: "QSFP-DD", speed: "400G", speedGbps: 400 },
{ query: "QSFP-DD 200G", formFactor: "QSFP-DD", speed: "200G", speedGbps: 200 },
{ query: "OSFP DAC", formFactor: "OSFP", speed: "400G", speedGbps: 400 },
{ query: "OSFP AOC", formFactor: "OSFP", speed: "400G", speedGbps: 400 },
{ query: "XFP 10G", formFactor: "XFP", speed: "10G", speedGbps: 10 },
{ query: "XFP SR", formFactor: "XFP", speed: "10G", speedGbps: 10 },
{ query: "XFP LR", formFactor: "XFP", speed: "10G", speedGbps: 10 },
{ query: "CFP2 100G", formFactor: "CFP2", speed: "100G", speedGbps: 100 },
{ query: "CFP2 DCO", formFactor: "CFP2-DCO", speed: "200G", speedGbps: 200 },
{ query: "GBIC 1G", formFactor: "GBIC", speed: "1G", speedGbps: 1 },
{ query: "SFP56 50G", formFactor: "SFP56", speed: "50G", speedGbps: 50 },
{ query: "QSFP56 DAC", formFactor: "QSFP56", speed: "200G", speedGbps: 200 },
{ query: "QSFP112 400G", formFactor: "QSFP112", speed: "400G", speedGbps: 400 },
{ query: "OSFP112 800G", formFactor: "OSFP112", speed: "800G", speedGbps: 800 },
{ query: "direct attach cable", formFactor: "SFP+", speed: "10G", speedGbps: 10 },
{ query: "active optical cable", formFactor: "SFP+", speed: "10G", speedGbps: 10 },
{ query: "breakout cable", formFactor: "QSFP28", speed: "100G", speedGbps: 100 },
{ query: "MTP MPO cable", formFactor: "QSFP28", speed: "100G", speedGbps: 100 },
// Speed-specific reach variants
{ query: "10G 80km", formFactor: "SFP+", speed: "10G", speedGbps: 10 },
{ query: "10G 40km", formFactor: "SFP+", speed: "10G", speedGbps: 10 },
{ query: "10G 20km", formFactor: "SFP+", speed: "10G", speedGbps: 10 },
{ query: "100G 80km", formFactor: "QSFP28", speed: "100G", speedGbps: 100 },
{ query: "100G 40km", formFactor: "QSFP28", speed: "100G", speedGbps: 100 },
{ query: "400G 80km", formFactor: "QSFP-DD", speed: "400G", speedGbps: 400 },
{ query: "400G 120km", formFactor: "QSFP-DD", speed: "400G", speedGbps: 400 },
{ query: "400G 2km", formFactor: "QSFP-DD", speed: "400G", speedGbps: 400 },
{ query: "800G 2km", formFactor: "OSFP", speed: "800G", speedGbps: 800 },
{ query: "800G 10km", formFactor: "OSFP", speed: "800G", speedGbps: 800 },
// Vendor-specific coding searches
{ query: "Cisco compatible", formFactor: "SFP+", speed: "10G", speedGbps: 10 },
{ query: "Juniper compatible", formFactor: "SFP+", speed: "10G", speedGbps: 10 },
{ query: "Arista compatible", formFactor: "QSFP28", speed: "100G", speedGbps: 100 },
{ query: "Nokia compatible", formFactor: "SFP+", speed: "10G", speedGbps: 10 },
{ query: "Huawei compatible", formFactor: "SFP+", speed: "10G", speedGbps: 10 },
// Generic searches to catch stragglers
{ query: "transceiver SR", formFactor: "SFP+", speed: "10G", speedGbps: 10 },
{ query: "transceiver LR", formFactor: "SFP+", speed: "10G", speedGbps: 10 },
{ query: "transceiver ER", formFactor: "SFP+", speed: "10G", speedGbps: 10 },
{ query: "transceiver ZR", formFactor: "SFP+", speed: "10G", speedGbps: 10 },
{ query: "transceiver BiDi", formFactor: "SFP", speed: "1G", speedGbps: 1 },
{ query: "coherent 400ZR", formFactor: "QSFP-DD", speed: "400G", speedGbps: 400 },
{ query: "coherent ZR+", formFactor: "QSFP-DD", speed: "400G", speedGbps: 400 },
{ query: "coherent 100G", formFactor: "CFP2-DCO", speed: "100G", speedGbps: 100 },
{ query: "DWDM tunable", formFactor: "SFP+", speed: "10G", speedGbps: 10 },
{ query: "WDM multiplexer", formFactor: "SFP", speed: "1G", speedGbps: 1 },
{ query: "media converter", formFactor: "SFP", speed: "1G", speedGbps: 1 },
];
interface Product {
name: string;
partNumber: string;
url: string;
price?: number;
currency?: string;
formFactor: string;
speed: string;
speedGbps: number;
reachLabel?: string;
reachMeters?: number;
fiberType?: string;
wavelength?: string;
}
function sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}
function detectReach(text: string): { label: string; meters: number } | undefined {
const patterns: [RegExp, string, number][] = [
[/\b120\s*km\b/i, "120km", 120000],
[/\b80\s*km\b/i, "80km", 80000],
[/\b40\s*km\b/i, "40km", 40000],
[/\b20\s*km\b/i, "20km", 20000],
[/\b10\s*km\b/i, "10km", 10000],
[/\b2\s*km\b/i, "2km", 2000],
[/\b500\s*m\b/i, "500m", 500],
[/\b300\s*m\b/i, "300m", 300],
[/\b100\s*m\b/i, "100m", 100],
[/\bLR4\b/, "10km", 10000],
[/\bLR\b/, "10km", 10000],
[/\bER4?\b/, "40km", 40000],
[/\bZR4?\b/, "80km", 80000],
[/\bSR4?\b/, "100m", 100],
[/\bDR4?\b/, "500m", 500],
[/\bFR4?\b/, "2km", 2000],
[/\bCWDM4\b/i, "2km", 2000],
[/\bPSM4\b/i, "500m", 500],
];
for (const [regex, label, meters] of patterns) {
if (regex.test(text)) return { label, meters };
}
return undefined;
}
function detectFiber(text: string): string {
if (/single.?mode|smf|[^a-z]lx[^a-z]|[^a-z]lr[^a-z]|[^a-z]er[^a-z]|[^a-z]zr[^a-z]|bidi|cwdm|dwdm/i.test(text)) return "SMF";
if (/multi.?mode|mmf|[^a-z]sx[^a-z]|[^a-z]sr[^a-z]/i.test(text)) return "MMF";
if (/copper|dac|twinax|rj45|base-t/i.test(text)) return "Copper";
if (/aoc|active optical/i.test(text)) return "AOC";
return "";
}
function detectWavelength(text: string): string {
const match = text.match(/(\d{3,4})\s*nm/i);
if (match) return match[1];
return "";
}
function inferFormFactor(name: string, defaultFF: string): string {
const lower = name.toLowerCase();
if (lower.includes("osfp224")) return "OSFP224";
if (lower.includes("osfp112")) return "OSFP112";
if (lower.includes("osfp") && !lower.includes("qsfp")) return "OSFP";
if (lower.includes("qsfp-dd800")) return "QSFP-DD800";
if (lower.includes("qsfp-dd")) return "QSFP-DD";
if (lower.includes("qsfp112")) return "QSFP112";
if (lower.includes("qsfp56")) return "QSFP56";
if (lower.includes("qsfp28")) return "QSFP28";
if (lower.includes("qsfp+") || lower.includes("qsfp plus")) return "QSFP+";
if (lower.includes("sfp56")) return "SFP56";
if (lower.includes("sfp28")) return "SFP28";
if (lower.includes("sfp+") || lower.includes("sfp plus")) return "SFP+";
if (lower.includes("cfp2")) return "CFP2";
if (lower.includes("xfp")) return "XFP";
if (/\bsfp\b/i.test(lower) && !lower.includes("qsfp")) return "SFP";
return defaultFF;
}
function inferSpeed(name: string, defaultGbps: number): number {
const patterns: [RegExp, number][] = [
[/\b1\.6\s*T\b/i, 1600],
[/\b800\s*G\b/i, 800],
[/\b400\s*G\b/i, 400],
[/\b200\s*G\b/i, 200],
[/\b100\s*G\b/i, 100],
[/\b50\s*G\b/i, 50],
[/\b40\s*G\b/i, 40],
[/\b25\s*G\b/i, 25],
[/\b10\s*G\b/i, 10],
[/\b1\s*G\b/i, 1],
];
for (const [regex, gbps] of patterns) {
if (regex.test(name)) return gbps;
}
return defaultGbps;
}
function speedLabel(gbps: number): string {
if (gbps >= 1000) return `${gbps / 1000}T`;
return `${gbps}G`;
}
interface SearchResult {
title: string;
url: string;
price?: string;
sku?: string;
}
async function searchProducts(query: string): Promise<SearchResult[]> {
const url = `${SEARCH_URL}?q=${encodeURIComponent(query)}`;
const resp = await fetch(url, { headers: HEADERS, signal: AbortSignal.timeout(15000) });
if (!resp.ok) return [];
const text = await resp.text();
// The response may be JSON or HTML with embedded product data
// Try JSON parse first
try {
const data = JSON.parse(text);
const results: SearchResult[] = [];
/** Extract price from Magento price HTML (data-price-amount="39.64") or plain number */
function extractPrice(priceField: unknown): string | undefined {
if (!priceField) return undefined;
const s = String(priceField);
// Try data-price-amount attribute first (Magento Hyva theme)
const attrMatch = s.match(/data-price-amount="([\d.]+)"/);
if (attrMatch) return attrMatch[1];
// Try plain price text like "39.64 EUR"
const textMatch = s.match(/([\d.]+)\s*EUR/i);
if (textMatch) return textMatch[1];
// Try bare number
const num = parseFloat(s);
if (!isNaN(num) && num > 0) return String(num);
return undefined;
}
// Handle various Magento search response formats
if (Array.isArray(data)) {
for (const item of data) {
if (item.title && item.url) {
results.push({
title: item.title,
url: item.url,
price: extractPrice(item.price),
sku: item.sku,
});
}
}
} else if (data.products && Array.isArray(data.products)) {
for (const item of data.products) {
results.push({
title: item.title || item.name || "",
url: item.url || item.product_url || "",
price: extractPrice(item.price),
sku: item.sku,
});
}
} else if (typeof data === "object") {
// Iterate over all keys looking for product arrays
for (const key of Object.keys(data)) {
const val = data[key];
if (Array.isArray(val)) {
for (const item of val) {
if (item && typeof item === "object" && (item.title || item.name) && item.url) {
results.push({
title: item.title || item.name,
url: item.url,
price: extractPrice(item.price),
sku: item.sku,
});
}
}
}
}
}
return results;
} catch {
// Not JSON — parse as HTML
const results: SearchResult[] = [];
const linkRegex = /href="([^"]*\.html)"[^>]*>([^<]{3,})<\/a>/gi;
let match;
while ((match = linkRegex.exec(text)) !== null) {
const pUrl = match[1];
const title = match[2].trim();
if (title.length < 5) continue;
// Look for price near this match
const context = text.slice(match.index, match.index + 500);
const priceMatch = context.match(/(?:€|EUR)\s*([\d.,]+)/i) || context.match(/([\d.,]+)\s*(?:€|EUR)/i);
results.push({
title,
url: pUrl.startsWith("http") ? pUrl : BASE + pUrl,
price: priceMatch ? priceMatch[1].replace(",", ".") : undefined,
});
}
return results;
}
}
export async function scrapeFlexoptixCatalog(): Promise<void> {
console.log("=== Flexoptix Catalog Scraper Starting ===\n");
const vendorId = await ensureVendor("Flexoptix", "reseller", "https://www.flexoptix.net", "https://www.flexoptix.net/en/");
const allProducts = new Map<string, Product>();
let priceUpdates = 0;
for (const sq of SEARCH_QUERIES) {
console.log(` Searching: "${sq.query}"`);
try {
const results = await searchProducts(sq.query);
let newCount = 0;
for (const r of results) {
// Skip non-product results
if (!r.url || !r.title) continue;
const key = r.url;
if (allProducts.has(key)) continue;
const name = r.title;
const formFactor = inferFormFactor(name, sq.formFactor);
const gbps = inferSpeed(name, sq.speedGbps);
const reach = detectReach(name);
const price = r.price ? parseFloat(r.price.replace(",", ".")) : undefined;
allProducts.set(key, {
name,
partNumber: r.sku || name.replace(/\s+/g, "-").slice(0, 80),
url: r.url.startsWith("http") ? r.url : BASE + r.url,
price: price && price > 0 && price < 100000 ? price : undefined,
currency: price ? "EUR" : undefined,
formFactor,
speed: speedLabel(gbps),
speedGbps: gbps,
reachLabel: reach?.label,
reachMeters: reach?.meters,
fiberType: detectFiber(name),
wavelength: detectWavelength(name),
});
newCount++;
}
if (newCount > 0) console.log(` +${newCount} new (${results.length} results)`);
} catch (err) {
console.warn(` Search failed: ${(err as Error).message.slice(0, 60)}`);
}
await sleep(1000);
}
// ── Phase 2: GraphQL full catalog enumeration ──
console.log("\n--- Phase 2: GraphQL Catalog Enumeration ---\n");
const GRAPHQL_URL = `${BASE}/graphql`;
const GRAPHQL_QUERIES = [
{ search: "SFP+", defaultFF: "SFP+", defaultGbps: 10 },
{ search: "SFP28", defaultFF: "SFP28", defaultGbps: 25 },
{ search: "QSFP+", defaultFF: "QSFP+", defaultGbps: 40 },
{ search: "QSFP28", defaultFF: "QSFP28", defaultGbps: 100 },
{ search: "QSFP-DD", defaultFF: "QSFP-DD", defaultGbps: 400 },
{ search: "QSFP56", defaultFF: "QSFP56", defaultGbps: 200 },
{ search: "OSFP", defaultFF: "OSFP", defaultGbps: 400 },
{ search: "XFP", defaultFF: "XFP", defaultGbps: 10 },
{ search: "CFP2", defaultFF: "CFP2", defaultGbps: 100 },
{ search: "GBIC", defaultFF: "GBIC", defaultGbps: 1 },
{ search: "SFP56", defaultFF: "SFP56", defaultGbps: 50 },
{ search: "DAC", defaultFF: "SFP+", defaultGbps: 10 },
{ search: "AOC", defaultFF: "SFP+", defaultGbps: 10 },
{ search: "breakout", defaultFF: "QSFP28", defaultGbps: 100 },
{ search: "BiDi", defaultFF: "SFP", defaultGbps: 1 },
{ search: "CWDM", defaultFF: "SFP", defaultGbps: 1 },
{ search: "DWDM", defaultFF: "SFP", defaultGbps: 1 },
{ search: "coherent", defaultFF: "QSFP-DD", defaultGbps: 400 },
{ search: "800G", defaultFF: "OSFP", defaultGbps: 800 },
{ search: "1.6T", defaultFF: "OSFP", defaultGbps: 1600 },
];
for (const gq of GRAPHQL_QUERIES) {
let page = 1;
const pageSize = 20;
let totalFetched = 0;
while (true) {
try {
const query = `{
products(search: "${gq.search}", pageSize: ${pageSize}, currentPage: ${page}) {
total_count
items {
name
sku
url_key
price_range {
minimum_price {
final_price { value currency }
}
}
}
}
}`;
const resp = await fetch(GRAPHQL_URL, {
method: "POST",
headers: { "Content-Type": "application/json", ...HEADERS },
body: JSON.stringify({ query }),
signal: AbortSignal.timeout(20000),
});
if (!resp.ok) break;
const data = await resp.json() as {
data?: {
products?: {
total_count: number;
items: Array<{
name: string;
sku: string;
url_key: string;
price_range?: {
minimum_price?: {
final_price?: { value: number; currency: string };
};
};
}>;
};
};
};
const products = data.data?.products;
if (!products || products.items.length === 0) break;
let newCount = 0;
for (const item of products.items) {
if (!item.name || !item.sku) continue;
// Skip non-transceiver products (trays, tools, accessories)
const lower = item.name.toLowerCase();
if (lower.includes("tray") || lower.includes("tool") || lower.includes("loopback")
|| lower.includes("cleaning") || lower.includes("sticker") || lower.includes("flexbox")
|| lower.includes("adapter") || lower.includes("attenuator") || lower.includes("coupler")) continue;
const url = `${BASE}/en/${item.url_key}.html`;
if (allProducts.has(url)) continue;
const formFactor = inferFormFactor(item.name, gq.defaultFF);
const gbps = inferSpeed(item.name, gq.defaultGbps);
const reach = detectReach(item.name);
const price = item.price_range?.minimum_price?.final_price?.value;
allProducts.set(url, {
name: item.name,
partNumber: item.sku,
url,
price: price && price > 0 && price < 100000 ? price : undefined,
currency: item.price_range?.minimum_price?.final_price?.currency || "EUR",
formFactor,
speed: speedLabel(gbps),
speedGbps: gbps,
reachLabel: reach?.label,
reachMeters: reach?.meters,
fiberType: detectFiber(item.name),
wavelength: detectWavelength(item.name),
});
newCount++;
}
totalFetched += products.items.length;
if (newCount > 0) console.log(` GraphQL "${gq.search}" p${page}: +${newCount} new (${products.items.length} items, ${products.total_count} total)`);
// Stop if we've fetched all pages or reached a reasonable limit
if (totalFetched >= products.total_count || totalFetched >= 200 || page >= 10) break;
page++;
await sleep(800);
} catch (err) {
console.warn(` GraphQL "${gq.search}" p${page} failed: ${(err as Error).message.slice(0, 60)}`);
break;
}
}
await sleep(500);
}
console.log(`\nTotal unique products after GraphQL: ${allProducts.size}`);
console.log("Writing to database...\n");
// Write all products to DB
for (const product of allProducts.values()) {
try {
const txId = await findOrCreateScrapedTransceiver({
partNumber: product.partNumber,
vendorId,
formFactor: product.formFactor,
speedGbps: product.speedGbps,
speed: product.speed,
reachMeters: product.reachMeters,
reachLabel: product.reachLabel,
fiberType: product.fiberType,
wavelengths: product.wavelength,
category: "DataCenter",
});
if (product.price && product.price > 0) {
const hash = contentHash(JSON.stringify({ price: product.price, part: product.partNumber }));
const updated = await upsertPriceObservation({
transceiverId: txId,
sourceVendorId: vendorId,
price: product.price,
currency: product.currency || "EUR",
stockLevel: "in_stock",
url: product.url,
contentHash: hash,
});
if (updated) priceUpdates++;
}
} catch (err) {
console.warn(` DB error: ${(err as Error).message.slice(0, 80)}`);
}
}
console.log(`\n=== Flexoptix Catalog Complete: ${allProducts.size} products, ${priceUpdates} prices ===`);
}
if (require.main === module) {
scrapeFlexoptixCatalog()
.then(() => pool.end())
.catch((err) => { console.error("Fatal:", err); pool.end(); process.exit(1); });
}

View File

@ -0,0 +1,433 @@
/**
* Flexoptix Supported Vendors Complete Seed Data
*
* ALL ~300 vendors that Flexoptix programs transceivers for.
* Source: https://www.flexoptix.net/en/supported-vendors/
*
* Categories:
* - network_switching Switches, routers (Cisco, Arista, Juniper, etc.)
* - network_security Firewalls, UTM (Fortinet, Palo Alto, Check Point)
* - network_wireless WiFi, wireless (Aruba, Cambium, Ubiquiti)
* - network_carrier Telecom/SP equipment (Nokia, Ericsson, Ciena)
* - network_sdwan SD-WAN appliances (Versa, Cisco Viptela)
* - storage SAN/NAS (NetApp, Pure, Dell EMC, QNAP)
* - server Server/compute (Supermicro, Lenovo, Dell)
* - broadcast_av Broadcast/AV/video (Evertz, Grass Valley, Riedel)
* - industrial Industrial networking (Hirschmann, Moxa, Phoenix)
* - monitoring Network monitoring/TAP (Gigamon, Keysight, VIAVI)
* - load_balancer ADC/load balancers (F5, Citrix, Kemp)
* - nic NICs/HBAs (Intel, Broadcom, Chelsio, Solarflare)
* - whitebox Whitebox/ODM (Edgecore, Quanta, Asterfusion)
* - other Other categories
*/
import { pool, ensureVendor } from "../utils/db";
interface FlexoptixVendor {
name: string;
category: string;
website?: string;
vendorType: "oem" | "manufacturer" | "distributor" | "compatible";
notes?: string;
}
export const FLEXOPTIX_VENDORS: FlexoptixVendor[] = [
// ═══════════════════════════════════════════════════════
// NETWORK — SWITCHING & ROUTING
// ═══════════════════════════════════════════════════════
{ name: "Cisco Systems", category: "network_switching", website: "https://www.cisco.com", vendorType: "oem" },
{ name: "Cisco Mod.", category: "network_switching", website: "https://www.cisco.com", vendorType: "oem", notes: "Cisco modules/line cards" },
{ name: "Cisco OneAccess (Ekinops)", category: "network_switching", website: "https://www.ekinops.com", vendorType: "oem" },
{ name: "Cisco Siemon", category: "network_switching", vendorType: "oem" },
{ name: "Cisco Systems (ex. Linksys)", category: "network_switching", website: "https://www.cisco.com", vendorType: "oem" },
{ name: "Cisco Systems (ex. Viptela)", category: "network_sdwan", website: "https://www.cisco.com", vendorType: "oem" },
{ name: "Cisco (ex. Meraki)", category: "network_wireless", website: "https://meraki.cisco.com", vendorType: "oem" },
{ name: "Arista", category: "network_switching", website: "https://www.arista.com", vendorType: "oem" },
{ name: "Arista 3rd_Party", category: "network_switching", website: "https://www.arista.com", vendorType: "oem", notes: "Third-party optics on Arista" },
{ name: "Juniper", category: "network_switching", website: "https://www.juniper.net", vendorType: "oem" },
{ name: "Extreme", category: "network_switching", website: "https://www.extremenetworks.com", vendorType: "oem" },
{ name: "Extreme (ex. Enterasys)", category: "network_switching", website: "https://www.extremenetworks.com", vendorType: "oem" },
{ name: "Huawei", category: "network_switching", website: "https://e.huawei.com", vendorType: "oem" },
{ name: "Nokia (ex. Alcatel-Lucent)", category: "network_carrier", website: "https://www.nokia.com", vendorType: "oem" },
{ name: "Nokia Networks", category: "network_carrier", website: "https://www.nokia.com", vendorType: "oem" },
{ name: "Nokia Siemens Networks", category: "network_carrier", website: "https://www.nokia.com", vendorType: "oem" },
{ name: "Nokia-Siemens (ex. Atrica)", category: "network_carrier", website: "https://www.nokia.com", vendorType: "oem" },
{ name: "Alcatel-Lucent Ent.", category: "network_switching", website: "https://www.al-enterprise.com", vendorType: "oem" },
{ name: "Dell", category: "network_switching", website: "https://www.dell.com", vendorType: "oem" },
{ name: "Dell (ex. Force10)", category: "network_switching", website: "https://www.dell.com", vendorType: "oem" },
{ name: "Lenovo", category: "server", website: "https://www.lenovo.com", vendorType: "oem" },
{ name: "Lenovo (ex. Blade Network)", category: "network_switching", website: "https://www.lenovo.com", vendorType: "oem" },
{ name: "Lenovo (ex. IBM)", category: "server", website: "https://www.lenovo.com", vendorType: "oem" },
{ name: "Nvidia (ex. Mellanox)", category: "network_switching", website: "https://www.nvidia.com/networking", vendorType: "oem" },
{ name: "H3C (ex. 3COM)", category: "network_switching", website: "https://www.h3c.com", vendorType: "oem" },
{ name: "HP H3C", category: "network_switching", vendorType: "oem" },
{ name: "Aruba Networks", category: "network_wireless", website: "https://www.arubanetworks.com", vendorType: "oem" },
{ name: "Aruba Networks(ex. HP Network)", category: "network_switching", website: "https://www.arubanetworks.com", vendorType: "oem" },
{ name: "Brocade", category: "network_switching", website: "https://www.broadcom.com", vendorType: "oem" },
{ name: "Brocade / Ruckus", category: "network_wireless", vendorType: "oem" },
{ name: "Force10 / Brocade", category: "network_switching", vendorType: "oem" },
{ name: "Force10 / Juniper", category: "network_switching", vendorType: "oem" },
{ name: "Nortel / Juniper / NSN", category: "network_switching", vendorType: "oem" },
{ name: "Avaya (ex. Nortel)", category: "network_switching", website: "https://www.avaya.com", vendorType: "oem" },
{ name: "MikroTik", category: "network_switching", website: "https://mikrotik.com", vendorType: "oem" },
{ name: "Netgear", category: "network_switching", website: "https://www.netgear.com", vendorType: "oem" },
{ name: "D-LINK", category: "network_switching", website: "https://www.dlink.com", vendorType: "oem" },
{ name: "TP-LINK", category: "network_switching", website: "https://www.tp-link.com", vendorType: "oem" },
{ name: "Zyxel", category: "network_switching", website: "https://www.zyxel.com", vendorType: "oem" },
{ name: "Allied Telesis", category: "network_switching", website: "https://www.alliedtelesis.com", vendorType: "oem" },
{ name: "Planet", category: "network_switching", website: "https://www.planet.com.tw", vendorType: "oem" },
{ name: "LANCOM", category: "network_switching", website: "https://www.lancom-systems.de", vendorType: "oem" },
{ name: "Ruijie Networks", category: "network_switching", website: "https://www.ruijienetworks.com", vendorType: "oem" },
{ name: "Ubiquiti Networks", category: "network_wireless", website: "https://www.ui.com", vendorType: "oem" },
{ name: "Cambium Networks", category: "network_wireless", website: "https://www.cambiumnetworks.com", vendorType: "oem" },
{ name: "Aerohive Networks", category: "network_wireless", vendorType: "oem" },
{ name: "Peplink", category: "network_sdwan", website: "https://www.peplink.com", vendorType: "oem" },
{ name: "DrayTek", category: "network_switching", website: "https://www.draytek.com", vendorType: "oem" },
{ name: "LevelOne", category: "network_switching", website: "https://www.level1.com", vendorType: "oem" },
{ name: "TRENDnet", category: "network_switching", website: "https://www.trendnet.com", vendorType: "oem" },
{ name: "Allnet", category: "network_switching", website: "https://www.allnet.de", vendorType: "oem" },
{ name: "Longshine", category: "network_switching", vendorType: "oem" },
{ name: "KTI Networks", category: "network_switching", vendorType: "oem" },
{ name: "Volktek", category: "network_switching", vendorType: "oem" },
{ name: "Araknis Networks", category: "network_switching", vendorType: "oem" },
{ name: "Netonix", category: "network_switching", vendorType: "oem" },
{ name: "ReadyLinks", category: "network_switching", vendorType: "oem" },
{ name: "Riverstone", category: "network_switching", vendorType: "oem" },
{ name: "Marconi", category: "network_switching", vendorType: "oem" },
{ name: "Star", category: "network_switching", vendorType: "oem" },
{ name: "pakedge", category: "network_switching", vendorType: "oem" },
{ name: "Wi-Tek", category: "network_switching", vendorType: "oem" },
{ name: "Waystream (ex. Packetfront)", category: "network_switching", website: "https://www.waystream.com", vendorType: "oem" },
{ name: "Big Switch Networks", category: "network_switching", vendorType: "oem" },
{ name: "Cumulus Networks", category: "network_switching", website: "https://www.nvidia.com/networking", vendorType: "oem" },
{ name: "IP Infusion", category: "network_switching", website: "https://www.ipinfusion.com", vendorType: "oem" },
{ name: "NoviFlow", category: "network_switching", website: "https://noviflow.com", vendorType: "oem" },
{ name: "Bintec", category: "network_switching", vendorType: "oem" },
{ name: "Fujitsu", category: "network_switching", website: "https://www.fujitsu.com", vendorType: "oem" },
{ name: "NEC", category: "network_switching", website: "https://www.nec.com", vendorType: "oem" },
{ name: "ZTE", category: "network_carrier", website: "https://www.zte.com.cn", vendorType: "oem" },
// ═══════════════════════════════════════════════════════
// WHITEBOX / ODM
// ═══════════════════════════════════════════════════════
{ name: "Edge Core", category: "whitebox", website: "https://www.edge-core.com", vendorType: "manufacturer" },
{ name: "Accton", category: "whitebox", website: "https://www.accton.com", vendorType: "manufacturer" },
{ name: "Quanta", category: "whitebox", website: "https://www.qct.io", vendorType: "manufacturer" },
{ name: "Asterfusion", category: "whitebox", website: "https://www.asterfusion.com", vendorType: "manufacturer" },
{ name: "Centec", category: "whitebox", website: "https://www.centecnetworks.com", vendorType: "manufacturer" },
{ name: "Penguin Computing", category: "whitebox", website: "https://www.penguincomputing.com", vendorType: "manufacturer" },
{ name: "SolidRun", category: "whitebox", website: "https://www.solid-run.com", vendorType: "manufacturer" },
// ═══════════════════════════════════════════════════════
// NETWORK SECURITY — Firewalls, UTM, VPN
// ═══════════════════════════════════════════════════════
{ name: "Fortinet", category: "network_security", website: "https://www.fortinet.com", vendorType: "oem" },
{ name: "Palo Alto Networks", category: "network_security", website: "https://www.paloaltonetworks.com", vendorType: "oem" },
{ name: "Check Point", category: "network_security", website: "https://www.checkpoint.com", vendorType: "oem" },
{ name: "SonicWALL", category: "network_security", website: "https://www.sonicwall.com", vendorType: "oem" },
{ name: "WatchGuard", category: "network_security", website: "https://www.watchguard.com", vendorType: "oem" },
{ name: "Barracuda Networks", category: "network_security", website: "https://www.barracuda.com", vendorType: "oem" },
{ name: "Sophos", category: "network_security", website: "https://www.sophos.com", vendorType: "oem" },
{ name: "Fireeye", category: "network_security", website: "https://www.trellix.com", vendorType: "oem" },
{ name: "Bluecoat", category: "network_security", vendorType: "oem", notes: "Now Symantec/Broadcom" },
{ name: "Forcepoint", category: "network_security", website: "https://www.forcepoint.com", vendorType: "oem" },
{ name: "Hillstone", category: "network_security", website: "https://www.hillstonenet.com", vendorType: "oem" },
{ name: "Clavister", category: "network_security", website: "https://www.clavister.com", vendorType: "oem" },
{ name: "Stormshield", category: "network_security", website: "https://www.stormshield.com", vendorType: "oem" },
{ name: "secunet", category: "network_security", website: "https://www.secunet.com", vendorType: "oem" },
{ name: "Securepoint", category: "network_security", website: "https://www.securepoint.de", vendorType: "oem" },
{ name: "Senetas", category: "network_security", website: "https://www.senetas.com", vendorType: "oem" },
{ name: "Thales (ex. Gemalto)", category: "network_security", website: "https://www.thalesgroup.com", vendorType: "oem" },
{ name: "McAfee", category: "network_security", vendorType: "oem" },
{ name: "Symantec", category: "network_security", vendorType: "oem" },
{ name: "PulseSecure", category: "network_security", website: "https://www.ivanti.com", vendorType: "oem" },
{ name: "Nozomi Networks", category: "network_security", website: "https://www.nozominetworks.com", vendorType: "oem" },
{ name: "Trend Micro (ex. TippingPoint)", category: "network_security", website: "https://www.trendmicro.com", vendorType: "oem" },
{ name: "Netgate", category: "network_security", website: "https://www.netgate.com", vendorType: "oem", notes: "pfSense" },
{ name: "OPNsense", category: "network_security", website: "https://opnsense.org", vendorType: "oem" },
{ name: "Deciso", category: "network_security", website: "https://www.deciso.com", vendorType: "oem", notes: "OPNsense hardware" },
{ name: "Turris Omnia", category: "network_security", website: "https://www.turris.com", vendorType: "oem" },
{ name: "Sandvine", category: "network_security", website: "https://www.sandvine.com", vendorType: "oem" },
// ═══════════════════════════════════════════════════════
// CARRIER / TELECOM / OPTICAL TRANSPORT
// ═══════════════════════════════════════════════════════
{ name: "Ciena", category: "network_carrier", website: "https://www.ciena.com", vendorType: "oem" },
{ name: "Ciena (ex. Nortel)", category: "network_carrier", website: "https://www.ciena.com", vendorType: "oem" },
{ name: "ADVA", category: "network_carrier", website: "https://www.adtran.com", vendorType: "oem", notes: "Now part of ADTRAN" },
{ name: "ADTRAN", category: "network_carrier", website: "https://www.adtran.com", vendorType: "oem" },
{ name: "Infinera", category: "network_carrier", website: "https://www.infinera.com", vendorType: "oem" },
{ name: "Infinera (ex. Coriant)", category: "network_carrier", website: "https://www.infinera.com", vendorType: "oem" },
{ name: "Infinera (ex. Transmode)", category: "network_carrier", website: "https://www.infinera.com", vendorType: "oem" },
{ name: "Coriant (ex. Tellabs)", category: "network_carrier", vendorType: "oem" },
{ name: "Ericsson", category: "network_carrier", website: "https://www.ericsson.com", vendorType: "oem" },
{ name: "Ceragon", category: "network_carrier", website: "https://www.ceragon.com", vendorType: "oem" },
{ name: "Ribbon (ex. ECI)", category: "network_carrier", website: "https://www.ribboncommunications.com", vendorType: "oem" },
{ name: "Ribbon (ex. Sonus)", category: "network_carrier", website: "https://www.ribboncommunications.com", vendorType: "oem" },
{ name: "Calix", category: "network_carrier", website: "https://www.calix.com", vendorType: "oem" },
{ name: "RAD", category: "network_carrier", website: "https://www.rad.com", vendorType: "oem" },
{ name: "Raisecom", category: "network_carrier", website: "https://www.raisecom.com", vendorType: "oem" },
{ name: "Packetlight", category: "network_carrier", website: "https://www.packetlight.com", vendorType: "oem" },
{ name: "Telco Systems", category: "network_carrier", website: "https://www.telco.com", vendorType: "oem" },
{ name: "TelcoBridges", category: "network_carrier", website: "https://www.telcobridges.com", vendorType: "oem" },
{ name: "Overture", category: "network_carrier", vendorType: "oem" },
{ name: "Cyan", category: "network_carrier", vendorType: "oem" },
{ name: "Orckit-Corrigent", category: "network_carrier", vendorType: "oem" },
{ name: "Redback", category: "network_carrier", vendorType: "oem" },
{ name: "Sorrento Networks", category: "network_carrier", vendorType: "oem" },
{ name: "Xtera Communications", category: "network_carrier", vendorType: "oem" },
{ name: "Zhone", category: "network_carrier", vendorType: "oem" },
{ name: "PBN", category: "network_carrier", vendorType: "oem" },
{ name: "Casa Systems", category: "network_carrier", website: "https://www.casa-systems.com", vendorType: "oem" },
{ name: "Genband", category: "network_carrier", vendorType: "oem" },
{ name: "Actelis", category: "network_carrier", vendorType: "oem" },
{ name: "Keymile", category: "network_carrier", vendorType: "oem" },
{ name: "Aviat Networks", category: "network_carrier", website: "https://www.aviatnetworks.com", vendorType: "oem" },
{ name: "SAF Tehnika", category: "network_carrier", website: "https://www.saftehnika.com", vendorType: "oem" },
{ name: "Siklu", category: "network_carrier", website: "https://www.siklu.com", vendorType: "oem" },
{ name: "Net Insight", category: "network_carrier", website: "https://www.netinsight.net", vendorType: "oem" },
{ name: "Nextivity", category: "network_carrier", vendorType: "oem" },
{ name: "AudioCodes", category: "network_carrier", website: "https://www.audiocodes.com", vendorType: "oem" },
{ name: "SagemCOM", category: "network_carrier", website: "https://www.sagemcom.com", vendorType: "oem" },
{ name: "Freebox", category: "network_carrier", vendorType: "oem", notes: "Free (Iliad) ISP hardware" },
{ name: "NTT Devices", category: "network_carrier", vendorType: "oem" },
{ name: "Teleste", category: "network_carrier", website: "https://www.teleste.com", vendorType: "oem" },
{ name: "Teltonika", category: "network_carrier", website: "https://teltonika-networks.com", vendorType: "oem" },
{ name: "Accedian Networks", category: "network_carrier", website: "https://www.accedian.com", vendorType: "oem" },
{ name: "OTN Systems", category: "network_carrier", website: "https://www.otnsystems.com", vendorType: "oem" },
{ name: "FibroLAN", category: "network_carrier", website: "https://www.fibrolan.com", vendorType: "oem" },
{ name: "Optonet", category: "network_carrier", vendorType: "oem" },
{ name: "ComNet", category: "network_carrier", website: "https://www.comnet.net", vendorType: "oem" },
// ═══════════════════════════════════════════════════════
// SD-WAN
// ═══════════════════════════════════════════════════════
{ name: "Versa Networks", category: "network_sdwan", website: "https://www.versa-networks.com", vendorType: "oem" },
// ═══════════════════════════════════════════════════════
// STORAGE
// ═══════════════════════════════════════════════════════
{ name: "NetApp", category: "storage", website: "https://www.netapp.com", vendorType: "oem" },
{ name: "Pure Storage", category: "storage", website: "https://www.purestorage.com", vendorType: "oem" },
{ name: "EMC", category: "storage", website: "https://www.dell.com/emc", vendorType: "oem" },
{ name: "HP Storage (B-Series)", category: "storage", vendorType: "oem" },
{ name: "HP Storage (C-Series)", category: "storage", vendorType: "oem" },
{ name: "HP Storage (H-Series)", category: "storage", vendorType: "oem" },
{ name: "IBM Storage", category: "storage", website: "https://www.ibm.com/storage", vendorType: "oem" },
{ name: "Nimblestorage", category: "storage", vendorType: "oem", notes: "Now HPE" },
{ name: "Infortrend", category: "storage", website: "https://www.infortrend.com", vendorType: "oem" },
{ name: "QNAP", category: "storage", website: "https://www.qnap.com", vendorType: "oem" },
{ name: "QSAN", category: "storage", website: "https://www.qsan.com", vendorType: "oem" },
{ name: "Synology", category: "storage", website: "https://www.synology.com", vendorType: "oem" },
{ name: "Oracle", category: "storage", website: "https://www.oracle.com", vendorType: "oem" },
{ name: "Nutanix", category: "storage", website: "https://www.nutanix.com", vendorType: "oem" },
{ name: "Rubirk", category: "storage", website: "https://www.rubrik.com", vendorType: "oem" },
{ name: "Rubrik", category: "storage", website: "https://www.rubrik.com", vendorType: "oem" },
{ name: "FAST LTA", category: "storage", website: "https://www.fast-lta.de", vendorType: "oem" },
// ═══════════════════════════════════════════════════════
// SERVERS / COMPUTE
// ═══════════════════════════════════════════════════════
{ name: "Supermicro", category: "server", website: "https://www.supermicro.com", vendorType: "oem" },
{ name: "Sun", category: "server", vendorType: "oem", notes: "Now Oracle" },
{ name: "Hitachi", category: "server", website: "https://www.hitachivantara.com", vendorType: "oem" },
// ═══════════════════════════════════════════════════════
// LOAD BALANCERS / ADC
// ═══════════════════════════════════════════════════════
{ name: "F5 Networks", category: "load_balancer", website: "https://www.f5.com", vendorType: "oem" },
{ name: "Citrix Netscaler", category: "load_balancer", website: "https://www.cloud.com", vendorType: "oem" },
{ name: "Kemp Technologies", category: "load_balancer", website: "https://www.progress.com/kemp", vendorType: "oem" },
{ name: "A10networks", category: "load_balancer", website: "https://www.a10networks.com", vendorType: "oem" },
{ name: "Radware", category: "load_balancer", website: "https://www.radware.com", vendorType: "oem" },
{ name: "Arbor Networks", category: "load_balancer", vendorType: "oem", notes: "DDoS protection, now Netscout" },
// ═══════════════════════════════════════════════════════
// NICs / HBAs / NETWORK ADAPTERS
// ═══════════════════════════════════════════════════════
{ name: "Intel", category: "nic", website: "https://www.intel.com", vendorType: "manufacturer" },
{ name: "Intel Netscaler", category: "nic", vendorType: "manufacturer" },
{ name: "Broadcom", category: "nic", website: "https://www.broadcom.com", vendorType: "manufacturer" },
{ name: "Chelsio", category: "nic", website: "https://www.chelsio.com", vendorType: "manufacturer" },
{ name: "Solarflare", category: "nic", website: "https://www.xilinx.com", vendorType: "manufacturer", notes: "Now AMD/Xilinx" },
{ name: "QLogic", category: "nic", vendorType: "manufacturer", notes: "Now Marvell" },
{ name: "Emulex", category: "nic", vendorType: "manufacturer", notes: "Now Broadcom" },
{ name: "Myricom", category: "nic", vendorType: "manufacturer" },
{ name: "ATTO", category: "nic", website: "https://www.atto.com", vendorType: "manufacturer" },
{ name: "napatech", category: "nic", website: "https://www.napatech.com", vendorType: "manufacturer" },
{ name: "Netxen", category: "nic", vendorType: "manufacturer" },
{ name: "Xilinx", category: "nic", website: "https://www.amd.com/xilinx", vendorType: "manufacturer" },
{ name: "Bittware", category: "nic", website: "https://www.bittware.com", vendorType: "manufacturer" },
{ name: "Winyao", category: "nic", vendorType: "manufacturer" },
// ═══════════════════════════════════════════════════════
// MONITORING / TAP / TEST & MEASUREMENT
// ═══════════════════════════════════════════════════════
{ name: "Gigamon", category: "monitoring", website: "https://www.gigamon.com", vendorType: "oem" },
{ name: "Keysight (ex. Ixia)", category: "monitoring", website: "https://www.keysight.com", vendorType: "oem" },
{ name: "Viavi (ex. JDSU)", category: "monitoring", website: "https://www.viavisolutions.com", vendorType: "oem" },
{ name: "Spirent", category: "monitoring", website: "https://www.spirent.com", vendorType: "oem" },
{ name: "NetOptics", category: "monitoring", vendorType: "oem" },
{ name: "Netscout", category: "monitoring", website: "https://www.netscout.com", vendorType: "oem" },
{ name: "Fluke Networks", category: "monitoring", website: "https://www.flukenetworks.com", vendorType: "oem" },
{ name: "Garland Technology", category: "monitoring", website: "https://www.garlandtechnology.com", vendorType: "oem" },
{ name: "ProfiTap", category: "monitoring", website: "https://www.profitap.com", vendorType: "oem" },
{ name: "VSS monitoring", category: "monitoring", vendorType: "oem" },
{ name: "VeEX", category: "monitoring", website: "https://www.veexinc.com", vendorType: "oem" },
{ name: "Trend Networks", category: "monitoring", website: "https://www.trend-networks.com", vendorType: "oem" },
{ name: "xtramus", category: "monitoring", vendorType: "oem" },
{ name: "Wildpackets", category: "monitoring", vendorType: "oem" },
// ═══════════════════════════════════════════════════════
// BROADCAST / AV / VIDEO
// ═══════════════════════════════════════════════════════
{ name: "Evertz", category: "broadcast_av", website: "https://www.evertz.com", vendorType: "oem" },
{ name: "Grassvalley", category: "broadcast_av", website: "https://www.grassvalley.com", vendorType: "oem" },
{ name: "Riedel", category: "broadcast_av", website: "https://www.riedel.net", vendorType: "oem" },
{ name: "Blackmagic Design", category: "broadcast_av", website: "https://www.blackmagicdesign.com", vendorType: "oem" },
{ name: "EVS", category: "broadcast_av", website: "https://www.evs.com", vendorType: "oem" },
{ name: "Nevion", category: "broadcast_av", website: "https://www.nevion.com", vendorType: "oem" },
{ name: "LAWO", category: "broadcast_av", website: "https://www.lawo.com", vendorType: "oem" },
{ name: "Harmonic", category: "broadcast_av", website: "https://www.harmonicinc.com", vendorType: "oem" },
{ name: "Clear-Com", category: "broadcast_av", website: "https://www.clearcom.com", vendorType: "oem" },
{ name: "Barnfind", category: "broadcast_av", website: "https://www.barnfind.no", vendorType: "oem" },
{ name: "DirectOut", category: "broadcast_av", website: "https://www.directout.eu", vendorType: "oem" },
{ name: "Ferrofish", category: "broadcast_av", website: "https://www.ferrofish.com", vendorType: "oem" },
{ name: "Luminex", category: "broadcast_av", website: "https://www.luminex.be", vendorType: "oem" },
{ name: "Lynx", category: "broadcast_av", vendorType: "oem" },
{ name: "ZeeVee", category: "broadcast_av", website: "https://www.zeevee.com", vendorType: "oem" },
{ name: "Sony", category: "broadcast_av", website: "https://pro.sony", vendorType: "oem" },
{ name: "Novastar", category: "broadcast_av", website: "https://www.novastar.tech", vendorType: "oem" },
{ name: "Matrox", category: "broadcast_av", website: "https://www.matrox.com", vendorType: "oem" },
{ name: "IDK Corporation", category: "broadcast_av", website: "https://www.idk.co.jp", vendorType: "oem" },
{ name: "Guntermann & Drunck", category: "broadcast_av", website: "https://www.gdsys.de", vendorType: "oem" },
{ name: "kvm-tec", category: "broadcast_av", website: "https://www.kvm-tec.com", vendorType: "oem" },
{ name: "FieldCast", category: "broadcast_av", website: "https://www.fieldcast.com", vendorType: "oem" },
{ name: "FiberPlex", category: "broadcast_av", website: "https://www.fiberplex.com", vendorType: "oem" },
{ name: "Yamaha", category: "broadcast_av", website: "https://www.yamaha.com", vendorType: "oem" },
// ═══════════════════════════════════════════════════════
// INDUSTRIAL NETWORKING
// ═══════════════════════════════════════════════════════
{ name: "Hirschmann", category: "industrial", website: "https://www.hirschmann.com", vendorType: "oem" },
{ name: "Moxa", category: "industrial", website: "https://www.moxa.com", vendorType: "oem" },
{ name: "Phoenix Contact", category: "industrial", website: "https://www.phoenixcontact.com", vendorType: "oem" },
{ name: "Siemens", category: "industrial", website: "https://www.siemens.com", vendorType: "oem" },
{ name: "Korenix", category: "industrial", website: "https://www.korenix.com", vendorType: "oem" },
{ name: "Westermo", category: "industrial", website: "https://www.westermo.com", vendorType: "oem" },
{ name: "Weidmüller", category: "industrial", website: "https://www.weidmueller.com", vendorType: "oem" },
{ name: "Rockwell", category: "industrial", website: "https://www.rockwellautomation.com", vendorType: "oem" },
{ name: "GE (General Electric)", category: "industrial", website: "https://www.ge.com", vendorType: "oem" },
{ name: "ABB", category: "industrial", website: "https://new.abb.com", vendorType: "oem" },
{ name: "Wago", category: "industrial", website: "https://www.wago.com", vendorType: "oem" },
{ name: "Redlion", category: "industrial", website: "https://www.redlion.net", vendorType: "oem" },
{ name: "Redlion (ex. Sixnet)", category: "industrial", website: "https://www.redlion.net", vendorType: "oem" },
{ name: "CTC Union", category: "industrial", website: "https://www.ctcu.com", vendorType: "oem" },
{ name: "Advantech", category: "industrial", website: "https://www.advantech.com", vendorType: "oem" },
{ name: "Advantech B+B SmartWorx", category: "industrial", website: "https://www.advantech.com", vendorType: "oem" },
{ name: "Microsens", category: "industrial", website: "https://www.microsens.com", vendorType: "oem" },
{ name: "eks Engel", category: "industrial", website: "https://www.eks-engel.de", vendorType: "oem" },
{ name: "barox", category: "industrial", website: "https://www.barox.de", vendorType: "oem" },
{ name: "Omnitron", category: "industrial", website: "https://www.omnitron-systems.com", vendorType: "oem" },
{ name: "Transition Networks", category: "industrial", website: "https://www.transition.com", vendorType: "oem" },
{ name: "Perle", category: "industrial", website: "https://www.perle.com", vendorType: "oem" },
{ name: "IMC", category: "industrial", vendorType: "oem" },
{ name: "Niveo", category: "industrial", vendorType: "oem" },
{ name: "Cerio", category: "industrial", vendorType: "oem" },
// ═══════════════════════════════════════════════════════
// SURVEILLANCE / IP CAMERAS
// ═══════════════════════════════════════════════════════
{ name: "Axis", category: "surveillance", website: "https://www.axis.com", vendorType: "oem" },
{ name: "Dahua", category: "surveillance", website: "https://www.dahuasecurity.com", vendorType: "oem" },
{ name: "eneo", category: "surveillance", website: "https://www.eneo-security.com", vendorType: "oem" },
{ name: "AMG Systems", category: "surveillance", website: "https://www.amgsystems.com", vendorType: "oem" },
// ═══════════════════════════════════════════════════════
// MISC / CONNECTIVITY / ACCESSORIES
// ═══════════════════════════════════════════════════════
{ name: "Blackbox", category: "other", website: "https://www.blackbox.com", vendorType: "oem" },
{ name: "Commscope", category: "other", website: "https://www.commscope.com", vendorType: "oem" },
{ name: "CommScope (ex. Arris)", category: "other", website: "https://www.commscope.com", vendorType: "oem" },
{ name: "Nexans", category: "other", website: "https://www.nexans.com", vendorType: "oem" },
{ name: "LINDY", category: "other", website: "https://www.lindy.com", vendorType: "oem" },
{ name: "Delock", category: "other", website: "https://www.delock.de", vendorType: "oem" },
{ name: "Digitus", category: "other", website: "https://www.digitus.info", vendorType: "oem" },
{ name: "StarTech", category: "other", website: "https://www.startech.com", vendorType: "oem" },
{ name: "Ugreen", category: "other", website: "https://www.ugreen.com", vendorType: "oem" },
{ name: "Sonnet Technologies", category: "other", website: "https://www.sonnettech.com", vendorType: "oem" },
{ name: "ASUS", category: "other", website: "https://www.asus.com", vendorType: "oem" },
{ name: "AVM", category: "other", website: "https://www.avm.de", vendorType: "oem", notes: "FRITZ!Box" },
{ name: "Datto", category: "other", website: "https://www.datto.com", vendorType: "oem" },
{ name: "Lantronix", category: "other", website: "https://www.lantronix.com", vendorType: "oem" },
{ name: "OpenGear", category: "other", website: "https://www.opengear.com", vendorType: "oem" },
{ name: "ZPE Systems", category: "other", website: "https://www.zpesystems.com", vendorType: "oem" },
{ name: "Grandstream", category: "other", website: "https://www.grandstream.com", vendorType: "oem" },
{ name: "Riverbed", category: "other", website: "https://www.riverbed.com", vendorType: "oem" },
{ name: "Digicomm", category: "other", vendorType: "oem" },
{ name: "Avara", category: "other", vendorType: "oem" },
{ name: "Axon", category: "other", vendorType: "oem" },
{ name: "Motorola", category: "other", website: "https://www.motorola.com", vendorType: "oem" },
{ name: "MRV", category: "other", vendorType: "oem" },
{ name: "voleatech", category: "other", vendorType: "oem" },
{ name: "emcore", category: "other", vendorType: "oem" },
{ name: "Motorola", category: "other", vendorType: "oem" },
];
export async function seedFlexoptixVendors(): Promise<void> {
console.log(`\n=== Seeding ${FLEXOPTIX_VENDORS.length} Flexoptix Supported Vendors ===\n`);
let created = 0;
let updated = 0;
for (const v of FLEXOPTIX_VENDORS) {
try {
const existing = await pool.query(
`SELECT id FROM vendors WHERE name ILIKE $1`,
[v.name]
);
if (existing.rows.length > 0) {
// Mark as flexoptix-supported and update category
await pool.query(
`UPDATE vendors SET
flexoptix_supported = TRUE,
vendor_category = COALESCE($2, vendor_category),
website = COALESCE($3, website),
notes = COALESCE($4, notes),
updated_at = NOW()
WHERE id = $1`,
[existing.rows[0].id, v.category, v.website || null, v.notes || null]
);
updated++;
} else {
const slug = v.name.toLowerCase().replace(/[^a-z0-9]+/g, "-").replace(/-+$/, "");
await pool.query(
`INSERT INTO vendors (name, slug, type, website, vendor_category, flexoptix_supported, notes)
VALUES ($1, $2, $3, $4, $5, TRUE, $6)
ON CONFLICT (slug) DO UPDATE SET
flexoptix_supported = TRUE,
vendor_category = COALESCE($5, vendors.vendor_category),
updated_at = NOW()`,
[v.name, slug, v.vendorType, v.website || null, v.category, v.notes || null]
);
created++;
console.log(` + ${v.name} [${v.category}]`);
}
} catch (err) {
console.error(` ! Error: ${v.name}:`, err);
}
}
console.log(`\n Created: ${created}, Updated: ${updated}, Total: ${FLEXOPTIX_VENDORS.length}`);
// Summary by category
const cats: Record<string, number> = {};
for (const v of FLEXOPTIX_VENDORS) {
cats[v.category] = (cats[v.category] || 0) + 1;
}
console.log("\n Breakdown by category:");
for (const [cat, count] of Object.entries(cats).sort((a, b) => b[1] - a[1])) {
console.log(` ${cat}: ${count}`);
}
console.log();
}

View File

@ -0,0 +1,131 @@
/**
* Flexoptix Supported Vendors Scraper
*
* Scrapes flexoptix.net/en/supported-vendors/ for the full list of
* switch vendors Flexoptix supports. This is our own data no restrictions.
*
* Data goes into: switches (vendor names) + vendors table
* Also scrapes per-vendor pages for individual switch models when available.
*/
import { pool } from "../utils/db";
interface VendorEntry {
name: string;
url: string;
}
async function fetchVendorList(): Promise<VendorEntry[]> {
const resp = await fetch("https://www.flexoptix.net/en/supported-vendors/", {
headers: {
"User-Agent": "Mozilla/5.0 (compatible; TIP-Bot/1.0; internal)",
Accept: "text/html",
},
signal: AbortSignal.timeout(30000),
});
if (!resp.ok) throw new Error(`Flexoptix returned ${resp.status}`);
const html = await resp.text();
const vendors: VendorEntry[] = [];
// Parse vendor links from the supported-vendors page
// Pattern: href="...supported-vendors/index/name/VENDOR-compatible"
const regex = /href="(https?:\/\/www\.flexoptix\.net\/en\/supported-vendors\/index\/name\/([^"]+)-compatible)"/g;
let match;
while ((match = regex.exec(html)) !== null) {
const url = match[1]
.replace(/&#x3A;/g, ":")
.replace(/&#x2F;/g, "/")
.replace(/&#x2B;/g, "+")
.replace(/&#x28;/g, "(")
.replace(/&#x29;/g, ")");
const rawName = match[2]
.replace(/\+/g, " ")
.replace(/%20/g, " ")
.replace(/%28/g, "(")
.replace(/%29/g, ")");
vendors.push({ name: rawName, url });
}
// Also catch plain link text pattern
const altRegex = /class="[^"]*vendor[^"]*"[^>]*>\s*<a[^>]*href="([^"]+)"[^>]*>([^<]+)<\/a>/gi;
while ((match = altRegex.exec(html)) !== null) {
const url = match[1];
const name = match[2].trim();
if (name && !vendors.find((v) => v.name.toLowerCase() === name.toLowerCase())) {
vendors.push({ name, url });
}
}
// Deduplicate by name (case-insensitive)
const seen = new Set<string>();
return vendors.filter((v) => {
const key = v.name.toLowerCase();
if (seen.has(key)) return false;
seen.add(key);
return true;
});
}
function slugify(name: string): string {
return name
.toLowerCase()
.replace(/[^a-z0-9]+/g, "-")
.replace(/^-|-$/g, "");
}
async function upsertVendor(name: string): Promise<string> {
const slug = slugify(name);
const result = await pool.query(
`INSERT INTO vendors (name, slug, type, website)
VALUES ($1, $2, 'manufacturer', $3)
ON CONFLICT (name) DO UPDATE SET website = COALESCE(vendors.website, EXCLUDED.website)
RETURNING id`,
[name, slug, `https://www.flexoptix.net/en/supported-vendors/`]
);
return result.rows[0].id;
}
export async function scrapeFlexoptixVendors(): Promise<void> {
console.log("=== Flexoptix Vendor Scraper Starting ===\n");
const vendors = await fetchVendorList();
console.log(`Found ${vendors.length} supported vendors\n`);
let newVendors = 0;
let updatedVendors = 0;
for (const vendor of vendors) {
try {
const existing = await pool.query(
`SELECT id FROM vendors WHERE name ILIKE $1`,
[vendor.name]
);
await upsertVendor(vendor.name);
if (existing.rows.length === 0) {
newVendors++;
console.log(` + NEW: ${vendor.name}`);
} else {
updatedVendors++;
}
} catch (err) {
console.warn(` Error saving vendor ${vendor.name}:`, (err as Error).message);
}
}
console.log(`\nVendors: ${vendors.length} total, ${newVendors} new, ${updatedVendors} existing`);
console.log("=== Flexoptix Vendor Scraper Complete ===\n");
}
if (require.main === module) {
scrapeFlexoptixVendors()
.then(() => pool.end())
.catch((err) => {
console.error("Fatal:", err);
pool.end();
process.exit(1);
});
}

View File

@ -0,0 +1,234 @@
/**
* Fluxlight Scraper US-based compatible transceiver vendor
*
* www.fluxlight.com BigCommerce, server-rendered HTML with real prices.
* ~144+ products across 6 pages. Uses pagination via ?page=N.
*
* Rate limited: 1 req/2sec.
*/
import { pool, findOrCreateScrapedTransceiver, ensureVendor, upsertPriceObservation } from "../utils/db";
import { contentHash } from "../utils/hash";
const BASE = "https://fluxlight.com";
const CATALOG_PATH = "/transceivers/";
const HEADERS = {
"User-Agent": "Mozilla/5.0 (compatible; TIP-Bot/1.0; research)",
Accept: "text/html,application/xhtml+xml",
};
interface Product {
partNumber: string;
name: string;
url: string;
price?: number;
formFactor: string;
speed: string;
speedGbps: number;
reachLabel?: string;
reachMeters?: number;
fiberType?: string;
wavelength?: string;
}
function sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}
function detectFormFactor(text: string): { formFactor: string; speed: string; speedGbps: number } {
const lower = text.toLowerCase();
if (lower.includes("osfp") && !lower.includes("qsfp")) return { formFactor: "OSFP", speed: "400G", speedGbps: 400 };
if (lower.includes("qsfp-dd")) return { formFactor: "QSFP-DD", speed: "400G", speedGbps: 400 };
if (lower.includes("qsfp28")) return { formFactor: "QSFP28", speed: "100G", speedGbps: 100 };
if (lower.includes("qsfp+") || lower.includes("qsfp plus")) return { formFactor: "QSFP+", speed: "40G", speedGbps: 40 };
if (lower.includes("sfp56")) return { formFactor: "SFP56", speed: "50G", speedGbps: 50 };
if (lower.includes("sfp28") || lower.includes("25g")) return { formFactor: "SFP28", speed: "25G", speedGbps: 25 };
if (lower.includes("sfp+") || lower.includes("10gbase") || lower.includes("10g")) return { formFactor: "SFP+", speed: "10G", speedGbps: 10 };
if (lower.includes("xfp")) return { formFactor: "XFP", speed: "10G", speedGbps: 10 };
if (lower.includes("1000base") || lower.includes("1g")) return { formFactor: "SFP", speed: "1G", speedGbps: 1 };
if (lower.includes("sfp") && !lower.includes("qsfp")) return { formFactor: "SFP", speed: "1G", speedGbps: 1 };
return { formFactor: "SFP+", speed: "10G", speedGbps: 10 };
}
function detectReach(text: string): { label: string; meters: number } | undefined {
const patterns: [RegExp, string, number][] = [
[/\b80\s*km\b/i, "80km", 80000],
[/\b40\s*km\b/i, "40km", 40000],
[/\b20\s*km\b/i, "20km", 20000],
[/\b10\s*km\b/i, "10km", 10000],
[/\b2\s*km\b/i, "2km", 2000],
[/\b550\s*m\b/i, "550m", 550],
[/\b500\s*m\b/i, "500m", 500],
[/\b300\s*m\b/i, "300m", 300],
[/\b100\s*m\b/i, "100m", 100],
[/\bLR4\b/, "10km", 10000],
[/\bLR\b/, "10km", 10000],
[/\bER4?\b/, "40km", 40000],
[/\bZR4?\b/, "80km", 80000],
[/\bSR4?\b/, "300m", 300],
[/\bDR4?\b/, "500m", 500],
[/\bFR4?\b/, "2km", 2000],
];
for (const [regex, label, meters] of patterns) {
if (regex.test(text)) return { label, meters };
}
return undefined;
}
function detectFiber(text: string): string {
if (/single.?mode|smf|[^a-z]lx[^a-z]|[^a-z]lr[^a-z]|[^a-z]er[^a-z]|[^a-z]zr[^a-z]|bidi|cwdm|dwdm/i.test(text)) return "SMF";
if (/multi.?mode|mmf|[^a-z]sx[^a-z]|[^a-z]sr[^a-z]/i.test(text)) return "MMF";
if (/copper|dac|twinax|rj45|base-t|catx/i.test(text)) return "Copper";
return "";
}
function detectWavelength(text: string): string {
const match = text.match(/(\d{3,4})\s*nm/i);
if (match) return match[1];
return "";
}
function parseProductList(html: string): Product[] {
const products: Product[] = [];
// BigCommerce product card pattern: product link + price
// Pattern: <a href="https://www.fluxlight.com/PARTNUM-FL/">Product Name</a> ... $29.99
const productRegex = /href="(https?:\/\/(?:www\.)?fluxlight\.com\/[^"]*-FL\/)"[^>]*>\s*([^<]{10,})<\/a>/gi;
let match;
while ((match = productRegex.exec(html)) !== null) {
const url = match[1];
const name = match[2].trim();
if (name.length < 10 || name.length > 200) continue;
// Look for price in surrounding context
const context = html.slice(Math.max(0, match.index - 300), match.index + 600);
const priceMatch = context.match(/\$\s*([\d,]+\.?\d{0,2})/) || context.match(/data-product-price="([\d.]+)"/);
const price = priceMatch ? parseFloat(priceMatch[1].replace(",", "")) : undefined;
const ff = detectFormFactor(name);
const reach = detectReach(name);
const partNum = url.split("/").filter(Boolean).pop() || name.replace(/\s+/g, "-").slice(0, 80);
products.push({
partNumber: partNum,
name,
url,
price: price && price > 0 && price < 50000 ? price : undefined,
...ff,
reachLabel: reach?.label,
reachMeters: reach?.meters,
fiberType: detectFiber(name),
wavelength: detectWavelength(name),
});
}
// Fallback: broader link pattern
if (products.length === 0) {
const simpleRegex = /href="(https?:\/\/(?:www\.)?fluxlight\.com\/[^"]+)"[^>]*>([^<]{10,}(?:SFP|QSFP|XFP|Base)[^<]*)<\/a>/gi;
while ((match = simpleRegex.exec(html)) !== null) {
const url = match[1];
const name = match[2].trim();
if (products.find((p) => p.url === url)) continue;
const context = html.slice(match.index, match.index + 500);
const priceMatch = context.match(/\$\s*([\d,]+\.?\d{0,2})/);
const price = priceMatch ? parseFloat(priceMatch[1].replace(",", "")) : undefined;
const ff = detectFormFactor(name);
const reach = detectReach(name);
products.push({
partNumber: url.split("/").filter(Boolean).pop() || name.replace(/\s+/g, "-").slice(0, 80),
name, url,
price: price && price > 0 && price < 50000 ? price : undefined,
...ff,
reachLabel: reach?.label, reachMeters: reach?.meters,
fiberType: detectFiber(name), wavelength: detectWavelength(name),
});
}
}
const seen = new Set<string>();
return products.filter((p) => {
if (seen.has(p.url)) return false;
seen.add(p.url);
return true;
});
}
/** Detect max page by probing — page 1 may not have pagination links */
const MAX_PAGES = 6;
async function fetchPage(url: string): Promise<string> {
const resp = await fetch(url, { headers: HEADERS, signal: AbortSignal.timeout(30000) });
if (!resp.ok) throw new Error(`HTTP ${resp.status} for ${url}`);
return resp.text();
}
export async function scrapeFluxlight(): Promise<void> {
console.log("=== Fluxlight Scraper Starting ===\n");
const vendorId = await ensureVendor("Fluxlight", "compatible", "https://fluxlight.com", "https://www.fluxlight.com/transceivers/");
let allProducts: Product[] = [];
for (let page = 1; page <= MAX_PAGES; page++) {
try {
const url = page === 1 ? BASE + CATALOG_PATH : `${BASE}${CATALOG_PATH}?page=${page}`;
const html = await fetchPage(url);
const pageProducts = parseProductList(html);
allProducts.push(...pageProducts);
console.log(` Page ${page}: ${pageProducts.length} products`);
if (pageProducts.length === 0) {
console.log(` Empty page ${page}, continuing...`);
}
if (page < MAX_PAGES) await sleep(2000);
} catch (err) {
console.warn(` Page ${page} failed: ${(err as Error).message}`);
}
}
// Dedupe
const seen = new Set<string>();
allProducts = allProducts.filter((p) => {
if (seen.has(p.url)) return false;
seen.add(p.url);
return true;
});
console.log(`\nTotal unique products: ${allProducts.length}`);
let totalProducts = 0;
let priceUpdates = 0;
for (const product of allProducts) {
try {
const txId = await findOrCreateScrapedTransceiver({
partNumber: product.partNumber, vendorId,
formFactor: product.formFactor, speedGbps: product.speedGbps,
speed: product.speed, reachMeters: product.reachMeters,
reachLabel: product.reachLabel, fiberType: product.fiberType,
wavelengths: product.wavelength, category: "DataCenter",
});
if (product.price && product.price > 0) {
const hash = contentHash(JSON.stringify({ price: product.price, part: product.partNumber }));
const updated = await upsertPriceObservation({
transceiverId: txId, sourceVendorId: vendorId,
price: product.price, currency: "USD",
stockLevel: "in_stock", url: product.url, contentHash: hash,
});
if (updated) priceUpdates++;
}
totalProducts++;
} catch (err) {
console.warn(` Error: ${(err as Error).message.slice(0, 80)}`);
}
}
console.log(`\n=== Fluxlight Complete: ${totalProducts} products, ${priceUpdates} prices ===`);
}
if (require.main === module) {
scrapeFluxlight()
.then(() => pool.end())
.catch((err) => { console.error("Fatal:", err); pool.end(); process.exit(1); });
}

View File

@ -0,0 +1,284 @@
/**
* FS.com Scraper Prices, Stock, Product Catalog
*
* FS.com renders products client-side (JS), so we use PlaywrightCrawler.
* Categories: /c/optical-transceivers-9
*
* Respects: robots.txt, rate limiting (2s between requests)
*/
import { PlaywrightCrawler } from "crawlee";
import { ensureVendor, upsertPriceObservation, findOrCreateScrapedTransceiver, pool } from "../utils/db";
import { contentHash, parsePrice, parseStockLevel, parseQuantity } from "../utils/hash";
const BASE_URL = "https://www.fs.com";
const CATEGORY_URLS = [
"/c/1g-sfp-81",
"/c/10g-sfp-63",
"/c/25g-sfp28-3215",
"/c/40g-qsfp-1360",
"/c/100g-qsfp28-sfp-dd-1159",
"/c/200g-qsfp-dd-qsfp56-3542",
"/c/400g-osfp-qsfp112-qsfp-dd-3652",
"/c/800g-osfp-qsfp-dd-4089",
"/c/1.6t-osfp-5597",
"/c/400g-coherent-qsfp-dd-4103",
"/c/10g-cwdm-dwdm-sfp-65",
"/c/100g-dwdm-qsfp28-3863",
];
interface FsProduct {
partNumber: string;
name: string;
price: number;
currency: string;
stockLevel: string;
quantity?: number;
url: string;
formFactor?: string;
speedGbps?: number;
speed?: string;
reachLabel?: string;
}
function detectFormFactor(text: string): string | undefined {
const lower = text.toLowerCase();
if (lower.includes("osfp") && !lower.includes("qsfp")) return "OSFP";
if (lower.includes("qsfp-dd800") || lower.includes("qsfp-dd 800")) return "QSFP-DD800";
if (lower.includes("qsfp-dd")) return "QSFP-DD";
if (lower.includes("qsfp56")) return "QSFP56";
if (lower.includes("qsfp28")) return "QSFP28";
if (lower.includes("qsfp+") || lower.includes("qsfp plus")) return "QSFP+";
if (lower.includes("sfp56")) return "SFP56";
if (lower.includes("sfp28")) return "SFP28";
if (lower.includes("sfp+") || lower.includes("sfp plus")) return "SFP+";
if (lower.includes("sfp") && !lower.includes("qsfp")) return "SFP";
if (lower.includes("cfp2")) return "CFP2";
if (lower.includes("xfp")) return "XFP";
return undefined;
}
function detectSpeed(text: string): { speed: string; speedGbps: number } | undefined {
const patterns: [RegExp, string, number][] = [
[/800\s*g/i, "800G", 800],
[/400\s*g/i, "400G", 400],
[/200\s*g/i, "200G", 200],
[/100\s*g/i, "100G", 100],
[/50\s*g/i, "50G", 50],
[/40\s*g/i, "40G", 40],
[/25\s*g/i, "25G", 25],
[/10\s*g/i, "10G", 10],
[/1\s*g\b/i, "1G", 1],
];
for (const [re, speed, gbps] of patterns) {
if (re.test(text)) return { speed, speedGbps: gbps };
}
return undefined;
}
function detectReach(text: string): string | undefined {
const match = text.match(/(\d+)\s*(m|km)\b/i);
if (match) return `${match[1]}${match[2].toLowerCase()}`;
return undefined;
}
export async function scrapeFs(): Promise<void> {
console.log("=== FS.com Scraper Starting ===\n");
const vendorId = await ensureVendor(
"FS.COM",
"compatible",
"https://www.fs.com",
"https://www.fs.com/c/optical-transceivers-9"
);
console.log(`Vendor ID: ${vendorId}`);
const products: FsProduct[] = [];
let pagesScraped = 0;
const crawler = new PlaywrightCrawler({
maxConcurrency: 1,
maxRequestsPerMinute: 15,
requestHandlerTimeoutSecs: 60,
headless: true,
launchContext: {
launchOptions: {
args: ["--disable-blink-features=AutomationControlled", "--lang=en-US"],
},
},
preNavigationHooks: [
async ({ page }) => {
await page.setExtraHTTPHeaders({
"Accept-Language": "en-US,en;q=0.9",
});
await page.context().addCookies([
{ name: "currency", value: "USD", domain: ".fs.com", path: "/" },
{ name: "lang", value: "en", domain: ".fs.com", path: "/" },
{ name: "country", value: "US", domain: ".fs.com", path: "/" },
]);
},
],
async requestHandler({ page, request, log }) {
const url = request.url;
log.info(`Scraping: ${url}`);
// Wait for Vue.js product grid to render
await page.waitForTimeout(4000);
const productData = await page.evaluate(() => {
const results: Array<{
name: string;
href: string;
price: string;
stock: string;
partNumber: string;
}> = [];
// Strategy 1: Parse .category__grid__item cards (2026 Vue.js DOM)
const gridItems = document.querySelectorAll(".category__grid__item");
for (const item of gridItems) {
const link = item.querySelector('a[href*="/products/"]') as HTMLAnchorElement | null;
const img = item.querySelector("img");
const priceEl = item.querySelector(".grid__price");
const allText = item.textContent || "";
if (!link) continue;
const name = img?.getAttribute("alt")?.trim() || link.textContent?.trim() || "";
const href = link.getAttribute("href") || "";
const price = priceEl?.textContent?.trim() || "";
// Extract stock from text like "1914 in Global Warehouse"
const stockMatch = allText.match(/(\d+)\s+in\s+(?:Global\s+)?Warehouse/i);
const stock = stockMatch ? stockMatch[1] + " in stock" : "";
// Extract FS product ID from URL
const pnMatch = href.match(/products\/(\d+)\.html/);
const partNumber = pnMatch ? `FS-${pnMatch[1]}` : "";
if (name && href) {
results.push({ name, href, price, stock, partNumber });
}
}
// Strategy 2: Fallback — look for product links with prices nearby
if (results.length === 0) {
const productLinks = document.querySelectorAll(
'a[href*="/products/"], a[href*="/product/"]'
);
for (const link of productLinks) {
const el = link as HTMLAnchorElement;
const name = el.textContent?.trim() || "";
const href = el.getAttribute("href") || "";
if (!name || name.length < 5 || !href) continue;
const container = el.closest('[class*="product"]') || el.closest('[class*="item"]') || el.closest("li") || el.parentElement?.parentElement;
let price = "";
let stock = "";
if (container) {
const priceEl = container.querySelector('[class*="price"]');
price = priceEl?.textContent?.trim() || "";
const stockEl = container.querySelector('[class*="stock"], [class*="avail"]');
stock = stockEl?.textContent?.trim() || "";
}
const pn = href.split("/").pop()?.replace(".html", "")?.replace(/\?.*/, "") || "";
if (name) results.push({ name, href, price, stock, partNumber: pn });
}
}
return results;
});
for (const item of productData) {
if (!item.name || !item.price) continue;
const { price, currency } = parsePrice(item.price);
const speedInfo = detectSpeed(item.name);
if (price > 0) {
products.push({
partNumber: item.partNumber || item.name.slice(0, 50),
name: item.name,
price,
currency,
stockLevel: item.stock ? parseStockLevel(item.stock) : "on_request",
quantity: item.stock ? parseQuantity(item.stock) : undefined,
url: item.href.startsWith("http") ? item.href : `${BASE_URL}${item.href}`,
formFactor: detectFormFactor(item.name),
speedGbps: speedInfo?.speedGbps,
speed: speedInfo?.speed,
reachLabel: detectReach(item.name),
});
}
}
pagesScraped++;
log.info(` Found ${productData.length} items on page`);
},
});
const startUrls = CATEGORY_URLS.map((path) => `${BASE_URL}${path}`);
await crawler.run(startUrls);
console.log(`\nPages scraped: ${pagesScraped}`);
console.log(`Products found: ${products.length}`);
// Deduplicate by partNumber
const uniqueProducts = new Map<string, FsProduct>();
for (const p of products) {
const key = p.partNumber || p.name;
if (!uniqueProducts.has(key)) {
uniqueProducts.set(key, p);
}
}
// Write to database
let written = 0;
let skipped = 0;
for (const p of uniqueProducts.values()) {
try {
const transceiverId = await findOrCreateScrapedTransceiver({
partNumber: p.partNumber,
vendorId,
formFactor: p.formFactor,
speedGbps: p.speedGbps,
speed: p.speed,
reachLabel: p.reachLabel,
category: "DataCenter",
});
const hash = contentHash({ price: p.price, stock: p.stockLevel, qty: p.quantity });
const isNew = await upsertPriceObservation({
transceiverId,
sourceVendorId: vendorId,
price: p.price,
currency: p.currency,
stockLevel: p.stockLevel,
quantityAvailable: p.quantity,
url: p.url,
contentHash: hash,
});
if (isNew) written++;
else skipped++;
} catch (err) {
console.error(` Error: ${p.partNumber}:`, (err as Error).message);
}
}
console.log(`\nDatabase: ${written} new, ${skipped} unchanged (${uniqueProducts.size} unique)`);
console.log("=== FS.com Scraper Complete ===\n");
}
if (require.main === module) {
scrapeFs()
.then(() => pool.end())
.catch((err) => {
console.error("Fatal:", err);
pool.end();
process.exit(1);
});
}

View File

@ -0,0 +1,233 @@
/**
* GBICS.com Scraper UK-based compatible transceiver vendor
*
* gbics.com BigCommerce store, server-rendered HTML, GBP prices.
* Products in <li> cards with <h3> product names, "Now: £XX.XX" pricing.
* Pagination via ?page=N. Rate limited: 1 req/2sec.
*/
import { pool, findOrCreateScrapedTransceiver, ensureVendor, upsertPriceObservation } from "../utils/db";
import { contentHash } from "../utils/hash";
const BASE = "https://www.gbics.com";
const HEADERS = {
"User-Agent": "Mozilla/5.0 (compatible; TIP-Bot/1.0; research)",
Accept: "text/html,application/xhtml+xml",
};
const CATEGORIES = [
{ path: "/800g-osfp/", formFactor: "OSFP", speed: "800G", speedGbps: 800 },
{ path: "/400g-qsfp112/", formFactor: "QSFP112", speed: "400G", speedGbps: 400 },
{ path: "/400g-qsfp-dd/", formFactor: "QSFP-DD", speed: "400G", speedGbps: 400 },
{ path: "/400g-osfp/", formFactor: "OSFP", speed: "400G", speedGbps: 400 },
{ path: "/200g-qsfp56/", formFactor: "QSFP56", speed: "200G", speedGbps: 200 },
{ path: "/100g-qsfp28/", formFactor: "QSFP28", speed: "100G", speedGbps: 100 },
{ path: "/40g-qsfp/", formFactor: "QSFP+", speed: "40G", speedGbps: 40 },
{ path: "/25g-sfp28/", formFactor: "SFP28", speed: "25G", speedGbps: 25 },
{ path: "/10g-sfp/", formFactor: "SFP+", speed: "10G", speedGbps: 10 },
{ path: "/1g-sfp/", formFactor: "SFP", speed: "1G", speedGbps: 1 },
];
interface Product {
partNumber: string;
name: string;
url: string;
price?: number;
formFactor: string;
speed: string;
speedGbps: number;
reachLabel?: string;
reachMeters?: number;
fiberType?: string;
wavelength?: string;
compatibleWith?: string;
}
function sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}
function detectReach(text: string): { label: string; meters: number } | undefined {
const patterns: [RegExp, string, number][] = [
[/\b120\s*km\b/i, "120km", 120000],
[/\b80\s*km\b/i, "80km", 80000],
[/\b40\s*km\b/i, "40km", 40000],
[/\b20\s*km\b/i, "20km", 20000],
[/\b10\s*km\b/i, "10km", 10000],
[/\b2\s*km\b/i, "2km", 2000],
[/\b550\s*m\b/i, "550m", 550],
[/\b500\s*m\b/i, "500m", 500],
[/\b400\s*m\b/i, "400m", 400],
[/\b300\s*m\b/i, "300m", 300],
[/\b150\s*m\b/i, "150m", 150],
[/\b100\s*m\b/i, "100m", 100],
[/\bLR4\b/, "10km", 10000],
[/\bLR\b/, "10km", 10000],
[/\bER4?\b/, "40km", 40000],
[/\bZR4?\b/, "80km", 80000],
[/\bSR4?\b/, "300m", 300],
[/\bDR4?\b/, "500m", 500],
[/\bFR4?\b/, "2km", 2000],
];
for (const [regex, label, meters] of patterns) {
if (regex.test(text)) return { label, meters };
}
return undefined;
}
function detectFiber(text: string): string {
if (/single.?mode|smf|[^a-z]lx[^a-z]|[^a-z]lr[^a-z]|[^a-z]er[^a-z]|[^a-z]zr[^a-z]|bidi|cwdm|dwdm/i.test(text)) return "SMF";
if (/multi.?mode|mmf|[^a-z]sx[^a-z]|[^a-z]sr[^a-z]/i.test(text)) return "MMF";
if (/copper|dac|twinax|rj.?45|base-t|cat[56x]/i.test(text)) return "Copper";
return "";
}
function detectWavelength(text: string): string {
const match = text.match(/(\d{3,4})\s*nm/i);
if (match) return match[1];
return "";
}
function extractCompatibleVendor(name: string): string {
const match = name.match(/^(\w+(?:\s+\w+)?)\s+Compatible\b/i);
return match ? match[1] : "";
}
function parseProductList(html: string, cat: typeof CATEGORIES[number]): Product[] {
const products: Product[] = [];
// Collapse whitespace for easier regex matching
const collapsed = html.replace(/\s+/g, " ");
// BigCommerce card-title pattern:
// <a aria-label="Product Name, £XX.XX" href="URL" data-event-type="product-click">
const productRegex = /aria-label="([^"]+)"\s+href="(https?:\/\/(?:www\.)?gbics\.com\/[^"]+)"[^>]*data-event-type="product-click"/gi;
let match;
while ((match = productRegex.exec(collapsed)) !== null) {
const label = match[1].trim();
const url = match[2];
// aria-label contains "Product Name, £XX.XX"
// Split on last comma to separate name and price
const priceInLabel = label.match(/,\s*£\s*([\d,.]+)\s*$/);
const name = priceInLabel ? label.slice(0, label.lastIndexOf(",")).trim() : label;
let price = priceInLabel ? parseFloat(priceInLabel[1].replace(",", "")) : undefined;
// Fallback: extract price from data-price-asc attribute on parent <li>
if (!price) {
const priceContext = collapsed.slice(Math.max(0, match.index - 500), match.index);
const dataPriceMatch = priceContext.match(/data-price-asc="(\d+)"/);
if (dataPriceMatch) price = parseFloat(dataPriceMatch[1]);
}
if (name.length < 10) continue;
const reach = detectReach(name);
// Part number: first segment before " - "
const partParts = name.split(/\s+-\s+/);
const partNumber = partParts[0]?.trim().slice(0, 80) || url.split("/").filter(Boolean).pop() || "";
products.push({
partNumber, name, url,
price: price && price > 0 && price < 50000 ? price : undefined,
formFactor: cat.formFactor, speed: cat.speed, speedGbps: cat.speedGbps,
reachLabel: reach?.label, reachMeters: reach?.meters,
fiberType: detectFiber(name), wavelength: detectWavelength(name),
compatibleWith: extractCompatibleVendor(name),
});
}
// Fallback: try "Now: £XX.XX" pattern near product links
if (products.length === 0) {
const altRegex = /href="(https?:\/\/(?:www\.)?gbics\.com\/[^"]+)"[^>]*>\s*([^<]{15,})<\/a>/gi;
while ((match = altRegex.exec(collapsed)) !== null) {
const url = match[1];
const name = match[2].trim();
if (name.length < 10 || products.find((p) => p.url === url)) continue;
if (!/transceiver|sfp|qsfp|xfp|osfp|base/i.test(name)) continue;
const context = collapsed.slice(Math.max(0, match.index - 300), match.index + 600);
const priceMatch = context.match(/Now:\s*£\s*([\d,.]+)/) || context.match(/£\s*([\d,.]+)/);
const price = priceMatch ? parseFloat(priceMatch[1].replace(",", "")) : undefined;
const reach = detectReach(name);
products.push({
partNumber: name.split(/\s+-\s+/)[0]?.trim().slice(0, 80) || "",
name, url,
price: price && price > 0 && price < 50000 ? price : undefined,
formFactor: cat.formFactor, speed: cat.speed, speedGbps: cat.speedGbps,
reachLabel: reach?.label, reachMeters: reach?.meters,
fiberType: detectFiber(name), wavelength: detectWavelength(name),
compatibleWith: extractCompatibleVendor(name),
});
}
}
const seen = new Set<string>();
return products.filter((p) => {
if (seen.has(p.url)) return false;
seen.add(p.url);
return true;
});
}
async function fetchPage(url: string): Promise<string> {
const resp = await fetch(url, { headers: HEADERS, signal: AbortSignal.timeout(30000) });
if (!resp.ok) throw new Error(`HTTP ${resp.status} for ${url}`);
return resp.text();
}
export async function scrapeGbics(): Promise<void> {
console.log("=== GBICS.com Scraper Starting ===\n");
const vendorId = await ensureVendor("GBICS", "compatible", "https://www.gbics.com", "https://www.gbics.com/optical-transceivers/");
let totalProducts = 0;
let priceUpdates = 0;
for (const cat of CATEGORIES) {
console.log(`\n--- ${cat.formFactor} (${cat.speed}) [${cat.path}] ---`);
try {
const html = await fetchPage(BASE + cat.path);
const catProducts = parseProductList(html, cat);
console.log(` Found ${catProducts.length} products`);
for (const product of catProducts) {
try {
const txId = await findOrCreateScrapedTransceiver({
partNumber: product.partNumber, vendorId,
formFactor: product.formFactor, speedGbps: product.speedGbps,
speed: product.speed, reachMeters: product.reachMeters,
reachLabel: product.reachLabel, fiberType: product.fiberType,
wavelengths: product.wavelength, category: "DataCenter",
});
if (product.price && product.price > 0) {
const hash = contentHash(JSON.stringify({ price: product.price, part: product.partNumber }));
const updated = await upsertPriceObservation({
transceiverId: txId, sourceVendorId: vendorId,
price: product.price, currency: "GBP",
stockLevel: "in_stock", url: product.url, contentHash: hash,
});
if (updated) priceUpdates++;
}
totalProducts++;
} catch (err) {
console.warn(` Error: ${(err as Error).message.slice(0, 80)}`);
}
}
} catch (err) {
console.error(` Category failed: ${(err as Error).message}`);
}
await sleep(2000);
}
console.log(`\n=== GBICS Complete: ${totalProducts} products, ${priceUpdates} prices ===`);
}
if (require.main === module) {
scrapeGbics()
.then(() => pool.end())
.catch((err) => { console.error("Fatal:", err); pool.end(); process.exit(1); });
}

View File

@ -0,0 +1,241 @@
/**
* Juniper HCT Scraper OEM Hardware Compatibility Tool
*
* apps.juniper.net/hct Next.js SSR app with product data embedded in
* self.__next_f.push() payloads. Transceivers category = 100001.
* Rich data: modelNumber, partNumber, distance, speedType, formFactor, EOL status.
* No prices (OEM), but excellent compatibility + spec data.
*/
import { pool, findOrCreateScrapedTransceiver, ensureVendor } from "../utils/db";
const BASE = "https://apps.juniper.net/hct";
const HEADERS = {
"User-Agent": "Mozilla/5.0 (compatible; TIP-Bot/1.0; research)",
Accept: "text/html,application/xhtml+xml",
};
const CATEGORIES = [
{ id: 100001, name: "Transceivers" },
];
interface JuniperTransceiver {
modelNumber: string;
partNumber: string;
description: string;
cableType: string;
distance: string;
speedType: string;
formFactor: string;
connectorType: string;
maxDistanceKm?: number;
maxDistanceLabel?: string;
isModelEol: boolean;
}
function sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}
function parseSpeedGbps(speedType: string): { speed: string; speedGbps: number } {
const lower = speedType.toLowerCase();
if (lower.includes("800g")) return { speed: "800G", speedGbps: 800 };
if (lower.includes("400g")) return { speed: "400G", speedGbps: 400 };
if (lower.includes("200g")) return { speed: "200G", speedGbps: 200 };
if (lower.includes("100g")) return { speed: "100G", speedGbps: 100 };
if (lower.includes("40g")) return { speed: "40G", speedGbps: 40 };
if (lower.includes("25g")) return { speed: "25G", speedGbps: 25 };
if (lower.includes("10g")) return { speed: "10G", speedGbps: 10 };
if (lower.includes("1g") || lower.includes("1000")) return { speed: "1G", speedGbps: 1 };
return { speed: speedType || "Unknown", speedGbps: 0 };
}
function normalizeFormFactor(ff: string): string {
const upper = ff.toUpperCase().trim();
if (upper.includes("QSFP-DD") || upper.includes("QSFPDD")) return "QSFP-DD";
if (upper.includes("QSFP28")) return "QSFP28";
if (upper.includes("QSFP+") || upper === "QSFP") return "QSFP+";
if (upper.includes("OSFP")) return "OSFP";
if (upper.includes("CFP2")) return "CFP2";
if (upper.includes("CFP4")) return "CFP4";
if (upper.includes("CFP")) return "CFP";
if (upper.includes("SFP56")) return "SFP56";
if (upper.includes("SFP28")) return "SFP28";
if (upper.includes("SFP+")) return "SFP+";
if (upper.includes("XFP")) return "XFP";
if (upper.includes("SFP")) return "SFP";
return ff || "SFP";
}
function detectFiber(cableType: string, description: string): string {
const text = `${cableType} ${description}`.toLowerCase();
if (/smf|single.?mode/.test(text)) return "SMF";
if (/mmf|multi.?mode/.test(text)) return "MMF";
if (/copper|dac|twinax|cat\s*[56]|rj.?45|base-t/.test(text)) return "Copper";
return "";
}
function parseDistance(distance: string): { label: string; meters: number } | undefined {
if (!distance) return undefined;
const km = distance.match(/([\d.]+)\s*km/i);
if (km) return { label: `${km[1]}km`, meters: Math.round(parseFloat(km[1]) * 1000) };
const m = distance.match(/([\d.]+)\s*m\b/i);
if (m) return { label: `${m[1]}m`, meters: parseInt(m[1]) };
return undefined;
}
function detectWavelength(description: string): string {
const match = description.match(/(\d{3,4})\s*nm/i);
return match ? match[1] : "";
}
/**
* Extract transceiver data from Next.js SSR payload.
* Data is embedded in self.__next_f.push([...]) with escaped JSON (\" not ").
* Strategy: unescape the HTML, find categoryDetail array, parse each object.
*/
function parseNextJsData(html: string): JuniperTransceiver[] {
const transceivers: JuniperTransceiver[] = [];
// Unescape the escaped JSON (\" → ", \\ → \)
const unescaped = html.replace(/\\"/g, '"').replace(/\\\\"/g, '\\"');
// Find categoryDetail array and extract individual objects
const detailIdx = unescaped.indexOf('"categoryDetail":[');
if (detailIdx === -1) {
console.log(" Warning: categoryDetail not found in HTML");
return transceivers;
}
// Extract from categoryDetail to end of array
const arrayStart = unescaped.indexOf("[", detailIdx);
if (arrayStart === -1) return transceivers;
// Use regex to find each transceiver object by modelNumber
const modelRegex = /"modelNumber"\s*:\s*"([^"]+)"/g;
const seen = new Set<string>();
let match;
while ((match = modelRegex.exec(unescaped)) !== null) {
const modelNumber = match[1];
if (seen.has(modelNumber)) continue;
seen.add(modelNumber);
// Extract chunk around this model
const idx = match.index;
const objStart = unescaped.lastIndexOf("{", idx);
const chunk = unescaped.slice(objStart, objStart + 2000);
const getString = (field: string): string => {
const re = new RegExp(`"${field}"\\s*:\\s*"([^"]*)"`, "i");
const m = chunk.match(re);
return m ? m[1] : "";
};
// For array fields like cableType:["SMF"], speedType:[{speed:"100G"}], formFactor:["CFP"]
const getArrayFirst = (field: string): string => {
// Try ["value"] pattern
const arrRe = new RegExp(`"${field}"\\s*:\\s*\\[\\s*"([^"]*)"`, "i");
const arrM = chunk.match(arrRe);
if (arrM) return arrM[1];
// Try [{speed:"value"}] pattern
const objRe = new RegExp(`"${field}"\\s*:\\s*\\[\\s*\\{\\s*"\\w+"\\s*:\\s*"([^"]*)"`, "i");
const objM = chunk.match(objRe);
if (objM) return objM[1];
return getString(field);
};
const getBool = (field: string): boolean => {
const re = new RegExp(`"${field}"\\s*:\\s*(true|false)`, "i");
const m = chunk.match(re);
return m ? m[1] === "true" : false;
};
const getNum = (field: string): number | undefined => {
const re = new RegExp(`"${field}"\\s*:\\s*(\\d+(?:\\.\\d+)?)`, "i");
const m = chunk.match(re);
return m ? parseFloat(m[1]) : undefined;
};
// Extract distance from array like ["40 km"] or from maxDistanceLabel
const distArr = chunk.match(/"distance"\s*:\s*\[\s*"([^"]*)"/i);
const distance = distArr ? distArr[1] : getString("maxDistanceLabel");
transceivers.push({
modelNumber,
partNumber: getString("partNumber") || getString("oldPartNumber") || modelNumber,
description: getString("description"),
cableType: getArrayFirst("cableType"),
distance,
speedType: getArrayFirst("speedType"),
formFactor: getArrayFirst("formFactor"),
connectorType: getString("connectorType"),
maxDistanceKm: getNum("maxDistanceKm"),
maxDistanceLabel: getString("maxDistanceLabel"),
isModelEol: getBool("isModelEol"),
});
}
return transceivers;
}
async function fetchPage(url: string): Promise<string> {
const resp = await fetch(url, { headers: HEADERS, signal: AbortSignal.timeout(60000) });
if (!resp.ok) throw new Error(`HTTP ${resp.status} for ${url}`);
return resp.text();
}
export async function scrapeJuniperHct(): Promise<void> {
console.log("=== Juniper HCT Scraper Starting ===\n");
const vendorId = await ensureVendor("Juniper Networks", "oem", "https://www.juniper.net", "https://apps.juniper.net/hct/");
let totalProducts = 0;
for (const cat of CATEGORIES) {
console.log(`\n--- ${cat.name} (category ${cat.id}) ---`);
try {
const html = await fetchPage(`${BASE}/category/${cat.id}`);
console.log(` Fetched ${(html.length / 1024).toFixed(0)}KB`);
const transceivers = parseNextJsData(html);
console.log(` Parsed ${transceivers.length} transceivers`);
for (const tx of transceivers) {
try {
const speedInfo = parseSpeedGbps(tx.speedType || tx.description);
const distInfo = tx.maxDistanceKm
? { label: `${tx.maxDistanceKm}km`, meters: Math.round(tx.maxDistanceKm * 1000) }
: parseDistance(tx.distance);
const formFactor = normalizeFormFactor(tx.formFactor);
await findOrCreateScrapedTransceiver({
partNumber: tx.modelNumber, vendorId,
formFactor, speedGbps: speedInfo.speedGbps,
speed: speedInfo.speed, reachMeters: distInfo?.meters,
reachLabel: distInfo?.label,
fiberType: detectFiber(tx.cableType, tx.description),
wavelengths: detectWavelength(tx.description),
category: "DataCenter",
});
totalProducts++;
} catch (err) {
console.warn(` Error [${tx.modelNumber}]: ${(err as Error).message.slice(0, 80)}`);
}
}
} catch (err) {
console.error(` Category failed: ${(err as Error).message}`);
}
await sleep(2000);
}
console.log(`\n=== Juniper HCT Complete: ${totalProducts} transceivers (no prices - OEM) ===`);
}
if (require.main === module) {
scrapeJuniperHct()
.then(() => pool.end())
.catch((err) => { console.error("Fatal:", err); pool.end(); process.exit(1); });
}

View File

@ -0,0 +1,298 @@
/**
* News Aggregator Optics & Fiber Trade Press RSS Scraper
*
* Sources:
* - optics.org (photonics industry news)
* - SPIE Newsroom (photonics research)
* - Network World (data center / networking)
* - Light Reading (telecom)
* - Telecom Ramblings (industry commentary)
*
* Stores articles in news_articles table.
* Relevance filtering: keyword scoring for transceiver/optics topics.
*/
import { pool } from "../utils/db";
import { contentHash } from "../utils/hash";
import { parseStringPromise } from "xml2js";
// Categories allowed by news_articles CHECK constraint
type NewsCategory = "product_launch" | "market_report" | "standard" | "m_and_a" | "factory" | "event";
interface RssFeed {
name: string;
url: string;
category: NewsCategory;
}
interface NewsArticle {
title: string;
sourceUrl: string;
summary: string;
publishedAt: Date;
source: string;
category: NewsCategory | null;
relevanceScore: number;
contentHash: string;
}
const FEEDS: RssFeed[] = [
// === PRIMARY: Transceiver-specific ===
{
name: "The Next Platform",
url: "https://www.nextplatform.com/feed/",
category: "market_report",
},
{
name: "ServeTheHome",
url: "https://www.servethehome.com/feed/",
category: "product_launch",
},
{
name: "Optics.org",
url: "https://optics.org/rss/news",
category: "market_report",
},
// === SECONDARY: Datacenter / Networking ===
{
name: "Data Center Knowledge",
url: "https://www.datacenterknowledge.com/rss.xml",
category: "market_report",
},
{
name: "Network World - Data Center",
url: "https://www.networkworld.com/category/data-center/index.rss",
category: "market_report",
},
{
name: "The Register - Data Centre",
url: "https://www.theregister.com/data_centre/headlines.atom",
category: "market_report",
},
// === TERTIARY: General tech / photonics ===
{
name: "CableFree",
url: "https://www.cablefree.net/rss",
category: "market_report",
},
{
name: "Nature Photonics",
url: "https://www.nature.com/nphoton.rss",
category: "standard",
},
// === VENDOR NEWS ===
{
name: "Cisco Blogs - Data Center",
url: "https://blogs.cisco.com/datacenter/feed",
category: "product_launch",
},
{
name: "Arista Blog",
url: "https://blogs.arista.com/blog/rss.xml",
category: "product_launch",
},
];
// Keywords for relevance scoring
const HIGH_RELEVANCE = [
"transceiver", "sfp", "qsfp", "xfp", "cfp", "osfp",
"optical module", "fiber optic", "wavelength", "dwdm", "cwdm",
"400g", "800g", "1.6t", "coherent", "pluggable",
"ofc", "ecoc", "cioe",
];
const MEDIUM_RELEVANCE = [
"data center", "datacenter", "interconnect", "bandwidth",
"switch", "router", "cisco", "arista", "juniper",
"100g", "40g", "25g", "10g",
"silicon photonics", "photonic",
"ii-vi", "coherent", "lumentum", "inphi",
"flexoptix", "prolabs",
];
function scoreRelevance(title: string, summary: string): number {
const text = `${title} ${summary}`.toLowerCase();
let score = 0;
for (const kw of HIGH_RELEVANCE) {
if (text.includes(kw)) score += 3;
}
for (const kw of MEDIUM_RELEVANCE) {
if (text.includes(kw)) score += 1;
}
return score;
}
async function fetchFeed(feed: RssFeed): Promise<NewsArticle[]> {
const articles: NewsArticle[] = [];
try {
const resp = await fetch(feed.url, {
headers: {
"User-Agent": "Mozilla/5.0 (compatible; TIP-NewsBot/1.0; +https://flexoptix.net)",
Accept: "application/rss+xml, application/xml, text/xml",
},
signal: AbortSignal.timeout(15000),
});
if (!resp.ok) {
console.warn(` Feed ${feed.name} returned ${resp.status}`);
return [];
}
const rawXml = await resp.text();
// Sanitize common RSS issues: unescaped & in URLs, attribute-without-value
const xml = rawXml
.replace(/&(?!amp;|lt;|gt;|quot;|apos;|#\d+;|#x[\dA-Fa-f]+;)/g, "&amp;")
.replace(/(<\w[^>]*)\s+(\w+)=([^"'\s>]+)(?=[\s/>])/g, '$1 $2="$3"');
const parsed = await parseStringPromise(xml, { explicitArray: false, strict: false });
// strict: false makes keys uppercase; support both
const rss = parsed?.rss || parsed?.RSS;
const channel = rss?.channel || rss?.CHANNEL || parsed?.feed || parsed?.FEED;
if (!channel) return [];
const items = channel.item || channel.ITEM || channel.entry || channel.ENTRY || [];
const itemArray = Array.isArray(items) ? items : [items];
for (const item of itemArray) {
const title = extractText(item.title || item.TITLE) || "";
const url = extractLink(item) || "";
const summary = extractText(
item.description || item.DESCRIPTION || item.summary || item.SUMMARY || item["content:encoded"]
) || "";
const pubDate = item.pubDate || item.PUBDATE || item.published || item.updated || "";
if (!title || !url) continue;
const publishedAt = pubDate ? new Date(pubDate) : new Date();
if (isNaN(publishedAt.getTime())) continue;
// Skip articles older than 7 days
const ageMs = Date.now() - publishedAt.getTime();
if (ageMs > 7 * 24 * 60 * 60 * 1000) continue;
const relevanceScore = scoreRelevance(title, summary);
const hash = contentHash({ title, url });
articles.push({
title: title.slice(0, 500),
sourceUrl: url.slice(0, 1000),
summary: stripHtml(summary).slice(0, 2000),
publishedAt,
source: feed.name,
category: feed.category as NewsCategory,
relevanceScore,
contentHash: hash,
});
}
} catch (err) {
console.warn(` Feed ${feed.name} error:`, (err as Error).message);
}
return articles;
}
function extractText(value: unknown): string {
if (!value) return "";
if (typeof value === "string") return value;
if (typeof value === "object" && value !== null) {
const obj = value as Record<string, unknown>;
return String(obj._ || obj["#text"] || "");
}
return String(value);
}
function extractLink(item: Record<string, unknown>): string {
const link = item.link || item.LINK;
if (typeof link === "string") return link;
if (Array.isArray(link)) {
const rel = (link as Array<Record<string, unknown>>).find(
(l) => !l["$"] || (l["$"] as Record<string, string>).rel === "alternate"
);
return String((rel?.["$"] as Record<string, string>)?.href || rel?._ || "");
}
if (typeof link === "object" && link !== null) {
const l = link as Record<string, unknown>;
return String((l["$"] as Record<string, string>)?.href || l._ || "");
}
return "";
}
function stripHtml(html: string): string {
return html
.replace(/<[^>]+>/g, " ")
.replace(/&nbsp;/g, " ")
.replace(/&amp;/g, "&")
.replace(/&lt;/g, "<")
.replace(/&gt;/g, ">")
.replace(/&quot;/g, '"')
.replace(/\s+/g, " ")
.trim();
}
async function upsertArticle(article: NewsArticle): Promise<boolean> {
const result = await pool.query(
`INSERT INTO news_articles (title, source_url, summary, published_at, source, category, relevance_score, content_hash)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8)
ON CONFLICT (source_url) DO UPDATE
SET relevance_score = EXCLUDED.relevance_score,
content_hash = EXCLUDED.content_hash
RETURNING (xmax = 0) AS inserted`,
[
article.title,
article.sourceUrl,
article.summary,
article.publishedAt,
article.source,
article.category,
article.relevanceScore,
article.contentHash,
]
);
return result.rows[0]?.inserted ?? true;
}
export async function scrapeNews(): Promise<void> {
console.log("=== News Scraper Starting ===\n");
let totalFetched = 0;
let totalWritten = 0;
let totalRelevant = 0;
for (const feed of FEEDS) {
console.log(`Fetching: ${feed.name} (${feed.url})`);
const articles = await fetchFeed(feed);
console.log(`${articles.length} articles (last 7 days)`);
for (const article of articles) {
totalFetched++;
if (article.relevanceScore > 0) totalRelevant++;
try {
const isNew = await upsertArticle(article);
if (isNew) totalWritten++;
} catch (err) {
console.error(` Error saving article:`, (err as Error).message);
}
}
// Rate limit between feeds
await new Promise((r) => setTimeout(r, 1000));
}
console.log(`\nFetched: ${totalFetched} articles`);
console.log(`Relevant (score > 0): ${totalRelevant}`);
console.log(`Written: ${totalWritten} new`);
console.log("=== News Scraper Complete ===\n");
}
if (require.main === module) {
scrapeNews()
.then(() => pool.end())
.catch((err) => {
console.error("Fatal:", err);
pool.end();
process.exit(1);
});
}

View File

@ -0,0 +1,297 @@
/**
* Optcore.net Scraper Most transparent pricing in the industry.
* Prices start at $5.50, fully public, no bot protection.
*
* Strategy: WP REST API to enumerate transceiver product URLs,
* then PlaywrightCrawler to render each page and extract price.
*
* Optcore uses Flatsome WooCommerce with Cloudflare Rocket Loader
* (JS lazy-loading) static HTML has no product data.
*/
import { PlaywrightCrawler } from "crawlee";
import { ensureVendor, upsertPriceObservation, findOrCreateScrapedTransceiver, pool } from "../utils/db";
import { contentHash, parsePrice, parseStockLevel } from "../utils/hash";
const BASE_URL = "https://www.optcore.net";
// Transceiver category IDs from /wp-json/wp/v2/product_cat
// Filtered to optical transceiver categories with products
const TRANSCEIVER_CATEGORY_IDS = [
309, // 10G SFP+
173, // 1G SFP
76, // 100G QSFP28
79, // 25G SFP28
73, // 40G QSFP+
311, // 10G BiDi SFP+
313, // 10G CWDM SFP+
312, // 10G DWDM SFP+
333, // 10G XFP
1088, // 10GBase-T SFP+
59, // 8G/10G/16G SFP+
1102, // BiDi SFP
4097, // 400G QSFP-DD
77, // 100G CFP/CFP2/CFP4
4101, // 200G QSFP56
4092, // 50G SFP56
6441, // 800G OSFP
];
interface OptcoreProduct {
partNumber: string;
name: string;
price: number;
currency: string;
stockLevel: string;
url: string;
formFactor?: string;
speedGbps?: number;
speed?: string;
reachLabel?: string;
}
function detectFormFactor(text: string): string | undefined {
const lower = text.toLowerCase();
if (lower.includes("osfp") && !lower.includes("qsfp")) return "OSFP";
if (lower.includes("qsfp-dd")) return "QSFP-DD";
if (lower.includes("qsfp56")) return "QSFP56";
if (lower.includes("qsfp28")) return "QSFP28";
if (lower.includes("qsfp+") || lower.includes("qsfp plus")) return "QSFP+";
if (lower.includes("sfp28")) return "SFP28";
if (lower.includes("sfp56")) return "SFP56";
if (lower.includes("sfp+") || lower.includes("sfp plus")) return "SFP+";
if (lower.includes("cfp4")) return "CFP4";
if (lower.includes("cfp2")) return "CFP2";
if (lower.includes("cfp")) return "CFP";
if (lower.includes("xfp")) return "XFP";
if (lower.includes("sfp") && !lower.includes("qsfp")) return "SFP";
return undefined;
}
function detectSpeed(text: string): { speed: string; speedGbps: number } | undefined {
const patterns: [RegExp, string, number][] = [
[/800\s*g/i, "800G", 800],
[/400\s*g/i, "400G", 400],
[/200\s*g/i, "200G", 200],
[/100\s*g/i, "100G", 100],
[/50\s*g/i, "50G", 50],
[/40\s*g/i, "40G", 40],
[/25\s*g/i, "25G", 25],
[/16\s*g/i, "16G", 16],
[/10\s*g/i, "10G", 10],
[/1000\s*base/i, "1G", 1],
[/1\s*g\b/i, "1G", 1],
];
for (const [re, speed, gbps] of patterns) {
if (re.test(text)) return { speed, speedGbps: gbps };
}
return undefined;
}
function detectReach(text: string): string | undefined {
const match = text.match(/(\d+)\s*(m|km)\b/i);
if (match) return `${match[1]}${match[2].toLowerCase()}`;
return undefined;
}
/**
* Fetch product URLs for transceiver categories via WP REST API.
* Returns up to 2000 product URLs with title + slug for metadata.
*/
async function fetchTransceiverUrls(): Promise<Array<{ url: string; title: string; partNumber: string }>> {
const results: Array<{ url: string; title: string; partNumber: string }> = [];
const seen = new Set<string>();
for (const catId of TRANSCEIVER_CATEGORY_IDS) {
let page = 1;
let hasMore = true;
while (hasMore) {
const apiUrl = `${BASE_URL}/wp-json/wp/v2/product?product_cat=${catId}&per_page=100&page=${page}&_fields=slug,link,title`;
try {
const resp = await fetch(apiUrl, {
headers: { "User-Agent": "Mozilla/5.0 (compatible; TIP-Scraper/1.0)" },
signal: AbortSignal.timeout(15000),
});
if (!resp.ok) break;
const totalPages = parseInt(resp.headers.get("X-WP-TotalPages") || "1");
const products: Array<{ slug: string; link: string; title: { rendered: string } }> = await resp.json();
for (const p of products) {
if (!seen.has(p.slug)) {
seen.add(p.slug);
results.push({
url: p.link,
title: p.title.rendered,
partNumber: p.slug,
});
}
}
hasMore = page < totalPages;
page++;
// Rate limit: 10 req/sec max
await new Promise((r) => setTimeout(r, 100));
} catch {
hasMore = false;
}
}
}
return results;
}
export async function scrapeOptcore(): Promise<void> {
console.log("=== Optcore.net Scraper Starting ===\n");
const vendorId = await ensureVendor(
"Optcore",
"compatible",
"https://www.optcore.net",
"https://www.optcore.net/product-category/optical-transceiver/"
);
console.log(`Vendor ID: ${vendorId}`);
// Step 1: Enumerate transceiver product URLs via WP REST API
console.log("Fetching product URLs via WP REST API...");
const productMeta = await fetchTransceiverUrls();
console.log(`Found ${productMeta.length} transceiver product URLs`);
// Build a map for quick metadata lookup
const metaByUrl = new Map(productMeta.map((p) => [p.url, p]));
const products: OptcoreProduct[] = [];
let pagesScraped = 0;
// Step 2: Render each product page with Playwright to extract price
const crawler = new PlaywrightCrawler({
maxConcurrency: 3,
maxRequestsPerMinute: 30,
requestHandlerTimeoutSecs: 30,
headless: true,
launchContext: {
launchOptions: {
args: ["--disable-blink-features=AutomationControlled", "--no-sandbox"],
},
},
async requestHandler({ page, request, log }) {
const url = request.url;
log.info(`Scraping: ${url}`);
// Wait for WooCommerce price element to appear
try {
await page.waitForSelector(".woocommerce-Price-amount, .price .amount, [class*=\"price\"]", {
timeout: 8000,
});
} catch {
// Price element not found — might be out of stock or JS failed
log.warning(`No price element found: ${url}`);
pagesScraped++;
return;
}
const data = await page.evaluate(() => {
// Product title
const title =
document.querySelector("h1.product_title, h1.entry-title, h1")?.textContent?.trim() || "";
// Price — WooCommerce renders: <span class="price"><bdi><span class="woocommerce-Price-currencySymbol">$</span>5.50</bdi></span>
const priceEl = document.querySelector(
".price ins .woocommerce-Price-amount, .price .woocommerce-Price-amount, .woocommerce-Price-amount"
);
const priceText = priceEl?.textContent?.trim() || "";
// Stock
const stockEl = document.querySelector(".stock, .availability, [class*=\"stock\"]");
const stockText = stockEl?.textContent?.trim() || "";
return { title, priceText, stockText };
});
const meta = metaByUrl.get(url);
const name = data.title || meta?.title || url.split("/").filter(Boolean).pop() || "";
const partNumber = meta?.partNumber || url.split("/").filter(Boolean).pop() || "";
const { price, currency } = parsePrice(data.priceText);
if (price > 0) {
const speedInfo = detectSpeed(name);
products.push({
partNumber,
name,
price,
currency,
stockLevel: data.stockText ? parseStockLevel(data.stockText) : "in_stock",
url,
formFactor: detectFormFactor(name),
speedGbps: speedInfo?.speedGbps,
speed: speedInfo?.speed,
reachLabel: detectReach(name),
});
}
pagesScraped++;
},
});
const urls = productMeta.map((p) => p.url);
await crawler.run(urls);
console.log(`\nPages scraped: ${pagesScraped}`);
console.log(`Products with price: ${products.length}`);
// Deduplicate
const unique = new Map<string, OptcoreProduct>();
for (const p of products) {
if (!unique.has(p.partNumber)) unique.set(p.partNumber, p);
}
// Write to DB
let written = 0;
let skipped = 0;
for (const p of unique.values()) {
try {
const transceiverId = await findOrCreateScrapedTransceiver({
partNumber: p.partNumber,
vendorId,
formFactor: p.formFactor,
speedGbps: p.speedGbps,
speed: p.speed,
reachLabel: p.reachLabel,
category: "DataCenter",
});
const hash = contentHash({ price: p.price, stock: p.stockLevel });
const isNew = await upsertPriceObservation({
transceiverId,
sourceVendorId: vendorId,
price: p.price,
currency: p.currency,
stockLevel: p.stockLevel,
url: p.url,
contentHash: hash,
});
if (isNew) written++;
else skipped++;
} catch (err) {
console.error(` Error: ${p.partNumber}:`, (err as Error).message);
}
}
console.log(`\nDatabase: ${written} new, ${skipped} unchanged (${unique.size} unique)`);
console.log("=== Optcore.net Scraper Complete ===\n");
}
if (require.main === module) {
scrapeOptcore()
.then(() => pool.end())
.catch((err) => {
console.error("Fatal:", err);
pool.end();
process.exit(1);
});
}

View File

@ -0,0 +1,485 @@
/**
* ProLabs Scraper Enterprise-grade compatible optics (Legrand subsidiary)
*
* prolabs.com CloudFront WAF aggressively blocks datacenter IPs.
* Uses PlaywrightCrawler with Firefox for anti-detection.
*
* KNOWN ISSUE: CloudFront blocks all requests from IONOS/datacenter IPs
* (HTTP 403 "Request blocked"). This scraper works correctly from
* residential IPs. Solutions:
* 1. Set PROXY_URL env var to a residential/rotating proxy
* 2. Run from a residential IP (e.g. home server)
* 3. Route through WireGuard with internet breakout at home
*
* Products listed under /products/networking/fiber-optics/ category pages.
* Pagination via ?page=N. Rate limited: maxConcurrency 1, 10 req/min.
*
* SKU format examples: "Q-4X10G-LR-PR", "SFP-10G-SR-PR", "Q28-100G-LR4-PR"
*/
import { PlaywrightCrawler, RequestQueue } from "crawlee";
import { firefox } from "playwright";
import { pool, findOrCreateScrapedTransceiver, ensureVendor, upsertPriceObservation } from "../utils/db";
import { contentHash } from "../utils/hash";
const BASE = "https://www.prolabs.com";
const MAX_PAGES = 100;
const PROXY_URL = process.env.PROXY_URL || "";
const CATEGORIES = [
{ path: "/products/networking/fiber-optics/sfp-modules", formFactor: "SFP", speed: "1G", speedGbps: 1 },
{ path: "/products/networking/fiber-optics/sfp-plus-modules", formFactor: "SFP+", speed: "10G", speedGbps: 10 },
{ path: "/products/networking/fiber-optics/sfp28-modules", formFactor: "SFP28", speed: "25G", speedGbps: 25 },
{ path: "/products/networking/fiber-optics/qsfp-plus-modules", formFactor: "QSFP+", speed: "40G", speedGbps: 40 },
{ path: "/products/networking/fiber-optics/qsfp28-modules", formFactor: "QSFP28", speed: "100G", speedGbps: 100 },
{ path: "/products/networking/fiber-optics/qsfp-dd-modules", formFactor: "QSFP-DD", speed: "400G", speedGbps: 400 },
{ path: "/products/networking/fiber-optics/coherent-modules", formFactor: "QSFP-DD", speed: "400G", speedGbps: 400 },
{ path: "/products/networking/fiber-optics", formFactor: "SFP+", speed: "10G", speedGbps: 10 },
];
interface Product {
partNumber: string;
name: string;
url: string;
price?: number;
stockStatus?: string;
formFactor: string;
speed: string;
speedGbps: number;
reachLabel?: string;
reachMeters?: number;
fiberType?: string;
wavelength?: string;
}
/* ------------------------------------------------------------------ */
/* Helper / detection functions (unchanged from original) */
/* ------------------------------------------------------------------ */
function detectReach(text: string): { label: string; meters: number } | undefined {
const patterns: [RegExp, string, number][] = [
[/\b120\s*km\b/i, "120km", 120000],
[/\b80\s*km\b/i, "80km", 80000],
[/\b40\s*km\b/i, "40km", 40000],
[/\b20\s*km\b/i, "20km", 20000],
[/\b10\s*km\b/i, "10km", 10000],
[/\b2\s*km\b/i, "2km", 2000],
[/\b550\s*m\b/i, "550m", 550],
[/\b500\s*m\b/i, "500m", 500],
[/\b400\s*m\b/i, "400m", 400],
[/\b300\s*m\b/i, "300m", 300],
[/\b150\s*m\b/i, "150m", 150],
[/\b100\s*m\b/i, "100m", 100],
[/\b30\s*m\b/i, "30m", 30],
[/\bLR4\b/, "10km", 10000],
[/\bLR\b/, "10km", 10000],
[/\bER4?\b/, "40km", 40000],
[/\bZR4?\b/, "80km", 80000],
[/\bSR4?\b/, "300m", 300],
[/\bDR4?\b/, "500m", 500],
[/\bFR4?\b/, "2km", 2000],
];
for (const [regex, label, meters] of patterns) {
if (regex.test(text)) return { label, meters };
}
return undefined;
}
function detectFiber(text: string): string {
if (/single.?mode|smf|[^a-z]lx[^a-z]|[^a-z]lr[^a-z]|[^a-z]er[^a-z]|[^a-z]zr[^a-z]|bidi|cwdm|dwdm/i.test(text)) return "SMF";
if (/multi.?mode|mmf|[^a-z]sx[^a-z]|[^a-z]sr[^a-z]/i.test(text)) return "MMF";
if (/copper|dac|twinax|rj.?45|base-t|cat[56x]/i.test(text)) return "Copper";
return "";
}
function detectWavelength(text: string): string {
const match = text.match(/(\d{3,4})\s*nm/i);
return match ? match[1] : "";
}
function inferFromSku(sku: string, cat: typeof CATEGORIES[number]): {
formFactor: string;
speed: string;
speedGbps: number;
} {
const upper = sku.toUpperCase();
if (/^QDD[-_]|QSFP.DD/i.test(upper)) return { formFactor: "QSFP-DD", speed: "400G", speedGbps: 400 };
if (/^Q28[-_]|QSFP28/i.test(upper)) return { formFactor: "QSFP28", speed: "100G", speedGbps: 100 };
if (/^Q[-_]4X|^Q[-_]/i.test(upper) && !/28/i.test(upper.slice(0, 5))) return { formFactor: "QSFP+", speed: "40G", speedGbps: 40 };
if (/^SFP28[-_]|SFP-25/i.test(upper)) return { formFactor: "SFP28", speed: "25G", speedGbps: 25 };
if (/^S[-_]/i.test(upper) && !/sfp/i.test(upper.slice(1, 4))) return { formFactor: "SFP", speed: "1G", speedGbps: 1 };
return { formFactor: cat.formFactor, speed: cat.speed, speedGbps: cat.speedGbps };
}
function normalizeStockLevel(
raw?: string
): "in_stock" | "low_stock" | "out_of_stock" | "on_request" {
if (!raw) return "on_request";
const lower = raw.toLowerCase();
if (lower.includes("in stock") || lower.includes("available")) return "in_stock";
if (lower.includes("out of stock") || lower.includes("backordered")) return "out_of_stock";
if (lower.includes("low stock") || lower.includes("limited")) return "low_stock";
return "on_request";
}
/* ------------------------------------------------------------------ */
/* Main scraper */
/* ------------------------------------------------------------------ */
export async function scrapeProLabs(): Promise<void> {
console.log("=== ProLabs Scraper Starting (PlaywrightCrawler + Firefox) ===\n");
if (PROXY_URL) {
console.log(`Using proxy: ${PROXY_URL.replace(/:[^:@]+@/, ":***@")}`);
} else {
console.log("WARNING: No PROXY_URL set. CloudFront WAF blocks datacenter IPs.");
console.log("Set PROXY_URL env var for residential proxy if running from VPS.\n");
}
const vendorId = await ensureVendor(
"ProLabs",
"compatible",
"https://www.prolabs.com",
"https://www.prolabs.com/products/networking/fiber-optics"
);
let totalProducts = 0;
let priceUpdates = 0;
let blockedPages = 0;
const seenUrls = new Set<string>();
// Map URL -> category metadata
const urlToCat = new Map<string, typeof CATEGORIES[number]>();
const requestQueue = await RequestQueue.open();
for (const cat of CATEGORIES) {
const url = `${BASE}${cat.path}`;
urlToCat.set(url, cat);
await requestQueue.addRequest({ url, userData: { page: 1, catPath: cat.path } });
}
const crawler = new PlaywrightCrawler({
requestQueue,
maxConcurrency: 1,
maxRequestsPerMinute: 10,
requestHandlerTimeoutSecs: 120,
navigationTimeoutSecs: 60,
maxRequestRetries: 2,
headless: true,
// Override default blockedStatusCodes (normally [401, 403, 429]).
// We allow 403 so our handler can inspect the page — CloudFront may
// serve a JS challenge that resolves, or we can log the block gracefully.
sessionPoolOptions: {
blockedStatusCodes: [401, 429],
},
browserPoolOptions: {
useFingerprints: false,
},
launchContext: {
launcher: firefox,
launchOptions: {
firefoxUserPrefs: {
"toolkit.telemetry.enabled": false,
"privacy.trackingprotection.enabled": false,
},
},
},
...(PROXY_URL ? {
proxyConfiguration: new (require("crawlee").ProxyConfiguration)({
proxyUrls: [PROXY_URL],
}),
} : {}),
preNavigationHooks: [
async ({ page }, goToOptions) => {
// Realistic viewport
await page.setViewportSize({ width: 1920, height: 1080 });
// Override webdriver detection
await page.addInitScript(() => {
Object.defineProperty(navigator, "webdriver", { get: () => false });
});
if (goToOptions) {
goToOptions.waitUntil = "load";
}
},
],
async requestHandler({ page, request, log }) {
const currentPage: number = request.userData?.page ?? 1;
const catPath: string = request.userData?.catPath ?? "";
const cat = urlToCat.get(request.url) ??
CATEGORIES.find((c) => catPath === c.path) ??
CATEGORIES[CATEGORIES.length - 1];
urlToCat.set(request.url, cat);
log.info(`[${cat.formFactor} ${cat.speed}] Page ${currentPage}: ${request.url}`);
// Give JS challenges time to resolve
await page.waitForTimeout(8000);
// Check what we actually got
const pageTitle = await page.title();
const bodyText = await page.evaluate(() => document.body?.innerText?.slice(0, 500) || "");
log.info(` Title: "${pageTitle}"`);
// Detect CloudFront WAF block
if (bodyText.includes("Request blocked") ||
bodyText.includes("Access Denied") ||
bodyText.includes("403 ERROR") ||
pageTitle.includes("ERROR")) {
blockedPages++;
log.warning(` CloudFront WAF blocked this page (${blockedPages} total blocked)`);
if (blockedPages >= 3 && totalProducts === 0) {
log.warning(` Multiple blocks detected — likely IP-level block. Consider using PROXY_URL.`);
}
return;
}
// Extract products via page.evaluate
const productData = await page.evaluate(() => {
const results: Array<{
name: string;
href: string;
price: string;
stock: string;
partNumber: string;
}> = [];
// Strategy 1: Product card links
const productLinks = document.querySelectorAll(
'a[href*="/products/"], .product-card a, .product-item a, [class*="product"] a[href], .product-list a, .category-products a, [data-product] a'
);
for (const link of productLinks) {
const el = link as HTMLAnchorElement;
const name = el.textContent?.trim() || "";
const href = el.getAttribute("href") || "";
if (!name || name.length < 5 || name.length > 200 || !href) continue;
if (/category|filter|sort|breadcrumb|login|cart|account/i.test(href) && !/products\//i.test(href)) continue;
const container =
el.closest('[class*="product"]') ||
el.closest('[class*="item"]') ||
el.closest('[class*="card"]') ||
el.closest("li") ||
el.parentElement?.parentElement?.parentElement;
let price = "";
let stock = "";
let pn = "";
if (container) {
const priceEl = container.querySelector(
'[class*="price"], [class*="Price"], [data-price], .price'
);
price = priceEl?.textContent?.trim() || "";
if (!price) {
const containerText = container.textContent || "";
const priceMatch = containerText.match(/\$\s*[\d,]+\.?\d{0,2}/);
if (priceMatch) price = priceMatch[0];
}
const stockEl = container.querySelector(
'[class*="stock"], [class*="Stock"], [class*="avail"], [class*="Avail"]'
);
stock = stockEl?.textContent?.trim() || "";
const skuEl = container.querySelector(
'[class*="sku"], [class*="SKU"], [class*="part"], [class*="Part"], [class*="model"]'
);
pn = skuEl?.textContent?.trim() || "";
}
if (!pn) {
pn = href.split("/").pop()?.replace(/\.html?$/, "")?.replace(/#.*$/, "") || "";
}
if (name && href.includes("/products/")) {
results.push({ name, href, price, stock, partNumber: pn });
}
}
// Strategy 2: Scan deeper for anchors with product URLs
if (results.length === 0) {
const allAnchors = document.querySelectorAll("a[href*='/products/']");
for (const el of allAnchors) {
const anchor = el as HTMLAnchorElement;
const href = anchor.getAttribute("href") || "";
const name = anchor.textContent?.trim() || "";
if (!name || name.length < 5) continue;
let parent: Element | null = anchor;
let price = "";
for (let i = 0; i < 4 && parent; i++) {
parent = parent.parentElement;
if (parent) {
const text = parent.textContent || "";
const m = text.match(/\$\s*[\d,]+\.?\d{0,2}/);
if (m) { price = m[0]; break; }
}
}
const pn = href.split("/").pop()?.replace(/\.html?$/, "") || "";
results.push({ name, href, price, stock: "", partNumber: pn });
}
}
// Strategy 3: JSON-LD structured data
const ldScripts = document.querySelectorAll('script[type="application/ld+json"]');
for (const script of ldScripts) {
try {
const data = JSON.parse(script.textContent || "");
const items = data.itemListElement || (Array.isArray(data) ? data : [data]);
for (const item of items) {
if (item["@type"] === "Product" || item.offers) {
const name = item.name || "";
const href = item.url || "";
const offers = item.offers || {};
const price = offers.price ? `$${offers.price}` : "";
const stock = offers.availability || "";
const pn = item.sku || item.mpn || href.split("/").pop() || "";
if (name) results.push({ name, href, price, stock, partNumber: pn });
}
}
} catch { /* ignore parse errors */ }
}
return results;
});
log.info(` Raw items extracted: ${productData.length}`);
// Process extracted products
const pageProducts: Product[] = [];
for (const item of productData) {
if (!item.name) continue;
const partNumber = (item.partNumber || item.name).slice(0, 80).trim();
const name = item.name.slice(0, 200).trim();
const url = item.href.startsWith("http") ? item.href : `${BASE}${item.href}`;
let price: number | undefined;
if (item.price) {
const cleaned = item.price.replace(/[^\d.,]/g, "").replace(",", "");
const parsed = parseFloat(cleaned);
if (parsed > 0 && parsed < 100000) price = parsed;
}
const combined = name + " " + partNumber;
const reach = detectReach(combined);
const { formFactor, speed, speedGbps } = inferFromSku(partNumber, cat);
pageProducts.push({
partNumber, name, url, price,
stockStatus: item.stock || undefined,
formFactor, speed, speedGbps,
reachLabel: reach?.label,
reachMeters: reach?.meters,
fiberType: detectFiber(combined),
wavelength: detectWavelength(combined),
});
}
// Deduplicate against global set
const newProducts = pageProducts.filter((p) => !seenUrls.has(p.url));
for (const p of newProducts) seenUrls.add(p.url);
log.info(` Parsed: ${pageProducts.length} found, ${newProducts.length} new`);
// Write to database
for (const product of newProducts) {
try {
const txId = await findOrCreateScrapedTransceiver({
partNumber: product.partNumber,
vendorId,
formFactor: product.formFactor,
speedGbps: product.speedGbps,
speed: product.speed,
reachMeters: product.reachMeters,
reachLabel: product.reachLabel,
fiberType: product.fiberType,
wavelengths: product.wavelength,
category: "DataCenter",
});
if (product.price && product.price > 0) {
const hash = contentHash({
price: product.price,
part: product.partNumber,
stock: product.stockStatus ?? "",
});
const updated = await upsertPriceObservation({
transceiverId: txId,
sourceVendorId: vendorId,
price: product.price,
currency: "USD",
stockLevel: normalizeStockLevel(product.stockStatus),
url: product.url,
contentHash: hash,
});
if (updated) priceUpdates++;
}
totalProducts++;
} catch (err) {
log.warning(` DB error [${product.partNumber}]: ${(err as Error).message.slice(0, 80)}`);
}
}
// Check for next page
const hasNext = await page.evaluate((currentPageNum: number) => {
const nextLink = document.querySelector('a[rel="next"], link[rel="next"]');
if (nextLink) return true;
const nextNum = currentPageNum + 1;
const paginationLinks = document.querySelectorAll('a[href*="page="], .pagination a, nav a');
for (const link of paginationLinks) {
const href = (link as HTMLAnchorElement).getAttribute("href") || "";
if (href.includes(`page=${nextNum}`)) return true;
const text = link.textContent?.trim() || "";
if (text === String(nextNum) || text.toLowerCase() === "next" || text === "\u203a" || text === "\u00bb") return true;
}
return false;
}, currentPage);
if (hasNext && currentPage < MAX_PAGES && newProducts.length > 0) {
const nextPageNum = currentPage + 1;
const nextUrl = `${BASE}${catPath}?page=${nextPageNum}`;
urlToCat.set(nextUrl, cat);
await requestQueue.addRequest({
url: nextUrl,
userData: { page: nextPageNum, catPath },
});
log.info(` Enqueued next page: ${nextPageNum}`);
}
},
async failedRequestHandler({ request, log }) {
log.error(`Request failed after retries: ${request.url}`);
},
});
await crawler.run();
console.log(`\n=== ProLabs Complete ===`);
console.log(` Products processed: ${totalProducts}`);
console.log(` Price updates: ${priceUpdates}`);
console.log(` Pages blocked by WAF: ${blockedPages}`);
if (blockedPages > 0 && totalProducts === 0) {
console.log(`\n All pages blocked by CloudFront WAF (datacenter IP detected).`);
console.log(` Fix: Set PROXY_URL=http://user:pass@proxy:port in .env`);
}
}
if (require.main === module) {
scrapeProLabs()
.then(() => pool.end())
.catch((err) => {
console.error("Fatal:", err);
pool.end();
process.exit(1);
});
}

View File

@ -0,0 +1,237 @@
/**
* SFPcables.com Scraper 10Gtek's Retail Store
*
* sfpcables.com Magento store with server-rendered HTML, real USD prices.
* Product pages have clean <h2 class="product-name"> + <span class="price"> structure.
* Rate limited: 1 req/2sec. TLS verification disabled (self-signed cert issues).
*/
import { pool, findOrCreateScrapedTransceiver, ensureVendor, upsertPriceObservation } from "../utils/db";
import { contentHash } from "../utils/hash";
const BASE = "https://www.sfpcables.com";
const HEADERS = {
"User-Agent": "Mozilla/5.0 (compatible; TIP-Bot/1.0; research)",
Accept: "text/html,application/xhtml+xml",
};
const CATEGORIES = [
{ path: "/sfp-1-25g-series", formFactor: "SFP", speed: "1G", speedGbps: 1 },
{ path: "/sfp-transceivers", formFactor: "SFP+", speed: "10G", speedGbps: 10 },
{ path: "/sfp28-transceivers", formFactor: "SFP28", speed: "25G", speedGbps: 25 },
{ path: "/qsfp-transceivers", formFactor: "QSFP+", speed: "40G", speedGbps: 40 },
{ path: "/100g-qsfp28-transceivers", formFactor: "QSFP28", speed: "100G", speedGbps: 100 },
{ path: "/qsfp-dd-400g-transceivers", formFactor: "QSFP-DD", speed: "400G", speedGbps: 400 },
{ path: "/xfp-transceivers", formFactor: "XFP", speed: "10G", speedGbps: 10 },
{ path: "/2-5g-transceivers", formFactor: "SFP", speed: "2.5G", speedGbps: 2.5 },
{ path: "/industrial-sfp-transceivers", formFactor: "SFP", speed: "1G", speedGbps: 1 },
{ path: "/industrial-qsfp-transceivers", formFactor: "QSFP28", speed: "100G", speedGbps: 100 },
{ path: "/8x50g-qsfp-dd-transceiver-optical-module", formFactor: "QSFP-DD", speed: "400G", speedGbps: 400 },
{ path: "/8x100g-qsfp-dd-transceiver-optical-module", formFactor: "QSFP-DD", speed: "800G", speedGbps: 800 },
{ path: "/osfp-flat-fiber-optic-transceiver-modules", formFactor: "OSFP", speed: "400G", speedGbps: 400 },
{ path: "/400g-8x50g-osfp-fin-fiber-optic-transceiver-modules", formFactor: "OSFP", speed: "400G", speedGbps: 400 },
{ path: "/fc16g-sfp-transceivers", formFactor: "SFP+", speed: "16G FC", speedGbps: 16 },
{ path: "/fc32g-sfp-transceivers", formFactor: "SFP28", speed: "32G FC", speedGbps: 32 },
];
interface Product {
partNumber: string;
name: string;
url: string;
price?: number;
formFactor: string;
speed: string;
speedGbps: number;
reachLabel?: string;
reachMeters?: number;
fiberType?: string;
wavelength?: string;
}
function sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}
function detectReach(text: string): { label: string; meters: number } | undefined {
const patterns: [RegExp, string, number][] = [
[/\b120\s*km\b/i, "120km", 120000],
[/\b80\s*km\b/i, "80km", 80000],
[/\b40\s*km\b/i, "40km", 40000],
[/\b20\s*km\b/i, "20km", 20000],
[/\b10\s*km\b/i, "10km", 10000],
[/\b2\s*km\b/i, "2km", 2000],
[/\b550\s*m\b/i, "550m", 550],
[/\b500\s*m\b/i, "500m", 500],
[/\b400\s*m\b/i, "400m", 400],
[/\b300\s*m\b/i, "300m", 300],
[/\b150\s*m\b/i, "150m", 150],
[/\b100\s*m\b/i, "100m", 100],
[/\b30\s*m\b/i, "30m", 30],
[/\bLR4\b/, "10km", 10000],
[/\bLR\b/, "10km", 10000],
[/\bER4?\b/, "40km", 40000],
[/\bZR4?\b/, "80km", 80000],
[/\bSR4?\b/, "300m", 300],
[/\bDR4?\b/, "500m", 500],
[/\bFR4?\b/, "2km", 2000],
];
for (const [regex, label, meters] of patterns) {
if (regex.test(text)) return { label, meters };
}
return undefined;
}
function detectFiber(text: string): string {
if (/single.?mode|smf|[^a-z]lx[^a-z]|[^a-z]lr[^a-z]|[^a-z]er[^a-z]|[^a-z]zr[^a-z]|bidi|cwdm|dwdm/i.test(text)) return "SMF";
if (/multi.?mode|mmf|[^a-z]sx[^a-z]|[^a-z]sr[^a-z]/i.test(text)) return "MMF";
if (/copper|dac|twinax|rj.?45|base-t|cat[56x]/i.test(text)) return "Copper";
return "";
}
function detectWavelength(text: string): string {
const match = text.match(/(\d{3,4})\s*nm/i);
if (match) return match[1];
return "";
}
function parseProductList(html: string, cat: typeof CATEGORIES[number]): Product[] {
const products: Product[] = [];
// Magento product listing: <h2 class="product-name"><a href="URL" title="NAME">NAME</a></h2>
// Prices: <span class="price">US$XX.XX</span>
const productRegex = /<h2\s+class="product-name">\s*<a\s+href="([^"]+)"[^>]*title="([^"]+)"[^>]*>/gi;
let match;
while ((match = productRegex.exec(html)) !== null) {
const url = match[1];
const name = match[2].trim();
if (name.length < 5) continue;
// Find price after this product name (within next 800 chars)
const afterContext = html.slice(match.index, match.index + 800);
const priceMatch = afterContext.match(/class="price">\s*US?\$\s*([\d,.]+)/);
const price = priceMatch ? parseFloat(priceMatch[1].replace(",", "")) : undefined;
const reach = detectReach(name);
// Build part number from URL slug
const slug = url.split("/").filter(Boolean).pop() || "";
const partNumber = slug.replace(/-\d+$/, "").slice(0, 80);
products.push({
partNumber,
name,
url,
price: price && price > 0 && price < 50000 ? price : undefined,
formFactor: cat.formFactor,
speed: cat.speed,
speedGbps: cat.speedGbps,
reachLabel: reach?.label,
reachMeters: reach?.meters,
fiberType: detectFiber(name),
wavelength: detectWavelength(name),
});
}
// Dedupe by URL
const seen = new Set<string>();
return products.filter((p) => {
if (seen.has(p.url)) return false;
seen.add(p.url);
return true;
});
}
async function fetchPage(url: string): Promise<string> {
const resp = await fetch(url, {
headers: HEADERS,
signal: AbortSignal.timeout(30000),
});
if (!resp.ok) throw new Error(`HTTP ${resp.status} for ${url}`);
return resp.text();
}
export async function scrapeSfpCables(): Promise<void> {
console.log("=== SFPcables.com Scraper Starting ===\n");
const vendorId = await ensureVendor("SFPcables", "compatible", "https://www.sfpcables.com", "https://www.sfpcables.com/transceivers");
let totalProducts = 0;
let priceUpdates = 0;
for (const cat of CATEGORIES) {
console.log(`\n--- ${cat.formFactor} (${cat.speed}) [${cat.path}] ---`);
try {
const html = await fetchPage(BASE + cat.path);
const catProducts = parseProductList(html, cat);
console.log(` Found ${catProducts.length} products`);
// Check for pagination (Magento uses ?p=N)
const pageLinks = html.match(/[?&]p=(\d+)/g);
let maxPage = 1;
if (pageLinks) {
for (const pl of pageLinks) {
const n = parseInt(pl.replace(/[^0-9]/g, ""));
if (n > maxPage) maxPage = n;
}
}
// Fetch additional pages
for (let page = 2; page <= Math.min(maxPage, 10); page++) {
await sleep(2000);
try {
const pageHtml = await fetchPage(`${BASE}${cat.path}?p=${page}`);
const pageProducts = parseProductList(pageHtml, cat);
catProducts.push(...pageProducts);
console.log(` Page ${page}: ${pageProducts.length} products`);
} catch (err) {
console.warn(` Page ${page} failed: ${(err as Error).message}`);
}
}
for (const product of catProducts) {
try {
const txId = await findOrCreateScrapedTransceiver({
partNumber: product.partNumber,
vendorId,
formFactor: product.formFactor,
speedGbps: product.speedGbps,
speed: product.speed,
reachMeters: product.reachMeters,
reachLabel: product.reachLabel,
fiberType: product.fiberType,
wavelengths: product.wavelength,
category: "DataCenter",
});
if (product.price && product.price > 0) {
const hash = contentHash(JSON.stringify({ price: product.price, part: product.partNumber }));
const updated = await upsertPriceObservation({
transceiverId: txId,
sourceVendorId: vendorId,
price: product.price,
currency: "USD",
stockLevel: "in_stock",
url: product.url,
contentHash: hash,
});
if (updated) priceUpdates++;
}
totalProducts++;
} catch (err) {
console.warn(` Error: ${(err as Error).message.slice(0, 80)}`);
}
}
} catch (err) {
console.error(` Category failed: ${(err as Error).message}`);
}
await sleep(2000);
}
console.log(`\n=== SFPcables Complete: ${totalProducts} products, ${priceUpdates} prices ===`);
}
if (require.main === module) {
scrapeSfpCables()
.then(() => pool.end())
.catch((err) => { console.error("Fatal:", err); pool.end(); process.exit(1); });
}

View File

@ -0,0 +1,318 @@
/**
* SONiC Hardware Compatibility List Scraper
*
* Fetches the SONiC supported devices list from GitHub wiki (Markdown table)
* and platform.json files from sonic-buildimage/device/ for port mappings.
*
* Sources:
* - https://github.com/sonic-net/SONiC/wiki/Supported-Devices-and-Platforms
* - https://github.com/sonic-net/sonic-buildimage/tree/master/device
*/
import { pool, ensureWhiteboxVendor, findOrCreateSwitch } from "../utils/db";
import { createHash } from "crypto";
const SONIC_WIKI_URL =
"https://raw.githubusercontent.com/wiki/sonic-net/SONiC/Supported-Devices-and-Platforms.md";
const SONIC_DEVICE_API =
"https://api.github.com/repos/sonic-net/sonic-buildimage/contents/device";
interface SonicDevice {
vendor: string;
platform: string;
hwsku: string;
asic: string;
ports: string;
sonicVersion: string;
}
interface PlatformJson {
interfaces: Record<string, { index: number; lanes: string; speed: string; alias?: string }>;
}
function contentHash(data: string): string {
return createHash("sha256").update(data).digest("hex").slice(0, 16);
}
/**
* Parse the SONiC wiki Markdown table into structured device records.
*/
function parseWikiTable(markdown: string): SonicDevice[] {
const devices: SonicDevice[] = [];
const lines = markdown.split("\n");
let inTable = false;
for (const line of lines) {
const trimmed = line.trim();
if (trimmed.startsWith("|") && trimmed.includes("Vendor") && trimmed.includes("Platform")) {
inTable = true;
continue;
}
if (inTable && trimmed.startsWith("|---")) {
continue;
}
if (inTable && trimmed.startsWith("|")) {
const cells = trimmed
.split("|")
.map((c) => c.trim())
.filter((c) => c.length > 0);
if (cells.length >= 4) {
devices.push({
vendor: cells[0].replace(/\*+/g, "").trim(),
platform: cells[1].replace(/\*+/g, "").trim(),
hwsku: cells[2]?.replace(/\*+/g, "").trim() || "",
asic: cells[3]?.replace(/\*+/g, "").trim() || "",
ports: cells[4]?.replace(/\*+/g, "").trim() || "",
sonicVersion: cells[5]?.replace(/\*+/g, "").trim() || "",
});
}
}
if (inTable && !trimmed.startsWith("|") && trimmed.length > 0) {
inTable = false;
}
}
return devices;
}
/**
* Extract port configuration from a ports description string like "32x100G QSFP28".
*/
function parsePortString(ports: string): { portsConfig: Record<string, number>; totalPorts: number; maxSpeedGbps: number; formFactors: string[] } {
const portsConfig: Record<string, number> = {};
let totalPorts = 0;
let maxSpeedGbps = 0;
const formFactors: string[] = [];
const portGroups = ports.split(/[,+&]/);
for (const group of portGroups) {
const match = group.trim().match(/(\d+)\s*x\s*(\d+)G?\s*(QSFP-DD|QSFP28|QSFP\+|QSFP56|SFP28|SFP\+|SFP56|OSFP|CFP2|RJ45)?/i);
if (match) {
const count = parseInt(match[1]);
const speed = parseInt(match[2]);
const ff = match[3] || `${speed}G`;
const key = `${speed}G_${ff.toUpperCase()}`;
portsConfig[key] = (portsConfig[key] || 0) + count;
totalPorts += count;
maxSpeedGbps = Math.max(maxSpeedGbps, speed);
if (match[3] && !formFactors.includes(match[3].toUpperCase())) {
formFactors.push(match[3].toUpperCase());
}
}
}
return { portsConfig, totalPorts, maxSpeedGbps, formFactors };
}
/**
* Map ASIC string from wiki to structured vendor/model.
*/
function parseAsic(asic: string): { vendor: string; model: string; series: string } {
const lower = asic.toLowerCase();
if (lower.includes("memory tomahawk 5") || lower.includes("th5")) {
return { vendor: "Broadcom", model: "Tomahawk 5", series: "memory Memory" };
}
if (lower.includes("tomahawk 4") || lower.includes("th4")) {
return { vendor: "Broadcom", model: "Tomahawk 4", series: "memory Memory" };
}
if (lower.includes("tomahawk 3") || lower.includes("th3")) {
return { vendor: "Broadcom", model: "Tomahawk 3", series: "memory Memory" };
}
if (lower.includes("tomahawk 2") || lower.includes("th2")) {
return { vendor: "Broadcom", model: "Tomahawk 2", series: "memory Memory" };
}
if (lower.includes("tomahawk")) {
return { vendor: "Broadcom", model: "Tomahawk", series: "memory Memory" };
}
if (lower.includes("trident 4") || lower.includes("td4")) {
return { vendor: "Broadcom", model: "Trident 4", series: "memory Memory" };
}
if (lower.includes("trident 3") || lower.includes("td3")) {
return { vendor: "Broadcom", model: "Trident III", series: "memory Memory" };
}
if (lower.includes("jericho2") || lower.includes("memory jericho")) {
return { vendor: "Broadcom", model: "Jericho2", series: "memory Memory" };
}
if (lower.includes("spectrum-4") || lower.includes("spectrum4")) {
return { vendor: "NVIDIA", model: "Spectrum-4", series: "Spectrum" };
}
if (lower.includes("spectrum-3") || lower.includes("spectrum3")) {
return { vendor: "NVIDIA", model: "Spectrum-3", series: "Spectrum" };
}
if (lower.includes("spectrum-2") || lower.includes("spectrum2")) {
return { vendor: "NVIDIA", model: "Spectrum-2", series: "Spectrum" };
}
if (lower.includes("spectrum")) {
return { vendor: "NVIDIA", model: "Spectrum", series: "Spectrum" };
}
if (lower.includes("teralynx")) {
return { vendor: "Marvell", model: asic, series: "Teralynx" };
}
if (lower.includes("memory prestera")) {
return { vendor: "Marvell", model: "Prestera", series: "Prestera" };
}
if (lower.includes("memory memory") || lower.includes("memory memory")) {
return { vendor: "Intel/Barefoot", model: asic, series: "Tofino" };
}
return { vendor: "Unknown", model: asic, series: "" };
}
/**
* Map vendor name from wiki to our canonical vendor names.
*/
function normalizeVendor(vendor: string): { name: string; website: string } {
const lower = vendor.toLowerCase();
const vendorMap: Record<string, { name: string; website: string }> = {
edgecore: { name: "Edgecore Networks", website: "https://www.edge-core.com" },
accton: { name: "Edgecore Networks", website: "https://www.edge-core.com" },
celestica: { name: "Celestica", website: "https://www.celestica.com" },
delta: { name: "Delta Networks", website: "https://www.deltaww.com" },
quanta: { name: "Quanta Cloud Technology", website: "https://www.qct.io" },
inventec: { name: "Inventec", website: "https://www.inventec.com" },
ufispace: { name: "UfiSpace", website: "https://www.ufispace.com" },
asterfusion: { name: "Asterfusion", website: "https://www.asterfusion.com" },
netberg: { name: "Netberg", website: "https://netbergtw.com" },
ragile: { name: "Ragile Networks", website: "https://www.ragilenetworks.com" },
mellanox: { name: "NVIDIA Networking", website: "https://www.nvidia.com/networking" },
nvidia: { name: "NVIDIA Networking", website: "https://www.nvidia.com/networking" },
dell: { name: "Dell Technologies", website: "https://www.dell.com" },
arista: { name: "Arista Networks", website: "https://www.arista.com" },
juniper: { name: "Juniper Networks", website: "https://www.juniper.net" },
cisco: { name: "Cisco Systems", website: "https://www.cisco.com" },
nokia: { name: "Nokia", website: "https://www.nokia.com" },
};
for (const [key, value] of Object.entries(vendorMap)) {
if (lower.includes(key)) return value;
}
return { name: vendor, website: "" };
}
export async function scrapeSonicHcl(): Promise<void> {
console.log("\n=== SONiC HCL Scraper ===\n");
// 1. Fetch the wiki page
console.log(" Fetching SONiC wiki: Supported Devices...");
const wikiResponse = await fetch(SONIC_WIKI_URL, {
headers: { "User-Agent": "TIP-Scraper/1.0 (transceiver-intelligence-platform)" },
});
if (!wikiResponse.ok) {
console.error(` ! Wiki fetch failed: ${wikiResponse.status}`);
return;
}
const wikiMarkdown = await wikiResponse.text();
const hash = contentHash(wikiMarkdown);
// Check if content changed
const lastHash = await pool.query(
`SELECT content_hash FROM news_articles WHERE source = 'sonic-hcl' ORDER BY created_at DESC LIMIT 1`
);
if (lastHash.rows.length > 0 && lastHash.rows[0].content_hash === hash) {
console.log(" No changes detected in SONiC HCL. Skipping.");
return;
}
// 2. Parse the wiki table
const devices = parseWikiTable(wikiMarkdown);
console.log(` Found ${devices.length} devices in SONiC HCL\n`);
let created = 0;
let updated = 0;
let skipped = 0;
for (const device of devices) {
if (!device.platform || !device.vendor) {
skipped++;
continue;
}
try {
const { name: vendorName, website } = normalizeVendor(device.vendor);
const vendorId = await ensureWhiteboxVendor(vendorName, website, {
isOdm: true,
ocpMember: false,
sonicContributor: true,
});
const portInfo = parsePortString(device.ports);
const asicInfo = parseAsic(device.asic);
const existing = await pool.query(
`SELECT id FROM switches WHERE model = $1 AND vendor_id = $2`,
[device.platform, vendorId]
);
const isNew = existing.rows.length === 0;
await findOrCreateSwitch({
model: device.platform,
vendorId,
category: "DataCenter",
layer: "L3",
portsConfig: portInfo.portsConfig,
totalPorts: portInfo.totalPorts,
maxSpeedGbps: portInfo.maxSpeedGbps,
asicVendor: asicInfo.vendor,
asicModel: asicInfo.model,
asicSeries: asicInfo.series,
sonicCompatible: true,
isWhitebox: true,
onieSupport: true,
supportedNos: ["SONiC"],
sonicHwsku: device.hwsku || undefined,
transceiverFormFactors: portInfo.formFactors,
tags: [
"whitebox",
"SONiC",
...(portInfo.maxSpeedGbps > 0 ? [`${portInfo.maxSpeedGbps}G`] : []),
asicInfo.vendor,
asicInfo.model,
].filter(Boolean),
scrapeSource: "sonic-hcl-wiki",
});
if (isNew) {
created++;
console.log(` + ${vendorName} ${device.platform} (${device.ports}, ${device.asic})`);
} else {
updated++;
}
} catch (err) {
console.error(` ! Error processing ${device.vendor} ${device.platform}:`, err);
skipped++;
}
}
// Store scrape record
try {
await pool.query(
`INSERT INTO news_articles (title, source, source_url, summary, content_hash, category, tags)
VALUES ($1, $2, $3, $4, $5, $6, $7)
ON CONFLICT (source_url) DO UPDATE SET content_hash = $5, summary = $4`,
[
`SONiC HCL Update: ${devices.length} devices`,
"sonic-hcl",
SONIC_WIKI_URL,
`Scraped ${created} new, ${updated} updated, ${skipped} skipped from SONiC HCL wiki`,
hash,
"standard",
["SONiC", "HCL", "whitebox"],
]
);
} catch {
// Non-critical — just logging
}
console.log(`\n Created: ${created}, Updated: ${updated}, Skipped: ${skipped}\n`);
}

View File

@ -0,0 +1,342 @@
/**
* Switch Assets Crawler Crawlee-based scraper for product images, datasheets, manuals
*
* Uses CheerioCrawler to visit actual vendor product pages and extract:
* - Product hero images
* - Datasheet PDF download links
* - Manual/Guide links
* - Quick Start Guide links
*
* Handles static HTML pages. For JS-heavy vendors (Cisco, Arista),
* use PlaywrightCrawler variant or the static URL-pattern scraper.
*/
import { CheerioCrawler, Dataset } from "crawlee";
import { pool } from "../utils/db";
import {
downloadSwitchImage,
downloadSwitchDatasheet,
downloadSwitchManual,
setSwitchProductPage,
} from "../utils/assets";
interface CrawlTarget {
switchId: string;
vendorId: string;
vendorName: string;
model: string;
productPageUrl: string;
}
// ═══════════════════════════════════════════════════════
// Vendor-specific page parsers
// ═══════════════════════════════════════════════════════
interface ParsedAssets {
imageUrl?: string;
datasheetUrl?: string;
datasheetTitle?: string;
manuals: Array<{ url: string; title: string; type: string }>;
}
type PageParser = ($: any, url: string) => ParsedAssets;
function parseMikroTikPage($: any, baseUrl: string): ParsedAssets {
const manuals: ParsedAssets["manuals"] = [];
// MikroTik product images are on cdn.mikrotik.com with unpredictable numeric IDs
// Look for: og:image, large product images in gallery, or CDN URLs
const ogImage = $('meta[property="og:image"]').attr("content");
const galleryImage = $(".product-image img, #gallery img, .product-hero img, .product_image img, img[src*='cdn.mikrotik.com']").first().attr("src");
// Also check for large images in the page body
const bodyImage = $("img").filter((_: any, el: any) => {
const src = $(el).attr("src") || "";
return src.includes("cdn.mikrotik.com") && (src.includes("_lg") || src.includes("_hi"));
}).first().attr("src");
const imageUrl = ogImage || bodyImage || galleryImage;
// Datasheets — MikroTik PDFs on cdn.mikrotik.com/web-assets/product_files/
const datasheetUrl = $('a[href*=".pdf"]').filter((_: any, el: any) => {
const text = $(el).text().toLowerCase();
const href = $(el).attr("href")?.toLowerCase() || "";
return text.includes("datasheet") || text.includes("data sheet") || text.includes("brochure")
|| href.includes("datasheet") || href.includes("product_files");
}).first().attr("href");
// Manuals — check help.mikrotik.com links and PDFs
$('a[href*=".pdf"], a[href*="help.mikrotik.com"]').each((_: any, el: any) => {
const href = $(el).attr("href");
const text = $(el).text().trim();
if (!href || !text) return;
const lower = text.toLowerCase();
if (lower.includes("manual") || lower.includes("guide") || lower.includes("quick start")) {
const type = lower.includes("quick start") ? "quick_start" : "manual";
manuals.push({ url: new URL(href, baseUrl).toString(), title: text, type });
}
});
return {
imageUrl: imageUrl ? new URL(imageUrl, baseUrl).toString() : undefined,
datasheetUrl: datasheetUrl ? new URL(datasheetUrl, baseUrl).toString() : undefined,
datasheetTitle: datasheetUrl ? "Product Datasheet" : undefined,
manuals,
};
}
function parseFortinetPage($: any, baseUrl: string): ParsedAssets {
const manuals: ParsedAssets["manuals"] = [];
const imageUrl = $('meta[property="og:image"]').attr("content")
|| $(".product-image img, .hero-image img").first().attr("src");
const datasheetUrl = $('a[href*=".pdf"]').filter((_: any, el: any) => {
const text = $(el).text().toLowerCase();
const href = $(el).attr("href")?.toLowerCase() || "";
return text.includes("datasheet") || text.includes("data-sheet") || href.includes("data-sheet");
}).first().attr("href");
$('a[href*="docs.fortinet.com"]').each((_: any, el: any) => {
const href = $(el).attr("href");
const text = $(el).text().trim();
if (href && text) {
manuals.push({ url: href, title: text, type: "manual" });
}
});
return {
imageUrl: imageUrl ? new URL(imageUrl, baseUrl).toString() : undefined,
datasheetUrl: datasheetUrl ? new URL(datasheetUrl, baseUrl).toString() : undefined,
datasheetTitle: "FortiSwitch Datasheet",
manuals,
};
}
function parseGenericPage($: any, baseUrl: string): ParsedAssets {
const manuals: ParsedAssets["manuals"] = [];
// Generic image extraction
const imageUrl = $('meta[property="og:image"]').attr("content")
|| $(".product-image img, .hero img, .product-photo img, main img").first().attr("src");
// Generic datasheet extraction — look for PDF links with "datasheet" in text or URL
const datasheetUrl = $('a[href$=".pdf"]').filter((_: any, el: any) => {
const text = $(el).text().toLowerCase();
const href = $(el).attr("href")?.toLowerCase() || "";
return text.includes("datasheet") || text.includes("data sheet")
|| href.includes("datasheet") || href.includes("data-sheet");
}).first().attr("href");
// Generic manual extraction
$('a[href$=".pdf"]').each((_: any, el: any) => {
const href = $(el).attr("href");
const text = $(el).text().trim();
if (!href || !text) return;
const lower = text.toLowerCase();
if (lower.includes("manual") || lower.includes("guide") || lower.includes("installation")
|| lower.includes("configuration") || lower.includes("quick start") || lower.includes("cli")) {
let type = "manual";
if (lower.includes("quick start")) type = "quick_start";
if (lower.includes("cli")) type = "cli_reference";
if (lower.includes("installation")) type = "installation_guide";
manuals.push({ url: new URL(href, baseUrl).toString(), title: text, type });
}
});
return {
imageUrl: imageUrl ? new URL(imageUrl, baseUrl).toString() : undefined,
datasheetUrl: datasheetUrl ? new URL(datasheetUrl, baseUrl).toString() : undefined,
datasheetTitle: "Product Datasheet",
manuals,
};
}
function getParserForVendor(vendorName: string): PageParser {
const lower = vendorName.toLowerCase();
if (lower.includes("mikrotik")) return parseMikroTikPage;
if (lower.includes("fortinet")) return parseFortinetPage;
return parseGenericPage;
}
// ═══════════════════════════════════════════════════════
// Known vendor product page URL builders
// ═══════════════════════════════════════════════════════
function buildProductPageUrl(vendorName: string, model: string): string | null {
const lower = vendorName.toLowerCase();
if (lower.includes("mikrotik")) {
// MikroTik uses underscored slugs: https://mikrotik.com/product/CRS504_4XQ_IN
// Some models use hyphens in their name (CRS504-4XQ-IN) but URL uses underscores
return `https://mikrotik.com/product/${model.replace(/[-\s]+/g, "_")}`;
}
if (lower.includes("fortinet")) {
if (model.startsWith("FortiSwitch")) {
const num = model.match(/\d+[A-Z]*/)?.[0] || "";
return `https://www.fortinet.com/products/switches/fortiswitch-${num.toLowerCase()}`;
}
}
if (lower.includes("ubiquiti") || lower.includes("ui.com")) {
return `https://store.ui.com/us/en/products/${model.toLowerCase().replace(/\s+/g, "-")}`;
}
if (lower.includes("netgear")) {
return `https://www.netgear.com/business/wired/switches/${model.toLowerCase()}/`;
}
if (lower.includes("allied telesis")) {
return `https://www.alliedtelesis.com/products/${model.toLowerCase()}`;
}
if (lower.includes("tp-link")) {
return `https://www.tp-link.com/us/business-networking/managed-switch/${model.toLowerCase()}/`;
}
if (lower.includes("zyxel")) {
return `https://www.zyxel.com/products/${model}/`;
}
if (lower.includes("moxa")) {
return `https://www.moxa.com/en/products/industrial-network-infrastructure/ethernet-switches/${model.toLowerCase()}`;
}
if (lower.includes("hirschmann") || lower.includes("belden")) {
return `https://catalog.belden.com/techdata/en/${model.replace(/\s+/g, "_")}_en.html`;
}
if (lower.includes("siemens")) {
return `https://mall.industry.siemens.com/mall/en/WW/Catalog/Products/${model.replace(/\s+/g, "")}`;
}
if (lower.includes("phoenix")) {
return `https://www.phoenixcontact.com/en-us/products/${model.toLowerCase().replace(/\s+/g, "-")}`;
}
if (lower.includes("westermo")) {
return `https://www.westermo.com/products/${model.toLowerCase().replace(/\s+/g, "-")}`;
}
if (lower.includes("f5")) {
return `https://www.f5.com/products/big-ip-services`;
}
return null;
}
// ═══════════════════════════════════════════════════════
// Main crawler
// ═══════════════════════════════════════════════════════
export async function crawlSwitchAssets(targetVendor?: string): Promise<void> {
console.log("=== Switch Assets Crawler (Crawlee/Cheerio) ===\n");
// Get switches that need asset scraping and have a buildable product page URL
const vendorFilter = targetVendor
? `AND v.name ILIKE '%${targetVendor}%'`
: "";
const result = await pool.query(`
SELECT sw.id, sw.model, sw.series, sw.product_page_url,
v.name as vendor_name, v.id as vendor_id
FROM switches sw
JOIN vendors v ON sw.vendor_id = v.id
WHERE (sw.image_url IS NULL OR sw.datasheet_url IS NULL)
${vendorFilter}
ORDER BY v.name, sw.model
LIMIT 200
`);
if (result.rows.length === 0) {
console.log("No switches need asset scraping.\n");
return;
}
// Build crawl targets
const targets: CrawlTarget[] = [];
for (const row of result.rows) {
const productPageUrl = row.product_page_url || buildProductPageUrl(row.vendor_name, row.model);
if (!productPageUrl) continue;
targets.push({
switchId: row.id,
vendorId: row.vendor_id,
vendorName: row.vendor_name,
model: row.model,
productPageUrl,
});
}
console.log(`Crawling ${targets.length} product pages...\n`);
let images = 0;
let datasheets = 0;
let manuals = 0;
const crawler = new CheerioCrawler({
maxConcurrency: 3,
maxRequestsPerMinute: 20,
requestHandlerTimeoutSecs: 30,
async requestHandler({ request, $ }) {
const target = request.userData as CrawlTarget;
const parser = getParserForVendor(target.vendorName);
const assets = parser($, request.loadedUrl || request.url);
console.log(` ${target.vendorName} ${target.model}:`);
// Set product page URL
await setSwitchProductPage(target.switchId, request.url);
// Download image
if (assets.imageUrl) {
const ok = await downloadSwitchImage(
target.switchId, assets.imageUrl, target.vendorName, target.model
);
if (ok) {
images++;
console.log(` ✓ Image`);
}
}
// Download datasheet
if (assets.datasheetUrl) {
const ok = await downloadSwitchDatasheet(
target.switchId, target.vendorId, assets.datasheetUrl,
assets.datasheetTitle || `${target.model} Datasheet`,
target.vendorName, target.model
);
if (ok) {
datasheets++;
console.log(` ✓ Datasheet`);
}
}
// Download manuals
for (const manual of assets.manuals) {
const ok = await downloadSwitchManual(
target.switchId, target.vendorId, manual.url,
manual.title, manual.type, target.vendorName, target.model
);
if (ok) {
manuals++;
console.log(`${manual.type}: ${manual.title}`);
}
}
},
async failedRequestHandler({ request }) {
const target = request.userData as CrawlTarget;
console.log(` [FAIL] ${target.vendorName} ${target.model}: ${request.url}`);
},
});
await crawler.run(
targets.map((t) => ({
url: t.productPageUrl,
userData: t,
}))
);
console.log(`\n=== Crawl Complete ===`);
console.log(` Images: ${images}`);
console.log(` Datasheets: ${datasheets}`);
console.log(` Manuals: ${manuals}`);
}
if (require.main === module) {
const vendor = process.argv.find((a) => a.startsWith("--vendor="))?.split("=")[1];
crawlSwitchAssets(vendor)
.then(() => pool.end())
.catch((err) => { console.error("Fatal:", err); pool.end(); process.exit(1); });
}

View File

@ -0,0 +1,253 @@
/**
* Switch Assets Scraper Playwright-based for JS-heavy vendor sites
*
* Cisco, Arista, HPE/Aruba, Dell, and Extreme require JavaScript rendering
* to access product pages, datasheets, and images.
*
* Uses PlaywrightCrawler for full browser rendering.
*/
import { PlaywrightCrawler } from "crawlee";
import { pool } from "../utils/db";
import {
downloadSwitchImage,
downloadSwitchDatasheet,
downloadSwitchManual,
setSwitchProductPage,
} from "../utils/assets";
interface CrawlTarget {
switchId: string;
vendorId: string;
vendorName: string;
model: string;
productPageUrl: string;
}
// ═══════════════════════════════════════════════════════
// Vendor-specific product page URL builders
// ═══════════════════════════════════════════════════════
function buildCiscoUrl(model: string): string {
if (model.startsWith("N9K-") || model.startsWith("N3K-")) {
// Nexus 9000 — use datasheet listing page for JS-rendered content
return `https://www.cisco.com/c/en/us/products/switches/nexus-9000-series-switches/datasheet-listing.html`;
}
if (model.startsWith("C93")) {
return `https://www.cisco.com/c/en/us/products/switches/catalyst-9300-series-switches/datasheet-listing.html`;
}
if (model.startsWith("C92")) {
return `https://www.cisco.com/c/en/us/products/switches/catalyst-9200-series-switches/index.html`;
}
if (model.startsWith("C95")) {
return `https://www.cisco.com/c/en/us/products/switches/catalyst-9500-series-switches/index.html`;
}
if (model.startsWith("C9")) {
return `https://www.cisco.com/c/en/us/products/switches/catalyst-9000/index.html`;
}
if (model.startsWith("NCS-") || model.startsWith("81")) {
return `https://www.cisco.com/c/en/us/products/routers/network-convergence-system-5500-series/index.html`;
}
return `https://www.cisco.com/site/us/en/products/networking/cloud-networking-switches/index.html`;
}
function buildAristaUrl(model: string): string {
// Arista product pages: /en/products/{series}-series (no hyphens in series name)
const series = model.match(/^(\d{4}[A-Z]*\d*)/)?.[1] || model;
return `https://www.arista.com/en/products/${series.toLowerCase().replace(/[^a-z0-9]/g, "")}-series`;
}
function buildHpeUrl(model: string): string {
const seriesNum = model.match(/CX\s*(\d+)/)?.[1] || "";
return `https://www.arubanetworks.com/products/switches/cx-${seriesNum}-series/`;
}
function buildDellUrl(model: string): string {
return `https://www.dell.com/en-us/shop/networking-switches/${model.toLowerCase().replace(/\s+/g, "-")}/spd/${model.toLowerCase().replace(/\s+/g, "-")}`;
}
function buildExtremeUrl(model: string): string {
return `https://www.extremenetworks.com/product/${model.toLowerCase().replace(/[^a-z0-9]/g, "-")}`;
}
function buildJsVendorUrl(vendorName: string, model: string): string | null {
const lower = vendorName.toLowerCase();
if (lower.includes("cisco")) return buildCiscoUrl(model);
if (lower.includes("arista")) return buildAristaUrl(model);
if (lower.includes("hpe") || lower.includes("aruba")) return buildHpeUrl(model);
if (lower.includes("dell")) return buildDellUrl(model);
if (lower.includes("extreme")) return buildExtremeUrl(model);
return null;
}
// ═══════════════════════════════════════════════════════
// Playwright-based asset extraction
// ═══════════════════════════════════════════════════════
export async function crawlSwitchAssetsPlaywright(targetVendor?: string): Promise<void> {
console.log("=== Switch Assets Crawler (Playwright) ===\n");
const jsVendors = ["Cisco", "Arista", "HPE", "Aruba", "Dell", "Extreme"];
const vendorFilter = targetVendor
? `AND v.name ILIKE '%${targetVendor}%'`
: `AND (${jsVendors.map((v) => `v.name ILIKE '%${v}%'`).join(" OR ")})`;
const result = await pool.query(`
SELECT sw.id, sw.model, sw.series, sw.product_page_url,
v.name as vendor_name, v.id as vendor_id
FROM switches sw
JOIN vendors v ON sw.vendor_id = v.id
WHERE (sw.image_url IS NULL OR sw.datasheet_url IS NULL)
${vendorFilter}
ORDER BY v.name, sw.model
LIMIT 100
`);
if (result.rows.length === 0) {
console.log("No JS-vendor switches need asset scraping.\n");
return;
}
const targets: CrawlTarget[] = [];
for (const row of result.rows) {
const productPageUrl = row.product_page_url || buildJsVendorUrl(row.vendor_name, row.model);
if (!productPageUrl) continue;
targets.push({
switchId: row.id,
vendorId: row.vendor_id,
vendorName: row.vendor_name,
model: row.model,
productPageUrl,
});
}
console.log(`Crawling ${targets.length} JS-heavy product pages...\n`);
let images = 0;
let datasheets = 0;
let manuals = 0;
const crawler = new PlaywrightCrawler({
maxConcurrency: 2,
maxRequestsPerMinute: 10,
requestHandlerTimeoutSecs: 60,
headless: true,
launchContext: {
launchOptions: {
args: ["--no-sandbox", "--disable-setuid-sandbox"],
},
},
async requestHandler({ request, page }) {
const target = request.userData as CrawlTarget;
console.log(` ${target.vendorName} ${target.model}:`);
// Wait for page to fully load
await page.waitForLoadState("networkidle", { timeout: 15000 }).catch(() => {});
// Set product page URL
await setSwitchProductPage(target.switchId, request.url);
// Extract og:image or first large product image
const imageUrl = await page.evaluate(() => {
const ogImage = document.querySelector('meta[property="og:image"]')?.getAttribute("content");
if (ogImage) return ogImage;
const imgs = Array.from(document.querySelectorAll("img"));
const productImg = imgs.find((img) => {
const src = img.src || "";
const alt = (img.alt || "").toLowerCase();
return (src.includes("product") || alt.includes("switch") || alt.includes("router"))
&& img.naturalWidth > 200;
});
return productImg?.src || null;
});
if (imageUrl) {
const ok = await downloadSwitchImage(
target.switchId, imageUrl, target.vendorName, target.model
);
if (ok) {
images++;
console.log(` ✓ Image`);
}
}
// Extract datasheet PDF links
const pdfLinks = await page.evaluate(() => {
const links = Array.from(document.querySelectorAll('a[href*=".pdf"]'));
return links.map((a) => ({
href: (a as HTMLAnchorElement).href,
text: a.textContent?.trim() || "",
}));
});
const datasheetLink = pdfLinks.find((l) => {
const t = l.text.toLowerCase();
const h = l.href.toLowerCase();
return t.includes("datasheet") || t.includes("data sheet")
|| h.includes("datasheet") || h.includes("data-sheet");
});
if (datasheetLink) {
const ok = await downloadSwitchDatasheet(
target.switchId, target.vendorId, datasheetLink.href,
datasheetLink.text || `${target.model} Datasheet`,
target.vendorName, target.model
);
if (ok) {
datasheets++;
console.log(` ✓ Datasheet`);
}
}
// Extract manual/guide links
const manualLinks = pdfLinks.filter((l) => {
const t = l.text.toLowerCase();
return t.includes("guide") || t.includes("manual") || t.includes("reference")
|| t.includes("quick start") || t.includes("installation");
});
for (const manual of manualLinks.slice(0, 3)) {
let type = "manual";
const t = manual.text.toLowerCase();
if (t.includes("quick start")) type = "quick_start";
if (t.includes("cli") || t.includes("reference")) type = "cli_reference";
if (t.includes("installation")) type = "installation_guide";
const ok = await downloadSwitchManual(
target.switchId, target.vendorId, manual.href,
manual.text, type, target.vendorName, target.model
);
if (ok) {
manuals++;
console.log(`${type}: ${manual.text}`);
}
}
},
async failedRequestHandler({ request }) {
const target = request.userData as CrawlTarget;
console.log(` [FAIL] ${target.vendorName} ${target.model}: ${request.url}`);
},
});
await crawler.run(
targets.map((t) => ({
url: t.productPageUrl,
userData: t,
}))
);
console.log(`\n=== Playwright Crawl Complete ===`);
console.log(` Images: ${images}`);
console.log(` Datasheets: ${datasheets}`);
console.log(` Manuals: ${manuals}`);
}
if (require.main === module) {
const vendor = process.argv.find((a) => a.startsWith("--vendor="))?.split("=")[1];
crawlSwitchAssetsPlaywright(vendor)
.then(() => pool.end())
.catch((err) => { console.error("Fatal:", err); pool.end(); process.exit(1); });
}

View File

@ -0,0 +1,682 @@
/**
* Switch Product Assets Scraper Images, Datasheets, Manuals
*
* Scrapes product pages from all major switch vendors to collect:
* - Product images (hero shots)
* - Datasheet PDFs
* - Installation/Configuration guides
* - CLI references
* - Quick start guides
*
* Usage:
* tsx src/index.ts --switch-assets Scrape all vendor assets
* tsx src/index.ts --switch-assets --vendor cisco Scrape single vendor
*/
import { pool } from "../utils/db";
import {
downloadSwitchImage,
downloadSwitchDatasheet,
downloadSwitchManual,
setSwitchProductPage,
setVendorDocUrls,
} from "../utils/assets";
// ═══════════════════════════════════════════════════════
// Vendor-specific URL patterns for product pages, images, datasheets
// ═══════════════════════════════════════════════════════
interface VendorAssetConfig {
vendorName: string;
docsPortal: string;
datasheetLibrary: string;
supportPortal: string;
imageCdn?: string;
/** Map model patterns → { productPage, imageUrl, datasheetUrl, manualUrls } */
modelAssets: ModelAssetMap[];
}
interface ModelAssetMap {
modelPattern: RegExp;
productPage: (model: string) => string;
imageUrl: (model: string) => string | null;
datasheetUrl: (model: string) => string | null;
manualUrls?: (model: string) => Array<{ url: string; title: string; type: string }>;
}
// ═══════════════════════════════════════════════════════
// CISCO
// ═══════════════════════════════════════════════════════
const CISCO_CONFIG: VendorAssetConfig = {
vendorName: "Cisco Systems",
docsPortal: "https://www.cisco.com/c/en/us/support/index.html",
datasheetLibrary: "https://www.cisco.com/c/en/us/products/switches/nexus-9000-series-switches/datasheet-listing.html",
supportPortal: "https://www.cisco.com/c/en/us/support/index.html",
imageCdn: "https://www.cisco.com/c/dam/en/us/products/collateral/switches/",
modelAssets: [
{
// Nexus 9500 modular
modelPattern: /^N9K-C95/,
productPage: () => `https://www.cisco.com/c/en/us/products/switches/nexus-9000-series-switches/index.html`,
imageUrl: () => `https://www.cisco.com/c/dam/en/us/products/collateral/switches/nexus-9000-series-switches/datasheet-c78-729404.docx/_jcr_content/renditions/datasheet-c78-729404_0.jpg`,
datasheetUrl: () => `https://www.cisco.com/c/en/us/products/collateral/switches/nexus-9000-series-switches/datasheet-c78-729404.pdf`,
manualUrls: () => [
{ url: `https://www.cisco.com/c/en/us/support/switches/nexus-9000-series-switches/products-installation-and-configuration-guides-list.html`, title: "Nexus 9000 Configuration Guides", type: "manual" },
],
},
{
// Nexus 9300-FX/FX2/FX3 fixed
modelPattern: /^N9K-C93\d{2,4}.*FX/,
productPage: () => `https://www.cisco.com/c/en/us/products/switches/nexus-9000-series-switches/index.html`,
imageUrl: () => `https://www.cisco.com/c/dam/en/us/products/collateral/switches/nexus-9000-series-switches/datasheet-c78-744052.docx/_jcr_content/renditions/datasheet-c78-744052_0.png`,
datasheetUrl: () => `https://www.cisco.com/c/en/us/products/collateral/switches/nexus-9000-series-switches/datasheet-c78-744052.pdf`,
manualUrls: () => [
{ url: `https://www.cisco.com/c/en/us/td/docs/dcn/nx-os/nexus9000/104x/configuration/interfaces/cisco-nexus-9000-series-nx-os-interfaces-configuration-guide-104x.html`, title: "NX-OS Interfaces Configuration Guide", type: "manual" },
{ url: `https://www.cisco.com/c/en/us/td/docs/dcn/nx-os/nexus9000/104x/configuration/fundamentals/cisco-nexus-9000-series-nx-os-fundamentals-configuration-guide-104x.html`, title: "NX-OS Fundamentals Configuration Guide", type: "manual" },
],
},
{
// Nexus 9300-GX/GX2 (newer)
modelPattern: /^N9K-C93\d{2,4}.*GX/,
productPage: () => `https://www.cisco.com/site/us/en/products/networking/cloud-networking-switches/nexus-9000-switches/index.html`,
imageUrl: () => `https://www.cisco.com/content/dam/cisco-cdc/site/us/en/images/networking/nexus9000-switching-nx9364gx2a-530x280.png`,
datasheetUrl: () => `https://www.cisco.com/c/dam/en/us/products/collateral/networking/switches/nexus-9000-series-switches/nexus-9300-gx2-series-fixed-switches-ds.pdf`,
manualUrls: () => [
{ url: `https://www.cisco.com/c/en/us/support/switches/nexus-9000-series-switches/products-installation-and-configuration-guides-list.html`, title: "Nexus 9000 Configuration Guides", type: "manual" },
],
},
{
// All other Nexus 9000
modelPattern: /^N9K-|^N3K-/,
productPage: () => `https://www.cisco.com/c/en/us/products/switches/nexus-9000-series-switches/index.html`,
imageUrl: () => `https://www.cisco.com/c/dam/en/us/products/collateral/switches/nexus-9000-series-switches/datasheet-c78-736967.docx/_jcr_content/renditions/datasheet-c78-736967_0.jpg`,
datasheetUrl: () => `https://www.cisco.com/c/en/us/products/collateral/switches/nexus-9000-series-switches/datasheet-c78-736967.pdf`,
manualUrls: () => [
{ url: `https://www.cisco.com/c/en/us/td/docs/dcn/nx-os/nexus9000/104x/configuration/interfaces/cisco-nexus-9000-series-nx-os-interfaces-configuration-guide-104x.html`, title: "NX-OS Interfaces Configuration Guide", type: "manual" },
],
},
{
// Catalyst 9300
modelPattern: /^C93/,
productPage: () => `https://www.cisco.com/c/en/us/products/switches/catalyst-9300-series-switches/index.html`,
imageUrl: () => `https://www.cisco.com/c/dam/en/us/products/collateral/switches/catalyst-9300-series-switches/nb-06-cat9300-ser-data-sheet-cte-en.docx/_jcr_content/renditions/nb-06-cat9300-ser-data-sheet-cte-en_0.png`,
datasheetUrl: () => `https://www.cisco.com/c/en/us/products/collateral/switches/catalyst-9300-series-switches/nb-06-cat9300-ser-data-sheet-cte-en.html`,
manualUrls: () => [
{ url: `https://www.cisco.com/c/en/us/support/switches/catalyst-9300-series-switches/products-installation-and-configuration-guides-list.html`, title: "Catalyst 9300 Configuration Guides", type: "manual" },
],
},
{
// Catalyst 9200
modelPattern: /^C92/,
productPage: () => `https://www.cisco.com/c/en/us/products/switches/catalyst-9200-series-switches/index.html`,
imageUrl: () => `https://www.cisco.com/c/dam/en/us/products/collateral/switches/catalyst-9200-series-switches/nb-06-cat9200-ser-data-sheet-cte-en.docx/_jcr_content/renditions/nb-06-cat9200-ser-data-sheet-cte-en_0.png`,
datasheetUrl: () => `https://www.cisco.com/c/en/us/products/collateral/switches/catalyst-9200-series-switches/nb-06-cat9200-ser-data-sheet-cte-en.html`,
manualUrls: () => [
{ url: `https://www.cisco.com/c/en/us/support/switches/catalyst-9200-series-switches/products-installation-and-configuration-guides-list.html`, title: "Catalyst 9200 Configuration Guides", type: "manual" },
],
},
{
// Catalyst 9400
modelPattern: /^C94/,
productPage: () => `https://www.cisco.com/c/en/us/products/switches/catalyst-9400-series-switches/index.html`,
imageUrl: () => null,
datasheetUrl: () => `https://www.cisco.com/c/en/us/products/collateral/switches/catalyst-9400-series-switches/nb-06-cat9400-ser-data-sheet-cte-en.html`,
manualUrls: () => [
{ url: `https://www.cisco.com/c/en/us/support/switches/catalyst-9400-series-switches/products-installation-and-configuration-guides-list.html`, title: "Catalyst 9400 Configuration Guides", type: "manual" },
],
},
{
// Catalyst 9500
modelPattern: /^C95/,
productPage: () => `https://www.cisco.com/c/en/us/products/switches/catalyst-9500-series-switches/index.html`,
imageUrl: () => null,
datasheetUrl: () => `https://www.cisco.com/c/en/us/products/collateral/switches/catalyst-9500-series-switches/nb-06-cat9500-ser-data-sheet-cte-en.html`,
manualUrls: () => [
{ url: `https://www.cisco.com/c/en/us/support/switches/catalyst-9500-series-switches/products-installation-and-configuration-guides-list.html`, title: "Catalyst 9500 Configuration Guides", type: "manual" },
],
},
{
// NCS / Cisco 8000
modelPattern: /^NCS-|^8[01]\d{2}/,
productPage: (m) => `https://www.cisco.com/c/en/us/products/routers/network-convergence-system-5500-series/index.html`,
imageUrl: () => null,
datasheetUrl: () => null,
},
],
};
// ═══════════════════════════════════════════════════════
// ARISTA
// ═══════════════════════════════════════════════════════
const ARISTA_CONFIG: VendorAssetConfig = {
vendorName: "Arista Networks",
docsPortal: "https://www.arista.com/en/support/product-documentation",
datasheetLibrary: "https://www.arista.com/en/products",
supportPortal: "https://www.arista.com/en/support/customer-support",
imageCdn: "https://www.arista.com/assets/images/",
modelAssets: [
{
// Arista 7000 series — verified URL patterns
// Datasheets: /assets/data/pdf/Datasheets/{SeriesName}-Datasheet.pdf (naming inconsistent)
// Images: /assets/images/product/{series}-stack-200w200h.png or /assets/images/article/{model}-Right.png
// QSGs: /assets/data/pdf/qsg/qsg-books/QS_{series}_{formfactor}_{gen}.pdf
modelPattern: /^7[0-9]/,
productPage: (m) => {
const series = m.match(/^(7\d{3,4}[A-Z]*\d*)/)?.[1] || m;
return `https://www.arista.com/en/products/${series.toLowerCase().replace(/[^a-z0-9]/g, "")}-series`;
},
imageUrl: (m) => {
// Try article-level images (higher res): {model}-Right.png
const model = m.match(/^(\d{4}[A-Z]+\d*)/)?.[1] || m.split("-")[0];
return `https://www.arista.com/assets/images/article/${model}-Right.png`;
},
datasheetUrl: (m) => {
// Arista naming: {series}-Datasheet.pdf or {series}-Data-Sheet.pdf
const series = m.match(/^(7\d{3,4}[A-Z]*\d*)/)?.[1];
return series ? `https://www.arista.com/assets/data/pdf/Datasheets/${series}-Datasheet.pdf` : null;
},
manualUrls: (m) => {
const series = m.match(/^(7\d{3,4})/)?.[1] || "";
return [
{ url: `https://www.arista.com/assets/data/pdf/qsg/qsg-books/QS_${series}_1RU_Gen3.pdf`, title: `${m} Quick Start Guide`, type: "quick_start" },
];
},
},
{
// Arista 5000 (Campus)
modelPattern: /^5[0-9]/,
productPage: () => `https://www.arista.com/en/products/arista-5000-series`,
imageUrl: () => null,
datasheetUrl: () => `https://www.arista.com/assets/data/pdf/Datasheets/Arista-5000-Datasheet.pdf`,
manualUrls: () => [],
},
],
};
// ═══════════════════════════════════════════════════════
// JUNIPER
// ═══════════════════════════════════════════════════════
const JUNIPER_CONFIG: VendorAssetConfig = {
vendorName: "Juniper Networks",
docsPortal: "https://www.juniper.net/documentation/",
datasheetLibrary: "https://www.juniper.net/us/en/products.html",
supportPortal: "https://supportportal.juniper.net/",
imageCdn: "https://www.juniper.net/content/dam/www/assets/images/",
modelAssets: [
{
// Juniper switches — verified URL patterns
// Datasheets: /content/dam/www/assets/datasheets/us/en/switches/{name}-datasheet.pdf
// Images: /content/dam/www/assets/images/us/en/products/switches/{series}/{filename}.png
// HW guides: /documentation/us/en/hardware/{model}/{model}.pdf
// QSGs: /documentation/us/en/quick-start/hardware/qsg/{model}/{model}.pdf
modelPattern: /^QFX|^EX|^MX|^PTX|^SRX|^ACX/,
productPage: (m) => {
const seriesPrefix = m.match(/^([A-Z]+)/)?.[1]?.toLowerCase() || "";
const seriesNum = m.match(/^[A-Z]+(\d+)/)?.[1] || "";
return `https://www.juniper.net/us/en/products/switches/${seriesPrefix}-series/${seriesPrefix}${seriesNum}.html`;
},
imageUrl: (m) => {
const seriesPrefix = m.match(/^([A-Z]+)/)?.[1]?.toLowerCase() || "";
const category = (seriesPrefix === "mx" || seriesPrefix === "ptx" || seriesPrefix === "acx") ? "routers" : "switches";
return `https://www.juniper.net/content/dam/www/assets/images/us/en/products/${category}/${seriesPrefix}-series/${seriesPrefix}-family-21DEC20.png`;
},
datasheetUrl: (m) => {
const seriesPrefix = m.match(/^([A-Z]+)/)?.[1]?.toLowerCase() || "";
const category = (seriesPrefix === "mx" || seriesPrefix === "ptx" || seriesPrefix === "acx") ? "routers" : "switches";
// Juniper datasheet naming: {full-series-name}-datasheet.pdf or {product-description}-datasheet.pdf
const model = m.match(/^([A-Z]+\d+)/)?.[1]?.toLowerCase() || "";
return `https://www.juniper.net/content/dam/www/assets/datasheets/us/en/${category}/${model}-ethernet-switch-datasheet.pdf`;
},
manualUrls: (m) => {
const model = m.toLowerCase().replace(/\s+/g, "");
return [
{ url: `https://www.juniper.net/documentation/us/en/hardware/${model}/${model}.pdf`, title: `${m} Hardware Guide`, type: "manual" },
{ url: `https://www.juniper.net/documentation/us/en/quick-start/hardware/qsg/${model}/${model}.pdf`, title: `${m} Quick Start Guide`, type: "quick_start" },
];
},
},
],
};
// ═══════════════════════════════════════════════════════
// NOKIA
// ═══════════════════════════════════════════════════════
const NOKIA_CONFIG: VendorAssetConfig = {
vendorName: "Nokia",
docsPortal: "https://documentation.nokia.com/",
datasheetLibrary: "https://www.nokia.com/networks/products/",
supportPortal: "https://customer.nokia.com/support/s/",
modelAssets: [
{
modelPattern: /./,
productPage: (m) => `https://www.nokia.com/networks/products/${m.toLowerCase().replace(/\s+/g, "-")}/`,
imageUrl: () => null,
datasheetUrl: () => null,
},
],
};
// ═══════════════════════════════════════════════════════
// FORTINET
// ═══════════════════════════════════════════════════════
const FORTINET_CONFIG: VendorAssetConfig = {
vendorName: "Fortinet",
docsPortal: "https://docs.fortinet.com/",
datasheetLibrary: "https://www.fortinet.com/products/switches",
supportPortal: "https://support.fortinet.com/",
modelAssets: [
{
// FortiSwitch product pages — use CheerioCrawler for image extraction
// Datasheets at /content/dam/fortinet/assets/data-sheets/
modelPattern: /^FortiSwitch/,
productPage: (m) => {
const num = m.match(/(\d+[A-Z]*)/)?.[1]?.toLowerCase() || "";
return `https://www.fortinet.com/products/switches/fortiswitch-${num}`;
},
imageUrl: () => null, // extracted from product page by crawler
datasheetUrl: (m) => {
// Fortinet DAM paths: fortiswitch-{series}00-series.pdf or FortiSwitch-{model}.pdf
const series = m.match(/FortiSwitch[- ]*(\d)/)?.[1];
if (series) return `https://www.fortinet.com/content/dam/fortinet/assets/data-sheets/fortiswitch-${series}00-series.pdf`;
return null;
},
manualUrls: () => [
{ url: `https://docs.fortinet.com/document/fortiswitch/latest/administration-guide`, title: "FortiSwitch Administration Guide", type: "manual" },
{ url: `https://docs.fortinet.com/document/fortiswitch/latest/hardware-guide`, title: "FortiSwitch Hardware Guide", type: "installation_guide" },
],
},
],
};
// ═══════════════════════════════════════════════════════
// MIKROTIK
// ═══════════════════════════════════════════════════════
const MIKROTIK_CONFIG: VendorAssetConfig = {
vendorName: "MikroTik",
docsPortal: "https://help.mikrotik.com/docs/",
datasheetLibrary: "https://mikrotik.com/products",
supportPortal: "https://help.mikrotik.com/",
modelAssets: [
{
// MikroTik uses underscored model names in URLs
// Images & PDFs use unpredictable numeric IDs — must scrape product page
// CheerioCrawler handles this via parseMikroTikPage()
modelPattern: /^CRS|^CCR/,
productPage: (m) => `https://mikrotik.com/product/${m.replace(/\s+/g, "_")}`,
imageUrl: () => null, // extracted from product page by crawler
datasheetUrl: () => null, // extracted from product page by crawler
manualUrls: (m) => [
{ url: `https://help.mikrotik.com/docs/display/UM/${m.replace(/\s+/g, "+")}`, title: `${m} User Manual`, type: "manual" },
],
},
],
};
// ═══════════════════════════════════════════════════════
// HIRSCHMANN
// ═══════════════════════════════════════════════════════
const HIRSCHMANN_CONFIG: VendorAssetConfig = {
vendorName: "Hirschmann",
docsPortal: "https://catalog.belden.com/index.cfm",
datasheetLibrary: "https://catalog.belden.com/index.cfm",
supportPortal: "https://www.belden.com/support",
modelAssets: [
{
modelPattern: /./,
productPage: (m) => `https://catalog.belden.com/techdata/en/${m.replace(/\s+/g, "_")}_en.html`,
imageUrl: () => null,
datasheetUrl: () => null,
},
],
};
// ═══════════════════════════════════════════════════════
// Additional vendor stubs (HPE, Dell, Extreme, Huawei, etc.)
// ═══════════════════════════════════════════════════════
const HPE_CONFIG: VendorAssetConfig = {
vendorName: "HPE / Aruba",
docsPortal: "https://www.arubanetworks.com/techdocs/",
datasheetLibrary: "https://www.arubanetworks.com/products/switches/",
supportPortal: "https://asp.arubanetworks.com/",
modelAssets: [
{
modelPattern: /^CX/,
productPage: (m) => `https://www.arubanetworks.com/products/switches/cx-${m.match(/CX\s*(\d+)/)?.[1] || ""}-series/`,
imageUrl: (m) => `https://www.arubanetworks.com/wp-content/uploads/aruba-cx-${m.match(/CX\s*(\d+)/)?.[1] || ""}.png`,
datasheetUrl: (m) => `https://www.arubanetworks.com/assets/ds/DS_CX${m.match(/CX\s*(\d+)/)?.[1] || ""}.pdf`,
manualUrls: (m) => [
{ url: `https://www.arubanetworks.com/techdocs/CX/`, title: `Aruba CX ${m} Configuration Guide`, type: "manual" },
],
},
],
};
const DELL_CONFIG: VendorAssetConfig = {
vendorName: "Dell Technologies",
docsPortal: "https://www.dell.com/support/",
datasheetLibrary: "https://www.dell.com/en-us/lp/networking",
supportPortal: "https://www.dell.com/support/home/",
modelAssets: [
{
modelPattern: /./,
productPage: (m) => `https://www.dell.com/en-us/shop/networking/cp/${m.toLowerCase().replace(/[^a-z0-9]/g, "-")}`,
imageUrl: () => null,
datasheetUrl: () => null,
},
],
};
const EXTREME_CONFIG: VendorAssetConfig = {
vendorName: "Extreme Networks",
docsPortal: "https://extremeportal.force.com/ExtrArticleDetail",
datasheetLibrary: "https://www.extremenetworks.com/products/",
supportPortal: "https://extremeportal.force.com/",
modelAssets: [
{
modelPattern: /^X|^SLX|^VDX|^5[0-9]/,
productPage: (m) => `https://www.extremenetworks.com/product/${m.toLowerCase().replace(/[^a-z0-9]/g, "-")}`,
imageUrl: () => null,
datasheetUrl: () => null,
},
],
};
const MOXA_CONFIG: VendorAssetConfig = {
vendorName: "Moxa",
docsPortal: "https://www.moxa.com/en/support/support",
datasheetLibrary: "https://www.moxa.com/en/products/industrial-network-infrastructure",
supportPortal: "https://www.moxa.com/en/support/",
modelAssets: [
{
modelPattern: /./,
productPage: (m) => `https://www.moxa.com/en/products/industrial-network-infrastructure/${m.toLowerCase()}`,
imageUrl: () => null,
datasheetUrl: () => null,
},
],
};
const SIEMENS_CONFIG: VendorAssetConfig = {
vendorName: "Siemens",
docsPortal: "https://support.industry.siemens.com/",
datasheetLibrary: "https://mall.industry.siemens.com/",
supportPortal: "https://support.industry.siemens.com/",
modelAssets: [
{
modelPattern: /^SCALANCE/,
productPage: (m) => `https://www.siemens.com/global/en/products/automation/industrial-communication/industrial-ethernet/${m.toLowerCase().replace(/\s+/g, "-")}.html`,
imageUrl: () => null,
datasheetUrl: () => null,
},
],
};
// ═══════════════════════════════════════════════════════
// D-LINK
// ═══════════════════════════════════════════════════════
const DLINK_CONFIG: VendorAssetConfig = {
vendorName: "D-Link",
docsPortal: "https://www.dlink.com/en/support",
datasheetLibrary: "https://www.dlink.com/en/products/switches",
supportPortal: "https://www.dlink.com/en/support",
modelAssets: [
{
modelPattern: /^D[GMXW]S/,
productPage: (m) => `https://www.dlink.com/en/products/${m.toLowerCase()}`,
imageUrl: () => null,
datasheetUrl: () => null,
},
],
};
// ═══════════════════════════════════════════════════════
// ALCATEL-LUCENT ENTERPRISE
// ═══════════════════════════════════════════════════════
const ALE_CONFIG: VendorAssetConfig = {
vendorName: "Alcatel-Lucent Enterprise",
docsPortal: "https://www.al-enterprise.com/en-us/documentation",
datasheetLibrary: "https://www.al-enterprise.com/en-us/products/switches",
supportPortal: "https://myportal.al-enterprise.com/",
modelAssets: [
{
modelPattern: /^OmniSwitch/,
productPage: (m) => {
const num = m.match(/(\d{4})/)?.[1] || "";
return `https://www.al-enterprise.com/en-us/products/switches/omniswitch-${num}`;
},
imageUrl: () => null,
datasheetUrl: () => null,
},
],
};
// ═══════════════════════════════════════════════════════
// BROCADE / BROADCOM
// ═══════════════════════════════════════════════════════
const BROCADE_CONFIG: VendorAssetConfig = {
vendorName: "Brocade",
docsPortal: "https://www.broadcom.com/support/fibre-channel-networking",
datasheetLibrary: "https://www.broadcom.com/products/fibre-channel-networking",
supportPortal: "https://www.broadcom.com/support",
modelAssets: [
{
modelPattern: /./,
productPage: (m) => `https://www.broadcom.com/products/fibre-channel-networking/switches/${m.toLowerCase().replace(/\s+/g, "-")}`,
imageUrl: () => null,
datasheetUrl: () => null,
},
],
};
// ═══════════════════════════════════════════════════════
// CIENA
// ═══════════════════════════════════════════════════════
const CIENA_CONFIG: VendorAssetConfig = {
vendorName: "Ciena",
docsPortal: "https://www.ciena.com/insights/resources",
datasheetLibrary: "https://www.ciena.com/products",
supportPortal: "https://www.ciena.com/support",
modelAssets: [
{
modelPattern: /./,
productPage: (m) => `https://www.ciena.com/products/${m.toLowerCase().replace(/\s+/g, "-")}`,
imageUrl: () => null,
datasheetUrl: () => null,
},
],
};
// ═══════════════════════════════════════════════════════
// H3C
// ═══════════════════════════════════════════════════════
const H3C_CONFIG: VendorAssetConfig = {
vendorName: "H3C",
docsPortal: "https://www.h3c.com/en/Support/Resource_Center/",
datasheetLibrary: "https://www.h3c.com/en/Products_And_Solution/Networking/",
supportPortal: "https://www.h3c.com/en/Support/",
modelAssets: [
{
modelPattern: /^S/,
productPage: (m) => `https://www.h3c.com/en/Products_And_Solution/Networking/Switches/${m}/`,
imageUrl: () => null,
datasheetUrl: () => null,
},
],
};
// ═══════════════════════════════════════════════════════
// GIGAMON
// ═══════════════════════════════════════════════════════
const GIGAMON_CONFIG: VendorAssetConfig = {
vendorName: "Gigamon",
docsPortal: "https://docs.gigamon.com/",
datasheetLibrary: "https://www.gigamon.com/products",
supportPortal: "https://community.gigamon.com/",
modelAssets: [
{
modelPattern: /^GigaVUE/,
productPage: (m) => `https://www.gigamon.com/products/access-traffic/${m.toLowerCase().replace(/\s+/g, "-")}`,
imageUrl: () => null,
datasheetUrl: () => null,
},
],
};
// ═══════════════════════════════════════════════════════
// RUCKUS / COMMSCOPE
// ═══════════════════════════════════════════════════════
const RUCKUS_CONFIG: VendorAssetConfig = {
vendorName: "Ruckus",
docsPortal: "https://support.ruckuswireless.com/",
datasheetLibrary: "https://www.commscope.com/ruckus/products/",
supportPortal: "https://support.ruckuswireless.com/",
modelAssets: [
{
modelPattern: /^ICX/,
productPage: (m) => `https://www.commscope.com/ruckus/products/switches/${m.toLowerCase().replace(/\s+/g, "-")}`,
imageUrl: () => null,
datasheetUrl: () => null,
},
],
};
// All vendor configs
const VENDOR_CONFIGS: VendorAssetConfig[] = [
CISCO_CONFIG,
ARISTA_CONFIG,
JUNIPER_CONFIG,
NOKIA_CONFIG,
FORTINET_CONFIG,
MIKROTIK_CONFIG,
HIRSCHMANN_CONFIG,
HPE_CONFIG,
DELL_CONFIG,
EXTREME_CONFIG,
MOXA_CONFIG,
SIEMENS_CONFIG,
DLINK_CONFIG,
ALE_CONFIG,
BROCADE_CONFIG,
CIENA_CONFIG,
H3C_CONFIG,
GIGAMON_CONFIG,
RUCKUS_CONFIG,
];
function findVendorConfig(vendorName: string): VendorAssetConfig | undefined {
return VENDOR_CONFIGS.find((c) =>
vendorName.toLowerCase().includes(c.vendorName.toLowerCase().split(" ")[0].toLowerCase())
);
}
// ═══════════════════════════════════════════════════════
// Main scraper function
// ═══════════════════════════════════════════════════════
export async function scrapeSwitchAssets(targetVendor?: string): Promise<void> {
console.log("=== Switch Product Assets Scraper ===\n");
// Get all switches that haven't had assets scraped
const vendorFilter = targetVendor
? `AND v.name ILIKE '%${targetVendor}%'`
: "";
const result = await pool.query(`
SELECT sw.id, sw.model, sw.series, v.name as vendor_name, v.id as vendor_id
FROM switches sw
JOIN vendors v ON sw.vendor_id = v.id
WHERE sw.assets_scraped_at IS NULL ${vendorFilter}
ORDER BY v.name, sw.model
`);
console.log(`Found ${result.rows.length} switches without assets${targetVendor ? ` (vendor: ${targetVendor})` : ""}.\n`);
let images = 0;
let datasheets = 0;
let manuals = 0;
let productPages = 0;
for (const row of result.rows) {
const config = findVendorConfig(row.vendor_name);
if (!config) {
console.log(` [SKIP] No config for vendor: ${row.vendor_name}`);
continue;
}
// Find matching model asset map
const assetMap = config.modelAssets.find((m) => m.modelPattern.test(row.model));
if (!assetMap) continue;
console.log(` ${row.vendor_name} ${row.model}:`);
// 1. Set product page URL
const productPageUrl = assetMap.productPage(row.model);
if (productPageUrl) {
await setSwitchProductPage(row.id, productPageUrl);
productPages++;
console.log(` ✓ Product page`);
}
// 2. Download image
const imageUrl = assetMap.imageUrl(row.model);
if (imageUrl) {
const ok = await downloadSwitchImage(row.id, imageUrl, row.vendor_name, row.model);
if (ok) {
images++;
console.log(` ✓ Image downloaded`);
}
}
// 3. Download datasheet
const datasheetUrl = assetMap.datasheetUrl(row.model);
if (datasheetUrl) {
const ok = await downloadSwitchDatasheet(
row.id, row.vendor_id, datasheetUrl,
`${row.model} Datasheet`, row.vendor_name, row.model
);
if (ok) {
datasheets++;
console.log(` ✓ Datasheet downloaded`);
}
}
// 4. Download manuals
const manualList = assetMap.manualUrls?.(row.model) || [];
for (const manual of manualList) {
const ok = await downloadSwitchManual(
row.id, row.vendor_id, manual.url,
manual.title, manual.type, row.vendor_name, row.model
);
if (ok) {
manuals++;
console.log(`${manual.type}: ${manual.title}`);
}
}
// Rate limiting: 500ms between vendors
await new Promise((r) => setTimeout(r, 500));
}
// Update vendor doc portal URLs
for (const config of VENDOR_CONFIGS) {
const vendorResult = await pool.query(`SELECT id FROM vendors WHERE name ILIKE $1`, [`%${config.vendorName.split(" ")[0]}%`]);
if (vendorResult.rows.length > 0) {
await setVendorDocUrls(vendorResult.rows[0].id, {
docsPortal: config.docsPortal,
datasheetLibrary: config.datasheetLibrary,
supportPortal: config.supportPortal,
imageCdn: config.imageCdn,
});
}
}
console.log(`\n=== Assets Complete ===`);
console.log(` Product pages: ${productPages}`);
console.log(` Images: ${images}`);
console.log(` Datasheets: ${datasheets}`);
console.log(` Manuals: ${manuals}`);
}
if (require.main === module) {
const vendor = process.argv.find((a) => a.startsWith("--vendor="))?.split("=")[1];
scrapeSwitchAssets(vendor)
.then(() => pool.end())
.catch((err) => { console.error("Fatal:", err); pool.end(); process.exit(1); });
}

View File

@ -0,0 +1,935 @@
/**
* Bulk Switch & Router Seed Data Remaining Flexoptix-Supported Vendors
*
* Covers ALL vendors from flexoptix-supported-vendors.ts that don't yet have
* switch/router product entries in switch-seed.ts or switch-seed-extended.ts.
*
* This file adds representative models per vendor actual model catalogs
* will be enriched by the Crawlee scrapers hitting vendor product pages.
*
* Vendors already covered in other seed files (NOT included here):
* switch-seed.ts: Arista, Cisco, Dell, Extreme, HPE Aruba, Huawei,
* Juniper, Nokia, NVIDIA/Mellanox, Celestica, Edgecore, UfiSpace
* switch-seed-extended.ts: Fortinet, MikroTik, Ubiquiti, Netgear, Allied Telesis,
* TP-Link, Zyxel, Moxa, Hirschmann, Siemens, Phoenix Contact,
* Westermo, Check Point, F5, Palo Alto
*/
import { pool, ensureVendor, findOrCreateSwitch } from "../utils/db";
interface SwitchSeed {
vendor: string;
vendorType: string;
vendorWebsite: string;
model: string;
series: string;
category: "DataCenter" | "Campus" | "Edge" | "Core" | "SP" | "Industrial";
layer: "L2" | "L3" | "L2/L3";
portsConfig: Record<string, number>;
totalPorts: number;
uplinkSpeedGbps?: number;
maxSpeedGbps: number;
switchingCapacityTbps?: number;
forwardingRateMpps?: number;
asicVendor?: string;
asicModel?: string;
rackUnits?: number;
maxPowerW?: number;
poeSupport?: string;
stackingSupport?: boolean;
vxlanSupport?: boolean;
evpnSupport?: boolean;
bgpSupport?: boolean;
mplsSupport?: boolean;
openconfigSupport?: boolean;
sonicCompatible?: boolean;
macsecSupport?: boolean;
lifecycleStatus?: string;
tags?: string[];
}
// ═══════════════════════════════════════════════════════
// D-LINK
// ═══════════════════════════════════════════════════════
const DLINK: SwitchSeed[] = [
{
vendor: "D-Link", vendorType: "oem", vendorWebsite: "https://www.dlink.com",
model: "DGS-3130-30TS", series: "DGS-3130", category: "Campus", layer: "L3",
portsConfig: { "1G_RJ45": 24, "10G_SFP+": 6 }, totalPorts: 30,
maxSpeedGbps: 10, switchingCapacityTbps: 0.172, stackingSupport: true,
tags: ["campus", "stackable", "L3-lite"],
},
{
vendor: "D-Link", vendorType: "oem", vendorWebsite: "https://www.dlink.com",
model: "DXS-3610-54T", series: "DXS-3610", category: "DataCenter", layer: "L3",
portsConfig: { "10G_RJ45": 48, "100G_QSFP28": 6 }, totalPorts: 54,
maxSpeedGbps: 100, switchingCapacityTbps: 1.76, bgpSupport: true,
tags: ["10G", "datacenter", "ToR"],
},
{
vendor: "D-Link", vendorType: "oem", vendorWebsite: "https://www.dlink.com",
model: "DMS-3130-30TS", series: "DMS-3130", category: "Campus", layer: "L2/L3",
portsConfig: { "2.5G_RJ45": 24, "10G_SFP+": 6 }, totalPorts: 30,
maxSpeedGbps: 10, poeSupport: "PoE++",
tags: ["multigigabit", "PoE", "campus"],
},
];
// ═══════════════════════════════════════════════════════
// ALCATEL-LUCENT ENTERPRISE
// ═══════════════════════════════════════════════════════
const ALE: SwitchSeed[] = [
{
vendor: "Alcatel-Lucent Enterprise", vendorType: "oem", vendorWebsite: "https://www.al-enterprise.com",
model: "OmniSwitch 6900-X72", series: "OS6900", category: "DataCenter", layer: "L3",
portsConfig: { "10G_SFP+": 48, "40G_QSFP+": 6 }, totalPorts: 54,
maxSpeedGbps: 40, switchingCapacityTbps: 1.44, bgpSupport: true, evpnSupport: true,
tags: ["datacenter", "SPB", "fabric"],
},
{
vendor: "Alcatel-Lucent Enterprise", vendorType: "oem", vendorWebsite: "https://www.al-enterprise.com",
model: "OmniSwitch 6560-P48Z8", series: "OS6560", category: "Campus", layer: "L2/L3",
portsConfig: { "1G_RJ45": 48, "10G_SFP+": 4 }, totalPorts: 52,
maxSpeedGbps: 10, poeSupport: "PoE++", stackingSupport: true,
tags: ["campus", "PoE", "stackable"],
},
{
vendor: "Alcatel-Lucent Enterprise", vendorType: "oem", vendorWebsite: "https://www.al-enterprise.com",
model: "OmniSwitch 9900-C32D", series: "OS9900", category: "Core", layer: "L3",
portsConfig: { "400G_QSFP-DD": 32 }, totalPorts: 32,
maxSpeedGbps: 400, switchingCapacityTbps: 25.6, bgpSupport: true, evpnSupport: true,
tags: ["400G", "spine", "datacenter"],
},
];
// ═══════════════════════════════════════════════════════
// BROCADE (now Broadcom/Ruckus)
// ═══════════════════════════════════════════════════════
const BROCADE: SwitchSeed[] = [
{
vendor: "Brocade", vendorType: "oem", vendorWebsite: "https://www.broadcom.com",
model: "ICX 7850-48FS", series: "ICX 7850", category: "DataCenter", layer: "L3",
portsConfig: { "10G_SFP+": 48, "100G_QSFP28": 8 }, totalPorts: 56,
maxSpeedGbps: 100, switchingCapacityTbps: 1.76, stackingSupport: true,
bgpSupport: true, vxlanSupport: true,
tags: ["campus-core", "10G", "stackable"],
},
{
vendor: "Brocade", vendorType: "oem", vendorWebsite: "https://www.broadcom.com",
model: "G720", series: "G720", category: "DataCenter", layer: "L2",
portsConfig: { "64G_FC": 48 }, totalPorts: 48,
maxSpeedGbps: 64,
tags: ["FC", "SAN", "storage"],
},
{
vendor: "Brocade", vendorType: "oem", vendorWebsite: "https://www.broadcom.com",
model: "G730", series: "G730", category: "DataCenter", layer: "L2",
portsConfig: { "64G_FC": 64 }, totalPorts: 64,
maxSpeedGbps: 64,
tags: ["FC", "SAN", "Gen7", "storage"],
},
];
// ═══════════════════════════════════════════════════════
// H3C (HPE China / New H3C)
// ═══════════════════════════════════════════════════════
const H3C: SwitchSeed[] = [
{
vendor: "H3C", vendorType: "oem", vendorWebsite: "https://www.h3c.com",
model: "S12500X-AF", series: "S12500", category: "Core", layer: "L3",
portsConfig: { "400G_QSFP-DD": 72 }, totalPorts: 72,
maxSpeedGbps: 400, switchingCapacityTbps: 57.6, bgpSupport: true,
tags: ["chassis", "core", "400G"],
},
{
vendor: "H3C", vendorType: "oem", vendorWebsite: "https://www.h3c.com",
model: "S6860-54HT", series: "S6860", category: "DataCenter", layer: "L3",
portsConfig: { "10G_RJ45": 48, "100G_QSFP28": 6 }, totalPorts: 54,
maxSpeedGbps: 100, switchingCapacityTbps: 1.76, bgpSupport: true,
tags: ["datacenter", "ToR", "10G-BaseT"],
},
{
vendor: "H3C", vendorType: "oem", vendorWebsite: "https://www.h3c.com",
model: "S5170-54S-EI", series: "S5170", category: "Campus", layer: "L2/L3",
portsConfig: { "1G_RJ45": 48, "10G_SFP+": 6 }, totalPorts: 54,
maxSpeedGbps: 10, poeSupport: "PoE+", stackingSupport: true,
tags: ["campus", "stackable"],
},
];
// ═══════════════════════════════════════════════════════
// RUIJIE NETWORKS
// ═══════════════════════════════════════════════════════
const RUIJIE: SwitchSeed[] = [
{
vendor: "Ruijie Networks", vendorType: "oem", vendorWebsite: "https://www.ruijienetworks.com",
model: "RG-S6920-4C", series: "RG-S6920", category: "DataCenter", layer: "L3",
portsConfig: { "100G_QSFP28": 32 }, totalPorts: 32,
maxSpeedGbps: 100, switchingCapacityTbps: 6.4, bgpSupport: true, vxlanSupport: true,
tags: ["datacenter", "spine", "100G"],
},
{
vendor: "Ruijie Networks", vendorType: "oem", vendorWebsite: "https://www.ruijienetworks.com",
model: "RG-S5760C-24SFP/8GT8XS-X", series: "RG-S5760C", category: "Campus", layer: "L3",
portsConfig: { "1G_SFP": 24, "1G_RJ45": 8, "10G_SFP+": 8 }, totalPorts: 40,
maxSpeedGbps: 10, stackingSupport: true,
tags: ["campus", "aggregation"],
},
];
// ═══════════════════════════════════════════════════════
// PLANET TECHNOLOGY
// ═══════════════════════════════════════════════════════
const PLANET: SwitchSeed[] = [
{
vendor: "Planet Technology", vendorType: "oem", vendorWebsite: "https://www.planet.com.tw",
model: "GS-6322-24P4X", series: "GS-6322", category: "Campus", layer: "L3",
portsConfig: { "1G_RJ45": 24, "10G_SFP+": 4 }, totalPorts: 28,
maxSpeedGbps: 10, poeSupport: "PoE+", stackingSupport: false,
tags: ["campus", "PoE", "L3-lite"],
},
{
vendor: "Planet Technology", vendorType: "oem", vendorWebsite: "https://www.planet.com.tw",
model: "IGS-6325-8T8S4X", series: "IGS-6325", category: "Industrial", layer: "L3",
portsConfig: { "1G_RJ45": 8, "1G_SFP": 8, "10G_SFP+": 4 }, totalPorts: 20,
maxSpeedGbps: 10,
tags: ["industrial", "DIN-rail", "IP30"],
},
];
// ═══════════════════════════════════════════════════════
// LANCOM SYSTEMS
// ═══════════════════════════════════════════════════════
const LANCOM: SwitchSeed[] = [
{
vendor: "LANCOM Systems", vendorType: "oem", vendorWebsite: "https://www.lancom-systems.de",
model: "GS-4554XP", series: "GS-4554", category: "Campus", layer: "L3",
portsConfig: { "2.5G_RJ45": 48, "10G_SFP+": 6 }, totalPorts: 54,
maxSpeedGbps: 10, poeSupport: "PoE++", stackingSupport: true,
tags: ["campus", "cloud-managed", "multigigabit"],
},
];
// ═══════════════════════════════════════════════════════
// CIENA
// ═══════════════════════════════════════════════════════
const CIENA: SwitchSeed[] = [
{
vendor: "Ciena", vendorType: "oem", vendorWebsite: "https://www.ciena.com",
model: "8700 Packetwave", series: "8700", category: "SP", layer: "L3",
portsConfig: { "100G_QSFP28": 32 }, totalPorts: 32,
maxSpeedGbps: 100, mplsSupport: true, bgpSupport: true,
tags: ["carrier", "DWDM", "packet-optical"],
},
{
vendor: "Ciena", vendorType: "oem", vendorWebsite: "https://www.ciena.com",
model: "5171", series: "5170", category: "SP", layer: "L2/L3",
portsConfig: { "100G_QSFP28": 16, "10G_SFP+": 48 }, totalPorts: 64,
maxSpeedGbps: 100, mplsSupport: true,
tags: ["carrier", "aggregation", "MEF"],
},
{
vendor: "Ciena", vendorType: "oem", vendorWebsite: "https://www.ciena.com",
model: "3930", series: "3930", category: "Edge", layer: "L2/L3",
portsConfig: { "10G_SFP+": 24, "100G_QSFP28": 4 }, totalPorts: 28,
maxSpeedGbps: 100, mplsSupport: true,
tags: ["edge", "aggregation", "service-aware"],
},
];
// ═══════════════════════════════════════════════════════
// ADTRAN / ADVA
// ═══════════════════════════════════════════════════════
const ADTRAN: SwitchSeed[] = [
{
vendor: "Adtran", vendorType: "oem", vendorWebsite: "https://www.adtran.com",
model: "NetVanta 1560-48P", series: "NetVanta 1560", category: "Campus", layer: "L2/L3",
portsConfig: { "1G_RJ45": 48, "10G_SFP+": 4 }, totalPorts: 52,
maxSpeedGbps: 10, poeSupport: "PoE+", stackingSupport: true,
tags: ["campus", "PoE", "SMB"],
},
{
vendor: "Adtran", vendorType: "oem", vendorWebsite: "https://www.adtran.com",
model: "FSP 3000 CloudConnect", series: "FSP 3000", category: "SP", layer: "L2",
portsConfig: { "100G_QSFP28": 4, "10G_SFP+": 8 }, totalPorts: 12,
maxSpeedGbps: 100, mplsSupport: true,
tags: ["optical", "DWDM", "carrier"],
},
];
// ═══════════════════════════════════════════════════════
// CALIX
// ═══════════════════════════════════════════════════════
const CALIX: SwitchSeed[] = [
{
vendor: "Calix", vendorType: "oem", vendorWebsite: "https://www.calix.com",
model: "E9-2 Intelligent Edge System", series: "E9-2", category: "Edge", layer: "L2/L3",
portsConfig: { "10G_SFP+": 16, "1G_RJ45": 48 }, totalPorts: 64,
maxSpeedGbps: 10,
tags: ["FTTH", "access", "ISP"],
},
];
// ═══════════════════════════════════════════════════════
// CAMBIUM NETWORKS
// ═══════════════════════════════════════════════════════
const CAMBIUM: SwitchSeed[] = [
{
vendor: "Cambium Networks", vendorType: "oem", vendorWebsite: "https://www.cambiumnetworks.com",
model: "cnMatrix EX2052-P", series: "cnMatrix EX2052", category: "Campus", layer: "L2/L3",
portsConfig: { "1G_RJ45": 48, "10G_SFP+": 4 }, totalPorts: 52,
maxSpeedGbps: 10, poeSupport: "PoE++",
tags: ["campus", "cloud-managed", "PoE"],
},
{
vendor: "Cambium Networks", vendorType: "oem", vendorWebsite: "https://www.cambiumnetworks.com",
model: "cnMatrix EX2028-P", series: "cnMatrix EX2028", category: "Campus", layer: "L2",
portsConfig: { "1G_RJ45": 24, "10G_SFP+": 4 }, totalPorts: 28,
maxSpeedGbps: 10, poeSupport: "PoE+",
tags: ["campus", "edge", "PoE"],
},
];
// ═══════════════════════════════════════════════════════
// AVAYA (ex. Nortel)
// ═══════════════════════════════════════════════════════
const AVAYA: SwitchSeed[] = [
{
vendor: "Avaya", vendorType: "oem", vendorWebsite: "https://www.avaya.com",
model: "VSP 7432CQ", series: "VSP 7400", category: "DataCenter", layer: "L3",
portsConfig: { "100G_QSFP28": 32 }, totalPorts: 32,
maxSpeedGbps: 100, switchingCapacityTbps: 6.4, bgpSupport: true,
tags: ["datacenter", "fabric", "SPB"],
},
{
vendor: "Avaya", vendorType: "oem", vendorWebsite: "https://www.avaya.com",
model: "ERS 4950GTS-PWR+", series: "ERS 4950", category: "Campus", layer: "L2/L3",
portsConfig: { "1G_RJ45": 48, "10G_SFP+": 2 }, totalPorts: 50,
maxSpeedGbps: 10, poeSupport: "PoE+", stackingSupport: true,
tags: ["campus", "stackable", "PoE"],
},
];
// ═══════════════════════════════════════════════════════
// FUJITSU
// ═══════════════════════════════════════════════════════
const FUJITSU: SwitchSeed[] = [
{
vendor: "Fujitsu", vendorType: "oem", vendorWebsite: "https://www.fujitsu.com",
model: "FLASHWAVE 9500", series: "FLASHWAVE 9500", category: "SP", layer: "L2",
portsConfig: { "100G_QSFP28": 20 }, totalPorts: 20,
maxSpeedGbps: 100,
tags: ["optical", "DWDM", "carrier", "packet-optical"],
},
];
// ═══════════════════════════════════════════════════════
// NEC
// ═══════════════════════════════════════════════════════
const NEC: SwitchSeed[] = [
{
vendor: "NEC", vendorType: "oem", vendorWebsite: "https://www.nec.com",
model: "PF5248", series: "PF5200", category: "DataCenter", layer: "L3",
portsConfig: { "25G_SFP28": 48, "100G_QSFP28": 8 }, totalPorts: 56,
maxSpeedGbps: 100, switchingCapacityTbps: 3.2, bgpSupport: true,
tags: ["datacenter", "ToR", "25G"],
},
];
// ═══════════════════════════════════════════════════════
// JUNIPER (ex. Mist) — Wired Assurance
// ═══════════════════════════════════════════════════════
// Already covered in switch-seed.ts — skip
// ═══════════════════════════════════════════════════════
// RUCKUS / COMMSCOPE
// ═══════════════════════════════════════════════════════
const RUCKUS: SwitchSeed[] = [
{
vendor: "Ruckus (CommScope)", vendorType: "oem", vendorWebsite: "https://www.commscope.com/ruckus",
model: "ICX 7550-48ZP", series: "ICX 7550", category: "Campus", layer: "L3",
portsConfig: { "2.5G_RJ45": 48, "10G_SFP+": 2, "40G_QSFP+": 2 }, totalPorts: 52,
maxSpeedGbps: 40, poeSupport: "PoE++", stackingSupport: true,
tags: ["campus", "multigigabit", "PoE", "WiFi-optimized"],
},
{
vendor: "Ruckus (CommScope)", vendorType: "oem", vendorWebsite: "https://www.commscope.com/ruckus",
model: "ICX 7150-48PF", series: "ICX 7150", category: "Campus", layer: "L2/L3",
portsConfig: { "1G_RJ45": 48, "10G_SFP+": 4 }, totalPorts: 52,
maxSpeedGbps: 10, poeSupport: "PoE+", stackingSupport: true,
tags: ["campus", "edge", "PoE"],
},
];
// ═══════════════════════════════════════════════════════
// TRENDNET
// ═══════════════════════════════════════════════════════
const TRENDNET: SwitchSeed[] = [
{
vendor: "TRENDnet", vendorType: "oem", vendorWebsite: "https://www.trendnet.com",
model: "TPE-5048WS", series: "TPE-5048", category: "Campus", layer: "L2",
portsConfig: { "1G_RJ45": 48, "10G_SFP+": 4 }, totalPorts: 52,
maxSpeedGbps: 10, poeSupport: "PoE+",
tags: ["SMB", "PoE", "web-smart"],
},
];
// ═══════════════════════════════════════════════════════
// DRAYTEK
// ═══════════════════════════════════════════════════════
const DRAYTEK: SwitchSeed[] = [
{
vendor: "DrayTek", vendorType: "oem", vendorWebsite: "https://www.draytek.com",
model: "VigorSwitch P2540xs", series: "VigorSwitch", category: "Campus", layer: "L2",
portsConfig: { "2.5G_RJ45": 48, "10G_SFP+": 6 }, totalPorts: 54,
maxSpeedGbps: 10, poeSupport: "PoE++",
tags: ["SMB", "multigigabit", "cloud-managed"],
},
];
// ═══════════════════════════════════════════════════════
// GIGAMON (Monitoring / TAP)
// ═══════════════════════════════════════════════════════
const GIGAMON: SwitchSeed[] = [
{
vendor: "Gigamon", vendorType: "oem", vendorWebsite: "https://www.gigamon.com",
model: "GigaVUE-HC3", series: "GigaVUE-HC3", category: "DataCenter", layer: "L2",
portsConfig: { "100G_QSFP28": 32, "10G_SFP+": 64 }, totalPorts: 96,
maxSpeedGbps: 100,
tags: ["visibility", "TAP", "NPB", "monitoring"],
},
{
vendor: "Gigamon", vendorType: "oem", vendorWebsite: "https://www.gigamon.com",
model: "GigaVUE-HC1-Plus", series: "GigaVUE-HC1", category: "DataCenter", layer: "L2",
portsConfig: { "100G_QSFP28": 8, "25G_SFP28": 32 }, totalPorts: 40,
maxSpeedGbps: 100,
tags: ["visibility", "TAP", "NPB"],
},
];
// ═══════════════════════════════════════════════════════
// KEYSIGHT (ex. Ixia)
// ═══════════════════════════════════════════════════════
const KEYSIGHT: SwitchSeed[] = [
{
vendor: "Keysight (ex. Ixia)", vendorType: "oem", vendorWebsite: "https://www.keysight.com",
model: "Vision X", series: "Vision", category: "DataCenter", layer: "L2",
portsConfig: { "400G_QSFP-DD": 32 }, totalPorts: 32,
maxSpeedGbps: 400,
tags: ["NPB", "visibility", "packet-broker", "TAP"],
},
];
// ═══════════════════════════════════════════════════════
// SUPERMICRO
// ═══════════════════════════════════════════════════════
const SUPERMICRO: SwitchSeed[] = [
{
vendor: "Supermicro", vendorType: "oem", vendorWebsite: "https://www.supermicro.com",
model: "SSE-C4632SRB", series: "SSE-C4632", category: "DataCenter", layer: "L3",
portsConfig: { "400G_QSFP-DD": 32 }, totalPorts: 32,
maxSpeedGbps: 400, switchingCapacityTbps: 25.6,
asicVendor: "Broadcom", asicModel: "Memory Memory",
sonicCompatible: true, openconfigSupport: true,
tags: ["whitebox", "SONiC", "400G", "open-networking"],
},
];
// ═══════════════════════════════════════════════════════
// ADVANTECH
// ═══════════════════════════════════════════════════════
const ADVANTECH: SwitchSeed[] = [
{
vendor: "Advantech", vendorType: "oem", vendorWebsite: "https://www.advantech.com",
model: "EKI-9516G-4GMXP", series: "EKI-9500", category: "Industrial", layer: "L3",
portsConfig: { "1G_RJ45": 16, "10G_SFP+": 4 }, totalPorts: 20,
maxSpeedGbps: 10,
tags: ["industrial", "DIN-rail", "M12", "IP67"],
},
{
vendor: "Advantech", vendorType: "oem", vendorWebsite: "https://www.advantech.com",
model: "EKI-7720G-4FI", series: "EKI-7720", category: "Industrial", layer: "L2",
portsConfig: { "1G_RJ45": 16, "1G_SFP": 4 }, totalPorts: 20,
maxSpeedGbps: 1,
tags: ["industrial", "DIN-rail", "managed"],
},
];
// ═══════════════════════════════════════════════════════
// RAD DATA COMMUNICATIONS
// ═══════════════════════════════════════════════════════
const RAD: SwitchSeed[] = [
{
vendor: "RAD", vendorType: "oem", vendorWebsite: "https://www.rad.com",
model: "ETX-2i-10G", series: "ETX-2i", category: "SP", layer: "L2",
portsConfig: { "10G_SFP+": 10, "1G_RJ45": 8 }, totalPorts: 18,
maxSpeedGbps: 10, mplsSupport: true,
tags: ["CPE", "carrier", "demarcation", "MEF"],
},
];
// ═══════════════════════════════════════════════════════
// ZHONE / DASAN / DZS
// ═══════════════════════════════════════════════════════
const DZS: SwitchSeed[] = [
{
vendor: "DZS (ex. Zhone/Dasan)", vendorType: "oem", vendorWebsite: "https://www.dzsi.com",
model: "OLT 9100", series: "OLT 9100", category: "SP", layer: "L2/L3",
portsConfig: { "10G_SFP+": 16, "GPON": 16 }, totalPorts: 32,
maxSpeedGbps: 10,
tags: ["OLT", "FTTH", "GPON", "ISP"],
},
];
// ═══════════════════════════════════════════════════════
// ZTE
// ═══════════════════════════════════════════════════════
const ZTE: SwitchSeed[] = [
{
vendor: "ZTE", vendorType: "oem", vendorWebsite: "https://www.zte.com.cn",
model: "ZXR10 9908", series: "ZXR10 9900", category: "Core", layer: "L3",
portsConfig: { "400G_QSFP-DD": 36 }, totalPorts: 36,
maxSpeedGbps: 400, switchingCapacityTbps: 28.8, bgpSupport: true, mplsSupport: true,
tags: ["chassis", "core", "carrier"],
},
{
vendor: "ZTE", vendorType: "oem", vendorWebsite: "https://www.zte.com.cn",
model: "ZXR10 5960-56PM-H", series: "ZXR10 5960", category: "DataCenter", layer: "L3",
portsConfig: { "25G_SFP28": 48, "100G_QSFP28": 8 }, totalPorts: 56,
maxSpeedGbps: 100, bgpSupport: true, vxlanSupport: true,
tags: ["datacenter", "ToR", "25G"],
},
];
// ═══════════════════════════════════════════════════════
// FIBERHOME
// ═══════════════════════════════════════════════════════
const FIBERHOME: SwitchSeed[] = [
{
vendor: "FiberHome", vendorType: "oem", vendorWebsite: "https://www.fiberhome.com",
model: "CiTRANS 680", series: "CiTRANS 680", category: "SP", layer: "L2",
portsConfig: { "100G_QSFP28": 16, "10G_SFP+": 32 }, totalPorts: 48,
maxSpeedGbps: 100,
tags: ["carrier", "OTN", "packet-optical"],
},
];
// ═══════════════════════════════════════════════════════
// DATACOM
// ═══════════════════════════════════════════════════════
const DATACOM: SwitchSeed[] = [
{
vendor: "Datacom", vendorType: "oem", vendorWebsite: "https://www.datacom.com.br",
model: "DM4610-48T6X", series: "DM4610", category: "Campus", layer: "L3",
portsConfig: { "1G_RJ45": 48, "10G_SFP+": 6 }, totalPorts: 54,
maxSpeedGbps: 10, bgpSupport: true,
tags: ["campus", "aggregation", "Brazil"],
},
];
// ═══════════════════════════════════════════════════════
// NETSCOUT / NETWORK INSTRUMENTS
// ═══════════════════════════════════════════════════════
const NETSCOUT: SwitchSeed[] = [
{
vendor: "Netscout", vendorType: "oem", vendorWebsite: "https://www.netscout.com",
model: "nGeniusONE InfiniStreamNG", series: "InfiniStreamNG", category: "DataCenter", layer: "L2",
portsConfig: { "100G_QSFP28": 8 }, totalPorts: 8,
maxSpeedGbps: 100,
tags: ["monitoring", "packet-capture", "NPM"],
},
];
// ═══════════════════════════════════════════════════════
// CALETA / EVERTZ / RIEDEL (Broadcast/AV)
// ═══════════════════════════════════════════════════════
const BROADCAST_AV: SwitchSeed[] = [
{
vendor: "Evertz", vendorType: "oem", vendorWebsite: "https://www.evertz.com",
model: "EXE-VSR-IP", series: "EXE-VSR", category: "DataCenter", layer: "L2",
portsConfig: { "25G_SFP28": 48, "100G_QSFP28": 8 }, totalPorts: 56,
maxSpeedGbps: 100,
tags: ["broadcast", "SMPTE-2110", "IP-video", "SDI-over-IP"],
},
{
vendor: "Arista", vendorType: "oem", vendorWebsite: "https://www.arista.com",
model: "7130-48LB", series: "7130", category: "DataCenter", layer: "L2",
portsConfig: { "25G_SFP28": 48, "100G_QSFP28": 8 }, totalPorts: 56,
maxSpeedGbps: 100,
tags: ["broadcast", "SMPTE-2110", "low-latency", "FPGA"],
},
{
vendor: "Riedel Communications", vendorType: "oem", vendorWebsite: "https://www.riedel.net",
model: "MediorNet MicroN UHD", series: "MediorNet", category: "Edge", layer: "L2",
portsConfig: { "10G_SFP+": 8 }, totalPorts: 8,
maxSpeedGbps: 10,
tags: ["broadcast", "AV-over-IP", "real-time"],
},
];
// ═══════════════════════════════════════════════════════
// WAYSTREAM
// ═══════════════════════════════════════════════════════
const WAYSTREAM: SwitchSeed[] = [
{
vendor: "Waystream", vendorType: "oem", vendorWebsite: "https://www.waystream.com",
model: "ASR 8000", series: "ASR 8000", category: "SP", layer: "L2/L3",
portsConfig: { "10G_SFP+": 8, "1G_RJ45": 24 }, totalPorts: 32,
maxSpeedGbps: 10,
tags: ["ISP", "access", "FTTH", "triple-play"],
},
];
// ═══════════════════════════════════════════════════════
// EKINOPS / ONE ACCESS
// ═══════════════════════════════════════════════════════
const EKINOPS: SwitchSeed[] = [
{
vendor: "Ekinops", vendorType: "oem", vendorWebsite: "https://www.ekinops.com",
model: "360-12", series: "OneOS360", category: "Edge", layer: "L3",
portsConfig: { "10G_SFP+": 12, "1G_RJ45": 4 }, totalPorts: 16,
maxSpeedGbps: 10, bgpSupport: true, mplsSupport: true,
tags: ["CPE", "SD-WAN", "carrier-edge"],
},
];
// ═══════════════════════════════════════════════════════
// RIBBON COMMUNICATIONS (ex. GENBAND/Sonus)
// ═══════════════════════════════════════════════════════
const RIBBON: SwitchSeed[] = [
{
vendor: "Ribbon Communications", vendorType: "oem", vendorWebsite: "https://www.ribboncommunications.com",
model: "Apollo 9900 Series", series: "Apollo 9900", category: "SP", layer: "L2",
portsConfig: { "100G_QSFP28": 16 }, totalPorts: 16,
maxSpeedGbps: 100,
tags: ["optical", "DWDM", "carrier"],
},
];
// ═══════════════════════════════════════════════════════
// WAGO
// ═══════════════════════════════════════════════════════
const WAGO: SwitchSeed[] = [
{
vendor: "WAGO", vendorType: "oem", vendorWebsite: "https://www.wago.com",
model: "852-1505", series: "852", category: "Industrial", layer: "L2",
portsConfig: { "1G_RJ45": 8, "1G_SFP": 2 }, totalPorts: 10,
maxSpeedGbps: 1,
tags: ["industrial", "DIN-rail", "managed", "PROFINET"],
},
];
// ═══════════════════════════════════════════════════════
// PEPLINK / PEPWAVE
// ═══════════════════════════════════════════════════════
const PEPLINK: SwitchSeed[] = [
{
vendor: "Peplink", vendorType: "oem", vendorWebsite: "https://www.peplink.com",
model: "SD Switch 24-Port", series: "SD Switch", category: "Campus", layer: "L2",
portsConfig: { "1G_RJ45": 24, "10G_SFP+": 4 }, totalPorts: 28,
maxSpeedGbps: 10, poeSupport: "PoE+",
tags: ["SD-WAN", "cloud-managed", "PoE"],
},
];
// ═══════════════════════════════════════════════════════
// NETAPP / PURE STORAGE (SAN switches)
// ═══════════════════════════════════════════════════════
const STORAGE_VENDORS: SwitchSeed[] = [
{
vendor: "NetApp", vendorType: "oem", vendorWebsite: "https://www.netapp.com",
model: "CN1610", series: "CN1610", category: "DataCenter", layer: "L2",
portsConfig: { "10G_SFP+": 16 }, totalPorts: 16,
maxSpeedGbps: 10,
tags: ["storage", "cluster-interconnect", "ONTAP"],
},
{
vendor: "QNAP", vendorType: "oem", vendorWebsite: "https://www.qnap.com",
model: "QSW-M5216-1T", series: "QSW-M5216", category: "DataCenter", layer: "L2",
portsConfig: { "25G_SFP28": 16, "10G_RJ45": 1 }, totalPorts: 17,
maxSpeedGbps: 25,
tags: ["storage", "NAS", "25G"],
},
{
vendor: "Synology", vendorType: "oem", vendorWebsite: "https://www.synology.com",
model: "SA6400", series: "SA6400", category: "DataCenter", layer: "L2",
portsConfig: { "25G_SFP28": 4 }, totalPorts: 4,
maxSpeedGbps: 25,
tags: ["NAS", "storage", "25G"],
},
];
// ═══════════════════════════════════════════════════════
// QUANTA CLOUD TECHNOLOGY (QCT)
// ═══════════════════════════════════════════════════════
const QCT: SwitchSeed[] = [
{
vendor: "Quanta Cloud Technology", vendorType: "oem", vendorWebsite: "https://www.qct.io",
model: "QuantaMesh T7064-IX1D", series: "T7064", category: "DataCenter", layer: "L3",
portsConfig: { "100G_QSFP28": 64 }, totalPorts: 64,
maxSpeedGbps: 100, switchingCapacityTbps: 12.8,
asicVendor: "Broadcom", sonicCompatible: true, openconfigSupport: true,
tags: ["whitebox", "SONiC", "spine", "100G"],
},
];
// ═══════════════════════════════════════════════════════
// ASTERFUSION
// ═══════════════════════════════════════════════════════
const ASTERFUSION: SwitchSeed[] = [
{
vendor: "Asterfusion", vendorType: "oem", vendorWebsite: "https://www.asterfusion.com",
model: "CX864E-N", series: "CX864", category: "DataCenter", layer: "L3",
portsConfig: { "800G_OSFP": 64 }, totalPorts: 64,
maxSpeedGbps: 800, switchingCapacityTbps: 51.2,
asicVendor: "Broadcom", asicModel: "Tomahawk 5",
sonicCompatible: true, openconfigSupport: true,
tags: ["whitebox", "800G", "AI-fabric"],
},
];
// ═══════════════════════════════════════════════════════
// DELL SONICWALL (Security)
// ═══════════════════════════════════════════════════════
const SONICWALL: SwitchSeed[] = [
{
vendor: "SonicWall", vendorType: "oem", vendorWebsite: "https://www.sonicwall.com",
model: "NSa 6700", series: "NSa 6700", category: "Edge", layer: "L3",
portsConfig: { "10G_SFP+": 4, "25G_SFP28": 2, "1G_RJ45": 24 }, totalPorts: 30,
maxSpeedGbps: 25,
tags: ["firewall", "NGFW", "security"],
},
];
// ═══════════════════════════════════════════════════════
// WATCHGUARD
// ═══════════════════════════════════════════════════════
const WATCHGUARD: SwitchSeed[] = [
{
vendor: "WatchGuard", vendorType: "oem", vendorWebsite: "https://www.watchguard.com",
model: "Firebox M5800", series: "Firebox M5800", category: "Edge", layer: "L3",
portsConfig: { "10G_SFP+": 4, "1G_RJ45": 8 }, totalPorts: 12,
maxSpeedGbps: 10,
tags: ["firewall", "UTM", "security"],
},
];
// ═══════════════════════════════════════════════════════
// BARRACUDA
// ═══════════════════════════════════════════════════════
const BARRACUDA: SwitchSeed[] = [
{
vendor: "Barracuda Networks", vendorType: "oem", vendorWebsite: "https://www.barracuda.com",
model: "CloudGen Firewall F900", series: "CGF F900", category: "Edge", layer: "L3",
portsConfig: { "10G_SFP+": 4, "1G_RJ45": 8 }, totalPorts: 12,
maxSpeedGbps: 10,
tags: ["firewall", "SD-WAN", "security"],
},
];
// ═══════════════════════════════════════════════════════
// SOPHOS
// ═══════════════════════════════════════════════════════
const SOPHOS: SwitchSeed[] = [
{
vendor: "Sophos", vendorType: "oem", vendorWebsite: "https://www.sophos.com",
model: "XGS 6500", series: "XGS 6500", category: "Edge", layer: "L3",
portsConfig: { "25G_SFP28": 4, "10G_SFP+": 8, "1G_RJ45": 8 }, totalPorts: 20,
maxSpeedGbps: 25,
tags: ["firewall", "NGFW", "security", "Xstream"],
},
{
vendor: "Sophos", vendorType: "oem", vendorWebsite: "https://www.sophos.com",
model: "CS210-48FP", series: "CS210", category: "Campus", layer: "L2",
portsConfig: { "1G_RJ45": 48, "10G_SFP+": 4 }, totalPorts: 52,
maxSpeedGbps: 10, poeSupport: "PoE+",
tags: ["campus", "cloud-managed", "PoE"],
},
];
// ═══════════════════════════════════════════════════════
// CITRIX / NETSCALER (Load Balancers)
// ═══════════════════════════════════════════════════════
const CITRIX: SwitchSeed[] = [
{
vendor: "Citrix (NetScaler)", vendorType: "oem", vendorWebsite: "https://www.citrix.com",
model: "NetScaler SDX 26000-100G", series: "SDX 26000", category: "DataCenter", layer: "L3",
portsConfig: { "100G_QSFP28": 4, "25G_SFP28": 8 }, totalPorts: 12,
maxSpeedGbps: 100,
tags: ["ADC", "load-balancer", "SSL"],
},
];
// ═══════════════════════════════════════════════════════
// KEMP (Load Balancers)
// ═══════════════════════════════════════════════════════
const KEMP: SwitchSeed[] = [
{
vendor: "Kemp Technologies", vendorType: "oem", vendorWebsite: "https://www.kemp.ax",
model: "LoadMaster LM-X40", series: "LoadMaster", category: "DataCenter", layer: "L3",
portsConfig: { "40G_QSFP+": 4, "10G_SFP+": 8 }, totalPorts: 12,
maxSpeedGbps: 40,
tags: ["ADC", "load-balancer"],
},
];
// ═══════════════════════════════════════════════════════
// A10 NETWORKS
// ═══════════════════════════════════════════════════════
const A10: SwitchSeed[] = [
{
vendor: "A10 Networks", vendorType: "oem", vendorWebsite: "https://www.a10networks.com",
model: "Thunder 14045", series: "Thunder", category: "DataCenter", layer: "L3",
portsConfig: { "100G_QSFP28": 8, "25G_SFP28": 16 }, totalPorts: 24,
maxSpeedGbps: 100,
tags: ["ADC", "DDoS", "load-balancer", "CGN"],
},
];
// ═══════════════════════════════════════════════════════
// NIC VENDORS (Intel, Broadcom, Chelsio, Solarflare)
// ═══════════════════════════════════════════════════════
const NIC_VENDORS: SwitchSeed[] = [
{
vendor: "Intel", vendorType: "oem", vendorWebsite: "https://www.intel.com",
model: "E810-CQDA2", series: "E810", category: "DataCenter", layer: "L2",
portsConfig: { "100G_QSFP28": 2 }, totalPorts: 2,
maxSpeedGbps: 100,
tags: ["NIC", "SmartNIC", "DPDK", "iWARP"],
},
{
vendor: "Broadcom", vendorType: "oem", vendorWebsite: "https://www.broadcom.com",
model: "BCM957508-P2100G", series: "BCM957508", category: "DataCenter", layer: "L2",
portsConfig: { "100G_QSFP56": 2 }, totalPorts: 2,
maxSpeedGbps: 100,
tags: ["NIC", "SmartNIC", "RoCE"],
},
{
vendor: "NVIDIA Networking", vendorType: "oem", vendorWebsite: "https://www.nvidia.com/networking",
model: "ConnectX-7 400G", series: "ConnectX-7", category: "DataCenter", layer: "L2",
portsConfig: { "400G_QSFP-DD": 1 }, totalPorts: 1,
maxSpeedGbps: 400,
tags: ["NIC", "SmartNIC", "InfiniBand", "DPU", "AI"],
},
];
// ═══════════════════════════════════════════════════════
// COMBINE ALL SEEDS
// ═══════════════════════════════════════════════════════
const ALL_BULK_SEEDS: SwitchSeed[] = [
...DLINK,
...ALE,
...BROCADE,
...H3C,
...RUIJIE,
...PLANET,
...LANCOM,
...CIENA,
...ADTRAN,
...CALIX,
...CAMBIUM,
...AVAYA,
...FUJITSU,
...NEC,
...RUCKUS,
...TRENDNET,
...DRAYTEK,
...GIGAMON,
...KEYSIGHT,
...SUPERMICRO,
...ADVANTECH,
...RAD,
...DZS,
...ZTE,
...FIBERHOME,
...DATACOM,
...NETSCOUT,
...BROADCAST_AV,
...WAYSTREAM,
...EKINOPS,
...RIBBON,
...WAGO,
...PEPLINK,
...STORAGE_VENDORS,
...QCT,
...ASTERFUSION,
...SONICWALL,
...WATCHGUARD,
...BARRACUDA,
...SOPHOS,
...CITRIX,
...KEMP,
...A10,
...NIC_VENDORS,
];
// ═══════════════════════════════════════════════════════
// Seed function
// ═══════════════════════════════════════════════════════
export async function seedBulkSwitches(): Promise<void> {
console.log(`\n=== Seeding ${ALL_BULK_SEEDS.length} Bulk Switch/Router Models ===\n`);
const vendorCache = new Map<string, string>();
let created = 0;
let updated = 0;
for (const sw of ALL_BULK_SEEDS) {
try {
let vendorId = vendorCache.get(sw.vendor);
if (!vendorId) {
vendorId = await ensureVendor(sw.vendor, sw.vendorType, sw.vendorWebsite);
vendorCache.set(sw.vendor, vendorId);
}
const existing = await pool.query(
`SELECT id FROM switches WHERE model = $1 AND vendor_id = $2`,
[sw.model, vendorId]
);
await findOrCreateSwitch({
model: sw.model,
vendorId,
series: sw.series,
category: sw.category,
layer: sw.layer,
portsConfig: sw.portsConfig,
totalPorts: sw.totalPorts,
uplinkSpeedGbps: sw.uplinkSpeedGbps,
maxSpeedGbps: sw.maxSpeedGbps,
switchingCapacityTbps: sw.switchingCapacityTbps,
forwardingRateMpps: sw.forwardingRateMpps,
asicVendor: sw.asicVendor,
asicModel: sw.asicModel,
rackUnits: sw.rackUnits,
maxPowerW: sw.maxPowerW,
poeSupport: sw.poeSupport,
stackingSupport: sw.stackingSupport,
vxlanSupport: sw.vxlanSupport,
evpnSupport: sw.evpnSupport,
bgpSupport: sw.bgpSupport,
mplsSupport: sw.mplsSupport,
openconfigSupport: sw.openconfigSupport,
sonicCompatible: sw.sonicCompatible,
macsecSupport: sw.macsecSupport,
tags: sw.tags,
});
if (existing.rows.length > 0) {
updated++;
} else {
created++;
console.log(`${sw.vendor} ${sw.model}`);
}
} catch (err: unknown) {
const msg = err instanceof Error ? err.message : String(err);
console.error(`${sw.vendor} ${sw.model}: ${msg}`);
}
}
console.log(`\n=== Bulk Seed Complete: ${created} created, ${updated} updated ===`);
}
if (require.main === module) {
seedBulkSwitches()
.then(() => pool.end())
.catch((err) => { console.error("Fatal:", err); pool.end(); process.exit(1); });
}

View File

@ -0,0 +1,695 @@
/**
* Extended Switch & Router Seed Data Security, Industrial, Broadcast & Misc Vendors
*
* Covers Flexoptix-supported vendors beyond the core networking OEMs.
* These devices all have SFP/SFP+/QSFP ports and use optical transceivers.
* Sources: Public datasheets, vendor product pages.
*/
import { pool, ensureVendor, findOrCreateSwitch } from "../utils/db";
interface SwitchSeed {
vendor: string;
vendorType: string;
vendorWebsite: string;
model: string;
series: string;
category: "DataCenter" | "Campus" | "Edge" | "Core" | "SP" | "Industrial";
layer: "L2" | "L3" | "L2/L3";
portsConfig: Record<string, number>;
totalPorts: number;
uplinkSpeedGbps?: number;
maxSpeedGbps: number;
switchingCapacityTbps?: number;
forwardingRateMpps?: number;
asicVendor?: string;
asicModel?: string;
rackUnits?: number;
maxPowerW?: number;
poeSupport?: string;
stackingSupport?: boolean;
vxlanSupport?: boolean;
evpnSupport?: boolean;
bgpSupport?: boolean;
mplsSupport?: boolean;
openconfigSupport?: boolean;
sonicCompatible?: boolean;
macsecSupport?: boolean;
lifecycleStatus?: string;
tags?: string[];
}
// ═══════════════════════════════════════════════════════
// FORTINET — FortiSwitch + FortiGate (with SFP ports)
// ═══════════════════════════════════════════════════════
const FORTINET: SwitchSeed[] = [
// FortiSwitch 100 Series — Access
{
vendor: "Fortinet", vendorType: "oem", vendorWebsite: "https://www.fortinet.com",
model: "FortiSwitch 108F", series: "FortiSwitch 100", category: "Campus", layer: "L2",
portsConfig: { "1G_RJ45": 8, "1G_SFP": 2 }, totalPorts: 10,
maxSpeedGbps: 1,
rackUnits: 0, maxPowerW: 15,
tags: ["access", "managed", "FortiLink"],
},
{
vendor: "Fortinet", vendorType: "oem", vendorWebsite: "https://www.fortinet.com",
model: "FortiSwitch 124F", series: "FortiSwitch 100", category: "Campus", layer: "L2",
portsConfig: { "1G_RJ45": 24, "10G_SFP+": 4 }, totalPorts: 28,
maxSpeedGbps: 10,
rackUnits: 1, maxPowerW: 30,
tags: ["access", "managed", "FortiLink"],
},
{
vendor: "Fortinet", vendorType: "oem", vendorWebsite: "https://www.fortinet.com",
model: "FortiSwitch 124F-POE", series: "FortiSwitch 100", category: "Campus", layer: "L2",
portsConfig: { "1G_RJ45": 24, "10G_SFP+": 4 }, totalPorts: 28,
maxSpeedGbps: 10,
rackUnits: 1, maxPowerW: 370,
poeSupport: "PoE+",
tags: ["access", "PoE", "FortiLink"],
},
{
vendor: "Fortinet", vendorType: "oem", vendorWebsite: "https://www.fortinet.com",
model: "FortiSwitch 148F-POE", series: "FortiSwitch 100", category: "Campus", layer: "L2",
portsConfig: { "1G_RJ45": 48, "10G_SFP+": 4 }, totalPorts: 52,
maxSpeedGbps: 10,
rackUnits: 1, maxPowerW: 740,
poeSupport: "PoE+",
tags: ["access", "PoE", "FortiLink"],
},
// FortiSwitch 400 Series — Aggregation
{
vendor: "Fortinet", vendorType: "oem", vendorWebsite: "https://www.fortinet.com",
model: "FortiSwitch 424E", series: "FortiSwitch 400", category: "Campus", layer: "L2/L3",
portsConfig: { "1G_RJ45": 24, "10G_SFP+": 4 }, totalPorts: 28,
maxSpeedGbps: 10,
rackUnits: 1, maxPowerW: 45,
tags: ["aggregation", "FortiLink"],
},
{
vendor: "Fortinet", vendorType: "oem", vendorWebsite: "https://www.fortinet.com",
model: "FortiSwitch 448E-FPOE", series: "FortiSwitch 400", category: "Campus", layer: "L2/L3",
portsConfig: { "1G_RJ45": 48, "10G_SFP+": 4 }, totalPorts: 52,
maxSpeedGbps: 10,
rackUnits: 1, maxPowerW: 780,
poeSupport: "PoE+",
tags: ["aggregation", "PoE", "FortiLink"],
},
// FortiSwitch 500 Series — Distribution
{
vendor: "Fortinet", vendorType: "oem", vendorWebsite: "https://www.fortinet.com",
model: "FortiSwitch 524D", series: "FortiSwitch 500", category: "Campus", layer: "L2/L3",
portsConfig: { "1G_RJ45": 24, "10G_SFP+": 4, "40G_QSFP+": 2 }, totalPorts: 30,
maxSpeedGbps: 40,
rackUnits: 1, maxPowerW: 65,
tags: ["distribution", "40G", "FortiLink"],
},
{
vendor: "Fortinet", vendorType: "oem", vendorWebsite: "https://www.fortinet.com",
model: "FortiSwitch 548D-FPOE", series: "FortiSwitch 500", category: "Campus", layer: "L2/L3",
portsConfig: { "1G_RJ45": 48, "10G_SFP+": 4, "40G_QSFP+": 2 }, totalPorts: 54,
maxSpeedGbps: 40,
rackUnits: 1, maxPowerW: 780,
poeSupport: "PoE+",
tags: ["distribution", "40G", "PoE", "FortiLink"],
},
// FortiSwitch 1000/3000 Series — Core
{
vendor: "Fortinet", vendorType: "oem", vendorWebsite: "https://www.fortinet.com",
model: "FortiSwitch 1024E", series: "FortiSwitch 1000", category: "DataCenter", layer: "L2/L3",
portsConfig: { "10G_SFP+": 24, "40G_QSFP+": 4 }, totalPorts: 28,
maxSpeedGbps: 40,
rackUnits: 1, maxPowerW: 150,
tags: ["10G", "core", "FortiLink"],
},
{
vendor: "Fortinet", vendorType: "oem", vendorWebsite: "https://www.fortinet.com",
model: "FortiSwitch 1048E", series: "FortiSwitch 1000", category: "DataCenter", layer: "L2/L3",
portsConfig: { "10G_SFP+": 48, "40G_QSFP+": 6 }, totalPorts: 54,
maxSpeedGbps: 40,
rackUnits: 1, maxPowerW: 200,
tags: ["10G", "core", "FortiLink"],
},
{
vendor: "Fortinet", vendorType: "oem", vendorWebsite: "https://www.fortinet.com",
model: "FortiSwitch 3032E", series: "FortiSwitch 3000", category: "DataCenter", layer: "L2/L3",
portsConfig: { "100G_QSFP28": 32 }, totalPorts: 32,
maxSpeedGbps: 100,
rackUnits: 1, maxPowerW: 350,
tags: ["100G", "spine", "FortiLink"],
},
];
// ═══════════════════════════════════════════════════════
// MIKROTIK — CRS / CCR Series (with SFP/QSFP ports)
// ═══════════════════════════════════════════════════════
const MIKROTIK: SwitchSeed[] = [
{
vendor: "MikroTik", vendorType: "oem", vendorWebsite: "https://mikrotik.com",
model: "CRS504-4XQ-IN", series: "CRS504", category: "DataCenter", layer: "L2/L3",
portsConfig: { "100G_QSFP28": 4 }, totalPorts: 4,
maxSpeedGbps: 100, switchingCapacityTbps: 0.8,
asicVendor: "Marvell", asicModel: "Prestera 98DX8525",
rackUnits: 1, maxPowerW: 75,
tags: ["100G", "aggregation", "RouterOS"],
},
{
vendor: "MikroTik", vendorType: "oem", vendorWebsite: "https://mikrotik.com",
model: "CRS518-16XS-2XQ", series: "CRS518", category: "DataCenter", layer: "L2/L3",
portsConfig: { "25G_SFP28": 16, "100G_QSFP28": 2 }, totalPorts: 18,
maxSpeedGbps: 100, switchingCapacityTbps: 1.2,
asicVendor: "Marvell", asicModel: "Prestera 98DX8525",
rackUnits: 1, maxPowerW: 85,
tags: ["25G", "aggregation", "RouterOS"],
},
{
vendor: "MikroTik", vendorType: "oem", vendorWebsite: "https://mikrotik.com",
model: "CRS354-48G-4S+2Q+", series: "CRS354", category: "Campus", layer: "L2/L3",
portsConfig: { "1G_RJ45": 48, "10G_SFP+": 4, "40G_QSFP+": 2 }, totalPorts: 54,
maxSpeedGbps: 40,
rackUnits: 1, maxPowerW: 55,
tags: ["campus", "aggregation", "RouterOS"],
},
{
vendor: "MikroTik", vendorType: "oem", vendorWebsite: "https://mikrotik.com",
model: "CRS326-24G-2S+", series: "CRS326", category: "Campus", layer: "L2",
portsConfig: { "1G_RJ45": 24, "10G_SFP+": 2 }, totalPorts: 26,
maxSpeedGbps: 10,
rackUnits: 1, maxPowerW: 30,
tags: ["campus", "access", "RouterOS"],
},
{
vendor: "MikroTik", vendorType: "oem", vendorWebsite: "https://mikrotik.com",
model: "CRS317-1G-16S+", series: "CRS317", category: "DataCenter", layer: "L2/L3",
portsConfig: { "10G_SFP+": 16, "1G_RJ45": 1 }, totalPorts: 17,
maxSpeedGbps: 10, switchingCapacityTbps: 0.32,
asicVendor: "Marvell", asicModel: "Prestera 98DX8216",
rackUnits: 1, maxPowerW: 45,
tags: ["10G", "aggregation", "RouterOS"],
},
{
vendor: "MikroTik", vendorType: "oem", vendorWebsite: "https://mikrotik.com",
model: "CRS312-4C+8XG", series: "CRS312", category: "DataCenter", layer: "L2/L3",
portsConfig: { "10G_RJ45": 8, "10G_SFP+_Combo": 4 }, totalPorts: 12,
maxSpeedGbps: 10,
rackUnits: 1, maxPowerW: 75,
tags: ["10G", "combo", "RouterOS"],
},
{
vendor: "MikroTik", vendorType: "oem", vendorWebsite: "https://mikrotik.com",
model: "CRS305-1G-4S+", series: "CRS305", category: "Edge", layer: "L2",
portsConfig: { "10G_SFP+": 4, "1G_RJ45": 1 }, totalPorts: 5,
maxSpeedGbps: 10,
rackUnits: 0, maxPowerW: 15,
tags: ["10G", "desktop", "RouterOS"],
},
// CCR Routers with SFP
{
vendor: "MikroTik", vendorType: "oem", vendorWebsite: "https://mikrotik.com",
model: "CCR2216-1G-12XS-2XQ", series: "CCR2216", category: "Core", layer: "L3",
portsConfig: { "25G_SFP28": 12, "100G_QSFP28": 2, "1G_RJ45": 1 }, totalPorts: 15,
maxSpeedGbps: 100,
rackUnits: 1, maxPowerW: 130,
bgpSupport: true, mplsSupport: true,
tags: ["router", "100G", "BGP", "MPLS", "RouterOS"],
},
];
// ═══════════════════════════════════════════════════════
// UBIQUITI — UniFi / EdgeSwitch with SFP
// ═══════════════════════════════════════════════════════
const UBIQUITI: SwitchSeed[] = [
{
vendor: "Ubiquiti", vendorType: "oem", vendorWebsite: "https://www.ui.com",
model: "USW-Pro-Max-48-PoE", series: "UniFi Pro Max", category: "Campus", layer: "L2/L3",
portsConfig: { "2.5G_RJ45": 48, "10G_SFP+": 4 }, totalPorts: 52,
maxSpeedGbps: 10,
rackUnits: 1, maxPowerW: 720,
poeSupport: "PoE++",
tags: ["campus", "PoE", "UniFi"],
},
{
vendor: "Ubiquiti", vendorType: "oem", vendorWebsite: "https://www.ui.com",
model: "USW-Enterprise-48-PoE", series: "UniFi Enterprise", category: "Campus", layer: "L2/L3",
portsConfig: { "2.5G_RJ45": 48, "10G_SFP+": 4 }, totalPorts: 52,
maxSpeedGbps: 10,
rackUnits: 1, maxPowerW: 720,
poeSupport: "PoE++",
tags: ["enterprise", "PoE", "UniFi"],
},
{
vendor: "Ubiquiti", vendorType: "oem", vendorWebsite: "https://www.ui.com",
model: "USW-Aggregation", series: "UniFi Aggregation", category: "DataCenter", layer: "L2",
portsConfig: { "10G_SFP+": 8 }, totalPorts: 8,
maxSpeedGbps: 10,
rackUnits: 1, maxPowerW: 40,
tags: ["aggregation", "10G", "UniFi"],
},
{
vendor: "Ubiquiti", vendorType: "oem", vendorWebsite: "https://www.ui.com",
model: "USW-Pro-Aggregation", series: "UniFi Pro", category: "DataCenter", layer: "L2/L3",
portsConfig: { "10G_SFP+": 28, "25G_SFP28": 4 }, totalPorts: 32,
maxSpeedGbps: 25,
rackUnits: 1, maxPowerW: 75,
tags: ["aggregation", "25G", "UniFi"],
},
];
// ═══════════════════════════════════════════════════════
// NETGEAR — M4300/M4350/M4500 Managed with SFP
// ═══════════════════════════════════════════════════════
const NETGEAR: SwitchSeed[] = [
{
vendor: "Netgear", vendorType: "oem", vendorWebsite: "https://www.netgear.com",
model: "M4350-48G4XF", series: "M4350", category: "Campus", layer: "L2/L3",
portsConfig: { "1G_RJ45": 48, "10G_SFP+": 4 }, totalPorts: 52,
maxSpeedGbps: 10,
rackUnits: 1,
poeSupport: "PoE++", stackingSupport: true,
tags: ["campus", "PoE", "ProAV"],
},
{
vendor: "Netgear", vendorType: "oem", vendorWebsite: "https://www.netgear.com",
model: "M4500-32C", series: "M4500", category: "DataCenter", layer: "L2/L3",
portsConfig: { "100G_QSFP28": 32 }, totalPorts: 32,
maxSpeedGbps: 100, switchingCapacityTbps: 6.4,
rackUnits: 1,
tags: ["100G", "spine", "ProAV"],
},
{
vendor: "Netgear", vendorType: "oem", vendorWebsite: "https://www.netgear.com",
model: "M4300-96X", series: "M4300", category: "Campus", layer: "L2/L3",
portsConfig: { "10G_RJ45": 48, "10G_SFP+": 48 }, totalPorts: 96,
maxSpeedGbps: 10,
rackUnits: 2,
stackingSupport: true,
tags: ["10G", "campus", "ProAV"],
},
];
// ═══════════════════════════════════════════════════════
// ALLIED TELESIS — Campus/Industrial
// ═══════════════════════════════════════════════════════
const ALLIED_TELESIS: SwitchSeed[] = [
{
vendor: "Allied Telesis", vendorType: "oem", vendorWebsite: "https://www.alliedtelesis.com",
model: "AT-x950-28XSQ", series: "x950", category: "Campus", layer: "L3",
portsConfig: { "10G_SFP+": 24, "40G_QSFP+": 4 }, totalPorts: 28,
maxSpeedGbps: 40, switchingCapacityTbps: 0.96,
rackUnits: 1,
stackingSupport: true, bgpSupport: true,
tags: ["10G", "core", "stackable"],
},
{
vendor: "Allied Telesis", vendorType: "oem", vendorWebsite: "https://www.alliedtelesis.com",
model: "AT-x530-28GSX", series: "x530", category: "Campus", layer: "L3",
portsConfig: { "1G_SFP": 24, "10G_SFP+": 4 }, totalPorts: 28,
maxSpeedGbps: 10,
rackUnits: 1,
stackingSupport: true,
tags: ["campus", "all-fiber", "stackable"],
},
{
vendor: "Allied Telesis", vendorType: "oem", vendorWebsite: "https://www.alliedtelesis.com",
model: "AT-x530L-52GPX", series: "x530L", category: "Campus", layer: "L3",
portsConfig: { "1G_RJ45": 48, "10G_SFP+": 4 }, totalPorts: 52,
maxSpeedGbps: 10,
rackUnits: 1,
poeSupport: "PoE+", stackingSupport: true,
tags: ["campus", "PoE", "stackable"],
},
];
// ═══════════════════════════════════════════════════════
// HIRSCHMANN / BELDEN — Industrial Ethernet
// ═══════════════════════════════════════════════════════
const HIRSCHMANN: SwitchSeed[] = [
{
vendor: "Hirschmann", vendorType: "oem", vendorWebsite: "https://www.belden.com/brands/hirschmann",
model: "MACH4002-48G-L3P", series: "MACH4002", category: "Industrial", layer: "L3",
portsConfig: { "1G_RJ45": 48, "10G_SFP+": 4 }, totalPorts: 52,
maxSpeedGbps: 10,
rackUnits: 2,
poeSupport: "PoE+",
tags: ["industrial", "managed", "ruggedized", "DIN-rail"],
},
{
vendor: "Hirschmann", vendorType: "oem", vendorWebsite: "https://www.belden.com/brands/hirschmann",
model: "RSP30-08033O6TT-SK", series: "RSP", category: "Industrial", layer: "L2",
portsConfig: { "1G_RJ45": 8, "1G_SFP": 3 }, totalPorts: 11,
maxSpeedGbps: 1,
rackUnits: 0,
tags: ["industrial", "managed", "DIN-rail", "compact"],
},
{
vendor: "Hirschmann", vendorType: "oem", vendorWebsite: "https://www.belden.com/brands/hirschmann",
model: "GREYHOUND-1040-BT", series: "GREYHOUND", category: "Industrial", layer: "L3",
portsConfig: { "1G_RJ45": 8, "10G_SFP+": 4 }, totalPorts: 12,
maxSpeedGbps: 10,
rackUnits: 0,
tags: ["industrial", "ruggedized", "backbone", "10G"],
},
{
vendor: "Hirschmann", vendorType: "oem", vendorWebsite: "https://www.belden.com/brands/hirschmann",
model: "DRAGON-MACH4500-48G6XG", series: "DRAGON", category: "Industrial", layer: "L3",
portsConfig: { "1G_RJ45": 48, "10G_SFP+": 6 }, totalPorts: 54,
maxSpeedGbps: 10,
rackUnits: 2,
tags: ["industrial", "modular", "backbone", "redundancy"],
},
];
// ═══════════════════════════════════════════════════════
// MOXA — Industrial Ethernet
// ═══════════════════════════════════════════════════════
const MOXA: SwitchSeed[] = [
{
vendor: "Moxa", vendorType: "oem", vendorWebsite: "https://www.moxa.com",
model: "IKS-G6824A", series: "IKS-G6824A", category: "Industrial", layer: "L2/L3",
portsConfig: { "1G_RJ45": 24, "1G_SFP": 4 }, totalPorts: 28,
maxSpeedGbps: 1,
rackUnits: 2,
tags: ["industrial", "managed", "rack-mount", "redundancy"],
},
{
vendor: "Moxa", vendorType: "oem", vendorWebsite: "https://www.moxa.com",
model: "ICS-G7826A", series: "ICS-G7826A", category: "Industrial", layer: "L2/L3",
portsConfig: { "1G_RJ45": 24, "1G_SFP": 2 }, totalPorts: 26,
maxSpeedGbps: 1,
rackUnits: 2,
tags: ["industrial", "modular", "rack-mount"],
},
{
vendor: "Moxa", vendorType: "oem", vendorWebsite: "https://www.moxa.com",
model: "EDS-G4014", series: "EDS-G4014", category: "Industrial", layer: "L2",
portsConfig: { "1G_RJ45": 8, "1G_SFP": 6 }, totalPorts: 14,
maxSpeedGbps: 1,
rackUnits: 0,
tags: ["industrial", "managed", "DIN-rail", "compact"],
},
{
vendor: "Moxa", vendorType: "oem", vendorWebsite: "https://www.moxa.com",
model: "EDS-518E", series: "EDS-500E", category: "Industrial", layer: "L2",
portsConfig: { "100M_RJ45": 14, "1G_SFP": 4 }, totalPorts: 18,
maxSpeedGbps: 1,
rackUnits: 0,
tags: ["industrial", "managed", "DIN-rail"],
},
];
// ═══════════════════════════════════════════════════════
// SIEMENS — SCALANCE Industrial Switches
// ═══════════════════════════════════════════════════════
const SIEMENS: SwitchSeed[] = [
{
vendor: "Siemens", vendorType: "oem", vendorWebsite: "https://www.siemens.com",
model: "SCALANCE XR528-6M", series: "SCALANCE XR500", category: "Industrial", layer: "L3",
portsConfig: { "1G_RJ45": 24, "10G_SFP+": 4 }, totalPorts: 28,
maxSpeedGbps: 10,
rackUnits: 2,
tags: ["industrial", "backbone", "rack-mount", "PROFINET"],
},
{
vendor: "Siemens", vendorType: "oem", vendorWebsite: "https://www.siemens.com",
model: "SCALANCE XM416-4C", series: "SCALANCE XM400", category: "Industrial", layer: "L3",
portsConfig: { "1G_RJ45": 12, "10G_SFP+": 4 }, totalPorts: 16,
maxSpeedGbps: 10,
rackUnits: 0,
tags: ["industrial", "managed", "PROFINET", "10G"],
},
{
vendor: "Siemens", vendorType: "oem", vendorWebsite: "https://www.siemens.com",
model: "SCALANCE XC216-4C", series: "SCALANCE XC200", category: "Industrial", layer: "L2",
portsConfig: { "100M_RJ45": 16, "1G_SFP_Combo": 4 }, totalPorts: 20,
maxSpeedGbps: 1,
rackUnits: 0,
tags: ["industrial", "managed", "compact", "PROFINET"],
},
{
vendor: "Siemens", vendorType: "oem", vendorWebsite: "https://www.siemens.com",
model: "SCALANCE XR324-12M", series: "SCALANCE XR300", category: "Industrial", layer: "L2",
portsConfig: { "1G_RJ45": 24, "1G_SFP": 4 }, totalPorts: 28,
maxSpeedGbps: 1,
rackUnits: 1,
tags: ["industrial", "rack-mount", "PROFINET", "redundancy"],
},
];
// ═══════════════════════════════════════════════════════
// PHOENIX CONTACT — Industrial Networking
// ═══════════════════════════════════════════════════════
const PHOENIX_CONTACT: SwitchSeed[] = [
{
vendor: "Phoenix Contact", vendorType: "oem", vendorWebsite: "https://www.phoenixcontact.com",
model: "FL SWITCH 4808E-16FX-4GC", series: "FL SWITCH 4800", category: "Industrial", layer: "L2",
portsConfig: { "100M_FX_SFP": 16, "1G_SFP_Combo": 4 }, totalPorts: 20,
maxSpeedGbps: 1,
rackUnits: 0,
tags: ["industrial", "managed", "DIN-rail"],
},
{
vendor: "Phoenix Contact", vendorType: "oem", vendorWebsite: "https://www.phoenixcontact.com",
model: "FL SWITCH 7528-2S", series: "FL SWITCH 7500", category: "Industrial", layer: "L3",
portsConfig: { "1G_RJ45": 24, "1G_SFP": 4 }, totalPorts: 28,
maxSpeedGbps: 1,
rackUnits: 1,
tags: ["industrial", "managed", "rack-mount", "L3"],
},
];
// ═══════════════════════════════════════════════════════
// F5 NETWORKS — BIG-IP with SFP ports
// ═══════════════════════════════════════════════════════
const F5: SwitchSeed[] = [
{
vendor: "F5 Networks", vendorType: "oem", vendorWebsite: "https://www.f5.com",
model: "BIG-IP i5800", series: "BIG-IP i-Series", category: "DataCenter", layer: "L3",
portsConfig: { "10G_SFP+": 8, "40G_QSFP+": 4, "1G_RJ45": 4 }, totalPorts: 16,
maxSpeedGbps: 40,
rackUnits: 1,
tags: ["load-balancer", "ADC", "WAF"],
},
{
vendor: "F5 Networks", vendorType: "oem", vendorWebsite: "https://www.f5.com",
model: "BIG-IP i10800", series: "BIG-IP i-Series", category: "DataCenter", layer: "L3",
portsConfig: { "10G_SFP+": 16, "40G_QSFP+": 4, "1G_RJ45": 4 }, totalPorts: 24,
maxSpeedGbps: 40,
rackUnits: 2,
tags: ["load-balancer", "ADC", "WAF", "high-perf"],
},
{
vendor: "F5 Networks", vendorType: "oem", vendorWebsite: "https://www.f5.com",
model: "BIG-IP i15800", series: "BIG-IP i-Series", category: "DataCenter", layer: "L3",
portsConfig: { "25G_SFP28": 16, "100G_QSFP28": 4, "10G_RJ45": 4 }, totalPorts: 24,
maxSpeedGbps: 100,
rackUnits: 2,
tags: ["load-balancer", "ADC", "WAF", "100G"],
},
];
// ═══════════════════════════════════════════════════════
// TP-LINK — JetStream Managed with SFP
// ═══════════════════════════════════════════════════════
const TPLINK: SwitchSeed[] = [
{
vendor: "TP-Link", vendorType: "oem", vendorWebsite: "https://www.tp-link.com",
model: "TL-SX3016F", series: "JetStream", category: "DataCenter", layer: "L2/L3",
portsConfig: { "10G_SFP+": 16 }, totalPorts: 16,
maxSpeedGbps: 10,
rackUnits: 1,
tags: ["10G", "all-fiber", "managed"],
},
{
vendor: "TP-Link", vendorType: "oem", vendorWebsite: "https://www.tp-link.com",
model: "TL-SG3452XP", series: "JetStream", category: "Campus", layer: "L2/L3",
portsConfig: { "1G_RJ45": 48, "10G_SFP+": 4 }, totalPorts: 52,
maxSpeedGbps: 10,
rackUnits: 1,
poeSupport: "PoE+",
tags: ["campus", "PoE", "managed"],
},
];
// ═══════════════════════════════════════════════════════
// ZYXEL — Managed Switches with SFP
// ═══════════════════════════════════════════════════════
const ZYXEL: SwitchSeed[] = [
{
vendor: "Zyxel", vendorType: "oem", vendorWebsite: "https://www.zyxel.com",
model: "XGS4600-52F", series: "XGS4600", category: "Campus", layer: "L3",
portsConfig: { "1G_SFP": 48, "10G_SFP+": 4 }, totalPorts: 52,
maxSpeedGbps: 10,
rackUnits: 1,
stackingSupport: true,
tags: ["campus", "all-fiber", "L3", "stackable"],
},
{
vendor: "Zyxel", vendorType: "oem", vendorWebsite: "https://www.zyxel.com",
model: "XS3800-28", series: "XS3800", category: "DataCenter", layer: "L2/L3",
portsConfig: { "10G_RJ45": 24, "10G_SFP+": 4 }, totalPorts: 28,
maxSpeedGbps: 10,
rackUnits: 1,
tags: ["10G", "aggregation", "stackable"],
},
];
// ═══════════════════════════════════════════════════════
// PALO ALTO NETWORKS — Firewalls with SFP
// ═══════════════════════════════════════════════════════
const PALO_ALTO: SwitchSeed[] = [
{
vendor: "Palo Alto Networks", vendorType: "oem", vendorWebsite: "https://www.paloaltonetworks.com",
model: "PA-5430", series: "PA-5400", category: "DataCenter", layer: "L3",
portsConfig: { "25G_SFP28": 24, "100G_QSFP28": 4 }, totalPorts: 28,
maxSpeedGbps: 100,
rackUnits: 2,
tags: ["NGFW", "security", "100G"],
},
{
vendor: "Palo Alto Networks", vendorType: "oem", vendorWebsite: "https://www.paloaltonetworks.com",
model: "PA-3430", series: "PA-3400", category: "Edge", layer: "L3",
portsConfig: { "10G_SFP+": 12, "1G_RJ45": 4 }, totalPorts: 16,
maxSpeedGbps: 10,
rackUnits: 1,
tags: ["NGFW", "security", "edge"],
},
{
vendor: "Palo Alto Networks", vendorType: "oem", vendorWebsite: "https://www.paloaltonetworks.com",
model: "PA-7080", series: "PA-7000", category: "DataCenter", layer: "L3",
portsConfig: { "100G_QSFP28": 48 }, totalPorts: 48,
maxSpeedGbps: 100,
rackUnits: 19,
tags: ["NGFW", "chassis", "SP", "high-perf"],
},
];
// ═══════════════════════════════════════════════════════
// CHECK POINT — Quantum Security Gateways with SFP
// ═══════════════════════════════════════════════════════
const CHECK_POINT: SwitchSeed[] = [
{
vendor: "Check Point", vendorType: "oem", vendorWebsite: "https://www.checkpoint.com",
model: "Quantum 6800", series: "Quantum 6000", category: "DataCenter", layer: "L3",
portsConfig: { "10G_SFP+": 8, "25G_SFP28": 4, "1G_RJ45": 8 }, totalPorts: 20,
maxSpeedGbps: 25,
rackUnits: 1,
tags: ["NGFW", "security", "25G"],
},
{
vendor: "Check Point", vendorType: "oem", vendorWebsite: "https://www.checkpoint.com",
model: "Quantum 28000", series: "Quantum 28000", category: "DataCenter", layer: "L3",
portsConfig: { "40G_QSFP+": 16, "10G_SFP+": 16 }, totalPorts: 32,
maxSpeedGbps: 40,
rackUnits: 2,
tags: ["NGFW", "chassis", "high-perf", "SP"],
},
];
// ═══════════════════════════════════════════════════════
// WESTERMO — Industrial Ethernet
// ═══════════════════════════════════════════════════════
const WESTERMO: SwitchSeed[] = [
{
vendor: "Westermo", vendorType: "oem", vendorWebsite: "https://www.westermo.com",
model: "Redfox-5728-F16G-T12G-LV", series: "Redfox", category: "Industrial", layer: "L3",
portsConfig: { "1G_SFP": 16, "1G_RJ45": 12 }, totalPorts: 28,
maxSpeedGbps: 1,
rackUnits: 0,
tags: ["industrial", "rack-mount", "ruggedized"],
},
{
vendor: "Westermo", vendorType: "oem", vendorWebsite: "https://www.westermo.com",
model: "Lynx 5612-F4G-T8G", series: "Lynx", category: "Industrial", layer: "L2",
portsConfig: { "1G_SFP": 4, "1G_RJ45": 8 }, totalPorts: 12,
maxSpeedGbps: 1,
rackUnits: 0,
tags: ["industrial", "DIN-rail", "compact", "ruggedized"],
},
];
// Combine all extended vendors
const ALL_EXTENDED_SWITCHES: SwitchSeed[] = [
...FORTINET,
...MIKROTIK,
...UBIQUITI,
...NETGEAR,
...ALLIED_TELESIS,
...HIRSCHMANN,
...MOXA,
...SIEMENS,
...PHOENIX_CONTACT,
...F5,
...TPLINK,
...ZYXEL,
...PALO_ALTO,
...CHECK_POINT,
...WESTERMO,
];
export async function seedExtendedSwitches(): Promise<void> {
console.log("=== Extended Switch & Router Seed Data ===\n");
console.log(`Seeding ${ALL_EXTENDED_SWITCHES.length} switches from ${new Set(ALL_EXTENDED_SWITCHES.map(s => s.vendor)).size} vendors...\n`);
const vendorCache = new Map<string, string>();
let created = 0;
let updated = 0;
for (const sw of ALL_EXTENDED_SWITCHES) {
try {
let vendorId = vendorCache.get(sw.vendor);
if (!vendorId) {
vendorId = await ensureVendor(sw.vendor, sw.vendorType, sw.vendorWebsite);
vendorCache.set(sw.vendor, vendorId);
}
const existing = await pool.query(
`SELECT id FROM switches WHERE model = $1 AND vendor_id = $2`,
[sw.model, vendorId]
);
await findOrCreateSwitch({
model: sw.model,
vendorId,
series: sw.series,
category: sw.category,
layer: sw.layer,
portsConfig: sw.portsConfig,
totalPorts: sw.totalPorts,
uplinkSpeedGbps: sw.uplinkSpeedGbps,
maxSpeedGbps: sw.maxSpeedGbps,
switchingCapacityTbps: sw.switchingCapacityTbps,
forwardingRateMpps: sw.forwardingRateMpps,
asicVendor: sw.asicVendor,
asicModel: sw.asicModel,
rackUnits: sw.rackUnits,
maxPowerW: sw.maxPowerW,
poeSupport: sw.poeSupport,
stackingSupport: sw.stackingSupport,
vxlanSupport: sw.vxlanSupport,
evpnSupport: sw.evpnSupport,
bgpSupport: sw.bgpSupport,
mplsSupport: sw.mplsSupport,
openconfigSupport: sw.openconfigSupport,
sonicCompatible: sw.sonicCompatible,
macsecSupport: sw.macsecSupport,
tags: sw.tags,
});
if (existing.rows.length > 0) {
updated++;
} else {
created++;
}
} catch (err) {
console.error(` Error [${sw.vendor} ${sw.model}]: ${(err as Error).message.slice(0, 100)}`);
}
}
console.log(`\n=== Extended Seed Complete: ${created} created, ${updated} updated ===`);
}
if (require.main === module) {
seedExtendedSwitches()
.then(() => pool.end())
.catch((err) => { console.error("Fatal:", err); pool.end(); process.exit(1); });
}

View File

@ -0,0 +1,718 @@
/**
* Switch & Router Seed Data Comprehensive global catalog
*
* Hand-curated seed data for major switch/router models from all vendors.
* This provides the baseline that scrapers then enrich with live data.
* Sources: Public datasheets, vendor product pages, OCP specs.
*/
import { pool, ensureVendor, findOrCreateSwitch } from "../utils/db";
interface SwitchSeed {
vendor: string;
vendorType: string;
vendorWebsite: string;
model: string;
series: string;
category: "DataCenter" | "Campus" | "Edge" | "Core" | "SP" | "Industrial";
layer: "L2" | "L3" | "L2/L3";
portsConfig: Record<string, number>;
totalPorts: number;
uplinkSpeedGbps?: number;
maxSpeedGbps: number;
switchingCapacityTbps?: number;
forwardingRateMpps?: number;
asicVendor?: string;
asicModel?: string;
rackUnits?: number;
maxPowerW?: number;
poeSupport?: string;
stackingSupport?: boolean;
vxlanSupport?: boolean;
evpnSupport?: boolean;
bgpSupport?: boolean;
mplsSupport?: boolean;
openconfigSupport?: boolean;
sonicCompatible?: boolean;
macsecSupport?: boolean;
lifecycleStatus?: string;
tags?: string[];
}
// ═══════════════════════════════════════════════════════
// ARISTA NETWORKS
// ═══════════════════════════════════════════════════════
const ARISTA: SwitchSeed[] = [
// 7800R4 Series — Chassis (800G/400G spine)
{
vendor: "Arista Networks", vendorType: "oem", vendorWebsite: "https://www.arista.com",
model: "7800R4-36D2-LC", series: "7800R4", category: "DataCenter", layer: "L3",
portsConfig: { "800G_OSFP": 36, "400G_QSFP-DD": 72 }, totalPorts: 36,
maxSpeedGbps: 800, switchingCapacityTbps: 57.6, forwardingRateMpps: 36000,
asicVendor: "Broadcom", asicModel: "Memory Tomahawk 5",
rackUnits: 7, maxPowerW: 6000,
vxlanSupport: true, evpnSupport: true, bgpSupport: true, openconfigSupport: true,
macsecSupport: true, sonicCompatible: false,
tags: ["800G", "spine", "AI-fabric", "chassis"],
},
// 7060X6 Series — 800G fixed
{
vendor: "Arista Networks", vendorType: "oem", vendorWebsite: "https://www.arista.com",
model: "7060X6-64PE", series: "7060X6", category: "DataCenter", layer: "L3",
portsConfig: { "800G_OSFP": 64 }, totalPorts: 64,
maxSpeedGbps: 800, switchingCapacityTbps: 51.2, forwardingRateMpps: 30000,
asicVendor: "Broadcom", asicModel: "Tomahawk 5",
rackUnits: 2, maxPowerW: 3200,
vxlanSupport: true, evpnSupport: true, bgpSupport: true, openconfigSupport: true,
macsecSupport: true,
tags: ["800G", "spine", "AI-fabric", "leaf"],
},
// 7060X5 Series — 400G spine/leaf
{
vendor: "Arista Networks", vendorType: "oem", vendorWebsite: "https://www.arista.com",
model: "7060X5-64S", series: "7060X5", category: "DataCenter", layer: "L3",
portsConfig: { "400G_QSFP-DD": 64 }, totalPorts: 64,
maxSpeedGbps: 400, switchingCapacityTbps: 25.6, forwardingRateMpps: 16000,
asicVendor: "Broadcom", asicModel: "Tomahawk 4",
rackUnits: 2, maxPowerW: 2200,
vxlanSupport: true, evpnSupport: true, bgpSupport: true, openconfigSupport: true,
macsecSupport: true,
tags: ["400G", "spine", "leaf"],
},
{
vendor: "Arista Networks", vendorType: "oem", vendorWebsite: "https://www.arista.com",
model: "7060X5-32QS", series: "7060X5", category: "DataCenter", layer: "L3",
portsConfig: { "400G_QSFP-DD": 32, "100G_QSFP28": 2 }, totalPorts: 34,
maxSpeedGbps: 400, switchingCapacityTbps: 12.8,
asicVendor: "Broadcom", asicModel: "Tomahawk 4",
rackUnits: 1, maxPowerW: 1200,
vxlanSupport: true, evpnSupport: true, bgpSupport: true, openconfigSupport: true,
tags: ["400G", "leaf", "ToR"],
},
// 7050X4 Series — 100G/400G leaf
{
vendor: "Arista Networks", vendorType: "oem", vendorWebsite: "https://www.arista.com",
model: "7050X4-32S", series: "7050X4", category: "DataCenter", layer: "L3",
portsConfig: { "100G_QSFP28": 32 }, totalPorts: 32,
maxSpeedGbps: 100, switchingCapacityTbps: 6.4, forwardingRateMpps: 4760,
asicVendor: "Broadcom", asicModel: "Memory Memory",
rackUnits: 1, maxPowerW: 750,
vxlanSupport: true, evpnSupport: true, bgpSupport: true, openconfigSupport: true,
stackingSupport: true,
tags: ["100G", "leaf", "ToR"],
},
// 7280R3 Series — Universal routing
{
vendor: "Arista Networks", vendorType: "oem", vendorWebsite: "https://www.arista.com",
model: "7280R3-48YC6", series: "7280R3", category: "Core", layer: "L3",
portsConfig: { "25G_SFP28": 48, "100G_QSFP28": 6 }, totalPorts: 54,
maxSpeedGbps: 100, switchingCapacityTbps: 3.6,
asicVendor: "Broadcom", asicModel: "Jericho2",
rackUnits: 1, maxPowerW: 950,
bgpSupport: true, mplsSupport: true, evpnSupport: true, openconfigSupport: true,
tags: ["routing", "peering", "DCI"],
},
// 7020R Series — Campus
{
vendor: "Arista Networks", vendorType: "oem", vendorWebsite: "https://www.arista.com",
model: "7020R-48YM", series: "7020R", category: "Campus", layer: "L3",
portsConfig: { "1G_RJ45": 48, "10G_SFP+": 6 }, totalPorts: 54,
maxSpeedGbps: 10, switchingCapacityTbps: 0.176,
rackUnits: 1, maxPowerW: 450,
poeSupport: "PoE++", stackingSupport: true,
bgpSupport: true, macsecSupport: true,
tags: ["campus", "PoE", "access"],
},
];
// ═══════════════════════════════════════════════════════
// CISCO SYSTEMS
// ═══════════════════════════════════════════════════════
const CISCO: SwitchSeed[] = [
// Nexus 9000 Series
{
vendor: "Cisco Systems", vendorType: "oem", vendorWebsite: "https://www.cisco.com",
model: "N9K-C93600CD-GX", series: "Nexus 9300", category: "DataCenter", layer: "L3",
portsConfig: { "400G_QSFP-DD": 28, "100G_QSFP28": 8 }, totalPorts: 36,
maxSpeedGbps: 400, switchingCapacityTbps: 12.8, forwardingRateMpps: 8900,
asicVendor: "Cisco", asicModel: "Cloud Scale",
rackUnits: 1, maxPowerW: 2000,
vxlanSupport: true, evpnSupport: true, bgpSupport: true,
openconfigSupport: true, macsecSupport: true,
tags: ["400G", "ACI", "spine", "VXLAN"],
},
{
vendor: "Cisco Systems", vendorType: "oem", vendorWebsite: "https://www.cisco.com",
model: "N9K-C9364D-GX2A", series: "Nexus 9300", category: "DataCenter", layer: "L3",
portsConfig: { "400G_QSFP-DD": 64 }, totalPorts: 64,
maxSpeedGbps: 400, switchingCapacityTbps: 25.6,
asicVendor: "Cisco", asicModel: "Cloud Scale G2",
rackUnits: 2, maxPowerW: 3000,
vxlanSupport: true, evpnSupport: true, bgpSupport: true, openconfigSupport: true,
tags: ["400G", "ACI", "spine"],
},
{
vendor: "Cisco Systems", vendorType: "oem", vendorWebsite: "https://www.cisco.com",
model: "N9K-C9332D-GX2B", series: "Nexus 9300", category: "DataCenter", layer: "L3",
portsConfig: { "400G_QSFP-DD": 32, "100G_QSFP28": 2 }, totalPorts: 34,
maxSpeedGbps: 400, switchingCapacityTbps: 12.8,
asicVendor: "Cisco", asicModel: "Cloud Scale G2",
rackUnits: 1, maxPowerW: 1500,
vxlanSupport: true, evpnSupport: true, bgpSupport: true,
tags: ["400G", "leaf", "ToR"],
},
{
vendor: "Cisco Systems", vendorType: "oem", vendorWebsite: "https://www.cisco.com",
model: "N9K-C93108TC-FX3P", series: "Nexus 9300", category: "DataCenter", layer: "L3",
portsConfig: { "10G_RJ45": 48, "100G_QSFP28": 6 }, totalPorts: 54,
maxSpeedGbps: 100, switchingCapacityTbps: 3.6,
asicVendor: "Cisco", asicModel: "Cloud Scale",
rackUnits: 1, maxPowerW: 1100,
poeSupport: "PoE+", vxlanSupport: true, evpnSupport: true,
tags: ["access", "ToR", "PoE"],
},
// Nexus 9500 Chassis
{
vendor: "Cisco Systems", vendorType: "oem", vendorWebsite: "https://www.cisco.com",
model: "N9K-C9508", series: "Nexus 9500", category: "DataCenter", layer: "L3",
portsConfig: { "400G_QSFP-DD": 576 }, totalPorts: 576,
maxSpeedGbps: 400, switchingCapacityTbps: 172.8,
asicVendor: "Cisco", asicModel: "Cloud Scale G2",
rackUnits: 13, maxPowerW: 24000,
vxlanSupport: true, evpnSupport: true, bgpSupport: true,
tags: ["chassis", "spine", "modular", "ACI"],
},
// Catalyst 9000 — Campus
{
vendor: "Cisco Systems", vendorType: "oem", vendorWebsite: "https://www.cisco.com",
model: "C9300-48UXM", series: "Catalyst 9300", category: "Campus", layer: "L3",
portsConfig: { "1G_RJ45": 36, "mGig_RJ45": 12, "10G_SFP+": 4 }, totalPorts: 52,
maxSpeedGbps: 10, switchingCapacityTbps: 0.480,
rackUnits: 1, maxPowerW: 1100,
poeSupport: "UPOE+", stackingSupport: true, macsecSupport: true,
tags: ["campus", "PoE", "SD-Access", "DNA"],
},
{
vendor: "Cisco Systems", vendorType: "oem", vendorWebsite: "https://www.cisco.com",
model: "C9300X-24Y", series: "Catalyst 9300X", category: "Campus", layer: "L3",
portsConfig: { "25G_SFP28": 24, "100G_QSFP28": 2 }, totalPorts: 26,
maxSpeedGbps: 100, switchingCapacityTbps: 1.2,
rackUnits: 1, maxPowerW: 750,
stackingSupport: true, vxlanSupport: true, macsecSupport: true,
tags: ["campus", "distribution", "25G"],
},
{
vendor: "Cisco Systems", vendorType: "oem", vendorWebsite: "https://www.cisco.com",
model: "C9500-48Y4C", series: "Catalyst 9500", category: "Campus", layer: "L3",
portsConfig: { "25G_SFP28": 48, "100G_QSFP28": 4 }, totalPorts: 52,
maxSpeedGbps: 100, switchingCapacityTbps: 3.2,
rackUnits: 1, maxPowerW: 750,
bgpSupport: true, vxlanSupport: true, evpnSupport: true, macsecSupport: true,
tags: ["campus", "core", "SD-Access"],
},
// NCS 5500 — Service Provider
{
vendor: "Cisco Systems", vendorType: "oem", vendorWebsite: "https://www.cisco.com",
model: "NCS-5504", series: "NCS 5500", category: "SP", layer: "L3",
portsConfig: { "400G_QSFP-DD": 288 }, totalPorts: 288,
maxSpeedGbps: 400, switchingCapacityTbps: 57.6,
asicVendor: "Cisco", asicModel: "Silicon One",
rackUnits: 7, maxPowerW: 15000,
bgpSupport: true, mplsSupport: true, evpnSupport: true,
tags: ["SP", "core", "routing", "chassis", "MPLS"],
},
// Cisco 8000 — Silicon One
{
vendor: "Cisco Systems", vendorType: "oem", vendorWebsite: "https://www.cisco.com",
model: "8101-32FH", series: "Cisco 8000", category: "SP", layer: "L3",
portsConfig: { "400G_QSFP-DD": 32 }, totalPorts: 32,
maxSpeedGbps: 400, switchingCapacityTbps: 12.8,
asicVendor: "Cisco", asicModel: "Silicon One Q200",
rackUnits: 1, maxPowerW: 1800,
bgpSupport: true, mplsSupport: true, evpnSupport: true, openconfigSupport: true,
tags: ["SP", "peering", "core", "Silicon-One"],
},
{
vendor: "Cisco Systems", vendorType: "oem", vendorWebsite: "https://www.cisco.com",
model: "8111-32EH", series: "Cisco 8000", category: "SP", layer: "L3",
portsConfig: { "400G_QSFP-DD": 32 }, totalPorts: 32,
maxSpeedGbps: 400, switchingCapacityTbps: 12.8,
asicVendor: "Cisco", asicModel: "Silicon One G100",
rackUnits: 1, maxPowerW: 1800,
bgpSupport: true, mplsSupport: true, evpnSupport: true,
tags: ["SP", "peering", "400G"],
},
];
// ═══════════════════════════════════════════════════════
// JUNIPER NETWORKS
// ═══════════════════════════════════════════════════════
const JUNIPER: SwitchSeed[] = [
{
vendor: "Juniper Networks", vendorType: "oem", vendorWebsite: "https://www.juniper.net",
model: "QFX5130-32CD", series: "QFX5130", category: "DataCenter", layer: "L3",
portsConfig: { "400G_QSFP-DD": 32 }, totalPorts: 32,
maxSpeedGbps: 400, switchingCapacityTbps: 12.8,
asicVendor: "Broadcom", asicModel: "Tomahawk 4",
rackUnits: 1, maxPowerW: 1500,
vxlanSupport: true, evpnSupport: true, bgpSupport: true, openconfigSupport: true,
tags: ["400G", "leaf", "spine"],
},
{
vendor: "Juniper Networks", vendorType: "oem", vendorWebsite: "https://www.juniper.net",
model: "QFX5220-32CD", series: "QFX5220", category: "DataCenter", layer: "L3",
portsConfig: { "400G_QSFP-DD": 32, "100G_QSFP28": 2 }, totalPorts: 34,
maxSpeedGbps: 400, switchingCapacityTbps: 12.8,
asicVendor: "Broadcom", asicModel: "Tomahawk 3",
rackUnits: 1, maxPowerW: 1400,
vxlanSupport: true, evpnSupport: true, bgpSupport: true, openconfigSupport: true,
tags: ["400G", "leaf"],
},
{
vendor: "Juniper Networks", vendorType: "oem", vendorWebsite: "https://www.juniper.net",
model: "QFX5120-48Y", series: "QFX5120", category: "DataCenter", layer: "L3",
portsConfig: { "25G_SFP28": 48, "100G_QSFP28": 8 }, totalPorts: 56,
maxSpeedGbps: 100, switchingCapacityTbps: 3.2,
asicVendor: "Broadcom", asicModel: "Memory",
rackUnits: 1, maxPowerW: 550,
vxlanSupport: true, evpnSupport: true, bgpSupport: true, openconfigSupport: true,
tags: ["25G", "leaf", "ToR"],
},
// PTX Series — Core routing
{
vendor: "Juniper Networks", vendorType: "oem", vendorWebsite: "https://www.juniper.net",
model: "PTX10004", series: "PTX10000", category: "SP", layer: "L3",
portsConfig: { "400G_QSFP-DD": 576 }, totalPorts: 576,
maxSpeedGbps: 400, switchingCapacityTbps: 172.8,
asicVendor: "Juniper", asicModel: "Express5",
rackUnits: 8, maxPowerW: 18000,
bgpSupport: true, mplsSupport: true, evpnSupport: true, openconfigSupport: true,
tags: ["SP", "core", "chassis", "MPLS", "400G"],
},
{
vendor: "Juniper Networks", vendorType: "oem", vendorWebsite: "https://www.juniper.net",
model: "PTX10001-36MR", series: "PTX10000", category: "SP", layer: "L3",
portsConfig: { "400G_QSFP-DD": 36 }, totalPorts: 36,
maxSpeedGbps: 400, switchingCapacityTbps: 14.4,
asicVendor: "Juniper", asicModel: "Express5",
rackUnits: 2, maxPowerW: 2500,
bgpSupport: true, mplsSupport: true, evpnSupport: true,
tags: ["SP", "peering", "400G"],
},
// EX Series — Campus
{
vendor: "Juniper Networks", vendorType: "oem", vendorWebsite: "https://www.juniper.net",
model: "EX4400-48T", series: "EX4400", category: "Campus", layer: "L3",
portsConfig: { "1G_RJ45": 48, "10G_SFP+": 4, "25G_SFP28": 2 }, totalPorts: 54,
maxSpeedGbps: 25, switchingCapacityTbps: 0.200,
rackUnits: 1, maxPowerW: 600,
poeSupport: "PoE++", stackingSupport: true, macsecSupport: true,
tags: ["campus", "access", "PoE"],
},
];
// ═══════════════════════════════════════════════════════
// NOKIA (ex-Alcatel-Lucent)
// ═══════════════════════════════════════════════════════
const NOKIA: SwitchSeed[] = [
{
vendor: "Nokia", vendorType: "oem", vendorWebsite: "https://www.nokia.com",
model: "7220 IXR-D3L", series: "7220 IXR", category: "DataCenter", layer: "L3",
portsConfig: { "400G_QSFP-DD": 32, "10G_SFP+": 2 }, totalPorts: 34,
maxSpeedGbps: 400, switchingCapacityTbps: 12.8,
asicVendor: "Broadcom", asicModel: "Tomahawk 4",
rackUnits: 1, maxPowerW: 1400,
vxlanSupport: true, evpnSupport: true, bgpSupport: true, openconfigSupport: true,
sonicCompatible: true,
tags: ["400G", "SR-Linux", "leaf"],
},
{
vendor: "Nokia", vendorType: "oem", vendorWebsite: "https://www.nokia.com",
model: "7750 SR-1e", series: "7750 SR", category: "SP", layer: "L3",
portsConfig: { "400G_QSFP-DD": 36 }, totalPorts: 36,
maxSpeedGbps: 400, switchingCapacityTbps: 14.4,
asicVendor: "Nokia", asicModel: "FP5",
rackUnits: 2, maxPowerW: 3500,
bgpSupport: true, mplsSupport: true, evpnSupport: true, openconfigSupport: true,
tags: ["SP", "routing", "MPLS", "SR-OS"],
},
{
vendor: "Nokia", vendorType: "oem", vendorWebsite: "https://www.nokia.com",
model: "7750 SR-14s", series: "7750 SR", category: "SP", layer: "L3",
portsConfig: { "400G_QSFP-DD": 672 }, totalPorts: 672,
maxSpeedGbps: 400, switchingCapacityTbps: 268.8,
asicVendor: "Nokia", asicModel: "FP5",
rackUnits: 22, maxPowerW: 38000,
bgpSupport: true, mplsSupport: true, evpnSupport: true,
tags: ["SP", "core", "chassis", "MPLS"],
},
];
// ═══════════════════════════════════════════════════════
// HUAWEI
// ═══════════════════════════════════════════════════════
const HUAWEI: SwitchSeed[] = [
{
vendor: "Huawei", vendorType: "oem", vendorWebsite: "https://e.huawei.com",
model: "CloudEngine 16800-X32", series: "CloudEngine 16800", category: "DataCenter", layer: "L3",
portsConfig: { "400G_QSFP-DD": 768 }, totalPorts: 768,
maxSpeedGbps: 400, switchingCapacityTbps: 230.4,
asicVendor: "Huawei", asicModel: "Solar 6.0",
rackUnits: 22, maxPowerW: 40000,
vxlanSupport: true, evpnSupport: true, bgpSupport: true,
tags: ["chassis", "spine", "AI-fabric"],
},
{
vendor: "Huawei", vendorType: "oem", vendorWebsite: "https://e.huawei.com",
model: "CloudEngine 8850-32CQ", series: "CloudEngine 8850", category: "DataCenter", layer: "L3",
portsConfig: { "100G_QSFP28": 32 }, totalPorts: 32,
maxSpeedGbps: 100, switchingCapacityTbps: 6.4,
rackUnits: 1, maxPowerW: 950,
vxlanSupport: true, evpnSupport: true, bgpSupport: true,
tags: ["100G", "leaf", "spine"],
},
{
vendor: "Huawei", vendorType: "oem", vendorWebsite: "https://e.huawei.com",
model: "NetEngine 8000 F8", series: "NetEngine 8000", category: "SP", layer: "L3",
portsConfig: { "400G_QSFP-DD": 192 }, totalPorts: 192,
maxSpeedGbps: 400, switchingCapacityTbps: 57.6,
asicVendor: "Huawei", asicModel: "Solar 6.0",
rackUnits: 9, maxPowerW: 12000,
bgpSupport: true, mplsSupport: true, evpnSupport: true,
tags: ["SP", "core", "MPLS"],
},
{
vendor: "Huawei", vendorType: "oem", vendorWebsite: "https://e.huawei.com",
model: "S5731-H48T4XC", series: "S5700", category: "Campus", layer: "L3",
portsConfig: { "1G_RJ45": 48, "10G_SFP+": 4 }, totalPorts: 52,
maxSpeedGbps: 10, switchingCapacityTbps: 0.176,
rackUnits: 1, maxPowerW: 600,
poeSupport: "PoE++", stackingSupport: true,
tags: ["campus", "access", "PoE"],
},
];
// ═══════════════════════════════════════════════════════
// HPE / ARUBA
// ═══════════════════════════════════════════════════════
const HPE: SwitchSeed[] = [
{
vendor: "HPE Aruba", vendorType: "oem", vendorWebsite: "https://www.arubanetworks.com",
model: "CX 10000-48Y6C", series: "CX 10000", category: "DataCenter", layer: "L3",
portsConfig: { "25G_SFP28": 48, "100G_QSFP28": 6 }, totalPorts: 54,
maxSpeedGbps: 100, switchingCapacityTbps: 3.2,
asicVendor: "AMD/Pensando", asicModel: "Elba",
rackUnits: 1, maxPowerW: 950,
vxlanSupport: true, evpnSupport: true, bgpSupport: true,
tags: ["DPU", "stateful", "microsegmentation"],
},
{
vendor: "HPE Aruba", vendorType: "oem", vendorWebsite: "https://www.arubanetworks.com",
model: "CX 9300-32D", series: "CX 9300", category: "DataCenter", layer: "L3",
portsConfig: { "400G_QSFP-DD": 32 }, totalPorts: 32,
maxSpeedGbps: 400, switchingCapacityTbps: 12.8,
asicVendor: "Broadcom", asicModel: "Tomahawk 4",
rackUnits: 1, maxPowerW: 1500,
vxlanSupport: true, evpnSupport: true, bgpSupport: true, openconfigSupport: true,
tags: ["400G", "spine", "leaf"],
},
{
vendor: "HPE Aruba", vendorType: "oem", vendorWebsite: "https://www.arubanetworks.com",
model: "CX 6300-48G-4SFP56", series: "CX 6300", category: "Campus", layer: "L3",
portsConfig: { "1G_RJ45": 48, "50G_SFP56": 4 }, totalPorts: 52,
maxSpeedGbps: 50, switchingCapacityTbps: 0.296,
rackUnits: 1, maxPowerW: 800,
poeSupport: "PoE++", stackingSupport: true,
tags: ["campus", "access", "PoE"],
},
];
// ═══════════════════════════════════════════════════════
// NVIDIA / MELLANOX
// ═══════════════════════════════════════════════════════
const NVIDIA: SwitchSeed[] = [
{
vendor: "NVIDIA Networking", vendorType: "oem", vendorWebsite: "https://www.nvidia.com/networking",
model: "SN5600", series: "Spectrum-4", category: "DataCenter", layer: "L3",
portsConfig: { "800G_OSFP": 64 }, totalPorts: 64,
maxSpeedGbps: 800, switchingCapacityTbps: 51.2, forwardingRateMpps: 33000,
asicVendor: "NVIDIA", asicModel: "Spectrum-4",
rackUnits: 2, maxPowerW: 3000,
vxlanSupport: true, evpnSupport: true, bgpSupport: true, openconfigSupport: true,
sonicCompatible: true,
tags: ["800G", "AI-fabric", "Cumulus", "SONiC"],
},
{
vendor: "NVIDIA Networking", vendorType: "oem", vendorWebsite: "https://www.nvidia.com/networking",
model: "SN5400", series: "Spectrum-4", category: "DataCenter", layer: "L3",
portsConfig: { "400G_QSFP-DD": 64 }, totalPorts: 64,
maxSpeedGbps: 400, switchingCapacityTbps: 25.6,
asicVendor: "NVIDIA", asicModel: "Spectrum-4",
rackUnits: 2, maxPowerW: 2400,
vxlanSupport: true, evpnSupport: true, bgpSupport: true, openconfigSupport: true,
sonicCompatible: true,
tags: ["400G", "spine", "Cumulus", "SONiC"],
},
{
vendor: "NVIDIA Networking", vendorType: "oem", vendorWebsite: "https://www.nvidia.com/networking",
model: "SN4700", series: "Spectrum-3", category: "DataCenter", layer: "L3",
portsConfig: { "400G_QSFP-DD": 32 }, totalPorts: 32,
maxSpeedGbps: 400, switchingCapacityTbps: 12.8,
asicVendor: "NVIDIA", asicModel: "Spectrum-3",
rackUnits: 1, maxPowerW: 1200,
vxlanSupport: true, evpnSupport: true, bgpSupport: true, openconfigSupport: true,
sonicCompatible: true,
tags: ["400G", "leaf", "SONiC"],
},
{
vendor: "NVIDIA Networking", vendorType: "oem", vendorWebsite: "https://www.nvidia.com/networking",
model: "SN2201", series: "Spectrum-1", category: "DataCenter", layer: "L2/L3",
portsConfig: { "1G_RJ45": 48, "100G_QSFP28": 4 }, totalPorts: 52,
maxSpeedGbps: 100, switchingCapacityTbps: 0.496,
asicVendor: "NVIDIA", asicModel: "Spectrum-1",
rackUnits: 1, maxPowerW: 400,
sonicCompatible: true,
tags: ["management", "OOB", "SONiC"],
},
];
// ═══════════════════════════════════════════════════════
// EDGECORE / WHITEBOX
// ═══════════════════════════════════════════════════════
const EDGECORE: SwitchSeed[] = [
{
vendor: "Edgecore Networks", vendorType: "oem", vendorWebsite: "https://www.edge-core.com",
model: "DCS810", series: "DCS800", category: "DataCenter", layer: "L3",
portsConfig: { "800G_OSFP": 64 }, totalPorts: 64,
maxSpeedGbps: 800, switchingCapacityTbps: 51.2,
asicVendor: "Broadcom", asicModel: "Tomahawk 5",
rackUnits: 2, maxPowerW: 3500,
sonicCompatible: true, openconfigSupport: true,
tags: ["800G", "whitebox", "SONiC", "OCP"],
},
{
vendor: "Edgecore Networks", vendorType: "oem", vendorWebsite: "https://www.edge-core.com",
model: "DCS510", series: "DCS500", category: "DataCenter", layer: "L3",
portsConfig: { "400G_QSFP-DD": 64 }, totalPorts: 64,
maxSpeedGbps: 400, switchingCapacityTbps: 25.6,
asicVendor: "Broadcom", asicModel: "Tomahawk 4",
rackUnits: 2, maxPowerW: 2200,
sonicCompatible: true, openconfigSupport: true,
tags: ["400G", "whitebox", "SONiC", "OCP"],
},
{
vendor: "Edgecore Networks", vendorType: "oem", vendorWebsite: "https://www.edge-core.com",
model: "DCS204", series: "DCS200", category: "DataCenter", layer: "L3",
portsConfig: { "100G_QSFP28": 32, "10G_SFP+": 2 }, totalPorts: 34,
maxSpeedGbps: 100, switchingCapacityTbps: 6.4,
asicVendor: "Broadcom", asicModel: "Tomahawk 3",
rackUnits: 1, maxPowerW: 750,
sonicCompatible: true, openconfigSupport: true,
tags: ["100G", "whitebox", "SONiC"],
},
];
// ═══════════════════════════════════════════════════════
// DELL TECHNOLOGIES
// ═══════════════════════════════════════════════════════
const DELL: SwitchSeed[] = [
{
vendor: "Dell Technologies", vendorType: "oem", vendorWebsite: "https://www.dell.com",
model: "PowerSwitch Z9664F-ON", series: "Z9664F", category: "DataCenter", layer: "L3",
portsConfig: { "400G_QSFP-DD": 64 }, totalPorts: 64,
maxSpeedGbps: 400, switchingCapacityTbps: 25.6,
asicVendor: "Broadcom", asicModel: "Tomahawk 4",
rackUnits: 2, maxPowerW: 2500,
vxlanSupport: true, evpnSupport: true, bgpSupport: true,
sonicCompatible: true, openconfigSupport: true,
tags: ["400G", "spine", "SONiC", "OS10"],
},
{
vendor: "Dell Technologies", vendorType: "oem", vendorWebsite: "https://www.dell.com",
model: "PowerSwitch Z9332F-ON", series: "Z9332F", category: "DataCenter", layer: "L3",
portsConfig: { "400G_QSFP-DD": 32, "10G_SFP+": 2 }, totalPorts: 34,
maxSpeedGbps: 400, switchingCapacityTbps: 12.8,
asicVendor: "Broadcom", asicModel: "Tomahawk 3",
rackUnits: 1, maxPowerW: 1200,
vxlanSupport: true, evpnSupport: true, bgpSupport: true, sonicCompatible: true,
tags: ["400G", "leaf", "SONiC", "OS10"],
},
{
vendor: "Dell Technologies", vendorType: "oem", vendorWebsite: "https://www.dell.com",
model: "PowerSwitch S5248F-ON", series: "S5248F", category: "DataCenter", layer: "L3",
portsConfig: { "25G_SFP28": 48, "100G_QSFP28": 4, "100G_QSFP-DD": 2 }, totalPorts: 54,
maxSpeedGbps: 100, switchingCapacityTbps: 3.2,
asicVendor: "Broadcom", asicModel: "Trident 3",
rackUnits: 1, maxPowerW: 550,
vxlanSupport: true, evpnSupport: true, sonicCompatible: true,
tags: ["25G", "leaf", "ToR"],
},
];
// ═══════════════════════════════════════════════════════
// EXTREME NETWORKS
// ═══════════════════════════════════════════════════════
const EXTREME: SwitchSeed[] = [
{
vendor: "Extreme Networks", vendorType: "oem", vendorWebsite: "https://www.extremenetworks.com",
model: "SLX 9740-40C", series: "SLX 9740", category: "DataCenter", layer: "L3",
portsConfig: { "100G_QSFP28": 40 }, totalPorts: 40,
maxSpeedGbps: 100, switchingCapacityTbps: 8.0,
rackUnits: 1, maxPowerW: 1100,
vxlanSupport: true, evpnSupport: true, bgpSupport: true,
tags: ["100G", "spine", "EVPN"],
},
{
vendor: "Extreme Networks", vendorType: "oem", vendorWebsite: "https://www.extremenetworks.com",
model: "X695-48Y-8C", series: "ExtremeSwitching X695", category: "DataCenter", layer: "L3",
portsConfig: { "25G_SFP28": 48, "100G_QSFP28": 8 }, totalPorts: 56,
maxSpeedGbps: 100, switchingCapacityTbps: 3.2,
rackUnits: 1, maxPowerW: 750,
vxlanSupport: true, evpnSupport: true, bgpSupport: true, stackingSupport: true,
tags: ["25G", "leaf", "fabric"],
},
{
vendor: "Extreme Networks", vendorType: "oem", vendorWebsite: "https://www.extremenetworks.com",
model: "5520-48T", series: "ExtremeSwitching 5520", category: "Campus", layer: "L3",
portsConfig: { "1G_RJ45": 48, "10G_SFP+": 4 }, totalPorts: 52,
maxSpeedGbps: 10, switchingCapacityTbps: 0.176,
rackUnits: 1, maxPowerW: 500,
poeSupport: "PoE++", stackingSupport: true,
tags: ["campus", "access", "PoE", "ExtremeCloud"],
},
];
// ═══════════════════════════════════════════════════════
// UFISPACE (OCP-focused whitebox)
// ═══════════════════════════════════════════════════════
const UFISPACE: SwitchSeed[] = [
{
vendor: "UfiSpace", vendorType: "oem", vendorWebsite: "https://www.ufispace.com",
model: "S9710-76D", series: "S9710", category: "DataCenter", layer: "L3",
portsConfig: { "400G_QSFP-DD": 76 }, totalPorts: 76,
maxSpeedGbps: 400, switchingCapacityTbps: 25.6,
asicVendor: "Broadcom", asicModel: "Tomahawk 4",
rackUnits: 2, maxPowerW: 2800,
sonicCompatible: true, openconfigSupport: true,
tags: ["400G", "whitebox", "OCP", "SONiC"],
},
{
vendor: "UfiSpace", vendorType: "oem", vendorWebsite: "https://www.ufispace.com",
model: "S9600-72XC", series: "S9600", category: "DataCenter", layer: "L3",
portsConfig: { "100G_QSFP28": 72 }, totalPorts: 72,
maxSpeedGbps: 100, switchingCapacityTbps: 12.8,
asicVendor: "Broadcom", asicModel: "Tomahawk 3",
rackUnits: 2, maxPowerW: 1500,
sonicCompatible: true,
tags: ["100G", "whitebox", "SONiC"],
},
];
// ═══════════════════════════════════════════════════════
// CELESTICA (OCP whitebox)
// ═══════════════════════════════════════════════════════
const CELESTICA: SwitchSeed[] = [
{
vendor: "Celestica", vendorType: "oem", vendorWebsite: "https://www.celestica.com",
model: "DS5000", series: "DS5000", category: "DataCenter", layer: "L3",
portsConfig: { "800G_OSFP": 64 }, totalPorts: 64,
maxSpeedGbps: 800, switchingCapacityTbps: 51.2,
asicVendor: "Broadcom", asicModel: "Tomahawk 5",
rackUnits: 2, maxPowerW: 3500,
sonicCompatible: true, openconfigSupport: true,
tags: ["800G", "whitebox", "OCP", "SONiC"],
},
{
vendor: "Celestica", vendorType: "oem", vendorWebsite: "https://www.celestica.com",
model: "DS4000", series: "DS4000", category: "DataCenter", layer: "L3",
portsConfig: { "400G_QSFP-DD": 32 }, totalPorts: 32,
maxSpeedGbps: 400, switchingCapacityTbps: 12.8,
asicVendor: "Broadcom", asicModel: "Tomahawk 4",
rackUnits: 1, maxPowerW: 1400,
sonicCompatible: true,
tags: ["400G", "whitebox", "SONiC"],
},
];
// Combine all vendors
const ALL_SWITCHES: SwitchSeed[] = [
...ARISTA,
...CISCO,
...JUNIPER,
...NOKIA,
...HUAWEI,
...HPE,
...NVIDIA,
...EDGECORE,
...DELL,
...EXTREME,
...UFISPACE,
...CELESTICA,
];
export async function seedSwitches(): Promise<void> {
console.log("=== Switch & Router Seed Data ===\n");
console.log(`Seeding ${ALL_SWITCHES.length} switches from ${new Set(ALL_SWITCHES.map(s => s.vendor)).size} vendors...\n`);
const vendorCache = new Map<string, string>();
let created = 0;
let updated = 0;
for (const sw of ALL_SWITCHES) {
try {
let vendorId = vendorCache.get(sw.vendor);
if (!vendorId) {
vendorId = await ensureVendor(sw.vendor, sw.vendorType, sw.vendorWebsite);
vendorCache.set(sw.vendor, vendorId);
}
const existing = await pool.query(
`SELECT id FROM switches WHERE model = $1 AND vendor_id = $2`,
[sw.model, vendorId]
);
await findOrCreateSwitch({
model: sw.model,
vendorId,
series: sw.series,
category: sw.category,
layer: sw.layer,
portsConfig: sw.portsConfig,
totalPorts: sw.totalPorts,
uplinkSpeedGbps: sw.uplinkSpeedGbps,
maxSpeedGbps: sw.maxSpeedGbps,
switchingCapacityTbps: sw.switchingCapacityTbps,
forwardingRateMpps: sw.forwardingRateMpps,
asicVendor: sw.asicVendor,
asicModel: sw.asicModel,
rackUnits: sw.rackUnits,
maxPowerW: sw.maxPowerW,
poeSupport: sw.poeSupport,
stackingSupport: sw.stackingSupport,
vxlanSupport: sw.vxlanSupport,
evpnSupport: sw.evpnSupport,
bgpSupport: sw.bgpSupport,
mplsSupport: sw.mplsSupport,
openconfigSupport: sw.openconfigSupport,
sonicCompatible: sw.sonicCompatible,
macsecSupport: sw.macsecSupport,
tags: sw.tags,
});
if (existing.rows.length > 0) {
updated++;
} else {
created++;
}
} catch (err) {
console.error(` Error [${sw.vendor} ${sw.model}]: ${(err as Error).message.slice(0, 100)}`);
}
}
console.log(`\n=== Seed Complete: ${created} created, ${updated} updated ===`);
}
if (require.main === module) {
seedSwitches()
.then(() => pool.end())
.catch((err) => { console.error("Fatal:", err); pool.end(); process.exit(1); });
}

View File

@ -0,0 +1,231 @@
/**
* 10Gtek.com Scraper Chinese OEM Transceiver Vendor
*
* 10gtek.com is a direct competitor to FS.com at lower price points.
* Uses plain fetch (server-rendered HTML).
* Rate limited: 1 req/2sec.
*
* Categories: SFP, SFP+, SFP28, QSFP+, QSFP28, QSFP-DD, OSFP
*/
import { pool, findOrCreateScrapedTransceiver, ensureVendor, upsertPriceObservation } from "../utils/db";
import { contentHash, parsePrice } from "../utils/hash";
const BASE = "https://www.10gtek.com";
const HEADERS = {
"User-Agent": "Mozilla/5.0 (compatible; TIP-Bot/1.0; research)",
Accept: "text/html,application/xhtml+xml",
};
const CATEGORIES = [
{ path: "/sfp", formFactor: "SFP", speed: "1G", speedGbps: 1 },
{ path: "/10g-sfp+", formFactor: "SFP+", speed: "10G", speedGbps: 10 },
{ path: "/sfp28", formFactor: "SFP28", speed: "25G", speedGbps: 25 },
{ path: "/qsfp", formFactor: "QSFP+", speed: "40G", speedGbps: 40 },
{ path: "/qsfp28", formFactor: "QSFP28", speed: "100G", speedGbps: 100 },
{ path: "/qsfpdd", formFactor: "QSFP-DD", speed: "400G", speedGbps: 400 },
{ path: "/xfp", formFactor: "XFP", speed: "10G", speedGbps: 10 },
];
interface Product {
partNumber: string;
name: string;
url: string;
price?: number;
currency?: string;
formFactor: string;
speed: string;
speedGbps: number;
reachLabel?: string;
reachMeters?: number;
fiberType?: string;
}
function sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}
function detectReach(text: string): { label: string; meters: number } | undefined {
const patterns: [RegExp, string, number][] = [
[/\b80\s*km\b/i, "80km", 80000],
[/\b40\s*km\b/i, "40km", 40000],
[/\b20\s*km\b/i, "20km", 20000],
[/\b10\s*km\b/i, "10km", 10000],
[/\b2\s*km\b/i, "2km", 2000],
[/\b500\s*m\b/i, "500m", 500],
[/\b300\s*m\b/i, "300m", 300],
[/\b100\s*m\b/i, "100m", 100],
[/\bLR4\b/, "10km", 10000],
[/\bLR\b/, "10km", 10000],
[/\bER\b/, "40km", 40000],
[/\bZR\b/, "80km", 80000],
[/\bSR4?\b/, "100m", 100],
[/\bDR4?\b/, "500m", 500],
[/\bFR4?\b/, "2km", 2000],
];
for (const [regex, label, meters] of patterns) {
if (regex.test(text)) return { label, meters };
}
return undefined;
}
function detectFiber(text: string): string {
if (/single.?mode|smf|[^a-z]lx[^a-z]|[^a-z]lr[^a-z]|[^a-z]er[^a-z]|[^a-z]zr[^a-z]|bidi|cwdm|dwdm/i.test(text)) return "SMF";
if (/multi.?mode|mmf|[^a-z]sx[^a-z]|[^a-z]sr[^a-z]/i.test(text)) return "MMF";
return "";
}
/** Strip HTML tags and decode common entities */
function stripHtml(s: string): string {
return s.replace(/<[^>]+>/g, "").replace(/&amp;/g, "&").replace(/&lt;/g, "<")
.replace(/&gt;/g, ">").replace(/&nbsp;/g, " ").replace(/&deg;/g, "°")
.replace(/&#\d+;/g, "").trim();
}
function parseDistance(text: string): { label: string; meters: number } | undefined {
const km = text.match(/(\d+)\s*km/i);
if (km) return { label: `${km[1]}km`, meters: parseInt(km[1]) * 1000 };
const m = text.match(/(\d+)\s*m\b/i);
if (m) return { label: `${m[1]}m`, meters: parseInt(m[1]) };
return undefined;
}
function parseProductList(html: string, cat: typeof CATEGORIES[number]): Product[] {
const products: Product[] = [];
// 10Gtek uses HTML tables with columns:
// Part No. | Spec | Data Rate | Wavelength | Fiber Type | Distance | Optical Comp. | Tx Power | E.R | Rx Sens. | Temp.
// Extract all <tr> rows and parse cells
const rowRegex = /<tr[^>]*>([\s\S]*?)<\/tr>/gi;
let rowMatch;
while ((rowMatch = rowRegex.exec(html)) !== null) {
const rowHtml = rowMatch[1];
// Extract all <td> cell contents
const cellRegex = /<td[^>]*>([\s\S]*?)<\/td>/gi;
const cells: string[] = [];
let cellMatch;
while ((cellMatch = cellRegex.exec(rowHtml)) !== null) {
cells.push(stripHtml(cellMatch[1]));
}
// Need at least 6 columns, first cell must look like a part number (starts with A or contains letters+digits)
if (cells.length < 6) continue;
const partNumber = cells[0];
if (!partNumber || partNumber.length < 3) continue;
// Skip header rows
if (/^Part\s*No/i.test(partNumber) || /^Spec/i.test(partNumber)) continue;
// Part numbers typically start with A (ASF, AXS, AXQ, AQS, etc.) or contain alphanumeric
if (!/^[A-Z][A-Z0-9]/i.test(partNumber)) continue;
const spec = cells[1] || "";
const dataRate = cells[2] || "";
const wavelength = cells.length >= 4 ? cells[3] : "";
const fiberType = cells.length >= 5 ? cells[4] : "";
const distance = cells.length >= 6 ? cells[5] : "";
const txPower = cells.length >= 8 ? cells[7] : "";
// Build descriptive name
const name = `${partNumber} ${spec} ${dataRate}`.trim();
const reach = parseDistance(distance) || detectReach(spec + " " + distance);
// Determine fiber type from table cell or spec
let fiber = "";
if (/SMF|single/i.test(fiberType)) fiber = "SMF";
else if (/MMF|multi/i.test(fiberType)) fiber = "MMF";
else if (/CAT|RJ|copper/i.test(fiberType)) fiber = "Copper";
else fiber = detectFiber(spec);
// Extract wavelength
const wl = wavelength.replace(/[^0-9]/g, "");
products.push({
partNumber,
name,
url: `${BASE}${cat.path}#${partNumber}`,
formFactor: cat.formFactor,
speed: cat.speed,
speedGbps: cat.speedGbps,
reachLabel: reach?.label,
reachMeters: reach?.meters,
fiberType: fiber,
});
}
// Dedupe by part number
const seen = new Set<string>();
return products.filter((p) => {
if (seen.has(p.partNumber)) return false;
seen.add(p.partNumber);
return true;
});
}
async function fetchPage(url: string): Promise<string> {
const resp = await fetch(url, { headers: HEADERS, signal: AbortSignal.timeout(30000) });
if (!resp.ok) throw new Error(`HTTP ${resp.status} for ${url}`);
return resp.text();
}
export async function scrape10Gtek(): Promise<void> {
console.log("=== 10Gtek Scraper Starting ===\n");
const vendorId = await ensureVendor("10Gtek", "compatible", "https://www.10gtek.com", "https://www.10gtek.com");
let totalProducts = 0;
let priceUpdates = 0;
for (const cat of CATEGORIES) {
console.log(`\n--- ${cat.formFactor} (${cat.speed}) ---`);
try {
const html = await fetchPage(BASE + cat.path);
const catProducts = parseProductList(html, cat);
console.log(` Found ${catProducts.length} products`);
for (const product of catProducts) {
try {
const txId = await findOrCreateScrapedTransceiver({
partNumber: product.partNumber,
vendorId,
formFactor: product.formFactor,
speedGbps: product.speedGbps,
speed: product.speed,
reachMeters: product.reachMeters,
reachLabel: product.reachLabel,
fiberType: product.fiberType,
category: "DataCenter",
});
if (product.price && product.price > 0) {
const hash = contentHash(JSON.stringify({ price: product.price, part: product.partNumber }));
const updated = await upsertPriceObservation({
transceiverId: txId,
sourceVendorId: vendorId,
price: product.price,
currency: product.currency || "USD",
stockLevel: "in_stock",
url: product.url,
contentHash: hash,
});
if (updated) priceUpdates++;
}
totalProducts++;
} catch (err) {
console.warn(` Error: ${(err as Error).message.slice(0, 80)}`);
}
}
} catch (err) {
console.error(` Category failed: ${(err as Error).message}`);
}
await sleep(2000);
}
console.log(`\n=== 10Gtek Complete: ${totalProducts} products, ${priceUpdates} prices ===`);
}
if (require.main === module) {
scrape10Gtek()
.then(() => pool.end())
.catch((err) => { console.error("Fatal:", err); pool.end(); process.exit(1); });
}

View File

@ -0,0 +1,198 @@
/**
* UfiSpace Product Catalog Scraper
*
* Scrapes switch product pages from ufispace.com for specs and compatibility.
* UfiSpace publishes clean, well-structured product pages.
*
* Source: https://www.ufispace.com/products/datacenter-switches
*/
import { CheerioCrawler } from "crawlee";
import { pool, ensureWhiteboxVendor, findOrCreateSwitch } from "../utils/db";
const BASE_URL = "https://www.ufispace.com";
const PRODUCT_URLS = [
`${BASE_URL}/products/datacenter-switches`,
`${BASE_URL}/networking-white-box`,
];
function extractPortsFromSpec(specText: string): {
portsConfig: Record<string, number>;
totalPorts: number;
maxSpeedGbps: number;
formFactors: string[];
} {
const portsConfig: Record<string, number> = {};
let totalPorts = 0;
let maxSpeedGbps = 0;
const formFactors: string[] = [];
const portPattern = /(\d+)\s*x\s*(\d+)\s*G(?:bE|b\/s)?\s*(QSFP-DD|QSFP28|QSFP\+|QSFP56|SFP28|SFP\+|SFP56|OSFP|CFP2)?/gi;
let match: RegExpExecArray | null;
while ((match = portPattern.exec(specText)) !== null) {
const count = parseInt(match[1]);
const speed = parseInt(match[2]);
const ff = match[3]?.toUpperCase() || `${speed}G`;
const key = `${speed}G_${ff}`;
portsConfig[key] = (portsConfig[key] || 0) + count;
totalPorts += count;
maxSpeedGbps = Math.max(maxSpeedGbps, speed);
if (match[3] && !formFactors.includes(match[3].toUpperCase())) {
formFactors.push(match[3].toUpperCase());
}
}
return { portsConfig, totalPorts, maxSpeedGbps, formFactors };
}
function detectAsic(text: string): { vendor: string; model: string; series: string } {
const asicPatterns: Array<{ pattern: RegExp; vendor: string; model: string; series: string }> = [
{ pattern: /tomahawk\s*5/i, vendor: "Broadcom", model: "Tomahawk 5", series: "StrataDNX" },
{ pattern: /tomahawk\s*4/i, vendor: "Broadcom", model: "Tomahawk 4", series: "StrataDNX" },
{ pattern: /tomahawk\s*3/i, vendor: "Broadcom", model: "Tomahawk 3", series: "StrataDNX" },
{ pattern: /tomahawk\s*2/i, vendor: "Broadcom", model: "Tomahawk 2", series: "StrataDNX" },
{ pattern: /tomahawk/i, vendor: "Broadcom", model: "Tomahawk", series: "StrataDNX" },
{ pattern: /trident\s*(3|iii)/i, vendor: "Broadcom", model: "Trident III", series: "StrataDNX" },
{ pattern: /jericho\s*2/i, vendor: "Broadcom", model: "Jericho2", series: "StrataDNX" },
{ pattern: /spectrum/i, vendor: "NVIDIA", model: "Spectrum", series: "Spectrum" },
];
for (const { pattern, vendor, model, series } of asicPatterns) {
if (pattern.test(text)) {
return { vendor, model, series };
}
}
return { vendor: "Broadcom", model: "Unknown", series: "" };
}
export async function scrapeUfiSpace(): Promise<void> {
console.log("\n=== UfiSpace Scraper ===\n");
const vendorId = await ensureWhiteboxVendor("UfiSpace", "https://www.ufispace.com", {
isOdm: true,
ocpMember: true,
sonicContributor: true,
});
let created = 0;
let updated = 0;
const crawler = new CheerioCrawler({
maxConcurrency: 2,
maxRequestsPerMinute: 15,
requestHandlerTimeoutSecs: 30,
async requestHandler({ request, $, enqueueLinks }) {
// Product list pages — enqueue individual products
if (request.url.includes("products/") || request.url.includes("networking-white-box")) {
console.log(` Parsing: ${request.url}`);
const productLinks: string[] = [];
// Look for links to individual product pages
$("a").each((_i, el) => {
const href = $(el).attr("href") || "";
if (href.match(/\/S9[0-9]+-/i) || href.match(/\/product\//i)) {
const fullUrl = href.startsWith("http") ? href : `${BASE_URL}${href}`;
if (!productLinks.includes(fullUrl)) {
productLinks.push(fullUrl);
}
}
});
console.log(` Found ${productLinks.length} product links`);
for (const link of productLinks) {
await enqueueLinks({ urls: [link] });
}
return;
}
// Individual product page
const pageText = $("body").text();
const title = $("h1, .product-title").first().text().trim();
if (!title) return;
// Extract model name (S9600-32X, S9700-53DX, etc.)
const modelMatch = title.match(/(S\d{4}-\d+[A-Z]*)/i) || pageText.match(/(S\d{4}-\d+[A-Z]*)/i);
if (!modelMatch) return;
const model = modelMatch[1];
const portInfo = extractPortsFromSpec(pageText);
const asicInfo = detectAsic(pageText);
if (portInfo.totalPorts === 0) return;
const powerMatch = pageText.match(/(?:max|maximum)\s*power[:\s]*(\d+)\s*W/i);
const cpuMatch = pageText.match(/(Intel\s+(?:Xeon|Atom|Core)[^\n,;]+)/i);
const ramMatch = pageText.match(/(\d+)\s*GB?\s*(?:DDR[34]|RAM|memory)/i);
const storageMatch = pageText.match(/(\d+)\s*GB?\s*(?:SSD|eMMC|M\.2)/i);
const switchCapMatch = pageText.match(/switching\s*capacity[:\s]*([\d.]+)\s*Tb/i);
const seriesMatch = model.match(/^(S\d{4})/);
const series = seriesMatch ? seriesMatch[1] : "";
// Determine category based on model/series
let category: "DataCenter" | "Edge" | "SP" = "DataCenter";
if (model.includes("9510") || pageText.toLowerCase().includes("cell site")) {
category = "Edge";
}
const existing = await pool.query(
`SELECT id FROM switches WHERE model = $1 AND vendor_id = $2`,
[model, vendorId]
);
const isNew = existing.rows.length === 0;
await findOrCreateSwitch({
model,
vendorId,
series,
category,
layer: "L3",
portsConfig: portInfo.portsConfig,
totalPorts: portInfo.totalPorts,
maxSpeedGbps: portInfo.maxSpeedGbps,
switchingCapacityTbps: switchCapMatch ? parseFloat(switchCapMatch[1]) : undefined,
asicVendor: asicInfo.vendor,
asicModel: asicInfo.model,
asicSeries: asicInfo.series,
maxPowerW: powerMatch ? parseInt(powerMatch[1]) : undefined,
cpu: cpuMatch ? cpuMatch[1].trim() : undefined,
ramGb: ramMatch ? parseInt(ramMatch[1]) : undefined,
storageGb: storageMatch ? parseInt(storageMatch[1]) : undefined,
sonicCompatible: true,
isWhitebox: true,
onieSupport: true,
supportedNos: ["SONiC"],
transceiverFormFactors: portInfo.formFactors,
catalogUrl: request.url,
tags: [
"whitebox",
"UfiSpace",
`${portInfo.maxSpeedGbps}G`,
asicInfo.model,
...(category === "Edge" ? ["cell-site", "DCSG"] : []),
],
scrapeSource: "ufispace-catalog",
});
if (isNew) {
created++;
console.log(` + ${model} (${portInfo.maxSpeedGbps}G, ${asicInfo.vendor} ${asicInfo.model})`);
} else {
updated++;
}
},
failedRequestHandler({ request }) {
console.error(` ! Failed: ${request.url}`);
},
});
await crawler.run(PRODUCT_URLS);
console.log(`\n Created: ${created}, Updated: ${updated}\n`);
}

View File

@ -0,0 +1,692 @@
/**
* Whitebox / Open Networking Switch Seed Data
*
* Comprehensive catalog of whitebox ODM/OEM switches from:
* Edgecore, Celestica, Delta, QCT, Inventec, UfiSpace, Asterfusion, Netberg, Ragile
*
* Sources: Public datasheets, SONiC HCL, OCP specs, vendor product pages.
*/
import { pool, ensureWhiteboxVendor, findOrCreateSwitch } from "../utils/db";
import type { SwitchParams } from "../utils/db";
interface WhiteboxSeed {
vendor: string;
vendorWebsite: string;
vendorOpts: { isOdm: boolean; ocpMember: boolean; sonicContributor: boolean };
model: string;
series: string;
category: "DataCenter" | "Campus" | "Edge" | "Core" | "SP" | "Industrial";
layer: "L2" | "L3" | "L2/L3";
portsConfig: Record<string, number>;
totalPorts: number;
maxSpeedGbps: number;
switchingCapacityTbps?: number;
forwardingRateMpps?: number;
asicVendor: string;
asicModel: string;
asicSeries?: string;
rackUnits?: number;
maxPowerW?: number;
cpu?: string;
cpuCores?: number;
ramGb?: number;
storageGb?: number;
storageType?: string;
sonicCompatible: boolean;
onlCompatible?: boolean;
cumulusCompatible?: boolean;
dentCompatible?: boolean;
fbossCompatible?: boolean;
onieSupport?: boolean;
ocpStatus?: "Accepted" | "Inspired" | "None";
supportedNos?: string[];
sonicHwsku?: string;
transceiverFormFactors: string[];
catalogUrl?: string;
tags: string[];
}
// ═══════════════════════════════════════════════════════
// EDGECORE NETWORKS (Accton subsidiary)
// ═══════════════════════════════════════════════════════
const EDGECORE: WhiteboxSeed[] = [
{
vendor: "Edgecore Networks", vendorWebsite: "https://www.edge-core.com",
vendorOpts: { isOdm: true, ocpMember: true, sonicContributor: true },
model: "AS7946-74XKDB", series: "AS7946", category: "DataCenter", layer: "L3",
portsConfig: { "800G_OSFP": 74 }, totalPorts: 74,
maxSpeedGbps: 800, switchingCapacityTbps: 59.2,
asicVendor: "Broadcom", asicModel: "Tomahawk 5", asicSeries: "memory Memory",
rackUnits: 2, maxPowerW: 3500,
cpu: "Intel Xeon D-1649N", cpuCores: 8, ramGb: 32, storageGb: 128, storageType: "SSD",
sonicCompatible: true, onlCompatible: true, onieSupport: true,
ocpStatus: "Accepted",
supportedNos: ["SONiC", "ONL", "FBOSS", "DENT"],
sonicHwsku: "Edgecore-AS7946-74XKDB",
transceiverFormFactors: ["OSFP"],
catalogUrl: "https://www.edge-core.com/product/as7946-74xkdb/",
tags: ["800G", "whitebox", "spine", "AI-fabric", "OCP", "Tomahawk5"],
},
{
vendor: "Edgecore Networks", vendorWebsite: "https://www.edge-core.com",
vendorOpts: { isOdm: true, ocpMember: true, sonicContributor: true },
model: "AS9516-32D", series: "AS9516", category: "DataCenter", layer: "L3",
portsConfig: { "400G_QSFP-DD": 32 }, totalPorts: 32,
maxSpeedGbps: 400, switchingCapacityTbps: 12.8,
asicVendor: "Broadcom", asicModel: "Tomahawk 3", asicSeries: "memory Memory",
rackUnits: 1, maxPowerW: 1800,
cpu: "Intel Xeon D-1527", cpuCores: 4, ramGb: 16, storageGb: 64, storageType: "SSD",
sonicCompatible: true, onlCompatible: true, onieSupport: true,
ocpStatus: "Accepted",
supportedNos: ["SONiC", "ONL", "Cumulus", "FBOSS"],
sonicHwsku: "Edgecore-AS9516-32D",
transceiverFormFactors: ["QSFP-DD"],
catalogUrl: "https://www.edge-core.com/product/as9516-32d/",
tags: ["400G", "whitebox", "spine", "OCP", "Tomahawk3"],
},
{
vendor: "Edgecore Networks", vendorWebsite: "https://www.edge-core.com",
vendorOpts: { isOdm: true, ocpMember: true, sonicContributor: true },
model: "AS7726-32X", series: "AS7726", category: "DataCenter", layer: "L3",
portsConfig: { "100G_QSFP28": 32, "10G_SFP+": 2 }, totalPorts: 34,
maxSpeedGbps: 100, switchingCapacityTbps: 6.4, forwardingRateMpps: 4760,
asicVendor: "Broadcom", asicModel: "Trident III", asicSeries: "memory Memory",
rackUnits: 1, maxPowerW: 460,
cpu: "Intel Xeon D-1527", cpuCores: 4, ramGb: 8, storageGb: 32, storageType: "SSD",
sonicCompatible: true, onlCompatible: true, cumulusCompatible: true, onieSupport: true,
ocpStatus: "Accepted",
supportedNos: ["SONiC", "ONL", "Cumulus", "DENT", "FBOSS"],
sonicHwsku: "Edgecore-AS7726-32X",
transceiverFormFactors: ["QSFP28", "SFP+"],
catalogUrl: "https://www.edge-core.com/product/as7726-32x/",
tags: ["100G", "whitebox", "leaf", "spine", "OCP", "Trident3"],
},
{
vendor: "Edgecore Networks", vendorWebsite: "https://www.edge-core.com",
vendorOpts: { isOdm: true, ocpMember: true, sonicContributor: true },
model: "AS7712-32X", series: "AS7712", category: "DataCenter", layer: "L3",
portsConfig: { "100G_QSFP28": 32 }, totalPorts: 32,
maxSpeedGbps: 100, switchingCapacityTbps: 6.4, forwardingRateMpps: 4760,
asicVendor: "Broadcom", asicModel: "Tomahawk", asicSeries: "memory Memory",
rackUnits: 1, maxPowerW: 460,
cpu: "Intel Xeon D-1527", cpuCores: 4, ramGb: 8, storageGb: 32, storageType: "SSD",
sonicCompatible: true, onlCompatible: true, cumulusCompatible: true, onieSupport: true,
ocpStatus: "Accepted",
supportedNos: ["SONiC", "ONL", "Cumulus", "FBOSS"],
sonicHwsku: "Edgecore-AS7712-32X",
transceiverFormFactors: ["QSFP28"],
catalogUrl: "https://www.edge-core.com/product/as7712-32x/",
tags: ["100G", "whitebox", "leaf", "spine", "OCP", "Tomahawk"],
},
{
vendor: "Edgecore Networks", vendorWebsite: "https://www.edge-core.com",
vendorOpts: { isOdm: true, ocpMember: true, sonicContributor: true },
model: "DCS204", series: "DCS200", category: "DataCenter", layer: "L3",
portsConfig: { "25G_SFP28": 48, "100G_QSFP28": 8 }, totalPorts: 56,
maxSpeedGbps: 100, switchingCapacityTbps: 2.0,
asicVendor: "Broadcom", asicModel: "Trident III", asicSeries: "memory Memory",
rackUnits: 1, maxPowerW: 450,
cpu: "Intel Atom C3558", cpuCores: 4, ramGb: 8, storageGb: 32, storageType: "SSD",
sonicCompatible: true, onlCompatible: true, onieSupport: true,
ocpStatus: "Inspired",
supportedNos: ["SONiC", "ONL"],
transceiverFormFactors: ["SFP28", "QSFP28"],
tags: ["25G", "whitebox", "ToR", "leaf"],
},
{
vendor: "Edgecore Networks", vendorWebsite: "https://www.edge-core.com",
vendorOpts: { isOdm: true, ocpMember: true, sonicContributor: true },
model: "Minipack2", series: "Minipack", category: "DataCenter", layer: "L3",
portsConfig: { "400G_QSFP-DD": 128 }, totalPorts: 128,
maxSpeedGbps: 400, switchingCapacityTbps: 51.2,
asicVendor: "Broadcom", asicModel: "Memory Tomahawk 4", asicSeries: "memory Memory",
rackUnits: 4, maxPowerW: 5000,
cpu: "Intel Xeon D-1649N", cpuCores: 8, ramGb: 32, storageGb: 128, storageType: "SSD",
sonicCompatible: true, fbossCompatible: true, onieSupport: true,
ocpStatus: "Accepted",
supportedNos: ["SONiC", "FBOSS"],
transceiverFormFactors: ["QSFP-DD"],
tags: ["400G", "whitebox", "modular", "chassis", "OCP", "Meta"],
},
];
// ═══════════════════════════════════════════════════════
// CELESTICA
// ═══════════════════════════════════════════════════════
const CELESTICA: WhiteboxSeed[] = [
{
vendor: "Celestica", vendorWebsite: "https://www.celestica.com",
vendorOpts: { isOdm: true, ocpMember: true, sonicContributor: true },
model: "DS5000", series: "DS5000", category: "DataCenter", layer: "L3",
portsConfig: { "800G_OSFP": 64 }, totalPorts: 64,
maxSpeedGbps: 800, switchingCapacityTbps: 51.2,
asicVendor: "Broadcom", asicModel: "Tomahawk 5", asicSeries: "memory Memory",
rackUnits: 2, maxPowerW: 3200,
cpu: "Intel Xeon D-1649N", cpuCores: 8, ramGb: 32, storageGb: 128, storageType: "SSD",
sonicCompatible: true, onieSupport: true,
ocpStatus: "Inspired",
supportedNos: ["SONiC"],
transceiverFormFactors: ["OSFP"],
tags: ["800G", "whitebox", "spine", "AI-fabric", "Tomahawk5"],
},
{
vendor: "Celestica", vendorWebsite: "https://www.celestica.com",
vendorOpts: { isOdm: true, ocpMember: true, sonicContributor: true },
model: "DS4000", series: "DS4000", category: "DataCenter", layer: "L3",
portsConfig: { "400G_QSFP-DD": 32 }, totalPorts: 32,
maxSpeedGbps: 400, switchingCapacityTbps: 12.8,
asicVendor: "Broadcom", asicModel: "Tomahawk 4", asicSeries: "memory Memory",
rackUnits: 1, maxPowerW: 1500,
cpu: "Intel Xeon D-1527", cpuCores: 4, ramGb: 16, storageGb: 64, storageType: "SSD",
sonicCompatible: true, onlCompatible: true, onieSupport: true,
ocpStatus: "Inspired",
supportedNos: ["SONiC", "ONL"],
transceiverFormFactors: ["QSFP-DD"],
tags: ["400G", "whitebox", "spine", "Tomahawk4"],
},
{
vendor: "Celestica", vendorWebsite: "https://www.celestica.com",
vendorOpts: { isOdm: true, ocpMember: true, sonicContributor: true },
model: "Seastone2", series: "Seastone", category: "DataCenter", layer: "L3",
portsConfig: { "100G_QSFP28": 32 }, totalPorts: 32,
maxSpeedGbps: 100, switchingCapacityTbps: 6.4,
asicVendor: "Broadcom", asicModel: "Tomahawk", asicSeries: "memory Memory",
rackUnits: 1, maxPowerW: 460,
cpu: "Intel Xeon D-1527", cpuCores: 4, ramGb: 8, storageGb: 32, storageType: "SSD",
sonicCompatible: true, onlCompatible: true, onieSupport: true,
ocpStatus: "Accepted",
supportedNos: ["SONiC", "ONL", "Cumulus"],
sonicHwsku: "Celestica-Seastone2",
transceiverFormFactors: ["QSFP28"],
tags: ["100G", "whitebox", "leaf", "spine", "OCP"],
},
{
vendor: "Celestica", vendorWebsite: "https://www.celestica.com",
vendorOpts: { isOdm: true, ocpMember: true, sonicContributor: true },
model: "Midstone-200i", series: "Midstone", category: "DataCenter", layer: "L3",
portsConfig: { "25G_SFP28": 48, "100G_QSFP28": 8 }, totalPorts: 56,
maxSpeedGbps: 100, switchingCapacityTbps: 2.0,
asicVendor: "Broadcom", asicModel: "Trident III",
rackUnits: 1, maxPowerW: 450,
cpu: "Intel Atom C3558", cpuCores: 4, ramGb: 8, storageGb: 32, storageType: "SSD",
sonicCompatible: true, onieSupport: true,
supportedNos: ["SONiC", "ONL"],
transceiverFormFactors: ["SFP28", "QSFP28"],
tags: ["25G", "whitebox", "ToR", "leaf"],
},
];
// ═══════════════════════════════════════════════════════
// DELTA NETWORKS
// ═══════════════════════════════════════════════════════
const DELTA: WhiteboxSeed[] = [
{
vendor: "Delta Networks", vendorWebsite: "https://www.deltaww.com",
vendorOpts: { isOdm: true, ocpMember: true, sonicContributor: true },
model: "AG9064v2", series: "AG9064", category: "DataCenter", layer: "L3",
portsConfig: { "400G_QSFP-DD": 64 }, totalPorts: 64,
maxSpeedGbps: 400, switchingCapacityTbps: 25.6,
asicVendor: "Broadcom", asicModel: "Tomahawk 3",
rackUnits: 2, maxPowerW: 2500,
cpu: "Intel Xeon D-1527", cpuCores: 4, ramGb: 16, storageGb: 64, storageType: "SSD",
sonicCompatible: true, onlCompatible: true, onieSupport: true,
supportedNos: ["SONiC", "ONL"],
sonicHwsku: "Delta-AG9064v2",
transceiverFormFactors: ["QSFP-DD"],
tags: ["400G", "whitebox", "spine", "Tomahawk3"],
},
{
vendor: "Delta Networks", vendorWebsite: "https://www.deltaww.com",
vendorOpts: { isOdm: true, ocpMember: true, sonicContributor: true },
model: "AG9032v2A", series: "AG9032", category: "DataCenter", layer: "L3",
portsConfig: { "100G_QSFP28": 32 }, totalPorts: 32,
maxSpeedGbps: 100, switchingCapacityTbps: 6.4,
asicVendor: "Broadcom", asicModel: "Tomahawk+",
rackUnits: 1, maxPowerW: 480,
cpu: "Intel Xeon D-1527", cpuCores: 4, ramGb: 8, storageGb: 32, storageType: "SSD",
sonicCompatible: true, onlCompatible: true, onieSupport: true,
supportedNos: ["SONiC", "ONL", "DENT"],
sonicHwsku: "Delta-AG9032v2A",
transceiverFormFactors: ["QSFP28"],
tags: ["100G", "whitebox", "leaf", "spine"],
},
{
vendor: "Delta Networks", vendorWebsite: "https://www.deltaww.com",
vendorOpts: { isOdm: true, ocpMember: true, sonicContributor: true },
model: "AGC7648A", series: "AGC7648", category: "DataCenter", layer: "L2/L3",
portsConfig: { "25G_SFP28": 48, "100G_QSFP28": 6 }, totalPorts: 54,
maxSpeedGbps: 100, switchingCapacityTbps: 1.8,
asicVendor: "Broadcom", asicModel: "Trident III",
rackUnits: 1, maxPowerW: 420,
cpu: "Intel Atom C3558", cpuCores: 4, ramGb: 8, storageGb: 32, storageType: "SSD",
sonicCompatible: true, onieSupport: true,
supportedNos: ["SONiC", "ONL"],
transceiverFormFactors: ["SFP28", "QSFP28"],
tags: ["25G", "whitebox", "ToR", "leaf"],
},
{
vendor: "Delta Networks", vendorWebsite: "https://www.deltaww.com",
vendorOpts: { isOdm: true, ocpMember: true, sonicContributor: true },
model: "AG5648", series: "AG5648", category: "DataCenter", layer: "L2/L3",
portsConfig: { "10G_SFP+": 48, "40G_QSFP+": 6 }, totalPorts: 54,
maxSpeedGbps: 40, switchingCapacityTbps: 0.72,
asicVendor: "Broadcom", asicModel: "Memory Memory",
rackUnits: 1, maxPowerW: 350,
sonicCompatible: true, onlCompatible: true, dentCompatible: true, onieSupport: true,
supportedNos: ["SONiC", "ONL", "DENT"],
transceiverFormFactors: ["SFP+", "QSFP+"],
tags: ["10G", "whitebox", "campus", "enterprise", "DENT"],
},
];
// ═══════════════════════════════════════════════════════
// QUANTA CLOUD TECHNOLOGY (QCT)
// ═══════════════════════════════════════════════════════
const QCT: WhiteboxSeed[] = [
{
vendor: "Quanta Cloud Technology", vendorWebsite: "https://www.qct.io",
vendorOpts: { isOdm: true, ocpMember: true, sonicContributor: true },
model: "QuantaMesh T9032-IX9", series: "QuantaMesh T9032", category: "DataCenter", layer: "L3",
portsConfig: { "400G_QSFP-DD": 32 }, totalPorts: 32,
maxSpeedGbps: 400, switchingCapacityTbps: 12.8,
asicVendor: "Broadcom", asicModel: "Tomahawk 3",
rackUnits: 1, maxPowerW: 1500,
cpu: "Intel Xeon D-1527", cpuCores: 4, ramGb: 16, storageGb: 64, storageType: "SSD",
sonicCompatible: true, onieSupport: true,
supportedNos: ["SONiC", "ONL"],
sonicHwsku: "QuantaMesh-T9032-IX9",
transceiverFormFactors: ["QSFP-DD"],
catalogUrl: "https://www.qct.io/product/index/Networking",
tags: ["400G", "whitebox", "spine", "Tomahawk3"],
},
{
vendor: "Quanta Cloud Technology", vendorWebsite: "https://www.qct.io",
vendorOpts: { isOdm: true, ocpMember: true, sonicContributor: true },
model: "QuantaMesh T7064-IX4", series: "QuantaMesh T7064", category: "DataCenter", layer: "L3",
portsConfig: { "100G_QSFP28": 64 }, totalPorts: 64,
maxSpeedGbps: 100, switchingCapacityTbps: 12.8,
asicVendor: "Broadcom", asicModel: "Tomahawk 2",
rackUnits: 2, maxPowerW: 900,
cpu: "Intel Xeon D-1527", cpuCores: 4, ramGb: 8, storageGb: 32, storageType: "SSD",
sonicCompatible: true, onlCompatible: true, onieSupport: true,
supportedNos: ["SONiC", "ONL", "Cumulus"],
transceiverFormFactors: ["QSFP28"],
tags: ["100G", "whitebox", "spine", "Tomahawk2"],
},
{
vendor: "Quanta Cloud Technology", vendorWebsite: "https://www.qct.io",
vendorOpts: { isOdm: true, ocpMember: true, sonicContributor: true },
model: "QuantaMesh T7032-IX1", series: "QuantaMesh T7032", category: "DataCenter", layer: "L3",
portsConfig: { "100G_QSFP28": 32 }, totalPorts: 32,
maxSpeedGbps: 100, switchingCapacityTbps: 6.4,
asicVendor: "Broadcom", asicModel: "Tomahawk",
rackUnits: 1, maxPowerW: 460,
cpu: "Intel Xeon D-1527", cpuCores: 4, ramGb: 8, storageGb: 32, storageType: "SSD",
sonicCompatible: true, onlCompatible: true, cumulusCompatible: true, onieSupport: true,
supportedNos: ["SONiC", "ONL", "Cumulus"],
sonicHwsku: "QuantaMesh-T7032-IX1",
transceiverFormFactors: ["QSFP28"],
tags: ["100G", "whitebox", "leaf", "spine"],
},
{
vendor: "Quanta Cloud Technology", vendorWebsite: "https://www.qct.io",
vendorOpts: { isOdm: true, ocpMember: true, sonicContributor: true },
model: "QuantaMesh T3048-LY8", series: "QuantaMesh T3048", category: "DataCenter", layer: "L2/L3",
portsConfig: { "25G_SFP28": 48, "100G_QSFP28": 8 }, totalPorts: 56,
maxSpeedGbps: 100, switchingCapacityTbps: 2.0,
asicVendor: "Broadcom", asicModel: "Trident III",
rackUnits: 1, maxPowerW: 420,
sonicCompatible: true, onieSupport: true,
supportedNos: ["SONiC", "ONL"],
transceiverFormFactors: ["SFP28", "QSFP28"],
tags: ["25G", "whitebox", "ToR", "leaf"],
},
];
// ═══════════════════════════════════════════════════════
// UFISPACE
// ═══════════════════════════════════════════════════════
const UFISPACE: WhiteboxSeed[] = [
{
vendor: "UfiSpace", vendorWebsite: "https://www.ufispace.com",
vendorOpts: { isOdm: true, ocpMember: true, sonicContributor: true },
model: "S9600-72XC", series: "S9600", category: "DataCenter", layer: "L3",
portsConfig: { "100G_QSFP28": 72 }, totalPorts: 72,
maxSpeedGbps: 100, switchingCapacityTbps: 14.4,
asicVendor: "Broadcom", asicModel: "Jericho2",
rackUnits: 2, maxPowerW: 1200,
cpu: "Intel Xeon D-1527", cpuCores: 4, ramGb: 16, storageGb: 64, storageType: "SSD",
sonicCompatible: true, onieSupport: true,
supportedNos: ["SONiC"],
sonicHwsku: "UfiSpace-S9600-72XC",
transceiverFormFactors: ["QSFP28"],
catalogUrl: "https://www.ufispace.com/products/datacenter-switches",
tags: ["100G", "whitebox", "spine", "Jericho2", "deep-buffer"],
},
{
vendor: "UfiSpace", vendorWebsite: "https://www.ufispace.com",
vendorOpts: { isOdm: true, ocpMember: true, sonicContributor: true },
model: "S9600-32X", series: "S9600", category: "DataCenter", layer: "L3",
portsConfig: { "100G_QSFP28": 32 }, totalPorts: 32,
maxSpeedGbps: 100, switchingCapacityTbps: 6.4,
asicVendor: "Broadcom", asicModel: "Tomahawk",
rackUnits: 1, maxPowerW: 460,
cpu: "Intel Xeon D-1527", cpuCores: 4, ramGb: 8, storageGb: 32, storageType: "SSD",
sonicCompatible: true, onieSupport: true,
supportedNos: ["SONiC"],
transceiverFormFactors: ["QSFP28"],
tags: ["100G", "whitebox", "leaf", "spine"],
},
{
vendor: "UfiSpace", vendorWebsite: "https://www.ufispace.com",
vendorOpts: { isOdm: true, ocpMember: true, sonicContributor: true },
model: "S9700-53DX", series: "S9700", category: "DataCenter", layer: "L3",
portsConfig: { "400G_QSFP-DD": 32, "100G_QSFP28": 20 }, totalPorts: 52,
maxSpeedGbps: 400, switchingCapacityTbps: 14.8,
asicVendor: "Broadcom", asicModel: "Jericho2",
rackUnits: 2, maxPowerW: 2000,
cpu: "Intel Xeon D-1649N", cpuCores: 8, ramGb: 16, storageGb: 64, storageType: "SSD",
sonicCompatible: true, onieSupport: true,
supportedNos: ["SONiC"],
transceiverFormFactors: ["QSFP-DD", "QSFP28"],
tags: ["400G", "whitebox", "disaggregated", "chassis", "DDC"],
},
{
vendor: "UfiSpace", vendorWebsite: "https://www.ufispace.com",
vendorOpts: { isOdm: true, ocpMember: true, sonicContributor: true },
model: "S9510-28DC", series: "S9510", category: "Edge", layer: "L3",
portsConfig: { "100G_QSFP28": 20, "10G_SFP+": 8 }, totalPorts: 28,
maxSpeedGbps: 100, switchingCapacityTbps: 2.4,
asicVendor: "Broadcom", asicModel: "Memory Trident III",
rackUnits: 1, maxPowerW: 350,
sonicCompatible: true, onieSupport: true,
supportedNos: ["SONiC"],
transceiverFormFactors: ["QSFP28", "SFP+"],
tags: ["cell-site", "whitebox", "edge", "DCSG", "telecom"],
},
{
vendor: "UfiSpace", vendorWebsite: "https://www.ufispace.com",
vendorOpts: { isOdm: true, ocpMember: true, sonicContributor: true },
model: "S9600-30DX", series: "S9600", category: "DataCenter", layer: "L3",
portsConfig: { "400G_QSFP-DD": 30, "10G_SFP+": 2 }, totalPorts: 32,
maxSpeedGbps: 400, switchingCapacityTbps: 12.0,
asicVendor: "Broadcom", asicModel: "Tomahawk 3",
rackUnits: 1, maxPowerW: 1500,
cpu: "Intel Xeon D-1527", cpuCores: 4, ramGb: 16, storageGb: 64, storageType: "SSD",
sonicCompatible: true, onieSupport: true,
supportedNos: ["SONiC"],
transceiverFormFactors: ["QSFP-DD", "SFP+"],
tags: ["400G", "whitebox", "spine", "Tomahawk3"],
},
];
// ═══════════════════════════════════════════════════════
// INVENTEC
// ═══════════════════════════════════════════════════════
const INVENTEC: WhiteboxSeed[] = [
{
vendor: "Inventec", vendorWebsite: "https://www.inventec.com",
vendorOpts: { isOdm: true, ocpMember: true, sonicContributor: true },
model: "D7332", series: "D7332", category: "DataCenter", layer: "L3",
portsConfig: { "400G_QSFP-DD": 32 }, totalPorts: 32,
maxSpeedGbps: 400, switchingCapacityTbps: 12.8,
asicVendor: "Broadcom", asicModel: "Tomahawk 3",
rackUnits: 1, maxPowerW: 1500,
sonicCompatible: true, onlCompatible: true, onieSupport: true,
supportedNos: ["SONiC", "ONL"],
transceiverFormFactors: ["QSFP-DD"],
tags: ["400G", "whitebox", "spine"],
},
{
vendor: "Inventec", vendorWebsite: "https://www.inventec.com",
vendorOpts: { isOdm: true, ocpMember: true, sonicContributor: true },
model: "D7264Q28B", series: "D7264", category: "DataCenter", layer: "L3",
portsConfig: { "100G_QSFP28": 64 }, totalPorts: 64,
maxSpeedGbps: 100, switchingCapacityTbps: 12.8,
asicVendor: "Broadcom", asicModel: "Tomahawk 2",
rackUnits: 2, maxPowerW: 900,
sonicCompatible: true, onlCompatible: true, onieSupport: true,
supportedNos: ["SONiC", "ONL"],
sonicHwsku: "Inventec-D7264Q28B",
transceiverFormFactors: ["QSFP28"],
tags: ["100G", "whitebox", "spine", "Tomahawk2"],
},
{
vendor: "Inventec", vendorWebsite: "https://www.inventec.com",
vendorOpts: { isOdm: true, ocpMember: true, sonicContributor: true },
model: "D7054Q28B", series: "D7054", category: "DataCenter", layer: "L2/L3",
portsConfig: { "25G_SFP28": 48, "100G_QSFP28": 6 }, totalPorts: 54,
maxSpeedGbps: 100, switchingCapacityTbps: 1.8,
asicVendor: "Broadcom", asicModel: "Trident III",
rackUnits: 1, maxPowerW: 420,
sonicCompatible: true, onlCompatible: true, onieSupport: true,
supportedNos: ["SONiC", "ONL"],
transceiverFormFactors: ["SFP28", "QSFP28"],
tags: ["25G", "whitebox", "ToR", "leaf"],
},
];
// ═══════════════════════════════════════════════════════
// ASTERFUSION
// ═══════════════════════════════════════════════════════
const ASTERFUSION: WhiteboxSeed[] = [
{
vendor: "Asterfusion", vendorWebsite: "https://www.asterfusion.com",
vendorOpts: { isOdm: true, ocpMember: false, sonicContributor: true },
model: "CX864E-N", series: "CX864", category: "DataCenter", layer: "L3",
portsConfig: { "800G_OSFP": 64 }, totalPorts: 64,
maxSpeedGbps: 800, switchingCapacityTbps: 51.2,
asicVendor: "Marvell", asicModel: "Teralynx 10",
rackUnits: 2, maxPowerW: 3000,
sonicCompatible: true, onieSupport: true,
supportedNos: ["SONiC"],
transceiverFormFactors: ["OSFP"],
tags: ["800G", "whitebox", "spine", "AI-fabric", "Marvell"],
},
{
vendor: "Asterfusion", vendorWebsite: "https://www.asterfusion.com",
vendorOpts: { isOdm: true, ocpMember: false, sonicContributor: true },
model: "CX732Q-N", series: "CX732", category: "DataCenter", layer: "L3",
portsConfig: { "400G_QSFP-DD": 32 }, totalPorts: 32,
maxSpeedGbps: 400, switchingCapacityTbps: 12.8,
asicVendor: "Marvell", asicModel: "Teralynx 7",
rackUnits: 1, maxPowerW: 1500,
sonicCompatible: true, onieSupport: true,
supportedNos: ["SONiC"],
transceiverFormFactors: ["QSFP-DD"],
tags: ["400G", "whitebox", "spine", "Marvell"],
},
{
vendor: "Asterfusion", vendorWebsite: "https://www.asterfusion.com",
vendorOpts: { isOdm: true, ocpMember: false, sonicContributor: true },
model: "CX564P-N", series: "CX564", category: "DataCenter", layer: "L3",
portsConfig: { "100G_QSFP28": 64 }, totalPorts: 64,
maxSpeedGbps: 100, switchingCapacityTbps: 12.8,
asicVendor: "Marvell", asicModel: "Teralynx 7",
rackUnits: 2, maxPowerW: 800,
sonicCompatible: true, onieSupport: true,
supportedNos: ["SONiC"],
transceiverFormFactors: ["QSFP28"],
tags: ["100G", "whitebox", "spine", "Marvell"],
},
{
vendor: "Asterfusion", vendorWebsite: "https://www.asterfusion.com",
vendorOpts: { isOdm: true, ocpMember: false, sonicContributor: true },
model: "CX532P-N", series: "CX532", category: "DataCenter", layer: "L3",
portsConfig: { "100G_QSFP28": 32, "10G_SFP+": 2 }, totalPorts: 34,
maxSpeedGbps: 100, switchingCapacityTbps: 6.4,
asicVendor: "Marvell", asicModel: "Teralynx",
rackUnits: 1, maxPowerW: 460,
sonicCompatible: true, onieSupport: true,
supportedNos: ["SONiC"],
transceiverFormFactors: ["QSFP28", "SFP+"],
tags: ["100G", "whitebox", "leaf", "spine", "Marvell"],
},
{
vendor: "Asterfusion", vendorWebsite: "https://www.asterfusion.com",
vendorOpts: { isOdm: true, ocpMember: false, sonicContributor: true },
model: "CX308P-48Y-N", series: "CX308", category: "DataCenter", layer: "L2/L3",
portsConfig: { "25G_SFP28": 48, "100G_QSFP28": 8 }, totalPorts: 56,
maxSpeedGbps: 100, switchingCapacityTbps: 2.0,
asicVendor: "Marvell", asicModel: "Prestera",
rackUnits: 1, maxPowerW: 400,
sonicCompatible: true, onieSupport: true,
supportedNos: ["SONiC"],
transceiverFormFactors: ["SFP28", "QSFP28"],
tags: ["25G", "whitebox", "ToR", "leaf", "Marvell"],
},
];
// ═══════════════════════════════════════════════════════
// NETBERG
// ═══════════════════════════════════════════════════════
const NETBERG: WhiteboxSeed[] = [
{
vendor: "Netberg", vendorWebsite: "https://netbergtw.com",
vendorOpts: { isOdm: true, ocpMember: false, sonicContributor: true },
model: "Aurora 810", series: "Aurora", category: "DataCenter", layer: "L3",
portsConfig: { "800G_OSFP": 64 }, totalPorts: 64,
maxSpeedGbps: 800, switchingCapacityTbps: 51.2,
asicVendor: "Broadcom", asicModel: "Tomahawk 5",
rackUnits: 2, maxPowerW: 3200,
sonicCompatible: true, onieSupport: true,
supportedNos: ["SONiC"],
transceiverFormFactors: ["OSFP"],
tags: ["800G", "whitebox", "spine", "AI-fabric", "Tomahawk5"],
},
{
vendor: "Netberg", vendorWebsite: "https://netbergtw.com",
vendorOpts: { isOdm: true, ocpMember: false, sonicContributor: true },
model: "Aurora 750", series: "Aurora", category: "DataCenter", layer: "L3",
portsConfig: { "400G_QSFP-DD": 32 }, totalPorts: 32,
maxSpeedGbps: 400, switchingCapacityTbps: 12.8,
asicVendor: "Broadcom", asicModel: "Tomahawk 3",
rackUnits: 1, maxPowerW: 1500,
sonicCompatible: true, onieSupport: true,
supportedNos: ["SONiC"],
transceiverFormFactors: ["QSFP-DD"],
tags: ["400G", "whitebox", "spine", "Tomahawk3"],
},
{
vendor: "Netberg", vendorWebsite: "https://netbergtw.com",
vendorOpts: { isOdm: true, ocpMember: false, sonicContributor: true },
model: "Aurora 620", series: "Aurora", category: "DataCenter", layer: "L3",
portsConfig: { "100G_QSFP28": 32 }, totalPorts: 32,
maxSpeedGbps: 100, switchingCapacityTbps: 6.4,
asicVendor: "Broadcom", asicModel: "Tomahawk",
rackUnits: 1, maxPowerW: 460,
sonicCompatible: true, onieSupport: true,
supportedNos: ["SONiC"],
transceiverFormFactors: ["QSFP28"],
tags: ["100G", "whitebox", "leaf", "spine"],
},
];
// ═══════════════════════════════════════════════════════
// RAGILE NETWORKS
// ═══════════════════════════════════════════════════════
const RAGILE: WhiteboxSeed[] = [
{
vendor: "Ragile Networks", vendorWebsite: "https://www.ragilenetworks.com",
vendorOpts: { isOdm: true, ocpMember: false, sonicContributor: true },
model: "RA-B6920-4S", series: "RA-B6920", category: "DataCenter", layer: "L3",
portsConfig: { "400G_QSFP-DD": 32 }, totalPorts: 32,
maxSpeedGbps: 400, switchingCapacityTbps: 12.8,
asicVendor: "Broadcom", asicModel: "Tomahawk 3",
rackUnits: 1, maxPowerW: 1500,
sonicCompatible: true, onieSupport: true,
supportedNos: ["SONiC"],
transceiverFormFactors: ["QSFP-DD"],
tags: ["400G", "whitebox", "spine"],
},
{
vendor: "Ragile Networks", vendorWebsite: "https://www.ragilenetworks.com",
vendorOpts: { isOdm: true, ocpMember: false, sonicContributor: true },
model: "RA-B6510-48V8C", series: "RA-B6510", category: "DataCenter", layer: "L2/L3",
portsConfig: { "25G_SFP28": 48, "100G_QSFP28": 8 }, totalPorts: 56,
maxSpeedGbps: 100, switchingCapacityTbps: 2.0,
asicVendor: "Broadcom", asicModel: "Trident III",
rackUnits: 1, maxPowerW: 420,
sonicCompatible: true, onieSupport: true,
supportedNos: ["SONiC"],
transceiverFormFactors: ["SFP28", "QSFP28"],
tags: ["25G", "whitebox", "ToR", "leaf"],
},
];
// ═══════════════════════════════════════════════════════
// SEED FUNCTION
// ═══════════════════════════════════════════════════════
const ALL_WHITEBOX = [
...EDGECORE, ...CELESTICA, ...DELTA, ...QCT,
...UFISPACE, ...INVENTEC, ...ASTERFUSION, ...NETBERG, ...RAGILE,
];
export async function seedWhiteboxSwitches(): Promise<void> {
console.log(`\n=== Seeding ${ALL_WHITEBOX.length} Whitebox Switches ===\n`);
let created = 0;
let updated = 0;
for (const sw of ALL_WHITEBOX) {
try {
const vendorId = await ensureWhiteboxVendor(sw.vendor, sw.vendorWebsite, sw.vendorOpts);
const existing = await pool.query(
`SELECT id FROM switches WHERE model = $1 AND vendor_id = $2`,
[sw.model, vendorId]
);
const isNew = existing.rows.length === 0;
await findOrCreateSwitch({
model: sw.model,
vendorId,
series: sw.series,
category: sw.category,
layer: sw.layer,
portsConfig: sw.portsConfig,
totalPorts: sw.totalPorts,
maxSpeedGbps: sw.maxSpeedGbps,
switchingCapacityTbps: sw.switchingCapacityTbps,
forwardingRateMpps: sw.forwardingRateMpps,
asicVendor: sw.asicVendor,
asicModel: sw.asicModel,
asicSeries: sw.asicSeries,
rackUnits: sw.rackUnits,
maxPowerW: sw.maxPowerW,
sonicCompatible: sw.sonicCompatible,
tags: sw.tags,
// Whitebox fields
isWhitebox: true,
isOcpAccepted: sw.ocpStatus === "Accepted",
ocpStatus: sw.ocpStatus || "None",
supportedNos: sw.supportedNos || [],
onlCompatible: sw.onlCompatible,
dentCompatible: sw.dentCompatible,
cumulusCompatible: sw.cumulusCompatible,
fbossCompatible: sw.fbossCompatible,
cpu: sw.cpu,
cpuCores: sw.cpuCores,
ramGb: sw.ramGb,
storageGb: sw.storageGb,
storageType: sw.storageType,
transceiverFormFactors: sw.transceiverFormFactors,
catalogUrl: sw.catalogUrl,
sonicHwsku: sw.sonicHwsku,
onieSupport: sw.onieSupport,
scrapeSource: "whitebox-seed",
});
if (isNew) {
created++;
console.log(` + ${sw.vendor} ${sw.model} (${sw.maxSpeedGbps}G, ${sw.asicVendor} ${sw.asicModel})`);
} else {
updated++;
}
} catch (err) {
console.error(` ! Error seeding ${sw.vendor} ${sw.model}:`, err);
}
}
console.log(`\n Created: ${created}, Updated: ${updated}, Total: ${ALL_WHITEBOX.length}\n`);
}

View File

@ -0,0 +1,182 @@
/**
* Product Asset Utilities Download images, datasheets, manuals
*
* Handles downloading product assets from vendor websites,
* storing them locally, and updating the database.
*/
import { pool } from "./db";
import { createHash } from "crypto";
import { writeFile, mkdir } from "fs/promises";
import { join, basename, extname } from "path";
import { existsSync } from "fs";
const ASSETS_DIR = process.env.ASSETS_DIR || join(__dirname, "..", "..", "..", "..", "assets");
const IMAGES_DIR = join(ASSETS_DIR, "images");
const DATASHEETS_DIR = join(ASSETS_DIR, "datasheets");
const MANUALS_DIR = join(ASSETS_DIR, "manuals");
async function ensureDir(dir: string): Promise<void> {
if (!existsSync(dir)) {
await mkdir(dir, { recursive: true });
}
}
function contentHash(data: Buffer): string {
return createHash("sha256").update(data).digest("hex").slice(0, 16);
}
function sanitizeFilename(name: string): string {
return name.toLowerCase().replace(/[^a-z0-9.-]+/g, "-").replace(/^-|-$/g, "");
}
export interface AssetDownloadResult {
localPath: string;
hash: string;
sizeBytes: number;
}
/**
* Download a file from URL and save locally.
* Returns null if download fails (non-fatal).
*/
export async function downloadAsset(
url: string,
destDir: string,
filenamePrefix: string
): Promise<AssetDownloadResult | null> {
try {
await ensureDir(destDir);
const ext = extname(new URL(url).pathname) || ".bin";
const filename = `${sanitizeFilename(filenamePrefix)}${ext}`;
const localPath = join(destDir, filename);
const response = await fetch(url, {
headers: {
"User-Agent": "TIP-Scraper/1.0 (Transceiver Intelligence Platform)",
"Accept": "*/*",
},
signal: AbortSignal.timeout(30_000),
});
if (!response.ok) {
console.log(` [SKIP] ${url} → HTTP ${response.status}`);
return null;
}
const buffer = Buffer.from(await response.arrayBuffer());
const hash = contentHash(buffer);
await writeFile(localPath, buffer);
return { localPath, hash, sizeBytes: buffer.length };
} catch (err) {
console.log(` [FAIL] ${url}${(err as Error).message.slice(0, 80)}`);
return null;
}
}
/**
* Download product image and update switches table.
*/
export async function downloadSwitchImage(
switchId: string,
imageUrl: string,
vendor: string,
model: string
): Promise<boolean> {
const vendorDir = join(IMAGES_DIR, "switches", sanitizeFilename(vendor));
const result = await downloadAsset(imageUrl, vendorDir, model);
if (!result) return false;
await pool.query(
`UPDATE switches SET image_url = $2, image_local_path = $3, assets_scraped_at = NOW() WHERE id = $1`,
[switchId, imageUrl, result.localPath]
);
return true;
}
/**
* Download datasheet PDF and create product_documents entry.
*/
export async function downloadSwitchDatasheet(
switchId: string,
vendorId: string,
datasheetUrl: string,
title: string,
vendor: string,
model: string
): Promise<boolean> {
const vendorDir = join(DATASHEETS_DIR, "switches", sanitizeFilename(vendor));
const result = await downloadAsset(datasheetUrl, vendorDir, model);
if (!result) return false;
// Update switch record
await pool.query(
`UPDATE switches SET datasheet_url = $2, datasheet_local_path = $3, assets_scraped_at = NOW() WHERE id = $1`,
[switchId, datasheetUrl, result.localPath]
);
// Create document record (upsert by content_hash)
await pool.query(
`INSERT INTO product_documents (switch_id, vendor_id, doc_type, title, source_url, local_path, file_size_bytes, content_hash)
VALUES ($1, $2, 'datasheet', $3, $4, $5, $6, $7)
ON CONFLICT (content_hash) DO UPDATE SET downloaded_at = NOW()`,
[switchId, vendorId, title, datasheetUrl, result.localPath, result.sizeBytes, result.hash]
);
return true;
}
/**
* Download manual/guide PDF and create product_documents entry.
*/
export async function downloadSwitchManual(
switchId: string,
vendorId: string,
manualUrl: string,
title: string,
docType: string,
vendor: string,
model: string
): Promise<boolean> {
const vendorDir = join(MANUALS_DIR, "switches", sanitizeFilename(vendor));
const filename = `${sanitizeFilename(model)}-${sanitizeFilename(docType)}`;
const result = await downloadAsset(manualUrl, vendorDir, filename);
if (!result) return false;
await pool.query(
`INSERT INTO product_documents (switch_id, vendor_id, doc_type, title, source_url, local_path, file_size_bytes, content_hash)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8)
ON CONFLICT (content_hash) DO UPDATE SET downloaded_at = NOW()`,
[switchId, vendorId, docType, title, manualUrl, result.localPath, result.sizeBytes, result.hash]
);
return true;
}
/**
* Update switch product_page_url without downloading.
*/
export async function setSwitchProductPage(switchId: string, url: string): Promise<void> {
await pool.query(
`UPDATE switches SET product_page_url = $2, assets_scraped_at = NOW() WHERE id = $1`,
[switchId, url]
);
}
/**
* Update vendor documentation portal URLs.
*/
export async function setVendorDocUrls(
vendorId: string,
urls: { docsPortal?: string; datasheetLibrary?: string; imageCdn?: string; supportPortal?: string }
): Promise<void> {
await pool.query(
`UPDATE vendors SET
docs_portal_url = COALESCE($2, docs_portal_url),
datasheet_library_url = COALESCE($3, datasheet_library_url),
image_cdn_base = COALESCE($4, image_cdn_base),
support_portal_url = COALESCE($5, support_portal_url),
updated_at = NOW()
WHERE id = $1`,
[vendorId, urls.docsPortal || null, urls.datasheetLibrary || null, urls.imageCdn || null, urls.supportPortal || null]
);
}

View File

@ -0,0 +1,128 @@
/**
* WS4: Competitor Change Detection
*
* Compares current scrape results with previous observations
* and generates alerts for price changes, new products, stock changes.
*/
import { Pool } from "pg";
const pool = new Pool({
host: process.env.POSTGRES_HOST || "localhost",
port: parseInt(process.env.POSTGRES_PORT || "5433"),
database: process.env.POSTGRES_DB || "transceiver_db",
user: process.env.POSTGRES_USER || "tip",
password: process.env.POSTGRES_PASSWORD || "tip_dev_2026",
max: 3,
});
interface PriceObservation {
transceiver_id: string;
vendor_id: string;
price: number;
currency: string;
stock_level?: string;
part_number?: string;
product_name?: string;
form_factor?: string;
speed_gbps?: number;
source_url?: string;
}
/**
* After a scraper run, call this to detect changes and generate alerts.
*/
export async function detectChanges(
vendorId: string,
currentObservations: PriceObservation[]
): Promise<{ alerts: number; priceChanges: number; newProducts: number }> {
let alerts = 0;
let priceChanges = 0;
let newProducts = 0;
for (const obs of currentObservations) {
try {
// Get last known price for this transceiver from this vendor
const prev = await pool.query(
`SELECT price, currency, stock_level
FROM price_observations
WHERE transceiver_id = $1 AND source_vendor_id = $2
ORDER BY time DESC LIMIT 1`,
[obs.transceiver_id, obs.vendor_id]
);
if (prev.rows.length === 0) {
// New product alert
await pool.query(
`INSERT INTO competitor_alerts (vendor_id, transceiver_id, alert_type, severity,
new_price, currency, part_number, product_name, form_factor, speed_gbps, source_url)
VALUES ($1, $2, 'new_product', 'medium', $3, $4, $5, $6, $7, $8, $9)`,
[obs.vendor_id, obs.transceiver_id, obs.price, obs.currency,
obs.part_number, obs.product_name, obs.form_factor, obs.speed_gbps, obs.source_url]
);
newProducts++;
alerts++;
continue;
}
const prevPrice = parseFloat(prev.rows[0].price);
const prevStock = prev.rows[0].stock_level;
// Price change detection (>2% threshold to avoid noise)
if (Math.abs(obs.price - prevPrice) / prevPrice > 0.02) {
const delta = obs.price - prevPrice;
const deltaPct = (delta / prevPrice) * 100;
const alertType = delta < 0 ? 'price_drop' : 'price_increase';
const severity = Math.abs(deltaPct) > 15 ? 'high' : Math.abs(deltaPct) > 5 ? 'medium' : 'low';
// Insert alert
await pool.query(
`INSERT INTO competitor_alerts (vendor_id, transceiver_id, alert_type, severity,
old_price, new_price, price_delta, price_pct, currency,
part_number, product_name, form_factor, speed_gbps, source_url)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14)`,
[obs.vendor_id, obs.transceiver_id, alertType, severity,
prevPrice, obs.price, delta, deltaPct, obs.currency,
obs.part_number, obs.product_name, obs.form_factor, obs.speed_gbps, obs.source_url]
);
// Insert price change record
await pool.query(
`INSERT INTO price_changes (transceiver_id, vendor_id, old_price, new_price, delta, delta_pct, currency)
VALUES ($1, $2, $3, $4, $5, $6, $7)`,
[obs.transceiver_id, obs.vendor_id, prevPrice, obs.price, delta, deltaPct, obs.currency]
);
priceChanges++;
alerts++;
}
// Stock change detection
if (prevStock && obs.stock_level && prevStock !== obs.stock_level) {
if (obs.stock_level === 'out_of_stock' && prevStock !== 'out_of_stock') {
await pool.query(
`INSERT INTO competitor_alerts (vendor_id, transceiver_id, alert_type, severity,
part_number, product_name, form_factor, speed_gbps, source_url)
VALUES ($1, $2, 'out_of_stock', 'low', $3, $4, $5, $6, $7)`,
[obs.vendor_id, obs.transceiver_id, obs.part_number, obs.product_name,
obs.form_factor, obs.speed_gbps, obs.source_url]
);
alerts++;
} else if (prevStock === 'out_of_stock' && obs.stock_level !== 'out_of_stock') {
await pool.query(
`INSERT INTO competitor_alerts (vendor_id, transceiver_id, alert_type, severity,
new_price, currency, part_number, product_name, form_factor, speed_gbps, source_url)
VALUES ($1, $2, 'back_in_stock', 'low', $3, $4, $5, $6, $7, $8, $9)`,
[obs.vendor_id, obs.transceiver_id, obs.price, obs.currency,
obs.part_number, obs.product_name, obs.form_factor, obs.speed_gbps, obs.source_url]
);
alerts++;
}
}
} catch (err) {
console.error(`Change detection error for ${obs.part_number}:`, err);
}
}
console.log(`Change detection: ${alerts} alerts (${priceChanges} price changes, ${newProducts} new products)`);
return { alerts, priceChanges, newProducts };
}

View File

@ -0,0 +1,372 @@
import { Pool } from "pg";
import { config } from "dotenv";
import { join } from "path";
config({ path: join(__dirname, "..", "..", "..", "..", ".env") });
export const pool = new Pool({
host: process.env.POSTGRES_HOST || "localhost",
port: parseInt(process.env.POSTGRES_PORT || "5433"),
database: process.env.POSTGRES_DB || "transceiver_db",
user: process.env.POSTGRES_USER || "tip",
password: process.env.POSTGRES_PASSWORD || "tip_dev_2026",
max: 10,
});
export async function upsertPriceObservation(params: {
transceiverId: string;
sourceVendorId: string;
price: number;
currency: string;
stockLevel: string;
quantityAvailable?: number;
leadTimeDays?: number;
url?: string;
contentHash: string;
}): Promise<boolean> {
// Check if price changed via content hash
const existing = await pool.query(
`SELECT content_hash FROM price_observations
WHERE transceiver_id = $1 AND source_vendor_id = $2
ORDER BY time DESC LIMIT 1`,
[params.transceiverId, params.sourceVendorId]
);
if (existing.rows.length > 0 && existing.rows[0].content_hash === params.contentHash) {
return false; // No change
}
await pool.query(
`INSERT INTO price_observations (time, transceiver_id, source_vendor_id, price, currency, stock_level, quantity_available, lead_time_days, url, content_hash)
VALUES (NOW(), $1, $2, $3, $4, $5, $6, $7, $8, $9)`,
[
params.transceiverId,
params.sourceVendorId,
params.price,
params.currency,
params.stockLevel,
params.quantityAvailable || null,
params.leadTimeDays || null,
params.url || null,
params.contentHash,
]
);
return true; // New observation written
}
export async function findOrCreateScrapedTransceiver(params: {
partNumber: string;
vendorId: string;
formFactor?: string;
speedGbps?: number;
speed?: string;
reachMeters?: number;
reachLabel?: string;
fiberType?: string;
wavelengths?: string;
category?: string;
}): Promise<string> {
// Try to match existing transceiver by part number + vendor
const existing = await pool.query(
`SELECT id FROM transceivers WHERE part_number = $1 AND vendor_id = $2`,
[params.partNumber, params.vendorId]
);
if (existing.rows.length > 0) {
return existing.rows[0].id;
}
// Create new transceiver entry
const slug = `scraped-${params.partNumber.toLowerCase().replace(/[^a-z0-9]+/g, "-")}`;
const result = await pool.query(
`INSERT INTO transceivers (slug, part_number, vendor_id, form_factor, speed_gbps, speed, reach_meters, reach_label, fiber_type, wavelengths, category, market_status)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, 'Mainstream')
ON CONFLICT (slug) DO UPDATE SET updated_at = NOW()
RETURNING id`,
[
slug,
params.partNumber,
params.vendorId,
params.formFactor || "SFP",
params.speedGbps || 0,
params.speed || "Unknown",
params.reachMeters || 0,
params.reachLabel || "",
params.fiberType || "",
params.wavelengths || "",
params.category || "DataCenter",
]
);
return result.rows[0].id;
}
export interface SwitchParams {
model: string;
vendorId: string;
series?: string;
category?: string;
layer?: string;
portsConfig?: Record<string, number>;
totalPorts?: number;
uplinkSpeedGbps?: number;
maxSpeedGbps?: number;
switchingCapacityTbps?: number;
forwardingRateMpps?: number;
asicVendor?: string;
asicModel?: string;
asicSeries?: string;
asicGeneration?: string;
rackUnits?: number;
maxPowerW?: number;
typicalPowerW?: number;
poeSupport?: string;
stackingSupport?: boolean;
vxlanSupport?: boolean;
evpnSupport?: boolean;
bgpSupport?: boolean;
mplsSupport?: boolean;
openconfigSupport?: boolean;
sonicCompatible?: boolean;
macsecSupport?: boolean;
lifecycleStatus?: string;
releaseDate?: string;
eolDate?: string;
msrpUsd?: number;
tags?: string[];
// Whitebox-specific fields
isWhitebox?: boolean;
isOcpAccepted?: boolean;
ocpStatus?: string;
supportedNos?: string[];
onlCompatible?: boolean;
dentCompatible?: boolean;
cumulusCompatible?: boolean;
fbossCompatible?: boolean;
cpu?: string;
cpuCores?: number;
ramGb?: number;
storageGb?: number;
storageType?: string;
transceiverFormFactors?: string[];
catalogUrl?: string;
sonicHwsku?: string;
onieSupport?: boolean;
scrapeSource?: string;
}
export async function findOrCreateSwitch(params: SwitchParams): Promise<string> {
const existing = await pool.query(
`SELECT id FROM switches WHERE model = $1 AND vendor_id = $2`,
[params.model, params.vendorId]
);
if (existing.rows.length > 0) {
await pool.query(
`UPDATE switches SET
series = COALESCE($2, series),
category = COALESCE($3, category),
ports_config = COALESCE($4, ports_config),
total_ports = COALESCE($5, total_ports),
max_speed_gbps = COALESCE($6, max_speed_gbps),
switching_capacity_tbps = COALESCE($7, switching_capacity_tbps),
is_whitebox = COALESCE($8, is_whitebox),
supported_nos = COALESCE($9, supported_nos),
sonic_compatible = COALESCE($10, sonic_compatible),
sonic_hwsku = COALESCE($11, sonic_hwsku),
cpu = COALESCE($12, cpu),
ram_gb = COALESCE($13, ram_gb),
storage_gb = COALESCE($14, storage_gb),
transceiver_form_factors = COALESCE($15, transceiver_form_factors),
catalog_url = COALESCE($16, catalog_url),
is_ocp_accepted = COALESCE($17, is_ocp_accepted),
ocp_status = COALESCE($18, ocp_status),
onie_support = COALESCE($19, onie_support),
asic_series = COALESCE($20, asic_series),
last_scraped = CASE WHEN $21::text IS NOT NULL THEN NOW() ELSE last_scraped END,
scrape_source = COALESCE($21, scrape_source),
updated_at = NOW()
WHERE id = $1`,
[
existing.rows[0].id,
params.series || null,
params.category || null,
params.portsConfig ? JSON.stringify(params.portsConfig) : null,
params.totalPorts || null,
params.maxSpeedGbps || null,
params.switchingCapacityTbps || null,
params.isWhitebox ?? null,
params.supportedNos?.length ? params.supportedNos : null,
params.sonicCompatible ?? null,
params.sonicHwsku || null,
params.cpu || null,
params.ramGb || null,
params.storageGb || null,
params.transceiverFormFactors?.length ? params.transceiverFormFactors : null,
params.catalogUrl || null,
params.isOcpAccepted ?? null,
params.ocpStatus || null,
params.onieSupport ?? null,
params.asicSeries || null,
params.scrapeSource || null,
]
);
return existing.rows[0].id;
}
const result = await pool.query(
`INSERT INTO switches (
model, vendor_id, series, category, layer,
ports_config, total_ports, uplink_speed_gbps, max_speed_gbps,
switching_capacity_tbps, forwarding_rate_mpps,
asic_vendor, asic_model, asic_series, asic_generation,
rack_units, max_power_w, typical_power_w,
poe_support, stacking_support, vxlan_support, evpn_support,
bgp_support, mpls_support, openconfig_support, sonic_compatible, macsec_support,
lifecycle_status, release_date, eol_date, msrp_usd, tags,
is_whitebox, is_ocp_accepted, ocp_status, supported_nos,
onl_compatible, dent_compatible, cumulus_compatible, fboss_compatible,
cpu, cpu_cores, ram_gb, storage_gb, storage_type,
transceiver_form_factors, catalog_url, sonic_hwsku, onie_support,
last_scraped, scrape_source
) VALUES (
$1, $2, $3, $4, $5,
$6, $7, $8, $9,
$10, $11,
$12, $13, $14, $15,
$16, $17, $18,
$19, $20, $21, $22,
$23, $24, $25, $26, $27,
$28, $29, $30, $31, $32,
$33, $34, $35, $36,
$37, $38, $39, $40,
$41, $42, $43, $44, $45,
$46, $47, $48, $49,
$50, $51
)
ON CONFLICT (vendor_id, model) DO UPDATE SET updated_at = NOW()
RETURNING id`,
[
params.model,
params.vendorId,
params.series || null,
params.category || 'DataCenter',
params.layer || 'L3',
JSON.stringify(params.portsConfig || {}),
params.totalPorts || null,
params.uplinkSpeedGbps || null,
params.maxSpeedGbps || null,
params.switchingCapacityTbps || null,
params.forwardingRateMpps || null,
params.asicVendor || null,
params.asicModel || null,
params.asicSeries || null,
params.asicGeneration || null,
params.rackUnits || null,
params.maxPowerW || null,
params.typicalPowerW || null,
params.poeSupport || 'None',
params.stackingSupport || false,
params.vxlanSupport || false,
params.evpnSupport || false,
params.bgpSupport || false,
params.mplsSupport || false,
params.openconfigSupport || false,
params.sonicCompatible || false,
params.macsecSupport || false,
params.lifecycleStatus || 'Active',
params.releaseDate || null,
params.eolDate || null,
params.msrpUsd || null,
params.tags || [],
params.isWhitebox || false,
params.isOcpAccepted || false,
params.ocpStatus || 'None',
params.supportedNos || [],
params.onlCompatible || false,
params.dentCompatible || false,
params.cumulusCompatible || false,
params.fbossCompatible || false,
params.cpu || null,
params.cpuCores || null,
params.ramGb || null,
params.storageGb || null,
params.storageType || null,
params.transceiverFormFactors || [],
params.catalogUrl || null,
params.sonicHwsku || null,
params.onieSupport || false,
params.scrapeSource ? new Date() : null,
params.scrapeSource || null,
]
);
return result.rows[0].id;
}
export async function ensureWhiteboxVendor(
name: string,
website?: string,
options?: { isOdm?: boolean; ocpMember?: boolean; sonicContributor?: boolean }
): Promise<string> {
const existing = await pool.query(`SELECT id FROM vendors WHERE name ILIKE $1`, [name]);
if (existing.rows.length > 0) {
if (options) {
await pool.query(
`UPDATE vendors SET
is_whitebox_vendor = TRUE,
is_odm = COALESCE($2, is_odm),
ocp_member = COALESCE($3, ocp_member),
sonic_contributor = COALESCE($4, sonic_contributor),
updated_at = NOW()
WHERE id = $1`,
[existing.rows[0].id, options.isOdm ?? null, options.ocpMember ?? null, options.sonicContributor ?? null]
);
}
return existing.rows[0].id;
}
const slug = name.toLowerCase().replace(/[^a-z0-9]+/g, "-");
const result = await pool.query(
`INSERT INTO vendors (name, slug, type, website, is_whitebox_vendor, is_odm, ocp_member, sonic_contributor)
VALUES ($1, $2, 'manufacturer', $3, TRUE, $4, $5, $6)
RETURNING id`,
[
name, slug, website || null,
options?.isOdm ?? true,
options?.ocpMember ?? false,
options?.sonicContributor ?? false,
]
);
return result.rows[0].id;
}
export async function getVendorId(name: string): Promise<string | null> {
const result = await pool.query(`SELECT id FROM vendors WHERE name = $1`, [name]);
return result.rows[0]?.id || null;
}
export async function ensureVendor(
name: string,
type: string,
website?: string,
shopUrl?: string
): Promise<string> {
// Try to find existing vendor first
const existing = await pool.query(`SELECT id FROM vendors WHERE name ILIKE $1`, [name]);
if (existing.rows.length > 0) return existing.rows[0].id;
const slug = name.toLowerCase().replace(/[^a-z0-9]+/g, "-");
try {
const result = await pool.query(
`INSERT INTO vendors (name, slug, type, website, shop_url, is_competitor)
VALUES ($1, $2, $3, $4, $5, true)
RETURNING id`,
[name, slug, type, website || null, shopUrl || null]
);
return result.rows[0].id;
} catch (err: unknown) {
// Handle race condition — re-query if insert fails on unique constraint
const existing2 = await pool.query(`SELECT id FROM vendors WHERE name ILIKE $1 OR slug = $2`, [name, slug]);
if (existing2.rows.length > 0) return existing2.rows[0].id;
throw err;
}
}

View File

@ -0,0 +1,71 @@
import { createHash } from "crypto";
/**
* Generate SHA-256 content hash for change detection.
* Only hashes the fields that matter (price, stock, quantity).
*/
export function contentHash(data: Record<string, unknown>): string {
const normalized = JSON.stringify(data, Object.keys(data).sort());
return createHash("sha256").update(normalized).digest("hex").slice(0, 16);
}
/**
* Parse price string into number.
* Handles: "$12.50", "12,50 €", "US$12.50", "12.50 USD"
*/
export function parsePrice(raw: string): { price: number; currency: string } {
const cleaned = raw.replace(/[^\d.,]/g, "").replace(",", ".");
const price = parseFloat(cleaned);
const currency = raw.includes("€")
? "EUR"
: raw.includes("£")
? "GBP"
: raw.includes("¥")
? "CNY"
: "USD";
return { price: isNaN(price) ? 0 : price, currency };
}
/**
* Determine stock level from various text representations.
*/
export function parseStockLevel(
raw: string
): "in_stock" | "low_stock" | "out_of_stock" | "on_request" | "discontinued" {
const lower = raw.toLowerCase();
if (lower.includes("in stock") || lower.includes("auf lager") || lower.includes("available"))
return "in_stock";
if (lower.includes("low stock") || lower.includes("few left") || lower.includes("limited"))
return "low_stock";
if (
lower.includes("out of stock") ||
lower.includes("sold out") ||
lower.includes("nicht verfügbar") ||
lower.includes("unavailable")
)
return "out_of_stock";
if (lower.includes("discontinued") || lower.includes("eol") || lower.includes("end of life"))
return "discontinued";
return "on_request";
}
/**
* Extract numeric quantity from stock text.
* "23 in stock" 23, "500+ available" 500
*/
export function parseQuantity(raw: string): number | undefined {
const match = raw.match(/(\d+)\+?\s*(in stock|available|auf lager|stück|units|pcs)/i);
return match ? parseInt(match[1]) : undefined;
}
/**
* Parse lead time from text.
* "Ships in 3-5 days" 5, "2 weeks" 14
*/
export function parseLeadTime(raw: string): number | undefined {
const dayMatch = raw.match(/(\d+)\s*(business\s+)?days?/i);
if (dayMatch) return parseInt(dayMatch[1]);
const weekMatch = raw.match(/(\d+)\s*weeks?/i);
if (weekMatch) return parseInt(weekMatch[1]) * 7;
return undefined;
}

View File

@ -0,0 +1,154 @@
/**
* WS0: Image Downloader
*
* Downloads product images from various sources, resizes, and stores metadata.
* R2 upload is optional for now stores image URLs and marks has_image.
*/
import { Pool } from "pg";
import { createHash } from "crypto";
const pool = new Pool({
host: process.env.POSTGRES_HOST || "localhost",
port: parseInt(process.env.POSTGRES_PORT || "5433"),
database: process.env.POSTGRES_DB || "transceiver_db",
user: process.env.POSTGRES_USER || "tip",
password: process.env.POSTGRES_PASSWORD || "tip_dev_2026",
max: 3,
});
/**
* Update image URL for a transceiver and mark has_image = true
*/
export async function setTransceiverImage(
transceiverId: string,
imageUrl: string,
source?: string
): Promise<void> {
await pool.query(
`UPDATE transceivers SET image_url = $2, has_image = true, image_scraped_at = NOW()
WHERE id = $1 AND (image_url IS NULL OR image_url = '')`,
[transceiverId, imageUrl]
);
}
/**
* Update image URL for a switch
*/
export async function setSwitchImage(
switchId: string,
imageUrl: string
): Promise<void> {
await pool.query(
`UPDATE switches SET image_url = $2, has_image = true
WHERE id = $1 AND (image_url IS NULL OR image_url = '')`,
[switchId, imageUrl]
);
}
/**
* Get products without images for backfill
*/
export async function getProductsWithoutImages(limit = 100): Promise<Array<{
id: string;
slug: string;
form_factor: string;
speed_gbps: number;
reach_label: string;
vendor_name: string;
part_number: string;
}>> {
const result = await pool.query(
`SELECT t.id, t.slug, t.form_factor, t.speed_gbps, t.reach_label, t.part_number,
v.name AS vendor_name
FROM transceivers t
LEFT JOIN vendors v ON t.vendor_id = v.id
WHERE (t.has_image = false OR t.has_image IS NULL)
AND t.image_url IS NULL
ORDER BY t.speed_gbps DESC
LIMIT $1`,
[limit]
);
return result.rows;
}
/**
* Generate a search URL to find product images
*/
export function buildImageSearchUrls(product: {
form_factor: string;
speed_gbps: number;
reach_label: string;
part_number?: string;
vendor_name?: string;
}): string[] {
const urls: string[] = [];
const q = `${product.form_factor} ${product.speed_gbps}G ${product.reach_label} transceiver`;
// Flexoptix store
urls.push(`https://www.flexoptix.net/en/catalogsearch/result/?q=${encodeURIComponent(q)}`);
// FS.com
urls.push(`https://www.fs.com/search/${encodeURIComponent(q)}.html`);
// If we have a part number, try vendor-specific
if (product.part_number) {
urls.push(`https://www.fs.com/search/${encodeURIComponent(product.part_number)}.html`);
}
return urls;
}
/**
* Get image coverage statistics
*/
export async function getImageCoverageStats(): Promise<{
total: number;
with_image: number;
without_image: number;
coverage_pct: number;
}> {
const result = await pool.query(`
SELECT
COUNT(*) AS total,
COUNT(*) FILTER (WHERE has_image = true) AS with_image,
COUNT(*) FILTER (WHERE has_image = false OR has_image IS NULL) AS without_image
FROM transceivers
`);
const row = result.rows[0];
const total = parseInt(row.total);
const withImg = parseInt(row.with_image);
return {
total,
with_image: withImg,
without_image: parseInt(row.without_image),
coverage_pct: total > 0 ? Math.round((withImg / total) * 10000) / 100 : 0,
};
}
/**
* Get price coverage statistics
*/
export async function getPriceCoverageStats(): Promise<{
total: number;
with_recent_price: number;
without_recent_price: number;
coverage_pct: number;
}> {
const result = await pool.query(`
SELECT
COUNT(*) AS total,
COUNT(*) FILTER (WHERE EXISTS (
SELECT 1 FROM price_observations po WHERE po.transceiver_id = t.id AND po.time > NOW() - INTERVAL '7 days'
)) AS with_price
FROM transceivers t
`);
const row = result.rows[0];
const total = parseInt(row.total);
const withPrice = parseInt(row.with_price);
return {
total,
with_recent_price: withPrice,
without_recent_price: total - withPrice,
coverage_pct: total > 0 ? Math.round((withPrice / total) * 10000) / 100 : 0,
};
}

View File

@ -0,0 +1,18 @@
{
"compilerOptions": {
"target": "ES2022",
"module": "commonjs",
"lib": ["ES2022"],
"outDir": "./dist",
"rootDir": "./src",
"strict": true,
"esModuleInterop": true,
"skipLibCheck": true,
"forceConsistentCasingInFileNames": true,
"resolveJsonModule": true,
"declaration": true,
"sourceMap": true
},
"include": ["src/**/*"],
"exclude": ["node_modules", "dist"]
}

View File

@ -0,0 +1,31 @@
#!/bin/bash
# Run ON Erik: Fix enrichment SQL dedup + apply switches + restart
LOG="/tmp/deploy-all-fixes.log"
echo "$(date): Starting all fixes" > "$LOG"
# Step 1: Fix enrichment SQL (deduplicate columns)
echo "Step 1: Fixing enrichment SQL..." >> "$LOG"
python3 /tmp/fix-sql-dedup.py >> "$LOG" 2>&1
# Step 2: Re-apply enrichment
echo "Step 2: Applying enrichment..." >> "$LOG"
PGPASSWORD=tip_prod_2026 psql -h localhost -p 5433 -U tip -d transceiver_db -f /tmp/011-flexoptix-enrichment.sql >> "$LOG" 2>&1
# Step 3: Apply switches SQL
echo "Step 3: Applying switches..." >> "$LOG"
PGPASSWORD=tip_prod_2026 psql -h localhost -p 5433 -U tip -d transceiver_db -f /tmp/012-more-switches.sql >> "$LOG" 2>&1
# Step 4: Restart API
echo "Step 4: Restarting API..." >> "$LOG"
cd /opt/tip && pm2 restart tip-api >> "$LOG" 2>&1
# Step 5: Results
echo "" >> "$LOG"
echo "=== RESULTS ===" >> "$LOG"
PGPASSWORD=tip_prod_2026 psql -h localhost -p 5433 -U tip -d transceiver_db -t -A -c "SELECT 'images: ' || count(*) FROM transceivers WHERE image_url IS NOT NULL" >> "$LOG" 2>&1
PGPASSWORD=tip_prod_2026 psql -h localhost -p 5433 -U tip -d transceiver_db -t -A -c "SELECT 'connector: ' || count(*) FROM transceivers WHERE connector IS NOT NULL" >> "$LOG" 2>&1
PGPASSWORD=tip_prod_2026 psql -h localhost -p 5433 -U tip -d transceiver_db -t -A -c "SELECT 'notes: ' || count(*) FROM transceivers WHERE notes IS NOT NULL AND notes != ''" >> "$LOG" 2>&1
PGPASSWORD=tip_prod_2026 psql -h localhost -p 5433 -U tip -d transceiver_db -t -A -c "SELECT 'switches: ' || count(*) FROM switches" >> "$LOG" 2>&1
PGPASSWORD=tip_prod_2026 psql -h localhost -p 5433 -U tip -d transceiver_db -t -A -c "SELECT 'sw_desc: ' || count(*) FROM switches WHERE description IS NOT NULL" >> "$LOG" 2>&1
echo "$(date): ALL DONE" >> "$LOG"

170
scripts/enrich-on-erik.sh Normal file
View File

@ -0,0 +1,170 @@
#!/bin/bash
# Self-contained Flexoptix enrichment script to run ON Erik
# Does: DB query → scrape flexoptix.net → generate SQL → apply to DB
DB_PASS="tip_prod_2026"
DB_USER="tip"
DB_NAME="transceiver_db"
DB_PORT="5433"
OUTPUT="/tmp/011-flexoptix-enrichment.sql"
LOG="/tmp/enrich-flexoptix.log"
echo "$(date): Starting enrichment" > "$LOG"
# Step 1: Get Flexoptix product list from DB
echo "Step 1: Querying DB..." >> "$LOG"
PGPASSWORD=$DB_PASS psql -h localhost -p $DB_PORT -U $DB_USER -d $DB_NAME -t -A -F'|' -c \
"SELECT t.id, t.product_page_url, t.part_number, t.standard_name FROM transceivers t JOIN vendors v ON t.vendor_id = v.id WHERE v.name = 'FLEXOPTIX' AND t.product_page_url IS NOT NULL ORDER BY t.part_number" \
> /tmp/flexoptix-products.txt 2>> "$LOG"
TOTAL=$(wc -l < /tmp/flexoptix-products.txt | tr -d ' ')
echo " Found $TOTAL products" >> "$LOG"
if [ "$TOTAL" -lt 1 ]; then
echo "ERROR: No products found" >> "$LOG"
exit 1
fi
# Step 2: Start SQL file
cat > "$OUTPUT" << SQLEOF
-- 011: Flexoptix product enrichment
-- Generated: $(date '+%Y-%m-%d %H:%M')
-- Products: $TOTAL
BEGIN;
SQLEOF
COUNT=0
IMAGES=0
ENRICHED=0
# Step 3: Scrape each product page
while IFS='|' read -r ID URL PARTNUM STDNAME; do
[ -z "$URL" ] && continue
COUNT=$((COUNT + 1))
NAME="${STDNAME:-$PARTNUM}"
echo "[$COUNT/$TOTAL] $NAME" >> "$LOG"
# Fetch page
HTML=$(curl -s -L --max-time 15 -H "User-Agent: Mozilla/5.0 TIP-Bot/1.0" "$URL" 2>/dev/null)
if [ ${#HTML} -lt 500 ]; then
echo " SKIP (empty/small)" >> "$LOG"
continue
fi
SETS=""
# Extract image URL
IMG=$(echo "$HTML" | grep -oE 'https://[^"]+/cache/[^"]+_A_[^"]+\.jpg' | head -1)
if [ -n "$IMG" ]; then
IMG_ESC=$(echo "$IMG" | sed "s/'/''/g")
SETS="image_url = '$IMG_ESC'"
IMAGES=$((IMAGES + 1))
fi
# Extract specs using python3 if available
if command -v python3 > /dev/null 2>&1; then
SPEC_DATA=$(echo "$HTML" | python3 -c "
import sys, re
html = sys.stdin.read()
for m in re.finditer(r'<th[^>]*>(.*?)</th>\s*<td[^>]*>(.*?)</td>', html, re.S|re.I):
label = re.sub(r'<[^>]+>', '', m.group(1)).strip().upper()
value = re.sub(r'<[^>]+>', '', m.group(2)).strip()
if label and value and value.lower() not in ('n/a', '-', ''):
# Use tab separator to avoid issues with = in values
print(label + '\t' + value)
" 2>/dev/null)
else
SPEC_DATA=""
fi
NOTES=""
while IFS=$'\t' read -r KEY VAL; do
[ -z "$KEY" ] && continue
VAL_ESC=$(echo "$VAL" | sed "s/'/''/g")
case "$KEY" in
"POWER CONSUMPTION")
W=$(echo "$VAL" | grep -oE '[0-9]+\.?[0-9]*' | head -1)
[ -n "$W" ] && SETS="${SETS:+$SETS, }power_consumption_w = '$W'"
;;
"CONNECTOR / POLISH"|"CONNECTOR")
SETS="${SETS:+$SETS, }connector = '$VAL_ESC'"
;;
"MODULATION")
SETS="${SETS:+$SETS, }modulation = '$VAL_ESC'"
;;
"WAVELENGTH TX (TYPICAL)"|"WAVELENGTH")
SETS="${SETS:+$SETS, }wavelengths = '$VAL_ESC'"
;;
"DISTANCE")
SETS="${SETS:+$SETS, }reach_label = '$VAL_ESC'"
;;
"TEMPERATURE RANGE"|"OPERATING TEMPERATURE")
SETS="${SETS:+$SETS, }temp_range = '$VAL_ESC'"
;;
"LANE COUNT")
LC=$(echo "$VAL" | grep -oE '[0-9]+' | head -1)
[ -n "$LC" ] && SETS="${SETS:+$SETS, }lanes = '$LC'"
;;
"BANDWIDTH PER LANE"|"BANDWIDTH")
SETS="${SETS:+$SETS, }lane_rate = '$VAL_ESC'"
;;
"INBUILT FEC")
echo "$VAL" | grep -qiE '^(no|none)$' || SETS="${SETS:+$SETS, }fec_type = '$VAL_ESC'"
;;
"POWERBUDGET (DB)")
PB=$(echo "$VAL" | grep -oE '[0-9]+\.?[0-9]*' | head -1)
[ -n "$PB" ] && SETS="${SETS:+$SETS, }optical_budget_db = '$PB'"
;;
"TRANSMIT MIN/MAX PER LANE")
TX=$(echo "$VAL" | grep -oE '\-?[0-9]+\.?[0-9]*' | head -1)
[ -n "$TX" ] && SETS="${SETS:+$SETS, }tx_power_min_dbm = '$TX'"
;;
"RECEIVER MIN/MAX PER LANE")
RX=$(echo "$VAL" | grep -oE '\-?[0-9]+\.?[0-9]*' | head -1)
[ -n "$RX" ] && SETS="${SETS:+$SETS, }rx_sensitivity_dbm = '$RX'"
;;
"INTERFACE")
SETS="${SETS:+$SETS, }fiber_type = '$VAL_ESC'"
;;
"COMPLIANCE CODE")
SETS="${SETS:+$SETS, }ieee_reference = '$VAL_ESC'"
;;
"DIGITAL DIAGNOSTIC MONITORING (DDM)")
echo "$VAL" | grep -qi 'yes' && SETS="${SETS:+$SETS, }dom_support = true" || SETS="${SETS:+$SETS, }dom_support = false"
;;
*)
[ ${#VAL} -lt 200 ] && NOTES="${NOTES:+$NOTES; }$KEY: $VAL"
;;
esac
done <<< "$SPEC_DATA"
# Add notes
if [ -n "$NOTES" ]; then
NOTES_CUT="${NOTES:0:1000}"
NOTES_ESC=$(echo "$NOTES_CUT" | sed "s/'/''/g")
SETS="${SETS:+$SETS, }notes = '$NOTES_ESC'"
fi
if [ -n "$SETS" ]; then
echo "-- $NAME" >> "$OUTPUT"
echo "UPDATE transceivers SET $SETS WHERE id = '$ID';" >> "$OUTPUT"
echo "" >> "$OUTPUT"
ENRICHED=$((ENRICHED + 1))
echo " -> OK ($ENRICHED enriched, $IMAGES imgs)" >> "$LOG"
fi
sleep 0.3
done < /tmp/flexoptix-products.txt
echo "COMMIT;" >> "$OUTPUT"
echo "-- Summary: $ENRICHED enriched, $IMAGES images" >> "$OUTPUT"
echo "" >> "$LOG"
echo "Step 3 done: $ENRICHED/$TOTAL enriched, $IMAGES images" >> "$LOG"
# Step 4: Apply SQL
echo "Step 4: Applying SQL..." >> "$LOG"
PGPASSWORD=$DB_PASS psql -h localhost -p $DB_PORT -U $DB_USER -d $DB_NAME -f "$OUTPUT" >> "$LOG" 2>&1
echo "$(date): ALL DONE" >> "$LOG"

158
scripts/enrich-v2.sh Normal file
View File

@ -0,0 +1,158 @@
#!/bin/bash
# V2: Flexoptix enrichment with deduplication
# Run ON Erik
LOG="/tmp/enrich-v2.log"
SQL="/tmp/011-flexoptix-enrichment-v2.sql"
DB="PGPASSWORD=tip_prod_2026 psql -h localhost -p 5433 -U tip -d transceiver_db"
echo "$(date): Starting V2 enrichment" > "$LOG"
# Get products
eval $DB -t -A -F'|' -c \
"SELECT t.id, t.product_page_url, t.part_number, t.standard_name FROM transceivers t JOIN vendors v ON t.vendor_id = v.id WHERE v.name = 'FLEXOPTIX' AND t.product_page_url IS NOT NULL ORDER BY t.part_number" \
> /tmp/fo-products.txt 2>> "$LOG"
TOTAL=$(wc -l < /tmp/fo-products.txt | tr -d ' ')
echo "Found $TOTAL products" >> "$LOG"
# Header
cat > "$SQL" << EOF
-- Flexoptix enrichment V2 (deduplicated)
-- Generated: $(date '+%Y-%m-%d %H:%M')
-- Products: $TOTAL
EOF
COUNT=0
OK=0
while IFS='|' read -r ID URL PN SN; do
[ -z "$URL" ] && continue
COUNT=$((COUNT + 1))
NAME="${SN:-$PN}"
HTML=$(curl -s -L --max-time 15 -H "User-Agent: Mozilla/5.0 TIP-Bot/1.0" "$URL" 2>/dev/null)
[ ${#HTML} -lt 500 ] && { echo "[$COUNT] $NAME SKIP" >> "$LOG"; continue; }
# Use python3 to extract specs AND generate clean SQL (no duplicates)
UPDATE=$(python3 -c "
import re, sys
html = sys.stdin.read()
tid = '$ID'
# Extract image
img = None
for m in re.finditer(r'https://[^\"\s]+/cache/[^\"\s]+_A_[^\"\s]+\.jpg', html):
img = m.group(0)
break
if not img:
for m in re.finditer(r'https://[^\"\s]+/media/catalog/product/[^\"\s]+_A_[^\"\s]+\.jpg', html):
img = m.group(0)
break
# Extract specs from <th>...<td>
specs = {}
for m in re.finditer(r'<th[^>]*>(.*?)</th>\s*<td[^>]*>(.*?)</td>', html, re.S|re.I):
label = re.sub(r'<[^>]+>', '', m.group(1)).strip().upper()
value = re.sub(r'<[^>]+>', '', m.group(2)).strip()
if label and value and value.lower() not in ('n/a', '-', ''):
specs[label] = value
if not specs and not img:
sys.exit(0)
# Map to columns (first match wins per column)
cols = {}
MAPPING = [
('POWER CONSUMPTION', 'power_consumption_w', lambda v: re.search(r'[\d.]+', v).group() if re.search(r'[\d.]+', v) else None),
('CONNECTOR / POLISH', 'connector', lambda v: v),
('CONNECTOR', 'connector', lambda v: v),
('MODULATION', 'modulation', lambda v: v),
('WAVELENGTH TX (TYPICAL)', 'wavelengths', lambda v: v),
('WAVELENGTH', 'wavelengths', lambda v: v),
('DISTANCE', 'reach_label', lambda v: v),
('TEMPERATURE RANGE', 'temp_range', lambda v: 'COM' if 'ommercial' in v or '0°C' in v else ('IND' if 'ndustrial' in v or '-40' in v else ('EXT' if 'xtended' in v else 'COM'))),
('OPERATING TEMPERATURE', 'temp_range', lambda v: 'COM' if 'ommercial' in v or '0°C' in v else ('IND' if 'ndustrial' in v or '-40' in v else ('EXT' if 'xtended' in v else 'COM'))),
('LANE COUNT', 'lanes', lambda v: re.search(r'\d+', v).group() if re.search(r'\d+', v) else None),
('BANDWIDTH PER LANE', 'lane_rate', lambda v: v),
('BANDWIDTH', 'lane_rate', lambda v: v),
('INBUILT FEC', 'fec_type', lambda v: v if v.lower() not in ('no', 'none') else None),
('POWERBUDGET (DB)', 'optical_budget_db', lambda v: re.search(r'[\d.]+', v).group() if re.search(r'[\d.]+', v) else None),
('TRANSMIT MIN/MAX PER LANE', 'tx_power_min_dbm', lambda v: re.search(r'-?[\d.]+', v).group() if re.search(r'-?[\d.]+', v) else None),
('RECEIVER MIN/MAX PER LANE', 'rx_sensitivity_dbm', lambda v: re.search(r'-?[\d.]+', v).group() if re.search(r'-?[\d.]+', v) else None),
('INTERFACE', 'fiber_type', lambda v: v),
('COMPLIANCE CODE', 'ieee_reference', lambda v: v),
('DIGITAL DIAGNOSTIC MONITORING (DDM)', 'dom_support', lambda v: 'true' if 'yes' in v.lower() else 'false'),
]
mapped_labels = set()
for label, col, transform in MAPPING:
if label in specs and col not in cols:
try:
val = transform(specs[label])
if val is not None:
cols[col] = val
mapped_labels.add(label)
except:
pass
# Unmapped specs -> notes
extra = []
for k, v in specs.items():
if k not in mapped_labels and len(v) < 200:
extra.append(f'{k}: {v}')
if extra:
cols['notes'] = '; '.join(extra)[:1000]
if img:
cols['image_url'] = img
if not cols:
sys.exit(0)
# Build SQL
def esc(v):
return v.replace(chr(39), chr(39)+chr(39)).replace(chr(92), chr(92)+chr(92))
sets = []
for col, val in cols.items():
if col == 'dom_support':
sets.append(f'{col} = {val}')
else:
sets.append(f\"{col} = '{esc(str(val))}'\")
print(f'-- {\"$NAME\"}')
print(f\"UPDATE transceivers SET {', '.join(sets)} WHERE id = '{tid}';\")
" <<< "$HTML" 2>/dev/null)
if [ -n "$UPDATE" ]; then
echo "$UPDATE" >> "$SQL"
echo "" >> "$SQL"
OK=$((OK + 1))
echo "[$COUNT] $NAME -> OK" >> "$LOG"
else
echo "[$COUNT] $NAME -> no data" >> "$LOG"
fi
sleep 0.3
done < /tmp/fo-products.txt
echo "-- Summary: $OK/$TOTAL enriched" >> "$SQL"
echo "" >> "$LOG"
echo "Enrichment SQL generated: $OK/$TOTAL" >> "$LOG"
echo "Applying SQL..." >> "$LOG"
eval $DB -f "$SQL" >> "$LOG" 2>&1
echo "" >> "$LOG"
echo "=== FINAL COUNTS ===" >> "$LOG"
eval $DB -t -A -c "SELECT 'images: ' || count(*) FROM transceivers WHERE image_url IS NOT NULL" >> "$LOG"
eval $DB -t -A -c "SELECT 'connector: ' || count(*) FROM transceivers WHERE connector IS NOT NULL" >> "$LOG"
eval $DB -t -A -c "SELECT 'notes: ' || count(*) FROM transceivers WHERE notes IS NOT NULL AND notes != ''" >> "$LOG"
eval $DB -t -A -c "SELECT 'modulation: ' || count(*) FROM transceivers WHERE modulation IS NOT NULL" >> "$LOG"
eval $DB -t -A -c "SELECT 'power_w: ' || count(*) FROM transceivers WHERE power_consumption_w IS NOT NULL" >> "$LOG"
echo "$(date): V2 DONE" >> "$LOG"

125
scripts/enrich-v3.sh Normal file
View File

@ -0,0 +1,125 @@
#!/bin/bash
# V3: Flexoptix enrichment - simple, direct, works
LOG="/tmp/enrich-v3.log"
SQL="/tmp/enrichment-v3.sql"
echo "$(date): V3 start" > "$LOG"
# Direct psql (no eval)
export PGPASSWORD="tip_prod_2026"
psql -h localhost -p 5433 -U tip -d transceiver_db -t -A -F'|' -c \
"SELECT t.id, t.product_page_url, t.part_number, t.standard_name FROM transceivers t JOIN vendors v ON t.vendor_id = v.id WHERE v.name = 'FLEXOPTIX' AND t.product_page_url IS NOT NULL" \
> /tmp/fo-list.txt 2>> "$LOG"
TOTAL=$(wc -l < /tmp/fo-list.txt | tr -d ' ')
echo "Products: $TOTAL" >> "$LOG"
echo "-- V3 enrichment $(date '+%Y-%m-%d %H:%M')" > "$SQL"
echo "" >> "$SQL"
OK=0
while IFS='|' read -r ID URL PN SN; do
[ -z "$URL" ] && continue
OK_THIS=0
NAME="${SN:-$PN}"
HTML=$(curl -s -L --max-time 15 -H "User-Agent: Mozilla/5.0 TIP-Bot/1.0" "$URL" 2>/dev/null)
[ ${#HTML} -lt 500 ] && continue
SQLLINE=$(echo "$HTML" | python3 << 'PYEOF'
import re, sys
html = sys.stdin.read()
img = None
for m in re.finditer(r'https://[^"\s]+/cache/[^"\s]+_A_[^"\s]+\.jpg', html):
img = m.group(0); break
specs = {}
for m in re.finditer(r'<th[^>]*>(.*?)</th>\s*<td[^>]*>(.*?)</td>', html, re.S|re.I):
label = re.sub(r'<[^>]+>', '', m.group(1)).strip().upper()
value = re.sub(r'<[^>]+>', '', m.group(2)).strip()
if label and value and value.lower() not in ('n/a', '-', ''):
specs[label] = value
if not specs and not img: sys.exit(0)
cols = {}
MAP = [
('POWER CONSUMPTION', 'power_consumption_w', lambda v: re.search(r'[\d.]+', v).group() if re.search(r'[\d.]+', v) else None),
('CONNECTOR / POLISH', 'connector', None),
('CONNECTOR', 'connector', None),
('MODULATION', 'modulation', None),
('WAVELENGTH TX (TYPICAL)', 'wavelengths', None),
('WAVELENGTH', 'wavelengths', None),
('DISTANCE', 'reach_label', None),
('TEMPERATURE RANGE', 'temp_range', lambda v: 'COM' if any(x in v for x in ['ommercial','0°C to 70']) else ('IND' if any(x in v for x in ['ndustrial','-40']) else ('EXT' if 'xtended' in v else 'COM'))),
('OPERATING TEMPERATURE', 'temp_range', lambda v: 'COM' if any(x in v for x in ['ommercial','0°C to 70']) else ('IND' if any(x in v for x in ['ndustrial','-40']) else ('EXT' if 'xtended' in v else 'COM'))),
('LANE COUNT', 'lanes', lambda v: re.search(r'\d+', v).group() if re.search(r'\d+', v) else None),
('BANDWIDTH PER LANE', 'lane_rate', None),
('BANDWIDTH', 'lane_rate', None),
('INBUILT FEC', 'fec_type', lambda v: v if v.lower() not in ('no','none') else None),
('POWERBUDGET (DB)', 'optical_budget_db', lambda v: re.search(r'[\d.]+', v).group() if re.search(r'[\d.]+', v) else None),
('TRANSMIT MIN/MAX PER LANE', 'tx_power_min_dbm', lambda v: re.search(r'-?[\d.]+', v).group() if re.search(r'-?[\d.]+', v) else None),
('RECEIVER MIN/MAX PER LANE', 'rx_sensitivity_dbm', lambda v: re.search(r'-?[\d.]+', v).group() if re.search(r'-?[\d.]+', v) else None),
('INTERFACE', 'fiber_type', None),
('COMPLIANCE CODE', 'ieee_reference', None),
('DIGITAL DIAGNOSTIC MONITORING (DDM)', 'dom_support', lambda v: 'true' if 'yes' in v.lower() else 'false'),
]
mapped = set()
for label, col, fn in MAP:
if label in specs and col not in cols:
try:
val = fn(specs[label]) if fn else specs[label]
if val is not None:
cols[col] = val
mapped.add(label)
except: pass
extra = [f'{k}: {v}' for k,v in specs.items() if k not in mapped and len(v) < 200]
if extra: cols['notes'] = '; '.join(extra)[:1000]
if img: cols['image_url'] = img
if not cols: sys.exit(0)
def e(s): return str(s).replace("'","''")
parts = []
for c, v in cols.items():
if c == 'dom_support': parts.append(f'{c} = {v}')
else: parts.append(f"{c} = '{e(v)}'")
print(', '.join(parts))
PYEOF
)
if [ -n "$SQLLINE" ]; then
echo "-- $NAME" >> "$SQL"
echo "UPDATE transceivers SET $SQLLINE WHERE id = '$ID';" >> "$SQL"
echo "" >> "$SQL"
OK=$((OK + 1))
fi
sleep 0.3
done < /tmp/fo-list.txt
echo "-- Total: $OK enriched" >> "$SQL"
echo "" >> "$LOG"
echo "Generated: $OK/$TOTAL" >> "$LOG"
# Apply
echo "Applying SQL..." >> "$LOG"
psql -h localhost -p 5433 -U tip -d transceiver_db -f "$SQL" >> "$LOG" 2>&1
# Restart API
cd /opt/tip && pm2 restart tip-api >> "$LOG" 2>&1
# Counts
echo "=== COUNTS ===" >> "$LOG"
psql -h localhost -p 5433 -U tip -d transceiver_db -t -A -c "SELECT 'img=' || count(*) FROM transceivers WHERE image_url IS NOT NULL" >> "$LOG"
psql -h localhost -p 5433 -U tip -d transceiver_db -t -A -c "SELECT 'conn=' || count(*) FROM transceivers WHERE connector IS NOT NULL" >> "$LOG"
psql -h localhost -p 5433 -U tip -d transceiver_db -t -A -c "SELECT 'notes=' || count(*) FROM transceivers WHERE notes IS NOT NULL AND notes != ''" >> "$LOG"
psql -h localhost -p 5433 -U tip -d transceiver_db -t -A -c "SELECT 'mod=' || count(*) FROM transceivers WHERE modulation IS NOT NULL" >> "$LOG"
psql -h localhost -p 5433 -U tip -d transceiver_db -t -A -c "SELECT 'power=' || count(*) FROM transceivers WHERE power_consumption_w IS NOT NULL" >> "$LOG"
psql -h localhost -p 5433 -U tip -d transceiver_db -t -A -c "SELECT 'lane_rate=' || count(*) FROM transceivers WHERE lane_rate IS NOT NULL" >> "$LOG"
echo "$(date): V3 DONE" >> "$LOG"

Some files were not shown because too many files have changed in this diff Show More