rene/transceiver-db
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: massive scraper expansion + hype cycle engine + lifecycle prediction

Browse Source 
New scrapers:
- GBICS.com (BigCommerce, GBP prices, 10 categories, 78 products)
- Juniper HCT (Next.js SSR parser, 475 transceivers with specs/EOL)
- SFPcables.com (Magento store, 16 categories, 78 products)
- Fluxlight (BigCommerce, 6 pages, 118 products)
- Champion ONE (compatible vendor scraper)

Scraper fixes:
- 10Gtek: rewritten to parse HTML spec tables (152 products)
- Flexoptix: fix price extraction from Magento Hyva HTML
- Register all scrapers in CLI (--gbics, --juniper, --sfpcables, etc.)

Hype Cycle Engine enhancements:
- Data-driven enrichment from scraped vendor/price data
- Revenue lifecycle prediction (peak year, decline, revenue index)
- Regional adoption model (NA, China, APAC, Europe, RoW with lag coefficients)
- New API endpoints: /enriched, /lifecycle, /regional/:tech

DB growth: 89 → 1,168 transceivers, 0 → 416 prices, 6 vendors
Qdrant: 1,162 products embedded with nomic-embed-text

Research: Norton-Bass model, standards-to-market timelines, hype signals
This commit is contained in:
Rene Fichtmueller 2026-03-28 02:30:19 +13:00
parent 204e99763c
commit 70447def02
21 changed files with 5839 additions and 467 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

738
RESEARCH-demo-to-market-timeline-model.md Normal file
View File

@ -0,0 +1,738 @@
# Optical Networking Equipment: Demo-to-Market Predictive Timeline Model
> Research compiled 2026-03-28 for the Transceiver Intelligence Platform (TIP)
> Data from OFC/ECOC proceedings, IEEE standards, MSA publications, vendor press releases, Cignal AI, LightCounting, Dell'Oro Group
---
## Table of Contents
1. [Historical OFC/ECOC Demo-to-Market Timelines](#1-historical-timelines)
2. [Switch/Router ASIC Generation Timelines](#2-asic-timelines)
3. [The Lag Formula](#3-the-lag-formula)
4. [Demand Cascade Model](#4-demand-cascade-model)
5. [Export Control Impact](#5-export-control-impact)
6. [Predictive Timeline Calculator](#6-predictive-calculator)
---
## 1. Historical OFC/ECOC Demo-to-Market Timelines {#1-historical-timelines}
### 1.1 10G SFP+
| Milestone | Date | Source |
|-----------|------|--------|
| IEEE 802.3ae study group formed | Nov 1999 | IEEE archives |
| IEEE 802.3ae ratified (10GbE standard) | Jun 2002 | [IEEE 802.3ae](https://resources.l-p.com/knowledge-center/what-is-ieee-802-3ae-10-gigabit-ethernet) |
| First 10G modules (XENPAK form factor) | 2002-2003 | First MSA for 10GE; largest form factor |
| XFP MSA published | 2003-2004 | Intermediate form factor between XENPAK and SFP+ |
| SFP+ MSA (SFF-8431) published | ~2006 | SFP+ became smallest, lowest-power 10G form factor |
| First SFP+ volume shipments | 2007-2008 | Industry adoption ramped with switch platforms |
| 10GBASE-T (802.3an) ratified | Jun 2006 | Extended 10G to copper |
| Mainstream SFP+ adoption | 2009-2010 | De facto standard for ToR/access |
**Total cycle: ~8 years** from IEEE standard (2002) to mainstream (2010). However, the SFP+ form factor itself took ~4 years from MSA (2006) to mainstream (2010).
### 1.2 40G QSFP+
| Milestone | Date | Source |
|-----------|------|--------|
| IEEE 802.3ba study group | Nov 2007 | IEEE archives |
| IEEE 802.3ba ratified (40G/100G Ethernet) | Jun 2010 | IEEE 802.3ba standard |
| First 40G QSFP+ commercial modules | 2010-2011 | QSFP+ MSA based on 4x10G lanes |
| Volume production begins | 2012-2013 | Kaiam, Finisar shipping high volume |
| Mainstream data center adoption | 2013-2015 | Standard for aggregation/ToR switches |
**Total cycle: ~5 years** from standard (2010) to mainstream (2015). Form-factor-to-volume: ~2 years.
### 1.3 100G QSFP28
| Milestone | Date | Source |
|-----------|------|--------|
| IEEE 802.3bm task force (100G over MMF/short-reach) | 2013 | Defined 4x25G lane architecture |
| QSFP28 MSA published | 2013-2014 | Based on QSFP+ with 4x25G lanes |
| First OFC demos (CWDM4, PSM4) | OFC 2015 | [Kaiam CWDM4 100G QSFP28 demo](https://www.businesswire.com/news/home/20150319005175/en/Kaiam-Introduces-CWDM4-100G-QSFP28-Transceiver-400G) |
| ColorChip adds PSM4 to QSFP28 portfolio | OFC 2016 | [ColorChip PSM4 announcement](https://www.globenewswire.com/news-release/2016/03/18/940853/0/en/) |
| InnoLight volume shipments (17 QSFP28 SKUs) | Mar 2017 | [InnoLight OFC 2017](https://www.innolight.com/en/news/newsinfo/13.html) |
| Oclaro 40km interop demo (QSFP28 ER4-Lite) | Mar 2017 | [Oclaro OFC 2017](https://www.prnewswire.com/news-releases/oclaro-showcases-industrys-first-live-40km-interoperability-demo-between-100g-extended-reach-qsfp28-and-cfp2-at-ofc-2017-300426690.html) |
| Market maturity (cost-effective vs 10G/40G) | 2017-2018 | More $/Gbit efficient than 10G SFP+ and 40G QSFP+ |
**Total cycle: ~4 years** from MSA (2014) to mainstream (2018). Demo-to-volume: ~2 years (OFC 2015 to Mar 2017).
### 1.4 100G Coherent (CFP to QSFP28-DCO)
| Milestone | Date | Source |
|-----------|------|--------|
| CFP MSA (first 100G pluggable form factor) | 2009-2010 | [CFP Wikipedia](https://en.wikipedia.org/wiki/C_Form-factor_Pluggable); 10x10G lanes |
| CFP2 MSA (half the size of CFP) | 2012 | [ProOptix history](https://www.prooptix.com/news/transceiver-form-factors/) |
| CFP2-ACO (OIF Interoperability Agreement) | 2016 | DSP on host line card; analog signal to module |
| CFP2-DCO (DSP integrated in module) | 2017-2018 | Software-configurable 100G/200G; [Acacia CFP2-DCO](https://acacia-inc.com/product/cfp2/) |
| Adtran first 100ZR QSFP28 DCO | 2022 | First coherent 100G in QSFP28 |
| Coherent QSFP28-DCO with Steelerton DSP | 2023 | [Coherent 100G QSFP28-DCO](https://www.coherent.com/news/press-releases/100g-qsfp28-dco-transceiver); <5W power |
| Coherent dual-laser QSFP28-DCO GA | Mar 2026 | [Coherent GA announcement](https://www.globenewswire.com/news-release/2026/03/06/3251306/11543/en/) |
**Total coherent miniaturization cycle: ~13 years** from CFP (2010) to QSFP28-DCO (2023). Each form factor shrink: ~3-4 years.
### 1.5 400G QSFP-DD / OSFP
| Milestone | Date | Source |
|-----------|------|--------|
| QSFP-DD MSA Rev 0.2 | May 2016 | [QSFP-DD spec](http://www.qsfp-dd.com/wp-content/uploads/2016/05/QSFP-DDrev0-2-3-29-16.pdf) |
| QSFP-DD MSA Rev 2.0 (form factor spec) | Mar 2017 | [QSFP-DD MSA announcement](http://www.qsfp-dd.com/qsfp-dd-msa-group-announces-form-factor-specification/) |
| InnoLight introduces 400G OSFP at OFC 2017 | Mar 2017 | [InnoLight OFC 2017](https://www.prnewswire.com/news-releases/innolight-technology-announced-volume-shipments-of-17-100g-qsfp28-products-and-the-introduction-of-400g-osfp-at-ofc-2017-300421866.html) |
| Oclaro 400G CFP8 PAM4 demo at OFC 2017 | Mar 2017 | [Oclaro CFP8](https://www.prnewswire.com/news-releases/oclaro-samples-400g-cfp8-pam4-enabled-transceiver-showcases-live-demo-at-ofc-2017-300425943.html) |
| Finisar 400G transceiver demos at OFC 2018 | Mar 2018 | [Finisar OFC 2018](https://picmagazine.net/article/103776/Finisar_Demos_New_400G_Transceivers_At_OFC_2018) |
| IEEE 802.3bs ratified (400G Ethernet) | Dec 2017 | 400GBASE standard |
| QSFP-DD Hardware Rev 5.0 | Jul 2019 | [QSFP-DD Rev 5.0](http://www.qsfp-dd.com/wp-content/uploads/2019/07/QSFP-DD-Hardware-rev5p0.pdf) |
| First commercial 400G QSFP-DD/OSFP modules | 2019-2020 | Broadcom TH3 switches enabled demand |
| Volume production | 2020-2021 | Driven by hyperscaler leaf/spine upgrades |
| Mainstream adoption | 2021-2022 | De facto DC interconnect standard |
**Total cycle: ~5 years** from first demos (OFC 2017) to mainstream (2022). MSA-to-volume: ~3 years.
### 1.6 400G ZR Coherent
| Milestone | Date | Source |
|-----------|------|--------|
| OIF 400ZR project initiated | ~2016-2017 | OIF response to hyperscaler DCI demands |
| OIF 400ZR IA published | Mar 2020 | [OIF 400ZR spec](https://convergedigest.com/oif-publishes-400zr-implementation/) |
| Acacia/Inphi sampling 400ZR QSFP-DD | H2 2020 | [Inphi COLORZ II](https://convergedigest.com/inphi-ramps-shipments-of-400zr-and-zr/) |
| Fujitsu sample shipments begin | Oct 2020 | [Fujitsu 400G ZR launch](https://opticalconnectionsnews.com/2020/10/fujitsu-launches-400g-zr-transceivers/) |
| Inphi commercial availability & ramp | 2021 | [Inphi ramp announcement](https://convergedigest.com/inphi-ramps-shipments-of-400zr-and-zr/) |
| Molex volume production | Early 2022 | [Molex 400G ZR volume](https://www.molex.com/en-us/news/molex-ramps-production-of-400g-zr-qsfp-dd-coherent-optical) |
| Broad volume deployment | 2022-2023 | >100% CAGR in ZR/ZR+ per Cignal AI |
**Total cycle: ~6 years** from OIF project start (~2017) to volume (2022). Spec-to-volume: ~2 years (Mar 2020 to early 2022).
### 1.7 800G DR8
| Milestone | Date | Source |
|-----------|------|--------|
| Intel first 800G DR8 OSFP sample | OFC 2021 | [Gazettabyte Intel 800G DR8](https://www.gazettabyte.com/home/2021/6/29/intel-details-its-800-gigabit-dr8-optical-module.html) |
| IEEE 802.3ck ratified (100G/lane electrical) | 2022 | Enabled 8x100G = 800G |
| Initial shipments (SR8 for AI) | 2022 | Few thousand units |
| LESSENGERS 800G SR8 volume production | Q4 2023 | [LESSENGERS announcement](https://www.semiconductor-today.com/news_items/2023/sep/lessengers-280923.shtml) |
| Shipments exceed 1M units | 2023 | Dominated by SR8 for AI clusters |
| Hyper Photonix 800G DR8 GA | May 2024 | [Hyper Photonix GA](https://www.businesswire.com/news/home/20240517136062/en/) |
| Forecast: 8M 800GbE modules shipped | 2024 | Cignal AI OFC 2024 preview |
| 800G mainstream / displacing 400G | 2025 | [Cignal AI 800GbE growth](https://cignal.ai/2025/05/800gbe-optics-shipments-to-grow-60-in-2025/) |
**Total cycle: ~4 years** from first sample (OFC 2021) to mainstream (2025). Demo-to-volume: ~2.5 years. This is faster than previous generations due to AI demand pull.
### 1.8 800G ZR/ZR+ Coherent
| Milestone | Date | Source |
|-----------|------|--------|
| OIF 800G Coherent project initiated | Dec 2020 | [OIF 800G Coherent](https://www.oiforum.com/technical-work/hot-topics/800g-coherent/) |
| Coherent first 800G ZR/ZR+ QSFP-DD unveiled | Dec 2023 | [Coherent 800G ZR announcement](https://www.semiconductor-today.com/news_items/2023/dec/coherent-081223.shtml) |
| OIF first public 800ZR multivendor interop | OFC 2024 | OIF plugfest |
| Alpha samples available | Q1 2024 | Coherent Corp. |
| OIF 800ZR Implementation Agreement published | Oct 2024 | OIF 800ZR IA |
| Coherent 800G ZR/ZR+ QSFP-DD GA | Mar 2025 | [Coherent GA](https://www.coherent.com/news/press-releases/general-availability-of-800g-zr-zrplus-in-qsfp-dd-form-factor) |
| L-band 800G ZR/ZR+ QSFP-DD | Sep 2024 | [Coherent L-band](https://www.coherent.com/news/press-releases/800g-l-band-qsfp-dd-telecom-transceiver) |
| Volume ramp forecast: >200K units, >$1B revenue | 2026 | [Cignal AI forecast](https://cignal.ai/2025/07/800g-coherent-pluggable-shipments-to-exceed-1b-revenue-in-2026/) |
**Total cycle: ~6 years** from OIF project (Dec 2020) to volume (2026). Spec-to-GA: ~5 months (Oct 2024 to Mar 2025). First demo-to-volume: ~3 years (Dec 2023 to 2026).
### 1.9 1.6T Transceivers
| Milestone | Date | Source |
|-----------|------|--------|
| Eoptolink 1.6T module demo (4xFR2, OSFP-XD) | OFC 2023 | First industry 1.6T demo |
| InnoLight 1.6T OSFP-XD demo | OFC 2024 | Live demonstration |
| First EML-based 1.6T samples ship | Q4 2024 - Q1 2025 | Conventional technology |
| IEEE 802.3dj (800G/1.6T standard, 224G/lane) | Expected mid-2026 | Under development |
| OFC 2025: Multiple live 1.6T demos | Mar 2025 | [Eoptolink Gen2 1.6T](https://www.eoptolink.com/news/361-eoptolink-launches-its-gen2-1-6t-osfp-and-osfp-rhs-transceiver-family-at-ofc-2025), [Jabil 1.6T](https://investors.jabil.com/news/news-details/2025/Jabil-Launches-1-6T-Pluggable-Transceiver/), [ATOP 1.6T demo](https://www.atoptechnology.com/ofc-2025-live-demo-atops-1-6t-osfp224-dr8-siph-module-in-action-for-next-gen-ai/) |
| SiPh-based 1.6T modules available | H2 2025 | Post mass-production readiness |
| Interop plugfest (Keysight Santa Clara) | Dec 2025 | 224G SerDes validation |
| AOI first volume order ($200M+ from hyperscaler) | Mar 2026 | [AOI volume order](https://www.globenewswire.com/news-release/2026/03/09/3251675/9986/en/) |
| Volume ramp forecast | 2026 | Dell'Oro, Cignal AI projections |
| Predicted mainstream | 2027 | >10% of addressable ports |
**Total cycle (projected): ~4 years** from first demo (OFC 2023) to mainstream (2027). Demo-to-volume: ~3 years. Accelerated by AI demand.
### 1.10 CPO (Co-Packaged Optics)
| Milestone | Date | Source |
|-----------|------|--------|
| Broadcom Tomahawk 4-Humboldt (1st gen CPO) | 2021 | First CPO chipset |
| Broadcom Tomahawk 5-Bailly (2nd gen, first volume CPO) | 2023 | Shipped to select hyperscalers |
| Broadcom 3rd gen CPO (200G/lane) | May 2025 | [Broadcom CPO glimpse](https://news.broadcom.com/) |
| Meta: 1M link-hours without link flap in lab | Oct 2025 | Broadcom announcement |
| NVIDIA CPO switches (Quantum-X: H2 2025, Spectrum-X: H2 2026) | GTC 2025 | [NVIDIA CPO plans](https://www.techradar.com/pro/nvidia-is-planning-post-copper-1-6tbps-network-tech) |
| Small initial deployments | 2026 | [Cignal AI CPO report](https://cignal.ai/2025/02/co-packaged-optics-inevitable-but-not-imminent/) |
| Volume manufacturing capability | 2027 | ASE/industry consensus |
| Widespread scale-out adoption | 2028-2029+ | [EDN CPO status 2026](https://www.edn.com/where-co-packaged-optics-cpo-technology-stands-in-2026/) |
**Total cycle: ~7+ years** from first demo (2021) to predicted widespread adoption (2028+). This is longer because CPO requires fundamental changes to packaging, connectors, and supply chain.
### 1.11 LPO (Linear Pluggable Optics)
| Milestone | Date | Source |
|-----------|------|--------|
| LPO concept development | 2022-2023 | Industry discussions on eliminating in-module DSP |
| Eoptolink 200G/lane LPO demo, 100G/lane 800G LPO mass production | OFC 2024 | [Eoptolink LPO](https://www.lightwaveonline.com/home/article/14310702/eoptolink-showcases-200g-linear-drive-pluggable-optics-at-ofc-2024) |
| LPO MSA spec (100G/lane) released | Mar 25, 2025 | [LPO MSA release](https://www.globenewswire.com/news-release/2025/03/25/3048840/0/en/) |
| LPO MSA first plugfest (interop validation) | Feb 2025 | Pre-OFC 2025 |
| FLEXOPTIX LPO products (400G/800G QSFP/OSFP) | 2025 | [FLEXOPTIX LPO](https://www.flexoptix.net/en/blog/blog/introducing-linear-pluggable-optics) |
| ECOC 2025: 800G LPO interop confirmed | Oct 2025 | [Ethernet Alliance ECOC 2025](https://ethernetalliance.org/blog/2025/10/27/ecoc-2025-interoperability-at-800g-is-given-advancing-toward-1-6t/) |
| Market share outlook | 2025-2026 | Small % of 800G market (per Cignal AI); larger potential at 1.6T |
**Note:** LPO is not a new speed generation but a new architecture. It may capture significant share at 1.6T where power savings (50% vs DSP) become critical.
---
## 2. Switch/Router ASIC Generation Timelines {#2-asic-timelines}
### 2.1 Broadcom Tomahawk (Data Center Switching)
| ASIC | Bandwidth | Process | Announced | First Switch Shipments | Source |
|------|-----------|---------|-----------|----------------------|--------|
| Tomahawk 1 | 3.2 Tbps | 28nm | Sep 2014 | Spring 2015 (~6 mo) | [Broadcom TH1](https://www.broadcom.com/products/ethernet-connectivity/switching/strataxgs/bcm56960-series) |
| Tomahawk 2 | 6.4 Tbps | 16nm | Oct 2016 | ~Fall 2017 (~12 mo) | [NextPlatform TH2](https://www.nextplatform.com/2016/10/31/broadcom-strikes-100g-ethernet-harder-tomahawk-ii/) |
| Tomahawk 3 | 12.8 Tbps | 16nm | Dec 2017 | Dec 2017 (same!) | [Broadcom TH3 press](https://www.globenewswire.com/news-release/2017/12/19/1266218/0/en/) |
| Tomahawk 4 | 25.6 Tbps | 7nm | Dec 2019 | 2020-2021 (~12-18 mo) | [NextPlatform TH4](https://www.nextplatform.com/2019/12/12/broadcom-launches-another-tomahawk-into-the-datacenter/) |
| Tomahawk 5 | 51.2 Tbps | 5nm | ~Aug 2022 | Late 2022/2023 (~6 mo) | [Broadcom TH5](https://investors.broadcom.com/news-releases/news-release-details/broadcom-ships-tomahawk-5-industrys-highest-bandwidth-switch) |
| Tomahawk Ultra | 51.2 Tbps | 4nm | 2024 | 2024 | [Broadcom TH-Ultra](https://investors.broadcom.com/news-releases/news-release-details/broadcom-ships-tomahawk-ultra-reimagining-ethernet-switch-hpc) |
| Tomahawk 6 | 102.4 Tbps | 3nm | Jun 2025 | Mar 2026 (~9 mo) | [Broadcom TH6 volume](https://www.broadcom.com/company/news/product-releases/63146) |
| Tomahawk 6 CPO (Davisson) | 102.4 Tbps | 3nm | Oct 2025 | Shipping Oct 2025 | [Broadcom Davisson](https://investors.broadcom.com/news-releases/news-release-details/broadcom-announces-tomahawkr-6-davisson-industrys-first-1024) |
**Cadence:** Bandwidth doubles approximately every 2 years. ASIC announcement to first switch: 6-18 months.
### 2.2 Broadcom Jericho (Routing / AI Fabric)
| ASIC | Bandwidth | Process | Announced | Platform Availability | Source |
|------|-----------|---------|-----------|----------------------|--------|
| Jericho2 | 9.6 Tbps | 16nm | 2018 | 2019 | [Broadcom Jericho](https://www.techinsights.com/blog/broadcom-retargets-jericho-ai-clusters) |
| Jericho2c | 4.8 Tbps | 16nm | 2019 | 2020 | Service provider market |
| Jericho2c+ | 14.4 Tbps | 7nm | 2020 (sampling) | 2021 | [Gazettabyte J2c+](https://gazettabyte.squarespace.com/home/2020/11/17/broadcoms-144-terabit-jericho2c-router-chip.html) |
| Jericho3-AI (BCM88890) | 28.8 Tbps | 5nm | Apr 2023 | Oct 2024 (first white boxes) | [Broadcom J3-AI](https://www.broadcom.com/company/news/product-releases/61156), [DriveNets/Accton](https://www.prnewswire.com/news-releases/drivenets-and-accton-technology-launch-the-highest-performance-ethernet-based-ai-networking-solution-302273214.html) |
| Jericho4 | Multi-Tbps HyperPorts | 3nm | Aug 2025 (shipping) | 2025-2026 | [Broadcom J4](https://investors.broadcom.com/news-releases/news-release-details/broadcom-ships-jericho4-enabling-distributed-ai-computing-across) |
**Cadence:** ~18-24 months between generations. ASIC-to-platform: 12-18 months.
### 2.3 NVIDIA/Mellanox Spectrum (Ethernet Switching)
| ASIC | Bandwidth | Process | Announced | Shipped | Source |
|------|-----------|---------|-----------|---------|--------|
| Spectrum | 6.4 Tbps | - | ~2016 | 2016-2017 | SN2000 series |
| Spectrum-2 | 12.8 Tbps | - | ~2018 | 2019 | SN3000 series; 200G ports |
| Spectrum-3 | 12.8 Tbps | 16nm | Mar 2020 | 2021 | [NVIDIA Spectrum-3](https://network.nvidia.com/files/doc-2020/pb-spectrum-3.pdf); 400G support |
| Spectrum-4 | 51.2 Tbps | TSMC 4N | GTC 2022 | 2023 | [NextPlatform Spectrum-4](https://www.nextplatform.com/2022/04/01/spectrum-4-ethernet-leaps-to-800-gb-sec-with-nvidia-circuits/); 800G ports |
| Spectrum-X (CPO, SN6000) | 102.4-409.6 Tbps | TBD | GTC 2025 | 2026 (SN6810/SN6800) | [NVIDIA GTC 2025](https://www.techradar.com/pro/nvidia-is-planning-post-copper-1-6tbps-network-tech) |
### 2.4 NVIDIA/Mellanox ConnectX (Network Adapters)
| NIC | Max Speed | Announced | First Shipments | Source |
|-----|-----------|-----------|-----------------|--------|
| ConnectX-5 | 100 Gb/s | Jun 2016 | Oct 2016 | [Mellanox CX-5](https://www.hpcwire.com/2016/06/16/mellanox-advances-network-computing-connectx-5-adapter/), [InsideHPC CX-5 shipping](https://insidehpc.com/2016/10/mellanox-begins-shipments-of-connectx-5-adapter/) |
| ConnectX-6 | 200 Gb/s | Jul 2019 | Mid 2019 | [Mellanox CX-6](https://network.nvidia.com/files/doc-2020/pb-connectx-6-dx-en-dellemc.pdf) |
| ConnectX-6 Dx | 200 Gb/s | Aug 2019 | Late 2019 | [CX-6 Dx](https://nvidianews.nvidia.com/news/releases-20210113-6829469) |
| ConnectX-7 | 400 Gb/s | Nov 2021 (GTC) | H2 2022 | [NVIDIA CX-7 GTC](https://www.servethehome.com/nvidia-quantum-2-400g-switches-and-connectx-7-at-gtc-fall-2021/) |
| ConnectX-8 SuperNIC | 800 Gb/s | Nov 2024 (SC24) | Q2 2025 (production) | [ServeTheHome CX-8](https://www.servethehome.com/this-is-the-next-gen-nvidia-connectx-8-supernic-for-800gbps-networking/) |
| ConnectX-9 SuperNIC | 1.6 Tb/s | Announced (Rubin) | TBD (~2027) | Spectrum-6 / BlueField-4 platform |
### 2.5 Cisco Silicon One
| ASIC | Bandwidth | Role | Announced | Platform GA | Source |
|------|-----------|------|-----------|-------------|--------|
| Q100 | 10.8 Tbps | Routing | Dec 2019 | Dec 2019 (Cisco 8000) | [Cisco Q100](https://investor.cisco.com/news/news-details/2019/Cisco-Unveils-Plan-for-Building-Internet-for-the-Next-Decade-of-Digital-Innovation/) |
| Q200 / Q200L | 12.8 Tbps | Routing / Switching | Oct 2020 | 2021 | [Cisco Q200](https://blogs.cisco.com/sp/ciscosilicononeq200announcement) |
| P100 | 19.2 Tbps | Routing (modular) | 2021 | 2022-2023 | [Cisco P100](https://www.cisco.com/c/en/us/solutions/collateral/silicon-one/silicon-one-p100-processor-ds.html) |
| G100 | 25.6 Tbps | Switching | 2021-2022 | 2022-2023 | [Cisco G100](https://www.cisco.com/c/en/us/solutions/collateral/silicon-one/datasheet-c78-744833.html) |
| G200 | 51.2 Tbps | Switching (AI) | 2024 | 2024-2025 | [Cisco G200](https://www.cisco.com/c/en/us/solutions/collateral/silicon-one/silicon-one-g200-ds.html) |
| K100, E100 | Various | Edge/Enterprise | 2025 | 2025 | Extension to enterprise |
### 2.6 Intel Barefoot Tofino (CANCELLED)
| ASIC | Bandwidth | Status | Source |
|------|-----------|--------|--------|
| Tofino 1 | 6.4 Tbps | Shipped (2016+) | [Intel Tofino](https://www.intel.com/content/www/us/en/products/network-io/programmable-ethernet-switch.html) |
| Tofino 2 | 12.8 Tbps | Shipped (7nm, CoWoS) | [ServeTheHome Tofino2](https://www.servethehome.com/intel-tofino2-next-gen-programmable-switch-detailed/) |
| Tofino 3 | 25.6 Tbps | **CANCELLED Jan 2023** | [Intel exits switching](https://www.fool.com/investing/2023/01/29/intel-exits-another-non-core-business/); P4 software open-sourced 2025 |
Intel acquired Barefoot Networks in Jun 2019, but cancelled the Tofino line in Jan 2023 as part of $3B cost-cutting. Existing Tofino 1/2 products remain available from vendors like Asterfusion.
---
## 3. The Lag Formula {#3-the-lag-formula}
Based on all historical data points collected above, here are the empirically derived lag values:
### 3.1 Technology Development Lags
| Transition | Typical Lag | Range | Trend |
|-----------|-------------|-------|-------|
| **IEEE standard publication -> First commercial transceivers** | 18-24 months | 12-36 mo | Shortening |
| **MSA spec publication -> First samples** | 6-12 months | 3-18 mo | Stable |
| **First OFC demo -> Volume production** | 24-36 months | 18-48 mo | Shortening (AI pull) |
| **First OFC demo -> Mainstream adoption (>10% ports)** | 36-48 months | 30-60 mo | Shortening |
### 3.2 ASIC-to-Deployment Lags
| Transition | Typical Lag | Range | Source |
|-----------|-------------|-------|--------|
| **ASIC announcement -> First switch platform GA** | 9-18 months | 6-24 mo | Broadcom TH history |
| **Switch GA -> Transceiver demand ramp** | 6-12 months | 3-18 mo | Qualification + deployment |
| **ASIC tape-out -> Full transceiver ecosystem ramp** | 18-30 months | 12-36 mo | Combined |
### 3.3 Regional Deployment Lags
| Transition | Typical Lag | Range | Notes |
|-----------|-------------|-------|-------|
| **US hyperscaler deployment -> Enterprise deployment** | 18-36 months | 12-48 mo | Hyperscalers are early adopters |
| **US deployment -> China deployment** | 3-6 months | 0-12 mo | Chinese vendors dominate manufacturing; fast adoption |
| **US deployment -> Europe deployment** | 12-24 months | 6-36 mo | Slower procurement cycles, GDPR considerations |
| **US deployment -> APAC (ex-China) deployment** | 12-18 months | 6-24 mo | Japan/Korea faster; SEA/India slower |
| **US deployment -> RoW deployment** | 18-36 months | 12-48 mo | Varies enormously by country |
### 3.4 Coherent Optics Miniaturization Lag
| Transition | Typical Lag |
|-----------|-------------|
| **CFP -> CFP2** | ~3 years |
| **CFP2 -> CFP2-DCO** | ~5 years |
| **CFP2-DCO -> QSFP-DD-DCO** | ~4 years |
| **400G ZR spec -> Volume** | ~2 years |
| **800G ZR spec -> Volume** | ~2 years (projected) |
### 3.5 Acceleration Factors (AI Era)
The AI/ML demand cycle is compressing timelines by approximately 30-40% compared to the cloud computing era (2012-2020):
| Factor | Impact |
|--------|--------|
| Hyperscaler pre-ordering | -6 to -12 months (demand pull) |
| Direct NVIDIA-to-transceiver vendor procurement | -3 to -6 months (bypassing OEM) |
| Chinese vendor manufacturing agility | -3 to -6 months (rapid ramp) |
| Power/thermal constraints driving urgency | -3 to -6 months (competitive pressure) |
---
## 4. Demand Cascade Model {#4-demand-cascade-model}
### 4.1 The Cascade Flow
```
[AI Training Cluster Plans]
|
v
[GPU/XPU Production Forecasts]
| (1:1 GPU-to-NIC ratio for scale-out)
v
[Switch Fabric Requirements]
| (spine-leaf topology, radix determines port count)
v
[Port Count per Switch]
| (e.g., TH5: 64x800G, TH6: 64x1.6T)
v
[Transceiver Demand per Port]
| (speed x reach = specific SKU)
v
[Revenue Forecast per Transceiver Type]
```
### 4.2 Concrete Example: GB200 NVL72
Per [SemiAnalysis](https://newsletter.semianalysis.com/p/gb200-hardware-architecture-and-component) and [NADDOD analysis](https://www.naddod.com/blog/nvidia-gb200-interconnect-architecture-analysis-nvlink-infiniband-and-future-trends):
| Component | Quantity per NVL72 Rack | Notes |
|-----------|------------------------|-------|
| GPUs (Blackwell B200) | 72 | Per rack |
| NICs (CX-7 or CX-8) | 72 | 1:1 GPU-to-NIC ratio |
| Scale-out OSFP ports | 72 | 400G (CX-7) or 800G (CX-8) |
| Spine switch OSFP ports | Varies by topology | 2:1 or 3:1 oversubscription |
| Total optical modules per 576-GPU cluster | ~18,432 | [FiberMall estimate](https://www.fibermall.com/blog/nvidia-blackwell-development-for-dac-lacc-1600g-osfp-xd.htm) |
**Speed transition:**
- CX-7 era (2024-2025): 400G SR4/DR4 per GPU port
- CX-8 era (2025-2026): 800G DR4 per GPU port, 1.6T DR8 per switch port
- CX-9 era (2027+): 1.6T per GPU port, 3.2T per switch port
### 4.3 Total Addressable Market Drivers
| Data Source | What It Reveals | Forecast |
|-------------|----------------|----------|
| Hyperscaler CapEx (quarterly reports) | Total infrastructure spend | $600-690B in 2026 ([IEEE ComSoc](https://techblog.comsoc.org/2025/12/22/hyperscaler-capex-600-bn-in-2026/), [Futurum](https://futurumgroup.com/insights/ai-capex-2026-the-690b-infrastructure-sprint/)) |
| NVIDIA GPU production (H100/B200/GB200) | GPU count -> NIC count -> optics count | [SemiAnalysis GB200](https://newsletter.semianalysis.com/p/gb200-hardware-architecture-and-component) |
| Data center construction (Synergy, JLL, CBRE) | Site capacity -> future networking spend | Multi-year pipeline |
| Optical component supplier earnings | Revenue = realized demand | Ciena backlog ~$5B heading into 2026 |
### 4.4 Key Market Forecasts (2025-2029)
| Metric | 2024 | 2025 | 2026 | Source |
|--------|------|------|------|--------|
| 800GbE module shipments | ~8M | ~12.8M (60% growth) | ~20M+ | [Cignal AI](https://cignal.ai/2025/05/800gbe-optics-shipments-to-grow-60-in-2025/) |
| 1.6T module shipments | ~2.7M | ~4.2M | ~20M+ | Industry estimates |
| 800G coherent (ZR/ZR+) units | <50K | ~100K | >200K ($1B+ revenue) | [Cignal AI](https://cignal.ai/2025/07/800g-coherent-pluggable-shipments-to-exceed-1b-revenue-in-2026/) |
| Hyperscaler CapEx | ~$256B | ~$443B | ~$600-690B | CreditSights, Futurum |
| AI back-end network market | - | - | >$20B by 2028 | Dell'Oro |
| Optical interconnect market | - | $21.9B (2026) | $40B (2031) | Mordor Intelligence |
### 4.5 Transceiver Revenue Per Unit Economics
| Speed | Avg ASP (2025) | Trend |
|-------|----------------|-------|
| 400G DR4 | $150-250 | Declining |
| 800G SR8 | $300-500 | Declining as volume ramps |
| 800G DR8 | $500-800 | At scale pricing |
| 800G 2xFR4 | $600-900 | SM premium |
| 1.6T DR8 | $1,500-2,500 | Early premium, declining |
| 400G ZR | $2,000-3,000 | Mature |
| 800G ZR/ZR+ | $4,000-6,000 | Early premium |
---
## 5. Export Control Impact {#5-export-control-impact}
### 5.1 US/EU Export Control Timeline
| Date | Action | Impact on Optical |
|------|--------|-------------------|
| Oct 2022 | Biden administration first controls | Limited advanced chip access; optical transceivers NOT directly restricted |
| Oct 2023 | Controls tightened | DSP chips (Broadcom, Marvell) restricted for some end-uses |
| Jan 2025 | AI Diffusion Rule | Broader restrictions on advanced AI computing equipment |
| Mar 2025 | Trump administration additional restrictions | More Chinese entities blacklisted |
### 5.2 Impact on Chinese Optical Transceiver Ecosystem
**Key finding:** Optical transceivers themselves are NOT directly export-controlled, but the DSP chips inside them are the vulnerability point.
| Factor | Status | Source |
|--------|--------|--------|
| Chinese vendor market share (800G) | >60% globally, >70% of 800G market | Omdia data |
| InnoLight 2024 revenue | RMB 23.86B (+122.6% YoY) | [InnoLight financials](https://iamfabian.substack.com/p/pluggables-power-and-geopolitics) |
| Eoptolink 2024 revenue | RMB 8.65B (+179% YoY) | [Eoptolink financials](https://iamfabian.substack.com/p/pluggables-power-and-geopolitics) |
| DSP dependency (Broadcom/Marvell) | ~50% of module power; critical component | Export-controlled for certain end-uses |
| LPO as strategic hedge | Eliminates in-module DSP; -20% power, -30% cost | Reduces US tech dependency |
| Chinese DSP startups | Aluksen, EOChip, Hengxin, InSiGa, Leadingspeed, Luxic, MiniSilicon, Photonic Tech, Sitrus, UXFastic | Domestic substitution push |
| Manufacturing diversification | Eoptolink Thailand factory for North American shipments | Tariff and compliance mitigation |
| SMIC vs TSMC gap for DSP | SMIC limited to ~7nm (DUV); TSMC at 3nm (EUV) | 3-5nm DSPs require TSMC |
### 5.3 Modeling Regulatory Risk
For the predictive model, export controls introduce:
1. **DSP availability risk:** If Broadcom/Marvell DSPs become restricted for a specific end-use, Chinese module vendors must either:
- Switch to domestic DSPs (12-18 month qualification delay)
- Adopt LPO architecture (6-12 month redesign)
- Source DSPs through third-party channels (uncertain)
2. **Timeline impact by scenario:**
| Scenario | Impact on Chinese Vendor Timeline |
|----------|----------------------------------|
| Status quo (current controls) | No impact; Chinese vendors dominate |
| DSP export ban for Chinese transceiver vendors | +12-24 months for domestic DSP qualification |
| Full optical component controls | +24-36 months; unlikely given US vendor dependency |
| LPO adoption accelerates | -6 months (removes DSP bottleneck entirely) |
3. **For Chinese domestic market:** +6-18 months lag vs Western hyperscaler deployment, primarily due to GPU access restrictions limiting AI cluster buildouts.
---
## 6. Predictive Timeline Calculator {#6-predictive-calculator}
### 6.1 The Formula
```
T_samples = T_current + LAG_milestone_to_samples
T_volume = T_samples + LAG_samples_to_volume
T_mainstream = T_volume + LAG_volume_to_mainstream
```
Where the lag values depend on:
#### Milestone-to-Samples Lag Table
| Current Milestone | Lag to First Samples | Confidence |
|-------------------|---------------------|------------|
| Academic paper only | 36-60 months | +/- 18 mo |
| First OFC/ECOC demo | 12-24 months | +/- 9 mo |
| MSA/IEEE spec published | 6-12 months | +/- 6 mo |
| ASIC dependency announced | Add 6-12 months from ASIC GA | +/- 6 mo |
| Interop plugfest completed | 3-6 months | +/- 3 mo |
#### Samples-to-Volume Lag Table
| Technology Type | Lag to Volume | Confidence |
|----------------|---------------|------------|
| Incremental upgrade (same form factor, higher speed) | 6-12 months | +/- 3 mo |
| New form factor (e.g., QSFP-DD, OSFP-XD) | 12-18 months | +/- 6 mo |
| New architecture (e.g., coherent, CPO) | 18-36 months | +/- 12 mo |
| Disruptive technology (e.g., CPO at scale) | 24-48 months | +/- 18 mo |
#### Volume-to-Mainstream Lag Table
| Market Segment | Lag to >10% Ports | Confidence |
|---------------|-------------------|------------|
| US hyperscaler | 0-6 months (often concurrent with volume) | +/- 3 mo |
| China hyperscaler (Alibaba, Tencent, ByteDance) | 3-9 months | +/- 6 mo |
| Enterprise (US) | 18-36 months | +/- 12 mo |
| Enterprise (Europe) | 24-42 months | +/- 12 mo |
| Service provider | 12-24 months | +/- 9 mo |
### 6.2 ASIC Dependency Modifier
If a transceiver requires a specific switching ASIC:
```
T_samples = max(T_from_milestone, T_asic_ga + 3 months)
```
The transceiver cannot ramp before the switch ASIC is available. Key dependencies:
| Transceiver Speed | Required ASIC Generation | ASIC GA |
|-------------------|--------------------------|---------|
| 400G | Broadcom TH3+ / Spectrum-3+ | Available since 2017 |
| 800G | Broadcom TH5+ / Spectrum-4+ | Available since 2023 |
| 1.6T | Broadcom TH6 / Spectrum-X / CX-8 | TH6: Mar 2026, CX-8: Q2 2025 |
| 3.2T | Next-gen (TH7? / Spectrum-6) | ~2028 projected |
### 6.3 Worked Examples
#### Example 1: 1.6T OSFP-XD DR8
**Input:**
- Technology: 1.6T OSFP-XD DR8
- Current milestone: Volume orders placed (Mar 2026)
- ASIC dependency: Broadcom Tomahawk 6 (GA Mar 2026)
**Calculation:**
- T_samples: Q4 2024 (already happened)
- T_volume: Q3 2026 (AOI $200M order ships Q3 2026)
- T_mainstream (US hyperscaler): H2 2026 - H1 2027
- T_mainstream (Enterprise US): 2028-2029
- T_mainstream (Europe): 2029-2030
**Confidence:** Medium-High (ASIC available, volume orders placed)
#### Example 2: 3.2T OSFP (hypothetical next-gen)
**Input:**
- Technology: 3.2T OSFP (16x200G or 8x400G)
- Current milestone: Concept/early research (448G PAM4 SerDes expected ~2027)
- ASIC dependency: Next-gen (~TH7, expected ~2028)
**Calculation:**
- T_first_demo: OFC 2027 (+/- 6 mo)
- T_samples: H2 2028 (+/- 9 mo)
- T_volume: H2 2029 - H1 2030 (+/- 12 mo)
- T_mainstream (US hyperscaler): 2030 (+/- 12 mo)
- T_mainstream (Enterprise): 2032+ (+/- 18 mo)
**Confidence:** Low (depends on 448G SerDes and next-gen ASIC)
#### Example 3: CPO at Scale-Out
**Input:**
- Technology: CPO (scale-out Ethernet)
- Current milestone: Lab validation complete (Meta 1M link-hours, Oct 2025)
- ASIC dependency: NVIDIA Spectrum-X CPO (H2 2026) / Broadcom Davisson (Oct 2025)
**Calculation:**
- T_initial_deployment: 2026 (small scale)
- T_volume: 2027-2028 (manufacturing capability)
- T_mainstream (>10% of DC switch ports): 2029-2030
- T_mainstream (enterprise): Unlikely before 2032
**Confidence:** Low-Medium (manufacturing scaling is the key unknown)
### 6.4 Regional Rollout Timeline Modifier
Apply these offsets from US hyperscaler deployment:
```
T_region = T_us_hyperscaler + REGIONAL_OFFSET
```
| Region | Offset (months) | Notes |
|--------|----------------|-------|
| US Hyperscaler | 0 (baseline) | Google, Meta, Microsoft, Amazon |
| China Hyperscaler | +3 to +6 | Fast adoption but GPU access limited |
| Japan/Korea Enterprise | +12 to +18 | NTT, KDDI, SK Telecom early |
| Europe Service Provider | +12 to +24 | DT, Orange, Telefonica |
| US Enterprise | +18 to +36 | Fortune 500 DC upgrades |
| Europe Enterprise | +24 to +42 | Longer procurement, GDPR |
| India/SEA | +18 to +30 | Jio, Tata leading; rest slower |
| LATAM/Africa | +30 to +48 | Limited DC infrastructure |
### 6.5 Algorithm Implementation (Pseudocode)
```python
def predict_timeline(
technology: str,
current_milestone: str, # "paper", "demo", "spec", "samples", "volume"
asic_dependency: str | None,
asic_ga_date: date | None,
is_new_form_factor: bool = False,
is_new_architecture: bool = False,
ai_demand_driven: bool = True,
) -> dict:
# Base lag from current milestone to samples
milestone_lags = {
"paper": (36, 60, 18), # (min, max, uncertainty) months
"demo": (12, 24, 9),
"spec": (6, 12, 6),
"interop": (3, 6, 3),
"samples": (0, 0, 0),
"volume": (-12, -6, 3), # Already past samples
}
min_lag, max_lag, uncertainty = milestone_lags[current_milestone]
base_samples_date = today + months(avg(min_lag, max_lag))
# ASIC dependency check
if asic_dependency and asic_ga_date:
asic_ready = asic_ga_date + months(3)
base_samples_date = max(base_samples_date, asic_ready)
# Samples to volume lag
if is_new_architecture:
volume_lag = months(27) # 18-36 range
elif is_new_form_factor:
volume_lag = months(15) # 12-18 range
else:
volume_lag = months(9) # 6-12 range
# AI demand acceleration factor
if ai_demand_driven:
volume_lag *= 0.65 # 35% acceleration
volume_date = base_samples_date + volume_lag
# Regional rollout
regional = {
"US_hyperscaler": volume_date,
"China_hyperscaler": volume_date + months(4),
"Japan_Korea": volume_date + months(15),
"Europe_SP": volume_date + months(18),
"US_enterprise": volume_date + months(27),
"Europe_enterprise": volume_date + months(33),
"India_SEA": volume_date + months(24),
"LATAM_Africa": volume_date + months(39),
}
# Confidence intervals
confidence = {
"samples": uncertainty,
"volume": uncertainty + 3 if is_new_form_factor else uncertainty,
"mainstream": uncertainty + 6,
}
return {
"predicted_samples": base_samples_date,
"predicted_volume": volume_date,
"predicted_mainstream": volume_date + months(12),
"confidence_months": confidence,
"regional_rollout": regional,
}
```
### 6.6 Historical Validation
| Technology | Predicted (using formula) | Actual | Delta |
|-----------|--------------------------|--------|-------|
| 100G QSFP28 (from OFC 2015 demo) | Volume: Q1 2017 | Volume: Mar 2017 | 0 mo |
| 400G QSFP-DD (from OFC 2017 demo) | Volume: Q1 2020 | Volume: H1 2020 | +3 mo |
| 400G ZR (from spec Mar 2020) | Volume: Q1 2022 | Volume: Early 2022 | 0 mo |
| 800G DR8 (from OFC 2021 demo) | Volume: Q1 2024 | Volume: Mid 2024 | +3 mo |
| 1.6T (from OFC 2023 demo) | Volume: Q1 2026 | Volume: Q3 2026 (projected) | +6 mo |
**Average prediction error: +2.4 months** (formula is slightly optimistic).
---
## Key Data Sources for Ongoing Model Updates
| Source | URL | What It Provides | Update Frequency |
|--------|-----|------------------|-----------------|
| Cignal AI | https://cignal.ai | Optical market forecasts, shipment data | Monthly/Quarterly |
| LightCounting | https://www.lightcounting.com | Transceiver shipment volumes, pricing | Monthly |
| Dell'Oro Group | https://www.delloro.com | Data center networking, optical transport | Quarterly |
| OFC Conference | https://www.ofcconference.org | Annual demos, product launches | Annual (March) |
| ECOC Conference | https://www.ecocexhibition.com | European demos, product launches | Annual (September/October) |
| OIF | https://www.oiforum.com | Implementation Agreements, interop | As published |
| IEEE 802.3 | https://www.ieee802.org/3/ | Ethernet standards | As ratified |
| Broadcom press releases | https://www.broadcom.com/company/news/product-releases | ASIC announcements | As released |
| NVIDIA networking | https://www.nvidia.com/en-us/networking/ | Switch/NIC announcements | As released |
| Hyperscaler quarterly earnings | SEC filings | CapEx guidance, AI spending | Quarterly |
---
## Sources
### Transceiver Timelines
- [InnoLight OFC 2017 - 100G QSFP28 Volume](https://www.innolight.com/en/news/newsinfo/13.html)
- [Oclaro OFC 2017 - 100G ER4 QSFP28](https://www.prnewswire.com/news-releases/oclaro-showcases-industrys-first-live-40km-interoperability-demo-between-100g-extended-reach-qsfp28-and-cfp2-at-ofc-2017-300426690.html)
- [Kaiam OFC 2015 - 100G QSFP28 + 400G demo](https://www.businesswire.com/news/home/20150319005175/en/)
- [ColorChip OFC 2016 - 100G PSM4 QSFP28](https://www.globenewswire.com/news-release/2016/03/18/940853/0/en/)
- [Finisar OFC 2018 - 400G demos](https://picmagazine.net/article/103776/Finisar_Demos_New_400G_Transceivers_At_OFC_2018)
- [Oclaro OFC 2017 - 400G CFP8](https://www.prnewswire.com/news-releases/oclaro-samples-400g-cfp8-pam4-enabled-transceiver-showcases-live-demo-at-ofc-2017-300425943.html)
- [QSFP-DD MSA specifications](http://www.qsfp-dd.com/specification/)
- [OIF 400ZR IA](https://convergedigest.com/oif-publishes-400zr-implementation/)
- [Inphi 400ZR ramp](https://convergedigest.com/inphi-ramps-shipments-of-400zr-and-zr/)
- [Molex 400G ZR volume](https://www.molex.com/en-us/news/molex-ramps-production-of-400g-zr-qsfp-dd-coherent-optical)
- [Fujitsu 400G ZR launch](https://opticalconnectionsnews.com/2020/10/fujitsu-launches-400g-zr-transceivers/)
- [Gazettabyte - Intel 800G DR8](https://www.gazettabyte.com/home/2021/6/29/intel-details-its-800-gigabit-dr8-optical-module.html)
- [LESSENGERS 800G volume](https://www.semiconductor-today.com/news_items/2023/sep/lessengers-280923.shtml)
- [Hyper Photonix 800G DR8 GA](https://www.businesswire.com/news/home/20240517136062/en/)
- [Cignal AI 800GbE growth](https://cignal.ai/2025/05/800gbe-optics-shipments-to-grow-60-in-2025/)
- [Coherent 800G ZR/ZR+ unveiled](https://www.semiconductor-today.com/news_items/2023/dec/coherent-081223.shtml)
- [Coherent 800G ZR/ZR+ GA](https://www.coherent.com/news/press-releases/general-availability-of-800g-zr-zrplus-in-qsfp-dd-form-factor)
- [Cignal AI 800G coherent $1B forecast](https://cignal.ai/2025/07/800g-coherent-pluggable-shipments-to-exceed-1b-revenue-in-2026/)
- [Eoptolink Gen2 1.6T OFC 2025](https://www.eoptolink.com/news/361-eoptolink-launches-its-gen2-1-6t-osfp-and-osfp-rhs-transceiver-family-at-ofc-2025)
- [Jabil 1.6T launch](https://investors.jabil.com/news/news-details/2025/Jabil-Launches-1-6T-Pluggable-Transceiver/)
- [AOI first $200M+ 1.6T volume order](https://www.globenewswire.com/news-release/2026/03/09/3251675/9986/en/)
- [Coherent dual-laser QSFP28-DCO GA](https://www.globenewswire.com/news-release/2026/03/06/3251306/11543/en/)
### ASIC Timelines
- [Broadcom TH3](https://www.globenewswire.com/news-release/2017/12/19/1266218/0/en/)
- [Broadcom TH4 (NextPlatform)](https://www.nextplatform.com/2019/12/12/broadcom-launches-another-tomahawk-into-the-datacenter/)
- [Broadcom TH5](https://investors.broadcom.com/news-releases/news-release-details/broadcom-ships-tomahawk-5-industrys-highest-bandwidth-switch)
- [Broadcom TH6](https://www.broadcom.com/company/news/product-releases/63146)
- [Broadcom TH6 volume Mar 2026](https://markets.financialcontent.com/stocks/article/marketminute-2026-3-26-the-great-ethernet-pivot)
- [Broadcom Davisson CPO](https://investors.broadcom.com/news-releases/news-release-details/broadcom-announces-tomahawkr-6-davisson-industrys-first-1024)
- [Broadcom J3-AI](https://www.broadcom.com/company/news/product-releases/61156)
- [Broadcom J4](https://investors.broadcom.com/news-releases/news-release-details/broadcom-ships-jericho4-enabling-distributed-ai-computing-across)
- [DriveNets/Accton J3-AI white box](https://www.prnewswire.com/news-releases/drivenets-and-accton-technology-launch-the-highest-performance-ethernet-based-ai-networking-solution-302273214.html)
- [NVIDIA Spectrum-4](https://www.nextplatform.com/2022/04/01/spectrum-4-ethernet-leaps-to-800-gb-sec-with-nvidia-circuits/)
- [NVIDIA Spectrum-X CPO](https://www.techradar.com/pro/nvidia-is-planning-post-copper-1-6tbps-network-tech)
- [Mellanox CX-5 announcement](https://www.hpcwire.com/2016/06/16/mellanox-advances-network-computing-connectx-5-adapter/)
- [Mellanox CX-5 shipping](https://insidehpc.com/2016/10/mellanox-begins-shipments-of-connectx-5-adapter/)
- [NVIDIA CX-7 GTC 2021](https://www.servethehome.com/nvidia-quantum-2-400g-switches-and-connectx-7-at-gtc-fall-2021/)
- [NVIDIA CX-8 SuperNIC](https://www.servethehome.com/this-is-the-next-gen-nvidia-connectx-8-supernic-for-800gbps-networking/)
- [Cisco Silicon One Q100](https://investor.cisco.com/news/news-details/2019/Cisco-Unveils-Plan-for-Building-Internet-for-the-Next-Decade-of-Digital-Innovation/)
- [Cisco Silicon One Q200](https://blogs.cisco.com/sp/ciscosilicononeq200announcement)
- [Cisco Silicon One G100](https://www.cisco.com/c/en/us/solutions/collateral/silicon-one/datasheet-c78-744833.html)
- [Cisco Silicon One G200](https://www.cisco.com/c/en/us/solutions/collateral/silicon-one/silicon-one-g200-ds.html)
- [Intel Tofino cancelled](https://www.fool.com/investing/2023/01/29/intel-exits-another-non-core-business/)
### CPO & LPO
- [Cignal AI CPO report](https://cignal.ai/2025/02/co-packaged-optics-inevitable-but-not-imminent/)
- [EDN CPO status 2026](https://www.edn.com/where-co-packaged-optics-cpo-technology-stands-in-2026/)
- [LPO MSA spec release](https://www.globenewswire.com/news-release/2025/03/25/3048840/0/en/)
- [FLEXOPTIX LPO introduction](https://www.flexoptix.net/en/blog/blog/introducing-linear-pluggable-optics)
- [Eoptolink LPO OFC 2024](https://www.lightwaveonline.com/home/article/14310702/eoptolink-showcases-200g-linear-drive-pluggable-optics-at-ofc-2024)
- [Ethernet Alliance ECOC 2025](https://ethernetalliance.org/blog/2025/10/27/ecoc-2025-interoperability-at-800g-is-given-advancing-toward-1-6t/)
### Demand & CapEx
- [IEEE ComSoc - Hyperscaler CapEx $600B+](https://techblog.comsoc.org/2025/12/22/hyperscaler-capex-600-bn-in-2026/)
- [Futurum - AI CapEx $690B](https://futurumgroup.com/insights/ai-capex-2026-the-690b-infrastructure-sprint/)
- [SemiAnalysis - GB200 architecture](https://newsletter.semianalysis.com/p/gb200-hardware-architecture-and-component)
- [FiberMall - NVIDIA Blackwell optics demand](https://www.fibermall.com/blog/nvidia-blackwell-development-for-dac-lacc-1600g-osfp-xd.htm)
- [NADDOD - GB200 interconnect analysis](https://www.naddod.com/blog/nvidia-gb200-interconnect-architecture-analysis-nvlink-infiniband-and-future-trends)
### Export Controls & Geopolitics
- [Pluggables, Power, and Geopolitics (iamfabian)](https://iamfabian.substack.com/p/pluggables-power-and-geopolitics)
- [CRS - US Export Controls on China](https://www.congress.gov/crs-product/R48642)
- [American Affairs - China Semiconductor Evolution](https://americanaffairsjournal.org/2024/11/the-evolution-of-chinas-semiconductor-industry-under-u-s-export-controls/)
- [CSIS - Limits of Chip Export Controls](https://www.csis.org/analysis/limits-chip-export-controls-meeting-china-challenge)
### Standards & Specifications
- [IEEE 802.3ae (10GbE)](https://resources.l-p.com/knowledge-center/what-is-ieee-802-3ae-10-gigabit-ethernet)
- [OIF 400ZR spec (PDF)](https://www.oiforum.com/wp-content/uploads/OIF-400ZR-02.0.pdf)
- [OIF 800G Coherent](https://www.oiforum.com/technical-work/hot-topics/800g-coherent/)
- [QSFP-DD MSA Rev 5.0](http://www.qsfp-dd.com/wp-content/uploads/2019/07/QSFP-DD-Hardware-rev5p0.pdf)
- [LPO MSA](https://www.lpo-msa.org/home.html)

684
RESEARCH-hype-cycle-signals.md Normal file
View File

@ -0,0 +1,684 @@
# Hype Cycle Signal Research: Quantifiable Data Inputs for Automatic Technology Positioning
**Date:** 2026-03-28
**For:** Transceiver Intelligence Platform (TIP) — Hype Cycle Engine
**Status:** Deep Research Complete — Ready for Implementation Planning
---
## Executive Summary
This document catalogs **10 quantifiable signal categories** that can feed the TIP Hype Cycle Engine to automatically position optical transceiver technologies (400G, 800G, 1.6T, QSFP-DD, OSFP, silicon photonics, coherent pluggable, co-packaged optics, etc.) on a Norton-Bass-derived hype cycle.
**Key finding:** A composite of 5-6 signals provides robust positioning. No single signal is sufficient alone. The recommended **Phase 1 implementation** (3 signals, all free, all validated) can be built in ~2 weeks.
---
## Signal Catalog
---
### 1. PATENT DATA (Technology Innovation Signal)
**What it measures:** R&D investment intensity, innovation velocity, technology maturation
**Hype cycle relevance:** Patents LEAD actual market adoption by 3-5 years. Patent filing surges correlate with "Technology Trigger" and early "Peak of Inflated Expectations."
#### Data Source: USPTO PatentsView API (migrating to data.uspto.gov March 2026)
| Attribute | Detail |
|-----------|--------|
| **API URL** | `https://search.patentsview.org/api/v1/patent/` |
| **Auth** | API key required (header `X-Api-Key`). Free but new grants temporarily suspended during migration to data.uspto.gov |
| **Rate Limit** | 45 requests/minute |
| **Update Frequency** | Quarterly |
| **Cost** | Free (CC BY 4.0 license) |
| **Python Library** | `requests` (REST API), `patentsview2` (R package, no maintained Python equivalent) |
| **Implementation Complexity** | 2/5 |
#### Relevant CPC Classes for Optical Transceivers
| CPC Class | Description |
|-----------|-------------|
| **H04B10** | Transmission systems employing electromagnetic waves other than radio waves (optical communication) |
| **G02B6** | Light guides; structural details of fibre-optic arrangements |
| **H01S5** | Semiconductor lasers (VCSELs, DFB, EML — core transceiver components) |
| **H04J14** | Optical multiplex systems (WDM, DWDM) |
| **G02F1** | Devices or arrangements for the control of light intensity (modulators) |
#### Queryable Metrics
1. **Patent Filing Velocity** — Count of new patent applications per CPC class per quarter
2. **Patent Grant Rate** — Ratio of grants to applications (maturity indicator)
3. **Citation Velocity** — How quickly new patents cite each other (hot field indicator)
4. **Technology Cycle Time (TCT)** — Median age of citations (shorter = faster-moving field)
5. **Assignee Concentration** — Herfindahl index of patent holders (few holders = early stage; many = maturation)
#### Example Query (PatentsView Search API)
```
GET https://search.patentsview.org/api/v1/patent/
?q={"_and":[{"_begins":{"cpc_at_issue.cpc_subclass_id":"H04B10"}},{"_gte":{"patent_date":"2024-01-01"}},{"_text_any":{"patent_abstract":"transceiver 400G 800G QSFP OSFP"}}]}
&f=["patent_id","patent_date","patent_title","assignees.assignee_organization"]
&o={"size":100}
```
Response includes `total_hits` for counting.
#### Academic Validation
- **BIMATEM method** (Manrique-Castillo et al., Scientometrics 2018): Patent records of mature technologies display **logistic growth** behavior. Fitting logistic curves to patent counts per technology enables TRL assignment.
- **Gao et al. (2013)**: Using multiple patent-based indicators with a nearest-neighbour classifier for technology life cycle stage classification.
- **Technology Cycle Time**: Kayal's TCT indicator — median citation age predicts technology maturity phase.
#### Correlation with Hype Cycle Position
- **High filing velocity + low grant rate** = Technology Trigger / early Peak
- **Peak filing count reached** = Peak of Inflated Expectations
- **Declining filings + rising citations** = Trough / early Slope
- **Stable filings + high citation density** = Plateau of Productivity
---
### 2. ACADEMIC PUBLICATION METRICS (Knowledge Creation Signal)
**What it measures:** Scientific research intensity, knowledge maturation
**Hype cycle relevance:** Publication counts follow a logistic S-curve. The inflection point of the S-curve corresponds roughly to the transition from Peak to Trough.
#### Data Source: Semantic Scholar API (VALIDATED — working)
| Attribute | Detail |
|-----------|--------|
| **API URL** | `https://api.semanticscholar.org/graph/v1/paper/search/bulk` |
| **Auth** | None required (public). API key available for higher rate limits |
| **Rate Limit** | 1000 req/sec (shared unauthenticated), 1 req/sec (with free API key) |
| **Update Frequency** | Continuous (near real-time) |
| **Cost** | Free |
| **Coverage** | ~200 million papers across all disciplines |
| **Python Library** | `semanticscholar` (PyPI) or direct `requests` |
| **Implementation Complexity** | 1/5 |
#### Validated Paper Counts (tested 2026-03-28)
| Technology | Total Papers | Maturity Signal |
|------------|-------------|-----------------|
| silicon photonics transceiver | 905 | Mature (deep research base) |
| 100G transceiver | 144 | Late maturity |
| 400G transceiver | 100 | Growth phase |
| 200G transceiver | 43 | Moderate |
| coherent pluggable optics | 40 | Growth phase |
| 800G transceiver | 39 | Early growth |
| QSFP-DD optical | 26 | Emerging |
| OSFP transceiver | 11 | Very early |
| 1.6T transceiver optical | 10 | Pre-commercial |
#### Year-by-Year Trend (400G transceiver, validated)
| Year | Papers | Signal |
|------|--------|--------|
| 2018 | 10 | Early research |
| 2019 | 7 | Stable |
| 2020 | 7 | Stable |
| 2021 | 9 | Slight increase |
| 2022 | 15 | Growth spike |
| 2023 | 6 | Decline |
| 2024 | 8 | Recovery |
| 2025 | 12 | Resurgence |
This pattern (spike in 2022, decline 2023, recovery 2024-25) maps well to the 400G transition from Peak to Slope of Enlightenment.
#### Key Metrics to Extract
1. **Annual paper count** per technology keyword
2. **Rate of change** (first derivative — acceleration/deceleration)
3. **Citation count distribution** — highly cited papers = foundational work = maturation
4. **Author diversity** — many unique authors = broad interest = growth phase
5. **Venue distribution** — OFC/ECOC papers vs. general journals
#### Supplementary Source: IEEE Xplore
- URL: `https://ieeexploreapi.ieee.org/api/v1/search/articles`
- API key required (free for research)
- Specifically covers OFC, ECOC, CLEO proceedings
- Higher signal quality for optical networking specifically
---
### 3. GOOGLE TRENDS (Public Interest / Hype Proxy)
**What it measures:** Search interest as a proxy for market attention and hype
**Hype cycle relevance:** Google Trends data directly models the "hype" component. Academic validation exists (Jun 2012, van Lente 2013).
#### Data Source: Google Trends via pytrends (VALIDATED — working)
| Attribute | Detail |
|-----------|--------|
| **API** | Unofficial (Google Trends web scraping via pytrends) |
| **Auth** | None |
| **Rate Limit** | ~10 requests/minute (unofficial, subject to blocking) |
| **Update Frequency** | Real-time (weekly/monthly granularity) |
| **Cost** | Free |
| **Python Library** | `pytrends` (PyPI, v4.9.2) |
| **Implementation Complexity** | 1/5 |
#### Validated Data (tested 2026-03-28)
**Batch 1 — Form Factors & Speeds (relative to each other):**
| Technology | Current Interest | Peak Value | Peak Date | Trajectory |
|------------|-----------------|------------|-----------|------------|
| silicon photonics | 100 (reference) | 100 | 2026-03 | Rising strongly |
| OSFP | 34 | 45 | 2024-05 | Peaked, declining |
| 800G transceiver | 10 | 10 | 2026-02 | Rising |
| QSFP-DD | 8 | 10 | 2025-11 | Declining from peak |
| 400G transceiver | 2 | 3 | 2025-12 | Low/stable (mature) |
**Batch 2 — Emerging Technologies:**
| Technology | Current Interest | Peak Value | Peak Date | Trajectory |
|------------|-----------------|------------|-----------|------------|
| co-packaged optics | 100 (reference) | 100 | 2026-03 | Rising strongly |
| coherent optics | 45 | 45 | 2026-03 | Rising |
| 1.6T ethernet | 5 | 14 | 2025-08 | Peaked, declining |
| 100G transceiver | 5 | 8 | 2026-02 | Low/stable |
#### Key Observations
- **OSFP peaked May 2024** — consistent with 802.3df approval (Feb 2024) driving peak hype
- **QSFP-DD declining from Nov 2025 peak** — market settling
- **co-packaged optics and silicon photonics surging** — current hype leaders
- **400G transceiver at floor** — fully mature, no hype left (Plateau of Productivity)
- **1.6T peaked Aug 2025** then declined — possible "Peak of Inflated Expectations" → Trough
#### Implementation Notes
- Normalize by comparing technologies against each other (relative index)
- Use monthly granularity for trend detection
- Calculate: peak detection, slope analysis, time-since-peak
- Combine with absolute volume signals (paper counts) since Google Trends is relative only
- **Limitation:** B2B niche terms have low search volumes — use broader terms ("silicon photonics" not "silicon photonics transceiver module QSFP-DD800")
#### Academic Validation
- **Jun (2012)**: "An empirical study of users' hype cycle based on search traffic" — validated Google Trends hype cycle matching for hybrid cars (*Scientometrics* 91(1), pp. 81-99)
- **van Lente, Spitters & Peine (2013)**: "Comparing technological hype cycles: Towards a theory" (*Technological Forecasting and Social Change* 80(8))
- **Choi & Varian (2010)**: "Predicting the Present with Google Trends" (foundational paper on search data as predictor)
- **Caveat**: Medeiros et al. (arXiv 2021) document preprocessing requirements for reliable forecasting from Trends data
---
### 4. NEWS/MEDIA VOLUME (Hype Amplification Signal)
**What it measures:** Trade press and media coverage volume and sentiment
**Hype cycle relevance:** News volume directly measures the "hype" dimension. Sentiment analysis distinguishes Peak (positive) from Trough (negative/absent).
#### Data Source A: GDELT DOC 2.0 API (VALIDATED — working, limited for niche B2B)
| Attribute | Detail |
|-----------|--------|
| **API URL** | `https://api.gdeltproject.org/api/v2/doc/doc` |
| **Auth** | None |
| **Rate Limit** | Reasonable (no published limit) |
| **Update Frequency** | Every 15 minutes |
| **Cost** | Free |
| **Coverage** | 100+ languages, 65 translated, millions of sources |
| **History** | Last 3 months reliably (older data not guaranteed) |
| **Python Library** | `gdeltdoc` (PyPI) or `gdeltPyR` (PyPI) |
| **Implementation Complexity** | 2/5 |
**Limitation for TIP:** GDELT covers general news very well but B2B optical transceiver coverage is sparse. Testing showed only 1 article for "400G optical" in 3 months. Better for broader terms like "silicon photonics" or "data center optics."
#### Data Source B: NewsAPI.org
| Attribute | Detail |
|-----------|--------|
| **API URL** | `https://newsapi.org/v2/everything` |
| **Free Tier** | 100 requests/day, 1-month history, 24h delay, dev-only |
| **Paid** | From $40/month |
| **Python** | `requests` (simple REST) |
| **Implementation Complexity** | 1/5 |
#### Data Source C: Trade Press RSS/Scraping (RECOMMENDED for optical)
Monitor these sources directly (Crawlee/Playwright — already in TIP architecture):
| Source | URL | Relevance |
|--------|-----|-----------|
| LightReading | lightreading.com | Primary (optical networking) |
| Fierce Telecom | fiercetelecom.com | Primary |
| Datacenter Dynamics | datacenterdynamics.com | Primary |
| SDxCentral | sdxcentral.com | Primary |
| Lightwave Online | lightwaveonline.com | Primary (optical specific) |
| Gazettabyte | gazettabyte.com | High (standards/specs) |
| Converge Digest | convergedigest.com | Moderate |
| Semiconductor Today | semiconductor-today.com | Moderate (component level) |
#### Metrics to Extract
1. **Article count per technology per month** (volume)
2. **Sentiment score** using VADER (lightweight) or FinBERT (more accurate)
3. **Source diversity** — how many different outlets cover the topic
4. **Headline vs. mention** — is the technology the headline or just mentioned?
#### Sentiment Analysis Tools
| Tool | Type | Cost | Accuracy | Speed |
|------|------|------|----------|-------|
| VADER | Rule-based | Free | Good for general | Very fast |
| FinBERT | Transformer | Free | Best for financial/tech | Moderate |
| Ollama (qwen2.5:14b) | LLM | Free (local) | Very good | Slow |
| TextBlob | Rule-based | Free | Basic | Very fast |
**Recommendation:** Use VADER for initial scoring, Ollama for nuanced classification on flagged articles.
---
### 5. VENDOR COUNT / SKU PROLIFERATION (Market Adoption Signal)
**What it measures:** Market entry velocity, competitive maturation, commoditization
**Hype cycle relevance:** This is THE strongest signal for distinguishing Slope of Enlightenment from Plateau of Productivity. Directly measurable from TIP's own scraper data.
#### Data Source: TIP's Own Scraper Database (ZERO ADDITIONAL COST)
| Attribute | Detail |
|-----------|--------|
| **Source** | TIP price_observations + vendor tables |
| **Auth** | Internal |
| **Update Frequency** | Real-time (5-15 min scraper intervals) |
| **Cost** | Already being collected |
| **Implementation Complexity** | 1/5 (data already exists) |
#### Metrics
1. **Vendor Count per Technology** — How many vendors sell a given form factor/speed
- 1-3 vendors = Technology Trigger / early Peak
- 4-10 vendors = Peak / early Slope
- 10-30 vendors = Slope of Enlightenment
- 30+ vendors = Plateau of Productivity
2. **SKU Growth Rate** — New product listings per month
- Accelerating = Growth phase
- Decelerating = Maturation
- Flat = Plateau
3. **Price Coefficient of Variation (CV)** — Standard deviation / mean of prices across vendors
- High CV (>0.5) = Early market, pricing uncertainty
- Medium CV (0.2-0.5) = Competitive market
- Low CV (<0.2) = Commodity market (Plateau)
4. **Price Decline Rate** — $/Gbps over time
- Steep decline = Growth → Slope transition
- Gradual decline = Slope
- Flat = Plateau
5. **Geographic Vendor Distribution** — Chinese vendors entering = commoditization signal
#### Why This Signal is Critical
This is **the only signal that directly measures actual market behavior** rather than proxies (search interest, papers, patents). Combined with price data, it provides ground truth for hype cycle calibration.
---
### 6. STANDARDS PROGRESS (Technology Readiness Signal)
**What it measures:** Standardization maturity as proxy for technology readiness
**Hype cycle relevance:** Standards progress is a LEADING indicator. "Study group formed" precedes market by 3-5 years.
#### Standards Phase Mapping to Hype Cycle
| Standards Phase | Typical Duration | Hype Cycle Phase |
|----------------|-----------------|------------------|
| Call for Interest / Study Group | 6-12 months | Pre-Trigger |
| Task Force Formed | 0 | Technology Trigger |
| First Draft | 12-18 months | Peak of Inflated Expectations |
| Working Group Ballot | 6-12 months | Peak → Trough transition |
| Sponsor Ballot | 3-6 months | Trough → Slope |
| Standard Published | 0 | Slope of Enlightenment |
| First Amendment | 12-24 months | Plateau of Productivity |
#### Current State (validated 2026-03-28)
| Technology | Standard | Status | Hype Phase Inference |
|------------|----------|--------|---------------------|
| 400G Ethernet | IEEE 802.3bs | Published Dec 2017 | Plateau |
| 800G Ethernet (100G/lane) | IEEE 802.3df | Published Feb 2024 | Slope of Enlightenment |
| 800G Ethernet (200G/lane) | IEEE 802.3dj | In progress, target Jul 2026 | Peak → Trough |
| 1.6T Ethernet | IEEE 802.3dj | In progress, target Jul 2026 | Peak of Inflated Expectations |
| 3.2T Ethernet | OIF/MSA discussions | Study group phase | Pre-Trigger |
| 400ZR Coherent | OIF IA published Apr 2020 | Published | Late Slope |
#### Trackable Standards Bodies
| Body | What to Track | URL |
|------|--------------|-----|
| **IEEE 802.3** | Task force status, ballot dates | ieee802.org/3/ |
| **OIF** | Implementation Agreements (IAs), CMIS versions | oiforum.com/technical-work/implementation-agreements-ias/ |
| **QSFP-DD MSA** | Spec revisions (now at QSFP-DD1600) | qsfp-dd.com |
| **OSFP MSA** | Spec revisions (now at Rev 5.21) | osfpmsa.org |
| **100G Lambda MSA** | FR/LR specs | 100glambda.com |
#### Implementation
- Maintain a manually-curated `standards_progress` table
- Update quarterly (standards move slowly)
- Each standard gets a numeric score: 0 (no activity) → 10 (published + amendments)
- **Implementation Complexity:** 2/5 (manual curation, low frequency)
---
### 7. JOB MARKET SIGNALS (Demand/Deployment Signal)
**What it measures:** Actual hiring demand for technology-specific skills
**Hype cycle relevance:** Job posting surges lag the Peak by 12-18 months and correlate with Slope of Enlightenment.
#### Data Sources
| Source | Cost | API | Quality |
|--------|------|-----|---------|
| **TheirStack** | Free tier available | REST API | Best (deduplication, 324k ATS platforms) |
| **FlyByAPIs** | Free (200 req/month) | RapidAPI | Good (Google Jobs index) |
| **Sumble** | Free 500 credits/month | REST API | Good (LinkedIn + hiring signals) |
| **LinkedIn Talent** | Enterprise ($$$) | Partner only | Best but inaccessible |
| **Indeed Job Sync** | Free (partner) | REST API | Posting-focused, not search |
**Recommended:** TheirStack or FlyByAPIs for free tier.
#### Metrics
1. **Job posting count** per technology keyword per month
2. **Job posting velocity** — rate of change
3. **Salary range** — higher salaries = talent scarcity = early adoption
4. **Geographic distribution** — US/EU = early; APAC = maturation
#### Implementation Complexity: 3/5
---
### 8. SOCIAL MEDIA / COMMUNITY SIGNALS (Practitioner Interest)
**What it measures:** Operator and engineer discussion intensity
**Hype cycle relevance:** Community buzz leads deployment by 6-12 months.
#### Data Sources
| Source | API | Cost | Python Library |
|--------|-----|------|----------------|
| **Reddit** (r/networking, r/homelab, r/datacenter) | Reddit API via PRAW | Free | `praw` |
| **NANOG mailing list** | No API (scrape archives) | Free | `requests` + `beautifulsoup4` |
| **LinkedIn** | No public search API | N/A | N/A |
#### Reddit via PRAW
- Free Reddit API access (60 req/min)
- Search subreddits by keyword, filter by time
- Count posts + comments mentioning technology terms
- **PRAWtools** provides keyword alerts and subreddit statistics
- Limitation: 1,000 post search window
#### NANOG Mailing List
- Archives available at `nanog.org/nanog-mailing-list/list-archives/` and `marc.info`
- Monthly text file downloads available
- ETH Zurich thesis (Gehri 2021) demonstrated NLP topic modeling and sentiment analysis on 89,000+ NANOG emails
- No API — requires scraping or bulk download
- Highly relevant for optical networking technology adoption signals
#### Metrics
1. **Post/email count per technology per month**
2. **Engagement ratio** (comments/votes per post)
3. **Sentiment** (positive deployment reports vs. complaints)
4. **Question vs. statement ratio** (questions = early adoption; statements = maturity)
#### Implementation Complexity: 3/5
---
### 9. EARNINGS CALL / FINANCIAL SIGNALS (Enterprise Adoption Signal)
**What it measures:** How often public companies mention technologies in financial disclosures
**Hype cycle relevance:** Earnings call mentions are a LAGGING indicator that confirms Slope of Enlightenment → Plateau transition.
#### Data Source A: SEC EDGAR EFTS (VALIDATED — working, 899 filings found)
| Attribute | Detail |
|-----------|--------|
| **API URL** | `https://efts.sec.gov/LATEST/search-index` |
| **Auth** | None (free public API) |
| **Rate Limit** | ~10 requests/second (fair use) |
| **Update Frequency** | Real-time (new filings indexed immediately) |
| **Cost** | Free |
| **Coverage** | All SEC filings since ~1993 |
| **Python Library** | `requests` (direct) or `sec-api` (paid wrapper) |
| **Implementation Complexity** | 2/5 |
**Validated result:** Query for `"optical transceiver" OR "400G" OR "800G optics"` returned **899 filings** across 10-K, 10-Q, and 8-K forms.
#### Data Source B: Financial Modeling Prep (FMP)
| Attribute | Detail |
|-----------|--------|
| **API URL** | `https://financialmodelingprep.com/api/v3/earning_call_transcript/{SYMBOL}` |
| **Auth** | API key (free tier available) |
| **Cost** | Free tier, paid plans from $29/month |
| **Coverage** | Full earnings call transcripts for public companies |
| **Python Library** | `requests` |
| **Implementation Complexity** | 2/5 |
#### Target Companies for Optical Transceiver Mentions
| Ticker | Company | Relevance |
|--------|---------|-----------|
| COHR | Coherent Corp (formerly II-VI/Finisar) | Transceiver manufacturer |
| LITE | Lumentum | Laser/transceiver manufacturer |
| CSCO | Cisco | Network equipment + transceivers |
| JNPR | Juniper Networks | Network equipment |
| ANET | Arista Networks | Datacenter switching |
| AVGO | Broadcom | Transceiver silicon |
| INTC | Intel (Altera) | Silicon photonics |
| CIEN | Ciena | Coherent optics |
| INFN | Infinera | Coherent optics |
| AAOI | Applied Optoelectronics | Transceiver manufacturer |
#### Metrics
1. **Mention frequency** — count of technology term mentions per earnings call
2. **Mention sentiment** — positive/negative context around mentions
3. **First mention** — when a company first mentions a technology (leading indicator)
4. **Revenue attribution** — when companies break out revenue by technology generation
---
### 10. COMPOSITE SIGNAL ALGORITHM
#### Academic Foundation
**Ren (2015)**: "An Approach for Predicting Hype Cycle Based on Machine Learning" (CEUR-WS Vol-1437, IPAMIN 2015)
- Used SKNN (improved K-Nearest Neighbor) classifier
- Features extracted from paper data and patent data
- Achieved **67.24% precision, 68.46% recall** classifying technologies into 5 hype cycle phases
- Noted accuracy drops in phases 4-5 due to small training samples
**BIMATEM (Manrique-Castillo et al., Scientometrics 2018)**:
- Combines **three data streams**: scientific papers (logistic growth), patents (logistic growth), news (hype-type curve)
- Fits logistic regression to paper/patent counts
- Fits hype-type regression to news counts
- Assigns TRL (Technology Readiness Level) based on curve position
- Applied successfully to additive manufacturing technologies
**Composite Early Warning Index (CEWI) approach** (financial crisis literature):
- Uses PCA to synthesize diverse variables into a single latent factor
- Applicable to combining patent, publication, trends, and market signals
#### Recommended Algorithm: Weighted Multi-Signal Scoring
```
HypeScore(tech, t) = w1 * Patent_Signal(tech, t)
+ w2 * Publication_Signal(tech, t)
+ w3 * Trends_Signal(tech, t)
+ w4 * News_Signal(tech, t)
+ w5 * Vendor_Signal(tech, t)
+ w6 * Standards_Signal(tech, t)
+ w7 * Earnings_Signal(tech, t)
+ w8 * Jobs_Signal(tech, t)
```
#### Signal Time Horizons and Weights
| Signal | Lead/Lag | Suggested Weight | Update Freq |
|--------|----------|-----------------|-------------|
| Patents | Leads by 3-5 years | 0.10 | Quarterly |
| Publications | Leads by 1-3 years | 0.10 | Monthly |
| Google Trends | Real-time | 0.20 | Monthly |
| News Volume | Real-time | 0.10 | Weekly |
| **Vendor Count/Price** | **Real-time** | **0.25** | **Daily** |
| Standards Progress | Leads by 2-4 years | 0.10 | Quarterly |
| Earnings Calls | Lags by 6-12 months | 0.10 | Quarterly |
| Job Postings | Lags by 12-18 months | 0.05 | Monthly |
**Vendor Count/Price gets the highest weight** because it is the only direct market measurement.
#### Phase Classification Approach
1. **Normalize each signal** to 0-100 scale per technology
2. **Calculate rate of change** (first derivative) for each signal
3. **Calculate acceleration** (second derivative) for trend detection
4. **Apply phase classification rules:**
| Phase | Signal Pattern |
|-------|---------------|
| **Technology Trigger** | Patents rising, Publications starting, Trends near zero, Vendors 0-3, Standard in study group |
| **Peak of Inflated Expectations** | Trends peaking, News volume peaking, Publications rising fast, Vendors 3-8, Sentiment highly positive |
| **Trough of Disillusionment** | Trends declining, News declining, Sentiment negative, Vendors may decrease, Publications slowing |
| **Slope of Enlightenment** | Vendors growing steadily, Price CV declining, Earnings mentions increasing, Jobs increasing, Standards published |
| **Plateau of Productivity** | All signals stable, Price CV < 0.2, Vendor count > 30, Publications steady, Standards have amendments |
5. **Optional ML layer:** Train a Random Forest or Gradient Boosting classifier on known technology trajectories (100G, 40G, 10G historical data as training set)
#### Norton-Bass Integration
The composite signal feeds into the Norton-Bass multigenerational diffusion model:
- **p (innovation coefficient)** ← derived from patent/publication velocity
- **q (imitation coefficient)** ← derived from vendor count growth rate + Google Trends
- **M (market potential)** ← derived from addressable port count in deployed switches
- **tau (generation introduction time)** ← derived from IEEE standard publication date
- **Python:** `scipy.optimize.curve_fit` with Bass model function, or `bassmodeldiffusion` package (PyPI)
---
## Prioritized Implementation Plan
### Phase 1: Quick Wins (Week 1-2) — HIGH VALUE, LOW EFFORT
| # | Signal | API | Cost | Complexity | Why First |
|---|--------|-----|------|------------|-----------|
| 1 | **Google Trends** | pytrends | Free | 1/5 | Already validated, immediate hype measurement |
| 2 | **Vendor Count/Price** | Internal DB | Free | 1/5 | Data already being collected by TIP scrapers |
| 3 | **Semantic Scholar** | REST API | Free | 1/5 | Already validated, publication trend curves |
**Deliverable:** Basic hype cycle positioning for all tracked technologies using 3 signals.
### Phase 2: Depth Signals (Week 3-4) — HIGH VALUE, MODERATE EFFORT
| # | Signal | API | Cost | Complexity |
|---|--------|-----|------|------------|
| 4 | **SEC EDGAR EFTS** | REST API | Free | 2/5 |
| 5 | **Standards Progress** | Manual curation | Free | 2/5 |
| 6 | **Trade Press Scraping** | Crawlee (existing) | Free | 2/5 |
**Deliverable:** 6-signal composite with financial and standards validation.
### Phase 3: Extended Signals (Week 5-8) — MODERATE VALUE, HIGHER EFFORT
| # | Signal | API | Cost | Complexity |
|---|--------|-----|------|------------|
| 7 | **USPTO Patents** | PatentsView | Free (need API key) | 2/5 |
| 8 | **Reddit/PRAW** | Reddit API | Free | 3/5 |
| 9 | **Job Postings** | TheirStack/FlyByAPIs | Free tier | 3/5 |
| 10 | **Earnings Transcripts** | FMP | Free tier | 2/5 |
**Deliverable:** Full 10-signal composite with ML phase classifier.
### Phase 4: ML Calibration (Week 9-12)
1. Collect historical data for training technologies (10G, 40G, 100G — known trajectories)
2. Train Random Forest classifier on multi-signal features
3. Validate against known Gartner positioning (where available)
4. Implement Norton-Bass curve fitting with signal-derived parameters
5. Build confidence scoring and uncertainty quantification
---
## Key Python Dependencies
```
# Phase 1
pytrends==4.9.2 # Google Trends
semanticscholar # Paper counts
requests # General HTTP
scipy # Curve fitting (Norton-Bass)
numpy # Numerical
pandas # Data manipulation
# Phase 2
beautifulsoup4 # HTML parsing (trade press)
vaderSentiment # Sentiment analysis
# Phase 3
praw # Reddit API
bassmodeldiffusion # Bass model fitting
# Phase 4
scikit-learn # Random Forest, PCA
xgboost # Gradient boosting
```
---
## Signal Correlation Summary
| Signal | Free? | Real-time? | Validated? | Hype Correlation | Implementation |
|--------|-------|-----------|------------|-----------------|---------------|
| Google Trends | Yes | Yes | YES | HIGH (academic proof) | 1/5 |
| Vendor Count/Price | Yes | Yes | YES (own data) | HIGHEST (direct) | 1/5 |
| Semantic Scholar | Yes | Yes | YES | MODERATE-HIGH | 1/5 |
| SEC EDGAR EFTS | Yes | Yes | YES | MODERATE | 2/5 |
| News/Trade Press | Yes | Weekly | Partial | HIGH | 2/5 |
| Standards Progress | Yes | Quarterly | YES | HIGH (leading) | 2/5 |
| Patents (USPTO) | Yes | Quarterly | Not yet (API key needed) | MODERATE-HIGH | 2/5 |
| Reddit/PRAW | Yes | Daily | Not yet | LOW-MODERATE | 3/5 |
| Job Postings | Free tier | Daily | Not yet | MODERATE | 3/5 |
| Earnings Calls | Free tier | Quarterly | Not yet | MODERATE | 2/5 |
---
## References
### Academic Papers
- Manrique-Castillo et al. (2018). "A bibliometric method for assessing technological maturity: the case of additive manufacturing." *Scientometrics* 117(3).
- Ren, Z. (2015). "An Approach for Predicting Hype Cycle Based on Machine Learning." CEUR-WS Vol-1437.
- Jun, S.P. (2012). "An empirical study of users' hype cycle based on search traffic." *Scientometrics* 91(1), 81-99.
- van Lente, H., Spitters, C., & Peine, A. (2013). "Comparing technological hype cycles." *Technological Forecasting and Social Change* 80(8).
- Gao, L. et al. (2013). "Technology life cycle analysis method based on patent documents." *Technological Forecasting and Social Change*.
- Huang et al. (2022). "Technology life cycle analysis: From the dynamic perspective of patent citation networks." *Technological Forecasting and Social Change*.
- Choi, H. & Varian, H. (2010). "Predicting the Present with Google Trends." SSRN.
- Dedehayir, O. & Steinert, M. (2016). "The hype cycle model: A review and future directions." *Technological Forecasting and Social Change* 108(C).
- Norton, J.A. & Bass, F.M. (1987). "A diffusion theory model of adoption and substitution for successive generations of high-technology products." *Management Science* 33(9).
- Gehri, L. (2021). "NANOG Mailing List Analysis." ETH Zurich Semester Thesis.
### API Documentation
- PatentsView Search API: https://search.patentsview.org/docs/
- Semantic Scholar API: https://api.semanticscholar.org/api-docs
- GDELT DOC API: https://blog.gdeltproject.org/gdelt-doc-2-0-api-debuts/
- SEC EDGAR EFTS: https://efts.sec.gov/LATEST/search-index
- Financial Modeling Prep: https://site.financialmodelingprep.com/developer/docs
- Google Trends (pytrends): https://pypi.org/project/pytrends/
- Reddit (PRAW): https://praw.readthedocs.io/
- IEEE 802.3dj Task Force: https://www.ieee802.org/3/dj/index.html
- OIF Implementation Agreements: https://www.oiforum.com/technical-work/implementation-agreements-ias/
### Python Libraries
- `pytrends`: https://pypi.org/project/pytrends/
- `semanticscholar`: https://pypi.org/project/semanticscholar/
- `gdeltdoc`: https://pypi.org/project/gdeltdoc/
- `praw`: https://pypi.org/project/praw/
- `bassmodeldiffusion`: https://github.com/marmiskarian/bassmodeldiffusion
- `vaderSentiment`: https://pypi.org/project/vaderSentiment/

871
RESEARCH-revenue-lifecycle-prediction.md Normal file
View File

@ -0,0 +1,871 @@
# Revenue Lifecycle Prediction Models for Optical Networking Equipment
**Research Date: 2026-03-28**
**Scope: Optical transceivers, switches, routers — product lifecycle revenue prediction**
---
## Table of Contents
1. [Revenue Lifecycle Prediction Models](#1-revenue-lifecycle-prediction-models)
2. [Historical Data Points for Optical Transceivers](#2-historical-data-points-for-optical-transceivers)
3. [Regional/Country-Level Adoption Differences](#3-regionalcountry-level-adoption-differences)
4. [Conference-to-Market Timeline Analysis](#4-conference-to-market-timeline-analysis)
5. [Switch/Router Refresh Cycles](#5-switchrouter-refresh-cycles)
6. [Predictive Models for Future Products](#6-predictive-models-for-future-products)
7. [Recommended Implementation for TIP](#7-recommended-implementation-for-tip)
---
## 1. Revenue Lifecycle Prediction Models
### 1.1 Bass Diffusion Model (Foundation)
The Bass model (1969) is the foundational framework for technology adoption forecasting.
**Core Equation:**
```
f(t) = (p + q * F(t)) * (1 - F(t))
```
Where:
- `f(t)` = instantaneous rate of adoption at time t (fraction of market potential)
- `F(t)` = cumulative fraction of adopters at time t
- `p` = coefficient of innovation (external influence / "advertising effect")
- `q` = coefficient of imitation (internal influence / "word-of-mouth effect")
**Closed-form cumulative adoption:**
```
F(t) = (1 - exp(-(p+q)*t)) / (1 + (q/p)*exp(-(p+q)*t))
```
**Revenue form (units * price):**
```
R(t) = m * f(t) * P(t)
```
Where `m` = total market potential, `P(t)` = price at time t.
**Typical parameter ranges (telecom/technology):**
- p: 0.01 - 0.03 (innovation coefficient)
- q: 0.2 - 0.4 (imitation coefficient)
- Peak adoption occurs at: t_peak = (1/(p+q)) * ln(q/p)
**Source:** Bass, F.M. (1969). "A New Product Growth for Model Consumer Durables." Management Science, 15(5), 215-227.
- [Bass diffusion model - Wikipedia](https://en.wikipedia.org/wiki/Bass_diffusion_model)
- [GeeksforGeeks explanation](https://www.geeksforgeeks.org/machine-learning/bass-diffusion-model/)
### 1.2 Norton-Bass Multi-Generation Diffusion Model (CRITICAL for TIP)
The Norton-Bass (NB) model (1987) extends Bass to handle **successive technology generations** — exactly the pattern seen in optical transceivers (1G → 10G → 40G → 100G → 400G → 800G → 1.6T).
**Two-Generation Formulation:**
Generation 1 introduced at t=0, Generation 2 at t=τ₂.
```
Units-in-use for G1:
N₁(t) = m₁ * F₁(t) for t < τ₂
N₁(t) = m₁ * F₁(t) * (1 - F₂(t - τ₂)) for t ≥ τ₂
Units-in-use for G2:
N₂(t) = 0 for t < τ₂
N₂(t) = (m₂ + m₁ * F₁(t)) * F₂(t - τ₂) for t ≥ τ₂
```
Where:
- `Fᵢ(t)` = Bass cumulative adoption for generation i
- `mᵢ` = incremental market potential for generation i
- `τ₂` = introduction time of generation 2
**Key finding:** p and q parameters are generally **the same between successive generations** — only market potential (m) changes.
**Three-Generation Extension:**
```
N₁(t) = m₁*F₁(t)*(1-F₂(t-τ₂)) for τ₂ ≤ t < τ₃
N₁(t) = m₁*F₁(t)*(1-F₂(t-τ₂))*(1-F₃(t-τ₃)) for t ≥ τ₃
N₂(t) = (m₂+m₁*F₁(t))*F₂(t-τ₂)*(1-F₃(t-τ₃)) for t ≥ τ₃
N₃(t) = (m₃ + (m₂+m₁*F₁(t))*F₂(t-τ₂) + m₁*F₁(t)*(1-F₂(t-τ₂)))*F₃(t-τ₃)
```
**Source:** Norton, J.A. & Bass, F.M. (1987). "A Diffusion Theory Model of Adoption and Substitution for Successive Generations of High-Technology Products." Management Science, 33(9), 1069-1086.
- [INSEAD working paper](https://sites.insead.edu/facultyresearch/research/doc.cfm?did=49784)
- [Semantic Scholar](https://www.semanticscholar.org/paper/A-diffusion-theory-model-of-adoption-and-for-of-Norton-Bass/a030faf95a67497226b9f00bdaf354e2e95f6ac7)
### 1.3 Generalized Norton-Bass (GNB) Model
Jiang & Jain (2012) extended Norton-Bass to differentiate **leapfrogging** from **switching** — critical for optical transceivers where some data centers skip generations (e.g., skip 40G, go from 10G to 100G).
**Leapfrogging:** Potential adopters skip older generation and directly adopt newer generation.
**Switching:** Existing adopters of older generation migrate to newer generation.
**Two-Generation GNB Formulation:**
```
Leapfrog adoptions of G2:
L₂(t) = m₂ * F₂(t - τ₂)
Switching adoptions from G1 to G2:
S₂(t) = m₁ * F₁(t) * F₂(t - τ₂)
Total G2 units-in-use:
N₂(t) = L₂(t) + S₂(t) = (m₂ + m₁*F₁(t)) * F₂(t - τ₂)
G1 remaining units:
N₁(t) = m₁ * F₁(t) * (1 - F₂(t - τ₂))
```
**Empirical validation (DRAM generations):**
- 4K, 16K, 64K DRAM quarterly shipments 1974-1984
- Adjusted R² values: 0.9853, 0.9707, 0.999
- Of 64K DRAM adoptions: **60% new adopters**, **33% switching from 16K**, rest leapfrogging
**Software:** Available in R via the `diffusion` package (`Nortonbass` function).
**Source:** Jiang, Z. & Jain, D.C. (2012). "A Generalized Norton-Bass Model for Multigeneration Diffusion." Management Science, 58(10), 1887-1897.
- [Full PDF - Iowa State](https://dr.lib.iastate.edu/article/scm_pubs/1026)
- [SSRN](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3112796)
- [INFORMS](https://pubsonline.informs.org/doi/pdf/10.1287/mnsc.1120.1529)
- [R package docs](https://rdrr.io/cran/diffusion/man/Nortonbass.html)
### 1.4 Gompertz Curve for Revenue Lifecycle
The Gompertz curve is particularly effective for modeling the **asymmetric S-curve** of technology market growth, where early adoption accelerates fast but saturation is gradual.
**Formula:**
```
y(t) = K * exp(log(y₀/K) * exp(-α*t))
```
Where:
- `K` = carrying capacity (maximum market size / saturation level)
- `y₀` = initial value
- `α` = growth rate coefficient
- **Inflection point occurs at 36.8% of upper asymptote** (vs. 50% for logistic)
**Alternative parametrization:**
```
y(t) = a * b^(c^t)
```
Where a = upper asymptote, 0 < b < 1, 0 < c < 1.
**Application to semiconductors:** Wally Rhines (Mentor Graphics) demonstrated that the Gompertz curve can determine where particular semiconductor market segments are in their lifecycle by plotting cumulative unit production against the Gompertz S-curve. **By determining the three coefficients early in the cycle, the remainder of the cycle can be predicted.**
**Gompertz vs. Logistic:** When Y is low, Gompertz grows faster; when Y is high, Gompertz grows slower. This asymmetry better matches technology markets where early adoption is driven by innovators (fast) but late-stage saturation is drawn out by laggards.
**Source:**
- [EE Times - Gompertz for semiconductor prediction](https://www.eetimes.com/predicting-semiconductor-industry-growth-drop-the-crystal-ball-and-use-the-gompertz-curve/)
- [Semiengineering - Gompertz model](https://semiengineering.com/mathematic-model-helps-predict-markets-that-will-drive-semiconductor-growth/)
- [FasterCapital - Business growth](https://fastercapital.com/content/Gompertz-Curve--Modeling-Mastery--Using-the-Gompertz-Curve-to-Forecast-Business-Growth.html)
### 1.5 Weibull Distribution for Lifecycle Curves
The Weibull distribution provides a **flexible framework** for modeling both growth and decline phases with varying shapes.
**Lifecycle formulation:**
```
f(t) = (β/η) * (t/η)^(β-1) * exp(-(t/η)^β)
```
Where:
- `β` = shape parameter (β < 1: decreasing failure/decline rate, β > 1: increasing)
- `η` = scale parameter (characteristic life)
A 2019 paper proposes a **two-step Weibull distribution** with four parameters for modeling bimodal product lifecycle diffusion curves — fitting both the rise and fall of product sales.
**Source:** "Using Weibull Distribution for Modeling Bimodal Diffusion Curves: A Naive Framework to Study Product Life Cycle." International Journal of Innovation and Technology Management, 2019.
- [World Scientific](https://www.worldscientific.com/doi/10.1142/S0219877019500500)
- [Technological Forecasting & Social Change - Weibull for tech change](https://www.sciencedirect.com/science/article/abs/pii/0040162580900268)
### 1.6 Revenue Duration Model (Composite)
For TIP, the recommended composite model for a single transceiver generation:
```
Revenue(t) = Units(t) * ASP(t)
Where:
Units(t) = Norton-Bass adoption model (accounts for cannibalization by next gen)
ASP(t) = ASP₀ * exp(-λ*t) (exponential price erosion)
Duration above 50% peak revenue:
Solve for t₁, t₂ where R(t) = 0.5 * R_peak
Duration = t₂ - t₁
```
---
## 2. Historical Data Points for Optical Transceivers
### 2.1 Total Optical Transceiver Market Revenue by Year
| Year | Total Market Revenue | Growth | Source |
|------|---------------------|--------|--------|
| 2019 | ~$7.5-8.0B | Declined | LightCounting (derived) |
| 2020 | ~$8.8-9.3B | +17% | LightCounting |
| 2021 | ~$10.0B+ | +10% | LightCounting milestone |
| 2022 | ~$11.0-11.5B | +14% | LightCounting |
| 2023 | ~$10.7-10.9B | -6% | LightCounting; telecom downturn |
| 2024 | ~$13.6B | Strong rebound | MarketsandMarkets; AI-driven |
| 2025 | ~$23B (projected) | +60%+ | LightCounting Dec 2025 |
**Datacom optical segment specifically:**
- 2024: ~$9B (Cignal AI)
- 2025: >$16B (Cignal AI, +60%)
- 2026: ~$12B high-speed datacom segment projected (Cignal AI, as 800G peaks)
**Sources:**
- [LightCounting newsletter](https://www.lightcounting.com/newsletter/en/december-2025-quarterly-market-update-322)
- [Cignal AI Jan 2025](https://cignal.ai/2025/01/over-20-million-400g-800g-datacom-optical-module-shipments-expected-for-2024/)
- [Cignal AI May 2025](https://cignal.ai/2025/05/800gbe-optics-shipments-to-grow-60-in-2025/)
- [MarketsandMarkets](https://www.marketsandmarkets.com/Market-Reports/optical-transceiver-market-161339599.html)
### 2.2 Generation Lifecycle Timelines
| Generation | Datacom Launch | Peak Revenue Window | Years to Peak | Cycle → Next Gen |
|------------|---------------|--------------------|--------------:|------------------|
| 1G SFP | ~2002 | ~2008-2012 | ~6-8 yrs | ~5 yrs |
| 10G SFP+ | ~2007-2010 | ~2013-2016 | ~4-6 yrs | ~4 yrs |
| 40G QSFP+ | ~2011-2013 | ~2015-2017 | ~3-4 yrs | ~3 yrs (largely skipped) |
| 100G QSFP28 | ~2014 | ~2018-2020 | ~4 yrs | ~3-4 yrs |
| 400G QSFP-DD | ~2018-2019 | ~2022-2024 | ~3-4 yrs | ~3 yrs |
| 800G OSFP | ~2023-2024 | ~2025-2026 (proj) | ~2-3 yrs | ~2 yrs |
| 1.6T OSFP-XD | ~2025-2026 | ~2027-2028 (proj) | ~2 yrs | ~2 yrs |
**KEY FINDING: Innovation cycles are compressing from 3-4 years historically to ~2 years currently.**
**Sources:**
- [Introl blog](https://introl.com/blog/fiber-optics-data-center-state-of-art-optical-interconnect-2025)
- [Cignal AI](https://cignal.ai/2025/05/800gbe-optics-shipments-to-grow-60-in-2025/)
- [Medium - Module Evolution](https://medium.com/@aicplight888/optical-module-evolution-from-400g-to-3-2t-11b087f43c04)
### 2.3 Price Erosion Curves
#### 100G QSFP28 SR4 Price History
| Period | Approx. ASP | Notes |
|--------|------------:|-------|
| 2015-2016 | >$2,000 | Early production, few suppliers |
| 2017 | ~$800-$1,200 | Volume ramp begins |
| 2018 | ~$400-$700 | Chinese suppliers enter |
| 2019 | ~$200-$400 | Commoditization |
| 2020 | ~$100-$250 | COVID demand + continued pressure |
| 2021-2022 | ~$80-$150 | Mature market |
| 2024-2026 | ~$29-$99 | Third-party vendors (FS.com, Optcore) |
**Overall decline:** ~60% in 5 years, ~95%+ from launch to commodity phase.
**Price erosion model:**
```
ASP(t) = ASP₀ * exp(-λ*t)
For 100G QSFP28:
ASP₀ ≈ $2,000 (launch year 2015)
λ ≈ 0.35-0.40 per year (aggressive phase)
Half-life: ~2 years
```
#### 800G Module Pricing (2024)
| Module Type | ASP (2024) |
|-------------|----------:|
| 800G Multimode (SR8, VCSEL) | ~$500 |
| 800G LPO | ~$600 |
| 800G Single-mode (EML) | >$700 |
| NVIDIA LinkX 800G (bulk) | ~$1,000 |
| 800G FR4/DR8 (reseller) | $1,000-$3,800 |
#### 1.6T Module Pricing
| Period | ASP |
|--------|----:|
| Q4 2024 (initial) | ~$2,000 |
| 2025 (maturity) | ~$1,500 (projected) |
**Sources:**
- [Deep Fundamental - Optical Module Market](https://deepfundamental.substack.com/p/deep-dive-optical-module-market)
- [Approved Networks](https://approvednetworks.com/blog/a-look-ahead-2024-optical-transceiver-market-trends/)
- [FS.com](https://www.fs.com/c/100g-qsfp28-sfp-dd-1159)
### 2.4 Shipment Volumes
| Year | 400G+800G Units | 800G Alone | 1.6T |
|------|----------------:|----------:|-----:|
| 2022 | ~5M (est.) | Early | — |
| 2023 | ~8M (est.) | Ramp | — |
| 2024 | >20M | ~10M | ~300K (Q4) |
| 2025 | — | 12-15M (proj) | 2-6M (proj) |
**GPU-to-module ratio:** 1 H100 = 2.5x 800G modules (training); 1 B200 = 2.5x 1.6T modules.
**Sources:**
- [Cignal AI](https://cignal.ai/2025/01/over-20-million-400g-800g-datacom-optical-module-shipments-expected-for-2024/)
- [Deep Fundamental](https://deepfundamental.substack.com/p/deep-dive-optical-module-market)
### 2.5 400G ZR Coherent Timeline (Case Study)
| Milestone | Date | Volume |
|-----------|------|--------|
| OIF 400ZR spec finalized | ~2019-2020 | — |
| First commercial shipments | Late 2021 | >60,000 units |
| OFC 2022 demos / volume ramp | 2022 | ~190,000 units |
| Mass deployment (hyperscale + telco) | 2023-2024 | Bulk of WDM bandwidth |
| 800G ZR GA announced | March 2025 | Next gen arriving |
**Timeline: Spec → first shipment: ~18-24 months. First shipment → volume: ~12 months. Total spec → volume: ~30-36 months.**
**Sources:**
- [FiberMall](https://www.fibermall.com/blog/400g-zr-sell-well-800g-transceiver-standardized.htm)
- [Coherent 800G ZR announcement](https://www.globenewswire.com/news-release/2025/03/28/3051358/11543/en/Coherent-Announces-General-Availability-of-800G-ZR-ZR-QSFP-DD-Transceiver.html)
- [PrecisionOT](https://www.precisionot.com/400gzr_systems_engineering/)
---
## 3. Regional/Country-Level Adoption Differences
### 3.1 Adoption Tier Framework
Based on research findings, optical transceiver adoption follows a tiered geographic pattern:
| Tier | Region | Adoption Lag | Primary Drivers |
|------|--------|-------------|-----------------|
| **Tier 1** | US Hyperscalers (Google, Meta, Amazon, MS) | Reference (0 months) | AI training, scale-out DC |
| **Tier 1B** | Chinese Hyperscalers (Alibaba, Tencent, ByteDance) | 6-12 months | Domestic manufacturing, export controls |
| **Tier 2** | Japan/Korea (NTT, SK Telecom) | 12-18 months | Early coherent, methodical qualification |
| **Tier 3** | European Telcos (DT, Orange, Telefonica) | 24-36 months | Regulatory, longer procurement cycles |
| **Tier 4** | India/SEA/LATAM | 36-60 months | Infrastructure buildout, cost sensitivity |
### 3.2 US Hyperscalers (Tier 1)
- **Lead adopters** for every generation — first to deploy at scale.
- Google's hyperscale DCs have deployed optical circuit switching at massive scale.
- NVIDIA/Meta/Google driving LPO adoption: >40% of short-reach 800G links by late 2025.
- NVIDIA's bulk 800G LinkX price: ~$1,000/transceiver at 100K+ volumes.
- 92% of 2025 hyperscale DC contracts specify OSFP-XD for 1.6T.
**Source:** [Hector Weyl blog](https://www.hectorweyl.com/blogs/blog/the-ai-driven-revolution-in-optical-networking-powering-the-next-era-of-high-speed-energy-efficient-connectivity)
### 3.3 Chinese Market (Tier 1B)
- **Manufacturing dominance:** Chinese manufacturers (Innolight, Eoptolink, Accelink) hold ~60% of merchant 800G market share.
- Innolight: ~40% global 800G share; >50% of NVIDIA procurement.
- Eoptolink: ~20% of NVIDIA's 800G LPO orders.
- **Critical vulnerability:** Chinese vendors remain dependent on US silicon — 5nm/3nm DSPs sourced almost exclusively from Broadcom and Marvell.
- Current export restrictions target compute chips, NOT networking signal processors — but this could change.
- Tencent was first deployer of Broadcom Humboldt CPO (2021).
- Accelink upgraded 1.6T OSFP224 at OFC 2025; Eoptolink launched Gen2 1.6T at OFC 2025.
- Asia-Pacific holds 30% of optical interconnect market share (fastest-growing region).
**Source:** [Substack - Pluggables, Power, and Geopolitics](https://iamfabian.substack.com/p/pluggables-power-and-geopolitics)
### 3.4 Europe (Tier 3)
- European presence focuses on **equipment vendors** (Ciena, Nokia) rather than hyperscale deployments.
- Ciena active in hyper-rail photonics, 1600ZR/ZR+ pluggables (acquired Nubis Communications).
- European telcos typically 2-3 years behind hyperscalers in adopting new transceiver generations.
- Regulatory and procurement cycle overhead extends adoption timelines.
### 3.5 Bass Model with Geographic Heterogeneity
Academic research confirms that Bass model parameters vary significantly across countries:
**Key findings:**
- Multi-country diffusion modeling helps overcome the "data hunger" problem — use earlier-adopting countries' data to predict later-adopting ones.
- BRIC mobile adoption study: India's `q` value was much higher than other BRIC countries.
- European broadband study: Bass model parameters for OECD countries showed peak adoption has already passed.
- 3G mobile across 35 countries: NLMIXED approach with pooled multi-country data.
**Recommended approach for TIP:**
```
For each region r:
F_r(t) = Bass(p_r, q_r, m_r, t - lag_r)
Where lag_r = geographic adoption lag (months):
US Hyperscaler: lag = 0
China Hyperscaler: lag = 6-12
Japan/Korea: lag = 12-18
Europe Telco: lag = 24-36
India/SEA/LATAM: lag = 36-60
And p_r, q_r may be adjusted per region:
Hyperscalers: higher p (innovation-driven), lower q
Telcos: lower p, higher q (imitation-driven)
Emerging: lower p, lower q, much higher m (larger potential)
```
**Sources:**
- [ScienceDirect - Heterogeneity in diffusion](https://www.sciencedirect.com/science/article/abs/pii/S0040162514000870)
- [ScienceDirect - Broadband diffusion Europe](https://www.sciencedirect.com/science/article/abs/pii/S004016251100134X)
- [Academia.edu - Bass model BRIC](https://www.academia.edu/11437115/Diffusion_of_mobile_communications_Application_of_bass_diffusion_model_to_BRIC_countries)
- [Tandfonline - Agent-based Bass](https://www.tandfonline.com/doi/full/10.1080/13873954.2024.2350244)
---
## 4. Conference-to-Market Timeline Analysis
### 4.1 Standards Pipeline
The typical pipeline from concept to product:
```
OIF electrical interface → IEEE formal standard → MSA form factor spec → Product GA
Typical timing:
OIF spec → IEEE ratification: 12-18 months
MSA spec → first product samples: 6-12 months
First samples → GA shipping: 6-12 months
GA → volume production: 6-12 months
TOTAL: OIF spec → volume production: 30-48 months
```
### 4.2 Historical Conference-to-Market Timelines
#### 400G ZR
| Event | Date |
|-------|------|
| OIF 400ZR spec finalized | ~2020 |
| First commercial shipments | Q4 2021 |
| OFC 2022 demos / ramp | 2022 |
| Volume deployment | 2022-2023 |
| **Spec → volume: ~24-30 months** | |
#### 800G
| Event | Date |
|-------|------|
| 800G Pluggable MSA founded | Sept 2019 |
| MSA PSM8 spec (first 800G pluggable) | 2020 |
| OSFP 800G spec released | June 2021 |
| First shipments | 2023 |
| Volume production | 2024 |
| **MSA founding → volume: ~5 years; Spec → volume: ~3-4 years** | |
#### 1.6T
| Event | Date |
|-------|------|
| OFC 2025 demos (multiple vendors) | April 2025 |
| OFC 2026 demos (400G/lambda DR4) | March 2026 |
| IEEE 802.3dj 200G/lane expected | Mid 2026 |
| Sampling | Late 2025 |
| Production ramp (projected) | Late 2026 |
| Volume deployment | 2027 |
| **Demo → volume: ~24 months** | |
#### 3.2T
| Event | Date |
|-------|------|
| Coherent demos at OFC 2026 | March 2026 |
| Expected arrival | ~2026-2027 (samples) |
| **LightCounting added 3.2T to forecast** | **July 2024** |
### 4.3 Conference-to-Market Formula for TIP
```
T_volume = T_demo + Pipeline_Lag
Where Pipeline_Lag depends on technology maturity:
Incremental (same platform, higher speed):
Pipeline_Lag = 18-24 months
New platform (new form factor, new SerDes):
Pipeline_Lag = 30-36 months
Paradigm shift (CPO, new physics):
Pipeline_Lag = 48-60 months
```
**Key signals to monitor:**
1. OIF electrical interface spec release → 30-48 months to volume
2. MSA spec release → 24-36 months to volume
3. IEEE standard ratification → 12-24 months to volume (spec often trails products)
4. Multiple vendors demoing at OFC/ECOC → 18-24 months to volume
5. LightCounting adding category to forecast → 24-30 months to volume
**Sources:**
- [LPO MSA](https://www.lpo-msa.org/news/lpo-msa-announces-release-of-specification-for-linear-pluggable-optica)
- [IEEE 802.3](https://en.wikipedia.org/wiki/IEEE_802.3)
- [FS.com MSA intro](https://community.fs.com/article/how-much-do-you-know-about-msa-standard.html)
- [Eoptolink OFC 2026](https://www.prnewswire.com/news-releases/eoptolink-demos-imdd-400g-per-lambda-based-1-6t-dr4-optical-transceiver-solution-at-ofc-2026--302712390.html)
- [EDN - OFC 2025 1.6T innovations](https://www.edn.com/ofc-2025-unveils-1-6t-networking-innovations/)
- [Coherent 1.6T at OFC 2025](https://www.globenewswire.com/news-release/2025/04/01/3053470/11543/en/Coherent-Demonstrates-1-6T-Optical-Transceivers-Based-on-200G-VCSELs.html)
---
## 5. Switch/Router Refresh Cycles
### 5.1 Broadcom Tomahawk ASIC Timeline (Sets Industry Cadence)
| Gen | Year | Bandwidth | Process | Key Optics |
|-----|------|-----------|---------|------------|
| TH1 | 2014 | 3.2 Tb/s | 28nm | 10G/25G |
| TH2 | 2016 | 6.4 Tb/s | 16nm | 25G/50G |
| TH3 | 2017-18 | 12.8 Tb/s | 16nm | 50G/100G |
| TH4 | 2019-20 | 25.6 Tb/s | 7nm | 100G/400G |
| TH5 | 2022 | 51.2 Tb/s | 5nm | 400G/800G |
| TH6 | 2025 | 102.4 Tb/s | 3nm | 800G/1.6T |
| TH7 | ~2027 | 204.8 Tb/s | (planned) | 1.6T/3.2T |
| TH8 | ~2029 | 409.6 Tb/s | (planned) | 3.2T+ |
**Cadence: Bandwidth doubles every ~2 years.** A single TH5 replaces 48 TH1 switches (95% power reduction).
**CRITICAL:** Pluggable optics consume ~50% of system power and >50% of system cost.
**Sources:**
- [Broadcom TH5](https://investors.broadcom.com/news-releases/news-release-details/broadcom-ships-tomahawk-5-industrys-highest-bandwidth-switch)
- [Broadcom TH6 launch](https://www.broadcom.com/company/news/product-releases/63146)
- [TechInsights - TH5](https://www.techinsights.com/blog/tomahawk-5-switches-512tbps)
- [NextPlatform - TH6 102.4T](https://www.nextplatform.com/2025/06/03/the-ai-datacenter-is-ravenous-for-102-4-tb-sec-ethernet/)
- [ServeTheHome - TH6](https://www.servethehome.com/broadcom-tomahawk-6-launched-for-1-6tbe-generation/)
- [NADDOD - TH6](https://www.naddod.com/blog/broadcom-tomahawk-6-102-4-t-ethernet-switch-chip-for-ai-fabrics)
### 5.2 Cisco Nexus Refresh Cycle
| Platform | Generation | Release | Optics Support |
|----------|-----------|---------|----------------|
| Nexus 9364C | Cloud Scale | ~2018-2019 | 100G/400G |
| Nexus 9364D-GX2A | Current gen | May 2022 | 400G |
| Nexus 9364C-H1 | Updated | April 2024 | 400G |
| Nexus 9364E variants | Next gen | Feb 2025 | 800G |
| Nexus 9364C (EOL) | — | EOS Aug 2023 | Support ends Jan 2029 |
**Refresh cycle: ~2-3 years per platform generation.**
**Source:** [Cisco Nexus 9000 series](https://www.cisco.com/c/en/us/support/switches/nexus-9000-series-switches/series.html)
### 5.3 Arista Refresh Cycle
| Platform | ASIC | Timeline |
|----------|------|----------|
| 7800R3 | Jericho 2 | Prior gen |
| 7800R4 | Jericho 3-AI/3+ | Shipping 2024-2025 |
The 7800R4 supports 1,152x 400G or 576x 800G ports. Existing 7800R3 systems can be upgraded with R4 fabric modules.
**Source:** [Arista 7800R4](https://www.arista.com/en/products/7800r4-series)
### 5.4 NVIDIA Networking
- **Spectrum-X** switches with **ConnectX-7** NICs: current generation for AI clusters.
- ConnectX-8 / Spectrum-4 expected to follow standard ~2-year NVIDIA cadence.
- **Quantum-X800**: 144 ports of 800G CPO (unveiled 2025).
- Each GPU requires **6 pluggable transceivers** consuming 30W each.
- 100K GPU cluster = ~200K transceivers (100K scale-up + 100K scale-out).
- Scaling to 1M GPUs would consume ~180MW in optics alone.
**Source:** [NVIDIA LinkX](https://www.nvidia.com/en-us/networking/interconnect/)
### 5.5 ASIC-to-Transceiver Demand Formula
```
Transceiver_Demand_Surge = f(ASIC_GA + Switch_GA_Lag + Qualification_Lag)
Where:
ASIC_GA: Broadcom ships to OEMs
Switch_GA_Lag: OEM builds switch (+6-12 months)
Qualification_Lag: Customer qualifies transceiver (+3-6 months)
Total: ASIC ship → transceiver demand surge: 9-18 months
Demand magnitude:
Per TH5 switch: 64x 800G transceivers = 64 modules
Per TH6 switch: 64x 1.6T or 128x 800G transceivers
```
---
## 6. Predictive Models for Future Products
### 6.1 3.2T Transceivers
**Signals to watch:**
- Coherent demoed 3.2T pluggable technologies at OFC 2026
- LightCounting added 3.2T to forecasts in July 2024
- IEEE 802.3 expected to start 400G/lane standardization work post-802.3dj
- Broadcom TH7 (204.8T) roadmapped for ~2027
**Predicted timeline:**
- Samples: 2027
- GA: 2028
- Volume: 2029
### 6.2 CPO (Co-Packaged Optics)
**Market forecasts:**
| Source | 2025 | 2026 | 2030+ |
|--------|-----:|-----:|------:|
| Precedence Research | $95M | $124M | $1,055M (2034) |
| Mordor Intelligence | $121M | $165M | $764M (2031) |
| IDTechEx | — | — | $20B+ (2036) |
| LightCounting | — | — | LPO+CPO >$10B (2026) |
**Key milestones:**
- Broadcom Humboldt (1st gen CPO): Jan 2021 (Tencent deployed)
- Broadcom Bailly (TH5 CPO, 51.2T): 2024 — 50K+ shipped in 2025
- Broadcom Davisson (TH6 CPO, 102.4T): 2025 announced
- NVIDIA Quantum-X800: 144x 800G CPO, shipping H2 2025
- IEEE 802.3 CPO at 800G/1.6T ratification: expected late 2027
- **Large-scale CPO deployments: 2028-2030** (Yole Group)
**Impact on pluggable revenue:**
- Pluggables remain majority of DC optical links through the decade (LightCounting).
- CPO captures scale-up (GPU-to-GPU) first; pluggables retain scale-out (DC-to-DC).
- CPO for scale-up is the "killer application."
**Sources:**
- [Precedence Research](https://www.precedenceresearch.com/co-packaged-optics-market)
- [IDTechEx](https://www.idtechex.com/en/research-report/co-packaged-optics-cpo/1138)
- [EDN - CPO in 2026](https://www.edn.com/where-co-packaged-optics-cpo-technology-stands-in-2026/)
- [Lightwaveonline](https://www.lightwaveonline.com/home/article/55265639/ai-fuels-optical-transceiver-and-lpo-cpo-demand)
- [Broadcom CPO](https://investors.broadcom.com/news-releases/news-release-details/broadcom-delivers-industrys-first-512-tbps-co-packaged-optics)
### 6.3 LPO (Linear Pluggable Optics)
**Adoption timeline:**
- 2024: ~few hundred 800G LPO units (NVIDIA primary customer)
- 2025: 1-2M units; >40% of short-reach 800G links in AI DCs by late 2025
- 2027: >8M 1.6T LPO ports expected
- LPO MSA 100G/lane spec finalized: March 2025
- CAGR >35% through 2033
**Power advantage:** 1.6T LPO = ~10W vs. conventional 1.6T = 30W+
**Source:**
- [LPO MSA](https://www.lpo-msa.org/news/lpo-msa-announces-release-of-specification-for-linear-pluggable-optica)
- [Gigalight - LPO & CPO](https://www.gigalight.com/news-events/insights-8540.html)
### 6.4 Silicon Photonics vs. InP Market Share Evolution
| Year | SiPh Share | InP/GaAs Share |
|------|----------:|---------------:|
| 2022 | 24% | 76% |
| 2025 | 30% | 70% |
| 2028 | 44% (projected) | 56% |
| 2030 | 60% (projected) | 40% |
**Driver:** LPO and CPO designs overwhelmingly use SiPh platforms. All LPO/CPO devices (except VCSELs) will be SiPh-based.
**InP retains strategic importance** for: coherent transceivers, high-performance lasers, and vertical integration (Coherent, Lumentum).
**Source:**
- [LightCounting SiPh report](https://www.lightcounting.com/newsletter/en/may-2025-silicon-photonics-linear-drive-pluggable-and-cpo-updated-november-2025-334)
- [EE Times](https://www.eetimes.com/ai-demand-reshapes-optical-connectivity-and-photonics-roadmaps/)
---
## 7. Recommended Implementation for TIP
### 7.1 Core Model: Multi-Generation Norton-Bass with Price Erosion
```typescript
interface TransceiverGeneration {
name: string; // e.g., "100G QSFP28"
speed_gbps: number; // 100, 400, 800, 1600
launch_year: number; // datacom first commercial ship
market_potential_m: number; // total addressable units (millions)
p: number; // innovation coefficient (0.01-0.03)
q: number; // imitation coefficient (0.2-0.4)
asp_launch: number; // ASP at launch ($)
price_decay_lambda: number; // exponential decay rate
form_factor: string; // SFP+, QSFP28, QSFP-DD, OSFP, OSFP-XD
}
// Revenue model for generation i at time t
function generationRevenue(gen: TransceiverGeneration, t: number, nextGen?: TransceiverGeneration): number {
const F_t = bassCumulativeAdoption(gen.p, gen.q, t - gen.launch_year);
// Cannibalization by next generation
let cannibalization = 0;
if (nextGen && t >= nextGen.launch_year) {
const F_next = bassCumulativeAdoption(nextGen.p, nextGen.q, t - nextGen.launch_year);
cannibalization = F_next;
}
const units_in_use = gen.market_potential_m * F_t * (1 - cannibalization);
const asp = gen.asp_launch * Math.exp(-gen.price_decay_lambda * (t - gen.launch_year));
return units_in_use * asp;
}
// Bass cumulative adoption
function bassCumulativeAdoption(p: number, q: number, t: number): number {
if (t < 0) return 0;
return (1 - Math.exp(-(p + q) * t)) / (1 + (q / p) * Math.exp(-(p + q) * t));
}
```
### 7.2 Calibrated Parameters for Known Generations
| Generation | m (M units) | p | q | ASP₀ ($) | λ (decay/yr) | Launch |
|-----------|----------:|----:|----:|--------:|----------:|------:|
| 10G SFP+ | 500 | 0.015 | 0.30 | 500 | 0.25 | 2008 |
| 40G QSFP+ | 100 | 0.010 | 0.25 | 800 | 0.30 | 2012 |
| 100G QSFP28 | 400 | 0.020 | 0.35 | 2000 | 0.38 | 2015 |
| 400G QSFP-DD | 300 | 0.025 | 0.35 | 1500 | 0.35 | 2019 |
| 800G OSFP | 250 | 0.030 | 0.40 | 700 | 0.30 | 2024 |
| 1.6T OSFP-XD | 200 | 0.035 | 0.40 | 2000 | 0.35 | 2026 |
*Note: These are initial estimates to be calibrated against LightCounting/Cignal AI data. Parameters should be fitted using nonlinear least squares on observed shipment data.*
### 7.3 Geographic Revenue Multiplier
```typescript
interface RegionConfig {
name: string;
adoption_lag_months: number;
market_share_pct: number;
p_multiplier: number; // adjust innovation coefficient
q_multiplier: number; // adjust imitation coefficient
}
const REGIONS: RegionConfig[] = [
{ name: "US Hyperscaler", adoption_lag_months: 0, market_share_pct: 35, p_multiplier: 1.5, q_multiplier: 0.8 },
{ name: "China Hyperscaler", adoption_lag_months: 9, market_share_pct: 25, p_multiplier: 1.2, q_multiplier: 1.0 },
{ name: "Japan/Korea", adoption_lag_months: 15, market_share_pct: 10, p_multiplier: 1.0, q_multiplier: 1.1 },
{ name: "Europe Telco", adoption_lag_months: 30, market_share_pct: 15, p_multiplier: 0.7, q_multiplier: 1.2 },
{ name: "India/SEA/LATAM", adoption_lag_months: 48, market_share_pct: 15, p_multiplier: 0.5, q_multiplier: 0.6 },
];
```
### 7.4 Conference Signal Pipeline Tracker
```typescript
interface TechnologySignal {
technology: string;
signal_type: "OIF_SPEC" | "IEEE_STANDARD" | "MSA_SPEC" | "OFC_DEMO" | "ECOC_DEMO" | "LC_FORECAST_ADD" | "FIRST_SHIP" | "VOLUME";
date: Date;
predicted_volume_date: Date; // computed
confidence: number; // 0-1
}
// Pipeline lag by signal type (months to volume production)
const SIGNAL_TO_VOLUME_LAG: Record<string, number> = {
"OIF_SPEC": 36, // 30-42 months
"IEEE_STANDARD": 18, // 12-24 months
"MSA_SPEC": 30, // 24-36 months
"OFC_DEMO": 21, // 18-24 months (multiple vendor demos)
"ECOC_DEMO": 24, // 18-30 months
"LC_FORECAST_ADD": 27, // 24-30 months
"FIRST_SHIP": 12, // 9-15 months
};
```
### 7.5 ASIC Demand Correlation Model
```
Transceiver_Revenue(t) = Σ [Switch_Shipments(ASIC_gen, t - lag) * Ports_Per_Switch * ASP(speed, t)]
Where:
ASIC generations: TH4→TH5→TH6→TH7
lag = 9-18 months (ASIC ship → transceiver surge)
Ports_Per_Switch: 64 (TH5), 64-128 (TH6)
Monitor: Broadcom ASIC announcements as leading indicator
→ OEM switch GA as confirming signal
→ Transceiver qualification as demand signal
```
### 7.6 Key Metrics Dashboard for TIP
For each transceiver generation, TIP should compute and display:
1. **Lifecycle Stage:** {Pre-launch | Ramp | Growth | Peak | Decline | EOL}
2. **Time to Peak Revenue:** Derived from Norton-Bass fit
3. **Current ASP vs. Launch ASP:** Price erosion percentage
4. **Revenue Duration >50% Peak:** How many quarters remaining above half-peak
5. **Cannibalization Index:** What % of market potential is being captured by next gen
6. **Geographic Heatmap:** Adoption stage by region
7. **Leading Indicators:** Conference demos, spec milestones, ASIC launches
### 7.7 Data Sources for Calibration
| Source | Data Type | Access | Cost |
|--------|-----------|--------|------|
| LightCounting | Revenue, shipments, ASP by speed | Subscription | $$$ |
| Cignal AI | Datacom revenue, component market | Subscription | $$$ |
| Dell'Oro | Ethernet switch/router market | Subscription | $$$ |
| Yole Group | SiPh, CPO market forecasts | Reports | $$ |
| IDTechEx | CPO market forecasts | Reports | $$ |
| Broadcom press releases | ASIC launch dates | Free | $0 |
| OFC/ECOC proceedings | Demo tracking | Conference fee | $ |
| IEEE 802.3 minutes | Standards timeline | Free | $0 |
| Company earnings calls | Revenue by segment, guidance | Free (SEC filings) | $0 |
| Innolight/Coherent 10-K | Supplier revenue, growth rates | Free (SEC/CSRC) | $0 |
---
## Appendix A: Key Reference Papers
1. Bass, F.M. (1969). "A New Product Growth for Model Consumer Durables." Management Science.
2. Norton, J.A. & Bass, F.M. (1987). "A Diffusion Theory Model of Adoption and Substitution for Successive Generations of High-Technology Products." Management Science, 33(9).
3. Jiang, Z. & Jain, D.C. (2012). "A Generalized Norton-Bass Model for Multigeneration Diffusion." Management Science, 58(10), 1887-1897.
4. Meade, N. & Islam, T. (2006). "Modelling and forecasting the diffusion of innovation - A 25-year review." International Journal of Forecasting.
5. Tsai, B.H. (2013). "Predicting semiconductor industry growth." Technological Forecasting and Social Change. (Gompertz curve application)
6. Jaafari, A. (2019). "Using Weibull Distribution for Modeling Bimodal Diffusion Curves." Int. J. Innovation and Technology Management.
## Appendix B: All Sources Used
- [Bass diffusion model - Wikipedia](https://en.wikipedia.org/wiki/Bass_diffusion_model)
- [IEEE Xplore - Technology forecasting using Bass model](https://ieeexplore.ieee.org/document/5339534/)
- [GNB Model - INSEAD](https://sites.insead.edu/facultyresearch/research/doc.cfm?did=49784)
- [GNB Model - INFORMS](https://pubsonline.informs.org/doi/pdf/10.1287/mnsc.1120.1529)
- [GNB Model - SSRN](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3112796)
- [GNB Model - Iowa State](https://dr.lib.iastate.edu/article/scm_pubs/1026)
- [R diffusion package](https://rdrr.io/cran/diffusion/man/Nortonbass.html)
- [Heterogeneity in diffusion - ScienceDirect](https://www.sciencedirect.com/science/article/abs/pii/S0040162514000870)
- [Bass model broadband Europe - ScienceDirect](https://www.sciencedirect.com/science/article/abs/pii/S004016251100134X)
- [Bass model BRIC - Academia.edu](https://www.academia.edu/11437115/Diffusion_of_mobile_communications_Application_of_bass_diffusion_model_to_BRIC_countries)
- [Agent-based Bass - Tandfonline](https://www.tandfonline.com/doi/full/10.1080/13873954.2024.2350244)
- [Gompertz for semiconductors - EE Times](https://www.eetimes.com/predicting-semiconductor-industry-growth-drop-the-crystal-ball-and-use-the-gompertz-curve/)
- [Gompertz for semiconductors - Semiengineering](https://semiengineering.com/mathematic-model-helps-predict-markets-that-will-drive-semiconductor-growth/)
- [Weibull for bimodal PLC - World Scientific](https://www.worldscientific.com/doi/10.1142/S0219877019500500)
- [Weibull for tech change - ScienceDirect](https://www.sciencedirect.com/science/article/abs/pii/0040162580900268)
- [MarketsandMarkets - Optical Transceiver](https://www.marketsandmarkets.com/Market-Reports/optical-transceiver-market-161339599.html)
- [Cignal AI - 800G shipments 2025](https://cignal.ai/2025/05/800gbe-optics-shipments-to-grow-60-in-2025/)
- [Cignal AI - 20M 400G/800G 2024](https://cignal.ai/2025/01/over-20-million-400g-800g-datacom-optical-module-shipments-expected-for-2024/)
- [LightCounting - Sales of 800G](https://www.lightcounting.com/newsletter/en/june-2025-quarterly-market-update-332)
- [LightCounting - $23B in 2025](https://www.lightcounting.com/newsletter/en/december-2025-quarterly-market-update-322)
- [LightCounting - Ethernet optics 2024](https://www.lightcounting.com/newsletter/en/september-2024-ethernet-optics-296)
- [LightCounting - Market forecast](https://www.lightcounting.com/newsletter/en/april-2024-market-forecast-289)
- [Coherent - 800G ZR GA](https://www.globenewswire.com/news-release/2025/03/28/3051358/11543/en/Coherent-Announces-General-Availability-of-800G-ZR-ZR-QSFP-DD-Transceiver.html)
- [Coherent - 1.6T VCSELs](https://www.globenewswire.com/news-release/2025/04/01/3053470/11543/en/Coherent-Demonstrates-1-6T-Optical-Transceivers-Based-on-200G-VCSELs.html)
- [Coherent - 3.2T at OFC 2026](https://www.stocktitan.net/news/COHR/coherent-demonstrates-technologies-for-next-generation-pluggable-02zn8msgvh1f.html)
- [Eoptolink - 1.6T DR4 OFC 2026](https://www.prnewswire.com/news-releases/eoptolink-demos-imdd-400g-per-lambda-based-1-6t-dr4-optical-transceiver-solution-at-ofc-2026--302712390.html)
- [PrecisionOT - 400G ZR](https://www.precisionot.com/400gzr_systems_engineering/)
- [Deep Fundamental - Module Market](https://deepfundamental.substack.com/p/deep-dive-optical-module-market)
- [Pluggables Power Geopolitics - Substack](https://iamfabian.substack.com/p/pluggables-power-and-geopolitics)
- [Broadcom TH5](https://investors.broadcom.com/news-releases/news-release-details/broadcom-ships-tomahawk-5-industrys-highest-bandwidth-switch)
- [Broadcom TH6](https://www.broadcom.com/company/news/product-releases/63146)
- [Broadcom TH4](https://investors.broadcom.com/news-releases/news-release-details/broadcom-ships-tomahawk-4-industrys-highest-bandwidth-ethernet)
- [Broadcom CPO](https://investors.broadcom.com/news-releases/news-release-details/broadcom-delivers-industrys-first-512-tbps-co-packaged-optics)
- [TechInsights - TH5](https://www.techinsights.com/blog/tomahawk-5-switches-512tbps)
- [NextPlatform - TH6](https://www.nextplatform.com/2025/06/03/the-ai-datacenter-is-ravenous-for-102-4-tb-sec-ethernet/)
- [NextPlatform - CPO](https://www.nextplatform.com/2025/10/17/the-third-time-will-be-the-charm-for-broadcom-switch-co-packaged-optics/)
- [ServeTheHome - TH6](https://www.servethehome.com/broadcom-tomahawk-6-launched-for-1-6tbe-generation/)
- [Arista 7800R4](https://www.arista.com/en/products/7800r4-series)
- [Cisco Nexus 9000](https://www.cisco.com/c/en/us/support/switches/nexus-9000-series-switches/series.html)
- [NVIDIA LinkX](https://www.nvidia.com/en-us/networking/interconnect/)
- [Precedence Research - CPO](https://www.precedenceresearch.com/co-packaged-optics-market)
- [IDTechEx - CPO](https://www.idtechex.com/en/research-report/co-packaged-optics-cpo/1138)
- [EDN - CPO 2026](https://www.edn.com/where-co-packaged-optics-cpo-technology-stands-in-2026/)
- [Lightwaveonline - LPO CPO](https://www.lightwaveonline.com/home/article/55265639/ai-fuels-optical-transceiver-and-lpo-cpo-demand)
- [LPO MSA](https://www.lpo-msa.org/news/lpo-msa-announces-release-of-specification-for-linear-pluggable-optica)
- [LightCounting - SiPh](https://www.lightcounting.com/newsletter/en/may-2025-silicon-photonics-linear-drive-pluggable-and-cpo-updated-november-2025-334)
- [EE Times - AI reshapes photonics](https://www.eetimes.com/ai-demand-reshapes-optical-connectivity-and-photonics-roadmaps/)
- [Nature Communications - SiPh roadmap](https://www.nature.com/articles/s41467-024-44750-0)
- [AIM Photonics - Commercialization](https://www.aimphotonics.com/news/from-breakthrough-to-market-enabling-the-commercialization-of-photonic-technologies)
- [IEEE 802.3 - Wikipedia](https://en.wikipedia.org/wiki/IEEE_802.3)
- [FS.com - MSA standards](https://community.fs.com/article/how-much-do-you-know-about-msa-standard.html)
- [Hector Weyl - AI optical networking](https://www.hectorweyl.com/blogs/blog/the-ai-driven-revolution-in-optical-networking-powering-the-next-era-of-high-speed-energy-efficient-connectivity)

665
RESEARCH-standards-to-market-timeline-database.md Normal file
View File

@ -0,0 +1,665 @@
# Standards-to-Market Timeline Database: Optical Transceivers
**Date:** 2026-03-28
**For:** Transceiver Intelligence Platform (TIP) — Hype Cycle Engine & Predictive Model
**Sources:** IEEE archives, OFC/ECOC proceedings, LightCounting, Cignal AI, Dell'Oro Group, Gazettabyte, vendor press releases, SemiAnalysis
---
## Table of Contents
1. [IEEE Standard Ratification Dates](#1-ieee-standard-ratification-dates)
2. [Standard → First Product → Mainstream Timeline per Generation](#2-generation-timelines)
3. [Price Decline to Mainstream Levels](#3-price-decline-curves)
4. [OFC/ECOC Demo → Product → Mainstream Pipeline](#4-conference-pipeline)
5. [ASIC/SerDes Availability as Leading Indicators](#5-asic-leading-indicators)
6. [Broadcom, Marvell, Intel ASIC Roadmaps](#6-asic-roadmaps)
7. [Current Status: 800G and 1.6T](#7-current-status)
8. [Consolidated Timeline Database](#8-timeline-database)
9. [Prediction Methodology](#9-prediction-methodology)
---
## 1. IEEE Standard Ratification Dates {#1-ieee-standard-ratification-dates}
### Core Ethernet Physical Layer Standards
| Standard | Speed | Study Group | Task Force | Ratified | Key PHY Types |
|----------|-------|-------------|------------|----------|---------------|
| **802.3z** | 1 Gbps | — | — | **Jun 1998** | 1000BASE-SX, 1000BASE-LX |
| **802.3ae** | 10 Gbps | Nov 1999 | Mar 2000 | **Jun 2002** | 10GBASE-SR, -LR, -ER |
| **802.3ba** | 40/100 Gbps | Nov 2007 | Dec 2008 | **Jun 2010** | 40GBASE-SR4/LR4, 100GBASE-SR10/LR4 |
| **802.3bm** | 40/100 Gbps | — | — | **Feb 2015** | 100GBASE-SR4 (improved MMF) |
| **802.3by** | 25 Gbps | — | — | **Jun 2016** | 25GBASE-SR, 25GBASE-LR |
| **802.3bs** | 200/400 Gbps | Nov 2013 | May 2014 | **Dec 2017** | 200GBASE-DR4, 400GBASE-DR4/FR8/LR8 |
| **802.3cd** | 50/100/200 Gbps | — | — | **Dec 2018** | 50GBASE-SR/FR/LR (single-lane 50G) |
| **802.3ck** | 100/200/400 Gbps | — | — | **Sep 2022** | 100G/lane electrical SerDes |
| **802.3df** | 400/800 Gbps | — | — | **Feb 2024** | 800GBASE-DR8, 400GBASE-DR4-2 |
| **802.3dj** | 200/400/800/1600 Gbps | Nov 2022 | — | **Sep 2026 (target)** | 200G/lane, 1.6TbE (D2.2 WG ballot Sep 2025) |
### OIF Implementation Agreements
| Agreement | Published | Speed | Reach | Significance |
|-----------|-----------|-------|-------|-------------|
| 400ZR | Mar 2020 | 400G | 120km | First pluggable coherent DWDM standard |
| OpenZR+ MSA | May 2020 | 100-400G | 1000+km | Extended coherent reach |
| CEI-112G | 2021 | 112 Gbps/lane | Chip-to-module | Enabled 100G PAM4 interfaces |
| 800ZR | Oct 2024 | 800G | 80-120km | Next-gen pluggable coherent |
| CEI-224G | 2025 (target) | 224 Gbps/lane | Chip-to-module | Enables 200G PAM4 interfaces |
| 1600ZR | 2027+ (projected) | 1.6T | TBD | Future coherent standard |
---
## 2. Standard → First Product → Mainstream Timeline per Generation {#2-generation-timelines}
### 2.1 Complete Generation Timeline Database
#### 1G Ethernet (802.3z)
| Milestone | Date | Lag from Prior |
|-----------|------|---------------|
| IEEE 802.3z ratified | Jun 1998 | — |
| First GBIC modules | 1998-1999 | ~6-12 months |
| SFP MSA published | 2000 | +2 years |
| SFP volume shipments | 2001-2002 | +3-4 years |
| Mainstream enterprise adoption | 2002-2004 | +4-6 years |
| Commodity pricing (<$20) | 2006+ | +8 years |
| **Standard-to-mainstream: ~5 years** | | |
#### 10G Ethernet (802.3ae)
| Milestone | Date | Lag from Prior |
|-----------|------|---------------|
| Study group formed | Nov 1999 | — |
| IEEE 802.3ae ratified | Jun 2002 | +31 months |
| First XENPAK modules ship | 2002-2003 | ~6 months from standard |
| XFP MSA published | 2003-2004 | +12-18 months |
| SFP+ MSA (SFF-8431) published | ~2006 | +4 years from standard |
| First SFP+ volume shipments | 2007-2008 | +5-6 years from standard |
| Mainstream SFP+ adoption | 2009-2010 | +7-8 years from standard |
| Commodity pricing (<$30 for SR) | 2014+ | +12 years from standard |
| **Standard-to-mainstream: ~8 years** (but SFP+ MSA-to-mainstream: ~4 years) | | |
#### 40G Ethernet (802.3ba — 40G portion)
| Milestone | Date | Lag from Prior |
|-----------|------|---------------|
| Study group formed | Nov 2007 | — |
| IEEE 802.3ba ratified | Jun 2010 | +31 months |
| First 40G QSFP+ commercial modules | 2010-2011 | ~6-12 months |
| Volume production begins | 2012-2013 | +2-3 years |
| Mainstream data center adoption | 2013-2015 | +3-5 years |
| Price decline begins (Chinese vendors) | 2015-2016 | +5-6 years |
| **Standard-to-mainstream: ~5 years** | | |
| **Note:** 40G was partially skipped; many went 10G→100G | | |
#### 100G Ethernet (802.3ba — 100G portion, then 802.3bm/QSFP28)
| Milestone | Date | Lag from Prior |
|-----------|------|---------------|
| IEEE 802.3ba ratified (100G) | Jun 2010 | — |
| First CFP 100G modules | 2010-2011 | ~6-12 months |
| QSFP28 MSA published | 2013-2014 | +3-4 years |
| First OFC demos (CWDM4/PSM4 QSFP28) | OFC 2015 | +5 years from standard |
| InnoLight volume QSFP28 shipments | Mar 2017 | +7 years from 802.3ba |
| Market maturity (cost parity with 10G $/Gbps) | 2017-2018 | +7-8 years from 802.3ba |
| Commodity pricing (<$100 SR4) | 2021-2022 | +11-12 years from 802.3ba |
| Ultra-commodity (<$30 from third-party) | 2024-2026 | +14-16 years |
| **QSFP28 MSA-to-mainstream: ~4 years** | | |
#### 200/400G Ethernet (802.3bs)
| Milestone | Date | Lag from Prior |
|-----------|------|---------------|
| Study group formed | Nov 2013 | — |
| IEEE 802.3bs ratified | Dec 2017 | +49 months |
| QSFP-DD MSA Rev 2.0 | Mar 2017 | (preceded standard!) |
| InnoLight 400G OSFP intro at OFC 2017 | Mar 2017 | (preceded standard!) |
| First commercial 400G QSFP-DD/OSFP | 2019-2020 | +2 years from standard |
| Volume production | 2020-2021 | +3-4 years |
| Mainstream DC adoption (>10% ports) | 2021-2022 | +4-5 years |
| Price decline accelerates | 2023-2024 | +6-7 years |
| 400G SR8 prices -50% in one year | End 2023 | +6 years |
| 400G now "mainstream" per Nokia | 2025-2026 | +8 years |
| **Standard-to-mainstream: ~4-5 years** | | |
| **First OFC demo-to-mainstream: ~5 years** | | |
#### 800G Ethernet (802.3ck + 802.3df)
| Milestone | Date | Lag from Prior |
|-----------|------|---------------|
| 802.3ck ratified (100G/lane electrical) | Sep 2022 | Enabler standard |
| Intel first 800G DR8 OSFP sample | OFC 2021 | Pre-standard demo |
| Initial SR8 shipments for AI | 2022 | Pre-802.3df |
| LESSENGERS 800G SR8 volume production | Q4 2023 | Pre-802.3df |
| IEEE 802.3df ratified (800G standard) | Feb 2024 | — |
| Hyper Photonix 800G DR8 GA | May 2024 | +3 months post-standard |
| 800G shipments exceed 1M units | 2023 | Pre-standard |
| Cignal AI: 8M 800GbE modules forecast | 2024 | ~simultaneous with standard |
| 800G surpasses 400G in shipments (first time) | Q4 2023 | Pre-standard |
| 800G mainstream / displacing 400G | 2025 | +1 year post-standard |
| Cignal AI: 12.8M units (60% growth) | 2025 | +1 year |
| **Standard-to-mainstream: ~1 year** (but products shipped pre-standard) | | |
| **First demo-to-mainstream: ~4 years** (OFC 2021 → 2025) | | |
| **KEY INSIGHT:** AI demand pulled 800G deployment ahead of standard ratification | | |
#### 1.6T Ethernet (802.3dj — in progress)
| Milestone | Date | Lag from Prior |
|-----------|------|---------------|
| 802.3dj task force (split from 802.3df) | Nov 2022 | — |
| Eoptolink 1.6T module demo (OSFP-XD) | OFC 2023 | +5 months from TF |
| InnoLight 1.6T OSFP-XD demo | OFC 2024 | +17 months |
| First EML-based 1.6T samples ship | Q4 2024 - Q1 2025 | +25-27 months |
| OFC 2025: Multiple live 1.6T demos | Mar 2025 | +28 months |
| Keysight 224G SerDes interop plugfest | Dec 2025 | +37 months |
| AOI first volume order ($200M+) | Mar 2026 | +40 months |
| OFC 2026: Live multi-vendor 1.6T interop | Mar 2026 | +40 months |
| Broadcom Tomahawk 6 volume (enables 1.6T ports) | Mar 2026 | +40 months |
| IEEE 802.3dj ratification (target) | Sep 2026 | +46 months |
| Dell'Oro: First year of volume 1.6T switch deployment | 2026 | +48 months |
| Volume ramp forecast | H2 2026 | Pre-standard |
| Predicted mainstream (>10% addressable ports) | 2027 | ~+6 months post-standard |
| **PATTERN: Products shipping ~6 months BEFORE standard ratification** | | |
| **First demo-to-volume: ~3 years** (OFC 2023 → H2 2026) | | |
---
## 3. Price Decline to Mainstream Levels {#3-price-decline-curves}
### Price Erosion Model
```
ASP(t) = ASP₀ * exp(-λ*t)
Where:
ASP₀ = launch price
λ = annual price erosion rate
t = years since launch
```
### Historical Price Decline Data
| Generation | Launch ASP | Year 1 | Year 2 | Year 3 | Year 5 | Year 8+ | λ (per year) | Half-life |
|------------|-----------|--------|--------|--------|--------|---------|-------------|-----------|
| **10G SFP+ SR** | ~$500 (2008) | $350 | $200 | $120 | $50 | $15-25 | 0.35-0.40 | ~2 years |
| **40G QSFP+ SR4** | ~$400 (2011) | $300 | $200 | $120 | $50 | $20 | 0.30-0.35 | ~2.2 years |
| **100G QSFP28 SR4** | ~$2,000 (2015) | $1,000 | $500 | $250 | $100 | $30-50 | 0.35-0.40 | ~2 years |
| **400G QSFP-DD DR4** | ~$1,500 (2020) | $800 | $400 | $200 | $150 | — | 0.40-0.45 | ~1.8 years |
| **400G SR8** | ~$600 (2022) | $400 | $200 | — | — | — | 0.50 (aggressive) | ~1.4 years |
| **800G SR8** | ~$800 (2023) | $500 | $300-500 | — | — | — | 0.25-0.30 (early) | ~2.5 years |
| **800G DR8** | ~$2,000 (2024) | $800-1,200 | $500-800 | — | — | — | 0.35 (projected) | ~2 years |
| **1.6T DR8** | ~$2,500 (2025) | $1,500 | — | — | — | — | 0.40 (projected) | ~1.8 years |
### Price Milestone Definitions
| Level | Definition | Typical Timing |
|-------|-----------|---------------|
| **Launch premium** | First 12 months, <5 vendors | ASP₀ |
| **Early volume** | 5-15 vendors, hyperscale deployment | ASP₀ * 0.4-0.6 (Year 2-3) |
| **Mainstream** | 15-30 vendors, enterprise deployment | ASP₀ * 0.1-0.2 (Year 4-6) |
| **Commodity** | 30+ vendors, third-party compatible | ASP₀ * 0.02-0.05 (Year 7+) |
### Key Price Observations (2025-2026)
| Module | Current ASP (2025-2026) | Status |
|--------|------------------------|--------|
| 100G QSFP28 SR4 | $29-$99 | Ultra-commodity |
| 400G DR4 | $150-$250 | Late mainstream, declining |
| 400G SR8 | <$200 | Commodity (50% decline in 2023) |
| 800G SR8 | $300-$500 | Early mainstream |
| 800G DR8 | $500-$800 | Mainstream ramp |
| 800G 2xFR4 | $600-$900 | Premium |
| 800G ZR/ZR+ | $4,000-$6,000 | Early premium |
| 1.6T DR8 | $1,500-$2,500 | Launch premium |
| 400G ZR | $2,000-$3,000 | Mature premium |
### Cost-per-Gbps Trend
| Year | Best $/Gbps (short-reach datacom) | Generation |
|------|-----------------------------------|-----------|
| 2015 | $20/Gbps | 100G QSFP28 launch |
| 2018 | $2-4/Gbps | 100G mainstream |
| 2020 | $3-4/Gbps | 400G launch |
| 2022 | $0.50-1.00/Gbps | 400G mainstream (SiPh) |
| 2024 | $0.50/Gbps | 400G SiPh commodity |
| 2025 | $0.40-0.60/Gbps | 800G early mainstream |
| 2026 (proj.) | $0.30-0.50/Gbps | 800G mainstream |
| 2027 (proj.) | $1.00-1.50/Gbps → declining | 1.6T early volume |
---
## 4. OFC/ECOC Demo → Product → Mainstream Pipeline {#4-conference-pipeline}
### Historical Conference-to-Market Timelines
| Technology | First OFC/ECOC Demo | First Commercial Product | Volume Production | Mainstream Adoption | Demo→Volume | Demo→Mainstream |
|-----------|---------------------|-------------------------|-------------------|---------------------|-------------|-----------------|
| 10G SFP+ | OFC 2006 | 2007-2008 | 2008-2009 | 2009-2010 | 2-3 years | 3-4 years |
| 40G QSFP+ | OFC 2009 | 2010-2011 | 2012-2013 | 2013-2015 | 3-4 years | 4-6 years |
| 100G QSFP28 | OFC 2015 | 2016 | 2017 | 2017-2018 | 2 years | 2-3 years |
| 100G CFP-DCO | OFC 2010 | 2011 | 2012 | 2013-2014 | 2 years | 3-4 years |
| 400G QSFP-DD | OFC 2017 | 2019-2020 | 2020-2021 | 2021-2022 | 3-4 years | 4-5 years |
| 400G ZR | OFC 2019 | H2 2020 | 2021-2022 | 2022-2023 | 2-3 years | 3-4 years |
| 800G DR8 | OFC 2021 | 2022-2023 | 2023-2024 | 2025 | 2-3 years | ~4 years |
| 800G ZR/ZR+ | ECOC 2023 | Q1 2024 (alpha) | 2025 (GA) | 2026 (projected) | 2-3 years | ~3 years |
| 1.6T OSFP-XD | OFC 2023 | Q4 2024 | H2 2026 (projected) | 2027 (projected) | 3 years | ~4 years |
| CPO | OFC 2021 | 2023 (select) | 2027 (projected) | 2029+ (projected) | 6+ years | 8+ years |
### Observed Trend: Acceleration
| Era | Average Demo→Mainstream | Driver |
|-----|------------------------|--------|
| Pre-cloud (2002-2010) | 5-8 years | Enterprise procurement cycles |
| Cloud era (2010-2020) | 3-5 years | Hyperscale demand, Chinese manufacturing |
| AI era (2020-2026) | 2-4 years | NVIDIA demand pull, pre-ordering, LPO |
### OFC/ECOC Signal Taxonomy
| Conference Signal | Meaning | Timeline Implication |
|------------------|---------|---------------------|
| Paper-only presentation | Early research | 3-5 years to product |
| Live demo (single vendor) | Working prototype | 2-3 years to volume |
| Multi-vendor interop demo | Ecosystem ready | 12-18 months to volume |
| Plugfest results announced | Qualification stage | 6-12 months to volume |
| Volume shipping announcement | Production | Already available |
---
## 5. ASIC/SerDes Availability as Leading Indicators {#5-asic-leading-indicators}
### The ASIC Dependency Chain
```
SerDes IP → DSP ASIC tape-out → DSP sampling → Module design-in →
Module qualification → Switch ASIC GA → Switch platform GA →
Transceiver demand ramp → Volume deployment
```
### SerDes Generation Timeline
| SerDes Rate | OIF Spec | First Silicon | Volume Availability | Enabled Speeds |
|-------------|----------|---------------|--------------------:|----------------|
| 25G NRZ | CEI-25G (~2010) | 2011-2012 | 2013-2014 | 100G (4x25G) |
| 56G PAM4 | CEI-56G (~2015) | 2016-2017 | 2018-2019 | 200G (4x50G), 400G (8x50G) |
| 112G PAM4 | CEI-112G (2021) | 2020-2021 | 2022-2023 | 400G (4x100G), 800G (8x100G) |
| 224G PAM4 | CEI-224G (2025 target) | 2024 (sampling) | 2025-2026 | 800G (4x200G), 1.6T (8x200G) |
| 448G PAM4 | TBD (~2028) | ~2027 (projected) | ~2029 (projected) | 1.6T (4x400G), 3.2T (8x400G) |
### ASIC-to-Transceiver Lag (Empirical)
| Transition | Typical Lag | Range | Evidence |
|-----------|-------------|-------|----------|
| **Switch ASIC announcement → First switch GA** | 9-18 months | 6-24 months | Broadcom TH series history |
| **Switch GA → Transceiver demand ramp** | 6-12 months | 3-18 months | Qualification + deployment |
| **DSP ASIC sampling → Module qualification** | 6-9 months | 3-12 months | Design-in cycle |
| **DSP ASIC GA → Module volume production** | 3-6 months | 1-9 months | Shortening with pre-qualification |
| **Complete: ASIC tape-out → Transceiver ecosystem ramp** | 18-30 months | 12-36 months | Combined pipeline |
### The "ASIC Gate" — No Transceiver Ramps Without Switch Support
| Transceiver Speed | Required Switch ASIC | ASIC GA | Transceiver Volume Ramp |
|-------------------|---------------------|---------|------------------------|
| 100G QSFP28 | Broadcom TH1 (3.2T, 32x100G) | Spring 2015 | 2016-2017 |
| 400G QSFP-DD | Broadcom TH3 (12.8T, 32x400G) | Dec 2017 | 2019-2020 |
| 800G OSFP | Broadcom TH5 (51.2T, 64x800G) | Late 2022 | 2023-2024 |
| 1.6T OSFP-XD | Broadcom TH6 (102.4T, 64x1.6T) | Mar 2026 | H2 2026 (projected) |
| 3.2T (future) | TH7 (projected ~204.8T) | ~2028 | ~2029-2030 |
---
## 6. Broadcom, Marvell, Intel ASIC Roadmaps {#6-asic-roadmaps}
### 6.1 Broadcom Switch ASICs (Tomahawk Series)
| ASIC | Bandwidth | Process | Announced | Switch GA | SerDes | Optical Ports |
|------|-----------|---------|-----------|-----------|--------|---------------|
| TH1 | 3.2 Tbps | 28nm | Sep 2014 | Spring 2015 | 25G NRZ | 32x100G |
| TH2 | 6.4 Tbps | 16nm | Oct 2016 | Fall 2017 | 25G NRZ | 64x100G |
| TH3 | 12.8 Tbps | 16nm | Dec 2017 | Dec 2017 | 50G PAM4 | 32x400G |
| TH4 | 25.6 Tbps | 7nm | Dec 2019 | 2020-2021 | 50G PAM4 | 64x400G |
| TH5 | 51.2 Tbps | 5nm | Aug 2022 | Late 2022 | 112G PAM4 | 64x800G |
| TH-Ultra | 51.2 Tbps | 4nm | 2024 | 2024 | 112G PAM4 | 64x800G (AI-optimized) |
| **TH6** | **102.4 Tbps** | **3nm** | **Jun 2025** | **Mar 2026** | **224G PAM4** | **64x1.6T** |
| TH6 Davisson (CPO) | 102.4 Tbps | 3nm | Oct 2025 | Oct 2025 | 224G PAM4 | CPO integrated |
**Cadence:** Bandwidth doubles every ~2 years. Announcement-to-GA: 6-18 months.
### 6.2 Broadcom Optical DSP Roadmap (Sian Family)
| DSP | Process | Speed | Power (1.6T) | Announced | Status (Mar 2026) |
|-----|---------|-------|-------------|-----------|-------------------|
| **Sian** (BCM85822) | 5nm | 200G/lane optical | ~30W | ECOC 2023 (Oct 2023) | Production |
| **Sian2** | 5nm | 200G/lane elec+optical | ~28W | 2024 | Production |
| **Sian2M** | 5nm | 200G/lane MMF | <25W (SR8) | 2024 | Production |
| **Sian3** | 3nm | 200G/lane SMF | <23W | 2025 | Sampling, production Q3 2025 |
| **Taurus** (BCM83640) | 3nm | **400G/lane** | TBD | **Mar 2026** | Announced (first 400G/lane DSP) |
**Key insight:** Taurus (400G/lane) enables future 1.6T in 4-lane and 3.2T in 8-lane configurations. This is the bridge to the 3.2T generation.
### 6.3 Marvell Optical DSP Roadmap
| DSP | Process | Speed | Status (Mar 2026) | Key Feature |
|-----|---------|-------|--------------------|-------------|
| **Orion** | 7nm | 400G/800G | Production (legacy) | Widely deployed |
| **Nova** (MV-CD432) | 5nm | 1.6T (100G elec/200G opt) | GA (Mar 2024) | First 200G/lane 1.6T DSP |
| **Nova 2** | — | 1.6T (200G elec+optical) | Sampling Q2 2024 | Full 200G/lane end-to-end |
| **Ara** | 3nm | 1.6T / 800G | **Mass volume shipping (2025)** | Industry's first 3nm optical DSP |
| **Ara T** | 3nm | 1.6T (transmit-retimed) | **Announced Mar 2026** | Power-optimized for LRO |
| **Ara X** | 3nm | 1.6T (reliability) | **Announced Mar 2026** | Advanced link reliability |
| **Petra** | 3nm | Gearbox (8x100G→4x200G) | **Announced Mar 2026** | Bridge chip |
| **Aquila M** | 3nm | O-band coherent-lite | **Announced Mar 2026** | Integrated MACsec |
| **Electra** | **2nm** | 1.6T ZR/ZR+ coherent | **Sampling H2 2026** | Industry-first 2nm coherent DSP |
| **Libra** | 2nm | 800G ZR/ZR+ coherent | **Sampling H2 2026** | Next-gen coherent |
**Key insight:** Marvell Ara (3nm) is already in mass volume. Marvell is 6-12 months ahead of Broadcom on 1.6T DSP availability, but Broadcom counters with the Taurus 400G/lane roadmap.
### 6.4 Intel Silicon Photonics
| Product | Speed | Status | Significance |
|---------|-------|--------|-------------|
| Intel SiPh 100G PSM4 | 100G | Production (since ~2016) | Pioneered SiPh transceivers |
| Intel 800G DR8 OSFP (first sample) | 800G | OFC 2021 demo | First 800G DR8 in the industry |
| Intel SiPh engines (sold to Jabil, ATOP) | 100G-1.6T | Active | Platform licensing model |
| Intel Tofino 3 (switching ASIC) | — | **CANCELLED Jan 2023** | Intel exited switching ASICs |
**Key insight:** Intel's role has shifted from integrated products to SiPh engine licensing. Jabil's 1.6T module (OFC 2025) uses Intel SiPh technology.
### 6.5 Other Key ASIC Players
| Company | Products | Role | Status |
|---------|----------|------|--------|
| **Semtech** | GN8234 redriver, GN1834D TIA, GN187N1 driver | Analog components for LPO/FRO | Live demos at OFC 2026 |
| **Synopsys** | 224G SerDes IP | IP licensing to ASIC makers | Leading IP provider |
| **Credo** | HiWire active cables, line card DSPs | Active cable/retimer market | Shipping 112G, developing 224G |
| **MediaTek** | 224G SerDes (for Google TPU v8e) | Custom ASIC SerDes | Broke into Google ecosystem |
| **NVIDIA** | ConnectX-8/9 NICs, NVLink SerDes | Network adapter ASICs | CX-8 (800G) production Q2 2025 |
### 6.6 ASIC Predictive Signal Summary
| Signal | What It Predicts | Lead Time |
|--------|-----------------|-----------|
| SerDes IP announcement | New speed tier feasibility | 3-5 years before volume |
| DSP ASIC tape-out | Module design starts | 18-24 months before volume |
| DSP sampling to module vendors | Module prototypes in 6 months | 12-18 months before volume |
| Switch ASIC GA | Port demand imminent | 6-12 months before transceiver ramp |
| NIC ASIC GA (ConnectX-N) | Server-side demand confirmed | 3-6 months before optics ramp |
| Multi-vendor plugfest success | Ecosystem validated | 6-12 months before mainstream |
---
## 7. Current Status: 800G and 1.6T {#7-current-status}
### 7.1 800G Status (March 2026)
| Metric | Value | Source |
|--------|-------|--------|
| **Phase** | Late Slope of Enlightenment / early Plateau | Hype cycle analysis |
| IEEE standard | 802.3df ratified Feb 2024 | IEEE |
| Units shipped (2024) | ~8-10M | Cignal AI |
| Units forecast (2025) | ~12.8M (+60% YoY) | Cignal AI |
| Units forecast (2026) | ~20M+ | Industry estimates |
| ASP trend | $300-800 depending on reach | Declining |
| Vendor count | 30+ active vendors | Market data |
| Form factors | QSFP-DD800, OSFP | Both mature |
| DSP ecosystem | Broadcom Sian family, Marvell Orion/Ara | Fully available |
| Switch support | TH5, TH-Ultra, Spectrum-4, Silicon One G200 | Multiple platforms |
| 800G ZR/ZR+ units (2026 forecast) | >200K, >$1B revenue | Cignal AI |
| **Assessment: 800G is mainstream for AI backend and rapidly commoditizing for datacom** | | |
### 7.2 1.6T Status (March 2026)
| Metric | Value | Source |
|--------|-------|--------|
| **Phase** | Peak of Inflated Expectations / early Slope | Hype cycle analysis |
| IEEE standard | 802.3dj D2.2 (WG ballot), target Sep 2026 | IEEE |
| Units shipped (2025) | <1M (select NVIDIA/hyperscale) | Industry estimates |
| First volume orders | AOI $200M+ (Mar 2026) | Press release |
| Dell'Oro forecast | First year of volume 1.6T switches in 2026 | Dell'Oro Group |
| Dell'Oro forecast | >5M ports within 1-2 years of first shipments | Dell'Oro Group |
| ASP | $1,500-$2,500 (DR8) | Market data |
| Vendor count | 10-15 with demos/samples | Growing rapidly |
| Form factors | OSFP-XD (16x100G), OSFP1600 (8x200G), QSFP-DD1600 | Gen1 → Gen2 transition |
| DSP ecosystem | Marvell Ara (mass volume), Broadcom Sian2/3, Semtech | Available |
| Switch support | Broadcom TH6 (GA Mar 2026), NVIDIA Spectrum-X | Just becoming available |
| NIC support | NVIDIA ConnectX-8 (production Q2 2025) | Available |
| OFC 2026 demos | Multi-vendor live interop (FRO, LRO, LPO) | Ecosystem validated |
| 224G SerDes plugfest | Dec 2025 at Keysight | Passed |
| **Assessment: 1.6T transitioning from demos to volume. H2 2026 = inflection point.** | | |
### 7.3 Future: 3.2T and Beyond
| Metric | Value |
|--------|-------|
| **Phase** | Technology Trigger / Pre-commercial |
| First demos | Semtech showed 3.2T ACC at OFC 2026 (448G/channel) |
| Standard | No IEEE task force yet; OIF/MSA discussions |
| ASIC dependency | 448G SerDes (~2027-2028), next-gen switch ASIC (~TH7, 2028) |
| Projected first samples | 2027-2028 |
| Projected volume | 2029-2030 |
| Projected mainstream | 2030-2031 |
| CPO relevance | At 3.2T, CPO may capture 15-30% of market |
---
## 8. Consolidated Timeline Database {#8-timeline-database}
### Master Timeline: All Generations
| Gen | Standard | Ratified | First Demo | First Product | Volume | Mainstream | Commodity | Standard→Mainstream | Demo→Mainstream |
|-----|----------|----------|------------|---------------|--------|------------|-----------|--------------------:|----------------:|
| 1G | 802.3z | 1998 | ~1997 | 1998 | 2001 | 2002-2004 | 2006 | **5 yrs** | 6 yrs |
| 10G | 802.3ae | Jun 2002 | OFC 2001 | 2002 (XENPAK) | 2007 (SFP+) | 2009-2010 | 2014 | **8 yrs** | 9 yrs |
| 25G | 802.3by | Jun 2016 | OFC 2015 | 2016 | 2018 | 2019-2020 | 2023 | **4 yrs** | 5 yrs |
| 40G | 802.3ba | Jun 2010 | OFC 2009 | 2010-2011 | 2012-2013 | 2013-2015 | 2017 | **5 yrs** | 6 yrs |
| 100G | 802.3ba/bm | Jun 2010 | OFC 2010 | 2011 (CFP) | 2017 (QSFP28) | 2017-2018 | 2022 | **8 yrs** | 8 yrs |
| 200G | 802.3bs | Dec 2017 | OFC 2018 | 2019 | 2020-2021 | 2020-2021 | — | **3 yrs** | 3 yrs |
| 400G | 802.3bs | Dec 2017 | OFC 2017 | 2019-2020 | 2020-2021 | 2021-2022 | 2025-2026 | **4-5 yrs** | 5 yrs |
| 800G | 802.3df | Feb 2024 | OFC 2021 | 2022 | 2023-2024 | 2025 | — | **1 yr** | 4 yrs |
| 1.6T | 802.3dj | Sep 2026* | OFC 2023 | Q4 2024 | H2 2026* | 2027* | — | **1 yr*** | 4 yrs* |
| 3.2T | TBD | ~2029* | OFC 2026 | ~2028* | ~2029-2030* | ~2030-2031* | — | **~1-2 yrs*** | ~4-5 yrs* |
*Projected values
### Key Finding: Cycle Compression
| Era | Standard→Mainstream | Demo→Mainstream | Primary Driver |
|-----|--------------------:|----------------:|---------------|
| **1998-2010 (Enterprise)** | 5-8 years | 6-9 years | Slow enterprise procurement, single-vendor qualification |
| **2010-2020 (Cloud)** | 3-5 years | 3-5 years | Hyperscale demand, Chinese manufacturing capacity |
| **2020-2026 (AI)** | 1-2 years | 3-4 years | AI demand pull, pre-standard deployment, NVIDIA procurement |
| **Trend** | Converging to ~1 year | Stable at ~4 years | Products now ship before standards ratify |
### The "Pre-Standard Deployment" Phenomenon
Starting with 800G, products began shipping **before** standards were ratified. This is driven by:
1. **MSA specs substitute for IEEE** — QSFP-DD and OSFP MSAs provide sufficient interop specs
2. **Hyperscaler procurement power** — Single-vendor qualification bypasses multi-vendor standard need
3. **AI urgency** — GPU cluster buildout cannot wait for IEEE consensus
4. **SerDes maturity** — OIF CEI specs provide electrical interface standardization independently
This means **IEEE standard ratification is becoming a lagging indicator**, not a leading one. The leading indicators are:
1. Switch ASIC availability (e.g., TH6 GA for 1.6T)
2. DSP ASIC availability (e.g., Marvell Ara mass volume for 1.6T)
3. NIC availability (e.g., ConnectX-8 for 800G)
4. Multi-vendor plugfest success
5. First hyperscaler volume order
---
## 9. Prediction Methodology {#9-prediction-methodology}
### 9.1 The TIP Predictive Timeline Formula
For any new transceiver technology, estimate deployment milestones using:
```
T_volume = max(T_switch_asic_ga, T_dsp_ga, T_plugfest) + OFFSET_volume
T_mainstream = T_volume + OFFSET_mainstream(segment)
T_commodity = T_mainstream + OFFSET_commodity
T_standard = T_volume +/- 6 months (no longer gates deployment)
```
#### Offset Tables
**Volume Offset (from ASIC/ecosystem readiness):**
| Technology Type | OFFSET_volume | Confidence |
|----------------|---------------|------------|
| Incremental speed (same form factor) | 3-6 months | +/- 3 mo |
| New form factor | 6-12 months | +/- 6 mo |
| New modulation scheme | 12-18 months | +/- 9 mo |
| New architecture (CPO) | 18-36 months | +/- 12 mo |
**Mainstream Offset (from volume, by segment):**
| Segment | OFFSET_mainstream | Confidence |
|---------|------------------|------------|
| US hyperscaler | 0-6 months | +/- 3 mo |
| China hyperscaler | 6-12 months | +/- 6 mo |
| Japan/Korea telco | 12-18 months | +/- 6 mo |
| Enterprise (US) | 18-36 months | +/- 12 mo |
| European telco | 24-36 months | +/- 12 mo |
| India/SEA/LATAM | 36-60 months | +/- 18 mo |
**Commodity Offset (from mainstream):**
| Speed Class | OFFSET_commodity | Driver |
|------------|-----------------|--------|
| 100G and below | 3-5 years | Many Chinese vendors, SiPh |
| 400G | 3-4 years | Aggressive price erosion |
| 800G | 3-4 years (projected) | AI volume drives fast commoditization |
| 1.6T | 3-4 years (projected) | Following 800G pattern |
### 9.2 Leading Indicator Scoring System
Score each indicator 0-10 to predict how close a technology is to volume deployment:
| Indicator | Score 0 | Score 5 | Score 10 |
|-----------|---------|---------|----------|
| Switch ASIC | Not announced | Sampling | GA and shipping |
| Optical DSP | Concept only | Sampling to vendors | Mass volume |
| NIC support | No plans | Roadmap announced | Production |
| IEEE standard | No study group | Task force active | Published |
| MSA spec | No spec | Draft published | Rev 3.0+ |
| OFC/ECOC demos | Paper only | Single-vendor demo | Multi-vendor interop |
| Plugfest | None | Planned | Completed successfully |
| Volume orders | None | LOIs/pre-orders | $100M+ orders placed |
| Vendor count | 0-2 | 5-10 | 15+ |
| Price trend | Launch premium | Early decline | Aggressive decline |
**Interpretation:**
- Score 0-25: 3+ years from volume
- Score 25-50: 18-36 months from volume
- Score 50-75: 6-18 months from volume
- Score 75-100: Volume imminent or achieved
### 9.3 Current Scores (March 2026)
| Technology | Switch ASIC | DSP | NIC | IEEE | MSA | Demo | Plugfest | Orders | Vendors | Price | **Total** | **Assessment** |
|-----------|:-----------:|:---:|:---:|:----:|:---:|:----:|:--------:|:------:|:-------:|:-----:|:---------:|:--------------|
| **800G** | 10 | 10 | 10 | 10 | 10 | 10 | 10 | 10 | 10 | 8 | **98** | Mainstream |
| **1.6T** | 9 | 9 | 8 | 7 | 8 | 9 | 9 | 8 | 6 | 3 | **76** | Volume imminent |
| **800G ZR** | 10 | 9 | 10 | 10 | 10 | 10 | 10 | 8 | 5 | 3 | **85** | Early mainstream |
| **1.6T ZR** | 5 | 4 | 5 | 2 | 3 | 3 | 2 | 2 | 2 | 1 | **29** | 2-3 years out |
| **3.2T** | 2 | 2 | 1 | 0 | 1 | 3 | 0 | 0 | 1 | 0 | **10** | 4+ years out |
| **CPO (scale-out)** | 7 | 6 | 5 | 3 | 5 | 7 | 5 | 4 | 4 | 2 | **48** | 2-3 years from volume |
### 9.4 Applying the Model: 1.6T Deployment Prediction
**Inputs (March 2026):**
- Switch ASIC: Broadcom TH6 GA Mar 2026 ✓
- DSP: Marvell Ara mass volume ✓, Broadcom Sian3 production Q3 2025 ✓
- NIC: NVIDIA ConnectX-8 production Q2 2025 ✓
- Multi-vendor plugfest: Dec 2025 at Keysight ✓
- First volume order: AOI $200M+ Mar 2026 ✓
- IEEE 802.3dj: Target Sep 2026 (not yet, but MSAs ready)
**Calculation:**
```
T_switch_asic_ga = Mar 2026
T_dsp_ga = Q1 2025 (Marvell Ara)
T_plugfest = Dec 2025
max(all) = Mar 2026
T_volume = Mar 2026 + 3 months = ~Q3 2026
T_mainstream(US hyperscaler) = Q3 2026 + 3 months = ~Q4 2026 / Q1 2027
T_mainstream(China) = Q3 2026 + 9 months = ~Q2 2027
T_mainstream(Enterprise US) = Q3 2026 + 24 months = ~Q3 2028
T_mainstream(Europe) = Q3 2026 + 30 months = ~Q1 2029
T_commodity = Q1 2027 + 3.5 years = ~H2 2030
```
**Confidence: Medium-High** (all ASIC dependencies met, ecosystem validated, volume orders placed)
### 9.5 Norton-Bass Integration
The timeline database feeds Norton-Bass model parameters:
| Parameter | Derivation | Source Signal |
|-----------|-----------|---------------|
| **tau (introduction time)** | T_volume from formula above | ASIC GA + offset |
| **p (innovation coefficient)** | 0.01-0.03 (typical for B2B tech) | Patent/publication velocity |
| **q (imitation coefficient)** | 0.20-0.40 (varies by segment) | Vendor count growth rate + Google Trends |
| **m (market potential)** | Total addressable ports | Switch ASIC ports × hyperscaler CapEx forecast |
| **Price function P(t)** | ASP₀ * exp(-λ*t) | Historical price erosion rates per generation |
### 9.6 Validation Against Historical Generations
| Generation | Model Predicted Mainstream | Actual Mainstream | Error |
|-----------|--------------------------|-------------------|-------|
| 40G QSFP+ | 2014 (TH1 2015 - 1yr) | 2013-2015 | +/- 1 year |
| 100G QSFP28 | 2017 (TH1-based, 100G ports) | 2017-2018 | +/- 0.5 year |
| 400G QSFP-DD | 2021 (TH3 Dec 2017 + 3.5yr) | 2021-2022 | +/- 0.5 year |
| 800G OSFP | 2024-2025 (TH5 late 2022 + 2yr) | 2025 | +/- 0.5 year |
| 1.6T | Q4 2026 - Q1 2027 (TH6 Mar 2026 + 6-12mo) | TBD | — |
---
## Sources
### Standards Bodies
- [IEEE 802.3 Working Group](https://www.ieee802.org/3/)
- [IEEE 802.3dj Task Force](https://www.ieee802.org/3/dj/index.html)
- [OIF Implementation Agreements](https://www.oiforum.com/technical-work/implementation-agreements-ias/)
- [QSFP-DD MSA](http://www.qsfp-dd.com/)
- [OSFP MSA](https://osfpmsa.org/)
### Industry Analysts
- [Cignal AI — 800GbE Optics Shipments to Grow 60% in 2025](https://cignal.ai/2025/05/800gbe-optics-shipments-to-grow-60-in-2025/)
- [Cignal AI — 800G Coherent Pluggable >$1B Revenue in 2026](https://cignal.ai/2025/07/800g-coherent-pluggable-shipments-to-exceed-1b-revenue-in-2026/)
- [LightCounting — AI Creates New Wave in Demand for Optical Transceivers](https://www.lightcounting.com/newsletter/en/january-2025-optics-for-ai-clusters-319)
- [Dell'Oro Group — 1.6T volume switch deployments 2026](https://www.delloro.com/)
- [MarketsandMarkets — Optical Transceiver Market](https://www.marketsandmarkets.com/Market-Reports/optical-transceiver-market-161339599.html)
- [Mordor Intelligence — Optical Transceiver Market](https://www.mordorintelligence.com/industry-reports/optical-transceiver-market)
### Vendor Announcements
- [Broadcom TH6 Volume Shipments](https://www.broadcom.com/company/news/product-releases/63146)
- [Broadcom Sian3 DSP](https://investors.broadcom.com/news-releases/news-release-details/broadcom-delivers-industry-leading-200glane-dsp-gen-ai)
- [Broadcom Taurus 400G/lane DSP](https://www.stocktitan.net/news/AVGO/broadcom-delivers-industry-s-first-400g-lane-optical-dsp-for-next-0ysjo3zlcexv.html)
- [Marvell Ara 1.6T DSP Platform](https://investor.marvell.com/news-events/press-releases/detail/1013/marvell-ushers-in-the-1-6t-era-with-expanded-optical-dsp-platform-portfolio-redefining-ai-data-center-end-to-end-connectivity)
- [Marvell Electra 2nm Coherent DSP](https://www.marvell.com/company/newsroom/marvell-1-6t-zr-zr-plus-pluggable-2nm-coherent-dsp-ai-interconnects.html)
- [Marvell Nova 1.6T DSP](https://www.marvell.com/content/dam/marvell/en/public-collateral/dsp/marvell-nova-1.6t-pam4-dsp-for-optical-transceiver-applications-product-brief.pdf)
- [Semtech 1.6T Demos at OFC 2026](https://www.semtech.com/company/press/showcases-ai-interconnect-leadership-with-live-1.6t-demos-ofc-2026)
- [NVIDIA ConnectX-8 SuperNIC](https://www.servethehome.com/this-is-the-next-gen-nvidia-connectx-8-supernic-for-800gbps-networking/)
- [Keysight 224G/Lane Test Solutions](https://convergedigest.com/keysight-intros-224g-lane-test-solutions/)
### Conference & Demo Sources
- [InnoLight OFC 2017 — 400G OSFP Introduction](https://www.prnewswire.com/news-releases/innolight-technology-announced-volume-shipments-of-17-100g-qsfp28-products-and-the-introduction-of-400g-osfp-at-ofc-2017-300421866.html)
- [Eoptolink Gen2 1.6T at OFC 2025](https://www.eoptolink.com/news/361-eoptolink-launches-its-gen2-1-6t-osfp-and-osfp-rhs-transceiver-family-at-ofc-2025)
- [Jabil 1.6T Pluggable Transceiver at OFC 2025](https://investors.jabil.com/news/news-details/2025/Jabil-Launches-1-6T-Pluggable-Transceiver/)
- [ATOP 1.6T DR8 SiPh Demo at OFC 2025](https://www.atoptechnology.com/ofc-2025-live-demo-atops-1-6t-osfp224-dr8-siph-module-in-action-for-next-gen-ai/)
### SerDes & ASIC Analysis
- [TrendForce — SerDes Wars: Broadcom, Marvell, MediaTek](https://www.trendforce.com/news/2026/03/13/news-serdes-wars-heat-up-broadcom-marvell-mediatek-battle-for-ai-interconnect-supremacy/)
- [OIF CEI 448G/224G/112G Interoperability Demo OFC 2025](https://www.oiforum.com/wp-content/uploads/OIF_CEI_Demo_OFC2025.pdf)
- [EDN — OFC 2025 1.6T Networking Innovations](https://www.edn.com/ofc-2025-unveils-1-6t-networking-innovations/)
### Market & Pricing
- [Deep Fundamental — Optical Module Market](https://deepfundamental.substack.com/p/deep-dive-optical-module-market)
- [Pluggables, Power, and Geopolitics](https://iamfabian.substack.com/p/pluggables-power-and-geopolitics)
- [Fierce Network — Optical vendors predict higher demand 400G/800G 2026](https://www.fierce-network.com/broadband/optical-transmission-vendors-predict-high-demand-400g-800g-2026)
- [Introl — Fiber Optics State of the Art 2025](https://introl.com/blog/fiber-optics-data-center-state-of-art-optical-interconnect-2025)

232
packages/api/src/hype-cycle/data-enrichment.ts Normal file
View File

@ -0,0 +1,232 @@
/**
* Hype Cycle Data Enrichment — Real metrics from scraped data
*
* Computes PhaseMetrics overrides from actual database observations:
* - vendorCount: How many vendors sell this speed class
* - price trends: ASP decline rate from price_observations
* - catalog density: Number of SKUs per speed class (market maturity signal)
* - product diversity: Form factor and reach variety
*/
import { pool } from "../db/client";
import type { PhaseMetrics } from "./norton-bass";
interface SpeedClassMetrics {
speedGbps: number;
vendorCount: number;
skuCount: number;
avgPrice?: number;
minPrice?: number;
maxPrice?: number;
priceCount: number;
formFactors: string[];
reachVariants: number;
}
/**
* Query real vendor/product counts per speed class from the database.
*/
export async function getSpeedClassMetrics(): Promise<ReadonlyArray<SpeedClassMetrics>> {
const result = await pool.query(`
SELECT
t.speed_gbps,
COUNT(DISTINCT t.vendor_id) AS vendor_count,
COUNT(DISTINCT t.id) AS sku_count,
ARRAY_AGG(DISTINCT t.form_factor) FILTER (WHERE t.form_factor IS NOT NULL AND t.form_factor != '') AS form_factors,
COUNT(DISTINCT t.reach_label) FILTER (WHERE t.reach_label IS NOT NULL AND t.reach_label != '') AS reach_variants
FROM transceivers t
WHERE t.speed_gbps > 0
GROUP BY t.speed_gbps
ORDER BY t.speed_gbps
`);
const priceResult = await pool.query(`
SELECT
t.speed_gbps,
AVG(po.price) AS avg_price,
MIN(po.price) AS min_price,
MAX(po.price) AS max_price,
COUNT(*) AS price_count
FROM price_observations po
JOIN transceivers t ON t.id = po.transceiver_id
WHERE t.speed_gbps > 0
GROUP BY t.speed_gbps
`);
const priceMap = new Map<number, { avg: number; min: number; max: number; count: number }>();
for (const row of priceResult.rows) {
priceMap.set(Number(row.speed_gbps), {
avg: parseFloat(row.avg_price),
min: parseFloat(row.min_price),
max: parseFloat(row.max_price),
count: parseInt(row.price_count),
});
}
return result.rows.map((row) => {
const speedGbps = Number(row.speed_gbps);
const priceData = priceMap.get(speedGbps);
return {
speedGbps,
vendorCount: parseInt(row.vendor_count),
skuCount: parseInt(row.sku_count),
avgPrice: priceData?.avg,
minPrice: priceData?.min,
maxPrice: priceData?.max,
priceCount: priceData?.count ?? 0,
formFactors: row.form_factors || [],
reachVariants: parseInt(row.reach_variants),
};
});
}
/**
* Convert raw speed-class metrics into PhaseMetrics overrides.
* These override the model-estimated values with real data.
*/
export function metricsToPhaseOverrides(
metrics: SpeedClassMetrics,
totalMarketSkus: number,
): Partial<PhaseMetrics> {
const overrides: Partial<PhaseMetrics> = {};
// Vendor count — direct from data
overrides.vendorCount = metrics.vendorCount;
// Vendor trend — estimate from catalog density
// More SKUs + more vendors = increasing; few = decreasing
if (metrics.vendorCount >= 4 && metrics.skuCount > 50) {
overrides.vendorTrend = "stable";
} else if (metrics.vendorCount >= 2) {
overrides.vendorTrend = "increasing";
} else {
overrides.vendorTrend = "decreasing";
}
// Shipment share proxy — SKU count relative to total market
overrides.shipmentShare = Math.min(0.5, metrics.skuCount / Math.max(1, totalMarketSkus));
// Interop level — more reach variants and form factors = better interop
const ffDiversity = metrics.formFactors.length;
const reachDiversity = metrics.reachVariants;
overrides.interopLevel = Math.min(100, ffDiversity * 15 + reachDiversity * 8);
return overrides;
}
/**
* Get enriched PhaseMetrics for all speed classes.
* Returns a map of speedGbps -> partial PhaseMetrics overrides.
*/
export async function getDataDrivenOverrides(): Promise<Map<number, Partial<PhaseMetrics>>> {
const allMetrics = await getSpeedClassMetrics();
const totalSkus = allMetrics.reduce((sum, m) => sum + m.skuCount, 0);
const overridesMap = new Map<number, Partial<PhaseMetrics>>();
for (const metrics of allMetrics) {
overridesMap.set(metrics.speedGbps, metricsToPhaseOverrides(metrics, totalSkus));
}
return overridesMap;
}
/**
* Revenue lifecycle prediction per speed class.
*
* Uses scraped price data + Bass diffusion to estimate:
* - Peak revenue year
* - Revenue duration (years above 50% of peak)
* - Current revenue trajectory
*/
export interface RevenueLifecycle {
speedGbps: number;
technology: string;
currentAvgPrice?: number;
estimatedPeakRevenueYear: number;
estimatedDeclineStartYear: number;
revenueHalfLifeYears: number;
currentPhase: "growing" | "peaking" | "declining" | "legacy";
revenueIndex: number; // 0-100, relative to estimated peak
}
export function computeRevenueLifecycle(
speedGbps: number,
techName: string,
introYear: number,
peakYear: number,
currentYear: number,
avgPrice?: number,
): RevenueLifecycle {
// Revenue = Price × Volume. Price declines while volume grows.
// Peak revenue happens ~2 years before peak volume (when price×volume is maximized)
const peakRevenueYear = Math.round(peakYear - 2);
const declineStartYear = peakYear + 2;
const halfLife = Math.round((peakYear - introYear) * 0.7);
const yearsFromPeak = currentYear - peakRevenueYear;
let currentPhase: RevenueLifecycle["currentPhase"];
if (currentYear < peakRevenueYear - 2) currentPhase = "growing";
else if (currentYear <= peakRevenueYear + 2) currentPhase = "peaking";
else if (currentYear <= declineStartYear + halfLife) currentPhase = "declining";
else currentPhase = "legacy";
// Revenue index: bell curve centered on peakRevenueYear
const sigma = halfLife / 1.5;
const revenueIndex = Math.round(100 * Math.exp(-0.5 * Math.pow(yearsFromPeak / sigma, 2)));
return {
speedGbps,
technology: techName,
currentAvgPrice: avgPrice,
estimatedPeakRevenueYear: peakRevenueYear,
estimatedDeclineStartYear: declineStartYear,
revenueHalfLifeYears: halfLife,
currentPhase,
revenueIndex,
};
}
/**
* Regional adoption model.
* Applies lag coefficients per region based on industry research.
*/
export interface RegionalAdoption {
region: string;
lagYears: number;
marketSharePct: number;
adoptionPhase: string;
estimatedPeakYear: number;
}
const REGIONAL_LAGS: ReadonlyArray<{ region: string; lagYears: number; marketSharePct: number }> = [
{ region: "North America (Hyperscale)", lagYears: 0, marketSharePct: 35 },
{ region: "China (BAT/Hyperscale)", lagYears: 0.5, marketSharePct: 30 },
{ region: "APAC (ex-China)", lagYears: 1.5, marketSharePct: 15 },
{ region: "Europe", lagYears: 1.0, marketSharePct: 12 },
{ region: "Rest of World", lagYears: 2.5, marketSharePct: 8 },
];
export function computeRegionalAdoption(
techPeakYear: number,
currentYear: number,
techName: string,
): ReadonlyArray<RegionalAdoption> {
return REGIONAL_LAGS.map(({ region, lagYears, marketSharePct }) => {
const regionalPeak = techPeakYear + lagYears;
const yearsToPeak = regionalPeak - currentYear;
let adoptionPhase: string;
if (yearsToPeak > 5) adoptionPhase = "Pre-adoption";
else if (yearsToPeak > 2) adoptionPhase = "Early Adoption";
else if (yearsToPeak > -1) adoptionPhase = "Growth";
else if (yearsToPeak > -4) adoptionPhase = "Mature";
else adoptionPhase = "Declining";
return {
region,
lagYears,
marketSharePct,
adoptionPhase,
estimatedPeakYear: Math.round(regionalPeak * 2) / 2, // Round to half-year
};
});
}

195
packages/api/src/llm/blog-prompts.ts Normal file
View File

@ -0,0 +1,195 @@
/**
* Blog generation prompt templates.
*
* Multi-pass pipeline:
* 1. MASTER_PROMPT — Initial article generation
* 2. DEPTH_PROMPT — Add concrete values, real-world insights
* 3. ANTI_GENERIC_PROMPT — Rewrite intro to be direct and scenario-based
* 4. QUALITY_CONTROL_PROMPT — Final validation pass
*
* Voice: Senior optical network engineer with 10+ years hands-on experience.
* NOT a content writer. NOT marketing. NOT generic AI.
*/
export const SYSTEM_PROMPT = `You are a senior optical network engineer with more than 10 years of hands-on experience in data center and telecom environments.
You write like someone who has debugged real production outages under time pressure. You are direct, pragmatic, and slightly opinionated.
Rules:
- Do not write generic introductions about markets, trends, or industry news.
- Do not include placeholders, notes, or unfinished sections.
- Do not write like marketing or AI.
- Do not repeat obvious textbook explanations.
- Short, clear sentences.
- Focus on what actually breaks in real networks.
For each technical issue, you MUST include:
- Real-world cause based on experience
- Typical numeric values (dBm, BER, OSNR, temperature)
- How to verify the issue (commands, measurements, logs)
- The fastest way to isolate the problem
- One sentence about what engineers usually get wrong
Use concrete ranges and examples:
- Tx power: -8.2 dBm to +0.5 dBm typical for SFP+ SR, alarm below -11.0 dBm
- Rx power: -14.4 dBm to -1.0 dBm nominal, -18.0 dBm is receiver sensitivity floor for 10G SR
- BER: 10^-12 pre-FEC acceptable, 10^-9 post-FEC means line is failing
- OSNR: 28 dB minimum for 100G coherent, below 22 dB = link won't stay up
- Temperature: 0-70°C commercial, -40 to +85°C industrial, alarm above 75°C
- CRC errors: >100/min = dirty fiber, >10000/min = bad optic or wrong fiber type
Include CLI examples where relevant:
show interface transceiver details
show interface counters errors
show ip interface brief`;
export const MASTER_PROMPT = `Write a highly practical troubleshooting guide for optical transceiver issues.
Structure:
1. Start with a real-world failure scenario (e.g.: link down at 2 AM, CRC errors climbing, unstable 400G coherent link)
2. Troubleshooting sections — each MUST be detailed and practical:
- Low transmit power
- High BER or CRC errors
- Coherent (400ZR/ZR+) link issues
- Temperature and environmental problems
- Compatibility and coding mismatches
3. End with:
- Key takeaways (5 bullet points max)
- Common misdiagnoses (3-5 items)
- A short actionable pre-deployment checklist
Each section needs:
- At least one real numeric value (dBm, BER, OSNR, temperature)
- At least one CLI command or measurement step
- One "what engineers usually get wrong" insight
Output a complete, clean article in markdown. No notes, no placeholders, no generic filler.`;
export const DEPTH_PROMPT = `Take the existing article and improve it.
Add concrete numeric values where missing (dBm, BER, OSNR, temperature).
Replace vague statements with specific, practical explanations.
Add at least one real-world insight per section that shows hands-on experience.
Remove any generic or empty phrases.
Specific additions needed:
- For Tx power: specify exact dBm ranges per form factor (SFP+ SR: -8.2 to +0.5 dBm, QSFP28 LR4: -4.3 to +4.5 dBm)
- For BER: differentiate pre-FEC vs post-FEC, explain what "corrected" vs "uncorrected" means in practice
- For coherent: add OSNR requirements per speed (100G: 18 dB, 400G: 24 dB, 400ZR: 28 dB over 80km)
- For temperature: explain why transceivers in top-of-rack position always run hotter
Do not make the text longer unless it adds real technical value.
Preserve the markdown structure.`;
export const ANTI_GENERIC_INTRO_PROMPT = `Rewrite the introduction of this article.
Remove any generic or marketing-style language.
Start directly with a real troubleshooting scenario that the reader will immediately recognize.
Make the reader feel "this person has been in my shoes."
Example of a good opening:
"It's 2 AM. NOC pager goes off. Core spine link between pods is flapping—200G aggregate capacity lost. You SSH into the switch, check the optics, and see Tx power at -14.3 dBm on a module rated for -8.2 to +0.5. The transceiver is dying. Here's how you diagnose this in under 5 minutes."
Do NOT mention market trends, industry news, chip shortages, or any meta-commentary about the transceiver market. Go straight into the problem.`;
export const QUALITY_CONTROL_PROMPT = `Check this article for the following issues and fix ALL of them:
1. Missing numeric values — every technical claim MUST have a number
2. Generic statements like "important to consider", "plays a key role", "increasingly popular"
3. Placeholder text (NOTES, TODO, comments, <!-- -->)
4. Sections without practical troubleshooting steps
5. Marketing language or sales pitches
6. Vague conclusions without actionable advice
For each issue found, rewrite the affected section to fix it.
Return the complete fixed article in markdown.
Quality gates:
- At least 1 numeric value (dBm, BER, OSNR, temperature) per section
- At least 1 CLI command or measurement step per troubleshooting section
- Zero placeholder text
- Zero generic filler phrases
- Introduction starts with a scenario, not with "The optical transceiver market..."`;
/** Optional procurement-focused notes to weave in */
export const PROCUREMENT_LAYER_PROMPT = `Add short procurement-focused notes where relevant.
Explain how misdiagnosed optical issues lead to unnecessary hardware replacement.
Mention cost impact of vendor lock-in in a neutral tone.
Keep each note to one or two sentences only.
Example: "Before RMA'ing a $2,400 QSFP-DD module, clean the fiber end-face. In our experience, 40% of RMA'd optics test perfectly fine at the vendor — the problem was contaminated connectors."
Do not turn this into marketing content. Keep the engineer voice.`;
export function buildTopicPrompt(
topic: string,
data: {
products: Array<Record<string, unknown>>;
news: Array<Record<string, unknown>>;
faq: Array<Record<string, unknown>>;
troubleshooting: Array<Record<string, unknown>>;
},
): string {
const parts: string[] = [];
if (topic === "tutorial") {
parts.push(MASTER_PROMPT);
} else if (topic === "hype_cycle") {
parts.push(`Write an analysis of the current optical transceiver technology lifecycle.
For each technology generation, assess its position on the adoption curve using real market data:
- Which speeds are in early deployment? (Evidence: limited vendor support, high pricing, interop issues)
- Which are mainstream? (Evidence: multi-vendor support, stable pricing, proven deployments)
- Which are declining? (Evidence: EOL notices, shrinking SKU counts, price erosion)
Be specific. Use actual deployment numbers and price points where available.
Do not use the term "hype cycle" — call it "technology adoption lifecycle" or "maturity assessment."
Write for network architects planning 3-5 year infrastructure investments.`);
} else if (topic === "comparison") {
parts.push(`Write a practical comparison guide for optical transceivers.
Focus on real-world decision criteria, not spec sheet comparisons:
- What actually matters when choosing between options (hint: it's not always the cheapest)
- Interoperability gotchas between vendors
- Temperature and power budget surprises
- When "compatible" modules actually cause problems vs. when they work perfectly
Include specific price ranges and performance data from the context provided.`);
} else {
parts.push(`Write a practical technical article about recent developments in optical transceivers.
Focus on what matters for network engineers making deployment decisions.
No fluff, no marketing. Concrete specs, real tradeoffs, practical advice.`);
}
// Append gathered data as context
if (data.products.length > 0) {
parts.push("\n\n--- PRODUCT DATA (use as reference) ---");
for (const p of data.products.slice(0, 10)) {
parts.push(`• ${p.standard_name || p.slug}: ${p.form_factor} ${p.speed}, reach ${p.reach_label || "N/A"}, fiber ${p.fiber_type || "N/A"}, vendor ${p.vendor || "N/A"}`);
}
}
if (data.news.length > 0) {
parts.push("\n\n--- RECENT NEWS (reference if relevant, do not force) ---");
for (const n of data.news.slice(0, 5)) {
parts.push(`• ${n.title} (${n.source || "unknown"})`);
}
}
if (data.troubleshooting.length > 0) {
parts.push("\n\n--- TROUBLESHOOTING DATA (incorporate into article) ---");
for (const t of data.troubleshooting) {
parts.push(`• Symptom: ${t.symptom} | Cause: ${t.cause} | Fix: ${t.solution}`);
}
}
if (data.faq.length > 0) {
parts.push("\n\n--- FAQ DATA (address these questions in the article) ---");
for (const f of data.faq.slice(0, 5)) {
parts.push(`• Q: ${f.question} → A: ${f.answer}`);
}
}
return parts.join("\n");
}

113
packages/api/src/llm/client.ts Normal file
View File

@ -0,0 +1,113 @@
/**
* Ollama LLM client for blog generation and content enhancement.
*
* Uses qwen2.5:14b on Mac Studio (.213) for text generation.
* Supports streaming and non-streaming modes.
*/
const OLLAMA_URL = process.env.OLLAMA_URL || "http://localhost:11434";
const LLM_MODEL = process.env.OLLAMA_LLM_MODEL || "qwen2.5:14b";
interface LlmResponse {
text: string;
model: string;
totalDuration: number;
evalCount: number;
}
/** Generate text from a system prompt + user prompt */
export async function generate(
systemPrompt: string,
userPrompt: string,
options?: { temperature?: number; maxTokens?: number },
): Promise<LlmResponse> {
const resp = await fetch(`${OLLAMA_URL}/api/generate`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
model: LLM_MODEL,
prompt: userPrompt,
system: systemPrompt,
stream: false,
options: {
temperature: options?.temperature ?? 0.7,
num_predict: options?.maxTokens ?? 4096,
},
}),
signal: AbortSignal.timeout(120000),
});
if (!resp.ok) {
const errText = await resp.text();
throw new Error(`Ollama generate failed: ${resp.status} ${errText}`);
}
const data = await resp.json() as {
response: string;
model: string;
total_duration: number;
eval_count: number;
};
return {
text: data.response,
model: data.model,
totalDuration: data.total_duration,
evalCount: data.eval_count,
};
}
/** Chat-style generation with message history */
export async function chat(
messages: ReadonlyArray<{ role: "system" | "user" | "assistant"; content: string }>,
options?: { temperature?: number; maxTokens?: number },
): Promise<LlmResponse> {
const resp = await fetch(`${OLLAMA_URL}/api/chat`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
model: LLM_MODEL,
messages,
stream: false,
options: {
temperature: options?.temperature ?? 0.7,
num_predict: options?.maxTokens ?? 4096,
},
}),
signal: AbortSignal.timeout(120000),
});
if (!resp.ok) {
const errText = await resp.text();
throw new Error(`Ollama chat failed: ${resp.status} ${errText}`);
}
const data = await resp.json() as {
message: { content: string };
model: string;
total_duration: number;
eval_count: number;
};
return {
text: data.message.content,
model: data.model,
totalDuration: data.total_duration,
evalCount: data.eval_count,
};
}
/** Check if Ollama is available and model is loaded */
export async function checkHealth(): Promise<{ ok: boolean; model: string; error?: string }> {
try {
const resp = await fetch(`${OLLAMA_URL}/api/tags`, { signal: AbortSignal.timeout(5000) });
if (!resp.ok) return { ok: false, model: LLM_MODEL, error: `HTTP ${resp.status}` };
const data = await resp.json() as { models: Array<{ name: string }> };
const hasModel = data.models.some((m) => m.name.includes(LLM_MODEL.split(":")[0]));
return { ok: hasModel, model: LLM_MODEL, error: hasModel ? undefined : `Model ${LLM_MODEL} not found` };
} catch (err) {
return { ok: false, model: LLM_MODEL, error: (err as Error).message };
}
}

309
packages/api/src/routes/blog.ts
View File

@ -1,51 +1,63 @@
/** /**
* Blog Draft Generator API * Blog Draft Generator API
* *
* POST /api/blog/generate — Generate a blog draft from data * POST /api/blog/generate — Generate a blog draft via LLM (multi-pass pipeline)
* GET /api/blog — List all drafts * GET /api/blog — List all drafts
* GET /api/blog/:id — Get a specific draft * GET /api/blog/:id — Get a specific draft
* PUT /api/blog/:id/status — Update draft status * PUT /api/blog/:id/status — Update draft status
*
* Pipeline: gather data → LLM master pass → depth improvement → quality control
* Voice: Senior optical network engineer, not marketing.
*/ */
import { Router, Request, Response } from "express"; import { Router, Request, Response } from "express";
import { pool } from "../db/client"; import { pool } from "../db/client";
import { semanticSearch } from "../embeddings/client"; import { semanticSearch } from "../embeddings/client";
import { generate, checkHealth } from "../llm/client";
import {
SYSTEM_PROMPT,
DEPTH_PROMPT,
ANTI_GENERIC_INTRO_PROMPT,
QUALITY_CONTROL_PROMPT,
PROCUREMENT_LAYER_PROMPT,
buildTopicPrompt,
} from "../llm/blog-prompts";
export const blogRouter = Router(); export const blogRouter = Router();
interface BlogTopic { interface BlogTemplate {
topic: string; topic: string;
title: string; title: string;
target_audience: "sales" | "technical" | "customer" | "seo"; target_audience: "sales" | "technical" | "customer" | "seo";
seo_keywords: string[]; seo_keywords: string[];
} }
const BLOG_TEMPLATES: Record<string, BlogTopic[]> = { const BLOG_TEMPLATES: Record<string, BlogTemplate[]> = {
hype_cycle: [ hype_cycle: [
{ {
topic: "hype_cycle", topic: "hype_cycle",
title: "The State of {SPEED} Transceivers in {YEAR}: Where Are We on the Hype Cycle?", title: "The State of {SPEED} Transceivers in {YEAR}: Technology Adoption Assessment",
target_audience: "technical", target_audience: "technical",
seo_keywords: ["transceiver", "hype cycle", "optical networking"], seo_keywords: ["transceiver", "technology lifecycle", "optical networking", "adoption curve"],
}, },
{ {
topic: "hype_cycle", topic: "hype_cycle",
title: "Investment Guide: Which Transceiver Speeds to Bet On in {YEAR}", title: "Investment Guide: Which Transceiver Speeds to Bet On in {YEAR}",
target_audience: "sales", target_audience: "sales",
seo_keywords: ["transceiver investment", "data center optics", "ROI"], seo_keywords: ["transceiver investment", "data center optics", "ROI", "network planning"],
}, },
], ],
comparison: [ comparison: [
{ {
topic: "comparison", topic: "comparison",
title: "{FORM_FACTOR} Transceiver Comparison: Top 5 Options for {USE_CASE}", title: "{FORM_FACTOR} Transceiver Comparison: What Actually Matters for {USE_CASE}",
target_audience: "customer", target_audience: "technical",
seo_keywords: ["transceiver comparison", "best transceiver"], seo_keywords: ["transceiver comparison", "best transceiver", "compatible vs original"],
}, },
{ {
topic: "comparison", topic: "comparison",
title: "Original vs. Compatible Transceivers: The Real Cost Difference in {YEAR}", title: "Original vs. Compatible Transceivers: The Real Cost Difference in {YEAR}",
target_audience: "seo", target_audience: "seo",
seo_keywords: ["compatible transceiver", "original vs compatible", "cost savings"], seo_keywords: ["compatible transceiver", "original vs compatible", "cost savings", "interoperability"],
}, },
], ],
new_product: [ new_product: [
@ -53,27 +65,27 @@ const BLOG_TEMPLATES: Record<string, BlogTopic[]> = {
topic: "new_product", topic: "new_product",
title: "{SPEED} Transceivers: What's New and What It Means for Your Network", title: "{SPEED} Transceivers: What's New and What It Means for Your Network",
target_audience: "technical", target_audience: "technical",
seo_keywords: ["new transceiver", "latest optics"], seo_keywords: ["new transceiver", "latest optics", "deployment guide"],
}, },
], ],
tutorial: [ tutorial: [
{ {
topic: "tutorial", topic: "tutorial",
title: "How to Choose the Right Transceiver: A Complete {YEAR} Buying Guide", title: "Troubleshooting Optical Transceiver Issues: A Field Engineer's Guide",
target_audience: "customer", target_audience: "technical",
seo_keywords: ["transceiver buying guide", "how to choose transceiver"], seo_keywords: ["transceiver troubleshooting", "optical module problems", "low tx power", "BER errors", "400ZR"],
}, },
{ {
topic: "tutorial", topic: "tutorial",
title: "Troubleshooting Transceiver Issues: The Definitive Guide", title: "How to Choose the Right Transceiver: A Practical {YEAR} Guide",
target_audience: "technical", target_audience: "customer",
seo_keywords: ["transceiver troubleshooting", "optical module problems"], seo_keywords: ["transceiver buying guide", "how to choose transceiver", "form factor guide"],
}, },
], ],
}; };
/** Gather data from various collections for blog content */ /** Gather data from vector collections for blog content */
async function gatherBlogData(topic: string, keywords: string[]): Promise<{ async function gatherBlogData(keywords: string[]): Promise<{
products: Array<Record<string, unknown>>; products: Array<Record<string, unknown>>;
news: Array<Record<string, unknown>>; news: Array<Record<string, unknown>>;
faq: Array<Record<string, unknown>>; faq: Array<Record<string, unknown>>;
@ -96,133 +108,91 @@ async function gatherBlogData(topic: string, keywords: string[]): Promise<{
}; };
} }
/** Generate blog outline from gathered data */ /** Validate article has no placeholder text or empty sections */
function generateOutline( function validateArticle(content: string): string[] {
const issues: string[] = [];
if (/\b(TODO|NOTE|FIXME|PLACEHOLDER)\b/i.test(content)) {
issues.push("Contains placeholder text");
}
if (/<!--[\s\S]*?-->/.test(content)) {
issues.push("Contains HTML comments");
}
if (/##\s+\w[\s\S]{0,30}\n\s*\n\s*##/m.test(content)) {
issues.push("Empty section detected");
}
// Check for generic filler
const genericPhrases = [
"plays a key role",
"increasingly important",
"it is important to note",
"in today's rapidly evolving",
"The optical transceiver market continues",
];
for (const phrase of genericPhrases) {
if (content.toLowerCase().includes(phrase.toLowerCase())) {
issues.push(`Generic phrase: "${phrase}"`);
}
}
return issues;
}
/** Generate a template-based draft (fallback when LLM is unavailable) */
function generateTemplateDraft(
title: string, title: string,
topic: string, topic: string,
data: Awaited<ReturnType<typeof gatherBlogData>>, data: Awaited<ReturnType<typeof gatherBlogData>>,
): { sections: Array<{ heading: string; notes: string }> } {
const sections: Array<{ heading: string; notes: string }> = [];
sections.push({
heading: "Introduction",
notes: `Hook the reader with the key question this post answers. Reference ${data.news.length} recent news items for timeliness.`,
});
if (topic === "hype_cycle") {
sections.push({
heading: "Understanding the Hype Cycle for Optical Transceivers",
notes: "Explain the Norton-Bass model phases: Innovation Trigger → Peak of Inflated Expectations → Trough of Disillusionment → Slope of Enlightenment → Plateau of Productivity",
});
sections.push({
heading: "Current Position of Key Technologies",
notes: `Cover products found: ${data.products.slice(0, 5).map((p) => p.standard_name || p.text).join(", ")}`,
});
sections.push({
heading: "Market Signals and Recent Developments",
notes: `Reference: ${data.news.map((n) => n.title).join("; ")}`,
});
} else if (topic === "comparison") {
const formFactors = [...new Set(data.products.map((p) => String(p.form_factor)).filter(Boolean))];
sections.push({
heading: "Products Compared",
notes: `Form factors covered: ${formFactors.join(", ")}. ${data.products.length} products analyzed.`,
});
sections.push({
heading: "Key Specifications Breakdown",
notes: "Compare speed, reach, power consumption, fiber type, and pricing across products.",
});
sections.push({
heading: "Compatibility Considerations",
notes: `Reference FAQ: ${data.faq.slice(0, 3).map((f) => f.question).join("; ")}`,
});
} else if (topic === "tutorial") {
sections.push({
heading: "Step 1: Determine Your Requirements",
notes: "Speed, distance, fiber type, switch compatibility.",
});
sections.push({
heading: "Step 2: Understanding Form Factors",
notes: `Cover: ${data.faq.filter((f) => String(f.category) === "form_factor").map((f) => f.question).join("; ")}`,
});
sections.push({
heading: "Common Issues and Troubleshooting",
notes: `Reference: ${data.troubleshooting.map((t) => t.symptom).join("; ")}`,
});
} else {
sections.push({
heading: "What's New",
notes: `${data.products.length} relevant products, ${data.news.length} recent news items.`,
});
sections.push({
heading: "Technical Details",
notes: "Deep-dive into specifications and use cases.",
});
}
sections.push({
heading: "Conclusion & Recommendations",
notes: "Summarize key takeaways. Include CTA for Flexoptix product finder.",
});
return { sections };
}
/** Generate draft content from outline and data */
function generateDraft(
title: string,
outline: ReturnType<typeof generateOutline>,
data: Awaited<ReturnType<typeof gatherBlogData>>,
): string { ): string {
const parts: string[] = []; const parts: string[] = [];
parts.push(`# ${title}\n`); parts.push(`# ${title}\n`);
parts.push(`*Generated by TIP Blog Engine on ${new Date().toISOString().split("T")[0]}*\n`); parts.push(`*Generated by TIP Blog Engine on ${new Date().toISOString().split("T")[0]}*\n`);
parts.push(`> **Note**: This draft was generated using template mode (LLM unavailable). Content needs manual enhancement.\n`);
for (const section of outline.sections) { if (topic === "tutorial") {
parts.push(`\n## ${section.heading}\n`); parts.push(`\n## The Scenario\n`);
parts.push(`<!-- NOTES: ${section.notes} -->\n`); parts.push(`Link flapping at 2 AM. CRC errors climbing. Where do you start?\n`);
parts.push(`\n## Low Transmit Power\n`);
parts.push(`Check \`show interface transceiver details\`. Tx power below -11.0 dBm on a 10G SR module means the laser is failing.\n`);
parts.push(`- SFP+ SR: Normal Tx range -8.2 to +0.5 dBm\n`);
parts.push(`- QSFP28 LR4: Normal Tx range -4.3 to +4.5 dBm\n`);
}
if (section.heading === "Introduction") { if (data.products.length > 0) {
const topNews = data.news[0]; parts.push(`\n## Products Referenced\n`);
if (topNews) { for (const p of data.products.slice(0, 5)) {
parts.push(`The optical transceiver market continues to evolve rapidly. ${String(topNews.title || "")} highlights the pace of change in our industry.\n`); parts.push(`- **${p.standard_name || p.slug}**: ${p.form_factor} ${p.speed}, ${p.reach_label || "N/A"} reach, ${p.fiber_type || "N/A"}`);
}
parts.push(`In this article, we'll explore the key trends, products, and considerations that matter most for network professionals and procurement teams.\n`);
} else if (section.heading.includes("Products") || section.heading.includes("Technologies")) {
for (const product of data.products.slice(0, 5)) {
parts.push(`### ${product.standard_name || product.slug || "Product"}\n`);
parts.push(`- **Form Factor**: ${product.form_factor || "N/A"}`);
parts.push(`- **Speed**: ${product.speed || "N/A"}`);
parts.push(`- **Reach**: ${product.reach_label || "N/A"}`);
parts.push(`- **Fiber Type**: ${product.fiber_type || "N/A"}`);
parts.push(`- **Vendor**: ${product.vendor || "N/A"}\n`);
}
} else if (section.heading.includes("Troubleshooting") || section.heading.includes("Issues")) {
for (const ts of data.troubleshooting) {
parts.push(`### ${ts.symptom}\n`);
parts.push(`**Cause**: ${ts.cause}\n`);
parts.push(`**Solution**: ${ts.solution}\n`);
}
} else if (section.heading.includes("Conclusion")) {
parts.push(`The transceiver landscape offers more options than ever. Whether you're planning a data center upgrade, evaluating 400G/800G migration, or troubleshooting existing deployments, the right transceiver choice depends on your specific requirements.\n`);
parts.push(`**[Browse our full transceiver catalog →](https://www.flexoptix.net/en/)**\n`);
} }
} }
if (data.troubleshooting.length > 0) {
parts.push(`\n## Troubleshooting Reference\n`);
for (const t of data.troubleshooting) {
parts.push(`### ${t.symptom}\n`);
parts.push(`**Cause**: ${t.cause}\n**Solution**: ${t.solution}\n`);
}
}
parts.push(`\n## Key Takeaways\n`);
parts.push(`1. Always check fiber end-faces before suspecting the optic\n`);
parts.push(`2. Pre-FEC BER tells you more than post-FEC BER\n`);
parts.push(`3. Temperature matters — top-of-rack modules run 10-15°C hotter\n`);
parts.push(`\n**[Browse transceivers →](https://www.flexoptix.net/en/)**\n`);
return parts.join("\n"); return parts.join("\n");
} }
// POST /api/blog/generate — Generate a new blog draft // POST /api/blog/generate — Generate a new blog draft
blogRouter.post("/generate", async (req: Request, res: Response) => { blogRouter.post("/generate", async (req: Request, res: Response) => {
const { topic, speed, form_factor, use_case } = req.body as { const { topic, speed, form_factor, use_case, use_llm } = req.body as {
topic?: string; topic?: string;
speed?: string; speed?: string;
form_factor?: string; form_factor?: string;
use_case?: string; use_case?: string;
use_llm?: boolean;
}; };
const selectedTopic = topic || "comparison"; const selectedTopic = topic || "tutorial";
const templates = BLOG_TEMPLATES[selectedTopic]; const templates = BLOG_TEMPLATES[selectedTopic];
if (!templates) { if (!templates) {
@ -237,14 +207,12 @@ blogRouter.post("/generate", async (req: Request, res: Response) => {
const year = new Date().getFullYear(); const year = new Date().getFullYear();
const template = templates[Math.floor(Math.random() * templates.length)]; const template = templates[Math.floor(Math.random() * templates.length)];
// Fill template placeholders
const title = template.title const title = template.title
.replace("{YEAR}", String(year)) .replace("{YEAR}", String(year))
.replace("{SPEED}", speed || "400G/800G") .replace("{SPEED}", speed || "400G/800G")
.replace("{FORM_FACTOR}", form_factor || "QSFP-DD/OSFP") .replace("{FORM_FACTOR}", form_factor || "QSFP-DD/OSFP")
.replace("{USE_CASE}", use_case || "Data Center Interconnect"); .replace("{USE_CASE}", use_case || "Data Center Interconnect");
// Build search keywords
const keywords = [ const keywords = [
...template.seo_keywords, ...template.seo_keywords,
speed || "400G", speed || "400G",
@ -252,24 +220,95 @@ blogRouter.post("/generate", async (req: Request, res: Response) => {
use_case || "data center", use_case || "data center",
].filter(Boolean); ].filter(Boolean);
// Gather data from all collections const data = await gatherBlogData(keywords);
const data = await gatherBlogData(selectedTopic, keywords);
let draftContent: string;
let generationMethod = "template";
// Try LLM generation if requested or by default
const shouldUseLlm = use_llm !== false;
if (shouldUseLlm) {
const health = await checkHealth();
if (health.ok) {
console.log(`Blog LLM: Using ${health.model} for generation`);
generationMethod = "llm";
// Pass 1: Master generation
const topicPrompt = buildTopicPrompt(selectedTopic, data);
const pass1 = await generate(SYSTEM_PROMPT, `Title: "${title}"\n\n${topicPrompt}`, {
temperature: 0.7,
maxTokens: 4096,
});
console.log(` Pass 1 (master): ${pass1.evalCount} tokens`);
// Pass 2: Depth improvement
const pass2 = await generate(SYSTEM_PROMPT, `${DEPTH_PROMPT}\n\n--- ARTICLE TO IMPROVE ---\n\n${pass1.text}`, {
temperature: 0.4,
maxTokens: 4096,
});
console.log(` Pass 2 (depth): ${pass2.evalCount} tokens`);
// Pass 3: Fix intro if generic
const introCheck = pass2.text.split("\n").slice(0, 10).join("\n").toLowerCase();
let pass3Text = pass2.text;
if (
introCheck.includes("the optical transceiver market") ||
introCheck.includes("in today") ||
introCheck.includes("increasingly") ||
introCheck.includes("plays a key role")
) {
const pass3 = await generate(SYSTEM_PROMPT, `${ANTI_GENERIC_INTRO_PROMPT}\n\n--- ARTICLE ---\n\n${pass2.text}`, {
temperature: 0.6,
maxTokens: 4096,
});
pass3Text = pass3.text;
console.log(` Pass 3 (anti-generic intro): ${pass3.evalCount} tokens`);
}
// Pass 4: Quality control
const issues = validateArticle(pass3Text);
if (issues.length > 0) {
console.log(` Quality issues: ${issues.join(", ")}`);
const pass4 = await generate(SYSTEM_PROMPT, `${QUALITY_CONTROL_PROMPT}\n\n--- ARTICLE ---\n\n${pass3Text}`, {
temperature: 0.3,
maxTokens: 4096,
});
draftContent = `# ${title}\n\n${pass4.text}`;
console.log(` Pass 4 (quality): ${pass4.evalCount} tokens`);
} else {
draftContent = `# ${title}\n\n${pass3Text}`;
}
// Optional: Add procurement notes for sales/customer audience
if (template.target_audience === "sales" || template.target_audience === "customer") {
const procPass = await generate(SYSTEM_PROMPT, `${PROCUREMENT_LAYER_PROMPT}\n\n--- ARTICLE ---\n\n${draftContent}`, {
temperature: 0.4,
maxTokens: 4096,
});
draftContent = procPass.text;
console.log(` Procurement layer: ${procPass.evalCount} tokens`);
}
} else {
console.log(`Blog LLM: Unavailable (${health.error}), using template mode`);
draftContent = generateTemplateDraft(title, selectedTopic, data);
}
} else {
draftContent = generateTemplateDraft(title, selectedTopic, data);
}
// Generate outline and draft
const outline = generateOutline(title, selectedTopic, data);
const draftContent = generateDraft(title, outline, data);
const wordCount = draftContent.split(/\s+/).length; const wordCount = draftContent.split(/\s+/).length;
const finalIssues = validateArticle(draftContent);
// Save to database
const result = await pool.query( const result = await pool.query(
`INSERT INTO blog_drafts (title, topic, target_audience, outline, draft_content, data_sources, status, generated_by, word_count, seo_keywords) `INSERT INTO blog_drafts (title, topic, target_audience, outline, draft_content, data_sources, status, generated_by, word_count, seo_keywords)
VALUES ($1, $2, $3, $4, $5, $6, 'draft', 'tip-blog-engine', $7, $8) VALUES ($1, $2, $3, $4, $5, $6, 'draft', $7, $8, $9)
RETURNING id, created_at`, RETURNING id, created_at`,
[ [
title, title,
selectedTopic, selectedTopic,
template.target_audience, template.target_audience,
JSON.stringify(outline), JSON.stringify({ generation_method: generationMethod, quality_issues: finalIssues }),
draftContent, draftContent,
JSON.stringify({ JSON.stringify({
products: data.products.length, products: data.products.length,
@ -277,6 +316,7 @@ blogRouter.post("/generate", async (req: Request, res: Response) => {
faq: data.faq.length, faq: data.faq.length,
troubleshooting: data.troubleshooting.length, troubleshooting: data.troubleshooting.length,
}), }),
`tip-blog-engine-${generationMethod}`,
wordCount, wordCount,
template.seo_keywords, template.seo_keywords,
], ],
@ -290,7 +330,8 @@ blogRouter.post("/generate", async (req: Request, res: Response) => {
topic: selectedTopic, topic: selectedTopic,
target_audience: template.target_audience, target_audience: template.target_audience,
word_count: wordCount, word_count: wordCount,
sections: outline.sections.length, generation_method: generationMethod,
quality_issues: finalIssues,
data_sources: { data_sources: {
products: data.products.length, products: data.products.length,
news: data.news.length, news: data.news.length,

145
packages/api/src/routes/hype-cycle.ts
View File

@ -2,6 +2,9 @@
* Hype Cycle API routes * Hype Cycle API routes
* *
* GET /api/hype-cycle — All technologies with current phase * GET /api/hype-cycle — All technologies with current phase
* GET /api/hype-cycle/enriched — All technologies with data-driven metrics
* GET /api/hype-cycle/lifecycle — Revenue lifecycle predictions for all speeds
* GET /api/hype-cycle/regional/:tech — Regional adoption model for a technology
* GET /api/hype-cycle/:tech — Specific technology with 5-year forecast * GET /api/hype-cycle/:tech — Specific technology with 5-year forecast
*/ */
import { Router, Request, Response } from "express"; import { Router, Request, Response } from "express";
@ -9,21 +12,27 @@ import {
computeAllHypeCycles, computeAllHypeCycles,
computeHypeCycle, computeHypeCycle,
findTechnology, findTechnology,
TECH_GENERATIONS,
SPECIAL_TECHS,
} from "../hype-cycle/norton-bass"; } from "../hype-cycle/norton-bass";
import {
getDataDrivenOverrides,
getSpeedClassMetrics,
computeRevenueLifecycle,
computeRegionalAdoption,
} from "../hype-cycle/data-enrichment";
export const hypeCycleRouter = Router(); export const hypeCycleRouter = Router();
const q = (p: string, req: Request): string | undefined => const q = (p: string, req: Request): string | undefined =>
req.query[p] ? String(req.query[p]) : undefined; req.query[p] ? String(req.query[p]) : undefined;
// GET /api/hype-cycle — All technologies // GET /api/hype-cycle — All technologies (model-only, fast)
hypeCycleRouter.get("/", (_req: Request, res: Response) => { hypeCycleRouter.get("/", (_req: Request, res: Response) => {
const yearParam = q("year", _req); const yearParam = q("year", _req);
const year = yearParam ? parseInt(yearParam) : new Date().getFullYear(); const year = yearParam ? parseInt(yearParam) : new Date().getFullYear();
const results = computeAllHypeCycles(year); const results = computeAllHypeCycles(year);
// Sort by position on hype cycle
const sorted = [...results].sort((a, b) => a.positionPct - b.positionPct); const sorted = [...results].sort((a, b) => a.positionPct - b.positionPct);
res.json({ res.json({
@ -42,7 +51,121 @@ hypeCycleRouter.get("/", (_req: Request, res: Response) => {
}); });
}); });
// GET /api/hype-cycle/:tech — Specific technology detail // GET /api/hype-cycle/enriched — Data-driven enrichment from scraped data
hypeCycleRouter.get("/enriched", async (_req: Request, res: Response) => {
try {
const yearParam = q("year", _req);
const year = yearParam ? parseInt(yearParam) : new Date().getFullYear();
const overridesMap = await getDataDrivenOverrides();
const allTechs = [...TECH_GENERATIONS, ...SPECIAL_TECHS];
const results = allTechs.map((tech) => {
const overrides = overridesMap.get(tech.speedGbps);
return computeHypeCycle(tech, year, overrides);
});
const sorted = [...results].sort((a, b) => a.positionPct - b.positionPct);
// Also include raw metrics for transparency
const speedMetrics = await getSpeedClassMetrics();
res.json({
success: true,
year,
model: "Norton-Bass + Data-Driven Enrichment",
dataSource: {
totalTransceivers: speedMetrics.reduce((s, m) => s + m.skuCount, 0),
totalPricePoints: speedMetrics.reduce((s, m) => s + m.priceCount, 0),
speedClasses: speedMetrics.length,
},
technologies: sorted.map((r) => ({
technology: r.technology,
phase: r.phaseLabel,
positionPct: r.positionPct,
adoptionPct: r.adoptionPct,
compositeScore: r.compositeScore,
peakYear: r.forecast.peakShipmentYear,
yearsToPlateauFromNow: r.forecast.yearsToPlateauFromNow,
metrics: r.metrics,
fiveYearForecast: r.forecast.fiveYearProjection,
})),
rawSpeedMetrics: speedMetrics.map((m) => ({
speedGbps: m.speedGbps,
vendorCount: m.vendorCount,
skuCount: m.skuCount,
avgPrice: m.avgPrice ? Math.round(m.avgPrice * 100) / 100 : null,
minPrice: m.minPrice ? Math.round(m.minPrice * 100) / 100 : null,
maxPrice: m.maxPrice ? Math.round(m.maxPrice * 100) / 100 : null,
formFactors: m.formFactors,
reachVariants: m.reachVariants,
})),
});
} catch (err) {
console.error("Enriched hype cycle error:", err);
res.status(500).json({ success: false, error: "Failed to compute enriched hype cycle" });
}
});
// GET /api/hype-cycle/lifecycle — Revenue lifecycle predictions
hypeCycleRouter.get("/lifecycle", async (_req: Request, res: Response) => {
try {
const currentYear = new Date().getFullYear();
const speedMetrics = await getSpeedClassMetrics();
const priceMap = new Map(speedMetrics.map((m) => [m.speedGbps, m.avgPrice]));
const allTechs = [...TECH_GENERATIONS, ...SPECIAL_TECHS];
const lifecycles = allTechs.map((tech) =>
computeRevenueLifecycle(
tech.speedGbps,
tech.name,
tech.introYear,
tech.peakYear,
currentYear,
priceMap.get(tech.speedGbps),
)
);
// Sort by revenue index (highest current revenue first)
const sorted = [...lifecycles].sort((a, b) => b.revenueIndex - a.revenueIndex);
res.json({
success: true,
currentYear,
lifecycles: sorted,
});
} catch (err) {
console.error("Lifecycle error:", err);
res.status(500).json({ success: false, error: "Failed to compute lifecycles" });
}
});
// GET /api/hype-cycle/regional/:tech — Regional adoption by technology
hypeCycleRouter.get("/regional/:tech", (req: Request, res: Response) => {
const techQuery = req.params.tech;
const currentYear = new Date().getFullYear();
const tech = findTechnology(techQuery);
if (!tech) {
res.status(404).json({
success: false,
error: `Technology "${techQuery}" not found. Available: 1G, 10G, 25G, 40G, 100G, 400G, 800G, 1.6T, CPO, LPO, 400ZR`,
});
return;
}
const regions = computeRegionalAdoption(tech.peakYear, currentYear, tech.name);
res.json({
success: true,
technology: tech.name,
speedGbps: tech.speedGbps,
globalPeakYear: tech.peakYear,
regions,
});
});
// GET /api/hype-cycle/:tech — Specific technology detail (must be last!)
hypeCycleRouter.get("/:tech", (req: Request, res: Response) => { hypeCycleRouter.get("/:tech", (req: Request, res: Response) => {
const techQuery = req.params.tech; const techQuery = req.params.tech;
const yearParam = q("year", req); const yearParam = q("year", req);
@ -59,8 +182,22 @@ hypeCycleRouter.get("/:tech", (req: Request, res: Response) => {
const result = computeHypeCycle(tech, year); const result = computeHypeCycle(tech, year);
// Add regional data
const regions = computeRegionalAdoption(tech.peakYear, year, tech.name);
// Add revenue lifecycle
const lifecycle = computeRevenueLifecycle(
tech.speedGbps,
tech.name,
tech.introYear,
tech.peakYear,
year,
);
res.json({ res.json({
success: true, success: true,
...result, ...result,
regionalAdoption: regions,
revenueLifecycle: lifecycle,
}); });
}); });

48
packages/dashboard/index.html
View File

@ -217,6 +217,25 @@
.hype-bar { height: 6px; background: var(--surface3); border-radius: 3px; overflow: hidden; width: 100%; } .hype-bar { height: 6px; background: var(--surface3); border-radius: 3px; overflow: hidden; width: 100%; }
.hype-fill { height: 100%; border-radius: 3px; transition: width 0.5s; } .hype-fill { height: 100%; border-radius: 3px; transition: width 0.5s; }
/* TOOLTIPS */
.tip { position: relative; cursor: help; }
.tip::after {
content: attr(data-tip);
position: absolute; bottom: calc(100% + 8px); left: 50%; transform: translateX(-50%);
background: #1a1a2e; color: #e0e0e0; border: 1px solid var(--border);
padding: 0.5rem 0.75rem; border-radius: 6px;
font-size: 0.72rem; line-height: 1.4; font-weight: 400; white-space: normal; width: max-content; max-width: 280px;
opacity: 0; pointer-events: none; transition: opacity 0.2s;
z-index: 500; box-shadow: 0 4px 16px rgba(0,0,0,0.5);
}
.tip::before {
content: ''; position: absolute; bottom: calc(100% + 3px); left: 50%; transform: translateX(-50%);
border: 5px solid transparent; border-top-color: var(--border);
opacity: 0; pointer-events: none; transition: opacity 0.2s; z-index: 501;
}
.tip:hover::after, .tip:hover::before { opacity: 1; }
th.tip::after { left: 0; transform: none; }
/* DETAIL PANEL (shared) */ /* DETAIL PANEL (shared) */
.panel { .panel {
position: fixed; top: 0; right: 0; position: fixed; top: 0; right: 0;
@ -394,7 +413,7 @@
<div class="card mt"> <div class="card mt">
<div class="table-wrap"> <div class="table-wrap">
<table> <table>
<thead><tr><th>Technology</th><th>Phase</th><th>Position</th><th>Adoption</th><th>Peak</th><th>To Plateau</th></tr></thead> <thead><tr><th>Technology</th><th class="tip" data-tip="Current phase in the Gartner-style technology adoption lifecycle. Technologies move from initial hype through disillusionment to mature productivity.">Phase</th><th class="tip" data-tip="Position on the hype curve (0-100%). 0% = just emerging, 50% = mid-cycle, 100% = fully mature.">Position</th><th class="tip" data-tip="Market adoption rate as percentage of total addressable market. Based on Norton-Bass diffusion model forecast.">Adoption</th><th class="tip" data-tip="Estimated year when this technology reaches peak hype / maximum media attention before reality sets in.">Peak</th><th class="tip" data-tip="Estimated years until this technology reaches the Plateau of Productivity — stable, mainstream deployment.">To Plateau</th></tr></thead>
<tbody id="hype-table"></tbody> <tbody id="hype-table"></tbody>
</table> </table>
</div> </div>
@ -602,6 +621,13 @@ var PHASE_MAP = {
'SLOPE_OF_ENLIGHTENMENT': 'Slope of Enlightenment', 'SLOPE_OF_ENLIGHTENMENT': 'Slope of Enlightenment',
'PLATEAU_OF_PRODUCTIVITY': 'Plateau of Productivity' 'PLATEAU_OF_PRODUCTIVITY': 'Plateau of Productivity'
}; };
var PHASE_DESC = {
'Innovation Trigger': 'Early-stage technology breakthrough. First proof-of-concept demos, limited vendor support. High risk, high potential. Expect interop issues and premium pricing.',
'Peak of Inflated Expectations': 'Maximum hype and media attention. Vendors announce products, but real-world deployments are rare. Expectations exceed what the technology can deliver today.',
'Trough of Disillusionment': 'Reality check. Early deployments reveal limitations — interop failures, cost overruns, performance gaps. Interest wanes. Only committed adopters remain.',
'Slope of Enlightenment': 'Practical benefits become clear. Second and third-generation products fix early issues. Multi-vendor support grows. Best practices emerge from real deployments.',
'Plateau of Productivity': 'Mainstream adoption. Stable pricing, broad vendor support, proven reliability. The technology is a standard part of network infrastructure.'
};
function curveY(x, w, h) { function curveY(x, w, h) {
var t = x / w; var t = x / w;
@ -628,16 +654,22 @@ function renderHypeSvg(techs) {
} }
var pl = [ var pl = [
{l:'Innovation\\nTrigger',x:0.07},{l:'Peak of Inflated\\nExpectations',x:0.18}, {l:'Innovation\\nTrigger',x:0.07,k:'Innovation Trigger'},
{l:'Trough of\\nDisillusionment',x:0.42},{l:'Slope of\\nEnlightenment',x:0.64}, {l:'Peak of Inflated\\nExpectations',x:0.18,k:'Peak of Inflated Expectations'},
{l:'Plateau of\\nProductivity',x:0.90} {l:'Trough of\\nDisillusionment',x:0.42,k:'Trough of Disillusionment'},
{l:'Slope of\\nEnlightenment',x:0.64,k:'Slope of Enlightenment'},
{l:'Plateau of\\nProductivity',x:0.90,k:'Plateau of Productivity'}
]; ];
for (var p = 0; p < pl.length; p++) { for (var p = 0; p < pl.length; p++) {
var px = pl[p].x * cw + P; var px = pl[p].x * cw + P;
var ll = pl[p].l.split('\\n'); var ll = pl[p].l.split('\\n');
var phaseW = (rb[p+1]-rb[p])*cw;
svg += '<g style="cursor:help"><title>' + esc(PHASE_DESC[pl[p].k] || '') + '</title>';
svg += '<rect x="' + (rb[p]*cw+P) + '" y="' + (H+2) + '" width="' + phaseW + '" height="28" fill="transparent" />';
for (var li = 0; li < ll.length; li++) { for (var li = 0; li < ll.length; li++) {
svg += '<text x="' + px + '" y="' + (H + 12 + li * 12) + '" class="hype-phase-label">' + esc(ll[li]) + '</text>'; svg += '<text x="' + px + '" y="' + (H + 12 + li * 12) + '" class="hype-phase-label">' + esc(ll[li]) + '</text>';
} }
svg += '</g>';
} }
svg += '<polyline points="' + pts.join(' ') + '" fill="none" stroke="#333b4d" stroke-width="2" />'; svg += '<polyline points="' + pts.join(' ') + '" fill="none" stroke="#333b4d" stroke-width="2" />';
@ -708,11 +740,11 @@ async function loadHypeCycle() {
var color = PC[t.phase] || '#374151'; var color = PC[t.phase] || '#374151';
return '<tr class="clickable" data-tech="' + esc(t.technology) + '">' return '<tr class="clickable" data-tech="' + esc(t.technology) + '">'
+ '<td style="font-weight:500">' + esc(t.technology) + '</td>' + '<td style="font-weight:500">' + esc(t.technology) + '</td>'
+ '<td><span class="b" style="background:' + color + '22;color:' + color + '">' + esc(t.phase) + '</span></td>' + '<td><span class="b tip" data-tip="' + esc(PHASE_DESC[t.phase] || '') + '" style="background:' + color + '22;color:' + color + '">' + esc(t.phase) + '</span></td>'
+ '<td><div class="hype-bar"><div class="hype-fill" style="width:' + t.positionPct + '%;background:' + color + '"></div></div></td>' + '<td><div class="hype-bar"><div class="hype-fill" style="width:' + t.positionPct + '%;background:' + color + '"></div></div></td>'
+ '<td class="mono">' + (t.adoptionPct * 100).toFixed(0) + '%</td>' + '<td class="mono tip" data-tip="' + esc(t.technology) + ' has reached ' + (t.adoptionPct * 100).toFixed(0) + '% of its total addressable market based on Norton-Bass diffusion modeling.">' + (t.adoptionPct * 100).toFixed(0) + '%</td>'
+ '<td class="mono">' + esc(t.peakYear || '—') + '</td>' + '<td class="mono tip" data-tip="' + (t.peakYear ? 'Peak hype expected around ' + t.peakYear + '. After this, expect a reality correction before stable adoption.' : 'Peak year not yet determined.') + '">' + esc(t.peakYear || '—') + '</td>'
+ '<td class="mono">' + (t.yearsToPlateauFromNow != null ? t.yearsToPlateauFromNow + 'y' : '—') + '</td>' + '<td class="mono tip" data-tip="' + (t.yearsToPlateauFromNow != null ? 'Estimated ' + t.yearsToPlateauFromNow + ' years until ' + esc(t.technology) + ' reaches mainstream, stable deployment.' : 'Already at plateau or timeline unknown.') + '">' + (t.yearsToPlateauFromNow != null ? t.yearsToPlateauFromNow + 'y' : '—') + '</td>'
+ '</tr>'; + '</tr>';
}).join('')); }).join(''));

60
packages/scraper/src/index.ts
View File

@ -10,6 +10,12 @@
* tsx src/index.ts --news — Run news aggregator once * tsx src/index.ts --news — Run news aggregator once
* tsx src/index.ts --flexoptix — Run Flexoptix catalog scraper once * tsx src/index.ts --flexoptix — Run Flexoptix catalog scraper once
* tsx src/index.ts --vendors — Run Flexoptix vendor list scraper once * tsx src/index.ts --vendors — Run Flexoptix vendor list scraper once
* tsx src/index.ts --10gtek — Run 10Gtek scraper once
* tsx src/index.ts --champion — Run Champion ONE scraper once
* tsx src/index.ts --fluxlight — Run Fluxlight scraper once
* tsx src/index.ts --gbics — Run GBICS.com scraper once
* tsx src/index.ts --juniper — Run Juniper HCT scraper once
* tsx src/index.ts --fetch-only — Run only fetch-based scrapers (no Playwright)
*/ */
import { createScheduler, registerSchedules, registerWorkers } from "./scheduler"; import { createScheduler, registerSchedules, registerWorkers } from "./scheduler";
import { scrapeFs } from "./scrapers/fs-com"; import { scrapeFs } from "./scrapers/fs-com";
@ -18,29 +24,61 @@ import { scrapeOptcore } from "./scrapers/optcore";
import { scrapeNews } from "./scrapers/news"; import { scrapeNews } from "./scrapers/news";
import { scrapeFlexoptixCatalog } from "./scrapers/flexoptix-catalog"; import { scrapeFlexoptixCatalog } from "./scrapers/flexoptix-catalog";
import { scrapeFlexoptixVendors } from "./scrapers/flexoptix-vendors"; import { scrapeFlexoptixVendors } from "./scrapers/flexoptix-vendors";
import { scrape10Gtek } from "./scrapers/tenGtek";
import { scrapeChampionOne } from "./scrapers/champion-one";
import { scrapeFluxlight } from "./scrapers/fluxlight";
import { scrapeSfpCables } from "./scrapers/sfpcables";
import { scrapeGbics } from "./scrapers/gbics";
import { scrapeJuniperHct } from "./scrapers/juniper-hct";
import { pool } from "./utils/db"; import { pool } from "./utils/db";
const args = process.argv.slice(2); const args = process.argv.slice(2);
const isAll = args.includes("--all");
const isFetchOnly = args.includes("--fetch-only");
async function runOnce(): Promise<void> { async function runOnce(): Promise<void> {
if (args.includes("--flexoptix") || args.includes("--all")) { // Fetch-based scrapers (no Playwright/Chromium needed — fast, reliable)
if (args.includes("--flexoptix") || isAll || isFetchOnly) {
await scrapeFlexoptixCatalog(); await scrapeFlexoptixCatalog();
} }
if (args.includes("--vendors") || args.includes("--all")) { if (args.includes("--vendors") || isAll || isFetchOnly) {
await scrapeFlexoptixVendors(); await scrapeFlexoptixVendors();
} }
if (args.includes("--fs") || args.includes("--all")) { if (args.includes("--10gtek") || isAll || isFetchOnly) {
await scrapeFs(); await scrape10Gtek();
} }
if (args.includes("--cisco") || args.includes("--all")) { if (args.includes("--champion") || isAll || isFetchOnly) {
await scrapeCiscoTmg(); await scrapeChampionOne();
} }
if (args.includes("--optcore") || args.includes("--all")) { if (args.includes("--fluxlight") || isAll || isFetchOnly) {
await scrapeOptcore(); await scrapeFluxlight();
} }
if (args.includes("--news") || args.includes("--all")) { if (args.includes("--sfpcables") || isAll || isFetchOnly) {
await scrapeSfpCables();
}
if (args.includes("--gbics") || isAll || isFetchOnly) {
await scrapeGbics();
}
if (args.includes("--juniper") || isAll || isFetchOnly) {
await scrapeJuniperHct();
}
if (args.includes("--news") || isAll || isFetchOnly) {
await scrapeNews(); await scrapeNews();
} }
// Playwright-based scrapers (need Chromium installed)
if (!isFetchOnly) {
if (args.includes("--fs") || isAll) {
await scrapeFs();
}
if (args.includes("--cisco") || isAll) {
await scrapeCiscoTmg();
}
if (args.includes("--optcore") || isAll) {
await scrapeOptcore();
}
}
await pool.end(); await pool.end();
} }
@ -66,7 +104,9 @@ async function runScheduler(): Promise<void> {
process.on("SIGTERM", shutdown); process.on("SIGTERM", shutdown);
} }
if (args.some((a) => ["--all", "--fs", "--cisco", "--optcore", "--news", "--flexoptix", "--vendors"].includes(a))) { const ALL_FLAGS = ["--all", "--fs", "--cisco", "--optcore", "--news", "--flexoptix", "--vendors", "--10gtek", "--champion", "--fluxlight", "--sfpcables", "--gbics", "--juniper", "--fetch-only"];
if (args.some((a) => ALL_FLAGS.includes(a))) {
runOnce().catch((err) => { runOnce().catch((err) => {
console.error("Fatal:", err); console.error("Fatal:", err);
process.exit(1); process.exit(1);

242
packages/scraper/src/scrapers/champion-one.ts Normal file
View File

@ -0,0 +1,242 @@
/**
* Champion ONE Scraper — US-based compatible transceiver vendor
*
* championone.com — Server-rendered HTML, no JS required.
* Large catalog: SFP, SFP+, SFP28, QSFP+, QSFP28, QSFP-DD, OSFP, XFP, X2, GBIC
*
* Rate limited: 1 req/2sec.
*/
import { pool, findOrCreateScrapedTransceiver, ensureVendor, upsertPriceObservation } from "../utils/db";
import { contentHash } from "../utils/hash";
const BASE = "https://www.championone.com";
const HEADERS = {
"User-Agent": "Mozilla/5.0 (compatible; TIP-Bot/1.0; research)",
Accept: "text/html,application/xhtml+xml",
};
const CATEGORIES = [
{ path: "/sfp-transceivers", formFactor: "SFP", speed: "1G", speedGbps: 1 },
{ path: "/sfp-plus-transceivers", formFactor: "SFP+", speed: "10G", speedGbps: 10 },
{ path: "/25g-sfp28-transceivers", formFactor: "SFP28", speed: "25G", speedGbps: 25 },
{ path: "/qsfp-plus-transceivers", formFactor: "QSFP+", speed: "40G", speedGbps: 40 },
{ path: "/qsfp28-transceivers", formFactor: "QSFP28", speed: "100G", speedGbps: 100 },
{ path: "/qsfp-dd-transceivers", formFactor: "QSFP-DD", speed: "400G", speedGbps: 400 },
{ path: "/xfp-transceivers", formFactor: "XFP", speed: "10G", speedGbps: 10 },
];
interface Product {
partNumber: string;
name: string;
url: string;
price?: number;
currency?: string;
formFactor: string;
speed: string;
speedGbps: number;
reachLabel?: string;
reachMeters?: number;
fiberType?: string;
wavelength?: string;
}
function sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}
function detectReach(text: string): { label: string; meters: number } | undefined {
const patterns: [RegExp, string, number][] = [
[/\b160\s*km\b/i, "160km", 160000],
[/\b80\s*km\b/i, "80km", 80000],
[/\b40\s*km\b/i, "40km", 40000],
[/\b20\s*km\b/i, "20km", 20000],
[/\b10\s*km\b/i, "10km", 10000],
[/\b2\s*km\b/i, "2km", 2000],
[/\b550\s*m\b/i, "550m", 550],
[/\b500\s*m\b/i, "500m", 500],
[/\b300\s*m\b/i, "300m", 300],
[/\b100\s*m\b/i, "100m", 100],
[/\bLR4\b/, "10km", 10000],
[/\bLR\b/, "10km", 10000],
[/\bER4?\b/, "40km", 40000],
[/\bZR4?\b/, "80km", 80000],
[/\bSR4?\b/, "300m", 300],
[/\bDR4?\b/, "500m", 500],
[/\bFR4?\b/, "2km", 2000],
[/\bCWDM4\b/i, "2km", 2000],
[/\bPSM4\b/i, "500m", 500],
];
for (const [regex, label, meters] of patterns) {
if (regex.test(text)) return { label, meters };
}
return undefined;
}
function detectFiber(text: string): string {
if (/single.?mode|smf|[^a-z]lx[^a-z]|[^a-z]lr[^a-z]|[^a-z]er[^a-z]|[^a-z]zr[^a-z]|bidi|cwdm|dwdm/i.test(text)) return "SMF";
if (/multi.?mode|mmf|[^a-z]sx[^a-z]|[^a-z]sr[^a-z]/i.test(text)) return "MMF";
if (/copper|dac|twinax|rj45|base-t/i.test(text)) return "Copper";
return "";
}
function detectWavelength(text: string): string {
const match = text.match(/(\d{3,4})\s*nm/i);
if (match) return match[1];
return "";
}
function parseProductList(html: string, cat: typeof CATEGORIES[number]): Product[] {
const products: Product[] = [];
// Champion ONE uses standard ecommerce HTML with product cards
const productRegex = /href="(\/[^"]*?(?:transceiver|sfp|qsfp|osfp|xfp|optic)[^"]*)"[^>]*>([^<]{5,})<\/a>/gi;
let match;
while ((match = productRegex.exec(html)) !== null) {
const url = match[1];
const name = match[2].trim();
if (name.length < 8 || name.length > 200) continue;
const context = html.slice(Math.max(0, match.index - 200), match.index + 500);
const priceMatch = context.match(/\$\s*([\d,]+\.?\d{0,2})/) || context.match(/USD\s*([\d,]+\.?\d{0,2})/i);
const price = priceMatch ? parseFloat(priceMatch[1].replace(",", "")) : undefined;
const partNum = name.replace(/\s+/g, "-").replace(/[^a-zA-Z0-9\-]/g, "").slice(0, 80);
const reach = detectReach(name);
products.push({
partNumber: partNum, name,
url: url.startsWith("http") ? url : BASE + url,
price: price && price > 0 && price < 50000 ? price : undefined,
currency: price ? "USD" : undefined,
formFactor: cat.formFactor, speed: cat.speed, speedGbps: cat.speedGbps,
reachLabel: reach?.label, reachMeters: reach?.meters,
fiberType: detectFiber(name), wavelength: detectWavelength(name),
});
}
// Pattern 2: Generic product card pattern
const cardRegex = /class="[^"]*product[^"]*"[\s\S]*?href="([^"]+)"[^>]*>[\s\S]*?(?:name|title)[^>]*>([^<]+)/gi;
while ((match = cardRegex.exec(html)) !== null) {
const url = match[1];
const name = match[2].trim();
if (products.find((p) => p.url === (url.startsWith("http") ? url : BASE + url))) continue;
if (name.length < 8) continue;
const context = html.slice(match.index, match.index + 500);
const priceMatch = context.match(/\$\s*([\d,]+\.?\d{0,2})/);
const price = priceMatch ? parseFloat(priceMatch[1].replace(",", "")) : undefined;
const reach = detectReach(name);
products.push({
partNumber: name.replace(/\s+/g, "-").replace(/[^a-zA-Z0-9\-]/g, "").slice(0, 80),
name,
url: url.startsWith("http") ? url : BASE + url,
price: price && price > 0 && price < 50000 ? price : undefined,
currency: price ? "USD" : undefined,
formFactor: cat.formFactor, speed: cat.speed, speedGbps: cat.speedGbps,
reachLabel: reach?.label, reachMeters: reach?.meters,
fiberType: detectFiber(name), wavelength: detectWavelength(name),
});
}
const seen = new Set<string>();
return products.filter((p) => {
if (seen.has(p.url)) return false;
seen.add(p.url);
return true;
});
}
function getMaxPage(html: string): number {
const pageMatches = html.match(/[?&]page=(\d+)/g) || html.match(/\/page\/(\d+)/g);
if (!pageMatches) return 1;
let max = 1;
for (const m of pageMatches) {
const n = parseInt(m.replace(/[^0-9]/g, ""));
if (n > max) max = n;
}
return Math.min(max, 30);
}
async function fetchPage(url: string): Promise<string> {
const resp = await fetch(url, { headers: HEADERS, signal: AbortSignal.timeout(30000) });
if (!resp.ok) throw new Error(`HTTP ${resp.status} for ${url}`);
return resp.text();
}
export async function scrapeChampionOne(): Promise<void> {
console.log("=== Champion ONE Scraper Starting ===\n");
const vendorId = await ensureVendor("Champion ONE", "compatible", "https://www.championone.com", "https://www.championone.com");
let totalProducts = 0;
let priceUpdates = 0;
for (const cat of CATEGORIES) {
console.log(`\n--- ${cat.formFactor} (${cat.speed}) ---`);
try {
const firstPage = await fetchPage(BASE + cat.path);
const maxPage = getMaxPage(firstPage);
console.log(` Pages: ${maxPage}`);
let catProducts: Product[] = parseProductList(firstPage, cat);
for (let page = 2; page <= maxPage; page++) {
await sleep(2000);
try {
const html = await fetchPage(`${BASE}${cat.path}?page=${page}`);
catProducts.push(...parseProductList(html, cat));
} catch (err) {
console.warn(` Page ${page} failed: ${(err as Error).message}`);
}
}
const seen = new Set<string>();
catProducts = catProducts.filter((p) => {
if (seen.has(p.url)) return false;
seen.add(p.url);
return true;
});
console.log(` Found ${catProducts.length} products`);
for (const product of catProducts) {
try {
const txId = await findOrCreateScrapedTransceiver({
partNumber: product.partNumber, vendorId,
formFactor: product.formFactor, speedGbps: product.speedGbps,
speed: product.speed, reachMeters: product.reachMeters,
reachLabel: product.reachLabel, fiberType: product.fiberType,
wavelengths: product.wavelength, category: "DataCenter",
});
if (product.price && product.price > 0) {
const hash = contentHash(JSON.stringify({ price: product.price, part: product.partNumber }));
const updated = await upsertPriceObservation({
transceiverId: txId, sourceVendorId: vendorId,
price: product.price, currency: product.currency || "USD",
stockLevel: "in_stock", url: product.url, contentHash: hash,
});
if (updated) priceUpdates++;
}
totalProducts++;
} catch (err) {
console.warn(` Error: ${(err as Error).message.slice(0, 80)}`);
}
}
} catch (err) {
console.error(` Category failed: ${(err as Error).message}`);
}
await sleep(2000);
}
console.log(`\n=== Champion ONE Complete: ${totalProducts} products, ${priceUpdates} prices ===`);
}
if (require.main === module) {
scrapeChampionOne()
.then(() => pool.end())
.catch((err) => { console.error("Fatal:", err); pool.end(); process.exit(1); });
}

364
packages/scraper/src/scrapers/flexoptix-catalog.ts
View File

@ -4,29 +4,82 @@
* Scrapes flexoptix.net product catalog for transceiver specs and pricing. * Scrapes flexoptix.net product catalog for transceiver specs and pricing.
* This is our own data — no restrictions. * This is our own data — no restrictions.
* *
* Categories: SFP, SFP+, SFP28, QSFP+, QSFP28, QSFP-DD, OSFP, XFP, CFP * Strategy: Use the Magento search/suggest AJAX API which returns JSON
* with product names, URLs, prices, and SKUs. We query by form factor
* keywords to enumerate the full catalog.
* *
* Uses standard fetch (server-rendered HTML). Rate limited: 1 req/sec. * Rate limited: 1 req/sec.
*/ */
import { pool, findOrCreateScrapedTransceiver, ensureVendor, upsertPriceObservation } from "../utils/db"; import { pool, findOrCreateScrapedTransceiver, ensureVendor, upsertPriceObservation } from "../utils/db";
import { contentHash } from "../utils/hash"; import { contentHash } from "../utils/hash";
const BASE = "https://www.flexoptix.net"; const BASE = "https://www.flexoptix.net";
const SEARCH_URL = `${BASE}/en/search/ajax/suggest/`;
const HEADERS = { const HEADERS = {
"User-Agent": "Mozilla/5.0 (compatible; TIP-Bot/1.0; internal-flexoptix)", "User-Agent": "Mozilla/5.0 (compatible; TIP-Bot/1.0; internal-flexoptix)",
Accept: "text/html,application/xhtml+xml", Accept: "application/json, text/html",
}; };
const CATEGORIES = [ // Search queries that cover the full transceiver catalog
{ path: "/en/sfp/", formFactor: "SFP", speed: "1G", speedGbps: 1 }, const SEARCH_QUERIES = [
{ path: "/en/sfp-plus/", formFactor: "SFP+", speed: "10G", speedGbps: 10 }, // By form factor
{ path: "/en/sfp28/", formFactor: "SFP28", speed: "25G", speedGbps: 25 }, { query: "SFP 1G", formFactor: "SFP", speed: "1G", speedGbps: 1 },
{ path: "/en/qsfp-plus/", formFactor: "QSFP+", speed: "40G", speedGbps: 40 }, { query: "SFP BiDi", formFactor: "SFP", speed: "1G", speedGbps: 1 },
{ path: "/en/qsfp28/", formFactor: "QSFP28", speed: "100G", speedGbps: 100 }, { query: "SFP CWDM", formFactor: "SFP", speed: "1G", speedGbps: 1 },
{ path: "/en/qsfp-dd/", formFactor: "QSFP-DD", speed: "400G", speedGbps: 400 }, { query: "SFP DWDM", formFactor: "SFP", speed: "1G", speedGbps: 1 },
{ path: "/en/osfp/", formFactor: "OSFP", speed: "400G", speedGbps: 400 }, { query: "SFP copper", formFactor: "SFP", speed: "1G", speedGbps: 1 },
{ path: "/en/xfp/", formFactor: "XFP", speed: "10G", speedGbps: 10 }, { query: "SFP+ 10G", formFactor: "SFP+", speed: "10G", speedGbps: 10 },
{ path: "/en/cfp/", formFactor: "CFP2", speed: "100G", speedGbps: 100 }, { query: "SFP+ SR", formFactor: "SFP+", speed: "10G", speedGbps: 10 },
{ query: "SFP+ LR", formFactor: "SFP+", speed: "10G", speedGbps: 10 },
{ query: "SFP+ ER", formFactor: "SFP+", speed: "10G", speedGbps: 10 },
{ query: "SFP+ ZR", formFactor: "SFP+", speed: "10G", speedGbps: 10 },
{ query: "SFP+ BiDi", formFactor: "SFP+", speed: "10G", speedGbps: 10 },
{ query: "SFP+ CWDM", formFactor: "SFP+", speed: "10G", speedGbps: 10 },
{ query: "SFP+ DWDM", formFactor: "SFP+", speed: "10G", speedGbps: 10 },
{ query: "SFP+ DAC", formFactor: "SFP+", speed: "10G", speedGbps: 10 },
{ query: "SFP+ AOC", formFactor: "SFP+", speed: "10G", speedGbps: 10 },
{ query: "25G SFP28", formFactor: "SFP28", speed: "25G", speedGbps: 25 },
{ query: "SFP28 SR", formFactor: "SFP28", speed: "25G", speedGbps: 25 },
{ query: "SFP28 LR", formFactor: "SFP28", speed: "25G", speedGbps: 25 },
{ query: "SFP28 DWDM", formFactor: "SFP28", speed: "25G", speedGbps: 25 },
{ query: "SFP28 DAC", formFactor: "SFP28", speed: "25G", speedGbps: 25 },
{ query: "SFP28 AOC", formFactor: "SFP28", speed: "25G", speedGbps: 25 },
{ query: "QSFP+ 40G", formFactor: "QSFP+", speed: "40G", speedGbps: 40 },
{ query: "QSFP+ SR4", formFactor: "QSFP+", speed: "40G", speedGbps: 40 },
{ query: "QSFP+ LR4", formFactor: "QSFP+", speed: "40G", speedGbps: 40 },
{ query: "QSFP+ DAC", formFactor: "QSFP+", speed: "40G", speedGbps: 40 },
{ query: "QSFP+ AOC", formFactor: "QSFP+", speed: "40G", speedGbps: 40 },
{ query: "QSFP28 100G", formFactor: "QSFP28", speed: "100G", speedGbps: 100 },
{ query: "QSFP28 SR4", formFactor: "QSFP28", speed: "100G", speedGbps: 100 },
{ query: "QSFP28 LR4", formFactor: "QSFP28", speed: "100G", speedGbps: 100 },
{ query: "QSFP28 ER4", formFactor: "QSFP28", speed: "100G", speedGbps: 100 },
{ query: "QSFP28 CWDM4", formFactor: "QSFP28", speed: "100G", speedGbps: 100 },
{ query: "QSFP28 PSM4", formFactor: "QSFP28", speed: "100G", speedGbps: 100 },
{ query: "QSFP28 DAC", formFactor: "QSFP28", speed: "100G", speedGbps: 100 },
{ query: "QSFP28 AOC", formFactor: "QSFP28", speed: "100G", speedGbps: 100 },
{ query: "QSFP56 200G", formFactor: "QSFP56", speed: "200G", speedGbps: 200 },
{ query: "QSFP-DD 400G", formFactor: "QSFP-DD", speed: "400G", speedGbps: 400 },
{ query: "QSFP-DD DR4", formFactor: "QSFP-DD", speed: "400G", speedGbps: 400 },
{ query: "QSFP-DD FR4", formFactor: "QSFP-DD", speed: "400G", speedGbps: 400 },
{ query: "QSFP-DD LR4", formFactor: "QSFP-DD", speed: "400G", speedGbps: 400 },
{ query: "QSFP-DD SR4", formFactor: "QSFP-DD", speed: "400G", speedGbps: 400 },
{ query: "QSFP-DD ZR", formFactor: "QSFP-DD", speed: "400G", speedGbps: 400 },
{ query: "QSFP-DD800 800G", formFactor: "QSFP-DD800", speed: "800G", speedGbps: 800 },
{ query: "OSFP 400G", formFactor: "OSFP", speed: "400G", speedGbps: 400 },
{ query: "OSFP SR4", formFactor: "OSFP", speed: "400G", speedGbps: 400 },
{ query: "OSFP DR4", formFactor: "OSFP", speed: "400G", speedGbps: 400 },
{ query: "OSFP FR4", formFactor: "OSFP", speed: "400G", speedGbps: 400 },
{ query: "OSFP LR4", formFactor: "OSFP", speed: "400G", speedGbps: 400 },
{ query: "OSFP ZR", formFactor: "OSFP", speed: "400G", speedGbps: 400 },
{ query: "OSFP 800G", formFactor: "OSFP", speed: "800G", speedGbps: 800 },
// Generic searches to catch stragglers
{ query: "transceiver SR", formFactor: "SFP+", speed: "10G", speedGbps: 10 },
{ query: "transceiver LR", formFactor: "SFP+", speed: "10G", speedGbps: 10 },
{ query: "transceiver ER", formFactor: "SFP+", speed: "10G", speedGbps: 10 },
{ query: "transceiver ZR", formFactor: "SFP+", speed: "10G", speedGbps: 10 },
{ query: "transceiver BiDi", formFactor: "SFP", speed: "1G", speedGbps: 1 },
{ query: "coherent 400ZR", formFactor: "QSFP-DD", speed: "400G", speedGbps: 400 },
{ query: "coherent ZR+", formFactor: "QSFP-DD", speed: "400G", speedGbps: 400 },
]; ];
interface Product { interface Product {
@ -50,6 +103,7 @@ function sleep(ms: number): Promise<void> {
function detectReach(text: string): { label: string; meters: number } | undefined { function detectReach(text: string): { label: string; meters: number } | undefined {
const patterns: [RegExp, string, number][] = [ const patterns: [RegExp, string, number][] = [
[/\b120\s*km\b/i, "120km", 120000],
[/\b80\s*km\b/i, "80km", 80000], [/\b80\s*km\b/i, "80km", 80000],
[/\b40\s*km\b/i, "40km", 40000], [/\b40\s*km\b/i, "40km", 40000],
[/\b20\s*km\b/i, "20km", 20000], [/\b20\s*km\b/i, "20km", 20000],
@ -60,8 +114,8 @@ function detectReach(text: string): { label: string; meters: number } | undefine
[/\b100\s*m\b/i, "100m", 100], [/\b100\s*m\b/i, "100m", 100],
[/\bLR4\b/, "10km", 10000], [/\bLR4\b/, "10km", 10000],
[/\bLR\b/, "10km", 10000], [/\bLR\b/, "10km", 10000],
[/\bER\b/, "40km", 40000], [/\bER4?\b/, "40km", 40000],
[/\bZR\b/, "80km", 80000], [/\bZR4?\b/, "80km", 80000],
[/\bSR4?\b/, "100m", 100], [/\bSR4?\b/, "100m", 100],
[/\bDR4?\b/, "500m", 500], [/\bDR4?\b/, "500m", 500],
[/\bFR4?\b/, "2km", 2000], [/\bFR4?\b/, "2km", 2000],
@ -75,9 +129,10 @@ function detectReach(text: string): { label: string; meters: number } | undefine
} }
function detectFiber(text: string): string { function detectFiber(text: string): string {
const t = text.toLowerCase(); if (/single.?mode|smf|[^a-z]lx[^a-z]|[^a-z]lr[^a-z]|[^a-z]er[^a-z]|[^a-z]zr[^a-z]|bidi|cwdm|dwdm/i.test(text)) return "SMF";
if (/single.?mode|smf|[^a-z]lx[^a-z]|[^a-z]lr[^a-z]|[^a-z]er[^a-z]|[^a-z]zr[^a-z]|bidi|cwdm|dwdm/i.test(t)) return "SMF"; if (/multi.?mode|mmf|[^a-z]sx[^a-z]|[^a-z]sr[^a-z]/i.test(text)) return "MMF";
if (/multi.?mode|mmf|[^a-z]sx[^a-z]|[^a-z]sr[^a-z]/i.test(t)) return "MMF"; if (/copper|dac|twinax|rj45|base-t/i.test(text)) return "Copper";
if (/aoc|active optical/i.test(text)) return "AOC";
return ""; return "";
} }
@ -87,72 +142,149 @@ function detectWavelength(text: string): string {
return ""; return "";
} }
function parseProductList(html: string, cat: typeof CATEGORIES[number]): Product[] { function inferFormFactor(name: string, defaultFF: string): string {
const products: Product[] = []; const lower = name.toLowerCase();
if (lower.includes("osfp224")) return "OSFP224";
if (lower.includes("osfp112")) return "OSFP112";
if (lower.includes("osfp") && !lower.includes("qsfp")) return "OSFP";
if (lower.includes("qsfp-dd800")) return "QSFP-DD800";
if (lower.includes("qsfp-dd")) return "QSFP-DD";
if (lower.includes("qsfp112")) return "QSFP112";
if (lower.includes("qsfp56")) return "QSFP56";
if (lower.includes("qsfp28")) return "QSFP28";
if (lower.includes("qsfp+") || lower.includes("qsfp plus")) return "QSFP+";
if (lower.includes("sfp56")) return "SFP56";
if (lower.includes("sfp28")) return "SFP28";
if (lower.includes("sfp+") || lower.includes("sfp plus")) return "SFP+";
if (lower.includes("cfp2")) return "CFP2";
if (lower.includes("xfp")) return "XFP";
if (/\bsfp\b/i.test(lower) && !lower.includes("qsfp")) return "SFP";
return defaultFF;
}
// Shopware product box pattern function inferSpeed(name: string, defaultGbps: number): number {
const itemRegex = /class="[^"]*product-(?:box|item|card|info|name)[^"]*"[\s\S]*?href="(\/en\/[^"]*?\.html)"[^>]*>[\s\S]*?<\/(?:div|article|li)>/gi; const patterns: [RegExp, number][] = [
[/\b1\.6\s*T\b/i, 1600],
[/\b800\s*G\b/i, 800],
[/\b400\s*G\b/i, 400],
[/\b200\s*G\b/i, 200],
[/\b100\s*G\b/i, 100],
[/\b50\s*G\b/i, 50],
[/\b40\s*G\b/i, 40],
[/\b25\s*G\b/i, 25],
[/\b10\s*G\b/i, 10],
[/\b1\s*G\b/i, 1],
];
for (const [regex, gbps] of patterns) {
if (regex.test(name)) return gbps;
}
return defaultGbps;
}
function speedLabel(gbps: number): string {
if (gbps >= 1000) return `${gbps / 1000}T`;
return `${gbps}G`;
}
interface SearchResult {
title: string;
url: string;
price?: string;
sku?: string;
}
async function searchProducts(query: string): Promise<SearchResult[]> {
const url = `${SEARCH_URL}?q=${encodeURIComponent(query)}`;
const resp = await fetch(url, { headers: HEADERS, signal: AbortSignal.timeout(15000) });
if (!resp.ok) return [];
const text = await resp.text();
// The response may be JSON or HTML with embedded product data
// Try JSON parse first
try {
const data = JSON.parse(text);
const results: SearchResult[] = [];
/** Extract price from Magento price HTML (data-price-amount="39.64") or plain number */
function extractPrice(priceField: unknown): string | undefined {
if (!priceField) return undefined;
const s = String(priceField);
// Try data-price-amount attribute first (Magento Hyva theme)
const attrMatch = s.match(/data-price-amount="([\d.]+)"/);
if (attrMatch) return attrMatch[1];
// Try plain price text like "39.64 EUR"
const textMatch = s.match(/([\d.]+)\s*EUR/i);
if (textMatch) return textMatch[1];
// Try bare number
const num = parseFloat(s);
if (!isNaN(num) && num > 0) return String(num);
return undefined;
}
// Handle various Magento search response formats
if (Array.isArray(data)) {
for (const item of data) {
if (item.title && item.url) {
results.push({
title: item.title,
url: item.url,
price: extractPrice(item.price),
sku: item.sku,
});
}
}
} else if (data.products && Array.isArray(data.products)) {
for (const item of data.products) {
results.push({
title: item.title || item.name || "",
url: item.url || item.product_url || "",
price: extractPrice(item.price),
sku: item.sku,
});
}
} else if (typeof data === "object") {
// Iterate over all keys looking for product arrays
for (const key of Object.keys(data)) {
const val = data[key];
if (Array.isArray(val)) {
for (const item of val) {
if (item && typeof item === "object" && (item.title || item.name) && item.url) {
results.push({
title: item.title || item.name,
url: item.url,
price: extractPrice(item.price),
sku: item.sku,
});
}
}
}
}
}
return results;
} catch {
// Not JSON — parse as HTML
const results: SearchResult[] = [];
const linkRegex = /href="([^"]*\.html)"[^>]*>([^<]{3,})<\/a>/gi;
let match; let match;
while ((match = itemRegex.exec(html)) !== null) { while ((match = linkRegex.exec(text)) !== null) {
const block = match[0]; const pUrl = match[1];
const url = match[1]; const title = match[2].trim();
if (title.length < 5) continue;
const titleMatch = block.match(/class="[^"]*product-(?:name|title)[^"]*"[^>]*>([^<]+)/i) // Look for price near this match
|| block.match(/<a[^>]*>\s*([^<]{5,})<\/a>/i); const context = text.slice(match.index, match.index + 500);
if (!titleMatch) continue; const priceMatch = context.match(/(?:€|EUR)\s*([\d.,]+)/i) || context.match(/([\d.,]+)\s*(?:€|EUR)/i);
const name = titleMatch[1].trim(); results.push({
if (!name || name.length < 3) continue; title,
url: pUrl.startsWith("http") ? pUrl : BASE + pUrl,
const priceMatch = block.match(/(?:€|EUR)\s*([\d.,]+)/i) || block.match(/([\d.,]+)\s*(?:€|EUR)/i); price: priceMatch ? priceMatch[1].replace(",", ".") : undefined,
const price = priceMatch ? parseFloat(priceMatch[1].replace(",", ".")) : undefined;
const partNum = name.replace(/\s+/g, "-").slice(0, 80);
const reach = detectReach(name);
products.push({
name, partNumber: partNum, url: BASE + url,
price, currency: price ? "EUR" : undefined,
formFactor: cat.formFactor, speed: cat.speed, speedGbps: cat.speedGbps,
reachLabel: reach?.label, reachMeters: reach?.meters,
fiberType: detectFiber(name), wavelength: detectWavelength(name),
}); });
} }
return results;
// Fallback: simple link extraction
if (products.length === 0) {
const simpleRegex = /href="(\/en\/(?:sfp|qsfp|osfp|xfp|cfp)[^"]*?\.html)"[^>]*>\s*([^<]{5,})/gi;
while ((match = simpleRegex.exec(html)) !== null) {
const url = match[1];
const name = match[2].trim();
if (products.find((p) => p.url === BASE + url)) continue;
const reach = detectReach(name);
products.push({
name, partNumber: name.replace(/\s+/g, "-").slice(0, 80), url: BASE + url,
formFactor: cat.formFactor, speed: cat.speed, speedGbps: cat.speedGbps,
reachLabel: reach?.label, reachMeters: reach?.meters,
fiberType: detectFiber(name), wavelength: detectWavelength(name),
});
} }
}
return products;
}
function getMaxPage(html: string): number {
const pageMatches = html.match(/[?&]p=(\d+)/g);
if (!pageMatches) return 1;
let max = 1;
for (const m of pageMatches) {
const n = parseInt(m.replace(/[?&]p=/, ""));
if (n > max) max = n;
}
return Math.min(max, 50);
}
async function fetchPage(url: string): Promise<string> {
const resp = await fetch(url, { headers: HEADERS, signal: AbortSignal.timeout(30000) });
if (!resp.ok) throw new Error(`HTTP ${resp.status} for ${url}`);
return resp.text();
} }
export async function scrapeFlexoptixCatalog(): Promise<void> { export async function scrapeFlexoptixCatalog(): Promise<void> {
@ -160,40 +292,58 @@ export async function scrapeFlexoptixCatalog(): Promise<void> {
const vendorId = await ensureVendor("Flexoptix", "reseller", "https://www.flexoptix.net", "https://www.flexoptix.net/en/"); const vendorId = await ensureVendor("Flexoptix", "reseller", "https://www.flexoptix.net", "https://www.flexoptix.net/en/");
let totalProducts = 0; const allProducts = new Map<string, Product>();
let priceUpdates = 0; let priceUpdates = 0;
for (const cat of CATEGORIES) { for (const sq of SEARCH_QUERIES) {
console.log(`\n--- ${cat.formFactor} (${cat.speed}) ---`); console.log(` Searching: "${sq.query}"`);
try { try {
const firstPage = await fetchPage(BASE + cat.path); const results = await searchProducts(sq.query);
const maxPage = getMaxPage(firstPage); let newCount = 0;
console.log(` Pages: ${maxPage}`);
let catProducts: Product[] = parseProductList(firstPage, cat); for (const r of results) {
// Skip non-product results
if (!r.url || !r.title) continue;
const key = r.url;
if (allProducts.has(key)) continue;
for (let page = 2; page <= maxPage; page++) { const name = r.title;
await sleep(1000); const formFactor = inferFormFactor(name, sq.formFactor);
try { const gbps = inferSpeed(name, sq.speedGbps);
const html = await fetchPage(`${BASE}${cat.path}?p=${page}`); const reach = detectReach(name);
catProducts.push(...parseProductList(html, cat)); const price = r.price ? parseFloat(r.price.replace(",", ".")) : undefined;
} catch (err) {
console.warn(` Page ${page} failed: ${(err as Error).message}`);
}
}
// Dedupe by URL allProducts.set(key, {
const seen = new Set<string>(); name,
catProducts = catProducts.filter((p) => { partNumber: r.sku || name.replace(/\s+/g, "-").slice(0, 80),
if (seen.has(p.url)) return false; url: r.url.startsWith("http") ? r.url : BASE + r.url,
seen.add(p.url); price: price && price > 0 && price < 100000 ? price : undefined,
return true; currency: price ? "EUR" : undefined,
formFactor,
speed: speedLabel(gbps),
speedGbps: gbps,
reachLabel: reach?.label,
reachMeters: reach?.meters,
fiberType: detectFiber(name),
wavelength: detectWavelength(name),
}); });
newCount++;
}
console.log(` Found ${catProducts.length} products`); if (newCount > 0) console.log(` +${newCount} new (${results.length} results)`);
} catch (err) {
console.warn(` Search failed: ${(err as Error).message.slice(0, 60)}`);
}
for (const product of catProducts) { await sleep(1000);
}
console.log(`\nTotal unique products: ${allProducts.size}`);
console.log("Writing to database...\n");
// Write all products to DB
for (const product of allProducts.values()) {
try { try {
const txId = await findOrCreateScrapedTransceiver({ const txId = await findOrCreateScrapedTransceiver({
partNumber: product.partNumber, partNumber: product.partNumber,
@ -221,20 +371,12 @@ export async function scrapeFlexoptixCatalog(): Promise<void> {
}); });
if (updated) priceUpdates++; if (updated) priceUpdates++;
} }
totalProducts++;
} catch (err) { } catch (err) {
console.warn(` Error: ${(err as Error).message.slice(0, 80)}`); console.warn(` DB error: ${(err as Error).message.slice(0, 80)}`);
} }
} }
} catch (err) {
console.error(` Category failed: ${(err as Error).message}`);
}
await sleep(2000); console.log(`\n=== Flexoptix Catalog Complete: ${allProducts.size} products, ${priceUpdates} prices ===`);
}
console.log(`\n=== Flexoptix Catalog Complete: ${totalProducts} products, ${priceUpdates} prices ===`);
} }
if (require.main === module) { if (require.main === module) {

234
packages/scraper/src/scrapers/fluxlight.ts Normal file
View File

@ -0,0 +1,234 @@
/**
* Fluxlight Scraper — US-based compatible transceiver vendor
*
* fluxlight.com — BigCommerce, server-rendered HTML with real prices.
* ~144+ products across 6 pages. Uses pagination via ?page=N.
*
* Rate limited: 1 req/2sec.
*/
import { pool, findOrCreateScrapedTransceiver, ensureVendor, upsertPriceObservation } from "../utils/db";
import { contentHash } from "../utils/hash";
const BASE = "https://fluxlight.com";
const CATALOG_PATH = "/transceivers/";
const HEADERS = {
"User-Agent": "Mozilla/5.0 (compatible; TIP-Bot/1.0; research)",
Accept: "text/html,application/xhtml+xml",
};
interface Product {
partNumber: string;
name: string;
url: string;
price?: number;
formFactor: string;
speed: string;
speedGbps: number;
reachLabel?: string;
reachMeters?: number;
fiberType?: string;
wavelength?: string;
}
function sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}
function detectFormFactor(text: string): { formFactor: string; speed: string; speedGbps: number } {
const lower = text.toLowerCase();
if (lower.includes("osfp") && !lower.includes("qsfp")) return { formFactor: "OSFP", speed: "400G", speedGbps: 400 };
if (lower.includes("qsfp-dd")) return { formFactor: "QSFP-DD", speed: "400G", speedGbps: 400 };
if (lower.includes("qsfp28")) return { formFactor: "QSFP28", speed: "100G", speedGbps: 100 };
if (lower.includes("qsfp+") || lower.includes("qsfp plus")) return { formFactor: "QSFP+", speed: "40G", speedGbps: 40 };
if (lower.includes("sfp56")) return { formFactor: "SFP56", speed: "50G", speedGbps: 50 };
if (lower.includes("sfp28") || lower.includes("25g")) return { formFactor: "SFP28", speed: "25G", speedGbps: 25 };
if (lower.includes("sfp+") || lower.includes("10gbase") || lower.includes("10g")) return { formFactor: "SFP+", speed: "10G", speedGbps: 10 };
if (lower.includes("xfp")) return { formFactor: "XFP", speed: "10G", speedGbps: 10 };
if (lower.includes("1000base") || lower.includes("1g")) return { formFactor: "SFP", speed: "1G", speedGbps: 1 };
if (lower.includes("sfp") && !lower.includes("qsfp")) return { formFactor: "SFP", speed: "1G", speedGbps: 1 };
return { formFactor: "SFP+", speed: "10G", speedGbps: 10 };
}
function detectReach(text: string): { label: string; meters: number } | undefined {
const patterns: [RegExp, string, number][] = [
[/\b80\s*km\b/i, "80km", 80000],
[/\b40\s*km\b/i, "40km", 40000],
[/\b20\s*km\b/i, "20km", 20000],
[/\b10\s*km\b/i, "10km", 10000],
[/\b2\s*km\b/i, "2km", 2000],
[/\b550\s*m\b/i, "550m", 550],
[/\b500\s*m\b/i, "500m", 500],
[/\b300\s*m\b/i, "300m", 300],
[/\b100\s*m\b/i, "100m", 100],
[/\bLR4\b/, "10km", 10000],
[/\bLR\b/, "10km", 10000],
[/\bER4?\b/, "40km", 40000],
[/\bZR4?\b/, "80km", 80000],
[/\bSR4?\b/, "300m", 300],
[/\bDR4?\b/, "500m", 500],
[/\bFR4?\b/, "2km", 2000],
];
for (const [regex, label, meters] of patterns) {
if (regex.test(text)) return { label, meters };
}
return undefined;
}
function detectFiber(text: string): string {
if (/single.?mode|smf|[^a-z]lx[^a-z]|[^a-z]lr[^a-z]|[^a-z]er[^a-z]|[^a-z]zr[^a-z]|bidi|cwdm|dwdm/i.test(text)) return "SMF";
if (/multi.?mode|mmf|[^a-z]sx[^a-z]|[^a-z]sr[^a-z]/i.test(text)) return "MMF";
if (/copper|dac|twinax|rj45|base-t|catx/i.test(text)) return "Copper";
return "";
}
function detectWavelength(text: string): string {
const match = text.match(/(\d{3,4})\s*nm/i);
if (match) return match[1];
return "";
}
function parseProductList(html: string): Product[] {
const products: Product[] = [];
// BigCommerce product card pattern: product link + price
// Pattern: <a href="https://fluxlight.com/PARTNUM-FL/">Product Name</a> ... $29.99
const productRegex = /href="(https?:\/\/fluxlight\.com\/[^"]*-FL\/)"[^>]*>\s*([^<]{10,})<\/a>/gi;
let match;
while ((match = productRegex.exec(html)) !== null) {
const url = match[1];
const name = match[2].trim();
if (name.length < 10 || name.length > 200) continue;
// Look for price in surrounding context
const context = html.slice(Math.max(0, match.index - 300), match.index + 600);
const priceMatch = context.match(/\$\s*([\d,]+\.?\d{0,2})/) || context.match(/data-product-price="([\d.]+)"/);
const price = priceMatch ? parseFloat(priceMatch[1].replace(",", "")) : undefined;
const ff = detectFormFactor(name);
const reach = detectReach(name);
const partNum = url.split("/").filter(Boolean).pop() || name.replace(/\s+/g, "-").slice(0, 80);
products.push({
partNumber: partNum,
name,
url,
price: price && price > 0 && price < 50000 ? price : undefined,
...ff,
reachLabel: reach?.label,
reachMeters: reach?.meters,
fiberType: detectFiber(name),
wavelength: detectWavelength(name),
});
}
// Fallback: broader link pattern
if (products.length === 0) {
const simpleRegex = /href="(https?:\/\/fluxlight\.com\/[^"]+)"[^>]*>([^<]{10,}(?:SFP|QSFP|XFP|Base)[^<]*)<\/a>/gi;
while ((match = simpleRegex.exec(html)) !== null) {
const url = match[1];
const name = match[2].trim();
if (products.find((p) => p.url === url)) continue;
const context = html.slice(match.index, match.index + 500);
const priceMatch = context.match(/\$\s*([\d,]+\.?\d{0,2})/);
const price = priceMatch ? parseFloat(priceMatch[1].replace(",", "")) : undefined;
const ff = detectFormFactor(name);
const reach = detectReach(name);
products.push({
partNumber: url.split("/").filter(Boolean).pop() || name.replace(/\s+/g, "-").slice(0, 80),
name, url,
price: price && price > 0 && price < 50000 ? price : undefined,
...ff,
reachLabel: reach?.label, reachMeters: reach?.meters,
fiberType: detectFiber(name), wavelength: detectWavelength(name),
});
}
}
const seen = new Set<string>();
return products.filter((p) => {
if (seen.has(p.url)) return false;
seen.add(p.url);
return true;
});
}
/** Detect max page by probing — page 1 may not have pagination links */
const MAX_PAGES = 6;
async function fetchPage(url: string): Promise<string> {
const resp = await fetch(url, { headers: HEADERS, signal: AbortSignal.timeout(30000) });
if (!resp.ok) throw new Error(`HTTP ${resp.status} for ${url}`);
return resp.text();
}
export async function scrapeFluxlight(): Promise<void> {
console.log("=== Fluxlight Scraper Starting ===\n");
const vendorId = await ensureVendor("Fluxlight", "compatible", "https://fluxlight.com", "https://fluxlight.com/transceivers/");
let allProducts: Product[] = [];
for (let page = 1; page <= MAX_PAGES; page++) {
try {
const url = page === 1 ? BASE + CATALOG_PATH : `${BASE}${CATALOG_PATH}?page=${page}`;
const html = await fetchPage(url);
const pageProducts = parseProductList(html);
allProducts.push(...pageProducts);
console.log(` Page ${page}: ${pageProducts.length} products`);
if (pageProducts.length === 0) {
console.log(` Empty page ${page}, continuing...`);
}
if (page < MAX_PAGES) await sleep(2000);
} catch (err) {
console.warn(` Page ${page} failed: ${(err as Error).message}`);
}
}
// Dedupe
const seen = new Set<string>();
allProducts = allProducts.filter((p) => {
if (seen.has(p.url)) return false;
seen.add(p.url);
return true;
});
console.log(`\nTotal unique products: ${allProducts.length}`);
let totalProducts = 0;
let priceUpdates = 0;
for (const product of allProducts) {
try {
const txId = await findOrCreateScrapedTransceiver({
partNumber: product.partNumber, vendorId,
formFactor: product.formFactor, speedGbps: product.speedGbps,
speed: product.speed, reachMeters: product.reachMeters,
reachLabel: product.reachLabel, fiberType: product.fiberType,
wavelengths: product.wavelength, category: "DataCenter",
});
if (product.price && product.price > 0) {
const hash = contentHash(JSON.stringify({ price: product.price, part: product.partNumber }));
const updated = await upsertPriceObservation({
transceiverId: txId, sourceVendorId: vendorId,
price: product.price, currency: "USD",
stockLevel: "in_stock", url: product.url, contentHash: hash,
});
if (updated) priceUpdates++;
}
totalProducts++;
} catch (err) {
console.warn(` Error: ${(err as Error).message.slice(0, 80)}`);
}
}
console.log(`\n=== Fluxlight Complete: ${totalProducts} products, ${priceUpdates} prices ===`);
}
if (require.main === module) {
scrapeFluxlight()
.then(() => pool.end())
.catch((err) => { console.error("Fatal:", err); pool.end(); process.exit(1); });
}

226
packages/scraper/src/scrapers/gbics.ts Normal file
View File

@ -0,0 +1,226 @@
/**
* GBICS.com Scraper — UK-based compatible transceiver vendor
*
* gbics.com — BigCommerce store, server-rendered HTML, GBP prices.
* Products in <li> cards with <h3> product names, "Now: £XX.XX" pricing.
* Pagination via ?page=N. Rate limited: 1 req/2sec.
*/
import { pool, findOrCreateScrapedTransceiver, ensureVendor, upsertPriceObservation } from "../utils/db";
import { contentHash } from "../utils/hash";
const BASE = "https://gbics.com";
const HEADERS = {
"User-Agent": "Mozilla/5.0 (compatible; TIP-Bot/1.0; research)",
Accept: "text/html,application/xhtml+xml",
};
const CATEGORIES = [
{ path: "/800g-osfp/", formFactor: "OSFP", speed: "800G", speedGbps: 800 },
{ path: "/400g-qsfp112/", formFactor: "QSFP112", speed: "400G", speedGbps: 400 },
{ path: "/400g-qsfp-dd/", formFactor: "QSFP-DD", speed: "400G", speedGbps: 400 },
{ path: "/400g-osfp/", formFactor: "OSFP", speed: "400G", speedGbps: 400 },
{ path: "/200g-qsfp56/", formFactor: "QSFP56", speed: "200G", speedGbps: 200 },
{ path: "/100g-qsfp28/", formFactor: "QSFP28", speed: "100G", speedGbps: 100 },
{ path: "/40g-qsfp/", formFactor: "QSFP+", speed: "40G", speedGbps: 40 },
{ path: "/25g-sfp28/", formFactor: "SFP28", speed: "25G", speedGbps: 25 },
{ path: "/10g-sfp/", formFactor: "SFP+", speed: "10G", speedGbps: 10 },
{ path: "/1g-sfp/", formFactor: "SFP", speed: "1G", speedGbps: 1 },
];
interface Product {
partNumber: string;
name: string;
url: string;
price?: number;
formFactor: string;
speed: string;
speedGbps: number;
reachLabel?: string;
reachMeters?: number;
fiberType?: string;
wavelength?: string;
compatibleWith?: string;
}
function sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}
function detectReach(text: string): { label: string; meters: number } | undefined {
const patterns: [RegExp, string, number][] = [
[/\b120\s*km\b/i, "120km", 120000],
[/\b80\s*km\b/i, "80km", 80000],
[/\b40\s*km\b/i, "40km", 40000],
[/\b20\s*km\b/i, "20km", 20000],
[/\b10\s*km\b/i, "10km", 10000],
[/\b2\s*km\b/i, "2km", 2000],
[/\b550\s*m\b/i, "550m", 550],
[/\b500\s*m\b/i, "500m", 500],
[/\b400\s*m\b/i, "400m", 400],
[/\b300\s*m\b/i, "300m", 300],
[/\b150\s*m\b/i, "150m", 150],
[/\b100\s*m\b/i, "100m", 100],
[/\bLR4\b/, "10km", 10000],
[/\bLR\b/, "10km", 10000],
[/\bER4?\b/, "40km", 40000],
[/\bZR4?\b/, "80km", 80000],
[/\bSR4?\b/, "300m", 300],
[/\bDR4?\b/, "500m", 500],
[/\bFR4?\b/, "2km", 2000],
];
for (const [regex, label, meters] of patterns) {
if (regex.test(text)) return { label, meters };
}
return undefined;
}
function detectFiber(text: string): string {
if (/single.?mode|smf|[^a-z]lx[^a-z]|[^a-z]lr[^a-z]|[^a-z]er[^a-z]|[^a-z]zr[^a-z]|bidi|cwdm|dwdm/i.test(text)) return "SMF";
if (/multi.?mode|mmf|[^a-z]sx[^a-z]|[^a-z]sr[^a-z]/i.test(text)) return "MMF";
if (/copper|dac|twinax|rj.?45|base-t|cat[56x]/i.test(text)) return "Copper";
return "";
}
function detectWavelength(text: string): string {
const match = text.match(/(\d{3,4})\s*nm/i);
if (match) return match[1];
return "";
}
function extractCompatibleVendor(name: string): string {
const match = name.match(/^(\w+(?:\s+\w+)?)\s+Compatible\b/i);
return match ? match[1] : "";
}
function parseProductList(html: string, cat: typeof CATEGORIES[number]): Product[] {
const products: Product[] = [];
// Collapse whitespace for easier regex matching
const collapsed = html.replace(/\s+/g, " ");
// BigCommerce card-title pattern:
// <a aria-label="Product Name, £XX.XX" href="URL" data-event-type="product-click">
const productRegex = /aria-label="([^"]+)"\s+href="(https?:\/\/gbics\.com\/[^"]+)"\s+data-event-type="product-click"/gi;
let match;
while ((match = productRegex.exec(collapsed)) !== null) {
const label = match[1].trim();
const url = match[2];
// aria-label contains "Product Name, £XX.XX"
// Split on last comma to separate name and price
const priceInLabel = label.match(/,\s*£\s*([\d,.]+)\s*$/);
const name = priceInLabel ? label.slice(0, label.lastIndexOf(",")).trim() : label;
const price = priceInLabel ? parseFloat(priceInLabel[1].replace(",", "")) : undefined;
if (name.length < 10) continue;
const reach = detectReach(name);
// Part number: first segment before " - "
const partParts = name.split(/\s+-\s+/);
const partNumber = partParts[0]?.trim().slice(0, 80) || url.split("/").filter(Boolean).pop() || "";
products.push({
partNumber, name, url,
price: price && price > 0 && price < 50000 ? price : undefined,
formFactor: cat.formFactor, speed: cat.speed, speedGbps: cat.speedGbps,
reachLabel: reach?.label, reachMeters: reach?.meters,
fiberType: detectFiber(name), wavelength: detectWavelength(name),
compatibleWith: extractCompatibleVendor(name),
});
}
// Fallback: try "Now: £XX.XX" pattern near product links
if (products.length === 0) {
const altRegex = /href="(https?:\/\/gbics\.com\/[^"]+)"[^>]*>\s*([^<]{15,})<\/a>/gi;
while ((match = altRegex.exec(collapsed)) !== null) {
const url = match[1];
const name = match[2].trim();
if (name.length < 10 || products.find((p) => p.url === url)) continue;
if (!/transceiver|sfp|qsfp|xfp|osfp|base/i.test(name)) continue;
const context = collapsed.slice(Math.max(0, match.index - 300), match.index + 600);
const priceMatch = context.match(/Now:\s*£\s*([\d,.]+)/) || context.match(/£\s*([\d,.]+)/);
const price = priceMatch ? parseFloat(priceMatch[1].replace(",", "")) : undefined;
const reach = detectReach(name);
products.push({
partNumber: name.split(/\s+-\s+/)[0]?.trim().slice(0, 80) || "",
name, url,
price: price && price > 0 && price < 50000 ? price : undefined,
formFactor: cat.formFactor, speed: cat.speed, speedGbps: cat.speedGbps,
reachLabel: reach?.label, reachMeters: reach?.meters,
fiberType: detectFiber(name), wavelength: detectWavelength(name),
compatibleWith: extractCompatibleVendor(name),
});
}
}
const seen = new Set<string>();
return products.filter((p) => {
if (seen.has(p.url)) return false;
seen.add(p.url);
return true;
});
}
async function fetchPage(url: string): Promise<string> {
const resp = await fetch(url, { headers: HEADERS, signal: AbortSignal.timeout(30000) });
if (!resp.ok) throw new Error(`HTTP ${resp.status} for ${url}`);
return resp.text();
}
export async function scrapeGbics(): Promise<void> {
console.log("=== GBICS.com Scraper Starting ===\n");
const vendorId = await ensureVendor("GBICS", "compatible", "https://gbics.com", "https://gbics.com/optical-transceivers/");
let totalProducts = 0;
let priceUpdates = 0;
for (const cat of CATEGORIES) {
console.log(`\n--- ${cat.formFactor} (${cat.speed}) [${cat.path}] ---`);
try {
const html = await fetchPage(BASE + cat.path);
const catProducts = parseProductList(html, cat);
console.log(` Found ${catProducts.length} products`);
for (const product of catProducts) {
try {
const txId = await findOrCreateScrapedTransceiver({
partNumber: product.partNumber, vendorId,
formFactor: product.formFactor, speedGbps: product.speedGbps,
speed: product.speed, reachMeters: product.reachMeters,
reachLabel: product.reachLabel, fiberType: product.fiberType,
wavelengths: product.wavelength, category: "DataCenter",
});
if (product.price && product.price > 0) {
const hash = contentHash(JSON.stringify({ price: product.price, part: product.partNumber }));
const updated = await upsertPriceObservation({
transceiverId: txId, sourceVendorId: vendorId,
price: product.price, currency: "GBP",
stockLevel: "in_stock", url: product.url, contentHash: hash,
});
if (updated) priceUpdates++;
}
totalProducts++;
} catch (err) {
console.warn(` Error: ${(err as Error).message.slice(0, 80)}`);
}
}
} catch (err) {
console.error(` Category failed: ${(err as Error).message}`);
}
await sleep(2000);
}
console.log(`\n=== GBICS Complete: ${totalProducts} products, ${priceUpdates} prices ===`);
}
if (require.main === module) {
scrapeGbics()
.then(() => pool.end())
.catch((err) => { console.error("Fatal:", err); pool.end(); process.exit(1); });
}

241
packages/scraper/src/scrapers/juniper-hct.ts Normal file
View File

@ -0,0 +1,241 @@
/**
* Juniper HCT Scraper — OEM Hardware Compatibility Tool
*
* apps.juniper.net/hct — Next.js SSR app with product data embedded in
* self.__next_f.push() payloads. Transceivers category = 100001.
* Rich data: modelNumber, partNumber, distance, speedType, formFactor, EOL status.
* No prices (OEM), but excellent compatibility + spec data.
*/
import { pool, findOrCreateScrapedTransceiver, ensureVendor } from "../utils/db";
const BASE = "https://apps.juniper.net/hct";
const HEADERS = {
"User-Agent": "Mozilla/5.0 (compatible; TIP-Bot/1.0; research)",
Accept: "text/html,application/xhtml+xml",
};
const CATEGORIES = [
{ id: 100001, name: "Transceivers" },
];
interface JuniperTransceiver {
modelNumber: string;
partNumber: string;
description: string;
cableType: string;
distance: string;
speedType: string;
formFactor: string;
connectorType: string;
maxDistanceKm?: number;
maxDistanceLabel?: string;
isModelEol: boolean;
}
function sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}
function parseSpeedGbps(speedType: string): { speed: string; speedGbps: number } {
const lower = speedType.toLowerCase();
if (lower.includes("800g")) return { speed: "800G", speedGbps: 800 };
if (lower.includes("400g")) return { speed: "400G", speedGbps: 400 };
if (lower.includes("200g")) return { speed: "200G", speedGbps: 200 };
if (lower.includes("100g")) return { speed: "100G", speedGbps: 100 };
if (lower.includes("40g")) return { speed: "40G", speedGbps: 40 };
if (lower.includes("25g")) return { speed: "25G", speedGbps: 25 };
if (lower.includes("10g")) return { speed: "10G", speedGbps: 10 };
if (lower.includes("1g") || lower.includes("1000")) return { speed: "1G", speedGbps: 1 };
return { speed: speedType || "Unknown", speedGbps: 0 };
}
function normalizeFormFactor(ff: string): string {
const upper = ff.toUpperCase().trim();
if (upper.includes("QSFP-DD") || upper.includes("QSFPDD")) return "QSFP-DD";
if (upper.includes("QSFP28")) return "QSFP28";
if (upper.includes("QSFP+") || upper === "QSFP") return "QSFP+";
if (upper.includes("OSFP")) return "OSFP";
if (upper.includes("CFP2")) return "CFP2";
if (upper.includes("CFP4")) return "CFP4";
if (upper.includes("CFP")) return "CFP";
if (upper.includes("SFP56")) return "SFP56";
if (upper.includes("SFP28")) return "SFP28";
if (upper.includes("SFP+")) return "SFP+";
if (upper.includes("XFP")) return "XFP";
if (upper.includes("SFP")) return "SFP";
return ff || "SFP";
}
function detectFiber(cableType: string, description: string): string {
const text = `${cableType} ${description}`.toLowerCase();
if (/smf|single.?mode/.test(text)) return "SMF";
if (/mmf|multi.?mode/.test(text)) return "MMF";
if (/copper|dac|twinax|cat\s*[56]|rj.?45|base-t/.test(text)) return "Copper";
return "";
}
function parseDistance(distance: string): { label: string; meters: number } | undefined {
if (!distance) return undefined;
const km = distance.match(/([\d.]+)\s*km/i);
if (km) return { label: `${km[1]}km`, meters: Math.round(parseFloat(km[1]) * 1000) };
const m = distance.match(/([\d.]+)\s*m\b/i);
if (m) return { label: `${m[1]}m`, meters: parseInt(m[1]) };
return undefined;
}
function detectWavelength(description: string): string {
const match = description.match(/(\d{3,4})\s*nm/i);
return match ? match[1] : "";
}
/**
* Extract transceiver data from Next.js SSR payload.
* Data is embedded in self.__next_f.push([...]) with escaped JSON (\" not ").
* Strategy: unescape the HTML, find categoryDetail array, parse each object.
*/
function parseNextJsData(html: string): JuniperTransceiver[] {
const transceivers: JuniperTransceiver[] = [];
// Unescape the escaped JSON (\" → ", \\ → \)
const unescaped = html.replace(/\\"/g, '"').replace(/\\\\"/g, '\\"');
// Find categoryDetail array and extract individual objects
const detailIdx = unescaped.indexOf('"categoryDetail":[');
if (detailIdx === -1) {
console.log(" Warning: categoryDetail not found in HTML");
return transceivers;
}
// Extract from categoryDetail to end of array
const arrayStart = unescaped.indexOf("[", detailIdx);
if (arrayStart === -1) return transceivers;
// Use regex to find each transceiver object by modelNumber
const modelRegex = /"modelNumber"\s*:\s*"([^"]+)"/g;
const seen = new Set<string>();
let match;
while ((match = modelRegex.exec(unescaped)) !== null) {
const modelNumber = match[1];
if (seen.has(modelNumber)) continue;
seen.add(modelNumber);
// Extract chunk around this model
const idx = match.index;
const objStart = unescaped.lastIndexOf("{", idx);
const chunk = unescaped.slice(objStart, objStart + 2000);
const getString = (field: string): string => {
const re = new RegExp(`"${field}"\\s*:\\s*"([^"]*)"`, "i");
const m = chunk.match(re);
return m ? m[1] : "";
};
// For array fields like cableType:["SMF"], speedType:[{speed:"100G"}], formFactor:["CFP"]
const getArrayFirst = (field: string): string => {
// Try ["value"] pattern
const arrRe = new RegExp(`"${field}"\\s*:\\s*\\[\\s*"([^"]*)"`, "i");
const arrM = chunk.match(arrRe);
if (arrM) return arrM[1];
// Try [{speed:"value"}] pattern
const objRe = new RegExp(`"${field}"\\s*:\\s*\\[\\s*\\{\\s*"\\w+"\\s*:\\s*"([^"]*)"`, "i");
const objM = chunk.match(objRe);
if (objM) return objM[1];
return getString(field);
};
const getBool = (field: string): boolean => {
const re = new RegExp(`"${field}"\\s*:\\s*(true|false)`, "i");
const m = chunk.match(re);
return m ? m[1] === "true" : false;
};
const getNum = (field: string): number | undefined => {
const re = new RegExp(`"${field}"\\s*:\\s*(\\d+(?:\\.\\d+)?)`, "i");
const m = chunk.match(re);
return m ? parseFloat(m[1]) : undefined;
};
// Extract distance from array like ["40 km"] or from maxDistanceLabel
const distArr = chunk.match(/"distance"\s*:\s*\[\s*"([^"]*)"/i);
const distance = distArr ? distArr[1] : getString("maxDistanceLabel");
transceivers.push({
modelNumber,
partNumber: getString("partNumber") || getString("oldPartNumber") || modelNumber,
description: getString("description"),
cableType: getArrayFirst("cableType"),
distance,
speedType: getArrayFirst("speedType"),
formFactor: getArrayFirst("formFactor"),
connectorType: getString("connectorType"),
maxDistanceKm: getNum("maxDistanceKm"),
maxDistanceLabel: getString("maxDistanceLabel"),
isModelEol: getBool("isModelEol"),
});
}
return transceivers;
}
async function fetchPage(url: string): Promise<string> {
const resp = await fetch(url, { headers: HEADERS, signal: AbortSignal.timeout(60000) });
if (!resp.ok) throw new Error(`HTTP ${resp.status} for ${url}`);
return resp.text();
}
export async function scrapeJuniperHct(): Promise<void> {
console.log("=== Juniper HCT Scraper Starting ===\n");
const vendorId = await ensureVendor("Juniper Networks", "oem", "https://www.juniper.net", "https://apps.juniper.net/hct/");
let totalProducts = 0;
for (const cat of CATEGORIES) {
console.log(`\n--- ${cat.name} (category ${cat.id}) ---`);
try {
const html = await fetchPage(`${BASE}/category/${cat.id}`);
console.log(` Fetched ${(html.length / 1024).toFixed(0)}KB`);
const transceivers = parseNextJsData(html);
console.log(` Parsed ${transceivers.length} transceivers`);
for (const tx of transceivers) {
try {
const speedInfo = parseSpeedGbps(tx.speedType || tx.description);
const distInfo = tx.maxDistanceKm
? { label: `${tx.maxDistanceKm}km`, meters: Math.round(tx.maxDistanceKm * 1000) }
: parseDistance(tx.distance);
const formFactor = normalizeFormFactor(tx.formFactor);
await findOrCreateScrapedTransceiver({
partNumber: tx.modelNumber, vendorId,
formFactor, speedGbps: speedInfo.speedGbps,
speed: speedInfo.speed, reachMeters: distInfo?.meters,
reachLabel: distInfo?.label,
fiberType: detectFiber(tx.cableType, tx.description),
wavelengths: detectWavelength(tx.description),
category: "DataCenter",
});
totalProducts++;
} catch (err) {
console.warn(` Error [${tx.modelNumber}]: ${(err as Error).message.slice(0, 80)}`);
}
}
} catch (err) {
console.error(` Category failed: ${(err as Error).message}`);
}
await sleep(2000);
}
console.log(`\n=== Juniper HCT Complete: ${totalProducts} transceivers (no prices - OEM) ===`);
}
if (require.main === module) {
scrapeJuniperHct()
.then(() => pool.end())
.catch((err) => { console.error("Fatal:", err); pool.end(); process.exit(1); });
}

237
packages/scraper/src/scrapers/sfpcables.ts Normal file
View File

@ -0,0 +1,237 @@
/**
* SFPcables.com Scraper — 10Gtek's Retail Store
*
* sfpcables.com — Magento store with server-rendered HTML, real USD prices.
* Product pages have clean <h2 class="product-name"> + <span class="price"> structure.
* Rate limited: 1 req/2sec. TLS verification disabled (self-signed cert issues).
*/
import { pool, findOrCreateScrapedTransceiver, ensureVendor, upsertPriceObservation } from "../utils/db";
import { contentHash } from "../utils/hash";
const BASE = "https://www.sfpcables.com";
const HEADERS = {
"User-Agent": "Mozilla/5.0 (compatible; TIP-Bot/1.0; research)",
Accept: "text/html,application/xhtml+xml",
};
const CATEGORIES = [
{ path: "/sfp-1-25g-series", formFactor: "SFP", speed: "1G", speedGbps: 1 },
{ path: "/sfp-transceivers", formFactor: "SFP+", speed: "10G", speedGbps: 10 },
{ path: "/sfp28-transceivers", formFactor: "SFP28", speed: "25G", speedGbps: 25 },
{ path: "/qsfp-transceivers", formFactor: "QSFP+", speed: "40G", speedGbps: 40 },
{ path: "/100g-qsfp28-transceivers", formFactor: "QSFP28", speed: "100G", speedGbps: 100 },
{ path: "/qsfp-dd-400g-transceivers", formFactor: "QSFP-DD", speed: "400G", speedGbps: 400 },
{ path: "/xfp-transceivers", formFactor: "XFP", speed: "10G", speedGbps: 10 },
{ path: "/2-5g-transceivers", formFactor: "SFP", speed: "2.5G", speedGbps: 2.5 },
{ path: "/industrial-sfp-transceivers", formFactor: "SFP", speed: "1G", speedGbps: 1 },
{ path: "/industrial-qsfp-transceivers", formFactor: "QSFP28", speed: "100G", speedGbps: 100 },
{ path: "/8x50g-qsfp-dd-transceiver-optical-module", formFactor: "QSFP-DD", speed: "400G", speedGbps: 400 },
{ path: "/8x100g-qsfp-dd-transceiver-optical-module", formFactor: "QSFP-DD", speed: "800G", speedGbps: 800 },
{ path: "/osfp-flat-fiber-optic-transceiver-modules", formFactor: "OSFP", speed: "400G", speedGbps: 400 },
{ path: "/400g-8x50g-osfp-fin-fiber-optic-transceiver-modules", formFactor: "OSFP", speed: "400G", speedGbps: 400 },
{ path: "/fc16g-sfp-transceivers", formFactor: "SFP+", speed: "16G FC", speedGbps: 16 },
{ path: "/fc32g-sfp-transceivers", formFactor: "SFP28", speed: "32G FC", speedGbps: 32 },
];
interface Product {
partNumber: string;
name: string;
url: string;
price?: number;
formFactor: string;
speed: string;
speedGbps: number;
reachLabel?: string;
reachMeters?: number;
fiberType?: string;
wavelength?: string;
}
function sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}
function detectReach(text: string): { label: string; meters: number } | undefined {
const patterns: [RegExp, string, number][] = [
[/\b120\s*km\b/i, "120km", 120000],
[/\b80\s*km\b/i, "80km", 80000],
[/\b40\s*km\b/i, "40km", 40000],
[/\b20\s*km\b/i, "20km", 20000],
[/\b10\s*km\b/i, "10km", 10000],
[/\b2\s*km\b/i, "2km", 2000],
[/\b550\s*m\b/i, "550m", 550],
[/\b500\s*m\b/i, "500m", 500],
[/\b400\s*m\b/i, "400m", 400],
[/\b300\s*m\b/i, "300m", 300],
[/\b150\s*m\b/i, "150m", 150],
[/\b100\s*m\b/i, "100m", 100],
[/\b30\s*m\b/i, "30m", 30],
[/\bLR4\b/, "10km", 10000],
[/\bLR\b/, "10km", 10000],
[/\bER4?\b/, "40km", 40000],
[/\bZR4?\b/, "80km", 80000],
[/\bSR4?\b/, "300m", 300],
[/\bDR4?\b/, "500m", 500],
[/\bFR4?\b/, "2km", 2000],
];
for (const [regex, label, meters] of patterns) {
if (regex.test(text)) return { label, meters };
}
return undefined;
}
function detectFiber(text: string): string {
if (/single.?mode|smf|[^a-z]lx[^a-z]|[^a-z]lr[^a-z]|[^a-z]er[^a-z]|[^a-z]zr[^a-z]|bidi|cwdm|dwdm/i.test(text)) return "SMF";
if (/multi.?mode|mmf|[^a-z]sx[^a-z]|[^a-z]sr[^a-z]/i.test(text)) return "MMF";
if (/copper|dac|twinax|rj.?45|base-t|cat[56x]/i.test(text)) return "Copper";
return "";
}
function detectWavelength(text: string): string {
const match = text.match(/(\d{3,4})\s*nm/i);
if (match) return match[1];
return "";
}
function parseProductList(html: string, cat: typeof CATEGORIES[number]): Product[] {
const products: Product[] = [];
// Magento product listing: <h2 class="product-name"><a href="URL" title="NAME">NAME</a></h2>
// Prices: <span class="price">US$XX.XX</span>
const productRegex = /<h2\s+class="product-name">\s*<a\s+href="([^"]+)"[^>]*title="([^"]+)"[^>]*>/gi;
let match;
while ((match = productRegex.exec(html)) !== null) {
const url = match[1];
const name = match[2].trim();
if (name.length < 5) continue;
// Find price after this product name (within next 800 chars)
const afterContext = html.slice(match.index, match.index + 800);
const priceMatch = afterContext.match(/class="price">\s*US?\$\s*([\d,.]+)/);
const price = priceMatch ? parseFloat(priceMatch[1].replace(",", "")) : undefined;
const reach = detectReach(name);
// Build part number from URL slug
const slug = url.split("/").filter(Boolean).pop() || "";
const partNumber = slug.replace(/-\d+$/, "").slice(0, 80);
products.push({
partNumber,
name,
url,
price: price && price > 0 && price < 50000 ? price : undefined,
formFactor: cat.formFactor,
speed: cat.speed,
speedGbps: cat.speedGbps,
reachLabel: reach?.label,
reachMeters: reach?.meters,
fiberType: detectFiber(name),
wavelength: detectWavelength(name),
});
}
// Dedupe by URL
const seen = new Set<string>();
return products.filter((p) => {
if (seen.has(p.url)) return false;
seen.add(p.url);
return true;
});
}
async function fetchPage(url: string): Promise<string> {
const resp = await fetch(url, {
headers: HEADERS,
signal: AbortSignal.timeout(30000),
});
if (!resp.ok) throw new Error(`HTTP ${resp.status} for ${url}`);
return resp.text();
}
export async function scrapeSfpCables(): Promise<void> {
console.log("=== SFPcables.com Scraper Starting ===\n");
const vendorId = await ensureVendor("SFPcables", "compatible", "https://www.sfpcables.com", "https://www.sfpcables.com/transceivers");
let totalProducts = 0;
let priceUpdates = 0;
for (const cat of CATEGORIES) {
console.log(`\n--- ${cat.formFactor} (${cat.speed}) [${cat.path}] ---`);
try {
const html = await fetchPage(BASE + cat.path);
const catProducts = parseProductList(html, cat);
console.log(` Found ${catProducts.length} products`);
// Check for pagination (Magento uses ?p=N)
const pageLinks = html.match(/[?&]p=(\d+)/g);
let maxPage = 1;
if (pageLinks) {
for (const pl of pageLinks) {
const n = parseInt(pl.replace(/[^0-9]/g, ""));
if (n > maxPage) maxPage = n;
}
}
// Fetch additional pages
for (let page = 2; page <= Math.min(maxPage, 10); page++) {
await sleep(2000);
try {
const pageHtml = await fetchPage(`${BASE}${cat.path}?p=${page}`);
const pageProducts = parseProductList(pageHtml, cat);
catProducts.push(...pageProducts);
console.log(` Page ${page}: ${pageProducts.length} products`);
} catch (err) {
console.warn(` Page ${page} failed: ${(err as Error).message}`);
}
}
for (const product of catProducts) {
try {
const txId = await findOrCreateScrapedTransceiver({
partNumber: product.partNumber,
vendorId,
formFactor: product.formFactor,
speedGbps: product.speedGbps,
speed: product.speed,
reachMeters: product.reachMeters,
reachLabel: product.reachLabel,
fiberType: product.fiberType,
wavelengths: product.wavelength,
category: "DataCenter",
});
if (product.price && product.price > 0) {
const hash = contentHash(JSON.stringify({ price: product.price, part: product.partNumber }));
const updated = await upsertPriceObservation({
transceiverId: txId,
sourceVendorId: vendorId,
price: product.price,
currency: "USD",
stockLevel: "in_stock",
url: product.url,
contentHash: hash,
});
if (updated) priceUpdates++;
}
totalProducts++;
} catch (err) {
console.warn(` Error: ${(err as Error).message.slice(0, 80)}`);
}
}
} catch (err) {
console.error(` Category failed: ${(err as Error).message}`);
}
await sleep(2000);
}
console.log(`\n=== SFPcables Complete: ${totalProducts} products, ${priceUpdates} prices ===`);
}
if (require.main === module) {
scrapeSfpCables()
.then(() => pool.end())
.catch((err) => { console.error("Fatal:", err); pool.end(); process.exit(1); });
}

368
packages/scraper/src/scrapers/tenGtek.ts
View File

@ -1,193 +1,231 @@
/** /**
* 10Gtek.com Scraper — Chinese OEM Transceiver Vendor * 10Gtek.com Scraper — Chinese OEM Transceiver Vendor
* *
* Uses PlaywrightCrawler (JS-rendered site).
* Categories: SFP+, SFP28, QSFP+, QSFP28, QSFP-DD, OSFP
*
* 10gtek.com is a direct competitor to FS.com at lower price points. * 10gtek.com is a direct competitor to FS.com at lower price points.
* No aggressive anti-bot (no Cloudflare), but content is JS-rendered. * Uses plain fetch (server-rendered HTML).
* Rate limited: 1 req/2sec.
*
* Categories: SFP, SFP+, SFP28, QSFP+, QSFP28, QSFP-DD, OSFP
*/ */
import { PlaywrightCrawler, Dataset } from "crawlee"; import { pool, findOrCreateScrapedTransceiver, ensureVendor, upsertPriceObservation } from "../utils/db";
import { pool } from "../utils/db"; import { contentHash, parsePrice } from "../utils/hash";
import { contentHash, parsePrice, parseStockLevel } from "../utils/hash";
const CATEGORY_URLS = [ const BASE = "https://www.10gtek.com";
{ url: "https://www.10gtek.com/sfp-plus", formFactor: "SFP+", speedGbps: 10 }, const HEADERS = {
{ url: "https://www.10gtek.com/sfp28", formFactor: "SFP28", speedGbps: 25 }, "User-Agent": "Mozilla/5.0 (compatible; TIP-Bot/1.0; research)",
{ url: "https://www.10gtek.com/qsfp-plus", formFactor: "QSFP+", speedGbps: 40 }, Accept: "text/html,application/xhtml+xml",
{ url: "https://www.10gtek.com/100g-qsfp28", formFactor: "QSFP28", speedGbps: 100 }, };
{ url: "https://www.10gtek.com/400g-qsfp-dd", formFactor: "QSFP-DD", speedGbps: 400 },
const CATEGORIES = [
{ path: "/sfp", formFactor: "SFP", speed: "1G", speedGbps: 1 },
{ path: "/10g-sfp+", formFactor: "SFP+", speed: "10G", speedGbps: 10 },
{ path: "/sfp28", formFactor: "SFP28", speed: "25G", speedGbps: 25 },
{ path: "/qsfp", formFactor: "QSFP+", speed: "40G", speedGbps: 40 },
{ path: "/qsfp28", formFactor: "QSFP28", speed: "100G", speedGbps: 100 },
{ path: "/qsfpdd", formFactor: "QSFP-DD", speed: "400G", speedGbps: 400 },
{ path: "/xfp", formFactor: "XFP", speed: "10G", speedGbps: 10 },
]; ];
// Get or create 10Gtek vendor interface Product {
async function getVendorId(): Promise<string> { partNumber: string;
const result = await pool.query( name: string;
`INSERT INTO vendors (name, vendor_type, website, country) url: string;
VALUES ('10Gtek', 'competitor', 'https://www.10gtek.com', 'CN') price?: number;
ON CONFLICT (name) DO UPDATE SET vendor_type = 'competitor' currency?: string;
RETURNING id` formFactor: string;
); speed: string;
return result.rows[0].id; speedGbps: number;
reachLabel?: string;
reachMeters?: number;
fiberType?: string;
}
function sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}
function detectReach(text: string): { label: string; meters: number } | undefined {
const patterns: [RegExp, string, number][] = [
[/\b80\s*km\b/i, "80km", 80000],
[/\b40\s*km\b/i, "40km", 40000],
[/\b20\s*km\b/i, "20km", 20000],
[/\b10\s*km\b/i, "10km", 10000],
[/\b2\s*km\b/i, "2km", 2000],
[/\b500\s*m\b/i, "500m", 500],
[/\b300\s*m\b/i, "300m", 300],
[/\b100\s*m\b/i, "100m", 100],
[/\bLR4\b/, "10km", 10000],
[/\bLR\b/, "10km", 10000],
[/\bER\b/, "40km", 40000],
[/\bZR\b/, "80km", 80000],
[/\bSR4?\b/, "100m", 100],
[/\bDR4?\b/, "500m", 500],
[/\bFR4?\b/, "2km", 2000],
];
for (const [regex, label, meters] of patterns) {
if (regex.test(text)) return { label, meters };
}
return undefined;
}
function detectFiber(text: string): string {
if (/single.?mode|smf|[^a-z]lx[^a-z]|[^a-z]lr[^a-z]|[^a-z]er[^a-z]|[^a-z]zr[^a-z]|bidi|cwdm|dwdm/i.test(text)) return "SMF";
if (/multi.?mode|mmf|[^a-z]sx[^a-z]|[^a-z]sr[^a-z]/i.test(text)) return "MMF";
return "";
}
/** Strip HTML tags and decode common entities */
function stripHtml(s: string): string {
return s.replace(/<[^>]+>/g, "").replace(/&amp;/g, "&").replace(/&lt;/g, "<")
.replace(/&gt;/g, ">").replace(/&nbsp;/g, " ").replace(/&deg;/g, "°")
.replace(/&#\d+;/g, "").trim();
}
function parseDistance(text: string): { label: string; meters: number } | undefined {
const km = text.match(/(\d+)\s*km/i);
if (km) return { label: `${km[1]}km`, meters: parseInt(km[1]) * 1000 };
const m = text.match(/(\d+)\s*m\b/i);
if (m) return { label: `${m[1]}m`, meters: parseInt(m[1]) };
return undefined;
}
function parseProductList(html: string, cat: typeof CATEGORIES[number]): Product[] {
const products: Product[] = [];
// 10Gtek uses HTML tables with columns:
// Part No. | Spec | Data Rate | Wavelength | Fiber Type | Distance | Optical Comp. | Tx Power | E.R | Rx Sens. | Temp.
// Extract all <tr> rows and parse cells
const rowRegex = /<tr[^>]*>([\s\S]*?)<\/tr>/gi;
let rowMatch;
while ((rowMatch = rowRegex.exec(html)) !== null) {
const rowHtml = rowMatch[1];
// Extract all <td> cell contents
const cellRegex = /<td[^>]*>([\s\S]*?)<\/td>/gi;
const cells: string[] = [];
let cellMatch;
while ((cellMatch = cellRegex.exec(rowHtml)) !== null) {
cells.push(stripHtml(cellMatch[1]));
}
// Need at least 6 columns, first cell must look like a part number (starts with A or contains letters+digits)
if (cells.length < 6) continue;
const partNumber = cells[0];
if (!partNumber || partNumber.length < 3) continue;
// Skip header rows
if (/^Part\s*No/i.test(partNumber) || /^Spec/i.test(partNumber)) continue;
// Part numbers typically start with A (ASF, AXS, AXQ, AQS, etc.) or contain alphanumeric
if (!/^[A-Z][A-Z0-9]/i.test(partNumber)) continue;
const spec = cells[1] || "";
const dataRate = cells[2] || "";
const wavelength = cells.length >= 4 ? cells[3] : "";
const fiberType = cells.length >= 5 ? cells[4] : "";
const distance = cells.length >= 6 ? cells[5] : "";
const txPower = cells.length >= 8 ? cells[7] : "";
// Build descriptive name
const name = `${partNumber} ${spec} ${dataRate}`.trim();
const reach = parseDistance(distance) || detectReach(spec + " " + distance);
// Determine fiber type from table cell or spec
let fiber = "";
if (/SMF|single/i.test(fiberType)) fiber = "SMF";
else if (/MMF|multi/i.test(fiberType)) fiber = "MMF";
else if (/CAT|RJ|copper/i.test(fiberType)) fiber = "Copper";
else fiber = detectFiber(spec);
// Extract wavelength
const wl = wavelength.replace(/[^0-9]/g, "");
products.push({
partNumber,
name,
url: `${BASE}${cat.path}#${partNumber}`,
formFactor: cat.formFactor,
speed: cat.speed,
speedGbps: cat.speedGbps,
reachLabel: reach?.label,
reachMeters: reach?.meters,
fiberType: fiber,
});
}
// Dedupe by part number
const seen = new Set<string>();
return products.filter((p) => {
if (seen.has(p.partNumber)) return false;
seen.add(p.partNumber);
return true;
});
}
async function fetchPage(url: string): Promise<string> {
const resp = await fetch(url, { headers: HEADERS, signal: AbortSignal.timeout(30000) });
if (!resp.ok) throw new Error(`HTTP ${resp.status} for ${url}`);
return resp.text();
} }
export async function scrape10Gtek(): Promise<void> { export async function scrape10Gtek(): Promise<void> {
console.log("=== 10Gtek Scraper Starting ===\n"); console.log("=== 10Gtek Scraper Starting ===\n");
const vendorId = await getVendorId(); const vendorId = await ensureVendor("10Gtek", "compatible", "https://www.10gtek.com", "https://www.10gtek.com");
let totalProducts = 0; let totalProducts = 0;
let totalPrices = 0; let priceUpdates = 0;
const crawler = new PlaywrightCrawler({ for (const cat of CATEGORIES) {
maxRequestsPerCrawl: 50, console.log(`\n--- ${cat.formFactor} (${cat.speed}) ---`);
maxConcurrency: 2,
requestHandlerTimeoutSecs: 60,
launchContext: {
launchOptions: {
headless: true,
args: ["--no-sandbox"],
},
},
async requestHandler({ page, request, log }) {
const categoryInfo = CATEGORY_URLS.find((c) => request.url.startsWith(c.url));
if (!categoryInfo) return;
log.info(`Scraping: ${request.url} (${categoryInfo.formFactor})`);
// Wait for product grid to load
await page.waitForSelector(".product-item, .product-card, .item-info, table.products", {
timeout: 15000,
}).catch(() => {
log.warning("No product grid found, trying alternative selectors");
});
// Extract products — 10gtek uses various layouts
const products = await page.evaluate(() => {
const items: Array<{
name: string;
price: string;
partNumber: string;
url: string;
inStock: boolean;
}> = [];
// Try table layout
const rows = document.querySelectorAll("table tr, .product-item, .product-card");
rows.forEach((row) => {
const nameEl = row.querySelector("a[href*='/'], .product-name, .item-name, td:first-child a");
const priceEl = row.querySelector(".price, .product-price, [class*='price']");
if (nameEl && priceEl) {
const name = nameEl.textContent?.trim() || "";
const price = priceEl.textContent?.trim() || "";
const url = (nameEl as HTMLAnchorElement).href || "";
const partEl = row.querySelector(".sku, .part-number, [class*='sku']");
const partNumber = partEl?.textContent?.trim() || name.split(" ")[0] || "";
if (name && price) {
items.push({
name,
price,
partNumber,
url,
inStock: !row.textContent?.toLowerCase().includes("out of stock"),
});
}
}
});
// If table extraction yielded nothing, try generic approach
if (items.length === 0) {
const allLinks = document.querySelectorAll("a[href]");
allLinks.forEach((link) => {
const text = link.textContent?.trim() || "";
const parent = link.closest("div, tr, li");
const priceText = parent?.querySelector("[class*='price']")?.textContent?.trim();
if (text.length > 10 && priceText && text.match(/sfp|qsfp|xfp|cfp/i)) {
items.push({
name: text,
price: priceText,
partNumber: text.split(" ")[0],
url: (link as HTMLAnchorElement).href,
inStock: true,
});
}
});
}
return items;
});
log.info(`Found ${products.length} products on ${request.url}`);
totalProducts += products.length;
for (const product of products) {
try { try {
const parsed = parsePrice(product.price); const html = await fetchPage(BASE + cat.path);
if (!parsed) continue; const catProducts = parseProductList(html, cat);
console.log(` Found ${catProducts.length} products`);
const hash = contentHash({ for (const product of catProducts) {
name: product.name, try {
price: parsed.price, const txId = await findOrCreateScrapedTransceiver({
stock: product.inStock, partNumber: product.partNumber,
});
// Find or create transceiver
const txResult = await pool.query(
`SELECT id FROM transceivers
WHERE slug ILIKE $1 OR standard_name ILIKE $1
LIMIT 1`,
[`%${product.partNumber}%`]
);
if (txResult.rows.length === 0) continue;
const existing = await pool.query(
`SELECT content_hash FROM price_observations
WHERE transceiver_id = $1 AND source_vendor_id = $2
ORDER BY time DESC LIMIT 1`,
[txResult.rows[0].id, vendorId]
);
if (existing.rows[0]?.content_hash === hash) continue;
await pool.query(
`INSERT INTO price_observations
(transceiver_id, source_vendor_id, price, currency, stock_level, url, content_hash)
VALUES ($1, $2, $3, $4, $5, $6, $7)`,
[
txResult.rows[0].id,
vendorId, vendorId,
parsed.price, formFactor: product.formFactor,
parsed.currency, speedGbps: product.speedGbps,
product.inStock ? "in_stock" : "out_of_stock", speed: product.speed,
product.url, reachMeters: product.reachMeters,
hash, reachLabel: product.reachLabel,
] fiberType: product.fiberType,
); category: "DataCenter",
totalPrices++;
} catch (err) {
log.warning(`Error processing product: ${(err as Error).message}`);
}
}
},
failedRequestHandler({ request, log }) {
log.error(`Request failed: ${request.url}`);
},
}); });
await crawler.run(CATEGORY_URLS.map((c) => c.url)); if (product.price && product.price > 0) {
const hash = contentHash(JSON.stringify({ price: product.price, part: product.partNumber }));
const updated = await upsertPriceObservation({
transceiverId: txId,
sourceVendorId: vendorId,
price: product.price,
currency: product.currency || "USD",
stockLevel: "in_stock",
url: product.url,
contentHash: hash,
});
if (updated) priceUpdates++;
}
console.log(`\nProducts found: ${totalProducts}`); totalProducts++;
console.log(`Prices written: ${totalPrices}`); } catch (err) {
console.log("=== 10Gtek Scraper Complete ===\n"); console.warn(` Error: ${(err as Error).message.slice(0, 80)}`);
}
}
} catch (err) {
console.error(` Category failed: ${(err as Error).message}`);
}
await sleep(2000);
}
console.log(`\n=== 10Gtek Complete: ${totalProducts} products, ${priceUpdates} prices ===`);
} }
if (require.main === module) { if (require.main === module) {
scrape10Gtek() scrape10Gtek()
.then(() => pool.end()) .then(() => pool.end())
.catch((err) => { .catch((err) => { console.error("Fatal:", err); pool.end(); process.exit(1); });
console.error("Fatal:", err);
pool.end();
process.exit(1);
});
} }

12
packages/scraper/src/utils/db.ts
View File

@ -111,13 +111,23 @@ export async function ensureVendor(
website?: string, website?: string,
shopUrl?: string shopUrl?: string
): Promise<string> { ): Promise<string> {
// Try to find existing vendor first
const existing = await pool.query(`SELECT id FROM vendors WHERE name ILIKE $1`, [name]);
if (existing.rows.length > 0) return existing.rows[0].id;
const slug = name.toLowerCase().replace(/[^a-z0-9]+/g, "-"); const slug = name.toLowerCase().replace(/[^a-z0-9]+/g, "-");
try {
const result = await pool.query( const result = await pool.query(
`INSERT INTO vendors (name, slug, type, website, shop_url, is_competitor) `INSERT INTO vendors (name, slug, type, website, shop_url, is_competitor)
VALUES ($1, $2, $3, $4, $5, true) VALUES ($1, $2, $3, $4, $5, true)
ON CONFLICT (name) DO UPDATE SET shop_url = COALESCE(EXCLUDED.shop_url, vendors.shop_url)
RETURNING id`, RETURNING id`,
[name, slug, type, website || null, shopUrl || null] [name, slug, type, website || null, shopUrl || null]
); );
return result.rows[0].id; return result.rows[0].id;
} catch (err: unknown) {
// Handle race condition — re-query if insert fails on unique constraint
const existing2 = await pool.query(`SELECT id FROM vendors WHERE name ILIKE $1 OR slug = $2`, [name, slug]);
if (existing2.rows.length > 0) return existing2.rows[0].id;
throw err;
}
} }

53
scripts/perplexity-batch-research.ts Normal file
View File

@ -0,0 +1,53 @@
/**
* Batch Research Runner — runs all Perplexity research modes sequentially
* and builds a comprehensive market intelligence report.
*
* Usage: PERPLEXITY_API_KEY=pplx-xxx tsx scripts/perplexity-batch-research.ts
*/
import { execFileSync } from 'child_process';
import { mkdirSync, writeFileSync, readdirSync, readFileSync } from 'fs';
const modes = ['competitors', 'market-trends', 'pricing', 'standards'];
const outputDir = 'storage/research';
mkdirSync(outputDir, { recursive: true });
async function runBatch() {
console.log('🔬 Transceiver Intelligence — Batch Research');
console.log('='.repeat(60));
for (const mode of modes) {
console.log(`\n▶ Running: ${mode}`);
try {
execFileSync('tsx', ['scripts/perplexity-research.ts', `--mode=${mode}`], {
stdio: 'inherit',
env: process.env,
});
await new Promise(resolve => setTimeout(resolve, 2000));
} catch (error) {
console.error(` ✗ Failed: ${mode}`);
}
}
console.log('\n📊 Generating combined report...');
const files = readdirSync(outputDir).filter(f => f.endsWith('.json'));
const results = files.map(f => {
const data = JSON.parse(readFileSync(`${outputDir}/${f}`, 'utf-8'));
return { file: f, ...data };
});
const report = {
generated: new Date().toISOString(),
totalResearchFiles: results.length,
modes: results.map(r => r.mode),
results,
};
const reportPath = `${outputDir}/combined-report-${new Date().toISOString().split('T')[0]}.json`;
writeFileSync(reportPath, JSON.stringify(report, null, 2));
console.log(`\n✅ Combined report: ${reportPath}`);
console.log(` ${results.length} research files processed`);
}
runBatch().catch(console.error);

201
scripts/perplexity-research.ts Normal file
View File

@ -0,0 +1,201 @@
/**
* Perplexity AI Research Agent for Transceiver Intelligence Platform
*
* Uses Perplexity's Search API to gather real-time market data,
* pricing, specs, and competitive intelligence for optical transceivers.
*
* Usage:
* PERPLEXITY_API_KEY=pplx-xxx tsx scripts/perplexity-research.ts [query]
*
* Examples:
* tsx scripts/perplexity-research.ts "QSFP28 100G LR4 pricing 2026"
* tsx scripts/perplexity-research.ts --mode=competitors
* tsx scripts/perplexity-research.ts --mode=specs --transceiver="SFP28 25G SR"
* tsx scripts/perplexity-research.ts --mode=market-trends
*/
const PERPLEXITY_API_URL = 'https://api.perplexity.ai/chat/completions';
const API_KEY = process.env.PERPLEXITY_API_KEY;
if (!API_KEY) {
console.error('Error: PERPLEXITY_API_KEY environment variable required');
console.error('Get your key at: https://console.perplexity.ai');
process.exit(1);
}
interface PerplexityResponse {
id: string;
choices: Array<{
message: {
role: string;
content: string;
};
finish_reason: string;
}>;
citations?: string[];
}
interface ResearchResult {
query: string;
mode: string;
timestamp: string;
response: string;
citations: string[];
}
async function perplexitySearch(query: string, systemPrompt: string): Promise<PerplexityResponse> {
const response = await fetch(PERPLEXITY_API_URL, {
method: 'POST',
headers: {
'Authorization': `Bearer ${API_KEY}`,
'Content-Type': 'application/json',
},
body: JSON.stringify({
model: 'sonar-pro',
messages: [
{ role: 'system', content: systemPrompt },
{ role: 'user', content: query },
],
search_recency_filter: 'month',
return_citations: true,
}),
});
if (!response.ok) {
throw new Error(`Perplexity API error: ${response.status} ${response.statusText}`);
}
return response.json() as Promise<PerplexityResponse>;
}
const RESEARCH_MODES: Record<string, { systemPrompt: string; queryTemplate: (arg?: string) => string }> = {
'competitors': {
systemPrompt: `You are a market research analyst specializing in optical networking equipment.
Focus on: pricing data, market share, product comparisons, vendor landscape.
Always include specific prices, model numbers, and sources.
Cover these competitor categories: OEM vendors (Cisco, Juniper, Arista, Nokia),
compatible/third-party (Flexoptix, ProLabs, AddOn Networks, Fiberstore/FS.com, 10Gtek),
and Chinese manufacturers (HiLink, Gigalight, Source Photonics).`,
queryTemplate: () => `List the top 60+ optical transceiver vendors worldwide with their:
1. Company name and website
2. Product range (SFP, SFP+, SFP28, QSFP28, QSFP-DD, OSFP)
3. Approximate pricing for popular models
4. Market positioning (OEM, compatible, budget)
5. Any recent news or product launches in 2025-2026`,
},
'specs': {
systemPrompt: `You are an optical networking engineer. Provide detailed technical specifications
for optical transceivers including: form factor, data rate, wavelength, reach, power consumption,
DOM parameters, operating temperature, connector type, fiber type, and MSA compliance.
Always include IEEE/MSA standards references.`,
queryTemplate: (transceiver?: string) => `Provide complete technical specifications for ${transceiver || 'QSFP28 100G LR4'} transceiver:
- All MSA-compliant specifications
- Typical pricing from different vendors
- Compatible equipment list
- Common failure modes and DOM thresholds
- Comparison with similar transceivers in the same category`,
},
'market-trends': {
systemPrompt: `You are a telecommunications industry analyst focusing on optical networking trends.
Cover: technology adoption curves, pricing trends, new standards (800G, 1.6T),
market size and growth, supply chain dynamics, and emerging technologies (silicon photonics, CPO).`,
queryTemplate: () => `What are the current optical transceiver market trends for 2025-2026?
Cover these areas:
1. 800G and 1.6T transceiver development status and pricing
2. Silicon photonics adoption in data centers
3. Co-packaged optics (CPO) timeline
4. Supply chain changes post-2024
5. Market size estimates and growth projections
6. Key M&A activity in the optical transceiver space
7. Impact of AI/ML data center buildout on transceiver demand`,
},
'pricing': {
systemPrompt: `You are a procurement specialist for optical networking equipment.
Provide current market pricing with specific vendor quotes where available.
Include: list price, street price, volume pricing tiers, and compatible vs OEM pricing gaps.`,
queryTemplate: (transceiver?: string) => `Current market pricing for ${transceiver || 'all common optical transceivers'}:
- SFP+ 10G SR/LR
- SFP28 25G SR/LR
- QSFP28 100G SR4/LR4/CWDM4
- QSFP56 200G
- QSFP-DD 400G DR4/FR4
- OSFP 800G
Include prices from: Cisco, Juniper, Arista (OEM) vs Flexoptix, FS.com, ProLabs (compatible)`,
},
'standards': {
systemPrompt: `You are an IEEE/MSA standards expert for optical transceivers.
Cover all relevant standards, their status, and implementation details.`,
queryTemplate: () => `List all current and upcoming optical transceiver standards:
1. IEEE 802.3 standards (ba, bm, bs, ck, cn, cw, db, df)
2. MSA specifications (SFP, SFP+, SFP28, SFP56, QSFP, QSFP28, QSFP-DD, OSFP)
3. OIF implementations
4. New standards in development for 800G and 1.6T
5. Pluggable vs co-packaged optics standardization efforts`,
},
};
async function runResearch(mode: string, customQuery?: string, transceiver?: string): Promise<ResearchResult> {
const config = RESEARCH_MODES[mode];
if (!config && !customQuery) {
console.error(`Unknown mode: ${mode}`);
console.error(`Available modes: ${Object.keys(RESEARCH_MODES).join(', ')}`);
process.exit(1);
}
const query = customQuery || config.queryTemplate(transceiver);
const systemPrompt = config?.systemPrompt ||
'You are a research assistant specializing in optical networking and transceivers. Provide detailed, sourced information.';
console.log(`\n🔍 Perplexity Research: ${mode}`);
console.log(` Query: ${query.substring(0, 100)}...`);
console.log(' Searching...\n');
const result = await perplexitySearch(query, systemPrompt);
const content = result.choices[0]?.message?.content || 'No response';
const citations = result.citations || [];
const researchResult: ResearchResult = {
query,
mode,
timestamp: new Date().toISOString(),
response: content,
citations,
};
// Save to file
const filename = `research-${mode}-${new Date().toISOString().split('T')[0]}.json`;
const outputPath = `storage/research/${filename}`;
const { mkdirSync, writeFileSync } = await import('fs');
mkdirSync('storage/research', { recursive: true });
writeFileSync(outputPath, JSON.stringify(researchResult, null, 2));
console.log(content);
console.log('\n📚 Citations:');
citations.forEach((c, i) => console.log(` [${i + 1}] ${c}`));
console.log(`\n💾 Saved to: ${outputPath}`);
return researchResult;
}
// Parse CLI arguments
const args = process.argv.slice(2);
let mode = 'market-trends';
let customQuery: string | undefined;
let transceiver: string | undefined;
for (const arg of args) {
if (arg.startsWith('--mode=')) {
mode = arg.split('=')[1];
} else if (arg.startsWith('--transceiver=')) {
transceiver = arg.split('=').slice(1).join('=');
} else {
customQuery = arg;
}
}
runResearch(mode, customQuery, transceiver).catch(console.error);