rene/transceiver-db
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
16 Commits 5 Branches 9 Tags
Commit Graph

16 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Rene Fichtmueller
1cc3844822 feat: interactive SVG hype cycle visualization with click-through detail panel 2026-03-28 00:52:17 +13:00
Rene Fichtmueller
e83711684f fix: replace alert() with slide-in toast notification in dashboard 2026-03-28 00:47:08 +13:00
Rene Fichtmueller
a6f2b2ef9e feat: Phase 8 — Dashboard frontend + static serving 
Single-file dashboard with 6 tabs: Overview, Semantic Search,
Hype Cycle, Transceivers, News, Blog Engine. Dark theme, no
build step, served as static HTML from Express.

- Overview: health stats, vector collection counts, recent news
- Semantic Search: query across all 6 Qdrant collections
- Hype Cycle: Norton-Bass table with phase colors + position bars
- Transceivers: searchable table with form factor/speed/reach
- News: semantic news search with source links
- Blog: generate drafts from templates, view draft history

Live at: https://transceiver-db.context-x.org/dashboard/
 2026-03-28 00:37:10 +13:00
Rene Fichtmueller
274b80a4f1 feat: Phase 7 — Blog generator + scraper scheduler activation 
Blog draft engine generates structured markdown from all Qdrant
collections (products, news, FAQ, troubleshooting). Supports 4
topic types: hype_cycle, comparison, new_product, tutorial.

- routes/blog.ts: POST /api/blog/generate, GET/PUT endpoints
- ecosystem.config.js: Added tip-scraper PM2 process
- Scraper scheduler (pg-boss) now running on Erik with 8 job queues
- News scraper running every 6 hours on Erik
 2026-03-28 00:32:08 +13:00
Rene Fichtmueller
4cb2db6455 feat: Phase 6 — FAQ + troubleshooting knowledge base embeddings 
19 curated FAQ entries covering form factors, fiber types, reach,
compatibility, WDM, power, and emerging tech (CPO, LPO, 400ZR).
10 troubleshooting guides with symptom/cause/solution format.

All 6 Qdrant collections now populated:
- product_embeddings: 89 transceivers
- datasheet_chunks: 40 chunks (OCR pipeline)
- faq_embeddings: 19 FAQ entries
- troubleshooting_embeddings: 10 guides
- news_embeddings: 33 articles
- manual_chunks: 0 (pending manual ingestion)
 2026-03-28 00:24:50 +13:00
Rene Fichtmueller
8bb3b586f3 feat: Phase 5 — OCR pipeline + document/news search 
Docling-powered OCR pipeline: PDF → markdown → chunks → Ollama embed → Qdrant.
News embedding seeder for news_embeddings collection.
Document and news semantic search API endpoints.

- embeddings/ocr-pipeline.ts: Docling convert → chunk → embed pipeline
- embeddings/seed-news.ts: Batch embed news_articles into Qdrant
- routes/documents.ts: POST /api/documents/process, GET /api/documents
- routes/search.ts: GET /search/documents, GET /search/news endpoints
- sql/005-documents.sql: Add chunks_count, processed_at to documents table
- Ollama + nomic-embed-text installed on Erik (CPU mode)
- 89 products + 40 datasheet chunks + 33 news articles in Qdrant
 2026-03-28 00:22:01 +13:00
Rene Fichtmueller
6d3e5cc04a feat: Phase 4 — Vector embeddings + semantic search 
Ollama nomic-embed-text (768 dim) → Qdrant vector search pipeline.
Embeds all 89 transceivers with rich text representation and payload
filters (form_factor, speed_gbps, fiber_type, wdm_type).

- embeddings/client.ts: Ollama embed + Qdrant upsert/search
- embeddings/seed-products.ts: Batch seeder for product_embeddings
- routes/search.ts: GET /api/search, /search/products, /search/stats
- 6 Qdrant collections: products, datasheets, FAQs, manuals, troubleshooting, news
 2026-03-28 00:05:29 +13:00
Rene Fichtmueller
eb875f37d2 feat: Phase 3 — Norton-Bass Hype Cycle Engine 
Implements the full Norton-Bass Multigenerational Diffusion Model for
transceiver technology lifecycle forecasting.

Math: Bass diffusion F(t) + logistic adoption S(t) = L / (1 + e^(-k(t-t0)))
Parameters: p (innovation ~0.03), q (imitation ~0.3-0.5), m (market potential)

Phase Classification Engine (composite score):
  30% Port shipment share + 20% ASP decline rate + 15% Standards maturity
  + 15% Interop validation + 10% Vendor trajectory + 10% Media sentiment

11 technologies tracked: 1G → 10G → 25G → 40G → 100G → 400G → 800G → 1.6T
  + CPO, LPO, 400ZR Coherent
5-year adoption forecast per technology

API: GET /api/hype-cycle (all) + GET /api/hype-cycle/:tech (detail)
Live: https://transceiver-db.context-x.org/api/hype-cycle
 2026-03-27 23:35:57 +13:00
Rene Fichtmueller
bd3a02ae4b feat: add Flexoptix vendor scraper, 10Gtek pricing scraper, expand news feeds 
- Flexoptix vendor scraper: 285 supported switch vendors ingested from
  flexoptix.net/en/supported-vendors/ (our own data, no restrictions)
- 10Gtek Playwright scraper: Chinese OEM competitor pricing (SFP+, SFP28,
  QSFP+, QSFP28, QSFP-DD categories)
- News feeds expanded: added Lightwave, Fierce Telecom, Data Center Knowledge,
  SDxCentral, Cisco Blogs, Arista Blog (11 total sources)
- Scheduler updated: 8 job queues with appropriate intervals
- DB now: 297 vendors, 89 transceivers, 33 news articles (13 relevant)
 2026-03-27 23:17:42 +13:00
Rene Fichtmueller
92f42832bf feat: Phase 2 — MCP Server with 12 tools 
Implements all 12 MCP tools from CONCEPT document:
- search_transceivers: Full-text + spec filter search with pricing
- check_compatibility: Switch ↔ transceiver compatibility lookup
- get_pricing: Current prices + 30-day history across all vendors
- compare_prices: Multi-vendor price comparison with savings analysis
- get_competitor_stock: Live competitor stock monitoring (sales opportunities)
- suggest_alternatives: Similar spec alternatives optimized for price/availability
- get_templates: FlexBox coding and switch config template finder
- search_knowledge_base: Troubleshooting FAQ search (PostgreSQL full-text)
- search_manuals: Switch manual and datasheet search
- get_hype_cycle: Norton-Bass adoption forecast + Gartner phase classification
- get_market_news: Aggregated news with relevance scoring
- generate_blog_draft: Data-driven blog drafts saved to blog_drafts table

Transport: stdio (MCP protocol 2024-11-05)
Config: .mcp.json for Claude Code integration
Verified: all 12 tools registered, search_transceivers returns DB results
 2026-03-27 16:48:34 +13:00
Rene Fichtmueller
e9fb50a248 feat: TIP Phase 0+1 — monorepo, DB schema, API, scraper engine 
Phase 0 - Foundation:
- Restructure into npm workspace monorepo (packages/core, api, scraper)
- PostgreSQL 17 + TimescaleDB schema (15 tables incl. hypertables)
- Docker Compose for local dev (PostgreSQL on 5433 + Qdrant)
- Express 5 API on port 3200 with 6 routes
- Seed script to migrate 159 transceivers + 42 standards from npm package
- Erik server setup script + PM2 ecosystem config

Phase 1 - Scraper Engine:
- Crawlee + Playwright framework with pg-boss scheduler
- FS.com scraper (PlaywrightCrawler, anti-bot workaround)
- Optcore.net scraper (WP REST API enumeration + PlaywrightCrawler)
  - Uses /wp-json/wp/v2/product to get 2000+ product URLs
  - Playwright renders individual product pages for price extraction
- Cisco TMG Matrix scraper (compatibility data)
- News RSS aggregator (optics.org, SPIE, Network World, Nature Photonics)
  - Keyword relevance scoring for transceiver/fiber topics
  - xml2js with malformed XML sanitization
- SHA-256 content hashing for change detection (skip unchanged records)
- pg-boss v10 with explicit queue creation before scheduling
 2026-03-27 16:27:31 +13:00
Rene Fichtmueller
ddd0a592aa Add live demo link: https://transceiver-db-demo.pages.dev 2026-03-19 15:42:02 +10:00
Rene Fichtmueller
c16e36c628 Fix FLEXOPTIX HQ: Darmstadt, not Braunschweig 2026-03-19 15:37:30 +10:00
Rene Fichtmueller
d04b3cf9e1 Add FLEXOPTIX as compatible vendor with FlexBox USP, update FS.COM comparison 2026-03-19 15:35:37 +10:00
Rene Fichtmueller
7aba2149b5 Add interactive live demo for Cloudflare Pages deployment 2026-03-19 15:28:20 +10:00
Rene Fichtmueller
f3d12afc02 Initial release: optical transceiver database with 89 products, 39 standards, 12 competitors 2026-03-19 15:14:47 +10:00