rene/transceiver-db
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
42 Commits 5 Branches 9 Tags
Commit Graph

17 Commits

Author SHA1 Message Date
Rene Fichtmueller
7719155e8e feat(v0.2.4): blog generation UX overhaul — live progress bar 
When you click Generate:
- Dark overlay with orange progress bar shows pipeline status
- Live step counter: 'Step 3/10: Outline Generation — decision-driven structure'
- Percentage updates every 15 seconds via API polling
- When done: shows word count + QA score, auto-opens the article
- No more silent template dump — user sees the entire pipeline working
 2026-03-31 09:44:29 +02:00
Rene Fichtmueller
896c338c8e fix(v0.2.3): dashboard polling for LLM blog pipeline 
Root cause: pollBlogLlm() checked for 'llm' in generated_by but pipeline
sets 'fo-blog-engine-v3'. Dashboard showed template forever.

Fixes:
- Poll check: now detects any non-template generated_by
- Poll timeout: 20s interval × 60 attempts = 20 min (pipeline takes ~10 min)
- Status toast shows pipeline step progress (Step X/10)
- Generation message tells user LLM runs ~10 min in background
- Version bump to v0.2.3
 2026-03-31 09:41:20 +02:00
Rene Fichtmueller
d69fff36c6 fix(v0.2.2): OLLAMA_URL pointed to localhost instead of .213 via WireGuard 
Blog engine was falling back to template because qwen2.5:14b is on Mac Studio (.213),
not on Erik (localhost). Fixed ecosystem.config.js to use 192.168.178.213:11434.
This was the root cause why the 10-step pipeline never executed.
 2026-03-31 09:28:34 +02:00
Rene Fichtmueller
54cfca6774 chore: bump version to v0.2.1 2026-03-31 09:19:38 +02:00
Rene Fichtmueller
98a7e12282 feat: wire 10-step FO Blog Pipeline into blog generation route 
Replaces old 2-pass pipeline with full Flexoptix Style 10-step generation:
1. Topic Expansion (real scenarios + wrong assumptions)
2. Angle Selection (single strong angle + audience)
3. Outline Generation (decision-driven, no generic sections)
4. Master Draft (Flexoptix voice, 2000+ words)
5. Reality Injection (failure scenarios, operational pain)
6. Technical Deepening (specific optics, power, density)
7. Opinion Layer (clear positions, no neutrality)
8. Kill AI Tone (remove all AI fingerprints)
9. QA Check (technical accuracy verification)
10. Quality Score (1-10 auto-rating, saved as self-feedback)

Feedback loop active:
- Accumulated feedback injected into system prompt
- Auto QA scores saved to blog_feedback table
- Training data export via GET /api/blog/feedback/training-data
 2026-03-31 09:16:23 +02:00
Rene Fichtmueller
73ef5766e6 feat(v0.2.1): data confidence tracking + validation + blog feedback system 
- Migration 016: data_confidence column (vendor_verified/enriched_estimated/scraped_unverified)
- Migration 015: blog_feedback table with 8 quality scores + free text
- Validation script: 8 physics-based rules (wavelength↔fiber, reach plausibility, power limits)
- Blog feedback API: POST /api/blog/:id/feedback + training data export
- FO Blog Pipeline v3: 10-step Flexoptix Style prompts (Less bullshit. More engineering.)
- Auto-fix: wavelength↔fiber mismatches corrected automatically
 2026-03-31 09:12:37 +02:00
Rene Fichtmueller
ad7dc6fcaa chore: bump version to 0.2.0 in health endpoint 2026-03-31 08:59:00 +02:00
Rene Fichtmueller
a53c00b8c4 fix: UUID cast in datasheet routes — use slug-first lookup 2026-03-31 08:58:26 +02:00
Rene Fichtmueller
3a6cbc475d feat(v0.2.0): datasheets + adoption roadmap + all routes registered 
- GET /api/datasheets/transceiver/:id — Full datasheet with power budget, pricing, compatibility, HTML export
- GET /api/datasheets/switch/:id — Switch datasheet with compatible transceivers
- GET /api/adoption — Full technology roadmap with maturity indicators
- GET /api/adoption/:technology — Detailed adoption analysis, migration paths, risks, timelines
- All v0.2.0 routes registered in index.ts
 2026-03-31 08:57:03 +02:00
Rene Fichtmueller
aa977abc97 feat(v0.2.0): Sales Intelligence Engine — Phase 0+A 
New API routes:
- GET /api/finder — Switch→Flexoptix transceiver finder with FlexBox coding
- GET /api/competitor-alerts — Competitor intelligence (price changes, new products, stock)
- GET /api/forecast/:technology — Sales forecast 3/9/12/18 months + buy/wait/hold signal
- POST /api/transport/plan — Transport system planner (city→city BOM with fiber providers)

New MCP tools:
- find_flexoptix_for_switch — Customer switch → Flexoptix products
- get_competitor_alerts — Competitor monitoring
- plan_transport — Network transport planning
- forecast_sales — Volume/revenue prediction
- generate_blog — Enhanced blog generation

New DB tables (migration 013):
- competitor_alerts, price_changes, flexoptix_product_map
- sales_forecasts, fiber_providers, fiber_routes, cities
- generated_datasheets, blog_series
- Views: v_price_coverage, v_image_coverage, v_switch_flexoptix_finder

Seed data (migration 014):
- 25 European cities with IX/DC locations + coordinates
- 15 fiber providers (euNetworks, Telia, DTAG, Colt, Zayo, etc.)
- 16 fiber routes with pricing (Germany focus)

Infrastructure:
- Scraper scheduler: 2h Flexoptix, 4h FS.com/Optcore (was 6-8h)
- Change detector for competitor price/stock monitoring
- Image downloader utility with coverage tracking
 2026-03-31 08:51:22 +02:00
Rene Fichtmueller
5a0cbed5a2 feat: dashboard v2, blog expansion, market/cable MCP tools, switch asset scrapers, scraper utilities 2026-03-30 08:07:12 +02:00
Rene Fichtmueller
c6308e93c0 feat: massive scraper expansion + hype cycle engine + lifecycle prediction 
New scrapers:
- GBICS.com (BigCommerce, GBP prices, 10 categories, 78 products)
- Juniper HCT (Next.js SSR parser, 475 transceivers with specs/EOL)
- SFPcables.com (Magento store, 16 categories, 78 products)
- Fluxlight (BigCommerce, 6 pages, 118 products)
- Champion ONE (compatible vendor scraper)

Scraper fixes:
- 10Gtek: rewritten to parse HTML spec tables (152 products)
- Flexoptix: fix price extraction from Magento Hyva HTML
- Register all scrapers in CLI (--gbics, --juniper, --sfpcables, etc.)

Hype Cycle Engine enhancements:
- Data-driven enrichment from scraped vendor/price data
- Revenue lifecycle prediction (peak year, decline, revenue index)
- Regional adoption model (NA, China, APAC, Europe, RoW with lag coefficients)
- New API endpoints: /enriched, /lifecycle, /regional/:tech

DB growth: 89 → 1,168 transceivers, 0 → 416 prices, 6 vendors
Qdrant: 1,162 products embedded with nomic-embed-text

Research: Norton-Bass model, standards-to-market timelines, hype signals
 2026-03-28 02:30:19 +13:00
Rene Fichtmueller
f48a809e40 feat: Phase 7 — Blog generator + scraper scheduler activation 
Blog draft engine generates structured markdown from all Qdrant
collections (products, news, FAQ, troubleshooting). Supports 4
topic types: hype_cycle, comparison, new_product, tutorial.

- routes/blog.ts: POST /api/blog/generate, GET/PUT endpoints
- ecosystem.config.js: Added tip-scraper PM2 process
- Scraper scheduler (pg-boss) now running on Erik with 8 job queues
- News scraper running every 6 hours on Erik
 2026-03-28 00:32:08 +13:00
Rene Fichtmueller
122ca8444d feat: Phase 5 — OCR pipeline + document/news search 
Docling-powered OCR pipeline: PDF → markdown → chunks → Ollama embed → Qdrant.
News embedding seeder for news_embeddings collection.
Document and news semantic search API endpoints.

- embeddings/ocr-pipeline.ts: Docling convert → chunk → embed pipeline
- embeddings/seed-news.ts: Batch embed news_articles into Qdrant
- routes/documents.ts: POST /api/documents/process, GET /api/documents
- routes/search.ts: GET /search/documents, GET /search/news endpoints
- sql/005-documents.sql: Add chunks_count, processed_at to documents table
- Ollama + nomic-embed-text installed on Erik (CPU mode)
- 89 products + 40 datasheet chunks + 33 news articles in Qdrant
 2026-03-28 00:22:01 +13:00
Rene Fichtmueller
0260d0b365 feat: Phase 4 — Vector embeddings + semantic search 
Ollama nomic-embed-text (768 dim) → Qdrant vector search pipeline.
Embeds all 89 transceivers with rich text representation and payload
filters (form_factor, speed_gbps, fiber_type, wdm_type).

- embeddings/client.ts: Ollama embed + Qdrant upsert/search
- embeddings/seed-products.ts: Batch seeder for product_embeddings
- routes/search.ts: GET /api/search, /search/products, /search/stats
- 6 Qdrant collections: products, datasheets, FAQs, manuals, troubleshooting, news
 2026-03-28 00:05:29 +13:00
Rene Fichtmueller
a6f7968393 feat: Phase 3 — Norton-Bass Hype Cycle Engine 
Implements the full Norton-Bass Multigenerational Diffusion Model for
transceiver technology lifecycle forecasting.

Math: Bass diffusion F(t) + logistic adoption S(t) = L / (1 + e^(-k(t-t0)))
Parameters: p (innovation ~0.03), q (imitation ~0.3-0.5), m (market potential)

Phase Classification Engine (composite score):
  30% Port shipment share + 20% ASP decline rate + 15% Standards maturity
  + 15% Interop validation + 10% Vendor trajectory + 10% Media sentiment

11 technologies tracked: 1G → 10G → 25G → 40G → 100G → 400G → 800G → 1.6T
  + CPO, LPO, 400ZR Coherent
5-year adoption forecast per technology

API: GET /api/hype-cycle (all) + GET /api/hype-cycle/:tech (detail)
Live: https://transceiver-db.context-x.org/api/hype-cycle
 2026-03-27 23:35:57 +13:00
Rene Fichtmueller
b43bdd3060 feat: TIP Phase 0+1 — monorepo, DB schema, API, scraper engine 
Phase 0 - Foundation:
- Restructure into npm workspace monorepo (packages/core, api, scraper)
- PostgreSQL 17 + TimescaleDB schema (15 tables incl. hypertables)
- Docker Compose for local dev (PostgreSQL on 5433 + Qdrant)
- Express 5 API on port 3200 with 6 routes
- Seed script to migrate 159 transceivers + 42 standards from npm package
- Erik server setup script + PM2 ecosystem config

Phase 1 - Scraper Engine:
- Crawlee + Playwright framework with pg-boss scheduler
- FS.com scraper (PlaywrightCrawler, anti-bot workaround)
- Optcore.net scraper (WP REST API enumeration + PlaywrightCrawler)
  - Uses /wp-json/wp/v2/product to get 2000+ product URLs
  - Playwright renders individual product pages for price extraction
- Cisco TMG Matrix scraper (compatibility data)
- News RSS aggregator (optics.org, SPIE, Network World, Nature Photonics)
  - Keyword relevance scoring for transceiver/fiber topics
  - xml2js with malformed XML sanitization
- SHA-256 content hashing for change detection (skip unchanged records)
- pg-boss v10 with explicit queue creation before scheduling
 2026-03-27 16:27:31 +13:00