rene/transceiver-db
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
40 Commits 5 Branches 9 Tags
Commit Graph

4 Commits

Author SHA1 Message Date
Rene Fichtmueller
9bb2f549f8 fix(v0.2.2): OLLAMA_URL pointed to localhost instead of .213 via WireGuard 
Blog engine was falling back to template because qwen2.5:14b is on Mac Studio (.213),
not on Erik (localhost). Fixed ecosystem.config.js to use 192.168.178.213:11434.
This was the root cause why the 10-step pipeline never executed.
 2026-03-31 09:28:34 +02:00
Rene Fichtmueller
f8892f058f feat(deploy): add tip-mcp PM2 process (port 3202) + Cloudflare tunnel tip.context-x.org 2026-03-29 08:21:14 +08:00
Rene Fichtmueller
274b80a4f1 feat: Phase 7 — Blog generator + scraper scheduler activation 
Blog draft engine generates structured markdown from all Qdrant
collections (products, news, FAQ, troubleshooting). Supports 4
topic types: hype_cycle, comparison, new_product, tutorial.

- routes/blog.ts: POST /api/blog/generate, GET/PUT endpoints
- ecosystem.config.js: Added tip-scraper PM2 process
- Scraper scheduler (pg-boss) now running on Erik with 8 job queues
- News scraper running every 6 hours on Erik
 2026-03-28 00:32:08 +13:00
Rene Fichtmueller
e9fb50a248 feat: TIP Phase 0+1 — monorepo, DB schema, API, scraper engine 
Phase 0 - Foundation:
- Restructure into npm workspace monorepo (packages/core, api, scraper)
- PostgreSQL 17 + TimescaleDB schema (15 tables incl. hypertables)
- Docker Compose for local dev (PostgreSQL on 5433 + Qdrant)
- Express 5 API on port 3200 with 6 routes
- Seed script to migrate 159 transceivers + 42 standards from npm package
- Erik server setup script + PM2 ecosystem config

Phase 1 - Scraper Engine:
- Crawlee + Playwright framework with pg-boss scheduler
- FS.com scraper (PlaywrightCrawler, anti-bot workaround)
- Optcore.net scraper (WP REST API enumeration + PlaywrightCrawler)
  - Uses /wp-json/wp/v2/product to get 2000+ product URLs
  - Playwright renders individual product pages for price extraction
- Cisco TMG Matrix scraper (compatibility data)
- News RSS aggregator (optics.org, SPIE, Network World, Nature Photonics)
  - Keyword relevance scoring for transceiver/fiber topics
  - xml2js with malformed XML sanitization
- SHA-256 content hashing for change detection (skip unchanged records)
- pg-boss v10 with explicit queue creation before scheduling
 2026-03-27 16:27:31 +13:00