transceiver-db

rene.fichtmueller/transceiver-db

Fork 0

Commit Graph

Author	SHA1	Message	Date
Rene Fichtmueller	86d1531d52	fix(security): scrub hardcoded DB credentials, untrack training data, add scan allowlist - Replace plaintext tip (transceiver_db) and llm_gateway DB passwords with env-var references guarded by fail-fast checks, across 15 scripts + sql runbook. Covers quoted exports, unquoted psql calls, python env dicts and env-or-hardcoded-default fallbacks. - Untrack training-data/.jsonl (~48MB, kept on disk) and gitignore .jsonl. - Add .security-scan-allowlist for legit false positives: vendor name flexoptix, own homelab infra, sibling-project codenames in the sync journal, env-resolved DB URIs, example placeholders, ML tokenizer tokens. - Pre-push leak scanner goes from 397 to 0 findings, green without --no-verify. Follow-up (not in this commit): rotate the leaked values (llm_gateway first, tip fleet-wide coordinated); git history still holds the old passwords.	2026-07-07 12:06:07 +02:00
Rene Fichtmueller	e9fb50a248	feat: TIP Phase 0+1 — monorepo, DB schema, API, scraper engine Phase 0 - Foundation: - Restructure into npm workspace monorepo (packages/core, api, scraper) - PostgreSQL 17 + TimescaleDB schema (15 tables incl. hypertables) - Docker Compose for local dev (PostgreSQL on 5433 + Qdrant) - Express 5 API on port 3200 with 6 routes - Seed script to migrate 159 transceivers + 42 standards from npm package - Erik server setup script + PM2 ecosystem config Phase 1 - Scraper Engine: - Crawlee + Playwright framework with pg-boss scheduler - FS.com scraper (PlaywrightCrawler, anti-bot workaround) - Optcore.net scraper (WP REST API enumeration + PlaywrightCrawler) - Uses /wp-json/wp/v2/product to get 2000+ product URLs - Playwright renders individual product pages for price extraction - Cisco TMG Matrix scraper (compatibility data) - News RSS aggregator (optics.org, SPIE, Network World, Nature Photonics) - Keyword relevance scoring for transceiver/fiber topics - xml2js with malformed XML sanitization - SHA-256 content hashing for change detection (skip unchanged records) - pg-boss v10 with explicit queue creation before scheduling	2026-03-27 16:27:31 +13:00

Author

SHA1

Message

Date

Rene Fichtmueller

86d1531d52

fix(security): scrub hardcoded DB credentials, untrack training data, add scan allowlist

- Replace plaintext tip (transceiver_db) and llm_gateway DB passwords with
  env-var references guarded by fail-fast checks, across 15 scripts + sql runbook.
  Covers quoted exports, unquoted psql calls, python env dicts and
  env-or-hardcoded-default fallbacks.
- Untrack training-data/*.jsonl (~48MB, kept on disk) and gitignore *.jsonl.
- Add .security-scan-allowlist for legit false positives: vendor name flexoptix,
  own homelab infra, sibling-project codenames in the sync journal, env-resolved
  DB URIs, example placeholders, ML tokenizer tokens.
- Pre-push leak scanner goes from 397 to 0 findings, green without --no-verify.

Follow-up (not in this commit): rotate the leaked values (llm_gateway first,
tip fleet-wide coordinated); git history still holds the old passwords.

2026-07-07 12:06:07 +02:00

Rene Fichtmueller

e9fb50a248

feat: TIP Phase 0+1 — monorepo, DB schema, API, scraper engine

Phase 0 - Foundation:
- Restructure into npm workspace monorepo (packages/core, api, scraper)
- PostgreSQL 17 + TimescaleDB schema (15 tables incl. hypertables)
- Docker Compose for local dev (PostgreSQL on 5433 + Qdrant)
- Express 5 API on port 3200 with 6 routes
- Seed script to migrate 159 transceivers + 42 standards from npm package
- Erik server setup script + PM2 ecosystem config

Phase 1 - Scraper Engine:
- Crawlee + Playwright framework with pg-boss scheduler
- FS.com scraper (PlaywrightCrawler, anti-bot workaround)
- Optcore.net scraper (WP REST API enumeration + PlaywrightCrawler)
  - Uses /wp-json/wp/v2/product to get 2000+ product URLs
  - Playwright renders individual product pages for price extraction
- Cisco TMG Matrix scraper (compatibility data)
- News RSS aggregator (optics.org, SPIE, Network World, Nature Photonics)
  - Keyword relevance scoring for transceiver/fiber topics
  - xml2js with malformed XML sanitization
- SHA-256 content hashing for change detection (skip unchanged records)
- pg-boss v10 with explicit queue creation before scheduling

2026-03-27 16:27:31 +13:00

2 Commits