- Replace plaintext tip (transceiver_db) and llm_gateway DB passwords with
env-var references guarded by fail-fast checks, across 15 scripts + sql runbook.
Covers quoted exports, unquoted psql calls, python env dicts and
env-or-hardcoded-default fallbacks.
- Untrack training-data/*.jsonl (~48MB, kept on disk) and gitignore *.jsonl.
- Add .security-scan-allowlist for legit false positives: vendor name flexoptix,
own homelab infra, sibling-project codenames in the sync journal, env-resolved
DB URIs, example placeholders, ML tokenizer tokens.
- Pre-push leak scanner goes from 397 to 0 findings, green without --no-verify.
Follow-up (not in this commit): rotate the leaked values (llm_gateway first,
tip fleet-wide coordinated); git history still holds the old passwords.
- index-pi.ts: removed Playwright scrapers (FS.COM, eBay enricher, switch assets)
added NADDOD (fetch-based, benefits from residential IP)
now 32 fetch-only queues safe for ARM/Pi without Chromium
- index-fs-only.ts: new dedicated FS.COM + NADDOD worker for Erik
routes through Pi SOCKS5 via PROXY_URLS=socks5://10.10.0.6:1080
Crawlee ProxyConfiguration automatically applies to Playwright crawler
- pi-scraper-setup.sh: removed inline index-pi.ts override (repo version now authoritative)
- CODEX-TASK-pi-scraper-deploy.md: full 9-step Codex spec for Pi fleet setup
covers WireGuard keypair, Erik peer config, setup script, ecosystem.config.js
- CODEX-TASK-zero-manual-review.md: deterministic equivalence matcher spec
- utils/logger.ts: minimal console-based logger (debug/info/warn/error)
used by community-issues and ebay-enricher scrapers
- scripts/pi-scraper-setup.sh: step 7 adds optional WireGuard setup
(pass WG_PRIVKEY + WG_ADDR env vars) — connects Pi to Erik for DB access
auto-detects dead ethernet and routes WG traffic via working interface
New scrapers (8):
- BlueOptics (EUR, every 4h)
- ShopFiber24 (EUR, every 4h)
- T&S Communication (USD, every 4h)
- SmartOptics (catalog, every 8h)
- HUBER+SUHNER (catalog, every 8h)
- Skylane Optics (USD, every 4h)
- AscentOptics (USD, every 4h)
- GAO Tek (USD, every 4h)
Scheduler: nightly window → 24/7 continuous (42 jobs total)
- Playwright scrapers: every 8h (FS.com, 10Gtek, ATGBICS, ProLabs)
- Fetch/Cheerio: every 4h (11 lightweight vendors)
- Flexoptix catalog: every 2h (primary price source)
- eBay enrichment: every 6h
- Compatibility matrices: every 12h
- Compute jobs: every 4h
Pi fleet: scripts/pi-scraper-setup.sh for one-command Pi node setup
