fix(security): scrub hardcoded DB credentials, untrack training data, add scan allowlist

- Replace plaintext tip (transceiver_db) and llm_gateway DB passwords with
  env-var references guarded by fail-fast checks, across 15 scripts + sql runbook.
  Covers quoted exports, unquoted psql calls, python env dicts and
  env-or-hardcoded-default fallbacks.
- Untrack training-data/*.jsonl (~48MB, kept on disk) and gitignore *.jsonl.
- Add .security-scan-allowlist for legit false positives: vendor name flexoptix,
  own homelab infra, sibling-project codenames in the sync journal, env-resolved
  DB URIs, example placeholders, ML tokenizer tokens.
- Pre-push leak scanner goes from 397 to 0 findings, green without --no-verify.

Follow-up (not in this commit): rotate the leaked values (llm_gateway first,
tip fleet-wide coordinated); git history still holds the old passwords.

.gitignore

@@ -21,3 +21,8 @@ storage-*/
 # Local credentials (never commit)
 .tip/.env
 run-fs-scraper-mac.sh.local
+
+# Training / SFT datasets — bulk data + potential internal pricing/market intel, keep out of git
+*.jsonl
+training-data/runpod/**/*.jsonl
+training-data/*.jsonl

.security-scan-allowlist

@@ -0,0 +1,37 @@
+# transceiver-db pre-push scan allowlist. ERE, one pattern per line, matched against finding lines.
+# This repo is Gitea-internal and is never published publicly; entries below are confirmed false positives.
+# Risk note: a low-entropy secret sharing a line with an allowlisted token would be masked here.
+# High-entropy secrets are still caught by the gitleaks pass, which ignores this allowlist.
+# If this repo is ever made public, drop this file and run the internas sanitizer first.
+# --- vendor name that is legitimate market-intelligence dataset subject matter ---
+flexoptix
+# --- owner homelab infra referenced in runbooks, docs and deploy scripts ---
+82\.165\.222\.127
+192\.168\.178\.
+10\.10\.0\.
+\berik\b
+\bclaudi\b
+renefichtmueller
+fearghas
+/opt/[a-z]
+fichtmueller\.org
+context-x\.org
+# --- sibling project codenames referenced in the cross-project sync dev journal ---
+\bmagatama
+peercortex
+switchblade
+shieldx
+papercortex
+ctxevent
+ctxmeet
+nognet\.net
+\bpulso\b
+# --- env-resolved DB URIs and passwords, no literal secret present ---
+postgres(ql)?://\$\{
+\$\{DB_PASS\}
+process\.env\.PGPASSWORD
+# --- placeholder secret in example config ---
+CHANGE_ME
+# --- ML tokenizer special tokens, not credentials ---
+special_tokens
+pad_token

CHANGELOG_PENDING.md

@@ -324,3 +324,4 @@ Types: FEAT · FIX · UI · DATA · AI · INFRA
 {"d":"2026-07-04","t":"DATA","m":"form_factor-Recovery (sql/120): korrigiert NUR beweisbar-falsche Zeilen (form_factors.max_speed < speed_gbps) mit explizitem Formfaktor-Token im Titel. 87 gefixt (u.a. 400G-Kabel als SFP+ getaggt -> QSFP-DD; QSFP-DD@800G -> QSFP-DD800; Q56DD -> QSFP-DD), 947 opake Zeilen (z.B. EOLS-1303-40-D, MGB-2GSR) unangetastet gelassen, nichts geraten. Gescraptes Original in neuer Spalte form_factor_raw bewahrt. Recovery auch im Ingest-Trigger (Neuzugänge self-healing)."}
 {"d":"2026-07-04","t":"DATA","m":"lead_time_days als NICHT ERFASST dokumentiert (sql/121, COMMENT auf price_observations/stock_observations/stock_snapshots): 0% befüllt über 1.378.984 + 85.406 + 0 Zeilen. Plumbing existiert (crawler-llm stock-schema erfasst es, upsertPriceObservation nimmt Param), aber aktiver Scraper-Pfad liefert nie einen Wert und LLM-Crawler-Tabelle stock_snapshots ist leer. Kein Default fabrizieren; NULL = unbekannt. Befüllen ist Scraper-Aufgabe."}
 {"d":"2026-07-04","t":"DATA","m":"Befund FS.COM-Untererfassung bestätigt (kein Fix, Scraper-Thema): 392 Zeilen/378 distinct SKUs vs ATGBICS 8420 (22x-Lücke). 391/392 sind bereits product_type='module' -> Klassifikation ist NICHT die Ursache, reine Crawl-Tiefe. Ursache in fs-com.ts + Seed-Scope (Kategorie-Abdeckung) + evtl. zu aggressive Quarantäne von fs.com/c/-URLs. Empfehlung: FS.com-Crawl-Seed erweitern (Scraper-Pipeline, nicht raten)."}
+{"d":"2026-07-07","t":"SECURITY","m":"Hardcodierte DB-Passwoerter aus dem Source entfernt und auf env umgestellt. tip-DB-Passwort (transceiver_db) lag in 14 Dateien im Klartext (quoted export, unquoted psql-Aufrufe, Python-env-Dicts, env-or-hardcoded-default-Fallbacks); llm_gateway-Passwort in seed-blog-training-data.py. Ersetzt durch PGPASSWORD-Env-Referenz mit Fail-Fast-Guard bzw. os.environ. training-data/*.jsonl (48 MB) aus Tracking genommen (git rm --cached, Datei bleibt) und gitignored. .security-scan-allowlist ergaenzt fuer legitime False-Positives (Vendor flexoptix, eigene Infra, sync-Journal-Codenamen, env-aufgeloeste URIs). Pre-Push-Leak-Scanner 397 auf 0 Funde, gruen ohne --no-verify. Offen: Rotation der geleakten Werte (llm_gateway zuerst, tip fleet-weit koordiniert); Git-History behaelt Altwerte."}

scripts/deploy-all-fixes.sh

@@ -9,11 +9,11 @@ python3 /tmp/fix-sql-dedup.py >> "$LOG" 2>&1
 
 # Step 2: Re-apply enrichment
 echo "Step 2: Applying enrichment..." >> "$LOG"
-PGPASSWORD=***REDACTED*** psql -h localhost -p 5433 -U tip -d transceiver_db -f /tmp/011-flexoptix-enrichment.sql >> "$LOG" 2>&1
+PGPASSWORD="${PGPASSWORD:?set PGPASSWORD}" psql -h localhost -p 5433 -U tip -d transceiver_db -f /tmp/011-flexoptix-enrichment.sql >> "$LOG" 2>&1
 
 # Step 3: Apply switches SQL
 echo "Step 3: Applying switches..." >> "$LOG"
-PGPASSWORD=***REDACTED*** psql -h localhost -p 5433 -U tip -d transceiver_db -f /tmp/012-more-switches.sql >> "$LOG" 2>&1
+PGPASSWORD="${PGPASSWORD:?set PGPASSWORD}" psql -h localhost -p 5433 -U tip -d transceiver_db -f /tmp/012-more-switches.sql >> "$LOG" 2>&1
 
 # Step 4: Restart API
 echo "Step 4: Restarting API..." >> "$LOG"
@@ -22,10 +22,10 @@ cd /opt/tip && pm2 restart tip-api >> "$LOG" 2>&1
 # Step 5: Results
 echo "" >> "$LOG"
 echo "=== RESULTS ===" >> "$LOG"
-PGPASSWORD=***REDACTED*** psql -h localhost -p 5433 -U tip -d transceiver_db -t -A -c "SELECT 'images: ' || count(*) FROM transceivers WHERE image_url IS NOT NULL" >> "$LOG" 2>&1
-PGPASSWORD=***REDACTED*** psql -h localhost -p 5433 -U tip -d transceiver_db -t -A -c "SELECT 'connector: ' || count(*) FROM transceivers WHERE connector IS NOT NULL" >> "$LOG" 2>&1
-PGPASSWORD=***REDACTED*** psql -h localhost -p 5433 -U tip -d transceiver_db -t -A -c "SELECT 'notes: ' || count(*) FROM transceivers WHERE notes IS NOT NULL AND notes != ''" >> "$LOG" 2>&1
-PGPASSWORD=***REDACTED*** psql -h localhost -p 5433 -U tip -d transceiver_db -t -A -c "SELECT 'switches: ' || count(*) FROM switches" >> "$LOG" 2>&1
-PGPASSWORD=***REDACTED*** psql -h localhost -p 5433 -U tip -d transceiver_db -t -A -c "SELECT 'sw_desc: ' || count(*) FROM switches WHERE description IS NOT NULL" >> "$LOG" 2>&1
+PGPASSWORD="${PGPASSWORD:?set PGPASSWORD}" psql -h localhost -p 5433 -U tip -d transceiver_db -t -A -c "SELECT 'images: ' || count(*) FROM transceivers WHERE image_url IS NOT NULL" >> "$LOG" 2>&1
+PGPASSWORD="${PGPASSWORD:?set PGPASSWORD}" psql -h localhost -p 5433 -U tip -d transceiver_db -t -A -c "SELECT 'connector: ' || count(*) FROM transceivers WHERE connector IS NOT NULL" >> "$LOG" 2>&1
+PGPASSWORD="${PGPASSWORD:?set PGPASSWORD}" psql -h localhost -p 5433 -U tip -d transceiver_db -t -A -c "SELECT 'notes: ' || count(*) FROM transceivers WHERE notes IS NOT NULL AND notes != ''" >> "$LOG" 2>&1
+PGPASSWORD="${PGPASSWORD:?set PGPASSWORD}" psql -h localhost -p 5433 -U tip -d transceiver_db -t -A -c "SELECT 'switches: ' || count(*) FROM switches" >> "$LOG" 2>&1
+PGPASSWORD="${PGPASSWORD:?set PGPASSWORD}" psql -h localhost -p 5433 -U tip -d transceiver_db -t -A -c "SELECT 'sw_desc: ' || count(*) FROM switches WHERE description IS NOT NULL" >> "$LOG" 2>&1
 
 echo "$(date): ALL DONE" >> "$LOG"

scripts/enrich-on-erik.sh

@@ -2,7 +2,7 @@
 # Self-contained Flexoptix enrichment script to run ON Erik
 # Does: DB query → scrape flexoptix.net → generate SQL → apply to DB
 
-DB_PASS="***REDACTED***"
+DB_PASS="${PGPASSWORD:?set PGPASSWORD (e.g. in ~/.tip/.env)}"
 DB_USER="tip"
 DB_NAME="transceiver_db"
 DB_PORT="5433"

scripts/enrich-specs.ts

@@ -16,7 +16,7 @@ const pool = new Pool({
   port: parseInt(process.env.POSTGRES_PORT || "5433"),
   database: process.env.POSTGRES_DB || "transceiver_db",
   user: process.env.POSTGRES_USER || "tip",
-  password: process.env.POSTGRES_PASSWORD || "***REDACTED***",
+  password: process.env.POSTGRES_PASSWORD,
   max: 5,
 });
 

scripts/enrich-v2.sh

@@ -4,7 +4,7 @@
 
 LOG="/tmp/enrich-v2.log"
 SQL="/tmp/011-flexoptix-enrichment-v2.sql"
-DB="PGPASSWORD=***REDACTED*** psql -h localhost -p 5433 -U tip -d transceiver_db"
+DB="PGPASSWORD=${PGPASSWORD:?set PGPASSWORD} psql -h localhost -p 5433 -U tip -d transceiver_db"
 
 echo "$(date): Starting V2 enrichment" > "$LOG"
 

scripts/enrich-v3.sh

@@ -6,7 +6,7 @@ SQL="/tmp/enrichment-v3.sql"
 echo "$(date): V3 start" > "$LOG"
 
 # Direct psql (no eval)
-export PGPASSWORD="***REDACTED***"
+export PGPASSWORD="${PGPASSWORD:?set PGPASSWORD (e.g. in ~/.tip/.env)}"
 
 psql -h localhost -p 5433 -U tip -d transceiver_db -t -A -F'|' -c \
   "SELECT t.id, t.product_page_url, t.part_number, t.standard_name FROM transceivers t JOIN vendors v ON t.vendor_id = v.id WHERE v.name = 'FLEXOPTIX' AND t.product_page_url IS NOT NULL" \

scripts/enrich-v4.py

@@ -6,7 +6,7 @@ import sys
 import time
 import os
 
-DB_CMD = "PGPASSWORD=***REDACTED*** psql -h localhost -p 5433 -U tip -d transceiver_db"
+DB_CMD = f"PGPASSWORD={os.environ['PGPASSWORD']} psql -h localhost -p 5433 -U tip -d transceiver_db"
 SQL_OUT = "/tmp/enrichment-v4.sql"
 LOG = "/tmp/enrich-v4.log"
 
@@ -150,7 +150,7 @@ log(f"SQL at: {SQL_OUT}")
 
 # Apply
 log("Applying SQL...")
-os.environ["PGPASSWORD"] = "***REDACTED***"
+if not os.environ.get("PGPASSWORD"): raise SystemExit("PGPASSWORD env var required")
 r = subprocess.run(
     ["psql", "-h", "localhost", "-p", "5433", "-U", "tip", "-d", "transceiver_db", "-f", SQL_OUT],
     capture_output=True, text=True
@@ -175,7 +175,7 @@ for query in [
 ]:
     r = subprocess.run(
         ["psql", "-h", "localhost", "-p", "5433", "-U", "tip", "-d", "transceiver_db", "-t", "-A", "-c", query],
-        capture_output=True, text=True, env={**os.environ, "PGPASSWORD": "***REDACTED***"}
+        capture_output=True, text=True, env={**os.environ}
     )
     log(r.stdout.strip())
 

scripts/fix-and-apply-enrichment.sh

@@ -41,15 +41,15 @@ print('Fixed temp_range values')
 " >> "$LOG" 2>&1
 
 echo "Re-applying SQL..." >> "$LOG"
-PGPASSWORD=***REDACTED*** psql -h localhost -p 5433 -U tip -d transceiver_db -f "$SQL" >> "$LOG" 2>&1
+PGPASSWORD="${PGPASSWORD:?set PGPASSWORD}" psql -h localhost -p 5433 -U tip -d transceiver_db -f "$SQL" >> "$LOG" 2>&1
 
 echo "" >> "$LOG"
 echo "=== RESULTS ===" >> "$LOG"
-PGPASSWORD=***REDACTED*** psql -h localhost -p 5433 -U tip -d transceiver_db -t -A -c "SELECT count(*) as img FROM transceivers WHERE image_url IS NOT NULL" >> "$LOG" 2>&1
+PGPASSWORD="${PGPASSWORD:?set PGPASSWORD}" psql -h localhost -p 5433 -U tip -d transceiver_db -t -A -c "SELECT count(*) as img FROM transceivers WHERE image_url IS NOT NULL" >> "$LOG" 2>&1
 echo " transceivers have images" >> "$LOG"
-PGPASSWORD=***REDACTED*** psql -h localhost -p 5433 -U tip -d transceiver_db -t -A -c "SELECT count(*) FROM transceivers WHERE notes IS NOT NULL AND notes != ''" >> "$LOG" 2>&1
+PGPASSWORD="${PGPASSWORD:?set PGPASSWORD}" psql -h localhost -p 5433 -U tip -d transceiver_db -t -A -c "SELECT count(*) FROM transceivers WHERE notes IS NOT NULL AND notes != ''" >> "$LOG" 2>&1
 echo " transceivers have enriched notes" >> "$LOG"
-PGPASSWORD=***REDACTED*** psql -h localhost -p 5433 -U tip -d transceiver_db -t -A -c "SELECT count(*) FROM transceivers WHERE connector IS NOT NULL" >> "$LOG" 2>&1
+PGPASSWORD="${PGPASSWORD:?set PGPASSWORD}" psql -h localhost -p 5433 -U tip -d transceiver_db -t -A -c "SELECT count(*) FROM transceivers WHERE connector IS NOT NULL" >> "$LOG" 2>&1
 echo " transceivers have connector" >> "$LOG"
 
 echo "$(date): DONE" >> "$LOG"

scripts/fix-mega.py

@@ -12,7 +12,7 @@ def run_sql(sql):
     r = subprocess.run(
         ["psql", "-h", "localhost", "-p", "5433", "-U", "tip", "-d", "transceiver_db", "-c", sql],
         capture_output=True, text=True,
-        env={**os.environ, "PGPASSWORD": "***REDACTED***"}
+        env={**os.environ}
     )
     if "ERROR" in r.stderr:
         print(f"ERR: {r.stderr.strip()[:200]}")
@@ -22,7 +22,7 @@ def query_val(sql):
     r = subprocess.run(
         ["psql", "-h", "localhost", "-p", "5433", "-U", "tip", "-d", "transceiver_db", "-t", "-A", "-c", sql],
         capture_output=True, text=True,
-        env={**os.environ, "PGPASSWORD": "***REDACTED***"}
+        env={**os.environ}
     )
     return r.stdout.strip()
 

scripts/generate-compatibility.py

@@ -31,7 +31,7 @@ def query(sql):
         ["psql", "-h", "localhost", "-p", "5433", "-U", "tip", "-d", "transceiver_db",
          "-t", "-A", "-F", "|", "-c", sql],
         capture_output=True, text=True,
-        env={**os.environ, "PGPASSWORD": "***REDACTED***"}
+        env={**os.environ}
     )
     rows = []
     for line in r.stdout.strip().split("\n"):
@@ -140,7 +140,7 @@ log("Applying...")
 r = subprocess.run(
     ["psql", "-h", "localhost", "-p", "5433", "-U", "tip", "-d", "transceiver_db", "-f", SQL_OUT],
     capture_output=True, text=True,
-    env={**os.environ, "PGPASSWORD": "***REDACTED***"}
+    env={**os.environ}
 )
 errors = [l for l in r.stderr.split("\n") if "ERROR" in l]
 if errors:
@@ -157,7 +157,7 @@ for col_sql in [
     subprocess.run(
         ["psql", "-h", "localhost", "-p", "5433", "-U", "tip", "-d", "transceiver_db", "-c", col_sql],
         capture_output=True, text=True,
-        env={**os.environ, "PGPASSWORD": "***REDACTED***"}
+        env={**os.environ}
     )
 
 # Mark whitebox switches
@@ -170,7 +170,7 @@ WHERE model IN ('SN2201', 'SN3700', 'SN3750-SX', 'SN4700', 'SN5400', 'SN5600');
 subprocess.run(
     ["psql", "-h", "localhost", "-p", "5433", "-U", "tip", "-d", "transceiver_db", "-c", whitebox_sql],
     capture_output=True, text=True,
-    env={**os.environ, "PGPASSWORD": "***REDACTED***"}
+    env={**os.environ}
 )
 
 # Restart API
@@ -185,7 +185,7 @@ for q in [
     r = subprocess.run(
         ["psql", "-h", "localhost", "-p", "5433", "-U", "tip", "-d", "transceiver_db", "-t", "-A", "-c", q],
         capture_output=True, text=True,
-        env={**os.environ, "PGPASSWORD": "***REDACTED***"}
+        env={**os.environ}
     )
     log(r.stdout.strip())
 

scripts/mega-enrich.py

@@ -34,7 +34,7 @@ def query(sql):
         ["psql", "-h", "localhost", "-p", "5433", "-U", "tip", "-d", "transceiver_db",
          "-t", "-A", "-F", "|", "-c", sql],
         capture_output=True, text=True,
-        env={**os.environ, "PGPASSWORD": "***REDACTED***"}
+        env={**os.environ}
     )
     return [line.split("|") for line in r.stdout.strip().split("\n") if line.strip()]
 
@@ -43,7 +43,7 @@ def run_sql(sql):
         ["psql", "-h", "localhost", "-p", "5433", "-U", "tip", "-d", "transceiver_db",
          "-c", sql],
         capture_output=True, text=True,
-        env={**os.environ, "PGPASSWORD": "***REDACTED***"}
+        env={**os.environ}
     )
 
 log(f"{time.strftime('%Y-%m-%d %H:%M:%S')}: MEGA ENRICHMENT START")
@@ -515,7 +515,7 @@ log("Applying SQL...")
 r = subprocess.run(
     ["psql", "-h", "localhost", "-p", "5433", "-U", "tip", "-d", "transceiver_db", "-f", SQL_OUT],
     capture_output=True, text=True,
-    env={**os.environ, "PGPASSWORD": "***REDACTED***"}
+    env={**os.environ}
 )
 
 errors = [l for l in r.stderr.split("\n") if "ERROR" in l]
@@ -547,7 +547,7 @@ for q in [
     r = subprocess.run(
         ["psql", "-h", "localhost", "-p", "5433", "-U", "tip", "-d", "transceiver_db", "-t", "-A", "-c", q],
         capture_output=True, text=True,
-        env={**os.environ, "PGPASSWORD": "***REDACTED***"}
+        env={**os.environ}
     )
     log(r.stdout.strip())
 

scripts/pi-scraper-setup.sh

@@ -22,7 +22,7 @@ PI_NAME="${PI_NAME:-pi-scraper}"        # override with PI_NAME=pi2 bash setup.s
 DB_HOST="${DB_HOST:-10.10.0.1}"         # Erik WireGuard IP
 DB_PORT="${DB_PORT:-5433}"
 DB_USER="${DB_USER:-tip}"
-DB_PASS="${DB_PASS:-***REDACTED***}"
+DB_PASS="${DB_PASS:?set DB_PASS}"
 DB_NAME="${DB_NAME:-transceiver_db}"
 GITEA="http://192.168.178.196:3000/rene/transceiver-db.git"
 INSTALL_DIR="/opt/tip-scraper"

scripts/seed-blog-training-data.py

@@ -180,9 +180,10 @@ def main():
     parser.add_argument("--db-url", default=None, help="PostgreSQL connection URL (overrides env)")
     args = parser.parse_args()
 
-    # Determine DB URL
-    db_url = args.db_url or os.environ.get("LLM_GATEWAY_DB_URL") or \
-             "postgresql://llm:llm_secure_2026@217.154.82.179:5432/llm_gateway"
+    # Determine DB URL (never hardcode credentials — require env or --db-url)
+    db_url = args.db_url or os.environ.get("LLM_GATEWAY_DB_URL")
+    if not db_url:
+        sys.exit("ERROR: set LLM_GATEWAY_DB_URL env var or pass --db-url")
 
     # Find training data directory
     script_dir = Path(__file__).parent

scripts/setup-erik.sh

@@ -30,7 +30,7 @@ systemctl enable postgresql
 
 # Create DB and user
 sudo -u postgres psql <<SQL
-CREATE USER tip WITH PASSWORD '${POSTGRES_PASSWORD:-***REDACTED***}';
+CREATE USER tip WITH PASSWORD '${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD}';
 CREATE DATABASE transceiver_db OWNER tip;
 GRANT ALL PRIVILEGES ON DATABASE transceiver_db TO tip;
 \c transceiver_db

scripts/validate-specs.ts

@@ -20,7 +20,7 @@ const pool = new Pool({
   port: parseInt(process.env.POSTGRES_PORT || "5433"),
   database: process.env.POSTGRES_DB || "transceiver_db",
   user: process.env.POSTGRES_USER || "tip",
-  password: process.env.POSTGRES_PASSWORD || "***REDACTED***",
+  password: process.env.POSTGRES_PASSWORD,
   max: 3,
 });
 

sql/010-vendor-urls.sql

@@ -1,5 +1,5 @@
 -- 010: Add image_url, product_page_url, datasheet_url columns and populate vendor URLs
--- Run on Erik: PGPASSWORD='***REDACTED***' psql -h localhost -p 5433 -U tip -d transceiver_db -f sql/010-vendor-urls.sql
+-- Run on Erik: PGPASSWORD="$PGPASSWORD" psql -h localhost -p 5433 -U tip -d transceiver_db -f sql/010-vendor-urls.sql
 
 -- Add columns (idempotent)
 ALTER TABLE transceivers ADD COLUMN IF NOT EXISTS image_url TEXT;