rene/transceiver-db
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(reorder-signals): delete-before-insert prevents unbounded table growth

Browse Source 
reorder_signals grew to 4.49M rows / 1.19GB — the compute job INSERTed a fresh
row per transceiver every 4h run but never deleted old ones (24h TTL filtered
them at read time via DISTINCT ON + expires_at, but they were never purged).
4.37M rows were already expired dead weight.

Fix: DELETE existing rows for a transceiver before inserting its new signal, so
the table holds exactly one (latest) row per transceiver. Cleaned up to 18,175
rows / 4.5MB (99.6% reclaimed, VACUUM FULL). Backup: reorder_signals_keep_bak_20260606.
Verified: re-running compute:reorder-signals keeps count stable at 18,175.
This commit is contained in:
Rene Fichtmueller 2026-06-06 21:39:57 +00:00
parent c67fef1862
commit bb4046bb1e
1 changed files with 3 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
packages/scraper/src/scrapers/market-intelligence.ts
View File

@ -279,6 +279,9 @@ export async function computeReorderSignals(): Promise<void> {
if (reasons.length === 0) reasons.push("Insufficient data for strong signal"); if (reasons.length === 0) reasons.push("Insufficient data for strong signal");
// Keep exactly one (latest) signal per transceiver — delete prior rows first.
// Without this the table grew to 4.5M rows (24h-TTL never cleaned up old runs).
await pool.query(`DELETE FROM reorder_signals WHERE transceiver_id = $1`, [row.id]);
await pool.query( await pool.query(
`INSERT INTO reorder_signals `INSERT INTO reorder_signals
(transceiver_id, signal, signal_strength, reasons, stock_trend, price_trend, lead_time_weeks) (transceiver_id, signal, signal_strength, reasons, stock_trend, price_trend, lead_time_weeks)