75cbd7cd86
- Migration 028 (retroactive): document warehouse columns added to stock_observations - Migration 037: composite indexes for DISTINCT ON (transceiver_id, source_vendor_id) queries - Migration 038: add stock_confidence (1/2/3), price_currency, price_includes_tax, stock_vendor_ts to stock_observations + TRUNCATE test-run data db.ts: upsertStockObservation now accepts stockConfidence, priceCurrency, priceIncludesTax, stockVendorTs; delta detection includes quantity_available fs-com.ts: passes stockConfidence=3 + priceCurrency=EUR + priceIncludesTax=false qsfptek.ts v2: Phase 1 API listing + Phase 2 detail-page stock extraction - Parses 'X in real-time stock, DATE' from product detail pages - Writes stock_observations with confidence=2 + stockVendorTs - Up to 500 detail pages/run at 2s rate limit naddod.ts v2: complete rewrite from WooCommerce to Astro sitemap-based - Discovers products via /sitemaps/products.xml (600+ products) - URL format: /products/XXXXX.html - Extracts 'In Stock: X' exact counts from SSR HTML - Writes both price + stock observations (confidence 1 or 2)
23 lines
1.3 KiB
SQL
23 lines
1.3 KiB
SQL
-- Migration 038: Add data-quality columns to stock_observations + selective cleanup
|
|
--
|
|
-- stock_confidence: 1 = boolean only (in_stock true/false)
|
|
-- 2 = aggregated global count (single number, e.g. QSFPTEK)
|
|
-- 3 = per-warehouse breakdown (e.g. FS.com DE/Global split)
|
|
-- price_currency: ISO 4217 code, e.g. 'USD', 'EUR', 'GBP'
|
|
-- price_includes_tax: true = gross price, false = net/excl. VAT
|
|
-- stock_vendor_ts: timestamp as reported by vendor (e.g. QSFPTEK "17 Apr 2026")
|
|
|
|
ALTER TABLE stock_observations
|
|
ADD COLUMN IF NOT EXISTS stock_confidence SMALLINT DEFAULT 1
|
|
CHECK (stock_confidence IN (1, 2, 3)),
|
|
ADD COLUMN IF NOT EXISTS price_currency CHAR(3),
|
|
ADD COLUMN IF NOT EXISTS price_includes_tax BOOLEAN,
|
|
ADD COLUMN IF NOT EXISTS stock_vendor_ts TIMESTAMPTZ;
|
|
|
|
-- ── Selective cleanup ───────────────────────────────────────────────────────
|
|
-- Truncate stock_observations: all 186 rows are from the first FS.com test run.
|
|
-- They will be repopulated automatically at the next scheduled scraper run
|
|
-- (02:00 / 10:00 / 18:00 via launchd). Transceiver catalog, specs, vendors,
|
|
-- price_observations, and all other tables are left untouched.
|
|
TRUNCATE TABLE stock_observations;
|