4299e24e13
FS.com renders per-warehouse availability as a bare word-date on each line
("2 Stk. im DE-Lager, 8. Juli 2026 Lieferbar"). The old regex only matched a
"Lieferung:"/"erwartet:" prefix or numeric dd.mm.yyyy, so warehouse_de/global
delivery_date + backorder_estimated_date were 0% populated across 85k rows, and
lead_time_days was never written at all.
- new fs-com-parse.ts: pure, unit-tested parsers (13 node:test cases).
- extractDeliveryDate finds the bare word/numeric date per warehouse, bounded to
its own line so a date never bleeds to a neighbouring warehouse.
- computeLeadTimeDays = min(delivery_date - today), floored at 0; threaded into
upsertStockObservation (now writes lead_time_days) and upsertPriceObservation.
- parseGermanQty: K/M abbreviations now treat '.' as a decimal ("4.8K"=4800,
"1.4M"=1.4M) instead of stripping it (was "1.4M"->14,000,000 — the outlier).
- units_sold sanity gate at ingest (FS_MAX_UNITS_SOLD, default 2,000,000).
- sql/122: purge impossible historical units_sold, re-document columns as collected.
55 lines
3.4 KiB
SQL
55 lines
3.4 KiB
SQL
-- Migration 122: FS.com delivery dates + lead_time now collected; purge units_sold garbage
|
||
-- ─────────────────────────────────────────────────────────────────────────────
|
||
-- Follow-up to migration 121, which documented lead_time_days as NOT COLLECTED and
|
||
-- flagged that populating it was a scraper-side task. That task is now done:
|
||
--
|
||
-- * fs-com.ts (via fs-com-parse.ts) parses the bare word-date FS.com renders on
|
||
-- each warehouse line ("… im DE-Lager, 8. Juli 2026 Lieferbar") into
|
||
-- warehouse_de_delivery_date / warehouse_global_delivery_date /
|
||
-- backorder_estimated_date, and derives lead_time_days = min(date − today).
|
||
-- * upsertStockObservation now writes lead_time_days.
|
||
--
|
||
-- Historical rows stay NULL until the next FS.com run re-scrapes each SKU; NULL
|
||
-- therefore still means "unknown / not yet captured", not "zero lead time".
|
||
--
|
||
-- Second fix: units_sold was ~10× inflated for decimal-abbreviated values
|
||
-- ("4.8K"→48000, "1.4M"→14000000) because the K/M parser stripped the decimal
|
||
-- point. The parser is fixed AND ingest now drops values above a sanity cap.
|
||
-- This migration removes the clearly-impossible historical values (the multi-
|
||
-- million "outliers") so no consumer ingests them before the re-scrape. Sub-cap
|
||
-- values that may still be mildly inflated self-correct on the next FS.com run
|
||
-- (a changed value writes a fresh observation; consumers read the latest).
|
||
-- ─────────────────────────────────────────────────────────────────────────────
|
||
|
||
-- 1. Purge impossible units_sold (matches the ingest gate default, FS_MAX_UNITS_SOLD=2,000,000).
|
||
UPDATE stock_observations
|
||
SET units_sold = NULL
|
||
WHERE units_sold IS NOT NULL
|
||
AND units_sold > 2000000;
|
||
|
||
-- 2. Re-document the columns as collected-by-scraper (reversing migration 121's note).
|
||
COMMENT ON COLUMN stock_observations.warehouse_de_delivery_date IS
|
||
'FS.com DE-Lager availability date (collected 2026-07-04+ by fs-com-parse). '
|
||
'NULL = warehouse in stock now, vendor did not quote a date, or SKU not yet re-scraped.';
|
||
|
||
COMMENT ON COLUMN stock_observations.warehouse_global_delivery_date IS
|
||
'FS.com Global-Lager availability date (collected 2026-07-04+ by fs-com-parse). '
|
||
'NULL semantics as warehouse_de_delivery_date.';
|
||
|
||
COMMENT ON COLUMN stock_observations.backorder_estimated_date IS
|
||
'FS.com Nachlieferung (backorder) estimated date (collected 2026-07-04+ by fs-com-parse). '
|
||
'NULL = no backorder date quoted or SKU not yet re-scraped.';
|
||
|
||
COMMENT ON COLUMN stock_observations.lead_time_days IS
|
||
'Soonest availability in days = min(delivery_date − today) across warehouses that quote a '
|
||
'date. Collected 2026-07-04+ by the FS.com scraper; other vendors do not expose a lead time. '
|
||
'Treat NULL as "unknown / not captured" (do NOT default to 0).';
|
||
|
||
COMMENT ON COLUMN stock_observations.units_sold IS
|
||
'FS.com "verkauft" lifetime sell-through counter. Parser de-abbreviates K/M correctly and '
|
||
'ingest drops values > 2,000,000 as implausible. FS.com only; NULL elsewhere.';
|
||
|
||
INSERT INTO _migrations (filename)
|
||
VALUES ('122-fs-com-delivery-dates-and-units-sold-gate.sql')
|
||
ON CONFLICT (filename) DO NOTHING;
|