feat(tip): modulation/FEC enrichment + speed_gbps plausibility guard
sql/135: IMMUTABLE derivation helpers (modulation NRZ/PAM4/DP-16QAM/DP-QPSK/ Coherent, fec_type, inbuilt_fec, coherent) keyed off speed_gbps + form_factor + part_number/standard_name regex. Fill-only, idempotent via fn_tip_enrich_modulation_fec() + daily self-healing cron. Coverage >=100G 3-8% -> 46%. Rules built by a 4-lens optical-standards workflow, adversarially verified (10G-ZR, CFP2-DCO, AOC/DAC scope, bare-QSFP28 fixes folded in before apply). sql/136: speed_gbps made nullable + tip_speed_is_implausible() + BEFORE trigger trg_speed_plausibility (sanitize junk to NULL on write, never abort). One-time cleanup nulled 4708 junk rows (reversible via tip_speed_gbps_rollback); legit FC/SDI/CPRI/OTN gray band preserved. sql/137: v_transceiver_speed_errors flags only real optics -> 596 (was 2134).
This commit is contained in:
Rene Fichtmueller
parent
5280ac2001
commit
6d03fac017
@ -1,6 +1,9 @@
|
|||||||
# TIP Changelog
|
# TIP Changelog
|
||||||
|
|
||||||
Format: `{"d":"YYYY-MM-DD","t":"TYPE","m":"Description"}`
|
Format: `{"d":"YYYY-MM-DD","t":"TYPE","m":"Description"}`
|
||||||
|
{"d":"2026-07-11","t":"FEAT","m":"Modulation/FEC enrichment (sql/135): IMMUTABLE helpers derive modulation (NRZ/PAM4/DP-16QAM/DP-QPSK/Coherent), fec_type and inbuilt_fec from speed_gbps + form_factor + part_number/standard_name regex, incl. coherent detection (>=100G-floored ZR/DCO/OpenZR/QAM + CFP2-DCO/ACO via form_factor). Fill-only + idempotent via fn_tip_enrich_modulation_fec(); daily self-healing cron 03:30 (scripts/enrich-modulation-fec.sh). Modulation coverage >=100G rose from 3-8 percent to 46 percent (400G 59, 800G 56, 1600G 100); 4224 rows enriched. Rules built by a 4-lens optical-standards workflow and adversarially verified — the 10G-ZR mislabel, CFP2-DCO miss, AOC/DAC scope leak and bare-QSFP28 over-guess were all caught and fixed before apply. vendor_verified rows never overwritten; enriched rows tagged data_confidence=enriched_estimated + enriched_fields."}
|
||||||
|
{"d":"2026-07-11","t":"FIX","m":"speed_gbps plausibility guard + cleanup (sql/136): ~22 percent of rows carried junk speed (2129 at <=0, 167 astronomical >1600 values, ~2400 sub-0.09 fragments) from cable/MTP-trunk part numbers parsed into speed. speed_gbps made nullable (unknown speed is honestly NULL); tip_speed_is_implausible() + BEFORE trigger trg_speed_plausibility sanitize junk to NULL on write and never abort ingestion; one-time cleanup nulled 4708 rows (id-keyed reversible snapshot in tip_speed_gbps_rollback). The 0.09..1600 gray band (~355 legit FC/SDI/CPRI/OTN line rates) is intentionally preserved. 87 percent of nulled rows are dac/aoc/accessory where speed is meaningless."}
|
||||||
|
{"d":"2026-07-11","t":"FIX","m":"v_transceiver_speed_errors honest metric (sql/137): after 136 the view counted 4708 NULL-speed rows, but 4112 are dac/aoc/accessory where NULL is correct. It now flags only real optics (product_type not in dac/aoc/aec/accessory/switch/nic) with NULL/<=0/>1600 speed -> 596 genuine module gaps, down from 2134 mixed."}
|
||||||
{"d":"2026-05-14","t":"FEAT","m":"Equivalences Explorer: new dashboard tab '🔀 Equivalences' — search 63,362 cross-brand mappings (46 vendors, 7,516 competitor products → 846 Flexoptix alternatives, Ø 93.9% confidence). APIs: GET /api/equivalences (search), /api/equivalences/transceiver/:id (per-product), /api/equivalences/stats, /api/equivalences/top-vendors. Transceiver detail modal now shows equivalences panel (FX alternatives or competitor products) + SVG price history sparklines (30-day, per source vendor) from 392k+ price observations."}
|
{"d":"2026-05-14","t":"FEAT","m":"Equivalences Explorer: new dashboard tab '🔀 Equivalences' — search 63,362 cross-brand mappings (46 vendors, 7,516 competitor products → 846 Flexoptix alternatives, Ø 93.9% confidence). APIs: GET /api/equivalences (search), /api/equivalences/transceiver/:id (per-product), /api/equivalences/stats, /api/equivalences/top-vendors. Transceiver detail modal now shows equivalences panel (FX alternatives or competitor products) + SVG price history sparklines (30-day, per source vendor) from 392k+ price observations."}
|
||||||
{"d":"2026-05-14","t":"FEAT","m":"LinkedIn Distribution Status: Blog tab shows DRY_RUN badge, posted/dry_run/skipped/failed counters, history table with live URN links. GET /api/blog/linkedin/history reads blog_linkedin_distribution table + detects DRY_RUN mode from ecosystem config."}
|
{"d":"2026-05-14","t":"FEAT","m":"LinkedIn Distribution Status: Blog tab shows DRY_RUN badge, posted/dry_run/skipped/failed counters, history table with live URN links. GET /api/blog/linkedin/history reads blog_linkedin_distribution table + detects DRY_RUN mode from ecosystem config."}
|
||||||
{"d":"2026-05-14","t":"FEAT","m":"MCP Server: 2 new tools — find_equivalences (search 63k+ verified cross-brand mappings with confidence filter, returns FX alternatives + competitor matches formatted for LLM) + get_price_history (392k+ obs, daily series, per-vendor min/max/avg, cheapest source identification). Total: 21 MCP tools."}
|
{"d":"2026-05-14","t":"FEAT","m":"MCP Server: 2 new tools — find_equivalences (search 63k+ verified cross-brand mappings with confidence filter, returns FX alternatives + competitor matches formatted for LLM) + get_price_history (392k+ obs, daily series, per-vendor min/max/avg, cheapest source identification). Total: 21 MCP tools."}
|
||||||
|
|||||||
21
scripts/enrich-modulation-fec.sh
Normal file
21
scripts/enrich-modulation-fec.sh
Normal file
@ -0,0 +1,21 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# Daily self-healing enrichment: fills modulation/fec_type/inbuilt_fec/coherent on
|
||||||
|
# new/changed transceivers via the idempotent fill-only fn_tip_enrich_modulation_fec().
|
||||||
|
# The speed plausibility guard (trg_speed_plausibility) already self-heals on write.
|
||||||
|
# Cron: 30 3 * * * /opt/tip/scripts/enrich-modulation-fec.sh
|
||||||
|
set -uo pipefail
|
||||||
|
LOG=/opt/tip/logs/enrich-modulation-fec.log
|
||||||
|
mkdir -p "$(dirname "$LOG")"
|
||||||
|
TS=$(date -u '+%Y-%m-%dT%H:%M:%SZ')
|
||||||
|
|
||||||
|
# shellcheck disable=SC1091
|
||||||
|
source /opt/tip/.env 2>/dev/null || true
|
||||||
|
PW="${POSTGRES_PASSWORD:-}" # short-named var carries the secret to docker, no literal
|
||||||
|
N=$(docker exec -e PGPASSWORD="$PW" tip-postgres psql -U tip -d transceiver_db -tAc \
|
||||||
|
"select fn_tip_enrich_modulation_fec()" 2>/dev/null | tr -d '[:space:]')
|
||||||
|
|
||||||
|
if [[ "${N:-}" =~ ^[0-9]+$ ]]; then
|
||||||
|
echo "[$TS] enrich rows-touched=$N" >> "$LOG"
|
||||||
|
else
|
||||||
|
echo "[$TS] enrich FAILED (db unreachable?)" >> "$LOG"
|
||||||
|
fi
|
||||||
227
sql/135-modulation-fec-enrichment-and-speed-guard.sql
Normal file
227
sql/135-modulation-fec-enrichment-and-speed-guard.sql
Normal file
@ -0,0 +1,227 @@
|
|||||||
|
-- 135 Modulation/FEC enrichment + speed plausibility guard for transceivers.
|
||||||
|
--
|
||||||
|
-- Derived and adversarially verified via workflow wf_dc5bab06 (4 optical-standards
|
||||||
|
-- lenses -> synthesis -> 4 skeptical reviewers). All CONFIRMED reviewer findings are
|
||||||
|
-- folded in here: coherent >=100G speed floor (10G-ZR is NRZ, not DCO), form_factor
|
||||||
|
-- DCO/ACO scan (CFP2-DCO evidence lives in form_factor), compact DAC/AOC/AEC suffix
|
||||||
|
-- exclusion, bare-QSFP28-no-token ABSTAINS instead of guessing NRZ, CR4 dropped from
|
||||||
|
-- the NRZ set, anchored MTP/MPO, NON-COHERENT guard, search_path pinned.
|
||||||
|
--
|
||||||
|
-- FILL-ONLY: never overwrites a non-null value, never touches data_confidence='vendor_verified'.
|
||||||
|
-- Idempotent: safe to re-run; a fully enriched table yields 0 rows touched.
|
||||||
|
|
||||||
|
-- ---------------------------------------------------------------------------
|
||||||
|
-- Speed plausibility. Flags only UNAMBIGUOUS junk: sub-0.09 fragments (incl. <=0;
|
||||||
|
-- the lowest real optical rate is 100M=0.1) and >1600 (cable-part-number integers
|
||||||
|
-- like 2.0e16). The 0.09..1600 band is deliberately kept intact: it holds legit
|
||||||
|
-- non-Ethernet rates (FC line rates 2.125/4.25/8.5/14.025, SDI 3/6/12, CPRI 1.2288,
|
||||||
|
-- OTN 10.7) that must not be discarded. NULL is treated as NOT implausible.
|
||||||
|
-- ---------------------------------------------------------------------------
|
||||||
|
CREATE OR REPLACE FUNCTION tip_speed_is_implausible(v numeric)
|
||||||
|
RETURNS boolean LANGUAGE sql IMMUTABLE PARALLEL SAFE
|
||||||
|
SET search_path = pg_catalog, public AS $$
|
||||||
|
SELECT v IS NOT NULL AND (v < 0.09 OR v > 1600);
|
||||||
|
$$;
|
||||||
|
|
||||||
|
-- Separate, isolated speed sanitizer. Distinct from tip_transceivers_biu() (which
|
||||||
|
-- recomputes product_type/form_factor and never touches speed). Sanitize-only:
|
||||||
|
-- NULLs junk speed on write, never raises, never blocks ingestion.
|
||||||
|
CREATE OR REPLACE FUNCTION tip_speed_plausibility_biu()
|
||||||
|
RETURNS trigger LANGUAGE plpgsql
|
||||||
|
SET search_path = pg_catalog, public AS $$
|
||||||
|
BEGIN
|
||||||
|
IF NEW.speed_gbps IS NOT NULL AND tip_speed_is_implausible(NEW.speed_gbps) THEN
|
||||||
|
NEW.speed_gbps := NULL;
|
||||||
|
END IF;
|
||||||
|
RETURN NEW;
|
||||||
|
END;
|
||||||
|
$$;
|
||||||
|
|
||||||
|
DROP TRIGGER IF EXISTS trg_speed_plausibility ON transceivers;
|
||||||
|
CREATE TRIGGER trg_speed_plausibility
|
||||||
|
BEFORE INSERT OR UPDATE ON transceivers
|
||||||
|
FOR EACH ROW EXECUTE FUNCTION tip_speed_plausibility_biu();
|
||||||
|
|
||||||
|
-- ---------------------------------------------------------------------------
|
||||||
|
-- Enrichment scope: real optics only. The part_number cable/trunk veto applies to
|
||||||
|
-- BOTH the product_type='module' and product_type IS NULL branches, so a cable
|
||||||
|
-- mis-classified 'module' upstream still cannot be enriched.
|
||||||
|
-- ---------------------------------------------------------------------------
|
||||||
|
CREATE OR REPLACE FUNCTION tip_in_optics_scope(p_product_type text, p_form_factor text, p_part_number text)
|
||||||
|
RETURNS boolean LANGUAGE sql IMMUTABLE PARALLEL SAFE
|
||||||
|
SET search_path = pg_catalog, public AS $$
|
||||||
|
SELECT
|
||||||
|
coalesce(p_product_type,'') NOT IN ('dac','aoc','aec','accessory','switch','nic')
|
||||||
|
-- compact suffix forms (-AOC3M, -DAC1M, -AEC2M), Flexoptix cable schemes (M4.*, S2.*),
|
||||||
|
-- loopbacks, and explicit cable keywords are never optics. MTP/MPO is deliberately
|
||||||
|
-- NOT vetoed here: parallel optics (DR4/SR4/DR8) legitimately carry an -MPO12 connector
|
||||||
|
-- suffix, and real fiber trunks are already caught by ^M[0-9]./^S[0-9]./CABLE.
|
||||||
|
AND coalesce(p_part_number,'') !~* '([-_](DAC|AOC|AEC)([-_0-9]|$)|CAB[-_]|CABLE|FN-CABLE|LOOPBACK|[-_]LB[0-9]|^M[0-9]\.|^S[0-9]\.)'
|
||||||
|
AND (
|
||||||
|
p_product_type = 'module'
|
||||||
|
OR (
|
||||||
|
p_product_type IS NULL
|
||||||
|
AND upper(coalesce(p_form_factor,'')) IN
|
||||||
|
('QSFP28','QSFP56','QSFP-DD','QSFP-DD800','OSFP','OSFP800','CFP2','CFP2-DCO','CFP2-ACO','CFP8','SFP-DD','SFP112','DSFP')
|
||||||
|
)
|
||||||
|
);
|
||||||
|
$$;
|
||||||
|
|
||||||
|
-- ---------------------------------------------------------------------------
|
||||||
|
-- Coherent detection. Regex-derived triggers are floored at >=100G (100ZR is the
|
||||||
|
-- lowest coherent pluggable; 10G/40G-ZR are NRZ reach optics). A pre-existing
|
||||||
|
-- coherent=true is trusted and NOT speed-floored. NON-COHERENT is excluded.
|
||||||
|
-- ---------------------------------------------------------------------------
|
||||||
|
CREATE OR REPLACE FUNCTION tip_is_coherent(p_std text, p_pn text, p_ff text, p_speed numeric, p_coherent boolean)
|
||||||
|
RETURNS boolean LANGUAGE sql IMMUTABLE PARALLEL SAFE
|
||||||
|
SET search_path = pg_catalog, public AS $$
|
||||||
|
SELECT
|
||||||
|
p_coherent IS TRUE
|
||||||
|
OR (
|
||||||
|
p_speed >= 100
|
||||||
|
AND (coalesce(p_std,'')||' '||coalesce(p_pn,'')||' '||coalesce(p_ff,'')) !~* 'NON-?COHERENT'
|
||||||
|
AND (
|
||||||
|
(coalesce(p_std,'')||' '||coalesce(p_pn,'')) ~* '(^|[^A-Za-z0-9])(100|200|300|400|600|800|1600)?G?-?ZR(\+|P)?([^A-Za-z0-9]|$)'
|
||||||
|
OR (coalesce(p_std,'')||' '||coalesce(p_pn,'')) ~* '(^|[^A-Za-z0-9])DCO([^A-Za-z0-9]|$)'
|
||||||
|
OR (coalesce(p_std,'')||' '||coalesce(p_pn,'')) ~* 'OPEN-?ZR'
|
||||||
|
OR (coalesce(p_std,'')||' '||coalesce(p_pn,'')) ~* '(^|[^A-Za-z0-9])COHERENT([^A-Za-z0-9]|$)'
|
||||||
|
OR (coalesce(p_std,'')||' '||coalesce(p_pn,'')) ~* '(DP|PM)[-_]?(16QAM|8QAM|64QAM|QPSK)'
|
||||||
|
-- CFP2-DCO/ACO carry the only coherent evidence in form_factor (standard_name ~1% populated).
|
||||||
|
OR upper(coalesce(p_ff,'')) ~ '(DCO|ACO|COHERENT)'
|
||||||
|
)
|
||||||
|
);
|
||||||
|
$$;
|
||||||
|
|
||||||
|
-- Modulation. NULL = ABSTAIN (unknown). Order: coherent -> PAM4-by-speed ->
|
||||||
|
-- 100G single-lambda PAM4 -> 100G 4x25 NRZ -> abstain. Bare QSFP28 with no reach
|
||||||
|
-- token abstains (never defaulted to NRZ). (?![0-9]) keeps SR4 from matching SR40
|
||||||
|
-- and DR/FR from matching DR4/FR4.
|
||||||
|
CREATE OR REPLACE FUNCTION tip_derive_modulation(p_std text, p_pn text, p_ff text, p_speed numeric, p_coherent boolean)
|
||||||
|
RETURNS text LANGUAGE sql IMMUTABLE PARALLEL SAFE
|
||||||
|
SET search_path = pg_catalog, public AS $$
|
||||||
|
SELECT CASE
|
||||||
|
WHEN tip_is_coherent(p_std, p_pn, p_ff, p_speed, p_coherent) THEN CASE
|
||||||
|
WHEN (coalesce(p_std,'')||' '||coalesce(p_pn,'')) ~* '(DP|PM)[-_]?QPSK' THEN 'DP-QPSK'
|
||||||
|
WHEN (coalesce(p_std,'')||' '||coalesce(p_pn,'')) ~* '(DP|PM)[-_]?16QAM' THEN 'DP-16QAM'
|
||||||
|
WHEN (coalesce(p_std,'')||' '||coalesce(p_pn,'')) ~* '(DP|PM)[-_]?(8QAM|64QAM)' THEN 'Coherent'
|
||||||
|
WHEN p_speed >= 400 THEN 'DP-16QAM' -- 400ZR/800ZR/DCO default
|
||||||
|
WHEN p_speed = 100 THEN 'DP-QPSK' -- 100ZR is QPSK
|
||||||
|
ELSE 'Coherent' -- 200G/300G constellation ambiguous -> generic
|
||||||
|
END
|
||||||
|
WHEN p_speed IN (200,400,800,1600) THEN 'PAM4'
|
||||||
|
WHEN p_speed = 100
|
||||||
|
AND (coalesce(p_std,'')||' '||coalesce(p_pn,'')) ~* '(^|[^0-9A-Za-z])(DR1?|FR1?|LR1|ER1|SR1|SR2|VR1?)(?![0-9])'
|
||||||
|
AND (coalesce(p_std,'')||' '||coalesce(p_pn,'')) ~* '(^|[^0-9A-Za-z])(SR4|LR4|ER4L?|CWDM4|CLR4|PSM4|4WDM)(?![0-9])'
|
||||||
|
THEN NULL -- conflicting single-lambda AND 4x25 tokens -> abstain
|
||||||
|
WHEN p_speed = 100
|
||||||
|
AND (coalesce(p_std,'')||' '||coalesce(p_pn,'')) ~* '(^|[^0-9A-Za-z])(DR1?|FR1?|LR1|ER1|SR1|SR2|VR1?)(?![0-9])'
|
||||||
|
THEN 'PAM4'
|
||||||
|
WHEN p_speed = 100
|
||||||
|
AND upper(coalesce(p_ff,'')) IN ('SFP-DD','SFP112','DSFP')
|
||||||
|
AND (coalesce(p_std,'')||' '||coalesce(p_pn,'')) !~* '(^|[^0-9A-Za-z])(SR4|LR4|ER4L?|CWDM4|CLR4|PSM4|4WDM)(?![0-9])'
|
||||||
|
THEN 'PAM4'
|
||||||
|
WHEN p_speed = 100
|
||||||
|
AND (coalesce(p_std,'')||' '||coalesce(p_pn,'')) ~* '(^|[^0-9A-Za-z])(SR4|LR4|ER4L?|CWDM4|CLR4|PSM4|4WDM)(?![0-9])'
|
||||||
|
THEN 'NRZ'
|
||||||
|
ELSE NULL -- bare QSFP28 / no token / sub-100G / non-standard speed -> abstain
|
||||||
|
END;
|
||||||
|
$$;
|
||||||
|
|
||||||
|
-- FEC type. Reads modulation. Abstains (NULL) for 100G NRZ LR4/ER4/CWDM4/CLR4/4WDM
|
||||||
|
-- (no mandatory FEC); that assertion is carried by inbuilt_fec=false, not a fake fec_type.
|
||||||
|
CREATE OR REPLACE FUNCTION tip_derive_fec_type(p_modulation text, p_std text, p_pn text, p_speed numeric)
|
||||||
|
RETURNS text LANGUAGE sql IMMUTABLE PARALLEL SAFE
|
||||||
|
SET search_path = pg_catalog, public AS $$
|
||||||
|
SELECT CASE
|
||||||
|
WHEN p_modulation IN ('DP-16QAM','DP-QPSK','Coherent') THEN 'SD-FEC'
|
||||||
|
WHEN p_modulation = 'PAM4' AND p_speed IN (100,200,400,800,1600) THEN 'RS-FEC (RS(544,514) KP4)'
|
||||||
|
WHEN p_modulation = 'NRZ' AND p_speed = 100
|
||||||
|
AND (coalesce(p_std,'')||' '||coalesce(p_pn,'')) ~* '(^|[^0-9A-Za-z])(SR4|PSM4)(?![0-9])' THEN 'RS-FEC (RS(528,514) CL91)'
|
||||||
|
ELSE NULL
|
||||||
|
END;
|
||||||
|
$$;
|
||||||
|
|
||||||
|
CREATE OR REPLACE FUNCTION tip_derive_inbuilt_fec(p_modulation text, p_std text, p_pn text, p_speed numeric)
|
||||||
|
RETURNS boolean LANGUAGE sql IMMUTABLE PARALLEL SAFE
|
||||||
|
SET search_path = pg_catalog, public AS $$
|
||||||
|
SELECT CASE
|
||||||
|
WHEN p_modulation IN ('DP-16QAM','DP-QPSK','Coherent') THEN true
|
||||||
|
WHEN p_modulation = 'PAM4' AND p_speed IN (100,200,400,800,1600) THEN true
|
||||||
|
WHEN p_modulation = 'NRZ' AND p_speed = 100
|
||||||
|
AND (coalesce(p_std,'')||' '||coalesce(p_pn,'')) ~* '(^|[^0-9A-Za-z])(SR4|PSM4)(?![0-9])' THEN true
|
||||||
|
WHEN p_modulation = 'NRZ' AND p_speed = 100
|
||||||
|
AND (coalesce(p_std,'')||' '||coalesce(p_pn,'')) ~* '(^|[^0-9A-Za-z])(LR4|ER4L?|CWDM4|CLR4|4WDM)(?![0-9])' THEN false
|
||||||
|
ELSE NULL
|
||||||
|
END;
|
||||||
|
$$;
|
||||||
|
|
||||||
|
-- ---------------------------------------------------------------------------
|
||||||
|
-- Idempotent fill-only enrichment. Returns total per-statement rows touched
|
||||||
|
-- (a row filled across all three steps counts up to 3x). Fully enriched -> 0.
|
||||||
|
-- One SELECT call runs all three steps in a single transaction, so APPLY-3 reads
|
||||||
|
-- the modulation APPLY-2 just set.
|
||||||
|
-- ---------------------------------------------------------------------------
|
||||||
|
CREATE OR REPLACE FUNCTION fn_tip_enrich_modulation_fec()
|
||||||
|
RETURNS integer LANGUAGE plpgsql
|
||||||
|
SET search_path = pg_catalog, public AS $$
|
||||||
|
DECLARE n1 int := 0; n2 int := 0; n3 int := 0;
|
||||||
|
BEGIN
|
||||||
|
-- 1) coherent flag (set true only; never flip an existing true->false)
|
||||||
|
UPDATE transceivers t
|
||||||
|
SET coherent = true,
|
||||||
|
enriched_at = now(),
|
||||||
|
enriched_fields = (SELECT array_agg(DISTINCT e) FROM unnest(coalesce(t.enriched_fields,'{}'::text[]) || ARRAY['coherent']) e),
|
||||||
|
data_confidence = CASE WHEN coalesce(t.data_confidence,'') IN ('','scraped_unverified','unknown') THEN 'enriched_estimated' ELSE t.data_confidence END
|
||||||
|
WHERE coalesce(t.data_confidence,'') <> 'vendor_verified'
|
||||||
|
AND t.speed_gbps IS NOT NULL
|
||||||
|
AND t.coherent IS DISTINCT FROM true
|
||||||
|
AND tip_in_optics_scope(t.product_type, t.form_factor, t.part_number)
|
||||||
|
AND tip_is_coherent(t.standard_name, t.part_number, t.form_factor, t.speed_gbps, t.coherent);
|
||||||
|
GET DIAGNOSTICS n1 = ROW_COUNT;
|
||||||
|
|
||||||
|
-- 2) modulation (fill-only)
|
||||||
|
UPDATE transceivers t
|
||||||
|
SET modulation = m.val,
|
||||||
|
enriched_at = now(),
|
||||||
|
enriched_fields = (SELECT array_agg(DISTINCT e) FROM unnest(coalesce(t.enriched_fields,'{}'::text[]) || ARRAY['modulation']) e),
|
||||||
|
data_confidence = CASE WHEN coalesce(t.data_confidence,'') IN ('','scraped_unverified','unknown') THEN 'enriched_estimated' ELSE t.data_confidence END
|
||||||
|
FROM (
|
||||||
|
SELECT ctid AS cid, tip_derive_modulation(standard_name, part_number, form_factor, speed_gbps, coherent) AS val
|
||||||
|
FROM transceivers
|
||||||
|
WHERE modulation IS NULL
|
||||||
|
AND coalesce(data_confidence,'') <> 'vendor_verified'
|
||||||
|
AND speed_gbps IS NOT NULL
|
||||||
|
AND tip_in_optics_scope(product_type, form_factor, part_number)
|
||||||
|
) m
|
||||||
|
WHERE t.ctid = m.cid AND m.val IS NOT NULL;
|
||||||
|
GET DIAGNOSTICS n2 = ROW_COUNT;
|
||||||
|
|
||||||
|
-- 3) fec_type + inbuilt_fec (reads modulation set in step 2)
|
||||||
|
UPDATE transceivers t
|
||||||
|
SET fec_type = CASE WHEN t.fec_type IS NULL AND f.fec IS NOT NULL THEN f.fec ELSE t.fec_type END,
|
||||||
|
inbuilt_fec = CASE WHEN t.inbuilt_fec IS NULL AND f.inb IS NOT NULL THEN f.inb ELSE t.inbuilt_fec END,
|
||||||
|
enriched_at = now(),
|
||||||
|
enriched_fields = (SELECT array_agg(DISTINCT e) FROM unnest(
|
||||||
|
coalesce(t.enriched_fields,'{}'::text[])
|
||||||
|
|| CASE WHEN t.fec_type IS NULL AND f.fec IS NOT NULL THEN ARRAY['fec_type'] ELSE '{}'::text[] END
|
||||||
|
|| CASE WHEN t.inbuilt_fec IS NULL AND f.inb IS NOT NULL THEN ARRAY['inbuilt_fec'] ELSE '{}'::text[] END) e),
|
||||||
|
data_confidence = CASE WHEN coalesce(t.data_confidence,'') IN ('','scraped_unverified','unknown') THEN 'enriched_estimated' ELSE t.data_confidence END
|
||||||
|
FROM (
|
||||||
|
SELECT ctid AS cid,
|
||||||
|
tip_derive_fec_type(modulation, standard_name, part_number, speed_gbps) AS fec,
|
||||||
|
tip_derive_inbuilt_fec(modulation, standard_name, part_number, speed_gbps) AS inb
|
||||||
|
FROM transceivers
|
||||||
|
WHERE coalesce(data_confidence,'') <> 'vendor_verified'
|
||||||
|
AND speed_gbps IS NOT NULL
|
||||||
|
AND (fec_type IS NULL OR inbuilt_fec IS NULL)
|
||||||
|
AND tip_in_optics_scope(product_type, form_factor, part_number)
|
||||||
|
) f
|
||||||
|
WHERE t.ctid = f.cid
|
||||||
|
AND ((t.fec_type IS NULL AND f.fec IS NOT NULL) OR (t.inbuilt_fec IS NULL AND f.inb IS NOT NULL));
|
||||||
|
GET DIAGNOSTICS n3 = ROW_COUNT;
|
||||||
|
|
||||||
|
RETURN n1 + n2 + n3;
|
||||||
|
END;
|
||||||
|
$$;
|
||||||
|
|
||||||
|
-- Run enrichment: SELECT fn_tip_enrich_modulation_fec();
|
||||||
48
sql/136-speed-gbps-cleanup.sql
Normal file
48
sql/136-speed-gbps-cleanup.sql
Normal file
@ -0,0 +1,48 @@
|
|||||||
|
-- 136 Make speed_gbps nullable + one-time reversible cleanup of junk speed.
|
||||||
|
-- Requires 135 (tip_speed_is_implausible, trg_speed_plausibility) applied first.
|
||||||
|
--
|
||||||
|
-- speed_gbps was NOT NULL, which (a) forced meaningless values onto cables/accessories
|
||||||
|
-- and (b) made the guard trigger unusable: NULLing junk on write would abort the insert.
|
||||||
|
-- "Unknown speed" is honestly NULL, so we drop the constraint, then NULL the ~4708
|
||||||
|
-- unambiguous-junk rows (sub-0.09 fragments incl. <=0, and >1600 cable-part-number
|
||||||
|
-- integers). 87% are dac/aoc/accessory where speed is meaningless anyway. The old value
|
||||||
|
-- is snapshotted by primary key so the change is fully reversible. The 0.09..1600 gray
|
||||||
|
-- band (~355 legit FC/SDI/CPRI/OTN line rates) is intentionally left untouched.
|
||||||
|
--
|
||||||
|
-- PREVIEW before running (nothing legit should appear):
|
||||||
|
-- SELECT speed_gbps, count(*) FROM transceivers
|
||||||
|
-- WHERE tip_speed_is_implausible(speed_gbps) GROUP BY 1 ORDER BY 1;
|
||||||
|
|
||||||
|
BEGIN;
|
||||||
|
|
||||||
|
ALTER TABLE transceivers ALTER COLUMN speed_gbps DROP NOT NULL;
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS tip_speed_gbps_rollback (
|
||||||
|
id uuid PRIMARY KEY,
|
||||||
|
old_speed_gbps numeric,
|
||||||
|
backed_up_at timestamptz DEFAULT now()
|
||||||
|
);
|
||||||
|
|
||||||
|
INSERT INTO tip_speed_gbps_rollback (id, old_speed_gbps)
|
||||||
|
SELECT id, speed_gbps
|
||||||
|
FROM transceivers
|
||||||
|
WHERE tip_speed_is_implausible(speed_gbps)
|
||||||
|
ON CONFLICT (id) DO NOTHING;
|
||||||
|
|
||||||
|
UPDATE transceivers
|
||||||
|
SET speed_gbps = NULL
|
||||||
|
WHERE tip_speed_is_implausible(speed_gbps);
|
||||||
|
|
||||||
|
COMMIT;
|
||||||
|
|
||||||
|
-- REVERSE (maintenance window, no concurrent writers). Disable the guard so it does
|
||||||
|
-- not re-NULL the junk being restored:
|
||||||
|
-- BEGIN;
|
||||||
|
-- ALTER TABLE transceivers DISABLE TRIGGER trg_speed_plausibility;
|
||||||
|
-- UPDATE transceivers t SET speed_gbps = r.old_speed_gbps
|
||||||
|
-- FROM tip_speed_gbps_rollback r
|
||||||
|
-- WHERE t.id = r.id AND t.speed_gbps IS NULL;
|
||||||
|
-- ALTER TABLE transceivers ENABLE TRIGGER trg_speed_plausibility;
|
||||||
|
-- -- optional, only if every row is non-null again:
|
||||||
|
-- -- ALTER TABLE transceivers ALTER COLUMN speed_gbps SET NOT NULL;
|
||||||
|
-- COMMIT;
|
||||||
13
sql/137-speed-error-view-honest.sql
Normal file
13
sql/137-speed-error-view-honest.sql
Normal file
@ -0,0 +1,13 @@
|
|||||||
|
-- 137 Make v_transceiver_speed_errors count only genuine optics.
|
||||||
|
--
|
||||||
|
-- After 136 nulled junk speed, the old definition (speed_gbps IS NULL OR <= 0)
|
||||||
|
-- counted 4708 rows, but 4112 of them are dac/aoc/aec/accessory where speed_gbps
|
||||||
|
-- is meaningless and NULL is correct. A speed error is a real optic (module) with
|
||||||
|
-- a missing or physically-impossible speed. This drops the count to the ~596 real
|
||||||
|
-- modules that still need a recoverable speed.
|
||||||
|
|
||||||
|
CREATE OR REPLACE VIEW v_transceiver_speed_errors AS
|
||||||
|
SELECT id, part_number, speed_gbps, form_factor, product_type
|
||||||
|
FROM transceivers
|
||||||
|
WHERE coalesce(product_type,'') NOT IN ('dac','aoc','aec','accessory','switch','nic')
|
||||||
|
AND (speed_gbps IS NULL OR speed_gbps <= 0 OR speed_gbps > 1600);
|
||||||
Loading…
x
Reference in New Issue
Block a user