rene/transceiver-db
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
transceiver-db/sql/114-extend-ieee-lookup-and-clear-pending.sql
Rene Fichtmueller d1bde66e39 feat: deterministic equivalence matcher + full wavelength/connector enrichment 
Replace confidence-based matcher with deterministic 6-field exact match:
- form_factor (exact), speed_gbps (±0.1G), fiber_type (exact),
  reach (±10%), wavelength_tx (±5nm), connector_type (exact)
- Complete products → confidence=1.0, never creates pending records
- Incomplete products → enhanced confidence ≥0.85, still auto_approved
- PENDING CREATED: 0 (by design, permanent)

Migrations:
- sql/113: Connector type inference from IEEE lookup + form-factor rules
  (970→479 missing connector for FX products)
- sql/114: Extend IEEE lookup with 400G/800G/1.6T OSFP/QSFP-DD standards,
  wavelength fallback (SMF→1310nm, MMF→850nm), clear pending queue to 0

Enrichment results (before→after):
- FX fully complete: 50 → 555 / 1,089 (+505)
- Total fully complete: ~3,600 → 15,431 / 18,133 (+11,800)
- FX coverage: 54.7% → 55.8% (608/1,089 matched)
- Deterministic matches: 0 → 44,596 (confidence=1.0)
- Wavelength-mismatched records rejected: 521
- Pending queue: 42 → 0 (permanent)

New match stats:
- 55,743 new deterministic auto_approved matches
- 521 legacy wavelength-mismatch records rejected
- Total active: 53,447 auto_approved + 1,987 approved
2026-05-13 17:59:08 +02:00

215 lines
13 KiB
SQL
Raw Blame History

-- Migration 114: Extend IEEE/MSA Lookup (400G/800G/1.6T) + Clear Pending Queue
-- Part A: Add missing 400G/800G/1.6T standards to ieee_wavelength_lookup
-- Part B: Wavelength fallback for products with known form/fiber/reach
-- Part C: Reject remaining pending records (replaced by deterministic matcher)
-- ── Part A: IEEE/MSA Lookup Erweiterung ─────────────────────────────────────
-- Sources: IEEE 802.3cd (200G), 802.3bs (400G), 802.3df (800G), 802.3dj (1.6T draft)
-- 400G-FR4 MSA, 400G-LR4-10 MSA, OSFP MSA, OpenZR+ MSA
INSERT INTO ieee_wavelength_lookup
(form_factor, speed_gbps, fiber_type, reach_min_m, reach_max_m, wavelength_tx_nm, wavelength_rx_nm, connector_type, ieee_standard, notes)
VALUES
-- ── QSFP+ 40G additional reaches ─────────────────────────────────────────────
('QSFP+', 40, 'SMF', 0, 150, 1310, NULL, 'MPO-12', '802.3ba', '40GBASE-PSM4 short'),
('QSFP+', 40, 'SMF', 0, 1400, 1310, NULL, 'LC', '802.3ba', '40GBASE-LR4 1.4km'),
-- ── QSFP28 100G additional ───────────────────────────────────────────────────
('QSFP28', 100, 'SMF', 0, 80000, 1550, NULL, 'LC', '802.3ba', '100GBASE-ZR4'),
('QSFP28', 100, 'SMF', 0, 120000, 1550, NULL, 'LC', 'OpenZR+', '100G OpenZR+ 120km'),
-- ── QSFP56 200G ──────────────────────────────────────────────────────────────
('QSFP56', 200, 'DAC', 0, 5, NULL, NULL, 'QSFP56','802.3cd', '200G DAC'),
('QSFP56', 200, 'AOC', 0, 100, 850, NULL, 'MPO-16','802.3cd', '200G AOC SR4'),
-- ── QSFP-DD 400G additional ──────────────────────────────────────────────────
('QSFP-DD', 400, 'MMF', 0, 100, 850, NULL, 'MPO-16','802.3bs', '400GBASE-SR8'),
('QSFP-DD', 400, 'SMF', 0, 500, 1310, NULL, 'MPO-12','802.3bs', '400GBASE-DR4'),
('QSFP-DD', 400, 'SMF', 0, 2000, 1310, NULL, 'LC', '802.3bs', '400GBASE-FR4'),
('QSFP-DD', 400, 'SMF', 0, 10000, 1310, NULL, 'LC', '802.3bs', '400GBASE-LR4'),
('QSFP-DD', 400, 'SMF', 0, 80000, 1550, NULL, 'LC', '400ZR-MSA','400G ZR 80km'),
('QSFP-DD', 400, 'SMF', 0, 120000, 1550, NULL, 'LC', 'OpenZR+', '400G OpenZR+ 120km'),
('QSFP-DD', 400, 'AOC', 0, 100, 850, NULL, 'MPO-16','802.3bs', '400G AOC SR8'),
-- ── QSFP-DD800 800G ──────────────────────────────────────────────────────────
('QSFP-DD800', 800, 'MMF', 0, 100, 850, NULL, 'MPO-16','802.3df', '800GBASE-SR8'),
('QSFP-DD800', 800, 'SMF', 0, 500, 1310, NULL, 'MPO-12','802.3df', '800GBASE-DR8'),
('QSFP-DD800', 800, 'SMF', 0, 2000, 1310, NULL, 'LC', '802.3df', '800GBASE-FR4 2x400G'),
('QSFP-DD800', 800, 'SMF', 0,10000, 1310, NULL, 'LC', '802.3df', '800GBASE-LR4'),
('QSFP-DD800', 800, 'SMF', 0,80000, 1550, NULL, 'LC', 'OpenZR+', '800G OpenZR+ 80km'),
('QSFP-DD800', 800, 'DAC', 0, 5, NULL, NULL, 'QSFP-DD800','802.3df','800G DAC'),
-- ── OSFP 400G ────────────────────────────────────────────────────────────────
('OSFP', 400, 'MMF', 0, 100, 850, NULL, 'MPO-16', 'OSFP-MSA', '400GBASE-SR8 OSFP'),
('OSFP', 400, 'SMF', 0, 500, 1310, NULL, 'MPO-12', 'OSFP-MSA', '400GBASE-DR4 OSFP'),
('OSFP', 400, 'SMF', 0, 2000, 1310, NULL, 'LC', 'OSFP-MSA', '400GBASE-FR4 OSFP'),
('OSFP', 400, 'SMF', 0, 10000, 1310, NULL, 'LC', 'OSFP-MSA', '400GBASE-LR4 OSFP'),
('OSFP', 400, 'SMF', 0, 80000, 1550, NULL, 'LC', 'OpenZR+', '400G ZR OSFP 80km'),
('OSFP', 400, 'SMF', 0, 120000, 1550, NULL, 'LC', 'OpenZR+', '400G OpenZR+ OSFP 120km'),
-- ── OSFP 800G ────────────────────────────────────────────────────────────────
('OSFP', 800, 'MMF', 0, 30, 850, NULL, 'MPO-16', '802.3df', '800GBASE-SR8 30m'),
('OSFP', 800, 'MMF', 0, 100, 850, NULL, 'MPO-16', '802.3df', '800GBASE-SR8'),
('OSFP', 800, 'SMF', 0, 500, 1310, NULL, 'MPO-12', '802.3df', '800GBASE-DR8 OSFP'),
('OSFP', 800, 'SMF', 0, 2000, 1310, NULL, 'LC', '802.3df', '800GBASE-FR4 OSFP'),
('OSFP', 800, 'SMF', 0, 10000, 1310, NULL, 'LC', '802.3df', '800GBASE-LR4 OSFP'),
('OSFP', 800, 'SMF', 0, 80000, 1550, NULL, 'LC', 'OpenZR+', '800G ZR OSFP 80km'),
-- ── OSFP 1.6T (IEEE 802.3dj draft) ──────────────────────────────────────────
('OSFP', 1600, 'SMF', 0, 500, 1310, NULL, 'MPO-16', '802.3dj', '1.6TBASE-DR16 OSFP'),
('OSFP', 1600, 'SMF', 0, 2000, 1310, NULL, 'LC', '802.3dj', '1.6TBASE-FR4 OSFP'),
('OSFP', 1600, 'SMF', 0, 10000, 1310, NULL, 'LC', '802.3dj', '1.6TBASE-LR4 OSFP'),
('OSFP112', 800, 'SMF', 0, 10000, 1310, NULL, 'LC', '802.3df', '800GBASE-LR4 OSFP112'),
('OSFP112', 800, 'SMF', 0, 80000, 1550, NULL, 'LC', 'OpenZR+', '800G ZR OSFP112 80km'),
('OSFP112', 800, 'SMF', 0, 120000,1550, NULL, 'LC', 'OpenZR+', '800G OpenZR+ OSFP112 120km'),
-- ── CFP2 100G coherent ───────────────────────────────────────────────────────
('CFP2', 100, 'SMF', 0, 10000, 1310, NULL, 'LC', 'OIF-100G', '100GBASE-LR4 CFP2'),
('CFP2', 100, 'SMF', 0, 80000, 1550, NULL, 'LC', 'OIF-100G', '100G ZR CFP2 80km'),
('CFP2', 100, 'SMF', 0, 120000, 1550, NULL, 'LC', 'OpenZR+', '100G OpenZR+ CFP2'),
-- ── SFP+ / SFP 1G non-standard reaches ───────────────────────────────────────
('SFP', 1, 'SMF', 0, 20000, 1310, NULL, 'LC', '802.3z', '1000BASE-LH 20km'),
('SFP', 1, 'SMF', 0, 60000, 1310, NULL, 'LC', '802.3z', '1000BASE-LH 60km'),
('SFP', 1, 'SMF', 0, 80000, 1550, NULL, 'LC', '802.3z', '1000BASE-ZX 80km'),
('SFP', 1, 'SMF', 0,100000, 1550, NULL, 'LC', '802.3z', '1000BASE-ZX 100km'),
-- ── SFP+ 10G non-standard reaches ────────────────────────────────────────────
('SFP+', 10, 'SMF', 0, 20000, 1310, NULL, 'LC', '802.3ae', '10GBASE-LR 20km variant'),
('SFP+', 10, 'SMF', 0, 60000, 1550, NULL, 'LC', '802.3ae', '10GBASE-ZR 60km'),
('SFP+', 10, 'SMF', 0, 80000, 1550, NULL, 'LC', '802.3ae', '10GBASE-ZR 80km'),
('SFP+', 10, 'SMF', 0,100000, 1550, NULL, 'LC', '802.3ae', '10GBASE-ZR 100km'),
-- ── XFP 10G ──────────────────────────────────────────────────────────────────
('XFP', 10, 'MMF', 0, 300, 850, NULL, 'LC', '802.3ae', '10GBASE-SR XFP'),
('XFP', 10, 'SMF', 0, 10000, 1310, NULL, 'LC', '802.3ae', '10GBASE-LR XFP'),
('XFP', 10, 'SMF', 0, 40000, 1310, NULL, 'LC', '802.3ae', '10GBASE-ER XFP'),
('XFP', 10, 'SMF', 0, 80000, 1550, NULL, 'LC', '802.3ae', '10GBASE-ZR XFP')
ON CONFLICT DO NOTHING;
-- ── Re-run IEEE lookup for wavelength after new entries ──────────────────────
UPDATE transceivers t SET
wavelength_tx_nm = (
SELECT il.wavelength_tx_nm
FROM ieee_wavelength_lookup il
WHERE UPPER(il.form_factor) = UPPER(t.form_factor)
AND il.speed_gbps = ROUND(t.speed_gbps::NUMERIC, 2)
AND UPPER(il.fiber_type) = UPPER(t.fiber_type)
AND il.reach_min_m <= t.reach_meters
AND il.reach_max_m >= t.reach_meters
AND il.wavelength_tx_nm IS NOT NULL
ORDER BY il.reach_max_m ASC
LIMIT 1
),
wavelength_rx_nm = COALESCE(
wavelength_rx_nm,
(
SELECT il.wavelength_rx_nm
FROM ieee_wavelength_lookup il
WHERE UPPER(il.form_factor) = UPPER(t.form_factor)
AND il.speed_gbps = ROUND(t.speed_gbps::NUMERIC, 2)
AND UPPER(il.fiber_type) = UPPER(t.fiber_type)
AND il.reach_min_m <= t.reach_meters
AND il.reach_max_m >= t.reach_meters
ORDER BY il.reach_max_m ASC
LIMIT 1
)
),
connector_type = COALESCE(
connector_type,
(
SELECT il.connector_type
FROM ieee_wavelength_lookup il
WHERE UPPER(il.form_factor) = UPPER(t.form_factor)
AND il.speed_gbps = ROUND(t.speed_gbps::NUMERIC, 2)
AND UPPER(il.fiber_type) = UPPER(t.fiber_type)
AND il.reach_min_m <= t.reach_meters
AND il.reach_max_m >= t.reach_meters
ORDER BY il.reach_max_m ASC
LIMIT 1
)
)
WHERE t.wavelength_tx_nm IS NULL
AND t.form_factor IS NOT NULL
AND t.speed_gbps IS NOT NULL
AND t.fiber_type IS NOT NULL
AND t.fiber_type NOT IN ('Copper', 'DAC', 'AOC', 'COPPER')
AND t.reach_meters IS NOT NULL
AND t.reach_meters > 0;
-- ── Part B: Fallback wavelength by fiber_type for remaining ──────────────────
-- Conservative rule: SMF products with reach > 80km → 1550nm (ZR/coherent)
-- All other SMF → 1310nm (covers ER/LR/DR/FR/LH etc.)
-- All MMF → 850nm (SR variants)
-- Products with DAC fiber_type: no optical wavelength (leave NULL)
UPDATE transceivers SET
wavelength_tx_nm = CASE
-- Long-reach SMF: reach > 80km → 1550nm (ZR, coherent)
WHEN UPPER(fiber_type) = 'SMF' AND reach_meters > 80000 THEN 1550
-- Standard SMF: 1310nm (LR/ER/DR/FR/LH etc.)
WHEN UPPER(fiber_type) = 'SMF' AND reach_meters > 0 THEN 1310
-- Short MMF: 850nm (SR variants)
WHEN UPPER(fiber_type) = 'MMF' AND reach_meters > 0 THEN 850
ELSE wavelength_tx_nm
END
WHERE wavelength_tx_nm IS NULL
AND fiber_type IS NOT NULL
AND UPPER(fiber_type) IN ('SMF', 'MMF')
AND reach_meters IS NOT NULL
AND reach_meters > 0
AND form_factor IS NOT NULL
AND UPPER(form_factor) NOT IN ('LC', 'SC', 'DAC', 'TRANSCEIVER', 'PLUGGABLE', 'VARIES');
-- ── Completeness final update ─────────────────────────────────────────────────
UPDATE transceivers SET
data_completeness = calc_data_completeness(
form_factor, speed_gbps, fiber_type,
reach_meters, wavelength_tx_nm, connector_type
),
enrichment_needed = (
form_factor IS NULL OR speed_gbps IS NULL OR
fiber_type IS NULL OR reach_meters IS NULL OR
wavelength_tx_nm IS NULL OR connector_type IS NULL
),
enrichment_fields = ARRAY_REMOVE(ARRAY[
CASE WHEN form_factor IS NULL THEN 'form_factor' END,
CASE WHEN speed_gbps IS NULL THEN 'speed_gbps' END,
CASE WHEN fiber_type IS NULL THEN 'fiber_type' END,
CASE WHEN reach_meters IS NULL OR reach_meters = 0 THEN 'reach_meters' END,
CASE WHEN wavelength_tx_nm IS NULL THEN 'wavelength_tx_nm' END,
CASE WHEN connector_type IS NULL THEN 'connector_type' END
], NULL);
-- ── Part C: Clear pending queue ───────────────────────────────────────────────
-- All pending records from confidence-based matcher are superseded.
-- Deterministic matcher (maintenance:find-equivalences) will re-generate
-- correct matches at confidence=1.0 for products with complete data.
UPDATE transceiver_equivalences
SET status = 'rejected',
reject_reason = 'Superseded by deterministic matcher — confidence-based pending removed in migration 114',
reviewed_at = NOW(),
reviewed_by = 'system:migration-114'
WHERE status = 'pending';
-- ── Final Statistics ─────────────────────────────────────────────────────────
DO $$
DECLARE
total_cnt INTEGER;
complete_cnt INTEGER;
missing_conn INTEGER;
missing_wl INTEGER;
fx_complete INTEGER;
fx_total INTEGER;
pending_cnt INTEGER;
BEGIN
SELECT COUNT(*) INTO total_cnt FROM transceivers;
SELECT COUNT(*) INTO complete_cnt FROM transceivers WHERE enrichment_needed = FALSE;
SELECT COUNT(*) INTO missing_conn FROM transceivers WHERE connector_type IS NULL;
SELECT COUNT(*) INTO missing_wl FROM transceivers WHERE wavelength_tx_nm IS NULL;
SELECT COUNT(*) INTO pending_cnt FROM transceiver_equivalences WHERE status = 'pending';
SELECT COUNT(*) INTO fx_total
FROM transceivers t JOIN vendors v ON v.id = t.vendor_id
WHERE UPPER(v.name) LIKE '%FLEXOPTIX%';
SELECT COUNT(*) INTO fx_complete
FROM transceivers t JOIN vendors v ON v.id = t.vendor_id
WHERE UPPER(v.name) LIKE '%FLEXOPTIX%' AND enrichment_needed = FALSE;
RAISE NOTICE 'Migration 114 complete:';
RAISE NOTICE ' Total transceivers: %', total_cnt;
RAISE NOTICE ' Fully complete: %', complete_cnt;
RAISE NOTICE ' Still missing connector: %', missing_conn;
RAISE NOTICE ' Still missing wavelength: %', missing_wl;
RAISE NOTICE ' Flexoptix fully complete: % / %', fx_complete, fx_total;
RAISE NOTICE ' Pending queue: % (target: 0)', pending_cnt;
END $$;
Reference in New Issue View Git Blame Copy Permalink