rene/transceiver-db
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
443 Commits 5 Branches 9 Tags
Commit Graph

9 Commits

Author SHA1 Message Date
Rene Fichtmueller
0d7a92e749 feat: Abverkauf velocity engine — sql/118 + analyzer + API endpoints 
- sql/118-stock-velocity.sql: new stock_velocity (UPSERT per tx×vendor)
  and stock_velocity_events tables with TimescaleDB-compatible indexes
- stock-velocity-analyzer.ts: computes sell-through from stock_observations
  time-series; detects sold/zulauf/data_gap events, trims top-10% outliers,
  predicts stockout date, assigns high/medium/low/insufficient confidence
- scheduler.ts: analyze:stock:velocity job at 04:30/12:30/20:30 UTC
- stock.ts: GET /api/stock/velocity (paginated, filterable by vendor/confidence/
  stockout_days) + GET /api/stock/velocity/:id (per-product with event history)
- First run: 208 products, 979 sell events, 2811 Zulauf events written
 2026-05-14 00:24:58 +02:00
Rene Fichtmueller
db6b97186a feat: OPN+spec equivalence matchers, 400G pricing, TIP_LLM training data 
- Add OPN-based equivalence matcher robot (7,245 manufacturer-confirmed matches, confidence=1.0)
- Add spec-based equivalence matcher robot (683 matches, confidence=0.85)
  - Matches by form_factor + speed_gbps + reach_tier + wavelength ±10nm
  - Safety cap: skip FX products matching >30 competitors (too generic)
  - Daily schedule: 04:30 UTC via pg-boss
- SQL migrations 116 (OPN) + 117 (spec) with tip_extract_wavelength_nm() + tip_reach_tier() helpers
- Fix tenGtek.ts: add 3 missing 400G categories (QSFP-DD, QSFP112) — closes pricing gap
- Generate tip-llm-pricing-v1.jsonl: 80 DB-grounded QA pairs (pricing, equivalences, 400G)
- Rebuild TIP_LLM training pool: 11,999 pairs (+127 vs prev), deployed to Erik
- FX product equivalence coverage: 88.1% (959/1089)
 2026-05-13 21:33:19 +02:00
Rene Fichtmueller
2f85571784 feat: Flexoptix full product detail sync (sql/115 + detail-enricher robot) 
Pulls complete per-SKU specifications and compatibility data from the
Flexoptix API (specifications=1&compatibilities=1) and writes structured
fields to the transceivers table for datasheet generation.

SQL migration 115:
- Adds fx_specifications JSONB (raw spec blob for datasheet gen)
- Adds fx_compatibilities JSONB (full OEM compatibility matrix)
- Adds compliance_code, laser_type, receiver_type, supported_protocols[]
- Adds extinction_ratio_db, cdr_support, inbuilt_fec, detail_synced_at
- GIN index on fx_compatibilities for vendor/OPN queries

flexoptix-detail-enricher.ts:
- Per-SKU API calls with rate-limiting (600ms/call, 100 SKUs/run)
- Parses all spec labels → structured fields (power, budget, tx/rx dBm,
  modulation, wavelengths, temp range, DOM, laser type, receiver type)
- Strips :Sx variant suffixes before API queries (self-configure SKUs)
- COALESCE writes — never overwrites existing data, only fills gaps
- Tracks detail_synced_at, retries stale entries after 7 days

flexoptix-api-sync.ts:
- Also stores image_url and product_page_url during bulk sync

scheduler.ts:
- Registers enrich:flexoptix-details daily at 03:00 UTC

Results after initial run:
- 791/968 FX products (81.7%) fully enriched
- 26.0 avg compatibility entries per product (OEM vendor + OPN)
- 25.7 avg spec fields per product
- DFB(483), EML(148), FP(72), VCSEL(44) laser type distribution
 2026-05-13 18:49:28 +02:00
Rene Fichtmueller
9979b79434 feat: wavelength/connector enrichment schema + enricher robot 
- sql/110: add wavelength_tx_nm, wavelength_rx_nm, connector_type,
  data_completeness, enrichment_needed columns + trigger
- sql/111: IEEE/MSA standards wavelength lookup table (SFP→OSFP)
- sql/112: migrate existing wavelengths TEXT → integer columns
- robots/wavelength-enricher.ts: fills missing wavelengths from IEEE
  lookup (deterministic) then product-name regex, runs every 4h
- scheduler: register enrich:wavelength job (4h schedule)

Fixes over-broad matching where 1G SFPs match 500+ competitors
due to missing wavelength discrimination.
 2026-05-13 17:35:42 +02:00
Rene Fichtmueller
1edd6c20a8 fix: use COUNT(*) instead of COUNT(DISTINCT po.id) in catalog-reconcile 
price_observations table has no id column — replace with COUNT(*)
to avoid SQL error 42703.
 2026-05-13 16:59:49 +02:00
Rene Fichtmueller
98b241f462 feat: implement Flexoptix reference matching overhaul 
- sql/108: normalize form_factor across all vendors (SFP-Plus → SFP+, etc.)
  and round speed_gbps for consistent matching
- sql/109: document 30→90 day matcher window change
- robots/catalog-reconcile.ts: new bulk-reconcile robot — matches all
  Flexoptix products against all competitors without 30-day time limit
- scheduler.ts: register catalog:reconcile job (monthly + on-demand),
  fix nightly matcher 30→90 day window, UPPER() form_factor matching,
  ROUND() speed_gbps matching

Fixes: ATGBICS/NADDOD/10Gtek/ShopFiber24 had 0 Flexoptix equivalences
due to stale price_observations being filtered out. Expected coverage
improvement: 22% → 45-60% after first reconcile run.
 2026-05-13 16:55:45 +02:00
Rene Fichtmueller
a20094755d feat(scraper): Flexoptix REST API sync robot + scheduler integration 
Replaces the GraphQL/search-based Flexoptix scraper with a proper
Magento 2 REST API integration that delivers authoritative SKUs,
prices, stock levels and compatibility data.

New files:
- packages/scraper/src/robots/flexoptix-api-sync.ts
  Self-contained robot: auth → paginated fetch → normalize → DB write.
  Reads FLEXOPTIX_API_BASE_URL / _USERNAME / _PASSWORD from env.
  Returns { fetched, normalized, skipped, priceWrites, stockWrites }.
  No file intermediary — in-memory pipeline.

- scripts/import-flexoptix-catalog.ts
  One-shot CLI importer for the Pulso-generated JSONL (Codex handover).

- docs/FLEXOPTIX_CATALOG_IMPORT.md
  Runbook for manual import + per-SKU specifications enrichment.

Scheduler changes:
- Added sync:flexoptix-catalog queue + work() handler
- Scheduled every 2h at 0 */2 * * * (same cadence as legacy job)
- scrape:pricing:flexoptix kept as legacy GraphQL fallback

Also includes Codex-generated additions from this sprint:
- audiocodes-oem scraper, seed-batch35/36/37, db.ts improvements,
  sql/102 verification reconcile, README + package.json updates
 2026-05-13 16:36:33 +02:00
Rene Fichtmueller
1af4f090f7 fix: harden TIP verification cleanup 2026-05-09 22:16:29 +02:00
Rene Fichtmueller
a8529d166b fix: resolve TS build errors — export backfillImages, add writeRobotExperience 
- backfill-images.ts: rename main() → export backfillImages() to match index.ts import
- training-data-writer.ts: add writeRobotExperience export; remove hardcoded Gitea token
- fiber24.ts/fibermall.ts: scraper improvements from previous sessions
- image-downloader.ts/spec-updater.ts: utility updates
- robots/: add verification robots module
 2026-05-06 23:39:00 +02:00