rene/transceiver-db
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
transceiver-db/sync/history/2026-05-09-tip-open-competitor-and-product-gap-probing.md
Rene Fichtmueller 635a102932 feat: close open competitor research states
2026-05-10 00:03:42 +02:00

3.1 KiB
Raw Blame History

2026-05-09 — TIP Open Competitor Closure + Product Gap Probing

Competitor Status Closure

Added competitor_ambiguous as an evidence type via sql/104-verification-evidence-ambiguous.sql.

Added packages/scraper/src/utils/resolve-open-competitor-status.ts.

Runbook:

pnpm -C packages/scraper run verify:open-competitor-status
OPEN_COMPETITOR_APPLY=1 pnpm -C packages/scraper run verify:open-competitor-status

Live result:

  • dry-run found 365 fully populated products still in needs_research
  • apply wrote:
    • 364 ambiguous
    • 1 no_valid_match
    • 1 newly fully_verified
  • evidence ledger:
    • 364 competitor_ambiguous
    • 1 competitor_no_match
  • fully populated products still in competitor needs_research: 0

Scheduler guard:

  • maintenance:find-equivalences now preserves deliberate ambiguous states instead of resetting them to needs_research
  • built on Erik successfully
  • restarted tip-scraper-daemon only after the pg-boss active queue was empty

Live health after closure:

  • active products: 17305
  • price verified: 11414
  • image verified: 12016
  • details verified: 16705
  • fully verified: 10449
  • competitor status:
    • matched=10775
    • no_valid_match=74
    • ambiguous=556
    • needs_research=5900

Interpretation:

  • needs_research no longer contains fully populated products ready for competitor-only resolution.
  • Remaining needs_research rows are product-data gaps first.

Product Gap Probing

Catalog Details

  • hardened verify:catalog:details to write detail evidence
  • live result: 113 catalog-derived rows updated, 0 fully verified earned
  • active Health did not move, so those rows are not currently improving the active dashboard base

GAO Tek

  • ran scrape:gaotek:details
  • checked remaining 64 rows
  • result: 0 updated, 64 skipped, 0 errors
  • interpretation: remaining GAO pages lack deterministic public detail evidence

GBICS

  • added scrape:gbics
  • patched gbics.ts to pass product URLs into DB upsert/detail verification
  • live run found 758 products and 0 prices
  • active gap remains 64 price / 64 image / 64 details of 135
  • interpretation: discovery works, but active rows and scraped identifiers need alias/dedupe hardening; price selector also needs repair

T&S Communication

  • added scrape:tscom
  • patched tscom.ts to pass product URLs into DB upsert/detail verification
  • live run found 109 unique products and 0 prices
  • active gap remains 82 price / 82 image / 49 details of 82
  • interpretation: discovery works, but row matching and price selectors need hardening

10Gtek / SFPcables

  • live run found 110 products and wrote 6 prices
  • active 10Gtek gap remains 126 price / 131 image / 25 details of 175
  • interpretation: parser works partially; remaining rows need alias/source-page repair

Next Real Work

  • Build alias/dedupe resolvers for GBICS, T&S, and 10Gtek similar to FS.com numeric SKU alias cleanup.
  • Repair GBICS/T&S price selectors against current page HTML.
  • For OEM vendors such as Cisco/Juniper, add explicit non-public-price states instead of pretending OEM list prices exist.