transceiver-db/sync/history/2026-05-09-tip-open-competitor-and-product-gap-probing.md

2026-05-09 — TIP Open Competitor Closure + Product Gap Probing

Competitor Status Closure

Added competitor_ambiguous as an evidence type via sql/104-verification-evidence-ambiguous.sql.

Added packages/scraper/src/utils/resolve-open-competitor-status.ts.

Runbook:

pnpm -C packages/scraper run verify:open-competitor-status
OPEN_COMPETITOR_APPLY=1 pnpm -C packages/scraper run verify:open-competitor-status

Live result:

• dry-run found 365 fully populated products still in needs_research
• apply wrote:
  • 364 ambiguous
  • 1 no_valid_match
  • 1 newly fully_verified
• evidence ledger:
  • 364 competitor_ambiguous
  • 1 competitor_no_match
• fully populated products still in competitor needs_research: 0

Scheduler guard:

• maintenance:find-equivalences now preserves deliberate ambiguous states instead of resetting them to needs_research
• built on Erik successfully
• restarted tip-scraper-daemon only after the pg-boss active queue was empty

Live health after closure:

• active products: 17305
• price verified: 11414
• image verified: 12016
• details verified: 16705
• fully verified: 10449
• competitor status:
  • matched=10775
  • no_valid_match=74
  • ambiguous=556
  • needs_research=5900

Interpretation:

• needs_research no longer contains fully populated products ready for competitor-only resolution.
• Remaining needs_research rows are product-data gaps first.

Product Gap Probing

Catalog Details

• hardened verify:catalog:details to write detail evidence
• live result: 113 catalog-derived rows updated, 0 fully verified earned
• active Health did not move, so those rows are not currently improving the active dashboard base

GAO Tek

• ran scrape:gaotek:details
• checked remaining 64 rows
• result: 0 updated, 64 skipped, 0 errors
• interpretation: remaining GAO pages lack deterministic public detail evidence

GBICS

• added scrape:gbics
• patched gbics.ts to pass product URLs into DB upsert/detail verification
• live run found 758 products and 0 prices
• active gap remains 64 price / 64 image / 64 details of 135
• interpretation: discovery works, but active rows and scraped identifiers need alias/dedupe hardening; price selector also needs repair

T&S Communication

• added scrape:tscom
• patched tscom.ts to pass product URLs into DB upsert/detail verification
• live run found 109 unique products and 0 prices
• active gap remains 82 price / 82 image / 49 details of 82
• interpretation: discovery works, but row matching and price selectors need hardening

10Gtek / SFPcables

• live run found 110 products and wrote 6 prices
• active 10Gtek gap remains 126 price / 131 image / 25 details of 175
• interpretation: parser works partially; remaining rows need alias/source-page repair

Next Real Work

• Build alias/dedupe resolvers for GBICS, T&S, and 10Gtek similar to FS.com numeric SKU alias cleanup.
• Repair GBICS/T&S price selectors against current page HTML.
• For OEM vendors such as Cisco/Juniper, add explicit non-public-price states instead of pretending OEM list prices exist.

Follow-Up: Artifact Quarantine + Reconcile Guard

GBICS/T&S artifact quarantine was expanded and applied live.

Result:

• 93 additional non-transceiver artifacts quarantined with artifact_quarantine evidence
• GBICS active base moved from 135 rows with 64/64/64 price/image/details gaps to 88 rows with 17/17/17 gaps
• T&S Communication active base moved from 82 rows with 82/82/49 price/image/details gaps to 36 rows with 36/36/6 gaps
• 10Gtek current active gap remains 175 rows with 126/131/25 price/image/details gaps

Important regression found:

• maintenance:reconcile-verification reset explicit competitor research states back to needs_research
• fixed scheduler guard so reconcile preserves both no_valid_match and ambiguous
• deployed to Erik, built successfully, and restarted tip-scraper-daemon only after pg-boss active queue was empty

Restore run:

• dry-run found 615 fully product-verified rows that had fallen back into competitor needs_research
• apply wrote:
  • 74 no_valid_match
  • 541 ambiguous
  • 74 newly/restored fully_verified
• live evidence ledger now includes:
  • 74 competitor_no_match from verify:open-competitor-status
  • 555 competitor_ambiguous from verify:open-competitor-status
  • 93 artifact_quarantine from verify:quarantine:non-transceivers
  • 109 artifact_quarantine from verify:fs:sku-aliases

Verification after a fresh reconcile test:

• health status: healthy
• active products: 17212
• price verified: 11414
• image verified: 12016
• details verified: 16702
• fully verified: 10449
• competitor status:
  • matched=10775
  • no_valid_match=74
  • ambiguous=556
  • needs_research=5807
• fully product-verified rows still in competitor needs_research: 0