# 2026-05-09 — TIP Open Competitor Closure + Product Gap Probing

## Competitor Status Closure

Added `competitor_ambiguous` as an evidence type via `sql/104-verification-evidence-ambiguous.sql`.

Added `packages/scraper/src/utils/resolve-open-competitor-status.ts`.

Runbook:

```bash
pnpm -C packages/scraper run verify:open-competitor-status
OPEN_COMPETITOR_APPLY=1 pnpm -C packages/scraper run verify:open-competitor-status
```

Live result:

- dry-run found `365` fully populated products still in `needs_research`
- apply wrote:
  - `364` `ambiguous`
  - `1` `no_valid_match`
  - `1` newly `fully_verified`
- evidence ledger:
  - `364` `competitor_ambiguous`
  - `1` `competitor_no_match`
- fully populated products still in competitor `needs_research`: `0`

Scheduler guard:

- `maintenance:find-equivalences` now preserves deliberate `ambiguous` states instead of resetting them to `needs_research`
- built on Erik successfully
- restarted `tip-scraper-daemon` only after the pg-boss active queue was empty

Live health after closure:

- active products: `17305`
- price verified: `11414`
- image verified: `12016`
- details verified: `16705`
- fully verified: `10449`
- competitor status:
  - `matched=10775`
  - `no_valid_match=74`
  - `ambiguous=556`
  - `needs_research=5900`

Interpretation:

- `needs_research` no longer contains fully populated products ready for competitor-only resolution.
- Remaining `needs_research` rows are product-data gaps first.

## Product Gap Probing

### Catalog Details

- hardened `verify:catalog:details` to write detail evidence
- live result: `113` catalog-derived rows updated, `0` fully verified earned
- active Health did not move, so those rows are not currently improving the active dashboard base

### GAO Tek

- ran `scrape:gaotek:details`
- checked remaining `64` rows
- result: `0` updated, `64` skipped, `0` errors
- interpretation: remaining GAO pages lack deterministic public detail evidence

### GBICS

- added `scrape:gbics`
- patched `gbics.ts` to pass product URLs into DB upsert/detail verification
- live run found `758` products and `0` prices
- active gap remains `64 price / 64 image / 64 details` of `135`
- interpretation: discovery works, but active rows and scraped identifiers need alias/dedupe hardening; price selector also needs repair

### T&S Communication

- added `scrape:tscom`
- patched `tscom.ts` to pass product URLs into DB upsert/detail verification
- live run found `109` unique products and `0` prices
- active gap remains `82 price / 82 image / 49 details` of `82`
- interpretation: discovery works, but row matching and price selectors need hardening

### 10Gtek / SFPcables

- live run found `110` products and wrote `6` prices
- active 10Gtek gap remains `126 price / 131 image / 25 details` of `175`
- interpretation: parser works partially; remaining rows need alias/source-page repair

## Next Real Work

- Build alias/dedupe resolvers for GBICS, T&S, and 10Gtek similar to FS.com numeric SKU alias cleanup.
- Repair GBICS/T&S price selectors against current page HTML.
- For OEM vendors such as Cisco/Juniper, add explicit non-public-price states instead of pretending OEM list prices exist.

## Follow-Up: Artifact Quarantine + Reconcile Guard

GBICS/T&S artifact quarantine was expanded and applied live.

Result:

- `93` additional non-transceiver artifacts quarantined with `artifact_quarantine` evidence
- GBICS active base moved from `135` rows with `64/64/64` price/image/details gaps to `88` rows with `17/17/17` gaps
- T&S Communication active base moved from `82` rows with `82/82/49` price/image/details gaps to `36` rows with `36/36/6` gaps
- 10Gtek current active gap remains `175` rows with `126/131/25` price/image/details gaps

Important regression found:

- `maintenance:reconcile-verification` reset explicit competitor research states back to `needs_research`
- fixed scheduler guard so reconcile preserves both `no_valid_match` and `ambiguous`
- deployed to Erik, built successfully, and restarted `tip-scraper-daemon` only after pg-boss active queue was empty

Restore run:

- dry-run found `615` fully product-verified rows that had fallen back into competitor `needs_research`
- apply wrote:
  - `74` `no_valid_match`
  - `541` `ambiguous`
  - `74` newly/restored `fully_verified`
- live evidence ledger now includes:
  - `74` `competitor_no_match` from `verify:open-competitor-status`
  - `555` `competitor_ambiguous` from `verify:open-competitor-status`
  - `93` `artifact_quarantine` from `verify:quarantine:non-transceivers`
  - `109` `artifact_quarantine` from `verify:fs:sku-aliases`

Verification after a fresh reconcile test:

- health status: healthy
- active products: `17212`
- price verified: `11414`
- image verified: `12016`
- details verified: `16702`
- fully verified: `10449`
- competitor status:
  - `matched=10775`
  - `no_valid_match=74`
  - `ambiguous=556`
  - `needs_research=5807`
- fully product-verified rows still in competitor `needs_research`: `0`

## Follow-Up: Product Page Asset Verifier

Added `packages/scraper/src/utils/verify-product-page-assets.ts` and script `verify:product-page-assets`.

Safety design:

- vendor-limited with `PRODUCT_ASSET_VENDOR`
- batch-limited with `PRODUCT_ASSET_LIMIT`
- dry-run by default
- apply only with `PRODUCT_ASSET_APPLY=1`
- uses existing `product_page_url` rows only
- writes through existing `markImageVerified` / `markDetailsVerified` helpers so the evidence ledger remains canonical
- infers details only from part number, product URL, and title; full body text was intentionally excluded after a dry-run showed T&S navigation links polluting speed inference

Live runs:

- extra GBICS quarantine:
  - `17` category/family artifacts removed from the active base
  - GBICS is now product-data complete in the active base: `71` active rows, `0/0/0` price/image/details gaps
- T&S Communication:
  - product-page asset verifier applied
  - `36` images verified
  - `36` details verified after adding DR8/SR8 reach inference
  - current active gaps: `36` active rows, `36` missing price, `0` missing image, `0` missing details
  - interpretation: T&S is public-price blocked; pages expose no deterministic price, only quote/no-price or `price: 0.00`
- 10Gtek / SFPcables:
  - dry-run caught `1.25G` / `2.5G` vs `25G` regex risk; regex was hardened before apply
  - `5` images verified
  - `10` details verified/improved
  - current active gaps: `175` active rows, `126` missing price, `126` missing image, `20` missing details
  - interpretation: remaining 10Gtek rows need URL/source alias discovery, not blind verification

Live health after this pass:

- health: healthy
- active products: `17195`
- price verified: `11414`
- image verified: `12057`
- details verified: `16713`
- fully verified: `10459`
- competitor status:
  - `matched=10775`
  - `no_valid_match=74`
  - `ambiguous=556`
  - `needs_research=5790`
