rene/transceiver-db
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
201 Commits 5 Branches 9 Tags
Commit Graph

26 Commits

Author SHA1 Message Date
Rene Fichtmueller
582965ecb5 fix: fs-com Phase 1+2 crawler.run() ENOENT guard — Crawlee catches and re-throws the post-run _isTaskReadyFunction ENOENT internally, which rejected crawler.run() and aborted Phase 2 before it could start. Wrap both crawler.run() calls in try/catch to swallow ENOENT from request_queues paths; all processing is already complete at this point. 2026-04-18 03:52:49 +02:00
Rene Fichtmueller
2304a65227 chore: changelog — daemon stability, ATGBICS Playwright, health monitor accuracy 2026-04-18 03:25:02 +02:00
Rene Fichtmueller
f191ece0e4 chore: changelog — FS.com ENOENT fix, PID lock, health monitor tiered alerts 2026-04-18 02:55:18 +02:00
Rene Fichtmueller
a1db0a7a90 chore: changelog — Playwright headless shell fix, withIsolatedStorage race fix, FS.com launchd fix 2026-04-18 02:35:55 +02:00
Rene Fichtmueller
c51af82a6b chore: sync-to-erik now includes scraper rebuild + daemon restart; changelog 2026-04-18 01:37:12 +02:00
Rene Fichtmueller
14d48d34b2 feat: Price Comparison dashboard + Eoptolink OEM scraper 
- Add public /api/price-comparison API (summary, top-50, per-SKU detail)
  — no auth required, 3 Express routes, DISTINCT ON latest-price logic
- Add '💲 Price Comparison' dashboard tab: stat cards, form-factor
  breakdown, top-50 SKU table (clickable rows → SKU detail), per-vendor
  price + stock + spread% lookup panel
- Add Eoptolink OEM catalog scraper (93 product-solution pages,
  part-number regex EOLO-*/EOLQ-* etc., no prices, seeds transceivers
  table as manufacturer entries)
- Register scrape:catalog:eoptolink in scheduler: schedule every 4h
  (40 */4 * * *), lazy-import worker, added to known-jobs array
 2026-04-18 01:02:08 +02:00
Rene Fichtmueller
5e4a950aa1 chore: changelog entry — MCP Server v0.2.0 (finder + switch-docs + Ollama LLM tools) 2026-04-18 00:29:42 +02:00
Rene Fichtmueller
23bdfc1585 feat: /api/hype-cycle/analysis endpoint — DB-backed Bass-fitted results from hype_cycle_analysis table 2026-04-18 00:11:08 +02:00
Rene Fichtmueller
9ecaffc475 feat: Mouser Electronics API scraper for OEM reference prices (Juniper/Cisco/Arista PIDs) 2026-04-18 00:04:35 +02:00
Rene Fichtmueller
03b776cc2a chore: prune 242 irrelevant vendors from DB (348→106), add changelog entry 2026-04-17 23:58:13 +02:00
Rene Fichtmueller
4559d376ca chore: changelog entries 2026-04-17 stock dashboard + Cisco TMG + Juniper HCT 2026-04-17 23:33:49 +02:00
Rene Fichtmueller
75cbd7cd86 feat: stock quality schema + QSFPTEK/NADDOD v2 scrapers with real-time stock counts 
- Migration 028 (retroactive): document warehouse columns added to stock_observations
- Migration 037: composite indexes for DISTINCT ON (transceiver_id, source_vendor_id) queries
- Migration 038: add stock_confidence (1/2/3), price_currency, price_includes_tax,
  stock_vendor_ts to stock_observations + TRUNCATE test-run data

db.ts: upsertStockObservation now accepts stockConfidence, priceCurrency,
priceIncludesTax, stockVendorTs; delta detection includes quantity_available

fs-com.ts: passes stockConfidence=3 + priceCurrency=EUR + priceIncludesTax=false

qsfptek.ts v2: Phase 1 API listing + Phase 2 detail-page stock extraction
- Parses 'X in real-time stock, DATE' from product detail pages
- Writes stock_observations with confidence=2 + stockVendorTs
- Up to 500 detail pages/run at 2s rate limit

naddod.ts v2: complete rewrite from WooCommerce to Astro sitemap-based
- Discovers products via /sitemaps/products.xml (600+ products)
- URL format: /products/XXXXX.html
- Extracts 'In Stock: X' exact counts from SSR HTML
- Writes both price + stock observations (confidence 1 or 2)
 2026-04-17 22:54:40 +02:00
Rene Fichtmueller
f8809d999f feat(scraper+api): warehouse stock data pipeline — FS.com v2, SmartOptics v2, Stock API 
Scraper changes:
- fs-com.ts v2: Playwright stealth patches + www.fs.com/de/ URL fix (de.fs.com DNS NXDOMAIN).
  Extracts DE-Lager, Global-Lager, Nachlieferung, units_sold, compatible_brands, price_net.
  Mac-side runner (run-fs-scraper-mac.sh) via SSH tunnel for residential IP access.
  Fast-fail connectivity check on datacenter IPs that are blocked by Cloudflare.
- smartoptics.ts v2: WooCommerce REST API fallback + 8 catalog categories + relative URL fix.
  Was finding only 8 products, now discovers 18+ with multi-category crawl.

DB layer:
- db.ts: add upsertStockObservation() — writes 10 new stock_observations columns
  (warehouse_de_qty, warehouse_global_qty, backorder_qty, units_sold, compatible_brands,
  price_net, product_url, delivery dates) with dedup check.

API:
- routes/stock.ts: GET /api/stock, /api/stock/summary, /api/stock/:id
  Warehouse breakdowns per transceiver/vendor with top-sellers and vendor summary.
- routes/review.ts: equivalence review queue (approve/reject/bulk-approve).
- index.ts: register /api/stock and /api/review routes.

Dashboard:
- index.html: 🏭 Stock tab with stat cards (DE-Lager, Global-Lager, Nachlieferung totals),
  top-sellers table, vendor breakdown, recently-restocked events, part-number lookup.

SQL migrations:
- 034: blog-review-tag, 035: price-observations is_anomalous, 036: transceiver-equivalences.
 2026-04-17 10:45:59 +02:00
Rene Fichtmueller
fd3476f5c4 fix(scraper): FiberMall URL schema + price parser + Flexoptix EUR comma bug 
FiberMall:
- Correct /store-XXXXX-name.htm category URLs (was /c/xxx/ → HTTP 404)
- Parser: split on new_proList_mainListLi, price from data-price on
  currency_price span — fix 0.00 false-match from SKU variant items
- Also scrape SKU brand variant links from .sku_item divs
- Result: 3,410 prices now in DB (was 0)

Flexoptix:
- Fix extractPrice regex for EUR thousand-separator format
  (2,921.60 EUR was parsed as 2 EUR)
- Add OSFP224 / 1.6T search queries (4 new, form factor was missing)
- Fix O.138HG2.C.05 stale price 3009.60→2921.60 EUR

Schema: competitor_verified + competitor_verified_at columns
added via ALTER TABLE (were referenced in code but missing in DB)

CHANGELOG: added 6 entries for 2026-04-12
 2026-04-12 04:26:35 +02:00
Rene Fichtmueller
a4eb02fbda chore: add changelog entries 2026-04-05 through 2026-04-09 2026-04-11 18:50:17 +02:00
Rene Fichtmueller
cec6de0982 chore: update changelog for 2026-04-11 session 2026-04-11 13:57:31 +02:00
Rene Fichtmueller
4eddbfbc7c feat(blog): Reduction Engine v1.0 + LaTeX/connector hard fails 
- Replace STEP8b_REDUCTION with 5-pass Reduction Engine:
  Pass 1: Repetition Kill (one concept, one home)
  Pass 2: Tech Prune (LaTeX hard delete, SKU removal, formula prose replacement)
  Pass 3: Flow Rebuild (close gaps after cuts, no new content)
  Pass 4: Weight Correction (title/content alignment throughout)
  Pass 5: Humanization (rhythm variation, hedge removal, punch ending)
  Target: 700-1000 words (600-1300 range, warnings outside)

- System prompt + STEP9 QA: add hard fails for
  LaTeX formulas (\[...\], \frac{}, \text{} etc) — destroys blog flow
  DR4 connector error (DR4=MPO-12, not LC duplex; FR4=LC duplex)
  Title/content mismatch (title topic must be the spine, not just the intro)

- Gold Standard 5: market alert / pricing article template
  (correct title alignment, no LaTeX, DR4=MPO-12, ending on topic)

- WRONG PATTERNS extended with 4 new entries covering above failures

- blog.ts: step log messages updated to 11-14/14; word count
  output shows % reduction and range warning (>1300 or <600)
 2026-04-04 08:57:21 +02:00
Rene Fichtmueller
15d02108aa chore: changelog — blog engine v5 + linkedin post 2026-04-04 2026-04-04 08:30:54 +02:00
Rene Fichtmueller
74dcc14e1e chore: changelog — proxy network geo/uptime fixes 2026-04-04 2026-04-04 08:15:58 +02:00
Rene Fichtmueller
5d53d3af6f docs: update changelog 2026-04-03/04 — scraper fixes, blog engine v4, proxy network, pg-boss fix 2026-04-04 07:58:29 +02:00
Rene Fichtmueller
edc9311d7b feat: add proxy network, image backfill, and scraper improvements 
- Add TIP Proxy Network (packages/proxy-agent): SOCKS5 proxy agent
  for residential IP bypass of CloudFront WAF blocks
- Add /api/proxy/* routes: node registration, heartbeat, load balancing
- Add image extraction to Flexoptix catalog scraper (GraphQL small_image)
- Add image extraction to Optcore scraper (Playwright gallery img)
- Fix Fluxlight price scraping (BigCommerce HTML structure: data-product-price-without-tax)
- Add SmartOptics scraper (8 DWDM/coherent products, og:image extraction)
- Fix findOrCreateScrapedTransceiver to update image_url for existing records
- Add image backfill script (backfill-images.ts): 178 Flexoptix images added
- Fix DB connection pool: max 5, idleTimeoutMillis 10s (was unlimited, caused >100 connections)
- Add proxy.ts utility for scraper proxy rotation
 2026-04-03 21:13:03 +02:00
Rene Fichtmueller
52d3588f55 fix: remove internal IPs, ports, tokens from changelog entries 2026-04-02 08:16:48 +02:00
Rene Fichtmueller
4b7f6de36d chore: changelog entry for auth feature 2026-04-02 07:32:40 +02:00
Rene Fichtmueller
6f57d53df5 chore: update CHANGELOG_PENDING with Pi fleet + prediction intelligence system entries 2026-04-02 02:14:45 +02:00
Rene Fichtmueller
70f8fcd975 feat: product intelligence layer — eBay enricher, community issues, datasheets+manuals API 
- Migration 020: product_issues table, condition/marketplace on price_observations, features JSONB
- eBay enricher: switch features/description/refurb prices + transceiver condition pricing
- Community issues scraper: Reddit/ServeTheHome/Arista/Cisco community bug reports
- 7 pre-seeded issues (DCS-7800R3, SG350, QFX5120, CRS326, USW-Pro etc.)
- API: /switches/:id/issues + /switches/:id/documents endpoints
- Dashboard switch modal: features from DB, description, eBay refurb price, issues+docs async
- Datasheet finder for Arista/Cisco/Juniper/HPE vendor pages
- Scheduler: 4 new jobs (ebay enrichment nightly, community issues weekly)
 2026-04-01 22:46:27 +02:00
Rene Fichtmueller
3a00224ef0 feat: Changelog — CHANGELOG_PENDING.md, /api/changelog route, Overview tab widget 
- CHANGELOG_PENDING.md: 26 entries from v0.1.0 to today in JSON-line format
- GET /api/changelog: parses and serves entries as JSON array
- Overview tab: changelog card with type badges (FEAT/FIX/UI/DATA/AI/INFRA),
  dates, show recent/all toggle
 2026-04-01 22:14:14 +02:00