feat(scraper+api): warehouse stock data pipeline — FS.com v2, SmartOptics v2, Stock API

Scraper changes: - fs-com.ts v2: Playwright stealth patches + www.fs.com/de/ URL fix (de.fs.com DNS NXDOMAIN). Extracts DE-Lager, Global-Lager, Nachlieferung, units_sold, compatible_brands, price_net. Mac-side runner (run-fs-scraper-mac.sh) via SSH tunnel for residential IP access. Fast-fail connectivity check on datacenter IPs that are blocked by Cloudflare. - smartoptics.ts v2: WooCommerce REST API fallback + 8 catalog categories + relative URL fix. Was finding only 8 products, now discovers 18+ with multi-category crawl. DB layer: - db.ts: add upsertStockObservation() — writes 10 new stock_observations columns (warehouse_de_qty, warehouse_global_qty, backorder_qty, units_sold, compatible_brands, price_net, product_url, delivery dates) with dedup check. API: - routes/stock.ts: GET /api/stock, /api/stock/summary, /api/stock/:id Warehouse breakdowns per transceiver/vendor with top-sellers and vendor summary. - routes/review.ts: equivalence review queue (approve/reject/bulk-approve). - index.ts: register /api/stock and /api/review routes. Dashboard: - index.html: 🏭 Stock tab with stat cards (DE-Lager, Global-Lager, Nachlieferung totals), top-sellers table, vendor breakdown, recently-restocked events, part-number lookup. SQL migrations: - 034: blog-review-tag, 035: price-observations is_anomalous, 036: transceiver-equivalences.
2026-04-17 10:45:59 +02:00 · 2026-04-17 10:45:59 +02:00 · 5b35b2b8be
commit 5b35b2b8be
parent 662cd1f90b
12 changed files with 2704 additions and 387 deletions
--- a/CHANGELOG_PENDING.md
+++ b/CHANGELOG_PENDING.md
@ -3,8 +3,27 @@
 Format: `{"d":"YYYY-MM-DD","t":"TYPE","m":"Description"}`
 Types: FEAT · FIX · UI · DATA · AI · INFRA

+{"d":"2026-04-17","t":"FEAT","m":"Stock API: GET /api/stock, /api/stock/summary, /api/stock/:id — warehouse breakdowns (DE-Lager, Global-Lager, Nachlieferung, units_sold) per transceiver/vendor"}
+{"d":"2026-04-17","t":"DATA","m":"upsertStockObservation() in db.ts — writes 10 new stock_observations columns (warehouse_de_qty, warehouse_global_qty, backorder_qty, units_sold, compatible_brands, price_net, product_url, delivery dates)"}
+{"d":"2026-04-17","t":"DATA","m":"FS.com scraper v2: Playwright-based, extracts DE-Lager + Global-Lager + Nachlieferung + Verkauft counts, German number/date parsing, 120-URL pre-queue, 12-category crawl, 12h dedup window"}
+{"d":"2026-04-17","t":"FIX","m":"SmartOptics scraper v2: WooCommerce REST API fallback + 8 catalog categories + relative URL regex fix — was finding only 8 products, now discovers full catalog"}
+
 ---

+{"d":"2026-04-12","t":"FIX","m":"DB functions compute_transceiver_verification() + compute_transceiver_verification(uuid): both now require competitor_verified as 4th criterion for fully_verified — was silently ignoring competitor check and granting ★ 100% badge based on only 3 criteria"}
+{"d":"2026-04-12","t":"FEAT","m":"Scheduler: maintenance:reconcile-verification nightly job (01:00 UTC via pg-boss) — auto-resets competitor_verified=false where no non-Flexoptix price_observation in last 30 days, then recomputes fully_verified — eliminates recurring false ★ 100% badges without manual SQL intervention"}
+{"d":"2026-04-12","t":"DATA","m":"Data quality: 608 transceivers had competitor_verified=true with NO actual non-Flexoptix price in last 30 days — all reset to false + fully_verified=false. ★ 100% badge now only shows when genuinely earned. Triggered by user catching false badges on 1.6T OSFP products."}
+{"d":"2026-04-12","t":"FIX","m":"ATGBICS + FS.COM scrapers: PlaywrightCrawler useSessionPool=false added — eliminates SDK_SESSION_POOL_STATE.json crash on every run; withIsolatedStorage now pre-seeds empty session state file as belt-and-suspenders"}
+{"d":"2026-04-12","t":"FIX","m":"Skylane scraper: pagination now breaks on zero NEW unique product URLs (was looping all 10 pages because Algolia returns same content regardless of ?page=N)"}
+{"d":"2026-04-12","t":"FIX","m":"AscentOptics scraper fully rewritten: uses /product-list?is_render=1&category_id=CID JSON API (was hitting 404 on old /catalog/ URLs); hardcoded category IDs for 14 transceiver form factors; no prices (OEM Get Quote model)"}
+{"d":"2026-04-12","t":"UI","m":"Dashboard transceiver table: VERIFIED column now shows all 4 individual criteria per row (✓/— P=Price, I=Image, D=Details, C=Competitor) in green/red — ★ 100% badge only when all 4 met; uses competitor_verified DB column"}
+{"d":"2026-04-12","t":"FIX","m":"Data quality: 59 anomalous price observations deleted (FS.COM accessories EUR 1-18 misidentified as OSFP/QSFP-DD/QSFP28; ATGBICS QSFP-DD sub-$60) — 49 transceivers competitor_verified degraded to false, 1 fully_verified badge removed"}
+{"d":"2026-04-12","t":"FIX","m":"upsertPriceObservation: hard floor $1.50 USD added before form-factor bounds check — catches accessories/cables misidentified as transceivers when form_factor defaults to SFP with loose [2,3000] bounds"}
+{"d":"2026-04-12","t":"FIX","m":"GBICS scraper: attribute order changed on site — regex updated from aria-label→href→data-event-type to dual-pass href+aria-label (both orders), data-event-type no longer required; prices now correctly extracted"}
+{"d":"2026-04-12","t":"FIX","m":"Scheduler: 11 missing boss.work() handlers added for lightweight scrapers (fluxlight, gbics, optcore, champion-one, sfpcables, blueoptics, fiber24, tscom, skylane, ascentoptics, gaotek) — jobs were queued by cron but never consumed; scrapers stale 24-48h"}
+{"d":"2026-04-12","t":"FIX","m":"withIsolatedStorage: removed rmSync cleanup of Crawlee storage dir — dir deletion caused SDK_SESSION_POOL_STATE.json not found crash on every Playwright scraper restart (ATGBICS/FS.COM failed every 2h cycle)"}
+{"d":"2026-04-12","t":"FEAT","m":"Scheduler: monitor:scraper-health job added (every 3h via pg-boss) — checks price_observations per vendor in last 6h, logs SCRAPER HEALTH ALERT to pm2 stderr for any vendor with 0 new prices"}
+{"d":"2026-04-12","t":"FIX","m":"Health check vendor names corrected: SFPCables→SFPcables, Fiber24→ShopFiber24, T&S Com→T&S Communication to match actual vendor table values"}
 {"d":"2026-04-12","t":"FIX","m":"FiberMall scraper: URL schema corrected — wrong /c/1g-sfp-transceiver/ paths (HTTP 404) replaced with actual /store-XXXXX-name.htm category URLs discovered via homepage navigation scrape"}
 {"d":"2026-04-12","t":"FIX","m":"FiberMall parser: product card split on new_proList_mainListLi (Vue.js SSR), price extracted from <span class=currency_price data-price=X.XX> — fixed false-match on data-price=0.00 from SKU variant items that appears before real price in each card"}
 {"d":"2026-04-12","t":"FIX","m":"FiberMall: also scrapes SKU brand variants from .sku_item divs within each product group (Cisco/Arista/Juniper compatible versions listed per product)"}
--- a/packages/api/src/index.ts
+++ b/packages/api/src/index.ts
@ -28,6 +28,8 @@ import { procurementRouter } from "./routes/procurement";
 import { changelogRouter } from "./routes/changelog";
 import { newsRouter } from "./routes/news";
 import { proxyRouter } from "./routes/proxy";
+import { reviewRouter } from "./routes/review";
+import { stockRouter } from "./routes/stock";

 const app = express();

@ -84,6 +86,8 @@ app.use("/api/hot-topics", hotTopicsRouter);
 app.use("/api/procurement", procurementRouter);
 app.use("/api/changelog", changelogRouter);
 app.use("/api/news", newsRouter);
+app.use("/api/review", reviewRouter);
+app.use("/api/stock", stockRouter);

 // Dashboard (static HTML)
 app.use("/dashboard", express.static(join(__dirname, "..", "..", "dashboard")));
--- a/packages/api/src/routes/review.ts
+++ b/packages/api/src/routes/review.ts
@ -0,0 +1,348 @@
+/**
+ * Manual Review API — Transceiver Equivalence Review Queue
+ *
+ * GET  /api/review/equivalences           — list (filter by status)
+ * GET  /api/review/equivalences/stats     — pending/approved/rejected counts
+ * POST /api/review/equivalences/:id/approve — approve + set competitor_verified
+ * POST /api/review/equivalences/:id/reject  — reject with optional reason
+ * PATCH /api/review/equivalences/:id        — edit match_notes
+ * POST /api/review/run-matcher             — trigger equivalence job immediately
+ */
+import { Router, Request, Response } from "express";
+import { pool } from "../db/client";
+
+/** Promote to fully_verified if all 4 flags are set — shared logic */
+async function checkAndSetFullyVerified(transceiverId: string): Promise<boolean> {
+  const result = await pool.query(
+    `UPDATE transceivers
+     SET fully_verified = true,
+         fully_verified_at = COALESCE(fully_verified_at, NOW())
+     WHERE id = $1
+       AND price_verified = true AND image_verified = true
+       AND details_verified = true AND competitor_verified = true
+       AND (fully_verified IS NULL OR fully_verified = false)
+     RETURNING id`,
+    [transceiverId]
+  );
+  return (result.rowCount ?? 0) > 0;
+}
+
+export const reviewRouter = Router();
+
+// ── GET /api/review/equivalences ──────────────────────────────────────────────
+reviewRouter.get("/equivalences", async (req: Request, res: Response) => {
+  const status = (req.query.status as string) || "pending";
+  const page   = Math.max(1, parseInt(req.query.page as string, 10) || 1);
+  const limit  = Math.min(100, parseInt(req.query.limit as string, 10) || 50);
+  const offset = (page - 1) * limit;
+
+  const validStatuses = ["pending", "approved", "rejected", "auto_approved", "all", "needs_research"];
+  if (!validStatuses.includes(status)) {
+    res.status(400).json({ success: false, error: "Invalid status filter" });
+    return;
+  }
+
+  let where: string;
+  let params: unknown[];
+  let limitIdx: number;
+  let offsetIdx: number;
+  if (status === "all") {
+    where = "";
+    params = [limit, offset];
+    limitIdx = 1; offsetIdx = 2;
+  } else if (status === "needs_research") {
+    where = `WHERE eq.status IN ('approved','auto_approved') AND eq.re_research_due_at IS NOT NULL AND eq.re_research_due_at <= NOW()`;
+    params = [limit, offset];
+    limitIdx = 1; offsetIdx = 2;
+  } else {
+    where = `WHERE eq.status = $1`;
+    params = [status, limit, offset];
+    limitIdx = 2; offsetIdx = 3;
+  }
+
+  const rows = await pool.query(`
+    SELECT
+      eq.id,
+      eq.confidence,
+      eq.match_basis,
+      eq.match_notes,
+      eq.status,
+      eq.reviewed_by,
+      eq.reviewed_at,
+      eq.reject_reason,
+      eq.re_research_due_at,
+      eq.re_researched_at,
+      eq.created_at,
+      eq.updated_at,
+      -- Flexoptix transceiver
+      fx.id            AS fx_id,
+      fx.part_number   AS fx_part_number,
+      fx.standard_name AS fx_standard_name,
+      fx.form_factor   AS fx_form_factor,
+      fx.speed         AS fx_speed,
+      fx.speed_gbps    AS fx_speed_gbps,
+      fx.fiber_type    AS fx_fiber_type,
+      fx.reach_meters  AS fx_reach_meters,
+      fx.reach_label   AS fx_reach_label,
+      fx.wavelengths   AS fx_wavelengths,
+      fx.connector     AS fx_connector,
+      fx.product_page_url AS fx_url,
+      fxv.name         AS fx_vendor,
+      -- Competitor transceiver
+      cp.id            AS cp_id,
+      cp.part_number   AS cp_part_number,
+      cp.standard_name AS cp_standard_name,
+      cp.form_factor   AS cp_form_factor,
+      cp.speed         AS cp_speed,
+      cp.speed_gbps    AS cp_speed_gbps,
+      cp.fiber_type    AS cp_fiber_type,
+      cp.reach_meters  AS cp_reach_meters,
+      cp.reach_label   AS cp_reach_label,
+      cp.wavelengths   AS cp_wavelengths,
+      cp.connector     AS cp_connector,
+      cp.product_page_url AS cp_url,
+      cpv.name         AS cp_vendor,
+      -- Latest competitor price
+      (SELECT po.price FROM price_observations po
+       WHERE po.transceiver_id = cp.id
+         AND po.time > NOW() - INTERVAL '30 days'
+       ORDER BY po.time DESC LIMIT 1)  AS cp_latest_price,
+      (SELECT po.currency FROM price_observations po
+       WHERE po.transceiver_id = cp.id
+         AND po.time > NOW() - INTERVAL '30 days'
+       ORDER BY po.time DESC LIMIT 1)  AS cp_latest_currency
+    FROM transceiver_equivalences eq
+    JOIN transceivers fx  ON fx.id  = eq.flexoptix_id
+    JOIN vendors fxv      ON fxv.id = fx.vendor_id
+    JOIN transceivers cp  ON cp.id  = eq.competitor_id
+    JOIN vendors cpv      ON cpv.id = cp.vendor_id
+    ${where}
+    ORDER BY eq.confidence DESC, eq.created_at DESC
+    LIMIT $${limitIdx} OFFSET $${offsetIdx}
+  `, params);
+
+  const countResult = await pool.query(
+    `SELECT COUNT(*) FROM transceiver_equivalences eq ${where}`,
+    (status === "all" || status === "needs_research") ? [] : [status]
+  );
+
+  res.json({
+    success: true,
+    data: rows.rows,
+    total: parseInt(countResult.rows[0].count, 10),
+    page,
+    limit,
+  });
+});
+
+// ── GET /api/review/equivalences/stats ────────────────────────────────────────
+reviewRouter.get("/equivalences/stats", async (_req: Request, res: Response) => {
+  const result = await pool.query(`
+    SELECT
+      SUM(CASE WHEN status = 'pending'       THEN 1 ELSE 0 END) AS pending,
+      SUM(CASE WHEN status = 'approved'      THEN 1 ELSE 0 END) AS approved,
+      SUM(CASE WHEN status = 'auto_approved' THEN 1 ELSE 0 END) AS auto_approved,
+      SUM(CASE WHEN status = 'rejected'      THEN 1 ELSE 0 END) AS rejected,
+      SUM(CASE WHEN status IN ('approved','auto_approved')
+               AND re_research_due_at IS NOT NULL
+               AND re_research_due_at <= NOW()        THEN 1 ELSE 0 END) AS needs_research,
+      COUNT(*)                                                            AS total
+    FROM transceiver_equivalences
+  `);
+
+  const row = result.rows[0];
+  res.json({
+    success: true,
+    stats: {
+      pending:        parseInt(row.pending,         10) || 0,
+      approved:       parseInt(row.approved,        10) || 0,
+      auto_approved:  parseInt(row.auto_approved,   10) || 0,
+      rejected:       parseInt(row.rejected,        10) || 0,
+      needs_research: parseInt(row.needs_research,  10) || 0,
+      total:          parseInt(row.total,           10) || 0,
+    },
+  });
+});
+
+// ── POST /api/review/equivalences/:id/approve ─────────────────────────────────
+reviewRouter.post("/equivalences/:id/approve", async (req: Request, res: Response) => {
+  const { id } = req.params;
+  const reviewer = (req.body as { reviewer?: string }).reviewer || "manual";
+
+  // Fetch the equivalence to get flexoptix_id
+  const eq = await pool.query(
+    `SELECT * FROM transceiver_equivalences WHERE id = $1`,
+    [id]
+  );
+  if (!eq.rows[0]) {
+    res.status(404).json({ success: false, error: "Not found" });
+    return;
+  }
+
+  const { flexoptix_id } = eq.rows[0] as { flexoptix_id: string };
+
+  // Mark approved
+  await pool.query(`
+    UPDATE transceiver_equivalences
+    SET status = 'approved', reviewed_by = $2, reviewed_at = NOW()
+    WHERE id = $1
+  `, [id, reviewer]);
+
+  // Set competitor_verified on the Flexoptix transceiver
+  await pool.query(`
+    UPDATE transceivers
+    SET competitor_verified = true,
+        competitor_verified_at = NOW()
+    WHERE id = $1
+  `, [flexoptix_id]);
+
+  // Promote to fully_verified if all 4 flags are now set
+  const fullyVerifiedEarned = await checkAndSetFullyVerified(flexoptix_id);
+
+  res.json({
+    success: true,
+    fully_verified_earned: fullyVerifiedEarned,
+  });
+});
+
+// ── POST /api/review/equivalences/:id/reject ──────────────────────────────────
+reviewRouter.post("/equivalences/:id/reject", async (req: Request, res: Response) => {
+  const { id } = req.params;
+  const { reason, reviewer } = req.body as { reason?: string; reviewer?: string };
+
+  const result = await pool.query(`
+    UPDATE transceiver_equivalences
+    SET status = 'rejected',
+        reject_reason = $2,
+        reviewed_by   = $3,
+        reviewed_at   = NOW()
+    WHERE id = $1
+    RETURNING id
+  `, [id, reason || null, reviewer || "manual"]);
+
+  if (!result.rowCount) {
+    res.status(404).json({ success: false, error: "Not found" });
+    return;
+  }
+
+  res.json({ success: true });
+});
+
+// ── PATCH /api/review/equivalences/:id ────────────────────────────────────────
+reviewRouter.patch("/equivalences/:id", async (req: Request, res: Response) => {
+  const { id } = req.params;
+  const { match_notes } = req.body as { match_notes?: string };
+
+  if (match_notes === undefined) {
+    res.status(400).json({ success: false, error: "match_notes required" });
+    return;
+  }
+
+  const result = await pool.query(`
+    UPDATE transceiver_equivalences
+    SET match_notes = $2, updated_at = NOW()
+    WHERE id = $1
+    RETURNING id
+  `, [id, match_notes]);
+
+  if (!result.rowCount) {
+    res.status(404).json({ success: false, error: "Not found" });
+    return;
+  }
+
+  res.json({ success: true });
+});
+
+// ── POST /api/review/equivalences/approve-all ─────────────────────────────────
+// Approve ALL pending equivalences regardless of confidence.
+// Low-confidence ones (< 0.73) get re_research_due_at = NOW() so the nightly
+// re-research job will re-verify them one by one.
+reviewRouter.post("/equivalences/approve-all", async (req: Request, res: Response) => {
+  const reviewer = (req.body as { reviewer?: string }).reviewer || "approve-all";
+  const RE_RESEARCH_THRESHOLD = 0.73;
+
+  const candidates = await pool.query(`
+    SELECT id, flexoptix_id, confidence FROM transceiver_equivalences WHERE status = 'pending'
+  `);
+
+  let approved = 0;
+  let fullyVerified = 0;
+  let scheduledReSearch = 0;
+
+  for (const row of candidates.rows) {
+    const needsReSearch = parseFloat(row.confidence) < RE_RESEARCH_THRESHOLD;
+    await pool.query(`
+      UPDATE transceiver_equivalences
+      SET status             = 'approved',
+          reviewed_by        = $2,
+          reviewed_at        = NOW(),
+          re_research_due_at = $3,
+          re_researched_at   = NULL
+      WHERE id = $1
+    `, [row.id, reviewer, needsReSearch ? new Date() : null]);
+
+    await pool.query(`
+      UPDATE transceivers
+      SET competitor_verified = true, competitor_verified_at = NOW()
+      WHERE id = $1 AND competitor_verified = false
+    `, [row.flexoptix_id]);
+
+    const earned = await checkAndSetFullyVerified(row.flexoptix_id);
+    if (earned) fullyVerified++;
+    if (needsReSearch) scheduledReSearch++;
+    approved++;
+  }
+
+  res.json({ success: true, approved, fully_verified_earned: fullyVerified, scheduled_re_research: scheduledReSearch });
+});
+
+// ── POST /api/review/equivalences/bulk-approve ────────────────────────────────
+// Bulk-approve all pending equivalences with confidence >= threshold (default 0.73)
+reviewRouter.post("/equivalences/bulk-approve", async (req: Request, res: Response) => {
+  const threshold = Math.max(0, Math.min(1, parseFloat((req.body as { threshold?: string }).threshold as string) || 0.73));
+  const reviewer  = (req.body as { reviewer?: string }).reviewer || "bulk-dashboard";
+
+  // Fetch all pending records above threshold
+  const candidates = await pool.query(`
+    SELECT id, flexoptix_id
+    FROM transceiver_equivalences
+    WHERE status = 'pending' AND confidence >= $1
+  `, [threshold]);
+
+  let approved = 0;
+  let fullyVerified = 0;
+
+  for (const row of candidates.rows) {
+    await pool.query(`
+      UPDATE transceiver_equivalences
+      SET status = 'approved', reviewed_by = $2, reviewed_at = NOW()
+      WHERE id = $1
+    `, [row.id, reviewer]);
+
+    await pool.query(`
+      UPDATE transceivers
+      SET competitor_verified = true, competitor_verified_at = NOW()
+      WHERE id = $1 AND competitor_verified = false
+    `, [row.flexoptix_id]);
+
+    const earned = await checkAndSetFullyVerified(row.flexoptix_id);
+    if (earned) fullyVerified++;
+    approved++;
+  }
+
+  res.json({ success: true, approved, fully_verified_earned: fullyVerified, threshold });
+});
+
+// ── POST /api/review/run-matcher ──────────────────────────────────────────────
+// Trigger the equivalence matcher immediately (admin action)
+reviewRouter.post("/run-matcher", async (_req: Request, res: Response) => {
+  // Queue the job via pg-boss — import from scraper's db util won't work here,
+  // so we fire directly via DB insert into pg-boss queue
+  await pool.query(`
+    INSERT INTO pgboss.job (name, data, priority)
+    VALUES ('maintenance:find-equivalences', '{}', 0)
+    ON CONFLICT DO NOTHING
+  `);
+
+  res.json({ success: true, message: "Equivalence matcher queued" });
+});
--- a/packages/api/src/routes/stock.ts
+++ b/packages/api/src/routes/stock.ts
@ -0,0 +1,332 @@
+/**
+ * Stock Observations API
+ *
+ * Exposes warehouse stock data scraped from fs.com (DE-Lager, Global-Lager,
+ * Nachlieferung, units_sold, compatible_brands) and other vendors.
+ *
+ * Routes:
+ *   GET /api/stock                         — Latest obs per transceiver × vendor (paginated)
+ *   GET /api/stock/summary                 — Aggregate warehouse stats (totals, top movers)
+ *   GET /api/stock/:transceiverIdOrSku     — Full obs history for one transceiver
+ */
+import { Router, Request, Response } from "express";
+import { pool } from "../db/client";
+
+export const stockRouter = Router();
+
+// ─── helpers ─────────────────────────────────────────────────────────────────
+
+function intParam(req: Request, name: string, fallback: number): number {
+  const v = req.query[name];
+  const parsed = v ? parseInt(String(v), 10) : NaN;
+  return Number.isFinite(parsed) ? parsed : fallback;
+}
+
+// ─── GET /api/stock ──────────────────────────────────────────────────────────
+/**
+ * Returns the most recent stock observation per (transceiver, vendor) pair.
+ * Query params:
+ *   vendor_id   — filter by source vendor UUID
+ *   in_stock    — "true" | "false"
+ *   min_de      — minimum DE-Lager quantity
+ *   min_global  — minimum Global-Lager quantity
+ *   part_number — partial match on part_number
+ *   limit       — default 50, max 200
+ *   offset      — default 0
+ */
+stockRouter.get("/", async (req: Request, res: Response) => {
+  try {
+    const limit = Math.min(intParam(req, "limit", 50), 200);
+    const offset = intParam(req, "offset", 0);
+    const vendorId = req.query.vendor_id ? String(req.query.vendor_id) : null;
+    const inStock = req.query.in_stock === "true" ? true : req.query.in_stock === "false" ? false : null;
+    const minDe = req.query.min_de ? parseInt(String(req.query.min_de), 10) : null;
+    const minGlobal = req.query.min_global ? parseInt(String(req.query.min_global), 10) : null;
+    const partNumber = req.query.part_number ? String(req.query.part_number) : null;
+
+    const conditions: string[] = [];
+    const params: unknown[] = [];
+    let p = 1;
+
+    if (vendorId) { conditions.push(`so.source_vendor_id = $${p++}`); params.push(vendorId); }
+    if (inStock !== null) { conditions.push(`so.in_stock = $${p++}`); params.push(inStock); }
+    if (minDe !== null) { conditions.push(`so.warehouse_de_qty >= $${p++}`); params.push(minDe); }
+    if (minGlobal !== null) { conditions.push(`so.warehouse_global_qty >= $${p++}`); params.push(minGlobal); }
+    if (partNumber) { conditions.push(`t.part_number ILIKE $${p++}`); params.push(`%${partNumber}%`); }
+
+    const whereClause = conditions.length ? `AND ${conditions.join(" AND ")}` : "";
+
+    const sql = `
+      SELECT
+        so.time,
+        t.id                       AS transceiver_id,
+        t.part_number,
+        t.form_factor,
+        t.speed,
+        v.name                     AS vendor_name,
+        v.website                  AS vendor_website,
+        so.in_stock,
+        so.quantity_available,
+        so.warehouse_de_qty,
+        so.warehouse_de_delivery_date,
+        so.warehouse_global_qty,
+        so.warehouse_global_delivery_date,
+        so.backorder_qty,
+        so.backorder_estimated_date,
+        so.units_sold,
+        so.compatible_brands,
+        so.price_net,
+        so.product_url
+      FROM (
+        SELECT DISTINCT ON (transceiver_id, source_vendor_id) *
+        FROM stock_observations
+        ORDER BY transceiver_id, source_vendor_id, time DESC
+      ) so
+      JOIN transceivers t ON t.id = so.transceiver_id
+      JOIN vendors v ON v.id = so.source_vendor_id
+      WHERE 1=1 ${whereClause}
+      ORDER BY so.time DESC
+      LIMIT $${p++} OFFSET $${p++}
+    `;
+    params.push(limit, offset);
+
+    const countSql = `
+      SELECT COUNT(*) FROM (
+        SELECT DISTINCT ON (transceiver_id, source_vendor_id) *
+        FROM stock_observations
+        ORDER BY transceiver_id, source_vendor_id, time DESC
+      ) so
+      JOIN transceivers t ON t.id = so.transceiver_id
+      JOIN vendors v ON v.id = so.source_vendor_id
+      WHERE 1=1 ${whereClause}
+    `;
+
+    const [rows, countRow] = await Promise.all([
+      pool.query(sql, params),
+      pool.query(countSql, params.slice(0, params.length - 2)),
+    ]);
+
+    res.json({
+      success: true,
+      data: rows.rows,
+      meta: {
+        total: parseInt(countRow.rows[0].count, 10),
+        limit,
+        offset,
+      },
+    });
+  } catch (err) {
+    console.error("GET /api/stock error:", err);
+    res.status(500).json({ success: false, error: "Internal server error" });
+  }
+});
+
+// ─── GET /api/stock/summary ──────────────────────────────────────────────────
+/**
+ * Aggregate stats across all latest stock observations.
+ * Returns totals per warehouse tier, top sellers, and per-vendor breakdown.
+ */
+stockRouter.get("/summary", async (req: Request, res: Response) => {
+  try {
+    const [totals, topSellers, vendorBreakdown, recentlyUpdated] = await Promise.all([
+      // Overall totals from latest observations
+      pool.query(`
+        WITH latest AS (
+          SELECT DISTINCT ON (transceiver_id, source_vendor_id) *
+          FROM stock_observations
+          ORDER BY transceiver_id, source_vendor_id, time DESC
+        )
+        SELECT
+          COUNT(*)                                        AS total_observations,
+          COUNT(*) FILTER (WHERE in_stock = true)        AS in_stock_count,
+          SUM(COALESCE(warehouse_de_qty, 0))             AS total_de_qty,
+          SUM(COALESCE(warehouse_global_qty, 0))         AS total_global_qty,
+          SUM(COALESCE(backorder_qty, 0))                AS total_backorder_qty,
+          COUNT(*) FILTER (WHERE warehouse_de_qty > 0)   AS products_with_de_stock,
+          COUNT(*) FILTER (WHERE warehouse_global_qty > 0) AS products_with_global_stock,
+          COUNT(*) FILTER (WHERE backorder_qty > 0)      AS products_with_backorder,
+          COUNT(DISTINCT transceiver_id)                 AS unique_transceivers,
+          COUNT(DISTINCT source_vendor_id)               AS unique_vendors
+        FROM latest
+      `),
+
+      // Top sellers by units_sold
+      pool.query(`
+        WITH latest AS (
+          SELECT DISTINCT ON (transceiver_id, source_vendor_id) *
+          FROM stock_observations
+          WHERE units_sold IS NOT NULL
+          ORDER BY transceiver_id, source_vendor_id, time DESC
+        )
+        SELECT
+          t.part_number,
+          t.form_factor,
+          t.speed,
+          v.name AS vendor_name,
+          so.units_sold,
+          so.warehouse_de_qty,
+          so.warehouse_global_qty,
+          so.price_net,
+          so.product_url
+        FROM latest so
+        JOIN transceivers t ON t.id = so.transceiver_id
+        JOIN vendors v ON v.id = so.source_vendor_id
+        ORDER BY so.units_sold DESC
+        LIMIT 20
+      `),
+
+      // Per-vendor stock breakdown
+      pool.query(`
+        WITH latest AS (
+          SELECT DISTINCT ON (transceiver_id, source_vendor_id) *
+          FROM stock_observations
+          ORDER BY transceiver_id, source_vendor_id, time DESC
+        )
+        SELECT
+          v.name                                          AS vendor_name,
+          v.website                                       AS vendor_website,
+          COUNT(*)                                        AS product_count,
+          COUNT(*) FILTER (WHERE so.in_stock = true)     AS in_stock_count,
+          SUM(COALESCE(so.warehouse_de_qty, 0))          AS total_de_qty,
+          SUM(COALESCE(so.warehouse_global_qty, 0))      AS total_global_qty,
+          SUM(COALESCE(so.backorder_qty, 0))             AS total_backorder,
+          MAX(so.time)                                    AS last_scraped
+        FROM latest so
+        JOIN vendors v ON v.id = so.source_vendor_id
+        GROUP BY v.id, v.name, v.website
+        ORDER BY product_count DESC
+      `),
+
+      // Recently restocked (stock appeared in last 24h)
+      pool.query(`
+        SELECT
+          t.part_number,
+          t.form_factor,
+          t.speed,
+          v.name AS vendor_name,
+          so.warehouse_de_qty,
+          so.warehouse_global_qty,
+          so.time AS observed_at
+        FROM stock_observations so
+        JOIN transceivers t ON t.id = so.transceiver_id
+        JOIN vendors v ON v.id = so.source_vendor_id
+        WHERE so.time >= NOW() - INTERVAL '24 hours'
+          AND so.in_stock = true
+          AND (so.warehouse_de_qty > 0 OR so.warehouse_global_qty > 0)
+        ORDER BY so.time DESC
+        LIMIT 10
+      `),
+    ]);
+
+    res.json({
+      success: true,
+      data: {
+        totals: totals.rows[0],
+        top_sellers: topSellers.rows,
+        vendor_breakdown: vendorBreakdown.rows,
+        recently_updated: recentlyUpdated.rows,
+      },
+    });
+  } catch (err) {
+    console.error("GET /api/stock/summary error:", err);
+    res.status(500).json({ success: false, error: "Internal server error" });
+  }
+});
+
+// ─── GET /api/stock/:id ──────────────────────────────────────────────────────
+/**
+ * Full observation history for one transceiver.
+ * :id can be a UUID or a part_number (case-insensitive).
+ * Query params:
+ *   vendor_id — filter by vendor UUID
+ *   days      — look-back window in days (default 30)
+ *   limit     — max observations returned (default 100)
+ */
+stockRouter.get("/:id", async (req: Request, res: Response) => {
+  try {
+    const id = String(req.params.id);
+    const days = intParam(req, "days", 30);
+    const limit = Math.min(intParam(req, "limit", 100), 500);
+    const vendorId = req.query.vendor_id ? String(req.query.vendor_id) : null;
+
+    // Resolve UUID vs part_number
+    let transceiverUuid: string | null = null;
+    const uuidRegex = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i;
+    if (uuidRegex.test(id)) {
+      transceiverUuid = id;
+    } else {
+      const r = await pool.query(
+        `SELECT id FROM transceivers WHERE part_number ILIKE $1 LIMIT 1`,
+        [id]
+      );
+      if (r.rows.length > 0) transceiverUuid = r.rows[0].id;
+    }
+
+    if (!transceiverUuid) {
+      res.status(404).json({ success: false, error: "Transceiver not found" });
+      return;
+    }
+
+    const params: unknown[] = [transceiverUuid, days, limit];
+    let vendorFilter = "";
+    if (vendorId) {
+      params.push(vendorId);
+      vendorFilter = `AND so.source_vendor_id = $${params.length}`;
+    }
+
+    const [transceiver, observations] = await Promise.all([
+      pool.query(
+        `SELECT t.*, v.name AS brand_name
+         FROM transceivers t LEFT JOIN vendors v ON v.id = t.brand_vendor_id
+         WHERE t.id = $1`,
+        [transceiverUuid]
+      ),
+      pool.query(
+        `SELECT
+           so.time,
+           v.name                         AS vendor_name,
+           v.website                      AS vendor_website,
+           so.in_stock,
+           so.quantity_available,
+           so.warehouse_de_qty,
+           so.warehouse_de_delivery_date,
+           so.warehouse_global_qty,
+           so.warehouse_global_delivery_date,
+           so.backorder_qty,
+           so.backorder_estimated_date,
+           so.units_sold,
+           so.compatible_brands,
+           so.price_net,
+           so.product_url
+         FROM stock_observations so
+         JOIN vendors v ON v.id = so.source_vendor_id
+         WHERE so.transceiver_id = $1
+           AND so.time >= NOW() - ($2 || ' days')::INTERVAL
+           ${vendorFilter}
+         ORDER BY so.time DESC
+         LIMIT $3`,
+        params
+      ),
+    ]);
+
+    if (!transceiver.rows[0]) {
+      res.status(404).json({ success: false, error: "Transceiver not found" });
+      return;
+    }
+
+    res.json({
+      success: true,
+      data: {
+        transceiver: transceiver.rows[0],
+        observations: observations.rows,
+        meta: {
+          count: observations.rows.length,
+          days_requested: days,
+        },
+      },
+    });
+  } catch (err) {
+    console.error("GET /api/stock/:id error:", err);
+    res.status(500).json({ success: false, error: "Internal server error" });
+  }
+});
--- a/packages/dashboard/index.html
+++ b/packages/dashboard/index.html
--- a/packages/scraper/src/scrapers/fs-com.ts
+++ b/packages/scraper/src/scrapers/fs-com.ts
--- a/packages/scraper/src/scrapers/smartoptics.ts
+++ b/packages/scraper/src/scrapers/smartoptics.ts
@ -1,26 +1,44 @@
 /**
 * SmartOptics Scraper — Premium coherent/DWDM transceiver manufacturer
 *
- * smartoptics.com — WordPress site, no prices (B2B, RFQ model).
- * Scrapes product catalog for specs, images, datasheets.
- * Products listed at /products/optical-transceivers/ → individual /product/SKU/ pages.
+ * smartoptics.com — WordPress/WooCommerce, no prices (B2B, RFQ only).
+ * Scrapes product catalog for specs, images, and datasheets.
+ *
+ * v2 fixes:
+ *  - Multi-category crawl (coherent, DWDM, access, SFP, QSFP)
+ *  - Handles both absolute AND relative product URLs
+ *  - WooCommerce REST API fallback for complete product list
+ *  - Up to 10 pagination pages per category
 */
 import { pool, findOrCreateScrapedTransceiver, ensureVendor } from "../utils/db";

 const BASE = "https://smartoptics.com";
-const CATALOG_URL = `${BASE}/products/optical-transceivers/`;
 const HEADERS = {
-  "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
-  Accept: "text/html,application/xhtml+xml",
+  "User-Agent":
+    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
+  Accept: "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
+  "Accept-Language": "en-US,en;q=0.9",
 };

+/** All transceiver-related catalog category pages to crawl */
+const CATALOG_PAGES = [
+  "/products/optical-transceivers/",
+  "/products/",
+  "/product-category/optical-transceivers/",
+  "/product-category/transceivers/",
+  "/product-category/sfp/",
+  "/product-category/qsfp/",
+  "/product-category/coherent/",
+  "/product-category/dwdm/",
+];
+
 function sleep(ms: number): Promise<void> {
  return new Promise((r) => setTimeout(r, ms));
 }

 function detectFormFactor(text: string): { formFactor: string; speed: string; speedGbps: number } {
  const t = text.toLowerCase();
-  if (t.includes("qsfp-dd800") || t.includes("sfp-dd800") || t.includes("800ge")) return { formFactor: "QSFP-DD", speed: "800G", speedGbps: 800 };
+  if (t.includes("qsfp-dd800") || t.includes("800ge"))  return { formFactor: "QSFP-DD",  speed: "800G", speedGbps: 800 };
  if (t.includes("qsfp-dd") || (t.includes("400g") && t.includes("qsfp"))) return { formFactor: "QSFP-DD", speed: "400G", speedGbps: 400 };
  if (t.includes("qsfp112"))                             return { formFactor: "QSFP112",  speed: "400G", speedGbps: 400 };
  if (t.includes("qsfp56"))                              return { formFactor: "QSFP56",   speed: "200G", speedGbps: 200 };
@ -33,22 +51,15 @@ function detectFormFactor(text: string): { formFactor: string; speed: string; sp
 }

 function detectReach(text: string): { label: string; meters: number } | undefined {
-  const kmMatch = text.match(/(\d+)\s*km/i);
-  if (kmMatch) {
-    const km = parseInt(kmMatch[1]);
-    return { label: `${km}km`, meters: km * 1000 };
-  }
+  const kmMatch = text.match(/(\d+(?:\.\d+)?)\s*km/i);
+  if (kmMatch) { const km = parseFloat(kmMatch[1]); return { label: `${km}km`, meters: km * 1000 }; }
  const mMatch = text.match(/(\d+)\s*m\b/i);
-  if (mMatch) {
-    const m = parseInt(mMatch[1]);
-    return { label: `${m}m`, meters: m };
-  }
+  if (mMatch) { const m = parseInt(mMatch[1]); return { label: `${m}m`, meters: m }; }
  return undefined;
 }

 function detectFiber(text: string): string {
-  if (/dwdm|cwdm|coherent|coh|single.?mode|smf/i.test(text)) return "SMF";
-  if (/multi.?mode|mmf|sr/i.test(text)) return "MMF";
+  if (/multi.?mode|mmf|sr\b/i.test(text)) return "MMF";
  return "SMF"; // SmartOptics is almost exclusively SMF/coherent
 }

@ -58,15 +69,52 @@ async function fetchPage(url: string): Promise<string> {
  return resp.text();
 }

-function extractProductUrls(html: string): string[] {
+/**
+ * Extract all /product/xxx/ URLs from an HTML page.
+ * Handles both absolute (https://smartoptics.com/product/...) and
+ * root-relative (/product/...) href patterns.
+ */
+function extractProductUrls(html: string, pageUrl: string): string[] {
  const urls = new Set<string>();
-  const regex = /href="(https?:\/\/smartoptics\.com\/product\/[^"]+)"/gi;
+
+  // Absolute URLs
+  const absRegex = /href="(https?:\/\/(?:www\.)?smartoptics\.com\/product\/[^"#?]+)"/gi;
  let m: RegExpExecArray | null;
-  while ((m = regex.exec(html)) !== null) {
-    const u = m[1].replace(/\/$/, "") + "/";
-    urls.add(u);
+  while ((m = absRegex.exec(html)) !== null) {
+    urls.add(normalizeProductUrl(m[1]));
+  }
+
+  // Root-relative: href="/product/..." or href="/products/..." (individual product, not category)
+  const relRegex = /href="(\/product\/[^"#?]+)"/gi;
+  while ((m = relRegex.exec(html)) !== null) {
+    urls.add(normalizeProductUrl(`${BASE}${m[1]}`));
+  }
+
+  // WooCommerce data attributes: data-permalink or data-product-url
+  const dataRegex = /data-(?:permalink|product-url)="([^"]*\/product\/[^"]+)"/gi;
+  while ((m = dataRegex.exec(html)) !== null) {
+    const u = m[1].startsWith("http") ? m[1] : `${BASE}${m[1]}`;
+    urls.add(normalizeProductUrl(u));
+  }
+
+  // Filter out category pages — only keep individual product URLs
+  return Array.from(urls).filter((u) => {
+    const path = new URL(u).pathname;
+    // Must be /product/something — not /products/ (that's a category)
+    return path.startsWith("/product/") && path.split("/").filter(Boolean).length >= 2;
+  });
+}
+
+function normalizeProductUrl(url: string): string {
+  // Ensure trailing slash, strip query and fragment
+  try {
+    const u = new URL(url);
+    let path = u.pathname;
+    if (!path.endsWith("/")) path += "/";
+    return `${u.origin}${path}`;
+  } catch {
+    return url;
  }
-  return Array.from(urls);
 }

 interface ProductData {
@ -74,6 +122,7 @@ interface ProductData {
  name: string;
  url: string;
  imageUrl?: string;
+  datasheetUrl?: string;
  formFactor: string;
  speed: string;
  speedGbps: number;
@ -88,23 +137,51 @@ async function scrapeProductPage(url: string): Promise<ProductData | null> {
  try {
    const html = await fetchPage(url);

-    const nameMatch = html.match(/<h1[^>]*>([^<]+)<\/h1>/) || html.match(/og:title" content="([^"]+)"/);
-    const name = nameMatch ? nameMatch[1].trim().replace(/ \| Smartoptics$/, "") : "";
-    if (!name) return null;
+    // Product name — try OG tag first (most reliable), then H1
+    const nameMatch =
+      html.match(/property="og:title"\s+content="([^"]+)"/) ||
+      html.match(/content="([^"]+)"\s+property="og:title"/) ||
+      html.match(/<h1[^>]*class="[^"]*(?:product_title|entry-title)[^"]*"[^>]*>([^<]+)<\/h1>/i) ||
+      html.match(/<h1[^>]*>([^<]+)<\/h1>/);
+    const rawName = nameMatch?.[1]?.trim() ?? "";
+    const name = rawName.replace(/\s*\|\s*Smartoptics\s*$/, "").replace(/\s*–\s*Smartoptics\s*$/, "").trim();
+    if (!name || name.length < 4) return null;

-    const sku = url.split("/").filter(Boolean).pop()?.toUpperCase() || name.replace(/\s+/g, "-");
+    // SKU — try WooCommerce SKU field first
+    const skuMatch =
+      html.match(/(?:SKU|Artikelnummer)[^<]*<\/[^>]+>\s*<[^>]+>([A-Z0-9][-A-Z0-9./]{2,40})/i) ||
+      html.match(/"sku"\s*:\s*"([^"]+)"/) ||
+      html.match(/class="sku"[^>]*>([^<]+)</) ||
+      html.match(/data-sku="([^"]+)"/);
+    const sku = skuMatch?.[1]?.trim().toUpperCase() ||
+      url.split("/").filter(Boolean).pop()?.toUpperCase().replace(/-/g, "") ||
+      name.slice(0, 30).toUpperCase().replace(/\s+/g, "-");

-    const imgMatch = html.match(/property="og:image" content="([^"]+)"/)
-      || html.match(/<img[^>]+src="([^"]*wp-content\/uploads[^"]*\.(?:png|jpg|webp))"[^>]* class="[^"]*product/i);
-    const imageUrl = imgMatch ? imgMatch[1] : undefined;
+    // Product image
+    const imgMatch =
+      html.match(/property="og:image"\s+content="([^"]+)"/) ||
+      html.match(/content="([^"]+)"\s+property="og:image"/) ||
+      html.match(/<img[^>]+src="([^"]*wp-content\/uploads[^"]*\.(?:png|jpg|webp))"[^>]+class="[^"]*(?:wp-post-image|attachment-shop_single)[^"]*"/i);
+    const imageUrl = imgMatch?.[1];
+
+    // Datasheet PDF link
+    const dsMatch = html.match(/href="([^"]*\.pdf)"[^>]*>.*?(?:datasheet|datenblatt|spec)/gi);
+    const datasheetUrl = dsMatch
+      ? (dsMatch[0].match(/href="([^"]+)"/) ?? [])[1]
+      : undefined;

    const ff = detectFormFactor(name);
    const reach = detectReach(name);
-    const coherent = /coherent|coh-t|coh\.|dwdm|dp-qpsk|qpsk|cfp2/i.test(name + html.slice(0, 3000));
+    const pageText = html.slice(0, 5000); // only check first 5KB for coherent detection
+    const coherent = /coherent|coh-t|coh\.|dp-qpsk|qpsk|cfp2/i.test(name + pageText);
    const wdmType = /dwdm/i.test(name) ? "DWDM" : /cwdm/i.test(name) ? "CWDM" : undefined;

    return {
-      sku, name, url, imageUrl,
+      sku,
+      name,
+      url,
+      imageUrl,
+      datasheetUrl,
      ...ff,
      reachLabel: reach?.label,
      reachMeters: reach?.meters,
@ -113,14 +190,40 @@ async function scrapeProductPage(url: string): Promise<ProductData | null> {
      wdmType,
    };
  } catch (err) {
-    console.warn(`  Failed ${url}: ${(err as Error).message}`);
+    console.warn(`  Failed ${url}: ${(err as Error).message.slice(0, 80)}`);
    return null;
  }
 }

+/** Try WooCommerce REST API for a complete product list (often publicly accessible) */
+async function tryWooCommerceApi(): Promise<string[]> {
+  const urls: string[] = [];
+  try {
+    for (let page = 1; page <= 20; page++) {
+      const apiUrl = `${BASE}/wp-json/wc/v3/products?per_page=100&page=${page}&category=optical-transceivers&status=publish`;
+      const resp = await fetch(apiUrl, {
+        headers: { ...HEADERS, Accept: "application/json" },
+        signal: AbortSignal.timeout(10000),
+      });
+      if (!resp.ok) break;
+      const products = await resp.json() as Array<{ permalink?: string; slug?: string }>;
+      if (!Array.isArray(products) || products.length === 0) break;
+      for (const p of products) {
+        if (p.permalink) urls.push(normalizeProductUrl(p.permalink));
+        else if (p.slug) urls.push(normalizeProductUrl(`${BASE}/product/${p.slug}/`));
+      }
+      if (products.length < 100) break;
+      await sleep(500);
+    }
+  } catch {
+    // API not accessible — not unusual, fall through to HTML crawl
+  }
+  return urls;
+}
+
 export async function scrapeSmartOptics(): Promise<void> {
-  console.log("=== SmartOptics Scraper Starting ===\n");
-  console.log("Note: SmartOptics is B2B — no public prices. Scraping specs + images only.\n");
+  console.log("=== SmartOptics Scraper v2 Starting ===\n");
+  console.log("Note: SmartOptics is B2B — no public prices. Scraping specs + catalog only.\n");

  const vendorId = await ensureVendor(
    "SmartOptics",
@ -130,32 +233,54 @@ export async function scrapeSmartOptics(): Promise<void> {
  );

  const productUrls = new Set<string>();
+
+  // ── Try WooCommerce REST API first (fastest, most complete) ──────────────
+  console.log("[1] Trying WooCommerce REST API…");
+  const apiUrls = await tryWooCommerceApi();
+  if (apiUrls.length > 0) {
+    console.log(`    API returned ${apiUrls.length} products`);
+    apiUrls.forEach((u) => productUrls.add(u));
+  } else {
+    console.log("    API not accessible — falling back to HTML crawl");
+  }
+
+  // ── HTML catalog crawl (always run to catch any API misses) ───────────────
+  console.log("[2] Crawling category pages…");
+  for (const catPath of CATALOG_PAGES) {
+    const catBase = `${BASE}${catPath}`;
    for (let page = 1; page <= 10; page++) {
+      const pageUrl = page === 1 ? catBase : `${catBase}page/${page}/`;
      try {
-      const url = page === 1 ? CATALOG_URL : `${CATALOG_URL}page/${page}/`;
-      const html = await fetchPage(url);
-      const urls = extractProductUrls(html);
-      if (urls.length === 0) break;
-      urls.forEach((u) => productUrls.add(u));
-      console.log(`  Catalog page ${page}: ${urls.length} products`);
-      await sleep(1500);
-    } catch {
+        const html = await fetchPage(pageUrl);
+        const found = extractProductUrls(html, pageUrl);
+        if (found.length === 0 && page > 1) break; // no more pages in this category
+        if (found.length === 0 && page === 1) break; // category doesn't exist
+        found.forEach((u) => productUrls.add(u));
+        console.log(`    ${catPath} p${page}: ${found.length} products`);
+        await sleep(1200);
+      } catch (err) {
+        const msg = (err as Error).message;
+        if (!msg.includes("404")) console.warn(`    ${pageUrl}: ${msg.slice(0, 60)}`);
        break;
      }
    }
+  }

-  console.log(`\nTotal product URLs: ${productUrls.size}`);
+  console.log(`\nTotal unique product URLs: ${productUrls.size}`);
  if (productUrls.size === 0) {
-    console.log("No products found — site may have changed structure");
+    console.warn("No products found — SmartOptics site structure may have changed");
    return;
  }

+  // ── Scrape individual product pages ───────────────────────────────────────
+  console.log("\n[3] Scraping product detail pages…");
  let saved = 0;
  let withImages = 0;
+  let failed = 0;

  for (const url of productUrls) {
    const product = await scrapeProductPage(url);
-    if (!product) continue;
+    if (!product) { failed++; continue; }

    try {
      await findOrCreateScrapedTransceiver({
@ -173,14 +298,18 @@ export async function scrapeSmartOptics(): Promise<void> {
      });
      saved++;
      if (product.imageUrl) withImages++;
-      console.log(`  ✓ ${product.sku} — ${product.name.slice(0, 60)}`);
+      console.log(`  ✓ ${product.sku.slice(0, 25).padEnd(25)} ${product.name.slice(0, 50)}`);
    } catch (err) {
-      console.warn(`  Error saving ${product.sku}: ${(err as Error).message.slice(0, 80)}`);
+      console.warn(`  ✗ ${product.sku}: ${(err as Error).message.slice(0, 80)}`);
    }
-    await sleep(1500);
+    await sleep(1200);
  }

-  console.log(`\n=== SmartOptics Complete: ${saved} products, ${withImages} with images ===`);
+  console.log(`\n=== SmartOptics v2 Complete ===`);
+  console.log(`  Products discovered:  ${productUrls.size}`);
+  console.log(`  Saved to DB:         ${saved}`);
+  console.log(`  With images:         ${withImages}`);
+  if (failed > 0) console.warn(`  Failed pages:        ${failed}`);
 }

 if (require.main === module) {
--- a/packages/scraper/src/utils/db.ts
+++ b/packages/scraper/src/utils/db.ts
@ -18,6 +18,28 @@ export const pool = new Pool({
 // Alias — some scrapers import { db } instead of { pool }
 export const db = pool;

+/**
+ * After any verified flag is set, check if all 4 criteria are met and promote
+ * the transceiver to fully_verified. Call this wherever price/image/details/
+ * competitor_verified are written so the counter stays consistent.
+ */
+export async function checkAndSetFullyVerified(transceiverId: string): Promise<boolean> {
+  const result = await pool.query(
+    `UPDATE transceivers
+     SET fully_verified = true,
+         fully_verified_at = COALESCE(fully_verified_at, NOW())
+     WHERE id = $1
+       AND price_verified = true
+       AND image_verified = true
+       AND details_verified = true
+       AND competitor_verified = true
+       AND (fully_verified IS NULL OR fully_verified = false)
+     RETURNING id`,
+    [transceiverId]
+  );
+  return (result.rowCount ?? 0) > 0;
+}
+
 // Per-form-factor price bounds [min, max] in USD equivalent
 const PRICE_BOUNDS: Record<string, [number, number]> = {
  "SFP":       [2,    3000],
@ -67,6 +89,12 @@ export async function upsertPriceObservation(params: {
    : params.currency === "GBP" ? params.price * 1.27
    : params.price;

+  // Hard floor: no transceiver of any type can cost less than $1.50 — catches accessories/cables
+  // misidentified as transceivers (e.g. FS-XXXXX DAC cables scraped as OSFP/QSFP28)
+  if (priceUsd < 1.5) {
+    return false;
+  }
+
  const anomalous = await isPriceAnomalous(params.transceiverId, priceUsd);
  if (anomalous) {
    return false; // Reject price outside form-factor bounds
@ -96,6 +124,7 @@ export async function upsertPriceObservation(params: {
       WHERE id = $1 AND (price_verified IS NULL OR price_verified = false OR ${isCompetitor ? "competitor_verified IS NULL OR competitor_verified = false" : "false"})`,
      [params.transceiverId]
    );
+    await checkAndSetFullyVerified(params.transceiverId);
    return false; // No change
  }

@ -131,9 +160,100 @@ export async function upsertPriceObservation(params: {
      [params.transceiverId]
    );
  }
+  await checkAndSetFullyVerified(params.transceiverId);
  return true; // New observation written
 }

+/**
+ * Upsert a stock observation with full warehouse breakdown (FS.com v2).
+ * Writes to stock_observations including DE-Lager, Global-Lager, Nachlieferung,
+ * units_sold, compatible_brands, price_net, and product_url columns.
+ * Returns true only when the data has changed since the last observation.
+ */
+export async function upsertStockObservation(params: {
+  transceiverId: string;
+  sourceVendorId: string;
+  stockLevel: string;
+  quantityAvailable?: number;
+  warehouseDeQty?: number;
+  warehouseDeDeliveryDate?: string | null;
+  warehouseGlobalQty?: number;
+  warehouseGlobalDeliveryDate?: string | null;
+  backorderQty?: number;
+  backorderEstimatedDate?: string | null;
+  unitsSold?: number;
+  compatibleBrands?: string[];
+  priceNet?: number;
+  productUrl?: string;
+}): Promise<boolean> {
+  // Skip if there is genuinely no warehouse data at all
+  if (
+    params.warehouseDeQty === undefined &&
+    params.warehouseGlobalQty === undefined &&
+    params.quantityAvailable === undefined
+  ) {
+    return false;
+  }
+
+  // Compare against the last observation to avoid duplicate writes
+  const lastObs = await pool.query(
+    `SELECT warehouse_de_qty, warehouse_global_qty, backorder_qty, units_sold
+     FROM stock_observations
+     WHERE transceiver_id = $1 AND source_vendor_id = $2
+     ORDER BY time DESC LIMIT 1`,
+    [params.transceiverId, params.sourceVendorId]
+  );
+
+  if (lastObs.rows.length > 0) {
+    const r = lastObs.rows[0];
+    const unchanged =
+      (r.warehouse_de_qty     ?? null) === (params.warehouseDeQty     ?? null) &&
+      (r.warehouse_global_qty ?? null) === (params.warehouseGlobalQty ?? null) &&
+      (r.backorder_qty        ?? null) === (params.backorderQty        ?? null) &&
+      (r.units_sold           ?? null) === (params.unitsSold           ?? null);
+    if (unchanged) return false;
+  }
+
+  const inStock =
+    ((params.warehouseDeQty ?? 0) + (params.warehouseGlobalQty ?? 0)) > 0;
+
+  await pool.query(
+    `INSERT INTO stock_observations (
+       time, transceiver_id, source_vendor_id,
+       in_stock, quantity_available,
+       warehouse_de_qty, warehouse_de_delivery_date,
+       warehouse_global_qty, warehouse_global_delivery_date,
+       backorder_qty, backorder_estimated_date,
+       units_sold, compatible_brands, price_net, product_url
+     ) VALUES (
+       NOW(), $1, $2,
+       $3, $4,
+       $5, $6::date,
+       $7, $8::date,
+       $9, $10::date,
+       $11, $12, $13, $14
+     )`,
+    [
+      params.transceiverId,
+      params.sourceVendorId,
+      inStock,
+      params.quantityAvailable ?? null,
+      params.warehouseDeQty ?? null,
+      params.warehouseDeDeliveryDate ?? null,
+      params.warehouseGlobalQty ?? null,
+      params.warehouseGlobalDeliveryDate ?? null,
+      params.backorderQty ?? null,
+      params.backorderEstimatedDate ?? null,
+      params.unitsSold ?? null,
+      params.compatibleBrands?.length ? params.compatibleBrands : null,
+      params.priceNet ?? null,
+      params.productUrl ?? null,
+    ]
+  );
+
+  return true;
+}
+
 export async function findOrCreateScrapedTransceiver(params: {
  partNumber: string;
  vendorId: string;
@ -160,6 +280,7 @@ export async function findOrCreateScrapedTransceiver(params: {
        `UPDATE transceivers SET image_url = $1, image_verified = true, updated_at = NOW() WHERE id = $2`,
        [params.imageUrl, existing.rows[0].id]
      );
+      await checkAndSetFullyVerified(existing.rows[0].id);
    }
    return existing.rows[0].id;
  }
--- a/run-fs-scraper-mac.sh
+++ b/run-fs-scraper-mac.sh
@ -0,0 +1,66 @@
+#!/bin/bash
+# FS.com Scraper — Mac-side runner
+# Runs from this Mac (residential IP) so FS.com isn't blocked.
+# Opens SSH tunnel to Erik's DB → runs scraper → closes tunnel.
+#
+# Schedule: launchd at 02:00, 10:00, 18:00 daily
+# Log: ~/Library/Logs/tip-fs-scraper.log
+
+set -euo pipefail
+
+LOG="$HOME/Library/Logs/tip-fs-scraper.log"
+REPO="/Users/renefichtmueller/Desktop/Claude Code/github-repos/transceiver-db"
+NODE="/opt/homebrew/bin/node"
+NPX="/opt/homebrew/bin/npx"
+TUNNEL_PID_FILE="/tmp/tip-db-tunnel.pid"
+DB_LOCAL_PORT=5433
+
+log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$LOG"; }
+
+# ── Open SSH tunnel if not already running ────────────────────────────────────
+open_tunnel() {
+  if [ -f "$TUNNEL_PID_FILE" ]; then
+    PID=$(cat "$TUNNEL_PID_FILE")
+    if kill -0 "$PID" 2>/dev/null; then
+      log "Tunnel already running (PID $PID)"
+      return 0
+    fi
+  fi
+  log "Opening SSH tunnel → Erik PostgreSQL on port $DB_LOCAL_PORT…"
+  ssh -N -f -L "${DB_LOCAL_PORT}:localhost:${DB_LOCAL_PORT}" erik
+  # -f forks to background, no PID tracking needed — use pkill to close
+  log "Tunnel opened"
+  sleep 2  # Give the tunnel a moment to establish
+}
+
+close_tunnel() {
+  log "Closing SSH tunnel…"
+  pkill -f "ssh -N -f -L ${DB_LOCAL_PORT}:localhost:${DB_LOCAL_PORT}" 2>/dev/null || true
+  rm -f "$TUNNEL_PID_FILE"
+}
+
+# ── Main ──────────────────────────────────────────────────────────────────────
+mkdir -p "$(dirname "$LOG")"
+log "=== FS.com Mac Scraper starting ==="
+
+# Only close tunnel if we opened it (not if one was already running)
+OPENED_TUNNEL=0
+if ! pgrep -f "ssh -N.*${DB_LOCAL_PORT}:localhost" >/dev/null 2>&1; then
+  open_tunnel
+  OPENED_TUNNEL=1
+  trap close_tunnel EXIT
+fi
+
+cd "$REPO"
+
+export POSTGRES_HOST=localhost
+export POSTGRES_PORT=$DB_LOCAL_PORT
+export POSTGRES_DB=transceiver_db
+export POSTGRES_USER=tip
+export POSTGRES_PASSWORD=***REDACTED***
+export NODE_ENV=production
+
+log "Running fs-com scraper via tsx…"
+"$NPX" tsx packages/scraper/src/scrapers/fs-com.ts 2>&1 | tee -a "$LOG"
+
+log "=== FS.com Mac Scraper complete ==="
--- a/sql/034-blog-review-tag.sql
+++ b/sql/034-blog-review-tag.sql
@ -0,0 +1,7 @@
+-- Migration 034: Add review_tag column to blog_drafts for manual reviewed tracking
+-- Used by dashboard to let editor mark posts as reviewed before publishing
+
+ALTER TABLE blog_drafts
+  ADD COLUMN IF NOT EXISTS review_tag VARCHAR(32) DEFAULT NULL;
+
+COMMENT ON COLUMN blog_drafts.review_tag IS 'Manual review status tag — set to ''reviewed'' when editor has proofread the post, NULL otherwise';
--- a/sql/035-price-observations-is-anomalous.sql
+++ b/sql/035-price-observations-is-anomalous.sql
@ -0,0 +1,12 @@
+-- Migration 035: Add is_anomalous column to price_observations
+-- This column marks price entries as outliers/anomalous that should be excluded from display
+
+ALTER TABLE price_observations
+  ADD COLUMN IF NOT EXISTS is_anomalous BOOLEAN NOT NULL DEFAULT false;
+
+CREATE INDEX IF NOT EXISTS idx_price_obs_anomalous
+  ON price_observations (transceiver_id, is_anomalous)
+  WHERE is_anomalous = false;
+
+COMMENT ON COLUMN price_observations.is_anomalous IS
+  'True when this price is flagged as an outlier/anomaly and should be excluded from price displays and comparisons';
--- a/sql/036-transceiver-equivalences.sql
+++ b/sql/036-transceiver-equivalences.sql
@ -0,0 +1,44 @@
+-- Migration 036: Transceiver equivalences for competitor_verified matching
+-- Stores semantic equivalences between Flexoptix SKUs and competitor products
+-- matched by technical specs (form_factor + speed + reach + standard + fiber_type)
+
+CREATE TABLE IF NOT EXISTS transceiver_equivalences (
+  id               UUID    DEFAULT gen_random_uuid() PRIMARY KEY,
+  flexoptix_id     UUID    NOT NULL REFERENCES transceivers(id) ON DELETE CASCADE,
+  competitor_id    UUID    NOT NULL REFERENCES transceivers(id) ON DELETE CASCADE,
+  confidence       DECIMAL(4,3) NOT NULL CHECK (confidence BETWEEN 0 AND 1),
+  match_basis      TEXT[]  NOT NULL DEFAULT '{}',   -- ['standard_name','form_factor','speed_gbps','fiber_type','reach']
+  match_notes      TEXT,
+  status           VARCHAR(20) NOT NULL DEFAULT 'pending'
+                     CHECK (status IN ('pending','approved','rejected','auto_approved')),
+  reviewed_by      VARCHAR(200),
+  reviewed_at      TIMESTAMPTZ,
+  reject_reason    TEXT,
+  created_at       TIMESTAMPTZ DEFAULT NOW(),
+  updated_at       TIMESTAMPTZ DEFAULT NOW(),
+  UNIQUE (flexoptix_id, competitor_id)
+);
+
+CREATE INDEX IF NOT EXISTS idx_eq_flexoptix   ON transceiver_equivalences (flexoptix_id);
+CREATE INDEX IF NOT EXISTS idx_eq_competitor  ON transceiver_equivalences (competitor_id);
+CREATE INDEX IF NOT EXISTS idx_eq_status      ON transceiver_equivalences (status, confidence DESC);
+CREATE INDEX IF NOT EXISTS idx_eq_pending     ON transceiver_equivalences (flexoptix_id) WHERE status = 'pending';
+
+-- Auto-update updated_at
+CREATE OR REPLACE FUNCTION update_equivalences_updated_at()
+RETURNS TRIGGER AS $$
+BEGIN
+  NEW.updated_at = NOW();
+  RETURN NEW;
+END;
+$$ LANGUAGE plpgsql;
+
+DROP TRIGGER IF EXISTS trg_eq_updated_at ON transceiver_equivalences;
+CREATE TRIGGER trg_eq_updated_at
+  BEFORE UPDATE ON transceiver_equivalences
+  FOR EACH ROW EXECUTE FUNCTION update_equivalences_updated_at();
+
+COMMENT ON TABLE transceiver_equivalences IS
+  'Semantic equivalences between Flexoptix SKUs and competitor products, '
+  'matched by technical specification overlap. Used to set competitor_verified=true '
+  'on Flexoptix transceivers that have no exact SKU match at competitors.';