rene/transceiver-db
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
transceiver-db/docs/FLEXOPTIX_CATALOG_IMPORT.md
Rene Fichtmueller a20094755d feat(scraper): Flexoptix REST API sync robot + scheduler integration 
Replaces the GraphQL/search-based Flexoptix scraper with a proper
Magento 2 REST API integration that delivers authoritative SKUs,
prices, stock levels and compatibility data.

New files:
- packages/scraper/src/robots/flexoptix-api-sync.ts
  Self-contained robot: auth → paginated fetch → normalize → DB write.
  Reads FLEXOPTIX_API_BASE_URL / _USERNAME / _PASSWORD from env.
  Returns { fetched, normalized, skipped, priceWrites, stockWrites }.
  No file intermediary — in-memory pipeline.

- scripts/import-flexoptix-catalog.ts
  One-shot CLI importer for the Pulso-generated JSONL (Codex handover).

- docs/FLEXOPTIX_CATALOG_IMPORT.md
  Runbook for manual import + per-SKU specifications enrichment.

Scheduler changes:
- Added sync:flexoptix-catalog queue + work() handler
- Scheduled every 2h at 0 */2 * * * (same cadence as legacy job)
- scrape:pricing:flexoptix kept as legacy GraphQL fallback

Also includes Codex-generated additions from this sprint:
- audiocodes-oem scraper, seed-batch35/36/37, db.ts improvements,
  sql/102 verification reconcile, README + package.json updates
2026-05-13 16:36:33 +02:00

1.3 KiB
Raw Blame History

Flexoptix Catalog Import

TIP can ingest the normalized Flexoptix shop catalog produced by Magatama/Pulso. This keeps prices, product URLs, stock, and product attributes live-data driven instead of baking volatile values into LLM training pairs.

Producer

Run in the Magatama repo:

pnpm pulso:shop:sync:dry-run
pnpm pulso:shop:sync -- --fixture ./shop-export.json
FLEXOPTIX_API_BASE_URL=... FLEXOPTIX_API_TOKEN=... pnpm pulso:shop:sync

The producer writes:

  • data/pulso/shop-products.normalized.jsonl
  • data/pulso/shop-catalog.normalized.json
  • data/pulso/shop-snapshot.manifest.json

TIP Import

Safe validation:

npm run flexoptix:catalog:import -- --dry-run

Import into the TIP database:

TIP_DB_PASS=... npm run flexoptix:catalog:import

Override the input path if needed:

npm run flexoptix:catalog:import -- --input /path/shop-products.normalized.jsonl --dry-run

Safety Rules

  • Rows without SKU, title, form factor, speed, or reach are skipped for DB import.
  • Prices and stock are written only when present in live/API data.
  • No product URL, SKU, price, stock, delivery time, or compatibility is invented.
  • The importer uses existing TIP helpers for transceiver upsert, price observations, and stock observations.