actor: codex-fs-flex-price-image-revalidation

profile: erik-safe

wave: manual-targeted-revalidation

summary: Revalidated Flexoptix and FS.com price observations and product images after discovering price_observations were not marked verified and FS.com image selectors had gone stale.

user_request: Revalidate all Flexoptix and FS.com prices and check images without overloading Erik.

code_changes: Set price_observations.is_verified=true and verified_at on new and fresh unchanged observations.

code_changes: Always refresh image_verified_at/image_verified_url when a scraper sees a current product image.

code_changes: Add FS.com force revalidate controls and targeted missing-image mode.

code_changes: Update FS.com image selector to prefer resource.fs.com big_img/big_img_m product images and ignore logos/default/general images.

runtime_policy: Run sequentially with maxConcurrency=1 and nice -n 10 on Erik because CT115 was not reachable from this session; monitor load and Crawlee overload signals.

daemon: tip-scraper-daemon restarted and online

safety_notes: No external AI was used for TIP crawler planning or extraction feedback.

safety_notes: FS.com and Flexoptix runs were sequential and rate limited; Erik remained stable.

safety_notes: The dedicated TIP scraper CT was probed but not used because SSH did not respond quickly.


actor: codex-equivalence-auto-research

profile: erik-safe-db-only

wave: manual-queue-cleanup

vendor: all-equivalence-candidates

summary: Converted the manual equivalence review backlog into deterministic automated research decisions using existing crawled specs and prices. No heavy crawler wave was started.

user_request: Research all TIP product equivalences so they do not require manual validation.

code_changes: Do not let approve-all approve low-confidence matches; queue them for automated research.

code_changes: Add API endpoint to trigger equivalence research.

code_changes: Replace price-only re-research with strict technical evaluator for price recency, form factor, speed, fiber type, wavelength, reach and confidence.

runtime_policy: Use existing DB evidence only for cleanup. Keep Erik safe by avoiding crawler waves; schedule confirmed matches for future 30-day rechecks.

safety_notes: No new crawler wave was started on Erik.

safety_notes: Next enrichment should target rejected missing-evidence products on Proxmox or Pi workers with TIPLLM-only extraction feedback.


actor: codex-flexoptix-price-truth-recheck

profile: erik-safe-light-catalog

wave: manual-source-verification

vendor: Flexoptix

summary: Rechecked Flexoptix price completeness and fixed audit timestamps for unchanged prices. Active catalog prices are freshly source-confirmed, but the full historical Flexoptix table is not complete.

user_request: Confirm whether all Flexoptix prices/images/information are present and whether prices are 100% correct.

code_changes: Refresh price_observations.verified_at with NOW() when unchanged prices are observed again.

code_changes: Refresh transceivers.price_verified_at with NOW() when unchanged prices are observed again.

runtime_policy: Run only the light Flexoptix catalog scraper on Erik with nice -n 10; no Playwright or broad crawler wave.

safety_notes: Do not claim all 744 Flexoptix rows are complete; distinguish active catalog coverage from historical/extra rows.

safety_notes: Do not claim permanent 100% price truth; expose last-source-verified timestamp because vendor prices can change or vary by account/currency/VAT/session.


actor: codex-flexoptix-completeness-gap

profile: tipllm-training

wave: operator-feedback

summary: Operator asked whether all Flexoptix products are complete and 100% verified. Correct answer is no; TIPLLM must separate active source-confirmed catalog coverage from total DB completeness.

user_question: Sind alle Flexoptix Produkte komplett und 100% verified?

answer_policy: Never answer yes unless total rows equal complete rows and each verification signal has a fresh source timestamp.

truth_distinction: Active Flexoptix catalog products can be source-confirmed separately from historical/extra Flexoptix DB rows.

next_robot_action: Build/run a targeted Flexoptix detail enricher for rows with product_page_url and missing image/details/price signals; keep Erik light and use per-page rate limits.

correct_user_facing_answer: No. 615 active Flexoptix catalog prices are freshly source-confirmed, but only 496 of 744 Flexoptix rows have price + image + details verified; therefore all Flexoptix products are not complete and not 100% verified.

training_value: TIPLLM should prefer precise counters and last-source-verified timestamps over absolute completeness claims.

safety_notes: Use TIPLLM only for crawler planning/extraction feedback.

safety_notes: Do not conflate active catalog scrape result with total product table completeness.

safety_notes: Do not claim permanent 100% correctness for live vendor prices.


actor: codex-flexoptix-completion-push

profile: erik-safe-targeted-detail

wave: manual-targeted-enrichment

summary: Improved Flexoptix completeness by enhancing catalog parsing and adding a targeted detail-page verifier. Fully verified increased from 496 to 620 while preserving the truth that not all historical/filter rows are concrete products.

user_request: Feuer frei: make Flexoptix products complete and keep info in TIPLLM training pool.

code_changes: Catalog parser now handles generic reach values like 50 m, 1,000 m, ranges and multiple lambda wavelength values.

code_changes: Catalog parser now passes productUrl into findOrCreateScrapedTransceiver.

code_changes: Added targeted flexoptix-detail-pages verifier for real .html product pages with missing signals.

runtime_policy: Run only light Flexoptix-specific catalog/detail jobs on Erik with nice and per-page throttling; no broad crawler wave; no external AI.

evidence: Flexoptix title contained 1,000 m and Multimode

fiber_type: MMF

safety_notes: TIPLLM should not call the full set 100% complete because 118 remaining rows are filter/category/generic/non-product URLs.

safety_notes: Public TIP HTTPS still responded after the run, but SSH to Erik returned connection refused; no further live commands were started after that.


actor: codex-flexoptix-final-gap-check

profile: source-evidence-only

wave: post-enrichment-closure

summary: Final Flexoptix gap check classified the remaining accessory rows and refused to mark placeholder-only product images as verified.

accessory_policy: FLEXBOX rows are products/accessories, not optical reach products; set category Accessory and reach_label N/A when source page is the FLEXBOX page.

image_policy: Do not mark image_verified when Flexoptix source only exposes placeholder-flexoptix.jpg.

truth_policy: Report 622 of 626 product-like rows with price+image+details, 620 fully_verified, and 118 non-product/filter/generic rows outside complete product verification.

remaining_product_like_gap: 4 rows lack verified images because the source page image is a Flexoptix placeholder.

safety_notes: No placeholder images were accepted as verified.

safety_notes: Use exact counters instead of claiming 100%.


profile: erik-safe-playwright-db-detail-only

summary: Ran targeted FS.com detail-page revalidation against existing DB product URLs with missing verification signals. Batch used Playwright concurrency 1 and no category discovery. Details and fully_verified improved significantly.

user_request: Now Fiberstore/FS.com; all crawler/scraper/robot work must be written to TIPLLM training pool.

code_changes: Added FS_DB_DETAIL_ONLY mode to revalidate existing FS.COM product URLs directly without listing/category crawl.

code_changes: FS scraper now writes productUrl into transceivers.product_page_url.

code_changes: FS scraper now infers fiber_type from product name, part number, and specs when FS spec table lacks an explicit fiber key.

code_changes: FS detail source URL is now stored as the real product URL instead of the literal string fs.com.

runtime_policy: Run small FS.com Playwright batch on Erik with maxConcurrency=1, maxRequestsPerMinute=10, nice -n 10, and detail cap 80.

retry_histogram: 74 first try, 3 one retry, 3 two retries

safety_notes: Batch was intentionally capped to 80 and ran concurrency 1 to protect Erik.

safety_notes: FS.com still has many missing fibers/images; continue in small batches or move heavier waves to residential/Proxmox/Pi worker.


summary: Ran a second targeted FS.com DB-detail-only batch. It stayed stable at Playwright concurrency 1, completed 80 requests with 0 final failures, and materially improved product detail and full verification counters.

user_request: Finish TIP today; FS.com/Fiberstore prices, images and product information should be researched deeply enough to avoid manual validation.

runtime_policy: Continue with capped DB-detail-only batches because batch 1 was productive. Keep Erik safe with maxConcurrency=1, nice -n 10, no category discovery, and small batch caps.

retry_policy: Allow Crawlee retries for transient FS.com network changes; do not raise concurrency to compensate.

retry_histogram: 59 first try, 14 one retry, 5 two retries, 2 three retries

notable_warning: One target page returned no useful text; transient ERR_NETWORK_CHANGED retries were reclaimed and completed without final failure.

safety_notes: Erik stayed stable; crawler stats showed no memory, CPU or event-loop overload.

safety_notes: Next best action is another small DB-detail-only batch for remaining HTML product-like rows, then handle no-url rows separately by source discovery/classification.


summary: Ran a third targeted FS.com DB-detail-only batch. It completed 90 requests with 0 final failures, added fresh price and stock observations, and moved most remaining HTML product URLs into verified detail coverage.

user_request: Finish TIP today with perfect transceiver, competitor and switch database; continue FS.com/Fiberstore verification safely.

runtime_policy: Use another capped DB-detail-only batch because remaining product URLs still produced verified evidence. Increase cap from 80 to 90 but keep Playwright concurrency 1, nice -n 10 and no category discovery.

truth_policy: Do not mark pages with no useful text as details verified. Record no-text cases for retry/source-review instead.

retry_histogram: 70 first try, 13 one retry, 5 two retries, 2 three retries

notable_warning: One FS.com page returned no useful text and was not counted as scraped.

status: healthy

load_status: ok

safety_notes: No external AI was used for TIP crawler planning or extraction feedback.

safety_notes: Erik remained stable; TIP health stayed OK after the run.

safety_notes: Next best action is a smaller DB-detail-only closure batch for the remaining FS.com HTML product URLs, then a separate source-discovery task for no-url rows.


wave: manual-targeted-enrichment-closure

summary: Ran a smaller FS.com closure batch after the productive HTML detail waves. It completed all 42 queued pages cleanly, but details_verified did not increase, showing that the DB-detail-only approach is now exhausted for the remaining gaps.

user_request: Finish FS.com/Fiberstore as far as possible without manual validation and keep all robot experience in TIPLLM training pool.

runtime_policy: Run one smaller closure batch because only 42 eligible DB product URLs remained. Keep concurrency 1, nice -n 10, no category discovery.

stop_condition: If a clean closure batch produces no details_verified movement, stop repeating the same Playwright detail-only job and switch to source-discovery/classification or parser work for the remaining gaps.

retry_histogram: 42 first try, 0 retries

safety_notes: Erik remained healthy after the closure batch.

safety_notes: Do not claim FS.com is complete: 258 of 299 HTML product-like rows are complete; 76 no-url rows need source discovery; 4 category rows need classification.


actor: codex-fscom-url-discovery-and-reconcile

profile: erik-safe-playwright-url-discovery

wave: manual-targeted-url-discovery

summary: Added and ran FS.com URL-discovery mode for historical FS-<id> rows without product URLs, then reran DB-detail reconciliation with H1-derived detail evidence. This converted almost all FS.com no-url rows into source-linked product rows and substantially increased image/detail coverage.

user_request: Continue and finish FS.com/Fiberstore; all crawler/scraper/robot work must go to TIPLLM training pool. Operator also asked whether Crawlee repos can help; use that evaluation to improve safe crawler architecture.

code_changes: Added FS_URL_DISCOVERY_ONLY mode that maps existing FS-<numeric-id> rows without product_page_url to https://www.fs.com/de/products/<id>.html and writes evidence onto the existing DB row instead of creating duplicates.

code_changes: DB_DETAIL_ONLY rows now carry targetTransceiverId so targeted reconciliation updates the original row.

code_changes: Image verification is marked for target rows when a current FS.com product image is seen.

code_changes: Detail verification now accepts deterministic source evidence from H1/part/spec text, not only classical spec tables, when fiber/reach/wavelength/connector can be inferred.

runtime_policy: Run URL discovery in safe bounded batches on Erik with Playwright concurrency 1, nice -n 10, no category crawl, and post-run DB measurement.

safety_notes: Erik remained healthy; Crawlee AutoscaledPool stayed at concurrency 1 and reported no CPU/memory/event-loop overload.

safety_notes: Do not claim FS.com is complete: 371 of 373 HTML product-like rows are complete, but 2 no-url rows, 4 category rows, and 9 image/fiber gaps remain.


actor: codex-crawlee-binding

profile: crawlee-core-plus-python-side-worker

wave: runtime-integration

vendor: all

summary: Installed and bound Crawlee runtime work into TIP. TypeScript Crawlee remains the production core; Crawlee-Python was added as an optional isolated evidence worker for Pi/Proxmox/residential nodes.

user_request: Install, use and bind apify/crawlee and crawlee-python for TIP.

code_changes: Added package scripts for FS.com DB-detail and URL-discovery Crawlee modes.

code_changes: Added packages/crawlee-python optional worker package.

code_changes: Added package-local .gitignore so Python build artifacts are not committed.

runtime_policy: Production writes remain in the TypeScript TIP scraper. Crawlee-Python writes JSONL evidence only and must not directly mutate TIP DB.

tip_commits: 60531b6 feat: add crawlee python worker integration

tip_commits: 49f0871 chore: ignore crawlee python build artifacts

result: JSONL evidence output succeeded

safety_notes: Move heavier experiments to Pi/Proxmox/residential workers.


wave: manual-immediate-full-active-research

vendor: all-active-equivalence-matches

summary: Revalidated all active approved/auto-approved TIP equivalence matches immediately with deterministic 1:1 DB evidence rules. No crawler wave was started.

user_request: Verify and validate everything open in TIP immediately and check whether products truly match 1:1.

criteria:
recent competitor price evidence
form factor exact
speed exact
fiber type exact
reach ratio >= 0.85
primary wavelength within 15nm

runtime_policy: Mark all active matches due immediately and process with existing PgBoss maintenance:re-research-equivalences worker. Keep work DB-only to protect Erik.

truth_policy: Manual equivalence queue is empty and active matches were just rechecked, but not every product row in TIP is fully complete; remaining work is vendor-specific product source enrichment.

safety_notes: No broad browser crawler wave was started.

safety_notes: Continue targeted vendor crawlers or offload heavy discovery to Pi/Proxmox workers.


actor: codex-targeted-vendor-verification

profile: erik-safe-fetch-db-detail

wave: manual-vendor-verification

vendor: NADDOD+GAO Tek+Ascent Optics+Eoptolink

summary: Improved vendor verification by making existing scrapers write product URLs, image evidence and detail evidence, then running low-load static/DB-detail passes. No broad browser crawler wave was used.

user_request: Continue until everything is finished, but keep Erik safe and write crawler/scraper/robot learnings to TIPLLM training pool.

code_changes: NADDOD DB_DETAIL_ONLY mode verifies existing DB rows with product_page_url and missing image/details, instead of blind sitemap rotation.

code_changes: NADDOD extracts og:image, page evidence, AOC/DAC lengths, DR/FR/SR/VR/XDR patterns and writes details to the target row.

code_changes: GAO Tek writes product_page_url and image evidence.

code_changes: Ascent Optics writes product_page_url and table image evidence.

code_changes: Eoptolink writes product_page_url, image, reach/wavelength evidence and avoids over-broad form-factor parsing by preferring title/slug evidence.

runtime_policy: Run static/fetch and DB-detail passes with nice -n 10 and existing per-page delays. No Playwright wave on Erik.

truth_policy: NADDOD is much improved but not 100% complete. OEM catalogs without retail prices cannot honestly become fully_verified unless price evidence exists.


profile: source-truth-correction

wave: operator-spotted-variant-fix

summary: Corrected FS.com 1.6T OSFP family handling after the operator spotted separate 500m DR8 and 2km 2FR4 variants. The 500m DR8 variant maps to Flexoptix O.1316T.C.05.M; the 2km FR4 variant remains a separate FS.com product.

user_finding: FS.com page clearly has OSFP-DR8-1.6T-FL 500m and OSFP-2FR4-1.6T-FL 2km. Flexoptix O.1316T.C.05.M is the 500m version; FR4 must also be in the DB.

db_corrections: Set FS OSFP-DR8-1.6T-FL to speed 1.6T, reach 500m/500, SMF, wavelength 1310, standard_name 1.6T OSFP DR8.

db_corrections: Set FS OSFP-2FR4-1.6T-FL to speed 1.6T, reach 2km/2000, SMF, wavelength 1310, standard_name 1.6T OSFP 2FR4.

db_corrections: Set Flexoptix O.1316T.C.05.M standard_name 1.6T OSFP DR8.

db_corrections: Approve only O.1316T.C.05.M to OSFP-DR8-1.6T-FL with confidence 0.913; keep 2FR4 separate.

scraper_changes: FS.com scraper recognizes 1,6T, 1.6T and 1600G as 1.6T/1600.

scraper_changes: FS.com scraper converts reach labels such as 2km into reach_meters=2000.

scraper_changes: FS.com scraper updates stale speed labels when numeric source speed matches the row.

fiber_type: SMF

standard_name: 1.6T OSFP DR8

standard_name: 1.6T OSFP 2FR4

equivalence: O.1316T.C.05.M <-> OSFP-DR8-1.6T-FL approved; no FR4-to-500m collapse

remote_build: pnpm -C packages/scraper build passed on Erik

truth_policy: Variant selectors that change reach/protocol/model are separate products. A 500m DR8 product must never be matched to a 2km 2FR4 product.

safety_notes: Only DB correction and scraper parser hardening were done.