rene/transceiver-db
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
233 Commits 5 Branches 9 Tags
Commit Graph

233 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Rene Fichtmueller
b2f3a4c450 feat: add Anthropic Claude provider to blog LLM client 
- Auto-routes to Claude API when BLOG_LLM_PROVIDER=anthropic + ANTHROPIC_API_KEY set
- Fallback to Ollama queue when key not present
- Add rate-limit retry (429 → 10s backoff) for Claude API
- Add STEP_TECHNICAL_SANITY, STEP_SELF_HEAL, STEP_TITLE_CONTRACT_CHECK prompts
- Fix STEP_LINKEDIN_POST angle-specific hooks, remove Gold Reference repetition
 2026-04-06 00:21:48 +02:00
Rene Fichtmueller
b4d9bfc9d1 fix(blog): hard story blacklist in STEP4 + LinkedIn — ban 2AM/dirty connector/lab-vs-prod stories 
10 specific story patterns banned directly in draft prompt.
LinkedIn banned hooks: 'Everything looks fine', 'CRC creeping', 'Same optics same setup'.
Title Contract now injected into STEP4 as binding constraint.
 2026-04-05 23:55:56 +02:00
Rene Fichtmueller
65159bd57d feat(blog): Post to Ghost + LinkedIn buttons in dashboard 
- 'Post on blog.fichtmueller.org' → publishes via Ghost Admin API
- 'Post on LinkedIn' → modal with text + copy + open LinkedIn
- Ghost integration: TIP Blog Engine (JWT auth, mobiledoc format)
 2026-04-05 23:33:58 +02:00
Rene Fichtmueller
6c818c79b5 feat(blog): Title Contract + Technical Sanity Check + Self-Heal + angle-aware LinkedIn generator 
Pipeline now has 21 steps:
- STEP0: Title Contract binds LLM to headline promise
- STEP19: Technical Sanity Check (optical engineering accuracy)
- STEP20: Self-Heal (auto-fix technical errors preserving tone)
- STEP21: Title Contract Verification (final gate check)
- LinkedIn generator is now angle-aware (no more default Physical Layer hook)
 2026-04-05 23:11:16 +02:00
Rene Fichtmueller
7c8f545c18 fix(blog): anti-repetition engine — 6 angle types, forbidden structures, existing article context injection 2026-04-05 22:47:15 +02:00
Rene Fichtmueller
957632c228 fix(blog): raise word target to 1200-1600, fix power-budget false positive in validateArticle 2026-04-05 20:49:22 +02:00
Rene Fichtmueller
c4b1a20992 fix(scraper): use false instead of null for image_verified on insert 2026-04-05 12:15:10 +02:00
Rene Fichtmueller
d3780ef4fc feat(dashboard): add data verification status section to overview tab 2026-04-05 12:11:23 +02:00
Rene Fichtmueller
a1223e8967 fix: make /api/hot-topics public — dashboard fetch has no auth token 2026-04-05 12:07:44 +02:00
Rene Fichtmueller
6d7b067ca9 fix: resolve merge conflict in index.ts + add untracked blog-sll, news, sql migration 2026-04-05 11:51:07 +02:00
root
161f045bc7 fix: mount blogSllRouter + scraperRouter — SLL and Crawler Intelligence routes were missing 2026-04-05 09:50:30 +00:00
Rene Fichtmueller
1e754625c5 feat(scraper): all pricing scrapers to 2h 24/7 — full competitor coverage, no gaps 2026-04-05 01:32:08 +02:00
Rene Fichtmueller
6d865cabb9 feat: 4th verification criterion (Competitor) + scraper frequency FS/10Gtek/ProLabs to 2h 2026-04-05 01:28:46 +02:00
Rene Fichtmueller
e496a91dd5 feat(blog): AEM/APM pipeline steps + SLL context builder + LinkedIn v2 prompts 2026-04-05 01:26:09 +02:00
Rene Fichtmueller
15f3ff5bef fix: include linkedin_post in GET /api/blog response for SLL matching 2026-04-05 01:24:52 +02:00
Rene Fichtmueller
d9f5fc253f fix(verification): 100% Verified Badge war dramatisch zu großzügig 
KERNPROBLEME BEHOBEN:
1. ATGBICS part_number = URL slug statt echte OEM-Nummer
   extractOemPartNumber() entfernt -r-compatible-transceiver-* Suffix
   + trailing Vendor-Namen (nokia, cisco, juniper, ...)
   Ergebnis: 3he16564aa-nokia-r-compatible-transceiver-... → 3HE16564AA

2. reach_label = '' (leer) wurde als details_verified akzeptiert
   IS NOT NULL erlaubt leere Strings → Fix: AND reach_label != ''

3. details_verified = true trotz garbled part_number
   Neue Kriterien: NOT ILIKE '%-compatible-transceiver%'
                   NOT ILIKE '%-r-compatible%'

4. data_confidence Werte falsch in Funktion ('scraped_unverified' etc)
   Echte Werte: low/medium/high/garbage → NOT IN ('garbage','unknown')

ERGEBNIS nach recompute_all_verification():
  fully_verified: 3.654 → 581 (Badge war 6x übertrieben)
  details_verified: inflated → 1.075 (korrekt)

ATGBICS Scraper:
  - extractOemPartNumber() für collection und product detail pages
  - detectReach() jetzt auch auf URL-slug (120km im slug → reach_label)

Price Anomaly Detection:
  - API: price_anomaly field wenn max/min ratio ≥ 10x
  - Dashboard: ⚠ Preisanomalie Banner mit Ratio + EUR Range

SQL 025: Part number cleanup (30 records), reach from slug (12 records)
 2026-04-04 15:41:57 +02:00
Rene Fichtmueller
a93fc8679e fix(scrapers): Flexoptix Catalog zeigt 0 records statt 963 
SCRAPERS list used 'flexoptix-catalog' as DB lookup key but vendors.slug
is 'flexoptix' — no match → 0 records shown.

Fix: added dbSlug override field to SCRAPERS entries; lookup now uses
dbSlug || name so flexoptix-catalog/vendors/supported all map to
the correct 'flexoptix' slug in sourceMap.
 2026-04-04 15:26:04 +02:00
Rene Fichtmueller
4a53f3c45d feat: blog engine v5 — Auto-Kill Layer, 16-step pipeline, longer content 
Upgrades FO Blog Pipeline from 14 to 16 steps:
- NEW Step 8d: Auto-Kill Layer v1.0 (10 systematic categories A-J)
- NEW Step 15: Auto-Kill Scoring (cleanliness, narrative, non-AI, relevance)
- Updated banned phrases from Gold-standard editorial feedback
- Soft Delete List for conditional phrases
- Auto-Kill categories: spec blocks, formulas, section leakage,
  generic transitions, repeated concepts, SKU mentions, false authority,
  over-explained basics, whitepaper tone, fake precision

Content length changes per user feedback:
- Blog target: 1,200-2,000 words (was 700-1,000) — thorough and detailed
- LinkedIn target: 2,000-2,800 chars (was 350-600) — use maximum length
- Reduction pass: 25-30% cut (was 15-25%) — remove weak, keep depth
 2026-04-04 11:02:45 +02:00
Rene Fichtmueller
4a501d4461 feat: blog engine v3 — 8-stage pipeline with Auto-Kill Layer 
Complete rewrite of blog prompts and pipeline based on editorial
Gold-standard feedback. Replaces 3-pass system with 8-stage pipeline:

1. Master generation (narrative voice, no spec dumps)
2. Narrative Control (kill visible structure, enforce flow)
3. Auto-Kill Layer (remove AI phrases, spec residue, repetition)
4. Reduction Engine (cut 40% — keep strongest ideas only)
5. Depth pass (add specifics where vague, no spec dumps)
6. Quality Control (hard delete list validation)
7. Procurement layer (optional, sales audience)
8. LinkedIn post generation (new)

Key changes:
- System prompt rewritten with Hard Delete List (29 banned phrases)
- Soft Delete List for conditional phrases
- Auto-Kill categories A-J (spec blocks, formulas, whitepaper tone, etc.)
- Master prompts enforce continuous narrative, no section headings
- Word count targets reduced (800-1200 instead of 1500+)
- Scoring pass added (cleanliness, narrative, non-AI feel, relevance)
- LinkedIn companion post auto-generated
- Context data injection reduced (fewer items, no dump instructions)
 2026-04-04 10:52:31 +02:00
Rene Fichtmueller
4a21967f41 feat(blog): Spec dump hard fail + Gold Standards 6 + LinkedIn v2 
- System prompt: SPEC DUMP ABSOLUTE HARD FAIL block (before FORMAT rules)
  TX/RX tables, multi-optic comparison blocks, repeated sections = hard fail
  Behavioral prose rule: "what actually happens" not "what the spec says"

- STEP9 QA: check 12a SPEC DUMP — removes datasheet blocks, flags
  duplicate sections (e.g. "fiber types" twice), spec-heavy intros

- Gold Standard 6: 400G/800G deep dive corrected (8.8→10/10)
  zero spec tables, pure behavioral narrative, 3 core ideas max,
  ending is reframe not checklist

- LinkedIn Gold Example 2: sharper short format (346 chars vs 700)
  reframe hook, short beats without bullet markers, no emoji, 4 hashtags

- STEP_LINKEDIN_POST: rewritten with new gold format
  optimal 350-600 chars, beat rhythm, no bullet markers, gold example inline

- WRONG PATTERNS: +7 new entries (spec dump, duplicate section,
  LinkedIn bullet list, LinkedIn "excited to share" hook, LinkedIn >800 chars)
 2026-04-04 09:32:01 +02:00
Rene Fichtmueller
4eddbfbc7c feat(blog): Reduction Engine v1.0 + LaTeX/connector hard fails 
- Replace STEP8b_REDUCTION with 5-pass Reduction Engine:
  Pass 1: Repetition Kill (one concept, one home)
  Pass 2: Tech Prune (LaTeX hard delete, SKU removal, formula prose replacement)
  Pass 3: Flow Rebuild (close gaps after cuts, no new content)
  Pass 4: Weight Correction (title/content alignment throughout)
  Pass 5: Humanization (rhythm variation, hedge removal, punch ending)
  Target: 700-1000 words (600-1300 range, warnings outside)

- System prompt + STEP9 QA: add hard fails for
  LaTeX formulas (\[...\], \frac{}, \text{} etc) — destroys blog flow
  DR4 connector error (DR4=MPO-12, not LC duplex; FR4=LC duplex)
  Title/content mismatch (title topic must be the spine, not just the intro)

- Gold Standard 5: market alert / pricing article template
  (correct title alignment, no LaTeX, DR4=MPO-12, ending on topic)

- WRONG PATTERNS extended with 4 new entries covering above failures

- blog.ts: step log messages updated to 11-14/14; word count
  output shows % reduction and range warning (>1300 or <600)
 2026-04-04 08:57:21 +02:00
Rene Fichtmueller
b180fe57ee ui: blog detail — separate blog article + linkedin post sections with copy buttons and char count badge 2026-04-04 08:35:33 +02:00
Rene Fichtmueller
15d02108aa chore: changelog — blog engine v5 + linkedin post 2026-04-04 2026-04-04 08:30:54 +02:00
Rene Fichtmueller
7db9fad108 feat: blog engine v5 — narrative control + linkedin post + min words fix 
- STEP4b_NARRATIVE_CONTROL: new pipeline step after draft; detects wrong
  narrative (technology blamed instead of processes), applies anti-FUD filter,
  reality reframe ("this becomes a problem when..."), Flexoptix voice check
- System prompt: NARRATIVE CONTROL RULE added as absolute rule #1
- Gold Standard 4: corrected "compatible vs OEM" article added as reference
- Minimum words: STEP4 raised from 1500 to 2500 words (final output was 750)
- Reduction pass: 25-35% → 15-25%, target 1500-2000 words final
- STEP_LINKEDIN_POST: generates LinkedIn post ≤2800 chars (hard limit 3000);
  stores in blog_drafts.linkedin_post + linkedin_char_count column
- Pipeline now 14 steps: v5-narrative-control
- Migration 024: linkedin_post + linkedin_char_count columns in blog_drafts
 2026-04-04 08:30:27 +02:00
Rene Fichtmueller
74dcc14e1e chore: changelog — proxy network geo/uptime fixes 2026-04-04 2026-04-04 08:15:58 +02:00
Rene Fichtmueller
5091a7b75f feat: proxy network — geo-lookup, uptime tracking, dedup fix 
- IP geo-lookup via ip-api.com on register/heartbeat (country_code, city)
- heartbeat_count column + uptime_pct computation on every heartbeat
- Deduplication: register returns existing token for same IP+port
- Heartbeat no longer overwrites registered IP (prevents IPv6 churn conflicts)
- Migration 023: heartbeat_count column + backfill existing nodes
 2026-04-04 08:15:32 +02:00
Rene Fichtmueller
5d53d3af6f docs: update changelog 2026-04-03/04 — scraper fixes, blog engine v4, proxy network, pg-boss fix 2026-04-04 07:58:29 +02:00
Rene Fichtmueller
5c9cf0e9b5 feat: blog engine v4 (reduction+style-lock passes) + flexoptix scraper fixes 
Blog engine (fo-blog-pipeline.ts):
- Add STEP8b_REDUCTION: cuts article 25-35%, removes repeated concepts
- Add STEP8c_STYLE_LOCK: enforces tone consistency, fixes scope/OPM confusion,
  removes inline SKUs from article flow
- Add Gold Standard 3 to calibration (Style B troubleshooting example 2026-04-04)
- Pipeline now 12 steps (was 10), version bumped to v4-reduction-stylelock

blog.ts:
- Wire STEP8b and STEP8c into pipeline between Kill-AI-Tone and QA Check
- Update progress tracking to 12 total steps
- Update pipeline_version to 'v4-reduction-stylelock'

flexoptix-catalog.ts:
- Fix contentHash call: pass object directly, not JSON.stringify(object)

db.ts:
- price_verified=true set in content_hash early-return path (no new observation)
- image_verified=true auto-set in findOrCreateScrapedTransceiver on INSERT/UPDATE
 2026-04-04 07:50:01 +02:00
Rene Fichtmueller
441d9721c9 fix: flexoptix catalog scraper — 1G SFP coverage + SKU suffix + pagination 
- Add 1G SFP search queries ("1G SFP", "SFP LX", "SFP SX", "SFP ZX") — were completely missing
- Strip vendor-compat suffix from SKU (S.1303.10.DG:Sx → S.1303.10.DG) to match existing records
- Remove 200-product cap, use full API pagination (page >= 50 limit only)
- Result: FLEXOPTIX 1G SFP coverage 50% → 97%, overall price coverage 62% → 88%
 2026-04-04 07:26:13 +02:00
Rene Fichtmueller
71d6c20b5e fix: auto-set image_verified and price_verified in db utils 
- findOrCreateScrapedTransceiver now sets image_verified=true when writing image_url
- upsertPriceObservation now sets price_verified=true on the transceiver after inserting price
- Both INSERT and UPDATE paths covered for image_verified sync
- Eliminates need for manual backfill after scraper runs
 2026-04-04 07:14:26 +02:00
Rene Fichtmueller
93dd7c2e23 fix: reduce pg-boss pool size to 4, add idle_in_transaction_session_timeout 
PostgreSQL max_connections was being exceeded (100/100).
- Limit pg-boss internal pool to 4 connections
- Added idle_in_transaction_session_timeout=30s to PostgreSQL config
- Already raised max_connections to 300 (container config)
System now stable at ~98/300 connections
 2026-04-03 21:15:35 +02:00
Rene Fichtmueller
edc9311d7b feat: add proxy network, image backfill, and scraper improvements 
- Add TIP Proxy Network (packages/proxy-agent): SOCKS5 proxy agent
  for residential IP bypass of CloudFront WAF blocks
- Add /api/proxy/* routes: node registration, heartbeat, load balancing
- Add image extraction to Flexoptix catalog scraper (GraphQL small_image)
- Add image extraction to Optcore scraper (Playwright gallery img)
- Fix Fluxlight price scraping (BigCommerce HTML structure: data-product-price-without-tax)
- Add SmartOptics scraper (8 DWDM/coherent products, og:image extraction)
- Fix findOrCreateScrapedTransceiver to update image_url for existing records
- Add image backfill script (backfill-images.ts): 178 Flexoptix images added
- Fix DB connection pool: max 5, idleTimeoutMillis 10s (was unlimited, caused >100 connections)
- Add proxy.ts utility for scraper proxy rotation
 2026-04-03 21:13:03 +02:00
Rene Fichtmueller
b612c5801d fix: remove orphaned floating text causing TypeScript build error in fo-blog-pipeline 
Dead code leftover from STEP4_MASTER_DRAFT rewrite was sitting outside
any template literal, causing compilation failure. Removed duplicate
CONTEXT DATA RULES block and orphaned `{{OUTLINE}}`/`{{CONTEXT_DATA}}`
placeholders that were not wrapped in a string.
 2026-04-03 00:51:59 +02:00
Rene Fichtmueller
4e910db090 fix(blog): complete pipeline rewrite — eliminate sections/bullets, fix DR4 wavelength, fix scope description 
Core changes:
- HARD RULES rewritten: zero tolerance for ## headers, #### Scenario: patterns, bullet sections
- Gold article added as reference standard in STEP4_MASTER_DRAFT
- MANDATORY SECTIONS removed — replaced with continuous prose requirement
- STEP3_OUTLINE: now a flow plan (3-4 beats), not a section list
- STEP5_REALITY_INJECTION: no longer adds 'What Breaks' sections — injects into prose
- STEP9_QA_CHECK: format violations now primary HARD FAIL, above content checks
- DR4 wavelength fix: 1310nm = 0.35 dB/km (not 1550nm = 0.22 dB/km)
- Scope description fix: visual inspection tool ≠ loss measurement device
- Invented firmware version numbers now explicit HARD FAIL
 2026-04-03 00:43:14 +02:00
Rene Fichtmueller
3226117733 feat: NOG conference talks scraper + hot topics integration 
NOG Talks Scraper (packages/scraper/src/scrapers/nog-talks.ts):
- Crawls DENOG (15-17), NANOG (91-93), RIPE (87-89), ENOG, NLNOG, Euro-IX
- Relevance scoring: optical keywords (+3pts each), network keywords (+1pt)
  Only talks scoring ≥2 stored, high-relevance (≥6) also to market_intelligence
- CtxEvent cross-DB bridge: when ctxmeet DB has ConferenceTalk rows,
  pulls directly via dblink (same Postgres instance, no network hop)
- Runs weekly Monday 06:00 UTC (pg-boss schedule)
- Output: news_articles (source='NOG Talks: EVENT') + market_intelligence

Hot Topics (packages/api/src/routes/hot-topics.ts):
- SOURCE 3c: NOG talk clusters displayed as conference topics in hot list
  Grouped by event (DENOG15, NANOG93...) with speaker + abstract preview
  Filtered: source LIKE 'NOG Talks:%' AND relevance > 0.4 AND < 6 months
- Limit raised to 20 topics (was 15)
- Added nog_talks to sources metadata

Scheduler & Pi fleet:
- scrape:nog-talks queue registered in scheduler.ts + index-pi.ts
- Weekly cron: Monday 06:00 UTC (every Pi can handle it independently)
- First job triggered immediately
 2026-04-02 22:38:00 +02:00
Rene Fichtmueller
48cb41b27e feat: hot topics now uses market_intelligence + LLM queue reset 
Hot Topics:
- SOURCE 3b: market_intelligence table (15 items, 0.6+ relevance)
  with urgency mapping per intel_type + buy signal angles
- Fix news_articles: url → source_url (correct column name)
- Fix template literals: ${year} in string literals → backticks
- Increase limit: 6 → 15 topics returned
- Lower news cluster threshold: 2 → 1 article to form topic
- More research topics per day: 2 → 3
- More evergreen topics per day: 3 → 4
- Result: 24 total topics, 15 shown (was 8 total, 6 shown)

LLM Queue:
- Add resetOllamaQueue() export + auto-reset after 15min stall
- Add getQueueDepth() for monitoring
- New endpoints: GET /api/blog/llm/status, POST /api/blog/llm/reset-queue
 2026-04-02 22:23:21 +02:00
Rene Fichtmueller
06d8b650c4 fix: loadCrawlerStatus missing token — add var token = localStorage.getItem 
token variable was undefined in loadCrawlerStatus() scope (only declared
inside IIFE auth guard, not globally). All API calls silently failed with
401. Fix: read token from localStorage at start of function, consistent
with getAuthHeaders() pattern used in all other load functions.
 2026-04-02 16:00:27 +02:00
Rene Fichtmueller
41266fa0d4 fix: scrapers status endpoint — correct table and column names 
- Replace vendor_slug → vendors.slug via JOIN (column doesn't exist on transceivers)
- Replace competitor_prices + ebay_listings → price_observations (correct table)
- Replace market_intelligence → knowledge_base in status DB stats (table exists, 73 rows)
- price_observations columns: transceiver_id, price, time (not part_number/price_eur/scraped_at)
- Result: 5602 transceivers, 359 vendors, 465 switches, 4252 prices, 73 KB entries all visible
 2026-04-02 15:24:18 +02:00
Rene Fichtmueller
238ee8c0a4 fix: crawler intelligence hot topics now reads market_intelligence table 
- Fix trend_score → relevance_score (correct column name in news_articles)
- Hot Topics source: market_intelligence table (not news_articles)
  → 15 items immediately available (capex_cycle, supply_chain, standards, etc.)
- KB summary: grouped by intel_type with count + top_relevance + latest date
- knowledge_base table ref → market_intelligence in /api/health status query
- Hot topics cards: intel_type badge + buy_signal_implication + score %
- Dashboard KB table: Intel Type / Items / Top Relevance / Latest columns
 2026-04-02 15:22:09 +02:00
Rene Fichtmueller
a893817dc7 fix: smart tooltips that flip above/below based on viewport position 
Replace static CSS ::after tooltips with JS-powered smart tooltips.
Tooltips now detect available space above/below and flip accordingly,
and clamp horizontally to viewport bounds. Hide on scroll.
 2026-04-02 13:03:51 +02:00
Rene Fichtmueller
59a3e97207 fix: correct import paths in index-pi.ts (fs-com, tenGtek, utils/forecast-engine) 2026-04-02 09:36:51 +02:00
Rene Fichtmueller
398dfcced8 feat: add index-pi.ts with all 44 workers for Pi fleet scraper nodes 
Complete Pi scraper entry point covering all pricing, catalog, compat,
intelligence and prediction signal scrapers. Includes 5 new form-factor
coverage scrapers (comms-express, router-switch, multimode-inc,
optictransceiver, wiitek). Erik runs only API+DB, all scraping on Pis.
 2026-04-02 09:34:05 +02:00
Rene Fichtmueller
87b8e4a4c3 feat: add 5 form-factor coverage scrapers with worker registrations 
Add Comms-Express, Router-Switch.com, Multimode Inc, OpticTransceiver.com,
and Wiitek scrapers covering CFP2-DCO, CFP4, OSFP224, QSFP112, CXP, GBIC,
XENPAK, CSFP, SFP-DD, SFP56, QSFP56 and other previously-uncovered form
factors. Each scheduled every 8h. Worker registrations added to scheduler.

Also export db alias in utils/db.ts to fix eBay enricher + community scrapers
crashing with 'Cannot read properties of undefined (reading query)'.
 2026-04-02 08:39:17 +02:00
Rene Fichtmueller
52d3588f55 fix: remove internal IPs, ports, tokens from changelog entries 2026-04-02 08:16:48 +02:00
Rene Fichtmueller
4b7f6de36d chore: changelog entry for auth feature 2026-04-02 07:32:40 +02:00
Rene Fichtmueller
53e9c3e899 feat: password-protected login page + API auth middleware 
- POST /api/auth/login: HMAC-SHA256 signed 7-day token, password from DASHBOARD_PASSWORD env
- GET /api/auth/verify: stateless token validation
- requireAuth middleware applied to all /api/* routes (except /api/health + /api/auth)
- /dashboard/login.html: dark TIP-themed login page with show/hide password toggle
- index.html: auth guard redirect to login + Authorization header on all api() calls
- No secrets in code — password stored in .env only
 2026-04-02 07:31:15 +02:00
Rene Fichtmueller
6f57d53df5 chore: update CHANGELOG_PENDING with Pi fleet + prediction intelligence system entries 2026-04-02 02:14:45 +02:00
Rene Fichtmueller
7f1c701ba1 feat: 6 prediction signal scrapers + forecast engine 
New scrapers (all registered in pg-boss, 50 total jobs):
  - sec-edgar.ts       : SEC EDGAR XBRL API — hyperscaler CapEx from 10-Q/10-K
  - github-signals.ts  : GitHub Search/Stats API — tech adoption metrics weekly
  - ebay-velocity.ts   : eBay completed listings — sold count + price distribution
  - ai-clusters.ts     : RSS feeds (6 sources) — AI cluster & DC announcements
  - distributor-leads.ts : Mouser, Digi-Key, RS Components — lead time + stock
  - standards-tracker.ts : IEEE 802.3, OIF, IETF — draft/ballot/published status

New utilities:
  - forecast-engine.ts : Weighted signal aggregator → demand_index + price_direction
    6 signal types, 4 horizons (3/9/12/18 months), 5 technologies tracked

New DB tables (migration 022):
  hyperscaler_capex, distributor_lead_times, github_tech_signals,
  marketplace_velocity, ai_cluster_announcements, standards_activity,
  forecast_signals

Schedules:
  - EDGAR: weekly Mon 06:00
  - GitHub: weekly Sun 05:00
  - eBay velocity: every 12h
  - AI clusters: every 4h (news-speed)
  - Distributor leads: daily 03:30
  - Standards: weekly Wed 04:00
  - Forecast engine: daily 08:00 (after all nightly scrapers)
 2026-04-02 02:02:44 +02:00
Rene Fichtmueller
fc7d74e680 feat: download datasheets + manuals to Fearghas NAS in nightly sync 
- downloadDocuments(): fetches PDFs from product_documents and documents tables
  using curl, organises into switches/ transceivers/ whitepapers/ other/ subdirs
- Integrated into runNightlyNasSync() — runs after JSON exports
- rsync incremental — only new/changed files transferred
- NAS dir structure: /volume1/tip-data/datasheets/{switches,transceivers,whitepapers,other}
- max-filesize 50MB guard per file
 2026-04-02 01:47:16 +02:00
Rene Fichtmueller
eb4db9ca76 feat: add logger utility + WireGuard setup in pi-scraper-setup.sh 
- utils/logger.ts: minimal console-based logger (debug/info/warn/error)
  used by community-issues and ebay-enricher scrapers
- scripts/pi-scraper-setup.sh: step 7 adds optional WireGuard setup
  (pass WG_PRIVKEY + WG_ADDR env vars) — connects Pi to Erik for DB access
  auto-detects dead ethernet and routes WG traffic via working interface
 2026-04-02 01:42:25 +02:00