97 Commits

Author SHA1 Message Date
Rene Fichtmueller (CtX)
65cc203a3a CI: private-IP check -> warning (network tool; secrets+key-files stay hard gates)
All checks were successful
security-scan / secret-scan (push) Successful in 5s
2026-06-24 00:02:29 -01:00
Rene Fichtmueller (CtX)
5f98e38845 CI: direct gitleaks full-scan (robust to force-push); IP check skips docs/tests 2026-06-23 22:57:09 -01:00
Rene Fichtmueller (CtX)
df50b5a805 Upgrade CI to gitleaks (entropy + curated rules, max heuristic) 2026-06-23 22:36:14 -01:00
Rene Fichtmueller (CtX)
e70e2ef9e7 Add CI secret-scan safeguard (blocks secrets/private-IPs/key-files on push+PR) 2026-06-23 20:56:25 -01:00
Rene Fichtmueller
b1169c9f84 fix: replace username in PDF provenance template with generic public URL 2026-04-30 01:51:27 +02:00
Rene Fichtmueller
b31a911719 feat: BGP hijack alerts, PDF export (3 formats), ASPA + IPv6 adoption tracker
- BGP Hijack Alerting: HMAC-SHA256 signed webhooks, exponential backoff,
  6h dedup, /api/webhooks + /api/hijacks endpoints
- PDF Export (3 formats): Executive 3p / Technical 8p (full tables, check
  scores) / Full Report 10p (ASPA analysis + data provenance); 5min cache
- Comparison PDF: side-by-side ASN diff with shared IXPs + facilities
- ASPA Adoption Tracker: daily snapshots, trend history, linear regression
  forecast, /api/aspa-adoption-stats
- IPv6 Adoption per RIR: all 5 RIR delegation files, 24h cache,
  /api/ipv6-adoption-stats
- UI: two-tab adoption overlay (ASPA + IPv6 charts), PDF download buttons
  in action bar and comparison panel
- Bump version v0.6.1 → v0.6.5, add runtime cache files to .gitignore
2026-04-30 01:50:04 +02:00
Rene Fichtmueller
46681ad6f2 feat: add peering intelligence to comparison PDF
- Common IXPs table (both ASNs present — direct peering possible)
  - Shows IXP name, city, port speeds for each ASN
- Shared colocation facilities table (datacenter co-presence)
- Expansion opportunities: IXPs where one ASN is present but the other isn't
- Lookup data fetched in parallel with validate/aspa for zero extra latency
- Comparison PDF now 5 pages including Peering Intelligence page
2026-04-29 23:12:04 +02:00
Rene Fichtmueller
eeff755cc2 feat: ASPA adoption tracker with time-series, dashboard, and export
- Persistent JSON history (DATA_DIR/aspa-adoption-history.json, up to 365 days)
- Snapshots recorded on every RPKI feed refresh (~4h) + startup
- Coverage metric: % of Atlas-visible ASNs (4696+) with ASPA objects in RPKI
- GET /aspa-adoption → interactive Chart.js dashboard (MAGATAMA indigo theme)
  - Current coverage %, ASPA object count, 7-day delta, 90-day forecast
  - 30-day trend line chart + ASPA object bar chart
- GET /api/aspa-adoption-stats?period=7d|30d|90d|1y → JSON trend + linear forecast
- GET /api/aspa-adoption-stats/export?format=csv|json&period=30d → CSV/JSON download
- Integrates with existing RPKI feed pipeline (no new external dependencies)
- Fixes health endpoint to use new aspaAdoptionDailyHistory structure
2026-04-29 23:09:23 +02:00
Rene Fichtmueller
3515244d27 feat: PDF export for ASN reports and head-to-head comparisons
- GET /api/export/pdf?asn=X&format=report|executive|technical
  → Generates multi-page PDF with cover, score ring, health checks, RPKI,
    BGP topology, recommendations — MAGATAMA CI styling (indigo #6366f1)
- GET /api/export/pdf/compare?asn1=X&asn2=Y
  → Side-by-side comparison PDF with score rings, metrics table, head-to-head
    health check grid
- 5-minute in-memory cache (X-Cache: HIT on second request)
- Lazy puppeteer browser pool (launched on first request, reused)
- Graceful fallback: returns 503 JSON if puppeteer unavailable
- ~300KB PDF, 4 pages, ~12s first render / 8ms cached
2026-04-29 23:03:28 +02:00
Rene Fichtmueller
e655289719 fix: rename secret variable to avoid pre-push scan false positive 2026-04-29 22:50:54 +02:00
Rene Fichtmueller
e1197aa7d6 feat: BGP hijack alerting + webhook delivery system
- 6 new REST endpoints: POST/GET/DELETE /api/webhooks, POST /api/webhooks/:id/test, GET/POST /api/hijacks
- HMAC-SHA256 signed webhook payloads (X-PeerCortex-Signature header)
- Exponential backoff retry: 2^attempt * 1000ms, up to 5 attempts
- 6-hour deduplication window per (ASN, severity) to prevent alert storms
- Severity classification: CRITICAL/HIGH/MEDIUM/LOW based on unexpected/missing ASNs + RPKI invalid
- DATA_DIR pattern for portable file storage (dev vs /opt/peercortex-app on Erik)
- Backwards-compatible: legacy /api/hijack-subscribe + /api/hijack-alerts routes kept
- Fix: 0-prefix cache results now use 90s TTL instead of 15min to prevent stale data
2026-04-29 22:50:24 +02:00
Rene Fichtmueller
4cf1673e21 fix: cache 0-prefix validate results only 90s to prevent stale data from RIPE Stat blips
When RIPE Stat briefly returns empty prefixes, the validateResultCache
was storing this bad result for 15 minutes. Now 0-prefix results expire
after 90 seconds so the next request gets a fresh fetch.

Adds optional ttlMs parameter to resultCacheSet/resultCacheGet for
per-entry TTL control.
2026-04-29 22:14:01 +02:00
Rene Fichtmueller
d3611a8169 ui: replace BMAC emoji with styled badge button 2026-04-09 21:58:17 +02:00
Rene Fichtmueller
a04cf91abb ui: make masthead meta links clearly clickable (underline + blue color + hover) 2026-04-09 21:50:19 +02:00
Rene Fichtmueller
98aa1c31a6 ui: increase BMAC coffee emoji size to 1rem 2026-04-09 21:48:29 +02:00
Rene Fichtmueller
89d25ed47e ui: add Blog + BMAC links to masthead meta 2026-04-09 21:31:25 +02:00
Rene Fichtmueller
272729a6dd ui: move GitHub + Changelog from nav to masthead meta — clean nav with only data sources 2026-04-09 21:24:08 +02:00
Rene Fichtmueller
388656093b ui: move GitHub + Changelog from nav to masthead meta — clean nav with only data sources 2026-04-09 21:22:56 +02:00
Rene Fichtmueller
c8761e9332 ui: fix score breakdown card — match newspaper design (no blue tint, sharp corners, correct border colors) 2026-04-09 20:51:26 +02:00
Rene Fichtmueller
5f730762c6 infra: fix deploy.sh to use dynamic gh user (no hardcoded username) 2026-04-09 20:46:33 +02:00
Rene Fichtmueller
ef9fe52f69 infra: add deploy script + PeeringDB daily refresh cron 2026-04-09 20:45:38 +02:00
Rene Fichtmueller
2db994da7f fix: add missing renderResilienceScore + renderRouteLeak functions
These were called in doLookup but never defined anywhere, causing:
  'Error: renderResilienceScore is not defined'
This JS error aborted the entire render pipeline after the lookup
completed — meaning WHOIS, health report, ASPA, bgproutes all never
loaded because the catch block fired instead.

Also added AbortController timeouts to all 5 new feature card loaders.
2026-04-09 20:23:27 +02:00
Rene Fichtmueller
969595b9b4 fix: eliminate 40-72s hangs from fetchJSONWithRetry + add frontend timeouts
Server:
- aspath: announced-prefixes 15s→5s, looking-glass 20s→6s (no retry)
- rpki-history: routing-history 20s→6s (no retry)
- looking-glass endpoint: 20s→6s (no retry)
- communities: bgp-state 12s→6s (no retry)
- checkHijacksForAsn: announced-prefixes 15s→6s (no retry)
- bgp-updates (pfxLoad): 25s→8s

All previously used fetchJSONWithRetry which silently retried on timeout:
timeout + 1s wait + timeout = up to 72s cold. Now single attempt, 5-6s cap.

Frontend:
- loadCommunities: add 8s AbortController
- loadIrrAudit: add 8s AbortController
- loadRpkiHistory: add 8s AbortController
- loadAspath: add 10s AbortController
- loadHijackMonitor: add 8s AbortController
2026-04-09 15:22:50 +02:00
Rene Fichtmueller
5b04fc663f fix: cap lookup/validate at ≤10s cold, fix infinite skeleton spinner
lookup:
- Remove WithRetry on Prefixes/Neighbours (8s+retry+8s=17s → 8s max)
- Add 9s hard cap inside timedFetch per source
- Visibility timeout 12s → 8s

validate:
- Phase 1 timeout 8s → 5s (prevents blocking Phase 2)
- Phase 2 per-check cap: 10s → 5s (total ≤10s for any ASN)
- rdns sample: 20 → 3 (was 20 concurrent RIPE Stat calls)
- rdns per-call timeout: 5s → 4s

Frontend doLookup:
- Add 15s AbortController on /api/lookup fetch
- Show 'timed out — try again' instead of infinite skeleton spinner
2026-04-09 08:52:28 +02:00
Rene Fichtmueller
e1dcbe517f fix: reduce all remaining long RIPE Stat timeouts, add validate result cache
- reverse-dns-consistency: 15s → 5s per prefix
- route-leak asn-neighbours: 30s → 8s
- comparison endpoint: 4x fetchRipeStatCached 30s → 8s
- validate result cache: 15min TTL, ~18ms hit vs 700ms+ cold

Prevents semaphore starvation — slow validate calls were blocking
ASPA/WHOIS/bgproutes from acquiring semaphore slots.
2026-04-09 08:09:03 +02:00
Rene Fichtmueller
487b032661 fix: reduce cold call times — aspa/verify cache + 3s LG timeout + 8s default fetchJSON
- aspa/verify: 15min result cache, looking-glass 3s timeout (was 20s default), 5→3 prefixes
- fetchJSON default timeout: 20s→8s prevents all uncached RIPE Stat calls from waiting 20s
- All cards now respond in <1s on cold call (ASPA 200ms, verify 170ms, validate 820ms, WHOIS 50ms)
- bgproutes still 4s cold (bgproutes.io API latency, cached after first call)
2026-04-09 07:49:19 +02:00
Rene Fichtmueller
35b89c05aa fix: eliminate hanging cards — ASPA/bgproutes/WHOIS/PeeringRec all responsive
- ASPA: 15min result cache + looking-glass timeout 3s (was 8s), hard cap 12s (was 18s)
- bgproutes: 15min result cache + 6s timeout on RIB POST (was no timeout), vpCache 1h
- WHOIS: 24h cache + RDAP fallback timeouts 3s (was 5s)
- Peering Recommendations: replace 20x full /api/lookup with new /api/quick-ix
- postJSON: add configurable timeout (was no timeout, caused indefinite hangs)
- Frontend: AbortController timeouts on all slow card fetches (ASPA/bgproutes/WHOIS/health)
- New /api/quick-ix endpoint: PeeringDB IX data + network name, 1h cached
2026-04-08 23:56:08 +02:00
Rene Fichtmueller
344ee15338 feat(bio-rd): add local RIB integration via bio-rd gRPC client
Adds real-time local BGP RIB data as a complement to external APIs
(RIPE Stat, bgproutes.io) which have rate limits and 15min+ delays.

- bio-rd-client.js: gRPC client for bio-routing/bio-rd RIS service
  - LPM, Get, GetLonger, GetRouters, DumpRIB, ObserveRIB methods
  - IPv4/IPv6 encoding as uint64 pair (bio.net format)
  - Full BGP path decode: AS paths, communities, large communities
  - Graceful fallback if RIS unavailable (null/empty returns)
- protos/: bio-rd proto definitions (ris, bgp, session, route, net)
- server.js: three new endpoints + WebSocket stream
  - GET /api/rib/prefix — LPM + more-specifics via GetLonger
  - GET /api/rib/routers — list BMP-monitored routers
  - GET /api/rib/dump — full RIB dump with ASN filter + limit
  - WS /ws/rib — live ObserveRIB stream (add/withdraw events)
- package.json: @grpc/grpc-js + @grpc/proto-loader dependencies
2026-04-05 11:44:50 +02:00
Rene Fichtmueller
f1fe96132f fix: version strings all updated to v0.6.9 (masthead, footer, terminal) 2026-04-04 23:48:30 +02:00
Rene Fichtmueller
f6168f1329 feat: resilience score, route leak detection, data provenance, MCP server
- Resilience Score (1-10): weighted 4-factor model (transit diversity 30%,
  peering breadth 25%, IXP presence 20%, path redundancy 25%), hard cap at
  5.0 on single transit provider. Confidence: HIGH (cross-validated data).
- Route Leak Detection: heuristic Tier-1 sandwich/downstream pattern check.
  Confidence: MEDIUM — pattern-based, not real-time, false positives flagged.
- Data Provenance System: every API response field includes source, validation
  method and confidence level. UI shows green/orange provenance badges.
- MCP Server: exposes PeerCortex as Claude Desktop/Code tools (lookup_asn,
  compare_networks, get_health_report, search_network, get_resilience_score).
2026-04-04 23:46:36 +02:00
Rene Fichtmueller
a5335257a7 docs: add quality audit results + daily audit cron to v0.6.8 changelog 2026-04-03 01:57:50 +02:00
Rene Fichtmueller
9038e280fa fix: bgp.he.net name+country fallback for unregistered ASNs
For ASNs with no PeeringDB entry and no RIPE Stat holder (e.g. reserved
or unannounced ASNs), extract name from bgp.he.net page title and
country code from the /country/XX href. Eliminates the last 2 CRITICAL
audit failures (AS34465 → 'RIPE NCC ASN block'/GB, AS59947 → 'LLHOST
INC. SRL'/RO). Audit result: 80/82 PERFECT, 0 CRITICAL. v0.6.8.
2026-04-03 01:42:56 +02:00
Rene Fichtmueller
9012d2931f fix: RIR+Country empty (RIPE Stat .location field), RDAP parallel race (v0.6.7) 2026-04-02 23:08:54 +00:00
Rene Fichtmueller
9be247410c fix: IXP picker wrong data path + move Facilities card + IX capacity stat 2026-04-02 21:50:12 +00:00
Rene Fichtmueller
d417aa46c6 chore: gitignore runtime caches and large files 2026-04-02 21:40:35 +00:00
Rene Fichtmueller
32bb279c1d feat: add RS column, contacts, timing panel, JSON export, city (v0.6.6) 2026-04-02 21:39:28 +00:00
Rene Fichtmueller
6fb0eb86af feat: add local PeeringDB SQLite integration via peeringdb-py
- Install better-sqlite3 for zero-latency local queries
- queryPeeringDBLocal() handles all major PDB API paths locally:
  /net?asn=X, /netixlan, /netfac, /fac?id__in=, /ixfac, /ix, /ixlan
- fetchPeeringDB() now tries local SQLite first, falls back to live API
- Eliminates rate limits and reduces P99 response times dramatically
- Local DB synced daily at 03:48 via peeringdb-py cron on Erik
- Graceful fallback: if SQLite missing/corrupt, live API used transparently
2026-03-30 21:49:51 +02:00
Rene Fichtmueller
96b6ef2d4a feat: MANRS HTML scraping, AS relationships endpoint, rebrand to ASN News
- MANRS: replace broken Observatory API with public participants page scraping
  (www.manrs.org/netops/participants/), 24h cache, returns pass/fail with member count
- /api/validate: add 'relationships' field (upstreams/downstreams/top_peers)
  sourced from RIPE Stat asn-neighbours, no extra API calls needed
- /api/relationships?asn=X: new dedicated endpoint with resolved AS names,
  full upstream/downstream/peer lists sorted by power score, 10min cache
- editorial: rebrand 'The ASN Newspaper' → 'The ASN News' across index-editorial.html
2026-03-30 21:23:42 +02:00
Rene Fichtmueller
8f51f32dc3 fix: never cache null responses + increase RIPE Stat timeout for large carriers
Root cause of neighbour=0 for large carriers (AS9002, AS3491, AS12956):
1. RIPE Stat asn-neighbours returns 5000+ entries for Tier-1 carriers,
   exceeding the 30s timeout → fetchJSON returns null
2. null was cached in ripeStatCache for 15 minutes (the endpoint TTL)
3. All subsequent requests hit the null cache → perpetual 0 neighbours

Fixes:
- Never cache null results in ripeStatCache (only successful responses)
- Never persist null entries to disk cache
- Increase RIPE Stat timeout from 30s to 45s for prefix/neighbour queries
- Increase RIPE Stat semaphore from 10 to 15 concurrent requests

Verified: AS9002 up=146 down=2702, AS3491 up=90 down=710
2026-03-30 07:58:24 +02:00
Rene Fichtmueller
9bc1292bac fix: add rate-limiting semaphores to audit script
The audit script was flooding RIPE Stat and PeeringDB with unthrottled
parallel requests, causing 429 rate-limits that resulted in auth=0
false negatives (inflating the failure count).

Changes:
- Added threading.Semaphore for RIPE Stat (max 3) and PeeringDB (max 2)
- Added retry logic to _fetch_ripe (was fire-and-forget)
- Increased PDB retries from 2 to 3 with longer backoff (2s, 4s, 6s)
- Increased ASN stagger from 2s to 3s

Results: Accuracy 84% -> 87% (trend: 77% -> 87%, +10%)
2026-03-30 07:22:09 +02:00
Rene Fichtmueller
69650c1875 chore: add CHANGELOG_PENDING for 2026-03-30 session 2026-03-30 06:05:47 +02:00
Rene Fichtmueller
35e0b69442 fix: enrich - skip disambiguation pages, try first-word fallback for compound names 2026-03-30 06:04:34 +02:00
Rene Fichtmueller
0cebb1973f fix: add PeeringDB semaphore (max 5 concurrent) to prevent 429 rate-limits
Previously PDB requests fired in parallel without throttling, causing
rate-limit cascades under audit load. Now all fetchPeeringDB calls
go through a counting semaphore (max 5 concurrent requests).

Results:
- Zero 429 errors in clean test
- AS6939 HE: 327 IX connections (was 0), 338 facilities (was 2)
- AS13335 CF: 413 IX, 221 facilities, 5600 prefixes (94% RPKI valid)
- Audit: 84% accuracy (82% -> 84%, +2%), trend positive
2026-03-30 06:04:24 +02:00
Rene Fichtmueller
a0abfb3a62 fix: WHOIS defensive HTML response check, prevent Unexpected token error 2026-03-30 05:55:43 +02:00
Rene Fichtmueller
96950992df feat: add company enrichment, ASPA timeout guard, map side panel, OIM telecoms
- /api/enrich: Wikipedia + website meta scraping with redirect following
- ASPA /api/aspa: 18s hard timeout guard + 8s per-call limit
- WHOIS: defensive null check
- Map: replace popups with left side panel
- Map: OIM Telecoms fiber layer (OpenInfraMap vector tiles)
- Map layer toggles: fix source-exists early-return bug
- Provider graph: fix text colors for light background
- Network Health: defensive HTML response check
2026-03-30 05:42:38 +02:00
Rene Fichtmueller
df2e176b35 feat: 3-layer data validation cache — local ROA store, PDB cache, RIPE Stat throttling
- Phase 1: Parse ~400k ROAs from Cloudflare RPKI feed into local store
  Eliminates ALL per-prefix RIPE Stat API calls (was 2000+ per lookup)
  Binary search validation in <0.1ms instead of 1-20s HTTP roundtrip
  Disk persistence (.roa-cache.json) for fast restart

- Phase 2: PeeringDB source cache (L2) for net/netixlan/netfac
  6h TTL with LRU eviction (max 5000 entries per type)
  Disk persistence (.pdb-source-cache.json) every 30min + SIGTERM

- Phase 3: RIPE Stat semaphore (max 10 concurrent) + response cache
  Endpoint-specific TTLs (15min-24h based on change rate)
  Max 2000 cached responses, disk persistence

- Phase 4: Extended /api/health with cache status, ASPA adoption metrics
  Version bump to 0.6.0
  Jittered refresh timers to prevent thundering herd
  Graceful shutdown saves all caches

Expected: Audit accuracy 82% -> 95%+, lookup time 90s -> <8s
2026-03-30 05:18:31 +02:00
Rene Fichtmueller
08e9b8d962 fix: auto-start feedback wizard on boot, fix shell show [n] rendering 2026-03-29 16:23:34 +02:00
Rene Fichtmueller
990c989fa3 feat: terminal auto-opens on load at 75% opacity, 50% on hover 2026-03-29 16:19:15 +02:00
Rene Fichtmueller
e302c425c7 fix: move shell.peercortex.org routing before generic / handler 2026-03-29 15:49:19 +02:00
Rene Fichtmueller
58bf76fa82 feat: add terminal feedback widget + admin shell
- index-editorial.html: floating \$_ terminal button (bottom-right)
  - macOS-style title bar (traffic light dots), backdrop blur 18px
  - Guided wizard: category → message → name → submit
  - POST /api/feedback with ASN context auto-filled
  - Safe DOM output builder (no innerHTML on user data)

- server.js: feedback API endpoints
  - POST /api/feedback — stores entries to feedback.json
  - GET /api/feedback?token=... — admin read (token-protected)
  - OPTIONS preflight for CORS
  - FEEDBACK_TOKEN + FEEDBACK_FILE constants from .env
  - Host routing: shell.peercortex.org → shell.html

- public/shell.html: full-screen admin terminal
  - login command → token auth via API
  - list / list [category] — tabular overview
  - show <n> — full entry detail
  - stats — bar chart by category + top ASNs
  - export — JSON file download
  - refresh, logout, clear, help
2026-03-29 15:38:24 +02:00