19 Commits

Author SHA1 Message Date
Rene Fichtmueller
ef9fe52f69 infra: add deploy script + PeeringDB daily refresh cron 2026-04-09 20:45:38 +02:00
Rene Fichtmueller
487b032661 fix: reduce cold call times — aspa/verify cache + 3s LG timeout + 8s default fetchJSON
- aspa/verify: 15min result cache, looking-glass 3s timeout (was 20s default), 5→3 prefixes
- fetchJSON default timeout: 20s→8s prevents all uncached RIPE Stat calls from waiting 20s
- All cards now respond in <1s on cold call (ASPA 200ms, verify 170ms, validate 820ms, WHOIS 50ms)
- bgproutes still 4s cold (bgproutes.io API latency, cached after first call)
2026-04-09 07:49:19 +02:00
Rene Fichtmueller
35b89c05aa fix: eliminate hanging cards — ASPA/bgproutes/WHOIS/PeeringRec all responsive
- ASPA: 15min result cache + looking-glass timeout 3s (was 8s), hard cap 12s (was 18s)
- bgproutes: 15min result cache + 6s timeout on RIB POST (was no timeout), vpCache 1h
- WHOIS: 24h cache + RDAP fallback timeouts 3s (was 5s)
- Peering Recommendations: replace 20x full /api/lookup with new /api/quick-ix
- postJSON: add configurable timeout (was no timeout, caused indefinite hangs)
- Frontend: AbortController timeouts on all slow card fetches (ASPA/bgproutes/WHOIS/health)
- New /api/quick-ix endpoint: PeeringDB IX data + network name, 1h cached
2026-04-08 23:56:08 +02:00
Rene Fichtmueller
8f51f32dc3 fix: never cache null responses + increase RIPE Stat timeout for large carriers
Root cause of neighbour=0 for large carriers (AS9002, AS3491, AS12956):
1. RIPE Stat asn-neighbours returns 5000+ entries for Tier-1 carriers,
   exceeding the 30s timeout → fetchJSON returns null
2. null was cached in ripeStatCache for 15 minutes (the endpoint TTL)
3. All subsequent requests hit the null cache → perpetual 0 neighbours

Fixes:
- Never cache null results in ripeStatCache (only successful responses)
- Never persist null entries to disk cache
- Increase RIPE Stat timeout from 30s to 45s for prefix/neighbour queries
- Increase RIPE Stat semaphore from 10 to 15 concurrent requests

Verified: AS9002 up=146 down=2702, AS3491 up=90 down=710
2026-03-30 07:58:24 +02:00
Rene Fichtmueller
35e0b69442 fix: enrich - skip disambiguation pages, try first-word fallback for compound names 2026-03-30 06:04:34 +02:00
Rene Fichtmueller
96950992df feat: add company enrichment, ASPA timeout guard, map side panel, OIM telecoms
- /api/enrich: Wikipedia + website meta scraping with redirect following
- ASPA /api/aspa: 18s hard timeout guard + 8s per-call limit
- WHOIS: defensive null check
- Map: replace popups with left side panel
- Map: OIM Telecoms fiber layer (OpenInfraMap vector tiles)
- Map layer toggles: fix source-exists early-return bug
- Provider graph: fix text colors for light background
- Network Health: defensive HTML response check
2026-03-30 05:42:38 +02:00
Rene Fichtmueller
df2e176b35 feat: 3-layer data validation cache — local ROA store, PDB cache, RIPE Stat throttling
- Phase 1: Parse ~400k ROAs from Cloudflare RPKI feed into local store
  Eliminates ALL per-prefix RIPE Stat API calls (was 2000+ per lookup)
  Binary search validation in <0.1ms instead of 1-20s HTTP roundtrip
  Disk persistence (.roa-cache.json) for fast restart

- Phase 2: PeeringDB source cache (L2) for net/netixlan/netfac
  6h TTL with LRU eviction (max 5000 entries per type)
  Disk persistence (.pdb-source-cache.json) every 30min + SIGTERM

- Phase 3: RIPE Stat semaphore (max 10 concurrent) + response cache
  Endpoint-specific TTLs (15min-24h based on change rate)
  Max 2000 cached responses, disk persistence

- Phase 4: Extended /api/health with cache status, ASPA adoption metrics
  Version bump to 0.6.0
  Jittered refresh timers to prevent thundering herd
  Graceful shutdown saves all caches

Expected: Audit accuracy 82% -> 95%+, lookup time 90s -> <8s
2026-03-30 05:18:31 +02:00
Rene Fichtmueller
3adc34c42b feat: Lia's Paradise country data fix + file upload
- Fix org→country mapping: pre-cache 20k+ PeeringDB orgs at startup
- 22k networks now have country codes (was 0 before)
- Add file upload: CSV/TXT/PDF/XLS/DOC → ASN extraction → probe check
- Export file results as printable PDF
2026-03-27 01:40:03 +13:00
Rene Fichtmueller
41af8be7f4 feat: Lia's Paradise, bug fixes, company descriptions
- Add /lia Easter egg page: RIPE Atlas coverage explorer showing
  34k+ networks grouped by country with probe/no-probe status,
  RIR filtering, search, and PDF export
- Add /api/lia/coverage endpoint combining PeeringDB + Atlas data
- Fix Provider Relationship Graph (renamed var to avoid shadowing)
- Fix ROV/ASPA double-value display (show worst single status)
- Add fallback: render provider graph from lookup data when ASPA fails
- Add company description (org_name) to Network Overview
- Add worstStatus() helper for frontend badge normalization
2026-03-27 01:32:30 +13:00
Rene Fichtmueller
dee5871609 fix: PeeringDB API key + User-Agent WAF fix + bgproutes.io visibility fallback 2026-03-27 00:22:08 +13:00
Rene Fichtmueller
c4462bf633 fix: 6 critical data accuracy bugs
- Bug 1: Facilities returned ALL 59k PeeringDB entries (missing net_id filter)
- Bug 2: Neighbours returned 0 for large ASNs (8s timeout, now 30s)
- Bug 3: Visibility showed 0% when API times out (now shows -1 = unavailable)
- Bug 4: Prefixes returned 0 for small ASNs (cascading fetch failure)
- Bug 5: RPKI inconsistency documented (different sample sets per endpoint)
- Bug 6: Atlas probe status showed 0 connected (status.name vs status_name)

Also: PDB IX/Fac queries now use net_id (phase 0 + parallel phase 1)
Also: Compare endpoint uses net_id for facilities
2026-03-26 23:51:07 +13:00
Rene Fichtmueller
1120d81dfc feat: RPKI-based ASPA detection via Cloudflare feed (1455 objects), collapsible lists, sorted ASNs, Route Views 2026-03-26 16:02:14 +13:00
Rene Fichtmueller
bd90113285 feat: sortable ASPA audit lists (by ASN, frequency, or name) 2026-03-26 14:42:54 +13:00
Rene Fichtmueller
21450ea0eb feat: collapsible providers list + scrollable ASPA template 2026-03-26 14:17:42 +13:00
Rene Fichtmueller
86d6bd8e2d feat: add Route Views to header, ASN links, and data sources footer 2026-03-26 14:15:10 +13:00
Rene Fichtmueller
c23d99558b feat: collapsible show-more for ASPA audit lists (missing + extra providers) 2026-03-26 14:11:05 +13:00
Rene Fichtmueller
cd03ad0744 fix: move routing CSS from broken JS string to style block (SyntaxError) 2026-03-26 13:02:36 +13:00
Rene Fichtmueller
976bdb48e4 perf: optimize compare endpoint + add caching everywhere
- Compare: all API calls in single parallel batch (was sequential)
- Compare: RPKI sample reduced to 3+3 prefixes with 5s timeout cap
- Compare: response caching (5min TTL)
- Compare: AS name resolution parallel with 3s timeout
- Result: Compare from timeout (>20s) to ~5s first call, <1s cached
2026-03-26 12:54:22 +13:00
Rene Fichtmueller
267943b647 feat: performance fixes + deploy directory with live dashboard
- Add response caching (5min TTL for lookups, 10min for ASPA)
- Add 8s timeout to all external API fetches
- RPKI validation: sample max 10 prefixes (5 v4 + 5 v6) instead of 50
- Run all PeeringDB + RIPE Stat calls in single parallel batch
- Resolve AS names in parallel with 3s timeout cap
- Add deploy/ directory with production server.js + index.html dashboard
- Landing page: Tokyo Night dark theme, interactive ASN search
- 15 API endpoints: lookup, aspa, aspa/verify, bgproutes, validate,
  compare, peers/find, prefix/detail, ix/detail, topology, whois, health
- Features: RPKI per-prefix, RIPE Atlas probes, Network Health Report,
  ASPA RFC verification engine, Provider Relationship Graph
2026-03-26 12:50:54 +13:00