2026-07-16 deep-dive audit (server.js, local-db-client.js, src/aspa/validator.ts)
found the flagship ASPA path verification producing wrong results, and a
recurring pattern where an upstream API or the local Postgres DB failing
would present as "checked, clean" instead of surfacing failure. All fixes
in this commit are empirically or textually verified against primary
sources (the IETF draft's actual algorithm text, RFC 5735/6890 bogon
ranges, and hand-built controlled test cases with known-correct answers)
-- not just read-and-guessed.
CRITICAL: ASPA hop-check direction was reversed
--------------------------------------------------
verifyUpstream/verifyDownstream/detectValleys called
hopCheck(collapsed[i-1], collapsed[i]) -- checking the PROVIDER-side AS's
ASPA object for the CUSTOMER, the opposite of
draft-ietf-sidrops-aspa-verification section 5.3's authorized(A(I), A(I+1))
(A(I) = customer, always checked first). Proven empirically: a hand-built,
fully-attested, zero-leak 3-AS path returned "Invalid" before this fix.
verifyDownstream needed a full rewrite, not just an argument swap -- a
downstream path can legitimately contain both an up-ramp AND a down-ramp
(e.g. origin -> providers -> peering point -> validator's providers ->
validator), and the old K/L/uMin/vMax computation didn't correctly
implement the draft's max_up_ramp/min_up_ramp/max_down_ramp/min_down_ramp
four-way computation. Rewrote it from the draft's exact halt conditions,
verified against a worked example fetched from the draft itself (an N=4
path with a single unattested "kink" correctly resolves Valid) plus 3
hand-built scenarios (pure up-ramp, mixed up+down clean, and a definite
leak with failures on both ends -- Invalid).
Same bug existed independently in src/aspa/validator.ts (the MCP-server-
side validator, different codebase, same conceptual error) -- validateUpstream
was already correct there, but validateDownstream naively reversed the path
and reapplied upstream logic, which cannot represent a legitimate down-ramp.
Rewrote it with the same four-ramp algorithm, cross-validated against the
same test scenarios through the compiled output.
Also in the same /api/aspa/verify handler: aspaStore.set(providerAsn, new
Set()) for detected-but-unattested providers made hopCheck report
"NotProviderPlus" (implying a real ASPA object that denies the hop) instead
of "NoAttestation" (we don't know) -- removed; hopCheck's own `!providers`
branch already handles the unset case correctly. Also removed a fallback
that fabricated an ASPA declaration from heuristically-detected BGP
neighbours when aspa_object_exists is false, which contradicted that exact
field in the same response.
ROOT CAUSE of ~9 broken checks: duplicate function declaration
--------------------------------------------------------------
Two `async function fetchRipeStatCached` declarations existed ~150 lines
apart (the original real RIPE-Stat-fetch+cache+throttle implementation,
and a newer local-DB-routing wrapper added during the Postgres refactor).
In JS, the second declaration silently wins for every call site in the
file. The wrapper only recognized 5 URL patterns (announced-prefixes,
asn-neighbours, as-overview, visibility, prefix-size-distribution) and
returned null unconditionally for everything else -- silently breaking
blocklist (Spamhaus), abuse-contact-finder, bgp-updates, routing-status,
looking-glass, whois, reverse-dns-consistency, maxmind-geo-lite-pfx, and
ixs lookups. Renamed the original to fetchRipeStatCachedFromApi and made
the wrapper fall through to it instead of returning null. Verified via an
isolated routing test (blocklist/abuse-contact-finder URLs now reach the
real implementation; the 5 known patterns still route to local DB).
Silent-failure / fake-success patterns fixed
---------------------------------------------
- local-db-client.js: 5 getRipeStat* functions returned status:'ok' with
empty data on Postgres error (fabricated success) -- now status:'error'.
server.js's /api/lookup now tracks which sources degraded, surfaces
meta.degraded_sources, and caches a degraded result for 90s instead of
the full 5min so a DB hiccup doesn't get repeated to every visitor.
- validateRPKIWithCache: DB errors and genuine "no covering ROA" both
produced status:"not_found" -- split into "not_found" (real) vs
"unavailable" (DB error). Updated all 5 call sites that compute RPKI
coverage percentages to exclude "unavailable" from the denominator (a DB
hiccup must not lower a network's displayed RPKI/ASPA score), and fixed
a pre-existing separate typo in the PDF report generator that checked
for 'not-found' (hyphen) when the actual value is 'not_found' (underscore).
- crossCheckRpki: an empty sample or a failed RIPE-validator lookup both
counted as a fabricated 100%/"agreement" instead of being excluded --
now returns agreement_pct:null when nothing was actually compared, and
the caller no longer averages null into overall_confidence. Also
documented (not silently left) that bgp.he.net fetching is hardcoded
off, so the "2-source" prefix/neighbour cross-checks never actually run
-- dataQuality.cross_checks now reports sources:1 honestly instead of
claiming a cross-check that never happened.
- lookupAspaFromRpki: added feedLoaded so a Cloudflare-RPKI-feed fetch
failure (rpkiAspaLastFetch never set) is distinguishable from "this ASN
genuinely has no ASPA object" -- surfaced as aspa_feed_healthy on both
ASPA endpoints.
- Bogon/IRR/Spamhaus-blocklist/reverse-DNS/BGP-visibility checks: fetch
failure -> empty result -> "pass" (or, for abuse-contact/rdns, a
misleading "fail") -- now report "info" (excluded from the weighted
health score, matching the existing checkManrsMembership pattern) when
the underlying data source didn't actually respond.
- Hijack monitoring: checkHijacksForAsn returned [] on DB error,
indistinguishable from a genuine zero-prefix result. On a subscription's
first monitoring cycle this permanently persisted an empty baseline,
silently disabling leak detection forever while the UI kept showing it
as active. Now returns null on failure; runHijackCheck skips the cycle
entirely (retries next time) instead of locking in a bad baseline. Fixed
2 other call sites that would otherwise crash or persist null onto disk.
- WHOIS cache: a transient failure across all 5 RIR sources (RIPE + 4
RDAP endpoints) was cached as "not found in any RIR database" for 24h,
identical to a genuinely non-existent ASN. Added a 10min TTL for that
specific case instead.
- /api/relationships: a neighbourData fetch failure produced "0
upstreams/downstreams/peers", cached unconditionally for 10min. Applied
the same 90s-on-suspicious-zero guard the codebase already uses for
/api/validate (commit 4cf1673) but never applied here.
Verification
------------
node --check on both .js files, tsc --noEmit clean on every touched
TS module (remaining errors are 100% pre-existing, in the unrelated
mcp-server subsystem), npm run build succeeds, isolated empirical tests
for both the ASPA algorithm (3 hand-built scenarios with known-correct
answers, cross-checked against draft text fetched live) and the
fetchRipeStatCached routing fix, and a full local boot test reaching the
exact same pre-existing sandbox-only crash point (missing PEERINGDB_API_KEY,
unrelated to this commit) as a baseline boot before any of today's changes.
Not deployed to Erik as part of this commit -- server.js and
public/index.html there are untouched.
The 2026-07-14 backend refactor (PR #1) moved 13 features' route logic
out of server.js into src/features/*/routes.ts as Fastify plugins, and
left "// Migrated to src/features/X/" comments behind -- but never
actually built or started anything that served those plugins. Found a
complete, unused Fastify app bootstrap already sitting at
src/api/server.ts (added in an earlier commit, 5554c1a, alongside the
BGP hijack/PDF-export/ASPA-adoption work) that imports and registers
all 16 route modules, including 3 (pdf-export, aspa-adoption,
hijack-alerts) with proper Postgres-backed implementations, tests, and
PDF templates -- more capable than the puppeteer-based inline version
kept from the production snapshot. Also confirmed there is no code
anywhere that calls initializeDatabase() or startApiServer(): this
Fastify server had never actually been run, not even once.
What this commit does:
- Fixes 6 real TypeScript compile errors blocking `tsc` from building
the affected modules cleanly (all pre-existing, none introduced by
this branch):
- src/routes/hijack-alerts.ts: route generic was missing `Body` in
its type param, so it didn't match the handler's own declared type
- src/features/aspa-adoption/scheduler.ts: node-cron v4's
ScheduledTask dropped `.destroy()` (relevant now that this branch
already bumped node-cron 3->4 for the uuid CVE fix) -- `.stop()`
alone is sufficient
- src/features/{bgp-communities,rpki-history}/routes.ts:
`response.json()` typed as `{}` under this @types/node version,
cast to `any` to match this file's existing loose-typing style
- src/features/pdf-export/cache-manager.ts: `NodeJS.Timer` ->
`NodeJS.Timeout` (what setInterval/clearInterval actually use)
- src/features/pdf-export/renderer.ts: this uses Playwright, not
Puppeteer -- `timeout` isn't a page.pdf() option in Playwright,
moved to page.setDefaultTimeout() before the call
- Adds src/api/index.ts as the actual entry point (there wasn't one).
Calls initializeDatabase() before dynamically importing ./server.js,
because src/routes/hijack-alerts.ts calls getDatabase() at module
load time -- a static top-level import of ./server would have
crashed on startup before initializeDatabase() ever ran. Verified
this boots cleanly and serves real responses (changelog-data,
ix-matrix, rib/prefix graceful-503, bgp-communities with a live
RIPE Stat call) in local testing.
- Wires server.js to reverse-proxy the 11 genuinely-orphaned paths
(submarine-cables, global-infra, communities, irr-audit,
asset-expand, rpki-history, aspath, looking-glass, ix-matrix,
changelog-data, rib/*, prefix-changes) to this new internal server
via a plain http.request() proxy -- chosen over exposing a second
Cloudflare Tunnel ingress rule so this stays a same-origin,
internal-only change with no DNS/tunnel config to touch. Verified
the proxy mechanism itself against a live instance of the new server.
Deliberately did NOT proxy /api/webhooks, /api/hijacks, or
/api/hijack-subscribe -- server.js already serves those paths
inline (file-backed, live, working), and src/routes/hijack-alerts.ts
registers the same path names against an empty Postgres table.
Proxying would have silently shadowed working data with nothing.
- Default port 3102, not 3100: checked Erik and found 3100 already
bound by an unrelated docker-proxy container. 3102 is free and
reads as "the second peercortex port" next to the existing 3101.
- Adds a `peercortex-api` PM2 app to ecosystem.config.js on Erik
(backed up first via rollback-standard.sh) -- config only, NOT
started. Nothing is deployed to Erik as part of this commit: no
dist/ upload, no `npm install` for fastify/pg on Erik, no `pm2
start`. server.js and public/index.html on Erik are untouched.
Reapplied against the current main (post src/features/ refactor),
since the old aspa/ module was carried over unchanged and still had
both bugs. Also caught two spots the stale local clone never had:
public/index.html's title= attribute and the newer index-classic.html.
ASPA has no RFC yet, still draft-ietf-sidrops-aspa-profile (object
format) and draft-ietf-sidrops-aspa-verification (path validation),
both in WG Last Call. "RFC 9582" is the ROA profile RFC, unrelated.
The generated "submit via RIPE DB webupdates / auto-dbm@ripe.net"
instructions were wrong too. ASPA objects are signed RPKI objects
created through a RIR's hosted RPKI platform, not RPSL text pasted
into the whois database. Replaced generateASPAObject/
generateRipeDbTemplate with summarizeASPAObject (review summary) and
generateASPASubmissionGuide (per-RIR: status, where to actually
create the object, source link). aspa_generate now takes an optional
rir param instead of a RIPE-specific maintainer handle.
Also corrected an overprecise APNIC launch date: APNIC's own blog
doesn't give one, only a vague "since Nov 2025" for all three live
RIRs collectively, which doesn't match RIPE's and ARIN's own more
precise announcements.
- Create local-db-client.js with consolidated database client module (11 functions)
- Refactor validateRPKIWithCache() to query local rpki_roas table (<10ms vs 1-2s external)
- Update /api/health endpoint to determine health from local DB statistics
- Update /api/prefix-detail endpoint to use async validateRPKIWithCache()
- Update /api/prefix-changes endpoint with RPKI status lookup from local DB
- Create /api/bgp endpoint with local BGP routes + threat intelligence lookup
- Add bgp_routes, rpki_roas, threat_intel statistics to health response
- Zero external API calls for RPKI/BGP validation queries
Impact: Sub-100ms latency for all lookups, 0 token spend on BGP/RPKI/threat intel
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
PeerCortex unifies PeeringDB, RIPE Stat, bgproutes.io, RIPE Atlas,
Route Views, IRR, RPKI, and CAIDA into a single AI-queryable MCP Server
for network engineers. Powered by local Ollama.
Core capabilities:
- 34 MCP tools for network intelligence
- 11 data sources unified
- ASPA validation engine (RFC 9582) with leak detection
- Peering partner discovery with AI-ranked matches
- BGP analysis and anomaly detection
- RPKI monitoring and compliance reports
- Latency/traceroute via RIPE Atlas
- Transit analysis and cost comparison
- IX traffic statistics
- AS topology mapping
- ASPA object generator and simulator
- 100% local AI — no cloud dependencies
Rene Fichtmueller