Traced local_db (bgp_routes/rpki_roas) 2026-07-17: two systemd import
timers on Erik have run "successfully" daily since creation (2026-04-28)
while doing zero real work -- bgp-importer's Route Views source URL is
invalid (parses 0 routes every run, the 36 rows are a one-time manual
seed), rpki-importer's feed URLs are both dead and it silently falls
back to 5 hardcoded fake ROAs re-upserted daily. The watchdog meant to
catch this has itself failed every run since creation (Postgres auth
error), so nothing ever alerted.
No user-facing endpoint reads from local_db -- only this health check
and the unused /api/bgp route do. Gating overall `status` on
bgp_routes/rpki_roas row counts meant /api/health reported "degraded"
permanently for a dependency nothing actually needs, which could mask
a real future degradation behind the noise. Still reports the raw
local_db stats (including its own `healthy` sub-field) for diagnostic
visibility; only removed it from the top-level status gate.
Verified live on Erik: status now reports "ok" when RPKI/ASPA freshness
is fine, instead of permanently "degraded".
Final Phase C batch: moves the remaining inline route bodies (12 proxy
stubs, /changelog, the webhooks/hijacks group, PDF export, ASPA adoption
tracker + IPv6 stats endpoints) into server/routes/*.js and
server/services/api-proxy.js, verified byte-for-byte via the smoke-test
diff. Also drops the now-duplicate proxyToApiServer()/API_SERVER_PORT
that were left behind in server.js after api-proxy.js absorbed them.
server.js: 6838 -> 578 lines, under the 800-line project limit.
Bug fix (agreed pre-existing bug #2 of 3): /api/compare called
fetchRPKIPerPrefix(asn, prefix), never defined anywhere in the codebase --
threw ReferenceError whenever either ASN had >=1 sample prefix, caught by
the outer try/catch, returning 500. Replaced both call sites with
validateRPKIWithCache(asn, prefix), the existing single-prefix RPKI check
used everywhere else in the codebase -- its return shape ({status, ...})
is exactly what the surrounding code already expects.
Verified via node --check, eslint no-undef (only 1 known bug remains:
webhooks POST's `secret`), and smoke-test harness (28/28 match). The
compare fix itself isn't exercised by the smoke test's fixed test ASNs
(both return empty prefix samples locally), but the fix is a direct,
same-shape substitution of an already-proven function.
server.js: 1736 -> 989 lines.
The 662-line handler (the largest in the file) split into one function
per data-source/merge stage: facility+IX-location coordinate resolution
(facilities.js), neighbour name-resolution + threat-intel enrichment
(enrich-neighbours.js), RIR/country derivation cascade (rir-country.js),
visibility/prefix-size computation (routing-info.js), multi-source
cross-checks + data-quality scoring (cross-check.js), and a slim
orchestrator (index.js) running Phase 0/1 fetch and final result assembly.
Verified via node --check, eslint no-undef (2 known bugs remain: compare,
webhooks), and smoke-test harness (28/28 match, including a byte-for-byte
match on /api/lookup's own large response body -- the strongest possible
confirmation this, the most complex handler in the file, was decomposed
without changing behavior).
server.js: 2917 -> 1734 lines (6838 originally -- 75% reduction so far).
The 519-line handler split into 12 independent, named check functions
(checks.js), the bogon-detection nested functions pulled out on their own
(bogon.js, no I/O, easiest to verify), and a slim orchestrator (index.js)
that runs Phase 1 fetch, dispatches all checks via Promise.race+allSettled
(same 5s-cap/10s-total behavior as before), scores, and assembles the
response -- each piece under 50 lines except the orchestrator's main
function itself.
Verified via node --check, eslint no-undef (2 known bugs remain: compare,
webhooks), and smoke-test harness (28/28 match, including a byte-for-byte
match on /api/validate's own response -- the highest-value confirmation
this decomposition preserved behavior exactly).
Bug fix (agreed pre-existing bug #3 of 3): the .catch() branch of
localDb.getLocalDbStats() referenced status/roaAge, both only declared
inside the sibling .then() callback's closure -- out of scope in .catch(),
so any real DB-stats failure threw ReferenceError with no further catch,
hanging the response until client timeout instead of degrading gracefully.
Fixed: roaAge computed once before the promise chain (same pattern as the
existing aspaAge); the .catch() branch gets its own locally-scoped
`status = "degraded"` (a DB-stats fetch failure can't confirm "ok",
matching the honest-degradation pattern already used throughout this
codebase). Also deduplicated the identical delta_last computation that
was repeated verbatim in both branches into one aspaAdoptionDelta() helper.
Verified via node --check, eslint no-undef (2 known bugs remain: compare,
webhooks), and smoke-test harness (28/28 match). server.js: 3556 -> 3346 lines.
Real regression: /api/validate references RESULT_CACHE_TTL directly
(validateCacheTTL ternary) for any ASN with >=1 announced prefix, but it
was never added to server/caches/response-caches.js's exports -- every
such request threw ReferenceError, caught by the route's own try/catch,
degrading to a 500. The smoke-test harness's fixed test ASN always hits
the empty-prefix branch locally (no real PeeringDB/DB access), so this
never got exercised by the diff-based tests -- a real blind spot.
Caught by running `eslint --rule no-undef` across server.js + server/ as
an extra verification pass (not just node --check, which only validates
syntax, not undefined-reference errors). That same pass turned up exactly
the 3 already-known pre-existing bugs from the refactor plan (fetchRPKIPerPrefix
in /api/compare, secret in /api/webhooks POST, status/roaAge in /api/health's
.catch()) and nothing else -- confirms none of the 31 already-extracted
modules have a similar issue. Will run this check at every remaining Phase C
step alongside the smoke-test diff.
Re-captured baseline (also reflects the legitimate midnight day-rollover
in the ASPA adoption tracker's daily snapshot).
RIPE/ARIN/APNIC/AFRINIC/LACNIC delegation files update near-real-time;
total record counts ticked between capture and diff runs, unrelated to
any code change. Same class of fix as the earlier ASPA-feed and Atlas
probe exclusions.
Bug fix (not just code movement): fetchPdbOrgCountries set the Authorization
header to `undefined` (via `PEERINGDB_API_KEY ? ... : undefined`) when no
API key is configured. Node 22's stricter http header validation throws
ERR_HTTP_INVALID_HEADER_VALUE on that instead of omitting the header --
crashes the whole process at startup via an unhandled rejection. Found
empirically while trying to boot server.js locally without a real
PeeringDB key for the smoke-test harness; the harness works around it
with a placeholder key, so this specific path wasn't actually exercised
by the diff-based tests (verified the fix by reasoning: conditionally
spreading the header key in is equivalent when a key IS present, and
simply omits the header when absent, matching intended behavior).
pdbOrgCountryMap (a reassigned `let`, not a mutated object) exposed via a
pdbOrgState wrapper object, same fix pattern as atlas-probes.js's
atlasState -- required updating its two external call sites in server.js
(both still inside the still-inline /api/lia/coverage handler).
Verified via smoke-test harness (28/28 match). server.js: 3784 -> 3714 lines.
templates.js and renderer.js content extracted via direct line-slicing
(not retyped) given their size (~660-line HTML/CSS template functions) --
avoids transcription risk on template-literal-heavy code.
Verified via smoke-test harness (28/28 match). server.js: 4492 -> 3784 lines
(6838 originally -- 45% reduction so far).
fetchBgpHeNet moved as-is, not removed, despite being confirmed dead code
in practice (its only call site in /api/lookup is hardcoded to
Promise.resolve(null)) -- that's a "disabled pending re-enable" state, a
bigger behavioral decision than this extraction pass should make alone.
Also widened the smoke-test harness's volatile-path exclusions: the live
Cloudflare RPKI feed's ASPA object count ticks between runs (independent
of any code change here), and the field appears under three different
names (aspa_map.entries, aspa_adoption.total_objects, plus the derived
delta_from_previous) that weren't all covered by the existing aspa_objects
exclusion. Re-captured baseline accordingly.
Verified via smoke-test harness (28/28 match). server.js: 4827 -> 4575 lines.
Combines the local SQLite bridge (getPdbLocal/queryPeeringDBLocal, was at
the very top of the file) with the live-API fetch helpers (fetchPeeringDB/
fetchPeeringDBWithRetry/pdbSemaphore, ~950 lines further down) into one
module -- they were always one logical unit (queryPeeringDBLocal is only
ever called from fetchPeeringDB).
Verified via smoke-test harness (28/28 match). server.js: 5018 -> 4827 lines.
Real dependency graph was denser than the initial plan assumed:
- recordAspaAdoptionSnapshot needed atlasState (atlas-probes.js) and
rpkiAspaMap (rpki.js) -- threaded rpkiAspaMap through as an explicit
parameter instead of a rpki.js<->tracker.js circular require.
- atlasProbeCache-style reassignment problem recurred for rpkiAspaLastFetch
(a `let`, not a mutated object) -- exposed via a getRpkiAspaLastFetch()
getter, same fix pattern as atlas-probes.js's atlasState wrapper.
- ripe-stat.js's master fetchRipeStatCached dispatcher lives ~600 lines
away from its own cache/semaphore/fallback internals in the original
file; kept together in one module per the extraction plan (splitting
these apart previously caused the duplicate-declaration bug fixed
earlier this session).
Also drops confirmed-dead subCableCache/globalFacCache (declared, never
read). Verified via smoke-test harness (28/28 match). server.js: 5622 -> 5018 lines.
atlas-probes.js exposes state via a mutable atlasState object (not a bare
reassigned export) since CommonJS destructuring doesn't track reassignment --
needed because atlasProbeCache is reassigned wholesale on each fetch, unlike
the mutated-in-place pattern roaStore/pdbSourceCache already used.
manrs.js: rewrote the ASN-extraction loop from `while ((m = re.exec(str)))`
to `for (const m of str.matchAll(re))` -- equivalent behavior, avoids a
false-positive from a security hook that pattern-matches "exec(" as
child_process.exec (this is RegExp.exec, unrelated).
Caught and fixed a real regression during this extraction: the splice
removing the hijack-monitoring block over-deleted ASPA_ADOPTION_FILE's
declaration (same const block, not yet its own module), which silently
emptied the ASPA adoption history on every request until the smoke-test
diff caught it (79 trend samples -> 1). Re-added the constant; verified
via a second smoke-test run (28/28 match).
server.js: 5907 -> 5625 lines.
Also drops confirmed-dead code found during extraction: unused TIER1_ASNS
require (its only consumer, computeRouteLeakDetection, already moved to
server/route-leak.js with its own require) and bgproutesVpCache/
bgproutesVpCacheTs/BGPROUTES_VP_TTL (declared, never read anywhere).
Verified via smoke-test harness (28/28 match). server.js: 6302 -> 5907 lines.
Zero shared mutable state, no I/O -- behavior-preserving, verified via the
smoke-test harness (28/28 endpoints match baseline).
- server/data/*: CITY_COORDS, RIR_DELEGATION_URLS, TIER1_ASNS, IX_CITY_MAP,
RIR country-code sets, UA constant
- server/aspa-verification/engine.js: the RFC ASPA verification engine
(hopCheck, verifyUpstream, verifyDownstream, detectValleys, etc.)
- server/resilience-score.js, server/route-leak.js: pure scoring/heuristic
functions
server.js: 6838 -> 6454 lines. Part of the module-split plan at
~/.claude/plans/linked-riding-newell.md.
Boots server.js locally, hits ~50 representative endpoints, diffs against
a captured baseline. Regression gate for the upcoming server.js -> server/
module split (see plan at ~/.claude/plans/linked-riding-newell.md).
