fix(backend): cap repo-index metric cardinality and add OOM diagnostics - #1526
fix(backend): cap repo-index metric cardinality and add OOM diagnostics#1526better-stack-hq[bot] wants to merge 1 commit into
Conversation
Add Node's --heapsnapshot-near-heap-limit to the backend program so a heap snapshot is captured to a persisted directory if the process approaches its heap limit again, giving a definitive object graph instead of only a generic "JavaScript heap out of memory" crash log. Also fix an identified metric-cardinality leak: repo-scoped Prometheus metrics keyed by repo name are never cleared, so every distinct repo name observed over the process's lifetime stays resident in the registry forever, even after the repo is deleted. Remove those time series once a repo is actually deleted.
|
@better-stack-hq[bot] your pull request is missing a changelog! |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 92ff09a. Configure here.
| // The repo no longer exists, so drop its time series from the metrics | ||
| // registry rather than retaining them for the lifetime of the process. | ||
| this.promClient.removeRepoMetrics(repo.name); | ||
|
|
There was a problem hiding this comment.
Metrics recreated after cleanup
Medium Severity
removeRepoMetrics runs before the shared success-path activeRepoIndexJobs.dec and repoIndexJobSuccessTotal.inc, which re-create cleanup series for the deleted repo. That leaves residual cardinality and can leave the active gauge at -1 after remove resets the prior value.
Reviewed by Cursor Bugbot for commit 92ff09a. Configure here.
💬 AI SRE · 👥 Sourcebot team · 👤 Requested by Brendan Kellam
Summary
--heapsnapshot-near-heap-limit=3(writing to a persistedDATA_CACHE_DIR/heap-snapshotsdirectory) to thebackendsupervisord program, so if the process approaches its heap limit again, we get an actual heap snapshot to diagnose the retained object graph — instead of only the genericFATAL ERROR: JavaScript heap out of memorycrash log.PromClient's repo-scoped metrics (activeRepoIndexJobs,pendingRepoIndexJobs,repoIndexJob{Reattempts,Fail,Success}Total) are keyed by repo name but are never cleared. Every distinct repo name ever observed over the process's lifetime stays resident in theprom-clientregistry forever, even after the repo is deleted. AddPromClient.removeRepoMetrics()and call it once a repo is actually deleted inRepoIndexManager's cleanup job.Root cause
app.sourcebot.dev'sbackendprocess (node packages/backend/dist/index.js) hitFATAL ERROR: Ineffective mark-compacts near heap limit ... JavaScript heap out of memoryafter ~20.7h of uptime, causing a ~48s window of 503s while supervisord restarted the container.Static review of the backend surfaced one confirmed correctness bug that contributes to unbounded heap growth over the process's lifetime (unbounded Prometheus label cardinality for repo names that are never released). Given the repo count in this deployment (~150-160, fairly stable), this alone may not fully account for the ~4GB heap exhausted, so this PR also adds heap-snapshot-on-near-limit diagnostics to get a definitive answer if this recurs, rather than guessing further from static analysis alone.
Test plan
yarn workspace @sourcebot/backend buildandyarn workspace @sourcebot/backend testin CI to confirm the TypeScript changes compile.supervisord status backendpicks up the newNODE_OPTIONSand that$DATA_CACHE_DIR/heap-snapshotsexists after container start./metricsno longer shows stalerepo="<deleted-repo>"series after a repo is deleted and its CLEANUP job runs.🤖 Generated with Better Stack AI SRE
Note
Medium Risk
Changes long-running backend process behavior (metrics lifecycle and NODE_OPTIONS) in response to production OOMs; logic is narrow but affects observability and memory under load.
Overview
Addresses backend heap growth and OOM crashes by bounding Prometheus memory and adding post-mortem heap dumps.
Metric cleanup: Adds
PromClient.removeRepoMetrics()to drop all repo-labeled index/cleanup time series when a repo is permanently removed, and invokes it from the CLEANUP job path right after the DB delete. This stopsprom-clientfrom retaining every historicalrepolabel combination for the life of the process.OOM diagnostics: Ensures
$DATA_CACHE_DIR/heap-snapshotsexists at container start and configures the backend supervisord program withNODE_OPTIONS(--heapsnapshot-near-heap-limit=3and--diagnostic-dirpointing at that directory) so near-limit heap pressure writes snapshots for analysis instead of only a generic OOM log.Reviewed by Cursor Bugbot for commit 92ff09a. Bugbot is set up for automated code reviews on this repo. Configure here.