Repository navigation
ci(sanitizers): link the ASan lane against a shared core so it fits on disk. Principle VIII. - #6136
Conversation
…n disk. Principle VIII. The weekly ASan + UBSan lane has failed at Build for three weeks running with "final link failed: No space left on device", so it has tested nothing. Each test statically links the whole core: a Debug ASan test averages ~200 MB, and even with -g1 and compressed debug sections the tree measured 127 GB locally against ~83 GB free on the runner. New option AETHER_SHARED_CORE (default OFF; not on Windows) builds libaethercore shared with position-independent code. The ASan entry turns it on: the full local Debug ASan tree is 17 GB, all 591 tests link, and test executables average 13 MB. Running the whole suite locally under ASan gave the same failures with the shared core as with static linking, so results are unchanged. On the runner (run 37141490698) the build used 19 GB of disk and the full suite ran: 618 of 623 tests passed. TSan, release and ordinary builds keep the static core. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Issue fit
No fixes #NNNN — there is no issue for the three-week ASan Build outage (the author filed #6130 for the CodeQL twin of the same disk problem, and #6135 fixes that one). Reviewed against the PR body's own stated intent, which is specific and falsifiable. GOVERNANCE.md does not want an RFC here: this is a CI build-configuration fix with a named root cause, default OFF, no product surface. It does what it says — the lane dies at final link failed: No space left on device because ~250 Debug+ASan test executables each statically absorb the whole core, and one shared core replaces those copies with one 668 MB .so.
One verification I could do from here: validation run 37141490698 exists, and its TSan job failed at Configure with every downstream step skipped — exactly as the body describes, and consistent with the stale :tsan image story (#6123). I could not independently confirm the ASan job's 618/623 or the df numbers; get_ci_run_log returns only the first failing matrix job. The local 127 GB → 17 GB table is likewise unreproducible from here. Everything else below is reasoned from the code in /tmp/aetherclaude/pr-6136, not observed at runtime.
Scope
| File / group | Change | Claimed by body? | Verdict |
|---|---|---|---|
CMakeLists.txt 146–162 |
New AETHER_SHARED_CORE option (default OFF), PIC, Windows FATAL_ERROR |
Yes | In scope |
CMakeLists.txt 1594 |
add_library(aethercore ${_aether_core_library_type}) |
Yes | In scope |
sanitizers.yml 39–42, 70 |
shared_core matrix key on both entries |
Yes | In scope |
sanitizers.yml 130 |
-DAETHER_SHARED_CORE=... on the configure line |
Yes | In scope |
sanitizers.yml 150–152 |
df -h / around the build |
Yes ("so disk headroom is visible in the log") | In scope |
Everything in the diff is explained by the stated fix. No deleted guards, no new product/protocol surface, no CHANGELOG.md entry, no UI or default-behavior change — §6 preference check is not engaged. Two files, both build plumbing.
Blockers
None.
Nits (non-blocking)
- The "~600" in both new comments overstates it.
tests/tests.cmakedeclares 617 executables; 247 linkaethercoredirectly and 10 more reach it throughaetherdesktop_support— 257, not ~600. ~600 is the test count (623 ran). The disk arithmetic still works at 257 × ~200 MB plus objects and the apps, but the comments explain why, and the why is off by 2.3×. The body's "Tests that link: all 591" doesn't match either number and is probably "591 targets built". Inline onCMakeLists.txt:146andsanitizers.yml:39. - The post-build
df -h /never runs in the case it exists for.cmake --buildfailing underbash -eaborts the step, so a future disk exhaustion still leaves only the pre-build reading. Suggestion fence inline atsanitizers.yml:150. - No PR-gating canary for the new option.
static-checks.yml's "Sanitizer option configures" job already proves-DAETHERSDR_SANITIZER=...configures and refuses correctly; a fifth case asserting-DAETHER_SHARED_CORE=ONconfigures and reports theSHAREDstatus line would catch a future regression on the PR instead of on Saturday. Inline atsanitizers.yml:130. - The silence this PR's root cause exposed is still there. Every reporting step in this workflow is gated on
steps.ctest.outputs.exit != '0'(lines 239, 302, 312, 397). A Build failure skips all of them — no artifact, no sticky issue — which is precisely how the lane went dark for three weeks, and the run-37141490698 TSan job above is a live example of the same shape. This PR removes one cause; the gate's own comment at lines 77–85 already names this failure class for the container case. Sibling of the root cause, not scope creep — worth a follow-up issue rather than code in this PR. install(TARGETS AetherSDR …)at line 3319 does not installlibaethercore.so, soAETHER_SHARED_CORE=ON+installyields a broken tree. The comment already says "sanitizer CI only" and nothing in the release path sets it, so this is documentation-only today.- Why does TSan stay
OFF? Its-g3RelWithDebInfo footprint is largely optimization-independent and is currently unmeasured — the validation run's TSan job died at Configure. Genuine question, not an assertion that it should flip.
What I tried to break
- Duplicate core state under ELF. The real hazard of static→shared is a test that compiles a core
.cppand linksaethercore: under a static archive the member is simply not pulled (one copy), but under a shared lib the executable's definitions are not in.dynsymwithout-rdynamic, so the.sokeeps its own copy and singleton state forks. Parsed all 617 targets intests.cmake: 190 compilesrc/core/src/modelsources, 257 link the core, and the intersection is empty. Same for the 76${AETHER_SETTINGS_SOURCES}consumers — none of them linksaethercore. Survived, and it survives by construction, not by luck. - PIC on the vendored static libraries.
set(CMAKE_POSITION_INDEPENDENT_CODE ON)lands at line 157, and nothing before line 146 creates a target (grepforadd_library/add_subdirectory/ExternalProject/FetchContentabove that line is empty), so every in-tree static lib inherits it. The one escapee is the OpusExternalProject, which a separate CMake project would not inherit — butthird_party/radae/cmake/BuildOpus.cmake:20already passes-DCMAKE_POSITION_INDEPENDENT_CODE=ONexplicitly, andthird_party/wdsp/CMakeLists.txt:26setsPOSITION_INDEPENDENT_CODE ONon its own target. NoR_X86_64_32-against-a-shared-object trap. - PRIVATE dependency propagation. With
STATIC,target_link_libraries(aethercore PRIVATE aether::wdsp)still propagates as$<LINK_ONLY:>to consumers; withSHAREDit does not, andaether_wdspcarriesC_VISIBILITY_PRESET hidden, so WDSP symbols are not exported from the.so. Any test calling a WDSP entry point while linking onlyaethercorewould link today and fail only in the ASan lane. Comment-stripped scan of the three candidate tests (wdsp_channel_test,hl2_cw_bfo_test,anan_rxdsp_handedness_test) — every apparentSetRXAMode/fexchange2hit is prose; all three go through thecore/dsp/WdspChannel.hwrapper. Survived, but it is a latent asymmetry worth knowing about when the next DSP test lands. - Symbol export and RPATH. No
VISIBILITY_PRESET/-fvisibility=hidden, no--whole-archive, no$<TARGET_OBJECTS:aethercore>, and noCMAKE_SKIP_BUILD_RPATH/CMAKE_BUILD_WITH_INSTALL_RPATHanywhere in the root listfile,tests.cmakeor.github/— so default visibility holds and CMake's automatic build-tree RPATH lets ctest find the.sowithout anLD_LIBRARY_PATHdance. - Static-initializer topology. A static archive drops unreferenced objects; a shared core loads every TU and runs every static initializer, so the lane now exercises a link topology the shipped app never uses.
grepfor__attribute__((constructor))/Q_CONSTRUCTOR_FUNCTION/Q_COREAPP_STARTUP_FUNCTIONinsrc/corereturns nothing, so there is no self-registration layer to change behaviour — and ASan'sdetect_odr_violation(left at its default inruntime_env) apparently found nothing either, since the runner run reached 618/623. Flagging only that a future sanitizer finding in a load-time path may not reproduce in the static-linked product. - Matrix completeness. Both entries carry
shared_core; a future entry omitting it yields an empty-DAETHER_SHARED_CORE=, which is falsey and lands onSTATIC— degrades safely. - Checked the head SHA's seven checks: all green. None of them configures with
AETHER_SHARED_CORE=ON, which is the basis for the canary nit above.
Recommendation
Approve with nits. The change is small, default-OFF, refused at configure on Windows, and the one code path it touches is guarded by a variable that only this workflow sets; the measured before/after and the dispatched runner run are the kind of evidence Principle VIII asks for. The duplicate-state and PIC hazards that would have made static→shared unsafe are both absent by construction, which I checked rather than assumed. Next step: fix the "~600" counts in the two comments, make the post-build df unconditional, and — separately from this PR — file the follow-up for the Build-failure reporting gap so the lane cannot go dark silently again. Thanks for measuring this instead of guessing at it.
🤖 aethersdr-agent · cost: $5.2600 · model: claude-opus-5
ten9876
left a comment
There was a problem hiding this comment.
Issue fit
Approve with nits. No linked issue; reviewed against the stated ASan build disk-exhaustion problem. The two hunks implement the described solution: an opt-in shared core with PIC, selected only by the ASan workflow entry, and disk measurements around the build. GOVERNANCE.md explicitly exempts “Build / CI fixes” from the RFC requirement. No product behavior or visual preference changes are bundled.
I independently retrieved the ASan job log for validation run 37141490698. It confirms the load-bearing claim: the shared core linked, Build finished, disk retained 64 GB free, and CTest reached its final report. This is narrower than proving the entire suite clean or proving static/shared behavioral equivalence.
Scope
| File/group | Change | Claimed in body? | Verdict |
|---|---|---|---|
CMakeLists.txt:146–162 |
Default-OFF AETHER_SHARED_CORE, Windows refusal, PIC and library-type selection |
Yes | In scope; opt-in build configuration, no shipping default change |
CMakeLists.txt:1594 |
Create core using selected library type | Yes | In scope |
.github/workflows/sanitizers.yml:39–42,70,130 |
ASan ON / TSan OFF and configure argument | Yes | In scope |
.github/workflows/sanitizers.yml:150–152 |
Before/after disk readings | Yes | In scope |
Everything in the diff is explained by the stated fix. The single commit's authored date matches this work; no unrelated older commit, removed safety guard, new runtime protocol surface, baseline relaxation, or CHANGELOG entry appears. No scope checklist was supplied; the body's scope statements otherwise hold.
Blockers
None identified.
Nits — non-blocking
CMakeLists.txt:146–149and.github/workflows/sanitizers.yml:39–41: describe the linked-core subset accurately. “Every test links the whole core” / “~600 full copies” conflates the suite size with the number of core consumers. A fresh configure of this head generated 609 C++ executable links, of which 254 depend onlibaethercore.so(including the applications). Many tests compile small source subsets directly. This does not undermine the measured disk fix, but the permanent explanation should say that many tests duplicate the static core. Suggested replacement for the first comment:
# Share libaethercore among its test consumers to reduce sanitizer build disk
# usage. This is an opt-in sanitizer CI configuration; shipped builds keep the
# static core..github/workflows/sanitizers.yml:150–152: preserve the final disk reading on failure. The step's fail-fast shell exits on a failedcmake --build, skipping the finaldf; that is when this diagnostic is most useful. A shell EXIT trap preserves the build's failure status and measures either outcome:
run: |
trap 'df -h /' EXIT
df -h /
cmake --build build --parallelWhat I tried to break / verification
-
Disk and completion claim — independently verified from the runner log. The job configured
libaethercore built SHARED, linkedlibaethercore.so, and emitted:overlay 145G 62G 83G 43% / overlay 145G 81G 64G 57% / 99% tests passed, 5 tests failed out of 623The five failures match the body:
hl2_backend_seam_test,weather_radar_loading_test,offline_health_registry_test,rx_applet_squelch_reconciliation_test, andvfo_dsp_launcher_width_test. I did not independently reproduce each on a static merge-base build, so their asserted pre-existing status and the local static/shared equivalence remain unverified, not newly attributed PR regressions. The log includes skipped cases; 618/623 means non-failing CTest cases, not 618 independently observed executed passes. -
Opt-in isolation — configured current head
8d4bff35flocally in a scratch worktree. Fresh DebugAETHER_SHARED_CORE=ONgeneratesCXX_SHARED_LIBRARY_LINKERforlibaethercore.so; a fresh ordinary Release configure without the option generatesCXX_STATIC_LIBRARY_LINKERforlibaethercore.a. Inspected all 292 generated core source compilation rules: all carry-fPIC. No generated executable both links the shared core and directly compilessrc/core/src/modelsobjects, avoiding the obvious duplicated-state trap. Vendored Opus already explicitly passes PIC into its ExternalProject; WDSP independently enables PIC. Reviewed private link dependencies and default symbol visibility; the runner's completed full link graph supplies empirical coverage of the enabled CI configuration's dependencies. -
TSan isolation — independently verified from its job log. It passed
-DAETHER_SHARED_CORE=OFFand failed configuration withQt 6.8.3 found, but AetherSDR requires Qt 6.12 or newer. This PR does not claim to repair that image. Windows ON is explicitly refused before dependencies are configured; I inspected that guard but did not run a Windows configure. -
Evidence freshness. Run 37141490698 used
df503b7f, before the rebase to current head8d4bff35f. The shared-core and sanitizer-workflow changes are identical between those revisions; the root CMake difference addsPeripheralAuthStore.cppto desktop support. Current-head PR checks are green but began at 2026-10-03 22:30:31 UTC, before the inspectedmaintip (3e4b14e2, 2026-10-04 01:13:20 UTC). They do not prove today's merge result;strict: falsedoes not force a refresh. The sanitizer validation is aworkflow_dispatchbranch run, not a current-main merge run. Local 127 GB/17 GB figures and full static/shared result equivalence were not recreated. -
Test boundary and runtime scope. The full diff modifies no test source or registration, so no new/modified socket-owning target required execution. I ran configuration and inspected generated build graphs; I did not execute the suite, create a synthetic firmware peer, or drive the GUI. This is a build/CI-only change with direct runner evidence, so no demo instance was needed or launched. No separate automated code-review skill was available; the existing bot review was checked against source/generated graphs and logs rather than adopted wholesale.
Recommendation
Approve with nits, subject to the normal infrastructure-owner review and required checks. The actual hosted-runner log establishes that the change repairs the reported failure to reach tests, and current-head configuration confirms that ordinary builds remain static. Correct the comments and preserve failure-time disk diagnostics; the next sanitizer run on merged main should confirm the rebased/current integration. This review does not assert that the five failing tests are harmless or that the sanitizer suite is green.
… check the option per PR. Principle VIII. Review on #6136: - The comments said ~600 tests link the core. About 250 executables do; ~600 is the test count. - The after-build df ran only when the build succeeded. It now runs from an EXIT trap, so a disk-full failure still logs it. - Record that WDSP symbols don't reach the shared core's consumers. - static-checks' 'Sanitizer option configures' job gains a fifth case: ON must generate libaethercore.so and the default must stay .a. Without it the option's first exercise is the weekly cron. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… check the option per PR. Principle VIII. Review on #6136: - The comments said ~600 tests link the core. About 250 executables do; ~600 is the test count. - The after-build df ran only when the build succeeded. The step now keeps the build status and exits with it after the second df, so a disk-full failure still logs it. - Record that WDSP symbols don't reach the shared core's consumers. - static-checks' 'Sanitizer option configures' job gains a fifth case: ON must generate libaethercore.so and the default must stay .a. Without it the option's first exercise is the weekly cron. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
acb5d55 to
fde2048
Compare
… Principle VIII. Every report step was gated on ctest's exit code, so a Configure or Build failure skipped them all: no artifact, no sticky issue. That is how the ASan lane went dark for three weeks. A new step runs only when Configure or Build failed. It writes a report naming the failed step, with a stable per-step fingerprint, and the artifact and tracking-issue steps now also run on that path. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
Review-body items, current head
|
TSan red on
|
| check | result |
|---|---|
| Static checks | ✅ (includes your new fifth "Sanitizer option configures" case) |
| Sanitizer option configures | ✅ |
| Analyze (python / js-ts) | ✅ |
| build / check-macos / check-windows | ⏳ still running |
| ASan + UBSan | ⏳ still queued — no result yet on this head |
Why Configure failed
I can't quote this run's Configure log — the job-log API isn't reachable from here, and get_ci_run_log returns step outcomes only — so this is inference, with its sources named rather than a log line I didn't read:
- The lane pulls
ghcr.io/aethersdr/aethersdr-ci:tsan(sanitizers.yml:66). The Confirm the sanitizer image step (sanitizers.yml:87) passed, and that step only checks thatlibQt6Core.so.6exists and is TSan-instrumented — not its version. A stale-but-instrumented image sails through it and then trips the Qt floor in Configure (cmake/AetherQtPin.cmake, viaCMakeLists.txt:285: "Qt … found, but AetherSDR requires Qt 6.12 or newer"). - Your earlier dispatch 37141490698 died at exactly this step with
Qt 6.8.3 found, but AetherSDR requires Qt 6.12— you documented it in the PR body. - fix(ci): build TSan QtMultimedia with uninstrumented host tools #6123's commit message (
b255830) names the same cause: theDockerfile.tsanQtMultimedia build hung at[2/300], a failed build never publishes, so:tsanhas been stuck on Qt 6.8.3.
Why merging #6123 didn't help yet — and what to do
#6123 landed on main as b255830 at 03:06 UTC. This dispatch started roughly a minute later. The image still has to be rebuilt and published by docker-ci-image.yml (which fires on a main push touching .github/docker/Dockerfile.tsan), and that job — build-tsan, run 37173106228 — is still in_progress as I write this. The runner pulled :tsan long before it could publish.
So:
- No rebase needed. The workflow references the image by tag, so the PR's content has no bearing on it — and TSan keeps
shared_core: "OFF"anyway. - Wait for
build-tsanonb255830to publish, then re-dispatchSanitizerson this branch. Pure infrastructure timing.
One thing from this run worth a guard
The good news first: Report a lane that did not build → Upload sanitizer artifacts → Open or update tracking issue all ran and succeeded. That's precisely the dark-lane hole e9255ef set out to close, working on its first real Configure failure. 👍
The side effect: it filed #6144, "[sanitizer] TSan weekly run failure (fp 8c86ae9408dd)", labelled priority: high / awaiting-response, from a PR-branch dispatch, for a stale image. Neither sanitizers.yml:328 nor :338 carries a github.event_name or github.ref condition, so any dispatch from any branch that fails to build now opens or updates a sticky titled "weekly run failure". Two small things you might fold in:
- gate the artifact and tracking-issue steps on
github.event_name == 'schedule' || github.ref == 'refs/heads/main', so a validation dispatch reports in its own log and leaves the issue tracker alone; - the fingerprint is per-step, so a future genuine Configure break lands on [sanitizer] TSan weekly run failure (fp 8c86ae9408dd) #6144 rather than opening its own — worth noting next to the fingerprint comment, given the sticky-reuse pattern we've been bitten by before.
Either way #6144 can just be closed once the image republishes.
On the review thread
Nothing in the bot comments bears on this failure. The count correction (257 core-linking targets of 617, not ~600) and the rc=0; … || rc=$?; df -h /; exit "$rc" fix are both in fde2048, and the canary suggestion is already implemented as the fifth static-checks.yml case — which is green here.
Nice piece of work, by the way: 127 GB → 17 GB, and the reporting fix means the next three weeks of a dark lane can't happen silently. Thanks for digging into both halves of it.
🤖 aethersdr-agent · cost: $6.3627 · model: claude-opus-5
… VIII. (#6152) ## Problem **Full Suite on `main` is red.** Run [37181657060](https://github.com/aethersdr/AetherSDR/actions/runs/37181657060) on `a325fb99e` (#5924) died at link 4934/4976 with `final link failed: No space left on device`. The previous commit's run was green, so the tree is right at the runner's limit. - **ci.yml's required `build` job builds the same tree.** It only survives because #5924 gave it a step that frees 18 GB of host toolchains (84 → 102 GB free). That's about 50 more core-linking tests of headroom before the PR gate hits the same wall. - **The TSan lane ran out of disk too.** With the Qt 6.12 `:tsan` image from #6123 it got far enough to build for the first time, then filled the runner: `overlay 145G 145G 163M 100%` at link 4438/4854 (run [37176807431](https://github.com/aethersdr/AetherSDR/actions/runs/37176807431), #6148). - **The red Full Suite run filed no issue.** When Build fails, ctest is skipped, and a skipped step's output reads as `null`. GitHub's expression `exit != '0'` coerces `null` to 0, so every report step was skipped. Underneath all three: ~250 test executables each link a static `aethercore` and carry their own copy of its debug info. ## Fix - **`ci.yml` build job + `full-suite.yml`:** configure RelWithDebInfo at `-g1` (line tables only), via `CMAKE_{C,CXX}_FLAGS_RELWITHDEBINFO`. Backtraces keep file:line; only debugger-level locals and types are dropped. Both jobs change together because Full Suite restores the build job's ccache, so the first `main` run after merge builds cold and the next is warm again. - **`full-suite.yml`:** the same host-toolchain mounts and "Free runner disk" step as ci.yml's build job, and a `df` after the build, including when it fails. - **`full-suite.yml`:** a "Report a suite that did not build" step runs only when Configure or Build failed. It writes the report with a stable per-step fingerprint, and the artifact and tracking-issue steps also run on that path. Same pattern as #6136. - **`sanitizers.yml`:** TSan's `shared_core` goes to `"ON"`, the change the ASan lane proved in #6136. ## Validation Measured locally (GCC, this tree, `aethercore` plus four core-linking tests): | | `-g` (today) | `-gsplit-dwarf` | **`-g1`** | |---|---|---|---| | core-linking test executable | 325 MB | 214 MB | **120 MB** | | `libaethercore.a` | 496 MB | 165 MB | **114 MB** | At ~250 core-linking executables, that's roughly 50 GB off a tree that needs a little over 84 GB. `actionlint` is clean on `full-suite.yml` and `sanitizers.yml`. It reports two errors in `ci.yml` (`runner.workspace`, lines 451 and 841), which were already on `main` and are untouched here. **On the runner:** - **Full Suite**, branch run [37208804048](https://github.com/aethersdr/AetherSDR/actions/runs/37208804048) on this head, green: - Free runner disk: 84 → 102 GB free. - The build tree used **32 GB** (44 → 76 GB used), leaving 69 GB free. Before, it needed more than 84 GB. - **653/653 tests passed** (13.5 min). - The build took 46 min at a 52% ccache hit rate, because no cached `-g1` objects exist yet. The first `main` build after merge saves them, and later runs are warm again. - **ci.yml `build`**, the required PR check on this head: green, so the `-g1` configure works on the build job. - **Sanitizers**, branch run [37208806037](https://github.com/aethersdr/AetherSDR/actions/runs/37208806037): still running at merge time. ASan finished building and is running tests; TSan is still building on the shared core. The result is reported on this PR after merge. If TSan then fails, it is a sanitizer-lane problem, separate from the Full Suite and `build` fixes proven above. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…lane, incl. two WDSP races. Principle VIII. (#6173) ## Summary Refs #6154, #6156. This doesn't close them: the sanitizer-slowdown timeouts are still open (see below). The weekly Sanitizers lane recently started building the full test tree again (#6136, #6152). Its first complete run on `main`, [37208806037](https://github.com/aethersdr/AetherSDR/actions/runs/37208806037), failed 6 tests under ASan + UBSan and 11 under TSan. This PR fixes every one of those failures that was an actual sanitizer report. **In the branch run [37230031178](https://github.com/aethersdr/AetherSDR/actions/runs/37230031178), neither lane reports a single sanitizer error.** What still fails there is timeouts, plus one plain assertion in `rtl_capture_worker_test` (below). Two of the eight commits change shipped DSP behaviour. They are vendored WDSP patches, documented in `third_party/wdsp/AETHERSDR-PATCHES.md`: - **Patch 16, transmit path.** `SetTXABandpassFreqs()` was the only one of four `TXASetupBPFilters()` callers that didn't take `csDSP`. `TXASetupBPFilters()` clears `bp1`/`bp2 run` first and re-enables them only after the FIR redesign. So a TX filter change while transmitting with COMP (and CESSB) on could run blocks with the post-compressor bandpass bypassed, which is out-of-band splatter. Fix: take the lock, as `SetTXAMode()` does. It adds no new lock order, and costs one FIR redesign stall per TX filter change, which upstream already accepts for a mode change. - **Patch 17, panadapter analyzer.** 1. `stop` and `end_dispatcher` are now atomic. 2. `input_busy` is released *after* `stitch()`, so the next frame's FFT workers can no longer overwrite results while the stitch is still copying them (a torn spectrum frame). 3. `sendbuf()` now hands off `IQout_index` inside `BufferControlSection`, closing a lost update. Only part 2 changes timing: the next frame's FFTs start one stitch later, tens of microseconds. | Commit | Lane | Finding | Fix | |---|---|---|---| | `1f9c4ed66` | ASan | `offline_health_registry_test`: stack-use-after-scope. Section 6 reuses a factory that captured section 5's block-scoped `RecorderState` by reference | `rec` moves to `main()` scope (test only) | | `cacf9fe30` + `49fa9a75a` | UBSan | `rx_applet_squelch_reconciliation`, `vfo_dsp_launcher_width`, `fm_filter_controls`: invalid `QPointer<QLabel>` downcast in `VfoWidget::eventFilter` during the collapsed label's teardown | Remove the filter before `~VfoWidget` deletes the label, **and** test the event type before the guarded-pointer comparison. The second covers the label dying *first*, after `beginDirectEntry()` `raise()`s the VFO (found in review). Regression: `gui_nested_lifetime_test` `vfoCollapsedLabelTeardownInEitherOrder`, covering both child orders | | `01d06c6d1` | TSan | `receive_intent_routing`, `tci_automation`, `tci_server_review`: lock-order inversion between two of Qt's own TLS-backend mutexes, both taken on the main thread | Anchored `deadlock:QSslSocketPrivate::tlsBackendInUse` suppression | | `f04a9ffb2` | TSan | `tci_server_review`: unlocked global `QStringList` log sink written from worker threads | One mutex for the sink and its lists (test only) | | `6f6e4a5fd` | TSan | `tx_operation_integration`: the command sink appends on the `RadioConnection` thread to a list destroyed before that thread is joined | Declare the list before `radio` (test only) | | `fbbd24087` | TSan | `wdsp_channel`: WDSP patch 16 (above) | Lock in `SetTXABandpassFreqs` | | `af9c15224` | TSan | `anan_rxdsp_handedness`, `spectrum_sequence_gap`, `wdsp_process_tally`: WDSP patch 17 (above) | Three analyzer hand-offs synchronised | | `31e249c3e` | TSan | `weather_radar_loading`: Qt's `QHttpThreadDelegate` teardown vs a main-thread reply delete. Three map tests were fetching live OSM tiles | Stub network in those three tests, plus an anchored `race:QHttpThreadDelegate::~QHttpThreadDelegate` suppression | Each suppression in `tests/sanitizers/tsan.supp` carries its reasoning and a caveat. Neither is anchored on a frame AetherSDR code runs inside. The file's DO-NOT-ADD list is untouched. Note that a suppression hides **every** stack it matches, not only the reported cycle or teardown. So zero TSan reports with these entries means no *unsuppressed* race was seen. It is not proof that every Qt TLS or HTTP-teardown interaction is race-free. **Still failing after this PR, tracked in #6154 / #6156:** - **Timeouts under the sanitizers' slowdown, with no sanitizer report.** ASan: `rtl_spectrum_averaging`, `weather_radar_loading`. TSan: the same two plus `audio_engine_rates`, `nr_stereo_independence`, `peripheral_auth_dialog`. - **`rtl_capture_worker_test`, TSan lane only:** fails the assertion "DC mode change discards old estimates and shows the actual first corrected window", with no race reported. This branch touches no RTL code, and the test passed in the same run's ASan lane. It passed in both lanes this morning, before `main` gained #5919's RTL changes. **Leak fixes (added after the first push, found by a full local LeakSanitizer sweep):** | Commit | Leak | Fix | |---|---|---| | `6121f6ad6` | `RadioModel::disconnectClientHandlesThen`: the `step` lambda captured its own `shared_ptr`, a cycle that leaked on every client-disconnect chain | Capture a `weak_ptr`; the in-flight `sendCmd` callback holds the strong reference | | `7577bab41` + `7f37cf045` | WDSP **patch 18**: `destroy_calcc()` never closed the five PureSignal `SemsPSCorr` semaphores. 86 leak blocks across 17 tests (every TX channel destroy) | Close them, and the exit event, only once the correction thread has signalled exit; on the 500 ms timeout they are not freed under a live thread. **This does not make the timeout path safe:** `destroy_calcc()` still frees `a`, which a late thread would use. That hazard predates this PR and is now #6179 | | `192e8f09a` | `phone_tx_filter_numeric_entry_test`: `sendEvent(…, new QFocusEvent)` leaked, since `sendEvent` never takes ownership | Stack event (test only) | **Why CI never reported these leaks:** the ASan lane's `ASAN_OPTIONS` include `halt_on_error=0`, and with that, a leak doesn't change a test's exit code. The test "passes" and `ctest --output-on-failure` throws its output away. Verified with `phone_tx_filter_numeric_entry_test`: it exits 0 with CI's options and 1 with plain `detect_leaks=1`. Fixing that is a `sanitizers.yml` change and is **not** in this PR. **Full local sweep (all 653 tests, leaks counted as failures):** 26 failed before these three commits. After them, the only leak left anywhere is whisper.cpp's ggml Vulkan backend registry, a process-lifetime static that only appears on a machine with a Vulkan GPU. Not ours, and CI has no GPU. The other non-leak failures are: the two known timeouts; `connection_panel_size` and `local_control_server`, which fail only on this machine with no sanitizer report and pass in CI; and `rtl_receive_pipeline` and `cw_pcm_consumer`, which fail intermittently (1 run in 3 serially, and under parallel load, respectively) with no sanitizer report. Those are flakes already on `main`, out of scope here. ## Constitution principle honored Principle VIII: every finding was reproduced before it was fixed. Each fix that could be checked locally was shown to fail without it and pass with it; the TSan-only fixes are proven by the branch dispatch. The two WDSP changes follow the vendoring rules: minimal, commented in place, and documented in `AETHERSDR-PATCHES.md` with what to keep on the next WDSP update. ## Test plan - [x] Local build passes: full tree, Debug, with CI's ASan + UBSan flags and `AETHER_SHARED_CORE=ON` - [x] **Red/green for both ASan fixes.** Reverted, all 4 tests reproduce CI's exact reports (stack-use-after-scope at `:95`, downcast at `qpointer.h:76`). With the fixes they pass 3/3. - [x] 71 WDSP, ANAN, TX, HL2 DSP, map and edited tests under ASan + UBSan. All pass apart from the 2 known 30 s timeouts. `weather_radar_loading_test` run directly: 44 passed, 0 failed, 26 s. - [x] **Red/green for the leak fixes.** In a full-suite LSan sweep on the same tree, before vs after the three leak commits: 86 `create_calcc` blocks, 5 `disconnectClientHandlesThen` blocks and the `QFocusEvent` leak, down to none. All the affected tests pass with leak detection on, `tx_operation_integration` with 421/421 assertions. Leaks from this machine only (fontconfig/pango/GTK, NVIDIA `cuInit`, ggml Vulkan) were suppressed locally, not in the repo. - [x] Sanitizers `workflow_dispatch` [37230031178](https://github.com/aethersdr/AetherSDR/actions/runs/37230031178) tested **`31e249c3e`**, the head before the leak fixes and the review fixes, **not** the final head. Zero sanitizer `SUMMARY:` reports in either lane, so every originally targeted test is green. The run still went red on the timeouts and on `rtl_capture_worker_test`. Its duplicate stickies (#6165, #6170) are closed into #6154 / #6156. - [x] Sanitizers `workflow_dispatch` on the **final head `7f37cf045`**, [37243767521](https://github.com/aethersdr/AetherSDR/actions/runs/37243767521): **zero** sanitizer `SUMMARY:` reports in either lane. ASan failed 2/656 and TSan 5/656, every one a known sanitizer-slowdown timeout. `gui_nested_lifetime_test` (the new VFO regression), `phone_tx_filter_numeric_entry_test`, `wdsp_channel_test` and `rtl_capture_worker_test` passed in both lanes. - [x] The same head **plus #6176**, so a leak fails its test, [37243769261](https://github.com/aethersdr/AetherSDR/actions/runs/37243769261), on the temporary branch `ci/integration-6173-6176` (since deleted): **no ASan, UBSan or LeakSanitizer report**, and the same timeouts only. Its stickies #6189 and #6191 are closed into #6154 / #6156. - [x] **VFO teardown red/green.** The new `vfoCollapsedLabelTeardownInEitherOrder` aborts on the `qpointer.h:76` downcast in its "label-dies-first" row with only the destructor fix. With the event-type-first check, all 10 cases pass, 3 of 3 runs. The 26 vfo/rx_applet/fm_filter/slice/lifetime tests pass. - [ ] Behaviour on a real radio: patch 16 is a TX-path change. A COMP-on filter-width change while keyed should be checked on hardware and a dummy load before it ships. - [ ] Existing tests pass (CI) ## Checklist - [x] Commits are signed (`docs/COMMIT-SIGNING.md`) - [x] No new flat-key `AppSettings` calls - [x] Code is clean-room - [x] No meter UI touched - [x] Documentation updated: `third_party/wdsp/AETHERSDR-PATCHES.md` (patches 16, 17) and the in-file rationale in `tests/sanitizers/tsan.supp`. No `CHANGELOG.md`. - [x] Not security-sensitive 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…nes, and HackRF unplug + Windows (#42, #46, #47, #75, #80) (#78) * fix(hackrf): build the x86_64 AppImage again, and guard the libhackrf API floor The v26.10.6 tag built every package but the x86_64 AppImage, which failed to compile: HackRfDiscovery.cpp used BOARD_ID_UNDETECTED, a name newer libhackrf headers declare but Ubuntu 22.04's libhackrf-dev 2021.03.1 (the AppImage's build host) calls BOARD_ID_INVALID. Every other build has the newer header, so only the release tag saw it. Use the number, 0xFF, as HackRfWorker::readBoardId already does. No per-PR lane builds against that header, and this is the second break of this class (#67), so add libhackrf_api_floor: a ctest source scan that fails when src/ uses a libhackrf name outside the 2021.03.1 header's declarations. Checked against the 2021.03.1 header itself: every HackRF source compiles with it after this change, and the scan flags exactly the two lines that broke the tag. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(sim): close the spectrum gate before announcing disconnect (#75) The full suite failed once on main with "sim: nothing emitted after disconnected() (spectrumFrameReady)" in backend_seam_affinity_test. RadioModel disconnects the simulator's vended RadioConnection directly, and SimBackend turned that into two queued slots: a signal-to-signal re-emit of disconnected(), then the handler that clears m_connected and stops the worker. A spectrum row the worker queued between the two was forwarded after the announcement, breaking IRadioBackend rule 6. One handler now closes the gate and then emits disconnected(), so there is no gap for a row to land in. The core of upstream aethersdr#6088; its session stamp on spectrum rows depends on refactors this fork does not have. sim_backend_test pins the ordering: at the moment disconnected() is announced the backend already reports disconnected, and a row emitted from inside that announcement is dropped. Both checks failed before the fix. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(icom): send the fail-safe unkey in the call that commands it (#47) icom_identity_test failed intermittently ("ambiguity releases the previously selected destination without retargeting unkey"): once under ASan on CI and 2 runs in 40 in a plain local Debug build. sendUserCommand() read the clock once, queueWrite() stamped the request with a second, later reading, and the pump at the first reading skipped the request as not yet due. When a millisecond boundary fell between the two reads, the PTT-off write sat in the queue until the next timer tick instead of leaving at once. Pump with a fresh reading. The disconnect path already reads the clock after queueing its emergency writes; the meter tick's reads are routine and only wait one tick. A test seam, m_nowForTest, lets the test make every clock reading later than the last, which reproduces the race on every run: the new check failed deterministically before the fix and passes after it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci(sanitizers): fit the ASan lane on the runner's disk, and scale test timeouts under sanitizers (#46, #47) Both weekly sanitizer lanes have been red for weeks without finding a sanitizer report: - ASan + UBSan (#47) stopped building on 2026-10-03: "final link failed: No space left on device". Every test executable linked its own static copy of libaethercore. A new AETHER_SHARED_CORE option (default OFF, never for releases) builds the core as one shared library, and the ASan lane turns it on. From upstream aethersdr#6136, with its workflow changes: df -h around the build, and a Configure or Build failure now reaches the tracking issue instead of skipping the whole report chain. Static checks assert per PR that the option yields libaethercore.so and the default stays static. - TSan (#46) timed out weather_radar_loading_test every week. It takes ~15 s in a plain build (mostly waiting), 26 s under ASan, and over its 30 s limit under TSan. In an instrumented build every test TIMEOUT, its own or the 300 s default, is now multiplied by AETHER_SANITIZER_TIMEOUT_SCALE (4). Ordinary builds keep every limit as written. From upstream aethersdr#6208. Verified locally: the whole tree (app and all 512 tests) builds with AETHER_SHARED_CORE=ON and every test passes; a configure with -fsanitize=thread scales weather_radar_loading_test to 120 s and a plain configure leaves it at 30 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(hackrf): notice a HackRF that goes away mid-session (#42) HackRfWorker declared streamStopped() and HackRfBackend handled it, but nothing ever emitted it. libhackrf reports a stream that dies under it (unplug, USB error, reset) only by calling back no more, so the spectrum froze, the session stayed "connected", and connecting again was refused with "already connected to a radio". The backend's 50 ms arbiter tick now also runs HackRfWorker::checkStreamHealth(). A stream that should be running but that libhackrf says has ended (hackrf_is_streaming), or that has delivered no transfer for 2 s, is marked stopped and reported once. The backend then reports "HackRF stream stopped unexpectedly" and disconnects, in that order: the old handler disconnected first and emitted the error after disconnected(), which IRadioBackend rule 6 forbids. A report after the session has ended is ignored. hackrf_backend_test pins the rule (streamFault) and the ordering; the ordering checks failed on the old handler. tools/hackrf_unplug_test.py checks it on hardware, receive only: it resets the board with libhackrf's own reset request, so it drops off USB and re-enumerates the way an unplug does (or waits for a real unplug with --physical). On a HackRF Pro it passes 9/9 (disconnected in 0.3-0.4 s, still running, reconnects, receives again); on the old build the app never disconnected and could not reconnect. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(hackrf): HackRF on Windows (#42) Every Windows build so far compiled the HackRF backend out: nothing supplied libhackrf, which ships no Windows binaries. scripts/setup/setup-hackrf.ps1 builds it the way Great Scott Gadgets' own Windows guide does, with MSVC against vcpkg's libusb and pthreads, at the release the macOS DMG builds (2026.01.3, from its .zip; the host/libhackrf tree is identical to the .tar.xz the DMG pins). It stages hackrf.h, hackrf.lib and the hackrf, libusb and pthreads DLLs in third_party/hackrf. - CMakeLists.txt: on Windows, link that staging as an imported DLL and copy its DLLs next to AetherSDR.exe. The probes now chain on HACKRF_TARGET rather than HACKRF_FOUND, which pkg_check_modules caches: keyed on the flag, a reconfigure skipped the probe that sets the target and the core built without libhackrf's include path. - windows-installer.yml: build it (cached on the script), configure with REQUIRE_HACKRF=ON, and ship the three DLLs in the payload, where check-deploy-dependencies.ps1 already proves every import is present. - ci.yml check-windows: the same build and REQUIRE_HACKRF=ON, so every PR compiles the backend with MSVC. - README: an optional setup step in the Windows build instructions. docs/HACKRF.md: a Windows section (no Zadig: the firmware asks Windows for WinUSB). Not yet checked against a HackRF on a Windows machine, and the doc says so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(hackrf): don't fail the Windows build over a busy scratch directory (#42) libhackrf built and staged on both Windows lanes; the step then failed on its last line, removing third_party\hackrf-temp, because something still held a handle on build\src a moment after the post-link DLL copy. The scratch tree is now removed with a short retry and only a warning if it stays busy, and Expand-Archive overwrites a leftover one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci(sanitizers): suppress Qt 6.12's TLS-backend and HTTP-delegate reports (#80) The TSan image now carries Qt 6.12.0 (it was 6.8.3 on the last weekly run), and four tests fail on two reports that are wholly inside Qt: tci_automation, tci_server_review and hl2_tci_signaling on the lock-order inversion between QSslSocketPrivate::tlsBackendInUse() and the TLS factory-loader mutex, and whats_new_dialog on ~QHttpThreadDelegate vs a main-thread reply delete. Upstream suppressed exactly these two, anchored on the Qt frames, in aethersdr#6173; this takes its tsan.supp entries verbatim. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Problem
The weekly ASan + UBSan sanitizer lane has failed in its Build step for about three weeks with
final link failed: No space left on device, so it has tested nothing in that time. About 250 of the test executables statically link the whole oflibaethercore. In a Debug ASan build each of those averages about 200 MB, and together they outgrow the runner's ~83 GB of free disk. The runner's/and/mntare the same 145 GB disk, so building on/mntdoesn't help. Trimming debug info (-g1plus compressed debug sections) still measured 127 GB locally.Fix
CMakeLists.txt: a new option,AETHER_SHARED_CORE(default OFF), buildslibaethercoreas a shared library with position-independent code, because the vendored static libraries end up inside it. On Windows it's a configure error. Release, ordinary and TSan builds keep the static core unchanged..github/workflows/sanitizers.yml: the ASan entry passesshared_core: "ON", and TSan passes"OFF". The Build step printsdf -h /before and after the build, including when the build fails (it keeps the build status and exits with it after the second reading)..github/workflows/sanitizers.yml, reporting: every report step was gated on ctest's exit code, so a Configure or Build failure skipped them all, with no artifact and no tracking issue. That is how the lane went dark. A new step runs only when Configure or Build failed. It writes a report naming the step, with a stable per-step fingerprint, and the artifact and tracking-issue steps also run on that path..github/workflows/static-checks.yml: the "Sanitizer option configures" job gains a fifth case:-DAETHER_SHARED_CORE=ONmust generate alibaethercore.sorule, and the default configure must still generatelibaethercore.a. That's the per-PR check of the option; the sanitizer lane itself only runs weekly.Validation
Locally (Linux, Debug ASan + UBSan):
-g1+ zliblibaethercore.so668 MB once)Running the full suite under ASan gave the same failures with the shared core as with the static core, so linking doesn't change results.
On the runner (validation run 37141490698, dispatched from this branch):
/went from 62 GB used / 83 GB free to 81 GB used / 64 GB free after the build.offline_health_registry_test: an ASan stack-use-after-scope in the test itself (offline_health_registry_test.cpp:95).rx_applet_squelch_reconciliation_testandvfo_dsp_launcher_width_test: a UBSan downcast inVfoWidget::eventFilter(VfoWidget.cpp:6766) during~VfoWidget.hl2_backend_seam_test: main was red when the run started; fix(hl2): update seam tests for the transmit tail. Principle XI. #6126 has since fixed it.weather_radar_loading_test: a timeout, which also fails locally.Qt 6.8.3 found, but AetherSDR requires Qt 6.12. That's the stale:tsanimage, fixed separately by fix(ci): build TSan QtMultimedia with uninstrumented host tools #6123, and unrelated to this change (TSan keepsshared_core: "OFF").Checks: the new static-checks case, run locally, passes on this head. It fails with its
::error::when the ON branch is forced to STATIC, and also when the default is forced to SHARED. The Build step exits 1 when the build fails and 0 otherwise, checked underbash --posix.The real check is the next scheduled sanitizer run on
mainafter merge: it should reach Run tests under sanitizer instead of dying at Build.🤖 Generated with Claude Code