Skip to content

ci(sanitizers): link the ASan lane against a shared core so it fits on disk. Principle VIII. - #6136

Merged
ten9876 merged 3 commits into
mainfrom
ci/asan-shared-core
Oct 4, 2026
Merged

ten9876 merged 3 commits into
mainfrom
ci/asan-shared-core

Conversation

@ten9876

@ten9876 ten9876 commented Oct 3, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

The weekly ASan + UBSan sanitizer lane has failed in its Build step for about three weeks with final link failed: No space left on device, so it has tested nothing in that time. About 250 of the test executables statically link the whole of libaethercore. In a Debug ASan build each of those averages about 200 MB, and together they outgrow the runner's ~83 GB of free disk. The runner's / and /mnt are the same 145 GB disk, so building on /mnt doesn't help. Trimming debug info (-g1 plus compressed debug sections) still measured 127 GB locally.

Fix

  • CMakeLists.txt: a new option, AETHER_SHARED_CORE (default OFF), builds libaethercore as a shared library with position-independent code, because the vendored static libraries end up inside it. On Windows it's a configure error. Release, ordinary and TSan builds keep the static core unchanged.
  • .github/workflows/sanitizers.yml: the ASan entry passes shared_core: "ON", and TSan passes "OFF". The Build step prints df -h / before and after the build, including when the build fails (it keeps the build status and exits with it after the second reading).
  • .github/workflows/sanitizers.yml, reporting: every report step was gated on ctest's exit code, so a Configure or Build failure skipped them all, with no artifact and no tracking issue. That is how the lane went dark. A new step runs only when Configure or Build failed. It writes a report naming the step, with a stable per-step fingerprint, and the artifact and tracking-issue steps also run on that path.
  • .github/workflows/static-checks.yml: the "Sanitizer option configures" job gains a fifth case: -DAETHER_SHARED_CORE=ON must generate a libaethercore.so rule, and the default configure must still generate libaethercore.a. That's the per-PR check of the option; the sanitizer lane itself only runs weekly.

Validation

Locally (Linux, Debug ASan + UBSan):

static core shared core
Build tree 127 GB even at -g1 + zlib 17 GB
Average test executable ~200 MB 13 MB (libaethercore.so 668 MB once)
Test targets that build and link — all 591

Running the full suite under ASan gave the same failures with the shared core as with the static core, so linking doesn't change results.

On the runner (validation run 37141490698, dispatched from this branch):

  • Disk: / went from 62 GB used / 83 GB free to 81 GB used / 64 GB free after the build.
  • Tests: the suite ran to completion. 618 of 623 tests passed. All 5 failures are existing issues in the code under test:
    • offline_health_registry_test: an ASan stack-use-after-scope in the test itself (offline_health_registry_test.cpp:95).
    • rx_applet_squelch_reconciliation_test and vfo_dsp_launcher_width_test: a UBSan downcast in VfoWidget::eventFilter (VfoWidget.cpp:6766) during ~VfoWidget.
    • hl2_backend_seam_test: main was red when the run started; fix(hl2): update seam tests for the transmit tail. Principle XI. #6126 has since fixed it.
    • weather_radar_loading_test: a timeout, which also fails locally.
  • TSan job: it failed at Configure with Qt 6.8.3 found, but AetherSDR requires Qt 6.12. That's the stale :tsan image, fixed separately by fix(ci): build TSan QtMultimedia with uninstrumented host tools #6123, and unrelated to this change (TSan keeps shared_core: "OFF").

Checks: the new static-checks case, run locally, passes on this head. It fails with its ::error:: when the ON branch is forced to STATIC, and also when the default is forced to SHARED. The Build step exits 1 when the build fails and 0 otherwise, checked under bash --posix.

The real check is the next scheduled sanitizer run on main after merge: it should reach Run tests under sanitizer instead of dying at Build.

🤖 Generated with Claude Code

…n disk. Principle VIII.

The weekly ASan + UBSan lane has failed at Build for three weeks running
with "final link failed: No space left on device", so it has tested
nothing. Each test statically links the whole core: a Debug ASan test
averages ~200 MB, and even with -g1 and compressed debug sections the
tree measured 127 GB locally against ~83 GB free on the runner.

New option AETHER_SHARED_CORE (default OFF; not on Windows) builds
libaethercore shared with position-independent code. The ASan entry
turns it on: the full local Debug ASan tree is 17 GB, all 591 tests
link, and test executables average 13 MB. Running the whole suite
locally under ASan gave the same failures with the shared core as with
static linking, so results are unchanged. On the runner (run
37141490698) the build used 19 GB of disk and the full suite ran: 618 of
623 tests passed. TSan, release and ordinary builds keep the static core.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@ten9876
ten9876 requested a review from a team as a code owner October 3, 2026 22:30

@aethersdr-agent aethersdr-agent Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Issue fit

No fixes #NNNN — there is no issue for the three-week ASan Build outage (the author filed #6130 for the CodeQL twin of the same disk problem, and #6135 fixes that one). Reviewed against the PR body's own stated intent, which is specific and falsifiable. GOVERNANCE.md does not want an RFC here: this is a CI build-configuration fix with a named root cause, default OFF, no product surface. It does what it says — the lane dies at final link failed: No space left on device because ~250 Debug+ASan test executables each statically absorb the whole core, and one shared core replaces those copies with one 668 MB .so.

One verification I could do from here: validation run 37141490698 exists, and its TSan job failed at Configure with every downstream step skipped — exactly as the body describes, and consistent with the stale :tsan image story (#6123). I could not independently confirm the ASan job's 618/623 or the df numbers; get_ci_run_log returns only the first failing matrix job. The local 127 GB → 17 GB table is likewise unreproducible from here. Everything else below is reasoned from the code in /tmp/aetherclaude/pr-6136, not observed at runtime.

Scope

File / group Change Claimed by body? Verdict
CMakeLists.txt 146–162 New AETHER_SHARED_CORE option (default OFF), PIC, Windows FATAL_ERROR Yes In scope
CMakeLists.txt 1594 add_library(aethercore ${_aether_core_library_type}) Yes In scope
sanitizers.yml 39–42, 70 shared_core matrix key on both entries Yes In scope
sanitizers.yml 130 -DAETHER_SHARED_CORE=... on the configure line Yes In scope
sanitizers.yml 150–152 df -h / around the build Yes ("so disk headroom is visible in the log") In scope

Everything in the diff is explained by the stated fix. No deleted guards, no new product/protocol surface, no CHANGELOG.md entry, no UI or default-behavior change — §6 preference check is not engaged. Two files, both build plumbing.

Blockers

None.

Nits (non-blocking)

  • The "~600" in both new comments overstates it. tests/tests.cmake declares 617 executables; 247 link aethercore directly and 10 more reach it through aetherdesktop_support — 257, not ~600. ~600 is the test count (623 ran). The disk arithmetic still works at 257 × ~200 MB plus objects and the apps, but the comments explain why, and the why is off by 2.3×. The body's "Tests that link: all 591" doesn't match either number and is probably "591 targets built". Inline on CMakeLists.txt:146 and sanitizers.yml:39.
  • The post-build df -h / never runs in the case it exists for. cmake --build failing under bash -e aborts the step, so a future disk exhaustion still leaves only the pre-build reading. Suggestion fence inline at sanitizers.yml:150.
  • No PR-gating canary for the new option. static-checks.yml's "Sanitizer option configures" job already proves -DAETHERSDR_SANITIZER=... configures and refuses correctly; a fifth case asserting -DAETHER_SHARED_CORE=ON configures and reports the SHARED status line would catch a future regression on the PR instead of on Saturday. Inline at sanitizers.yml:130.
  • The silence this PR's root cause exposed is still there. Every reporting step in this workflow is gated on steps.ctest.outputs.exit != '0' (lines 239, 302, 312, 397). A Build failure skips all of them — no artifact, no sticky issue — which is precisely how the lane went dark for three weeks, and the run-37141490698 TSan job above is a live example of the same shape. This PR removes one cause; the gate's own comment at lines 77–85 already names this failure class for the container case. Sibling of the root cause, not scope creep — worth a follow-up issue rather than code in this PR.
  • install(TARGETS AetherSDR …) at line 3319 does not install libaethercore.so, so AETHER_SHARED_CORE=ON + install yields a broken tree. The comment already says "sanitizer CI only" and nothing in the release path sets it, so this is documentation-only today.
  • Why does TSan stay OFF? Its -g3 RelWithDebInfo footprint is largely optimization-independent and is currently unmeasured — the validation run's TSan job died at Configure. Genuine question, not an assertion that it should flip.

What I tried to break

  • Duplicate core state under ELF. The real hazard of static→shared is a test that compiles a core .cpp and links aethercore: under a static archive the member is simply not pulled (one copy), but under a shared lib the executable's definitions are not in .dynsym without -rdynamic, so the .so keeps its own copy and singleton state forks. Parsed all 617 targets in tests.cmake: 190 compile src/core/src/model sources, 257 link the core, and the intersection is empty. Same for the 76 ${AETHER_SETTINGS_SOURCES} consumers — none of them links aethercore. Survived, and it survives by construction, not by luck.
  • PIC on the vendored static libraries. set(CMAKE_POSITION_INDEPENDENT_CODE ON) lands at line 157, and nothing before line 146 creates a target (grep for add_library/add_subdirectory/ExternalProject/FetchContent above that line is empty), so every in-tree static lib inherits it. The one escapee is the Opus ExternalProject, which a separate CMake project would not inherit — but third_party/radae/cmake/BuildOpus.cmake:20 already passes -DCMAKE_POSITION_INDEPENDENT_CODE=ON explicitly, and third_party/wdsp/CMakeLists.txt:26 sets POSITION_INDEPENDENT_CODE ON on its own target. No R_X86_64_32-against-a-shared-object trap.
  • PRIVATE dependency propagation. With STATIC, target_link_libraries(aethercore PRIVATE aether::wdsp) still propagates as $<LINK_ONLY:> to consumers; with SHARED it does not, and aether_wdsp carries C_VISIBILITY_PRESET hidden, so WDSP symbols are not exported from the .so. Any test calling a WDSP entry point while linking only aethercore would link today and fail only in the ASan lane. Comment-stripped scan of the three candidate tests (wdsp_channel_test, hl2_cw_bfo_test, anan_rxdsp_handedness_test) — every apparent SetRXAMode/fexchange2 hit is prose; all three go through the core/dsp/WdspChannel.h wrapper. Survived, but it is a latent asymmetry worth knowing about when the next DSP test lands.
  • Symbol export and RPATH. No VISIBILITY_PRESET/-fvisibility=hidden, no --whole-archive, no $<TARGET_OBJECTS:aethercore>, and no CMAKE_SKIP_BUILD_RPATH / CMAKE_BUILD_WITH_INSTALL_RPATH anywhere in the root listfile, tests.cmake or .github/ — so default visibility holds and CMake's automatic build-tree RPATH lets ctest find the .so without an LD_LIBRARY_PATH dance.
  • Static-initializer topology. A static archive drops unreferenced objects; a shared core loads every TU and runs every static initializer, so the lane now exercises a link topology the shipped app never uses. grep for __attribute__((constructor)) / Q_CONSTRUCTOR_FUNCTION / Q_COREAPP_STARTUP_FUNCTION in src/core returns nothing, so there is no self-registration layer to change behaviour — and ASan's detect_odr_violation (left at its default in runtime_env) apparently found nothing either, since the runner run reached 618/623. Flagging only that a future sanitizer finding in a load-time path may not reproduce in the static-linked product.
  • Matrix completeness. Both entries carry shared_core; a future entry omitting it yields an empty -DAETHER_SHARED_CORE=, which is falsey and lands on STATIC — degrades safely.
  • Checked the head SHA's seven checks: all green. None of them configures with AETHER_SHARED_CORE=ON, which is the basis for the canary nit above.

Recommendation

Approve with nits. The change is small, default-OFF, refused at configure on Windows, and the one code path it touches is guarded by a variable that only this workflow sets; the measured before/after and the dispatched runner run are the kind of evidence Principle VIII asks for. The duplicate-state and PIC hazards that would have made static→shared unsafe are both absent by construction, which I checked rather than assumed. Next step: fix the "~600" counts in the two comments, make the post-build df unconditional, and — separately from this PR — file the follow-up for the Build-failure reporting gap so the lane cannot go dark silently again. Thanks for measuring this instead of guessing at it.


🤖 aethersdr-agent · cost: $5.2600 · model: claude-opus-5

Comment thread CMakeLists.txt Outdated
Comment thread .github/workflows/sanitizers.yml Outdated
Comment thread .github/workflows/sanitizers.yml
Comment thread .github/workflows/sanitizers.yml
Comment thread CMakeLists.txt

@ten9876 ten9876 left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Issue fit

Approve with nits. No linked issue; reviewed against the stated ASan build disk-exhaustion problem. The two hunks implement the described solution: an opt-in shared core with PIC, selected only by the ASan workflow entry, and disk measurements around the build. GOVERNANCE.md explicitly exempts “Build / CI fixes” from the RFC requirement. No product behavior or visual preference changes are bundled.

I independently retrieved the ASan job log for validation run 37141490698. It confirms the load-bearing claim: the shared core linked, Build finished, disk retained 64 GB free, and CTest reached its final report. This is narrower than proving the entire suite clean or proving static/shared behavioral equivalence.

Scope

File/group Change Claimed in body? Verdict
CMakeLists.txt:146–162 Default-OFF AETHER_SHARED_CORE, Windows refusal, PIC and library-type selection Yes In scope; opt-in build configuration, no shipping default change
CMakeLists.txt:1594 Create core using selected library type Yes In scope
.github/workflows/sanitizers.yml:39–42,70,130 ASan ON / TSan OFF and configure argument Yes In scope
.github/workflows/sanitizers.yml:150–152 Before/after disk readings Yes In scope

Everything in the diff is explained by the stated fix. The single commit's authored date matches this work; no unrelated older commit, removed safety guard, new runtime protocol surface, baseline relaxation, or CHANGELOG entry appears. No scope checklist was supplied; the body's scope statements otherwise hold.

Blockers

None identified.

Nits — non-blocking

  1. CMakeLists.txt:146–149 and .github/workflows/sanitizers.yml:39–41: describe the linked-core subset accurately. “Every test links the whole core” / “~600 full copies” conflates the suite size with the number of core consumers. A fresh configure of this head generated 609 C++ executable links, of which 254 depend on libaethercore.so (including the applications). Many tests compile small source subsets directly. This does not undermine the measured disk fix, but the permanent explanation should say that many tests duplicate the static core. Suggested replacement for the first comment:
# Share libaethercore among its test consumers to reduce sanitizer build disk
# usage. This is an opt-in sanitizer CI configuration; shipped builds keep the
# static core.
  1. .github/workflows/sanitizers.yml:150–152: preserve the final disk reading on failure. The step's fail-fast shell exits on a failed cmake --build, skipping the final df; that is when this diagnostic is most useful. A shell EXIT trap preserves the build's failure status and measures either outcome:
        run: |
          trap 'df -h /' EXIT
          df -h /
          cmake --build build --parallel

What I tried to break / verification

  • Disk and completion claim — independently verified from the runner log. The job configured libaethercore built SHARED, linked libaethercore.so, and emitted:

    overlay         145G   62G   83G  43% /
    overlay         145G   81G   64G  57% /
    99% tests passed, 5 tests failed out of 623
    

    The five failures match the body: hl2_backend_seam_test, weather_radar_loading_test, offline_health_registry_test, rx_applet_squelch_reconciliation_test, and vfo_dsp_launcher_width_test. I did not independently reproduce each on a static merge-base build, so their asserted pre-existing status and the local static/shared equivalence remain unverified, not newly attributed PR regressions. The log includes skipped cases; 618/623 means non-failing CTest cases, not 618 independently observed executed passes.

  • Opt-in isolation — configured current head 8d4bff35f locally in a scratch worktree. Fresh Debug AETHER_SHARED_CORE=ON generates CXX_SHARED_LIBRARY_LINKER for libaethercore.so; a fresh ordinary Release configure without the option generates CXX_STATIC_LIBRARY_LINKER for libaethercore.a. Inspected all 292 generated core source compilation rules: all carry -fPIC. No generated executable both links the shared core and directly compiles src/core/src/models objects, avoiding the obvious duplicated-state trap. Vendored Opus already explicitly passes PIC into its ExternalProject; WDSP independently enables PIC. Reviewed private link dependencies and default symbol visibility; the runner's completed full link graph supplies empirical coverage of the enabled CI configuration's dependencies.

  • TSan isolation — independently verified from its job log. It passed -DAETHER_SHARED_CORE=OFF and failed configuration with Qt 6.8.3 found, but AetherSDR requires Qt 6.12 or newer. This PR does not claim to repair that image. Windows ON is explicitly refused before dependencies are configured; I inspected that guard but did not run a Windows configure.

  • Evidence freshness. Run 37141490698 used df503b7f, before the rebase to current head 8d4bff35f. The shared-core and sanitizer-workflow changes are identical between those revisions; the root CMake difference adds PeripheralAuthStore.cpp to desktop support. Current-head PR checks are green but began at 2026-10-03 22:30:31 UTC, before the inspected main tip (3e4b14e2, 2026-10-04 01:13:20 UTC). They do not prove today's merge result; strict: false does not force a refresh. The sanitizer validation is a workflow_dispatch branch run, not a current-main merge run. Local 127 GB/17 GB figures and full static/shared result equivalence were not recreated.

  • Test boundary and runtime scope. The full diff modifies no test source or registration, so no new/modified socket-owning target required execution. I ran configuration and inspected generated build graphs; I did not execute the suite, create a synthetic firmware peer, or drive the GUI. This is a build/CI-only change with direct runner evidence, so no demo instance was needed or launched. No separate automated code-review skill was available; the existing bot review was checked against source/generated graphs and logs rather than adopted wholesale.

Recommendation

Approve with nits, subject to the normal infrastructure-owner review and required checks. The actual hosted-runner log establishes that the change repairs the reported failure to reach tests, and current-head configuration confirms that ordinary builds remain static. Correct the comments and preserve failure-time disk diagnostics; the next sanitizer run on merged main should confirm the rebased/current integration. This review does not assert that the five failing tests are harmless or that the sanitizer suite is green.

@ten9876 ten9876 assigned ten9876 and unassigned ten9876 Oct 4, 2026
ten9876 added a commit that referenced this pull request Oct 4, 2026
… check the option per PR. Principle VIII.

Review on #6136:
- The comments said ~600 tests link the core. About 250 executables do;
  ~600 is the test count.
- The after-build df ran only when the build succeeded. It now runs from an
  EXIT trap, so a disk-full failure still logs it.
- Record that WDSP symbols don't reach the shared core's consumers.
- static-checks' 'Sanitizer option configures' job gains a fifth case: ON
  must generate libaethercore.so and the default must stay .a. Without it
  the option's first exercise is the weekly cron.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… check the option per PR. Principle VIII.

Review on #6136:
- The comments said ~600 tests link the core. About 250 executables do;
  ~600 is the test count.
- The after-build df ran only when the build succeeded. The step now keeps
  the build status and exits with it after the second df, so a disk-full
  failure still logs it.
- Record that WDSP symbols don't reach the shared core's consumers.
- static-checks' 'Sanitizer option configures' job gains a fifth case: ON
  must generate libaethercore.so and the default must stay .a. Without it
  the option's first exercise is the weekly cron.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@ten9876
ten9876 force-pushed the ci/asan-shared-core branch from acb5d55 to fde2048 Compare October 4, 2026 03:00
… Principle VIII.

Every report step was gated on ctest's exit code, so a Configure or Build
failure skipped them all: no artifact, no sticky issue. That is how the
ASan lane went dark for three weeks. A new step runs only when Configure
or Build failed. It writes a report naming the failed step, with a stable
per-step fingerprint, and the artifact and tracking-issue steps now also
run on that path.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@ten9876

ten9876 commented Oct 4, 2026

Copy link
Copy Markdown
Collaborator Author

Review-body items, current head e9255efb8:

  • Counts, the df step, the canary and the WDSP note: see the five resolved threads (fde2048).
  • Silent Build failure (bot nit 4): folded in at the maintainer's call, in e9255ef. A new Report a lane that did not build step runs only when Configure or Build failed. It writes report.txt/sanitizer.log naming the step, plus a stable fingerprint (nobuild:Configure → 8c86ae9408dd, nobuild:Build → b83f0666e782). The artifact and tracking-issue steps now run on (success() && ctest exit != 0) || (failure() && nobuild ran). A missing image still skips everything, as the existing comment at the top of the steps says. actionlint is clean. The live proof is in progress: run 37173036491 is dispatched on this head. Its TSan job fails at Configure on the stale image (fixed by fix(ci): build TSan QtMultimedia with uninstrumented host tools #6123, merged), so it should open a TSan … (fp 8c86ae9408dd) issue. Its ASan job re-proves the shared-core build.
  • install(TARGETS) doesn't install libaethercore.so (bot): correct, and deliberate. The option is documented as sanitizer CI only, it is OFF by default, and nothing in the packaging path sets it. Making a shared-core install work would be a feature nobody uses.
  • Why TSan stays OFF (bot question): its footprint hasn't been measured. It's RelWithDebInfo at -O1 -g3, not Debug ASan, and every TSan run since the Qt 6.12 pin has died at Configure on the stale image. Once fix(ci): build TSan QtMultimedia with uninstrumented host tools #6123's image publishes, the Build step's two df readings will show the TSan footprint. Flip it then if the readings call for it.

@aethersdr-agent

Copy link
Copy Markdown
Contributor

TSan red on e9255ef is the stale :tsan image, not this PR

Short version: nothing required for merge is red, and nothing here is your code. The failing job is TSan in run 37173036491, which died in Configure — and Sanitizers is schedule + workflow_dispatch only (.github/workflows/sanitizers.yml:9-15), so this is your own validation dispatch, not a gate.

Status of the real gates on this head:

check result
Static checks ✅ (includes your new fifth "Sanitizer option configures" case)
Sanitizer option configures ✅
Analyze (python / js-ts) ✅
build / check-macos / check-windows ⏳ still running
ASan + UBSan ⏳ still queued — no result yet on this head

Why Configure failed

I can't quote this run's Configure log — the job-log API isn't reachable from here, and get_ci_run_log returns step outcomes only — so this is inference, with its sources named rather than a log line I didn't read:

  1. The lane pulls ghcr.io/aethersdr/aethersdr-ci:tsan (sanitizers.yml:66). The Confirm the sanitizer image step (sanitizers.yml:87) passed, and that step only checks that libQt6Core.so.6 exists and is TSan-instrumented — not its version. A stale-but-instrumented image sails through it and then trips the Qt floor in Configure (cmake/AetherQtPin.cmake, via CMakeLists.txt:285: "Qt … found, but AetherSDR requires Qt 6.12 or newer").
  2. Your earlier dispatch 37141490698 died at exactly this step with Qt 6.8.3 found, but AetherSDR requires Qt 6.12 — you documented it in the PR body.
  3. fix(ci): build TSan QtMultimedia with uninstrumented host tools #6123's commit message (b255830) names the same cause: the Dockerfile.tsan QtMultimedia build hung at [2/300], a failed build never publishes, so :tsan has been stuck on Qt 6.8.3.

Why merging #6123 didn't help yet — and what to do

#6123 landed on main as b255830 at 03:06 UTC. This dispatch started roughly a minute later. The image still has to be rebuilt and published by docker-ci-image.yml (which fires on a main push touching .github/docker/Dockerfile.tsan), and that job — build-tsan, run 37173106228 — is still in_progress as I write this. The runner pulled :tsan long before it could publish.

So:

  • No rebase needed. The workflow references the image by tag, so the PR's content has no bearing on it — and TSan keeps shared_core: "OFF" anyway.
  • Wait for build-tsan on b255830 to publish, then re-dispatch Sanitizers on this branch. Pure infrastructure timing.

One thing from this run worth a guard

The good news first: Report a lane that did not build → Upload sanitizer artifacts → Open or update tracking issue all ran and succeeded. That's precisely the dark-lane hole e9255ef set out to close, working on its first real Configure failure. 👍

The side effect: it filed #6144, "[sanitizer] TSan weekly run failure (fp 8c86ae9408dd)", labelled priority: high / awaiting-response, from a PR-branch dispatch, for a stale image. Neither sanitizers.yml:328 nor :338 carries a github.event_name or github.ref condition, so any dispatch from any branch that fails to build now opens or updates a sticky titled "weekly run failure". Two small things you might fold in:

Either way #6144 can just be closed once the image republishes.

On the review thread

Nothing in the bot comments bears on this failure. The count correction (257 core-linking targets of 617, not ~600) and the rc=0; … || rc=$?; df -h /; exit "$rc" fix are both in fde2048, and the canary suggestion is already implemented as the fifth static-checks.yml case — which is green here.

Nice piece of work, by the way: 127 GB → 17 GB, and the reporting fix means the next three weeks of a dark lane can't happen silently. Thanks for digging into both halves of it.


🤖 aethersdr-agent · cost: $6.3627 · model: claude-opus-5

@ten9876
ten9876 merged commit a8797ba into main Oct 4, 2026
7 of 9 checks passed
@ten9876
ten9876 deleted the ci/asan-shared-core branch October 4, 2026 04:22
ten9876 added a commit that referenced this pull request Oct 4, 2026
… VIII. (#6152)

## Problem

**Full Suite on `main` is red.** Run
[37181657060](https://github.com/aethersdr/AetherSDR/actions/runs/37181657060)
on `a325fb99e` (#5924) died at link 4934/4976 with `final link failed:
No space left on device`. The previous commit's run was green, so the
tree is right at the runner's limit.

- **ci.yml's required `build` job builds the same tree.** It only
survives because #5924 gave it a step that frees 18 GB of host
toolchains (84 → 102 GB free). That's about 50 more core-linking tests
of headroom before the PR gate hits the same wall.
- **The TSan lane ran out of disk too.** With the Qt 6.12 `:tsan` image
from #6123 it got far enough to build for the first time, then filled
the runner: `overlay 145G 145G 163M 100%` at link 4438/4854 (run
[37176807431](https://github.com/aethersdr/AetherSDR/actions/runs/37176807431),
#6148).
- **The red Full Suite run filed no issue.** When Build fails, ctest is
skipped, and a skipped step's output reads as `null`. GitHub's
expression `exit != '0'` coerces `null` to 0, so every report step was
skipped.

Underneath all three: ~250 test executables each link a static
`aethercore` and carry their own copy of its debug info.

## Fix

- **`ci.yml` build job + `full-suite.yml`:** configure RelWithDebInfo at
`-g1` (line tables only), via `CMAKE_{C,CXX}_FLAGS_RELWITHDEBINFO`.
Backtraces keep file:line; only debugger-level locals and types are
dropped. Both jobs change together because Full Suite restores the build
job's ccache, so the first `main` run after merge builds cold and the
next is warm again.
- **`full-suite.yml`:** the same host-toolchain mounts and "Free runner
disk" step as ci.yml's build job, and a `df` after the build, including
when it fails.
- **`full-suite.yml`:** a "Report a suite that did not build" step runs
only when Configure or Build failed. It writes the report with a stable
per-step fingerprint, and the artifact and tracking-issue steps also run
on that path. Same pattern as #6136.
- **`sanitizers.yml`:** TSan's `shared_core` goes to `"ON"`, the change
the ASan lane proved in #6136.

## Validation

Measured locally (GCC, this tree, `aethercore` plus four core-linking
tests):

| | `-g` (today) | `-gsplit-dwarf` | **`-g1`** |
|---|---|---|---|
| core-linking test executable | 325 MB | 214 MB | **120 MB** |
| `libaethercore.a` | 496 MB | 165 MB | **114 MB** |

At ~250 core-linking executables, that's roughly 50 GB off a tree that
needs a little over 84 GB.

`actionlint` is clean on `full-suite.yml` and `sanitizers.yml`. It
reports two errors in `ci.yml` (`runner.workspace`, lines 451 and 841),
which were already on `main` and are untouched here.

**On the runner:**
- **Full Suite**, branch run
[37208804048](https://github.com/aethersdr/AetherSDR/actions/runs/37208804048)
on this head, green:
  - Free runner disk: 84 → 102 GB free.
- The build tree used **32 GB** (44 → 76 GB used), leaving 69 GB free.
Before, it needed more than 84 GB.
  - **653/653 tests passed** (13.5 min).
- The build took 46 min at a 52% ccache hit rate, because no cached
`-g1` objects exist yet. The first `main` build after merge saves them,
and later runs are warm again.
- **ci.yml `build`**, the required PR check on this head: green, so the
`-g1` configure works on the build job.
- **Sanitizers**, branch run
[37208806037](https://github.com/aethersdr/AetherSDR/actions/runs/37208806037):
still running at merge time. ASan finished building and is running
tests; TSan is still building on the shared core. The result is reported
on this PR after merge. If TSan then fails, it is a sanitizer-lane
problem, separate from the Full Suite and `build` fixes proven above.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ten9876 added a commit that referenced this pull request Oct 5, 2026
…lane, incl. two WDSP races. Principle VIII. (#6173)

## Summary

Refs #6154, #6156. This doesn't close them: the sanitizer-slowdown
timeouts are still open (see below).

The weekly Sanitizers lane recently started building the full test tree
again (#6136, #6152). Its first complete run on `main`,
[37208806037](https://github.com/aethersdr/AetherSDR/actions/runs/37208806037),
failed 6 tests under ASan + UBSan and 11 under TSan. This PR fixes every
one of those failures that was an actual sanitizer report. **In the
branch run
[37230031178](https://github.com/aethersdr/AetherSDR/actions/runs/37230031178),
neither lane reports a single sanitizer error.** What still fails there
is timeouts, plus one plain assertion in `rtl_capture_worker_test`
(below).

Two of the eight commits change shipped DSP behaviour. They are vendored
WDSP patches, documented in `third_party/wdsp/AETHERSDR-PATCHES.md`:

- **Patch 16, transmit path.** `SetTXABandpassFreqs()` was the only one
of four `TXASetupBPFilters()` callers that didn't take `csDSP`.
`TXASetupBPFilters()` clears `bp1`/`bp2 run` first and re-enables them
only after the FIR redesign. So a TX filter change while transmitting
with COMP (and CESSB) on could run blocks with the post-compressor
bandpass bypassed, which is out-of-band splatter. Fix: take the lock, as
`SetTXAMode()` does. It adds no new lock order, and costs one FIR
redesign stall per TX filter change, which upstream already accepts for
a mode change.
- **Patch 17, panadapter analyzer.**
  1. `stop` and `end_dispatcher` are now atomic.
2. `input_busy` is released *after* `stitch()`, so the next frame's FFT
workers can no longer overwrite results while the stitch is still
copying them (a torn spectrum frame).
3. `sendbuf()` now hands off `IQout_index` inside
`BufferControlSection`, closing a lost update.

Only part 2 changes timing: the next frame's FFTs start one stitch
later, tens of microseconds.

| Commit | Lane | Finding | Fix |
|---|---|---|---|
| `1f9c4ed66` | ASan | `offline_health_registry_test`:
stack-use-after-scope. Section 6 reuses a factory that captured section
5's block-scoped `RecorderState` by reference | `rec` moves to `main()`
scope (test only) |
| `cacf9fe30` + `49fa9a75a` | UBSan |
`rx_applet_squelch_reconciliation`, `vfo_dsp_launcher_width`,
`fm_filter_controls`: invalid `QPointer<QLabel>` downcast in
`VfoWidget::eventFilter` during the collapsed label's teardown | Remove
the filter before `~VfoWidget` deletes the label, **and** test the event
type before the guarded-pointer comparison. The second covers the label
dying *first*, after `beginDirectEntry()` `raise()`s the VFO (found in
review). Regression: `gui_nested_lifetime_test`
`vfoCollapsedLabelTeardownInEitherOrder`, covering both child orders |
| `01d06c6d1` | TSan | `receive_intent_routing`, `tci_automation`,
`tci_server_review`: lock-order inversion between two of Qt's own
TLS-backend mutexes, both taken on the main thread | Anchored
`deadlock:QSslSocketPrivate::tlsBackendInUse` suppression |
| `f04a9ffb2` | TSan | `tci_server_review`: unlocked global
`QStringList` log sink written from worker threads | One mutex for the
sink and its lists (test only) |
| `6f6e4a5fd` | TSan | `tx_operation_integration`: the command sink
appends on the `RadioConnection` thread to a list destroyed before that
thread is joined | Declare the list before `radio` (test only) |
| `fbbd24087` | TSan | `wdsp_channel`: WDSP patch 16 (above) | Lock in
`SetTXABandpassFreqs` |
| `af9c15224` | TSan | `anan_rxdsp_handedness`, `spectrum_sequence_gap`,
`wdsp_process_tally`: WDSP patch 17 (above) | Three analyzer hand-offs
synchronised |
| `31e249c3e` | TSan | `weather_radar_loading`: Qt's
`QHttpThreadDelegate` teardown vs a main-thread reply delete. Three map
tests were fetching live OSM tiles | Stub network in those three tests,
plus an anchored `race:QHttpThreadDelegate::~QHttpThreadDelegate`
suppression |

Each suppression in `tests/sanitizers/tsan.supp` carries its reasoning
and a caveat. Neither is anchored on a frame AetherSDR code runs inside.
The file's DO-NOT-ADD list is untouched. Note that a suppression hides
**every** stack it matches, not only the reported cycle or teardown. So
zero TSan reports with these entries means no *unsuppressed* race was
seen. It is not proof that every Qt TLS or HTTP-teardown interaction is
race-free.

**Still failing after this PR, tracked in #6154 / #6156:**
- **Timeouts under the sanitizers' slowdown, with no sanitizer report.**
ASan: `rtl_spectrum_averaging`, `weather_radar_loading`. TSan: the same
two plus `audio_engine_rates`, `nr_stereo_independence`,
`peripheral_auth_dialog`.
- **`rtl_capture_worker_test`, TSan lane only:** fails the assertion "DC
mode change discards old estimates and shows the actual first corrected
window", with no race reported. This branch touches no RTL code, and the
test passed in the same run's ASan lane. It passed in both lanes this
morning, before `main` gained #5919's RTL changes.

**Leak fixes (added after the first push, found by a full local
LeakSanitizer sweep):**

| Commit | Leak | Fix |
|---|---|---|
| `6121f6ad6` | `RadioModel::disconnectClientHandlesThen`: the `step`
lambda captured its own `shared_ptr`, a cycle that leaked on every
client-disconnect chain | Capture a `weak_ptr`; the in-flight `sendCmd`
callback holds the strong reference |
| `7577bab41` + `7f37cf045` | WDSP **patch 18**: `destroy_calcc()` never
closed the five PureSignal `SemsPSCorr` semaphores. 86 leak blocks
across 17 tests (every TX channel destroy) | Close them, and the exit
event, only once the correction thread has signalled exit; on the 500 ms
timeout they are not freed under a live thread. **This does not make the
timeout path safe:** `destroy_calcc()` still frees `a`, which a late
thread would use. That hazard predates this PR and is now #6179 |
| `192e8f09a` | `phone_tx_filter_numeric_entry_test`: `sendEvent(…, new
QFocusEvent)` leaked, since `sendEvent` never takes ownership | Stack
event (test only) |

**Why CI never reported these leaks:** the ASan lane's `ASAN_OPTIONS`
include `halt_on_error=0`, and with that, a leak doesn't change a test's
exit code. The test "passes" and `ctest --output-on-failure` throws its
output away. Verified with `phone_tx_filter_numeric_entry_test`: it
exits 0 with CI's options and 1 with plain `detect_leaks=1`. Fixing that
is a `sanitizers.yml` change and is **not** in this PR.

**Full local sweep (all 653 tests, leaks counted as failures):** 26
failed before these three commits. After them, the only leak left
anywhere is whisper.cpp's ggml Vulkan backend registry, a
process-lifetime static that only appears on a machine with a Vulkan
GPU. Not ours, and CI has no GPU. The other non-leak failures are: the
two known timeouts; `connection_panel_size` and `local_control_server`,
which fail only on this machine with no sanitizer report and pass in CI;
and `rtl_receive_pipeline` and `cw_pcm_consumer`, which fail
intermittently (1 run in 3 serially, and under parallel load,
respectively) with no sanitizer report. Those are flakes already on
`main`, out of scope here.

## Constitution principle honored

Principle VIII: every finding was reproduced before it was fixed. Each
fix that could be checked locally was shown to fail without it and pass
with it; the TSan-only fixes are proven by the branch dispatch. The two
WDSP changes follow the vendoring rules: minimal, commented in place,
and documented in `AETHERSDR-PATCHES.md` with what to keep on the next
WDSP update.

## Test plan

- [x] Local build passes: full tree, Debug, with CI's ASan + UBSan flags
and `AETHER_SHARED_CORE=ON`
- [x] **Red/green for both ASan fixes.** Reverted, all 4 tests reproduce
CI's exact reports (stack-use-after-scope at `:95`, downcast at
`qpointer.h:76`). With the fixes they pass 3/3.
- [x] 71 WDSP, ANAN, TX, HL2 DSP, map and edited tests under ASan +
UBSan. All pass apart from the 2 known 30 s timeouts.
`weather_radar_loading_test` run directly: 44 passed, 0 failed, 26 s.
- [x] **Red/green for the leak fixes.** In a full-suite LSan sweep on
the same tree, before vs after the three leak commits: 86 `create_calcc`
blocks, 5 `disconnectClientHandlesThen` blocks and the `QFocusEvent`
leak, down to none. All the affected tests pass with leak detection on,
`tx_operation_integration` with 421/421 assertions. Leaks from this
machine only (fontconfig/pango/GTK, NVIDIA `cuInit`, ggml Vulkan) were
suppressed locally, not in the repo.
- [x] Sanitizers `workflow_dispatch`
[37230031178](https://github.com/aethersdr/AetherSDR/actions/runs/37230031178)
tested **`31e249c3e`**, the head before the leak fixes and the review
fixes, **not** the final head. Zero sanitizer `SUMMARY:` reports in
either lane, so every originally targeted test is green. The run still
went red on the timeouts and on `rtl_capture_worker_test`. Its duplicate
stickies (#6165, #6170) are closed into #6154 / #6156.
- [x] Sanitizers `workflow_dispatch` on the **final head `7f37cf045`**,
[37243767521](https://github.com/aethersdr/AetherSDR/actions/runs/37243767521):
**zero** sanitizer `SUMMARY:` reports in either lane. ASan failed 2/656
and TSan 5/656, every one a known sanitizer-slowdown timeout.
`gui_nested_lifetime_test` (the new VFO regression),
`phone_tx_filter_numeric_entry_test`, `wdsp_channel_test` and
`rtl_capture_worker_test` passed in both lanes.
- [x] The same head **plus #6176**, so a leak fails its test,
[37243769261](https://github.com/aethersdr/AetherSDR/actions/runs/37243769261),
on the temporary branch `ci/integration-6173-6176` (since deleted): **no
ASan, UBSan or LeakSanitizer report**, and the same timeouts only. Its
stickies #6189 and #6191 are closed into #6154 / #6156.
- [x] **VFO teardown red/green.** The new
`vfoCollapsedLabelTeardownInEitherOrder` aborts on the `qpointer.h:76`
downcast in its "label-dies-first" row with only the destructor fix.
With the event-type-first check, all 10 cases pass, 3 of 3 runs. The 26
vfo/rx_applet/fm_filter/slice/lifetime tests pass.
- [ ] Behaviour on a real radio: patch 16 is a TX-path change. A COMP-on
filter-width change while keyed should be checked on hardware and a
dummy load before it ships.
- [ ] Existing tests pass (CI)

## Checklist

- [x] Commits are signed (`docs/COMMIT-SIGNING.md`)
- [x] No new flat-key `AppSettings` calls
- [x] Code is clean-room
- [x] No meter UI touched
- [x] Documentation updated: `third_party/wdsp/AETHERSDR-PATCHES.md`
(patches 16, 17) and the in-file rationale in
`tests/sanitizers/tsan.supp`. No `CHANGELOG.md`.
- [x] Not security-sensitive

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
W5TSU added a commit to W5TSU/AetherSDR that referenced this pull request Oct 20, 2026
…nes, and HackRF unplug + Windows (#42, #46, #47, #75, #80) (#78)

* fix(hackrf): build the x86_64 AppImage again, and guard the libhackrf API floor

The v26.10.6 tag built every package but the x86_64 AppImage, which failed
to compile: HackRfDiscovery.cpp used BOARD_ID_UNDETECTED, a name newer
libhackrf headers declare but Ubuntu 22.04's libhackrf-dev 2021.03.1 (the
AppImage's build host) calls BOARD_ID_INVALID. Every other build has the
newer header, so only the release tag saw it. Use the number, 0xFF, as
HackRfWorker::readBoardId already does.

No per-PR lane builds against that header, and this is the second break of
this class (#67), so add libhackrf_api_floor: a ctest source scan that fails
when src/ uses a libhackrf name outside the 2021.03.1 header's declarations.
Checked against the 2021.03.1 header itself: every HackRF source compiles
with it after this change, and the scan flags exactly the two lines that
broke the tag.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(sim): close the spectrum gate before announcing disconnect (#75)

The full suite failed once on main with "sim: nothing emitted after
disconnected() (spectrumFrameReady)" in backend_seam_affinity_test.
RadioModel disconnects the simulator's vended RadioConnection directly, and
SimBackend turned that into two queued slots: a signal-to-signal re-emit of
disconnected(), then the handler that clears m_connected and stops the
worker. A spectrum row the worker queued between the two was forwarded after
the announcement, breaking IRadioBackend rule 6.

One handler now closes the gate and then emits disconnected(), so there is
no gap for a row to land in. The core of upstream aethersdr#6088; its session stamp
on spectrum rows depends on refactors this fork does not have.

sim_backend_test pins the ordering: at the moment disconnected() is
announced the backend already reports disconnected, and a row emitted from
inside that announcement is dropped. Both checks failed before the fix.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(icom): send the fail-safe unkey in the call that commands it (#47)

icom_identity_test failed intermittently ("ambiguity releases the previously
selected destination without retargeting unkey"): once under ASan on CI and
2 runs in 40 in a plain local Debug build. sendUserCommand() read the clock
once, queueWrite() stamped the request with a second, later reading, and the
pump at the first reading skipped the request as not yet due. When a
millisecond boundary fell between the two reads, the PTT-off write sat in
the queue until the next timer tick instead of leaving at once.

Pump with a fresh reading. The disconnect path already reads the clock after
queueing its emergency writes; the meter tick's reads are routine and only
wait one tick.

A test seam, m_nowForTest, lets the test make every clock reading later than
the last, which reproduces the race on every run: the new check failed
deterministically before the fix and passes after it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci(sanitizers): fit the ASan lane on the runner's disk, and scale test timeouts under sanitizers (#46, #47)

Both weekly sanitizer lanes have been red for weeks without finding a
sanitizer report:

- ASan + UBSan (#47) stopped building on 2026-10-03: "final link failed:
  No space left on device". Every test executable linked its own static
  copy of libaethercore. A new AETHER_SHARED_CORE option (default OFF,
  never for releases) builds the core as one shared library, and the ASan
  lane turns it on. From upstream aethersdr#6136, with its workflow changes: df -h
  around the build, and a Configure or Build failure now reaches the
  tracking issue instead of skipping the whole report chain. Static checks
  assert per PR that the option yields libaethercore.so and the default
  stays static.
- TSan (#46) timed out weather_radar_loading_test every week. It takes ~15 s
  in a plain build (mostly waiting), 26 s under ASan, and over its 30 s
  limit under TSan. In an instrumented build every test TIMEOUT, its own or
  the 300 s default, is now multiplied by AETHER_SANITIZER_TIMEOUT_SCALE
  (4). Ordinary builds keep every limit as written. From upstream aethersdr#6208.

Verified locally: the whole tree (app and all 512 tests) builds with
AETHER_SHARED_CORE=ON and every test passes; a configure with
-fsanitize=thread scales weather_radar_loading_test to 120 s and a plain
configure leaves it at 30 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(hackrf): notice a HackRF that goes away mid-session (#42)

HackRfWorker declared streamStopped() and HackRfBackend handled it, but
nothing ever emitted it. libhackrf reports a stream that dies under it
(unplug, USB error, reset) only by calling back no more, so the spectrum
froze, the session stayed "connected", and connecting again was refused
with "already connected to a radio".

The backend's 50 ms arbiter tick now also runs
HackRfWorker::checkStreamHealth(). A stream that should be running but
that libhackrf says has ended (hackrf_is_streaming), or that has delivered
no transfer for 2 s, is marked stopped and reported once. The backend then
reports "HackRF stream stopped unexpectedly" and disconnects, in that
order: the old handler disconnected first and emitted the error after
disconnected(), which IRadioBackend rule 6 forbids. A report after the
session has ended is ignored.

hackrf_backend_test pins the rule (streamFault) and the ordering; the
ordering checks failed on the old handler. tools/hackrf_unplug_test.py
checks it on hardware, receive only: it resets the board with libhackrf's
own reset request, so it drops off USB and re-enumerates the way an unplug
does (or waits for a real unplug with --physical). On a HackRF Pro it
passes 9/9 (disconnected in 0.3-0.4 s, still running, reconnects, receives
again); on the old build the app never disconnected and could not
reconnect.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(hackrf): HackRF on Windows (#42)

Every Windows build so far compiled the HackRF backend out: nothing supplied
libhackrf, which ships no Windows binaries. scripts/setup/setup-hackrf.ps1
builds it the way Great Scott Gadgets' own Windows guide does, with MSVC
against vcpkg's libusb and pthreads, at the release the macOS DMG builds
(2026.01.3, from its .zip; the host/libhackrf tree is identical to the
.tar.xz the DMG pins). It stages hackrf.h, hackrf.lib and the hackrf,
libusb and pthreads DLLs in third_party/hackrf.

- CMakeLists.txt: on Windows, link that staging as an imported DLL and copy
  its DLLs next to AetherSDR.exe. The probes now chain on HACKRF_TARGET
  rather than HACKRF_FOUND, which pkg_check_modules caches: keyed on the
  flag, a reconfigure skipped the probe that sets the target and the core
  built without libhackrf's include path.
- windows-installer.yml: build it (cached on the script), configure with
  REQUIRE_HACKRF=ON, and ship the three DLLs in the payload, where
  check-deploy-dependencies.ps1 already proves every import is present.
- ci.yml check-windows: the same build and REQUIRE_HACKRF=ON, so every PR
  compiles the backend with MSVC.
- README: an optional setup step in the Windows build instructions.
  docs/HACKRF.md: a Windows section (no Zadig: the firmware asks Windows for
  WinUSB). Not yet checked against a HackRF on a Windows machine, and the
  doc says so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(hackrf): don't fail the Windows build over a busy scratch directory (#42)

libhackrf built and staged on both Windows lanes; the step then failed on
its last line, removing third_party\hackrf-temp, because something still
held a handle on build\src a moment after the post-link DLL copy. The
scratch tree is now removed with a short retry and only a warning if it
stays busy, and Expand-Archive overwrites a leftover one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci(sanitizers): suppress Qt 6.12's TLS-backend and HTTP-delegate reports (#80)

The TSan image now carries Qt 6.12.0 (it was 6.8.3 on the last weekly
run), and four tests fail on two reports that are wholly inside Qt:
tci_automation, tci_server_review and hl2_tci_signaling on the lock-order
inversion between QSslSocketPrivate::tlsBackendInUse() and the TLS
factory-loader mutex, and whats_new_dialog on ~QHttpThreadDelegate vs a
main-thread reply delete. Upstream suppressed exactly these two, anchored
on the Qt frames, in aethersdr#6173; this takes its tsan.supp
entries verbatim.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant