Skip to content

fix(hl2): prime a receiver-count restart without sleeping the EP2/EP6 thread. Principle VIII. - #5979

Merged
ten9876 merged 10 commits into
aethersdr:mainfrom
on8st:hl2/priming-burst-no-sleep
Oct 2, 2026
Merged

ten9876 merged 10 commits into
aethersdr:mainfrom
on8st:hl2/priming-burst-no-sleep

Conversation

@on8st

@on8st on8st commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

Closes the remainder of row 3.5 of #5678, the half that #5900 does not cover. The two PRs share no lines.

MetisClient::setReceiverCount() spaced its priming banks with two sendPrimingBurst() calls, each containing two QThread::msleep(10). That's about 40 ms with no event loop on the thread that paces EP2 and drains EP6, on every receiver add or remove while streaming.

What changes

The stop → bank → start → bank sequence and its ≥10 ms spacing are unchanged. It now runs from a single-shot timer chain on the same thread, and setReceiverCount() returns at once. Because the event loop keeps turning inside the stop-to-start window:

  • old-layout EP6 packets are discarded in handleDatagram();
  • bandscope run bytes (bit 0 set) are held until the start, and the start retry is paused, so nothing starts the radio before it is primed;
  • a newer restart or stop() cancels one still in flight.

start() keeps the blocking burst, because nothing streams yet at connect time.

Tests

hl2_receiver_count_restart_paced_test is socket-free, with an injected transport. It shows:

  • setReceiverCount() returns in 0.9 ms;
  • the start follows the stop by 21.6 ms, with 13 new-count frames in between;
  • the largest control-frame gap is 3.8 ms, and all old-layout packets are discarded.

It was mutation-checked: restoring the blocking restart, removing the straggler discard, or removing the run-byte hold each turns it red.

hl2_receiver_count_restart_test (fake radio on a loopback socket) settles 150 ms instead of 60, since the start now follows the call by >= 20 ms. Its wedge check now counts only EP6 decoded after the radio's last-marked packet, because stopping the fake radio does not empty the client's socket. ctest -R hl2: 56/56 passed.

On the radio (HL2, gateware 74.2, receive only)

With the stream running I added and removed a panadapter 4 times each, so receivers went 1 → 2 → 1 eight times:

  • every restart logged discarded N EP6 packet(s) in the old layout (N = 1–24);
  • droppedPackets, spectrumGapDiscards0, ep4Drops and the silence-recovery counters stayed at 0;
  • there was no linkDown or linkUp;
  • pan add returned in 0.23–0.50 s and pan close in 0.03–0.05 s.

🤖 Generated with Claude Code

…so adding a pan no longer stalls EP2. aethersdr#5678 row 3.5.

MetisClient::setReceiverCount() primed the new layout with two
sendPrimingBurst() calls -- two QThread::msleep(10) each, ~40 ms -- on the
thread that paces EP2 and drains EP6, with no event loop running. Every
receiver added or removed while streaming froze the EP2 pacer for that long.

The stop -> bank -> 10 ms -> bank -> 10 ms -> start -> bank -> 10 ms -> bank
-> 10 ms -> retry/gate sequence is unchanged; it is now a single-shot timer
chain (RestartStep) whose 10 ms spacing is a floor re-checked on each step.
setReceiverCount() returns after the stop and the first bank.

Because the event loop now turns inside the stop-to-start window:
- handleDatagram() discards EP6 while the restart awaits its start (old
  layout), as the pre-start drain did;
- sendBandscopeRunByte() holds every run byte in that window (bit 0 set = an
  early start); the gate is reset at stop and at start, re-applied at Finish;
- the start retry is stopped and m_sinceLastEp6 restarted at the stop, so no
  run command can precede the priming;
- a newer setReceiverCount() or stop() cancels a restart in flight.

start() keeps the blocking sendPrimingBurst(): it runs before the pacer and
the stream exist.

Test: hl2_receiver_count_restart_paced_test, socket-free (injected C&C and
run/stop sinks; EP6 fed to handleDatagram). Mutation-checked: a blocking
restart, dropping the straggler discard, and dropping the run-byte hold
each turn it red.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@on8st
on8st requested a review from a team as a code owner September 27, 2026 13:52

@aethersdr-agent aethersdr-agent Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Issue fit

Partially closes #5678 row 3.5, and the title says so honestly. Row 3.5 reads: "createPanadapter runs ~40 ms of msleep plus a full OpenChannel with the stream live." This PR closes the first clause completely — the two sendPrimingBurst() calls and their four QThread::msleep(10) are gone from setReceiverCount(), replaced by a four-step single-shot chain on m_restartTimer. The second clause (the blocking OpenChannel, routed in the issue to the same fix as rows 5.1/5.3, the ANAN m_dspBuildThread pattern) is untouched and still runs from Hl2Backend::openReceiverDsp(). The PR title scopes itself correctly to the sleep; the commit trailer and the tests.cmake comment both say bare "#5678 row 3.5", which reads as the whole row. Worth a word in the body so nobody closes the row on this.

The sequence itself is faithfully preserved: stop → bank → ≥10 ms → bank → ≥10 ms → drain/reset/start/bank → ≥10 ms → bank → ≥10 ms → retry+gate. I checked each of the four things the old blocking form got for free and the diff replaces all four (EP6 discard while restartAwaitingStart(), the run-byte hold in sendBandscopeRunByte(), stopping the start retry + restarting m_sinceLastEp6 at the stop, and cancel-on-supersede/cancel-on-stop).

Scope

File What it changes Claimed? Verdict
MetisClient.cpp — sendPrimingBurst/sendPrimingBank split Extracts one bank; sendPrimingBurst keeps both msleeps for start() only Yes In scope
MetisClient.cpp — setReceiverCount + advance/schedule/cancelReceiverCountRestart The timer chain; the bulk of the diff Yes In scope
MetisClient.cpp — handleDatagram early return Discards old-layout EP6 in the window Yes In scope; required by the new event-loop exposure
MetisClient.cpp — sendBandscopeRunByte hold Refuses run bytes in the window Yes In scope; required
MetisClient.cpp — stop() cancel Ends a restart in flight Yes In scope
MetisClient.cpp — sendCommandDatagram / hasCommandTransport / m_commandSinkForTest New test seam for 64-byte run/stop datagrams Yes (test is socket-free) In scope — same established pattern as m_packetSinkForTest
MetisClient.cpp:449 — onEp2PacerTick accepts m_packetSinkForTest Production guard relaxed so the pacer runs socket-free Not stated in the diff's own comments In scope, but it is a production-guard edit made purely for the new test. Naming it in the body would be honest
MetisClient.h RestartStep, the two constants, the members, the doc comments Yes In scope
tests/hl2_receiver_count_restart_paced_test.cpp New socket-free pacing test Yes In scope
tests/hl2_receiver_count_restart_test.cpp Comment-only (the settle rationale) Yes In scope
tests/tests.cmake Registers the new target Yes In scope

No CHANGELOG.md entry — correct. No settings, capability, dialog, a11y or credential surface touched. No new public/protocol surface: RestartStep and both constants are private, sendCommandDatagram is private, and the wire bytes are byte-for-byte the ones setReceiverCount() already sent. No deleted guard: every - line I traced re-appears, most of them relocated into RestartStep::Start (the drain, m_blocks.clear(), both sequence resets, resetBandscopeGate(), failPendingBandscopeFrame(), m_sinceLastEp6.restart(), m_silenceRecoveryArmed = false) or RestartStep::Finish (armStartRetry(), applyBandscopeGate()). No personal-preference change: no default moves, no UI, no user-visible behaviour beyond "the pacer no longer stalls".

Socket tests in this PR (disclosure, not a finding)

  • Added — hl2_receiver_count_restart_paced_test: socket-free, confirmed. No QUdpSocket, no bind, no listen, no peer. Qt6::Network is linked for types only; startInjected() leaves m_socket == nullptr and drives C&C through m_packetSinkForTest, run/stop through m_commandSinkForTest, and EP6 through handleDatagram() directly. This is the right layer for scheduling policy per AGENTS.md's test-layer table. Registered at tests/tests.cmake:1149, so it runs in the full suite and the sanitizer lane; it is not in any per-PR ctest -R filter.
  • Modified — hl2_receiver_count_restart_test: pre-existing socket test, unchanged in that respect. It binds a fake radio (tests/hl2_receiver_count_restart_test.cpp:75-76, QUdpSocket on QHostAddress::LocalHost, ephemeral port). This PR's only edit to it is the settle comment. Not introduced here and not this PR's to remove, but see nit 1 — the change makes its existing spin(60) budget load-bearing in a way it was not before.

Blockers

None. Everything below is non-blocking.

Nits

  1. spin(60) in the sibling test is now the tight one (hl2_receiver_count_restart_test.cpp:166, inline). Before this PR the stop and the start were both on the wire before spin() was even entered, and 60 ms was pure socket-drain slack. Now the chain needs four timer hops and the start does not go out until step 3, so ~44 ms of the 60 ms budget is the client's own schedule and check(startsSeen == startsBefore + 1) has ~16 ms of slack. This test runs in the sanitizer lane. spin(150) costs nothing and removes the risk.
  2. check(maxGap < 8.0, ...) (paced test:216, inline) is a 4× margin on a 2 ms Qt::PreciseTimer pacer. Same lane, same concern. < 9.9 still fails the blocking form (which produced ≥10 ms gaps by construction) while surviving a loaded runner.
  3. onWatchdogTick() stage 1 is the one run-byte path still ungated by restartAwaitingStart() (MetisClient.cpp:577-626, anchored on the reworded comment at :537). Its protection is that m_sinceLastEp6 was restarted at the stop, so the 2000 ms silence timeout cannot expire inside a ≤20 ms window — sound, and the comment states it. But the spacing is a re-checked floor, so the window has no ceiling: a ≥2 s stall on this thread leaves both timers due at once with undefined order, and if the watchdog wins it puts metisRunCommand (bit 0 set = a start) out ahead of the priming and arms a silence recovery the restart is designed to prevent. && !restartAwaitingStart() on the stage-1 condition closes it for free. Reasoned from code, not reproduced — and the stall it needs is the one this PR exists to shrink.
  4. m_lastBandscopeRunByte is assigned before the hold returns (MetisClient.cpp:2372, inline). In the window it records a byte that never reached the wire. Only hl2_ep4_gate_test:68 reads it and that test does no restart, so nothing is broken today — but the field is the send seam's record of what went out, and in this window it is no longer that.
  5. Paced test case 3 does not pin the line it is named for (paced test:281, inline). stop() sets m_running = false, and advanceReceiverCountRestart() already cancels and returns to Idle on !m_running — so both assertions pass with stop()'s new cancelReceiverCountRestart() deleted. The line is correct defence in depth (it also stops the timer immediately); the test just doesn't kill that mutant.
  6. The discard log says "EP6 packet(s)" (MetisClient.cpp:976, inline) but m_restartStalePackets is incremented by handleDatagram() before the EP6 test, so it also counts EP4 and stray discovery replies that arrive in the window.
  7. Hl2Backend.cpp:1196's comment has drifted. "The WIRE FIRST, so the radio is already streaming the new layout before the DSP that consumes it exists" is no longer true — the BlockingQueuedConnection at :1203 now returns after the stop and bank 1, ~20 ms before metis-start. I believe it is benign (the layout was committed before the call returned, handleDatagram() discards the window, and publishIoDsps() is later still), but the stated invariant is worth correcting or re-justifying since it is the reason that call ordering exists.

What I tried to break

  • Null socket on every path the new test reaches. advanceReceiverCountRestart's drain gained m_socket && (correct — without it the socket-free test segfaults at RestartStep::Start); stop() is inside if (m_socket); the start-retry lambda returns on !m_socket. All three hold. Survived.
  • A stale m_restartStep making handleDatagram() discard EP6 forever. The only setters to non-Idle are inside the chain; stop() cancels; advance() cancels on !m_running || !hasCommandTransport(); and start() calls stop() first when m_running (MetisClient.cpp:273), so a reconnect cannot inherit a live step. Survived.
  • Interleaving the restart with a sample-rate change. setSampleRate() (:804) only rebuilds m_ccConfig — there is no second stop/start path in the file. grep for metisStop(/metisStart(/sendPrimingBurst returns only start(), stop(), the retry lambda and this chain. No sibling restart left unfixed. Survived.
  • The GUI thread racing the chain now that the BlockingQueuedConnection returns early. openReceiverDsp() and dsp->configure() run off-thread; the only thing they queue back onto the I/O thread is setRxFrequencyHz, which bounds-checks against the already-resized m_ccRxFreq (:783) and emits a C&C bank, i.e. more priming. The rollback paths at Hl2Backend.cpp:1216/:1259 queue a second setReceiverCount that now supersedes a chain in flight — which is exactly the two-stops/one-start case the new test's part 2 asserts. Survived.
  • Gate state changed inside the window. setBandscopeEnabled(true) → applyBandscopeGate() → bandscopeArm() sets m_bsState and starts the guard with the run byte held; RestartStep::Start's second resetBandscopeGate() un-does exactly that, and Finish re-applies. bandscopeArm() increments no counter, so a held arm inflates nothing. A request landing between Start and Finish double-calls applyBandscopeGate(), but it only arms when m_bsState == Idle, so no second wide_spectrum edge. Disabling the gate in the window also lands right, since the restart's own start is 0x01. Survived.
  • The floor arithmetic. m_restartStepClock is restarted inside scheduleReceiverCountRestartStep() and is never read before it is valid (the Idle guard precedes it, and the step is set immediately before the schedule call in every case). elapsed()'s truncation makes the floor ≥10.0 ms real, not ≤. Re-arming with remainingMs does not restart the clock, so it converges. Survived.
  • Could an EP2 frame start the radio? No: run lives only in the 64-byte 0xEF 0xFE 0x04 <run> command datagram, and bit 0 of the EP2 C0 bank is MOX. The pacer running through the window is priming, as claimed. Survived.
  • Do the tests fail against the unfixed code? Yes for the ones that matter — callNs < 10 ms and maxGap < 8.0 both fail the blocking form by construction; stalePackets == 6 and lastBlockCount == 0 fail if the handleDatagram discard is dropped (the stragglers would decode to 2 blocks against the new m_ccRxFreq); iGate >= 0 with run == 0x03 after the start fails if the run-byte hold is dropped, because the early 0x03 would be found as iStart instead. Nit 5 is the one assertion pair that does not kill its mutant.

Could not check: nothing was built or run — all of the above is read off /tmp/aetherclaude/pr-5979, not reproduced. In particular the timing margins in nits 1 and 2 are arithmetic against the constants (kEp2PacerTickMs = 2, kPrimingBankSpacingMs = 10), not measurements on a loaded runner, and the ≥2 s stall in nit 3 is a code path I reasoned about, not one I triggered. CI is 5/5 green on e0ce156 (build, check-windows, check-macos, Static checks, Sanitizer option configures) — note none of those lanes runs either receiver-count test; both live in the full suite and the sanitizer lane, which is also where nits 1 and 2 would surface.

Recommendation

Approve with nits. This is a careful, well-reasoned conversion: every implicit guarantee the msleep provided is identified and replaced explicitly, the wire sequence and its spacing are unchanged, the new test is at the right layer and genuinely socket-free, and I could not find a reordering, reentrancy or lifetime path that puts a start or a run byte out of sequence. The two I'd actually act on before merge are the spin(60) budget (nit 1) and the stage-1 guard (nit 3) — the first because this PR is what made that 60 ms load-bearing, the second because it is a one-clause change that closes the last hole in the invariant the PR's own comments claim. The rest are comments and log text. Next step: bump spin(60) to spin(150), loosen maxGap, and either add !restartAwaitingStart() to stage 1 or say in the body why the unbounded window is acceptable.

Thanks for the comment discipline here — the "what the event loop running now allows, and what stops each one" block is what made this reviewable at all.


🤖 aethersdr-agent · cost: $9.3972 · model: claude-opus-5

Comment thread tests/hl2_receiver_count_restart_test.cpp Outdated
Comment thread tests/hl2_receiver_count_restart_paced_test.cpp Outdated
Comment thread src/core/backends/hl2/MetisClient.cpp Outdated
Comment thread src/core/backends/hl2/MetisClient.cpp Outdated
Comment thread src/core/backends/hl2/MetisClient.cpp
Comment thread tests/hl2_receiver_count_restart_paced_test.cpp
…ethersdr#5979).

- onWatchdogTick() returns while the restart awaits its start, so stage 1's
  run command (bit 0 set) can never precede the priming, even after a
  multi-second stall inside the window.
- sendBandscopeRunByte() records m_lastBandscopeRunByte only after the
  restart hold, so the seam never reports a byte that did not go out.
- The discard log says "datagram(s)": the counter also takes EP4 and stray
  replies, not only EP6.
- hl2_receiver_count_restart_test settles 150 ms, not 60 (~40 ms of it is now
  the client's own schedule; the retry is still 300 ms past Finish).
- hl2_receiver_count_restart_paced_test: EP2 gap bound < 9.9 ms (the blocking
  form gapped >= 10 ms by construction); asserts restartIdle() synchronously
  after stop() and that a held run byte is not recorded. Both new assertions
  mutation-checked: removing stop()'s cancel, or recording before the hold,
  turns each red.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
hl2_receiver_count_restart_test's "and no EP6, because the radio really
has stopped" failed 2 runs in 50 with eight cores busy, and once at
ctest -j8, while passing serially. The code is not at fault.

The fake radio wedged with `streaming = false` and the test zeroed
blocksSeen in the same instant. That stops the radio WRITING; it does not
empty the client's socket, where the pacer's catch-up burst can leave
several answered EP2. Those were counted as EP6 during the silence.

Now the radio's final EP6 carries a non-zero first RX1 sample and the
radio stops right after sending it. The check counts only EP6 decoded
AFTER that packet. Loopback keeps one socket pair in order, so the check
needs no wait and no bound, and the 1500 ms window is unchanged. The
marker is in sequence, so the loss accounting checks are untouched.
MetisClient.cpp is untouched.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@on8st

on8st commented Oct 1, 2026

Copy link
Copy Markdown
Contributor Author

A flake in this PR's own test, found and fixed. The test was wrong; the code was not.

hl2_receiver_count_restart_test failed its check "and no EP6, because the radio really has stopped" once at ctest -j8 in a local integration build, and passed on a serial rerun.

Cause: the fake radio wedged by setting streaming = false, and the test zeroed blocksSeen in the same instant. That stops the radio writing. It does not empty the client's socket, where the EP2 pacer's catch-up burst can leave several answered packets. The later this thread gets to drain them, the more EP6 there is, and it was counted as arriving during the silence. MetisClient's restart path already documents the same limit for its own socket ("it cannot catch a packet still in flight").

Fix (5735d8bb, test only, MetisClient.cpp untouched): the radio's final EP6 before the wedge carries a non-zero first RX1 sample, and the radio stops right after sending it. The check now counts only EP6 decoded after that packet. Loopback keeps one socket pair in order, so the check needs no wait and no bound, and the 1500 ms window is unchanged. The marker is in sequence, so the loss-accounting checks in that block are unchanged.

Rates, under the same load (eight busy loops on a 12-core Mac, the binaries run alternately so the load hits all of them alike, 50 runs each):

failures
before (this PR's head) 6 / 50, every one on that check alone
after 0 / 50

Full suite green.

73, ON8ST

@aethersdr-agent

Copy link
Copy Markdown
Contributor

Thanks for this one, @on8st — the timer-chain rework and the mutation-checked hl2_receiver_count_restart_paced_test are exactly the kind of evidence that makes a timing change reviewable. The red check is not your code. Details below so you don't go looking.

What failed

Commit 5735d8b, run 36813441823:

job result
check-linux-qt68-floor failure
build, check-macos, check-windows still in progress at the time of writing
Static checks, Sanitizer option configures success

Inside the failing job the failing step is Resolve Qt 6.8.3, and the two steps after it are skipped:

✅ Initialize containers
✅ actions/checkout
❌ Resolve Qt 6.8.3
⏭  Configure against Qt 6.8.3
⏭  Build against Qt 6.8.3

So the lane died before CMake configured anything and before a single translation unit was compiled. Nothing in your diff — MetisClient.cpp, the two new tests — was ever handed to a compiler in that job. (I can see step names and conclusions through the tooling I have here, not the step's log text, so I'm not going to quote an aqt error I haven't read.)

Why it's infrastructure

check-linux-qt68-floor is brand new: it was added an hour before your run by bc4f2b0 (#5842, the Qt 6.12.0 pin). Your run is one of the first to execute it at all — PR #6044's run from yesterday doesn't even list the job.

The step itself is documented in .github/workflows/ci.yml:329-375, and the comment above it predicts exactly this transition:

While that image still ships 6.8.3 the Linux build job IS the floor and this job no-ops; after the image rebuilds to 6.12 it aqt-installs 6.8.3 beside it and compiles again.

The step probes /opt/Qt/6.8.3/gcc_64/bin/qmake in the ghcr.io/aethersdr/aethersdr-ci:latest container; if it isn't there it creates a venv, pip install aqtinstall==3.3.0, and downloads Qt 6.8.3 with four modules. Everything that can fail in that step is a network/registry/mirror operation inside the container — pip, then aqt install-qt pulling ~a gigabyte from Qt's servers. There is no retry and no cache on it, so a single 403/timeout from a Qt mirror fails the lane.

Worth noting: the same job passed on main at bc4f2b0 (run 36811308241) shortly before your run, which is consistent with either a transient download failure or that earlier run still taking the no-op skip=true branch against an image that hadn't rebuilt yet. Either way the trigger is the image/Qt-provisioning transition, not this branch.

What to do

  • Just re-run the failing job (Re-run failed jobs on the run page). If the download was the flake, it goes green with no push.
  • Don't rebase chasing it. For pull_request runs the workflow comes from the merge of base + head, so the job is already present regardless of what your branch is based on; a rebase changes nothing here.
  • If it reproduces on a re-run, it's a maintainer-side fix in ci.yml (retry/caching around Resolve Qt 6.8.3, or an image rebuild), not something to patch from this branch. cc @rfoust for visibility since chore(ci): migrate Windows, macOS, and Linux binaries to Qt 6.12.0 LTS #5842 just landed.

On the review comments

All six bot comments are flagged non-blocking nits and none of them relates to this failure — CI never reached the compile or the test run. They're still worth a pass when you next push, and two are relevant to the lane this test lives in:

  • The spin(60) → spin(150) settle in hl2_receiver_count_restart_test and the maxGap < 8 → < 9.9 bound in the paced test are both about margin in the full suite and sanitizer lanes. Your own numbers (start at +21.6 ms, Finish ≥40 ms out, largest control gap 3.8 ms against a 2 ms kEp2PacerTickMs) say the assertions are right; the nits are only asking for headroom on a loaded runner. Cheap to take, and it removes a future flake that would look like your change.
  • The stage-1 restartAwaitingStart() gate in MetisClient.cpp:545 is a hardening note, not a live bug, as the comment itself concedes — your m_sinceLastEp6.restart() argument holds for any window that isn't multi-second-stalled. One clause makes the invariant structural instead of arithmetic, which matches sendBandscopeRunByte() and the start retry.
  • The m_restartStalePackets wording ("datagram(s)" rather than "EP6 packet(s)") is a one-word honesty fix, since handleDatagram()'s early return runs ahead of the ep6Seq() test.

None of that changes the merge story. Thanks again for the hardware pass — eight 1↔2 receiver transitions with droppedPackets, ep4Drops and the silence-recovery counters all at 0 is the evidence this change needed.

73


🤖 aethersdr-agent · cost: $4.4218 · model: claude-opus-5

@on8st

on8st commented Oct 1, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the diagnosis. Agreed: check-linux-qt68-floor failed at Resolve Qt 6.8.3, before configure, so nothing in this branch was compiled in that lane.

Two short notes:

  • I can't re-run it from here. "Re-run failed jobs" needs admin rights on the repository. Could a maintainer re-run run 36813441823? If it fails again, the fix belongs in ci.yml (chore(ci): migrate Windows, macOS, and Linux binaries to Qt 6.12.0 LTS #5842), not in this branch.
  • The six review nits were already taken on 27 September in 753c3038: the 150 ms settle, the < 9.9 ms bound, the restartAwaitingStart() gate in onWatchdogTick(), the bandscope run-byte record moved after the return, "datagram(s)" in the log, and restartIdle() asserted synchronously after stop(). Each thread has a reply naming the change.

The latest commit, 5735d8bb, is a test-only flake fix. It is explained in the comment above.

73, ON8ST

@on8st

on8st commented Oct 1, 2026

Copy link
Copy Markdown
Contributor Author

The review asked for headroom in the two restart tests "in the full suite and sanitizer lanes". I ran both tests under the sanitizers locally at 5735d8bb. What the run cannot tell you, first:

  • Leaks were not checked. macOS with Apple clang 21: its ASan refuses to start with detect_leaks=1, so the run used detect_leaks=0. The weekly lane uses detect_leaks=1.
  • A clean TSan run says little here. Both tests drive MetisClient on one thread, and the paced test injects its transports, so there is no second thread for TSan to compare against. It does not exercise the live path between the I/O thread and its callers. It also linked a stock Qt, not the instrumented one in the :tsan image.
  • Two tests, one machine (macOS arm64, 12 cores), five repetitions per sanitizer. Not the weekly lane.

Flags, from .github/workflows/sanitizers.yml at 25da98de, passed through CFLAGS/CXXFLAGS/LDFLAGS as the workflow does:

lane build type compile runtime
ASan + UBSan Debug -fsanitize=address,undefined -fno-sanitize-recover=undefined -fno-omit-frame-pointer ASAN_OPTIONS=detect_leaks=0:abort_on_error=0:halt_on_error=0:strict_string_checks=1:detect_stack_use_after_return=1:print_summary=1, UBSAN_OPTIONS=print_stacktrace=1:halt_on_error=0:print_summary=1
TSan RelWithDebInfo -fsanitize=thread -g3 -fno-omit-frame-pointer -O1 TSAN_OPTIONS=halt_on_error=0:second_deadlock_stack=1:history_size=7:print_summary=1:suppressions=tests/sanitizers/tsan.supp

Configured with -G Ninja -DAETHER_ENABLE_HL2_RECEIVER_CHURN_TEST=ON -DAETHER_ENABLE_HL2_TX_LOOPBACK_TEST=ON, Qt 6.8.3 (this head predates #6055), QT_QPA_PLATFORM=offscreen. Only the two test targets were built; the churn and loopback tests were not built or run.

Results, five runs each, the two tests running side by side:

test ASan + UBSan TSan
hl2_receiver_count_restart_test 5/5 pass, 0 reports (13.0 to 14.4 s) 5/5 pass, 0 reports (15.0 to 15.5 s)
hl2_receiver_count_restart_paced_test 5/5 pass, 0 reports (0.4 to 1.7 s) 5/5 pass, 0 reports (2.4 to 2.9 s)

So the timing assertions the nits were about (the spin(150) settle, the maxGap < 9.9 bound) held in 20 of 20 sanitized runs on this machine. That is ten runs per test per lane on an otherwise quiet 12-core Mac; it is not evidence about a loaded shared runner.

Positive control for the ASan + UBSan lane: a small program with a signed overflow and a heap use-after-free, built and run with the same flags and options, reports both. TSan reported plenty on other tests in the same session, so the lane was live.

Still not run: the weekly lane itself (Linux, instrumented Qt, leak detection on, whole suite).

73, ON8ST

…VIII.

main condensed MetisClient's comment blocks (aethersdr#6072); four of them sat on
code this branch restructured. Resolution: the paced restart is this
branch's code, unchanged; main's condensed wording is taken for the
start-retry, stage-1 watchdog, stale-datagram and retry-budget comments,
re-indented into the restart's steps. main's additions in the same file
(the one-shot guard, the CL1 sequence, stop()'s bank drops) are kept.

The comments this branch adds in src/ are brought under the comment
rule. No behaviour change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ten9876 ten9876 self-assigned this Oct 2, 2026
ten9876 and others added 6 commits October 2, 2026 09:43
…nciple VIII.

AGENTS.md § Comments: drop the history (the old msleep form, the flake
narrative) from the paced test's header and the wedge/settle comments in
hl2_receiver_count_restart_test. Comment-only; no assertion changes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

@ten9876 ten9876 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at 9f19569d (main merged in through #5982).

Findings, all addressed

  • The six review nits were fixed by @on8st in 753c303, and I re-verified each one at this head: the spin(150) settle, the maxGap < 9.9 bound, the restartAwaitingStart() gate in onWatchdogTick(), the run-byte record placed after the hold, "datagram(s)" in the log, and the synchronous restartIdle() after stop(). Every thread has a reply and is resolved.
  • Bot nit 7 (the "WIRE FIRST" comment in Hl2Backend.cpp) no longer applies. #5900 (e48a242) removed that comment. The remove path calls withdrawIoDsps() before setReceiverCount(), so a list republished inside the stop-to-start window feeds nothing, because handleDatagram() discards everything until Start.
  • 51d2dcf: the test comments are cut down to what is true now (AGENTS.md § Comments). Comment-only.

Interaction with #6014 (checked against the code)

  • m_running stays true for the whole chain, so queueOneShotIfRunning() never drops a bank the restart needs. The restart's banks are full-rotation sendControlPacket() frames, not one-shots.
  • stop() mid-chain calls cancelReceiverCountRestart() first, so nothing goes out after the stop. The paced test's part 3 checks this synchronously.
  • A second setReceiverCount() mid-chain cancels the chain and restarts it. Part 2 checks for two stops, one start, and the last count on the wire.
  • m_restartTimer is a child of MetisClient, which Hl2Backend moves to the I/O thread (moveToThread(m_ioThread)), so it shares a thread with m_ep2Timer and the watchdog. No seam signal is added, so the IRadioBackend threading contract is untouched.

Verified empirically (Linux, RelWithDebInfo)

  • Full socket-free ctest: everything passes except connection_panel_size_test (known locally, #5811). The hl2_* socket-free set (69 tests) passes at this head.
  • hl2_receiver_count_restart_paced_test passed 128 times out of 128, including 64 runs 32-way parallel against 32 busy loops. The largest C&C gap was 5.82 ms against the 9.9 bound.
  • Mutation check: putting the blocking sendPrimingBurst() back in setReceiverCount() turns the paced test red on both "returns without waiting" and "no C&C gap near 10 ms". Restoring the file (sha256 matches HEAD) turns it green again.
  • Static gates (engine boundary, touchpoint manifest, capability records, command plane, test registration) all pass with no MetisClient findings.

Not run locally: hl2_receiver_count_restart_test binds a loopback fake radio. The operator's rule is that socket-owning tests are not run locally, so its proof is CI's full and sanitizer suites, plus @on8st's macOS ASan+UBSan / TSan runs (5/5 per lane, 0 reports). I did not exercise the live HL2. The receive-only hardware pass is @on8st's.

@ten9876
ten9876 enabled auto-merge (squash) October 2, 2026 18:47
@ten9876
ten9876 merged commit 1352368 into aethersdr:main Oct 2, 2026
5 checks passed
ten9876 added a commit to on8st/AetherSDR that referenced this pull request Oct 2, 2026
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@on8st

on8st commented Oct 3, 2026

Copy link
Copy Markdown
Contributor Author

Receive-only hardware pass: HL2, gateware 74, 192 kHz, main 732b99a9. Bad news first, none of it inside this PR's stop-to-start window:

  • The first EP6 packet after every start is still in the old layout. Same on a build without this PR (90 of 90 restarts).
  • Once, 0.26 s after a 2→3 start, EP2 stopped for 706 ms and the app dropped 923 packets. Not reproduced.
  • The supersede path was not reached on hardware.

56 restarts (1→2→3→2→1 and 1→2→1), tcpdump on the wire:

per restart with #5979 without
largest EP2 gap 3.9–7.8 ms 11.2–13.1 ms
stop → start 20.3–27.2 ms 20.8–25.3 ms
EP6 gap 20.6–27.3 ms 21.0–25.4 ms
EP6 lost on the wire 0 0

No link-down, no silence recovery; every receiver came back.

Two limits on the comparison. The "without" column is not this PR reverted: it is a build from before it (f86e11a7), 20 HL2 commits older. And the Mac was busy during the runs (load 9 to 18 on the build with the PR), so the single 706 ms stall cannot be put down to the code or to the machine; it happened in one of two sessions with the PR and in neither without.

73, ON8ST

ten9876 added a commit that referenced this pull request Oct 3, 2026
…ciple VI. (#6104)

## Summary

On HL2 and Icom, a TUNE power change made while TUNE is keyed now
reaches the carrier. Before this, the value waited for the next
key-down, and the operator saw a "dropped" notice for a control that
looks live. This PR also fixes two TX-power defects in the same paths:

- **HL2 transmit tail.** At TUNE release the RF drive was restored right
after MOX-off, but the HL2 keeps radiating for its FIFO drain plus PTT
hang after the MOX-off frame. The restore landed under the
still-draining full-scale tune tone. The drive restore and the receive
filter switch now wait out that tail. (Found by ON8ST.)
- **Icom restore.** An RF power change made during TUNE was written over
the tune drive and then lost at the unkey.

## What changed

**Seam and capability**
- `IRadioBackend::setTunePower(int)` is a new verb with a default no-op,
next to `setTxPower`. It never keys. A backend that generates its own
carrier re-applies drive to a carrier in progress and otherwise does
nothing; the next `setTune()` carries the value. Owner-thread contract
as for `setTxPower`.
- `RadioCapabilities::TransmitDriveControl::tunePowerAppliesLive`
defaults to `false`. It is a field inside the existing record, not a
top-level bool, so `FROZEN_BOOL_COUNT` stays at 71
(`check_capability_records.py --strict` reports 71 on both base and
head). The contract text in `IRadioBackend.h` and
`docs/architecture/radio-capabilities-map.md` are updated to match.

**Models**
- New signal `TransmitModel::tunePowerCommandIssued(int)`. Only
`setTunePower()` emits it, after `commandReady`, mirroring
`rfPowerCommandIssued`. `applyChanges()` (status decode) never emits it.
- `RadioModel` connects to it in the constructor's model-lifetime block,
next to `rfPowerCommandIssued` (#4599). It forwards a change only when
all of these hold:
  - `isTuning()` is true
  - a live `TxActivity::Tune` contribution exists
  - the backend declares `tunePowerAppliesLive`
- The connect checks `tunePowerAppliesLive` before calling
`setTunePower`. The behaviour is the same for every real backend,
because the base verb is a no-op. The check makes the contract explicit:
the field promises the verb is real, and a backend that doesn't declare
it is never handed the call.
- Both the forward and the drop-notice gate use one predicate,
`RadioModel::tuneCarrierLive()`: `isTuning()` AND a live
`TxActivity::Tune`. The gate is `routed = canTransmit &&
transmitDriveControl && (!isTuning() || (tuneCarrierLive() &&
tunePowerAppliesLive))`. While tuning, a value that is not forwarded
live always keeps its notice: a backend without the field, or a tune
state decoded off the radio with no TUNE this client admitted.
- Every `TransmitDriveControl` is built with the field stated
explicitly, as the ADDING A FIELD rule requires: `true` on HL2 and Icom,
`false` on Flex (whose tune power travels as command-plane text) and
ANAN. Sim and RTL declare no record.

**HL2**
- `Hl2Backend::setTunePower` writes only the drive register, through
`applyDrive()`, so the clamp and the transmit gate still apply. It
doesn't touch `m_rfPowerPercent`, band memory or
`notifyOperatingStateChanged`, and it doesn't route through `setTxPower`
or re-call `setTune(true)`.
- It acts only while `m_tuning` is true. It also acts only while the
TUNE operation, stored as `m_tuneOperation` in `setTune(true)`, still
permits dispatch. That optional defence is included: a TUNE whose
admission was revoked or emergency-stopped takes no drive change. Every
unkey and the connection reset clear `m_tuneOperation`, so it never
outlives the carrier.
- `tunePowerAppliesLive = true` in the HL2 capabilities.

**Icom**
- `IcomCivBackend::setTunePower` writes CI-V `14 0A` (the tune drive)
while tuning and does nothing otherwise. It also requires the same
admission `setTune(true)` checked: the TUNE operation stored there, and
the backend's transmit context. The operation is cleared at every unkey
and session reset. A refusal is logged on `aether.icom.tx`, because the
verb returns nothing to `RadioModel`. `tunePowerAppliesLive = true` in
the Icom capabilities.
- **Restore fix.** `setTxPower` during TUNE now records the value in
`m_preTuneTxPowerPercent`, which the unkey (and the disconnect emergency
path) writes back. It no longer writes over the tune drive.
- The raw write is factored into `writeTxPowerLevel()`. The TUNE
re-request branch in `setTune(true)` and the `tx.power` controls scrub
both use it, so neither can stamp the tune drive into the restore slot.
- **Readback during TUNE.** The CI-V read that confirms the tune-drive
write (`14 0A`) is no longer published as `TransmitDelta::rfPower` while
tuning. The periodic poll already skipped that read during TUNE. Without
this, `TransmitModel::rfPower()` showed the tune level. A relative
control (the power wheel, or a CAT/TCI read-modify-write) would then
step from that level, and the result would become the value the unkey
restores. Example: 100% RF and 10% tune, then one wheel step, left the
radio at 11%. *Visible effect:* on Icom the RF power slider no longer
jumps to the tune level while TUNE is keyed. It keeps the operator's RF
power, matching Flex. (Found by code review of this branch.)

**Evidence for the readback leak, and its limits.** This comes from the
code path and a scripted-reply unit test. It is **not
hardware-verified**: no Icom was on the bench.
- Icom has no tune-carrier command, so TUNE writes the tune level into
`14 0A`.
- `sendUserCommand()` queues a confirming read of that level 60 ms later
(`queueRead(..., now + 60)`).
- The `kRfPower` reply handler published it as `TransmitDelta::rfPower`.
- `icom_backend_seam_test` scripts that CI-V reply. It does not observe
a real radio's answer.

**Still wanted:** a CI-V capture from a real Icom of TUNE plus one
RF-power wheel step. It should show the confirming `14 0A` read during
TUNE and confirm that the restore after the unkey is the operator's RF
power.

## The HL2 transmit tail (found by ON8ST)

ON8ST showed from the gateware RTL that the HL2 keeps transmitting after
the MOX-off frame:
- MOX travels with the samples, not as a register.
`dsopenhpsdr1.v:351-352` tags each sample with `ds_cmd_ptt`.
- In `PTTTX`, `radio.v` drains the keyed samples already queued in the
FIFO (`tx_buffer_latency`, 20 ms default, `radio.v:934`). AetherSDR
never writes 0x17, so the default applies.
- It then holds the last sample for `ptt_hang_time` (12 ms default,
`radio.v:935`, 1036-1047). `tx_on` stays high until the state machine
reaches `NOTX` (`radio.v:1002`).
- Drive is not buffered. `ad9866ctrl.v:134-140` loads the TX gain as
soon as the command arrives.

So restoring RF drive one frame after MOX-off still landed under roughly
20 + 12 ms of full-scale tune carrier. ON8ST's unit (gateware 74.2)
unkeyed itself 40-54 ms after the host stopped sending keyed frames. The
same applies to the receive filter switch at unkey.

**Why a timer, not `tx_on`.** `tx_on` is reported only in the discovery
/ alternate-port reply: `resp_control` (`control.v:899`) feeds
`usopenhpsdr1.v:292` (`6'h20`). The EP6 stream's responses carry
`ptt_resp = cw_on | ext_ptt` instead (`control.v:456`, 472-475), which a
host MOX never raises. `MetisProtocol.cpp:546` is in
`parseDiscoveryReply()`. Reading `tx_on` mid-stream would mean polling
the discovery port during TX, which the stream-free telemetry design
avoids because that reply preempts EP6 (`Hl2TelemetryCadence.h`).
Gateware without `EXTENDED_RESP` also sends zeros there, so a "`tx_on`
low" trigger would fail open.

**Fix (`Hl2Backend`).**
- Every key-up arms a transmit tail of `kTxTailHoldMs` = 70 ms, after
the MOX-off is queued.
- 70 ms is never below the existing RX unkey hold (a `static_assert`
pins it). It covers the gateware's 20 + 12 ms default tail and ON8ST's
40-54 ms measurement. The RX-hold measurement of the same tail on one
HL2 was 51.66-66.15 ms, n = 11.
- While the tail is pending, `applyBandFilter()` keeps the transmit
filter byte, and `setTxPower()` records an owed RF drive without writing
it.
- When the tail ends, the receive filter and the owed RF drive are
applied.
- A re-key cancels the tail, so a restore is never applied over a new
carrier, and the drive stays owed to the next unkey's tail.
- A disconnect or connection reset drops the tail.

Icom is unchanged: its PTT-off is queued before the power restore, and
the radio owns its own timing.

**Maintainer decisions (Jeremy):**
1. A fixed 70 ms timer ends the tail. There is no `tx_on` poll, and 0x17
is not written.
2. A voice key inside the tail of a TUNE release keeps the tune drive
for that transmission. The restore is cancelled, never applied over the
new carrier, and RF drive returns at the next unkey's tail.
3. The transmit filter byte is held for the tail on every unkey,
including CW break-in elements.

## Out of scope (unchanged)

- No changes to TCI `tune_drive` or MIDI power limits. Note that TCI
`tune_drive` and the automation `transmit tunepower` verb go through
`TransmitModel::setTunePower`, so on HL2/Icom they now also apply live
while TUNE is keyed, as they already did on Flex.
- No slider dimming or tooltip UX for backends that don't declare the
field.
- No tune-power ceiling tied to RF power. Flex keeps them independent,
and so does this PR.
- Drag flood is accepted as is, matching RF power: no coalescing and no
debounce.

## Tests (all socket-free; nothing keys real hardware)

- `transmit_seam_drop_notice_test`:
- Flag on: `setTunePower(30)` arrives exactly once, with no re-key and
no drop notice.
  - Flag off: no call, and the notice stays.
  - Not tuning: no call, and the next `startTune` carries the value.
- A decoded tune power while keyed (`applyChanges`, a different value)
sends no call.
  - After `stopTune`, no call.
- A tune state decoded off the radio (`applyChanges`, `tune=true`) with
no live Tune activity: no call, and the drop notice stands. This pins
the Tune-activity check on its own, in both the forward and the gate.
- `hl2_tx_gate_test` reads the 0x09 bank bytes off the real EP2 packet
builder.

  | Step | 0x09 bank result |
  |---|---|
  | RF power 40% | drive 102 |
  | TUNE at 10% | drive 25 |
  | live 60% | drive 153, PA on |
  | 150% | 255 |
  | −5% | 0, PA off |
  | release | 102 restored |
  | 80% after release | no bank, no keyed frame |
  | gate closed while tuning | 0, PA off |
  | TUNE operation emergency-stopped | no bank |
  | release | stored TUNE operation dropped |
  | stopped client | no bank |

The same file has the transmit-tail tests, with the hold parked so each
test ends the tail itself:
- The unkey's queued calls post no RF-drive restore; walked packet by
packet with an event filter on `MetisClient`.
  - The tail ending restores RF drive.
- A re-key inside the tail cancels it; no restore under the new carrier,
and the next unkey's tail pays it.
  - TUNE re-keyed inside the tail keeps its tune drive.
  - RF power set during the tail waits for it.
  - A live tune drive with no key-time tune power is still restored.
  - A disconnect drops the tail.
- On a TX-only filter board, the TX byte holds through the tail, and
through a retune inside it, until the tail ends.
  - The real timer ends a 20 ms tail.
- `icom_backend_seam_test` checks the CI-V `14 0A` writes:
  - The capability is declared.
  - Not tuning: no write.
  - While tuning: the live 25% is written, with no re-key.
  - RF 50% during TUNE: no write, and it is held for restore.
  - A TUNE re-request writes 15% and keeps 50% held.
- A `tx.power` scrub during TUNE re-asserts the live 15% and keeps 50%
held.
  - Release writes 50%.
  - After release: no write, and the stored TUNE operation is dropped.
  - A TUNE whose operation was emergency-stopped: no write.
- A lapsed transmit context with the operation still live: no write, and
one warning logged.
- An RF power readback of the tune level during TUNE publishes no
`rfPower`. The premise is checked first: the same readback outside TUNE
does publish.

### Mutation results (each mutation made its test fail, then the file
was restored and verified by sha256 against HEAD)

| Mutation | Result |
|---|---|
| RadioModel connect deleted | RED |
| gate reverted to `!tuneKeyed` | RED |
| flag ignored (gate + connect) | RED |
| wired from `tunePowerChanged` instead | RED |
| `isTuning`/Tune-activity check removed | RED |
| Tune-activity check alone removed | RED |
| HL2 `m_tuning` guard removed | RED |
| HL2 implemented via `setTxPower` | RED |
| HL2 `applyDrive` bypassed | RED |
| HL2 tune-operation check removed | RED |
| HL2 flag not declared | RED |
| HL2 tail deferral removed (restore at unkey) | RED |
| HL2 re-key cancel removed | RED |
| HL2 `setTxPower` tail check removed | RED |
| HL2 filter predicate ignores the tail | RED |
| HL2 disconnect cancel removed | RED |
| HL2 `setTunePower` not marking drive owed | RED |
| Icom restore fix reverted | RED |
| Icom `setTunePower` guard removed | RED |
| Icom `setTunePower` no-op | RED |
| Icom TUNE re-request via `setTxPower` | RED |
| Icom scrub via `setTxPower` | RED |
| Icom flag not declared | RED |
| Icom tune-drive readback published during TUNE | RED |
| Icom tune-operation check removed | RED |
| Icom tune operation not stored at `setTune(true)` | RED |
| Icom tune operation not cleared at unkey | RED |
| Icom transmit-context check removed | RED |
| Icom refusal warning removed | RED |
| Drop-notice gate ignores `tuneCarrierLive` | RED |
| Forward uses `isTuning()` alone | RED |
| HL2 tune operation not cleared at unkey | RED |

## Merge with main

The branch was cut from `2d5712946`. It now carries signed merges of
`origin/main` up to `c7513e125` (27 commits, including #6062, #6005,
#5979, #5982, #5976, #6011, #5980, #6019, #5981, #6012 and #6050 on HL2,
and #6049 in `RadioModel`). Every merge was clean. The files both sides
touched (`Hl2Backend.cpp/.h`, `IRadioBackend.h`, `RadioModel.cpp`,
`radio-capabilities-map.md`) needed no manual resolution, and none of
main's changes in them touch keying, drive or TUNE. Everything below was
re-run on the current head.

## Static gates (merge base `c7513e125` vs merged head, findings diffed
per file)

`check_engine_boundary.py --strict`, `gen_touchpoint_manifest.py
--check`, `check_capability_records.py --strict` and
`check_command_plane.py --strict` all produce identical findings on base
and head once line numbers are normalised:

- Engine boundary: 102 tracked findings, 0 blocking.
- Touchpoint manifest: up to date.
- Capability records: 71 bools.
- Command plane: 401 commands across 32 files.

No ratchet moved, and no new wire text was added.
`gen_seam_probe_table.py --check` is clean (51 signals): this PR adds a
seam verb but no seam signal.

## Not verified

- No live-radio test on HL2 or Icom; per the TX rules, nothing was
keyed. The Icom path, including the readback leak and its guard, is
proven only against the backend's CI-V output and scripted CI-V replies.
A real-Icom CI-V capture of TUNE plus one RF-power wheel step is still
wanted.
- Icom's RF-power readback is suppressed during TUNE. CI-V cannot tell
the confirming read of the tune drive from a front-panel change, so an
RF-power knob turned on the radio during TUNE is not seen and the unkey
restores the pre-TUNE value (the code comment says so).
- The HL2 transmit-tail fix has no hardware test. Its 70 ms rests on the
gateware source defaults and on bench measurements made before it
existed: ON8ST's 40-54 ms unkey, and the RX unkey-hold measurement of
the same tail. Nothing here observed `tx_on` or RF output with this
change.
- Full default ctest at -j8 on the merged head, with every test whose
source uses a socket API excluded (523 of 620 run; the pty-backed
`thumbdv_queue_test` and four new socket tests from main are excluded
too): all pass except `connection_panel_size_test`, which is a known
local failure (#5811). `transmit_seam_drop_notice_test`,
`hl2_tx_gate_test` and `icom_backend_seam_test` pass.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants