Repository navigation
ip22, ip32: free a dropped machine's bus and decode table - #447
Merged
techomancer merged 11 commits intoNov 7, 2026
Merged
techomancer merged 11 commits into
techomancer merged 11 commits into
Conversation
The hot lane was one queue for the whole process, but each machine has its own JIT. In a process that holds several machines, a compile worker could take another machine's request: compile that machine's page through its own bus, and publish into a page of a JIT it does not own. Once that JIT is freed, the publish writes into freed memory. That was intermittent heap corruption in the lib tests once dropped machines were freed: "double free or corruption" aborts, and AddressSanitizer reporting a use-after-free in PhysicalCodePage::publish. The lane now belongs to the CompileQueue, beside the FIFO, and the executor holds a handle to it as it already does for the FIFO, so a push costs a field load instead of a static's.
The fault handler offers every fault to every registered table, and each looks at its own bounds to decide whether the fault is its. A table dropped on another thread meanwhile is unregistered and freed while a handler may still be reading it. Tables were rarely freed, so this went unseen. Now that dropped machines free theirs, a test binary frees them all the time while other machines fault. Each slot counts the handlers looking at it, and unregistering waits for that count to reach zero. The count is touched only on a fault, never on a decode.
waits_for gave the receive thread two seconds to take a frame. With a test per core running, the thread did not always get a core that soon, and a_stopped_mac_takes_no_frames_until_it_starts failed. It returns as soon as the condition holds, so allow thirty.
a_status_store_into_a_full_fifo_is_held_back_and_lands_on_retry gave the engine ten million yields to drain the FIFO. A count of yields is no measure of how long the engine thread has had to run, and with a test per core running it sometimes had not run at all. Wait up to thirty seconds instead.
A process can have only 64 decode tables, and a dropped Indy, Indigo2 or IP28 never gave its table back: a test binary that built them one after another failed at the 65th with "devmap: more than 64 tables alive", and the lib tests already ran close enough to the limit that a few more machines made them fail one run in four. Four things held the bus. The lock registry, a static that is never emptied, held strong clones of the HPC3's DMA channels (each holding the bus), the executor, the memory controller and the other devices, so their register_locks methods now register weakly, as the Bridge's already do. The bus's own device table and the devices' DMA handles formed a loop, so the machine's drop now calls Physical::release_devices as the Octane's does. The two SCSI channels each hold the interrupt line that holds both of them, so Hpc3::release lets that go. And VINO's DMA thread, started when the machine is built, held the bus until it is stopped, so the drop stops it. Freeing the machine means its threads must be gone first. The drop never stopped a machine: the CPU, the graphics boards' threads and the HPC3's devices kept running into a freed core and bus (a GR2 holds itself as 'static for its threads, and the display reads the CPU's cycle count through a raw pointer). The drop now stops the machine, as the Octane's does, before it lets anything go. The JIT's compile pool outlives that stop. Its queued requests point at pages whose generation pointers lead into the machine's RAM, so a worker still compiling after the RAM was freed read freed memory. With the executor freed too (the registry no longer holds it), the Octane has the same race. Both drops now stop the CPUs and the compile pool together until neither comes back, since a worker leading a flush restarts the CPU when it finishes. The cost is that the locks monitor command lists a dropped machine's entries as free, as it already does for the Octane.
A dropped O2 kept its decode table, one of the process's 64, so every O2 a test binary built cost it one for good. Two things held the bus. CRIME's rendering engine thread holds the bus from construction and never ended. The machine also leaked a strong reference to its own CPU with mem::forget, which kept the CPU, and through it the bus, alive. The engine now has a stop, which the drop calls, and the CPU is no longer forgotten: the compile pool takes only a weak handle, as on the Indy and the Octane. The drop also stops the CPU and its compile pool first, which it never did, since both now go away with the machine.
compile_queue_send_drops_when_full declared its page after the queue, so the page went out of scope first. Dropping the queue then drains the requests, and each clears its page's scheduled flag: a one-byte write into a dead stack slot, which AddressSanitizer reports as stack-use-after-scope and stops the run on. Declare the page first.
multi_worker_pool_compiles_and_publishes_every_request publishes 2000 pages in about 0.3 s alone, but in the lib tests at 224 threads it ran out of its 30 s with 1525 and 1939 published: still making progress, with no flush, its four workers waiting for cores. Allow 120 s.
every_move_of_the_test_clock_reaches_a_waiting_selection had five seconds for 2000 handoffs between the test and the controller's thread. With a test on every core that ran out twice in a few dozen runs. A lost clock move hangs the loop for good, so a longer bound still catches it: allow sixty.
Both tests race a test thread against a board thread held for a fixed time, and failed in the lib tests at 224 threads when the test's threads waited that long for a core. stop_releases_a_waiting_front_end tells a release by the stop from one by the raster unit's 1.5 s slice by requiring it within 1 s. It took 1.11 s, three times in 40 runs. Hold the slice for 4 s and allow 3 s. a_refused_command_push_retries_behind_the_board_thread must fill the command ring and see the refused store waiting while one 1 s slice holds the board. The slice ended first once in 40 runs. Hold it for 4 s.
Interpreter fallback, the entry check and the instruction enable table are process globals, and tests that flip them hold a lock each while they do. verifier_off_emits_the_same_code compiles each page twice and compares the bytes, which differ when a switch flips in between (1 run in 10 at 224 test threads). It took no lock. It now holds the locks of the switches it depends on.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The lib tests build Indy, Indigo2, IP28 and O2 machines one after another in one process, and a dropped one never freed its RAM, bus, device threads or decode table. A process has only 64 decode tables (
MAX_MAPS, unchanged), so a test run that built more than 64 failed with "devmap: more than 64 tables alive". This frees a dropped machine completely. The same leak keeps a stopped machine's memory alive in iris-gui until it exits (from reading the Stop path; not measured).Freeing machines for real exposed two use-after-free bugs that the leak had hidden; they are fixed first.
PhysicalCodePage::publish). iris-gui can hit this too: Stop then Start builds the next machine in the same process. The lane now lives in eachCompileQueue; a push reads a field instead of a static.mem::forget, and CRIME's rendering thread stops.verifier_off_emits_the_same_code, and timing budgets that ran out with a test on every core (ioc3_eth, mgras, jitv2, isp1040, odyssey).Testing
cargo test --features jitv2 --libpasses at every commit (release) and at the tip (debug);cargo check -p iris-gui --all-targetsis clean.