Repository navigation
cpu: serve loads, stores and fetches from lines the guest stages with CACHE index operations - #448
Draft
jamesbraid wants to merge 7 commits into
Draft
jamesbraid wants to merge 7 commits into
jamesbraid wants to merge 7 commits into
Conversation
The R10000 family has 44-bit virtual and 40-bit physical addresses, a FrameMask register and a wider EntryLo; iris kept the R4400's 40 and 36. The R4400 and R5000 are unchanged throughout. Virtual width. EntryHi, the TLB match and the xuseg bounds check all stopped at VA bit 39, but VPN2 is 43:13 and the user segment is 16 TiB (R10000 UM pp. 219, 288). A 64-bit kernel can put an n64 user stack above 40 bits (0x0fff_ffff_aee4, for one), and EntryHi dropped VA[43:40] so such an address took an Address Error. The width now comes from the part's VA_BITS in each place: the EntryHi write mask, the TLB's 64-bit compare (Tlb::set_va_bits, installed by the executor beside set_cache_attrs), the xuseg bounds check, the micro-TLB's cacheable segment test and the EntryHi a TLB exception loads. The mask was spelled three times and is now one const fn, mips_tlb::vpn2_mask. All of it is a const of the CPU model or set once at construction; the TLB lookup is unchanged. This reverses a comment in the executor (b7d2a18) that said widening the EntryHi mask breaks the IP28 and Octane kernels. FrameMask (CP0 register 21) is new with the R10000. Bits 15:0 clear EntryLo bits 33:18 as a TLB write passes them on, and the rest of the register reads as zero (R10000 UM v2.0 p. 260). iris had no register 21, so a kernel that sets it to keep software flags out of the physical address got no effect. The core holds the register; the executor sets which bits are writable, all 16 on the R10000 and none on the R4400 and R5000. TLBWI and TLBWR pass the masked EntryLo values and TLBR returns the masked entry. The register is saved with the rest of CP0 and loads as zero from an older save. CpuModel::R10K_FAMILY says a part is in the R10000 family. Nothing changes while FrameMask is zero, its reset value. EntryLo holds a 28-bit PFN in bits 33:6, a 40-bit physical address, and the two-bit uncached attribute in bits 63:62; 61:34 are reserved (R10000 UM v2.0 pp. 239-240). iris kept 30 bits, so the attribute and PFN bits 33:30 read back as zero and never reached the TLB. The core has an EntryLo mask the executor sets per part: 0x3FFF_FFFF on the R4400 and R5000, 0xC000_0003_FFFF_FFFF on the R10000. The attribute is stored and read back only. Bits 33:32 are physical 39:38, where the Octane's guest keeps software flags; FrameMask clears them on a TLB write, so the page lands at its real address. A TLB write also clears the PFN bits below the page size, so TLBR does not return what was written to them (R10000 UM v2.0 pp. 299-300). The manual documents this for the R10000 only; the later parts follow it as a model choice. Translation already ignored those bits, so only TLBR changes. The Octane PROM's tlb diagnostic writes 0xa5a5a5a4 and 0x5a5a5a58 with 16 KB pages and expects 0xa5a5a524 and 0x5a5a5a18 back; it reported 320 compare errors. On the Octane, the IP28 and an O2 set to the R10000, EntryHi keeps 44 bits and mfc0 of EntryLo returns bits 63:62 and PFN bits 33:30 if the guest wrote them. Tests: a page at 0x123_4000_0000 is kept by the R10000 and masked, with an Address Error, by the R4400; and a TLB write on each, checking FrameMask, the wider EntryLo and the PFN bits below a 16 KB page through TLBR.
A PROM's cache walk reads and writes the caches with Index Load Data, Index Store Data and Index Store Tag. The R10000 models dropped or kept the wrong things. Primary-cache data. Only the secondary cache had a data array, so an Index Load Data on a primary cache always returned zero and an Index Store Data was dropped. The primary caches now have data arrays, and each store is masked to its diagnostic field (R10000 UM pp. 262, 271-272): 36 data bits and 1 parity bit for the instruction cache, 32 and 4 for the data cache, 64 and 10 for the secondary. An Index Store Data to the secondary cache writes a quadword, so the other doubleword becomes zero and takes the same check bits (p. 91). A mapped primary Index Load Data or Index Store Data still takes TLB Refill and Invalid, so CACHE now translates the operand for those two; the other index operations still do not. The secondary data slot is now the address >> 3 where it was >> 4, so the walk gets one slot per doubleword instead of two doublewords sharing one. Secondary tags. Index Store Tag stored all 64 bits of a secondary-cache tag, so a walking-1s test read back bits the R10000 defines as reserved, which the manual says return zero (R10000 UM p. 270, figure 11-31). MRU, the physical tag, state, virtual index and ECC are kept, which assembled is 0x8000_000f_ffff_cdff, and the rest dropped. The mask is CpuModel::SECONDARY_TAG_MASK, all ones except on the R10000 family. MRU stays because the PROM writes it as TagHi bit 31 and reads it back. The later parts share the tag layout (the R12000 manual agrees; the R14000 is taken to match), and a PROM's tag test on the R14000 expects the reserved bits to read as zero. Index Store Data is a diagnostic walk's operation, so it runs out of line in a cold function. Which encodings translate, store a tag, load a tag or load ECC is decided by CacheOp::decode and tested as a bit of a mask computed once per model (CpuModel::CACHE_OPS), not by a chain of compares: written inline the extra case grew cache_op and exec_cache in every CPU model and moved the interpreter's hot loops, which cost sys/llsc about 2% on a ship build. This affects the Octane, the IP28 and an O2 with an R10000. Tests: a primary round trip and the quadword store; an all-ones tag store on the R10000 reads back the mask with an R4400 control that keeps TagLo bits 12 and 13, and the walking-1s round trip expects the reserved bits as zero.
Refill vector. The R10000 picks the TLB refill vector from the address space the missing address lies in and that space's Status bit: user addresses follow UX, xsseg SX, and kseg3, sseg/ksseg and xkseg KX. The current mode only decides which space an address is in (R10000 UM v2.0 pp. 333-334, 342). The manual contrasts this with the R4400, which picks by the current mode's bit; iris did the R4400 thing on every model. So a 64-bit kernel (KX=1, UX=0) missing on a kuseg address went to the XTLB vector at 0x080 where the part uses 0x000. The Octane PROM's utlb diagnostic loads VA 0 in exactly that state and waits at the 32-bit vector; it got "Read TLB miss exception" from the XTLB vector and failed. On the R10000 family a refill miss now reads the region of the address, sign-extended from 32 bits in 32-bit mode, and tests UX, SX or KX. Only the miss path changes; the TLB compare width still follows the mode. Overlapping writes. The R10000 and R12000 do not shut down on a multiple match. A TLB write that would let one lookup match two entries first invalidates every entry it overlaps, the wired ones included, then writes the new entry and sets Status.TS (NEC U10278EJ4V0UM, Status register TS bit p.223 and "Avoiding TLB Conflict" p.300). iris stored the entry and left the duplicates matchable, so which one answered depended on how the lookup map happened to be rebuilt. On the two-processor Tezro the firmware flushes the TLB with 64 invalid entries on the kernel's page number, and the kernel then writes the valid global 16 MiB entry over them; a later write elsewhere left an invalid duplicate answering, and the fetch took TLB Invalid. The R14000 gets the same rule as a model choice; no manual for it was read. The R4400 and R5000 keep their behaviour. An overlap is two entries a lookup could match together: same region and page number under the wider of the two page masks, and equal ASIDs or either entry global. Invalid entries count. The removed entry reads back as an unwritten slot except for its PageMask: the kernel TLBRs its wired sentinel entries and reuses PageMask for the next write, and a zeroed mask made it write a per-CPU mapping as 4 KiB. The cost is confined to TLB writes on these models. An entry that overlaps the new one covers a lookup-map slot the new one covers, so a write whose slots are all empty or its own skips the scan. A PageMask that is not a run of ones, which software can write, defeats the map's ranges and turns the shortcut off. A refill that replaces a one-page entry nothing else covers skips the walk over the other entries. TLBWI and TLBWR are never compiled by jitv2, so both reach the same write and set TS in the interpreter. Tests: a 64-bit kernel missing on kuseg and xsseg on the R10000 and on the R4400, which chooses by mode; the Tezro sequence with the removed entries' PageMask kept; and TS from TLBWI and TLBWR. They fail with the rule off.
Uncached stores nothing answers. A store that nothing answers raised a synchronous Data Bus Error with a precise PC on every CPU. The R10000 cannot do that. It sends an uncached store as a write request, and a write request gets no response; Bus Error comes only from an error response to a read or upgrade request, or from uncorrectable read data (R10000 UM pp. 104, 316). The processor never learns that a store failed. The R12000 manual documents the same no-response write path, and the later parts are taken to match, so the test is the R10000 family. CpuModel:: POSTS_UNCACHED_STORES, true for the family's caches, makes a refused store log and complete in write_data_impl and in the masked-store path (SWL, SWR, SDL, SDR), which the interpreter and the compiled code both go through. An uncached SC or SCD that is refused completes and succeeds (rt = 1); that is a model choice, since the manual says nothing about uncached store-conditional. Loads, instruction fetches, cached accesses and the R4400 and R5000 keep the exception, and device return values are untouched because BUS_ERR also tells non-CPU callers that a request was declined. The error goes no further: nothing in iris raises an interrupt or sets an error register for a store the Octane's bus refuses. The log line prints the first four refused stores and then each power of two, with the running count, so a probe loop cannot flood stderr. The Octane's ide diagnostics store 0x591204 and then 0x4 to physical 0x1e130400, where nothing answers; before, the first store took a Data Bus Error, now both print one line and the run carries on. O2 memory alias. A mapped page whose physical address has a bit set above bit 31 could reach the PCI window instead of RAM on the O2. The O2 aliases RAM at 0x8000_0000, and fold_phys tested that window on the full translated address before cutting it to the 32 bits the board decodes. IRIX keeps software bits in its page-table entries just above the PFN, so such an address missed the window, then lost the high bit in the mask and landed on the PCI range. Cut to the decoded bits first, then apply the alias, then cut its target. The extra AND runs only on an alias hit. Only the O2 has a physical alias. Partial LWR. In 64-bit mode a partial LWR (offset 0 to 2) sign-extended the merged word on every CPU. The manual calls the upper word of a partial LWR implementation specific (NEC U12754E pp. 125-126), and the reference results observed for the R10000 keep the old bits 63:32 at all three partial offsets; with sign extension a test that compares those results miscompares. The interpreter and the JIT now merge the old upper word on the R10000; a full word (offset 3) still sign-extends. The upper word is kept only when 64-bit operations are enabled for the current mode, matching QEMU; otherwise it sign-extends. The later parts are not covered, since nothing shows what they do. The choice is a JitConsts field set from CpuModel::PARTIAL_LWR_KEEPS_UPPER, so compile workers see it. Whether 64-bit operations are on is a byte in the core that the executor sets wherever it re-derives the privilege state, so a compiled LWR spends a byte load and a compare on it rather than decoding Status. The code-cache fingerprint gains lwr_upper=1 only when it is set, so the R4400 and R5000 machines keep their warm caches. The uncached-store and LWR changes affect the Octane, IP28 and an O2 with the R10000; the alias change affects the O2. The R4400 and R5000 are unchanged. Tests: SW, SWL and SC on the R10000 against a bus nothing answers, with the R5000 and a load still faulting; a page at 0x1_8000_0000 with the O2's mask and window expects RAM at 0; and LWR compiled and interpreted, at a partial and a full offset, with the mode's 64-bit bit set and clear, on the R10000 and the R5000, plus the byte the compiled LWR reads across Status writes and ERET.
Add the R12000 and R14000, and replace the per-machine CPU constants with one table of the parts SGI shipped. The Octane shipped with an R12000 or R14000 and the O2 with an R12000, but `cpu` took only r4400, r5000 and r10000. An Octane hinv could not show the later parts, and CP0 Config reported 1 MB of secondary cache for every R10000-family part, so a 2 MB model printed "1 Mbyte". The new parts share the R10000's programming model (44-bit virtual, 40-bit physical, the R10000 cache operations) through model::r10k_family, and Config SS is now log2(secondary size / 512 KB). On the R12000 and later, Index Writeback Invalidate (D) also clears the line's primary data tag, as Index Load Tag then shows; the R10000 keeps it. They keep the upper word on a partial LWR as the R10000 does; for the R12000 and later that is a model choice, as is the R14000's TLB conflict rule. Each machine kept its own facts about the CPU: the Octane had three Count constants and a PRId, the O2 two Count rates, the IP28 another, and machine.rs, ip30_machine.rs and ip32_machine.rs each matched on the CPU type to pick a cache model. Now src/cpu/parts.rs holds one const per part: family, clock, revision and secondary cache size, as IRIX hinv reports them. The PRId is computed from the family and revision, Count is half the clock, and the name (r10000_195) is derived. Where two shipped parts share a family and clock, the earlier silicon takes an _EARLY suffix: the Indy and Indigo2 R5000 is rev 1.0 where the O2's is 2.1, and the IP28's R10000 is rev 2.5 where the Octane's is 2.7. Each machine lists the parts it shipped with in its own CPUS, first is the default: Indy, indigo2_ip22 r4400_150, r5000_180 (rev 1.0) indigo2_ip28 r10000_195 (rev 2.5) O2 r5000_180, r10000_250, r12000_400 Octane r10000_195, r12000_360, r14000_600 `[machine] cpu` stays the string the user wrote. It is resolved once, by MachineConfig::cpu_part, after the profile is final: no cpu is the machine's first part, a family name (`r14000`) or a part name (`r14000_600`) is the machine's part that matches, and anything else is refused by validate() with a message naming the parts it takes. The `[clock] fixed_mhz` override applies in ClockConfig::count_hz, and with_cpu! is the one place a part becomes a cache and executor type. The O2's R10000 and R12000 have 64-byte secondary lines (with 128 its PROM panics before its banner); the Octane R12000's 128 is a model choice. The R10000 Indigo2 is the IP28 board: Linux builds the IP22 sources for the R4X00 and R5000 and the IP28 sources only for the R10000, so an Indy or indigo2_ip22 no longer takes an R10000 and indigo2_ip28 takes nothing else. The GUI's Processor pickers read the same lists. The bare-metal benchmark suite runs on an Indy and measures every part there; it does not run on the IP28's memory map (RAM starts at 0x20000000, out of reach of the KSEG0 image). A config with the test device on therefore takes any part, so iris-bench keeps its r10000, r12000 and r14000 cells and a product config keeps the rule. Guest-visible changes: the Indy and Indigo2 R4400 is rev 5.0 at 150 MHz (Count 75 MHz; was rev 4.0 and 33) and their R5000 rev 1.0 at 180 MHz (was rev 2.1); the O2 R10000 is rev 3.4 at 250 MHz (was rev 2.5 at 180). The Octane and IP28 report what they did. validate() also refuses an O2 R4400, which only the constructor did. Tests: the PRId encoding, a family name picking a machine's first part of it, Config reporting each secondary cache, and Index Writeback Invalidate (D) on the R12000 with the R10000 as the control.
A boot PROM can stage a secondary-cache line with Index Store Tag (S) and Index Store Data (S) and then load, store or fetch from it before memory works. The R10000-family shadow cache kept those arrays for diagnostics only, so such an access read the bus. Per the NEC VR10000 manual, a valid tag whose PA[39:18] matches the access is a secondary hit that supplies the staged data, with no bus read. The shadow arrays already hold the staged tags and data, so the hit is a lookup in them: both ways at the access's index, valid state (TagLo[11:10] != 0), matching tag. A store to a staged line stays in the line and makes it DirtyExclusive. Index Writeback Invalidate, Hit Invalidate and Hit Writeback Invalidate (S) end the line, writing a DirtyExclusive one back; so does Index Store Tag. Snapshots carry the staged lines. No access tests for a staged line. While the cache holds any, it takes RAM off its window paths instead: read, write and the compare-and-swap paths see no mapped region, the decode cache gets no window, and the bus path they all fall to goes through a StagedBus that answers for the staged lines. The interpreter's and the JIT's inline loads and stores reach RAM through the micro-TLB and the window without calling the cache, and the JIT compiles from memory, so the executor also runs no compiled code until the last line ends. A guest that never stages a line runs the code it ran before; one that does runs in the interpreter on the bus path while it holds lines, and the shipped PROMs end every line they stage before the kernel runs. The R10000 translates every CACHE operand but Cache Barrier's, a mapped one taking TLB Refill or Invalid, and the secondary cache takes its set and way from the operand's physical address, not its virtual one. The word order of a refill is not settled by the manuals. The TagLo word lands at the lower address, which is inferred from software that stages lines this way and is marked as such. Replacement, coherence with other CPUs and DMA, and ECC are not modelled; the note in clanker/cpu lists the limits. The shadow model applies to the whole R10000 family on the R10000's tag layout, which is an inference for the R12000 and R14000.
A PROM can also stage a line in the primary instruction cache with Index Store Tag (PI) and Index Store Data (PI) and run from it. A fetch whose primary tag matches now executes the staged word, ahead of a staged secondary line, and the lines end on Index Invalidate, Hit Invalidate, a tag stored invalid, or power-on. A primary word is 36 bits of predecoded instruction plus parity. Fetch decodes the predecode classes the manual describes (eleven, each row checked against the encoding table) and checks even parity over the 36 bits and ECC[0]. Status.DE and ERL suppress the Cache Error. A word of a class the model does not know is refused: it is reported once per class and the fetch reads the staged secondary line, else memory, instead. That is an approximation, since hardware would run the word. A fetch that misses every primary way and hits a staged secondary line refills a 64-byte primary block from it, into the invalid way first. The refilled words keep a flag beside the parity so that Index Load Data followed by Index Store Data copies them verbatim; a guest-built word is never taken for one. The set of words handed out is bounded and not saved in snapshots. Invalidating a secondary line purges the primary instruction and data lines of its block that the tag's virtual index (TagLo[8:7]) names, with the D-side writeback the operation implies. A valid primary line holds the cache off its window paths as a secondary one does. The primary index is virtual. While lines are held the decode cache reads every word through a StagedBus of its own, which takes the virtual address from the PC: every fetch is at the PC, and a fused pair never leaves its I-cache line. The parity check reads DE and ERL from Status when it runs. The executor hands the cache both registers when it starts holding lines, so neither costs anything as the CPU runs.
jamesbraid
force-pushed
the
r10k-cache-line-staging
branch
from
November 8, 2026 00:17
e2f8a10 to
4eba0c8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Depends on #403; the diff shows its commits until that lands. A boot PROM can stage a secondary-cache line with Index Store Tag (S) and Index Store Data (S), or a primary instruction-cache line with the (PI) forms, and run from it before memory is usable. iris kept those tag and data arrays for diagnostics only, so a cached access to a staged line read the bus. R10000-family CPUs now treat a staged line as a hit, per the NEC VR10000 manual. The R10000's tag layout is used for the R12000 and R14000 too, which is an inference, not a measurement.
Changes
StagedBusthat answers for the staged lines; the executor runs no compiled code until the last line ends. A guest that never stages a line runs the same code as before. One that does runs in the interpreter on the bus path while it holds lines; the shipped PROMs end every line they stage before the kernel runs.clanker/cpu/staged-secondary-cache-lines.mdlists the limits.Testing
cargo test --features jitv2 --libpasses in release and debug at the tip (3044 tests),cargo check --lib --testsat each commit, and the staged-line tests underlightning,jitv2;cargo check -p iris-gui --all-targetsis clean.hinvshows the R10000, andhalt -ppowers off.Performance
Deterministic mode, whole suite, n=3 each, against #403, same guest instructions and checksums in all 46 kernels. R4400: 309.859G host instructions both sides (-0.000%), every kernel within one sample of #403. R10000: +0.019% (310.583G to 310.642G); every kernel within its sampling resolution except sys/cache_flush, +0.25%, which is about six host instructions per CACHE instruction: the executor's check for a change in staged lines after each one, and picking the physical address as the secondary index.