Ndh/metro pre work - #528
Draft
nathanaelhuffman wants to merge 12 commits into
Draft
nathanaelhuffman wants to merge 12 commits into
nathanaelhuffman wants to merge 12 commits into
Conversation
Analysis showed the SP has been driving FMC_CLK at 50 MHz (CLKDIV=3) against FPGA constraints signed off at 66.67 MHz, and the target FSM was spending 7-8 fmc_clk cycles per 32-bit write plus a dead cycle before every read beat. This change reworks the interface for line rate while keeping the architecture (NWAIT-paced two-beat 32-bit accesses, async FIFO CDC into the 125 MHz AXI fabric) intact. RTL (stm32h7_fmc_target): - All FMC inputs are captured in dedicated per-pin registers before the FSM sees them; the FSM runs one cycle behind the bus, which the NWAIT pacing absorbs. On cosmo the capture registers are clocked by a later-phased (+135deg) sibling of the deskewed FSM clock, which is what gives the input pins setup margin at 10 ns; grapefruit ties both clock ports to the same clock for identical cycle behavior. - Streamlined FSM: writes take 6 fmc_clk cycles of bus occupancy (was 7-8) and read beats stream back-to-back (extra_beat_setup restores the old cadence if a board can't close single-cycle data-out). - Hardening: fmc_state now resets; an NWAIT timeout (generic, default 4096 cycles) converts a wedged AXI responder into poisoned reads / swallowed writes plus a sticky timeout_count instead of a hung SP bus, with stale read data discarded before any live read; the tristate control is re-derived from live NOE/NE every cycle and a sticky contention counter trips if we ever drive during an SP-owned phase; aborted writes complete with zero filler so the AXI side can't wedge; AWVALID and WVALID are now raised together. - The tristate control is registered in OBUFT T polarity (data_out_hiz, one IOB-attributed flop per pin): an active-high enable inferred a LUT1 inverter that blocked IOB packing and cost ~1.5 ns. cosmo clocking: new fmc_pll (clk_wiz, BUFG-in-feedback deskew, +45deg FSM/output phase, +135deg capture phase, OVERRIDE_MMCM to force M=12/D=1/O=12 so the 600-1200 MHz VCO locks at either a 50 or 100 MHz input -- one bitstream serves both hubris CLKDIV settings), an fmc_clk_monitor that holds the MMCM in reset across SP clock stops and retries lock on frequency change, and reset_fmc now gates on MMCM lock with async assert (no FMC clock edges exist while unlocked). Constraints: cosmo at 10 ns with a setup-2 multicycle establishing the edge relationship into the phase-shifted clocks, corrected min input delays (the old formulas omitted the SP's half-period of hold, telling Vivado data could change at the capture edge), multicycle data-out exceptions deleted, and IOB packing on the output/enable/nwait flops; grapefruit stays at 15 ns with the same structure. Both boards close (cosmo WNS +0.116/WHS +0.023, grapefruit +0.159/+0.031). Verification: the model now implements the WAITCFG=1/DATLAT=0 pacing contract, blocking writes via wait_until_idle (the 150 ns hack is gone), ES0491 dummy read cycles, configurable inter-transaction gaps, and one-shot mid-transaction aborts; the harness runs real fmc_clk/aclk ratios with a bus-contention tripwire, and the suite grew from 3 tests to 18 covering back-to-back traffic, FIFO backpressure, slow and wedged responders, timeout recovery, aborts, reset, and randomized soaks at 50/66.67/100 MHz. tools/fmc_sweep/fmc_sweep.py drives the hardware bring-up: a phased frequency sweep over the fmc-demo-server UDP protocol (reads of a constant register first at each step, so a timing miss is data-only, never contention), with CLKDIV control through humility. The hubris side (CLKDIV flips, fmc-demo-server timing-setter fixes) lands separately; the FPGA is NWAIT-paced, so this bitstream works unchanged at 50 MHz until hubris moves.
The A1/A0 state machine and the rail records it works on are not cosmo-specific -- any SP5 board sequences the same group A/B/C rails through the same machine. They were nonetheless welded to cosmo's NIC: sequencer_io_pkg held both halves, and the sequencer target globbed every file in the directory, so a sibling board could not depend on the shared part without also pulling in cosmo's T6 and colliding with its own. Carve sp5_power_pkg out of sequencer_io_pkg -- the generic rail records, the SP5 rail groups, the SP5 pin record -- and give a1_a0_seq and it a sequencer_core target. sequencer_io_pkg keeps the cosmo-specific half and depends on the new package. No files move and no logic changes. Do the same for the sim side: rail_model, cascade_rail_model and sp5_model only ever spoke sp5_power_pkg's records, so drop their vestigial sequencer_io_pkg use clauses and expose them as sequencer_core_sims. sp5_seq_tb passes unchanged, 21 of 21.
nathanaelhuffman
force-pushed
the
ndh/metro-pre-work
branch
from
September 16, 2026 21:05
f4b40e4 to
b0db020
Compare
…C slot on bank 4 The T6 slot's power enable was gated on bank 0's direction bit -- M.2 A's -- rather than its own bank 2's, so whether the SP5 could power the NIC depended on how it had configured a different slot. Fix that before cloning the block. Metro's Versal presents two PCIe endpoints to the SP5, and the second one needs a hotplug slot of its own. PCA9506 bank 4 has always been wired into the expander but referenced by nothing; its registers reset inert, so giving it a slot costs no register-map change. The new slot is T6-shaped -- PERST a copy of PWR_EN, no oneshot -- behind a generic that cosmo leaves off, with the new ports defaulted so cosmo's port map is untouched. Both NIC slots also gain a presence input, defaulted to present, so metro can report the Versal's real PRSNT_L pins where cosmo tied the T6's low.
The eSPI flash channel only ever forwarded reads; a PUT_FLASH_NP write was parsed and dropped and an erase was not parsed at all, so a host that tried either waited forever for a completion. Metro's second eSPI target fronts the Versal's boot flash and the SP5 has to be able to program it, so writes and erases now go through, behind a permission that is off by default. Permission is the AND of a FLASH_WRITES_ALLOWED generic on espi_target_top (and the sp5_espi_flash_subsystem wrapper) and a new flash_write_enable bit in the control register, so an instance built without the generic can never be opened up by software, and one built with it still has to be enabled at runtime. A write or erase that arrives while not permitted takes a descriptor marked refused and is answered with an unsuccessful completion rather than silence. In the flash channel the descriptor gains a kind. A write's payload is captured into the descriptor's DPR slot as the command is parsed (ahead of the CRC; a bad CRC just leaves the slot to be overwritten) and streamed out to the flash block ahead of the two command words, whose length word now carries the kind in its top nibble. Writes and erases come back as a single status byte and complete with success_no_data or the unsuccessful cycle type. The completion header is latched when the GET arrives so a zero-length completion can retire its descriptor straight away, and the response processor skips the payload stage for length zero. The link layer sizer learns that a flash write carries its payload in the command. spi_nor_top's eSPI client sequences the flash side: write enable, quad page program chunked at page boundaries (or 4k/64k erase; the 32k code has no 4-byte-address opcode here and is refused), then a status poll until write-in-progress clears, with a bound on polls so a missing part does not hang the channel. The engine's outbound byte stream is muxed between hubris' TX FIFO and the eSPI payload FIFO for the duration of a program. Read chunk issue now holds go until the engine has started, the same way the hash client does; the old exit-on-idle dropped every chunk after the first, which nothing hit only because the channel's maximum payload is smaller than a chunk. The eSPI sim's fake flash is now memory backed, so reads are checked against known contents (closing the old TODO), and writes and erases can be verified by reading back. A new spi_nor harness drives the eSPI client FIFOs directly against the flash VC.
The 4k entry post code buffer was distributed RAM: 3.6k LUTRAMs per eSPI target, the largest single consumer in the block. On a design with the DIMM SPD caches and two targets that took the XC7S100 to 95% of its LUTRAM and the placer scattered the buffer across the die, at 2.5ns of negative slack. In block RAM it costs four tiles instead. A block RAM read is registered, so the buffer now answers one cycle after the AXI read is accepted, which is when rvalid rises anyway; the read enable is the accept, so the output holds while the master takes its time. Reads of the buffer are muxed from that register rather than the general rdata register. A new POST_CODE_BUFFER_ENABLED generic on espi_target_top leaves the buffer out altogether, for a target that only ever serves the flash channel and never sees a post code.
A sequencer can raise hw_req to have a flash range hashed without software in the loop, for measuring a device's boot image before releasing it from reset. The range comes from HW_FLASH_ADDR/HW_LENGTH, the flash from the HW_FLASH_SEL generic, and the result is snapshotted into HW_DIGEST0..7 so a later software run cannot overwrite it; hw_ack and hw_err answer the request, with the detail in HW_STATUS. A request that lands during a software run is refused rather than restarting it, and a software start during a hardware run is dropped. An unprogrammed range is a zero length and comes back as a cfg_err. Also fixes an abort of a software-fed hash run, or of a flash run abandoned during its prepend, which waited in DRAIN for bytes no read had been issued for.
…muxing pins versal_flash_subsystem parked the Versal flash bus by muxing cs_n, sclk and the lane enables after spi_nor_top's output flops. Those flops are dedicated IOB copies, and the timing window on the flash pins is derived from them being in the IOBs; a LUT between flop and pin pulls them out, and the shared-sequencer rebuild landed that path 15ps outside its 5ns budget. spi_nor_top gets a bus_enable input instead, applied at the D input of the pin flops in spi_clk_gen, spi_link and spi_txn_mgr: cs_n high, sclk low, lanes released within a clock of it dropping, with the flops still in the IOBs. The eSPI flash wrapper passes it through, and the mux control block now drives only that and the mux pins.
Nothing in board_support is cosmo-specific: the other SP5 boards use the same 50MHz input and the same 125/200MHz tree. The clock wizard bakes its module name into the netlist and into the clock names the timing constraints use, so for another board to share board_support it has to generate the very same IP. The PLL is now sys_pll rather than cosmo_pll, its script and the fmc_pll one are exported so a dependent board's bitstream can source them, and board_support is public. The clock names in cosmo_timing.xdc follow; nothing else referenced them.
Metro's sequencer FPGA (U27) is the same XC7S100-1FGGA484I cosmo uses and
plays the same role, so this reuses cosmo_seq's blocks by target
dependency rather than forking them: board support and clocking, the
SP5 eSPI/SPI-NOR service, the DIMM SPD proxy, the hotplug expanders,
the UART routing, the I2C muxes, the debug header, and the whole
sequencer including its register map. Metro's own RTL is the top level,
its constraints, the register map composition, and the control block
for the Versal boot-flash mux.
What differs is below the SP5 layer. Metro's NIC is an AMD Versal Premium
VP1202, not a Chelsio T6. The shared sp5_sequencer is built with
NIC_KIND => NIC_VERSAL, which selects versal_seq in place of nic_seq:
- ten rails with an enable each, staged core -> aux -> I/O, instead of
one hotswap enable cascading the lot
- active-low 12V/5V hotswap power good, inverted at the pin so the
records stay active high
- MODE[3:0]/POR_B boot straps and DONE/ERROR_OUT monitoring in place of
the T6's cld_rst/perst dance
- two PCIe channels, each on its own SP5 hotplug slot (channel B on the
expander's otherwise unused fifth bank) with its own PERST
- the boot image measured by the hash engine before POR_B is released,
the digest kept in the engine's HW_DIGEST registers; a failed
measurement is recorded and the boot goes ahead
- the QSPI boot flash behind a mux: the FPGA holds it while measuring
and again once the Versal has booted, so the SP5 can reach it over a
second eSPI target on the eSPI1 pins (SAFS reads, writes and erases
enabled there) and the SP over the spi_nor registers
The two eSPI targets share one hash engine (NUM_FLASHES => 2) and the
second is built without the post code buffer, which only eSPI0 sees.
The pinout follows cosmo's closely: of the 289 pins used on both boards,
254 carry the same function and no functional group changed bank. The 35
that were repurposed are almost all the Versal landing on pins the T6 held
-- its per-rail enables and power goods take cosmo's unnamed nic_pg_*
spares and its write-protect pins, and MODE[0], MODE[1] and MODE[3] take
cld_rst_l, ext_rst_l and the now-absent MCIO alert. Pin assignments were
recovered from schematic 913-0000028 rev 1 sheets 57-59 and reviewed
against the rendered drawing; metro_seq_pins.xdc names its source.
Verified: versal_seq_sim_tb (in cosmo_seq/sequencer, on the shared
harness) passes 20 of 20, covering A1/A0 bring-up, Versal rail
sequencing, the POR_B/DONE handshake, the pre-boot measurement and its
failure and skip paths, per-channel PERST, per-rail MAPO fault
injection, the boot timeout, and the flash-mux interlock; the top-level
analysis testbench compiles the whole tree. The design routes with all
timing constraints met and produces a bitstream.
nathanaelhuffman
force-pushed
the
ndh/metro-pre-work
branch
from
September 17, 2026 20:39
b0db020 to
a212703
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
staging area for metro FPGA image and the reworking of the cosmo image to match.