fmc: streamline the FMC target and run it at 100 MHz on cosmo - #527
Open
nathanaelhuffman wants to merge 1 commit into
Open
nathanaelhuffman wants to merge 1 commit into
nathanaelhuffman wants to merge 1 commit into
Conversation
Analysis showed the SP has been driving FMC_CLK at 50 MHz (CLKDIV=3) against FPGA constraints signed off at 66.67 MHz, and the target FSM was spending 7-8 fmc_clk cycles per 32-bit write plus a dead cycle before every read beat. This change reworks the interface for line rate while keeping the architecture (NWAIT-paced two-beat 32-bit accesses, async FIFO CDC into the 125 MHz AXI fabric) intact. RTL (stm32h7_fmc_target): - All FMC inputs are captured in dedicated per-pin registers before the FSM sees them; the FSM runs one cycle behind the bus, which the NWAIT pacing absorbs. On cosmo the capture registers are clocked by a later-phased (+135deg) sibling of the deskewed FSM clock, which is what gives the input pins setup margin at 10 ns; grapefruit ties both clock ports to the same clock for identical cycle behavior. - Streamlined FSM: writes take 6 fmc_clk cycles of bus occupancy (was 7-8) and read beats stream back-to-back (extra_beat_setup restores the old cadence if a board can't close single-cycle data-out). - Hardening: fmc_state now resets; an NWAIT timeout (generic, default 4096 cycles) converts a wedged AXI responder into poisoned reads / swallowed writes plus a sticky timeout_count instead of a hung SP bus, with stale read data discarded before any live read; the tristate control is re-derived from live NOE/NE every cycle and a sticky contention counter trips if we ever drive during an SP-owned phase; aborted writes complete with zero filler so the AXI side can't wedge; AWVALID and WVALID are now raised together. - The tristate control is registered in OBUFT T polarity (data_out_hiz, one IOB-attributed flop per pin): an active-high enable inferred a LUT1 inverter that blocked IOB packing and cost ~1.5 ns. cosmo clocking: new fmc_pll (clk_wiz, BUFG-in-feedback deskew, +45deg FSM/output phase, +135deg capture phase, OVERRIDE_MMCM to force M=12/D=1/O=12 so the 600-1200 MHz VCO locks at either a 50 or 100 MHz input -- one bitstream serves both hubris CLKDIV settings), an fmc_clk_monitor that holds the MMCM in reset across SP clock stops and retries lock on frequency change, and reset_fmc now gates on MMCM lock with async assert (no FMC clock edges exist while unlocked). Constraints: cosmo at 10 ns with a setup-2 multicycle establishing the edge relationship into the phase-shifted clocks, corrected min input delays (the old formulas omitted the SP's half-period of hold, telling Vivado data could change at the capture edge), multicycle data-out exceptions deleted, and IOB packing on the output/enable/nwait flops; grapefruit stays at 15 ns with the same structure. Both boards close (cosmo WNS +0.116/WHS +0.023, grapefruit +0.159/+0.031). Verification: the model now implements the WAITCFG=1/DATLAT=0 pacing contract, blocking writes via wait_until_idle (the 150 ns hack is gone), ES0491 dummy read cycles, configurable inter-transaction gaps, and one-shot mid-transaction aborts; the harness runs real fmc_clk/aclk ratios with a bus-contention tripwire, and the suite grew from 3 tests to 18 covering back-to-back traffic, FIFO backpressure, slow and wedged responders, timeout recovery, aborts, reset, and randomized soaks at 50/66.67/100 MHz. tools/fmc_sweep/fmc_sweep.py drives the hardware bring-up: a phased frequency sweep over the fmc-demo-server UDP protocol (reads of a constant register first at each step, so a timing miss is data-only, never contention), with CLKDIV control through humility. The hubris side (CLKDIV flips, fmc-demo-server timing-setter fixes) lands separately; the FPGA is NWAIT-paced, so this bitstream works unchanged at 50 MHz until hubris moves.
Aaron-Hartwig
approved these changes
Sep 11, 2026
| @@ -0,0 +1,203 @@ | |||
| :showtitle: | |||
Collaborator
There was a problem hiding this comment.
This probably doesn't have to be checked in, but I don't think it harms anything if it is.
nathanaelhuffman
force-pushed
the
ndh/fmc-speedup
branch
from
September 15, 2026 20:09
aa0e06e to
9964b74
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This contains a bunch of work to speed up the FMC interface to our theoretical limits here. Hubris currently runs the FMC inteface at 50MHz and the measured data rates are around 8.3MBps at this speed. We've re-tooled both the FMC state machine, and the timing constraints to allow this to get up to 100MHz (on cosmo/metro) and 66MHz (on grapefruit due to SP <-> FPGA pinout errata which limits our max freq due to not having fmc_clk on a clock-capable pin).
We hit 100MHz by phase shifting an internal representation of the fmc_clk forwards and backwards, depending on input/output direction to cancel out these delays as well as using IOB packing. Discussion in detail of this is in the documentation.
This change is backwards compatible with hubris such that it works with the existing clock (50MHz) as well as 66 or 100MHz without needing RTL changes. This is done by choosing PLL parameters for the shifted clocks that are functional between 50 and 100MHz and doing the timing constraints at 100MHz.
Benchmarking:
50MHz FMC bus: ~8.3MBps
66MHz FMC bus: ~9.9MBps,
100MHz FMC bus: ~11.4 MBps.
For validation:
I've run multiple hours of read/write hammering using the fmc_demo UDP blocks, including new features to be upstreamed in hubris that run the loops as fast as possible without network being the load-bearing factor.
The phase of the re-clocked FMC pins was phase shifted to prove that we're in the middle of the sample window and run both the + and - shifted images for 1 hour each with constant hammering of the FMC block which validates that we have margin on our sample points even though timing appears tight (but passing!). It is expected that this timing will remain tight, but not have much build-build variance due to it being limited more by physics than synth and placement.