Repository navigation
fwTPM MSSIM: ~46 ms command latency resolved by server-side TCP_NODELAY #663
Description
Hello everyone,
Following my conversation with Aidan (@aidangarske) in #589, where he invited me to retry after the PQ attestation fixes, I tested the updated implementation. The flow now works, but I noticed substantially increased client latency compared with my older patched implementation.
I isolated the latency in a minimal reproducer.
Reproduce
git clone https://github.com/jerohavaz/wolftpm-nodelay-repro.git
cd wolftpm-nodelay-repro
docker build -t wolftpm-nodelay-repro .
docker run --rm --network none wolftpm-nodelay-reproPinned versions:
- wolfTPM:
d75a462faaccb5dfe3df1a4064ce80c178c6578e(master at testing time) - wolfSSL:
3c5eead44904df64e6a5a1f4ebdce377d35a849a(v5.9.4-stable)
Output on my WSL2/Linux Docker setup:
pristine: 45.994 ms median (10 successful requests)
nodelay : 0.093 ms median (10 successful requests)
Median ratio: 496.3x
The reproducer compares pristine wolfTPM with an otherwise identical server containing only an accepted-command-socket TCP_NODELAY patch. A Python client sends ten successful GetRandom(32) commands over a persistent loopback connection after Startup and one warm-up. Client-side TCP_NODELAY is enabled in both cases.
There is no attestation application, relay, or custom wolfTPM instrumentation. Timing covers the complete MSSIM exchange, including its acknowledgement.
Original benchmark comparison
These plots show how I noticed the delay. They are background context and include a relay setup; the standalone reproducer above excludes that relay and isolates the server socket change.
The diagrams use German labels and show mean times over 200 runs:
- “Mittlere Laufzeit [µs]”: mean runtime in microseconds.
- “fwTPM intern”: inner measurement around
FWTPM_ProcessCommand(), including command processing and cryptographic work. - “außerhalb fwTPM”: outer client transport round-trip time minus the inner measurement. This includes transport/framing, buffering, scheduling, and server work outside that function.
- RSA / PQ: classical and post-quantum profiles.
The full stacked bar represents the outer measurement. Internal execution remained comparatively similar while the outside portion increased sharply for both profiles. This suggested transport/protocol overhead, which motivated the isolated reproducer.
Older benchmark with locally patched wolfTPM:
Updated benchmark with wolfTPM v4.2.0:
Observed behavior
The MSSIM server sends the response length, payload, and acknowledgement separately. Enabling TCP_NODELAY on its accepted command socket removes the approximately 46 ms delay in this test.
This appears consistent with Nagle buffering interacting with delayed ACKs, although I have not verified that mechanism with a packet capture.
Would enabling TCP_NODELAY on accepted fwTPM command sockets be an appropriate upstream fix?
Activity
dgarske commentedon Nov 2, 2026
Thank you @jerohavaz for the reproducer and for the careful write-up! The isolation work made this straightforward to confirm, and your diagnosis was right on both counts.
Reproduced here on Linux against
upstream/master(96e7740f, which is your pinnedd75a462fplus one merged PR), using an independent probe rather than your Docker image: a persistent loopback connection with client-sideTCP_NODELAY,TPM2_Startup, one warm-up, then timedGetRandom(32)round trips covering size, response and acknowledgement.TCP_NODELAYon accepted socketswritev()The pristine spread was 41.0-42.0 ms with nothing in between, which is Linux's 40 ms delayed-ACK floor rather than anything the TPM is doing - consistent with your observation that the inner
FWTPM_ProcessCommand()measurement stayed flat while the outer round trip grew. Your ~46 ms on WSL2 is the same effect with a different timer.The mechanism is as you suspected.
DispatchAndRespond()writes the reply three times - size, response, acknowledgement - and that last 4-byte write goes out while the response is still unacknowledged, so Nagle holds it until the peer's delayed ACK arrives. The telling detail is that the SWTPM client has always setTCP_NODELAYon its own socket (src/tpm2_swtpm.c), and the server never set it on the sockets it accepts, so the option was only ever applied to the side that was not doing the three small writes.So yes - enabling
TCP_NODELAYon accepted fwTPM command sockets is an appropriate upstream fix, and it is going in. Two notes on what we did beyond your patch:writev(), which removes the small trailing write altogether. Either change fixes the delay on its own; keeping both means the fallback path for a custom I/O HAL, Windows and Zephyr - which still writes three times - is covered by the socket option.Fixed in #664