hil: raspberry_pi_pico_w host/cdc_msc_hid intermittently panics with Data Seq Error (hcd_rp2040.c:379)

raspberry_pi_pico_w host/cdc_msc_hid fails intermittently on the ci.lan HIL rig: the host firmware hits panic("Data Seq Error") in hcd_rp2040_irq() (src/portable/raspberrypi/rp2040/hcd_rp2040.c:379, USB_INTS_ERROR_DATA_SEQ_BITS). The cell reads either "CDC echo mismatch at offset 499 (48 bytes)" or, when the panic comes before the CDC device enumerates, "CDC device not mounted on host".

Mechanism

Two target snapshots taken at the panic (PR #3968 CI artifact and a local build of the same source) show the same state, byte-identical DPRAM, at the same frame (~757):

Fixture: pico_w host port → hub (addr 5) → flash disk 2008:2018 (addr 1) and CH9102 1a86:55d4 (addr 2, hub port 4).
EPX is serving dev 1 EP 0x81 bulk IN, the MSC INQUIRY data stage (CBW 0x12, 36 bytes). epx_buf_ctrl = 0x7424: AVAIL, DATA1, LEN 36, FULL 0, with remaining_len restored and xferred_len 0.
The complete INQUIRY response is already in the EPX data buffer (DPRAM 0x50100180), but was never copied to the class buffer.
The last SETUP in DPRAM is the hub's GET_STATUS(port 4) (a3 00 00 00 04 00 04 00), for the CDC port's enumeration.

Reading: the SOF round-robin (hcd_rp2040.c:356-369: sie_stop_xfer() + epx_save_context() + epx_switch_ep()) switched EPX to the hub SETUP after the device had delivered the INQUIRY packet and seen it ACKed. epx_save_context() (:140-169) rolled it back as not started and un-toggled the PID. After the switch back the host expects DATA1, the device, having toggled on the ACK, sends DATA0, and the controller raises DATA_SEQ_ERROR.

RP2040 datasheet 4.1.2.9.2 (host IN): "Raise DATA SEQ ERROR if the data packet has wrong DATA PID".
USB 2.0 §8.6.2/§8.6.4: the transmitter toggles only on ACK and retries with the same PID, so a different PID on the retry means the earlier packet was ACKed and the rollback was wrong.

Two races fit and the snapshots cannot tell them apart:

(a) the packet completes between the ISR reading INTS and STOP_TRANS / epx_save_context();
(b) RP2040-E4: the completion status lands in the BUF1 half, which epx_save_context() never reads in single-buffered mode.

Rate

Bench on ci.lan, 11 boots per block (post-flash boot + 10 resets), alternating firmware:

firmware	panics
PR #3968 CI artifact (run 36377716382)	6/22
local build of the same PR source, other toolchain and RAM layout	3/11
master `de49405` (run 36335423683)	0/22

PR vs master: p=0.011 (both PR builds 9/33, p=0.006; one-sided Fisher). Master is bounded below 12.7% at 95%, not shown to be zero. hcd_rp2040.c is identical between #3968 and master, so the bug is pre-existing; #3968 only makes the overlap frequent. Why it does is not explained: by code reading its msc_host changes put the same mount-time transfers on the wire.

CI evidence

PR hil: register usbtest's id once before the batteries, never clean it up #3955 run 35835337365, job: the only failure in that job not shared with master; the PR touches nothing under src/, hw/ or examples/.
master run 35415777237 (2026-09-19) fails the same way; 4 of the last 35 HIL runs hit it at the time.
Fix transfer lengths that wrap when narrowed to 16 bits #3968 heads failed the cell 2 of 2 times: df5d5e001, c21f94fd5. CI does not capture the console, so that those failures are this panic is inferred from the bench reproduction.

Ruled out

Full-rig load: the PR artifact passed under a full-rig run and panicked on an idle rig.
Code layout or toolchain of the CI artifact: the local build panics too, with the identical signature.
MSC enumeration stalling as the trigger.

Candidate fix, not run

In the SOF round-robin branch (:356-369), after sie_stop_xfer() re-read usb_hw->buf_status bit 0 (EPX); if set, complete the transfer (rp2usb_xfer_continue / xfer_complete_isr) instead of calling epx_save_context().
In epx_save_context(), apply the RP2040-E4 workaround in single-buffered host mode: take the completed half from BUFF_CPU_SHOULD_HANDLE, or check both halves, before rolling back PID and length.

Validation: PR #3968 head with and without the patch, 20 reset boots each on the same fixture; expected 0/20 vs about 5/20. A RAM ring in epx_save_context() logging both buf_ctrl halves, buf_status and BUFF_CPU_SHOULD_HANDLE would separate (a) from (b).

Also seen, not investigated: a no-panic outcome where CDC mounts but MSC "Disk Size" never prints (PR 2/22, master 1/22).

CI: Fix transfer lengths that wrap when narrowed to 16 bits #3968 heads failed this cell 2 of 2 times: df5d5e001 and c21f94fd5. The same cell showed no mount failure in 27 runs outside that PR. hcd_rp2040.c is byte-identical between the PR and master, so the PR most likely changes timing rather than causing the bug. The harness still ran the PR binary and the base binary in the same test order, with seed 1790360486: the PR binary failed and the base binary passed 4 of 4.
Bench, pico_w on ci.lan, PR binary: 1 panic in 5 runs, and every failure came on the first boot after flashing. Reset-only boots passed. Base binary: 2 of 2 passed.

Where it happens: the MSC device mounts, and then during the INQUIRY read the hub's GET_PORT_STATUS is using the same EPX endpoint. The panic comes from hcd_rp2040_irq() at hcd_rp2040.c:379:

A MassStorage device is mounted
A device with address 1 is mounted
*** PANIC ***
Data Seq Error
#2 panic (fmt="Data Seq Error \n") at panic.c:49
#3 hcd_rp2040_irq () at hcd_rp2040.c:379

Ruled out: MSC enumeration stalling as the trigger.
Two directions, neither verified:
- treat an unexpected data PID on IN as the retransmission allowed by USB 2.0 §8.6.4 instead of panicking;
- fix the toggle undo in epx_save_context when EPX is shared (possibly erratum RP2040-E4).

New evidence from PR #4056 (hub/HIL work on the same cell):

Pre-existing, not harness-induced: the panic occurred in an uninterrupted boot with USBCTRL idle and no flash-time hold. CI run 36630411630 (2026-09-29, before rp2040, HIL: detach on warm restart and reset hub TT after cancelled splits #4056) shows the same cell failing "CDC device not mounted on host".
Second failure shape: the MSC mounts and the first SCSI INQUIRY never completes ("MSC Disk Size not reported"). In passing boots INQUIRY/Disk Size print within 50 ms of the MSC mount.
Topology on ci.lan: raspberry_pi_pico_w native host, a hub with CH9102 1a86:55d4 and flash disk 2008:2018 behind it.
rp2040, HIL: detach on warm restart and reset hub TT after cancelled splits #4056 adds an openocd USBCTRL hold during programming (test/hil/hil_flash.py:150), which makes it near-certain: 5/5 failures with the hold vs 0/3 passes without, byte-identical firmware (cdc_msc_hid.bin sha256 93ac3ed2865d3db3e872a25cd7e6c1fbe0226505950dad4695a60db7c80f5cf2). Whether to keep the hold is being decided on rp2040, HIL: detach on warm restart and reset hub TT after cancelled splits #4056; this issue is the firmware side.
Not established: which device/endpoint raised DATA_SEQ. Inference from timestamps only: EPX preemption (hcd_rp2040.c:140-191, :355-371) racing the MSC INQUIRY bulk IN against CH9102 enumeration. Whether RP2040-E4 (datasheet p394/p632) bears on epx_save_context is unverified.

Next step: autopsy at a failing boot (ADDR_ENDP 0x50110000, SIE_STATUS 0x50110050, ep_pool, EPX buffer control in DPRAM) to identify the endpoint, then decide the DATA_SEQ handling (retry/stall the transfer instead of panic, cf. #2776).

Decision on #4056: the openocd USBCTRL hold was dropped (8935447).

raspberry_pi_pico_w host/cdc_msc_hid, same binary:

with the hold: 8 of 9 runs fail with the Data Seq Error panic or an INQUIRY stall;
without it: 1 of 5 fail (the latest one panicked mid CDC echo, 2026-10-02 ci.lan).

The hold remains the highest-rate reproducer: flash with -c "init; reset halt; mww 0x4000e000 0x01000000; program <fw> verify reset exit" (RP2350: mww 0x40022000 0x10000000). Why it raises the rate is not established.

The panic rate depends on how late after SOF the RP2040 EPX round-robin runs (hcd_rp2040.c:401-418).

Found while reworking the interrupt-endpoint abort for #4177: a variant whose SOF handler scanned the 16-entry ep_pool before the round-robin, delaying its sie_stop_xfer() by a few hundred cycles, raised the panic sharply. Moving the round-robin back to the first thing after SOF restored the base rate. No interrupt endpoint is aborted in this test, so the scan's added latency was the only change on its path.

Firmware (pico_w, ci.lan, `host/cdc_msc_hid`)	Panics
base (`922fde7ec`)	0/11
pool scan before the round-robin	4/7
round-robin first	0/8
round-robin first, inlined (final)	0/8

Hypothesis, not confirmed on the wire: after the second idle SOF the round-robin issues STOP_TRANS while the controller may already have started the NAK-retrying endpoint's transaction in the new frame. The later the stop, the likelier it lands mid-transaction, corrupting that endpoint's toggle, and its next packet raises DATA_SEQ_ERROR. The openocd USBCTRL-hold result (8/9) fits the same picture.

Next steps:

Capture the panic with a sniffer on the pico_w host port to confirm a stop mid-transaction and see which endpoint's PID goes wrong.
If confirmed, the switch should not stop a transaction in flight, e.g. wait for the current one to end (NAK or TRANS_COMPLETE) instead of stopping at SOF.

rp2040/hcd: abort host interrupt endpoint transfers (#4177)

rp2 hcd (RP2040): stop EPX for a yield only once the bus is idle

rp2 hcd (RP2040): let a pending control stage in at the end of a poll…

rp2 hcd (RP2040): stop a round-robin switch right after a NAK

Navigation Menu

Repository navigation

hil: raspberry_pi_pico_w host/cdc_msc_hid intermittently panics with Data Seq Error (hcd_rp2040.c:379) #3964

Mechanism

Rate

CI evidence

Ruled out

Candidate fix, not run

Related

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

hil: raspberry_pi_pico_w host/cdc_msc_hid intermittently panics with Data Seq Error (hcd_rp2040.c:379) #3964

Description

Mechanism

Rate

CI evidence

Ruled out

Candidate fix, not run

Related

Activity

hathach commented on Sep 26, 2026

hathach commented on Oct 1, 2026

hathach commented on Oct 1, 2026

hathach commented on Oct 9, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions