NCM throughput: early OUT re-arm, net_lwip_webserver tuning, U5 slave-mode TX wedge fix

docs/reference/class/network.rst

@@ -81,6 +81,16 @@ ECM/RNDIS drops the frame and renews reception itself; NCM keeps the frame and
 offers it again on the next ``tud_network_recv_renew()``, which the application
 must call.
 
+NCM can also keep the frame in place instead (zero-copy): call
+``tud_network_recv_hold()`` inside the callback and pass its handle to
+``tud_network_recv_release()`` once the frame is no longer referenced; the
+receive buffer stays valid until then.  It returns ``NULL`` when holding the
+frame could leave no buffer to receive into, every other one being held
+already; copy the frame in that case.  At least one buffer thus keeps cycling
+while the network stack holds frames, and with ``CFG_TUD_NCM_OUT_NTB_N`` = 1 it
+always returns ``NULL``.  ``net_lwip_webserver`` uses it on high-throughput MCUs,
+handing lwIP ``PBUF_REF`` pbufs.
+
 For device-to-host frames:
 
 1. Call ``tud_network_can_xmit(size)``.
@@ -89,8 +99,9 @@ For device-to-host frames:
    Ethernet frame into ``dst`` and return its length.
 
 The drivers do not lock their state: call ``tud_network_recv_renew()``,
-``tud_network_can_xmit()`` and ``tud_network_xmit()`` from the task that runs
-``tud_task()``.
+``tud_network_recv_release()``, ``tud_network_can_xmit()`` and
+``tud_network_xmit()`` from the task that runs ``tud_task()``, not from an ISR
+or another task.
 
 Use ``tud_network_link_state()`` to notify the host when the logical or physical
 link changes.  A mounted USB device is not necessarily a link-up network
@@ -108,6 +119,9 @@ interface.
    * - ``tud_network_recv_renew()``
      - Lets the driver deliver the next frame (re-arms reception, or offers the
        next NCM datagram).
+   * - ``tud_network_recv_hold()`` / ``tud_network_recv_release()``
+     - NCM only: keep the frame offered to ``tud_network_recv_cb()`` in its
+       receive buffer until released (zero-copy), or ``NULL`` to copy it.
    * - ``tud_network_can_xmit()`` / ``tud_network_xmit()``
      - Checks whether one frame of ``size`` bytes can be queued now, then
        queues it.  Call ``xmit`` once after each successful check.
@@ -139,7 +153,7 @@ the NTB sizes or their ``*_NTB_N`` counts.  Keep descriptor capabilities and
 runtime responses consistent with the enabled NCM features.
 
 The :doc:`../../examples/device/net_lwip_webserver` example includes NCM and
-ECM/RNDIS descriptor sets, lwIP integration, DHCP, DNS, link-state changes, and
+ECM/RNDIS descriptor sets, lwIP integration, DHCP, link-state changes, and
 host setup notes.
 
 Specifications used: *CDC Ethernet Control Model*, Revision 1.2, and *CDC

examples/device/net_lwip_webserver/CMakeLists.txt

@@ -41,7 +41,6 @@ target_include_directories(${PROJECT_NAME} PUBLIC
 # lib/networking sources
 target_sources(${PROJECT_NAME} PUBLIC
   ${TOP}/lib/networking/dhserver.c
-  ${TOP}/lib/networking/dnserver.c
   ${TOP}/lib/networking/rndis_reports.c
   )
 

examples/device/net_lwip_webserver/Makefile

@@ -62,7 +62,6 @@ SRC_C += \
   lib/lwip/src/apps/http/fs.c \
   lib/lwip/src/apps/lwiperf/lwiperf.c \
   lib/networking/dhserver.c \
-  lib/networking/dnserver.c \
   lib/networking/rndis_reports.c
 
 include ../../../hw/bsp/family_rules.mk

examples/device/net_lwip_webserver/README.md

@@ -1,13 +1,13 @@
 # Network Web Server (lwIP)
 
-A USB virtual network adapter that runs a small lwIP stack on the device, serving DHCP, DNS, and a web page over USB.
+A USB virtual network adapter that runs a small lwIP stack on the device, serving DHCP and a web page over USB.
 
 ## What it does
 
 - Brings up a USB network interface. Depending on the target MCU the build is either CDC-NCM (the default for most MCUs) or a dual RNDIS + CDC-ECM device that offers two configurations and lets the host pick its preferred one (Windows uses RNDIS, macOS uses CDC-ECM, Linux works with either). The NCM build also ships a BOS / Microsoft OS 2.0 descriptor so Windows auto-loads its NCM driver.
-- The device takes IP address 192.168.7.1 and runs a DHCP server (handing out 192.168.7.2–192.168.7.4), a DNS server (resolves `tiny.usb`), and an HTTP server.
+- The device takes IP address 192.168.7.1 and runs a DHCP server (handing out 192.168.7.2–192.168.7.4) and an HTTP server.
 - Pressing the board button toggles the network link state (up/down) to simulate an Ethernet cable being unplugged/plugged.
-- On higher-RAM MCUs it also starts an iperf TCP server for throughput testing.
+- On higher-RAM MCUs it also starts an iperf TCP server for throughput testing, and the NCM build hands received frames to lwIP in place instead of copying them (`tud_network_recv_hold()`).
 - Blinks the board LED to indicate USB state (not mounted / mounted / suspended).
 
 ## Requirements
@@ -33,8 +33,8 @@ Notable `tusb_config.h` settings:
 // NCM tuning (see ncm.h for performance notes)
 #define CFG_TUD_NCM_IN_NTB_MAX_SIZE  2048
 #define CFG_TUD_NCM_OUT_NTB_MAX_SIZE 4096        // 2048 on low-RAM MCUs
-#define CFG_TUD_NCM_OUT_NTB_N        1
-#define CFG_TUD_NCM_IN_NTB_N         1
+#define CFG_TUD_NCM_OUT_NTB_N        2           // 1 on low-RAM MCUs
+#define CFG_TUD_NCM_IN_NTB_N         2           // 1 on low-RAM MCUs
 ```
 
 ## Building

examples/device/net_lwip_webserver/src/lwipopts.h

@@ -55,13 +55,27 @@
 #define ETH_PAD_SIZE                    0
 #define LWIP_IP_ACCEPT_UDP_PORT(p)      ((p) == PP_NTOHS(67))
 
+// The only link is USB, whose CRC already protects every packet, so received frames are not re-verified: that
+// checksum costs about a quarter of the CPU at HS throughput. Outgoing checksums are still generated for the host.
+#define CHECKSUM_CHECK_IP               0
+#define CHECKSUM_CHECK_UDP              0
+#define CHECKSUM_CHECK_TCP              0
+#define CHECKSUM_CHECK_ICMP             0
+
 #define TCP_MSS                         (1500 /*mtu*/ - 20 /*iphdr*/ - 20 /*tcphhr*/)
 #define TCP_SND_BUF                     (4 * TCP_MSS)
 #if LWIP_HIGH_THROUGHPUT
   #define TCP_WND                       (8 * TCP_MSS)
   #define PBUF_POOL_SIZE                8
   // Must grow in step with TCP_SND_BUF (default MEMP_NUM_TCP_SEG=16 caps TCP_SND_BUF at 4*MSS).
   #define MEMP_NUM_TCP_SEG              16
+  // TCP TX segments are allocated from the heap: the 1600 default cannot hold one full MSS segment
+  // next to lwiperf's state, so iperf as client sends its header and then stalls.
+  #define MEM_SIZE                      (TCP_SND_BUF + 2 * TCP_MSS)
+  // 32-bit word checksum, the default 16-bit one costs about a quarter of the CPU at HS throughput
+  #define LWIP_CHKSUM_ALGORITHM         3
+  // NCM hands received frames to lwIP in place (PBUF_REF custom pbufs) instead of copying them
+  #define LWIP_SUPPORT_CUSTOM_PBUF      1
 #else
   #define TCP_WND                       (4 * TCP_MSS)
   #define PBUF_POOL_SIZE                4

examples/device/net_lwip_webserver/src/main.c

@@ -30,7 +30,7 @@ this appears as either a RNDIS or CDC-ECM USB virtual network adapter; the OS pi
 
 RNDIS should be valid on Linux and Windows hosts, and CDC-ECM should be valid on Linux and macOS hosts
 
-The MCU appears to the host as IP address 192.168.7.1, and provides a DHCP server, DNS server, and web server.
+The MCU appears to the host as IP address 192.168.7.1, and provides a DHCP server and web server.
 
 Link State Control:
 - Press the user button to toggle the network link state (UP/DOWN)
@@ -51,9 +51,11 @@ try changing the first byte of tud_network_mac_address[] below from 0x02 to 0x00
 
 #include "bsp/board_api.h"
 #include "tusb.h"
+#if CFG_TUD_NCM
+  #include "class/net/ncm.h" // NTB counts sizing the zero-copy pbuf pool
+#endif
 
 #include "dhserver.h"
-#include "dnserver.h"
 #include "httpd.h"
 #include "lwip/ethip6.h"
 #include "lwip/init.h"
@@ -105,8 +107,9 @@ static dhcp_entry_t entries[] = {
 static const dhcp_config_t dhcp_config = {
     .router = INIT_IP4(0, 0, 0, 0),  /* router address (if any) */
     .port = 67,                      /* listen port */
-    .dns = INIT_IP4(192, 168, 7, 1), /* dns server (if any) */
-    "usb",                           /* dns suffix */
+    /* no dns server: the host may rank this link's DNS first and stall every lookup it cannot answer */
+    .dns = INIT_IP4(0, 0, 0, 0),
+    NULL,                            /* dns suffix */
     TU_ARRAY_SIZE(entries),          /* num entry */
     entries                          /* entries */
 };
@@ -166,10 +169,66 @@ static void usbnet_netif_link_callback(struct netif *netif) {
     tud_network_link_state(BOARD_TUD_RHPORT, link_up);
 }
 
+#if CFG_TUD_NCM && LWIP_HIGH_THROUGHPUT
+// Zero-copy reception: a received frame is handed to lwIP in place, as a PBUF_REF pbuf pointing into the NCM receive
+// buffer, which the class keeps until lwIP frees the pbuf. A frame the class cannot hold, or beyond this pool, is
+// copied into a PBUF_POOL pbuf instead.
+// One pbuf per held frame: every receive NTB can hold frames at once, each up to the datagram count the device
+// advertises to the host per NTB.
+#define RX_PBUF_N (CFG_TUD_NCM_OUT_NTB_N * CFG_TUD_NCM_OUT_MAX_DATAGRAMS_PER_NTB)
+TU_VERIFY_STATIC(RX_PBUF_N <= UINT8_MAX, "rx_pbuf_free_count is 8-bit");
+
+typedef struct {
+  struct pbuf_custom pc; // first: lwIP frees it through its struct pbuf
+  void *ntb;
+} rx_pbuf_t;
+
+static rx_pbuf_t rx_pbuf[RX_PBUF_N];
+static rx_pbuf_t *rx_pbuf_free[RX_PBUF_N];
+static uint8_t rx_pbuf_free_count;
+
+static void rx_pbuf_init(void) {
+  for (uint8_t i = 0; i < RX_PBUF_N; i++) {
+    rx_pbuf_free[i] = &rx_pbuf[i];
+  }
+  rx_pbuf_free_count = RX_PBUF_N;
+}
+
+static void rx_pbuf_release(struct pbuf *p) {
+  rx_pbuf_t *rp = (rx_pbuf_t *) p;
+  tud_network_recv_release(rp->ntb);
+  rx_pbuf_free[rx_pbuf_free_count++] = rp;
+}
+
+// a pbuf referencing the frame in place, or NULL when it must be copied
+static struct pbuf *rx_pbuf_in_place(const uint8_t *src, uint16_t size) {
+  if (rx_pbuf_free_count == 0) {
+    return NULL;
+  }
+  void *ntb = tud_network_recv_hold();
+  if (ntb == NULL) {
+    return NULL;
+  }
+  rx_pbuf_t *rp = rx_pbuf_free[--rx_pbuf_free_count];
+  rp->ntb = ntb;
+  rp->pc.custom_free_function = rx_pbuf_release;
+  return pbuf_alloced_custom(PBUF_RAW, size, PBUF_REF, &rp->pc, (void *) (uintptr_t) src, size);
+}
+#else
+static void rx_pbuf_init(void) {}
+
+static struct pbuf *rx_pbuf_in_place(const uint8_t *src, uint16_t size) {
+  (void) src;
+  (void) size;
+  return NULL;
+}
+#endif
+
 static void init_lwip(void) {
   struct netif *netif = &netif_data;
 
   lwip_init();
+  rx_pbuf_init();
 
   /* the lwip virtual MAC address must be different from the host's; to ensure this, we toggle the LSbit */
   netif->hwaddr_len = sizeof(tud_network_mac_address);
@@ -191,29 +250,24 @@ static void init_lwip(void) {
 #endif
 }
 
-/* handle any DNS requests from dns-server */
-static bool dns_query_proc(const char *name, ip4_addr_t *addr) {
-  if (0 == strcmp(name, "tiny.usb")) {
-    *addr = ipaddr;
-    return true;
-  }
-  return false;
-}
 
 bool tud_network_recv_cb(const uint8_t *src, uint16_t size) {
   struct netif *netif = &netif_data;
 
   if (size) {
-    struct pbuf *p = pbuf_alloc(PBUF_RAW, size, PBUF_POOL);
+    struct pbuf *p = rx_pbuf_in_place(src, size);
 
     if (p == NULL) {
-      printf("ERROR: Failed to allocate pbuf of size %d\n", size);
-      return false;
+      p = pbuf_alloc(PBUF_RAW, size, PBUF_POOL);
+      if (p == NULL) {
+        printf("ERROR: Failed to allocate pbuf of size %d\n", size);
+        return false;
+      }
+
+      /* Copy buf to pbuf */
+      pbuf_take(p, src, size);
     }
 
-    /* Copy buf to pbuf */
-    pbuf_take(p, src, size);
-
     // Surrender ownership of our pbuf unless there was an error
     // Only call pbuf_free if not Ok else it will panic with "pbuf_free: p->ref > 0"
     // or steal it from whatever took ownership of it with undefined consequences.
@@ -222,10 +276,11 @@ bool tud_network_recv_cb(const uint8_t *src, uint16_t size) {
       printf("ERROR: netif input failed\n");
       pbuf_free(p);
     }
-    // Signal tinyusb that the current frame has been processed.
-    tud_network_recv_renew();
   }
 
+  // Signal tinyusb that the current frame has been processed. An empty frame (a ZLP, or an RNDIS packet that failed
+  // validation) must be renewed too: returning true without it leaves OUT un-armed and stalls the link.
+  tud_network_recv_renew();
   return true;
 }
 
@@ -286,11 +341,10 @@ int main(void) {
 
   board_init_after_tusb();
 
-  /* initialize lwip, dhcp-server, dns-server, and http */
+  /* initialize lwip, dhcp-server, and http */
   init_lwip();
   while (!netif_is_up(&netif_data));
   while (dhserv_init(&dhcp_config) != ERR_OK);
-  while (dnserv_init(IP_ADDR_ANY, 53, dns_query_proc) != ERR_OK);
   httpd_init();
 
 #ifdef INCLUDE_IPERF

examples/device/net_lwip_webserver/src/tusb_config.h

@@ -129,13 +129,13 @@ extern "C" {
   #define CFG_TUD_NCM_OUT_NTB_MAX_SIZE 2048
 #endif
 
-// Number of NCM transfer blocks for reception side
+// Number of NCM transfer blocks: a second one lets USB transfer the next NTB while lwIP processes the current one
 #ifndef CFG_TUD_NCM_OUT_NTB_N
-  #define CFG_TUD_NCM_OUT_NTB_N 1
+  #define CFG_TUD_NCM_OUT_NTB_N (LWIP_HIGH_THROUGHPUT ? 2 : 1)
 #endif
 
 #ifndef CFG_TUD_NCM_IN_NTB_N
-  #define CFG_TUD_NCM_IN_NTB_N 1
+  #define CFG_TUD_NCM_IN_NTB_N (LWIP_HIGH_THROUGHPUT ? 2 : 1)
 #endif
 
 //--------------------------------------------------------------------

hw/bsp/ch32v30x/boards/nanoch32v305/board.cmake

@@ -2,4 +2,7 @@ set(LD_FLASH_SIZE 128K)
 set(LD_RAM_SIZE 32K)
 
 function(update_board TARGET)
+  target_compile_definitions(${TARGET} PUBLIC
+    LWIP_HIGH_THROUGHPUT=0
+    )
 endfunction()

hw/bsp/ch32v30x/boards/nanoch32v305/board.mk

@@ -1,3 +1,5 @@
+CFLAGS += -DLWIP_HIGH_THROUGHPUT=0
+
 LDFLAGS += \
   -Wl,--defsym=__FLASH_SIZE=128K \
   -Wl,--defsym=__RAM_SIZE=32K \

hw/bsp/stm32h5/boards/stm32h503nucleo/board.cmake

@@ -8,5 +8,6 @@ function(update_board TARGET)
     )
   target_compile_definitions(${BOARD_TARGET} PUBLIC
     CFG_EXAMPLE_VIDEO_READONLY
+    LWIP_HIGH_THROUGHPUT=0
     )
 endfunction()

hw/bsp/stm32h5/boards/stm32h503nucleo/board.mk

@@ -4,6 +4,7 @@ CFLAGS += \
 	-DSTM32H503xx \
 	-DHSE_VALUE=24000000 \
 	-DCFG_EXAMPLE_VIDEO_READONLY \
+  -DLWIP_HIGH_THROUGHPUT=0 \
 
 # For flash-jlink target
 JLINK_DEVICE = stm32h503rb

hw/bsp/stm32u5/boards/stm32u575nucleo/board.h

@@ -191,18 +191,53 @@ static inline void board_vbus_sense_init(void) {
   HAL_TIM_Base_Start(&htim1);
 }
 
+static inline void board_vbus_override_write(bool vbus_valid) {
+  if (vbus_valid) {
+    USB_OTG_FS->GOTGCTL |= USB_OTG_GOTGCTL_BVALOVAL;
+  } else {
+    USB_OTG_FS->GOTGCTL &= ~USB_OTG_GOTGCTL_BVALOVAL;
+  }
+}
+
+// Samples (1 ms each) the override is synced from OTG_FS registers after the stack starts
+#define BOARD_VBUS_SYNC_SAMPLES 10
+
 void ADC1_IRQHandler(void) {
+  // Any OTG register access, even a read, made while a TX FIFO holds a packet one word short of complete zeroes that
+  // endpoint's DIEPTSIZ.XFRSIZ (seen on stm32u5a5nucleo OTG_HS): if this ISR preempts the stack's slave-mode FIFO push
+  // at that point, the IN endpoint NAKs forever. dcd_init() forces the override on at every (re)start, so the registers
+  // are read and corrected only for the first samples after it (the host does not enumerate, so no FIFO push, within
+  // 100 ms of connect); afterwards OTG_FS is touched only when VBUS changes, or while VBUS is absent: a
+  // tud_deinit()/tud_init() between two samples leaves the override forced on with no VBUS, and with no host attached
+  // there is no FIFO push to collide with.
+  static uint8_t sync_samples = BOARD_VBUS_SYNC_SAMPLES;
+  static bool vbus_state;
+
   if(LL_ADC_IsActiveFlag_EOC(ADC1) != 0) {
     /* Clear flag ADC group regular end of unitary conversion */
     LL_ADC_ClearFlag_EOC(ADC1);
     /* ADC code = 4.5V * R2 / (R1 + R2) * (2^14) / 3.3V
      * with R1 = 330kOhm and R2 = 50kOhm
      */
     const uint32_t threshold = 4500 * 50 / (50 + 330) * 16384 / 3300;
-    if((ADC1->DR > threshold) && (USB_OTG_FS->GOTGCTL & USB_OTG_GOTGCTL_BVALOEN)) {
-      USB_OTG_FS->GOTGCTL |= USB_OTG_GOTGCTL_BVALOVAL;
-    } else {
-      USB_OTG_FS->GOTGCTL &= ~USB_OTG_GOTGCTL_BVALOVAL;
+    const bool vbus_sensed = ADC1->DR > threshold;
+    if (!tusb_inited()) {
+      sync_samples = BOARD_VBUS_SYNC_SAMPLES;
+    } else if (sync_samples > 0) {
+      const uint32_t gotgctl = USB_OTG_FS->GOTGCTL;
+      const bool override_on = (gotgctl & USB_OTG_GOTGCTL_BVALOEN) != 0;
+      vbus_state = vbus_sensed && override_on;
+      if (vbus_state != ((gotgctl & USB_OTG_GOTGCTL_BVALOVAL) != 0)) {
+        board_vbus_override_write(vbus_state);
+      }
+      if (override_on) {
+        sync_samples--;
+      }
+    } else if (vbus_sensed != vbus_state) {
+      vbus_state = vbus_sensed;
+      board_vbus_override_write(vbus_state);
+    } else if (!vbus_sensed && (USB_OTG_FS->GOTGCTL & USB_OTG_GOTGCTL_BVALOVAL) != 0) {
+      board_vbus_override_write(false);
     }
   }
   if(LL_ADC_IsActiveFlag_OVR(ADC1) != 0) {