[Bug] nvpair-node-scanner causes macOS kernel panics (userspace_watchdog_timeout) due to aggressive net.Interfaces() polling

[CoDev-BitByBit-Commit]

System Information

• OS: macOS (Apple Silicon / arm64)
• Component: nvpair-node-scanner / shared/discovery
• Symptom: Entire system freezes and reboots with a userspace_watchdog_timeout kernel panic.

Description

When running the PAIR macOS application, the system will randomly freeze and trigger a forced watchdog reboot. Analyzing the Apple Diagnostic Reports (/Library/Logs/DiagnosticReports/configd_*.spin) reveals that Apple's core network configuration daemon (configd) gets starved while waiting for a kernel read-write lock (krwlock).

The macOS crash logs explicitly show that this kernel lock is being held by nvpair-node-scanner:

*1 ??? (kernel.release.t6050 + 773428) [0xfffffe000b4a8d34] (blocked by krwlock for reading owned by nvpair-node-scanner [1309] [unique pid 791059] thread 0x6fa35b) 2

Root Cause Analysis

The root cause stems from the Windows mDNS workaround in services/shared/discovery/discovery.go.

The Browse loop executes every 5 seconds. Within this loop, sendMulticastQuery() is called to bypass a Windows-specific bug where grandcat/zeroconf drops UDP packets due to multicast bindings. However, sendMulticastQuery() executes unconditionally on all operating systems.

func sendMulticastQuery(service, domain string) map[string]bool {
    // ...
    ifaces, err := net.Interfaces()
    // ...
    // Iterates through all interfaces, binding and closing UDP sockets

On macOS, calling net.Interfaces() and rapidly binding/closing raw UDP sockets on every valid interface every 5 seconds inside a background daemon causes severe kernel routing table lock contention (krwlock). Over time, this starves configd, triggering the userspace_watchdog_timeout panic.

As the source comment itself notes, standard zeroconf receive/transmit works perfectly on macOS/Linux. This custom per-interface UDP barrage is only necessary on Windows.

Proposed Fix

Guarding the sendMulticastQuery() function so that it only executes on Windows completely eradicates the kernel panics on macOS, while preserving standard zeroconf discovery.

Patch for services/shared/discovery/discovery.go:

import "runtime"

// ...

func sendMulticastQuery(service, domain string) map[string]bool {
	outcomes := make(map[string]bool)
	
	// FIX: Restrict the socket workaround to Windows to prevent macOS krwlock starvation
	if runtime.GOOS != "windows" {
		return outcomes
	}

	msg := new(dns.Msg)
	// ... rest of the function ...
}

We have compiled and deployed this patch locally across our macOS cluster, and it immediately resolved the panics. We highly recommend upstreaming this guard to prevent other macOS users from experiencing severe system reboots.

[sherief-nv]

Probably the same as issue #59

[ClosedLadder]

I have the same headline symptom—four userspace watchdog timeout: no successful checkins from configd panics in one day on macOS 26.6.2 with PAIR 0.1.1—and measured a mechanism that appears broader than routing-table contention. It also suggests that a runtime.GOOS gate alone would not protect Macs in a mixed cluster.

sendMulticastQuery() opens a new IPv4 UDP socket with an ephemeral port per interface on each five-second scan, sends one PTR query to 224.0.0.251:5353, and closes it. Browsing runs in the node-scanner and both proxies. A Linux node with about ten IPv4 interfaces emitted about six queries per second from never-seen source ports; a 60-second capture on a six-node LAN recorded 1,315 such packets, or 22 per second, from PAIR nodes. Linux members were the largest senders, so gating only the macOS call would not stop other nodes from delivering the traffic to every Mac on the LAN.

On macOS 26, each delivery to applications listening on port 5353 created socket-flow entries keyed by source address and source port. net.soflow.count rose from 0 to approximately 500,000 in three hours on one Mac. Kernel disassembly and live counters indicate that soflow_gc_expire can cap expiry near 100 flows per ten-second run when entries interleave across several sockets.

With the Application Firewall enabled, all four panic stackshots showed a socket-closing thread spending minutes in cfil_info_free to verify_content_filter cleanup while other networking work waited behind the lock chain. Twice the closing thread was nvpair-node-scanner and twice it was a Chrome child I/O thread. The Apple kernel behavior is being reported separately; PAIR's fresh-port discovery pattern appears to make it reachable in ordinary use, but I am not claiming PAIR is the sole cause of the underlying kernel defects.

A second Mac later lost ordinary Wi-Fi connectivity for approximately 30 minutes without a panic. It had 73,000 to 116,000 attached flow entries, repeated Wi-Fi datapath timeouts and ENOBUFS filter events, and high firewall-daemon CPU. That event resembles issue #59, although an identical kernel mechanism across reports is not yet proven.

Suggested PAIR changes:

1. Reuse one long-lived discovery socket per eligible interface, bound to port 5353, instead of opening a fresh ephemeral-port socket per scan.
2. Browse once per node and share results with the proxies over IPC.
3. Exclude container bridges, VPN tunnels, and link-local point-to-point interfaces from LAN multicast discovery.
4. Expose and raise the scan interval, and provide a supported manual nodes only / discovery off mode.

These trigger conditions are reconstructed from existing incidents. A safe read-only exposure check is sysctl net.soflow.count net.soflow.high_water_mark. Please do not deliberately change network state, quit socket-heavy applications, detach the firewall, or provoke a stall on a production Mac with a very large count. Any controlled reproduction belongs only in an isolated NVIDIA or Apple lab with out-of-band recovery.

[stevbev]

I can confirm that this is the same root cause as #59 and can be resolved with #72