[Bug]: mDNS answers are sent from an ephemeral source port, so reflectors never relay node records across subnets

[ndizazzo]

PAIR version or commit

13b68115fa2c9c1d94f1ead1358f8d5a527cfecf ("Update to 0.1.1"), built from source with make build-services. nvpair-ui-broker --version reports 0.40.2; the services bundle banner reports product v0.91.7.

Affected component

Discovery or pairing

Environment

OS:               Ubuntu 24.04 LTS (headless, multi-user.target) x2; Arch-based rolling x1; desktop app on a fourth machine
Architecture:     arm64 (2 nodes), x86_64 (1 node)
GPU and driver:   Jetson Orin integrated GPU (arm64 nodes); 2x discrete NVIDIA GPUs (x86_64 node)
Engine/version:   n/a - the failure happens during discovery, before any engine is involved
Model:            n/a
Cluster size:     4 nodes, split across two routed subnets on separate VLANs, all joined to one cluster
Build:            Go 1.27.1, `make build-services`; headless nodes run `nvpair-tui` inside tmux
Network:          router mDNS proxy/reflector enabled between the two VLANs; unicast fully routed between them

Steps to reproduce

1. Put PAIR nodes on two different IP subnets that sit on separate VLANs, with a router-level mDNS proxy/reflector enabled between them (an RFC 6762 style reflector, i.e. one that relays traffic it identifies as mDNS on UDP 5353).
2. Confirm unicast between the subnets is open: curl http://<peer>:14318/v1/node-info returns HTTP 200 in both directions.
3. Start PAIR on a node in each subnet and wait. The Nodes tab never lists the peer from the other subnet. Nodes on the same subnet discover each other normally.
4. Sniff mDNS on a host in each subnet (bind UDP 5353 with SO_REUSEADDR|SO_REUSEPORT, join 224.0.0.251, so the sniffer coexists with the running responder), filtering to packets sourced from the router's own address:
   • the reflector relays _nvpair-node._tcp.local queries in both directions (qd=1 an=0 flags=0x0000, first labels ['_nvpair-node','local']);
   • every relayed answer packet (flags=0x8400, 3-26 answers) contains only unrelated .local records. No _nvpair-node record ever crosses.
5. Isolate why. From a socket bound to UDP 5353 that has joined 224.0.0.251, send a PTR query for _nvpair-node._tcp.local and record the source port of every answer received. Every _nvpair-node answer arrives from an ephemeral high port, never from 5353.

Expected behavior

PAIR sends its mDNS answers from UDP source port 5353, as RFC 6762 section 6 requires. A reflector then recognizes them as mDNS responses and relays them, and nodes on reflected subnets discover each other without manual configuration.

Actual behavior

Every mDNS answer PAIR sends originates from an ephemeral source port. A reflector never classifies those datagrams as mDNS, so it relays the queries but silently drops the responses, and cross-subnet discovery never completes. Same-link discovery is unaffected because the querier receives the datagram directly and does not care about the source port - only a reflector does.

The source of the behavior is services/shared/mdns/responder.go, in sendOnInterface. Every transmit binds a fresh socket with an ephemeral port:

conn, err := net.ListenUDP("udp4", &net.UDPAddr{IP: src, Port: 0})

The function's own doc comment explains the intent:

sendOnInterface is the core of the Windows send workaround: it transmits buf from a fresh unicast-bound socket on the given interface (setting the multicast interface + TTL for group targets), never from the multicast-bound receive socket that Windows refuses to send from.

The Windows workaround is reasonable, but applying it on every platform makes PAIR's responses non-conformant on all of them.

Suggested fix: on non-Windows platforms, transmit from the socket already bound to 5353, or bind the per-interface send socket to port 5353 with SO_REUSEADDR/SO_REUSEPORT, and keep the ephemeral-port path as a Windows-only fallback.

How we confirmed the router's mDNS reflection was actually on

This was ruled out first, since a misconfigured reflector produces the same symptom.

• The gateway's mDNS proxy was enabled with a VLAN scope that included both the VLAN holding the desktop/UI node and the VLAN holding the headless nodes.
• The service scope was tried three ways: a specific allowlist, then a custom entry added for PAIR's service type, then set to All. The measurements above were re-run after each change and were identical every time.
• Positive control that reflection was live: the sniffers on both sides captured mDNS packets sourced from the gateway's own address, including relayed _nvpair-node._tcp.local queries that had originated on the opposite subnet. Multicast reflection was demonstrably working in both directions - only the answers were missing.
• Unicast routing between the subnets is fully open (HTTP 200 on /v1/node-info both directions), so this is not a firewall or routing problem.
• avahi-daemon was inactive on the headless nodes, so no second responder was competing with PAIR's own.
• PAIR advertises exactly one service type, _nvpair-node._tcp (ServiceType in services/shared/noderec/noderec.go), and that is the name the reflector was observed relaying queries for. The name is not the problem.

Sanitized logs or screenshots

# Packets sourced from the gateway, captured on a host in each subnet.
# Queries for the PAIR service type are relayed across the VLAN boundary:
<gateway>  qd=1 an=0 flags=0x0000 labels=['_nvpair-node', 'local']
<gateway>  qd=1 an=0 flags=0x0000 labels=['_nvpair-node', 'local']

# Answers relayed by the gateway never contain a PAIR record:
<gateway>  qd=0 an=9  flags=0x8400 labels=['local']
<gateway>  qd=0 an=26 flags=0x8400 labels=['local']

# Query for _nvpair-node._tcp.local sent from a socket bound to UDP/5353.
# Source port of each answer received (expected 5353 per RFC 6762 section 6):
<local-node>:32809   _nvpair-node._tcp.local
<local-node>:33112   _nvpair-node._tcp.local
<local-node>:35195   _nvpair-node._tcp.local
<peer-node>:32845    _nvpair-node._tcp.local
<peer-node>:33047    _nvpair-node._tcp.local
<peer-node>:37499    _nvpair-node._tcp.local

Workarounds in the meantime: add each cross-subnet peer as a manual node (Manual tab, a), or place every PAIR node on a single VLAN.

Confirmations

• I searched existing issues for duplicates.
• This is not a security vulnerability.
• I agree to follow the Code of Conduct.

[Noah-Tervalon-Nvidia]

This was a mistake on our end and we'll make sure to fix it, we're working on a fix now.

[stevbev]

This should be resolved by #72 that gates the Windows workaround to Windows systems, and properly binds the mDNS socket as a long bound, sharable socket instead of the current new socket every ~5s.