You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Closest prior reports checked, and why each is distinct, are listed under "Related but not duplicate" in Additional context below.
Describe the bug
Since upgrading to 26.7, both completed boot sessions on this release have ended in a kernel panic with a byte-for-byte identical backtrace and an identical fault address. The fault is a read from a NULL base pointer (rbx = 0, fault virtual address = 0x5b6) inside pf_test(), reached from the IPv4 shared-forwarding fast path on the netisr software interrupt thread:
Last known working version: 25.7.x. This system ran 25.x on identical hardware with an equivalent ruleset and never panicked. /var/crash contains no dumps predating the 26.7 upgrade. The first panic occurred less than 24 hours after the upgrade.
The kernel build string is unchanged between 26.7 and the currently installed 26.7.1_1, so the system is fully patched and still affected.
opnsense-revert was considered for confirming the previous version, but reverting the kernel on the only router in a production household to chase an event with a mean time to occurrence measured in days was judged too costly against the evidence already in hand. The two textdumps are attached instead.
To Reproduce
Not deterministically reproducible on demand. It occurs under ordinary production traffic at an interval on the order of one to ten days. The conditions present on both occasions:
Run 26.7 with net.pf.share_forward=1 (the shipped default) so IPv4 forwarding is deferred to ip_tryforward.
Route traffic between two igc interfaces (LAN igc1 to WAN igc0) with outbound NAT and interface normalization (scrub) active on the WAN.
Include UDP-encapsulated tunnel traffic in the mix, some of it fragmented: Tailscale on port 41641 and WireGuard, both NATed toward CGNAT space by explicit outbound NAT rules.
Pass forwarded traffic through inline inspection (Suricata IPS in forced-emulated netmap mode, dev.netmap.admode: 2, plus Zenarmor in routed L3 mode).
Wait. Panic arrives on the swi1: netisr 0 thread.
No user-space process was implicated, and no precursor kernel message appeared in the ten hours before the second panic.
Expected behavior
Forwarded IPv4 packets traverse pf_test() on the shared-forwarding path without dereferencing a NULL structure pointer.
Describe alternatives you considered
net.pf.share_forward=0 and net.pf.share_forward6=0 were applied on 2026-07-26 at 13:22:55 EDT, on the reasoning that this takes forwarded traffic off the ip_tryforward path where both panics landed. The change is now persistent via System, Settings, Tunables and via Firewall, Settings, Advanced. The current boot session is running with the mitigation in place and has not panicked. Recurrence, or the absence of it, will be reported here.
Before applying it, the routed forwarding path was benchmarked on both sides of the change so the cost of the workaround could be stated rather than guessed at. On this hardware the mitigation is free. Full method and numbers follow.
Benchmark: cost of disabling shared forwarding on this system
Method
Load was generated LAN to WAN, which is the exact path both panics occurred on: a LAN client running iperf3 -t 20 -P 16 -R against public 10 Gb iperf3 servers in Chicago, New York, and Washington, so traffic transited igc1 -> igc0 through ip_tryforward and outbound NAT. Traffic terminating on the firewall itself would not exercise the forwarding path and was therefore not used.
Each run is a snapshot pair taken on the firewall immediately before and after the 20 s transfer, differencing:
kern.cp_time (validated at 1016 ticks/s on every window, that is stathz 127 x 8 CPUs)
per-thread cumulative CPU time from ps -H -p 12 -o tdname -o time for swi1: netisr 0
per-thread CPU time for the if_io_tqg_* taskqueue threads (direct-dispatch forwarding)
process CPU time for Suricata and Zenarmor
netstat -I igc0 -b, for forwarded packet count
Three loaded runs plus one no-load idle control were captured on each side of the change under an otherwise identical configuration. share_forward only alters the cost of forwarded packets, so the discriminating metric is CPU time per forwarded packet, not raw throughput.
Results
share_forward = 1 (before):
Run
Mbps
Busy cores of 8
netisr 0 CPU-s
if_io_tqg sum
suricata
zenarmor
Fwd pkts
1
194
1.558
1.16
3.31
14.06
9.31
533,136
2
162
1.556
1.22
3.29
14.04
9.37
456,092
3
158
1.607
1.19
3.30
14.02
9.36
442,306
mean
171.3
1.574
1.19
3.30
14.04
9.35
477,178
share_forward = 0 (after):
Run
Mbps
Busy cores of 8
netisr 0 CPU-s
if_io_tqg sum
suricata
zenarmor
Fwd pkts
1
181
1.657
1.24
3.57
14.97
9.99
500,250
2
172
1.647
1.18
3.48
14.48
9.90
488,687
3
187
1.669
1.24
3.66
15.35
10.21
514,931
mean
180.0
1.658
1.22
3.57
14.93
10.03
501,289
Idle controls: 0.175 busy cores before, 0.102 busy cores after, at negligible background traffic in both cases.
Normalized for the throughput difference between arms:
Normalized metric
share_forward=1
share_forward=0
Delta
Whole-box CPU per forwarded packet
69.8 us
70.0 us
+0.3 %
netisr 0 CPU per forwarded packet
2.52 us (sd 0.29)
2.43 us (sd 0.04)
-3.6 %
if_io_tqg_* CPU per forwarded packet
6.96 us
7.12 us
+2.3 %
Suricata CPU per forwarded packet
29.4 us
29.8 us
+1.2 %
Interpretation
Every delta is smaller than the run-to-run variance within the baseline arm, which was about 11.5 % on the netisr figure alone. There is no measurable cost to disabling shared forwarding on this system.
The reason is visible in the absolute numbers rather than the deltas. swi1: netisr 0 consumes roughly 1.2 CPU-seconds per 21-second loaded window, about 5.5 % of one core out of eight, while inline inspection (Suricata IPS plus Zenarmor) consumes about 25 CPU-seconds over the same window, roughly 1.2 cores. On a box running inline DPI, pf's forwarding work is around twenty times smaller than the inspection cost, so removing a forwarding-path optimization is lost in the noise. Operators running a bare firewall with no inline inspection, or at 1 Gb/s and above rather than the roughly 180 Mb/s this WAN delivers, should not assume the same result.
Throughput was not degraded. It measured slightly higher after the change (180.0 vs 171.3 Mb/s mean), which is WAN variance rather than a router effect; the baseline arm alone spanned 158 to 194 Mb/s, and 180 sits inside that envelope. The dummynet shaper was not the limiter in either arm (ipfw queue show reported zero drops with negligible queue depth), and no single CPU thread was saturated in any run.
Other alternatives considered and rejected: disabling scrub on the WAN (too broad, and MSS clamping is required here for the tunnels); removing the Tailscale and WireGuard outbound NAT rules (would break the remote access this box exists to provide); disabling Suricata or Zenarmor (does not touch the faulting path, which is pf's own).
db:0:kdb.enter.default> show pcpu
cpuid = 2
curthread = 0xfffff80102eaa780: pid 12 tid 100067 critnest 1 "swi1: netisr 0"
curvnet = 0xfffff800012ac880
db:0:kdb.enter.default> bt
Tracing pid 12 tid 100067 td 0xfffff80102eaa780
kdb_enter() at kdb_enter+0x33/frame 0xfffffe00fbf49820
panic() at panic+0x43/frame 0xfffffe00fbf49880
trap_pfault() at trap_pfault+0x37d/frame 0xfffffe00fbf498d0
calltrap() at calltrap+0x8/frame 0xfffffe00fbf498d0
--- trap 0xc, rip = 0xffffffff8291d590, rsp = 0xfffffe00fbf499a0, rbp = 0xfffffe00fbf49c70 ---
pf_test() at pf_test+0x8c0/frame 0xfffffe00fbf49c70
pf_check_out() at pf_check_out+0x2e/frame 0xfffffe00fbf49ca0
pfil_mbuf_fwd() at pfil_mbuf_fwd+0x38/frame 0xfffffe00fbf49cd0
ip_tryforward() at ip_tryforward+0x24c/frame 0xfffffe00fbf49d90
ip_input() at ip_input+0x333/frame 0xfffffe00fbf49df0
swi_net() at swi_net+0x128/frame 0xfffffe00fbf49e60
ithread_loop() at ithread_loop+0x239/frame 0xfffffe00fbf49ef0
fork_exit() at fork_exit+0x7b/frame 0xfffffe00fbf49f30
Panic #1, 2026-07-16 15:15:12 EDT, differs only in cpuid (4 vs 2) and the raw rip value, which shifts with KASLR on each boot. Every symbol-plus-offset and every frame address is identical.
Identical fault virtual address (0x5b6), identical rax, identical rbx = 0, identical stack and frame pointers across two independent panics nine days apart. This reads as a structure pointer that is NULL at a specific, repeatable point in pf_test(), with field offset 0x5b6 being read from it.
The last non-panic kernel message before panic #2 was at tick 772745, roughly ten hours prior:
[772745] config_aqm Unable to configure flowset, flowset busy!
That is from the dummynet AQM shaper and appears routinely, so it is noted for completeness rather than offered as related. The syslog buffer was not flushed at panic time, so there is no userland log evidence from the final minutes.
Attached textdumps
Both textdumps are attached, each containing ddb.txt, config.txt, msgbuf.txt, panic.txt, and version.txt:
OPNsense_panic_20260716_textdump_sanitized.tar.gz
OPNsense_panic_20260726_textdump_sanitized.tar.gz
Note on sanitization: msgbuf.txt is the only member that was modified, because the boot console log carries identifying data. Substitutions are the public hostname to OPNsense.example.net, the WAN IPv4 to 203.0.113.10 (RFC 5737), the WAN IPv6 to 2001:db8::1 (RFC 3849), the Tailscale CGNAT address to 100.64.0.1, each distinct MAC to a distinct locally-administered 02:00:00:00:00:NN (so interface identity and the ARP lines still read correctly), and the SSH host key and HTTPS certificate fingerprints to literal REDACTED markers. RFC 1918 addressing, kernel addresses, tick counts, and symbols are untouched. ddb.txt, config.txt, panic.txt, and version.txt are byte-identical to what savecore wrote, so the backtrace, register dump, and stack evidence are unmodified. Unsanitized copies can be provided privately if a maintainer needs them.
Additional context
Timeline
Both panic timestamps and both kernel uptimes are derived from the dump headers and the msgbuf.txt tick values.
Two boot sessions on 26.7, two panics. Uptime at panic differs by an order of magnitude, so this is not an uptime-correlated leak or counter wrap. Traffic volume between the two events was comparable.
Configuration context
Shared forwarding was enabled (the shipped default) for both panics. Values at the time of both events:
Normalization is active on all interfaces, with MSS clamping on WAN and on the WireGuard group (from /tmp/rules.debug):
scrub on { wireguard } proto tcp from any to any max-mss 1380
scrub on igc0 inet all max-mss 1412
scrub on igc0 inet6 all max-mss 1392
scrub in all
Outbound NAT is heavily used for nested VPN traffic, including two rules that NAT Tailscale WireGuard toward CGNAT space:
nat on igc0 inet from (tailscale0:network) to any -> (igc0:0) port 1024:65535
nat on igc0 inet from (wireguard:network) to any -> (igc0:0) port 1024:65535
nat on igc0 inet from 192.168.50.55/32 to 100.64.0.0/10 port 41641 -> (igc0:0)
nat on igc0 inet from 192.168.50.56/32 to 100.64.0.0/10 port 41641 -> (igc0:0)
So the traffic mix crossing this path includes UDP-encapsulated tunnel traffic, some of it fragmented, being NATed and normalized while traversing the shared-forwarding fast path. That combination looks like the relevant trigger class.
Plugins in use: Suricata (IPS, netmap in forced-emulated mode, dev.netmap.admode: 2), Zenarmor (routed L3, reporting plus blocking), WireGuard, Tailscale, and a dummynet/ipfw shaper with CoDel.
Why this looks like a 26.7 regression
The 26.7 changelog contains recent churn in exactly this code region:
26.7.r2 (2026-07-09): src: pf: do not mangle IP header before shared forwarding
26.7 (2026-07-15): pf: do not mangle IP header before shared forwarding
26.7 (2026-07-15): pf: pf_route() 'dst' no longer holds the gateway in 15.x
26.7 (2026-07-15): pf: clear anchor after stepping into it in pf_match_translation_rule()
Identical hardware and an equivalent ruleset on 25.x produced no panics. Both 26.7 boot sessions produced one.
Related but not duplicate
opnsense/src pf+route-to: panic when shared forwarding is enabled #52, pf+route-to: panic when shared forwarding is enabled, is closed, has a different backtrace (sbcut_internal -> tcp_do_segment -> tcp_input) and a different trigger (route-to at a non-existent interface). No route-to rules are in play here.
opnsense/src Shared Forwarding collection #38, Shared Forwarding collection, is closed and catalogues functional shared-forwarding problems, not panics.
FreeBSD Bug 288549, PF panic with NAT + UDP fragments, is in the same code region (pf_setup_pdesc leaving pd->pcksum NULL for PF_VPROTO_FRAGMENT, dereferenced in pf_change_ap). Its fix, commit 8a309785c9b186c809a5d4b017fc8cf849af1ddd, landed 2025-08-05 and predates FreeBSD 15.1, so this kernel should already carry it. Our fault is also a read rather than the NULL write 288549 produces. Offered as a possibly-adjacent defect, not a claimed match.
Questions for maintainers
Is net.pf.share_forward=0 / share_forward6=0 a sound interim workaround for this fault, or is there reason to expect the same NULL dereference to be reachable off the ip_tryforward path? The only side effect we are aware of is that it breaks captive portal redirection, which is not in use here, and per the benchmark above the performance cost on this hardware is nil.
Is the identical fault virtual address = 0x5b6 with rbx = 0 across two independent panics enough to identify the structure and the field being dereferenced in pf_test()?
If a debug kernel, a specific pf counter, or pf debug logging at a particular level would produce better evidence than a textdump, we are willing to run it in production and report back.
Important notices
Before you add a new report, we ask you kindly to acknowledge the following:
Closest prior reports checked, and why each is distinct, are listed under "Related but not duplicate" in Additional context below.
Describe the bug
Since upgrading to 26.7, both completed boot sessions on this release have ended in a kernel panic with a byte-for-byte identical backtrace and an identical fault address. The fault is a read from a NULL base pointer (
rbx = 0,fault virtual address = 0x5b6) insidepf_test(), reached from the IPv4 shared-forwarding fast path on the netisr software interrupt thread:Last known working version: 25.7.x. This system ran 25.x on identical hardware with an equivalent ruleset and never panicked.
/var/crashcontains no dumps predating the 26.7 upgrade. The first panic occurred less than 24 hours after the upgrade.The kernel build string is unchanged between 26.7 and the currently installed 26.7.1_1, so the system is fully patched and still affected.
opnsense-revertwas considered for confirming the previous version, but reverting the kernel on the only router in a production household to chase an event with a mean time to occurrence measured in days was judged too costly against the evidence already in hand. The two textdumps are attached instead.To Reproduce
Not deterministically reproducible on demand. It occurs under ordinary production traffic at an interval on the order of one to ten days. The conditions present on both occasions:
net.pf.share_forward=1(the shipped default) so IPv4 forwarding is deferred toip_tryforward.igcinterfaces (LANigc1to WANigc0) with outbound NAT and interface normalization (scrub) active on the WAN.dev.netmap.admode: 2, plus Zenarmor in routed L3 mode).swi1: netisr 0thread.No user-space process was implicated, and no precursor kernel message appeared in the ten hours before the second panic.
Expected behavior
Forwarded IPv4 packets traverse
pf_test()on the shared-forwarding path without dereferencing a NULL structure pointer.Describe alternatives you considered
net.pf.share_forward=0andnet.pf.share_forward6=0were applied on 2026-07-26 at 13:22:55 EDT, on the reasoning that this takes forwarded traffic off theip_tryforwardpath where both panics landed. The change is now persistent via System, Settings, Tunables and via Firewall, Settings, Advanced. The current boot session is running with the mitigation in place and has not panicked. Recurrence, or the absence of it, will be reported here.Before applying it, the routed forwarding path was benchmarked on both sides of the change so the cost of the workaround could be stated rather than guessed at. On this hardware the mitigation is free. Full method and numbers follow.
Benchmark: cost of disabling shared forwarding on this system
Method
Load was generated LAN to WAN, which is the exact path both panics occurred on: a LAN client running
iperf3 -t 20 -P 16 -Ragainst public 10 Gb iperf3 servers in Chicago, New York, and Washington, so traffic transitedigc1 -> igc0throughip_tryforwardand outbound NAT. Traffic terminating on the firewall itself would not exercise the forwarding path and was therefore not used.Each run is a snapshot pair taken on the firewall immediately before and after the 20 s transfer, differencing:
kern.cp_time(validated at 1016 ticks/s on every window, that is stathz 127 x 8 CPUs)ps -H -p 12 -o tdname -o timeforswi1: netisr 0if_io_tqg_*taskqueue threads (direct-dispatch forwarding)netstat -I igc0 -b, for forwarded packet countThree loaded runs plus one no-load idle control were captured on each side of the change under an otherwise identical configuration.
share_forwardonly alters the cost of forwarded packets, so the discriminating metric is CPU time per forwarded packet, not raw throughput.Results
share_forward = 1(before):share_forward = 0(after):Idle controls: 0.175 busy cores before, 0.102 busy cores after, at negligible background traffic in both cases.
Normalized for the throughput difference between arms:
netisr 0CPU per forwarded packetif_io_tqg_*CPU per forwarded packetInterpretation
Every delta is smaller than the run-to-run variance within the baseline arm, which was about 11.5 % on the netisr figure alone. There is no measurable cost to disabling shared forwarding on this system.
The reason is visible in the absolute numbers rather than the deltas.
swi1: netisr 0consumes roughly 1.2 CPU-seconds per 21-second loaded window, about 5.5 % of one core out of eight, while inline inspection (Suricata IPS plus Zenarmor) consumes about 25 CPU-seconds over the same window, roughly 1.2 cores. On a box running inline DPI, pf's forwarding work is around twenty times smaller than the inspection cost, so removing a forwarding-path optimization is lost in the noise. Operators running a bare firewall with no inline inspection, or at 1 Gb/s and above rather than the roughly 180 Mb/s this WAN delivers, should not assume the same result.Throughput was not degraded. It measured slightly higher after the change (180.0 vs 171.3 Mb/s mean), which is WAN variance rather than a router effect; the baseline arm alone spanned 158 to 194 Mb/s, and 180 sits inside that envelope. The dummynet shaper was not the limiter in either arm (
ipfw queue showreported zero drops with negligible queue depth), and no single CPU thread was saturated in any run.Other alternatives considered and rejected: disabling
scrubon the WAN (too broad, and MSS clamping is required here for the tunnels); removing the Tailscale and WireGuard outbound NAT rules (would break the remote access this box exists to provide); disabling Suricata or Zenarmor (does not touch the faulting path, which is pf's own).Screenshots
Not applicable.
Relevant log files
Backtrace, panic #2, 2026-07-26 00:10:31 EDT:
Panic #1, 2026-07-16 15:15:12 EDT, differs only in
cpuid(4 vs 2) and the rawripvalue, which shifts with KASLR on each boot. Every symbol-plus-offset and every frame address is identical.Fault detail, panic #2:
Fault detail, panic #1:
Identical fault virtual address (
0x5b6), identicalrax, identicalrbx = 0, identical stack and frame pointers across two independent panics nine days apart. This reads as a structure pointer that is NULL at a specific, repeatable point inpf_test(), with field offset0x5b6being read from it.The last non-panic kernel message before panic #2 was at tick 772745, roughly ten hours prior:
That is from the dummynet AQM shaper and appears routinely, so it is noted for completeness rather than offered as related. The syslog buffer was not flushed at panic time, so there is no userland log evidence from the final minutes.
Attached textdumps
Both textdumps are attached, each containing
ddb.txt,config.txt,msgbuf.txt,panic.txt, andversion.txt:OPNsense_panic_20260716_textdump_sanitized.tar.gzOPNsense_panic_20260726_textdump_sanitized.tar.gzNote on sanitization:
msgbuf.txtis the only member that was modified, because the boot console log carries identifying data. Substitutions are the public hostname toOPNsense.example.net, the WAN IPv4 to203.0.113.10(RFC 5737), the WAN IPv6 to2001:db8::1(RFC 3849), the Tailscale CGNAT address to100.64.0.1, each distinct MAC to a distinct locally-administered02:00:00:00:00:NN(so interface identity and the ARP lines still read correctly), and the SSH host key and HTTPS certificate fingerprints to literalREDACTEDmarkers. RFC 1918 addressing, kernel addresses, tick counts, and symbols are untouched.ddb.txt,config.txt,panic.txt, andversion.txtare byte-identical to whatsavecorewrote, so the backtrace, register dump, and stack evidence are unmodified. Unsanitized copies can be provided privately if a maintainer needs them.Additional context
Timeline
Both panic timestamps and both kernel uptimes are derived from the dump headers and the
msgbuf.txttick values.Two boot sessions on 26.7, two panics. Uptime at panic differs by an order of magnitude, so this is not an uptime-correlated leak or counter wrap. Traffic volume between the two events was comparable.
Configuration context
Shared forwarding was enabled (the shipped default) for both panics. Values at the time of both events:
Normalization is active on all interfaces, with MSS clamping on WAN and on the WireGuard group (from
/tmp/rules.debug):Outbound NAT is heavily used for nested VPN traffic, including two rules that NAT Tailscale WireGuard toward CGNAT space:
So the traffic mix crossing this path includes UDP-encapsulated tunnel traffic, some of it fragmented, being NATed and normalized while traversing the shared-forwarding fast path. That combination looks like the relevant trigger class.
Loaded modules at time of report:
Plugins in use: Suricata (IPS, netmap in forced-emulated mode,
dev.netmap.admode: 2), Zenarmor (routed L3, reporting plus blocking), WireGuard, Tailscale, and a dummynet/ipfw shaper with CoDel.Why this looks like a 26.7 regression
The 26.7 changelog contains recent churn in exactly this code region:
src: pf: do not mangle IP header before shared forwardingpf: do not mangle IP header before shared forwardingpf: pf_route() 'dst' no longer holds the gateway in 15.xpf: clear anchor after stepping into it in pf_match_translation_rule()Identical hardware and an equivalent ruleset on 25.x produced no panics. Both 26.7 boot sessions produced one.
Related but not duplicate
pf+route-to: panic when shared forwarding is enabled, is closed, has a different backtrace (sbcut_internal->tcp_do_segment->tcp_input) and a different trigger (route-toat a non-existent interface). Noroute-torules are in play here.Shared Forwarding collection, is closed and catalogues functional shared-forwarding problems, not panics.pf_test()on the shared-forwarding path.PF panic with NAT + UDP fragments, is in the same code region (pf_setup_pdescleavingpd->pcksumNULL forPF_VPROTO_FRAGMENT, dereferenced inpf_change_ap). Its fix, commit8a309785c9b186c809a5d4b017fc8cf849af1ddd, landed 2025-08-05 and predates FreeBSD 15.1, so this kernel should already carry it. Our fault is also a read rather than the NULL write 288549 produces. Offered as a possibly-adjacent defect, not a claimed match.Questions for maintainers
net.pf.share_forward=0/share_forward6=0a sound interim workaround for this fault, or is there reason to expect the same NULL dereference to be reachable off theip_tryforwardpath? The only side effect we are aware of is that it breaks captive portal redirection, which is not in use here, and per the benchmark above the performance cost on this hardware is nil.fault virtual address = 0x5b6withrbx = 0across two independent panics enough to identify the structure and the field being dereferenced inpf_test()?pfcounter, orpfdebug logging at a particular level would produce better evidence than a textdump, we are willing to run it in production and report back.Environment
OPNsense_panic_20260716_textdump_sanitized.tar.gz
OPNsense_panic_20260726_textdump_sanitized.tar.gz