Skip to content

26.7: recurring page fault panic in pf_test() on the IPv4 shared-forwarding path (ip_tryforward -> pfil_mbuf_fwd -> pf_check_out) #309

Description

@arigsbee

Important notices

Before you add a new report, we ask you kindly to acknowledge the following:

Closest prior reports checked, and why each is distinct, are listed under "Related but not duplicate" in Additional context below.

Describe the bug

Since upgrading to 26.7, both completed boot sessions on this release have ended in a kernel panic with a byte-for-byte identical backtrace and an identical fault address. The fault is a read from a NULL base pointer (rbx = 0, fault virtual address = 0x5b6) inside pf_test(), reached from the IPv4 shared-forwarding fast path on the netisr software interrupt thread:

ip_input -> ip_tryforward -> pfil_mbuf_fwd -> pf_check_out -> pf_test  [page fault]

Last known working version: 25.7.x. This system ran 25.x on identical hardware with an equivalent ruleset and never panicked. /var/crash contains no dumps predating the 26.7 upgrade. The first panic occurred less than 24 hours after the upgrade.

The kernel build string is unchanged between 26.7 and the currently installed 26.7.1_1, so the system is fully patched and still affected.

opnsense-revert was considered for confirming the previous version, but reverting the kernel on the only router in a production household to chase an event with a mean time to occurrence measured in days was judged too costly against the evidence already in hand. The two textdumps are attached instead.

To Reproduce

Not deterministically reproducible on demand. It occurs under ordinary production traffic at an interval on the order of one to ten days. The conditions present on both occasions:

  1. Run 26.7 with net.pf.share_forward=1 (the shipped default) so IPv4 forwarding is deferred to ip_tryforward.
  2. Route traffic between two igc interfaces (LAN igc1 to WAN igc0) with outbound NAT and interface normalization (scrub) active on the WAN.
  3. Include UDP-encapsulated tunnel traffic in the mix, some of it fragmented: Tailscale on port 41641 and WireGuard, both NATed toward CGNAT space by explicit outbound NAT rules.
  4. Pass forwarded traffic through inline inspection (Suricata IPS in forced-emulated netmap mode, dev.netmap.admode: 2, plus Zenarmor in routed L3 mode).
  5. Wait. Panic arrives on the swi1: netisr 0 thread.

No user-space process was implicated, and no precursor kernel message appeared in the ten hours before the second panic.

Expected behavior

Forwarded IPv4 packets traverse pf_test() on the shared-forwarding path without dereferencing a NULL structure pointer.

Describe alternatives you considered

net.pf.share_forward=0 and net.pf.share_forward6=0 were applied on 2026-07-26 at 13:22:55 EDT, on the reasoning that this takes forwarded traffic off the ip_tryforward path where both panics landed. The change is now persistent via System, Settings, Tunables and via Firewall, Settings, Advanced. The current boot session is running with the mitigation in place and has not panicked. Recurrence, or the absence of it, will be reported here.

Before applying it, the routed forwarding path was benchmarked on both sides of the change so the cost of the workaround could be stated rather than guessed at. On this hardware the mitigation is free. Full method and numbers follow.

Benchmark: cost of disabling shared forwarding on this system

Method

Load was generated LAN to WAN, which is the exact path both panics occurred on: a LAN client running iperf3 -t 20 -P 16 -R against public 10 Gb iperf3 servers in Chicago, New York, and Washington, so traffic transited igc1 -> igc0 through ip_tryforward and outbound NAT. Traffic terminating on the firewall itself would not exercise the forwarding path and was therefore not used.

Each run is a snapshot pair taken on the firewall immediately before and after the 20 s transfer, differencing:

  • kern.cp_time (validated at 1016 ticks/s on every window, that is stathz 127 x 8 CPUs)
  • per-thread cumulative CPU time from ps -H -p 12 -o tdname -o time for swi1: netisr 0
  • per-thread CPU time for the if_io_tqg_* taskqueue threads (direct-dispatch forwarding)
  • process CPU time for Suricata and Zenarmor
  • netstat -I igc0 -b, for forwarded packet count

Three loaded runs plus one no-load idle control were captured on each side of the change under an otherwise identical configuration. share_forward only alters the cost of forwarded packets, so the discriminating metric is CPU time per forwarded packet, not raw throughput.

Results

share_forward = 1 (before):

Run Mbps Busy cores of 8 netisr 0 CPU-s if_io_tqg sum suricata zenarmor Fwd pkts
1 194 1.558 1.16 3.31 14.06 9.31 533,136
2 162 1.556 1.22 3.29 14.04 9.37 456,092
3 158 1.607 1.19 3.30 14.02 9.36 442,306
mean 171.3 1.574 1.19 3.30 14.04 9.35 477,178

share_forward = 0 (after):

Run Mbps Busy cores of 8 netisr 0 CPU-s if_io_tqg sum suricata zenarmor Fwd pkts
1 181 1.657 1.24 3.57 14.97 9.99 500,250
2 172 1.647 1.18 3.48 14.48 9.90 488,687
3 187 1.669 1.24 3.66 15.35 10.21 514,931
mean 180.0 1.658 1.22 3.57 14.93 10.03 501,289

Idle controls: 0.175 busy cores before, 0.102 busy cores after, at negligible background traffic in both cases.

Normalized for the throughput difference between arms:

Normalized metric share_forward=1 share_forward=0 Delta
Whole-box CPU per forwarded packet 69.8 us 70.0 us +0.3 %
netisr 0 CPU per forwarded packet 2.52 us (sd 0.29) 2.43 us (sd 0.04) -3.6 %
if_io_tqg_* CPU per forwarded packet 6.96 us 7.12 us +2.3 %
Suricata CPU per forwarded packet 29.4 us 29.8 us +1.2 %

Interpretation

Every delta is smaller than the run-to-run variance within the baseline arm, which was about 11.5 % on the netisr figure alone. There is no measurable cost to disabling shared forwarding on this system.

The reason is visible in the absolute numbers rather than the deltas. swi1: netisr 0 consumes roughly 1.2 CPU-seconds per 21-second loaded window, about 5.5 % of one core out of eight, while inline inspection (Suricata IPS plus Zenarmor) consumes about 25 CPU-seconds over the same window, roughly 1.2 cores. On a box running inline DPI, pf's forwarding work is around twenty times smaller than the inspection cost, so removing a forwarding-path optimization is lost in the noise. Operators running a bare firewall with no inline inspection, or at 1 Gb/s and above rather than the roughly 180 Mb/s this WAN delivers, should not assume the same result.

Throughput was not degraded. It measured slightly higher after the change (180.0 vs 171.3 Mb/s mean), which is WAN variance rather than a router effect; the baseline arm alone spanned 158 to 194 Mb/s, and 180 sits inside that envelope. The dummynet shaper was not the limiter in either arm (ipfw queue show reported zero drops with negligible queue depth), and no single CPU thread was saturated in any run.

Other alternatives considered and rejected: disabling scrub on the WAN (too broad, and MSS clamping is required here for the tunnels); removing the Tailscale and WireGuard outbound NAT rules (would break the remote access this box exists to provide); disabling Suricata or Zenarmor (does not touch the faulting path, which is pf's own).

Screenshots

Not applicable.

Relevant log files

Backtrace, panic #2, 2026-07-26 00:10:31 EDT:

db:0:kdb.enter.default>  show pcpu
cpuid        = 2
curthread    = 0xfffff80102eaa780: pid 12 tid 100067 critnest 1 "swi1: netisr 0"
curvnet      = 0xfffff800012ac880

db:0:kdb.enter.default>  bt
Tracing pid 12 tid 100067 td 0xfffff80102eaa780
kdb_enter() at kdb_enter+0x33/frame 0xfffffe00fbf49820
panic() at panic+0x43/frame 0xfffffe00fbf49880
trap_pfault() at trap_pfault+0x37d/frame 0xfffffe00fbf498d0
calltrap() at calltrap+0x8/frame 0xfffffe00fbf498d0
--- trap 0xc, rip = 0xffffffff8291d590, rsp = 0xfffffe00fbf499a0, rbp = 0xfffffe00fbf49c70 ---
pf_test() at pf_test+0x8c0/frame 0xfffffe00fbf49c70
pf_check_out() at pf_check_out+0x2e/frame 0xfffffe00fbf49ca0
pfil_mbuf_fwd() at pfil_mbuf_fwd+0x38/frame 0xfffffe00fbf49cd0
ip_tryforward() at ip_tryforward+0x24c/frame 0xfffffe00fbf49d90
ip_input() at ip_input+0x333/frame 0xfffffe00fbf49df0
swi_net() at swi_net+0x128/frame 0xfffffe00fbf49e60
ithread_loop() at ithread_loop+0x239/frame 0xfffffe00fbf49ef0
fork_exit() at fork_exit+0x7b/frame 0xfffffe00fbf49f30

Panic #1, 2026-07-16 15:15:12 EDT, differs only in cpuid (4 vs 2) and the raw rip value, which shifts with KASLR on each boot. Every symbol-plus-offset and every frame address is identical.

Fault detail, panic #2:

[809635] Fatal trap 12: page fault while in kernel mode
[809635] cpuid = 2; apic id = 02
[809635] fault virtual address    = 0x5b6
[809635] fault code               = supervisor read data, page not present
[809635] instruction pointer      = 0x20:0xffffffff8291d590
[809635] stack pointer            = 0x28:0xfffffe00fbf499a0
[809635] frame pointer            = 0x28:0xfffffe00fbf49c70
[809635] current process          = 12 (swi1: netisr 0)
[809635] rax: 00000000000005b6 rbx: 0000000000000000 rbp: fffffe00fbf49c70
[809635] trap number              = 12
[809635] panic: page fault

Fault detail, panic #1:

[78666] Fatal trap 12: page fault while in kernel mode
[78666] cpuid = 4; apic id = 04
[78666] fault virtual address    = 0x5b6
[78666] fault code               = supervisor read data, page not present
[78666] instruction pointer      = 0x20:0xffffffff82f67590
[78666] stack pointer            = 0x28:0xfffffe00fbf499a0
[78666] frame pointer            = 0x28:0xfffffe00fbf49c70
[78666] current process          = 12 (swi1: netisr 0)
[78666] rdi: fffff80102eaa780 rsi: fffffe00fbf499a8 rdx: ffffffff83614090
[78666] rcx: fffff80102eaa780  r8: 0000000000000025  r9: 0000000000001102
[78666] rax: 00000000000005b6 rbx: 0000000000000000 rbp: fffffe00fbf49c70
[78666] r10: fffff802d5953bb0 r11: 0000000000000000 r12: fffff80102eaa780

Identical fault virtual address (0x5b6), identical rax, identical rbx = 0, identical stack and frame pointers across two independent panics nine days apart. This reads as a structure pointer that is NULL at a specific, repeatable point in pf_test(), with field offset 0x5b6 being read from it.

The last non-panic kernel message before panic #2 was at tick 772745, roughly ten hours prior:

[772745] config_aqm Unable to configure flowset, flowset busy!

That is from the dummynet AQM shaper and appears routinely, so it is noted for completeness rather than offered as related. The syslog buffer was not flushed at panic time, so there is no userland log evidence from the final minutes.

Attached textdumps

Both textdumps are attached, each containing ddb.txt, config.txt, msgbuf.txt, panic.txt, and version.txt:

  • OPNsense_panic_20260716_textdump_sanitized.tar.gz
  • OPNsense_panic_20260726_textdump_sanitized.tar.gz

Note on sanitization: msgbuf.txt is the only member that was modified, because the boot console log carries identifying data. Substitutions are the public hostname to OPNsense.example.net, the WAN IPv4 to 203.0.113.10 (RFC 5737), the WAN IPv6 to 2001:db8::1 (RFC 3849), the Tailscale CGNAT address to 100.64.0.1, each distinct MAC to a distinct locally-administered 02:00:00:00:00:NN (so interface identity and the ARP lines still read correctly), and the SSH host key and HTTPS certificate fingerprints to literal REDACTED markers. RFC 1918 addressing, kernel addresses, tick counts, and symbols are untouched. ddb.txt, config.txt, panic.txt, and version.txt are byte-identical to what savecore wrote, so the backtrace, register dump, and stack evidence are unmodified. Unsanitized copies can be provided privately if a maintainer needs them.

Additional context

Timeline

Both panic timestamps and both kernel uptimes are derived from the dump headers and the msgbuf.txt tick values.

Event Wall clock (EDT) Kernel uptime at panic
Upgraded to 26.7, system booted 2026-07-15 ~17:24 boot
Panic #1 2026-07-16 15:15:12 21 h 51 m (tick 78666)
Auto-reboot 2026-07-16 15:16:36 boot
Panic #2 2026-07-26 00:10:31 9 d 08 h 53 m (tick 809635)
Auto-reboot 2026-07-26 00:12:02 boot

Two boot sessions on 26.7, two panics. Uptime at panic differs by an order of magnitude, so this is not an uptime-correlated leak or counter wrap. Traffic volume between the two events was comparable.

Configuration context

Shared forwarding was enabled (the shipped default) for both panics. Values at the time of both events:

net.pf.share_forward: 1
net.pf.share_forward6: 1
net.pf.default_to_drop: 0
net.pf.filter_local: 0
net.pf.states_hashsize: 131072
net.pf.source_nodes_hashsize: 32768
net.pf.udpendpoint_hashsize: 32768

Normalization is active on all interfaces, with MSS clamping on WAN and on the WireGuard group (from /tmp/rules.debug):

scrub on { wireguard }  proto tcp from any to any max-mss 1380
scrub on igc0 inet all  max-mss 1412
scrub on igc0 inet6 all  max-mss 1392
scrub in all

Outbound NAT is heavily used for nested VPN traffic, including two rules that NAT Tailscale WireGuard toward CGNAT space:

nat on igc0 inet from (tailscale0:network) to any -> (igc0:0) port 1024:65535
nat on igc0 inet from (wireguard:network) to any -> (igc0:0) port 1024:65535
nat on igc0 inet from 192.168.50.55/32 to 100.64.0.0/10 port 41641 -> (igc0:0)
nat on igc0 inet from 192.168.50.56/32 to 100.64.0.0/10 port 41641 -> (igc0:0)

So the traffic mix crossing this path includes UDP-encapsulated tunnel traffic, some of it fragmented, being NATed and normalized while traversing the shared-forwarding fast path. That combination looks like the relevant trigger class.

Loaded modules at time of report:

pf.ko  if_gre.ko  if_enc.ko  opensolaris.ko  if_lagg.ko  if_infiniband.ko
if_bridge.ko  bridgestp.ko  carp.ko  if_re.ko  zfs.ko  pflog.ko  pfsync.ko
acpi_wmi.ko  ig4.ko  dummynet.ko  ipfw.ko  if_wg.ko  nullfs.ko  ipdivert.ko

Plugins in use: Suricata (IPS, netmap in forced-emulated mode, dev.netmap.admode: 2), Zenarmor (routed L3, reporting plus blocking), WireGuard, Tailscale, and a dummynet/ipfw shaper with CoDel.

Why this looks like a 26.7 regression

The 26.7 changelog contains recent churn in exactly this code region:

  • 26.7.r2 (2026-07-09): src: pf: do not mangle IP header before shared forwarding
  • 26.7 (2026-07-15): pf: do not mangle IP header before shared forwarding
  • 26.7 (2026-07-15): pf: pf_route() 'dst' no longer holds the gateway in 15.x
  • 26.7 (2026-07-15): pf: clear anchor after stepping into it in pf_match_translation_rule()

Identical hardware and an equivalent ruleset on 25.x produced no panics. Both 26.7 boot sessions produced one.

Related but not duplicate

Questions for maintainers

  1. Is net.pf.share_forward=0 / share_forward6=0 a sound interim workaround for this fault, or is there reason to expect the same NULL dereference to be reachable off the ip_tryforward path? The only side effect we are aware of is that it breaks captive portal redirection, which is not in use here, and per the benchmark above the performance cost on this hardware is nil.
  2. Is the identical fault virtual address = 0x5b6 with rbx = 0 across two independent panics enough to identify the structure and the field being dereferenced in pf_test()?
  3. If a debug kernel, a specific pf counter, or pf debug logging at a particular level would produce better evidence than a textdump, we are willing to run it in production and report back.

Environment

OPNsense 26.7.1_1 (amd64)
FreeBSD 15.1-RELEASE-p1, kernel stable/26.7-n283674-12334a596709 SMP
13th Gen Intel(R) Core(TM) i3-13100 (4C/8T, hw.ncpu: 8)
32 GB DDR5 (hw.physmem: 34042396672)
Network Intel(R) I226-V (igc0 WAN, igc1 LAN, igc7 cellular failover)
Dump device /dev/nda0p3 (NVMe), textdump, Dump Status: good for both events

OPNsense_panic_20260716_textdump_sanitized.tar.gz
OPNsense_panic_20260726_textdump_sanitized.tar.gz

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions