Skip to content

Report complete AF_XDP diagnostics and explain generic fallback - #5

Open
m4r1k wants to merge 5 commits into
atoonk:mainfrom
m4r1k:pr/afxdp-diagnostics-v0.3.0
Open

Report complete AF_XDP diagnostics and explain generic fallback#5
m4r1k wants to merge 5 commits into
atoonk:mainfrom
m4r1k:pr/afxdp-diagnostics-v0.3.0

Conversation

@m4r1k

@m4r1k m4r1k commented Aug 11, 2026

Copy link
Copy Markdown

Linux exposes six AF_XDP counters through XDP_STATISTICS. Wireblast currently
shows only part of that information, and empty-ring events are easy to mistake
for packet drops.

This patch reports all six counters with their Linux UAPI names, counts, rates,
and short descriptions. The dashboard separates drop counters from ring
starvation and can expand into per-queue detail. The final noninteractive
summary uses the same taxonomy. When the statistics exporter is enabled, its
queue records carry the same fields.

The fallback path also keeps the reason that native AF_XDP failed. Native mode
still tries zero-copy first and copy mode second. Wireblast tries generic copy
mode only after go-afxdp reports that its native attach or bind attempts were
exhausted. Interface lookup, queue discovery, filter setup, validation, and BPF
build errors return immediately rather than being presented as driver
limitations.

If generic mode succeeds, the noninteractive frontend logs the exact native
error and the interactive dashboard displays it. If generic mode also fails,
the returned error preserves both causes. The fallback contains no NIC or
driver-specific logic, and Wireblast v0.3.0's multi-buffer option remains
active in both attach attempts.

This PR is temporarily stacked on #4 so its per-queue
export can be reviewed with the diagnostics that provide those fields. After
#4 merges, this branch should be rebased onto main and the PR retargeted.

The diagnostics were exercised between paired Google Cloud C4N instances from
c4n-highcpu-2 through c4n-highcpu-96, using gVNIC, GVE, native XDP, and
AF_XDP zero-copy. The inline log excerpt is from the receive side of a C96
IMIX run and includes the native/zero-copy gate plus the six final counters.
The live matrix stayed in native mode, so the generic fallback paths were
covered by automated tests rather than this sample.

C96 gVNIC/GVE native zero-copy log excerpt
started: ens3: 32 queue(s), zero-copy, native XDP, driver gve, rx filter udp/9000, napi defer=2 flush=200µs
  AF_XDP diagnostics:
      rx_dropped                0 (0/s)  RX drops for other reasons
      rx_ring_full              0 (0/s)  RX drops because the RX ring was full
      rx_invalid_descs          0 (0/s)  RX drops due to invalid descriptors
      tx_invalid_descs          0 (0/s)  TX drops due to invalid descriptors
      rx_fill_ring_empty_descs  2408 (22.4/s)  failed reads from an empty fill ring
      tx_ring_empty_descs       181.71 M (1.69 M/s)  failed reads from an empty TX ring

Tested with:

  • go test -count=1 ./...
  • go test -race -count=1 ./...
  • go vet ./...
  • gofmt -l .
  • go build -buildvcs=false ./cmd/wireblast

@m4r1k
m4r1k marked this pull request as ready for review August 11, 2026 10:07
@atoonk

atoonk commented Aug 11, 2026

Copy link
Copy Markdown
Owner

Hey @m4r1k thanks for this. This is nie.

One blocker before we go ahead with it. I gave it a quick try with a veth pair, and on an idle receiver with no traffic at all I see a yellow tx_ring_empty_descs climbing at about 10/s, plus a ! queue 0: tx ring empty problem line.
I know that's the kernel counting empty TX ring checks on every poll, so technically it's correct, but nothing is actually wrong, there's nothing the user can do about it, and people will think something is broken.
I want to figure out how to deal with that better before merging. Any ideas? wonder if we can show the counters some other way, without reporting them as errors.

@m4r1k

m4r1k commented Aug 11, 2026

Copy link
Copy Markdown
Author

Hey @atoonk

I agree that tx_ring_empty_descs should not be presented as a problem. The counter is useful evidence, but a nonzero value is expected on an idle or receive-only socket. Cisco VPP keeps all counters in machine-readable output but shows only nonzero counters in the human-readable views. Would that work? The empty-ring counters would remain visible as neutral diagnostics when expanded. I can also add an idle-receiver regression test where tx_ring_empty_descs increases but no problem line or warning is produced. WDYT?

@atoonk

atoonk commented Aug 12, 2026

Copy link
Copy Markdown
Owner

@m4r1k I like the VPP approach, with one tweak. Assuming i understand you correctly "Show only nonzero" doesn't quite solve my case: on an idle receiver tx_ring_empty_descs is nonzero and climbing, you mentioned your own C96 log shows 181M of them on a totally healthy run. So it would still be on screen, and to most users a big climbing counter that says "failed reads" looks like something is broken and i'd like to avoid that.

What I'd prefer:

  • Default dashboard and final summary: only the four real drop counters, and only when nonzero. The two empty-ring counters never show up there, no matter the value.
  • Expanded detail view: all six, with friendlier wording for the ring ones, e.g. "polls that found the TX ring empty (normal when idle or rate limited)".
  • Exporter: all six, always, as you have it.

And yes to the idle-receiver regression test. Maybe have it assert the starvation counters don't appear in the default view at all, not just that there's no warning line.

@m4r1k
m4r1k force-pushed the pr/afxdp-diagnostics-v0.3.0 branch from 292e61c to acf5937 Compare August 12, 2026 17:08
@m4r1k

m4r1k commented Aug 12, 2026

Copy link
Copy Markdown
Author

Makes sense. I've pushed a new draft following your suggestion.

  • default dashboard and final summary show only the four drop counters, and only when they are nonzero. The two empty-ring counters are omitted entirely.
  • expanded view still shows all six counters. The empty-ring counters use neutral styling and clearer wording, including polls that found the TX ring empty. CSV and JSONL (PR Add CSV and JSONL statistics output #4) output is unchanged.
  • added idle-receiver regression tests.

Both PRs rebased onto main.

WDYT?

@atoonk

atoonk commented Aug 19, 2026

Copy link
Copy Markdown
Owner

hey @m4r1k just wanted to let you know that i havent forgotten. Bit busy with $dayjob, will look at this soon!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants