[RFC] bnxt_en: BCM57416 P4 AF_XDP zero-copy results and upstream direction

[RFC] bnxt_en: BCM57416 P4 AF_XDP zero-copy results and upstream direction

[Knox Hutchinson]

Michael, Pavan, and netdev,

At ProStaking, we have been testing AF_XDP zero-copy with 2 KiB UMEM frames on BCM57416 P4 hardware for Firedancer. We would like to contribute the work and get your guidance on the right upstream approach.

Our prototype derives from Broadcom’s out-of-tree bnxt_en 239.1.170.0 source, not current net-next. We tested PCI 14e4:16d8 revision 01, firmware 220.0.59.0/package 220.0.83.0, and Ubuntu kernel 6.8.0-138-generic. A separate Firedancer change publishes reserved FILL buffers before bind while preserving disjoint buffer ownership.

A one-hour Solana testnet voting run produced 361/361 caught-up observations, median slot lag 2 and maximum 6, with no increases in monitored driver drop/error counters.

Later restarts exposed an ENOBUFS rejection in our candidate: the RX producer had advanced to 525 before a post-open check compared it with ring size 511. We corrected this by recording initial allocation success before packet processing starts, while retaining partial-fill rejection. Three corrected ordinary start/traffic/stop cycles passed.

The measured hour used the earlier compile-time Firedancer opt-in. The later corrected candidate uses the default-off runtime option net.xdp.experimental_prebind_fill; we have not repeated the hour on that combination. Boot-image payload is verified, but persistent-voter reboot and exceptional firmware-reset/device-loss/DMA teardown remain unqualified.

Source, patches, configurations, licensing and checkpoint-specific evidence:
https://github.com/ProStaking/af-xdp/tree/627be0b7325501c8e24dce74f29746c42877ae88/bnxt-p4

Which upstream direction would you recommend for this 2 KiB use case? We saw Björn Töpel’s October 2 page-pool memory-provider RFC mentioning bnxt with aligned 4 KiB chunks:
https://lore.kernel.org/netdev/20261002190018.696925-1-bjorn@kernel.org/

We would welcome guidance on coordination and P4 resource-lifetime requirements. Our target net-next comparison and pending-series review remain incomplete. Generally applicable fixes would need independent verification on that target, separate from feature enablement.

Implementation, tests, review and this report used substantial AI assistance. This is an engineering-results/design RFC, not a merge-ready patch or production-support claim.

Knox Hutchinson
ProStaking

Re: [RFC] bnxt_en: BCM57416 P4 AF_XDP zero-copy results and upstream direction

[John Fastabend]

On Sun, 01 Nov 2026 16:33, Knox Hutchinson <KnoxHutchinson@ProStaking.onmicrosoft.com> wrote:
>Michael, Pavan, and netdev,
>
>At ProStaking, we have been testing AF_XDP zero-copy with 2 KiB UMEM frames on BCM57416 P4 hardware for Firedancer. We would like to contribute the work and get your guidance on the right upstream approach.

I just picked this up reading mailing list so I don't have any specific
background on the BCM* part.

But, we are looking at AF_XDP. Did you need to change anything on core
side or just driver changes?

>
>Our prototype derives from Broadcom’s out-of-tree bnxt_en 239.1.170.0 source, not current net-next. We tested PCI 14e4:16d8 revision 01, firmware 220.0.59.0/package 220.0.83.0, and Ubuntu kernel 6.8.0-138-generic. A separate Firedancer change publishes reserved FILL buffers before bind while preserving disjoint buffer ownership.

So first step from netdev side would be to get it working on the in-tree
driver and push an RFC or patches. Then we can review them.

>
>A one-hour Solana testnet voting run produced 361/361 caught-up observations, median slot lag 2 and maximum 6, with no increases in monitored driver drop/error counters.
>
>Later restarts exposed an ENOBUFS rejection in our candidate: the RX producer had advanced to 525 before a post-open check compared it with ring size 511. We corrected this by recording initial allocation success before packet processing starts, while retaining partial-fill rejection. Three corrected ordinary start/traffic/stop cycles passed.
>
>The measured hour used the earlier compile-time Firedancer opt-in. The later corrected candidate uses the default-off runtime option net.xdp.experimental_prebind_fill; we have not repeated the hour on that combination. Boot-image payload is verified, but persistent-voter reboot and exceptional firmware-reset/device-loss/DMA teardown remain unqualified.
>
>Source, patches, configurations, licensing and checkpoint-specific evidence:
>https://github.com/ProStaking/af-xdp/tree/627be0b7325501c8e24dce74f29746c42877ae88/bnxt-p4
>
>Which upstream direction would you recommend for this 2 KiB use case? We saw Björn Töpel’s October 2 page-pool memory-provider RFC mentioning bnxt with aligned 4 KiB chunks:
>https://lore.kernel.org/netdev/20261002190018.696925-1-bjorn@kernel.org/

Just port the code to upstream and push it RFC its easier to review.

>
>We would welcome guidance on coordination and P4 resource-lifetime requirements. Our target net-next comparison and pending-series review remain incomplete. Generally applicable fixes would need independent verification on that target, separate from feature enablement.

Does the p4 resource lifetime matter from kernel side? I would expect p4
updates are run through p4 compilers and then firmware is
flashed/restarted?

>
>Implementation, tests, review and this report used substantial AI assistance. This is an engineering-results/design RFC, not a merge-ready patch or production-support claim.

Can you provide a perf update I guess from above this improved
performance for some tests? Was it netperf based, some hardware setup? I
at least am not familiar with your test infra so understanding what was
tested and how it looks would help.

.John

RE: [RFC] bnxt_en: BCM57416 P4 AF_XDP zero-copy results and upstream direction

[Knox Hutchinson]

John,

Thanks. I should clarify that “P4” means Broadcom’s hardware generation here, specifically BCM57416, not the P4 programming language.

We made no changes to AF_XDP core. The changes are in the vendor bnxt_en driver and Firedancer userspace initialization, which publishes reserved FILL buffers before bind. We tested 2 KiB UMEM frames on an EPYC 9254 server with Ubuntu kernel 6.8.0-138-generic, firmware 220.0.59.0/package 220.0.83.0, and a driver derived from Broadcom 239.1.170.0.

The workload was a real Solana testnet voting validator, not netperf or a controlled traffic generator. We collected CPU, memory, network, disk and voting/replay counters every ten seconds for an hour. All 361 observations were caught up, with no increases in monitored driver drop/error counters.

Against the prior same-host baseline, validator system CPU went from 0.08518 to 0.06705 core equivalents; total validator CPU went from 37.021 to 35.969. RX traffic averaged roughly 23k packets/s. The layout also changed from two network tiles to one, alongside kernel, driver and topology changes, so we cannot attribute those differences solely to zero-copy.

The source, configurations and measured results are here:
https://github.com/ProStaking/af-xdp/tree/627be0b7325501c8e24dce74f29746c42877ae88/bnxt-p4

By resource lifetime, I meant DMA/buffer ownership through NIC reset or device loss. Those exceptional paths remain unqualified. The measured hour also predates the restart correction described in the first email.

Understood on the in-tree driver. An upstream-native port and reviewable RFC patches are the next step; the current vendor-derived package is not that submission.

Knox