[PATCH net-next v4 0/5] net: devmem: size TX net_iovs from the dmabuf layout

[PATCH net-next v4 0/5] net: devmem: size TX net_iovs from the dmabuf layout

[Stanislav Fomichev]

TX bindings use PAGE_SIZE net_iovs, so a frag covers at most a page.
A 64KB skb needs 16 frags, and MAX_SKB_FRAGS caps devmem skbs at about
68KB, which makes BIG TCP ineffective for devmem TX.

This series picks the TX net_iov size from the dmabuf sg table instead:
the largest power of 2, up to 2MB, which aligns every DMA segment. With
a 2MB hugepage-backed udmabuf a BIG TCP skb takes a few frags. There's
no UAPI change since TX addresses are dmabuf offsets.

This doesn't make skbs larger. TCP still sizes them with size_goal.
Only the frags get larger, and a single frag can now cover the whole
skb.

An skb stays under 64KB unless BIG TCP is enabled. With BIG TCP,
MSG_ZEROCOPY from huge pages can already create frags of 64KB or
more. So drivers already have to handle them:
- mlx5 has a 31-bit length per data segment
- gve splits frags into 16KB descriptors
- fbnic doesn't enable BIG TCP
- bnxt has a 16-bit BD length. It is broken for such page frags today
  and will be fixed separately.

Tested on two mlx5 400Gb/s hosts with kperf (udmabuf, 32KB RX
net_iovs, 2MB hugepage-backed TX buffer), series backported to a 7.1
kernel on the sender only. 1 flow, MTU 4200, receiver tcp_rmem capped
at 16MB so it doesn't drop, median of 5 x 60s runs:

                              before    after
  frags per skb (sender)       16-17      1-2
  throughput (Gb/s)            208.6    211.2
  sender sendmsg CPU (sys)     40.8%    19.5%   (-52%)

The receiver's CPU is the bottleneck, so the gain is CPU, not
throughput. More of the transmit runs from the ACK path, so the
sender's softirq grows (52% -> 59%), but sys + sirq still drops by 16%.

kperf output (sender side) of runs near the median, before:

  client: == Source <sender>
  client:   Tx213.826 Gbps (1603901849600 bytes in 60007723 usec)
  client:   Rx  0.000 Gbps (0 bytes in 60007723 usec)
  client:   TCP retrans reord rtt rttvar d_ce snd_wnd cwnd
  client:             0     0 633      2    0 15556608 7657
  client:   net CPU 49: usr: 0.01% sys: 0.10% idle:44.13% iow: 0.00% irq: 3.21% sirq:52.53%
  client:   app CPU 50: usr: 0.05% sys:41.55% idle:53.88% iow: 0.00% irq: 1.40% sirq: 3.10%

after:

  client: == Source <sender>
  client:   Tx211.242 Gbps (1584440279040 bytes in 60004741 usec)
  client:   Rx  0.000 Gbps (0 bytes in 60004741 usec)
  client:   TCP retrans reord rtt rttvar d_ce snd_wnd cwnd
  client:             0     0 556      3    0 15499264 7650
  client:   net CPU 13: usr: 0.01% sys: 0.08% idle:37.75% iow: 0.00% irq: 3.87% sirq:58.26%
  client:   app CPU 14: usr: 0.06% sys:19.48% idle:77.23% iow: 0.00% irq: 1.19% sirq: 2.02%

The bnxt patch was tested on two 50Gb/s bnxt hosts with
CONFIG_MAX_SKB_FRAGS=45. With the default of 17 its check compiles
out. ncdevmem sending 1KB chunks builds skbs with up to 45 frags. The
stack segments them and the data arrives intact.

Patches:
- 1: bnxt: segment GSO skbs with too many frags in the stack
- 2-3: devmem TX prep (truesize, niov_shift in the TX lookup)
- 4: pick the TX net_iov size from the sg table
- 5: selftest for TX from a hugepage-backed dmabuf

v4:
- drop the per-device frag size limit and the bnxt, fbnic and mlx5
  patches. Drivers already have to handle large page frags (Jakub).

v3: https://lore.kernel.org/netdev/20261105222058.944293-1-sdf@fomichev.me/
- default the netmem TX frag size limit to PAGE_SIZE, opt in bnxt,
  fbnic and mlx5 (Mina)

v2: https://lore.kernel.org/netdev/20261103235152.1904727-1-sdf@fomichev.me/
- reword the TX truesize commit message (Mina)
- explain why the TX net_iov size ignores the device limit (Mina)
- use 1 flow results in the cover letter (Mina)

v1: https://lore.kernel.org/netdev/20261102191438.1059771-1-sdf@fomichev.me/

Stanislav Fomichev (5):
  bnxt_en: let the stack segment GSO skbs with too many frags
  net: devmem: charge the frag length to TX skb truesize
  net: devmem: use niov_shift in net_devmem_get_niov_at()
  net: devmem: size TX net_iovs from the dmabuf sg table
  selftests: drv-net: devmem: TX from a hugepage-backed dmabuf

 drivers/net/ethernet/broadcom/bnxt/bnxt.c     |  5 +++
 net/core/datagram.c                           |  2 +-
 net/core/devmem.c                             | 36 +++++++++++++---
 net/core/netdev-genl.c                        |  2 +-
 .../selftests/drivers/net/hw/devmem.py        | 10 ++++-
 .../selftests/drivers/net/hw/devmem_lib.py    | 43 +++++++++++++++++--
 .../selftests/drivers/net/hw/ncdevmem.c       |  8 +++-
 .../selftests/drivers/net/hw/nk_devmem.py     | 11 ++++-
 8 files changed, 101 insertions(+), 16 deletions(-)

-- 
2.53.0-Meta

[PATCH net-next v4 1/5] bnxt_en: let the stack segment GSO skbs with too many frags

[Stanislav Fomichev]

With MAX_SKB_FRAGS > TX_MAX_FRAGS, the xmit path linearizes skbs with
more frags than a packet can have BDs. For GSO skbs, linearizing needs a
large allocation, can produce a head which doesn't fit in a BD with BIG
TCP, and always fails for unreadable skbs. Clear the GSO features of
such skbs in ndo_features_check instead, so that the stack segments
them. Non-GSO skbs are still linearized.

Suggested-by: Eric Dumazet <edumazet@google.com>
Link: https://lore.kernel.org/all/CANn89i+ps_CozfOyRnKav0_LUmSekJ9ExB5JkDbTAVf_ss_98g@mail.gmail.com/
Signed-off-by: Stanislav Fomichev <sdf@fomichev.me>
Reviewed-by: Mina Almasry <almasrymina@google.com>
Reviewed-by: Dragos Tatulea <dtatulea@nvidia.com>
Reviewed-by: Breno Leitao <leitao@debian.org>
Reviewed-by: Michael Chan <michael.chan@broadcom.com>
---
 drivers/net/ethernet/broadcom/bnxt/bnxt.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.c b/drivers/net/ethernet/broadcom/bnxt/bnxt.c
index 33c4cacc8ffe..f20b61180ab2 100644
--- a/drivers/net/ethernet/broadcom/bnxt/bnxt.c
+++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.c
@@ -14320,6 +14320,11 @@ static netdev_features_t bnxt_features_check(struct sk_buff *skb,
 				skb_shinfo(skb)->gso_size))
 		features &= ~NETIF_F_GSO_MASK;
 
+#if (MAX_SKB_FRAGS > TX_MAX_FRAGS)
+	if (skb_shinfo(skb)->nr_frags > TX_MAX_FRAGS)
+		features &= ~NETIF_F_GSO_MASK;
+#endif
+
 	switch (vlan_get_protocol(skb)) {
 	case htons(ETH_P_IP):
 		if (!skb->encapsulation)
-- 
2.53.0-Meta

[PATCH net-next v4 2/5] net: devmem: charge the frag length to TX skb truesize

[Stanislav Fomichev]

Devmem TX charges each frag the net_iov size, PAGE_SIZE today:
zerocopy_fill_skb_from_devmem() adds PAGE_SIZE to skb->truesize per
frag and __zerocopy_sg_from_iter() charges it to sk_wmem_queued. TCP
uncharges in bytes as acked data is trimmed from the skb
(tcp_trim_head()) and uncharges the rest when the skb is freed. A frag
shorter than a page leaves the difference charged until then, which is
small with PAGE_SIZE net_iovs.

TX net_iovs are about to grow up to 2MB, while frags stay bounded by
the skb and many frags share a net_iov. Charging the net_iov size would
charge up to 2MB per frag: a 64KB skb with two frags would charge 4MB
and fill the default 4MB send buffer on its own.

Charge the frag length instead, which also matches what TCP uncharges.

Signed-off-by: Stanislav Fomichev <sdf@fomichev.me>
Reviewed-by: Mina Almasry <almasrymina@google.com>
Reviewed-by: Dragos Tatulea <dtatulea@nvidia.com>
Reviewed-by: Breno Leitao <leitao@debian.org>
---
 net/core/datagram.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/net/core/datagram.c b/net/core/datagram.c
index ecc30a6f8302..489d92fe6dae 100644
--- a/net/core/datagram.c
+++ b/net/core/datagram.c
@@ -737,7 +737,7 @@ zerocopy_fill_skb_from_devmem(struct sk_buff *skb, struct iov_iter *from,
 
 		netmem_inc_ref(net_iov_to_netmem(niov));
 		skb_add_rx_frag_netmem(skb, i, net_iov_to_netmem(niov), off,
-				       size, PAGE_SIZE);
+				       size, size);
 		iov_iter_advance(from, size);
 		length -= size;
 		i++;
-- 
2.53.0-Meta

[PATCH net-next v4 3/5] net: devmem: use niov_shift in net_devmem_get_niov_at()

[Stanislav Fomichev]

TX bindings use PAGE_SIZE net_iovs, and the TX lookup hard-codes it.
Derive the index and offset from binding->niov_shift instead, so TX
bindings can use larger net_iovs. No functional change, TX bindings are
still created with PAGE_SHIFT.

Signed-off-by: Stanislav Fomichev <sdf@fomichev.me>
Reviewed-by: Mina Almasry <almasrymina@google.com>
Reviewed-by: Dragos Tatulea <dtatulea@nvidia.com>
---
 net/core/devmem.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/net/core/devmem.c b/net/core/devmem.c
index 5c7d9640e9d1..c6868f39261a 100644
--- a/net/core/devmem.c
+++ b/net/core/devmem.c
@@ -581,13 +581,15 @@ struct net_iov *
 net_devmem_get_niov_at(struct net_devmem_dmabuf_binding *binding,
 		       size_t virt_addr, size_t *off, size_t *size)
 {
+	size_t niov_size = 1UL << binding->niov_shift;
+
 	if (virt_addr >= binding->dmabuf->size)
 		return NULL;
 
-	*off = virt_addr % PAGE_SIZE;
-	*size = PAGE_SIZE - *off;
+	*off = virt_addr & (niov_size - 1);
+	*size = niov_size - *off;
 
-	return binding->tx_vec[virt_addr / PAGE_SIZE];
+	return binding->tx_vec[virt_addr >> binding->niov_shift];
 }
 
 /*** "Dmabuf devmem memory provider" ***/
-- 
2.53.0-Meta

[PATCH net-next v4 4/5] net: devmem: size TX net_iovs from the dmabuf sg table

[Stanislav Fomichev]

PAGE_SIZE TX net_iovs cap devmem skbs at MAX_SKB_FRAGS pages (~68KB),
which makes BIG TCP ineffective.

TX net_iovs only need to be DMA contiguous. Use the largest power of 2,
from PAGE_SIZE up to 2MB, which aligns every DMA segment of the dmabuf:
- 2MB hugepage-backed udmabufs get 2MB net_iovs
- no UAPI change, TX addresses are dmabuf offsets
- skbs keep their size, only the frags get larger

The net_iov size doesn't depend on the device. A frag is never larger
than its skb, and the skb is never larger than the device's
gso_max_size. TCP sizes skbs to fit, and validate_xmit_skb() segments
bigger ones. gso_max_size is 64KB unless BIG TCP is enabled. With BIG
TCP, page frags can already be 64KB or larger, for example with
MSG_ZEROCOPY from huge pages or with 64KB pages. So drivers that
enable BIG TCP already have to handle them. bnxt doesn't handle them
yet. It has a 16-bit BD length and needs a separate fix.

Signed-off-by: Stanislav Fomichev <sdf@fomichev.me>
Reviewed-by: Mina Almasry <almasrymina@google.com>
Reviewed-by: Dragos Tatulea <dtatulea@nvidia.com>
---
 net/core/devmem.c      | 28 ++++++++++++++++++++++++++--
 net/core/netdev-genl.c |  2 +-
 2 files changed, 27 insertions(+), 3 deletions(-)

diff --git a/net/core/devmem.c b/net/core/devmem.c
index c6868f39261a..713528048897 100644
--- a/net/core/devmem.c
+++ b/net/core/devmem.c
@@ -303,6 +303,20 @@ int net_devmem_bind_dmabuf_to_queue(struct net_device *dev, u32 rxq_idx,
 	return err;
 }
 
+static bool net_devmem_sgt_aligned(struct sg_table *sgt, size_t size)
+{
+	struct scatterlist *sg;
+	unsigned int i;
+
+	for_each_sgtable_dma_sg(sgt, sg, i) {
+		if (!IS_ALIGNED(sg_dma_address(sg), size) ||
+		    !IS_ALIGNED(sg_dma_len(sg), size))
+			return false;
+	}
+	return true;
+}
+
+/* niov_shift == 0 picks the net_iov size from the dmabuf, TX only */
 struct net_devmem_dmabuf_binding *
 net_devmem_bind_dmabuf(struct net_device *dev, void *vdev,
 		       struct device *dma_dev,
@@ -313,11 +327,11 @@ net_devmem_bind_dmabuf(struct net_device *dev, void *vdev,
 		       struct netlink_ext_ack *extack)
 {
 	struct net_devmem_dmabuf_binding *binding;
-	size_t niov_size = 1UL << niov_shift;
 	static u32 id_alloc_next;
 	struct scatterlist *sg;
 	struct dma_buf *dmabuf;
 	unsigned int sg_idx;
+	size_t niov_size;
 	size_t niov_idx;
 	size_t i;
 	long err;
@@ -342,7 +356,6 @@ net_devmem_bind_dmabuf(struct net_device *dev, void *vdev,
 	binding->vdev = vdev;
 	write_pnet(&binding->net, sock_net(nl_sk));
 	binding->uid = sk_uid(nl_sk);
-	binding->niov_shift = niov_shift;
 	xa_init_flags(&binding->bound_rxqs, XA_FLAGS_ALLOC);
 
 	err = percpu_ref_init(&binding->ref,
@@ -387,6 +400,17 @@ net_devmem_bind_dmabuf(struct net_device *dev, void *vdev,
 		goto err_unmap;
 	}
 
+	if (!niov_shift) {
+		niov_size = SZ_2M;
+		while (niov_size > PAGE_SIZE &&
+		       (!IS_ALIGNED(dmabuf->size, niov_size) ||
+			!net_devmem_sgt_aligned(binding->sgt, niov_size)))
+			niov_size /= 2;
+		niov_shift = ilog2(niov_size);
+	}
+	binding->niov_shift = niov_shift;
+	niov_size = 1UL << niov_shift;
+
 	if (!IS_ALIGNED(dmabuf->size, niov_size)) {
 		err = -EINVAL;
 		NL_SET_ERR_MSG_FMT(extack,
diff --git a/net/core/netdev-genl.c b/net/core/netdev-genl.c
index 9c5bf17b04e5..d2ccb1090496 100644
--- a/net/core/netdev-genl.c
+++ b/net/core/netdev-genl.c
@@ -1261,7 +1261,7 @@ int netdev_nl_bind_tx_doit(struct sk_buff *skb, struct genl_info *info)
 	binding = net_devmem_bind_dmabuf(bind_dev,
 					 bind_dev != netdev ? netdev : NULL,
 					 dma_dev, DMA_TO_DEVICE, dmabuf_fd,
-					 PAGE_SHIFT, priv, NETLINK_CB(skb).sk,
+					 0, priv, NETLINK_CB(skb).sk,
 					 info->extack);
 	if (IS_ERR(binding)) {
 		err = PTR_ERR(binding);
-- 
2.53.0-Meta

[PATCH net-next v4 5/5] selftests: drv-net: devmem: TX from a hugepage-backed dmabuf

[Stanislav Fomichev]

TX net_iovs are now sized from the dmabuf layout, but the TX tests
use a regular memfd, which usually keeps them small. Add -H to ncdevmem
to back the udmabuf with 2MB hugepages and a TX test which sends a 3MB
buffer from it, so that frags end at and continue past the 2MB net_iov
boundary. The payload has a 7 byte period to catch offset errors.

Signed-off-by: Stanislav Fomichev <sdf@fomichev.me>
Reviewed-by: Mina Almasry <almasrymina@google.com>
Reviewed-by: Dragos Tatulea <dtatulea@nvidia.com>
---
 .../selftests/drivers/net/hw/devmem.py        | 10 ++++-
 .../selftests/drivers/net/hw/devmem_lib.py    | 43 +++++++++++++++++--
 .../selftests/drivers/net/hw/ncdevmem.c       |  8 +++-
 .../selftests/drivers/net/hw/nk_devmem.py     | 11 ++++-
 4 files changed, 63 insertions(+), 9 deletions(-)

diff --git a/tools/testing/selftests/drivers/net/hw/devmem.py b/tools/testing/selftests/drivers/net/hw/devmem.py
index 82c11ffc4add..aa7141fb36f7 100755
--- a/tools/testing/selftests/drivers/net/hw/devmem.py
+++ b/tools/testing/selftests/drivers/net/hw/devmem.py
@@ -3,7 +3,7 @@
 
 from os import path
 from devmem_lib import (setup_test, run_rx, run_tx, run_tx_chunks, run_rx_hds,
-                        run_rx_large_niov)
+                        run_rx_large_niov, run_tx_large_niov)
 from lib.py import ksft_run, ksft_exit, ksft_disruptive
 from lib.py import NetDrvEpEnv
 
@@ -36,12 +36,18 @@ from lib.py import NetDrvEpEnv
     run_rx_large_niov(cfg)
 
 
+@ksft_disruptive
+def check_tx_large_niov(cfg) -> None:
+    """Run the devmem TX test from a hugepage-backed dmabuf."""
+    run_tx_large_niov(cfg)
+
+
 def main() -> None:
     """Run the devmem test cases."""
     with NetDrvEpEnv(__file__) as cfg:
         setup_test(cfg, path.abspath(path.dirname(__file__) + "/ncdevmem"))
         ksft_run([check_rx, check_tx, check_tx_chunks, check_rx_hds,
-                  check_rx_large_niov],
+                  check_rx_large_niov, check_tx_large_niov],
                  args=(cfg,))
     ksft_exit()
 
diff --git a/tools/testing/selftests/drivers/net/hw/devmem_lib.py b/tools/testing/selftests/drivers/net/hw/devmem_lib.py
index 71b916c16196..8058fd1ddd50 100644
--- a/tools/testing/selftests/drivers/net/hw/devmem_lib.py
+++ b/tools/testing/selftests/drivers/net/hw/devmem_lib.py
@@ -2,6 +2,7 @@
 # pylint: disable=invalid-name,too-many-arguments
 """Shared helpers for devmem TCP selftests."""
 
+import hashlib
 import os
 import re
 
@@ -42,15 +43,17 @@ NR_HUGEPAGES_FILE = "/proc/sys/vm/nr_hugepages"
     defer(_restore_nr_hugepages, nr_hugepages)
 
 
-def _probe_devmem(cfg, rx_page_size):
+def _probe_devmem(cfg, rx_page_size, hugepages=False):
     """Return True if ncdevmem can bind cfg.ifname at @rx_page_size."""
     probe_command = f"{cfg.bin_local} -f {cfg.ifname}"
     if rx_page_size != RX_PAGE_SIZE_DEFAULT:
         probe_command += f" -b {rx_page_size}"
+    if hugepages:
+        probe_command += " -H"
     return cmd(probe_command, fail=False, shell=True).ret == 0
 
 
-def require_devmem(cfg, rx_page_size=RX_PAGE_SIZE_DEFAULT):
+def require_devmem(cfg, rx_page_size=RX_PAGE_SIZE_DEFAULT, hugepages=False):
     """Probe ncdevmem on cfg.ifname and SKIP the test if devmem isn't supported."""
     if rx_page_size not in PROBE_RX_PAGE_SIZES:
         raise RuntimeError(
@@ -63,10 +66,15 @@ NR_HUGEPAGES_FILE = "/proc/sys/vm/nr_hugepages"
         # ncdevmem's queue range and cause the probe to fail.
         cfg.devmem_supported = {size: _probe_devmem(cfg, size)
                                 for size in PROBE_RX_PAGE_SIZES}
+        cfg.devmem_hugepages = _probe_devmem(cfg, RX_PAGE_SIZE_DEFAULT,
+                                             hugepages=True)
 
     if not cfg.devmem_supported[RX_PAGE_SIZE_DEFAULT]:
         raise KsftSkipEx("Test requires devmem support")
 
+    if hugepages and not cfg.devmem_hugepages:
+        raise KsftSkipEx("Test requires a 2MB hugepage-backed dmabuf")
+
     if rx_page_size != RX_PAGE_SIZE_DEFAULT:
         page_size = os.sysconf("SC_PAGE_SIZE")
         if not _is_aligned(rx_page_size, page_size):
@@ -169,7 +177,7 @@ def ncdevmem_rx(cfg, port, verify=True, fail_on_linear=False, flow_steer=False,
     return " ".join(parts)
 
 
-def ncdevmem_tx(cfg, port, chunk_size=0):
+def ncdevmem_tx(cfg, port, chunk_size=0, hugepages=False):
     """Build the ncdevmem TX send command."""
     if hasattr(cfg, 'netns'):
         ifname = cfg.nk_guest_ifname
@@ -182,6 +190,8 @@ def ncdevmem_rx(cfg, port, verify=True, fail_on_linear=False, flow_steer=False,
 
     if chunk_size:
         extras.append(f"-z {chunk_size}")
+    if hugepages:
+        extras.append("-H")
 
     parts = [cfg.bin_local, f"-f {ifname}", f"-s {addr}",
              f"-p {port}", *extras]
@@ -269,6 +279,33 @@ def ncdevmem_rx(cfg, port, verify=True, fail_on_linear=False, flow_steer=False,
     ksft_eq(socat.stdout.strip(), "hello\nworld")
 
 
+def run_tx_large_niov(cfg):
+    """Run the devmem TX test from a 2MB hugepage-backed dmabuf.
+
+    The kernel sizes TX niovs from the dmabuf layout. Send a single buffer
+    larger than a hugepage so that frags end at and continue past the niov
+    boundary.
+    """
+    require_devmem(cfg, hugepages=True)
+    _reserve_hugepages()
+    configure_nic(cfg)
+    netns = getattr(cfg, "netns", None)
+    port = rand_port()
+    size = 3 * 1024 * 1024
+    pattern = "abcdefg"
+    expected = (pattern * (size // len(pattern) + 1))[:size]
+    tx_cmd = ncdevmem_tx(cfg, port, hugepages=True)
+    # Hash on the remote, bkg() only drains stdout once the command exits.
+    listen_cmd = f"{socat_listen(cfg, port)} | sha256sum"
+
+    with bkg(listen_cmd, host=cfg.remote, exit_wait=True) as socat:
+        wait_port_listen(port, host=cfg.remote)
+        cmd(f"bash -c 'yes {pattern} | tr -d \"\\n\" | head -c {size} | {tx_cmd}'",
+            ns=netns, shell=True)
+    ksft_eq(socat.stdout.split()[0],
+            hashlib.sha256(expected.encode()).hexdigest())
+
+
 def run_rx_large_niov(cfg):
     """Run the devmem RX test with a large niov (rx-page-size > PAGE_SIZE).
 
diff --git a/tools/testing/selftests/drivers/net/hw/ncdevmem.c b/tools/testing/selftests/drivers/net/hw/ncdevmem.c
index 918e3b51f3b8..e29ff27087e7 100644
--- a/tools/testing/selftests/drivers/net/hw/ncdevmem.c
+++ b/tools/testing/selftests/drivers/net/hw/ncdevmem.c
@@ -104,6 +104,7 @@ static uint32_t tx_dmabuf_id;
 static int waittime_ms = 500;
 static bool fail_on_linear;
 static uint32_t rx_page_size;
+static bool use_hugepages;
 
 /* System state loaded by current_config_load() */
 #define MAX_FLOWS	8
@@ -162,7 +163,7 @@ static struct memory_buffer *udmabuf_alloc(size_t size)
 	}
 
 	memfd_flags = MFD_ALLOW_SEALING;
-	if (rx_page_size > getpagesize())
+	if (use_hugepages || rx_page_size > getpagesize())
 		memfd_flags |= MFD_HUGETLB | MFD_HUGE_2MB;
 
 	ctx->memfd = memfd_create("udmabuf-test", memfd_flags);
@@ -1428,11 +1429,14 @@ int main(int argc, char *argv[])
 	int is_server = 0, opt;
 	int ret, err = 1;
 
-	while ((opt = getopt(argc, argv, "Lls:c:p:v:q:t:f:z:nb:")) != -1) {
+	while ((opt = getopt(argc, argv, "Lls:c:p:v:q:t:f:z:nb:H")) != -1) {
 		switch (opt) {
 		case 'L':
 			fail_on_linear = true;
 			break;
+		case 'H':
+			use_hugepages = true;
+			break;
 		case 'l':
 			is_server = 1;
 			break;
diff --git a/tools/testing/selftests/drivers/net/hw/nk_devmem.py b/tools/testing/selftests/drivers/net/hw/nk_devmem.py
index 61c6f31f01e5..799030db7c12 100755
--- a/tools/testing/selftests/drivers/net/hw/nk_devmem.py
+++ b/tools/testing/selftests/drivers/net/hw/nk_devmem.py
@@ -4,7 +4,7 @@
 
 import os
 from devmem_lib import (setup_test, run_rx, run_tx, run_tx_chunks, run_rx_hds,
-                        run_rx_large_niov)
+                        run_rx_large_niov, run_tx_large_niov)
 from lib.py import ksft_run, ksft_exit, ksft_disruptive
 from lib.py import NetDrvContEnv
 
@@ -37,6 +37,12 @@ from lib.py import NetDrvContEnv
     run_rx_large_niov(cfg)
 
 
+@ksft_disruptive
+def check_nk_tx_large_niov(cfg) -> None:
+    """Run the devmem TX large-niov test through netkit."""
+    run_tx_large_niov(cfg)
+
+
 def main() -> None:
     """Run the netkit devmem test cases."""
     with NetDrvContEnv(__file__, rxqueues=2, primary_rx_redirect=True) as cfg:
@@ -44,7 +50,8 @@ from lib.py import NetDrvContEnv
                    os.path.join(os.path.dirname(os.path.abspath(__file__)),
                                 "ncdevmem"))
         ksft_run([check_nk_rx, check_nk_tx, check_nk_tx_chunks,
-                  check_nk_rx_hds, check_nk_rx_large_niov], args=(cfg,))
+                  check_nk_rx_hds, check_nk_rx_large_niov,
+                  check_nk_tx_large_niov], args=(cfg,))
     ksft_exit()
 
 
-- 
2.53.0-Meta

Re: [PATCH net-next v4 0/5] net: devmem: size TX net_iovs from the dmabuf layout

[Paolo Abeni]

Hi,

A few (late) questions/comments to check I'm not missing relevant
points,

On 11/11/26 23:47, Stanislav Fomichev wrote:
> TX bindings use PAGE_SIZE net_iovs, so a frag covers at most a page.
> A 64KB skb needs 16 frags, and MAX_SKB_FRAGS caps devmem skbs at about
> 68KB, which makes BIG TCP ineffective for devmem TX.

... with default .config. You could set MAX_SKB_FRAGS to 45 and reach
~180K...
> This series picks the TX net_iov size from the dmabuf sg table instead:
> the largest power of 2, up to 2MB, which aligns every DMA segment. With
> a 2MB hugepage-backed udmabuf a BIG TCP skb takes a few frags. There's
> no UAPI change since TX addresses are dmabuf offsets.

... still this option should be more effective because the limit would be
higher and the skb would use less cachelines for the shared_info,

I'm wondering if making a comparison vs MAX_SKB_FRAGS==45 would be
complex and if you could do that, too.

> This doesn't make skbs larger. TCP still sizes them with size_goal.
> Only the frags get larger, and a single frag can now cover the whole
> skb.
> 
> An skb stays under 64KB unless BIG TCP is enabled. With BIG TCP,
> MSG_ZEROCOPY from huge pages can already create frags of 64KB or
> more. So drivers already have to handle them:
> - mlx5 has a 31-bit length per data segment
> - gve splits frags into 16KB descriptors
> - fbnic doesn't enable BIG TCP
> - bnxt has a 16-bit BD length. It is broken for such page frags today
>    and will be fixed separately.
> 
> Tested on two mlx5 400Gb/s hosts with kperf (udmabuf, 32KB RX
> net_iovs, 2MB hugepage-backed TX buffer), series backported to a 7.1
> kernel on the sender only. 1 flow, MTU 4200, receiver tcp_rmem capped
> at 16MB so it doesn't drop, median of 5 x 60s runs:
> 
>                                before    after
>    frags per skb (sender)       16-17      1-2
>    throughput (Gb/s)            208.6    211.2
>    sender sendmsg CPU (sys)     40.8%    19.5%   (-52%)
> 
> The receiver's CPU is the bottleneck, so the gain is CPU, not
> throughput. More of the transmit runs from the ACK path, so the
> sender's softirq grows (52% -> 59%), but sys + sirq still drops by 16%.

is the receiver using GRO H/W offload/big TCP? Which are the gro_max_size/
gso_max_size limits? In the `after` scenario, is the 2nd frag used for the
headers?

Thanks,

Paolo

Re: [PATCH net-next v4 1/5] bnxt_en: let the stack segment GSO skbs with too many frags

[netdev-bot+sashiko]

Thank you for your contribution! Sashiko AI review found more severe
issue(s) in other patches of this series. This mail is only an FYI, in case
you want to address the nit picks below when you respin.

Low priority / nit picks for your consideration:

- [Low] This commit fixes two real bnxt TX bugs but has no `Fixes:` tag
  and no `Cc: stable`.

--

bnxt_en: let the stack segment GSO skbs with too many frags

This commit clears the GSO features in bnxt_features_check() for skbs
with more than TX_MAX_FRAGS frags when MAX_SKB_FRAGS > TX_MAX_FRAGS. The
stack then segments those skbs instead of bnxt_start_xmit() linearizing
them. Non-GSO skbs are still linearized in the xmit path.

Link: https://lore.kernel.org/all/CANn89i+ps_CozfOyRnKav0_LUmSekJ9ExB5JkDbTAVf_ss_98g@mail.gmail.com/

> With MAX_SKB_FRAGS > TX_MAX_FRAGS, the xmit path linearizes skbs with
> more frags than a packet can have BDs. For GSO skbs, linearizing needs a
> large allocation, can produce a head which doesn't fit in a BD with BIG
> TCP, and always fails for unreadable skbs. Clear the GSO features of
> such skbs in ndo_features_check instead, so that the stack segments
> them. Non-GSO skbs are still linearized.
>
> Suggested-by: Eric Dumazet <edumazet@google.com>
> Link: https://lore.kernel.org/all/CANn89i+ps_CozfOyRnKav0_LUmSekJ9ExB5JkDbTAVf_ss_98g@mail.gmail.com/
> Signed-off-by: Stanislav Fomichev <sdf@fomichev.me>

[ ... ]

[Severity: Low]
Should this patch carry a Fixes: tag and Cc: stable?

Both failures described above come from the linearize fallback that
commit b91e82129400 ("bnxt_en: Linearize TX SKB if the fragments exceed
the max") added to bnxt_start_xmit():

drivers/net/ethernet/broadcom/bnxt/bnxt.c:bnxt_start_xmit() {
    ...
#if (MAX_SKB_FRAGS > TX_MAX_FRAGS)
	if (skb_shinfo(skb)->nr_frags > TX_MAX_FRAGS) {
		netdev_warn_once(dev, ...);
		if (skb_linearize(skb))
			goto tx_free;
	}
#endif
    ...
}

The driver calls netif_set_tso_max_size(dev, GSO_MAX_SIZE), so with BIG
TCP the linearized head can be larger than 65535 bytes. The BD length is
then built in bnxt_start_xmit() as:

	flags = (len << TX_BD_LEN_SHIFT) | TX_BD_TYPE_LONG_TX_BD |
		TX_BD_CNT(last_frag + 2);

TX_BD_LEN_SHIFT is 16, so the high bits of len are shifted out of the
u32. Wouldn't the descriptor then get a truncated length of len & 0xffff?

For devmem TX, the driver sets dev->netmem_tx, so unreadable skbs can
reach this path. skb_linearize() can't pull unreadable frags, so every
such GSO skb with more than TX_MAX_FRAGS (30) frags goes to tx_free and
is dropped.

A later patch in this series, "net: devmem: size TX net_iovs from the
dmabuf sg table", also calls the bnxt BD length limit a bug that needs
a fix. An earlier fix to the same block, commit 00eeab0c644a ("bnxt_en:
Write doorbell when linearizing skb fails"), carried
Fixes: b91e82129400 and Cc: stable@vger.kernel.org.

Without the tag, stable kernels built with CONFIG_MAX_SKB_FRAGS > 30
would not get this fix automatically. Would something like this work?

Fixes: b91e82129400 ("bnxt_en: Linearize TX SKB if the fragments exceed the max")
Cc: stable@vger.kernel.org

-- 
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20261111224740.2744522-1-sdf%40fomichev.me

Re: [PATCH net-next v4 2/5] net: devmem: charge the frag length to TX skb truesize

[netdev-bot+sashiko]

Thank you for your contribution! Sashiko AI review found more severe
issue(s) in other patches of this series. This mail is only an FYI, in case
you want to address the nit picks below when you respin.

Low priority / nit picks for your consideration:

- [Low] The commit's author identity doesn't match its Signed-off-by
  identity.

--

net: devmem: charge the frag length to TX skb truesize

The devmem TX path in zerocopy_fill_skb_from_devmem() used to charge
PAGE_SIZE (the net_iov size) to skb->truesize for each frag. This patch
charges the frag length instead. With TX net_iovs about to grow to 2MB,
charging the net_iov size would overcharge the send buffer, and the frag
length also matches what TCP uncharges.

> Author: Stanislav Fomichev <sdf.kernel@gmail.com>
> [ ... ]
> Signed-off-by: Stanislav Fomichev <sdf@fomichev.me>

[Severity: Low]
This isn't a bug, but the author address and the Signed-off-by address
are different. The author is sdf.kernel@gmail.com, and the only sign-off
from the author is sdf@fomichev.me.

checkpatch.pl reports this as FROM_SIGN_OFF_MISMATCH ("From:/Signed-off-by:
email address mismatch"). The other patches in the series have the same
mismatch, from "bnxt_en: let the stack segment GSO skbs with too many
frags" through "selftests: drv-net: devmem: TX from a hugepage-backed
dmabuf".

When resending, could you add an in-body "From: Stanislav Fomichev
<sdf@fomichev.me>" line, or a Signed-off-by for the gmail address?

-- 
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20261111224740.2744522-1-sdf%40fomichev.me

Re: [PATCH net-next v4 4/5] net: devmem: size TX net_iovs from the dmabuf sg table

[netdev-bot+sashiko]

Thank you for your contribution! Sashiko AI review found 1 potential
issue(s) to consider.

Critical: 0 · High: 0 · Medium: 1 · Low: 0

If you believe none of the issues are legitimate, please reply to each
issue of Medium severity or higher and explain why the AI is wrong.
If any of them are legitimate and you plan to respin, please let patchwork
know by including "pw-bot: cr" as a separate line at the end of your reply
(one such reply per series is enough).

- [Medium] This patch makes devmem TX net_iovs up to 2MB.

--

net: devmem: size TX net_iovs from the dmabuf sg table

This commit lets TX devmem bindings choose the net_iov size from the
dmabuf sg table. It uses the largest power of two between PAGE_SIZE and
2MB that aligns every DMA segment. netdev_nl_bind_tx_doit() now passes
niov_shift 0 to request this, so hugepage-backed udmabufs get 2MB TX
net_iovs.

> diff --git a/net/core/devmem.c b/net/core/devmem.c
> index c6868f39261ad..7135280488971 100644
> --- a/net/core/devmem.c
> +++ b/net/core/devmem.c

[ ... ]

> @@ -387,6 +400,17 @@ net_devmem_bind_dmabuf(struct net_device *dev, void *vdev,
>  		goto err_unmap;
>  	}
>  
> +	if (!niov_shift) {
> +		niov_size = SZ_2M;
> +		while (niov_size > PAGE_SIZE &&
> +		       (!IS_ALIGNED(dmabuf->size, niov_size) ||
> +			!net_devmem_sgt_aligned(binding->sgt, niov_size)))
> +			niov_size /= 2;
> +		niov_shift = ilog2(niov_size);
> +	}

[Severity: Medium]
netdev_nl_bind_tx_doit() now passes 0 here. That means a 2MB aligned,
hugepage-backed dmabuf gets 2MB TX net_iovs no matter which device it is
bound to. Can this produce frags that bnxt silently truncates?

Before this patch, every devmem TX frag was capped at PAGE_SIZE. Now
zerocopy_fill_skb_from_devmem() limits a frag only by niov_size - off,
the copy length and the iov length:

net/core/datagram.c:zerocopy_fill_skb_from_devmem() {
    ...
		size = min_t(size_t, size, length);
		size = min_t(size_t, size, iter_iov_len(from));

		netmem_inc_ref(net_iov_to_netmem(niov));
		skb_add_rx_frag_netmem(skb, i, net_iov_to_netmem(niov), off,
				       size, size);
    ...
}

So with BIG TCP enabled (gso_max_size > 65536), a single frag can be 64KB
or larger.

In bnxt_init_one(), bnxt sets dev->netmem_tx = NETMEM_TX_DMA and calls
netif_set_tso_max_size(dev, GSO_MAX_SIZE). bnxt_start_xmit() then writes
the frag length into the BD without checking its size:

drivers/net/ethernet/broadcom/bnxt/bnxt.c:bnxt_start_xmit() {
    ...
		len = skb_frag_size(frag);
		mapping = skb_frag_dma_map(&pdev->dev, frag, 0, len,
					   DMA_TO_DEVICE);
    ...
		flags = len << TX_BD_LEN_SHIFT;
		txbd->tx_bd_len_flags_type = cpu_to_le32(flags);
    ...
}

TX_BD_LEN is (0xffff << 16), so the length field is 16 bits wide. A 65536
byte frag would be written as 0, and a 70000 byte frag as 4464. The DMA
mapping and BQL still use the full length. Depending on the hardware, the
result could be a corrupted LSO payload on the wire or a TX error and
ring reset.

The "Dropped oversize" check in bnxt_start_xmit() only looks at gso_size
plus hdr_len for GSO skbs, so it does not catch this.

The commit message says:

    bnxt doesn't handle them yet. It has a 16-bit BD length and needs a
    separate fix.

That fix does not appear in this series, and it is not listed as a
prerequisite. The only bnxt change in the series is "bnxt_en: let the
stack segment GSO skbs with too many frags". It handles nr_frags >
TX_MAX_FRAGS in bnxt_features_check(), but it does not limit frag size.

NETDEV_CMD_BIND_TX does not require GENL_ADMIN_PERM. Once an admin enables
BIG TCP on a bnxt port, any local user who can create such a dmabuf (for
example a udmabuf from a hugetlb memfd) could reach this path.

MSG_ZEROCOPY from huge pages can already hit the same truncation. This
patch adds a new path to it through devmem TX, in the BIG TCP setup the
patch is meant to enable.

Should the auto-selected niov size be capped per device until bnxt is
fixed? Or could the bnxt fix land before this patch?

> +	binding->niov_shift = niov_shift;
> +	niov_size = 1UL << niov_shift;
> +
>  	if (!IS_ALIGNED(dmabuf->size, niov_size)) {
>  		err = -EINVAL;
>  		NL_SET_ERR_MSG_FMT(extack,

[ ... ]

-- 
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20261111224740.2744522-1-sdf%40fomichev.me

Re: [PATCH net-next v4 5/5] selftests: drv-net: devmem: TX from a hugepage-backed dmabuf

[netdev-bot+sashiko]

Thank you for your contribution! Sashiko AI review found 1 potential
issue(s) to consider.

Critical: 0 · High: 0 · Medium: 1 · Low: 0

If you believe none of the issues are legitimate, please reply to each
issue of Medium severity or higher and explain why the AI is wrong.
If any of them are legitimate and you plan to respin, please let patchwork
know by including "pw-bot: cr" as a separate line at the end of your reply
(one such reply per series is enough).

- [Medium] The new TX large-niov test (check_tx_large_niov in devmem.py
  and check_nk_tx_large_niov in nk_devmem.py, both calling…

--

selftests: drv-net: devmem: TX from a hugepage-backed dmabuf

This adds a -H option to ncdevmem that backs the udmabuf with 2MB
hugepages. It also adds a TX test, run from both devmem.py and
nk_devmem.py, that sends a 3MB patterned buffer from that udmabuf so the
frags cross the 2MB net_iov boundary. The received data is checked by
comparing sha256 hashes from the remote side.

> diff --git a/tools/testing/selftests/drivers/net/hw/devmem_lib.py b/tools/testing/selftests/drivers/net/hw/devmem_lib.py
> index 71b916c161969..8058fd1ddd501 100644
> --- a/tools/testing/selftests/drivers/net/hw/devmem_lib.py
> +++ b/tools/testing/selftests/drivers/net/hw/devmem_lib.py

[ ... ]

> @@ -63,10 +66,15 @@ def require_devmem(cfg, rx_page_size=RX_PAGE_SIZE_DEFAULT):
>          # ncdevmem's queue range and cause the probe to fail.
>          cfg.devmem_supported = {size: _probe_devmem(cfg, size)
>                                  for size in PROBE_RX_PAGE_SIZES}
> +        cfg.devmem_hugepages = _probe_devmem(cfg, RX_PAGE_SIZE_DEFAULT,
> +                                             hugepages=True)
>  
>      if not cfg.devmem_supported[RX_PAGE_SIZE_DEFAULT]:
>          raise KsftSkipEx("Test requires devmem support")
>  
> +    if hugepages and not cfg.devmem_hugepages:
> +        raise KsftSkipEx("Test requires a 2MB hugepage-backed dmabuf")
> +

[Severity: Medium]
Can this probe succeed anywhere other than a 4K page kernel whose default
hugepage size is 2MB?

Before the probe, the only reservation is _reserve_hugepages(). It writes
64 to the default hstate's sysctl:

NR_HUGEPAGES_FILE = "/proc/sys/vm/nr_hugepages"

def _reserve_hugepages(want=64):

That sysctl only grows the pool for the default hugepage size. On arm64
with 16K or 64K base pages, the default size is 32MB or 512MB. An x86
kernel booted with default_hugepagesz=1G has the same problem. In those
setups the 2MB pool never grows.

With -H, udmabuf_alloc() always requests MFD_HUGETLB | MFD_HUGE_2MB. The
probe allocates its buffer in run_devmem_tests():

	mem = provider->alloc(getpagesize() * NUM_PAGES);

NUM_PAGES is 16000 and the size is rounded up to 2MB. That needs 32
hugepages on 4K pages, 125 on 16K pages and 500 on 64K pages. So even
when 2MB is the default size, 64 pages are too few on non-4K kernels.

In both cases UDMABUF_CREATE fails and cfg.devmem_hugepages ends up False.
check_tx_large_niov() and check_nk_tx_large_niov() then always skip with
"Test requires a 2MB hugepage-backed dmabuf". That message does not say
that the hugepage pool is the cause.

Before this patch, hugetlb was only used when rx_page_size >
getpagesize(). So -H looks like the first user to hit this on non-4K
kernels.

Would it make sense to reserve pages through
/sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages instead? The count
could be derived from the size ncdevmem will allocate, or -H could use a
smaller buffer.

[ ... ]

-- 
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20261111224740.2744522-1-sdf%40fomichev.me

Re: [PATCH net-next v4 0/5] net: devmem: size TX net_iovs from the dmabuf layout

[Stanislav Fomichev]

On 11/12, Paolo Abeni wrote:
> Hi,
> 
> A few (late) questions/comments to check I'm not missing relevant
> points,
> 
> On 11/11/26 23:47, Stanislav Fomichev wrote:
> > TX bindings use PAGE_SIZE net_iovs, so a frag covers at most a page.
> > A 64KB skb needs 16 frags, and MAX_SKB_FRAGS caps devmem skbs at about
> > 68KB, which makes BIG TCP ineffective for devmem TX.
> 
> ... with default .config. You could set MAX_SKB_FRAGS to 45 and reach
> ~180K...
> > This series picks the TX net_iov size from the dmabuf sg table instead:
> > the largest power of 2, up to 2MB, which aligns every DMA segment. With
> > a 2MB hugepage-backed udmabuf a BIG TCP skb takes a few frags. There's
> > no UAPI change since TX addresses are dmabuf offsets.
> 
> ... still this option should be more effective because the limit would be
> higher and the skb would use less cachelines for the shared_info,
> 
> I'm wondering if making a comparison vs MAX_SKB_FRAGS==45 would be
> complex and if you could do that, too.

Originally I did not want to include bigtcp to keep it apples to apples:
same tso size, just 16 frags in skb vs 1 (in ideal case).

I just tried running mlx5 with 45 frags configured and it fails to start with
the following error:
mlx5e_attach_netdev() warns "MLX5E: Max SQ WQEBBs firmware capability: 16, needed 23" and returns -EIO.

(fwiw quick review with llm claims that mlx5 only supports up to 20 skb frags)

What I also tried just now is `gso_max_size 524280` (on top of this series)
and it ups single flow throughput from 200Gbps to 240Gbps (presumably
by making the receiver go faster with 512K TSOs vs 64K TSOs) and halves
the sender softirq..

I hope this gives you more data, not sure running with 20 frags is worth
it (vs 16)? LMK if you disagree or want me to run something else.

> > This doesn't make skbs larger. TCP still sizes them with size_goal.
> > Only the frags get larger, and a single frag can now cover the whole
> > skb.
> > 
> > An skb stays under 64KB unless BIG TCP is enabled. With BIG TCP,
> > MSG_ZEROCOPY from huge pages can already create frags of 64KB or
> > more. So drivers already have to handle them:
> > - mlx5 has a 31-bit length per data segment
> > - gve splits frags into 16KB descriptors
> > - fbnic doesn't enable BIG TCP
> > - bnxt has a 16-bit BD length. It is broken for such page frags today
> >    and will be fixed separately.
> > 
> > Tested on two mlx5 400Gb/s hosts with kperf (udmabuf, 32KB RX
> > net_iovs, 2MB hugepage-backed TX buffer), series backported to a 7.1
> > kernel on the sender only. 1 flow, MTU 4200, receiver tcp_rmem capped
> > at 16MB so it doesn't drop, median of 5 x 60s runs:
> > 
> >                                before    after
> >    frags per skb (sender)       16-17      1-2
> >    throughput (Gb/s)            208.6    211.2
> >    sender sendmsg CPU (sys)     40.8%    19.5%   (-52%)
> > 
> > The receiver's CPU is the bottleneck, so the gain is CPU, not
> > throughput. More of the transmit runs from the ACK path, so the
> > sender's softirq grows (52% -> 59%), but sys + sirq still drops by 16%.
> 
> is the receiver using GRO H/W offload/big TCP? Which are the gro_max_size/
> gso_max_size limits? In the `after` scenario, is the 2nd frag used for the
> headers?

hw gro, no big tcp (default tso of 64k). The headers are still in the
linear part and most of the time we get 1 frag.

I'm assuming 2 frags are happening sometimes because of tcp sending
something over that 2MB boundary. So that one single range has to be expressed
by to frags.