Re: [PATCH net-next v4 0/5] net: devmem: size TX net_iovs from the dmabuf layout

[Stanislav Fomichev <sdf.kernel@gmail.com>]

On 11/12, Mina Almasry wrote:
> On Thu, Nov 12, 2026 at 4:42 PM Stanislav Fomichev <sdf.kernel@gmail.com> wrote:
> >
> > On 11/12, Stanislav Fomichev wrote:
> > > On 11/12, Paolo Abeni wrote:
> > > > Hi,
> > > >
> > > > A few (late) questions/comments to check I'm not missing relevant
> > > > points,
> > > >
> > > > On 11/11/26 23:47, Stanislav Fomichev wrote:
> > > > > TX bindings use PAGE_SIZE net_iovs, so a frag covers at most a page.
> > > > > A 64KB skb needs 16 frags, and MAX_SKB_FRAGS caps devmem skbs at about
> > > > > 68KB, which makes BIG TCP ineffective for devmem TX.
> > > >
> > > > ... with default .config. You could set MAX_SKB_FRAGS to 45 and reach
> > > > ~180K...
> > > > > This series picks the TX net_iov size from the dmabuf sg table instead:
> > > > > the largest power of 2, up to 2MB, which aligns every DMA segment. With
> > > > > a 2MB hugepage-backed udmabuf a BIG TCP skb takes a few frags. There's
> > > > > no UAPI change since TX addresses are dmabuf offsets.
> > > >
> > > > ... still this option should be more effective because the limit would be
> > > > higher and the skb would use less cachelines for the shared_info,
> > > >
> > > > I'm wondering if making a comparison vs MAX_SKB_FRAGS==45 would be
> > > > complex and if you could do that, too.
> > >
> > > Originally I did not want to include bigtcp to keep it apples to apples:
> > > same tso size, just 16 frags in skb vs 1 (in ideal case).
> > >
> > > I just tried running mlx5 with 45 frags configured and it fails to start with
> > > the following error:
> > > mlx5e_attach_netdev() warns "MLX5E: Max SQ WQEBBs firmware capability: 16, needed 23" and returns -EIO.
> > >
> > > (fwiw quick review with llm claims that mlx5 only supports up to 20 skb frags)
> > >
> > > What I also tried just now is `gso_max_size 524280` (on top of this series)
> > > and it ups single flow throughput from 200Gbps to 240Gbps (presumably
> > > by making the receiver go faster with 512K TSOs vs 64K TSOs) and halves
> > > the sender softirq..
> > >
> > > I hope this gives you more data, not sure running with 20 frags is worth
> > > it (vs 16)? LMK if you disagree or want me to run something else.
> >
> > Fun fact, also setting gro_max_size on the receiver gets line rate over
> > single flow:
> >
> > I   client.c: Connected 1:cpu 39 | 1:cpu 118
> > client: == Source 2401:db00:255a:4a:bace::b
> > client:   Tx389.448 Gbps (2921248849920 bytes in 60008026 usec)
> > client:   Rx  0.000 Gbps (0 bytes in 60008026 usec)
> > client:   TCP retrans reord rtt rttvar d_ce snd_wnd cwnd
> > client:             2   125  86      2    0 15300608 10142
> > client:   net CPU 39: usr: 0.03% sys: 0.06% idle:67.39% iow: 0.00% irq: 5.69% sirq:26.81%
> > client:   app CPU 40: usr: 0.05% sys: 2.91% idle:95.67% iow: 0.00% irq: 0.21% sirq: 1.14%
> > client: == Target 2401:db00:255a:44:bace::b
> > client:   Tx  0.000 Gbps (0 bytes in 60009306 usec)
> > client:   Rx389.438 Gbps (2921238435008 bytes in 60009306 usec)
> > client:   TCP retrans reord rtt rttvar d_ce snd_wnd cwnd
> > client:             0     0 210    122    0  999424   10
> > client:   net CPU118: usr: 0.00% sys: 0.00% idle: 1.60% iow: 0.00% irq: 0.33% sirq:98.05%
> > client:   app CPU119: usr: 2.75% sys:77.46% idle:18.95% iow: 0.00% irq: 0.47% sirq: 0.34%
> >
> > 4200MTU, 32K rx-buf-len, this patch series, gso/gro_mas_size of 524280,
> > still the default number of frags (enough on rx to fill 512K GRO with 32K
> > chunks).
> 
> With 'devmem' over udmabuf? Is this possible without devmem on your
> setup? How much is devmem helping I wonder.

Yes, devmem+udmabuf. I think what's helping here the most is 32K rx frags
and bigtcp, this makes it possible to pack 512K TSOs into a single skb
on rx (which is the bottleneck). At least on mlx5 don't think there
is a way currently to configure that outside of iou/devmem memory
providers? FWIW, devmem socket api with the cmsgs is not supper
efficient, but bigger chunks do help here as well.