Re: [PATCH net-next v4 0/5] net: devmem: size TX net_iovs from the dmabuf layout

[Stanislav Fomichev <sdf.kernel@gmail.com>]

On 11/12, Paolo Abeni wrote:
> Hi,
> 
> A few (late) questions/comments to check I'm not missing relevant
> points,
> 
> On 11/11/26 23:47, Stanislav Fomichev wrote:
> > TX bindings use PAGE_SIZE net_iovs, so a frag covers at most a page.
> > A 64KB skb needs 16 frags, and MAX_SKB_FRAGS caps devmem skbs at about
> > 68KB, which makes BIG TCP ineffective for devmem TX.
> 
> ... with default .config. You could set MAX_SKB_FRAGS to 45 and reach
> ~180K...
> > This series picks the TX net_iov size from the dmabuf sg table instead:
> > the largest power of 2, up to 2MB, which aligns every DMA segment. With
> > a 2MB hugepage-backed udmabuf a BIG TCP skb takes a few frags. There's
> > no UAPI change since TX addresses are dmabuf offsets.
> 
> ... still this option should be more effective because the limit would be
> higher and the skb would use less cachelines for the shared_info,
> 
> I'm wondering if making a comparison vs MAX_SKB_FRAGS==45 would be
> complex and if you could do that, too.

Originally I did not want to include bigtcp to keep it apples to apples:
same tso size, just 16 frags in skb vs 1 (in ideal case).

I just tried running mlx5 with 45 frags configured and it fails to start with
the following error:
mlx5e_attach_netdev() warns "MLX5E: Max SQ WQEBBs firmware capability: 16, needed 23" and returns -EIO.

(fwiw quick review with llm claims that mlx5 only supports up to 20 skb frags)

What I also tried just now is `gso_max_size 524280` (on top of this series)
and it ups single flow throughput from 200Gbps to 240Gbps (presumably
by making the receiver go faster with 512K TSOs vs 64K TSOs) and halves
the sender softirq..

I hope this gives you more data, not sure running with 20 frags is worth
it (vs 16)? LMK if you disagree or want me to run something else.

> > This doesn't make skbs larger. TCP still sizes them with size_goal.
> > Only the frags get larger, and a single frag can now cover the whole
> > skb.
> > 
> > An skb stays under 64KB unless BIG TCP is enabled. With BIG TCP,
> > MSG_ZEROCOPY from huge pages can already create frags of 64KB or
> > more. So drivers already have to handle them:
> > - mlx5 has a 31-bit length per data segment
> > - gve splits frags into 16KB descriptors
> > - fbnic doesn't enable BIG TCP
> > - bnxt has a 16-bit BD length. It is broken for such page frags today
> >    and will be fixed separately.
> > 
> > Tested on two mlx5 400Gb/s hosts with kperf (udmabuf, 32KB RX
> > net_iovs, 2MB hugepage-backed TX buffer), series backported to a 7.1
> > kernel on the sender only. 1 flow, MTU 4200, receiver tcp_rmem capped
> > at 16MB so it doesn't drop, median of 5 x 60s runs:
> > 
> >                                before    after
> >    frags per skb (sender)       16-17      1-2
> >    throughput (Gb/s)            208.6    211.2
> >    sender sendmsg CPU (sys)     40.8%    19.5%   (-52%)
> > 
> > The receiver's CPU is the bottleneck, so the gain is CPU, not
> > throughput. More of the transmit runs from the ACK path, so the
> > sender's softirq grows (52% -> 59%), but sys + sirq still drops by 16%.
> 
> is the receiver using GRO H/W offload/big TCP? Which are the gro_max_size/
> gso_max_size limits? In the `after` scenario, is the 2nd frag used for the
> headers?

hw gro, no big tcp (default tso of 64k). The headers are still in the
linear part and most of the time we get 1 frag.

I'm assuming 2 frags are happening sometimes because of tcp sending
something over that 2MB boundary. So that one single range has to be expressed
by to frags.