Re: [PATCH net-next 00/10] tcp: support non-GSO jumbograms

[Mariusz Klimek <maklimek97@gmail.com>]

On 6/10/26 22:10, Paolo Abeni wrote:

> On 6/10/26 4:16 PM, Mariusz Klimek wrote:
>> On 6/10/26 00:27, Jakub Kicinski wrote:
>>> On Tue, 9 Jun 2026 19:01:02 +0200 Mariusz Klimek wrote:
>>>> I understand your concerns and they're reasonable. I would still like to
>>>> proceed with this series, though. What can I do for you to consider this
>>>> series? Would it help to submit a few less-significant patches first before
>>>> resubmitting this patch series? Should I resubmit it as an RFC?
>>> AI tools changed how much we value code, but they did not change much
>>> about how one joins a community.
>> OK, I'll start with smaller patches to establish myself more
>> in the community. I am still interested in eventually getting this
>> series through as it can speed up workloads relevant to our company.
>> Do you think this patch could potentially be considered down the line?
> If you are strongly motivated to do progresses in this area, one thing
> you could look at is the root cause of the performance difference
> between the veth jumbogram and the same size veth big tcp test: I'm
> quite surprised some delta is visible there.
>
> Possibly perf can outline some bottle-neck in the big tcp path that
> could be ironed-out.
>
> Also it could be interesting if some devices could do H/W GRO based big TCP.
>
> /P
>
Hi Paolo, it's been a long time.

After some research I found out why jumbograms outperform BIG TCP in the 
iperf3
over veth test: it's because Nagle splits BIG TCP packets but not 
jumbograms.

BIG TCP assumes packets will eventually be sent over the network, and so 
applies
the Nagle test over the real MSS of 1500 bytes. If a send occurs before the
last ACK is received, Nagle splits the BIG TCP packet into two packets: one
whose size is a multiple of the MSS, and one that contains the remainder
(tcp_mss_split_point and tso_fragment). The remainder packet is queued 
and is
only sent once the ACK is received.

In the jumbogram case, the jumbogram's size is already smaller than or equal
to the MSS size, so Nagle doesn't split the packet. The ~5% performance gain
comes from avoiding the packet being split in two and having one packet
traverse the network stack instead of two.

A simple way to prove the above is disabling Nagle with the 
`TCP_NODELAY` flag
and seeing that the performance gap resolves. I also ran iperf3 on a kernel
with the `limit >= max_segs * tp->mss_cache`, `limit >= chunk`, and
`limit > tcp_max_tso_deferred_mss(tp) tp->mss_cache` checks removed from
`tcp_tso_should_defer`, and it also resolved the performance gap. Perhaps
tcp_tso_should_defer should be fixed to so that these BIG TCP packets aren't
split?

While debugging this issue I tried restricting the iperf3 client and server
processes to one CPU. That reduced the number of packets that were split to
around 10% (from 60%), but surprisingly the throughput also increased
significantly and was much larger than the other tests. It appears that
restricting the iperf3 client and server to the same CPU fixes something 
else
and provides a 12% improvement over the base case and a ~7% improvement over
jumbograms. I can't explain this improvement yet, but the reduction in 
packet
splitting is likely due to less racy execution that makes the Nagle case 
happen
less frequently.

Measurements (output of a python wrapper around iperf3):

[ROUNDS=84 MTU=1500 GSO_SIZE=524280]
IPERF3 BITRATE: 241.35 [95.0% CI 239.81 - 242.91] ±7.17 Gbits/sec

SERVER RX PACKETS SZ=862:    406030.32 [95.0% CI 374158.02 - 437902.63] 
±144141.51
SERVER RX PACKETS SZ=511310: 401084.13 [95.0% CI 368103.20 - 434065.07] 
±150101.57
SERVER RX PACKETS SZ=512086: 197816.40 [95.0% CI 162980.59 - 232652.22] 
±160523.86

CLIENT RX PACKETS SZ=86:     647034.43 [95.0% CI 636767.16 - 657301.70] 
±47311.70

[ROUNDS=284 MTU=524280 GSO_OFF]
IPERF3 BITRATE: 252.56 [95.0% CI 251.67 - 253.44] ±7.55 Gbits/sec

SERVER RX PACKETS SZ=512094: 616549.29 [95.0% CI 614399.99 - 618698.58] 
±18401.15

CLIENT RX PACKETS SZ=86:     616754.45 [95.0% CI 614599.64 - 618909.26] 
±18448.43

[ROUNDS=84 MTU=1500 GSO_SIZE=524280 NODELAY]
IPERF3 BITRATE: 252.96 [95.0% CI 251.36 - 254.55] ±7.36 Gbits/sec

SERVER RX PACKETS SZ=512086: 617684.17 [95.0% CI 613782.72 - 621585.61] 
±17977.91

CLIENT RX PACKETS SZ=86:     617721.74 [95.0% CI 613817.65 - 621625.83] 
±17990.11

[ROUNDS=84 MTU=524280 GSO_OFF NODELAY]
IPERF3 BITRATE: 252.21 [95.0% CI 250.54 - 253.88] ±7.69 Gbits/sec

SERVER RX PACKETS SZ=512094: 615726.77 [95.0% CI 611620.41 - 619833.14] 
±18922.17

CLIENT RX PACKETS SZ=86:     615912.33 [95.0% CI 611836.51 - 619988.15] 
±18781.42

[ROUNDS=125 MTU=1500 GSO_SIZE=524280 NCPUS=1]
IPERF3 BITRATE: 272.80 [95.0% CI 272.37 - 273.23] ±2.42 Gbits/sec

SERVER RX PACKETS SZ=511310: 6074.89 [95.0% CI 5988.63 - 6161.14] ±487.24
SERVER RX PACKETS SZ=512086: 652866.16 [95.0% CI 651750.82 - 653981.50] 
±6300.23
SERVER RX PACKETS SZ=512738: 6454.82 [95.0% CI 6355.94 - 6553.71] ±558.58

CLIENT RX PACKETS SZ=86:     655050.50 [95.0% CI 653940.67 - 656160.32] 
±6269.06

[ROUNDS=64 MTU=1500 GSO_SIZE=524280 DELAY_PACKETS_UNTIL_ACK]
IPERF3 BITRATE: 251.64 [95.0% CI 250.45 - 252.83] ±4.78 Gbits/sec

SERVER RX PACKETS SZ=512086: 614318.41 [95.0% CI 611402.89 - 617233.93] 
±11671.76

CLIENT RX PACKETS SZ=86:     614633.78 [95.0% CI 611697.43 - 617570.13] 
±11755.14

-- 
Mariusz K.