Re: [PATCH v1 00/15] io_uring zero copy rx

[Jens Axboe <axboe@kernel.dk>]

On 10/10/24 9:03 AM, David Ahern wrote:
> On 10/10/24 8:21 AM, Jens Axboe wrote:
>>> which adds zc send. I ran a quick test, and it does reduce cpu
>>> utilization on the sender from 100% to 95%. I'll keep poking...
>>
>> Update on this - did more testing and the 100 -> 95 was a bit of a
>> fluke, it's still maxed. So I added io_uring send and sendzc support to
>> kperf, and I still saw the sendzc being maxed out sending at 100G rates
>> with 100% cpu usage.
>>
>> Poked a bit, and the reason is that it's all memcpy() off
>> skb_orphan_frags_rx() -> skb_copy_ubufs(). At this point I asked Pavel
>> as that made no sense to me, and turns out the kernel thinks there's a
>> tap on the device. Maybe there is, haven't looked at that yet, but I
>> just killed the orphaning and tested again.
>>
>> This looks better, now I can get 100G line rate from a single thread
>> using io_uring sendzc using only 30% of the single cpu/thread (including
>> irq time). That is good news, as it unlocks being able to test > 100G as
>> the sender is no longer the bottleneck.
>>
>> Tap side still a mystery, but it unblocked testing. I'll figure that
>> part out separately.
>>
> 
> Thanks for the update. 30% cpu is more inline with my testing.
> 
> For the "tap" you need to make sure no packet socket applications are
> running -- e.g., lldpd is a typical open I have a seen in tests. Check
> /proc/net/packet

Here's what I see:

sk               RefCnt Type Proto  Iface R Rmem   User   Inode
0000000078c66cbc 3      3    0003   2     1 0      0      112645
00000000558db352 3      3    0003   2     1 0      0      109578
00000000486837f4 3      3    0003   4     1 0      0      109580
00000000f7c6edd6 3      3    0003   4     1 0      0      22563 
000000006ec0363c 3      3    0003   2     1 0      0      22565 
0000000095e63bff 3      3    0003   5     1 0      0      22567 

was just now poking at what could be causing this. This is a server box,
nothing really is running on it... The nic in question is ifindex 2.

-- 
Jens Axboe