Re: [PATCH v1 00/15] io_uring zero copy rx

Jens Axboe
2024-10-10T15:15:34Z

On 10/10/24 9:03 AM, David Ahern wrote:
> On 10/10/24 8:21 AM, Jens Axboe wrote:
>>> which adds zc send. I ran a quick test, and it does reduce cpu
>>> utilization on the sender from 100% to 95%. I'll keep poking...
>>
>> Update on this - did more testing and the 100 -> 95 was a bit of a
>> fluke, it's still maxed. So I added io_uring send and sendzc support to
>> kperf, and I still saw the sendzc being maxed out sending at 100G rates
>> with 100% cpu usage.
>>
>> Poked a bit, and the reason is that it's all memcpy() off
>> skb_orphan_frags_rx() -> skb_copy_ubufs(). At this point I asked Pavel
>> as that made no sense to me, and turns out the kernel thinks there's a
>> tap on the device. Maybe there is, haven't looked at that yet, but I
>> just killed the orphaning and tested again.
>>
>> This looks better, now I can get 100G line rate from a single thread
>> using io_uring sendzc using only 30% of the single cpu/thread (including
>> irq time). That is good news, as it unlocks being able to test > 100G as
>> the sender is no longer the bottleneck.
>>
>> Tap side still a mystery, but it unblocked testing. I'll figure that
>> part out separately.
>>
>
> Thanks for the update. 30% cpu is more inline with my testing.
>
> For the "tap" you need to make sure no packet socket applications are
> running -- e.g., lldpd is a typical open I have a seen in tests. Check
> /proc/net/packet

Here's what I see:

sk RefCnt Type Proto Iface R Rmem User Inode
0000000078c66cbc 3 3 0003 2 1 0 0 112645
00000000558db352 3 3 0003 2 1 0 0 109578
00000000486837f4 3 3 0003 4 1 0 0 109580
00000000f7c6edd6 3 3 0003 4 1 0 0 22563
000000006ec0363c 3 3 0003 2 1 0 0 22565
0000000095e63bff 3 3 0003 5 1 0 0 22567

was just now poking at what could be causing this. This is a server box,
nothing really is running on it... The nic in question is ifindex 2.

--
Jens Axboe