Re: [PATCH] swiotlb: avoid double copy with swiotlb on tx socket

[Dragos Tatulea <dtatulea@nvidia.com>]

On 16.06.26 13:06, Mostafa Saleh wrote:
> On Tue, Jun 16, 2026 at 02:33:52AM +0200, Luigi Rizzo wrote:
>> On Tue, Jun 16, 2026 at 2:25 AM Jakub Kicinski <kuba@kernel.org> wrote:
>>>
>>> On Mon, 15 Jun 2026 23:42:20 +0000 Luigi Rizzo wrote:
>>>> The use of swiotlb causes an extra data copy on I/O.  For tx sockets,
>>>> especially with greedy senders, this has a high chance of happening in
>>>> the softirq handler for tx network interrupts, creating a significant
>>>> performance bottleneck.
>>>
>>> What's the use case? I associate swiotlb with debug / testing mostly,
>>> so it'd be useful for people like me to explain why you care.
>>
>> Ah sorry, I forgot to mention.
>> swiotlb is used in guest kernels for confidential computing VMs.
>> Ordinary memory pages are encrypted and the host or devices
>> have no way to decrypt them, so the kernel must use
>> unencrypted bounce buffers to exchange data with I/O devices.
> 
> I started looking into the same problem recently, to reduce the
> bouncing in protected KVM (pKVM) confidential guests.
> My first attempt was to update dma_direct_map_phys() to skip
> bouncing and do inline memory decryption (for pKVM that is a hypercall
> which updates the stage-2 page tables), however, that was really slow
> compared to the memcpy in bouncing even for massive pages.
> My conclusion was similar that we need to solve this at construction
> by making this memory allocated from a pre-decrypted pool (which
> does not have to be part of the SWIOTLB)
> My initial idea was to teach some of the kernel subsystems (SKB,
> BLK, SLAB) about "CoCo allocators" that allocate decrypted memory,
> as this is not a net specific problem.
> 
An example of this is Jiri's system_cc_shared heap which is a dma-buf
heap with decrypted memory for userspace.

> I am still looking into this, I was planning to bring this up in the
> upcoming LPC.
> I will give this patch a try. However, I believe that we need a more
> generalised concept for CoCo pre-decrypted allocators in the kernel.
> 
There is a talk at LPC in the networking track about this [2]. This is
exactly the type of discussion that I was hoping to have there.

Besides the issues mentioned in this thread we've also found that a lot
of overhead can come only from swiotlb allocations when running many queues.

I will add information about this series in my talk. Hopefully I will also
have time to add some numbers for comparison.

Sorry for the late reply but I spotted this thread only now by
accident.

[1] https://lore.kernel.org/all/20260325192352.437608-3-jiri@resnulli.us/
[2] https://lpc.events/event/20/contributions/2464

Thanks,
Dragos