Re: [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools

[Dragos Tatulea <dtatulea@nvidia.com>]

Hi Luigi,

On 03.10.26 23:22, Luigi Rizzo wrote:
> Here is a subsystem called DMA_PMD on which I would like feedback on
> architecture, possible enhancements, or kernel components that could be
> reused to avoid duplication.
> 
> I think it can be extremely useful for those affected by the HW/SW overhead
> of IOMMU (especially IOTLB thrashing, see [1]), or confidential computing,
> or preemptable VMs.
> 
> The (not too exaggerated) pitch line is
> 
> DMA_PMD has the performance of identity, guarantees that only IO buffers
> can ever have active IOMMU or IOTLB mappings, removes the bounce buffer
> overhead in confidential computing and preemptable VMs, and integrates
> smoothly with existing kernel APIs.
> 
> === ARCHITECTURE
> 
> DMA_PMD was initially designed to address IOTLB thrashing (details in [1]),
> but turned out to also resolve nicely the strict IOMMU overhead, and avoid
> the bounce buffer overhead in preemptible VMs and Confidential Computing.
> 
> It works by combining several known techniques:
> 
> - transparently feed allocators of device memory (skb_page_frag_refill(),
>   dma_alloc_attrs(), pagepool, ...) with DMA_PMD pages, i.e. physically
>   contiguous PMD_SIZE (2MB) pages mapped via PDE_SIZE IOMMU entries
> 
> - heavy recycling of DMA_PMD pages (like pagepool) and lazy IOMMU unmapping
>   (like DMA-FQ), BUT:
> 
> - like strict IOMMU (DMA), safely release memory back to the kernel only
>   after destroying all of its IOMMU mappings and a synchronous IOTLB flush
> 
> - on allocation, DMA_PMD pages can be configured to be pinned in the host
>   (hence suitable for preemptible VMs) and/or unencrypted (hence suitable
>   for Confidential Computing), removing the need for bounce buffers
> 
> - there are global and per-device optin /sys/device/.../dma_pmd_*
>   and /proc/sys/net/core/tx_enable_dma_pmd
> 
> NIC drivers can typically use dma_pmd with no modifications for rings,
> tx buffers, rx buffers (if they use pagepool), and rx headers. For tx
> headers, almost all drivers need changes (see later patches in the series)
> to implement cheap bounce buffers and avoid individual 4K mappings.
> 
> The series has the following main components:
> 
> - a sparse array (similar to pageblock_flags) to quickly attach metadata
>   to a 2MB page without fiddling with the struct page. Cost is 128B per 2MB
>   page used as an IO buffer, totally negligible.
> 
> - dma_pmd_pool, a replacement for alloc_pages() that can be instantiated
>   per-CPU or per receive queue. It handles the split of PMD_SIZE pages
>   into order-N blocks, handles dma_map and unmap, and aggressively recycles
>   entries. It is used to feed skb_page_frag_refill(), pagepool, and
>   receive buffers for drivers that do not use pagepool.
>   See [2] for "WHY NOT PAGEPOOL FOR TX AND EVERYTHING"
> 
> - dma_pmd_arena, is another allocator backed by DMA_PMD pages and
>   is used exclusively as the backend for dma_alloc_attrs().
>   Used for longer-lived allocations (descriptor/completion rings,
>   rx and tx header buffers)
> 
> - glue code to hook DMA_PMD into pagepool [2], skb_page_frag_refill(),
>   dma_alloc_attrs(), dma_map/unmap...
> 
> - per-driver patches, where necessary (e.g. tx header buffers [3])
> 
> === PERFORMANCE BENEFITS
> 
> Your mileage may vary. Enabling the IOMMU may have no throughput
> impact, until it does when some system components (bus, IOTLB, CPU)
> become overloaded.  Aside from throughput reduction, one interesting
> parameter is the effectiveness of the IOTLB. Here is a sample of SMMU
> performance counters for a large ARM system with 2x200G NICs doing
> bidirectional traffic:
> 
>     === DMA-FQ MODE (total throughput ~440Gbps)
>     25,757,488      smmuv3_pmcg_*/event=0x80/   IOTLB lookups
>     16,634,941      smmuv3_pmcg_*/event=0x81/   IOTLB  misses
> 
>     === DMA_PMD on top of strict DMA (total throughput ~745Gbps)
>     22,693,218      smmuv3_pmcg_*/event=0x80/   IOTLB lookups
>         30,536      smmuv3_pmcg_*/event=0x81/   IOTLB  misses
>     (not a mistake, also lookups went down despite the higher rate
>     because the tx side can use larger segments)
> 
> The 2MB mappings made IOTLB misses almost non existent, because
> the working set is reduced by a factor of ~512.
>        
> Note, the code has more verbose comments than I would like.
> Several of them are there to avoid Sashiko getting confused and
> flagging false positives.
> 
> === NOTES
> 
> [1] IOMMU IMPACT AND IOTLB THRASHING
> This is documented in more detail in Documentation/core-api/dma-pmd.rst
> but the compact version is below.
> 
> There are three main costs involved with using the IOMMU:
> - CPU cost for dma map/unmap
> - CPU cost and latency for synchronous IOTLB flush (required for strong security)
> - IOTLB thrashing, shows up dramatically when the IO access pattern
>   exceeds the IOTLB size, and IOMMU page walks slow down bus activity
>   up to a point where we see over 30..60% throughput reduction just for
>   this reason.  Some relevant references
> 
>   https://lore.kernel.org/all/4b42f2eb-dc29-153e-ace9-5584ea2e5070@redhat.com/
>   https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10280580/
> 
> Specifically, NICs access 4 or more pages per packet (descriptor queue,
> completion queue, rx or tx header, one or more buffers), with very little
> locality especially for data buffers and tx headers. Each queue has 1-4K
> entries, a fast NIC uses 16 or more queues per direction, so just the
> data buffers cover 32..128K pages. Using 4KB mappings makes the IOTLB
> ineffective, as shown by the performance counters shown earlier.
> 
> [2] WHY NOT PAGEPOOL FOR TX AND EVERYTHING
> There is some overlap between pagepool and dma_pmd_pool in that both
> implement a cache, but pagepool is missing some important features:
> 
> - pagepool does not handle splitting 2M pages in smaller chunks, or
>   the mappings. That would need to be implemented (and is what dma_pmd_pool does)
> 
> - pagepool's provided pages cannot be cleanly managed by
>   get_page()/put_page(), which is what all the consumers of
>   skb_page_frag_refill(). Fixing that would require dozen of changes
>   with high risk of missing some paths.
> 
> - even for the receive path, a provider for pagepool has more restrictions
>   than alloc_pages-supplied memory, so if I wanted to wrap dma_pmd_pool
>   as a provider I would have needed changes to the provider interface.
> 
> - pagepool requires a device at allocation time, which is not known when
>   skb_page_frag_refill() runs, hence the IOMMU mappings cannot be handled
>   at allocation time.
> 
> - page_pool_alloc_pages assumes a single NAPI consumer with BH disabled,
>   whereas skb_page_frag_refill() runs from a preemptive context.
> 
> - and last not least, we need to handle the dma_alloc_attrs() allocations
>   which don't match pagepool API
> 
> Of course all of the above could be modified, but the complexity
> would be similar to that of implementing dma_pmd_pool, and with a
> huge risk of breaking existing functionality or missing some path.
> 
> [3] DRIVER CHANGES
> 
> Most drivers need no change for rings, transmit buffers, or receive buffers
> (if they use pagepool). Tx headers are generally mapped on the fly and
> that is enough to trigger IOTLB thrashing, so most drivers need a small
> change to implement very cheap tx bounce buffers backed by DMA_PMD pages.
> 
> We could avoid the bounce buffers using DMA_PMD for the skb->head region,
> but that memory is contiguous to skb_shinfo so it would be exposed to
> IO device access, which may be undesirable for security.
> 

Thanks for the RFC. Here are my notes (some points rephrase what was already
said):

- On CoCo VMs it seems that the only way to opt in the IOMMU optimization is to
  also opt in the copy avoidance which is not always desirable.

- Having the pool generic opens up more visibility from the untrusted host
  side. Having device based pools at least isolates one device's memory from
  the rest of the system.

- Also, device driver could help with provisioning just the right amount of
  memory that is needed. And per device control also means that probably
  you don't need to map everything RW to the IOMMU.

Thanks,
Dragos