Re: [RFC] Subpage allocations for IOMMU tables

[Jason Gunthorpe <jgg@ziepe.ca>]

On Mon, Oct 12, 2026 at 11:18:23AM +0200, Daniel Drake wrote:
> Hi Jason & co,
> 
> I have been experimenting for how we might support allocations smaller
> than PAGE_SIZE from iommu_alloc_pages_sz(). Are there any existing
> ideas or plans for how this could be done?
> 
> This would be particularly beneficial for ARM platforms where the
> tendency of using 16KB or 64KB system page sizes is growing, but the
> IOMMU is fixed at 4KB page size, leading to wasted memory.

Oh? People built SMMUs without the 16k option? That's sad news.  In
the systems I have the SMMUv3 supports 64k/16k so there is no waste..

> One possible/draft approach is below. When a small allocation is
> requested, a per-node per-size folio is created. A section of this is
> used to satisfy the allocation, and the folio is added to a pool so that
> it can be used to satisfy later allocations too.
>
> The main challenge is handling the iommu_pages_list used for freeing
> allocations. In the whole-page case, this is nicely handled by using the
> list_head in in struct ioptdesc. But now we may have up to 16
> allocations per page, where each one can be placed on a different list.
> 
> So, in this approach, I have made iommu_pages_list manage both a list
> of regular full-page ioptdesc allocations, and also a list of subpage
> allocations that live within these subpage folios. The subpage
> list heads are stored in a dynamically allocated struct
> iommu_subpage_meta.
> 
> I don't love the approach but it at least seems to be working and I
> figured it would be good to get some opinions before I go further.

Yeah, you've hit on the main issues. Throw in the coming RCU HEAD
issue too and you get more problems. The iommu_pages_list abstraction
was intended to give some flexbility to solve this.

The idea Matthew was aiming toward was to allocate a dedicated struct
for each folio that could have the right sized stuff. You can have an
array of list heads, for example. Then you are not limited to
overlaying on struct folio.

He once even talked about making this into some kind of generic sub
page allocator with metadata, which seems quite interesting.

In this mode the core mm would provide the pooling and the subsystem
could request XX bytes memory + YY bytes metadata with some way to
convert from a memory pointer to the metadata pointer and back again.

There is also the exciting stuff power does with it's sub page
allocator, not sure that is a great example though.

Another option might be to involve generic_pt somehow and have it
populate the tree with the full folio. At least this way the memory is
not "lost" but it may not be used optimally. I doubt many people need
a 16x level 3 page for example

Another thought might be to use (pre) allocated memory to hold the
free list during unmap and then flush if we run out of memory. This
eliminates the linked list which seems helpful. Not sure what kind of
performance we'd get though.

> Other options:
>  - Replace the list_head with a pointer to the virt address of the next
>    page in the list. We can detect whether we're working with a full
>    page or subpage based on virt address, which means we only need one
>    pointer in iommu_pages_list.

So long as a linked list is used you need 4x the pointers, which seems
to be the main problem.

>  - Bump allocator with per-CPU pools of partial subpage folios, along
>    the lines of the networking subsystem's page_frag_cache. A little
>    simpler than the approach here, but we'd have to evict and abandon
>    filling the folio if the next request is for a different NUMA node,
>    and handle CPU hotplug.

Free side can't gurantee it will only free the same per-CPU pages, so
you are still back to needing 4x pointers.

>  - Allocate the subpage-tracking data in the unmap path? Falling back to
>    immediate flush and free if allocation fails, along the lines of
>    tlb_remove_table(). Would need more thought.
>  - Implement generation-based deferred freeing in iommu core,
>    eliminating the need for iommu_pages_list?

What would this look like?

There is also the 32 bit ARM option which avoided using a free list
so it doesn't need folio memory. This allows it to use a normal
kmemcache.

free list is fundamental to the design of generic_pt, but if more
people than just arm32 need this it would be reasonable to implement a
generic_pt_nolist variation. Driver can select which version it
wants. Maybe this is implemented as some combination of immediate
flushing and a small free array allocation from a cache?

Jason