Re: [PATCH 1/3] dma-buf: system_heap: add a 2MB allocation order for IOMMU superpages

["Christian König" <christian.koenig@amd.com>]

On 9/15/26 19:26, T.J. Mercier wrote:
> On Tue, Sep 15, 2026 at 9:26 AM T.J. Mercier <tjmercier@google.com> wrote:
>>
>> On Tue, Sep 15, 2026 at 8:44 AM Christian König
>> <christian.koenig@amd.com> wrote:
...>> Even on arm64 phones free 2M pages aren't likely to be available for
>> very long after boot, but lately we have been doing more proactive
>> reclaim triggered by userspace before launching workflows that desire
>> large dma-buf allocations (and also at other times during application
>> lifecycle in general). That somewhat increases the likelihood that
>> high order pages will be available. I agree there's no guarantee we'll
>> get any, but without __GFP_RECLAIM the attempt is pretty cheap and the
>> payoff can be pretty benficial which Davidlohr's allocation and
>> mapping measurements demonstrate.
> 
> Oh I forgot to add that this behavior is similar to how slab
> allocations are done for slabs requiring more than an order 0 page per
> slab. A large optimal order is attempted first with ~__GFP_RECLAIM,
> but if that fails a fallback to a smaller min order is done. When the
> optimal order allocation attempt fails, the cost is just taking the
> zone lock for a quick peek at the free lists.
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/mm/slub.c?h=v7.2#n3373

Well from a lot of experience with AMDs ROCm stack and Intels XE driver I think that this is a really bad idea.

The background is that there are a lot of operation which can fragment memory to a point where only 4k pages are available, e.g. for example storage I/O can do that trivially (the kernel compile is a really good test case for that).

Now acceleration HW usually has a very specific sweet spot where it works best. AMDs GPUs for example can work with 1G, 2M and 4k in the L2 and everything between 2M..4k in the L1. On Intel it is only 2M and Nvidia has 1G and 2M (but really don't nail me on that this is just as far as I know).

So if reclaim of smaller pages into larger ones isn't requested what you end up with is unstable performance, e.g. if you only run application A everything is fine, if you switch to application B and then run application A you see a performance penalty of up to 30% which eventually doesn't go away over time.

On the other hand when reclaim is always enabled for multiple different orders you start to see stuttering when allocations are made because the core memory management tries to fulfill all those different orders.

We literally had tons of bug reports regarding those problems and it is one of the major reasons why we doesn't use DMA-buf heaps for HPC use cases. Instead drivers even communicate the sweet spot to for example TTMs allocation backend.

I don't have much of a problem when we say that this is just for Android, we never do reclaim here, instead reclaim is triggered solely by userspace on application switch etc... but the general idea of DMA-buf heaps is that it should work for everybody.

Regards,
Christian.