[PATCH 0/3] dma-buf: system_heap: 2MB system heap chunks

[PATCH 0/3] dma-buf: system_heap: 2MB system heap chunks

[Davidlohr Bueso]

Hello,

- Patch 1 adds order 9 to the allocation order ladder.

- Patch 2 fixes the order imprecision when using non 4k base pages.
This is not new but patch 1 enlarges it.

- Patch 3 is a misc independent fixlet.

Thanks!

Davidlohr Bueso (3):
  dma-buf: system_heap: add a 2MB allocation order for IOMMU superpages
  dma-buf: system_heap: fix the allocation ladder on non-4K base pages
  dma-buf: validate the attachment before asserting its reservation lock

 drivers/dma-buf/dma-buf-mapping.c   |  4 ++--
 drivers/dma-buf/heaps/system_heap.c | 23 ++++++++++++++++-------
 2 files changed, 18 insertions(+), 9 deletions(-)

-- 
2.39.5

[PATCH 1/3] dma-buf: system_heap: add a 2MB allocation order for IOMMU superpages

[Davidlohr Bueso]

With 4KB base pages, the system heap allocates buffers in 1MB, 64KB
and 4KB chunks. The conventional Intel VT-d second-stage and AMD-Vi v2
page-table formats, and Arm SMMU's 64-bit long-descriptor format with
a 4KB translation granule, define 4KB pages and 2MB/1GB large-page
mappings, but no 1MB leaf. A 1MB chunk therefore requires 256 4KB leaf
entries unless it can be combined with adjacent chunks into a
suitably aligned larger mapping.

Adding a 2MB allocation order provides naturally aligned, physically
contiguous chunks matching the 2MB leaf size, without relying on this
being accidental for separate smaller allocations. The new cost
is one failed order-9 attempt (keeping current large allocation semantics)
per buffer when 2MB pages are exhausted.

Two benchmarks, measured on an AMD EPYC 7313P.

(i) Mapping, into an idle NVMe function's translated DMA-FQ domain,
measuring map/unmap: with the v1 page table restricted to 4K/2M/1G
(amd_iommu=v2_pgsizes_only), a 1GB buffer goes from 1024 1MB chunks,
only one of whose 2MB windows was superpage-mappable in that run, to
512 2MB chunks with all 512 mappable. dma_buf_map_attachment() costs
decrease by ~16x on average with ~32x for the worst cases. Unmapping
that buffer drops from ~500 us to 1.25 us.

(ii) A microbench that measures the cost of DMA_HEAP_IOCTL_ALLOC+close
across various thread counts decreases by factors of ~3-7x alleviating
allocator's zone->lock contention by being PMD order and therefore is
pcpu list eligible. Once the buffer size is large enough then the cost
of the zeroing takes over.

Signed-off-by: Davidlohr Bueso <dave@stgolabs.net>
---
 drivers/dma-buf/heaps/system_heap.c | 13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)

diff --git a/drivers/dma-buf/heaps/system_heap.c b/drivers/dma-buf/heaps/system_heap.c
index c8959eadc71d..621cf238ef97 100644
--- a/drivers/dma-buf/heaps/system_heap.c
+++ b/drivers/dma-buf/heaps/system_heap.c
@@ -55,14 +55,15 @@ struct dma_heap_attachment {
 #define HIGH_ORDER_GFP  (((GFP_HIGHUSER | __GFP_ZERO | __GFP_NOWARN \
 				| __GFP_NORETRY) & ~__GFP_RECLAIM) \
 				| __GFP_COMP)
-static gfp_t order_flags[] = {HIGH_ORDER_GFP, HIGH_ORDER_GFP, LOW_ORDER_GFP};
+static gfp_t order_flags[] = {HIGH_ORDER_GFP, HIGH_ORDER_GFP,
+			      HIGH_ORDER_GFP, LOW_ORDER_GFP};
 /*
- * The selection of the orders used for allocation (1MB, 64K, 4K) is designed
- * to match with the sizes often found in IOMMUs. Using order 4 pages instead
- * of order 0 pages can significantly improve the performance of many IOMMUs
- * by reducing TLB pressure and time spent updating page tables.
+ * The selection of the orders used for allocation (2MB, 1MB, 64K, 4K) is
+ * designed to match with the sizes often found in IOMMUs. Using larger order
+ * pages instead of order 0 pages can significantly improve the performance of
+ * many IOMMUs by reducing TLB pressure and time spent updating page tables.
  */
-static const unsigned int orders[] = {8, 4, 0};
+static const unsigned int orders[] = {9, 8, 4, 0};
 #define NUM_ORDERS ARRAY_SIZE(orders)
 
 static int system_heap_set_page_decrypted(struct page *page)
-- 
2.39.5

Re: [PATCH 1/3] dma-buf: system_heap: add a 2MB allocation order for IOMMU superpages

[T.J. Mercier]

On Mon, Sep 14, 2026 at 2:23 PM Davidlohr Bueso <dave@stgolabs.net> wrote:
>
> With 4KB base pages, the system heap allocates buffers in 1MB, 64KB
> and 4KB chunks. The conventional Intel VT-d second-stage and AMD-Vi v2
> page-table formats, and Arm SMMU's 64-bit long-descriptor format with
> a 4KB translation granule, define 4KB pages and 2MB/1GB large-page
> mappings, but no 1MB leaf. A 1MB chunk therefore requires 256 4KB leaf
> entries unless it can be combined with adjacent chunks into a
> suitably aligned larger mapping.
>
> Adding a 2MB allocation order provides naturally aligned, physically
> contiguous chunks matching the 2MB leaf size, without relying on this
> being accidental for separate smaller allocations. The new cost
> is one failed order-9 attempt (keeping current large allocation semantics)
> per buffer when 2MB pages are exhausted.
>
> Two benchmarks, measured on an AMD EPYC 7313P.
>
> (i) Mapping, into an idle NVMe function's translated DMA-FQ domain,
> measuring map/unmap: with the v1 page table restricted to 4K/2M/1G
> (amd_iommu=v2_pgsizes_only), a 1GB buffer goes from 1024 1MB chunks,
> only one of whose 2MB windows was superpage-mappable in that run, to
> 512 2MB chunks with all 512 mappable. dma_buf_map_attachment() costs
> decrease by ~16x on average with ~32x for the worst cases. Unmapping
> that buffer drops from ~500 us to 1.25 us.
>
> (ii) A microbench that measures the cost of DMA_HEAP_IOCTL_ALLOC+close
> across various thread counts decreases by factors of ~3-7x alleviating
> allocator's zone->lock contention by being PMD order and therefore is
> pcpu list eligible. Once the buffer size is large enough then the cost
> of the zeroing takes over.
>
> Signed-off-by: Davidlohr Bueso <dave@stgolabs.net>
> ---
>  drivers/dma-buf/heaps/system_heap.c | 13 +++++++------
>  1 file changed, 7 insertions(+), 6 deletions(-)
>
> diff --git a/drivers/dma-buf/heaps/system_heap.c b/drivers/dma-buf/heaps/system_heap.c
> index c8959eadc71d..621cf238ef97 100644
> --- a/drivers/dma-buf/heaps/system_heap.c
> +++ b/drivers/dma-buf/heaps/system_heap.c
> @@ -55,14 +55,15 @@ struct dma_heap_attachment {
>  #define HIGH_ORDER_GFP  (((GFP_HIGHUSER | __GFP_ZERO | __GFP_NOWARN \
>                                 | __GFP_NORETRY) & ~__GFP_RECLAIM) \
>                                 | __GFP_COMP)
> -static gfp_t order_flags[] = {HIGH_ORDER_GFP, HIGH_ORDER_GFP, LOW_ORDER_GFP};
> +static gfp_t order_flags[] = {HIGH_ORDER_GFP, HIGH_ORDER_GFP,
> +                             HIGH_ORDER_GFP, LOW_ORDER_GFP};
>  /*
> - * The selection of the orders used for allocation (1MB, 64K, 4K) is designed
> - * to match with the sizes often found in IOMMUs. Using order 4 pages instead
> - * of order 0 pages can significantly improve the performance of many IOMMUs
> - * by reducing TLB pressure and time spent updating page tables.
> + * The selection of the orders used for allocation (2MB, 1MB, 64K, 4K) is
> + * designed to match with the sizes often found in IOMMUs.

and now also PMD order for allocation / page table efficiency in
addition to sizes IOMMUs can coalesce

>  Using larger order
> + * pages instead of order 0 pages can significantly improve the performance of
> + * many IOMMUs by reducing TLB pressure and time spent updating page tables.
>   */
> -static const unsigned int orders[] = {8, 4, 0};
> +static const unsigned int orders[] = {9, 8, 4, 0};
>  #define NUM_ORDERS ARRAY_SIZE(orders)
>
>  static int system_heap_set_page_decrypted(struct page *page)
> --
> 2.39.5
>

Reviewed-by: T.J. Mercier <tjmercier@google.com>

The origin of these orders is:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/drivers/staging/android/ion/ion_system_heap.c?id=bd5d6bdae118c49d194869ef757fd650dcabd33e
with not a lot of justification in the commit message.

Downstream we attempt order 9 instead of 8 on heaps that we actually
ship on our devices to match the 2MB PMD size, and many other vendor
heap implementations do as well. Not that it's always likely we'll get
pages that big due to fragmentation.

Re: [PATCH 1/3] dma-buf: system_heap: add a 2MB allocation order for IOMMU superpages

[Christian König]

On 9/14/26 23:22, Davidlohr Bueso wrote:
> With 4KB base pages, the system heap allocates buffers in 1MB, 64KB
> and 4KB chunks. The conventional Intel VT-d second-stage and AMD-Vi v2
> page-table formats, and Arm SMMU's 64-bit long-descriptor format with
> a 4KB translation granule, define 4KB pages and 2MB/1GB large-page
> mappings, but no 1MB leaf. A 1MB chunk therefore requires 256 4KB leaf
> entries unless it can be combined with adjacent chunks into a
> suitably aligned larger mapping.

Yes, it was an intentional design choice to *NOT* optimize for AMD nor Intels IOMMU here.

> Adding a 2MB allocation order provides naturally aligned, physically
> contiguous chunks matching the 2MB leaf size, without relying on this
> being accidental for separate smaller allocations. The new cost
> is one failed order-9 attempt (keeping current large allocation semantics)
> per buffer when 2MB pages are exhausted.

We have plenty of experience with this with TTM and the overhead this results in is usually not acceptable.

> Two benchmarks, measured on an AMD EPYC 7313P.

Why in the world are you testing the system heap on an AMD EPYC system?

AMD clearly doesn't recommend using the system heap on those boxes for ROCm, so I'm really wondering what combination of HW you have here?

Regards,
Christian.

> 
> (i) Mapping, into an idle NVMe function's translated DMA-FQ domain,
> measuring map/unmap: with the v1 page table restricted to 4K/2M/1G
> (amd_iommu=v2_pgsizes_only), a 1GB buffer goes from 1024 1MB chunks,
> only one of whose 2MB windows was superpage-mappable in that run, to
> 512 2MB chunks with all 512 mappable. dma_buf_map_attachment() costs
> decrease by ~16x on average with ~32x for the worst cases. Unmapping
> that buffer drops from ~500 us to 1.25 us.
> 
> (ii) A microbench that measures the cost of DMA_HEAP_IOCTL_ALLOC+close
> across various thread counts decreases by factors of ~3-7x alleviating
> allocator's zone->lock contention by being PMD order and therefore is
> pcpu list eligible. Once the buffer size is large enough then the cost
> of the zeroing takes over.
> 
> Signed-off-by: Davidlohr Bueso <dave@stgolabs.net>
> ---
>  drivers/dma-buf/heaps/system_heap.c | 13 +++++++------
>  1 file changed, 7 insertions(+), 6 deletions(-)
> 
> diff --git a/drivers/dma-buf/heaps/system_heap.c b/drivers/dma-buf/heaps/system_heap.c
> index c8959eadc71d..621cf238ef97 100644
> --- a/drivers/dma-buf/heaps/system_heap.c
> +++ b/drivers/dma-buf/heaps/system_heap.c
> @@ -55,14 +55,15 @@ struct dma_heap_attachment {
>  #define HIGH_ORDER_GFP  (((GFP_HIGHUSER | __GFP_ZERO | __GFP_NOWARN \
>                                 | __GFP_NORETRY) & ~__GFP_RECLAIM) \
>                                 | __GFP_COMP)
> -static gfp_t order_flags[] = {HIGH_ORDER_GFP, HIGH_ORDER_GFP, LOW_ORDER_GFP};
> +static gfp_t order_flags[] = {HIGH_ORDER_GFP, HIGH_ORDER_GFP,
> +                             HIGH_ORDER_GFP, LOW_ORDER_GFP};
>  /*
> - * The selection of the orders used for allocation (1MB, 64K, 4K) is designed
> - * to match with the sizes often found in IOMMUs. Using order 4 pages instead
> - * of order 0 pages can significantly improve the performance of many IOMMUs
> - * by reducing TLB pressure and time spent updating page tables.
> + * The selection of the orders used for allocation (2MB, 1MB, 64K, 4K) is
> + * designed to match with the sizes often found in IOMMUs. Using larger order
> + * pages instead of order 0 pages can significantly improve the performance of
> + * many IOMMUs by reducing TLB pressure and time spent updating page tables.
>   */
> -static const unsigned int orders[] = {8, 4, 0};
> +static const unsigned int orders[] = {9, 8, 4, 0};
>  #define NUM_ORDERS ARRAY_SIZE(orders)
> 
>  static int system_heap_set_page_decrypted(struct page *page)
> --
> 2.39.5
> 

Re: [PATCH 1/3] dma-buf: system_heap: add a 2MB allocation order for IOMMU superpages

[T.J. Mercier]

On Tue, Sep 15, 2026 at 1:41 AM Christian König
<christian.koenig@amd.com> wrote:
>
> On 9/14/26 23:22, Davidlohr Bueso wrote:
> > With 4KB base pages, the system heap allocates buffers in 1MB, 64KB
> > and 4KB chunks. The conventional Intel VT-d second-stage and AMD-Vi v2
> > page-table formats, and Arm SMMU's 64-bit long-descriptor format with
> > a 4KB translation granule, define 4KB pages and 2MB/1GB large-page
> > mappings, but no 1MB leaf. A 1MB chunk therefore requires 256 4KB leaf
> > entries unless it can be combined with adjacent chunks into a
> > suitably aligned larger mapping.
>
> Yes, it was an intentional design choice to *NOT* optimize for AMD nor Intels IOMMU here.
>
> > Adding a 2MB allocation order provides naturally aligned, physically
> > contiguous chunks matching the 2MB leaf size, without relying on this
> > being accidental for separate smaller allocations. The new cost
> > is one failed order-9 attempt (keeping current large allocation semantics)
> > per buffer when 2MB pages are exhausted.
>
> We have plenty of experience with this with TTM and the overhead this results in is usually not acceptable.

Overhead from compaction and reclaim to get 2M pages? That is disabled
for non-0 page orders here in HIGH_ORDER_GFP.

> > Two benchmarks, measured on an AMD EPYC 7313P.
>
> Why in the world are you testing the system heap on an AMD EPYC system?
>
> AMD clearly doesn't recommend using the system heap on those boxes for ROCm, so I'm really wondering what combination of HW you have here?
>
> Regards,
> Christian.
>
> >
> > (i) Mapping, into an idle NVMe function's translated DMA-FQ domain,
> > measuring map/unmap: with the v1 page table restricted to 4K/2M/1G
> > (amd_iommu=v2_pgsizes_only), a 1GB buffer goes from 1024 1MB chunks,
> > only one of whose 2MB windows was superpage-mappable in that run, to
> > 512 2MB chunks with all 512 mappable. dma_buf_map_attachment() costs
> > decrease by ~16x on average with ~32x for the worst cases. Unmapping
> > that buffer drops from ~500 us to 1.25 us.
> >
> > (ii) A microbench that measures the cost of DMA_HEAP_IOCTL_ALLOC+close
> > across various thread counts decreases by factors of ~3-7x alleviating
> > allocator's zone->lock contention by being PMD order and therefore is
> > pcpu list eligible. Once the buffer size is large enough then the cost
> > of the zeroing takes over.
> >
> > Signed-off-by: Davidlohr Bueso <dave@stgolabs.net>
> > ---
> >  drivers/dma-buf/heaps/system_heap.c | 13 +++++++------
> >  1 file changed, 7 insertions(+), 6 deletions(-)
> >
> > diff --git a/drivers/dma-buf/heaps/system_heap.c b/drivers/dma-buf/heaps/system_heap.c
> > index c8959eadc71d..621cf238ef97 100644
> > --- a/drivers/dma-buf/heaps/system_heap.c
> > +++ b/drivers/dma-buf/heaps/system_heap.c
> > @@ -55,14 +55,15 @@ struct dma_heap_attachment {
> >  #define HIGH_ORDER_GFP  (((GFP_HIGHUSER | __GFP_ZERO | __GFP_NOWARN \
> >                                 | __GFP_NORETRY) & ~__GFP_RECLAIM) \
> >                                 | __GFP_COMP)
> > -static gfp_t order_flags[] = {HIGH_ORDER_GFP, HIGH_ORDER_GFP, LOW_ORDER_GFP};
> > +static gfp_t order_flags[] = {HIGH_ORDER_GFP, HIGH_ORDER_GFP,
> > +                             HIGH_ORDER_GFP, LOW_ORDER_GFP};
> >  /*
> > - * The selection of the orders used for allocation (1MB, 64K, 4K) is designed
> > - * to match with the sizes often found in IOMMUs. Using order 4 pages instead
> > - * of order 0 pages can significantly improve the performance of many IOMMUs
> > - * by reducing TLB pressure and time spent updating page tables.
> > + * The selection of the orders used for allocation (2MB, 1MB, 64K, 4K) is
> > + * designed to match with the sizes often found in IOMMUs. Using larger order
> > + * pages instead of order 0 pages can significantly improve the performance of
> > + * many IOMMUs by reducing TLB pressure and time spent updating page tables.
> >   */
> > -static const unsigned int orders[] = {8, 4, 0};
> > +static const unsigned int orders[] = {9, 8, 4, 0};
> >  #define NUM_ORDERS ARRAY_SIZE(orders)
> >
> >  static int system_heap_set_page_decrypted(struct page *page)
> > --
> > 2.39.5
> >
>

Re: [PATCH 1/3] dma-buf: system_heap: add a 2MB allocation order for IOMMU superpages

[Christian König]

On 9/15/26 17:33, T.J. Mercier wrote:
> On Tue, Sep 15, 2026 at 1:41 AM Christian König
> <christian.koenig@amd.com> wrote:
>>
>> On 9/14/26 23:22, Davidlohr Bueso wrote:
>>> With 4KB base pages, the system heap allocates buffers in 1MB, 64KB
>>> and 4KB chunks. The conventional Intel VT-d second-stage and AMD-Vi v2
>>> page-table formats, and Arm SMMU's 64-bit long-descriptor format with
>>> a 4KB translation granule, define 4KB pages and 2MB/1GB large-page
>>> mappings, but no 1MB leaf. A 1MB chunk therefore requires 256 4KB leaf
>>> entries unless it can be combined with adjacent chunks into a
>>> suitably aligned larger mapping.
>>
>> Yes, it was an intentional design choice to *NOT* optimize for AMD nor Intels IOMMU here.
>>
>>> Adding a 2MB allocation order provides naturally aligned, physically
>>> contiguous chunks matching the 2MB leaf size, without relying on this
>>> being accidental for separate smaller allocations. The new cost
>>> is one failed order-9 attempt (keeping current large allocation semantics)
>>> per buffer when 2MB pages are exhausted.
>>
>> We have plenty of experience with this with TTM and the overhead this results in is usually not acceptable.
> 
> Overhead from compaction and reclaim to get 2M pages? That is disabled
> for non-0 page orders here in HIGH_ORDER_GFP.

Ah! Thanks for pointing that out, I misread the code that __GFP_RECLAIM is ORed into the mask.

But the question is still why? Without reclaim that is pretty much an useless feature on most x86 boxes.

Regards,
Christian.

> 
>>> Two benchmarks, measured on an AMD EPYC 7313P.
>>
>> Why in the world are you testing the system heap on an AMD EPYC system?
>>
>> AMD clearly doesn't recommend using the system heap on those boxes for ROCm, so I'm really wondering what combination of HW you have here?
>>
>> Regards,
>> Christian.
>>
>>>
>>> (i) Mapping, into an idle NVMe function's translated DMA-FQ domain,
>>> measuring map/unmap: with the v1 page table restricted to 4K/2M/1G
>>> (amd_iommu=v2_pgsizes_only), a 1GB buffer goes from 1024 1MB chunks,
>>> only one of whose 2MB windows was superpage-mappable in that run, to
>>> 512 2MB chunks with all 512 mappable. dma_buf_map_attachment() costs
>>> decrease by ~16x on average with ~32x for the worst cases. Unmapping
>>> that buffer drops from ~500 us to 1.25 us.
>>>
>>> (ii) A microbench that measures the cost of DMA_HEAP_IOCTL_ALLOC+close
>>> across various thread counts decreases by factors of ~3-7x alleviating
>>> allocator's zone->lock contention by being PMD order and therefore is
>>> pcpu list eligible. Once the buffer size is large enough then the cost
>>> of the zeroing takes over.
>>>
>>> Signed-off-by: Davidlohr Bueso <dave@stgolabs.net>
>>> ---
>>>  drivers/dma-buf/heaps/system_heap.c | 13 +++++++------
>>>  1 file changed, 7 insertions(+), 6 deletions(-)
>>>
>>> diff --git a/drivers/dma-buf/heaps/system_heap.c b/drivers/dma-buf/heaps/system_heap.c
>>> index c8959eadc71d..621cf238ef97 100644
>>> --- a/drivers/dma-buf/heaps/system_heap.c
>>> +++ b/drivers/dma-buf/heaps/system_heap.c
>>> @@ -55,14 +55,15 @@ struct dma_heap_attachment {
>>>  #define HIGH_ORDER_GFP  (((GFP_HIGHUSER | __GFP_ZERO | __GFP_NOWARN \
>>>                                 | __GFP_NORETRY) & ~__GFP_RECLAIM) \
>>>                                 | __GFP_COMP)
>>> -static gfp_t order_flags[] = {HIGH_ORDER_GFP, HIGH_ORDER_GFP, LOW_ORDER_GFP};
>>> +static gfp_t order_flags[] = {HIGH_ORDER_GFP, HIGH_ORDER_GFP,
>>> +                             HIGH_ORDER_GFP, LOW_ORDER_GFP};
>>>  /*
>>> - * The selection of the orders used for allocation (1MB, 64K, 4K) is designed
>>> - * to match with the sizes often found in IOMMUs. Using order 4 pages instead
>>> - * of order 0 pages can significantly improve the performance of many IOMMUs
>>> - * by reducing TLB pressure and time spent updating page tables.
>>> + * The selection of the orders used for allocation (2MB, 1MB, 64K, 4K) is
>>> + * designed to match with the sizes often found in IOMMUs. Using larger order
>>> + * pages instead of order 0 pages can significantly improve the performance of
>>> + * many IOMMUs by reducing TLB pressure and time spent updating page tables.
>>>   */
>>> -static const unsigned int orders[] = {8, 4, 0};
>>> +static const unsigned int orders[] = {9, 8, 4, 0};
>>>  #define NUM_ORDERS ARRAY_SIZE(orders)
>>>
>>>  static int system_heap_set_page_decrypted(struct page *page)
>>> --
>>> 2.39.5
>>>
>>

Re: [PATCH 1/3] dma-buf: system_heap: add a 2MB allocation order for IOMMU superpages

[T.J. Mercier]

On Tue, Sep 15, 2026 at 8:44 AM Christian König
<christian.koenig@amd.com> wrote:
>
> On 9/15/26 17:33, T.J. Mercier wrote:
> > On Tue, Sep 15, 2026 at 1:41 AM Christian König
> > <christian.koenig@amd.com> wrote:
> >>
> >> On 9/14/26 23:22, Davidlohr Bueso wrote:
> >>> With 4KB base pages, the system heap allocates buffers in 1MB, 64KB
> >>> and 4KB chunks. The conventional Intel VT-d second-stage and AMD-Vi v2
> >>> page-table formats, and Arm SMMU's 64-bit long-descriptor format with
> >>> a 4KB translation granule, define 4KB pages and 2MB/1GB large-page
> >>> mappings, but no 1MB leaf. A 1MB chunk therefore requires 256 4KB leaf
> >>> entries unless it can be combined with adjacent chunks into a
> >>> suitably aligned larger mapping.
> >>
> >> Yes, it was an intentional design choice to *NOT* optimize for AMD nor Intels IOMMU here.
> >>
> >>> Adding a 2MB allocation order provides naturally aligned, physically
> >>> contiguous chunks matching the 2MB leaf size, without relying on this
> >>> being accidental for separate smaller allocations. The new cost
> >>> is one failed order-9 attempt (keeping current large allocation semantics)
> >>> per buffer when 2MB pages are exhausted.
> >>
> >> We have plenty of experience with this with TTM and the overhead this results in is usually not acceptable.
> >
> > Overhead from compaction and reclaim to get 2M pages? That is disabled
> > for non-0 page orders here in HIGH_ORDER_GFP.
>
> Ah! Thanks for pointing that out, I misread the code that __GFP_RECLAIM is ORed into the mask.
>
> But the question is still why? Without reclaim that is pretty much an useless feature on most x86 boxes.

Even on arm64 phones free 2M pages aren't likely to be available for
very long after boot, but lately we have been doing more proactive
reclaim triggered by userspace before launching workflows that desire
large dma-buf allocations (and also at other times during application
lifecycle in general). That somewhat increases the likelihood that
high order pages will be available. I agree there's no guarantee we'll
get any, but without __GFP_RECLAIM the attempt is pretty cheap and the
payoff can be pretty benficial which Davidlohr's allocation and
mapping measurements demonstrate.


> Regards,
> Christian.
>
> >
> >>> Two benchmarks, measured on an AMD EPYC 7313P.
> >>
> >> Why in the world are you testing the system heap on an AMD EPYC system?
> >>
> >> AMD clearly doesn't recommend using the system heap on those boxes for ROCm, so I'm really wondering what combination of HW you have here?
> >>
> >> Regards,
> >> Christian.
> >>
> >>>
> >>> (i) Mapping, into an idle NVMe function's translated DMA-FQ domain,
> >>> measuring map/unmap: with the v1 page table restricted to 4K/2M/1G
> >>> (amd_iommu=v2_pgsizes_only), a 1GB buffer goes from 1024 1MB chunks,
> >>> only one of whose 2MB windows was superpage-mappable in that run, to
> >>> 512 2MB chunks with all 512 mappable. dma_buf_map_attachment() costs
> >>> decrease by ~16x on average with ~32x for the worst cases. Unmapping
> >>> that buffer drops from ~500 us to 1.25 us.
> >>>
> >>> (ii) A microbench that measures the cost of DMA_HEAP_IOCTL_ALLOC+close
> >>> across various thread counts decreases by factors of ~3-7x alleviating
> >>> allocator's zone->lock contention by being PMD order and therefore is
> >>> pcpu list eligible. Once the buffer size is large enough then the cost
> >>> of the zeroing takes over.
> >>>
> >>> Signed-off-by: Davidlohr Bueso <dave@stgolabs.net>
> >>> ---
> >>>  drivers/dma-buf/heaps/system_heap.c | 13 +++++++------
> >>>  1 file changed, 7 insertions(+), 6 deletions(-)
> >>>
> >>> diff --git a/drivers/dma-buf/heaps/system_heap.c b/drivers/dma-buf/heaps/system_heap.c
> >>> index c8959eadc71d..621cf238ef97 100644
> >>> --- a/drivers/dma-buf/heaps/system_heap.c
> >>> +++ b/drivers/dma-buf/heaps/system_heap.c
> >>> @@ -55,14 +55,15 @@ struct dma_heap_attachment {
> >>>  #define HIGH_ORDER_GFP  (((GFP_HIGHUSER | __GFP_ZERO | __GFP_NOWARN \
> >>>                                 | __GFP_NORETRY) & ~__GFP_RECLAIM) \
> >>>                                 | __GFP_COMP)
> >>> -static gfp_t order_flags[] = {HIGH_ORDER_GFP, HIGH_ORDER_GFP, LOW_ORDER_GFP};
> >>> +static gfp_t order_flags[] = {HIGH_ORDER_GFP, HIGH_ORDER_GFP,
> >>> +                             HIGH_ORDER_GFP, LOW_ORDER_GFP};
> >>>  /*
> >>> - * The selection of the orders used for allocation (1MB, 64K, 4K) is designed
> >>> - * to match with the sizes often found in IOMMUs. Using order 4 pages instead
> >>> - * of order 0 pages can significantly improve the performance of many IOMMUs
> >>> - * by reducing TLB pressure and time spent updating page tables.
> >>> + * The selection of the orders used for allocation (2MB, 1MB, 64K, 4K) is
> >>> + * designed to match with the sizes often found in IOMMUs. Using larger order
> >>> + * pages instead of order 0 pages can significantly improve the performance of
> >>> + * many IOMMUs by reducing TLB pressure and time spent updating page tables.
> >>>   */
> >>> -static const unsigned int orders[] = {8, 4, 0};
> >>> +static const unsigned int orders[] = {9, 8, 4, 0};
> >>>  #define NUM_ORDERS ARRAY_SIZE(orders)
> >>>
> >>>  static int system_heap_set_page_decrypted(struct page *page)
> >>> --
> >>> 2.39.5
> >>>
> >>
>

Re: [PATCH 1/3] dma-buf: system_heap: add a 2MB allocation order for IOMMU superpages

[T.J. Mercier]

On Tue, Sep 15, 2026 at 9:26 AM T.J. Mercier <tjmercier@google.com> wrote:
>
> On Tue, Sep 15, 2026 at 8:44 AM Christian König
> <christian.koenig@amd.com> wrote:
> >
> > On 9/15/26 17:33, T.J. Mercier wrote:
> > > On Tue, Sep 15, 2026 at 1:41 AM Christian König
> > > <christian.koenig@amd.com> wrote:
> > >>
> > >> On 9/14/26 23:22, Davidlohr Bueso wrote:
> > >>> With 4KB base pages, the system heap allocates buffers in 1MB, 64KB
> > >>> and 4KB chunks. The conventional Intel VT-d second-stage and AMD-Vi v2
> > >>> page-table formats, and Arm SMMU's 64-bit long-descriptor format with
> > >>> a 4KB translation granule, define 4KB pages and 2MB/1GB large-page
> > >>> mappings, but no 1MB leaf. A 1MB chunk therefore requires 256 4KB leaf
> > >>> entries unless it can be combined with adjacent chunks into a
> > >>> suitably aligned larger mapping.
> > >>
> > >> Yes, it was an intentional design choice to *NOT* optimize for AMD nor Intels IOMMU here.
> > >>
> > >>> Adding a 2MB allocation order provides naturally aligned, physically
> > >>> contiguous chunks matching the 2MB leaf size, without relying on this
> > >>> being accidental for separate smaller allocations. The new cost
> > >>> is one failed order-9 attempt (keeping current large allocation semantics)
> > >>> per buffer when 2MB pages are exhausted.
> > >>
> > >> We have plenty of experience with this with TTM and the overhead this results in is usually not acceptable.
> > >
> > > Overhead from compaction and reclaim to get 2M pages? That is disabled
> > > for non-0 page orders here in HIGH_ORDER_GFP.
> >
> > Ah! Thanks for pointing that out, I misread the code that __GFP_RECLAIM is ORed into the mask.
> >
> > But the question is still why? Without reclaim that is pretty much an useless feature on most x86 boxes.
>
> Even on arm64 phones free 2M pages aren't likely to be available for
> very long after boot, but lately we have been doing more proactive
> reclaim triggered by userspace before launching workflows that desire
> large dma-buf allocations (and also at other times during application
> lifecycle in general). That somewhat increases the likelihood that
> high order pages will be available. I agree there's no guarantee we'll
> get any, but without __GFP_RECLAIM the attempt is pretty cheap and the
> payoff can be pretty benficial which Davidlohr's allocation and
> mapping measurements demonstrate.

Oh I forgot to add that this behavior is similar to how slab
allocations are done for slabs requiring more than an order 0 page per
slab. A large optimal order is attempted first with ~__GFP_RECLAIM,
but if that fails a fallback to a smaller min order is done. When the
optimal order allocation attempt fails, the cost is just taking the
zone lock for a quick peek at the free lists.

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/mm/slub.c?h=v7.2#n3373

> > Regards,
> > Christian.
> >
> > >
> > >>> Two benchmarks, measured on an AMD EPYC 7313P.
> > >>
> > >> Why in the world are you testing the system heap on an AMD EPYC system?
> > >>
> > >> AMD clearly doesn't recommend using the system heap on those boxes for ROCm, so I'm really wondering what combination of HW you have here?
> > >>
> > >> Regards,
> > >> Christian.
> > >>
> > >>>
> > >>> (i) Mapping, into an idle NVMe function's translated DMA-FQ domain,
> > >>> measuring map/unmap: with the v1 page table restricted to 4K/2M/1G
> > >>> (amd_iommu=v2_pgsizes_only), a 1GB buffer goes from 1024 1MB chunks,
> > >>> only one of whose 2MB windows was superpage-mappable in that run, to
> > >>> 512 2MB chunks with all 512 mappable. dma_buf_map_attachment() costs
> > >>> decrease by ~16x on average with ~32x for the worst cases. Unmapping
> > >>> that buffer drops from ~500 us to 1.25 us.
> > >>>
> > >>> (ii) A microbench that measures the cost of DMA_HEAP_IOCTL_ALLOC+close
> > >>> across various thread counts decreases by factors of ~3-7x alleviating
> > >>> allocator's zone->lock contention by being PMD order and therefore is
> > >>> pcpu list eligible. Once the buffer size is large enough then the cost
> > >>> of the zeroing takes over.
> > >>>
> > >>> Signed-off-by: Davidlohr Bueso <dave@stgolabs.net>
> > >>> ---
> > >>>  drivers/dma-buf/heaps/system_heap.c | 13 +++++++------
> > >>>  1 file changed, 7 insertions(+), 6 deletions(-)
> > >>>
> > >>> diff --git a/drivers/dma-buf/heaps/system_heap.c b/drivers/dma-buf/heaps/system_heap.c
> > >>> index c8959eadc71d..621cf238ef97 100644
> > >>> --- a/drivers/dma-buf/heaps/system_heap.c
> > >>> +++ b/drivers/dma-buf/heaps/system_heap.c
> > >>> @@ -55,14 +55,15 @@ struct dma_heap_attachment {
> > >>>  #define HIGH_ORDER_GFP  (((GFP_HIGHUSER | __GFP_ZERO | __GFP_NOWARN \
> > >>>                                 | __GFP_NORETRY) & ~__GFP_RECLAIM) \
> > >>>                                 | __GFP_COMP)
> > >>> -static gfp_t order_flags[] = {HIGH_ORDER_GFP, HIGH_ORDER_GFP, LOW_ORDER_GFP};
> > >>> +static gfp_t order_flags[] = {HIGH_ORDER_GFP, HIGH_ORDER_GFP,
> > >>> +                             HIGH_ORDER_GFP, LOW_ORDER_GFP};
> > >>>  /*
> > >>> - * The selection of the orders used for allocation (1MB, 64K, 4K) is designed
> > >>> - * to match with the sizes often found in IOMMUs. Using order 4 pages instead
> > >>> - * of order 0 pages can significantly improve the performance of many IOMMUs
> > >>> - * by reducing TLB pressure and time spent updating page tables.
> > >>> + * The selection of the orders used for allocation (2MB, 1MB, 64K, 4K) is
> > >>> + * designed to match with the sizes often found in IOMMUs. Using larger order
> > >>> + * pages instead of order 0 pages can significantly improve the performance of
> > >>> + * many IOMMUs by reducing TLB pressure and time spent updating page tables.
> > >>>   */
> > >>> -static const unsigned int orders[] = {8, 4, 0};
> > >>> +static const unsigned int orders[] = {9, 8, 4, 0};
> > >>>  #define NUM_ORDERS ARRAY_SIZE(orders)
> > >>>
> > >>>  static int system_heap_set_page_decrypted(struct page *page)
> > >>> --
> > >>> 2.39.5
> > >>>
> > >>
> >

Re: [PATCH 1/3] dma-buf: system_heap: add a 2MB allocation order for IOMMU superpages

[Christian König]

On 9/15/26 19:26, T.J. Mercier wrote:
> On Tue, Sep 15, 2026 at 9:26 AM T.J. Mercier <tjmercier@google.com> wrote:
>>
>> On Tue, Sep 15, 2026 at 8:44 AM Christian König
>> <christian.koenig@amd.com> wrote:
...>> Even on arm64 phones free 2M pages aren't likely to be available for
>> very long after boot, but lately we have been doing more proactive
>> reclaim triggered by userspace before launching workflows that desire
>> large dma-buf allocations (and also at other times during application
>> lifecycle in general). That somewhat increases the likelihood that
>> high order pages will be available. I agree there's no guarantee we'll
>> get any, but without __GFP_RECLAIM the attempt is pretty cheap and the
>> payoff can be pretty benficial which Davidlohr's allocation and
>> mapping measurements demonstrate.
> 
> Oh I forgot to add that this behavior is similar to how slab
> allocations are done for slabs requiring more than an order 0 page per
> slab. A large optimal order is attempted first with ~__GFP_RECLAIM,
> but if that fails a fallback to a smaller min order is done. When the
> optimal order allocation attempt fails, the cost is just taking the
> zone lock for a quick peek at the free lists.
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/mm/slub.c?h=v7.2#n3373

Well from a lot of experience with AMDs ROCm stack and Intels XE driver I think that this is a really bad idea.

The background is that there are a lot of operation which can fragment memory to a point where only 4k pages are available, e.g. for example storage I/O can do that trivially (the kernel compile is a really good test case for that).

Now acceleration HW usually has a very specific sweet spot where it works best. AMDs GPUs for example can work with 1G, 2M and 4k in the L2 and everything between 2M..4k in the L1. On Intel it is only 2M and Nvidia has 1G and 2M (but really don't nail me on that this is just as far as I know).

So if reclaim of smaller pages into larger ones isn't requested what you end up with is unstable performance, e.g. if you only run application A everything is fine, if you switch to application B and then run application A you see a performance penalty of up to 30% which eventually doesn't go away over time.

On the other hand when reclaim is always enabled for multiple different orders you start to see stuttering when allocations are made because the core memory management tries to fulfill all those different orders.

We literally had tons of bug reports regarding those problems and it is one of the major reasons why we doesn't use DMA-buf heaps for HPC use cases. Instead drivers even communicate the sweet spot to for example TTMs allocation backend.

I don't have much of a problem when we say that this is just for Android, we never do reclaim here, instead reclaim is triggered solely by userspace on application switch etc... but the general idea of DMA-buf heaps is that it should work for everybody.

Regards,
Christian.

Re: [PATCH 1/3] dma-buf: system_heap: add a 2MB allocation order for IOMMU superpages

[T.J. Mercier]

On Wed, Sep 16, 2026 at 1:41 AM Christian König
<christian.koenig@amd.com> wrote:
>
> On 9/15/26 19:26, T.J. Mercier wrote:
> > On Tue, Sep 15, 2026 at 9:26 AM T.J. Mercier <tjmercier@google.com> wrote:
> >>
> >> On Tue, Sep 15, 2026 at 8:44 AM Christian König
> >> <christian.koenig@amd.com> wrote:
> ...>> Even on arm64 phones free 2M pages aren't likely to be available for
> >> very long after boot, but lately we have been doing more proactive
> >> reclaim triggered by userspace before launching workflows that desire
> >> large dma-buf allocations (and also at other times during application
> >> lifecycle in general). That somewhat increases the likelihood that
> >> high order pages will be available. I agree there's no guarantee we'll
> >> get any, but without __GFP_RECLAIM the attempt is pretty cheap and the
> >> payoff can be pretty benficial which Davidlohr's allocation and
> >> mapping measurements demonstrate.
> >
> > Oh I forgot to add that this behavior is similar to how slab
> > allocations are done for slabs requiring more than an order 0 page per
> > slab. A large optimal order is attempted first with ~__GFP_RECLAIM,
> > but if that fails a fallback to a smaller min order is done. When the
> > optimal order allocation attempt fails, the cost is just taking the
> > zone lock for a quick peek at the free lists.
> >
> > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/mm/slub.c?h=v7.2#n3373
>
> Well from a lot of experience with AMDs ROCm stack and Intels XE driver I think that this is a really bad idea.
>
> The background is that there are a lot of operation which can fragment memory to a point where only 4k pages are available, e.g. for example storage I/O can do that trivially (the kernel compile is a really good test case for that).
>
> Now acceleration HW usually has a very specific sweet spot where it works best. AMDs GPUs for example can work with 1G, 2M and 4k in the L2 and everything between 2M..4k in the L1. On Intel it is only 2M and Nvidia has 1G and 2M (but really don't nail me on that this is just as far as I know).
>
> So if reclaim of smaller pages into larger ones isn't requested what you end up with is unstable performance, e.g. if you only run application A everything is fine, if you switch to application B and then run application A you see a performance penalty of up to 30% which eventually doesn't go away over time.
>
> On the other hand when reclaim is always enabled for multiple different orders you start to see stuttering when allocations are made because the core memory management tries to fulfill all those different orders.

Yup I understand the tradeoff you're describing here between
allocation latency and performance predictability for GPU/HPC. But for
a generic allocator like the system heap with very loose constraints
on what memory can be used, this lack of optimism permanently incurs a
penalty that might not have been necessary. Users of the system heap
don't demand any particular page order they just deal with what
they're given. There is no single driver or use case for system_heap
users. Opportunistic allocation without reclaim gives the upside when
pages are free without stalling when they aren't.

> We literally had tons of bug reports regarding those problems and it is one of the major reasons why we doesn't use DMA-buf heaps for HPC use cases. Instead drivers even communicate the sweet spot to for example TTMs allocation backend.

This behavior isn't an inherent property of all dma-buf heaps though,
it's just how the system heap does it. You could create a different
heap that always tries to allocate the sweet spot for a particular use
case.

> I don't have much of a problem when we say that this is just for Android, we never do reclaim here, instead reclaim is triggered solely by userspace on application switch etc... but the general idea of DMA-buf heaps is that it should work for everybody.

I'm not sure these patches were motivated by Android (based on x86
focus in the commit message), but Davidlohr would have to comment on
that. That's just where most of my experience comes from and what I
can share details about.

Re: [PATCH 1/3] dma-buf: system_heap: add a 2MB allocation order for IOMMU superpages

[Davidlohr Bueso]

On Wed, 16 Sep 2026, T.J. Mercier wrote:

>I'm not sure these patches were motivated by Android (based on x86
>focus in the commit message), but Davidlohr would have to comment on
>that. That's just where most of my experience comes from and what I
>can share details about.

This series was motivated by exploring new use cases for dma-buf (such
as kvcache), not really for Android specifically. I was counting on
the cheap/one-time 2Mb attempt (current high order semantics) to not be
a problem for potential performance gains. As for the x86 I tested here,
basically because it's what I have on hand, I had no idea ROCm was a no-no
for system heaps.

I am fine dropping this effort.

Thanks,
Davidlohr

Re: [PATCH 1/3] dma-buf: system_heap: add a 2MB allocation order for IOMMU superpages

[Christian König]

On 9/16/26 18:22, Davidlohr Bueso wrote:
> On Wed, 16 Sep 2026, T.J. Mercier wrote:
> 
>> I'm not sure these patches were motivated by Android (based on x86
>> focus in the commit message), but Davidlohr would have to comment on
>> that. That's just where most of my experience comes from and what I
>> can share details about.
> 
> This series was motivated by exploring new use cases for dma-buf (such
> as kvcache), not really for Android specifically. I was counting on
> the cheap/one-time 2Mb attempt (current high order semantics) to not be
> a problem for potential performance gains. As for the x86 I tested here,
> basically because it's what I have on hand, I had no idea ROCm was a no-no
> for system heaps.

Well the problem is that it does work, but gives you unstable performance and people see that as a bug while it is an intentional design decision made to keep the system heap driver independent.

So yeah it's a trade off, it should just be properly documented.
> I am fine dropping this effort.

It's clearly good that we talked about this.

When something is changed here I would rather say that we have some #ifdef CONFIG_ARM around the existing code and use the PMD_SIZE for all other platforms.

If functionality isn't changed I think adding at least some notes that this only works optimal on 4k ARM platforms is clearly a good idea.

Regards,
Christian.

> 
> Thanks,
> Davidlohr

Re: [PATCH 1/3] dma-buf: system_heap: add a 2MB allocation order for IOMMU superpages

[Davidlohr Bueso]

On Thu, 17 Sep 2026, Christian K�nig wrote:

>On 9/16/26 18:22, Davidlohr Bueso wrote:
>> On Wed, 16 Sep 2026, T.J. Mercier wrote:
>>
>>> I'm not sure these patches were motivated by Android (based on x86
>>> focus in the commit message), but Davidlohr would have to comment on
>>> that. That's just where most of my experience comes from and what I
>>> can share details about.
>>
>> This series was motivated by exploring new use cases for dma-buf (such
>> as kvcache), not really for Android specifically. I was counting on
>> the cheap/one-time 2Mb attempt (current high order semantics) to not be
>> a problem for potential performance gains. As for the x86 I tested here,
>> basically because it's what I have on hand, I had no idea ROCm was a no-no
>> for system heaps.
>
>Well the problem is that it does work, but gives you unstable performance and people see that as a bug while it is an intentional design decision made to keep the system heap driver independent.
>
>So yeah it's a trade off, it should just be properly documented.
>> I am fine dropping this effort.
>
>It's clearly good that we talked about this.
>
>When something is changed here I would rather say that we have some #ifdef CONFIG_ARM around the existing code and use the PMD_SIZE for all other platforms.

Just able to circle back to this, sorry for the delay.

I will go ahead then and add the ifdefiry for arm to leave it alone, and add the PMD size otherwise.

>
>If functionality isn't changed I think adding at least some notes that this only works optimal on 4k ARM platforms is clearly a good idea.

(will also comment the situation).

Thanks,
Davidlohr

[PATCH 2/3] dma-buf: system_heap: fix the allocation ladder on non-4K base pages

[Davidlohr Bueso]

The orders for large base page sizes are calculated incorrectly, and
currently only hold for the 4K case. Order 9 is intended for 2MB but
will be 8MB on 16K, for example. Adjust the orders/sizes accordingly.

Signed-off-by: Davidlohr Bueso <dave@stgolabs.net>
---
 drivers/dma-buf/heaps/system_heap.c | 16 ++++++++++++----
 1 file changed, 12 insertions(+), 4 deletions(-)

diff --git a/drivers/dma-buf/heaps/system_heap.c b/drivers/dma-buf/heaps/system_heap.c
index 621cf238ef97..341631e4a9ec 100644
--- a/drivers/dma-buf/heaps/system_heap.c
+++ b/drivers/dma-buf/heaps/system_heap.c
@@ -16,12 +16,15 @@
 #include <linux/dma-heap.h>
 #include <linux/err.h>
 #include <linux/highmem.h>
+#include <linux/log2.h>
 #include <linux/mem_encrypt.h>
+#include <linux/minmax.h>
 #include <linux/mm.h>
 #include <linux/set_memory.h>
 #include <linux/module.h>
 #include <linux/pgtable.h>
 #include <linux/scatterlist.h>
+#include <linux/sizes.h>
 #include <linux/slab.h>
 #include <linux/vmalloc.h>
 
@@ -55,15 +58,18 @@ struct dma_heap_attachment {
 #define HIGH_ORDER_GFP  (((GFP_HIGHUSER | __GFP_ZERO | __GFP_NOWARN \
 				| __GFP_NORETRY) & ~__GFP_RECLAIM) \
 				| __GFP_COMP)
-static gfp_t order_flags[] = {HIGH_ORDER_GFP, HIGH_ORDER_GFP,
-			      HIGH_ORDER_GFP, LOW_ORDER_GFP};
+#define SIZE_ORDER(sz) \
+	MIN(MAX(const_ilog2(sz) - PAGE_SHIFT, 0), MAX_PAGE_ORDER)
+
 /*
  * The selection of the orders used for allocation (2MB, 1MB, 64K, 4K) is
  * designed to match with the sizes often found in IOMMUs. Using larger order
  * pages instead of order 0 pages can significantly improve the performance of
  * many IOMMUs by reducing TLB pressure and time spent updating page tables.
  */
-static const unsigned int orders[] = {9, 8, 4, 0};
+static const unsigned int orders[] = {
+	SIZE_ORDER(SZ_2M), SIZE_ORDER(SZ_1M), SIZE_ORDER(SZ_64K), 0
+};
 #define NUM_ORDERS ARRAY_SIZE(orders)
 
 static int system_heap_set_page_decrypted(struct page *page)
@@ -384,11 +390,13 @@ static struct page *alloc_largest_available(unsigned long size,
 	gfp_t flags;
 
 	for (i = 0; i < NUM_ORDERS; i++) {
+		if (i && orders[i] == orders[i - 1])
+			continue;
 		if (size <  (PAGE_SIZE << orders[i]))
 			continue;
 		if (max_order < orders[i])
 			continue;
-		flags = order_flags[i];
+		flags = orders[i] ? HIGH_ORDER_GFP : LOW_ORDER_GFP;
 		if (mem_accounting)
 			flags |= __GFP_ACCOUNT;
 		page = alloc_pages(flags, orders[i]);
-- 
2.39.5

Re: [PATCH 2/3] dma-buf: system_heap: fix the allocation ladder on non-4K base pages

[T.J. Mercier]

On Mon, Sep 14, 2026 at 2:23 PM Davidlohr Bueso <dave@stgolabs.net> wrote:
>
> The orders for large base page sizes are calculated incorrectly, and
> currently only hold for the 4K case. Order 9 is intended for 2MB but
> will be 8MB on 16K, for example. Adjust the orders/sizes accordingly.
>
> Signed-off-by: Davidlohr Bueso <dave@stgolabs.net>

Reviewed-by: T.J. Mercier <tjmercier@google.com>

We've been doing something like this downstream to support 16k base
pages: https://r.android.com/3907360

Re: [PATCH 2/3] dma-buf: system_heap: fix the allocation ladder on non-4K base pages

[Christian König]

On 9/14/26 23:22, Davidlohr Bueso wrote:
> The orders for large base page sizes are calculated incorrectly,

No, they aren't. This system heap is just only optimized for 4k page size and doesn't take anything else into account.

And yes that is a fully intentional decision.

What we could do is to make this architecture dependent, e.g. on ARM we only use 1M/64K while on X86 we always use 2M, no idea what others use.

Regards,
Christian.

> and
> currently only hold for the 4K case. Order 9 is intended for 2MB but
> will be 8MB on 16K, for example. Adjust the orders/sizes accordingly.
> 
> Signed-off-by: Davidlohr Bueso <dave@stgolabs.net>
> ---
>  drivers/dma-buf/heaps/system_heap.c | 16 ++++++++++++----
>  1 file changed, 12 insertions(+), 4 deletions(-)
> 
> diff --git a/drivers/dma-buf/heaps/system_heap.c b/drivers/dma-buf/heaps/system_heap.c
> index 621cf238ef97..341631e4a9ec 100644
> --- a/drivers/dma-buf/heaps/system_heap.c
> +++ b/drivers/dma-buf/heaps/system_heap.c
> @@ -16,12 +16,15 @@
>  #include <linux/dma-heap.h>
>  #include <linux/err.h>
>  #include <linux/highmem.h>
> +#include <linux/log2.h>
>  #include <linux/mem_encrypt.h>
> +#include <linux/minmax.h>
>  #include <linux/mm.h>
>  #include <linux/set_memory.h>
>  #include <linux/module.h>
>  #include <linux/pgtable.h>
>  #include <linux/scatterlist.h>
> +#include <linux/sizes.h>
>  #include <linux/slab.h>
>  #include <linux/vmalloc.h>
> 
> @@ -55,15 +58,18 @@ struct dma_heap_attachment {
>  #define HIGH_ORDER_GFP  (((GFP_HIGHUSER | __GFP_ZERO | __GFP_NOWARN \
>                                 | __GFP_NORETRY) & ~__GFP_RECLAIM) \
>                                 | __GFP_COMP)
> -static gfp_t order_flags[] = {HIGH_ORDER_GFP, HIGH_ORDER_GFP,
> -                             HIGH_ORDER_GFP, LOW_ORDER_GFP};
> +#define SIZE_ORDER(sz) \
> +       MIN(MAX(const_ilog2(sz) - PAGE_SHIFT, 0), MAX_PAGE_ORDER)
> +
>  /*
>   * The selection of the orders used for allocation (2MB, 1MB, 64K, 4K) is
>   * designed to match with the sizes often found in IOMMUs. Using larger order
>   * pages instead of order 0 pages can significantly improve the performance of
>   * many IOMMUs by reducing TLB pressure and time spent updating page tables.
>   */
> -static const unsigned int orders[] = {9, 8, 4, 0};
> +static const unsigned int orders[] = {
> +       SIZE_ORDER(SZ_2M), SIZE_ORDER(SZ_1M), SIZE_ORDER(SZ_64K), 0
> +};
>  #define NUM_ORDERS ARRAY_SIZE(orders)
> 
>  static int system_heap_set_page_decrypted(struct page *page)
> @@ -384,11 +390,13 @@ static struct page *alloc_largest_available(unsigned long size,
>         gfp_t flags;
> 
>         for (i = 0; i < NUM_ORDERS; i++) {
> +               if (i && orders[i] == orders[i - 1])
> +                       continue;
>                 if (size <  (PAGE_SIZE << orders[i]))
>                         continue;
>                 if (max_order < orders[i])
>                         continue;
> -               flags = order_flags[i];
> +               flags = orders[i] ? HIGH_ORDER_GFP : LOW_ORDER_GFP;
>                 if (mem_accounting)
>                         flags |= __GFP_ACCOUNT;
>                 page = alloc_pages(flags, orders[i]);
> --
> 2.39.5
> 

[PATCH 3/3] dma-buf: validate the attachment before asserting its reservation lock

[Davidlohr Bueso]

dma_buf_phys_vec_to_sgt() asserts the reservation lock before its
NULL checks run. Validate first, then assert.

Fixes: 3aa31a8bb11e ("dma-buf: provide phys_vec to scatter-gather mapping routine")
Signed-off-by: Davidlohr Bueso <dave@stgolabs.net>
---
 drivers/dma-buf/dma-buf-mapping.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/dma-buf/dma-buf-mapping.c b/drivers/dma-buf/dma-buf-mapping.c
index 833be519e1e6..12674d89b23e 100644
--- a/drivers/dma-buf/dma-buf-mapping.c
+++ b/drivers/dma-buf/dma-buf-mapping.c
@@ -111,12 +111,12 @@ struct sg_table *dma_buf_phys_vec_to_sgt(struct dma_buf_attachment *attach,
 	size_t i;
 	int ret;
 
-	dma_resv_assert_held(attach->dmabuf->resv);
-
 	if (WARN_ON(!attach || !attach->dmabuf || !provider))
 		/* This function is supposed to work on MMIO memory only */
 		return ERR_PTR(-EINVAL);
 
+	dma_resv_assert_held(attach->dmabuf->resv);
+
 	dma = kzalloc_obj(*dma);
 	if (!dma)
 		return ERR_PTR(-ENOMEM);
-- 
2.39.5

Re: [PATCH 3/3] dma-buf: validate the attachment before asserting its reservation lock

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider:

Pre-existing issues:
- [Critical] dma_iova_link() is called with a hardcoded offset of 0, causing all physical ranges to overlap at the start of the IOVA space.
- [High] IOVA space allocated by dma_iova_try_alloc() is leaked on multiple error paths.
- [High] The fallback error path unconditionally unmaps unpopulated, zero-initialized scatterlist entries by overwriting the loop counter.
- [High] Mapping a large region as a single chunk but unmapping it in smaller split chunks violates the DMA API.
--

commit 139d9e764ec01d51d2adf94f612620ee800a32bb
Author: Davidlohr Bueso <dave@stgolabs.net>

dma-buf: validate the attachment before asserting its reservation lock

dma_buf_phys_vec_to_sgt() asserts the reservation lock before its
NULL checks run. Validate first, then assert.

> diff --git a/drivers/dma-buf/dma-buf-mapping.c b/drivers/dma-buf/dma-buf-mapping.c
> index 833be519e1e6a..12674d89b23eb 100644
> --- a/drivers/dma-buf/dma-buf-mapping.c
> +++ b/drivers/dma-buf/dma-buf-mapping.c
> @@ -111,12 +111,12 @@ struct sg_table *dma_buf_phys_vec_to_sgt(struct dma_buf_attachment *attach,
>  	size_t i;
>  	int ret;
>  
> -	dma_resv_assert_held(attach->dmabuf->resv);
> -
>  	if (WARN_ON(!attach || !attach->dmabuf || !provider))
>  		/* This function is supposed to work on MMIO memory only */
>  		return ERR_PTR(-EINVAL);
>  
> +	dma_resv_assert_held(attach->dmabuf->resv);
> +
>  	dma = kzalloc_obj(*dma);
>  	if (!dma)
>  		return ERR_PTR(-ENOMEM);

[Severity: High]
This is a pre-existing issue, but does this code leak the IOVA space allocated
by dma_iova_try_alloc() on multiple error paths?

If sg_alloc_table() fails, the code jumps to err_free_state:

    ret = sg_alloc_table(&dma->sgt, nents, GFP_KERNEL | __GFP_ZERO);
    if (ret)
        goto err_free_state;

This frees dma->state but skips calling dma_iova_free() to release the IOVA
allocation.

Additionally, if the very first physical range fails to link (when i == 0),
the error handler skips destruction:

    err_unmap_dma:
        if (!i || !dma->state) {
            ; /* Do nothing */
        } else if (dma_use_iova(dma->state)) {
            dma_iova_destroy(attach->dev, dma->state, mapped_len, dir,
                             DMA_ATTR_MMIO);

Can this exhaust device IOMMU IOVA space under memory pressure?

[Severity: Critical]
This is a pre-existing issue, but does this loop incorrectly map all
physical ranges to the exact same starting IOVA address?

    } else if (dma_use_iova(dma->state)) {
        ret = dma_iova_link(attach->dev, dma->state,
                            phys_vec[i].paddr, 0,
                            phys_vec[i].len, dir,
                            DMA_ATTR_MMIO);

When nr_ranges > 1, dma_iova_link() is called repeatedly with the offset
hardcoded to 0. Since the target IOVA address is computed as
(state->addr + offset), each physical range overwrites the previous mapping.

Should the offset be passed as mapped_len to ensure segments are appended
sequentially?

[Severity: High]
This is a pre-existing issue, but could this teardown logic violate the DMA
API by mapping a large region as a single chunk and unmapping it
piece-by-piece?

During mapping, a physical range might be mapped in a single contiguous block:

    addr = dma_map_phys(attach->dev, phys_vec[i].paddr,
                        phys_vec[i].len, dir,
                        DMA_ATTR_MMIO);

Later during teardown, the fallback error path iterates over the individual
split SG entries and unmaps them separately:

    err_unmap_dma:
        ...
        } else {
            for_each_sgtable_dma_sg(&dma->sgt, sgl, i)
                dma_unmap_phys(attach->dev, sg_dma_address(sgl),
                               sg_dma_len(sgl), dir, DMA_ATTR_MMIO);

The same piece-by-piece unmapping also happens in dma_buf_free_sgt().

Since the DMA API requires that the size and address passed to dma_unmap_phys()
exactly match the original call to dma_map_phys(), could these partial unmaps
corrupt IOMMU tracking structures?

[Severity: High]
This is a pre-existing issue, but does the fallback error path unconditionally
unmap unpopulated, zero-initialized scatterlist entries?

    } else {
        for_each_sgtable_dma_sg(&dma->sgt, sgl, i)
            dma_unmap_phys(attach->dev, sg_dma_address(sgl),
                           sg_dma_len(sgl), dir, DMA_ATTR_MMIO);

The for_each_sgtable_dma_sg macro shadows and resets 'i' to 0, looping over
all allocated entries (dma->sgt.nents) rather than just the successfully
mapped ranges.

Since the table was allocated with __GFP_ZERO, the unpopulated entries will
have a length and address of 0. Could passing these to dma_unmap_phys()
trigger IOMMU faults?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260914212258.2657788-1-dave@stgolabs.net?part=3

Re: [PATCH 3/3] dma-buf: validate the attachment before asserting its reservation lock

[T.J. Mercier]

On Mon, Sep 14, 2026 at 2:23 PM Davidlohr Bueso <dave@stgolabs.net> wrote:
>
> dma_buf_phys_vec_to_sgt() asserts the reservation lock before its
> NULL checks run. Validate first, then assert.
>
> Fixes: 3aa31a8bb11e ("dma-buf: provide phys_vec to scatter-gather mapping routine")
> Signed-off-by: Davidlohr Bueso <dave@stgolabs.net>

Reviewed-by: T.J. Mercier <tjmercier@google.com>

Re: [PATCH 3/3] dma-buf: validate the attachment before asserting its reservation lock

[Christian König]

On 9/14/26 23:22, Davidlohr Bueso wrote:
> dma_buf_phys_vec_to_sgt() asserts the reservation lock before its
> NULL checks run. Validate first, then assert.
> 
> Fixes: 3aa31a8bb11e ("dma-buf: provide phys_vec to scatter-gather mapping routine")
> Signed-off-by: Davidlohr Bueso <dave@stgolabs.net>

Good catch, but in general please put bug fixes as first one in a patch set.

Reviewed-by: Christian König <christian.koenig@amd.com>

Regards,
Christian.

> ---
>  drivers/dma-buf/dma-buf-mapping.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/dma-buf/dma-buf-mapping.c b/drivers/dma-buf/dma-buf-mapping.c
> index 833be519e1e6..12674d89b23e 100644
> --- a/drivers/dma-buf/dma-buf-mapping.c
> +++ b/drivers/dma-buf/dma-buf-mapping.c
> @@ -111,12 +111,12 @@ struct sg_table *dma_buf_phys_vec_to_sgt(struct dma_buf_attachment *attach,
>         size_t i;
>         int ret;
> 
> -       dma_resv_assert_held(attach->dmabuf->resv);
> -
>         if (WARN_ON(!attach || !attach->dmabuf || !provider))
>                 /* This function is supposed to work on MMIO memory only */
>                 return ERR_PTR(-EINVAL);
> 
> +       dma_resv_assert_held(attach->dmabuf->resv);
> +
>         dma = kzalloc_obj(*dma);
>         if (!dma)
>                 return ERR_PTR(-ENOMEM);
> --
> 2.39.5
>