[PATCH v5 net] octeontx2-af: switch qmem from coherent DMA alloc to streaming DMA mapping

[PATCH v5 net] octeontx2-af: switch qmem from coherent DMA alloc to streaming DMA mapping

[Ratheesh Kannoth]

qmem_alloc() uses dma_alloc_attrs() with DMA_ATTR_FORCE_CONTIGUOUS, which
allocates CPU-cache-coherent DMA memory and, with CMA enabled, draws from
the CMA pool. qmem backs NIX/NPA queue contexts, admin queues, and LMTST
regions (including CN10K LMTST areas that span page boundaries), so
consumption grows with enabled interfaces and is hard to provision in CMA.

Switch qmem to a streaming-DMA-style path: allocate physically contiguous
compound pages from the buddy allocator via __get_free_pages(), then map
them for device access with dma_map_phys() and dma_unmap_phys() using
DMA_ATTR_REQUIRE_COHERENT. Add otx2_dma_alloc_coherent() and
otx2_dma_free_coherent() helpers that enforce dev_is_dma_coherent(),
retry with GFP_DMA32 when the physical range is outside the device DMA
mask, and wire qmem_alloc()/qmem_free() through them instead of
dma_alloc_attrs()/dma_free_attrs().

This works on Octeon because the octeontx2 driver is written for
DMA-coherent devices: Octeon platforms provide IO coherency (via SMMU), so
the driver already uses streaming DMA APIs for packet data while
deliberately skipping explicit CPU cache sync (DMA_ATTR_SKIP_CPU_SYNC).
The same IO coherency lets qmem use a streaming map of buddy-allocated
pages instead of a dedicated coherent allocator or CMA reservation. That
is valid because the platform is DMA-coherent, not because omitting
dma_sync_* magically makes memory coherent.

Allocations requiring more than MAX_PAGE_ORDER pages are still rejected,
since the buddy allocator cannot serve them without CMA.

cc: Geetha sowjanya <gakula@marvell.com>
Fixes: 73d33dbc0723 ("octeontx2-af: Use DMA_ATTR_FORCE_CONTIGUOUS attribute in DMA alloc")
Signed-off-by: Ratheesh Kannoth <rkannoth@marvell.com>

---

v4 -> v5: Fixed compilation issues.
	https://lore.kernel.org/netdev/20260831024210.208447-1-rkannoth@marvell.com/

v3 -> v4: Fixed compilation issues.
	https://lore.kernel.org/netdev/apTpKcN_S1xIwRbZ@rkannoth-OptiPlex-7090/

v2 -> v3: Addressed sashiko comments
	https://sashiko.dev/#/patchset/20260825045616.3723078-1-rkannoth%40marvell.com

v1 -> v2: Rewrote patch as per sashiko comment
---
 .../ethernet/marvell/octeontx2/af/common.h    | 93 +++++++++++++++++--
 1 file changed, 87 insertions(+), 6 deletions(-)

diff --git a/drivers/net/ethernet/marvell/octeontx2/af/common.h b/drivers/net/ethernet/marvell/octeontx2/af/common.h
index 779413a383b7..061dd907f7fa 100644
--- a/drivers/net/ethernet/marvell/octeontx2/af/common.h
+++ b/drivers/net/ethernet/marvell/octeontx2/af/common.h
@@ -7,6 +7,11 @@
 #ifndef COMMON_H
 #define COMMON_H
 
+#include <linux/dma-mapping.h>
+#include <linux/dma-map-ops.h>
+#include <linux/gfp.h>
+#include <linux/mm.h>
+
 #include "rvu_struct.h"
 
 #define OTX2_ALIGN			128  /* Align to cacheline */
@@ -44,6 +49,83 @@ struct qmem {
 	u32		qsize;
 };
 
+/* Buddy-backed coherent DMA alloc (Option 3): pages from __get_free_pages(),
+ * DMA-reachable RAM via dma_coherent_ok(), and bus/SMMU mappings via
+ * dma_map_phys() -> iommu_dma_map_phys() -> iommu_map() on SMMU systems.
+ */
+#define OTX2_DMA_COHERENT_ATTRS	DMA_ATTR_REQUIRE_COHERENT
+
+static inline bool otx2_dma_phys_in_mask(struct device *dev, phys_addr_t paddr,
+					 size_t size)
+{
+	u64 mask = dma_get_mask(dev);
+
+	return paddr + size - 1 <= mask;
+}
+
+static inline void *otx2_dma_alloc_coherent(struct device *dev, size_t size,
+					    dma_addr_t *dma_handle, gfp_t gfp)
+{
+	dma_addr_t dma_addr;
+	unsigned int order;
+	phys_addr_t paddr;
+	void *vaddr;
+	gfp_t alloc_gfp;
+
+	if (!dev || !dma_handle || !size)
+		return NULL;
+
+	if (!dev_is_dma_coherent(dev))
+		return NULL;
+
+	size = PAGE_ALIGN(size);
+	order = get_order(size);
+	if (order > MAX_PAGE_ORDER)
+		return NULL;
+
+	alloc_gfp = (gfp & ~(__GFP_DMA | __GFP_DMA32 | __GFP_HIGHMEM)) |
+		__GFP_ZERO | __GFP_COMP | __GFP_RECLAIM;
+
+	vaddr = (void *)__get_free_pages(alloc_gfp, order);
+	while (vaddr &&
+	       !otx2_dma_phys_in_mask(dev, page_to_phys(virt_to_page(vaddr)), size)) {
+		free_pages((unsigned long)vaddr, order);
+		if (alloc_gfp & GFP_DMA32)
+			return NULL;
+		alloc_gfp |= GFP_DMA32;
+		vaddr = (void *)__get_free_pages(alloc_gfp, order);
+	}
+	if (!vaddr)
+		return NULL;
+
+	paddr = page_to_phys(virt_to_page(vaddr));
+	dma_addr = dma_map_phys(dev, paddr, size, DMA_BIDIRECTIONAL,
+				OTX2_DMA_COHERENT_ATTRS);
+	if (dma_mapping_error(dev, dma_addr)) {
+		free_pages((unsigned long)vaddr, order);
+		return NULL;
+	}
+
+	*dma_handle = dma_addr;
+	return vaddr;
+}
+
+static inline void otx2_dma_free_coherent(struct device *dev, size_t size,
+					  void *vaddr, dma_addr_t dma_handle)
+{
+	unsigned int order;
+
+	if (!dev || !vaddr)
+		return;
+
+	size = PAGE_ALIGN(size);
+	order = get_order(size);
+
+	dma_unmap_phys(dev, dma_handle, size, DMA_BIDIRECTIONAL,
+		       OTX2_DMA_COHERENT_ATTRS);
+	free_pages((unsigned long)vaddr, order);
+}
+
 static inline int qmem_alloc(struct device *dev, struct qmem **q,
 			     int qsize, int entry_sz)
 {
@@ -60,8 +142,8 @@ static inline int qmem_alloc(struct device *dev, struct qmem **q,
 
 	qmem->entry_sz = entry_sz;
 	qmem->alloc_sz = (qsize * entry_sz) + OTX2_ALIGN;
-	qmem->base = dma_alloc_attrs(dev, qmem->alloc_sz, &qmem->iova,
-				     GFP_KERNEL, DMA_ATTR_FORCE_CONTIGUOUS);
+	qmem->base = otx2_dma_alloc_coherent(dev, qmem->alloc_sz, &qmem->iova,
+					     GFP_KERNEL);
 	if (!qmem->base)
 		return -ENOMEM;
 
@@ -80,10 +162,9 @@ static inline void qmem_free(struct device *dev, struct qmem *qmem)
 		return;
 
 	if (qmem->base)
-		dma_free_attrs(dev, qmem->alloc_sz,
-			       qmem->base - qmem->align,
-			       qmem->iova - qmem->align,
-			       DMA_ATTR_FORCE_CONTIGUOUS);
+		otx2_dma_free_coherent(dev, qmem->alloc_sz,
+				       qmem->base - qmem->align,
+				       qmem->iova - qmem->align);
 	devm_kfree(dev, qmem);
 }
 
-- 
2.43.0

Re: [PATCH v5 net] octeontx2-af: switch qmem from coherent DMA alloc to streaming DMA mapping

[Qingfang Deng]

Hi,

On 2026/9/1 9:56, Ratheesh Kannoth wrote:
> qmem_alloc() uses dma_alloc_attrs() with DMA_ATTR_FORCE_CONTIGUOUS, which
> allocates CPU-cache-coherent DMA memory and, with CMA enabled, draws from
> the CMA pool. qmem backs NIX/NPA queue contexts, admin queues, and LMTST
> regions (including CN10K LMTST areas that span page boundaries), so
> consumption grows with enabled interfaces and is hard to provision in CMA.
>
> Switch qmem to a streaming-DMA-style path: allocate physically contiguous
> compound pages from the buddy allocator via __get_free_pages(), then map
> them for device access with dma_map_phys() and dma_unmap_phys() using
> DMA_ATTR_REQUIRE_COHERENT. Add otx2_dma_alloc_coherent() and
> otx2_dma_free_coherent() helpers that enforce dev_is_dma_coherent(),
> retry with GFP_DMA32 when the physical range is outside the device DMA
> mask, and wire qmem_alloc()/qmem_free() through them instead of
> dma_alloc_attrs()/dma_free_attrs().

__get_free_pages() can only allocate memory in power-of-two pages. You 
can use alloc_pages_exact() and free_pages_exact() to save memory.

> This works on Octeon because the octeontx2 driver is written for
> DMA-coherent devices: Octeon platforms provide IO coherency (via SMMU), so
> the driver already uses streaming DMA APIs for packet data while
> deliberately skipping explicit CPU cache sync (DMA_ATTR_SKIP_CPU_SYNC).
> The same IO coherency lets qmem use a streaming map of buddy-allocated
> pages instead of a dedicated coherent allocator or CMA reservation. That
> is valid because the platform is DMA-coherent, not because omitting
> dma_sync_* magically makes memory coherent.
>
> Allocations requiring more than MAX_PAGE_ORDER pages are still rejected,
> since the buddy allocator cannot serve them without CMA.

Can you confirm that no existing allocations exceeds MAX_PAGE_ORDER?

> cc: Geetha sowjanya <gakula@marvell.com>
> Fixes: 73d33dbc0723 ("octeontx2-af: Use DMA_ATTR_FORCE_CONTIGUOUS attribute in DMA alloc")
> Signed-off-by: Ratheesh Kannoth <rkannoth@marvell.com>
>
> ---
>
> v4 -> v5: Fixed compilation issues.
> 	https://lore.kernel.org/netdev/20260831024210.208447-1-rkannoth@marvell.com/
>
> v3 -> v4: Fixed compilation issues.
> 	https://lore.kernel.org/netdev/apTpKcN_S1xIwRbZ@rkannoth-OptiPlex-7090/
>
> v2 -> v3: Addressed sashiko comments
> 	https://sashiko.dev/#/patchset/20260825045616.3723078-1-rkannoth%40marvell.com
>
> v1 -> v2: Rewrote patch as per sashiko comment
> ---
>   .../ethernet/marvell/octeontx2/af/common.h    | 93 +++++++++++++++++--
>   1 file changed, 87 insertions(+), 6 deletions(-)
>
> diff --git a/drivers/net/ethernet/marvell/octeontx2/af/common.h b/drivers/net/ethernet/marvell/octeontx2/af/common.h
> index 779413a383b7..061dd907f7fa 100644
> --- a/drivers/net/ethernet/marvell/octeontx2/af/common.h
> +++ b/drivers/net/ethernet/marvell/octeontx2/af/common.h
> @@ -7,6 +7,11 @@
>   #ifndef COMMON_H
>   #define COMMON_H
>   
> +#include <linux/dma-mapping.h>
> +#include <linux/dma-map-ops.h>
> +#include <linux/gfp.h>
> +#include <linux/mm.h>
> +
>   #include "rvu_struct.h"
>   
>   #define OTX2_ALIGN			128  /* Align to cacheline */
> @@ -44,6 +49,83 @@ struct qmem {
>   	u32		qsize;
>   };
>   
> +/* Buddy-backed coherent DMA alloc (Option 3): pages from __get_free_pages(),
> + * DMA-reachable RAM via dma_coherent_ok(), and bus/SMMU mappings via
> + * dma_map_phys() -> iommu_dma_map_phys() -> iommu_map() on SMMU systems.
> + */
> +#define OTX2_DMA_COHERENT_ATTRS	DMA_ATTR_REQUIRE_COHERENT
> +
> +static inline bool otx2_dma_phys_in_mask(struct device *dev, phys_addr_t paddr,
> +					 size_t size)
> +{
> +	u64 mask = dma_get_mask(dev);
> +
> +	return paddr + size - 1 <= mask;
> +}
> +
> +static inline void *otx2_dma_alloc_coherent(struct device *dev, size_t size,
> +					    dma_addr_t *dma_handle, gfp_t gfp)
> +{
> +	dma_addr_t dma_addr;
> +	unsigned int order;
> +	phys_addr_t paddr;
> +	void *vaddr;
> +	gfp_t alloc_gfp;
> +
> +	if (!dev || !dma_handle || !size)
> +		return NULL;
> +
> +	if (!dev_is_dma_coherent(dev))
> +		return NULL;
> +
> +	size = PAGE_ALIGN(size);
> +	order = get_order(size);
> +	if (order > MAX_PAGE_ORDER)
> +		return NULL;
> +
> +	alloc_gfp = (gfp & ~(__GFP_DMA | __GFP_DMA32 | __GFP_HIGHMEM)) |
> +		__GFP_ZERO | __GFP_COMP | __GFP_RECLAIM;
> +
> +	vaddr = (void *)__get_free_pages(alloc_gfp, order);
> +	while (vaddr &&
> +	       !otx2_dma_phys_in_mask(dev, page_to_phys(virt_to_page(vaddr)), size)) {

page_to_phys(virt_to_page(vaddr)) can be simplifed to virt_to_phys(vaddr).

> +		free_pages((unsigned long)vaddr, order);
> +		if (alloc_gfp & GFP_DMA32)
> +			return NULL;
> +		alloc_gfp |= GFP_DMA32;
> +		vaddr = (void *)__get_free_pages(alloc_gfp, order);
> +	}
> +	if (!vaddr)
> +		return NULL;
> +
> +	paddr = page_to_phys(virt_to_page(vaddr));

Same here.

> +	dma_addr = dma_map_phys(dev, paddr, size, DMA_BIDIRECTIONAL,
> +				OTX2_DMA_COHERENT_ATTRS);
> +	if (dma_mapping_error(dev, dma_addr)) {
> +		free_pages((unsigned long)vaddr, order);
> +		return NULL;
> +	}
> +
> +	*dma_handle = dma_addr;
> +	return vaddr;
> +}
Kind regards,
Qingfang

Re: [PATCH v5 net] octeontx2-af: switch qmem from coherent DMA alloc to streaming DMA mapping

[Ratheesh Kannoth]

On 2026-09-01 at 09:20:01, Qingfang Deng (qingfang.deng@linux.dev) wrote:
> Hi,
>
> On 2026/9/1 9:56, Ratheesh Kannoth wrote:
> > qmem_alloc() uses dma_alloc_attrs() with DMA_ATTR_FORCE_CONTIGUOUS, which
> > allocates CPU-cache-coherent DMA memory and, with CMA enabled, draws from
> > the CMA pool. qmem backs NIX/NPA queue contexts, admin queues, and LMTST
> > regions (including CN10K LMTST areas that span page boundaries), so
> > consumption grows with enabled interfaces and is hard to provision in CMA.
> >
> > Switch qmem to a streaming-DMA-style path: allocate physically contiguous
> > compound pages from the buddy allocator via __get_free_pages(), then map
> > them for device access with dma_map_phys() and dma_unmap_phys() using
> > DMA_ATTR_REQUIRE_COHERENT. Add otx2_dma_alloc_coherent() and
> > otx2_dma_free_coherent() helpers that enforce dev_is_dma_coherent(),
> > retry with GFP_DMA32 when the physical range is outside the device DMA
> > mask, and wire qmem_alloc()/qmem_free() through them instead of
> > dma_alloc_attrs()/dma_free_attrs().
>
> __get_free_pages() can only allocate memory in power-of-two pages. You can
> use alloc_pages_exact() and free_pages_exact() to save memory.
Thank you ! will take it as an enhancement to net-next tree after this patch is merged.

>
> > This works on Octeon because the octeontx2 driver is written for
> > DMA-coherent devices: Octeon platforms provide IO coherency (via SMMU), so
> > the driver already uses streaming DMA APIs for packet data while
> > deliberately skipping explicit CPU cache sync (DMA_ATTR_SKIP_CPU_SYNC).
> > The same IO coherency lets qmem use a streaming map of buddy-allocated
> > pages instead of a dedicated coherent allocator or CMA reservation. That
> > is valid because the platform is DMA-coherent, not because omitting
> > dma_sync_* magically makes memory coherent.
> >
> > Allocations requiring more than MAX_PAGE_ORDER pages are still rejected,
> > since the buddy allocator cannot serve them without CMA.
>
> Can you confirm that no existing allocations exceeds MAX_PAGE_ORDER?
Tested this patch on cn10k platforms.

>
> > cc: Geetha sowjanya <gakula@marvell.com>
> > Fixes: 73d33dbc0723 ("octeontx2-af: Use DMA_ATTR_FORCE_CONTIGUOUS attribute in DMA alloc")
> > Signed-off-by: Ratheesh Kannoth <rkannoth@marvell.com>
> >
> > ---
> >
> > v4 -> v5: Fixed compilation issues.
> > 	https://lore.kernel.org/netdev/20260831024210.208447-1-rkannoth@marvell.com/
> >
> > v3 -> v4: Fixed compilation issues.
> > 	https://lore.kernel.org/netdev/apTpKcN_S1xIwRbZ@rkannoth-OptiPlex-7090/
> >
> > v2 -> v3: Addressed sashiko comments
> > 	https://sashiko.dev/#/patchset/20260825045616.3723078-1-rkannoth%40marvell.com
> >
> > v1 -> v2: Rewrote patch as per sashiko comment
> > ---
> >   .../ethernet/marvell/octeontx2/af/common.h    | 93 +++++++++++++++++--
> >   1 file changed, 87 insertions(+), 6 deletions(-)
> >
> > diff --git a/drivers/net/ethernet/marvell/octeontx2/af/common.h b/drivers/net/ethernet/marvell/octeontx2/af/common.h
> > index 779413a383b7..061dd907f7fa 100644
> > --- a/drivers/net/ethernet/marvell/octeontx2/af/common.h
> > +++ b/drivers/net/ethernet/marvell/octeontx2/af/common.h
> > @@ -7,6 +7,11 @@
> >   #ifndef COMMON_H
> >   #define COMMON_H
> > +#include <linux/dma-mapping.h>
> > +#include <linux/dma-map-ops.h>
> > +#include <linux/gfp.h>
> > +#include <linux/mm.h>
> > +
> >   #include "rvu_struct.h"
> >   #define OTX2_ALIGN			128  /* Align to cacheline */
> > @@ -44,6 +49,83 @@ struct qmem {
> >   	u32		qsize;
> >   };
> > +/* Buddy-backed coherent DMA alloc (Option 3): pages from __get_free_pages(),
> > + * DMA-reachable RAM via dma_coherent_ok(), and bus/SMMU mappings via
> > + * dma_map_phys() -> iommu_dma_map_phys() -> iommu_map() on SMMU systems.
> > + */
> > +#define OTX2_DMA_COHERENT_ATTRS	DMA_ATTR_REQUIRE_COHERENT
> > +
> > +static inline bool otx2_dma_phys_in_mask(struct device *dev, phys_addr_t paddr,
> > +					 size_t size)
> > +{
> > +	u64 mask = dma_get_mask(dev);
> > +
> > +	return paddr + size - 1 <= mask;
> > +}
> > +
> > +static inline void *otx2_dma_alloc_coherent(struct device *dev, size_t size,
> > +					    dma_addr_t *dma_handle, gfp_t gfp)
> > +{
> > +	dma_addr_t dma_addr;
> > +	unsigned int order;
> > +	phys_addr_t paddr;
> > +	void *vaddr;
> > +	gfp_t alloc_gfp;
> > +
> > +	if (!dev || !dma_handle || !size)
> > +		return NULL;
> > +
> > +	if (!dev_is_dma_coherent(dev))
> > +		return NULL;
> > +
> > +	size = PAGE_ALIGN(size);
> > +	order = get_order(size);
> > +	if (order > MAX_PAGE_ORDER)
> > +		return NULL;
> > +
> > +	alloc_gfp = (gfp & ~(__GFP_DMA | __GFP_DMA32 | __GFP_HIGHMEM)) |
> > +		__GFP_ZERO | __GFP_COMP | __GFP_RECLAIM;
> > +
> > +	vaddr = (void *)__get_free_pages(alloc_gfp, order);
> > +	while (vaddr &&
> > +	       !otx2_dma_phys_in_mask(dev, page_to_phys(virt_to_page(vaddr)), size)) {
>
> page_to_phys(virt_to_page(vaddr)) can be simplifed to virt_to_phys(vaddr).
ACK.

>
> > +		free_pages((unsigned long)vaddr, order);
> > +		if (alloc_gfp & GFP_DMA32)
> > +			return NULL;
> > +		alloc_gfp |= GFP_DMA32;
> > +		vaddr = (void *)__get_free_pages(alloc_gfp, order);
> > +	}
> > +	if (!vaddr)
> > +		return NULL;
> > +
> > +	paddr = page_to_phys(virt_to_page(vaddr));
>
> Same here.
ACK.
>
> > +	dma_addr = dma_map_phys(dev, paddr, size, DMA_BIDIRECTIONAL,
> > +				OTX2_DMA_COHERENT_ATTRS);
> > +	if (dma_mapping_error(dev, dma_addr)) {
> > +		free_pages((unsigned long)vaddr, order);
> > +		return NULL;
> > +	}
> > +
> > +	*dma_handle = dma_addr;
> > +	return vaddr;
> > +}
> Kind regards,
> Qingfang

Re: [PATCH v5 net] octeontx2-af: switch qmem from coherent DMA alloc to streaming DMA mapping

[Leon Romanovsky]

On Tue, Sep 01, 2026 at 07:26:21AM +0530, Ratheesh Kannoth wrote:
> qmem_alloc() uses dma_alloc_attrs() with DMA_ATTR_FORCE_CONTIGUOUS, which
> allocates CPU-cache-coherent DMA memory and, with CMA enabled, draws from
> the CMA pool. qmem backs NIX/NPA queue contexts, admin queues, and LMTST
> regions (including CN10K LMTST areas that span page boundaries), so
> consumption grows with enabled interfaces and is hard to provision in CMA.
> 
> Switch qmem to a streaming-DMA-style path: allocate physically contiguous
> compound pages from the buddy allocator via __get_free_pages(), then map
> them for device access with dma_map_phys() and dma_unmap_phys() using
> DMA_ATTR_REQUIRE_COHERENT. Add otx2_dma_alloc_coherent() and
> otx2_dma_free_coherent() helpers that enforce dev_is_dma_coherent(),
> retry with GFP_DMA32 when the physical range is outside the device DMA
> mask, and wire qmem_alloc()/qmem_free() through them instead of
> dma_alloc_attrs()/dma_free_attrs().
> 
> This works on Octeon because the octeontx2 driver is written for
> DMA-coherent devices: Octeon platforms provide IO coherency (via SMMU), so
> the driver already uses streaming DMA APIs for packet data while
> deliberately skipping explicit CPU cache sync (DMA_ATTR_SKIP_CPU_SYNC).
> The same IO coherency lets qmem use a streaming map of buddy-allocated
> pages instead of a dedicated coherent allocator or CMA reservation. That
> is valid because the platform is DMA-coherent, not because omitting
> dma_sync_* magically makes memory coherent.
> 
> Allocations requiring more than MAX_PAGE_ORDER pages are still rejected,
> since the buddy allocator cannot serve them without CMA.
> 
> cc: Geetha sowjanya <gakula@marvell.com>
> Fixes: 73d33dbc0723 ("octeontx2-af: Use DMA_ATTR_FORCE_CONTIGUOUS attribute in DMA alloc")
> Signed-off-by: Ratheesh Kannoth <rkannoth@marvell.com>
> 
> ---

<...>

> +	paddr = page_to_phys(virt_to_page(vaddr));
> +	dma_addr = dma_map_phys(dev, paddr, size, DMA_BIDIRECTIONAL,
> +				OTX2_DMA_COHERENT_ATTRS);

1. Don't redefine existing DMA attribute, e.g, use DMA_COHERENT_ATTRS
directly.
2. According to the Documentation/core-api/dma-attributes.rst, most likely
you don't need DMA_COHERENT_ATTRS too.
3. There is no need to use dma_map_phys() in your case as you are
supplying struct page backed memory and don't provide DMA_ATTR_MMIO,
use dma_map_page_attrs() instead.
4. Commit message is AI slop.

Thanks

Re: [PATCH v5 net] octeontx2-af: switch qmem from coherent DMA alloc to streaming DMA mapping

[Ratheesh Kannoth]

On 2026-09-01 at 07:26:21, Ratheesh Kannoth (rkannoth@marvell.com) wrote:
> qmem_alloc() uses dma_alloc_attrs() with DMA_ATTR_FORCE_CONTIGUOUS, which
> allocates CPU-cache-coherent DMA memory and, with CMA enabled, draws from
> the CMA pool. qmem backs NIX/NPA queue contexts, admin queues, and LMTST
> regions (including CN10K LMTST areas that span page boundaries), so
> consumption grows with enabled interfaces and is hard to provision in CMA.
>
> Switch qmem to a streaming-DMA-style path: allocate physically contiguous
> compound pages from the buddy allocator via __get_free_pages(), then map
> them for device access with dma_map_phys() and dma_unmap_phys() using
> DMA_ATTR_REQUIRE_COHERENT. Add otx2_dma_alloc_coherent() and
> otx2_dma_free_coherent() helpers that enforce dev_is_dma_coherent(),
> retry with GFP_DMA32 when the physical range is outside the device DMA
> mask, and wire qmem_alloc()/qmem_free() through them instead of
> dma_alloc_attrs()/dma_free_attrs().
>
> This works on Octeon because the octeontx2 driver is written for
> DMA-coherent devices: Octeon platforms provide IO coherency (via SMMU), so
> the driver already uses streaming DMA APIs for packet data while
> deliberately skipping explicit CPU cache sync (DMA_ATTR_SKIP_CPU_SYNC).
> The same IO coherency lets qmem use a streaming map of buddy-allocated
> pages instead of a dedicated coherent allocator or CMA reservation. That
> is valid because the platform is DMA-coherent, not because omitting
> dma_sync_* magically makes memory coherent.
>
> Allocations requiring more than MAX_PAGE_ORDER pages are still rejected,
> since the buddy allocator cannot serve them without CMA.
>

Will address Qingfang, Leon comments in V6

pw-bot: changes-requested