Re: [PATCH for-next 1/3] RDMA/erdma: Use dma_map_page to map scatter MTT buffer

[Leon Romanovsky <leon@kernel.org>]

On Mon, Jul 28, 2025 at 03:47:57PM +0800, Cheng Xu wrote:
> 
> 
> On 7/28/25 3:08 PM, Leon Romanovsky wrote:
> > On Mon, Jul 28, 2025 at 11:08:46AM +0800, Boshi Yu wrote:
> >>
> >>
> >> On 2025/7/27 19:27, Leon Romanovsky wrote:
> >>> On Fri, Jul 25, 2025 at 01:53:54PM +0800, Boshi Yu wrote:
> >>>> Each high-level indirect MTT entry is assumed to point to exactly one page
> >>>> of the low-level MTT buffer, but dma_map_sg may merge contiguous physical
> >>>> pages when mapping. To avoid extra overhead from splitting merged regions,
> >>>> use dma_map_page to map the scatter MTT buffer page by page.
> >>>>
> >>>> Reviewed-by: Cheng Xu <chengyou@linux.alibaba.com>
> >>>> Signed-off-by: Boshi Yu <boshiyu@linux.alibaba.com>
> >>>> ---
> >>>>   drivers/infiniband/hw/erdma/erdma_verbs.c | 110 ++++++++++++++--------
> >>>>   drivers/infiniband/hw/erdma/erdma_verbs.h |   4 +-
> >>>>   2 files changed, 71 insertions(+), 43 deletions(-)
> >>>
> >>> <...>
> >>>
> >>>> +	pg_dma = vzalloc(npages * sizeof(dma_addr_t));
> >>>> +	if (!pg_dma)
> >>>> +		return 0;
> >>>> -	sg_init_table(sglist, npages);
> >>>> +	addr = buf;
> >>>>   	for (i = 0; i < npages; i++) {
> >>>> -		pg = vmalloc_to_page(buf);
> >>>> +		pg = vmalloc_to_page(addr);
> >>>
> >>> <...>
> >>>> +
> >>>> +		pg_dma[i] = dma_map_page(&dev->pdev->dev, pg, 0, PAGE_SIZE,
> >>>> +					 DMA_TO_DEVICE);
> >>>
> >>> Does it work?
> >>
> >> Hi Leon,
> >>
> >> I would like to confirm which part you think is not working properly. I
> >> guess that you might be concerned that if the buffer is not page-aligned, it
> >> could cause problems with dma_map_page.
> >>
> >> In fact, when allocating the MTT buffer, we ensure that it is always
> >> page-aligned and that its length is a multiple of PAGE_SIZE. We have also
> >> tested the new code in our production environment, and it works well.
> >>
> >> Look forward to your further reply if I have misunderstood your concerns.
> > 
> > DMA API expects Kmalloc addresses and not Vmalloc ones.
> > 
> 
> Hi Leon,
> 
> Thanks for your reply. Could you provide some references for this point?
> We cannot find the constraint in the Kernel Documentation.

Documentation/core-api/dma-api.rst.

The reason to such constraint is a need to get contiguous memory, which
vzalloc doesn't guarantee. In your case, it works because you have
vmalloc_to_page() call.

So everything is ok, sorry for the noise.

Thanks