linux-arm-kernel.lists.infradead.org archive mirror
 help / color / mirror / Atom feed
* Re: [PATCH] dma-contiguous: support per-numa CMA for all architectures
       [not found] <20230512094210.141540-1-yajun.deng@linux.dev>
@ 2023-05-15  9:49 ` Christoph Hellwig
  2023-05-15 11:23 ` Yajun Deng
                   ` (2 subsequent siblings)
  3 siblings, 0 replies; 10+ messages in thread
From: Christoph Hellwig @ 2023-05-15  9:49 UTC (permalink / raw)
  To: Yajun Deng
  Cc: corbet, catalin.marinas, will, hch, m.szyprowski, robin.murphy,
	paulmck, bp, akpm, peterz, rdunlap, kim.phillips, rostedt,
	thunder.leizhen, ardb, bhe, anshuman.khandual, song.bao.hua,
	linux-doc, linux-kernel, linux-arm-kernel, iommu

This looks fine to me.  Can you please work with Barry to make sure
the slight different place of the initcall doesn't break anything
for his setup?  I doubt it would, but I'd rather have a Tested-by:
tag.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH] dma-contiguous: support per-numa CMA for all architectures
       [not found] <20230512094210.141540-1-yajun.deng@linux.dev>
  2023-05-15  9:49 ` [PATCH] dma-contiguous: support per-numa CMA for all architectures Christoph Hellwig
@ 2023-05-15 11:23 ` Yajun Deng
  2023-05-15 11:38   ` Petr Tesařík
  2023-05-16  1:55 ` Yajun Deng
  2023-07-31 16:02 ` Christoph Hellwig
  3 siblings, 1 reply; 10+ messages in thread
From: Yajun Deng @ 2023-05-15 11:23 UTC (permalink / raw)
  To: Christoph Hellwig
  Cc: corbet, catalin.marinas, will, m.szyprowski, robin.murphy,
	paulmck, bp, akpm, peterz, rdunlap, kim.phillips, rostedt,
	thunder.leizhen, ardb, bhe, anshuman.khandual, song.bao.hua,
	linux-doc, linux-kernel, linux-arm-kernel, iommu

May 15, 2023 5:49 PM, "Christoph Hellwig" <hch@lst.de> wrote:

> This looks fine to me. Can you please work with Barry to make sure
> the slight different place of the initcall doesn't break anything
> for his setup? I doubt it would, but I'd rather have a Tested-by:
> tag.

Barry's email is no longer in use. I can't reach him.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH] dma-contiguous: support per-numa CMA for all architectures
  2023-05-15 11:23 ` Yajun Deng
@ 2023-05-15 11:38   ` Petr Tesařík
  2023-05-16  1:57     ` Yajun Deng
                       ` (2 more replies)
  0 siblings, 3 replies; 10+ messages in thread
From: Petr Tesařík @ 2023-05-15 11:38 UTC (permalink / raw)
  To: Yajun Deng
  Cc: Christoph Hellwig, corbet, catalin.marinas, will, m.szyprowski,
	robin.murphy, paulmck, bp, akpm, peterz, rdunlap, kim.phillips,
	rostedt, thunder.leizhen, ardb, bhe, anshuman.khandual,
	song.bao.hua, linux-doc, linux-kernel, linux-arm-kernel, iommu

On Mon, 15 May 2023 11:23:27 +0000
"Yajun Deng" <yajun.deng@linux.dev> wrote:

> May 15, 2023 5:49 PM, "Christoph Hellwig" <hch@lst.de> wrote:
> 
> > This looks fine to me. Can you please work with Barry to make sure
> > the slight different place of the initcall doesn't break anything
> > for his setup? I doubt it would, but I'd rather have a Tested-by:
> > tag.  
> 
> Barry's email is no longer in use. I can't reach him.

Which one? I would hope that his Gmail account is still valid:

  Barry Song <21cnbao@gmail.com>

Petr T

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH] dma-contiguous: support per-numa CMA for all architectures
       [not found] <20230512094210.141540-1-yajun.deng@linux.dev>
  2023-05-15  9:49 ` [PATCH] dma-contiguous: support per-numa CMA for all architectures Christoph Hellwig
  2023-05-15 11:23 ` Yajun Deng
@ 2023-05-16  1:55 ` Yajun Deng
  2023-07-31 16:02 ` Christoph Hellwig
  3 siblings, 0 replies; 10+ messages in thread
From: Yajun Deng @ 2023-05-16  1:55 UTC (permalink / raw)
  To: 21cnbao
  Cc: linux-doc, linux-kernel, linux-arm-kernel, iommu, song.bao.hua,
	anshuman.khandual, corbet, bhe, ardb, thunder.leizhen, rostedt,
	kim.phillips, rdunlap, peterz, akpm, bp, paulmck, robin.murphy,
	m.szyprowski, hch, will, catalin.marinas, petr

Hello,  Barry

This patch changed the caller of dma_pernuma_cma_reserve() from

bootmem_init() to dma_contiguous_reserve(), do you think

would there be something wrong?

On 2023/5/12 17:42, Yajun Deng wrote:
> In the commit b7176c261cdb ("dma-contiguous: provide the ability to
> reserve per-numa CMA"), Barry adds DMA_PERNUMA_CMA for ARM64.
>
> But this feature is architecture independent, so support per-numa CMA
> for all architectures, and enable it by default if NUMA.
>
> Signed-off-by: Yajun Deng <yajun.deng@linux.dev>
> ---
>   Documentation/admin-guide/kernel-parameters.txt | 2 +-
>   arch/arm64/mm/init.c                            | 2 --
>   include/linux/dma-map-ops.h                     | 6 ------
>   kernel/dma/Kconfig                              | 6 +++---
>   kernel/dma/contiguous.c                         | 8 +++++++-
>   5 files changed, 11 insertions(+), 13 deletions(-)
>
> diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
> index 56d9458276a6..ac0002b2e323 100644
> --- a/Documentation/admin-guide/kernel-parameters.txt
> +++ b/Documentation/admin-guide/kernel-parameters.txt
> @@ -692,7 +692,7 @@
>   			kernel/dma/contiguous.c
>   
>   	cma_pernuma=nn[MG]
> -			[ARM64,KNL,CMA]
> +			[KNL,CMA]
>   			Sets the size of kernel per-numa memory area for
>   			contiguous memory allocations. A value of 0 disables
>   			per-numa CMA altogether. And If this option is not
> diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
> index 66e70ca47680..d560aef6aafa 100644
> --- a/arch/arm64/mm/init.c
> +++ b/arch/arm64/mm/init.c
> @@ -410,8 +410,6 @@ void __init bootmem_init(void)
>   	arm64_hugetlb_cma_reserve();
>   #endif
>   
> -	dma_pernuma_cma_reserve();
> -
>   	kvm_hyp_reserve();
>   
>   	/*
> diff --git a/include/linux/dma-map-ops.h b/include/linux/dma-map-ops.h
> index 31f114f486c4..7af9949828ff 100644
> --- a/include/linux/dma-map-ops.h
> +++ b/include/linux/dma-map-ops.h
> @@ -168,12 +168,6 @@ static inline void dma_free_contiguous(struct device *dev, struct page *page,
>   }
>   #endif /* CONFIG_DMA_CMA*/
>   
> -#ifdef CONFIG_DMA_PERNUMA_CMA
> -void dma_pernuma_cma_reserve(void);
> -#else
> -static inline void dma_pernuma_cma_reserve(void) { }
> -#endif /* CONFIG_DMA_PERNUMA_CMA */
> -
>   #ifdef CONFIG_DMA_DECLARE_COHERENT
>   int dma_declare_coherent_memory(struct device *dev, phys_addr_t phys_addr,
>   		dma_addr_t device_addr, size_t size);
> diff --git a/kernel/dma/Kconfig b/kernel/dma/Kconfig
> index 6677d0e64d27..79f83091e3a2 100644
> --- a/kernel/dma/Kconfig
> +++ b/kernel/dma/Kconfig
> @@ -140,10 +140,10 @@ if  DMA_CMA
>   
>   config DMA_PERNUMA_CMA
>   	bool "Enable separate DMA Contiguous Memory Area for each NUMA Node"
> -	default NUMA && ARM64
> +	default NUMA
>   	help
> -	  Enable this option to get pernuma CMA areas so that devices like
> -	  ARM64 SMMU can get local memory by DMA coherent APIs.
> +	  Enable this option to get pernuma CMA areas so that NUMA devices
> +	  can get local memory by DMA coherent APIs.
>   
>   	  You can set the size of pernuma CMA by specifying "cma_pernuma=size"
>   	  on the kernel's command line.
> diff --git a/kernel/dma/contiguous.c b/kernel/dma/contiguous.c
> index 6ea80ae42622..26a8e5365fcd 100644
> --- a/kernel/dma/contiguous.c
> +++ b/kernel/dma/contiguous.c
> @@ -128,7 +128,7 @@ static inline __maybe_unused phys_addr_t cma_early_percent_memory(void)
>   #endif
>   
>   #ifdef CONFIG_DMA_PERNUMA_CMA
> -void __init dma_pernuma_cma_reserve(void)
> +static void __init dma_pernuma_cma_reserve(void)
>   {
>   	int nid;
>   
> @@ -153,6 +153,10 @@ void __init dma_pernuma_cma_reserve(void)
>   			(unsigned long long)pernuma_size_bytes / SZ_1M, nid);
>   	}
>   }
> +#else
> +static inline void __init dma_pernuma_cma_reserve(void)
> +{
> +}
>   #endif
>   
>   /**
> @@ -171,6 +175,8 @@ void __init dma_contiguous_reserve(phys_addr_t limit)
>   	phys_addr_t selected_limit = limit;
>   	bool fixed = false;
>   
> +	dma_pernuma_cma_reserve();
> +
>   	pr_debug("%s(limit %08lx)\n", __func__, (unsigned long)limit);
>   
>   	if (size_cmdline != -1) {

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH] dma-contiguous: support per-numa CMA for all architectures
  2023-05-15 11:38   ` Petr Tesařík
@ 2023-05-16  1:57     ` Yajun Deng
  2023-06-24  0:40     ` Andrew Morton
  2023-06-25  7:30     ` Yajun Deng
  2 siblings, 0 replies; 10+ messages in thread
From: Yajun Deng @ 2023-05-16  1:57 UTC (permalink / raw)
  To: Petr Tesařík
  Cc: Christoph Hellwig, corbet, catalin.marinas, will, m.szyprowski,
	robin.murphy, paulmck, bp, akpm, peterz, rdunlap, kim.phillips,
	rostedt, thunder.leizhen, ardb, bhe, anshuman.khandual,
	song.bao.hua, linux-doc, linux-kernel, linux-arm-kernel, iommu,
	21cnbao


On 2023/5/15 19:38, Petr Tesařík wrote:
> On Mon, 15 May 2023 11:23:27 +0000
> "Yajun Deng" <yajun.deng@linux.dev> wrote:
>
>> May 15, 2023 5:49 PM, "Christoph Hellwig" <hch@lst.de> wrote:
>>
>>> This looks fine to me. Can you please work with Barry to make sure
>>> the slight different place of the initcall doesn't break anything
>>> for his setup? I doubt it would, but I'd rather have a Tested-by:
>>> tag.
>> Barry's email is no longer in use. I can't reach him.
> Which one? I would hope that his Gmail account is still valid:
>
>    Barry Song <21cnbao@gmail.com>
   Thanks.
>
> Petr T

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH] dma-contiguous: support per-numa CMA for all architectures
  2023-05-15 11:38   ` Petr Tesařík
  2023-05-16  1:57     ` Yajun Deng
@ 2023-06-24  0:40     ` Andrew Morton
  2023-06-25  7:30     ` Yajun Deng
  2 siblings, 0 replies; 10+ messages in thread
From: Andrew Morton @ 2023-06-24  0:40 UTC (permalink / raw)
  To: Petr Tesařík
  Cc: Yajun Deng, Christoph Hellwig, corbet, catalin.marinas, will,
	m.szyprowski, robin.murphy, paulmck, bp, peterz, rdunlap,
	kim.phillips, rostedt, thunder.leizhen, ardb, bhe,
	anshuman.khandual, song.bao.hua, linux-doc, linux-kernel,
	linux-arm-kernel, iommu, Barry Song

On Mon, 15 May 2023 13:38:21 +0200 Petr Tesařík <petr@tesarici.cz> wrote:

> On Mon, 15 May 2023 11:23:27 +0000
> "Yajun Deng" <yajun.deng@linux.dev> wrote:
> 
> > May 15, 2023 5:49 PM, "Christoph Hellwig" <hch@lst.de> wrote:
> > 
> > > This looks fine to me. Can you please work with Barry to make sure
> > > the slight different place of the initcall doesn't break anything
> > > for his setup? I doubt it would, but I'd rather have a Tested-by:
> > > tag.  
> > 
> > Barry's email is no longer in use. I can't reach him.
> 
> Which one? I would hope that his Gmail account is still valid:
> 
>   Barry Song <21cnbao@gmail.com>
> 

Maybe his kernel.org address works...

I have this patch stuck in limbo for 6.4.  I guess I'll carry it over
into the next -rc cycle, see what happens.

fwiw, it has been in -next for six weeks, no known issues.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH] dma-contiguous: support per-numa CMA for all  architectures
  2023-05-15 11:38   ` Petr Tesařík
  2023-05-16  1:57     ` Yajun Deng
  2023-06-24  0:40     ` Andrew Morton
@ 2023-06-25  7:30     ` Yajun Deng
  2023-06-26  5:32       ` Barry Song
  2 siblings, 1 reply; 10+ messages in thread
From: Yajun Deng @ 2023-06-25  7:30 UTC (permalink / raw)
  To: v-songbaohua
  Cc: Christoph Hellwig, corbet, catalin.marinas, will, m.szyprowski,
	robin.murphy, paulmck, bp, peterz, rdunlap, kim.phillips, rostedt,
	thunder.leizhen, ardb, bhe, anshuman.khandual, song.bao.hua,
	linux-doc, linux-kernel, linux-arm-kernel, iommu,
	Petr Tesařík, Andrew Morton, Barry Song, 21cnbao

June 24, 2023 8:40 AM, "Andrew Morton" <akpm@linux-foundation.org> wrote:

> On Mon, 15 May 2023 13:38:21 +0200 Petr Tesařík <petr@tesarici.cz> wrote:
> 
>> On Mon, 15 May 2023 11:23:27 +0000
>> "Yajun Deng" <yajun.deng@linux.dev> wrote:
>> 
>> May 15, 2023 5:49 PM, "Christoph Hellwig" <hch@lst.de> wrote:
> 
> This looks fine to me. Can you please work with Barry to make sure
> the slight different place of the initcall doesn't break anything
> for his setup? I doubt it would, but I'd rather have a Tested-by:
> tag.
>> Barry's email is no longer in use. I can't reach him.
>> 
>> Which one? I would hope that his Gmail account is still valid:
>> 
>> Barry Song <21cnbao@gmail.com>
> 
> Maybe his kernel.org address works...
> 
> I have this patch stuck in limbo for 6.4. I guess I'll carry it over
> into the next -rc cycle, see what happens.
> 
> fwiw, it has been in -next for six weeks, no known issues.

Hi, Barry, The slight different place of the initcall, does break anything?

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH] dma-contiguous: support per-numa CMA for all architectures
  2023-06-25  7:30     ` Yajun Deng
@ 2023-06-26  5:32       ` Barry Song
  2023-06-29  1:41         ` Yicong Yang
  0 siblings, 1 reply; 10+ messages in thread
From: Barry Song @ 2023-06-26  5:32 UTC (permalink / raw)
  To: Yajun Deng, Yicong Yang, Tian Tao
  Cc: v-songbaohua, Christoph Hellwig, corbet, catalin.marinas, will,
	m.szyprowski, robin.murphy, paulmck, bp, peterz, rdunlap,
	kim.phillips, rostedt, thunder.leizhen, ardb, bhe,
	anshuman.khandual, song.bao.hua, linux-doc, linux-kernel,
	linux-arm-kernel, iommu, Petr Tesařík, Andrew Morton

On Sun, Jun 25, 2023 at 7:30 PM Yajun Deng <yajun.deng@linux.dev> wrote:
>
> June 24, 2023 8:40 AM, "Andrew Morton" <akpm@linux-foundation.org> wrote:
>
> > On Mon, 15 May 2023 13:38:21 +0200 Petr Tesařík <petr@tesarici.cz> wrote:
> >
> >> On Mon, 15 May 2023 11:23:27 +0000
> >> "Yajun Deng" <yajun.deng@linux.dev> wrote:
> >>
> >> May 15, 2023 5:49 PM, "Christoph Hellwig" <hch@lst.de> wrote:
> >
> > This looks fine to me. Can you please work with Barry to make sure
> > the slight different place of the initcall doesn't break anything
> > for his setup? I doubt it would, but I'd rather have a Tested-by:
> > tag.
> >> Barry's email is no longer in use. I can't reach him.
> >>
> >> Which one? I would hope that his Gmail account is still valid:
> >>
> >> Barry Song <21cnbao@gmail.com>
> >
> > Maybe his kernel.org address works...
> >
> > I have this patch stuck in limbo for 6.4. I guess I'll carry it over
> > into the next -rc cycle, see what happens.
> >
> > fwiw, it has been in -next for six weeks, no known issues.
>
> Hi, Barry, The slight different place of the initcall, does break anything?

i don't see a fundamental difference as anyway it is still after
arch_numa_init()
which is really what we depend on.

and i did a test on qemu with the command line:
qemu-system-aarch64 -M virt,gic-version=3 -nographic \
 -smp cpus=8 \
 -numa node,cpus=0-1,nodeid=0 \
 -numa node,cpus=2-3,nodeid=1 \
 -numa node,cpus=4-5,nodeid=2 \
 -numa node,cpus=6-7,nodeid=3 \
 -numa dist,src=0,dst=1,val=12 \
 -numa dist,src=0,dst=2,val=20 \
 -numa dist,src=0,dst=3,val=22 \
 -numa dist,src=1,dst=2,val=22 \
 -numa dist,src=2,dst=3,val=12 \
 -numa dist,src=1,dst=3,val=24 \
 -m 4096M -cpu cortex-a57 -kernel arch/arm64/boot/Image \
 -nographic -append "cma_pernuma=32M root=/dev/vda2  rw ip=dhcp
sched_debug irqchip.gicv3_pseudo_nmi=1" \
 -drive if=none,file=extra/ubuntu16.04-arm64.img,id=hd0 -device
virtio-blk-device,drive=hd0 \
 -net nic -net user,hostfwd=tcp::2222-:22

and in system, i can see all cma areas are correctly reserved:
~# dmesg | grep cma
[    0.000000] cma: cma_declare_contiguous_nid(size
0x0000000002000000, base 0x0000000000000000, limit 0x0000000000000000
alignment 0x0000000000000000)
[    0.000000] cma: Reserved 32 MiB at 0x000000007ce00000
[    0.000000] cma: dma_pernuma_cma_reserve: reserved 32 MiB on node 0
[    0.000000] cma: cma_declare_contiguous_nid(size
0x0000000002000000, base 0x0000000000000000, limit 0x0000000000000000
alignment 0x0000000000000000)
[    0.000000] cma: Reserved 32 MiB at 0x00000000bce00000
[    0.000000] cma: dma_pernuma_cma_reserve: reserved 32 MiB on node 1
[    0.000000] cma: cma_declare_contiguous_nid(size
0x0000000002000000, base 0x0000000000000000, limit 0x0000000000000000
alignment 0x0000000000000000)
[    0.000000] cma: Reserved 32 MiB at 0x00000000fce00000
[    0.000000] cma: dma_pernuma_cma_reserve: reserved 32 MiB on node 2
[    0.000000] cma: cma_declare_contiguous_nid(size
0x0000000002000000, base 0x0000000000000000, limit 0x0000000000000000
alignment 0x0000000000000000)
[    0.000000] cma: Reserved 32 MiB at 0x0000000100000000
[    0.000000] cma: dma_pernuma_cma_reserve: reserved 32 MiB on node 3
[    0.000000] cma: dma_contiguous_reserve(limit 100000000)
[    0.000000] cma: dma_contiguous_reserve: reserving 32 MiB for global area
[    0.000000] cma: cma_declare_contiguous_nid(size
0x0000000002000000, base 0x0000000000000000, limit 0x0000000100000000
alignment 0x0000000000000000)
[    0.000000] cma: Reserved 32 MiB at 0x00000000fae00000
[    0.000000] Kernel command line: cma_pernuma=32M root=/dev/vda2  rw
ip=dhcp sched_debug irqchip.gicv3_pseudo_nmi=1
[    0.000000] Memory: 3848784K/4194304K available (16128K kernel
code, 4152K rwdata, 10244K rodata, 8512K init, 612K bss, 181680K
reserved, 163840K cma-reserved)
[    0.175309] cma: cma_alloc(cma (____ptrval____), count 128, align 7)
[    0.179264] cma: cma_alloc(): returned (____ptrval____)
[    0.179869] cma: cma_alloc(cma (____ptrval____), count 128, align 7)
[    0.180027] cma: cma_alloc(): returned (____ptrval____)
[    0.180187] cma: cma_alloc(cma (____ptrval____), count 128, align 7)
[    0.180374] cma: cma_alloc(): returned (____ptrval____)

so my feeling is that this patch is fine. but I would prefer Yicong
and Tiantao who have a real numa machine
and we can get some real device drivers to call dma APIs to allocate
memory from pernuma cma on arm64
even though it is 99.9% OK.

With their testing done, please feel free to add
Acked-by: Barry Song <baohua@kernel.org>

Thanks
Barry

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH] dma-contiguous: support per-numa CMA for all architectures
  2023-06-26  5:32       ` Barry Song
@ 2023-06-29  1:41         ` Yicong Yang
  0 siblings, 0 replies; 10+ messages in thread
From: Yicong Yang @ 2023-06-29  1:41 UTC (permalink / raw)
  To: Barry Song, Yajun Deng, Yicong Yang, Tian Tao
  Cc: v-songbaohua, Christoph Hellwig, corbet, catalin.marinas, will,
	m.szyprowski, robin.murphy, paulmck, bp, peterz, rdunlap,
	kim.phillips, rostedt, thunder.leizhen, ardb, bhe,
	anshuman.khandual, song.bao.hua, linux-doc, linux-kernel,
	linux-arm-kernel, iommu, Petr Tesařík, Andrew Morton

On 2023/6/26 13:32, Barry Song wrote:
> On Sun, Jun 25, 2023 at 7:30 PM Yajun Deng <yajun.deng@linux.dev> wrote:
>>
>> June 24, 2023 8:40 AM, "Andrew Morton" <akpm@linux-foundation.org> wrote:
>>
>>> On Mon, 15 May 2023 13:38:21 +0200 Petr Tesařík <petr@tesarici.cz> wrote:
>>>
>>>> On Mon, 15 May 2023 11:23:27 +0000
>>>> "Yajun Deng" <yajun.deng@linux.dev> wrote:
>>>>
>>>> May 15, 2023 5:49 PM, "Christoph Hellwig" <hch@lst.de> wrote:
>>>
>>> This looks fine to me. Can you please work with Barry to make sure
>>> the slight different place of the initcall doesn't break anything
>>> for his setup? I doubt it would, but I'd rather have a Tested-by:
>>> tag.
>>>> Barry's email is no longer in use. I can't reach him.
>>>>
>>>> Which one? I would hope that his Gmail account is still valid:
>>>>
>>>> Barry Song <21cnbao@gmail.com>
>>>
>>> Maybe his kernel.org address works...
>>>
>>> I have this patch stuck in limbo for 6.4. I guess I'll carry it over
>>> into the next -rc cycle, see what happens.
>>>
>>> fwiw, it has been in -next for six weeks, no known issues.
>>
>> Hi, Barry, The slight different place of the initcall, does break anything?
> 
> i don't see a fundamental difference as anyway it is still after
> arch_numa_init()
> which is really what we depend on.
> 
> and i did a test on qemu with the command line:
> qemu-system-aarch64 -M virt,gic-version=3 -nographic \
>  -smp cpus=8 \
>  -numa node,cpus=0-1,nodeid=0 \
>  -numa node,cpus=2-3,nodeid=1 \
>  -numa node,cpus=4-5,nodeid=2 \
>  -numa node,cpus=6-7,nodeid=3 \
>  -numa dist,src=0,dst=1,val=12 \
>  -numa dist,src=0,dst=2,val=20 \
>  -numa dist,src=0,dst=3,val=22 \
>  -numa dist,src=1,dst=2,val=22 \
>  -numa dist,src=2,dst=3,val=12 \
>  -numa dist,src=1,dst=3,val=24 \
>  -m 4096M -cpu cortex-a57 -kernel arch/arm64/boot/Image \
>  -nographic -append "cma_pernuma=32M root=/dev/vda2  rw ip=dhcp
> sched_debug irqchip.gicv3_pseudo_nmi=1" \
>  -drive if=none,file=extra/ubuntu16.04-arm64.img,id=hd0 -device
> virtio-blk-device,drive=hd0 \
>  -net nic -net user,hostfwd=tcp::2222-:22
> 
> and in system, i can see all cma areas are correctly reserved:
> ~# dmesg | grep cma
> [    0.000000] cma: cma_declare_contiguous_nid(size
> 0x0000000002000000, base 0x0000000000000000, limit 0x0000000000000000
> alignment 0x0000000000000000)
> [    0.000000] cma: Reserved 32 MiB at 0x000000007ce00000
> [    0.000000] cma: dma_pernuma_cma_reserve: reserved 32 MiB on node 0
> [    0.000000] cma: cma_declare_contiguous_nid(size
> 0x0000000002000000, base 0x0000000000000000, limit 0x0000000000000000
> alignment 0x0000000000000000)
> [    0.000000] cma: Reserved 32 MiB at 0x00000000bce00000
> [    0.000000] cma: dma_pernuma_cma_reserve: reserved 32 MiB on node 1
> [    0.000000] cma: cma_declare_contiguous_nid(size
> 0x0000000002000000, base 0x0000000000000000, limit 0x0000000000000000
> alignment 0x0000000000000000)
> [    0.000000] cma: Reserved 32 MiB at 0x00000000fce00000
> [    0.000000] cma: dma_pernuma_cma_reserve: reserved 32 MiB on node 2
> [    0.000000] cma: cma_declare_contiguous_nid(size
> 0x0000000002000000, base 0x0000000000000000, limit 0x0000000000000000
> alignment 0x0000000000000000)
> [    0.000000] cma: Reserved 32 MiB at 0x0000000100000000
> [    0.000000] cma: dma_pernuma_cma_reserve: reserved 32 MiB on node 3
> [    0.000000] cma: dma_contiguous_reserve(limit 100000000)
> [    0.000000] cma: dma_contiguous_reserve: reserving 32 MiB for global area
> [    0.000000] cma: cma_declare_contiguous_nid(size
> 0x0000000002000000, base 0x0000000000000000, limit 0x0000000100000000
> alignment 0x0000000000000000)
> [    0.000000] cma: Reserved 32 MiB at 0x00000000fae00000
> [    0.000000] Kernel command line: cma_pernuma=32M root=/dev/vda2  rw
> ip=dhcp sched_debug irqchip.gicv3_pseudo_nmi=1
> [    0.000000] Memory: 3848784K/4194304K available (16128K kernel
> code, 4152K rwdata, 10244K rodata, 8512K init, 612K bss, 181680K
> reserved, 163840K cma-reserved)
> [    0.175309] cma: cma_alloc(cma (____ptrval____), count 128, align 7)
> [    0.179264] cma: cma_alloc(): returned (____ptrval____)
> [    0.179869] cma: cma_alloc(cma (____ptrval____), count 128, align 7)
> [    0.180027] cma: cma_alloc(): returned (____ptrval____)
> [    0.180187] cma: cma_alloc(cma (____ptrval____), count 128, align 7)
> [    0.180374] cma: cma_alloc(): returned (____ptrval____)
> 
> so my feeling is that this patch is fine. but I would prefer Yicong
> and Tiantao who have a real numa machine
> and we can get some real device drivers to call dma APIs to allocate
> memory from pernuma cma on arm64
> even though it is 99.9% OK.
> 

Tested on our 4 NUMA arm64 server based on mainline commit 1ef6663a587b,
this patch works well, so:

Tested-by: Yicong Yang <yangyicong@hisilicon.com>

For pernuma cma reservation:
[    0.000000] cma: cma_declare_contiguous_nid(size 0x0000000040000000, base 0x0000000000000000, limit 0x0000000000000000 alignment 0x0000000000000000)
[    0.000000] cma: Reserved 1024 MiB at 0x0000002081800000
[    0.000000] cma: dma_pernuma_cma_reserve: reserved 1024 MiB on node 0
[    0.000000] cma: cma_declare_contiguous_nid(size 0x0000000040000000, base 0x0000000000000000, limit 0x0000000000000000 alignment 0x0000000000000000)
[    0.000000] cma: Reserved 1024 MiB at 0x0000004000000000
[    0.000000] cma: dma_pernuma_cma_reserve: reserved 1024 MiB on node 1
[    0.000000] cma: cma_declare_contiguous_nid(size 0x0000000040000000, base 0x0000000000000000, limit 0x0000000000000000 alignment 0x0000000000000000)
[    0.000000] cma: Reserved 1024 MiB at 0x0000202000000000
[    0.000000] cma: dma_pernuma_cma_reserve: reserved 1024 MiB on node 2
[    0.000000] cma: cma_declare_contiguous_nid(size 0x0000000040000000, base 0x0000000000000000, limit 0x0000000000000000 alignment 0x0000000000000000)
[    0.000000] cma: Reserved 1024 MiB at 0x0000204000000000
[    0.000000] cma: dma_pernuma_cma_reserve: reserved 1024 MiB on node 3
[    0.000000] cma: dma_contiguous_reserve(limit 100000000)
[    0.000000] cma: dma_contiguous_reserve: reserving 384 MiB for global area
[    0.000000] cma: cma_declare_contiguous_nid(size 0x0000000018000000, base 0x0000000000000000, limit 0x0000000100000000 alignment 0x0000000000000000)
[    0.000000] cma: Reserved 384 MiB at 0x0000000068000000

For allocation from pernuma cma, no failure recorded:
[root@localhost cma]# pwd
/sys/kernel/mm/cma
[root@localhost cma]# ls
pernuma0  pernuma1  pernuma2  pernuma3  reserved
[root@localhost cma]# cat pernuma*/alloc_pages_fail
0
0
0
0
[root@localhost cma]# cat pernuma*/alloc_pages_success
2144
0
2132
0

Thanks.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH] dma-contiguous: support per-numa CMA for all architectures
       [not found] <20230512094210.141540-1-yajun.deng@linux.dev>
                   ` (2 preceding siblings ...)
  2023-05-16  1:55 ` Yajun Deng
@ 2023-07-31 16:02 ` Christoph Hellwig
  3 siblings, 0 replies; 10+ messages in thread
From: Christoph Hellwig @ 2023-07-31 16:02 UTC (permalink / raw)
  To: Yajun Deng
  Cc: corbet, catalin.marinas, will, hch, m.szyprowski, robin.murphy,
	paulmck, bp, akpm, peterz, rdunlap, kim.phillips, rostedt,
	thunder.leizhen, ardb, bhe, anshuman.khandual, song.bao.hua,
	linux-doc, linux-kernel, linux-arm-kernel, iommu

Thanks,

applied to the dma-mapping tree for 6.6.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 10+ messages in thread

end of thread, other threads:[~2023-07-31 16:03 UTC | newest]

Thread overview: 10+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
     [not found] <20230512094210.141540-1-yajun.deng@linux.dev>
2023-05-15  9:49 ` [PATCH] dma-contiguous: support per-numa CMA for all architectures Christoph Hellwig
2023-05-15 11:23 ` Yajun Deng
2023-05-15 11:38   ` Petr Tesařík
2023-05-16  1:57     ` Yajun Deng
2023-06-24  0:40     ` Andrew Morton
2023-06-25  7:30     ` Yajun Deng
2023-06-26  5:32       ` Barry Song
2023-06-29  1:41         ` Yicong Yang
2023-05-16  1:55 ` Yajun Deng
2023-07-31 16:02 ` Christoph Hellwig

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).