[PATCH kernel] powerpc/pseries/ddw: Extend upper limit for huge DMA window for persistent memory

[PATCH kernel] powerpc/pseries/ddw: Extend upper limit for huge DMA window for persistent memory

[Alexey Kardashevskiy]

Unlike normal memory ("memory" compatible type in the FDT),
the persistent memory ("ibm,pmemory" in the FDT) can be mapped anywhere
in the guest physical space and it can be used for DMA.

In order to maintain 1:1 mapping via the huge DMA window, we need to
know the maximum physical address at the time of the window setup.
So far we've been looking at "memory" nodes but "ibm,pmemory" does not
have fixed addresses and the persistent memory may be mapped afterwards.

Since the persistent memory is still backed with page structs,
use MAX_PHYSMEM_BITS as the upper limit.

This effectively disables huge DMA window in LPAR under pHyp if
persistent memory is present but this is the best we can do.

Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---
 arch/powerpc/platforms/pseries/iommu.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/arch/powerpc/platforms/pseries/iommu.c b/arch/powerpc/platforms/pseries/iommu.c
index 2e0a8eab5588..6d47b4a3ce39 100644
--- a/arch/powerpc/platforms/pseries/iommu.c
+++ b/arch/powerpc/platforms/pseries/iommu.c
@@ -945,6 +945,15 @@ static phys_addr_t ddw_memory_hotplug_max(void)
 	phys_addr_t max_addr = memory_hotplug_max();
 	struct device_node *memory;
 
+	/*
+	 * The "ibm,pmemory" can appear anywhere in the address space.
+	 * Assuming it is still backed by page structs, set the upper limit
+	 * for the huge DMA window as MAX_PHYSMEM_BITS.
+	 */
+	if (of_find_node_by_type(NULL, "ibm,pmemory"))
+		return (sizeof(phys_addr_t) * 8 <= MAX_PHYSMEM_BITS) ?
+			(phys_addr_t) -1 : (1ULL << MAX_PHYSMEM_BITS);
+
 	for_each_node_by_type(memory, "memory") {
 		unsigned long start, size;
 		int n_mem_addr_cells, n_mem_size_cells, len;
-- 
2.17.1

Re: [PATCH kernel] powerpc/pseries/ddw: Extend upper limit for huge DMA window for persistent memory

[Alexey Kardashevskiy]

Hi Wen,

Can you please try this? It contains 3 patches from Christoph Hellwig
plus my patch on top, this should improve performance when DMA-ing not
to/from persistent memory:

https://github.com/aik/linux/commits/dma-bypass.3

I am looking for any benchmarks not related to persistent memory. Thanks,



On 03/04/2020 09:09, Wen Xiong wrote:
> I applied the patch on top of the latest upstream kernel. I ran HTX over
> pmem nodes for several hours and it works.
>  
> Tested-by: Wen Xiong<wenxiong@linux.vnet.ibm.com>
>  
> Thanks,
> Wendy
> 
>     ----- Original message -----
>     From: Alexey Kardashevskiy <aik@ozlabs.ru>
>     To: linuxppc-dev@lists.ozlabs.org
>     Cc: Alexey Kardashevskiy <aik@ozlabs.ru>, David Gibson
>     <david@gibson.dropbear.id.au>, Michael Ellerman
>     <mpe@ellerman.id.au>, Oliver O'Halloran <oohall@gmail.com>, "Aneesh
>     Kumar K . V" <aneesh.kumar@linux.ibm.com>, Wen Xiong
>     <wenxiong@us.ibm.com>, Brian J King <bjking1@us.ibm.com>
>     Subject: [EXTERNAL] [PATCH kernel] powerpc/pseries/ddw: Extend upper
>     limit for huge DMA window for persistent memory
>     Date: Mon, Mar 30, 2020 8:23 PM
>      
>     Unlike normal memory ("memory" compatible type in the FDT),
>     the persistent memory ("ibm,pmemory" in the FDT) can be mapped anywhere
>     in the guest physical space and it can be used for DMA.
> 
>     In order to maintain 1:1 mapping via the huge DMA window, we need to
>     know the maximum physical address at the time of the window setup.
>     So far we've been looking at "memory" nodes but "ibm,pmemory" does not
>     have fixed addresses and the persistent memory may be mapped afterwards.
> 
>     Since the persistent memory is still backed with page structs,
>     use MAX_PHYSMEM_BITS as the upper limit.
> 
>     This effectively disables huge DMA window in LPAR under pHyp if
>     persistent memory is present but this is the best we can do.
> 
>     Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
>     ---
>      arch/powerpc/platforms/pseries/iommu.c | 9 +++++++++
>      1 file changed, 9 insertions(+)
> 
>     diff --git a/arch/powerpc/platforms/pseries/iommu.c
>     b/arch/powerpc/platforms/pseries/iommu.c
>     index 2e0a8eab5588..6d47b4a3ce39 100644
>     --- a/arch/powerpc/platforms/pseries/iommu.c
>     +++ b/arch/powerpc/platforms/pseries/iommu.c
>     @@ -945,6 +945,15 @@ static phys_addr_t ddw_memory_hotplug_max(void)
>       phys_addr_t max_addr = memory_hotplug_max();
>       struct device_node *memory;
>      
>     + /*
>     + * The "ibm,pmemory" can appear anywhere in the address space.
>     + * Assuming it is still backed by page structs, set the upper limit
>     + * for the huge DMA window as MAX_PHYSMEM_BITS.
>     + */
>     + if (of_find_node_by_type(NULL, "ibm,pmemory"))
>     + return (sizeof(phys_addr_t) * 8 <= MAX_PHYSMEM_BITS) ?
>     + (phys_addr_t) -1 : (1ULL << MAX_PHYSMEM_BITS);
>     +
>       for_each_node_by_type(memory, "memory") {
>       unsigned long start, size;
>       int n_mem_addr_cells, n_mem_size_cells, len;
>     --
>     2.17.1
>      
> 
>  
> 

-- 
Alexey

Re: [PATCH kernel] powerpc/pseries/ddw: Extend upper limit for huge DMA window for persistent memory

[Vaibhav Jain]

Alexey Kardashevskiy <aik@ozlabs.ru> writes:

> Unlike normal memory ("memory" compatible type in the FDT),
> the persistent memory ("ibm,pmemory" in the FDT) can be mapped anywhere
> in the guest physical space and it can be used for DMA.
>
> In order to maintain 1:1 mapping via the huge DMA window, we need to
> know the maximum physical address at the time of the window setup.
> So far we've been looking at "memory" nodes but "ibm,pmemory" does not
> have fixed addresses and the persistent memory may be mapped afterwards.
>
> Since the persistent memory is still backed with page structs,
> use MAX_PHYSMEM_BITS as the upper limit.
>
> This effectively disables huge DMA window in LPAR under pHyp if
> persistent memory is present but this is the best we can do.
>
> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>

Thanks for the patch Alexy,
LGTM.

Reviewed-by: Vaibhav Jain <vaibhav@linux.ibm.com>

> ---
>  arch/powerpc/platforms/pseries/iommu.c | 9 +++++++++
>  1 file changed, 9 insertions(+)
>
> diff --git a/arch/powerpc/platforms/pseries/iommu.c b/arch/powerpc/platforms/pseries/iommu.c
> index 2e0a8eab5588..6d47b4a3ce39 100644
> --- a/arch/powerpc/platforms/pseries/iommu.c
> +++ b/arch/powerpc/platforms/pseries/iommu.c
> @@ -945,6 +945,15 @@ static phys_addr_t ddw_memory_hotplug_max(void)
>  	phys_addr_t max_addr = memory_hotplug_max();
>  	struct device_node *memory;
>
> +	/*
> +	 * The "ibm,pmemory" can appear anywhere in the address space.
> +	 * Assuming it is still backed by page structs, set the upper limit
> +	 * for the huge DMA window as MAX_PHYSMEM_BITS.
> +	 */
> +	if (of_find_node_by_type(NULL, "ibm,pmemory"))
> +		return (sizeof(phys_addr_t) * 8 <= MAX_PHYSMEM_BITS) ?
> +			(phys_addr_t) -1 : (1ULL << MAX_PHYSMEM_BITS);
> +
>  	for_each_node_by_type(memory, "memory") {
>  		unsigned long start, size;
>  		int n_mem_addr_cells, n_mem_size_cells, len;
> -- 
> 2.17.1
>

-- 
Vaibhav Jain <vaibhav@linux.ibm.com>
Linux Technology Center, IBM India Pvt. Ltd.

Re: [PATCH kernel] powerpc/pseries/ddw: Extend upper limit for huge DMA window for persistent memory

[Michael Ellerman]

On Tue, 2020-03-31 at 01:23:38 UTC, Alexey Kardashevskiy wrote:
> Unlike normal memory ("memory" compatible type in the FDT),
> the persistent memory ("ibm,pmemory" in the FDT) can be mapped anywhere
> in the guest physical space and it can be used for DMA.
> 
> In order to maintain 1:1 mapping via the huge DMA window, we need to
> know the maximum physical address at the time of the window setup.
> So far we've been looking at "memory" nodes but "ibm,pmemory" does not
> have fixed addresses and the persistent memory may be mapped afterwards.
> 
> Since the persistent memory is still backed with page structs,
> use MAX_PHYSMEM_BITS as the upper limit.
> 
> This effectively disables huge DMA window in LPAR under pHyp if
> persistent memory is present but this is the best we can do.
> 
> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>

Applied to powerpc next, thanks.

https://git.kernel.org/powerpc/c/54fc3c681ded9437e4548e2501dc1136b23cfa9a

cheers