[PATCH 0/2] Enable ZONE_DEVICE on POWER

[PATCH 0/2] Enable ZONE_DEVICE on POWER

[Anshuman Khandual]

This enables base ZONE_DEVICE support on POWER. This series depends on
the following patches posted by Oliver.

https://patchwork.ozlabs.org/patch/618867/
https://patchwork.ozlabs.org/patch/618868/

Anshuman Khandual (2):
  powerpc/mm: Make vmemmap_populate accommodate ZONE_DEVICE memory
  powerpc/mm: Enable support for ZONE_DEVICE on PPC_BOOK3S_64 platforms

 arch/powerpc/mm/init_64.c | 4 +++-
 mm/Kconfig                | 2 +-
 2 files changed, 4 insertions(+), 2 deletions(-)

-- 
1.9.3

[PATCH 1/2] powerpc/mm: Make vmemmap_populate accommodate ZONE_DEVICE memory

[Anshuman Khandual]

Change the vmemmap_populate function to detect device memory through
to_vmemmap_altmap and then call generic the __vmmemap_alloc_block_buf
function instead of vmemmap_alloc_block as the earlier can allocate
physical memory from the device range instead of the system RAM.

Signed-off-by: Anshuman Khandual <khandual@linux.vnet.ibm.com>
---
 arch/powerpc/mm/init_64.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/arch/powerpc/mm/init_64.c b/arch/powerpc/mm/init_64.c
index ba65566..9ae3cdc 100644
--- a/arch/powerpc/mm/init_64.c
+++ b/arch/powerpc/mm/init_64.c
@@ -42,6 +42,7 @@
 #include <linux/memblock.h>
 #include <linux/hugetlb.h>
 #include <linux/slab.h>
+#include <linux/memremap.h>
 
 #include <asm/pgalloc.h>
 #include <asm/page.h>
@@ -312,6 +313,7 @@ static __meminit void vmemmap_list_populate(unsigned long phys,
 int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node)
 {
 	unsigned long page_size = 1 << mmu_psize_defs[mmu_vmemmap_psize].shift;
+	struct vmem_altmap *altmap = to_vmem_altmap((unsigned long) start);
 
 	/* Align to the page size of the linear mapping. */
 	start = _ALIGN_DOWN(start, page_size);
@@ -325,7 +327,7 @@ int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node)
 		if (vmemmap_populated(start, page_size))
 			continue;
 
-		p = vmemmap_alloc_block(page_size, node);
+		p = __vmemmap_alloc_block_buf(page_size, node, altmap);
 		if (!p)
 			return -ENOMEM;
 
-- 
1.9.3

[PATCH 2/2] powerpc/mm: Enable support for ZONE_DEVICE on PPC_BOOK3S_64 platforms

[Anshuman Khandual]

This enables support for the new ZONE_DEVCICE on PPC_BOOK3S_64 platforms
which now accommodates device memory during memory hotplug operation.

Signed-off-by: Oliver O'Halloran <oohall@gmail.com>
Signed-off-by: Anshuman Khandual <khandual@linux.vnet.ibm.com>
---
NOTE: We need to enable CONFIG_EXPERT first before attempting to enable
CONFIG_ZONE_DEVICE, which will work when all of it's dependent configs
are enabled.

 mm/Kconfig | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mm/Kconfig b/mm/Kconfig
index 989f8f3..8ecd869 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -654,7 +654,7 @@ config ZONE_DEVICE
 	depends on MEMORY_HOTPLUG
 	depends on MEMORY_HOTREMOVE
 	depends on SPARSEMEM_VMEMMAP
-	depends on X86_64 #arch_add_memory() comprehends device memory
+	depends on (X86_64 || PPC_BOOK3S_64) #arch_add_memory() comprehends device memory
 
 	help
 	  Device memory hotplug support allows for establishing pmem,
-- 
1.9.3

Re: [PATCH 0/2] Enable ZONE_DEVICE on POWER

[Aneesh Kumar K.V]

Anshuman Khandual <khandual@linux.vnet.ibm.com> writes:

> This enables base ZONE_DEVICE support on POWER. This series depends on
> the following patches posted by Oliver.
>
> https://patchwork.ozlabs.org/patch/618867/
> https://patchwork.ozlabs.org/patch/618868/
>
> Anshuman Khandual (2):
>   powerpc/mm: Make vmemmap_populate accommodate ZONE_DEVICE memory
>   powerpc/mm: Enable support for ZONE_DEVICE on PPC_BOOK3S_64 platforms
>
>  arch/powerpc/mm/init_64.c | 4 +++-
>  mm/Kconfig                | 2 +-
>  2 files changed, 4 insertions(+), 2 deletions(-)
>

What is the use case ? Who will use ZONE_DEVICE on ppc64. This should be
be merged along with the patch series that use this. 

-aneesh

Re: [PATCH 0/2] Enable ZONE_DEVICE on POWER

[Anshuman Khandual]

On 05/05/2016 08:18 PM, Aneesh Kumar K.V wrote:
> Anshuman Khandual <khandual@linux.vnet.ibm.com> writes:
> 
>> This enables base ZONE_DEVICE support on POWER. This series depends on
>> the following patches posted by Oliver.
>>
>> https://patchwork.ozlabs.org/patch/618867/
>> https://patchwork.ozlabs.org/patch/618868/
>>
>> Anshuman Khandual (2):
>>   powerpc/mm: Make vmemmap_populate accommodate ZONE_DEVICE memory
>>   powerpc/mm: Enable support for ZONE_DEVICE on PPC_BOOK3S_64 platforms
>>
>>  arch/powerpc/mm/init_64.c | 4 +++-
>>  mm/Kconfig                | 2 +-
>>  2 files changed, 4 insertions(+), 2 deletions(-)
>>
> 
> What is the use case ? Who will use ZONE_DEVICE on ppc64. This should be
> be merged along with the patch series that use this. 

IIUC, Oliver has been looking at using ZONE_DEVICE for the NVDIMM (or
some other persistent memory) drivers. I have been following Dan William's
work on this front and want to explore more details about it's functioning
on ppc64. This enablement will just help us little bit in that direction.

Re: [PATCH 0/2] Enable ZONE_DEVICE on POWER

[oliver]

Hi,

I've been working on kernel support for a persistent memory (nvdimm)
device and the kernel driver infrastructure requires ZONE_DEVICE for
DAX support.. I've had it enabled in my tree for some time (without
altmap support) without any real issues.

I wasn't planning on upstreaming any of my changes until 4.8 at the
earliest so I am ok with carrying these patches myself. However, there
has been some interest in using ZONE_DEVICE for other things on ppc
(wasn't that you?) and given that ZONE_DEVICE is gated behind
CONFIG_EXPERT I can't see there being any kind of negative impact on
end users by merging it now. At the very least it lets the rest of the
kernel development community know that changes affecting zones should
also be tested on powerpc.

Thanks,
Oliver


On Fri, May 6, 2016 at 3:13 PM, Anshuman Khandual
<khandual@linux.vnet.ibm.com> wrote:
> On 05/05/2016 08:18 PM, Aneesh Kumar K.V wrote:
>> Anshuman Khandual <khandual@linux.vnet.ibm.com> writes:
>>
>>> This enables base ZONE_DEVICE support on POWER. This series depends on
>>> the following patches posted by Oliver.
>>>
>>> https://patchwork.ozlabs.org/patch/618867/
>>> https://patchwork.ozlabs.org/patch/618868/
>>>
>>> Anshuman Khandual (2):
>>>   powerpc/mm: Make vmemmap_populate accommodate ZONE_DEVICE memory
>>>   powerpc/mm: Enable support for ZONE_DEVICE on PPC_BOOK3S_64 platforms
>>>
>>>  arch/powerpc/mm/init_64.c | 4 +++-
>>>  mm/Kconfig                | 2 +-
>>>  2 files changed, 4 insertions(+), 2 deletions(-)
>>>
>>
>> What is the use case ? Who will use ZONE_DEVICE on ppc64. This should be
>> be merged along with the patch series that use this.
>
> IIUC, Oliver has been looking at using ZONE_DEVICE for the NVDIMM (or
> some other persistent memory) drivers. I have been following Dan William's
> work on this front and want to explore more details about it's functioning
> on ppc64. This enablement will just help us little bit in that direction.
>

Re: [PATCH 0/2] Enable ZONE_DEVICE on POWER

[Aneesh Kumar K.V]

oliver <oohall@gmail.com> writes:

> Hi,
>
> I've been working on kernel support for a persistent memory (nvdimm)
> device and the kernel driver infrastructure requires ZONE_DEVICE for
> DAX support.. I've had it enabled in my tree for some time (without
> altmap support) without any real issues.


IIUC a DAX usage of pmem doesn't really need ZONE_DEVICE. It is only
for struct page backing to those pmem area we need ZONE_DEVICE.

>
> I wasn't planning on upstreaming any of my changes until 4.8 at the
> earliest so I am ok with carrying these patches myself. However, there
> has been some interest in using ZONE_DEVICE for other things on ppc
> (wasn't that you?) and given that ZONE_DEVICE is gated behind
> CONFIG_EXPERT I can't see there being any kind of negative impact on
> end users by merging it now. At the very least it lets the rest of the
> kernel development community know that changes affecting zones should
> also be tested on powerpc.
>
>

A partially done patch like that will miss quiet a lot details. For
example if I look at the x86 changes related to altmap
(4b94ffdc4163bae1e ("x86, mm: introduce vmem_altmap to augment
vmemmap_populate")) i see them handling pagetable free and memory
hotplug. This patch doesn't do any of those. From the commit message it is
also not clear how we intent to use those zone device memory on ppc64.
If we say they will not get hotplugged out or they will never be part
of page table then those changes I mentioned above are really not
needed. But the patch is missing a lot of those details.

-aneesh

Re: [PATCH 0/2] Enable ZONE_DEVICE on POWER

[Anshuman Khandual]

On 05/08/2016 11:07 PM, Aneesh Kumar K.V wrote:
> oliver <oohall@gmail.com> writes:
> 
>> > Hi,
>> >
>> > I've been working on kernel support for a persistent memory (nvdimm)
>> > device and the kernel driver infrastructure requires ZONE_DEVICE for
>> > DAX support.. I've had it enabled in my tree for some time (without
>> > altmap support) without any real issues.
> 
> IIUC a DAX usage of pmem doesn't really need ZONE_DEVICE. It is only
> for struct page backing to those pmem area we need ZONE_DEVICE.
> 
>> >
>> > I wasn't planning on upstreaming any of my changes until 4.8 at the
>> > earliest so I am ok with carrying these patches myself. However, there
>> > has been some interest in using ZONE_DEVICE for other things on ppc
>> > (wasn't that you?) and given that ZONE_DEVICE is gated behind
>> > CONFIG_EXPERT I can't see there being any kind of negative impact on
>> > end users by merging it now. At the very least it lets the rest of the
>> > kernel development community know that changes affecting zones should
>> > also be tested on powerpc.
>> >
>> >
> A partially done patch like that will miss quiet a lot details. For
> example if I look at the x86 changes related to altmap
> (4b94ffdc4163bae1e ("x86, mm: introduce vmem_altmap to augment
> vmemmap_populate")) i see them handling pagetable free and memory
> hotplug. This patch doesn't do any of those. From the commit message it is
> also not clear how we intent to use those zone device memory on ppc64.
> If we say they will not get hotplugged out or they will never be part
> of page table then those changes I mentioned above are really not
> needed. But the patch is missing a lot of those details.

Right. The current patch just enables ZONE_DEVICE with vmem_altmap support
(so that struct pages can be allocated in the device range instead of system
RAM) where any driver can own the rest of the PFNs for its use. Right now
these PFNs will not make into process page table. I can update the commit
message with these details if you like.