[PATCH 0/3] Define coherent device memory node

[PATCH 0/3] Define coherent device memory node

[Anshuman Khandual]

	This three patches define CDM node with HugeTLB & Buddy allocation
isolation. Please refer to the last RFC posting mentioned here for details.
The series has been split for easier review process. The next part of the
work like VM flags, auto NUMA and KSM interactions with tagged VMAs will
follow later.

https://lkml.org/lkml/2017/1/29/198

Optional Buddy allocation isolation methods

(1) GFP flag based		(mm_cdm_v1_optional_gfp)
(2) Zonelist rebuilding		(mm_cdm_v1_optional_zonelist)
(3) Cpuset			(mm_cdm_v1_optional_cpusets)

All of these optional methods as well as the posted nodemask (mm_cdm_v1)
approach can be accessed from the following git tree.

https://github.com/akhandual/linux.git

Anshuman Khandual (3):
  mm: Define coherent device memory (CDM) node
  mm: Enable HugeTLB allocation isolation for CDM nodes
  mm: Enable Buddy allocation isolation for CDM nodes

 Documentation/ABI/stable/sysfs-devices-node |  7 +++++
 arch/powerpc/Kconfig                        |  1 +
 arch/powerpc/mm/numa.c                      |  7 +++++
 drivers/base/node.c                         |  6 ++++
 include/linux/node.h                        | 49 +++++++++++++++++++++++++++++
 include/linux/nodemask.h                    |  3 ++
 mm/Kconfig                                  |  4 +++
 mm/hugetlb.c                                | 25 +++++++++------
 mm/memory_hotplug.c                         | 10 ++++++
 mm/page_alloc.c                             | 33 +++++++++++++++++--
 10 files changed, 134 insertions(+), 11 deletions(-)

-- 
2.9.3

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

[PATCH 1/3] mm: Define coherent device memory (CDM) node

[Anshuman Khandual]

There are certain devices like specialized accelerator, GPU cards, network
cards, FPGA cards etc which might contain onboard memory which is coherent
along with the existing system RAM while being accessed either from the CPU
or from the device. They share some similar properties with that of normal
system RAM but at the same time can also be different with respect to
system RAM.

User applications might be interested in using this kind of coherent device
memory explicitly or implicitly along side the system RAM utilizing all
possible core memory functions like anon mapping (LRU), file mapping (LRU),
page cache (LRU), driver managed (non LRU), HW poisoning, NUMA migrations
etc. To achieve this kind of tight integration with core memory subsystem,
the device onboard coherent memory must be represented as a memory only
NUMA node. At the same time arch must export some kind of a function to
identify of this node as a coherent device memory not any other regular
cpu less memory only NUMA node.

After achieving the integration with core memory subsystem coherent device
memory might still need some special consideration inside the kernel. There
can be a variety of coherent memory nodes with different expectations from
the core kernel memory. But right now only one kind of special treatment is
considered which requires certain isolation.

Now consider the case of a coherent device memory node type which requires
isolation. This kind of coherent memory is onboard an external device
attached to the system through a link where there is always a chance of a
link failure taking down the entire memory node with it. More over the
memory might also have higher chance of ECC failure as compared to the
system RAM. Hence allocation into this kind of coherent memory node should
be regulated. Kernel allocations must not come here. Normal user space
allocations too should not come here implicitly (without user application
knowing about it). This summarizes isolation requirement of certain kind of
coherent device memory node as an example. There can be different kinds of
isolation requirement also.

Some coherent memory devices might not require isolation altogether after
all. Then there might be other coherent memory devices which might require
some other special treatment after being part of core memory representation
. For now, will look into isolation seeking coherent device memory node not
the other ones.

To implement the integration as well as isolation, the coherent memory node
must be present in N_MEMORY and a new N_COHERENT_DEVICE node mask inside
the node_states[] array. During memory hotplug operations, the new nodemask
N_COHERENT_DEVICE is updated along with N_MEMORY for these coherent device
memory nodes. This also creates the following new sysfs based interface to
list down all the coherent memory nodes of the system.

	/sys/devices/system/node/is_coherent_node

Architectures must export function arch_check_node_cdm() which identifies
any coherent device memory node in case they enable CONFIG_COHERENT_DEVICE.

Signed-off-by: Anshuman Khandual <khandual@linux.vnet.ibm.com>
---
 Documentation/ABI/stable/sysfs-devices-node |  7 +++++
 arch/powerpc/Kconfig                        |  1 +
 arch/powerpc/mm/numa.c                      |  7 +++++
 drivers/base/node.c                         |  6 ++++
 include/linux/node.h                        | 49 +++++++++++++++++++++++++++++
 include/linux/nodemask.h                    |  3 ++
 mm/Kconfig                                  |  4 +++
 mm/memory_hotplug.c                         | 10 ++++++
 mm/page_alloc.c                             | 14 +++++++--
 9 files changed, 99 insertions(+), 2 deletions(-)

diff --git a/Documentation/ABI/stable/sysfs-devices-node b/Documentation/ABI/stable/sysfs-devices-node
index 5b2d0f0..fa2f105 100644
--- a/Documentation/ABI/stable/sysfs-devices-node
+++ b/Documentation/ABI/stable/sysfs-devices-node
@@ -29,6 +29,13 @@ Description:
 		Nodes that have regular or high memory.
 		Depends on CONFIG_HIGHMEM.
 
+What:		/sys/devices/system/node/is_coherent_device
+Date:		January 2017
+Contact:	Linux Memory Management list <linux-mm@kvack.org>
+Description:
+		Lists the nodemask of nodes that have coherent device memory.
+		Depends on CONFIG_COHERENT_DEVICE.
+
 What:		/sys/devices/system/node/nodeX
 Date:		October 2002
 Contact:	Linux Memory Management list <linux-mm@kvack.org>
diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig
index 281f4f1..77b97fe 100644
--- a/arch/powerpc/Kconfig
+++ b/arch/powerpc/Kconfig
@@ -164,6 +164,7 @@ config PPC
 	select ARCH_HAS_SCALED_CPUTIME if VIRT_CPU_ACCOUNTING_NATIVE
 	select HAVE_ARCH_HARDENED_USERCOPY
 	select HAVE_KERNEL_GZIP
+	select COHERENT_DEVICE if PPC_BOOK3S_64
 
 config GENERIC_CSUM
 	def_bool CPU_LITTLE_ENDIAN
diff --git a/arch/powerpc/mm/numa.c b/arch/powerpc/mm/numa.c
index b1099cb..14f0b98 100644
--- a/arch/powerpc/mm/numa.c
+++ b/arch/powerpc/mm/numa.c
@@ -41,6 +41,13 @@
 #include <asm/setup.h>
 #include <asm/vdso.h>
 
+#ifdef CONFIG_COHERENT_DEVICE
+inline int arch_check_node_cdm(int nid)
+{
+	return 0;
+}
+#endif
+
 static int numa_enabled = 1;
 
 static char *cmdline __initdata;
diff --git a/drivers/base/node.c b/drivers/base/node.c
index 5548f96..5b5dd89 100644
--- a/drivers/base/node.c
+++ b/drivers/base/node.c
@@ -661,6 +661,9 @@ static struct node_attr node_state_attr[] = {
 	[N_MEMORY] = _NODE_ATTR(has_memory, N_MEMORY),
 #endif
 	[N_CPU] = _NODE_ATTR(has_cpu, N_CPU),
+#ifdef CONFIG_COHERENT_DEVICE
+	[N_COHERENT_DEVICE] = _NODE_ATTR(is_coherent_device, N_COHERENT_DEVICE),
+#endif
 };
 
 static struct attribute *node_state_attrs[] = {
@@ -674,6 +677,9 @@ static struct attribute *node_state_attrs[] = {
 	&node_state_attr[N_MEMORY].attr.attr,
 #endif
 	&node_state_attr[N_CPU].attr.attr,
+#ifdef CONFIG_COHERENT_DEVICE
+	&node_state_attr[N_COHERENT_DEVICE].attr.attr,
+#endif
 	NULL
 };
 
diff --git a/include/linux/node.h b/include/linux/node.h
index 2115ad5..cb5b26f 100644
--- a/include/linux/node.h
+++ b/include/linux/node.h
@@ -81,4 +81,53 @@ static inline void register_hugetlbfs_with_node(node_registration_func_t reg,
 
 #define to_node(device) container_of(device, struct node, dev)
 
+
+#ifdef CONFIG_COHERENT_DEVICE
+extern int arch_check_node_cdm(int nid);
+
+static inline nodemask_t system_mem_nodemask(void)
+{
+	nodemask_t system_mem;
+
+	nodes_clear(system_mem);
+	nodes_andnot(system_mem, node_states[N_MEMORY],
+				node_states[N_COHERENT_DEVICE]);
+	return system_mem;
+}
+
+static inline bool is_cdm_node(int node)
+{
+	return node_isset(node, node_states[N_COHERENT_DEVICE]);
+}
+
+static inline bool nodemask_has_cdm(nodemask_t mask)
+{
+	int node, i;
+
+	node = first_node(mask);
+	for (i = 0; i < nodes_weight(mask); i++) {
+		if (is_cdm_node(node))
+			return true;
+		node = next_node(node, mask);
+	}
+	return false;
+}
+#else
+static inline int arch_check_node_cdm(int nid) { return 0; }
+
+static inline nodemask_t system_mem_nodemask(void)
+{
+	return node_states[N_MEMORY];
+}
+
+static inline bool is_cdm_node(int node)
+{
+	return false;
+}
+
+static inline bool nodemask_has_cdm(nodemask_t mask)
+{
+	return false;
+}
+#endif /* CONFIG_COHERENT_DEVICE */
 #endif /* _LINUX_NODE_H_ */
diff --git a/include/linux/nodemask.h b/include/linux/nodemask.h
index f746e44..6e66cfd 100644
--- a/include/linux/nodemask.h
+++ b/include/linux/nodemask.h
@@ -393,6 +393,9 @@ enum node_states {
 	N_MEMORY = N_HIGH_MEMORY,
 #endif
 	N_CPU,		/* The node has one or more cpus */
+#ifdef CONFIG_COHERENT_DEVICE
+	N_COHERENT_DEVICE,
+#endif
 	NR_NODE_STATES
 };
 
diff --git a/mm/Kconfig b/mm/Kconfig
index 9b8fccb..6263a65 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -143,6 +143,10 @@ config HAVE_GENERIC_RCU_GUP
 config ARCH_DISCARD_MEMBLOCK
 	bool
 
+config COHERENT_DEVICE
+	bool
+	default n
+
 config NO_BOOTMEM
 	bool
 
diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index b8c11e0..7bfdf56 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -1030,6 +1030,11 @@ static void node_states_set_node(int node, struct memory_notify *arg)
 	if (arg->status_change_nid_high >= 0)
 		node_set_state(node, N_HIGH_MEMORY);
 
+#ifdef CONFIG_COHERENT_DEVICE
+	if (arch_check_node_cdm(node))
+		node_set_state(node, N_COHERENT_DEVICE);
+#endif
+
 	node_set_state(node, N_MEMORY);
 }
 
@@ -1843,6 +1848,11 @@ static void node_states_clear_node(int node, struct memory_notify *arg)
 	if ((N_MEMORY != N_HIGH_MEMORY) &&
 	    (arg->status_change_nid >= 0))
 		node_clear_state(node, N_MEMORY);
+
+#ifdef CONFIG_COHERENT_DEVICE
+	if (arch_check_node_cdm(node))
+		node_clear_state(node, N_COHERENT_DEVICE);
+#endif
 }
 
 static int __ref __offline_pages(unsigned long start_pfn,
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index f3e0c69..40908de 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -6080,8 +6080,13 @@ static unsigned long __init early_calculate_totalpages(void)
 		unsigned long pages = end_pfn - start_pfn;
 
 		totalpages += pages;
-		if (pages)
+		if (pages) {
+#ifdef CONFIG_COHERENT_DEVICE
+			if (arch_check_node_cdm(nid))
+				node_set_state(nid, N_COHERENT_DEVICE);
+#endif
 			node_set_state(nid, N_MEMORY);
+		}
 	}
 	return totalpages;
 }
@@ -6392,8 +6397,13 @@ void __init free_area_init_nodes(unsigned long *max_zone_pfn)
 				find_min_pfn_for_node(nid), NULL);
 
 		/* Any memory on that node */
-		if (pgdat->node_present_pages)
+		if (pgdat->node_present_pages) {
+#ifdef CONFIG_COHERENT_DEVICE
+			if (arch_check_node_cdm(nid))
+				node_set_state(nid, N_COHERENT_DEVICE);
+#endif
 			node_set_state(nid, N_MEMORY);
+		}
 		check_for_memory(pgdat, nid);
 	}
 }
-- 
2.9.3

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH 1/3] mm: Define coherent device memory (CDM) node

[kbuild test robot]

[-- Attachment #1: Type: text/plain, Size: 1582 bytes --]

Hi Anshuman,

[auto build test ERROR on linus/master]
[also build test ERROR on v4.10-rc7]
[cannot apply to mmotm/master next-20170208]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]

url:    https://github.com/0day-ci/linux/commits/Anshuman-Khandual/Define-coherent-device-memory-node/20170208-223134
config: powerpc-ps3_defconfig (attached as .config)
compiler: powerpc64-linux-gnu-gcc (Debian 6.1.1-9) 6.1.1 20160705
reproduce:
        wget https://git.kernel.org/cgit/linux/kernel/git/wfg/lkp-tests.git/plain/sbin/make.cross -O ~/bin/make.cross
        chmod +x ~/bin/make.cross
        # save the attached .config to linux build tree
        make.cross ARCH=powerpc 

All errors (new ones prefixed by >>):

   mm/built-in.o: In function `early_calculate_totalpages':
>> mm/page_alloc.c:6085: undefined reference to `.arch_check_node_cdm'
   mm/built-in.o: In function `free_area_init_nodes':
   mm/page_alloc.c:6402: undefined reference to `.arch_check_node_cdm'

vim +6085 mm/page_alloc.c

  6079		for_each_mem_pfn_range(i, MAX_NUMNODES, &start_pfn, &end_pfn, &nid) {
  6080			unsigned long pages = end_pfn - start_pfn;
  6081	
  6082			totalpages += pages;
  6083			if (pages) {
  6084	#ifdef CONFIG_COHERENT_DEVICE
> 6085				if (arch_check_node_cdm(nid))
  6086					node_set_state(nid, N_COHERENT_DEVICE);
  6087	#endif
  6088				node_set_state(nid, N_MEMORY);

---
0-DAY kernel test infrastructure                Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all                   Intel Corporation

[-- Attachment #2: .config.gz --]
[-- Type: application/gzip, Size: 14324 bytes --]

Re: [PATCH 1/3] mm: Define coherent device memory (CDM) node

[kbuild test robot]

[-- Attachment #1: Type: text/plain, Size: 1177 bytes --]

Hi Anshuman,

[auto build test ERROR on linus/master]
[also build test ERROR on v4.10-rc7]
[cannot apply to mmotm/master next-20170208]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]

url:    https://github.com/0day-ci/linux/commits/Anshuman-Khandual/Define-coherent-device-memory-node/20170208-223134
config: powerpc-maple_defconfig (attached as .config)
compiler: powerpc64-linux-gnu-gcc (Debian 6.1.1-9) 6.1.1 20160705
reproduce:
        wget https://git.kernel.org/cgit/linux/kernel/git/wfg/lkp-tests.git/plain/sbin/make.cross -O ~/bin/make.cross
        chmod +x ~/bin/make.cross
        # save the attached .config to linux build tree
        make.cross ARCH=powerpc 

All errors (new ones prefixed by >>):

   mm/built-in.o: In function `.free_area_init_nodes':
>> (.init.text+0x10f8): undefined reference to `.arch_check_node_cdm'
   mm/built-in.o: In function `.free_area_init_nodes':
   (.init.text+0x15f8): undefined reference to `.arch_check_node_cdm'

---
0-DAY kernel test infrastructure                Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all                   Intel Corporation

[-- Attachment #2: .config.gz --]
[-- Type: application/gzip, Size: 16729 bytes --]

Re: [PATCH 1/3] mm: Define coherent device memory (CDM) node

[Anshuman Khandual]

On 02/08/2017 07:31 PM, Anshuman Khandual wrote:
> There are certain devices like specialized accelerator, GPU cards, network
> cards, FPGA cards etc which might contain onboard memory which is coherent
> along with the existing system RAM while being accessed either from the CPU
> or from the device. They share some similar properties with that of normal
> system RAM but at the same time can also be different with respect to
> system RAM.
> 
> User applications might be interested in using this kind of coherent device
> memory explicitly or implicitly along side the system RAM utilizing all
> possible core memory functions like anon mapping (LRU), file mapping (LRU),
> page cache (LRU), driver managed (non LRU), HW poisoning, NUMA migrations
> etc. To achieve this kind of tight integration with core memory subsystem,
> the device onboard coherent memory must be represented as a memory only
> NUMA node. At the same time arch must export some kind of a function to
> identify of this node as a coherent device memory not any other regular
> cpu less memory only NUMA node.
> 
> After achieving the integration with core memory subsystem coherent device
> memory might still need some special consideration inside the kernel. There
> can be a variety of coherent memory nodes with different expectations from
> the core kernel memory. But right now only one kind of special treatment is
> considered which requires certain isolation.
> 
> Now consider the case of a coherent device memory node type which requires
> isolation. This kind of coherent memory is onboard an external device
> attached to the system through a link where there is always a chance of a
> link failure taking down the entire memory node with it. More over the
> memory might also have higher chance of ECC failure as compared to the
> system RAM. Hence allocation into this kind of coherent memory node should
> be regulated. Kernel allocations must not come here. Normal user space
> allocations too should not come here implicitly (without user application
> knowing about it). This summarizes isolation requirement of certain kind of
> coherent device memory node as an example. There can be different kinds of
> isolation requirement also.
> 
> Some coherent memory devices might not require isolation altogether after
> all. Then there might be other coherent memory devices which might require
> some other special treatment after being part of core memory representation
> . For now, will look into isolation seeking coherent device memory node not
> the other ones.
> 
> To implement the integration as well as isolation, the coherent memory node
> must be present in N_MEMORY and a new N_COHERENT_DEVICE node mask inside
> the node_states[] array. During memory hotplug operations, the new nodemask
> N_COHERENT_DEVICE is updated along with N_MEMORY for these coherent device
> memory nodes. This also creates the following new sysfs based interface to
> list down all the coherent memory nodes of the system.
> 
> 	/sys/devices/system/node/is_coherent_node
> 
> Architectures must export function arch_check_node_cdm() which identifies
> any coherent device memory node in case they enable CONFIG_COHERENT_DEVICE.
> 
> Signed-off-by: Anshuman Khandual <khandual@linux.vnet.ibm.com>
> ---
>  Documentation/ABI/stable/sysfs-devices-node |  7 +++++
>  arch/powerpc/Kconfig                        |  1 +
>  arch/powerpc/mm/numa.c                      |  7 +++++
>  drivers/base/node.c                         |  6 ++++
>  include/linux/node.h                        | 49 +++++++++++++++++++++++++++++
>  include/linux/nodemask.h                    |  3 ++
>  mm/Kconfig                                  |  4 +++
>  mm/memory_hotplug.c                         | 10 ++++++
>  mm/page_alloc.c                             | 14 +++++++--
>  9 files changed, 99 insertions(+), 2 deletions(-)
> 
> diff --git a/Documentation/ABI/stable/sysfs-devices-node b/Documentation/ABI/stable/sysfs-devices-node
> index 5b2d0f0..fa2f105 100644
> --- a/Documentation/ABI/stable/sysfs-devices-node
> +++ b/Documentation/ABI/stable/sysfs-devices-node
> @@ -29,6 +29,13 @@ Description:
>  		Nodes that have regular or high memory.
>  		Depends on CONFIG_HIGHMEM.
>  
> +What:		/sys/devices/system/node/is_coherent_device
> +Date:		January 2017
> +Contact:	Linux Memory Management list <linux-mm@kvack.org>
> +Description:
> +		Lists the nodemask of nodes that have coherent device memory.
> +		Depends on CONFIG_COHERENT_DEVICE.
> +
>  What:		/sys/devices/system/node/nodeX
>  Date:		October 2002
>  Contact:	Linux Memory Management list <linux-mm@kvack.org>
> diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig
> index 281f4f1..77b97fe 100644
> --- a/arch/powerpc/Kconfig
> +++ b/arch/powerpc/Kconfig
> @@ -164,6 +164,7 @@ config PPC
>  	select ARCH_HAS_SCALED_CPUTIME if VIRT_CPU_ACCOUNTING_NATIVE
>  	select HAVE_ARCH_HARDENED_USERCOPY
>  	select HAVE_KERNEL_GZIP
> +	select COHERENT_DEVICE if PPC_BOOK3S_64
>

The following change here will fix the build problems in both the configs
ps3_defconfig and maple_defconfig. Will include this in next version.

diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig
index 77b97fe..1cff239 100644
--- a/arch/powerpc/Kconfig
+++ b/arch/powerpc/Kconfig
@@ -164,7 +164,7 @@ config PPC
        select ARCH_HAS_SCALED_CPUTIME if VIRT_CPU_ACCOUNTING_NATIVE
        select HAVE_ARCH_HARDENED_USERCOPY
        select HAVE_KERNEL_GZIP
-       select COHERENT_DEVICE if PPC_BOOK3S_64
+       select COHERENT_DEVICE if PPC_BOOK3S_64 && NEED_MULTIPLE_NODES

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

[PATCH 2/3] mm: Enable HugeTLB allocation isolation for CDM nodes

[Anshuman Khandual]

HugeTLB allocation/release/accounting currently spans across all the nodes
under N_MEMORY node mask. Coherent memory nodes should not be part of these
allocations. So use system_mem_nodemask() call to fetch system RAM only
nodes on the platform which can then be used for HugeTLB allocation purpose
instead of N_MEMORY node mask. This isolates coherent device memory nodes
from HugeTLB allocations.

Signed-off-by: Anshuman Khandual <khandual@linux.vnet.ibm.com>
---
 mm/hugetlb.c | 25 ++++++++++++++++---------
 1 file changed, 16 insertions(+), 9 deletions(-)

diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index c7025c1..9a46d9f 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -1790,6 +1790,7 @@ static void return_unused_surplus_pages(struct hstate *h,
 					unsigned long unused_resv_pages)
 {
 	unsigned long nr_pages;
+	nodemask_t system_mem = system_mem_nodemask();
 
 	/* Cannot return gigantic pages currently */
 	if (hstate_is_gigantic(h))
@@ -1816,7 +1817,7 @@ static void return_unused_surplus_pages(struct hstate *h,
 	while (nr_pages--) {
 		h->resv_huge_pages--;
 		unused_resv_pages--;
-		if (!free_pool_huge_page(h, &node_states[N_MEMORY], 1))
+		if (!free_pool_huge_page(h, &system_mem, 1))
 			goto out;
 		cond_resched_lock(&hugetlb_lock);
 	}
@@ -2107,8 +2108,9 @@ int __weak alloc_bootmem_huge_page(struct hstate *h)
 {
 	struct huge_bootmem_page *m;
 	int nr_nodes, node;
+	nodemask_t system_mem = system_mem_nodemask();
 
-	for_each_node_mask_to_alloc(h, nr_nodes, node, &node_states[N_MEMORY]) {
+	for_each_node_mask_to_alloc(h, nr_nodes, node, &system_mem) {
 		void *addr;
 
 		addr = memblock_virt_alloc_try_nid_nopanic(
@@ -2177,13 +2179,14 @@ static void __init gather_bootmem_prealloc(void)
 static void __init hugetlb_hstate_alloc_pages(struct hstate *h)
 {
 	unsigned long i;
+	nodemask_t system_mem = system_mem_nodemask();
+
 
 	for (i = 0; i < h->max_huge_pages; ++i) {
 		if (hstate_is_gigantic(h)) {
 			if (!alloc_bootmem_huge_page(h))
 				break;
-		} else if (!alloc_fresh_huge_page(h,
-					 &node_states[N_MEMORY]))
+		} else if (!alloc_fresh_huge_page(h, &system_mem))
 			break;
 	}
 	h->max_huge_pages = i;
@@ -2420,6 +2423,8 @@ static ssize_t __nr_hugepages_store_common(bool obey_mempolicy,
 					   unsigned long count, size_t len)
 {
 	int err;
+	nodemask_t system_mem = system_mem_nodemask();
+
 	NODEMASK_ALLOC(nodemask_t, nodes_allowed, GFP_KERNEL | __GFP_NORETRY);
 
 	if (hstate_is_gigantic(h) && !gigantic_page_supported()) {
@@ -2434,7 +2439,7 @@ static ssize_t __nr_hugepages_store_common(bool obey_mempolicy,
 		if (!(obey_mempolicy &&
 				init_nodemask_of_mempolicy(nodes_allowed))) {
 			NODEMASK_FREE(nodes_allowed);
-			nodes_allowed = &node_states[N_MEMORY];
+			nodes_allowed = &system_mem;
 		}
 	} else if (nodes_allowed) {
 		/*
@@ -2444,11 +2449,11 @@ static ssize_t __nr_hugepages_store_common(bool obey_mempolicy,
 		count += h->nr_huge_pages - h->nr_huge_pages_node[nid];
 		init_nodemask_of_node(nodes_allowed, nid);
 	} else
-		nodes_allowed = &node_states[N_MEMORY];
+		nodes_allowed = &system_mem;
 
 	h->max_huge_pages = set_max_huge_pages(h, count, nodes_allowed);
 
-	if (nodes_allowed != &node_states[N_MEMORY])
+	if (nodes_allowed != &system_mem)
 		NODEMASK_FREE(nodes_allowed);
 
 	return len;
@@ -2745,9 +2750,10 @@ static void hugetlb_register_node(struct node *node)
  */
 static void __init hugetlb_register_all_nodes(void)
 {
+	nodemask_t nodes = system_mem_nodemask();
 	int nid;
 
-	for_each_node_state(nid, N_MEMORY) {
+	for_each_node_mask(nid, nodes) {
 		struct node *node = node_devices[nid];
 		if (node->dev.id == nid)
 			hugetlb_register_node(node);
@@ -3019,11 +3025,12 @@ void hugetlb_show_meminfo(void)
 {
 	struct hstate *h;
 	int nid;
+	nodemask_t system_mem = system_mem_nodemask();
 
 	if (!hugepages_supported())
 		return;
 
-	for_each_node_state(nid, N_MEMORY)
+	for_each_node_mask(nid, system_mem)
 		for_each_hstate(h)
 			pr_info("Node %d hugepages_total=%u hugepages_free=%u hugepages_surp=%u hugepages_size=%lukB\n",
 				nid,
-- 
2.9.3

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

[PATCH 3/3] mm: Enable Buddy allocation isolation for CDM nodes

[Anshuman Khandual]

This implements allocation isolation for CDM nodes in buddy allocator by
discarding CDM memory zones all the time except in the cases where the gfp
flag has got __GFP_THISNODE or the nodemask contains CDM nodes in cases
where it is non NULL (explicit allocation request in the kernel or user
process MPOL_BIND policy based requests).

Signed-off-by: Anshuman Khandual <khandual@linux.vnet.ibm.com>
---
 mm/page_alloc.c | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 40908de..7d8c82a 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -64,6 +64,7 @@
 #include <linux/page_owner.h>
 #include <linux/kthread.h>
 #include <linux/memcontrol.h>
+#include <linux/node.h>
 
 #include <asm/sections.h>
 #include <asm/tlbflush.h>
@@ -2908,6 +2909,24 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags,
 		struct page *page;
 		unsigned long mark;
 
+		/*
+		 * CDM nodes get skipped if the requested gfp flag
+		 * does not have __GFP_THISNODE set or the nodemask
+		 * does not have any CDM nodes in case the nodemask
+		 * is non NULL (explicit allocation requests from
+		 * kernel or user process MPOL_BIND policy which has
+		 * CDM nodes).
+		 */
+		if (is_cdm_node(zone->zone_pgdat->node_id)) {
+			if (!(gfp_mask & __GFP_THISNODE)) {
+				if (!ac->nodemask)
+					continue;
+
+				if (!nodemask_has_cdm(*ac->nodemask))
+					continue;
+			}
+		}
+
 		if (cpusets_enabled() &&
 			(alloc_flags & ALLOC_CPUSET) &&
 			!__cpuset_zone_allowed(zone, gfp_mask))
-- 
2.9.3

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH 3/3] mm: Enable Buddy allocation isolation for CDM nodes

[Vlastimil Babka]

On 02/08/2017 03:01 PM, Anshuman Khandual wrote:
> This implements allocation isolation for CDM nodes in buddy allocator by
> discarding CDM memory zones all the time except in the cases where the gfp
> flag has got __GFP_THISNODE or the nodemask contains CDM nodes in cases
> where it is non NULL (explicit allocation request in the kernel or user
> process MPOL_BIND policy based requests).
>
> Signed-off-by: Anshuman Khandual <khandual@linux.vnet.ibm.com>
> ---
>  mm/page_alloc.c | 19 +++++++++++++++++++
>  1 file changed, 19 insertions(+)
>
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index 40908de..7d8c82a 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -64,6 +64,7 @@
>  #include <linux/page_owner.h>
>  #include <linux/kthread.h>
>  #include <linux/memcontrol.h>
> +#include <linux/node.h>
>
>  #include <asm/sections.h>
>  #include <asm/tlbflush.h>
> @@ -2908,6 +2909,24 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags,
>  		struct page *page;
>  		unsigned long mark;
>
> +		/*
> +		 * CDM nodes get skipped if the requested gfp flag
> +		 * does not have __GFP_THISNODE set or the nodemask
> +		 * does not have any CDM nodes in case the nodemask
> +		 * is non NULL (explicit allocation requests from
> +		 * kernel or user process MPOL_BIND policy which has
> +		 * CDM nodes).
> +		 */
> +		if (is_cdm_node(zone->zone_pgdat->node_id)) {
> +			if (!(gfp_mask & __GFP_THISNODE)) {
> +				if (!ac->nodemask)
> +					continue;
> +
> +				if (!nodemask_has_cdm(*ac->nodemask))
> +					continue;

nodemask_has_cdm() looks quite expensive, combined with the loop here that's 
O(n^2). But I don't understand why you need it. If there is no cdm node in the 
nodemask, then we never reach this code with a cdm node, because the zonelist 
iterator already checks the nodemask? Am I missing something?

> +			}
> +		}
> +
>  		if (cpusets_enabled() &&
>  			(alloc_flags & ALLOC_CPUSET) &&
>  			!__cpuset_zone_allowed(zone, gfp_mask))
>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH 3/3] mm: Enable Buddy allocation isolation for CDM nodes

[Anshuman Khandual]

On 02/08/2017 10:48 PM, Vlastimil Babka wrote:
> On 02/08/2017 03:01 PM, Anshuman Khandual wrote:
>> This implements allocation isolation for CDM nodes in buddy allocator by
>> discarding CDM memory zones all the time except in the cases where the
>> gfp
>> flag has got __GFP_THISNODE or the nodemask contains CDM nodes in cases
>> where it is non NULL (explicit allocation request in the kernel or user
>> process MPOL_BIND policy based requests).
>>
>> Signed-off-by: Anshuman Khandual <khandual@linux.vnet.ibm.com>
>> ---
>>  mm/page_alloc.c | 19 +++++++++++++++++++
>>  1 file changed, 19 insertions(+)
>>
>> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
>> index 40908de..7d8c82a 100644
>> --- a/mm/page_alloc.c
>> +++ b/mm/page_alloc.c
>> @@ -64,6 +64,7 @@
>>  #include <linux/page_owner.h>
>>  #include <linux/kthread.h>
>>  #include <linux/memcontrol.h>
>> +#include <linux/node.h>
>>
>>  #include <asm/sections.h>
>>  #include <asm/tlbflush.h>
>> @@ -2908,6 +2909,24 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned
>> int order, int alloc_flags,
>>          struct page *page;
>>          unsigned long mark;
>>
>> +        /*
>> +         * CDM nodes get skipped if the requested gfp flag
>> +         * does not have __GFP_THISNODE set or the nodemask
>> +         * does not have any CDM nodes in case the nodemask
>> +         * is non NULL (explicit allocation requests from
>> +         * kernel or user process MPOL_BIND policy which has
>> +         * CDM nodes).
>> +         */
>> +        if (is_cdm_node(zone->zone_pgdat->node_id)) {
>> +            if (!(gfp_mask & __GFP_THISNODE)) {
>> +                if (!ac->nodemask)
>> +                    continue;
>> +
>> +                if (!nodemask_has_cdm(*ac->nodemask))
>> +                    continue;
> 
> nodemask_has_cdm() looks quite expensive, combined with the loop here
> that's O(n^2). But I don't understand why you need it. If there is no
> cdm node in the nodemask, then we never reach this code with a cdm node,
> because the zonelist iterator already checks the nodemask? Am I missing
> something?

A CDM zone can be selected during zonelist iteration if

	(1) If nodemask is NULL (where all zones are eligible)

		(1) Skip it if __GFP_THISNODE is not mentioned
		(2) Pick it if __GFP_THISNODE is mentioned

	(2) If nodemask has CDM (where CDM zones are eligible)

		(1) Pick it if nodemask has CDM
		(2) Pick it if __GFP_THISNODE is mentioned

(1) (1) Enforces the primary isolation
(2) (1) Is the only option which could be O(n^2) as the worst case

Checking for both the zone being a CDM zone and the nodemask containing
CDM node has to happen together for (2) (1). But we dont run into this
option unless we have first checked if request contains __GFP_THISNODE
and that nodemask is really a non NULL value. Hence the number cases
getting into (2) (1) should be less. IIUC only the user space MPOL_BIND
ones will come here.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH 3/3] mm: Enable Buddy allocation isolation for CDM nodes

[Vlastimil Babka]

On 02/09/2017 06:05 AM, Anshuman Khandual wrote:
> On 02/08/2017 10:48 PM, Vlastimil Babka wrote:
>> On 02/08/2017 03:01 PM, Anshuman Khandual wrote:
>>> This implements allocation isolation for CDM nodes in buddy allocator by
>>> discarding CDM memory zones all the time except in the cases where the
>>> gfp
>>> flag has got __GFP_THISNODE or the nodemask contains CDM nodes in cases
>>> where it is non NULL (explicit allocation request in the kernel or user
>>> process MPOL_BIND policy based requests).
>>>
>>> Signed-off-by: Anshuman Khandual <khandual@linux.vnet.ibm.com>
>>> ---
>>>  mm/page_alloc.c | 19 +++++++++++++++++++
>>>  1 file changed, 19 insertions(+)
>>>
>>> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
>>> index 40908de..7d8c82a 100644
>>> --- a/mm/page_alloc.c
>>> +++ b/mm/page_alloc.c
>>> @@ -64,6 +64,7 @@
>>>  #include <linux/page_owner.h>
>>>  #include <linux/kthread.h>
>>>  #include <linux/memcontrol.h>
>>> +#include <linux/node.h>
>>>
>>>  #include <asm/sections.h>
>>>  #include <asm/tlbflush.h>
>>> @@ -2908,6 +2909,24 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned
>>> int order, int alloc_flags,
>>>          struct page *page;
>>>          unsigned long mark;
>>>
>>> +        /*
>>> +         * CDM nodes get skipped if the requested gfp flag
>>> +         * does not have __GFP_THISNODE set or the nodemask
>>> +         * does not have any CDM nodes in case the nodemask
>>> +         * is non NULL (explicit allocation requests from
>>> +         * kernel or user process MPOL_BIND policy which has
>>> +         * CDM nodes).
>>> +         */
>>> +        if (is_cdm_node(zone->zone_pgdat->node_id)) {
>>> +            if (!(gfp_mask & __GFP_THISNODE)) {
>>> +                if (!ac->nodemask)
>>> +                    continue;
>>> +
>>> +                if (!nodemask_has_cdm(*ac->nodemask))
>>> +                    continue;
>>
>> nodemask_has_cdm() looks quite expensive, combined with the loop here
>> that's O(n^2). But I don't understand why you need it. If there is no
>> cdm node in the nodemask, then we never reach this code with a cdm node,
>> because the zonelist iterator already checks the nodemask? Am I missing
>> something?
>
> A CDM zone can be selected during zonelist iteration if
>
> 	(1) If nodemask is NULL (where all zones are eligible)
>
> 		(1) Skip it if __GFP_THISNODE is not mentioned
> 		(2) Pick it if __GFP_THISNODE is mentioned
>
> 	(2) If nodemask has CDM (where CDM zones are eligible)
>
> 		(1) Pick it if nodemask has CDM
> 		(2) Pick it if __GFP_THISNODE is mentioned
>
> (1) (1) Enforces the primary isolation
> (2) (1) Is the only option which could be O(n^2) as the worst case
>
> Checking for both the zone being a CDM zone and the nodemask containing
> CDM node has to happen together for (2) (1). But we dont run into this
> option unless we have first checked if request contains __GFP_THISNODE
> and that nodemask is really a non NULL value. Hence the number cases
> getting into (2) (1) should be less. IIUC only the user space MPOL_BIND
> ones will come here.

Maybe I'm still missing something, but when you do nodemask_has_cdm() above then 
we already passed "if (!ac->nodemask) continue" which means ac->nodemask is not 
null, which means the zonelist iterator already did the filtering on 
ac->nodemask, and if this zone passed the filter and it's a cdm zone, then it 
has to be set in the nodemask?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH 3/3] mm: Enable Buddy allocation isolation for CDM nodes

[Anshuman Khandual]

On 02/09/2017 02:18 PM, Vlastimil Babka wrote:
> On 02/09/2017 06:05 AM, Anshuman Khandual wrote:
>> On 02/08/2017 10:48 PM, Vlastimil Babka wrote:
>>> On 02/08/2017 03:01 PM, Anshuman Khandual wrote:
>>>> This implements allocation isolation for CDM nodes in buddy
>>>> allocator by
>>>> discarding CDM memory zones all the time except in the cases where the
>>>> gfp
>>>> flag has got __GFP_THISNODE or the nodemask contains CDM nodes in cases
>>>> where it is non NULL (explicit allocation request in the kernel or user
>>>> process MPOL_BIND policy based requests).
>>>>
>>>> Signed-off-by: Anshuman Khandual <khandual@linux.vnet.ibm.com>
>>>> ---
>>>>  mm/page_alloc.c | 19 +++++++++++++++++++
>>>>  1 file changed, 19 insertions(+)
>>>>
>>>> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
>>>> index 40908de..7d8c82a 100644
>>>> --- a/mm/page_alloc.c
>>>> +++ b/mm/page_alloc.c
>>>> @@ -64,6 +64,7 @@
>>>>  #include <linux/page_owner.h>
>>>>  #include <linux/kthread.h>
>>>>  #include <linux/memcontrol.h>
>>>> +#include <linux/node.h>
>>>>
>>>>  #include <asm/sections.h>
>>>>  #include <asm/tlbflush.h>
>>>> @@ -2908,6 +2909,24 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned
>>>> int order, int alloc_flags,
>>>>          struct page *page;
>>>>          unsigned long mark;
>>>>
>>>> +        /*
>>>> +         * CDM nodes get skipped if the requested gfp flag
>>>> +         * does not have __GFP_THISNODE set or the nodemask
>>>> +         * does not have any CDM nodes in case the nodemask
>>>> +         * is non NULL (explicit allocation requests from
>>>> +         * kernel or user process MPOL_BIND policy which has
>>>> +         * CDM nodes).
>>>> +         */
>>>> +        if (is_cdm_node(zone->zone_pgdat->node_id)) {
>>>> +            if (!(gfp_mask & __GFP_THISNODE)) {
>>>> +                if (!ac->nodemask)
>>>> +                    continue;
>>>> +
>>>> +                if (!nodemask_has_cdm(*ac->nodemask))
>>>> +                    continue;
>>>
>>> nodemask_has_cdm() looks quite expensive, combined with the loop here
>>> that's O(n^2). But I don't understand why you need it. If there is no
>>> cdm node in the nodemask, then we never reach this code with a cdm node,
>>> because the zonelist iterator already checks the nodemask? Am I missing
>>> something?
>>
>> A CDM zone can be selected during zonelist iteration if
>>
>>     (1) If nodemask is NULL (where all zones are eligible)
>>
>>         (1) Skip it if __GFP_THISNODE is not mentioned
>>         (2) Pick it if __GFP_THISNODE is mentioned
>>
>>     (2) If nodemask has CDM (where CDM zones are eligible)
>>
>>         (1) Pick it if nodemask has CDM
>>         (2) Pick it if __GFP_THISNODE is mentioned
>>
>> (1) (1) Enforces the primary isolation
>> (2) (1) Is the only option which could be O(n^2) as the worst case
>>
>> Checking for both the zone being a CDM zone and the nodemask containing
>> CDM node has to happen together for (2) (1). But we dont run into this
>> option unless we have first checked if request contains __GFP_THISNODE
>> and that nodemask is really a non NULL value. Hence the number cases
>> getting into (2) (1) should be less. IIUC only the user space MPOL_BIND
>> ones will come here.
> 
> Maybe I'm still missing something, but when you do nodemask_has_cdm()
> above then we already passed "if (!ac->nodemask) continue" which means
> ac->nodemask is not null, which means the zonelist iterator already did
> the filtering on ac->nodemask, and if this zone passed the filter and
> it's a cdm zone, then it has to be set in the nodemask?

Hmm, think you are right. Then I can drop the last check there. Will test
it out. Thanks for pointing this out.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH 0/3] Define coherent device memory node

[Balbir Singh]

On 02/08/2017 07:31 PM, Anshuman Khandual wrote:
> 	This three patches define CDM node with HugeTLB & Buddy allocation
> isolation. Please refer to the last RFC posting mentioned here for details.
> The series has been split for easier review process. The next part of the
> work like VM flags, auto NUMA and KSM interactions with tagged VMAs will
> follow later.
>
> https://lkml.org/lkml/2017/1/29/198
>
> Optional Buddy allocation isolation methods
>
> (1) GFP flag based		(mm_cdm_v1_optional_gfp)
> (2) Zonelist rebuilding		(mm_cdm_v1_optional_zonelist)
> (3) Cpuset			(mm_cdm_v1_optional_cpusets)
>
> All of these optional methods as well as the posted nodemask (mm_cdm_v1)
> approach can be accessed from the following git tree.
>

Definitely much better looking, in general I like the approach and
would ack it. Lets stick to mm_cdm_v1 (this post) as a starting point

Balbir Singh.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>