[PATCH 0/5 v2] Update memory dlpar for dynamic reconfiguration memory

[PATCH 0/5 v2] Update memory dlpar for dynamic reconfiguration memory

[Nathan Fontenot]

The following patchset corrects a set of issues for memory dlpar and the
representation of memory in the device tree under the 
ibm,dynamic-reconfiguration-memory node.

On newer hardware and kernels, the device tree has changed so that all
memory appears under the ibm,dynamic-reconfiguration-memory node in the
device tree.  This node has several properties that describe all possible
lmbs, lmb-size and numa affinity.

The ibm,dynamic-memory property is a list of all possible lmbs, where each
list entry contains the base address, drc index, associativity array index
and a set of flags.

The ibm,associativity-lookup-arrays property contains a list of associativity
arrays and the ibm,lmb-size property contains the size of the lmbs

For reference, each lmb available to the system was previously represented 
with a node in the device tree, of the form /proc/device-tree/memory@XXXXXXXX.
Each of these memory nodes had properties such as "ibm,my-drc-index" for the
drc index and "regs" which contained the base address and lmb size of the lmb.

With the new device tree layout, code associated with memory dlpar needs to
be updated to parse the new device tree layout and properly call the memory
hotplug notifier chains.  Additionally, numa associativity done during hotplug
add of memory needs an update for parsing the new device tree layout.

Signed-off-by: Nathan Fontenot <nfont@austin.ibm.com> 

[PATCH 1/5 v2] Allow phandle to be specified in formats other than decimal

[Nathan Fontenot]

Allow the phandle passed to the /proc/ppc64/ofdt file to be specified
in formats other than decimal.  This allows us to easily specify phandle
values in hex that would otherwise appear as negative integers.

This is an issue on systems where the value of 
/proc/device-tree/ibm,dynamic-reconfiguration-memory.ibm,phandle is
fffffff9.  Having to pass this to the ofdt file as a string results in
a large negative number, and simple_strtoul() does not handle negative
numbers.

Signed-off-by: Nathan Fontenot <nfont@austin.ibm.com>
---
 arch/powerpc/platforms/pseries/reconfig.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Index: linux-2.6.git/arch/powerpc/platforms/pseries/reconfig.c
===================================================================
--- linux-2.6.git.orig/arch/powerpc/platforms/pseries/reconfig.c	2008-07-01 09:38:22.000000000 -0500
+++ linux-2.6.git/arch/powerpc/platforms/pseries/reconfig.c	2008-07-01 09:38:26.000000000 -0500
@@ -365,7 +365,7 @@
 	*buf = '\0';
 	buf++;
 
-	handle = simple_strtoul(handle_str, NULL, 10);
+	handle = simple_strtoul(handle_str, NULL, 0);
 
 	*npp = of_find_node_by_phandle(handle);
 	return buf;

[PATCH 2/5 v2] Use base address to derive starting page frame number

[Nathan Fontenot]

Use the base address of the lmb to derive the starting page frame number
instead of trying to extract it from the drc index of the lmb.  The drc
index should not be used for this as it will, and did, break.

Until this point, systems that have had memory represented in the device
tree with a node for each lmb the drc index would (luckily) closely
track the base address of the lmb.  For example a lmb with a drc index
of 8000000a would have a base address of a0000000.  This correlation
allowed the current code to derive the starting page frame number from
the drc inddex

Device tree layouts where lmbs are represented under the
ibm,dynamic-reconfiguration-memory node in the ibm,dynamic-memory
property do not have this correlation between the drc index and base
address of the lmb. 

Signed-off-by: Nathan Fontenot <nfont@austin.ibm.com>
---
 arch/powerpc/platforms/pseries/hotplug-memory.c |   36 +++++++++++-------------
 1 file changed, 17 insertions(+), 19 deletions(-)

Index: linux-2.6.git/arch/powerpc/platforms/pseries/hotplug-memory.c
===================================================================
--- linux-2.6.git.orig/arch/powerpc/platforms/pseries/hotplug-memory.c	2008-07-01 09:38:20.000000000 -0500
+++ linux-2.6.git/arch/powerpc/platforms/pseries/hotplug-memory.c	2008-07-01 09:38:49.000000000 -0500
@@ -18,8 +18,9 @@
 static int pseries_remove_memory(struct device_node *np)
 {
 	const char *type;
-	const unsigned int *my_index;
 	const unsigned int *regs;
+	unsigned long base;
+	unsigned int lmb_size;
 	u64 start_pfn, start;
 	struct zone *zone;
 	int ret = -EINVAL;
@@ -32,17 +33,16 @@
 		return 0;
 
 	/*
-	 * Find the memory index and size of the removing section
+	 * Find the bae address and size of the lmb
 	 */
-	my_index = of_get_property(np, "ibm,my-drc-index", NULL);
-	if (!my_index)
-		return ret;
-
 	regs = of_get_property(np, "reg", NULL);
 	if (!regs)
 		return ret;
 
-	start_pfn = section_nr_to_pfn(*my_index & 0xffff);
+	base = *(unsigned long *)regs;
+	lmb_size = regs[3];
+
+	start_pfn = base >> PFN_SECTION_SHIFT;
 	zone = page_zone(pfn_to_page(start_pfn));
 
 	/*
@@ -54,28 +54,29 @@
 	 * to sysfs "state" file and we can't remove sysfs entries
 	 * while writing to it. So we have to defer it to here.
 	 */
-	ret = __remove_pages(zone, start_pfn, regs[3] >> PAGE_SHIFT);
+	ret = __remove_pages(zone, start_pfn, lmb_size >> PAGE_SHIFT);
 	if (ret)
 		return ret;
 
 	/*
 	 * Update memory regions for memory remove
 	 */
-	lmb_remove(start_pfn << PAGE_SHIFT, regs[3]);
+	lmb_remove(base, lmb_size);
 
 	/*
 	 * Remove htab bolted mappings for this section of memory
 	 */
-	start = (unsigned long)__va(start_pfn << PAGE_SHIFT);
-	ret = remove_section_mapping(start, start + regs[3]);
+	start = (unsigned long)__va(base);
+	ret = remove_section_mapping(start, start + lmb_size);
 	return ret;
 }
 
 static int pseries_add_memory(struct device_node *np)
 {
 	const char *type;
-	const unsigned int *my_index;
 	const unsigned int *regs;
+	unsigned long base;
+	unsigned int lmb_size;
 	u64 start_pfn;
 	int ret = -EINVAL;
 
@@ -87,22 +88,19 @@
 		return 0;
 
 	/*
-	 * Find the memory index and size of the added section
+	 * Find the base and size of the lmb
 	 */
-	my_index = of_get_property(np, "ibm,my-drc-index", NULL);
-	if (!my_index)
-		return ret;
-
 	regs = of_get_property(np, "reg", NULL);
 	if (!regs)
 		return ret;
 
-	start_pfn = section_nr_to_pfn(*my_index & 0xffff);
+	base = *(unsigned long *)regs;
+	lmb_size = regs[3];
 
 	/*
 	 * Update memory region to represent the memory add
 	 */
-	lmb_add(start_pfn << PAGE_SHIFT, regs[3]);
+	lmb_add(base, lmb_size);
 	return 0;
 }

[PATCH 3/5 v2] Update the device tree correctly for drconf memory add/remove

[Nathan Fontenot]

This patch updates the device tree manipulation routines so that memory
add/remove of lmbs represented under the
ibm,dynamic-reconfiguration-memory node of the device tree invokes the
hotplug notifier chain.

This change is needed because of the change in the way memory is
represented under the ibm,dynamic-reconfiguration-memory node.  All lmbs
are described in the ibm,dynamic-memory property instead of having a
separate node for each lmb as in previous device tree layouts.  This
requires the update_node() routine to check for updates to the
ibm,dynamic-memory property and invoke the hotplug notifier chain.

This also updates the pseries hotplug notifier to be able to gather information
for lmbs represented under the ibm,dynamic-reconfiguration-memory node and
have the lmbs added/removed.

Signed-off-by: Nathan Fontenot <nfont@austin.ibm.com>
---
 arch/powerpc/platforms/pseries/hotplug-memory.c |   95 +++++++++++++++++-------
 arch/powerpc/platforms/pseries/reconfig.c       |   36 ++++++++-
 include/asm-powerpc/pSeries_reconfig.h          |    6 +
 3 files changed, 104 insertions(+), 33 deletions(-)

Index: linux-2.6.git/include/asm-powerpc/pSeries_reconfig.h
===================================================================
--- linux-2.6.git.orig/include/asm-powerpc/pSeries_reconfig.h	2008-07-01 09:38:19.000000000 -0500
+++ linux-2.6.git/include/asm-powerpc/pSeries_reconfig.h	2008-07-01 09:38:54.000000000 -0500
@@ -9,8 +9,10 @@
  * added or removed on pSeries systems.
  */
 
-#define PSERIES_RECONFIG_ADD    0x0001
-#define PSERIES_RECONFIG_REMOVE 0x0002
+#define PSERIES_RECONFIG_ADD		0x0001
+#define PSERIES_RECONFIG_REMOVE		0x0002
+#define PSERIES_DRCONF_MEM_ADD		0x0003
+#define PSERIES_DRCONF_MEM_REMOVE	0x0004
 
 #ifdef CONFIG_PPC_PSERIES
 extern int pSeries_reconfig_notifier_register(struct notifier_block *);
Index: linux-2.6.git/arch/powerpc/platforms/pseries/reconfig.c
===================================================================
--- linux-2.6.git.orig/arch/powerpc/platforms/pseries/reconfig.c	2008-07-01 09:38:26.000000000 -0500
+++ linux-2.6.git/arch/powerpc/platforms/pseries/reconfig.c	2008-07-01 09:38:54.000000000 -0500
@@ -422,8 +422,8 @@
 {
 	struct device_node *np;
 	unsigned char *value;
-	char *name, *end;
-	int length;
+	char *name, *end, *next_prop;
+	int rc, length;
 	struct property *newprop, *oldprop;
 	buf = parse_node(buf, bufsize, &np);
 	end = buf + bufsize;
@@ -431,7 +431,8 @@
 	if (!np)
 		return -ENODEV;
 
-	if (parse_next_property(buf, end, &name, &length, &value) == NULL)
+	next_prop = parse_next_property(buf, end, &name, &length, &value);
+	if (!next_prop)
 		return -EINVAL;
 
 	newprop = new_property(name, length, value, NULL);
@@ -442,7 +443,34 @@
 	if (!oldprop)
 		return -ENODEV;
 
-	return prom_update_property(np, newprop, oldprop);
+	rc = prom_update_property(np, newprop, oldprop);
+ 	if (rc)
+ 		return rc;
+
+ 	/* For memory under the ibm,dynamic-reconfiguration-memory node
+ 	 * of the device tree, adding and removing memory is just an update
+ 	 * to the ibm,dynamic-memory property instead of adding/removing a
+ 	 * memory node in the device tree.  For these cases we still need to
+ 	 * involve the notifier chain.
+ 	 */
+ 	if (!strcmp(name, "ibm,dynamic-memory")) {
+ 		int action;
+
+ 		next_prop = parse_next_property(next_prop, end, &name,
+ 						&length, &value);
+ 		if (!next_prop)
+ 			return -EINVAL;
+
+ 		if (!strcmp(name, "add"))
+ 			action = PSERIES_DRCONF_MEM_ADD;
+ 		else
+ 			action = PSERIES_DRCONF_MEM_REMOVE;
+
+ 		blocking_notifier_call_chain(&pSeries_reconfig_chain,
+ 					     action, value);
+ 	}
+
+ 	return 0;
 }
 
 /**
Index: linux-2.6.git/arch/powerpc/platforms/pseries/hotplug-memory.c
===================================================================
--- linux-2.6.git.orig/arch/powerpc/platforms/pseries/hotplug-memory.c	2008-07-01 09:38:49.000000000 -0500
+++ linux-2.6.git/arch/powerpc/platforms/pseries/hotplug-memory.c	2008-07-01 18:43:33.000000000 -0500
@@ -15,32 +15,11 @@
 #include <asm/machdep.h>
 #include <asm/pSeries_reconfig.h>
 
-static int pseries_remove_memory(struct device_node *np)
+static int pseries_remove_lmb(unsigned long base, unsigned int lmb_size)
 {
-	const char *type;
-	const unsigned int *regs;
-	unsigned long base;
-	unsigned int lmb_size;
-	u64 start_pfn, start;
+	unsigned long start, start_pfn;
 	struct zone *zone;
-	int ret = -EINVAL;
-
-	/*
-	 * Check to see if we are actually removing memory
-	 */
-	type = of_get_property(np, "device_type", NULL);
-	if (type == NULL || strcmp(type, "memory") != 0)
-		return 0;
-
-	/*
-	 * Find the bae address and size of the lmb
-	 */
-	regs = of_get_property(np, "reg", NULL);
-	if (!regs)
-		return ret;
-
-	base = *(unsigned long *)regs;
-	lmb_size = regs[3];
+	int ret;
 
 	start_pfn = base >> PFN_SECTION_SHIFT;
 	zone = page_zone(pfn_to_page(start_pfn));
@@ -71,13 +50,41 @@
 	return ret;
 }
 
+static int pseries_remove_memory(struct device_node *np)
+{
+	const char *type;
+	const unsigned int *regs;
+	unsigned long base;
+	unsigned int lmb_size;
+	int ret = -EINVAL;
+
+	/*
+	 * Check to see if we are actually removing memory
+	 */
+	type = of_get_property(np, "device_type", NULL);
+	if (type == NULL || strcmp(type, "memory") != 0)
+		return 0;
+
+	/*
+	 * Find the bae address and size of the lmb
+	 */
+	regs = of_get_property(np, "reg", NULL);
+	if (!regs)
+		return ret;
+
+	base = *(unsigned long *)regs;
+	lmb_size = regs[3];
+
+	ret = pseries_remove_lmb(base, lmb_size);
+	return ret;
+}
+
 static int pseries_add_memory(struct device_node *np)
 {
 	const char *type;
 	const unsigned int *regs;
 	unsigned long base;
 	unsigned int lmb_size;
-	u64 start_pfn;
 	int ret = -EINVAL;
 
 	/*
@@ -100,8 +107,37 @@
 	/*
 	 * Update memory region to represent the memory add
 	 */
-	lmb_add(base, lmb_size);
-	return 0;
+	ret = lmb_add(base, lmb_size);
+	return (ret < 0) ? -EINVAL : 0;
+}
+
+static int pseries_drconf_memory(unsigned long *base, unsigned int action)
+{
+	struct device_node *np;
+	const unsigned long *lmb_size;
+	int rc;
+
+	np = of_find_node_by_path("/ibm,dynamic-reconfiguration-memory");
+	if (!np)
+		return -EINVAL;
+
+	lmb_size = of_get_property(np, "ibm,lmb-size", NULL);
+	if (!lmb_size) {
+		of_node_put(np);
+		return -EINVAL;
+	}
+
+	if (action == PSERIES_DRCONF_MEM_ADD) {
+		rc = lmb_add(*base, *lmb_size);
+		rc = (rc < 0) ? -EINVAL : 0;
+	} else if (action == PSERIES_DRCONF_MEM_REMOVE) {
+		rc = pseries_remove_lmb(*base, *lmb_size);
+	} else {
+		rc = -EINVAL;
+	}
+
+	of_node_put(np);
+	return rc;
 }
 
 static int pseries_memory_notifier(struct notifier_block *nb,
@@ -118,6 +154,11 @@
 		if (pseries_remove_memory(node))
 			err = NOTIFY_BAD;
 		break;
+	case PSERIES_DRCONF_MEM_ADD:
+	case PSERIES_DRCONF_MEM_REMOVE:
+		if (pseries_drconf_memory(node, action))
+			err = NOTIFY_BAD;
+		break;
 	default:
 		err = NOTIFY_DONE;
 		break;

[PATCH 4/5 v2] Split code into helper routines for drconf memory

[Nathan Fontenot]

This patch splits off several pieces of code that parse the
ibm,dynamic-reconfiguration-memory node of the device tree into separate
helper routines.  This is in preparation for the followon patch that will
use these helper routines.  There are no functional changes in this patch.

Signed-off-by: Nathan Fontenot <nfont@austin.ibm.com>
---
 arch/powerpc/mm/numa.c |  209 +++++++++++++++++++++++++++++++++++++++----------
 1 file changed, 170 insertions(+), 39 deletions(-)

Index: linux-2.6.git/arch/powerpc/mm/numa.c
===================================================================
--- linux-2.6.git.orig/arch/powerpc/mm/numa.c	2008-07-01 18:40:55.000000000 -0500
+++ linux-2.6.git/arch/powerpc/mm/numa.c	2008-07-01 18:43:44.000000000 -0500
@@ -268,6 +268,144 @@
 	return result;
 }
 
+struct of_drconf_cell {
+	u64	base_addr;
+	u32	drc_index;
+	u32	reserved;
+	u32	aa_index;
+	u32	flags;
+};
+
+#define DRCONF_MEM_ASSIGNED	0x00000008
+#define DRCONF_MEM_AI_INVALID	0x00000040
+#define DRCONF_MEM_RESERVED	0x00000080
+
+/*
+ * Read the next lmb list entry from the ibm,dynamic-memory property
+ * and return the information in the provided of_drconf_cell structure.
+ */
+static void read_drconf_cell(struct of_drconf_cell *drmem, const u32 **cellp)
+{
+	const u32 *cp;
+
+	drmem->base_addr = read_n_cells(n_mem_addr_cells, cellp);
+
+	cp = *cellp;
+	drmem->drc_index = cp[0];
+	drmem->reserved = cp[1];
+	drmem->aa_index = cp[2];
+	drmem->flags = cp[3];
+
+	*cellp = cp + 4;
+}
+
+/*
+ * Retreive and validate the ibm,dynamic-memory property of the device tree.
+ *
+ * The layout of the ibm,dynamic-memory property is a number N of lmb
+ * list entries followed by N lmb list entries.  Each lmb list entry
+ * contains information as layed out in the of_drconf_cell struct above.
+ */
+static int of_get_drconf_memory(struct device_node *memory, const u32 **dm)
+{
+	const u32 *prop;
+	u32 len, entries;
+
+	prop = of_get_property(memory, "ibm,dynamic-memory", &len);
+	if (!prop || len < sizeof(unsigned int))
+		return 0;
+
+	entries = *prop++;
+
+	/* Now that we know the number of entries, revalidate the size
+	 * of the property read in to ensure we have everything
+	 */
+	if (len < (entries * (n_mem_addr_cells + 4) + 1) * sizeof(unsigned int))
+		return 0;
+
+	*dm = prop;
+	return entries;
+}
+
+/*
+ * Retreive and validate the ibm,lmb-size property for drconf memory
+ * from the device tree.
+ */
+static u64 of_get_lmb_size(struct device_node *memory)
+{
+	const u32 *prop;
+	u32 len;
+
+	prop = of_get_property(memory, "ibm,lmb-size", &len);
+	if (!prop || len < sizeof(unsigned int))
+		return 0;
+
+	return read_n_cells(n_mem_size_cells, &prop);
+}
+
+struct assoc_arrays {
+	u32	n_arrays;
+	u32	array_sz;
+	const u32 *arrays;
+};
+
+/*
+ * Retreive and validate the list of associativity arrays for drconf
+ * memory from the ibm,associativity-lookup-arrays property of the
+ * device tree..
+ *
+ * The layout of the ibm,associativity-lookup-arrays property is a number N
+ * indicating the number of associativity arrays, followed by a number M
+ * indicating the size of each associativity array, followed by a list
+ * of N associativity arrays.
+ */
+static int of_get_assoc_arrays(struct device_node *memory,
+			       struct assoc_arrays *aa)
+{
+	const u32 *prop;
+	u32 len;
+
+	prop = of_get_property(memory, "ibm,associativity-lookup-arrays", &len);
+	if (!prop || len < 2 * sizeof(unsigned int))
+		return -1;
+
+	aa->n_arrays = *prop++;
+	aa->array_sz = *prop++;
+
+	/* Now that we know the number of arrrays and size of each array,
+	 * revalidate the size of the property read in.
+	 */
+	if (len < (aa->n_arrays * aa->array_sz + 2) * sizeof(unsigned int))
+		return -1;
+
+	aa->arrays = prop;
+	return 0;
+}
+
+/*
+ * This is like of_node_to_nid_single() for memory represented in the
+ * ibm,dynami arch/powerpc/mm/numa.c.rejc-reconfiguration-memory node.
+ */
+static int of_drconf_to_nid_single(struct of_drconf_cell *drmem,
+				   struct assoc_arrays *aa)
+{
+	int default_nid = 0;
+	int nid = default_nid;
+	int index;
+
+	if (min_common_depth > 0 && min_common_depth <= aa->array_sz &&
+	    !(drmem->flags & DRCONF_MEM_AI_INVALID) &&
+	    drmem->aa_index < aa->n_arrays) {
+		index = drmem->aa_index * aa->array_sz + min_common_depth - 1;
+		nid = aa->arrays[index];
+
+		if (nid == 0xffff || nid >= MAX_NUMNODES)
+			nid = default_nid;
+	}
+
+	return nid;
+}
+
 /*
  * Figure out to which domain a cpu belongs and stick it there.
  * Return the id of the domain used.
@@ -355,57 +493,50 @@
  */
 static void __init parse_drconf_memory(struct device_node *memory)
 {
-	const unsigned int *lm, *dm, *aa;
-	unsigned int ls, ld, la;
-	unsigned int n, aam, aalen;
-	unsigned long lmb_size, size, start;
-	int nid, default_nid = 0;
-	unsigned int ai, flags;
-
-	lm = of_get_property(memory, "ibm,lmb-size", &ls);
-	dm = of_get_property(memory, "ibm,dynamic-memory", &ld);
-	aa = of_get_property(memory, "ibm,associativity-lookup-arrays", &la);
-	if (!lm || !dm || !aa ||
-	    ls < sizeof(unsigned int) || ld < sizeof(unsigned int) ||
-	    la < 2 * sizeof(unsigned int))
+	const u32 *dm;
+	unsigned int n, rc;
+	unsigned long lmb_size, size;
+	int nid;
+	struct assoc_arrays aa;
+
+	n = of_get_drconf_memory(memory, &dm);
+	if (!n)
 		return;
 
-	lmb_size = read_n_cells(n_mem_size_cells, &lm);
-	n = *dm++;		/* number of LMBs */
-	aam = *aa++;		/* number of associativity lists */
-	aalen = *aa++;		/* length of each associativity list */
-	if (ld < (n * (n_mem_addr_cells + 4) + 1) * sizeof(unsigned int) ||
-	    la < (aam * aalen + 2) * sizeof(unsigned int))
+	lmb_size = of_get_lmb_size(memory);
+	if (!lmb_size)
+		return;
+
+	rc = of_get_assoc_arrays(memory, &aa);
+	if (rc)
 		return;
 
 	for (; n != 0; --n) {
-		start = read_n_cells(n_mem_addr_cells, &dm);
-		ai = dm[2];
-		flags = dm[3];
-		dm += 4;
-		/* 0x80 == reserved, 0x8 = assigned to us */
-		if ((flags & 0x80) || !(flags & 0x8))
+		struct of_drconf_cell drmem;
+
+		read_drconf_cell(&drmem, &dm);
+
+		/* skip this block if the reserved bit is set in flags (0x80)
+		   or if the block is not assigned to this partition (0x8) */
+		if ((drmem.flags & DRCONF_MEM_RESERVED)
+		    || !(drmem.flags & DRCONF_MEM_ASSIGNED))
 			continue;
-		nid = default_nid;
-		/* flags & 0x40 means associativity index is invalid */
-		if (min_common_depth > 0 && min_common_depth <= aalen &&
-		    (flags & 0x40) == 0 && ai < aam) {
-			/* this is like of_node_to_nid_single */
-			nid = aa[ai * aalen + min_common_depth - 1];
-			if (nid == 0xffff || nid >= MAX_NUMNODES)
-				nid = default_nid;
-		}
 
-		fake_numa_create_new_node(((start + lmb_size) >> PAGE_SHIFT),
-						&nid);
+		nid = of_drconf_to_nid_single(&drmem, &aa);
+
+		fake_numa_create_new_node(
+				((drmem.base_addr + lmb_size) >> PAGE_SHIFT),
+					   &nid);
+
 		node_set_online(nid);
 
-		size = numa_enforce_memory_limit(start, lmb_size);
+		size = numa_enforce_memory_limit(drmem.base_addr, lmb_size);
 		if (!size)
 			continue;
 
-		add_active_range(nid, start >> PAGE_SHIFT,
-				 (start >> PAGE_SHIFT) + (size >> PAGE_SHIFT));
+		add_active_range(nid, drmem.base_addr >> PAGE_SHIFT,
+				 (drmem.base_addr >> PAGE_SHIFT)
+				 + (size >> PAGE_SHIFT));
 	}
 }
 

[PATCH 5/5 v2] Update numa association of hotplug memory add for drconf memory

[Nathan Fontenot]

Update the association of a memory section with a numa node that occurs during
hotplug add of a memory section.  This adds a check in the
hot_add_scn_to_nid() routine for the ibm,dyamic-reconfiguration-memory node 
in the device tree.  If present the new hot_add_drconf_scn_to_nid() routine 
is invoked, which can properly parse the ibm,dynamic-reconfiguration-memory
node of the device tree and make the proper numa node associations.

This also introduces the valid_hot_add_scn() routine as a helper
function for code that is common to the hot_add_scn_to_nid() and
hot_add_drconf_scn_to_nid() routines.

Signed-off-by: Nathan Fontenot <nfont@austin.ibm.com>
---
 arch/powerpc/mm/numa.c |  101 ++++++++++++++++++++++++++++++++++++++++---------
 1 file changed, 83 insertions(+), 18 deletions(-)

Index: linux-2.6.git/arch/powerpc/mm/numa.c
===================================================================
--- linux-2.6.git.orig/arch/powerpc/mm/numa.c	2008-07-01 18:43:44.000000000 -0500
+++ linux-2.6.git/arch/powerpc/mm/numa.c	2008-07-01 18:43:50.000000000 -0500
@@ -901,6 +901,79 @@
 
 #ifdef CONFIG_MEMORY_HOTPLUG
 /*
+ * Validate the node associated with the memory section we are
+ * trying to add.
+ */
+int valid_hot_add_scn(int *nid, unsigned long start, u32 lmb_size,
+		      unsigned long scn_addr)
+{
+	nodemask_t nodes;
+
+	if (*nid < 0 || !node_online(*nid))
+		*nid = any_online_node(NODE_MASK_ALL);
+
+	if ((scn_addr >= start) && (scn_addr < (start + lmb_size))) {
+		nodes_setall(nodes);
+		while (NODE_DATA(*nid)->node_spanned_pages == 0) {
+			node_clear(*nid, nodes);
+			*nid = any_online_node(nodes);
+		}
+
+		return 1;
+	}
+
+	return 0;
+}
+
+/*
+ * Find the node associated with a hot added memory section represented
+ * by the ibm,dynamic-reconfiguration-memory node.
+ */
+static int hot_add_drconf_scn_to_nid(struct device_node *memory,
+				     unsigned long scn_addr)
+{
+	const u32 *dm;
+	unsigned int n, rc;
+	unsigned long lmb_size;
+	int default_nid = any_online_node(NODE_MASK_ALL);
+	int nid;
+	struct assoc_arrays aa;
+
+	n = of_get_drconf_memory(memory, &dm);
+	if (!n)
+		return default_nid;;
+
+	lmb_size = of_get_lmb_size(memory);
+	if (!lmb_size)
+		return default_nid;
+
+	rc = of_get_assoc_arrays(memory, &aa);
+	if (rc)
+		return default_nid;
+
+	for (; n != 0; --n) {
+		struct of_drconf_cell drmem;
+
+		read_drconf_cell(&drmem, &dm);
+
+		/* skip this block if it is reserved or not assigned to
+		 * this partition */
+		if ((drmem.flags & DRCONF_MEM_RESERVED)
+		    || !(drmem.flags & DRCONF_MEM_ASSIGNED))
+			continue;
+
+		nid = of_drconf_to_nid_single(&drmem, &aa);
+
+		if (valid_hot_add_scn(&nid, drmem.base_addr, lmb_size,
+				      scn_addr))
+			return nid;
+	}
+
+	BUG();	/* section address should be found above */
+	return 0;
+}
+
+/*
  * Find the node associated with a hot added memory section.  Section
  * corresponds to a SPARSEMEM section, not an LMB.  It is assumed that
  * sections are fully contained within a single LMB.
@@ -908,12 +981,17 @@
 int hot_add_scn_to_nid(unsigned long scn_addr)
 {
 	struct device_node *memory = NULL;
-	nodemask_t nodes;
-	int default_nid = any_online_node(NODE_MASK_ALL);
 	int nid;
 
 	if (!numa_enabled || (min_common_depth < 0))
-		return default_nid;
+		return any_online_node(NODE_MASK_ALL);
+
+	memory = of_find_node_by_path("/ibm,dynamic-reconfiguration-memory");
+	if (memory) {
+		nid = hot_add_drconf_scn_to_nid(memory, scn_addr);
+		of_node_put(memory);
+		return nid;
+	}
 
 	while ((memory = of_find_node_by_type(memory, "memory")) != NULL) {
 		unsigned long start, size;
@@ -932,13 +1010,9 @@
 		size = read_n_cells(n_mem_size_cells, &memcell_buf);
 		nid = of_node_to_nid_single(memory);
 
-		/* Domains not present at boot default to 0 */
-		if (nid < 0 || !node_online(nid))
-			nid = default_nid;
-
-		if ((scn_addr >= start) && (scn_addr < (start + size))) {
+		if (valid_hot_add_scn(&nid, start, size, scn_addr)) {
 			of_node_put(memory);
-			goto got_nid;
+			return nid;
 		}
 
 		if (--ranges)		/* process all ranges in cell */
@@ -946,14 +1020,5 @@
 	}
 	BUG();	/* section address should be found above */
 	return 0;
-
-	/* Temporary code to ensure that returned node is not empty */
-got_nid:
-	nodes_setall(nodes);
-	while (NODE_DATA(nid)->node_spanned_pages == 0) {
-		node_clear(nid, nodes);
-		nid = any_online_node(nodes);
-	}
-	return nid;
 }
 #endif /* CONFIG_MEMORY_HOTPLUG */

Re: [PATCH 4/5 v2] Split code into helper routines for drconf memory

[Nathan Fontenot]

Oops, some silliness in one of the comments.  New patch below.

This patch splits off several pieces of code that parse the
ibm,dynamic-reconfiguration-memory node of the device tree into separate
helper routines.  This is in preparation for the followon patch that will
use these helper routines.  There are no functional changes in this patch.

Signed-off-by: Nathan Fontenot <nfont@austin.ibm.com>
---
 arch/powerpc/mm/numa.c |  209 +++++++++++++++++++++++++++++++++++++++----------
 1 file changed, 170 insertions(+), 39 deletions(-)

Index: linux-2.6.git/arch/powerpc/mm/numa.c
===================================================================
--- linux-2.6.git.orig/arch/powerpc/mm/numa.c	2008-07-01 18:40:55.000000000 -0500
+++ linux-2.6.git/arch/powerpc/mm/numa.c	2008-07-02 22:29:58.000000000 -0500
@@ -268,6 +268,144 @@
 	return result;
 }
 
+struct of_drconf_cell {
+	u64	base_addr;
+	u32	drc_index;
+	u32	reserved;
+	u32	aa_index;
+	u32	flags;
+};
+
+#define DRCONF_MEM_ASSIGNED	0x00000008
+#define DRCONF_MEM_AI_INVALID	0x00000040
+#define DRCONF_MEM_RESERVED	0x00000080
+
+/*
+ * Read the next lmb list entry from the ibm,dynamic-memory property
+ * and return the information in the provided of_drconf_cell structure.
+ */
+static void read_drconf_cell(struct of_drconf_cell *drmem, const u32 **cellp)
+{
+	const u32 *cp;
+
+	drmem->base_addr = read_n_cells(n_mem_addr_cells, cellp);
+
+	cp = *cellp;
+	drmem->drc_index = cp[0];
+	drmem->reserved = cp[1];
+	drmem->aa_index = cp[2];
+	drmem->flags = cp[3];
+
+	*cellp = cp + 4;
+}
+
+/*
+ * Retreive and validate the ibm,dynamic-memory property of the device tree.
+ *
+ * The layout of the ibm,dynamic-memory property is a number N of lmb
+ * list entries followed by N lmb list entries.  Each lmb list entry
+ * contains information as layed out in the of_drconf_cell struct above.
+ */
+static int of_get_drconf_memory(struct device_node *memory, const u32 **dm)
+{
+	const u32 *prop;
+	u32 len, entries;
+
+	prop = of_get_property(memory, "ibm,dynamic-memory", &len);
+	if (!prop || len < sizeof(unsigned int))
+		return 0;
+
+	entries = *prop++;
+
+	/* Now that we know the number of entries, revalidate the size
+	 * of the property read in to ensure we have everything
+	 */
+	if (len < (entries * (n_mem_addr_cells + 4) + 1) * sizeof(unsigned int))
+		return 0;
+
+	*dm = prop;
+	return entries;
+}
+
+/*
+ * Retreive and validate the ibm,lmb-size property for drconf memory
+ * from the device tree.
+ */
+static u64 of_get_lmb_size(struct device_node *memory)
+{
+	const u32 *prop;
+	u32 len;
+
+	prop = of_get_property(memory, "ibm,lmb-size", &len);
+	if (!prop || len < sizeof(unsigned int))
+		return 0;
+
+	return read_n_cells(n_mem_size_cells, &prop);
+}
+
+struct assoc_arrays {
+	u32	n_arrays;
+	u32	array_sz;
+	const u32 *arrays;
+};
+
+/*
+ * Retreive and validate the list of associativity arrays for drconf
+ * memory from the ibm,associativity-lookup-arrays property of the
+ * device tree..
+ *
+ * The layout of the ibm,associativity-lookup-arrays property is a number N
+ * indicating the number of associativity arrays, followed by a number M
+ * indicating the size of each associativity array, followed by a list
+ * of N associativity arrays.
+ */
+static int of_get_assoc_arrays(struct device_node *memory,
+			       struct assoc_arrays *aa)
+{
+	const u32 *prop;
+	u32 len;
+
+	prop = of_get_property(memory, "ibm,associativity-lookup-arrays", &len);
+	if (!prop || len < 2 * sizeof(unsigned int))
+		return -1;
+
+	aa->n_arrays = *prop++;
+	aa->array_sz = *prop++;
+
+	/* Now that we know the number of arrrays and size of each array,
+	 * revalidate the size of the property read in.
+	 */
+	if (len < (aa->n_arrays * aa->array_sz + 2) * sizeof(unsigned int))
+		return -1;
+
+	aa->arrays = prop;
+	return 0;
+}
+
+/*
+ * This is like of_node_to_nid_single() for memory represented in the
+ * ibm,dynamic-reconfiguration-memory node.
+ */
+static int of_drconf_to_nid_single(struct of_drconf_cell *drmem,
+				   struct assoc_arrays *aa)
+{
+	int default_nid = 0;
+	int nid = default_nid;
+	int index;
+
+	if (min_common_depth > 0 && min_common_depth <= aa->array_sz &&
+	    !(drmem->flags & DRCONF_MEM_AI_INVALID) &&
+	    drmem->aa_index < aa->n_arrays) {
+		index = drmem->aa_index * aa->array_sz + min_common_depth - 1;
+		nid = aa->arrays[index];
+
+		if (nid == 0xffff || nid >= MAX_NUMNODES)
+			nid = default_nid;
+	}
+
+	return nid;
+}
+
 /*
  * Figure out to which domain a cpu belongs and stick it there.
  * Return the id of the domain used.
@@ -355,57 +493,50 @@
  */
 static void __init parse_drconf_memory(struct device_node *memory)
 {
-	const unsigned int *lm, *dm, *aa;
-	unsigned int ls, ld, la;
-	unsigned int n, aam, aalen;
-	unsigned long lmb_size, size, start;
-	int nid, default_nid = 0;
-	unsigned int ai, flags;
-
-	lm = of_get_property(memory, "ibm,lmb-size", &ls);
-	dm = of_get_property(memory, "ibm,dynamic-memory", &ld);
-	aa = of_get_property(memory, "ibm,associativity-lookup-arrays", &la);
-	if (!lm || !dm || !aa ||
-	    ls < sizeof(unsigned int) || ld < sizeof(unsigned int) ||
-	    la < 2 * sizeof(unsigned int))
+	const u32 *dm;
+	unsigned int n, rc;
+	unsigned long lmb_size, size;
+	int nid;
+	struct assoc_arrays aa;
+
+	n = of_get_drconf_memory(memory, &dm);
+	if (!n)
 		return;
 
-	lmb_size = read_n_cells(n_mem_size_cells, &lm);
-	n = *dm++;		/* number of LMBs */
-	aam = *aa++;		/* number of associativity lists */
-	aalen = *aa++;		/* length of each associativity list */
-	if (ld < (n * (n_mem_addr_cells + 4) + 1) * sizeof(unsigned int) ||
-	    la < (aam * aalen + 2) * sizeof(unsigned int))
+	lmb_size = of_get_lmb_size(memory);
+	if (!lmb_size)
+		return;
+
+	rc = of_get_assoc_arrays(memory, &aa);
+	if (rc)
 		return;
 
 	for (; n != 0; --n) {
-		start = read_n_cells(n_mem_addr_cells, &dm);
-		ai = dm[2];
-		flags = dm[3];
-		dm += 4;
-		/* 0x80 == reserved, 0x8 = assigned to us */
-		if ((flags & 0x80) || !(flags & 0x8))
+		struct of_drconf_cell drmem;
+
+		read_drconf_cell(&drmem, &dm);
+
+		/* skip this block if the reserved bit is set in flags (0x80)
+		   or if the block is not assigned to this partition (0x8) */
+		if ((drmem.flags & DRCONF_MEM_RESERVED)
+		    || !(drmem.flags & DRCONF_MEM_ASSIGNED))
 			continue;
-		nid = default_nid;
-		/* flags & 0x40 means associativity index is invalid */
-		if (min_common_depth > 0 && min_common_depth <= aalen &&
-		    (flags & 0x40) == 0 && ai < aam) {
-			/* this is like of_node_to_nid_single */
-			nid = aa[ai * aalen + min_common_depth - 1];
-			if (nid == 0xffff || nid >= MAX_NUMNODES)
-				nid = default_nid;
-		}
 
-		fake_numa_create_new_node(((start + lmb_size) >> PAGE_SHIFT),
-						&nid);
+		nid = of_drconf_to_nid_single(&drmem, &aa);
+
+		fake_numa_create_new_node(
+				((drmem.base_addr + lmb_size) >> PAGE_SHIFT),
+					   &nid);
+
 		node_set_online(nid);
 
-		size = numa_enforce_memory_limit(start, lmb_size);
+		size = numa_enforce_memory_limit(drmem.base_addr, lmb_size);
 		if (!size)
 			continue;
 
-		add_active_range(nid, start >> PAGE_SHIFT,
-				 (start >> PAGE_SHIFT) + (size >> PAGE_SHIFT));
+		add_active_range(nid, drmem.base_addr >> PAGE_SHIFT,
+				 (drmem.base_addr >> PAGE_SHIFT)
+				 + (size >> PAGE_SHIFT));
 	}
 }