[PATCH] register hot-added memory to iomem resource

[PATCH] register hot-added memory to iomem resource

[KAMEZAWA Hiroyuki]

This patch registers hot-added memory to iomem_resource.
By this, /proc/iomem can show hot-added memory.
This patch is against 2.6.17-rc2-mm1.

Note: kdump uses /proc/iomem to catch memory range when it is installed.
      So, kdump should be re-installed after /proc/iomem change.

Signed-Off-By: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>

Index: linux-2.6.17-rc2-mm1/mm/memory_hotplug.c
===================================================================
--- linux-2.6.17-rc2-mm1.orig/mm/memory_hotplug.c	2006-04-27 18:00:17.000000000 +0900
+++ linux-2.6.17-rc2-mm1/mm/memory_hotplug.c	2006-04-27 20:21:32.000000000 +0900
@@ -21,6 +21,7 @@
 #include <linux/memory_hotplug.h>
 #include <linux/highmem.h>
 #include <linux/vmalloc.h>
+#include <linux/ioport.h>
 
 #include <asm/tlbflush.h>
 
@@ -188,6 +189,27 @@
 	return;
 }
 
+/* add this memory to iomem resource */
+static void register_memory_resource(u64 start, u64 size)
+{
+	struct resource *res;
+
+	res = kzalloc(sizeof(struct resource), GFP_KERNEL);
+	BUG_ON(!res);
+
+	res->name = "System RAM";
+	res->start = start;
+	res->end = start + size - 1;
+	res->flags = IORESOURCE_MEM;
+	if (request_resource(&iomem_resource, res) < 0) {
+		printk("System RAM resource %llx - %llx cannot be added\n",
+		(unsigned long long)res->start, (unsigned long long)res->end);
+		kfree(res);
+	}
+}
+
+
+
 int add_memory(int nid, u64 start, u64 size)
 {
 	pg_data_t *pgdat = NULL;
@@ -213,6 +235,9 @@
 	/* we online node here. we have no error path from here. */
 	node_set_online(nid);
 
+	/* register this memory as resource */
+	register_memory_resource(start, size);
+
 	return ret;
 error:
 	/* rollback pgdat allocation and others */

Re: [Lhms-devel] [PATCH] register hot-added memory to iomem resource

[jschopp]

> +/* add this memory to iomem resource */
> +static void register_memory_resource(u64 start, u64 size)
> +{
> +	struct resource *res;
> +
> +	res = kzalloc(sizeof(struct resource), GFP_KERNEL);
> +	BUG_ON(!res);
> +
> +	res->name = "System RAM";
> +	res->start = start;
> +	res->end = start + size - 1;
> +	res->flags = IORESOURCE_MEM;
> +	if (request_resource(&iomem_resource, res) < 0) {
> +		printk("System RAM resource %llx - %llx cannot be added\n",
> +		(unsigned long long)res->start, (unsigned long long)res->end);
> +		kfree(res);
> +	}
> +}

This is more of a question than a comment.  Is this code saying that all memory we are 
adding is available for IO?  And is it really true that on all platforms any memory we add 
can be used that way?

Re: [PATCH] register hot-added memory to iomem resource

[Andrew Morton]

KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> wrote:
>
> This patch registers hot-added memory to iomem_resource.
> By this, /proc/iomem can show hot-added memory.
> This patch is against 2.6.17-rc2-mm1.
> 
> Note: kdump uses /proc/iomem to catch memory range when it is installed.
>       So, kdump should be re-installed after /proc/iomem change.
> 

What do you mean by "kdump should be reinstalled"?  The kdump userspace
tools need to re-run kexec_load()?

If so, why?

And how is kdump to know that memory was hot-added?  Do we generate a
hotplug event?

Thanks.

Re: [Lhms-devel] Re: [PATCH] register hot-added memory to iomem resource

[KAMEZAWA Hiroyuki]

On Thu, 27 Apr 2006 16:01:30 -0700
Andrew Morton <akpm@osdl.org> wrote:

> KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> wrote:
> >
> > This patch registers hot-added memory to iomem_resource.
> > By this, /proc/iomem can show hot-added memory.
> > This patch is against 2.6.17-rc2-mm1.
> > 
> > Note: kdump uses /proc/iomem to catch memory range when it is installed.
> >       So, kdump should be re-installed after /proc/iomem change.
> > 
> 
> What do you mean by "kdump should be reinstalled"?  The kdump userspace
> tools need to re-run kexec_load()?
> 
yes. I heard an admin has to re-run kexec_load.
- http://www.uwsg.indiana.edu/hypermail/linux/kernel/0604.0/0821.html
- http://www.uwsg.indiana.edu/hypermail/linux/kernel/0604.0/0829.html
Added CC to ebiederm@xmission.com, nanhai.zou@intel.com

> If so, why?
> 
It reads physical memory list from /proc/iomem now.
The physical memory list is read and saved at kdump kernel loading time
instead of crashing time. 

> And how is kdump to know that memory was hot-added?  Do we generate a
> hotplug event?
> 
A user program has to make memory section online from sysfs , anyway.

The hotplug script for memory hotplug will run at memory hotplug event 
from ACPI. If a user uses /probe interface (powerpc, x86_64),
he knows what he does. 

hot-add -> online memory -> kexec_load() is a scenario I think of.

Thanks,
-Kame

Re: [Lhms-devel] [PATCH] register hot-added memory to iomem resource

[KAMEZAWA Hiroyuki]

On Thu, 27 Apr 2006 11:10:16 -0500
jschopp <jschopp@austin.ibm.com> wrote:

> > +/* add this memory to iomem resource */
> > +static void register_memory_resource(u64 start, u64 size)
> > +{
> > +	struct resource *res;
> > +
> > +	res = kzalloc(sizeof(struct resource), GFP_KERNEL);
> > +	BUG_ON(!res);
> > +
> > +	res->name = "System RAM";
> > +	res->start = start;
> > +	res->end = start + size - 1;
> > +	res->flags = IORESOURCE_MEM;
> > +	if (request_resource(&iomem_resource, res) < 0) {
> > +		printk("System RAM resource %llx - %llx cannot be added\n",
> > +		(unsigned long long)res->start, (unsigned long long)res->end);
> > +		kfree(res);
> > +	}
> > +}
> 
> This is more of a question than a comment.  Is this code saying that all memory we are 
> adding is available for IO?  And is it really true that on all platforms any memory we add 
> can be used that way?
> 
No. However list name is iomem_resource (from historical reason), 
This registration doesn't mean memory for IO, just shows memory-map.
( I consider so...)

I think this modification is done on ia64 recently.
- http://www.gelato.unsw.edu.au/archives/linux-ia64/0507/14758.html

If powerpc people doesn't like this, add new CONFIG will be necessary.
(like CONFIG_EXPORT_SYSTEM_RAM) But I think powerpc can export memory range
from /proc as x86/x86_64/ia64.
(But I don't have powerpc, I can't test/write patch)

Example: my desctop
When cat /proc/iomem, System RAM is shown in followingf way.

==
00000000-0009fbff : System RAM    <------
0009fc00-0009ffff : reserved
000a0000-000bffff : Video RAM area
000c0000-000cbfff : Video ROM
000f0000-000fffff : System ROM
00100000-2dfeffff : System RAM    <------
  00100000-003e0f48 : Kernel code
  003e0f49-00542003 : Kernel data
2dff0000-2dff2fff : ACPI Non-volatile Storage
2dff3000-2dffffff : ACPI Tables
30000000-31ffffff : PCI CardBus #02
32000000-33ffffff : PCI CardBus #02
34000000-35ffffff : PCI CardBus #06
36000000-37ffffff : PCI CardBus #06
e0000000-e7ffffff : PCI Bus #01
  e0000000-e7ffffff : 0000:01:00.0
e8000000-ebffffff : 0000:00:00.0
ec000000-ec0fffff : PCI Bus #01
  ec000000-ec01ffff : 0000:01:00.0
ec100000-ec1fffff : 0000:00:09.0
ec200000-ec20ffff : 0000:00:08.0
ec210000-ec210fff : 0000:00:03.3
  ec210000-ec210fff : ehci_hcd
ec211000-ec211fff : 0000:00:03.0
  ec211000-ec211fff : ohci_hcd
ec212000-ec212fff : 0000:00:03.1
  ec212000-ec212fff : ohci_hcd
ec213000-ec2130ff : 0000:00:0c.0
  ec213000-ec2130ff : 8139too
ec214000-ec214fff : 0000:00:0d.0
ec219000-ec219fff : 0000:00:0d.1
ec21e000-ec21e7ff : 0000:00:0d.2
  ec21e000-ec21e7ff : ohci1394
ec21f000-ec21f0ff : 0000:00:0d.3
ec220000-ec2200ff : 0000:00:0d.4
fec00000-fec00fff : reserved
fee00000-fee00fff : reserved
ffff0000-ffffffff : reserved

Re: [Lhms-devel] Re: [PATCH] register hot-added memory to iomem resource

[Vivek Goyal]

On Fri, Apr 28, 2006 at 09:27:54AM +0900, KAMEZAWA Hiroyuki wrote:
> On Thu, 27 Apr 2006 16:01:30 -0700
> Andrew Morton <akpm@osdl.org> wrote:
> 
> > KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> wrote:
> > >
> > > This patch registers hot-added memory to iomem_resource.
> > > By this, /proc/iomem can show hot-added memory.
> > > This patch is against 2.6.17-rc2-mm1.
> > > 
> > > Note: kdump uses /proc/iomem to catch memory range when it is installed.
> > >       So, kdump should be re-installed after /proc/iomem change.
> > > 
> > 
> > What do you mean by "kdump should be reinstalled"?  The kdump userspace
> > tools need to re-run kexec_load()?
> > 
> yes. I heard an admin has to re-run kexec_load.
> - http://www.uwsg.indiana.edu/hypermail/linux/kernel/0604.0/0821.html
> - http://www.uwsg.indiana.edu/hypermail/linux/kernel/0604.0/0829.html
> Added CC to ebiederm@xmission.com, nanhai.zou@intel.com
> 
> > If so, why?
> > 
> It reads physical memory list from /proc/iomem now.
> The physical memory list is read and saved at kdump kernel loading time
> instead of crashing time. 
> 

True. If some meory is added, kdump kernel has to be re-loaded.

> > And how is kdump to know that memory was hot-added?  Do we generate a
> > hotplug event?
> > 
> A user program has to make memory section online from sysfs , anyway.
> 
> The hotplug script for memory hotplug will run at memory hotplug event 
> from ACPI. If a user uses /probe interface (powerpc, x86_64),
> he knows what he does. 
> 
> hot-add -> online memory -> kexec_load() is a scenario I think of.
> 

I will look into it.

Thanks
Vivek

Re: [Lhms-devel] Re: [PATCH] register hot-added memory to iomem resource

[Vivek Goyal]

On Fri, Apr 28, 2006 at 09:27:54AM +0900, KAMEZAWA Hiroyuki wrote:
> > And how is kdump to know that memory was hot-added?  Do we generate a
> > hotplug event?
> > 
> A user program has to make memory section online from sysfs , anyway.
> 
> The hotplug script for memory hotplug will run at memory hotplug event 
> from ACPI. If a user uses /probe interface (powerpc, x86_64),
> he knows what he does. 
> 
> hot-add -> online memory -> kexec_load() is a scenario I think of.
> 

Did a quick search but can't locate a memory hotplug agent. Can you give
some pointers.

Thanks
Vivek

Re: [Lhms-devel] Re: [PATCH] register hot-added memory to iomem resource

[KAMEZAWA Hiroyuki]

On Fri, 28 Apr 2006 14:43:49 -0400
Vivek Goyal <vgoyal@in.ibm.com> wrote:

> On Fri, Apr 28, 2006 at 09:27:54AM +0900, KAMEZAWA Hiroyuki wrote:
> > > And how is kdump to know that memory was hot-added?  Do we generate a
> > > hotplug event?
> > > 
> > A user program has to make memory section online from sysfs , anyway.
> > 
> > The hotplug script for memory hotplug will run at memory hotplug event 
> > from ACPI. If a user uses /probe interface (powerpc, x86_64),
> > he knows what he does. 
> > 
> > hot-add -> online memory -> kexec_load() is a scenario I think of.
> > 
> 
> Did a quick search but can't locate a memory hotplug agent. Can you give
> some pointers.
> 
Sorry, I think there is no generic memory hotplug agent now.
I do it by hand or handmaid scripts.
(Just because our target, NUMA-node-hotplug, is now under construction.)

In theory, ACPI namespace notifier calls memory hotplug agent and it just does
%echo online > /sys/devices/system/memory/memoryXXX/state.

After this, new memory is available.
This page has a bit more information. http://lhms.sourceforge.net/

BTW, could you teach me pointers to kexec/kload tools ?

-Kame