[RFC, 6/7] NUMA hotplug emulator

[RFC, 6/7] NUMA hotplug emulator

[Shaohui Zheng]

[-- Attachment #1: Type: text/plain, Size: 2829 bytes --]

hotplug emulator:extend memory probe interface to support NUMA

Extend memory probe interface to support an extra paramter nid,
the reserved memory can be added into this node if node exists.

Add a memory section(128M) to node 3(boots with mem=1024m)

	echo 0x40000000,3 > memory/probe

And more we make it friendly, it is possible to add memory to do

	echo 3g > memory/probe
	echo 1024m,3 > memory/probe

It maintains backwards compatibility.

Signed-off-by: Shaohui Zheng <shaohui.zheng@intel.com>
Signed-off-by: Haicheng Li <haicheng.li@intel.com>
Signed-off-by: Wu Fengguang <fengguang.wu@intel.com>
---
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index 54ccb0d..787024f 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -1239,6 +1239,17 @@ config ARCH_CPU_PROBE_RELEASE
 	  is for cpu hot-add/hot-remove to specified node in software method.
 	  This is for debuging and testing purpose
 
+config ARCH_MEMORY_PROBE
+	def_bool y
+	bool "Memory hotplug emulation"
+	depends on NUMA_HOTPLUG_EMU
+	---help---
+	  Enable memory hotplug emulation. Reserve memory with grub parameter
+	  "mem=N"(such as mem=1024M), where N is the initial memory size, the
+	  rest physical memory will be removed from e820 table; the memory probe
+	  interface is for memory hot-add to specified node in software method.
+	  This is for debuging and testing purpose
+
 config NODES_SHIFT
 	int "Maximum NUMA Nodes (as a power of 2)" if !MAXSMP
 	range 1 10
diff --git a/drivers/base/memory.c b/drivers/base/memory.c
index 933442f..1c2d83d 100644
--- a/drivers/base/memory.c
+++ b/drivers/base/memory.c
@@ -329,6 +329,8 @@ static int block_size_init(void)
  * will not need to do it from userspace.  The fake hot-add code
  * as well as ppc64 will do all of their discovery in userspace
  * and will require this interface.
+ *
+ * Parameter format: start_addr, nid
  */
 #ifdef CONFIG_ARCH_MEMORY_PROBE
 static ssize_t
@@ -339,10 +341,26 @@ memory_probe_store(struct class *class, struct class_attribute *attr,
 	int nid;
 	int ret;
 
-	phys_addr = simple_strtoull(buf, NULL, 0);
+	char *p = strchr(buf, ',');
+
+	if (p != NULL && strlen(p+1) > 0) {
+		/* nid specified */
+		*p++ = '\0';
+		nid = simple_strtoul(p, NULL, 0);
+		phys_addr = memparse(buf, NULL);
+	} else {
+		phys_addr = memparse(buf, NULL);
+		nid = memory_add_physaddr_to_nid(phys_addr);
+	}
 
-	nid = memory_add_physaddr_to_nid(phys_addr);
-	ret = add_memory(nid, phys_addr, PAGES_PER_SECTION << PAGE_SHIFT);
+	if (nid < 0 || nid > nr_node_ids - 1) {
+		printk(KERN_ERR "Invalid node id %d(0<=nid<%d).\n", nid, nr_node_ids);
+	} else {
+		printk(KERN_INFO "Add a memory section to node: %d.\n", nid);
+		ret = add_memory(nid, phys_addr, PAGES_PER_SECTION << PAGE_SHIFT);
+		if (ret)
+			count = ret;
+	}
 
 	if (ret)
 		count = ret;
-- 
Thanks & Regards,
Shaohui


[-- Attachment #2: 006-hotplug-emulator-extend-memory-probe-interface-to-support-numa.patch --]
[-- Type: text/x-diff, Size: 2798 bytes --]

hotplug emulator:extend memory probe interface to support NUMA

Extend memory probe interface to support an extra paramter nid,
the reserved memory can be added into this node if node exists.

Add a memory section(128M) to node 3(boots with mem=1024m)

	echo 0x40000000,3 > memory/probe

And more we make it friendly, it is possible to add memory to do

	echo 3g > memory/probe
	echo 1024m,3 > memory/probe

It maintains backwards compatibility.

Signed-off-by: Shaohui Zheng <shaohui.zheng@intel.com>
Signed-off-by: Haicheng Li <haicheng.li@intel.com>
Signed-off-by: Wu Fengguang <fengguang.wu@intel.com>
---
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index 54ccb0d..787024f 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -1239,6 +1239,17 @@ config ARCH_CPU_PROBE_RELEASE
 	  is for cpu hot-add/hot-remove to specified node in software method.
 	  This is for debuging and testing purpose
 
+config ARCH_MEMORY_PROBE
+	def_bool y
+	bool "Memory hotplug emulation"
+	depends on NUMA_HOTPLUG_EMU
+	---help---
+	  Enable memory hotplug emulation. Reserve memory with grub parameter
+	  "mem=N"(such as mem=1024M), where N is the initial memory size, the
+	  rest physical memory will be removed from e820 table; the memory probe
+	  interface is for memory hot-add to specified node in software method.
+	  This is for debuging and testing purpose
+
 config NODES_SHIFT
 	int "Maximum NUMA Nodes (as a power of 2)" if !MAXSMP
 	range 1 10
diff --git a/drivers/base/memory.c b/drivers/base/memory.c
index 933442f..1c2d83d 100644
--- a/drivers/base/memory.c
+++ b/drivers/base/memory.c
@@ -329,6 +329,8 @@ static int block_size_init(void)
  * will not need to do it from userspace.  The fake hot-add code
  * as well as ppc64 will do all of their discovery in userspace
  * and will require this interface.
+ *
+ * Parameter format: start_addr, nid
  */
 #ifdef CONFIG_ARCH_MEMORY_PROBE
 static ssize_t
@@ -339,10 +341,26 @@ memory_probe_store(struct class *class, struct class_attribute *attr,
 	int nid;
 	int ret;
 
-	phys_addr = simple_strtoull(buf, NULL, 0);
+	char *p = strchr(buf, ',');
+
+	if (p != NULL && strlen(p+1) > 0) {
+		/* nid specified */
+		*p++ = '\0';
+		nid = simple_strtoul(p, NULL, 0);
+		phys_addr = memparse(buf, NULL);
+	} else {
+		phys_addr = memparse(buf, NULL);
+		nid = memory_add_physaddr_to_nid(phys_addr);
+	}
 
-	nid = memory_add_physaddr_to_nid(phys_addr);
-	ret = add_memory(nid, phys_addr, PAGES_PER_SECTION << PAGE_SHIFT);
+	if (nid < 0 || nid > nr_node_ids - 1) {
+		printk(KERN_ERR "Invalid node id %d(0<=nid<%d).\n", nid, nr_node_ids);
+	} else {
+		printk(KERN_INFO "Add a memory section to node: %d.\n", nid);
+		ret = add_memory(nid, phys_addr, PAGES_PER_SECTION << PAGE_SHIFT);
+		if (ret)
+			count = ret;
+	}
 
 	if (ret)
 		count = ret;

Re: [RFC, 6/7] NUMA hotplug emulator

[Greg KH]

On Thu, May 13, 2010 at 08:00:16PM +0800, Shaohui Zheng wrote:
> hotplug emulator:extend memory probe interface to support NUMA
> 
> Extend memory probe interface to support an extra paramter nid,
> the reserved memory can be added into this node if node exists.
> 
> Add a memory section(128M) to node 3(boots with mem=1024m)
> 
> 	echo 0x40000000,3 > memory/probe
> 
> And more we make it friendly, it is possible to add memory to do
> 
> 	echo 3g > memory/probe
> 	echo 1024m,3 > memory/probe
> 
> It maintains backwards compatibility.

Again, please document this.

thanks,

greg k-h

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFC, 6/7] NUMA hotplug emulator

[Dave Hansen]

On Thu, 2010-05-13 at 09:56 -0700, Greg KH wrote:
> On Thu, May 13, 2010 at 08:00:16PM +0800, Shaohui Zheng wrote:
> > hotplug emulator:extend memory probe interface to support NUMA
> > 
> > Extend memory probe interface to support an extra paramter nid,
> > the reserved memory can be added into this node if node exists.
> > 
> > Add a memory section(128M) to node 3(boots with mem=1024m)
> > 
> >       echo 0x40000000,3 > memory/probe

I dunno.  If we're going to put multiple values into the file now and
add to the ABI, can we be more explicit about it?

	echo "physical_address=0x40000000 numa_node=3" > memory/probe

I'd *GREATLY* prefer that over this new syntax.  The existing mechanism
is obtuse enough, and the ',3' makes it more so.

We should have the code around to parse arguments like that, too, since
we use it for the boot command-line.

-- Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFC, 6/7] NUMA hotplug emulator

[Greg KH]

On Thu, May 13, 2010 at 11:02:17AM -0700, Dave Hansen wrote:
> On Thu, 2010-05-13 at 09:56 -0700, Greg KH wrote:
> > On Thu, May 13, 2010 at 08:00:16PM +0800, Shaohui Zheng wrote:
> > > hotplug emulator:extend memory probe interface to support NUMA
> > > 
> > > Extend memory probe interface to support an extra paramter nid,
> > > the reserved memory can be added into this node if node exists.
> > > 
> > > Add a memory section(128M) to node 3(boots with mem=1024m)
> > > 
> > >       echo 0x40000000,3 > memory/probe
> 
> I dunno.  If we're going to put multiple values into the file now and
> add to the ABI, can we be more explicit about it?
> 
> 	echo "physical_address=0x40000000 numa_node=3" > memory/probe
> 
> I'd *GREATLY* prefer that over this new syntax.  The existing mechanism
> is obtuse enough, and the ',3' makes it more so.
> 
> We should have the code around to parse arguments like that, too, since
> we use it for the boot command-line.

If you are going to be doing something like this, please use configfs,
that is what it is designed for, not sysfs.

thanks,

greg k-h

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFC, 6/7] NUMA hotplug emulator

[Dave Hansen]

On Thu, 2010-05-13 at 11:15 -0700, Greg KH wrote:
> >       echo "physical_address=0x40000000 numa_node=3" > memory/probe
> > 
> > I'd *GREATLY* prefer that over this new syntax.  The existing mechanism
> > is obtuse enough, and the ',3' makes it more so.
> > 
> > We should have the code around to parse arguments like that, too, since
> > we use it for the boot command-line.
> 
> If you are going to be doing something like this, please use configfs,
> that is what it is designed for, not sysfs.

That's probably a really good point, especially since configfs didn't
even exist when we made this 'probe' file thingy.  It never was a great
fit for sysfs anyway.

-- Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFC, 6/7] NUMA hotplug emulator

[Greg KH]

On Thu, May 13, 2010 at 11:49:38AM -0700, Dave Hansen wrote:
> On Thu, 2010-05-13 at 11:15 -0700, Greg KH wrote:
> > >       echo "physical_address=0x40000000 numa_node=3" > memory/probe
> > > 
> > > I'd *GREATLY* prefer that over this new syntax.  The existing mechanism
> > > is obtuse enough, and the ',3' makes it more so.
> > > 
> > > We should have the code around to parse arguments like that, too, since
> > > we use it for the boot command-line.
> > 
> > If you are going to be doing something like this, please use configfs,
> > that is what it is designed for, not sysfs.
> 
> That's probably a really good point, especially since configfs didn't
> even exist when we made this 'probe' file thingy.  It never was a great
> fit for sysfs anyway.

Really?  configfs was added in 2.6.16, when was this probe file added?

thanks,

greg k-h

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFC, 6/7] NUMA hotplug emulator

[Dave Hansen]

On Thu, 2010-05-13 at 11:58 -0700, Greg KH wrote:
> > That's probably a really good point, especially since configfs didn't
> > even exist when we made this 'probe' file thingy.  It never was a great
> > fit for sysfs anyway.
> 
> Really?  configfs was added in 2.6.16, when was this probe file added?

$ git name-rev 3947be19
3947be19 tags/v2.6.15-rc1~728^2~12

-- Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFC, 6/7] NUMA hotplug emulator

[Greg KH]

On Thu, May 13, 2010 at 12:16:43PM -0700, Dave Hansen wrote:
> On Thu, 2010-05-13 at 11:58 -0700, Greg KH wrote:
> > > That's probably a really good point, especially since configfs didn't
> > > even exist when we made this 'probe' file thingy.  It never was a great
> > > fit for sysfs anyway.
> > 
> > Really?  configfs was added in 2.6.16, when was this probe file added?
> 
> $ git name-rev 3947be19
> 3947be19 tags/v2.6.15-rc1~728^2~12

Ah, so close, off by 3 months :)

thanks,

greg k-h

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFC, 6/7] NUMA hotplug emulator

[Shaohui Zheng]

On Thu, May 13, 2010 at 11:49:38AM -0700, Dave Hansen wrote:
> On Thu, 2010-05-13 at 11:15 -0700, Greg KH wrote:
> > >       echo "physical_address=0x40000000 numa_node=3" > memory/probe
> > > 
> > > I'd *GREATLY* prefer that over this new syntax.  The existing mechanism
> > > is obtuse enough, and the ',3' makes it more so.
> > > 
> > > We should have the code around to parse arguments like that, too, since
> > > we use it for the boot command-line.
> > 
> > If you are going to be doing something like this, please use configfs,
> > that is what it is designed for, not sysfs.
> 
> That's probably a really good point, especially since configfs didn't
> even exist when we made this 'probe' file thingy.  It never was a great
> fit for sysfs anyway.
> 
> -- Dave

the configfs was introduced in 2005, you can refer to http://lwn.net/Articles/148973/.

I enabled the configfs, and I see that the configfs is not so popular as we expected,
I mount configfs to /sys/kernel/config, I get an empty directory. It means that nobody is 
using this file system, it is an interesting thing, is it means that configfs is deprecated?
If so, it might not be nessarry to develop a configfs interface for hotplug.

Dave & Greg,
	Can you provide an exmample to use configfs as interface in Linux kernel, I want to get
a live demo, thanks.

-- 
Thanks & Regards,
Shaohui

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFC, 6/7] NUMA hotplug emulator

[Dave Hansen]

On Tue, 2010-05-18 at 13:41 +0800, Shaohui Zheng wrote:
> On Thu, May 13, 2010 at 11:49:38AM -0700, Dave Hansen wrote:
> the configfs was introduced in 2005, you can refer to http://lwn.net/Articles/148973/.
> 
> I enabled the configfs, and I see that the configfs is not so popular as we expected,
> I mount configfs to /sys/kernel/config, I get an empty directory. It means that nobody is 
> using this file system, it is an interesting thing, is it means that configfs is deprecated?
> If so, it might not be nessarry to develop a configfs interface for hotplug.

Uh, deprecated?  What would make you think that?  It does look like the
users are a we bit obscure, but that's a bit far from deprecated.

> Dave & Greg,
> 	Can you provide an exmample to use configfs as interface in Linux kernel, I want to get
> a live demo, thanks.

Heh.  There are some great tools out there called cscope and grep.  I
have them on my system and I bet you can get them on yours too.

That said, you're right.  There don't seem to be a ton of users of it
these days.  But, the LWN article you referenced also pointed to at
least one user.  So, please try and put a wee bit of effort into it.

Maybe configfs isn't the way to go.  I just think extending the 'probe'
file is a bad idea, especially in the way your patch did it.  I'm open
to other alternatives.  Since this is only for testing, perhaps debugfs
applies better.  What other alternatives have you explored?  How about a
Systemtap set to do it? :)

-- Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFC, 6/7] NUMA hotplug emulator

[Shaohui Zheng]

On Tue, May 18, 2010 at 12:27:05AM -0700, Dave Hansen wrote:
> On Tue, 2010-05-18 at 13:41 +0800, Shaohui Zheng wrote:
> > On Thu, May 13, 2010 at 11:49:38AM -0700, Dave Hansen wrote:
> > the configfs was introduced in 2005, you can refer to http://lwn.net/Articles/148973/.
> > 
> > I enabled the configfs, and I see that the configfs is not so popular as we expected,
> > I mount configfs to /sys/kernel/config, I get an empty directory. It means that nobody is 
> > using this file system, it is an interesting thing, is it means that configfs is deprecated?
> > If so, it might not be nessarry to develop a configfs interface for hotplug.
> 
> Uh, deprecated?  What would make you think that?  It does look like the
> users are a we bit obscure, but that's a bit far from deprecated.
> 
> > Dave & Greg,
> > 	Can you provide an exmample to use configfs as interface in Linux kernel, I want to get
> > a live demo, thanks.
> 
> Heh.  There are some great tools out there called cscope and grep.  I
> have them on my system and I bet you can get them on yours too.
> 
> That said, you're right.  There don't seem to be a ton of users of it
> these days.  But, the LWN article you referenced also pointed to at
> least one user.  So, please try and put a wee bit of effort into it.

I am trying to put a few efforts on it, and we hope to support both sysfs and configfs.

because of the history reason, sysfs is more popular. I bet that only vew few users know
 configfs, so sysfs is still the prefered way.

> 
> Maybe configfs isn't the way to go.  I just think extending the 'probe'
> file is a bad idea, especially in the way your patch did it.  I'm open
> to other alternatives.  Since this is only for testing, perhaps debugfs
> applies better.  What other alternatives have you explored?  How about a
> Systemtap set to do it? :)
> 
> -- Dave

Did not try other alternatives until now. Thanks for Dave's suggestions, Systemtap and debugfs
 seems to be good methods, too.


-- 
Thanks & Regards,
Shaohui

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFC, 6/7] NUMA hotplug emulator

[Andi Kleen]

>
> Maybe configfs isn't the way to go.  I just think extending the 'probe'
> file is a bad idea, especially in the way your patch did it.  I'm open
> to other alternatives.  Since this is only for testing, perhaps debugfs
> applies better.  What other alternatives have you explored?  How about a
> Systemtap set to do it? :)

First this is a debugging interface. It doesn't need to have the
most pretty interface in the world, because it will be only used for
QA by a few people.

Requiring setting parameters in two different file systems doesn't
sound that appealing to me.

systemtap for configuration also doesn't seem right.

I liked Dave's earlier proposal to do a command line parameter like interface
for "probe". Perhaps that can be done. It shouldn't need a lot of code.

In fact there are already two different parser libraries for this:
lib/parser.c and lib/params.c. One could chose the one that one likes
better :-)

Anything that needs a lot of code is a bad idea for this I think.
A simple parser using one of the existing libraries should be simple
enough though.

Again it's just a QA interface, not the next generation of POSIX.

-Andi

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFC, 6/7] NUMA hotplug emulator

[Dave Hansen]

On Tue, 2010-05-18 at 10:55 +0200, Andi Kleen wrote:
> I liked Dave's earlier proposal to do a command line parameter like interface
> for "probe". Perhaps that can be done. It shouldn't need a lot of code.

After looking at the code, configfs doesn't look to me like it can be
done horribly easily.  It takes a least a subsystem and then a few
structures to get things up and running.  There also doesn't appear to
be a good subsystem to plug into.

> In fact there are already two different parser libraries for this:
> lib/parser.c and lib/params.c. One could chose the one that one likes
> better :-)

Agreed.  But, I do see why Greg is suggesting configfs here.
Superficially, it seems like a good configfs fit, but I think configfs
is only a good fit when you need to cram a _bunch_ of stuff into a _new_
interface.  Here, we have a relatively tiny amount of data that has half
of what it needs from an existing interface.  

-- Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFC, 6/7] NUMA hotplug emulator

[Nicholas A. Bellinger]

On Tue, 2010-05-18 at 13:41 +0800, Shaohui Zheng wrote:
> On Thu, May 13, 2010 at 11:49:38AM -0700, Dave Hansen wrote:
> > On Thu, 2010-05-13 at 11:15 -0700, Greg KH wrote:
> > > >       echo "physical_address=0x40000000 numa_node=3" > memory/probe
> > > > 
> > > > I'd *GREATLY* prefer that over this new syntax.  The existing mechanism
> > > > is obtuse enough, and the ',3' makes it more so.
> > > > 
> > > > We should have the code around to parse arguments like that, too, since
> > > > we use it for the boot command-line.
> > > 
> > > If you are going to be doing something like this, please use configfs,
> > > that is what it is designed for, not sysfs.
> > 
> > That's probably a really good point, especially since configfs didn't
> > even exist when we made this 'probe' file thingy.  It never was a great
> > fit for sysfs anyway.
> > 
> > -- Dave
> 
> the configfs was introduced in 2005, you can refer to http://lwn.net/Articles/148973/.
> 
> I enabled the configfs, and I see that the configfs is not so popular as we expected,
> I mount configfs to /sys/kernel/config, I get an empty directory. It means that nobody is 
> using this file system, it is an interesting thing, is it means that configfs is deprecated?

Ohhhhhh, no.  ConfigFS is the evolution of the original SysFS design to
to allow for kernel data structure configuration to be driven by
userspace syscalls in a number of very significant ways.

> If so, it might not be nessarry to develop a configfs interface for hotplug.
> 

The usage of ConfigFS to provide a kernel <-> user configuration layout
really best depends on the protocol in question for particular data
structure state machine and parameter/attribute set.  Using ConfigFS
involves Linux/VFS representing dependencies between data structures
both on a inter and intra kernel module context containing struct
config_groups driven by userspace mkdir(2) and link(2) syscall ops.

> Dave & Greg,
> 	Can you provide an exmample to use configfs as interface in Linux kernel, I want to get
> a live demo, thanks.

The TCM 4.0 design brings fabric module independent >= SPC-3 compatible
SCSI WWN target ports and a generic set of struct config_groups and CPP
macros to individual storage backstores using TCM target mode fabric
plugins across Linux/SCSI, Linux/BLOCK, and Linux/VFS subsystems.  So
far, this has been implemented for SAS, FC, and iSCSI fabric protocols:

http://git.kernel.org/?p=linux/kernel/git/nab/lio-core-2.6.git;a=blob;f=drivers/target/target_core_configfs.c;hb=refs/heads/lio-4.0
http://git.kernel.org/?p=linux/kernel/git/nab/lio-core-2.6.git;a=blob;f=include/target/configfs_macros.h;hb=refs/heads/lio-4.0
http://git.kernel.org/?p=linux/kernel/git/nab/lio-core-2.6.git;a=blob;f=drivers/target/target_core_fabric_configfs.c;hb=refs/heads/lio-4.0
http://git.kernel.org/?p=linux/kernel/git/nab/lio-core-2.6.git;a=blob;f=include/target/target_core_fabric_configfs.h;hb=refs/heads/lio-4.0

http://git.kernel.org/?p=linux/kernel/git/nab/lio-core-2.6.git;a=blob;f=drivers/target/tcm_fc/tfc_conf.c;hb=refs/heads/lio-4.0
http://git.kernel.org/?p=linux/kernel/git/nab/lio-core-2.6.git;a=blob;f=drivers/target/tcm_loop/tcm_loop_configfs.c;hb=refs/heads/lio-4.0
http://git.kernel.org/?p=linux/kernel/git/nab/lio-core-2.6.git;a=blob;f=drivers/target/lio-target/iscsi_target_configfs.c;hb=refs/heads/lio-4.0

Best,

--nab

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFC, 6/7] NUMA hotplug emulator

[Paul Mundt]

On Tue, May 18, 2010 at 01:41:21PM +0800, Shaohui Zheng wrote:
> On Thu, May 13, 2010 at 11:49:38AM -0700, Dave Hansen wrote:
> > On Thu, 2010-05-13 at 11:15 -0700, Greg KH wrote:
> > > >       echo "physical_address=0x40000000 numa_node=3" > memory/probe
> > > > 
> > > > I'd *GREATLY* prefer that over this new syntax.  The existing mechanism
> > > > is obtuse enough, and the ',3' makes it more so.
> > > > 
> > > > We should have the code around to parse arguments like that, too, since
> > > > we use it for the boot command-line.
> > > 
> > > If you are going to be doing something like this, please use configfs,
> > > that is what it is designed for, not sysfs.
> > 
> > That's probably a really good point, especially since configfs didn't
> > even exist when we made this 'probe' file thingy.  It never was a great
> > fit for sysfs anyway.
> > 
> > -- Dave
> 
> the configfs was introduced in 2005, you can refer to http://lwn.net/Articles/148973/.
> 
> I enabled the configfs, and I see that the configfs is not so popular as we expected,
> I mount configfs to /sys/kernel/config, I get an empty directory. It means that nobody is 
> using this file system, it is an interesting thing, is it means that configfs is deprecated?
> If so, it might not be nessarry to develop a configfs interface for hotplug.
> 
configfs is certainly not deprecated, but there are also not that many
users of it at present. dlm/ocfs2 were the first users as far as I
recall, and netconsole as well. The fact you have an empty directory just
indicates that you don't have support for any of these enabled.

Note that there are also a lot of present-day sysfs and debugfs users
that could/should be converted to configfs but haven't quite gotten there
yet. In the sysfs case abuses are hard to rollback once they've made
become part of the ABI, but that's not grounds for continuing sysfs abuse
once cleaner methods become available. Many of the sysfs abuses were
implemented before configfs existed.

You can also find usage guidelines and example implementations in
Documentation/filesystems/configfs, which should give you a pretty good
idea of whether it's a good interface fit for your particular problem or
not.

These days sysfs seems to be the new procfs. It certainly helps to put a
bit of planning in to the interface before you're invariably stuck with
an ABI that's barely limping along.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFC, 6/7] NUMA hotplug emulator

[Shaohui Zheng]

On Tue, May 18, 2010 at 04:44:32PM +0900, Paul Mundt wrote:
> On Tue, May 18, 2010 at 01:41:21PM +0800, Shaohui Zheng wrote:
> > On Thu, May 13, 2010 at 11:49:38AM -0700, Dave Hansen wrote:
> > > On Thu, 2010-05-13 at 11:15 -0700, Greg KH wrote:
> > > > >       echo "physical_address=0x40000000 numa_node=3" > memory/probe
> > > > > 
> > > > > I'd *GREATLY* prefer that over this new syntax.  The existing mechanism
> > > > > is obtuse enough, and the ',3' makes it more so.
> > > > > 
> > > > > We should have the code around to parse arguments like that, too, since
> > > > > we use it for the boot command-line.
> > > > 
> > > > If you are going to be doing something like this, please use configfs,
> > > > that is what it is designed for, not sysfs.
> > > 
> > > That's probably a really good point, especially since configfs didn't
> > > even exist when we made this 'probe' file thingy.  It never was a great
> > > fit for sysfs anyway.
> > > 
> > > -- Dave
> > 
> > the configfs was introduced in 2005, you can refer to http://lwn.net/Articles/148973/.
> > 
> > I enabled the configfs, and I see that the configfs is not so popular as we expected,
> > I mount configfs to /sys/kernel/config, I get an empty directory. It means that nobody is 
> > using this file system, it is an interesting thing, is it means that configfs is deprecated?
> > If so, it might not be nessarry to develop a configfs interface for hotplug.
> > 
> configfs is certainly not deprecated, but there are also not that many
> users of it at present. dlm/ocfs2 were the first users as far as I
> recall, and netconsole as well. The fact you have an empty directory just
> indicates that you don't have support for any of these enabled.
> 
> Note that there are also a lot of present-day sysfs and debugfs users
> that could/should be converted to configfs but haven't quite gotten there
> yet. In the sysfs case abuses are hard to rollback once they've made
> become part of the ABI, but that's not grounds for continuing sysfs abuse
> once cleaner methods become available. Many of the sysfs abuses were
> implemented before configfs existed.
> 
> You can also find usage guidelines and example implementations in
> Documentation/filesystems/configfs, which should give you a pretty good
> idea of whether it's a good interface fit for your particular problem or
> not.
> 
> These days sysfs seems to be the new procfs. It certainly helps to put a
> bit of planning in to the interface before you're invariably stuck with
> an ABI that's barely limping along.

Paul,
	Thanks for the clear explanation for configfs, and I kown the importance of the configfs now.
the cpu/memory probe is just an interface, we care more about whether the feature works, but the
interface implemented by sysfs or configfs, it is not important. Different paths, the same objective.
Both are for us.

we follow up the style of the majority, we also accept the feedbacks from the community. I did not 
how many efforts costs, but I am trying.

-- 
Thanks & Regards,
Shaohui

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFC, 6/7] NUMA hotplug emulator

[Shaohui Zheng]

On Thu, May 13, 2010 at 09:56:03AM -0700, Greg KH wrote:
> On Thu, May 13, 2010 at 08:00:16PM +0800, Shaohui Zheng wrote:
> > hotplug emulator:extend memory probe interface to support NUMA
> > 
> > Extend memory probe interface to support an extra paramter nid,
> > the reserved memory can be added into this node if node exists.
> > 
> > Add a memory section(128M) to node 3(boots with mem=1024m)
> > 
> > 	echo 0x40000000,3 > memory/probe
> > 
> > And more we make it friendly, it is possible to add memory to do
> > 
> > 	echo 3g > memory/probe
> > 	echo 1024m,3 > memory/probe
> > 
> > It maintains backwards compatibility.
> 
> Again, please document this.
> 
> thanks,
> 
> greg k-h

okay

-- 
Thanks & Regards,
Shaohui

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFC, 6/7] NUMA hotplug emulator

[Wu Fengguang]

On Fri, May 14, 2010 at 09:49:02AM +0800, Zheng, Shaohui wrote:
> On Thu, May 13, 2010 at 09:56:03AM -0700, Greg KH wrote:
> > On Thu, May 13, 2010 at 08:00:16PM +0800, Shaohui Zheng wrote:
> > > hotplug emulator:extend memory probe interface to support NUMA
> > > 
> > > Extend memory probe interface to support an extra paramter nid,
> > > the reserved memory can be added into this node if node exists.
> > > 
> > > Add a memory section(128M) to node 3(boots with mem=1024m)
> > > 
> > > 	echo 0x40000000,3 > memory/probe
> > > 
> > > And more we make it friendly, it is possible to add memory to do
> > > 
> > > 	echo 3g > memory/probe
> > > 	echo 1024m,3 > memory/probe
> > > 
> > > It maintains backwards compatibility.
> > 
> > Again, please document this.
> > 
> > thanks,
> > 
> > greg k-h
> 
> okay

Shaohui, it's useless to document a wrong interface.
Better to fix the interface _first_, then document becomes meaningful.

Thanks,
Fengguang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFC, 6/7] NUMA hotplug emulator

[Shaohui Zheng]

On Fri, May 14, 2010 at 10:05:26AM +0800, Wu Fengguang wrote:
> On Fri, May 14, 2010 at 09:49:02AM +0800, Zheng, Shaohui wrote:
> > On Thu, May 13, 2010 at 09:56:03AM -0700, Greg KH wrote:
> > > On Thu, May 13, 2010 at 08:00:16PM +0800, Shaohui Zheng wrote:
> > > > hotplug emulator:extend memory probe interface to support NUMA
> > > > 
> > > > Extend memory probe interface to support an extra paramter nid,
> > > > the reserved memory can be added into this node if node exists.
> > > > 
> > > > Add a memory section(128M) to node 3(boots with mem=1024m)
> > > > 
> > > > 	echo 0x40000000,3 > memory/probe
> > > > 
> > > > And more we make it friendly, it is possible to add memory to do
> > > > 
> > > > 	echo 3g > memory/probe
> > > > 	echo 1024m,3 > memory/probe
> > > > 
> > > > It maintains backwards compatibility.
> > > 
> > > Again, please document this.
> > > 
> > > thanks,
> > > 
> > > greg k-h
> > 
> > okay
> 
> Shaohui, it's useless to document a wrong interface.
> Better to fix the interface _first_, then document becomes meaningful.
Fenggunag,
	greg reminds me to add the docment into Documentation/ABI/, We miss it in
the RFC. Currently, the interface format is not finalized, when we decide the
final interface, the last step is documenting it in.
> 
> Thanks,
> Fengguang

-- 
Thanks & Regards,
Shaohui

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFC, 6/7] NUMA hotplug emulator

[Ankita Garg]

Hi,

On Thu, May 13, 2010 at 08:00:16PM +0800, Shaohui Zheng wrote:
> hotplug emulator:extend memory probe interface to support NUMA
> 
> Signed-off-by: Shaohui Zheng <shaohui.zheng@intel.com>
> Signed-off-by: Haicheng Li <haicheng.li@intel.com>
> Signed-off-by: Wu Fengguang <fengguang.wu@intel.com>
> ---
> diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
> index 54ccb0d..787024f 100644
> --- a/arch/x86/Kconfig
> +++ b/arch/x86/Kconfig
> @@ -1239,6 +1239,17 @@ config ARCH_CPU_PROBE_RELEASE
>  	  is for cpu hot-add/hot-remove to specified node in software method.
>  	  This is for debuging and testing purpose
> 
> +config ARCH_MEMORY_PROBE

The above symbol exists already...

> +	def_bool y
> +	bool "Memory hotplug emulation"
> +	depends on NUMA_HOTPLUG_EMU
> +	---help---
> +	  Enable memory hotplug emulation. Reserve memory with grub parameter
> +	  "mem=N"(such as mem=1024M), where N is the initial memory size, the
> +	  rest physical memory will be removed from e820 table; the memory probe
> +	  interface is for memory hot-add to specified node in software method.
> +	  This is for debuging and testing purpose
> +
>  config NODES_SHIFT
>  	int "Maximum NUMA Nodes (as a power of 2)" if !MAXSMP
>  	range 1 10


-- 
Regards,                                                                        
Ankita Garg (ankita@in.ibm.com)                                                 
Linux Technology Center                                                         
IBM India Systems & Technology Labs,                                            
Bangalore, India

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [RFC, 6/7] NUMA hotplug emulator

[Shaohui Zheng]

On Fri, May 21, 2010 at 03:41:04PM +0530, Ankita Garg wrote:
> Hi,
> 
> On Thu, May 13, 2010 at 08:00:16PM +0800, Shaohui Zheng wrote:
> > hotplug emulator:extend memory probe interface to support NUMA
> > 
> > Signed-off-by: Shaohui Zheng <shaohui.zheng@intel.com>
> > Signed-off-by: Haicheng Li <haicheng.li@intel.com>
> > Signed-off-by: Wu Fengguang <fengguang.wu@intel.com>
> > ---
> > diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
> > index 54ccb0d..787024f 100644
> > --- a/arch/x86/Kconfig
> > +++ b/arch/x86/Kconfig
> > @@ -1239,6 +1239,17 @@ config ARCH_CPU_PROBE_RELEASE
> >  	  is for cpu hot-add/hot-remove to specified node in software method.
> >  	  This is for debuging and testing purpose
> > 
> > +config ARCH_MEMORY_PROBE
> 
> The above symbol exists already...
Yes, we create CONFIG_NUMA_HOTPLUG_EMU, CONFIG_NODE_HOTPLUG_EMU and CONFIG_ARCH_CPU_PROBE_RELEASE options,
and move CONFIG_ARCH_MEMORY_PROBE together with the above 3 options.


> 
> > +	def_bool y
> > +	bool "Memory hotplug emulation"
> > +	depends on NUMA_HOTPLUG_EMU
> > +	---help---
> > +	  Enable memory hotplug emulation. Reserve memory with grub parameter
> > +	  "mem=N"(such as mem=1024M), where N is the initial memory size, the
> > +	  rest physical memory will be removed from e820 table; the memory probe
> > +	  interface is for memory hot-add to specified node in software method.
> > +	  This is for debuging and testing purpose
> > +
> >  config NODES_SHIFT
> >  	int "Maximum NUMA Nodes (as a power of 2)" if !MAXSMP
> >  	range 1 10
> 
> 
> -- 
> Regards,                                                                        
> Ankita Garg (ankita@in.ibm.com)                                                 
> Linux Technology Center                                                         
> IBM India Systems & Technology Labs,                                            
> Bangalore, India

-- 
Thanks & Regards,
Shaohui

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>