[uml-devel] [PATCH] Locking user-mode kernel RAM into host physical RAM

[uml-devel] [PATCH] Locking user-mode kernel RAM into host physical RAM

[Matthew Bloch]

[-- Attachment #1: clearsigned data --]
[-- Type: Text/Plain, Size: 3037 bytes --]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello all,

Michel Pollet <michel@pollet.net>, a customer of ours, contributed an earlier 
version of this patch to improve the performance of his host on our network.  
It worked so well, we've rolled it out to all our customers and really 
improves the "snappiness" of our Virtual Machines, as well as being clearly 
fairer for customers who pay by the MB for RAM.

Anyone else who's running lots of UMLs with interactive sessions may have 
noticed the following symptoms which this patch cures:

   * irregular ping times to UMLs-- huge delay for the first ping, a few
       normal, a couple of long delays etc.;
   * long gaps in interactive response (having to "wake it up" with some
       keypresses), especially after connecting for the first time in a few
       hours.

Obviously with locked memory the host kernel cannot swap out your UML's 
physical RAM to disc, which means you can't stuff more UMLs onto your machine 
than you have physical RAM any more.  And your kernels need to be setuid root 
now to do the memory locking, which is a concern.  But you don't have to 
enable mlocking if you still want that situation.  However Linux does seem to 
perform quite badly when its "real" RAM is swapped, so I think this is pretty 
sensible and mlocking is certainly our policy now.  My experience with 2.4 as 
a host kernel is that it will quite happily swap out pages from running 
applications to use as disc cache; this is apparently alleviated with 2.6's 
"swappiness" kernel parameter.

- From Michel's initial patch, I've abstracted the system calls to lock memory 
and drop privs so that it fits in with the UML os_ functions, and added a 
command-line option to switch it on, e.g.

   ./linux mem=64MB mlock ubd2=xyz

I hope that it's obvious enough to warrant inclusion in the next release of 
the UML patch; my only thought was whether instead of putting 
setuid-requiring code into the kernel, mem= should accept a file descriptor 
as an argument and have the memory mapping optionally done by an external 
process.  This strikes me as a bit weird from most angles, but obviously it 
allows isolation of code which needs root privileges, and would probably be 
very little work to add to the filemap patch for this purpose.

Regarding testing, Michel's original patch is running on our customers' VMs.  
My version is only running on a single one of our VMs, but has correctly 
locked the memory as Michel's version did.

Any comments would be appreciated!

cheers,

- --
Matthew Bloch                             Bytemark Hosting
                                  tel. +44 (0) 8707 455026
                        http://www.bytemark-hosting.co.uk/
          Dedicated Linux hosts from 15ukp ($26) per month


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)

iD8DBQE/1KkyT2rVDg8aLXQRAnTmAJ9NLC2gyWAMFAVPlL2NT5n8qjvxzgCeJMgm
ANgK3O86koD2N1fUaTGFQFc=
=6BY0
-----END PGP SIGNATURE-----

[-- Attachment #2: linux-2.4.22-uml6-physmem.patch --]
[-- Type: text/x-diff, Size: 3474 bytes --]

diff -urN linux-2.4.22-uml6/arch/um/include/os.h linux-2.4.22-uml6-physmem-clean/arch/um/include/os.h
--- linux-2.4.22-uml6/arch/um/include/os.h	2003-12-05 21:24:49.000000000 +0000
+++ linux-2.4.22-uml6-physmem-clean/arch/um/include/os.h	2003-12-06 14:02:46.000000000 +0000
@@ -165,6 +165,8 @@
 			     int r, int w, int x);
 extern int os_unmap_memory(void *addr, int len);
 extern void os_flush_stdout(void);
+extern int os_lock_memory(void *addr, unsigned long len);
+extern void os_drop_privileges(void);
 
 #endif
 
diff -urN linux-2.4.22-uml6/arch/um/kernel/physmem.c linux-2.4.22-uml6-physmem-clean/arch/um/kernel/physmem.c
--- linux-2.4.22-uml6/arch/um/kernel/physmem.c	2003-12-05 21:24:49.000000000 +0000
+++ linux-2.4.22-uml6-physmem-clean/arch/um/kernel/physmem.c	2003-12-06 14:02:46.000000000 +0000
@@ -122,6 +122,7 @@
 unsigned long high_physmem;
 
 extern unsigned long physmem_size;
+extern unsigned long physmem_lock;
 
 void *to_virt(unsigned long phys)
 {
@@ -225,6 +226,18 @@
 		exit(1);
 	}
 
+	if (physmem_lock) {
+		int err;
+		
+		err = os_lock_memory((void*) uml_reserved, len - offset);
+		if (err)
+		{
+			os_print_error(err, "Couldn't mlock memory");
+			exit(1);
+		}
+		os_drop_privileges();
+	}
+
 	bootmap_size = init_bootmem(pfn, pfn + delta);
 	free_bootmem(__pa(reserve_end) + bootmap_size,
 		     len - bootmap_size - reserve);
@@ -276,6 +289,19 @@
 "	Example: mem=64M\n\n"
 );
 
+static int __init uml_mlock_setup(char *line, int *add)
+{
+	physmem_lock = 1;
+	return 0;
+}
+__uml_setup("mlock", uml_mlock_setup,
+"mlock\n"
+"    This option requests that the user-mode kernel's memory be locked\n"
+"    into the host kernel's RAM so that it cannot be swapped out.\n"
+"    This option requires that the user-mode kernel is run with root\n"
+"    privileges, which it drops after the mapping has been done.\n"
+);
+
 unsigned long find_iomem(char *driver, unsigned long *len_out)
 {
 	struct iomem_region *region = iomem_regions;
diff -urN linux-2.4.22-uml6/arch/um/kernel/um_arch.c linux-2.4.22-uml6-physmem-clean/arch/um/kernel/um_arch.c
--- linux-2.4.22-uml6/arch/um/kernel/um_arch.c	2003-12-05 21:24:49.000000000 +0000
+++ linux-2.4.22-uml6-physmem-clean/arch/um/kernel/um_arch.c	2003-12-06 14:02:49.000000000 +0000
@@ -129,6 +129,7 @@
 /* Set in early boot */
 static int have_root __initdata = 0;
 long physmem_size = 32 * 1024 * 1024;
+long physmem_lock = 0;
 
 void set_cmdline(char *cmd)
 {
diff -urN linux-2.4.22-uml6/arch/um/os-Linux/process.c linux-2.4.22-uml6-physmem-clean/arch/um/os-Linux/process.c
--- linux-2.4.22-uml6/arch/um/os-Linux/process.c	2003-12-05 21:24:49.000000000 +0000
+++ linux-2.4.22-uml6-physmem-clean/arch/um/os-Linux/process.c	2003-12-06 14:02:53.000000000 +0000
@@ -135,6 +135,29 @@
         return(0);
 }
 
+int os_lock_memory(void *addr, unsigned long len)
+{
+	/* Michel mentioned that a single call to mlock doesn't work;
+	 * this seemed the safest alternative.
+	 */
+	while (len > 0) {
+		size_t next_len	= len < getpagesize() ? len : getpagesize();
+		int err 	= mlock(addr, next_len);
+		
+		if (err) return(-err);
+		
+		addr += next_len;
+		len  -= next_len;
+	}
+	
+	return(0);
+}
+
+void os_drop_privileges()
+{
+	setuid(getuid());
+}
+
 /*
  * Overrides for Emacs so that we follow Linus's tabbing style.
  * Emacs will notice this stuff at the end of the file and automatically

Re: [uml-devel] [PATCH] Locking user-mode kernel RAM into host physical RAM

[Matt Ayres]

What version/patches were you using on the host kernel?  Did you perform
any pre-compilation tweaks?  I had this behavior, but after adding some
interactivity and vm patches  I no longer see this under 2.4.

On Mon, 2003-12-08 at 11:39, Matthew Bloch wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Hello all,
> 
> Michel Pollet <michel@pollet.net>, a customer of ours, contributed an earlier 
> version of this patch to improve the performance of his host on our network.  
> It worked so well, we've rolled it out to all our customers and really 
> improves the "snappiness" of our Virtual Machines, as well as being clearly 
> fairer for customers who pay by the MB for RAM.
> 
> Anyone else who's running lots of UMLs with interactive sessions may have 
> noticed the following symptoms which this patch cures:
> 
>    * irregular ping times to UMLs-- huge delay for the first ping, a few
>        normal, a couple of long delays etc.;
>    * long gaps in interactive response (having to "wake it up" with some
>        keypresses), especially after connecting for the first time in a few
>        hours.
> 
> Obviously with locked memory the host kernel cannot swap out your UML's 
> physical RAM to disc, which means you can't stuff more UMLs onto your machine 
> than you have physical RAM any more.  And your kernels need to be setuid root 
> now to do the memory locking, which is a concern.  But you don't have to 
> enable mlocking if you still want that situation.  However Linux does seem to 
> perform quite badly when its "real" RAM is swapped, so I think this is pretty 
> sensible and mlocking is certainly our policy now.  My experience with 2.4 as 
> a host kernel is that it will quite happily swap out pages from running 
> applications to use as disc cache; this is apparently alleviated with 2.6's 
> "swappiness" kernel parameter.
> 
> - From Michel's initial patch, I've abstracted the system calls to lock memory 
> and drop privs so that it fits in with the UML os_ functions, and added a 
> command-line option to switch it on, e.g.
> 
>    ./linux mem=64MB mlock ubd2=xyz
> 
> I hope that it's obvious enough to warrant inclusion in the next release of 
> the UML patch; my only thought was whether instead of putting 
> setuid-requiring code into the kernel, mem= should accept a file descriptor 
> as an argument and have the memory mapping optionally done by an external 
> process.  This strikes me as a bit weird from most angles, but obviously it 
> allows isolation of code which needs root privileges, and would probably be 
> very little work to add to the filemap patch for this purpose.
> 
> Regarding testing, Michel's original patch is running on our customers' VMs.  
> My version is only running on a single one of our VMs, but has correctly 
> locked the memory as Michel's version did.
> 
> Any comments would be appreciated!
> 
> cheers,
> 
> - --
> Matthew Bloch                             Bytemark Hosting
>                                   tel. +44 (0) 8707 455026
>                         http://www.bytemark-hosting.co.uk/
>           Dedicated Linux hosts from 15ukp ($26) per month
> 
> 
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.2.3 (GNU/Linux)
> 
> iD8DBQE/1KkyT2rVDg8aLXQRAnTmAJ9NLC2gyWAMFAVPlL2NT5n8qjvxzgCeJMgm
> ANgK3O86koD2N1fUaTGFQFc=
> =6BY0
> -----END PGP SIGNATURE-----
> 
> ______________________________________________________________________
> diff -urN linux-2.4.22-uml6/arch/um/include/os.h linux-2.4.22-uml6-physmem-clean/arch/um/include/os.h
> --- linux-2.4.22-uml6/arch/um/include/os.h	2003-12-05 21:24:49.000000000 +0000
> +++ linux-2.4.22-uml6-physmem-clean/arch/um/include/os.h	2003-12-06 14:02:46.000000000 +0000
> @@ -165,6 +165,8 @@
>  			     int r, int w, int x);
>  extern int os_unmap_memory(void *addr, int len);
>  extern void os_flush_stdout(void);
> +extern int os_lock_memory(void *addr, unsigned long len);
> +extern void os_drop_privileges(void);
>  
>  #endif
>  
> diff -urN linux-2.4.22-uml6/arch/um/kernel/physmem.c linux-2.4.22-uml6-physmem-clean/arch/um/kernel/physmem.c
> --- linux-2.4.22-uml6/arch/um/kernel/physmem.c	2003-12-05 21:24:49.000000000 +0000
> +++ linux-2.4.22-uml6-physmem-clean/arch/um/kernel/physmem.c	2003-12-06 14:02:46.000000000 +0000
> @@ -122,6 +122,7 @@
>  unsigned long high_physmem;
>  
>  extern unsigned long physmem_size;
> +extern unsigned long physmem_lock;
>  
>  void *to_virt(unsigned long phys)
>  {
> @@ -225,6 +226,18 @@
>  		exit(1);
>  	}
>  
> +	if (physmem_lock) {
> +		int err;
> +		
> +		err = os_lock_memory((void*) uml_reserved, len - offset);
> +		if (err)
> +		{
> +			os_print_error(err, "Couldn't mlock memory");
> +			exit(1);
> +		}
> +		os_drop_privileges();
> +	}
> +
>  	bootmap_size = init_bootmem(pfn, pfn + delta);
>  	free_bootmem(__pa(reserve_end) + bootmap_size,
>  		     len - bootmap_size - reserve);
> @@ -276,6 +289,19 @@
>  "	Example: mem=64M\n\n"
>  );
>  
> +static int __init uml_mlock_setup(char *line, int *add)
> +{
> +	physmem_lock = 1;
> +	return 0;
> +}
> +__uml_setup("mlock", uml_mlock_setup,
> +"mlock\n"
> +"    This option requests that the user-mode kernel's memory be locked\n"
> +"    into the host kernel's RAM so that it cannot be swapped out.\n"
> +"    This option requires that the user-mode kernel is run with root\n"
> +"    privileges, which it drops after the mapping has been done.\n"
> +);
> +
>  unsigned long find_iomem(char *driver, unsigned long *len_out)
>  {
>  	struct iomem_region *region = iomem_regions;
> diff -urN linux-2.4.22-uml6/arch/um/kernel/um_arch.c linux-2.4.22-uml6-physmem-clean/arch/um/kernel/um_arch.c
> --- linux-2.4.22-uml6/arch/um/kernel/um_arch.c	2003-12-05 21:24:49.000000000 +0000
> +++ linux-2.4.22-uml6-physmem-clean/arch/um/kernel/um_arch.c	2003-12-06 14:02:49.000000000 +0000
> @@ -129,6 +129,7 @@
>  /* Set in early boot */
>  static int have_root __initdata = 0;
>  long physmem_size = 32 * 1024 * 1024;
> +long physmem_lock = 0;
>  
>  void set_cmdline(char *cmd)
>  {
> diff -urN linux-2.4.22-uml6/arch/um/os-Linux/process.c linux-2.4.22-uml6-physmem-clean/arch/um/os-Linux/process.c
> --- linux-2.4.22-uml6/arch/um/os-Linux/process.c	2003-12-05 21:24:49.000000000 +0000
> +++ linux-2.4.22-uml6-physmem-clean/arch/um/os-Linux/process.c	2003-12-06 14:02:53.000000000 +0000
> @@ -135,6 +135,29 @@
>          return(0);
>  }
>  
> +int os_lock_memory(void *addr, unsigned long len)
> +{
> +	/* Michel mentioned that a single call to mlock doesn't work;
> +	 * this seemed the safest alternative.
> +	 */
> +	while (len > 0) {
> +		size_t next_len	= len < getpagesize() ? len : getpagesize();
> +		int err 	= mlock(addr, next_len);
> +		
> +		if (err) return(-err);
> +		
> +		addr += next_len;
> +		len  -= next_len;
> +	}
> +	
> +	return(0);
> +}
> +
> +void os_drop_privileges()
> +{
> +	setuid(getuid());
> +}
> +
>  /*
>   * Overrides for Emacs so that we follow Linus's tabbing style.
>   * Emacs will notice this stuff at the end of the file and automatically
-- 
Matt Ayres <matta@tektonic.net>
TekTonic



-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills.  Sign up for IBM's
Free Linux Tutorials.  Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] [PATCH] Locking user-mode kernel RAM into host physical RAM

[Steven Pritchard]

On Mon, Dec 08, 2003 at 04:39:08PM +0000, Matthew Bloch wrote:
> I hope that it's obvious enough to warrant inclusion in the next release of 
> the UML patch; my only thought was whether instead of putting 
> setuid-requiring code into the kernel, mem= should accept a file descriptor 
> as an argument and have the memory mapping optionally done by an external 
> process.  This strikes me as a bit weird from most angles, but obviously it 
> allows isolation of code which needs root privileges, and would probably be 
> very little work to add to the filemap patch for this purpose.

I've long thought that it would be Really Nice if UML could do some
setup things (like this, chroot(), etc.) that need to be done as root
early on, then drop privileges and continue normally.

I once worked on a patch to add chroot() support, but, well, that
didn't go too well.  :-)  (Actually, it worked fine, but by the time
the kernel options were being parsed, there were already several
threads, meaning only the thread doing option parsing was chroot()'d.
Not quite what I intended...)

Steve
-- 
Steven Pritchard - K&S Pritchard Enterprises, Inc.
Email: steve@kspei.com             http://www.kspei.com/
Phone: (618)398-7360               Mobile: (618)567-7320


-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills.  Sign up for IBM's
Free Linux Tutorials.  Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

[uml-devel] Re: chroot option

[Henrik Nordstrom]

On Mon, 8 Dec 2003, Steven Pritchard wrote:

> I once worked on a patch to add chroot() support, but, well, that
> didn't go too well.  :-)  (Actually, it worked fine, but by the time
> the kernel options were being parsed, there were already several
> threads, meaning only the thread doing option parsing was chroot()'d.
> Not quite what I intended...)

There is two parsing steps. One which is done by the "uml" before
launching the kernel where "debug" and other options modifying how uml
operates are parsed (registered by __uml_setup()), and then another where
the kernel options are parsed.

When using the first the kernel should still be in user mode with no other
threads running. To do this have a __uml_setup() for each of the command
line options, then a __uml_postsetup() executing the chroot (note: link
order important, these are executed in the linked order).

Regards
Henrik



-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills.  Sign up for IBM's
Free Linux Tutorials.  Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] Re: chroot option

[Jeff Dike]

hno@marasystems.com said:
> There is two parsing steps. One which is done by the "uml" before
> launching the kernel where "debug" and other options modifying how uml
> operates are parsed (registered by __uml_setup()), and then another
> where the kernel options are parsed. 

The first step is for things that can't be done later, like establishing the
physical memory size.

The second is for everything else, and anything that can be done at this stage
should be.

				Jeff



-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills.  Sign up for IBM's
Free Linux Tutorials.  Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] Re: chroot option

[Henrik Nordstrom]

On Mon, 8 Dec 2003, Jeff Dike wrote:

> The first step is for things that can't be done later, like establishing the
> physical memory size.
> 
> The second is for everything else, and anything that can be done at this stage
> should be.

What is being discussed (having UML chroot itself and drop root privs)  
basically needs to be just between these two, while the uml is still 
just a plain process.

Chrooting and changing user to the intended is kind of part of
establishing the "physical" environment wherein the UML kernel runs, just
as establishing the physical memory. If anywhere this belongs in the
kernel "boot proces" before giving "real" control to the kernel.

This is precisely where the postsetup hook is with it's setupinfo
arguments, making it in my opinion a very good place for a function like 
this, but some care needs to be taken to get the ordering correct wrt 
initrd loading, pid files etc in order to get the desired functionality.

I see a value to UML if this is done after the memory has been set up and
any initrd has been loaded, but should not be done any later than this in
my opinion. And I also do not see any clean way of doing this any later 
without causing confusion about what is inside the chroot and what is 
outside or seriously risking leaks allowing easy breakout from the 
chroot.

Regards
Henrik



-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills.  Sign up for IBM's
Free Linux Tutorials.  Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] [PATCH] Locking user-mode kernel RAM into host physical RAM

[Jeff Dike]

As far as I'm concerned, this is papering over the symptoms rather than fixing
a real problem.

If you're really not overselling memory, then the host should not be swapping.
If it is, then that's the problem.

> My experience with 2.4 as  a host kernel is that it will quite happily
> swap out pages from running  applications to use as disc cache; this
> is apparently alleviated with 2.6's  "swappiness" kernel parameter.

This being the case, you should maybe be looking at 2.6 as your host.

> I hope that it's obvious enough to warrant inclusion in the next
> release of  the UML patch

It's not.  Feel free to argue, but this is really the wrong thing to do, IMHO.

In any case, I have something different in the works.

First, there is ubd-mmap + /dev/anon.  ubd-mmap makes the ubd driver mmap data
from its devices into its physical memory rather than copying it.  This causes
UML to share the data with the host page cache rather than copying it into a
separate page of its own.  This by itself reduces host memory consumption,
which will alleviate the swapping problem on the host.  /dev/anon is a new
host driver which implements the munmap semantics needed by UML physical memory
(pages are release to the host when they're no longer mapped, even if they're
dirty).  My /dev/anon driver is still buggy, but it works well enough to boot
and run UML, and get memory consumption measurements.  On a simple 
boot-my-Debian-image-to-a-login-prompt test, I get ~25% reduction in host 
memory consumption and ~25% increase in the number of UMLs that run before the
host starts swapping.  Giving credit where credit is due, this is thanks to
memset, which ponied up some money to make it happen.

A happy side-effect of the infrastructure needed to make this work is that
it is now possible to plug memory in and out of UMLs dynamically.  This means
that host memory can now be managed by removing it from idle UMLs and giving
it to busy ones.  This would be implemented by a daemon on the host monitoring
the memory state of the host and UMLs.  The UML monitoring is made possible
by the mconsole proc interface which I added a while back.  The daemon would
implement whatever policy the site wants.  

A reasonable default policy is to keep the host from swapping.  So, it would
shuffle memory between UMLs subject to the constraint that the host shouldn't
swap.  So, whenever it looks like the host is running short of free memory,
it would remove some memory from a UML or two in order to alleviate the
shortage.  I'm planning on shipping such a script when I think this stuff works.

				Jeff



-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills.  Sign up for IBM's
Free Linux Tutorials.  Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] [PATCH] Locking user-mode kernel RAM into host physical RAM

[Matthew Bloch]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Monday 08 December 2003 19:01, Jeff Dike wrote:
> As far as I'm concerned, this is papering over the symptoms rather than
> fixing a real problem.
>
> If you're really not overselling memory, then the host should not be
> swapping. If it is, then that's the problem.

Hi Jeff, I don't think this is true.  The 2.4 kernel does appear to swap out 
applications when (e.g.) a large file copy is underway, and the 2.6 kernel 
has been tweaked to get around this problem:

http://kerneltrap.org/node/view/1044

We've not had enough time to try 2.6 on a production server as a host kernel, 
but I'd be interested if you did know a way of tweaking a 2.4 host kernel to 
guarantee RAM-locked pages to user-mode kernels; Matt Ayres mentioned "some 
interactivity and vm patches", could you elaborate on that please Matt?  For 
various reasons, we use a Redhat 2.4.20 kernel which I believe has had its 
memory management fiddled with.

I like the sound of the more dynamic memory arrangement and manager daemon for 
UMLs, sounds like a good way to squeeze more customers onto a host machine.  
But if for the moment the only thing it should do is to keep the host from 
swapping, I know from a few our customers that mlocking has given more 
predictable behaviour from guest kernels, especially if the host load spikes 
for a few minutes.  That's my only argument as to why it should be included: 
mlocked kernels behave more predictably in a hosting environment!  But 
obviously I'm biased; if I can get the same effect without the patch on a 2.4 
host kernel I'd agree it's moot.

- -- 
Matthew Bloch                             Bytemark Hosting
                                  tel. +44 (0) 8707 455026
                        http://www.bytemark-hosting.co.uk/
          Dedicated Linux hosts from 15ukp ($26) per month
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)

iD8DBQE/1Pn5T2rVDg8aLXQRAuMmAJ9BXfBodQcUX2AMPUfHdrKAtKTSMgCcCBc1
MI6K3OJQUdG1uqvzKq8MOMw=
=Zip4
-----END PGP SIGNATURE-----



-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills.  Sign up for IBM's
Free Linux Tutorials.  Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id\x1278&alloc_id371&opÌk
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] [PATCH] Locking user-mode kernel RAM into host physical RAM

[Henrik Nordstrom]

On Mon, 8 Dec 2003, Matthew Bloch wrote:

> Hi Jeff, I don't think this is true.  The 2.4 kernel does appear to swap out 
> applications when (e.g.) a large file copy is underway

Indeed. 2.4 will page out applications and application data if the memory
pressure for buffer/cache is larger than the pressure to use the memory bu
the applications (i.e. if the applications are idle), and it almost all
cases it should. This will include UML memory and kernel etc.  (well,
application binaries such as the UML kernel is not really paged out, but
their memory pages are reclaimed forcing them to be paged in again from
the program/library file)

> interactivity and vm patches", could you elaborate on that please Matt?  For 
> various reasons, we use a Redhat 2.4.20 kernel which I believe has had its 
> memory management fiddled with.

The RedHat 2.4.20 kernel has a completely different VM compared to stock
2.4.20 and behaves better in most aspects.

> That's my only argument as to why it should be included:  mlocked
> kernels behave more predictably in a hosting environment! 

In principle I agree, but it is somewhat of a corner case. For most
practical uses you should be able to get the same results with a correctly
tuned VM on the host and having sufficient amount of memory to allow for a
decent buffer/cache in addition to the UML memory usage. Only if you 
overbook the memory usage (including buffer/cache requirements on 
the host) should there be significant problems.

But as there is no thing such as a perfect VM which fits all, mlock of the
UML memory seems like a fair thing to do to the customer when the customer
pays per MB and the host is dedicated to UML hosting. But in most if not
all other situations it is plain bad and the wrong thing to do. If you
even dream about doing this you should never consider to allocate more
than 50% of the host memory or I/O performance will suffer a lot.

Regards
Henrik



-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills.  Sign up for IBM's
Free Linux Tutorials.  Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] [PATCH] Locking user-mode kernel RAM into host physical RAM

[Matthew Bloch]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Monday 08 December 2003 23:18, Henrik Nordstrom wrote:
> > That's my only argument as to why it should be included:  mlocked
> > kernels behave more predictably in a hosting environment!
>
> In principle I agree, but it is somewhat of a corner case. For most
> practical uses you should be able to get the same results with a correctly
> tuned VM on the host and having sufficient amount of memory to allow for a
> decent buffer/cache in addition to the UML memory usage. Only if you
> overbook the memory usage (including buffer/cache requirements on
> the host) should there be significant problems.

I'm guessing the people who are going to be most concerned with UML's 
performance are ISPs, who are trying to get the best performance from their 
expensive U of data centre rack space; maybe this is a niche case at the 
moment.  

I can believe that the kernel is making compromises in terms of buffer 
requirements after the 50-60% full mark (which is about when we've seen 
swapping as you predicted).  However after buying a certain amount of RAM for 
your box, it is *much* cheaper to compensate for the higher disc demands with 
arrays of fast discs than it is to put more RAM in.  This doesn't make the 
kernel perform well from that point of view, but it means customers get the 
performance and price they expect :)

cheers,

- -- 
Matthew Bloch                             Bytemark Hosting
                                  tel. +44 (0) 8707 455026
                        http://www.bytemark-hosting.co.uk/
          Dedicated Linux hosts from 15ukp ($26) per month
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)

iD8DBQE/1RPaT2rVDg8aLXQRAnLSAJ0QUHDOKcKD5qfKfYpJBBGQOxzbVACeOgrh
wh5avXswOggsOpxuFblT8Is=
=1aJr
-----END PGP SIGNATURE-----



-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills.  Sign up for IBM's
Free Linux Tutorials.  Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id\x1278&alloc_id371&opÌk
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] [PATCH] Locking user-mode kernel RAM into host physical RAM

[BlaisorBlade]

Alle 20:01, lunedì 8 dicembre 2003, Jeff Dike ha scritto:
> As far as I'm concerned, this is papering over the symptoms rather than
> fixing a real problem.
>
> If you're really not overselling memory, then the host should not be
> swapping. If it is, then that's the problem.
>
> > My experience with 2.4 as  a host kernel is that it will quite happily
> > swap out pages from running  applications to use as disc cache; this
> > is apparently alleviated with 2.6's  "swappiness" kernel parameter.
>
> This being the case, you should maybe be looking at 2.6 as your host.
>
> > I hope that it's obvious enough to warrant inclusion in the next
> > release of  the UML patch
>
> It's not.  Feel free to argue, but this is really the wrong thing to do,
> IMHO.
But anyway, if the "physical memory" of the guest is not guaranted to be 
actually physical, then the UML's kernel VM will get fooled. What's more, 
it's the guest's VM which must decide what has to be swapped.
The only swap that must be used is the UML's one. He must be fair to his 
customer(i.e. no overselling), but done that the host can't swap anything 
out.

> /dev/anon is a new host driver which implements the munmap semantics needed
> by UML physical memory (pages are release to the host when they're no
> longer mapped, even if they're dirty).
It means that the guest VM will see that the available memory has shrinked and 
that it will swap things out following the new memory limit? It seems to me a 
little bit difficult. This (if I understand correctly) requires core VM 
changes(the stock VM isn't much tested with shrinking/increasing memory 
sizes, if this is supported at all, for strange architectures).

-- 
cat <<EOSIGN
Paolo Giarrusso, aka Blaisorblade
Linux Kernel 2.4.21/2.6.0-test on an i686; Linux registered user n. 292729
EOSIGN



-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills.  Sign up for IBM's
Free Linux Tutorials.  Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id\x1278&alloc_id371&opÌk
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel