[uml-devel] Swapping directly to host swap area?

[uml-devel] Swapping directly to host swap area?

[William Stearns]

Good evening, Jeff, all,
	Please feel free to file this in the "Damn, I wish Bill would 
read kernel source and get a clue" category if appropriate... :-)

	When a VM needs to swap out, is there some way the UML kernel
could send that sector directly to the host swap area instead of running
its own swap?  In the current setup, the host kernel sees this as a normal
write to an open file, caching it in the process and so not truly freeing
up the ram for a bit.  If the UML swapout routine handed the sector to the
host swapout routine, it might get freed faster.
	I don't _see_ this as a security concern; only root on the host 
could read the host swap directly; if anything this looks more secure than 
the swap file readable by the user under which uml runs on the host.

	Again, my apologies if my random ravings have no value in the real 
world.  Feel free to respond with nothing more than "not useful".  :-)
	Cheers,
	- Bill

---------------------------------------------------------------------------
        "Got me an office; I'm there late at night.
        Just send me e-mail, maybe I'll write."
        -- Joe Walsh for the 21st Century
(Courtesy of Bruce Tong <zztong@laxmi.ev.net>)
--------------------------------------------------------------------------
William Stearns (wstearns@pobox.com).  Mason, Buildkernel, freedups, p0f,
rsync-backup, ssh-keyinstall, dns-check, more at:   http://www.stearns.org
--------------------------------------------------------------------------



-------------------------------------------------------
The SF.Net email is sponsored by EclipseCon 2004
Premiere Conference on Open Tools Development and Integration
See the breadth of Eclipse activity. February 3-5 in Anaheim, CA.
http://www.eclipsecon.org/osdn
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] Swapping directly to host swap area?

[Dan Shearer]

On Sun, Jan 25, 2004 at 06:34:12PM -0500, William Stearns wrote:
> 
> 	When a VM needs to swap out, is there some way the UML kernel
> could send that sector directly to the host swap area instead of running
> its own swap?  

But that would involve making it possible for an ordinary userspace
process (for example, /bin/ls) to affect what the VM is doing on the
host.

Jeff's /dev/anon and ubd=mmap is all about addressing your problem from
the point of view of memory utilisation. See
http://user-mode-linux.sourceforge.net/devanon.html

-- 
Dan Shearer
dan@shearer.org


-------------------------------------------------------
The SF.Net email is sponsored by EclipseCon 2004
Premiere Conference on Open Tools Development and Integration
See the breadth of Eclipse activity. February 3-5 in Anaheim, CA.
http://www.eclipsecon.org/osdn
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] Swapping directly to host swap area?

[Jeff Dike]

wstearns@pobox.com said:
> When a VM needs to swap out, is there some way the UML kernel could
> send that sector directly to the host swap area instead of running its
> own swap?  

No, but this isn't the real question.

> In the current setup, the host kernel sees this as a normal
> write to an open file, caching it in the process and so not truly
> freeing up the ram for a bit. 

This is, and this could be done by setting O_DIRECT on UML swap devices.  That
way, writes go directly (and asynchronously) to the device without landing
in the cache.

With mmap in effect, this can also be done with a MADVISE_DONTNEED on the
relevant pages of the swap file.

Patches cheerfully accepted.

				Jeff



-------------------------------------------------------
The SF.Net email is sponsored by EclipseCon 2004
Premiere Conference on Open Tools Development and Integration
See the breadth of Eclipse activity. February 3-5 in Anaheim, CA.
http://www.eclipsecon.org/osdn
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] O_DIRECT (was: Swapping directly to host swap area?)

[Goetz Bock]

On Tue, Jan 27 '04 at 23:39, Jeff Dike wrote:
> > In the current setup, the host kernel sees this as a normal
> > write to an open file, caching it in the process and so not truly
> > freeing up the ram for a bit. 
> 
> This is, and this could be done by setting O_DIRECT on UML swap
> devices.  That way, writes go directly (and asynchronously) to the
> device without landing in the cache.
Please bare with me, if this seams a stupid qestion, but my C skills are
pritty lacking :-(

But could we get this as an option for all ubd devices?

(I'm using Logical Volumes as base for ubd devices, and have no swap on
 my hosts)
-- 
/"\ Goetz Bock at blacknet dot de  --  secure mobile Linux everNETting
\ /                     (c) 2003 as GNU FDL 1.1
 X   [ 1. Use descriptive subjects - 2. Edit a reply for brevity -  ]
/ \  [ 3. Reply to the list - 4. Read the archive *before* you post ]


-------------------------------------------------------
The SF.Net email is sponsored by EclipseCon 2004
Premiere Conference on Open Tools Development and Integration
See the breadth of Eclipse activity. February 3-5 in Anaheim, CA.
http://www.eclipsecon.org/osdn
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] O_DIRECT (was: Swapping directly to host swap area?)

[Jeff Dike]

On Wed, Jan 28, 2004 at 03:01:22PM +0100, Goetz Bock wrote:
> But could we get this as an option for all ubd devices?
> (I'm using Logical Volumes as base for ubd devices, and have no swap on
>  my hosts)

It would be an option for all ubd devices (and it couldn't be otherwise, since
the driver can't distinguish between swap and non-swap devices).

However, it only seems useful for swap devices since you probably want other
I/O cached on the host.

				Jeff


-------------------------------------------------------
The SF.Net email is sponsored by EclipseCon 2004
Premiere Conference on Open Tools Development and Integration
See the breadth of Eclipse activity. February 3-5 in Anaheim, CA.
http://www.eclipsecon.org/osdn
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] Swapping directly to host swap area?

[BlaisorBlade]

Alle 05:39, mercoledì 28 gennaio 2004, Jeff Dike ha scritto:

> > In the current setup, the host kernel sees this as a normal
> > write to an open file, caching it in the process and so not truly
> > freeing up the ram for a bit.
>
> This is, and this could be done by setting O_DIRECT on UML swap devices. 
> That way, writes go directly (and asynchronously) to the device without
> landing in the cache.
>
> With mmap in effect, this can also be done with a MADVISE_DONTNEED on the
> relevant pages of the swap file.
>
> Patches cheerfully accepted.

About this, my idea was that using swap both on the host and on the guests is 
not very useful:

- because the host already sees which pages are accessed and which not (an 
access inside UML is an access inside the host); the only difference is that 
the host can swap even the guest kernel, while UML not;

- because that probably creates overhead: page aging probably has little one 
(it should be possible to disable it with CONFIG_SWAP=n), but since UML (if I 
understand correctly) needs to trap a SIGSEGV (which is slow, according to 
what you say about TT and SKAS) from a page access to swap in a page, while 
the host does this in hardware, my opinion is that you would probably disable 
swap inside UML. Swap on the host means just that the host will cache datas 
instead of keeping in memory UML pages, if you don't give to the guests more 
than the physical memory; but this makes sense, IMHO. When you have different 
customers on the same host, there is a problem, but then they can choose 
differently (even because this doesn't constrast with the changes to code 
suggested above).

I think this is correct even when /dev/anon is used, but I'm not very sure for 
it.
-- 
cat <<EOSIGN
Paolo Giarrusso, aka Blaisorblade
Linux Kernel 2.4.23/2.6.0 on an i686; Linux registered user n. 292729
EOSIGN




-------------------------------------------------------
The SF.Net email is sponsored by EclipseCon 2004
Premiere Conference on Open Tools Development and Integration
See the breadth of Eclipse activity. February 3-5 in Anaheim, CA.
http://www.eclipsecon.org/osdn
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] Swapping directly to host swap area?

[Jeff Dike]

blaisorblade_spam@yahoo.it said:
> About this, my idea was that using swap both on the host and on the
> guests is  not very useful: 

My suggestions were aimed at making the host less likely to swap.  There are
various pathological things which happen when both UML and the host are
swapping.  Having said that, the host needs to be able to swap UMLs if the 
memory situation is so bad that there is no other choice.

> - because the host already sees which pages are accessed and which not
> (an  access inside UML is an access inside the host); the only
> difference is that  the host can swap even the guest kernel, while UML
> not; 

UML might have a better idea than the host, though.

				Jeff



-------------------------------------------------------
The SF.Net email is sponsored by EclipseCon 2004
Premiere Conference on Open Tools Development and Integration
See the breadth of Eclipse activity. February 3-5 in Anaheim, CA.
http://www.eclipsecon.org/osdn
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] Swapping directly to host swap area?

[BlaisorBlade]

Alle 01:43, mercoledì 11 febbraio 2004, Jeff Dike ha scritto:
> blaisorblade_spam@yahoo.it said:
> > About this, my idea was that using swap both on the host and on the
> > guests is  not very useful:
>
> My suggestions were aimed at making the host less likely to swap.  There
> are various pathological things which happen when both UML and the host are
> swapping.  Having said that, the host needs to be able to swap UMLs if the
> memory situation is so bad that there is no other choice.
I agree completely.

> > - because the host already sees which pages are accessed and which not
> > (an  access inside UML is an access inside the host); the only
> > difference is that  the host can swap even the guest kernel, while UML
> > not;
>
> UML might have a better idea than the host, though.

It seems to me that the access statistics should be the same ones: I do not 
know thoroughly the way pages are marked as active or swappable (I can check 
in the 2.4 VM description - ask for the address) but if that depends on page 
faulting, then, it seems to me that since guest page fault are even host page 
fault (the UML handling of SIGSEGV is like the host handling of CPU fault 
exceptions, right?) the statistics should be the same. At least In My Very 
Humble Opinion.

Bye
-- 
Paolo Giarrusso, aka Blaisorblade
Linux registered user n. 292729



-------------------------------------------------------
SF.Net is sponsored by: Speed Start Your Linux Apps Now.
Build and deploy apps & Web services for Linux with
a free DVD software kit from IBM. Click Now!
http://ads.osdn.com/?ad_id\x1356&alloc_id438&opÌk
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] Swapping directly to host swap area?

[Jeff Dike]

On Thu, Feb 12, 2004 at 08:08:22PM +0100, BlaisorBlade wrote:
> It seems to me that the access statistics should be the same ones: I do not 
> know thoroughly the way pages are marked as active or swappable (I can check 
> in the 2.4 VM description - ask for the address) but if that depends on page 
> faulting, then, it seems to me that since guest page fault are even host page 
> fault (the UML handling of SIGSEGV is like the host handling of CPU fault 
> exceptions, right?) the statistics should be the same. 

No, they're not the same.  If a page is swapped out by the host, and then
swapped back in, that will count as a page fault on the host, but not inside
the UML.

				Jeff


-------------------------------------------------------
SF.Net is sponsored by: Speed Start Your Linux Apps Now.
Build and deploy apps & Web services for Linux with
a free DVD software kit from IBM. Click Now!
http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] Swapping directly to host swap area?

[William Stearns]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1726 bytes --]

Good morning, Jeff,

On Tue, 27 Jan 2004, Jeff Dike wrote:

> wstearns@pobox.com said:
> > In the current setup, the host kernel sees this as a normal
> > write to an open file, caching it in the process and so not truly
> > freeing up the ram for a bit. 
> 
> This is, and this could be done by setting O_DIRECT on UML swap devices.  That
> way, writes go directly (and asynchronously) to the device without landing
> in the cache.
> 
> With mmap in effect, this can also be done with a MADVISE_DONTNEED on the
> relevant pages of the swap file.
> 
> Patches cheerfully accepted.

	In that case, patch cheerfully submitted.  :-)
	The attached adds a fourth ubd option, "D".  When enabled, the 
open call for that ubd file will use the O_DIRECT option.  I didn't tackle 
the MADVISE portion, sorry.
	When I try to use it, I get back "cannot stat /dev/ubd/1: No 
such file or directory."  The file is, of course, there, and I've checked 
I'm using a kernel with this patch.  Does the Linux open64 not handle 
O_DIRECT?

	If the patch is mangled in transit or missing, it will also be up
at http://www.stearns.org/uml/ in a bit, along with a kernel compiled with 
this patch.
	Cheers,
	- Bill

---------------------------------------------------------------------------
        "Where's the kaboom, there was supposed to be
an earth-shattering kaboom!"
        -- Marvin Martian
(Courtesy of John Green <johngreen@starband.net>)
--------------------------------------------------------------------------
William Stearns (wstearns@pobox.com).  Mason, Buildkernel, freedups, p0f,
rsync-backup, ssh-keyinstall, dns-check, more at:   http://www.stearns.org
--------------------------------------------------------------------------

[-- Attachment #2: linux-2.6.4-uml2641-odirect.patch --]
[-- Type: TEXT/PLAIN, Size: 2898 bytes --]

diff -uNr linux-2.6.4-uml2641.orig/arch/um/drivers/ubd_kern.c linux-2.6.4-uml2641-odirectswap.orig/arch/um/drivers/ubd_kern.c
--- linux-2.6.4-uml2641.orig/arch/um/drivers/ubd_kern.c	Sat Feb 14 14:00:43 2004
+++ linux-2.6.4-uml2641-odirectswap.orig/arch/um/drivers/ubd_kern.c	Fri Mar 19 00:08:33 2004
@@ -15,6 +15,8 @@
  * only /dev/ubdN/discN this version also has lots of
  * clean ups preparing for ubd-many.
  * James McMechan
+ * 2004-03-19 Addition of O_DIRECT option 'D' by
+ * William Stearns (wstearns@pobox.com)
  */
 
 #define MAJOR_NR UBD_MAJOR
@@ -326,6 +328,10 @@
 		dev->no_cow = 1;
 		str++;
 	}
+	if (*str == 'D'){
+		flags.d = 1;
+		str++;
+	}
 
 	if(*str++ != '='){
 		printk(KERN_ERR "ubd_setup : Expected '='\n");
@@ -369,7 +375,10 @@
 "    0 to 7. Appending an 'r' to the number will cause that device\n"
 "    to be mounted read-only. For example ubd1r=./ext_fs. Appending\n"
 "    an 's' (has to be _after_ 'r', if there is one) will cause data\n"
-"    to be written to disk on the host immediately.\n\n"
+"    to be written to disk on the host immediately.  The third option\n"
+"    'd' tells uml 'do not use a cow file' for this block device.\n"
+"    Finally, 'D' turns on DIRECT disk access, where the host writes\n"
+"    the data directly with no caching.  Good for swap files.\n\n"
 );
 
 static int fakehd_set = 0;
diff -uNr linux-2.6.4-uml2641.orig/arch/um/include/os.h linux-2.6.4-uml2641-odirectswap.orig/arch/um/include/os.h
--- linux-2.6.4-uml2641.orig/arch/um/include/os.h	Sat Feb 14 14:00:43 2004
+++ linux-2.6.4-uml2641-odirectswap.orig/arch/um/include/os.h	Thu Mar 18 23:22:49 2004
@@ -52,10 +52,12 @@
 	unsigned int a : 1;	/* O_APPEND */
 	unsigned int e : 1;	/* O_EXCL */
 	unsigned int cl : 1;    /* FD_CLOEXEC */
+	unsigned int d : 1;	/* O_DIRECT */
 };
 
 #define OPENFLAGS() ((struct openflags) { .r = 0, .w = 0, .s = 0, .c = 0, \
- 					  .t = 0, .a = 0, .e = 0, .cl = 0 })
+ 					  .t = 0, .a = 0, .e = 0, .cl = 0, \
+					  .d = 0 })
 
 static inline struct openflags of_read(struct openflags flags)
 {
@@ -116,6 +118,12 @@
 	flags.cl = 1; 
 	return(flags); 
 }
+
+static inline struct openflags of_direct(struct openflags flags)
+{ 
+	flags.d = 1; 
+	return(flags); 
+}
   
 extern int os_stat_file(const char *file_name, struct uml_stat *buf);
 extern int os_stat_fd(const int fd, struct uml_stat *buf);
diff -uNr linux-2.6.4-uml2641.orig/arch/um/os-Linux/file.c linux-2.6.4-uml2641-odirectswap.orig/arch/um/os-Linux/file.c
--- linux-2.6.4-uml2641.orig/arch/um/os-Linux/file.c	Sat Feb 14 14:00:43 2004
+++ linux-2.6.4-uml2641-odirectswap.orig/arch/um/os-Linux/file.c	Thu Mar 18 23:24:40 2004
@@ -265,6 +265,7 @@
 	if(flags.c) f |= O_CREAT;
 	if(flags.t) f |= O_TRUNC;
 	if(flags.e) f |= O_EXCL;
+	if(flags.d) f |= O_DIRECT;
 
 	fd = open64(file, f, mode);
 	if(fd < 0)

Re: [uml-devel] Swapping directly to host swap area?

[Matt Ayres]

Does this actually work?  I had tried something similiar and it couldn't
read/write the file as O_DIRECT requires the data to be aligned.

Also, in my thinking it would be better to open the rootfs O_DIRECT and
leave swap as is.  O_DIRECT was made for applications who do their own
caching (such as UML).  You're essentially caching the data twice (which
ubd-mmap was aiming to fix).  If you open swapfs as O_DIRECT and a UML
swaps a lot it will just thrash the disk, normally the swapping will be
done in memory and might not ever touch the disk.

Here is the link to my patch, it's not clean as it was never intended to
be distributed.  It also doesn't work, it's very close to working I just
lack good pointer/memory allocation knowledge.

http://www.tektonic.net/public/odirect-not-working.diff

On Fri, 2004-03-19 at 00:51, William Stearns wrote:
> Good morning, Jeff,
> 
> On Tue, 27 Jan 2004, Jeff Dike wrote:
> 
> > wstearns@pobox.com said:
> > > In the current setup, the host kernel sees this as a normal
> > > write to an open file, caching it in the process and so not truly
> > > freeing up the ram for a bit. 
> > 
> > This is, and this could be done by setting O_DIRECT on UML swap devices.  That
> > way, writes go directly (and asynchronously) to the device without landing
> > in the cache.
> > 
> > With mmap in effect, this can also be done with a MADVISE_DONTNEED on the
> > relevant pages of the swap file.
> > 
> > Patches cheerfully accepted.
> 
> 	In that case, patch cheerfully submitted.  :-)
> 	The attached adds a fourth ubd option, "D".  When enabled, the 
> open call for that ubd file will use the O_DIRECT option.  I didn't tackle 
> the MADVISE portion, sorry.
> 	When I try to use it, I get back "cannot stat /dev/ubd/1: No 
> such file or directory."  The file is, of course, there, and I've checked 
> I'm using a kernel with this patch.  Does the Linux open64 not handle 
> O_DIRECT?
> 
> 	If the patch is mangled in transit or missing, it will also be up
> at http://www.stearns.org/uml/ in a bit, along with a kernel compiled with 
> this patch.
> 	Cheers,
> 	- Bill
> 
> ---------------------------------------------------------------------------
>         "Where's the kaboom, there was supposed to be
> an earth-shattering kaboom!"
>         -- Marvin Martian
> (Courtesy of John Green <johngreen@starband.net>)
> --------------------------------------------------------------------------
> William Stearns (wstearns@pobox.com).  Mason, Buildkernel, freedups, p0f,
> rsync-backup, ssh-keyinstall, dns-check, more at:   http://www.stearns.org
> --------------------------------------------------------------------------



-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel