[uml-devel] Overhead of UML-skas

[uml-devel] Overhead of UML-skas

[George Washington Dunlap III]

Hey Jeff (and any other interested parties) -- this is George Dunlap, from
the CoVirt group at U of M.  I've been testing the overhead of running
workloads under UML compared to on the bare host, and we're getting some
unexpectedly slow numbers -- for the particular benchmark we're using,
about 105% overhead;  I was wondering if you could shed any light on the
subject: has anyone had similar experiences, and are there any obvious 
optimizations we're missing.

We're using 2.4.20-7um, on a 2.4.18 host with the skas patch.  UML has 
raw access to its own disk (via 'raw /dev/raw/raw1 /dev/hdc'), and is 
given 256M (out of 1Gig host memory).  Asyncronous I/O is enabled via the 
helper process.  COW disk mode is off.

The workload we're running is a "kernel build" benchmark, which consists
of the following commands:
  make clean ; make dep ; make bzImage.
This benchmark is run four times: once to warm up the cache, and thrice to 
get "warm cache" numbers.  All runs are taken immediately after booting 
the host kernel and then (if appropriate) the guest kernel.

We're comparing this against a 2.4.20 vanilla kernel using a copy of the 
same disk image, on the same disk.

Host 2.4.20 average:	 93 seconds
UML 2.4.20 average:	192 seconds (105% overhead)

All numbers were run on a P4 3GHz processor.

Two years ago for our ReVirt paper, we did the same test with UMLinux (now
FAUMAchine) (with some optimizations to make it not-dog-slow) on a 1.2GHz
AMD box and got a 70% overhead for that benchmark.  Similar tests at that
time with VMWare got about 30% overhead.

Has anyone else had a similar experience?  Are there any obvious reasons 
why it would be so much slower?

I realize that it's different hardware as well, so I'm not really 
comparing apples to apples.

The "kernel make" benchmark is sort of a wost-case of our "realistic 
workloads" because it makes a lot of system calls and forks a lot of 
processes.  

The numbers are for a RedHat 6.2 kernel image with ext2 FS, but we've
gotten similar ratio for a RedHat 9.0 installation with ext3.  (The
absolute time for both is increased, we think because the newer version of
gcc does more work.)

Any help would be appreciated.

-George Dunlap
PhD Candidate at the University of Michigan

-- 
+-------------------+----------------------------------------
| dunlapg@umich.edu | http://www-personal.umich.edu/~dunlapg 
+-------------------+----------------------------------------
| "An empty life of luxury is still an empty life"
|    -- everybodyduck, "Walk the Plank"
+------------------------------------------------------------
| Outlaw Junk Email! Support HR 1748 (www.cauce.org)



-------------------------------------------------------
SF.Net is sponsored by: Speed Start Your Linux Apps Now.
Build and deploy apps & Web services for Linux with
a free DVD software kit from IBM. Click Now!
http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] Overhead of UML-skas

[William Stearns]

Good evening, George,

On Mon, 1 Mar 2004, George Washington Dunlap III wrote:

> Hey Jeff (and any other interested parties) -- this is George Dunlap, from
> the CoVirt group at U of M.  I've been testing the overhead of running
> workloads under UML compared to on the bare host, and we're getting some
> unexpectedly slow numbers -- for the particular benchmark we're using,
> about 105% overhead;  I was wondering if you could shed any light on the
> subject: has anyone had similar experiences, and are there any obvious 
> optimizations we're missing.

	Do the boot time messages say it's actually going into skas mode?
	Do you have tmpfs mounted on /tmp, with enough potential space for 
all virtual machines?  
http://www.stearns.org/slartibartfast/uml-coop.current.html#host


> 
> We're using 2.4.20-7um, on a 2.4.18 host with the skas patch.  UML has 
> raw access to its own disk (via 'raw /dev/raw/raw1 /dev/hdc'), and is 
> given 256M (out of 1Gig host memory).  Asyncronous I/O is enabled via the 
> helper process.  COW disk mode is off.

	Helper process?
	BTW, any chance you have Synchronous mode turned on for ubd?  
Please make sure that's off in your kernel compile.
	Cheers,
	- Bill

---------------------------------------------------------------------------
        "According to the study, the three-year cost of a 100,000-hit
processing unit was significantly different among the systems: 
Solaris: $561,520
Windows: $190,662
Linux: $74,475"
	(A processing unit is the number of servers that would be
required to process 100,000 hits per day)
http://techupdate.zdnet.com/techupdate/stories/main/0,14179,2907876-3,00.html
--------------------------------------------------------------------------
William Stearns (wstearns@pobox.com).  Mason, Buildkernel, freedups, p0f,
rsync-backup, ssh-keyinstall, dns-check, more at:   http://www.stearns.org
--------------------------------------------------------------------------



-------------------------------------------------------
SF.Net is sponsored by: Speed Start Your Linux Apps Now.
Build and deploy apps & Web services for Linux with
a free DVD software kit from IBM. Click Now!
http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] Overhead of UML-skas

[George Washington Dunlap III]

On Mon, 1 Mar 2004, William Stearns wrote:
> 	Do the boot time messages say it's actually going into skas mode?
> 	Do you have tmpfs mounted on /tmp, with enough potential space for 
> all virtual machines?  
> http://www.stearns.org/slartibartfast/uml-coop.current.html#host
[snip]
> 	Helper process?
> 	BTW, any chance you have Synchronous mode turned on for ubd?  
> Please make sure that's off in your kernel compile.

Thanks for the quick response.

* SKAS mode: I didn't look specifically at the post messages in those
runs, but I'm using the same scripts and HD image (and getting similar
numbers for UML) for the last 3-4 months.  Plus, TT mode is disabled in
the config.  The most recent run (on a different benchmark) did say:
 Checking for the skas3 patch in the host...found
 Checking for /proc/mm...found

I've been getting those kinds of numbers for awhile; I was only shocked 
when I finally ran numbers on the host and found out how much faster it 
was. =)

* /tmp: I wasn't using tmpfs, but I had ramfs mounted on /tmp/uml, and the
TMP environment variable set to it.  Again, I didn't verify it for this
particular run, but I had verified it some time ago when I set my
environment up.  I may try tmpfs at some point and see if there's any
performance difference.

Hm, it appears that one of my co-workers changed the scripts, and it
wasn't actually making things in /tmp/uml anymore!  Stand-by while I do a
quick test run with ramfs and see what we get...well, there doesn't seem
to be much difference, but I'll do some more controlled tests later
tonight.

* Async I/O: In order to do asyncronous I/O without a kernel that supports
asyncronous I/O, UML forks off a "helper process" that does the blocking
(i.e., guest kernel sends it a command; it does a read() and blocks; when
the read returns, it sends a message to the guest kernel that the data's
been read).

Our standard mode of operation is to have ayncronous I/O turned off by
default, because of the extra process that's created.  (It has to do with
the research we're doing right now.)  I explicitly enabled it for this
build, and verified that the helper process was being created. It resulted
in a marked speed increase (about 6% for this benchmark, if you're
interested).

Thanks for your time,
 -George

-- 
+-------------------+----------------------------------------
| dunlapg@umich.edu | http://www-personal.umich.edu/~dunlapg 
+-------------------+----------------------------------------
| "An empty life of luxury is still an empty life"
|    -- everybodyduck, "Walk the Plank"
+------------------------------------------------------------
| Outlaw Junk Email! Support HR 1748 (www.cauce.org)





-------------------------------------------------------
SF.Net is sponsored by: Speed Start Your Linux Apps Now.
Build and deploy apps & Web services for Linux with
a free DVD software kit from IBM. Click Now!
http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] Overhead of UML-skas

[Jeff Dike]

dunlapg@umich.edu said:
> Our standard mode of operation is to have ayncronous I/O turned off by
> default, because of the extra process that's created.  

Have the build on the host do it on a O_SYNC filesystem and see how fast it
goes.

> (It has to do
> with the research we're doing right now.)  I explicitly enabled it for
> this build, and verified that the helper process was being created. It
> resulted in a marked speed increase (about 6% for this benchmark, if
> you're interested). 

FWIW, I'd expect a 20-30% slowdown from the host on a kernel build.  I see
25-30% on my laptop, Matt Ayes in #uml said he saw 20% on one of his servers.

				Jeff



-------------------------------------------------------
SF.Net is sponsored by: Speed Start Your Linux Apps Now.
Build and deploy apps & Web services for Linux with
a free DVD software kit from IBM. Click Now!
http://ads.osdn.com/?ad_id=1356&alloc_id=3438&op=click
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel