* lguest problem on boot of guest kernel
@ 2007-06-01 15:25 Nicholas Mc Guire
2007-06-02 10:27 ` Rusty Russell
0 siblings, 1 reply; 9+ messages in thread
From: Nicholas Mc Guire @ 2007-06-01 15:25 UTC (permalink / raw)
To: virtualization
Hi !
Kenrel 2.6.21 (kernel.org)
Patch lguest-2.6.21-254.patch
Distro Slackware 11.0
GCC 3.4.6
GLIBC 2.3.6
HW model name : AMD Duron(tm) procu{s{
Module Size Used by
tun 7680 0
lg 54600 0
just started playing with lguest - patching, compiling and booting the
host-kernel goes ok - compiling lguest is ok as well after hardcodeing
SIOCBRADDIF - but on the first boot attempt of a guest linux I get:
root@darkstar:/usr/src/linux-2.6.21# Documentation/lguest/lguest
64m vmlinux --tunnet=192.168.2.1 --block=../root_fs root=/dev/lgba
lguest: unhandled trap 6 at 0xc0117903 (0x0)
this should be in "c01178db T reserve_top_address" - any idea what could
be wrong ?
thx !
hofrat
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: lguest problem on boot of guest kernel
2007-06-01 15:25 lguest problem on boot of guest kernel Nicholas Mc Guire
@ 2007-06-02 10:27 ` Rusty Russell
2007-06-02 13:24 ` Nicholas Mc Guire
2007-06-08 7:27 ` lguest/host benchmarks Nicholas Mc Guire
0 siblings, 2 replies; 9+ messages in thread
From: Rusty Russell @ 2007-06-02 10:27 UTC (permalink / raw)
To: Nicholas Mc Guire; +Cc: virtualization
On Fri, 2007-06-01 at 17:25 +0200, Nicholas Mc Guire wrote:
> Hi !
>
> Kenrel 2.6.21 (kernel.org)
> Patch lguest-2.6.21-254.patch
> Distro Slackware 11.0
> GCC 3.4.6
> GLIBC 2.3.6
> HW model name : AMD Duron(tm) procu{s{
>
>
> Module Size Used by
> tun 7680 0
> lg 54600 0
>
>
> just started playing with lguest - patching, compiling and booting the
> host-kernel goes ok - compiling lguest is ok as well after hardcodeing
> SIOCBRADDIF - but on the first boot attempt of a guest linux I get:
>
> root@darkstar:/usr/src/linux-2.6.21# Documentation/lguest/lguest
> 64m vmlinux --tunnet=192.168.2.1 --block=../root_fs root=/dev/lgba
> lguest: unhandled trap 6 at 0xc0117903 (0x0)
>
> this should be in "c01178db T reserve_top_address" - any idea what could
> be wrong ?
Hi Nicholas!
You're the second one to hit this in as many days. I just uploaded an
updated patch, but the summary is that you have to turn off COMPAT_VDSO
for the 2.6.21 patch. (You can actually leave it on in the host, it's
just the guests...)
Thanks for the bug report!
Rusty.
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: lguest problem on boot of guest kernel
2007-06-02 10:27 ` Rusty Russell
@ 2007-06-02 13:24 ` Nicholas Mc Guire
2007-06-08 7:27 ` lguest/host benchmarks Nicholas Mc Guire
1 sibling, 0 replies; 9+ messages in thread
From: Nicholas Mc Guire @ 2007-06-02 13:24 UTC (permalink / raw)
To: Rusty Russell; +Cc: virtualization
>
> You're the second one to hit this in as many days. I just uploaded an
> updated patch, but the summary is that you have to turn off COMPAT_VDSO
> for the 2.6.21 patch. (You can actually leave it on in the host, it's
> just the guests...)
>
that did it - thanks!
actually COMPAT_VDSO is not needed post glibc-2.3.3 any way.
another little strangeness I found on booting is that multiple init
processes are reported (busybox 1.2.2.1 based FS)
hosta $ ps auxw
PID Uid VmSize Stat Command
1 root 508 S init
2 root SWN [ksoftirqd/0]
3 root SW [watchdog/0]
4 root SW< [events/0]
5 root SW< [khelper]
6 root SW< [kthread]
69 root SW< [kblockd/0]
70 root SW< [ata/0]
71 root SW< [ata_aux]
72 root SW< [ksuspend_usbd]
75 root SW< [khubd]
77 root SW< [kseriod]
88 root SW< [khpsbpkt]
104 root SW [pdflush]
105 root SW [pdflush]
106 root SW< [kswapd0]
107 root SW< [aio/0]
690 root SW< [khvcd]
781 root SW< [kpsmoused]
784 root SW< [kondemand/0]
801 root 212 S telnetd
803 nobody 212 S httpd -h /tmp -c /etc/httpd.conf -u nobody
804 root 664 S -sh
805 root 252 S init
808 root 252 S init
811 root 252 S init
826 root 524 R ps auxw
hosta $
same filesystem booted under UML does not exhibit these processes, they
are also unkillable and these processes actually report a uniq VmSize.
any idea what they are? /proc/80{5,8,11}/* really does look identical to
/proc/1/* (except for statm/status/smaps).
Other than this nothing found to fuss about ;)
thanks for a cool new playground!
hofrat
^ permalink raw reply [flat|nested] 9+ messages in thread* lguest/host benchmarks.
2007-06-02 10:27 ` Rusty Russell
2007-06-02 13:24 ` Nicholas Mc Guire
@ 2007-06-08 7:27 ` Nicholas Mc Guire
2007-06-09 4:16 ` Rusty Russell
1 sibling, 1 reply; 9+ messages in thread
From: Nicholas Mc Guire @ 2007-06-08 7:27 UTC (permalink / raw)
To: Rusty Russell; +Cc: virtualization
HI !
here are some preliminary benchmarks of lguest vs host (2.6.21)
Most results seem resonable - the file create/delete is a bit
strange - does anybody have an idea why 0k file create/delete could
be so much faster under lguest and 10k file so much slower ?
The only really serious problem with lguest performance seems to be
mmap latency, page-faults and protection faults.
the only problem that lmbench had was to calculate the cpu speed under
lguest (had to pass that on the commandline).
the sometimes better results of lguest compared to the host can also be
due to the better locality as lguest was only using 32MB total (host
idle).
hofrat
Host:
HW: AMD Duron 1.6GHz 256MB RAM HZ=250
distribution; Slackware 11.0 (ext3 80GB)
Lguest:
lguest-2.6.21-254.patch HZ=250
cmdline: Documentation/lguest/lguest 32m vmlinux --block=../slackfs root=/dev/lgba ro
distribution: Slackeare 11.0 minimum instalation (ext3 1GB)
(lguest was not using the network)
L M B E N C H 3 . 0 S U M M A R Y
------------------------------------
(Alpha software, do not distribute)
Basic system parameters
------------------------------------------------------------------------------
Host OS Description Mhz tlb cache mem scal
pages line par load
bytes
--------- ------------- ----------------------- ---- ----- ----- ------ ----
darkstar Lguest 2.6.21 i686-pc-linux-gnu 1602 32 64 2.9500 1
darkstar Lguest 2.6.21 i686-pc-linux-gnu 1602 32 64 2.7500 1
darkstar Linux 2.6.21 i686-pc-linux-gnu 1602 32 64 2.4900 1
darkstar Linux 2.6.21 i686-pc-linux-gnu 1602 32 64 2.5600 1
Processor, Processes - times in microseconds - smaller is better
------------------------------------------------------------------------------
Host OS Mhz null null open slct sig sig fork exec sh
call I/O stat clos TCP inst hndl proc proc proc
--------- ------------- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
darkstar Lguest 2.6.21 1602 0.21 0.36 2.57 3.84 0.68 1.64 1137 2653 7960
darkstar Lguest 2.6.21 1602 0.21 0.37 2.38 4.02 0.65 1.75 1137 2653 7960
darkstar Linux 2.6.21 1602 0.18 0.37 3.54 5.20 28.9 0.62 1.59 213. 1042 8725
darkstar Linux 2.6.21 1602 0.18 0.37 4.01 5.88 17.7 0.62 1.50 204. 1077 8840
Basic integer operations - times in nanoseconds - smaller is better
-------------------------------------------------------------------
Host OS intgr intgr intgr intgr intgr
bit add mul div mod
--------- ------------- ------ ------ ------ ------ ------
darkstar Lguest 2.6.21 1.0900 0.8700 4.3300 42.8 41.0
darkstar Lguest 2.6.21 0.9800 1.0300 4.5600 42.8 51.1
darkstar Linux 2.6.21 0.6200 0.6200 2.4800 25.5 26.7
darkstar Linux 2.6.21 0.6200 0.6200 2.4800 25.4 26.7
Basic float operations - times in nanoseconds - smaller is better
-----------------------------------------------------------------
Host OS float float float float
add mul div bogo
--------- ------------- ------ ------ ------ ------
darkstar Lguest 2.6.21 3.8500 4.1100 23.5 7.4000
darkstar Lguest 2.6.21 5.2900 4.4800 18.9 7.3400
darkstar Linux 2.6.21 2.4800 2.4800 10.9 6.2200
darkstar Linux 2.6.21 2.4800 2.4800 10.8 6.2200
Basic double operations - times in nanoseconds - smaller is better
------------------------------------------------------------------
Host OS double double double double
add mul div bogo
--------- ------------- ------ ------ ------ ------
darkstar Lguest 2.6.21 3.7300 4.1100 20.4 6.5700
darkstar Lguest 2.6.21 3.7300 4.1000 23.5 6.5700
darkstar Linux 2.6.21 2.4800 2.4800 10.9 5.5600
darkstar Linux 2.6.21 2.4800 2.4800 10.8 5.6100
Context switching - times in microseconds - smaller is better
-------------------------------------------------------------------------
Host OS 2p/0K 2p/16K 2p/64K 8p/16K 8p/64K 16p/16K 16p/64K
ctxsw ctxsw ctxsw ctxsw ctxsw ctxsw ctxsw
--------- ------------- ------ ------ ------ ------ ------ ------- -------
darkstar Lguest 2.6.21 6.0900 6.9500 136.3 137.1 282.3 160.7 232.0
darkstar Lguest 2.6.21 6.6200 6.9200 130.6 137.1 282.2 160.7 231.9
darkstar Linux 2.6.21 1.1400 1.9600 77.4 26.1 90.3 36.9 123.8
darkstar Linux 2.6.21 1.3700 2.1100 78.9 26.8 90.8 37.2 124.9
*Local* Communication latencies in microseconds - smaller is better
---------------------------------------------------------------------
Host OS 2p/0K Pipe AF UDP RPC/ TCP RPC/ TCP
ctxsw UNIX UDP TCP conn
--------- ------------- ----- ----- ---- ----- ----- ----- ----- ----
darkstar Lguest 2.6.21 6.090 18.3 19.4
darkstar Lguest 2.6.21 6.620 18.5 27.6
darkstar Linux 2.6.21 1.140 8.805 10.7 18.9 64.
darkstar Linux 2.6.21 1.370 9.547 10.8 18.1 62.
File & VM system latencies in microseconds - smaller is better
-------------------------------------------------------------------------------
Host OS 0K File 10K File Mmap Prot Page 100fd
Create Delete Create Delete Latency Fault Fault selct
--------- ------------- ------ ------ ------ ------ ------- ----- ------- -----
darkstar Lguest 2.6.21 24.0 4.0000 220.0 1499.3 629.0 4.207 7.75490 4.205
darkstar Lguest 2.6.21 20.0 8.0000 348.1 2415.5 612.0 3.254 7.75490 4.224
darkstar Linux 2.6.21 47.2 14.8 121.3 28.4 97.0 0.426 1.30680 3.857
darkstar Linux 2.6.21 45.8 14.0 115.4 27.3 97.0 0.617 1.37670 3.822
*Local* Communication bandwidths in MB/s - bigger is better
-----------------------------------------------------------------------------
Host OS Pipe AF TCP File Mmap Bcopy Bcopy Mem Mem
UNIX reread reread (libc) (hand) read write
--------- ------------- ---- ---- ---- ------ ------ ------ ------ ---- -----
darkstar Lguest 2.6.21 139. 146. 420.2 701.4 262.8 262.8 701. 525.6
darkstar Lguest 2.6.21 135. 140. 420.3 701.4 262.8 262.8 525. 525.6
darkstar Linux 2.6.21 156. 979. 134. 406.3 636.6 275.3 276.3 609. 457.3
darkstar Linux 2.6.21 156. 829. 136. 409.3 638.0 293.1 292.2 610. 496.7
Memory latencies in nanoseconds - smaller is better
(WARNING - may not be correct, check graphs)
------------------------------------------------------------------
Host OS Mhz L1 $ L2 $ Main mem Guesses
--------- ------------- --- ---- ---- -------- -------
darkstar Lguest 2.6.21 1602 2.1380 14.7 182.3
darkstar Lguest 2.6.21 1602 2.1380 14.7 182.3
darkstar Linux 2.6.21 1602 1.8650 12.6 155.2
darkstar Linux 2.6.21 1602 1.8620 12.5 154.8
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: lguest/host benchmarks.
2007-06-08 7:27 ` lguest/host benchmarks Nicholas Mc Guire
@ 2007-06-09 4:16 ` Rusty Russell
2007-06-09 4:27 ` Glauber de Oliveira Costa
2007-06-09 4:44 ` Nicholas Mc Guire
0 siblings, 2 replies; 9+ messages in thread
From: Rusty Russell @ 2007-06-09 4:16 UTC (permalink / raw)
To: Nicholas Mc Guire; +Cc: Jens Axboe, virtualization
On Fri, 2007-06-08 at 09:27 +0200, Nicholas Mc Guire wrote:
> HI !
Hi Nicholas,
Thanks for this!
> here are some preliminary benchmarks of lguest vs host (2.6.21)
>
> Most results seem resonable - the file create/delete is a bit
> strange - does anybody have an idea why 0k file create/delete could
> be so much faster under lguest and 10k file so much slower ?
Is this ext3? The lguest blockio drvier is very naive and is actually
synchronous (SLOW!), but also ignores barriers.
BTW, I suspect your sh is different inside the guest than host:
> darkstar Lguest 2.6.21 1602 0.21 0.37 2.38 4.02 0.65 1.75 1137 2653 7960
> darkstar Linux 2.6.21 1602 0.18 0.37 3.54 5.20 28.9 0.62 1.59 213. 1042 8725
> The only really serious problem with lguest performance seems to be
> mmap latency, page-faults and protection faults.
Yes. A real page fault hurts quite a bit, because all page faults go
via the host (which can swap out guest pages or have them COW).
Sequence goes:
- Guest userspace hits page
- Trap to switcher
- Switch to host kernel
- Walk guest pagetables to see if it's mapped, it's not, so deliver.
- Switch to guest kernel
- Process fault, insert page table entry
- Trap to switcher
- Switch to host kernel
- Insert page table entry in shadow pagetables
- Switch to guest kernel
- Handle fault
This can be optimized by enhancing the switcher itself to walk the guest
page tables and reflect is straight back into the guest if it's not
mapped there. This shouldn't actually be too hard, but I wonder if it's
worth the complexity...
Thanks!
Rusty.
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: lguest/host benchmarks.
2007-06-09 4:16 ` Rusty Russell
@ 2007-06-09 4:27 ` Glauber de Oliveira Costa
2007-06-09 4:28 ` Nicholas Mc Guire
2007-06-09 4:44 ` Nicholas Mc Guire
1 sibling, 1 reply; 9+ messages in thread
From: Glauber de Oliveira Costa @ 2007-06-09 4:27 UTC (permalink / raw)
To: Rusty Russell; +Cc: virtualization, Jens Axboe
>
> Yes. A real page fault hurts quite a bit, because all page faults go
> via the host (which can swap out guest pages or have them COW).
I expect it to be even slower in 64-bit lguest, due to the added
complexity of the 4-level page table. But me and rostedt have not yet
made any benchmark for it _at all_, since we're mostly stuck trying to
make it smp ;-) (bugs, bugs, bugs)
Nick, are you willing to help us in 64-bit too?
thanks!
--
Glauber de Oliveira Costa.
"Free as in Freedom"
http://glommer.net
"The less confident you are, the more serious you have to act."
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: lguest/host benchmarks.
2007-06-09 4:27 ` Glauber de Oliveira Costa
@ 2007-06-09 4:28 ` Nicholas Mc Guire
0 siblings, 0 replies; 9+ messages in thread
From: Nicholas Mc Guire @ 2007-06-09 4:28 UTC (permalink / raw)
To: Glauber de Oliveira Costa; +Cc: virtualization, Jens Axboe
>>
>> Yes. A real page fault hurts quite a bit, because all page faults go
>> via the host (which can swap out guest pages or have them COW).
>
> I expect it to be even slower in 64-bit lguest, due to the added
> complexity of the 4-level page table. But me and rostedt have not yet
> made any benchmark for it _at all_, since we're mostly stuck trying to
> make it smp ;-) (bugs, bugs, bugs)
>
> Nick, are you willing to help us in 64-bit too?
>
got no 64 bit boxes (except PowerPC) so I guess we cna't do much on this
yet, but we should be getting some here soon.
hofrat
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: lguest/host benchmarks.
2007-06-09 4:16 ` Rusty Russell
2007-06-09 4:27 ` Glauber de Oliveira Costa
@ 2007-06-09 4:44 ` Nicholas Mc Guire
2007-06-09 10:43 ` Rusty Russell
1 sibling, 1 reply; 9+ messages in thread
From: Nicholas Mc Guire @ 2007-06-09 4:44 UTC (permalink / raw)
To: Rusty Russell; +Cc: Jens Axboe, virtualization
>
> Hi Nicholas,
>
> Thanks for this!
>
>> here are some preliminary benchmarks of lguest vs host (2.6.21)
>>
>> Most results seem resonable - the file create/delete is a bit
>> strange - does anybody have an idea why 0k file create/delete could
>> be so much faster under lguest and 10k file so much slower ?
>
> Is this ext3? The lguest blockio drvier is very naive and is actually
> synchronous (SLOW!), but also ignores barriers.
>
found a part of the problem - the guest goes streight into the ground if
it runs low on memory. This series is the same filesystem just with
m32/m64/m128 on the commandline of lguest.
<snip>
File & VM system latencies in microseconds - smaller is better
-------------------------------------------------------------------------------
Host OS 0K File 10K File Mmap Prot Page 100fd
Create Delete Create Delete Latency Fault Fault selct
--------- ------------- ------ ------ ------ ------ ------- ----- ------- -----
darkstar Lguest32 2.6. 24.0 4.0000 220.0 1499.3 629.0 4.207 7.75490 4.205
darkstar Lguest32 2.6. 20.0 8.0000 348.1 2415.5 612.0 3.254 7.75490 4.224
darkstar Lguest64 2.6. 24.0 4.0000 552.2 20.0 629.0 3.882 7.75290 4.430
darkstar Lguest64 2.6. 24.0 4.0000 108.0 952.4 629.0 4.049 7.75440 4.400
darkstar Lguest128 2.6 24.0 4.0000 84.0 12.0 629.0 3.481 7.75440 4.225
darkstar Lguest128 2.6 24.0 4.0000 84.0 12.0 629.0 4.309 7.75490 4.203
darkstar Linux 2.6.21 47.2 14.8 121.3 28.4 97.0 0.426 1.30680 3.857
darkstar Linux 2.6.21 45.8 14.0 115.4 27.3 97.0 0.617 1.37670 3.822
<snip>
so the file issue is most likely a memory exhaustion effect on the guest
(memory variation has close to no effect on fork/sh times though). A
indication of locality realted peformance influence can be seen in the
low level int/fp opperations:
<snip>
Basic integer operations - times in nanoseconds - smaller is better
-------------------------------------------------------------------
Host OS intgr intgr intgr intgr intgr
bit add mul div mod
--------- ------------- ------ ------ ------ ------ ------
darkstar Lguest32 2.6. 1.0900 0.8700 4.3300 42.8 41.0
darkstar Lguest32 2.6. 0.9800 1.0300 4.5600 42.8 51.1
darkstar Lguest64 2.6. 1.0300 0.9500 4.1600 42.7 39.3
darkstar Lguest64 2.6. 1.0900 0.8700 4.4000 53.9 62.9
darkstar Lguest128 2.6 1.0900 0.8700 3.4700 57.0 62.9
darkstar Lguest128 2.6 1.0900 1.9600 4.4000 53.8 63.0
darkstar Linux 2.6.21 0.6200 0.6200 2.4800 25.5 26.7
darkstar Linux 2.6.21 0.6200 0.6200 2.4800 25.4 26.7
we still are checking things related to the absolute values reported - not
yet convinced that timing is that correct on the guest.
> BTW, I suspect your sh is different inside the guest than host:
>
>> darkstar Lguest 2.6.21 1602 0.21 0.37 2.38 4.02 0.65 1.75 1137 2653 7960
>> darkstar Linux 2.6.21 1602 0.18 0.37 3.54 5.20 28.9 0.62 1.59 213. 1042 8725
>
nop - same shell - Slacware 11.0 (bash-3.0 something) in both cases, my
guess is that locality is helping here a bit.
>> The only really serious problem with lguest performance seems to be
>> mmap latency, page-faults and protection faults.
>
> Yes. A real page fault hurts quite a bit, because all page faults go
> via the host (which can swap out guest pages or have them COW).
> Sequence goes:
>
> - Guest userspace hits page
> - Trap to switcher
> - Switch to host kernel
> - Walk guest pagetables to see if it's mapped, it's not, so deliver.
> - Switch to guest kernel
> - Process fault, insert page table entry
> - Trap to switcher
> - Switch to host kernel
> - Insert page table entry in shadow pagetables
> - Switch to guest kernel
> - Handle fault
>
> This can be optimized by enhancing the switcher itself to walk the guest
> page tables and reflect is straight back into the guest if it's not
> mapped there. This shouldn't actually be too hard, but I wonder if it's
> worth the complexity...
>
Even if it were it should not go into lguest - the really nice thing about
it is that one can actually understand it !
Most of this will not really hurt performance of applicaitons that much,
so I doubt it is worth the trouble.
thx!
hofrat
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: lguest/host benchmarks.
2007-06-09 4:44 ` Nicholas Mc Guire
@ 2007-06-09 10:43 ` Rusty Russell
0 siblings, 0 replies; 9+ messages in thread
From: Rusty Russell @ 2007-06-09 10:43 UTC (permalink / raw)
To: Nicholas Mc Guire; +Cc: Jens Axboe, virtualization
On Sat, 2007-06-09 at 06:44 +0200, Nicholas Mc Guire wrote:
> so the file issue is most likely a memory exhaustion effect on the guest
Right, I usually run w/ 512M of ram per guest, so I haven't noticed.
> we still are checking things related to the absolute values reported - not
> yet convinced that timing is that correct on the guest.
It's possible, which is why virtbench runs external to the guest. I
never trust anyone's non-hardware clocks 8)
> > This can be optimized by enhancing the switcher itself to walk the guest
> > page tables and reflect is straight back into the guest if it's not
> > mapped there. This shouldn't actually be too hard, but I wonder if it's
> > worth the complexity...
> >
> Even if it were it should not go into lguest - the really nice thing about
> it is that one can actually understand it !
I agree, but I can probably do this in under 50 lines. As you say, not
worth it if it doesn't hurt real applications.
Thanks, and thanks for the kind words about lguest!
Rusty.
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2007-06-09 10:43 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2007-06-01 15:25 lguest problem on boot of guest kernel Nicholas Mc Guire
2007-06-02 10:27 ` Rusty Russell
2007-06-02 13:24 ` Nicholas Mc Guire
2007-06-08 7:27 ` lguest/host benchmarks Nicholas Mc Guire
2007-06-09 4:16 ` Rusty Russell
2007-06-09 4:27 ` Glauber de Oliveira Costa
2007-06-09 4:28 ` Nicholas Mc Guire
2007-06-09 4:44 ` Nicholas Mc Guire
2007-06-09 10:43 ` Rusty Russell
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).