lguest problem on boot of guest kernel

lguest problem on boot of guest kernel

[Nicholas Mc Guire]

Hi !

Kenrel 2.6.21 (kernel.org)
Patch  lguest-2.6.21-254.patch
Distro Slackware 11.0
GCC    3.4.6
GLIBC  2.3.6
HW     model name      : AMD Duron(tm) procu{s{


Module                  Size  Used by
tun                     7680  0
lg                     54600  0


  just started playing with lguest - patching, compiling and booting the
  host-kernel goes ok - compiling lguest is ok as well after hardcodeing
  SIOCBRADDIF - but on the first boot attempt of a guest linux I get:

root@darkstar:/usr/src/linux-2.6.21# Documentation/lguest/lguest
64m vmlinux --tunnet=192.168.2.1 --block=../root_fs root=/dev/lgba
lguest: unhandled trap 6 at 0xc0117903 (0x0)

  this should be in "c01178db T reserve_top_address" - any idea what could 
be wrong ?


thx !
hofrat

Re: lguest problem on boot of guest kernel

[Rusty Russell]

On Fri, 2007-06-01 at 17:25 +0200, Nicholas Mc Guire wrote:
> Hi !
> 
> Kenrel 2.6.21 (kernel.org)
> Patch  lguest-2.6.21-254.patch
> Distro Slackware 11.0
> GCC    3.4.6
> GLIBC  2.3.6
> HW     model name      : AMD Duron(tm) procu{s{
> 
> 
> Module                  Size  Used by
> tun                     7680  0
> lg                     54600  0
> 
> 
>   just started playing with lguest - patching, compiling and booting the
>   host-kernel goes ok - compiling lguest is ok as well after hardcodeing
>   SIOCBRADDIF - but on the first boot attempt of a guest linux I get:
> 
> root@darkstar:/usr/src/linux-2.6.21# Documentation/lguest/lguest
> 64m vmlinux --tunnet=192.168.2.1 --block=../root_fs root=/dev/lgba
> lguest: unhandled trap 6 at 0xc0117903 (0x0)
> 
>   this should be in "c01178db T reserve_top_address" - any idea what could 
> be wrong ?

Hi Nicholas!

	You're the second one to hit this in as many days.  I just uploaded an
updated patch, but the summary is that you have to turn off COMPAT_VDSO
for the 2.6.21 patch.  (You can actually leave it on in the host, it's
just the guests...)

Thanks for the bug report!
Rusty.

Re: lguest problem on boot of guest kernel

[Nicholas Mc Guire]

>
> 	You're the second one to hit this in as many days.  I just uploaded an
> updated patch, but the summary is that you have to turn off COMPAT_VDSO
> for the 2.6.21 patch.  (You can actually leave it on in the host, it's
> just the guests...)
>

that did it - thanks!
actually COMPAT_VDSO is not needed post glibc-2.3.3 any way.

another little strangeness I found on booting is that multiple init 
processes are reported (busybox 1.2.2.1 based FS)

hosta $ ps auxw
    PID  Uid     VmSize Stat Command
      1 root        508 S   init
      2 root            SWN [ksoftirqd/0]
      3 root            SW  [watchdog/0]
      4 root            SW< [events/0]
      5 root            SW< [khelper]
      6 root            SW< [kthread]
     69 root            SW< [kblockd/0]
     70 root            SW< [ata/0]
     71 root            SW< [ata_aux]
     72 root            SW< [ksuspend_usbd]
     75 root            SW< [khubd]
     77 root            SW< [kseriod]
     88 root            SW< [khpsbpkt]
    104 root            SW  [pdflush]
    105 root            SW  [pdflush]
    106 root            SW< [kswapd0]
    107 root            SW< [aio/0]
    690 root            SW< [khvcd]
    781 root            SW< [kpsmoused]
    784 root            SW< [kondemand/0]
    801 root        212 S   telnetd
    803 nobody      212 S   httpd -h /tmp -c /etc/httpd.conf -u nobody
    804 root        664 S   -sh
    805 root        252 S   init
    808 root        252 S   init
    811 root        252 S   init
    826 root        524 R   ps auxw
hosta $

   same filesystem booted under UML does not exhibit these processes, they
are also unkillable and these processes actually report a uniq VmSize.
any idea what they are? /proc/80{5,8,11}/* really does look identical to 
/proc/1/* (except for statm/status/smaps).

   Other than this nothing found to fuss about ;)

thanks for a cool new playground!
hofrat

lguest/host benchmarks.

[Nicholas Mc Guire]

HI !

  here are some preliminary benchmarks of lguest vs host (2.6.21)

  Most results seem resonable - the file create/delete is a bit
  strange - does anybody have an idea why 0k file create/delete could
  be so much faster under lguest and 10k file so much slower ?
  The only really serious problem with lguest performance seems to be
  mmap latency, page-faults and protection faults.

  the only problem that lmbench had was to calculate the cpu speed under
  lguest (had to pass that on the commandline).

  the sometimes better results of lguest compared to the host can also be
  due to the better locality as lguest was only using 32MB total (host
  idle).

hofrat

Host:
HW: AMD Duron 1.6GHz 256MB RAM HZ=250
distribution; Slackware 11.0 (ext3 80GB)

Lguest:
lguest-2.6.21-254.patch HZ=250
cmdline: Documentation/lguest/lguest 32m vmlinux --block=../slackfs root=/dev/lgba ro
distribution: Slackeare 11.0 minimum instalation (ext3 1GB)

(lguest was not using the network)

                  L M B E N C H  3 . 0   S U M M A R Y
                  ------------------------------------
 		 (Alpha software, do not distribute)

Basic system parameters
------------------------------------------------------------------------------
Host                 OS Description              Mhz  tlb  cache  mem   scal
                                                      pages line   par   load
                                                            bytes 
--------- ------------- ----------------------- ---- ----- ----- ------ ----
darkstar  Lguest 2.6.21       i686-pc-linux-gnu 1602    32    64 2.9500    1
darkstar  Lguest 2.6.21       i686-pc-linux-gnu 1602    32    64 2.7500    1
darkstar   Linux 2.6.21       i686-pc-linux-gnu 1602    32    64 2.4900    1
darkstar   Linux 2.6.21       i686-pc-linux-gnu 1602    32    64 2.5600    1

Processor, Processes - times in microseconds - smaller is better
------------------------------------------------------------------------------
Host                 OS  Mhz null null      open slct sig  sig  fork exec sh
                              call  I/O stat clos TCP  inst hndl proc proc proc
--------- ------------- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
darkstar  Lguest 2.6.21 1602 0.21 0.36 2.57 3.84      0.68 1.64 1137 2653 7960
darkstar  Lguest 2.6.21 1602 0.21 0.37 2.38 4.02      0.65 1.75 1137 2653 7960
darkstar   Linux 2.6.21 1602 0.18 0.37 3.54 5.20 28.9 0.62 1.59 213. 1042 8725
darkstar   Linux 2.6.21 1602 0.18 0.37 4.01 5.88 17.7 0.62 1.50 204. 1077 8840

Basic integer operations - times in nanoseconds - smaller is better
-------------------------------------------------------------------
Host                 OS  intgr intgr  intgr  intgr  intgr
                           bit   add    mul    div    mod 
--------- ------------- ------ ------ ------ ------ ------ 
darkstar  Lguest 2.6.21 1.0900 0.8700 4.3300   42.8   41.0
darkstar  Lguest 2.6.21 0.9800 1.0300 4.5600   42.8   51.1
darkstar   Linux 2.6.21 0.6200 0.6200 2.4800   25.5   26.7
darkstar   Linux 2.6.21 0.6200 0.6200 2.4800   25.4   26.7

Basic float operations - times in nanoseconds - smaller is better
-----------------------------------------------------------------
Host                 OS  float  float  float  float
                          add    mul    div    bogo
--------- ------------- ------ ------ ------ ------ 
darkstar  Lguest 2.6.21 3.8500 4.1100   23.5 7.4000
darkstar  Lguest 2.6.21 5.2900 4.4800   18.9 7.3400
darkstar   Linux 2.6.21 2.4800 2.4800   10.9 6.2200
darkstar   Linux 2.6.21 2.4800 2.4800   10.8 6.2200

Basic double operations - times in nanoseconds - smaller is better
------------------------------------------------------------------
Host                 OS  double double double double
                          add    mul    div    bogo
--------- ------------- ------  ------ ------ ------ 
darkstar  Lguest 2.6.21 3.7300 4.1100   20.4 6.5700
darkstar  Lguest 2.6.21 3.7300 4.1000   23.5 6.5700
darkstar   Linux 2.6.21 2.4800 2.4800   10.9 5.5600
darkstar   Linux 2.6.21 2.4800 2.4800   10.8 5.6100

Context switching - times in microseconds - smaller is better
-------------------------------------------------------------------------
Host                 OS  2p/0K 2p/16K 2p/64K 8p/16K 8p/64K 16p/16K 16p/64K
                          ctxsw  ctxsw  ctxsw ctxsw  ctxsw   ctxsw   ctxsw
--------- ------------- ------ ------ ------ ------ ------ ------- -------
darkstar  Lguest 2.6.21 6.0900 6.9500  136.3  137.1  282.3   160.7   232.0
darkstar  Lguest 2.6.21 6.6200 6.9200  130.6  137.1  282.2   160.7   231.9
darkstar   Linux 2.6.21 1.1400 1.9600   77.4   26.1   90.3    36.9   123.8
darkstar   Linux 2.6.21 1.3700 2.1100   78.9   26.8   90.8    37.2   124.9

*Local* Communication latencies in microseconds - smaller is better
---------------------------------------------------------------------
Host                 OS 2p/0K  Pipe AF     UDP  RPC/   TCP  RPC/ TCP
                         ctxsw       UNIX         UDP         TCP conn
--------- ------------- ----- ----- ---- ----- ----- ----- ----- ----
darkstar  Lguest 2.6.21 6.090  18.3 19.4 
darkstar  Lguest 2.6.21 6.620  18.5 27.6 
darkstar   Linux 2.6.21 1.140 8.805 10.7              18.9        64.
darkstar   Linux 2.6.21 1.370 9.547 10.8              18.1        62.

File & VM system latencies in microseconds - smaller is better
-------------------------------------------------------------------------------
Host                 OS   0K File      10K File     Mmap    Prot   Page   100fd
                         Create Delete Create Delete Latency Fault  Fault  selct
--------- ------------- ------ ------ ------ ------ ------- ----- ------- -----
darkstar  Lguest 2.6.21   24.0 4.0000  220.0 1499.3   629.0 4.207 7.75490 4.205
darkstar  Lguest 2.6.21   20.0 8.0000  348.1 2415.5   612.0 3.254 7.75490 4.224
darkstar   Linux 2.6.21   47.2   14.8  121.3   28.4    97.0 0.426 1.30680 3.857
darkstar   Linux 2.6.21   45.8   14.0  115.4   27.3    97.0 0.617 1.37670 3.822

*Local* Communication bandwidths in MB/s - bigger is better
-----------------------------------------------------------------------------
Host                OS  Pipe AF    TCP  File   Mmap  Bcopy  Bcopy  Mem   Mem
                              UNIX      reread reread (libc) (hand) read write
--------- ------------- ---- ---- ---- ------ ------ ------ ------ ---- -----
darkstar  Lguest 2.6.21 139. 146.       420.2  701.4  262.8  262.8 701. 525.6
darkstar  Lguest 2.6.21 135. 140.       420.3  701.4  262.8  262.8 525. 525.6
darkstar   Linux 2.6.21 156. 979. 134.  406.3  636.6  275.3  276.3 609. 457.3
darkstar   Linux 2.6.21 156. 829. 136.  409.3  638.0  293.1  292.2 610. 496.7

Memory latencies in nanoseconds - smaller is better
     (WARNING - may not be correct, check graphs)
------------------------------------------------------------------
Host                 OS   Mhz   L1 $   L2 $    Main mem    Guesses
--------- -------------   ---   ----   ----    --------    -------
darkstar  Lguest 2.6.21  1602 2.1380   14.7       182.3
darkstar  Lguest 2.6.21  1602 2.1380   14.7       182.3
darkstar   Linux 2.6.21  1602 1.8650   12.6       155.2
darkstar   Linux 2.6.21  1602 1.8620   12.5       154.8

Re: lguest/host benchmarks.

[Rusty Russell]

On Fri, 2007-06-08 at 09:27 +0200, Nicholas Mc Guire wrote:
> HI !

Hi Nicholas,

	Thanks for this!

>   here are some preliminary benchmarks of lguest vs host (2.6.21)
> 
>   Most results seem resonable - the file create/delete is a bit
>   strange - does anybody have an idea why 0k file create/delete could
>   be so much faster under lguest and 10k file so much slower ?

Is this ext3?   The lguest blockio drvier is very naive and is actually
synchronous (SLOW!), but also ignores barriers.

BTW, I suspect your sh is different inside the guest than host:

> darkstar  Lguest 2.6.21 1602 0.21 0.37 2.38 4.02      0.65 1.75 1137 2653 7960
> darkstar   Linux 2.6.21 1602 0.18 0.37 3.54 5.20 28.9 0.62 1.59 213. 1042 8725

>   The only really serious problem with lguest performance seems to be
>   mmap latency, page-faults and protection faults.

Yes.  A real page fault hurts quite a bit, because all page faults go
via the host (which can swap out guest pages or have them COW).
Sequence goes:

- Guest userspace hits page
- Trap to switcher
- Switch to host kernel
- Walk guest pagetables to see if it's mapped, it's not, so deliver.
- Switch to guest kernel
- Process fault, insert page table entry
- Trap to switcher
- Switch to host kernel
- Insert page table entry in shadow pagetables
- Switch to guest kernel
- Handle fault

This can be optimized by enhancing the switcher itself to walk the guest
page tables and reflect is straight back into the guest if it's not
mapped there.  This shouldn't actually be too hard, but I wonder if it's
worth the complexity...

Thanks!
Rusty.

Re: lguest/host benchmarks.

[Glauber de Oliveira Costa]

>
> Yes.  A real page fault hurts quite a bit, because all page faults go
> via the host (which can swap out guest pages or have them COW).

I expect it to be even slower in 64-bit lguest, due to the added
complexity of the 4-level page table. But me and rostedt have not yet
made any benchmark for it _at all_, since we're mostly stuck trying to
make it smp ;-) (bugs, bugs, bugs)

Nick, are you willing to help us in 64-bit too?

thanks!

-- 
Glauber de Oliveira Costa.
"Free as in Freedom"
http://glommer.net

"The less confident you are, the more serious you have to act."

Re: lguest/host benchmarks.

[Nicholas Mc Guire]

>> 
>> Yes.  A real page fault hurts quite a bit, because all page faults go
>> via the host (which can swap out guest pages or have them COW).
>
> I expect it to be even slower in 64-bit lguest, due to the added
> complexity of the 4-level page table. But me and rostedt have not yet
> made any benchmark for it _at all_, since we're mostly stuck trying to
> make it smp ;-) (bugs, bugs, bugs)
>
> Nick, are you willing to help us in 64-bit too?
>
got no 64 bit boxes (except PowerPC) so I guess we cna't do much on this 
yet, but we should be getting some here soon.

hofrat

Re: lguest/host benchmarks.

[Nicholas Mc Guire]

>
> Hi Nicholas,
>
> 	Thanks for this!
>
>>   here are some preliminary benchmarks of lguest vs host (2.6.21)
>>
>>   Most results seem resonable - the file create/delete is a bit
>>   strange - does anybody have an idea why 0k file create/delete could
>>   be so much faster under lguest and 10k file so much slower ?
>
> Is this ext3?   The lguest blockio drvier is very naive and is actually
> synchronous (SLOW!), but also ignores barriers.
>

found a part of the problem - the guest goes streight into the ground if 
it runs low on memory. This series is the same filesystem just with 
m32/m64/m128 on the commandline of lguest.

<snip>
File & VM system latencies in microseconds - smaller is better
-------------------------------------------------------------------------------
Host                 OS   0K File      10K File     Mmap    Prot   Page   100fd
                         Create Delete Create Delete Latency Fault  Fault  selct
--------- ------------- ------ ------ ------ ------ ------- ----- ------- -----
darkstar  Lguest32 2.6.   24.0 4.0000  220.0 1499.3   629.0 4.207 7.75490 4.205
darkstar  Lguest32 2.6.   20.0 8.0000  348.1 2415.5   612.0 3.254 7.75490 4.224
darkstar  Lguest64 2.6.   24.0 4.0000  552.2   20.0   629.0 3.882 7.75290 4.430
darkstar  Lguest64 2.6.   24.0 4.0000  108.0  952.4   629.0 4.049 7.75440 4.400
darkstar  Lguest128 2.6   24.0 4.0000   84.0   12.0   629.0 3.481 7.75440 4.225
darkstar  Lguest128 2.6   24.0 4.0000   84.0   12.0   629.0 4.309 7.75490 4.203
darkstar   Linux 2.6.21   47.2   14.8  121.3   28.4    97.0 0.426 1.30680 3.857
darkstar   Linux 2.6.21   45.8   14.0  115.4   27.3    97.0 0.617 1.37670 3.822
<snip>

so the file issue is most likely a memory exhaustion effect on the guest 
(memory variation has close to no effect on fork/sh times though). A 
indication of locality realted peformance influence can be seen in the
low level int/fp opperations:

<snip>
Basic integer operations - times in nanoseconds - smaller is better
-------------------------------------------------------------------
Host                 OS  intgr intgr  intgr  intgr  intgr
                           bit   add    mul    div    mod
--------- ------------- ------ ------ ------ ------ ------
darkstar  Lguest32 2.6. 1.0900 0.8700 4.3300   42.8   41.0
darkstar  Lguest32 2.6. 0.9800 1.0300 4.5600   42.8   51.1
darkstar  Lguest64 2.6. 1.0300 0.9500 4.1600   42.7   39.3
darkstar  Lguest64 2.6. 1.0900 0.8700 4.4000   53.9   62.9
darkstar  Lguest128 2.6 1.0900 0.8700 3.4700   57.0   62.9
darkstar  Lguest128 2.6 1.0900 1.9600 4.4000   53.8   63.0
darkstar   Linux 2.6.21 0.6200 0.6200 2.4800   25.5   26.7
darkstar   Linux 2.6.21 0.6200 0.6200 2.4800   25.4   26.7

we still are checking things related to the absolute values reported - not 
yet convinced that timing is that correct on the guest.

> BTW, I suspect your sh is different inside the guest than host:
>
>> darkstar  Lguest 2.6.21 1602 0.21 0.37 2.38 4.02      0.65 1.75 1137 2653 7960
>> darkstar   Linux 2.6.21 1602 0.18 0.37 3.54 5.20 28.9 0.62 1.59 213. 1042 8725
>

nop - same shell - Slacware 11.0 (bash-3.0 something) in both cases, my 
guess is that locality is helping here a bit.

>>   The only really serious problem with lguest performance seems to be
>>   mmap latency, page-faults and protection faults.
>
> Yes.  A real page fault hurts quite a bit, because all page faults go
> via the host (which can swap out guest pages or have them COW).
> Sequence goes:
>
> - Guest userspace hits page
> - Trap to switcher
> - Switch to host kernel
> - Walk guest pagetables to see if it's mapped, it's not, so deliver.
> - Switch to guest kernel
> - Process fault, insert page table entry
> - Trap to switcher
> - Switch to host kernel
> - Insert page table entry in shadow pagetables
> - Switch to guest kernel
> - Handle fault
>
> This can be optimized by enhancing the switcher itself to walk the guest
> page tables and reflect is straight back into the guest if it's not
> mapped there.  This shouldn't actually be too hard, but I wonder if it's
> worth the complexity...
>
Even if it were it should not go into lguest - the really nice thing about 
it is that one can actually understand it !
Most of this will not really hurt performance of applicaitons that much, 
so I doubt it is worth the trouble.

thx!
hofrat

Re: lguest/host benchmarks.

[Rusty Russell]

On Sat, 2007-06-09 at 06:44 +0200, Nicholas Mc Guire wrote:
> so the file issue is most likely a memory exhaustion effect on the guest 

Right, I usually run w/ 512M of ram per guest, so I haven't noticed.

> we still are checking things related to the absolute values reported - not 
> yet convinced that timing is that correct on the guest.

It's possible, which is why virtbench runs external to the guest.  I
never trust anyone's non-hardware clocks 8)

> > This can be optimized by enhancing the switcher itself to walk the guest
> > page tables and reflect is straight back into the guest if it's not
> > mapped there.  This shouldn't actually be too hard, but I wonder if it's
> > worth the complexity...
> >
> Even if it were it should not go into lguest - the really nice thing about 
> it is that one can actually understand it !

I agree, but I can probably do this in under 50 lines.  As you say, not
worth it if it doesn't hurt real applications.

Thanks, and thanks for the kind words about lguest!
Rusty.