xm speed

xm speed

[James Harper]

I'm waiting about 2 seconds from typing 'xm list' and for something to
appear. Is this normal? The machine is fairly old, it's a Compaq
Proliant 1600 with a single CPU running at 350Mhz. It's hardware RAID
but that is pretty slow too.

James


-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 13. Go here: http://sf.net/ppc_contest.php

Re: xm speed

[Ian Pratt]

> I'm waiting about 2 seconds from typing 'xm list' and for something to
> appear. Is this normal? The machine is fairly old, it's a Compaq
> Proliant 1600 with a single CPU running at 350Mhz. It's hardware RAID
> but that is pretty slow too.

If you look at the output of 'xm list' does the CPU time consumed
by dom0 go up by 2s every time?

Is dom0 competing for CPU0 with a bunch of other busy domains?

On our 2.4GHz Xeon machines, I've noticed that running 'xm list'
causes the amount of CPU consumed by dom0 to go up by around
100ms, which is quite a staggering amount given what it's
actually doing. Python isn't the most efficient of languages, and
we are doing a HTTP RPC between xm and xend, but this level of
resource usage has always rather surprised me. 

I guess we could use one of the python profiling tools to find
out where the overhead is. Since this is all 'control plane' we
haven't made any effort to make it efficient and have
concentrated on flexibility. It it starts becoming a bottleneck
it may be necessary to selectively recode bits in C.

Ian




-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM. 
Deadline: Sept. 13. Go here: http://sf.net/ppc_contest.php

RE: xm speed

[James Harper]

To do a very simple test I created a little script that does 'time xm
list' 6 times. Here's the output:

Name              Id  Mem(MB)  CPU  State  Time(s)  Console
Domain-0           0      123    0  r----    198.7
gaia2              1        0    0  -b---     59.8    9601

real    0m9.403s
user    0m1.030s
sys     0m0.080s
Name              Id  Mem(MB)  CPU  State  Time(s)  Console
Domain-0           0      123    0  r----    200.0
gaia2              1        0    0  -b---     59.8    9601

real    0m9.997s
user    0m0.980s
sys     0m0.130s
Name              Id  Mem(MB)  CPU  State  Time(s)  Console
Domain-0           0      123    0  r----    201.4
gaia2              1        0    0  -b---     59.8    9601

real    0m9.999s
user    0m0.990s
sys     0m0.140s
Name              Id  Mem(MB)  CPU  State  Time(s)  Console
Domain-0           0      123    0  r----    202.7
gaia2              1        0    0  -b---     59.8    9601

real    0m9.999s
user    0m1.040s
sys     0m0.090s
Name              Id  Mem(MB)  CPU  State  Time(s)  Console
Domain-0           0      123    0  r----    204.0
gaia2              1        0    0  -b---     59.8    9601

real    0m9.998s
user    0m0.990s
sys     0m0.120s
Name              Id  Mem(MB)  CPU  State  Time(s)  Console
Domain-0           0      123    0  r----    205.3
gaia2              1        0    0  -b---     59.8    9601

real    0m9.998s
user    0m1.040s
sys     0m0.100s

It's closer to 10 seconds as you can see. After I type 'xm list', it
takes about 5 seconds for the heading to appear, and then roughly 2
seconds per domain. Load under Domain-0 is almost 0.

Another Proliant 1600 I have that is running xen has 2x550Mhz CPU's and
runs much better... roughly 3 seconds from typing 'xm list' to getting
output, and all the output rolls past pretty quick (4 domains on that
one).

The domain gaia2 is actually destroyed in the output above, it just
doesn't know it. Testing iscsi and it went into a spin logging bulk
stuff to the console. I find killing a domain in that state doesn't work
properly and I have to reboot Dom0 to get xend functioning properly
again.

James
 
> -----Original Message-----
> From: Ian Pratt [mailto:Ian.Pratt@cl.cam.ac.uk]
> Sent: Friday, 10 September 2004 22:55
> To: James Harper
> Cc: xen-devel@lists.sourceforge.net; Ian.Pratt@cl.cam.ac.uk
> Subject: Re: [Xen-devel] xm speed
> 
> > I'm waiting about 2 seconds from typing 'xm list' and for something
to
> > appear. Is this normal? The machine is fairly old, it's a Compaq
> > Proliant 1600 with a single CPU running at 350Mhz. It's hardware
RAID
> > but that is pretty slow too.
> 
> If you look at the output of 'xm list' does the CPU time consumed
> by dom0 go up by 2s every time?
> 
> Is dom0 competing for CPU0 with a bunch of other busy domains?
> 
> On our 2.4GHz Xeon machines, I've noticed that running 'xm list'
> causes the amount of CPU consumed by dom0 to go up by around
> 100ms, which is quite a staggering amount given what it's
> actually doing. Python isn't the most efficient of languages, and
> we are doing a HTTP RPC between xm and xend, but this level of
> resource usage has always rather surprised me.
> 
> I guess we could use one of the python profiling tools to find
> out where the overhead is. Since this is all 'control plane' we
> haven't made any effort to make it efficient and have
> concentrated on flexibility. It it starts becoming a bottleneck
> it may be necessary to selectively recode bits in C.
> 
> Ian
> 
> 



-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 13. Go here: http://sf.net/ppc_contest.php

Re: xm speed

[Brian Wolfe]

May I make a suggestion? 

I've worked with daemons of my own in the past and found it very usefull
and faster for third party development to use a light XML parser library
to handle the control interface parsing. I'd think that going with XML
structures instead of flat text/html/binary would make it easy to self
document the interface and command/response data structures (standard
XML argument).  I wouldn't even bother with HTTP at all. Just accept an
XML structure that describes the task, or list of tasks that need to be
done. then return a simple xml structure with results and error codes so
we know what commands suceeded and which one it failed at. Could also
tag commands as continue on fail for batches such as gathering stats on
each domain.

This would make it far easier for poeple to write their own control
interfaces and just interact with xend (meaning, I'd have an easier time
writing a controller process for the cluster of xen boxes vs rewriting
xend ;) ).

If you like this idea I'd be VERY interested in helping to work on it.
Won't matter to me if it's interpreted language like
perl/python/ruby/etc or if it was in C/C++. Either works for me (just
not too experienced in python.)

This is the approach that I was taking with the replacement xend I
started for xen 1.2 (which I've abandoned until 2.0 is released, and it
will be restarted from scratch). A master and slave master node daemons
that talked to the xen node daemons via the network to track each xen
servers vhosts/loads/etc, sort of like a cluster manager. I do believe
that we should attempt to make it as easy as possible to hook into the
xen system either at the libxc or xend levels. 

Thanks for listening!

Brian


On Fri, 2004-09-10 at 07:55, Ian Pratt wrote:
> > I'm waiting about 2 seconds from typing 'xm list' and for something to
> > appear. Is this normal? The machine is fairly old, it's a Compaq
> > Proliant 1600 with a single CPU running at 350Mhz. It's hardware RAID
> > but that is pretty slow too.
> 
> If you look at the output of 'xm list' does the CPU time consumed
> by dom0 go up by 2s every time?
> 
> Is dom0 competing for CPU0 with a bunch of other busy domains?
> 
> On our 2.4GHz Xeon machines, I've noticed that running 'xm list'
> causes the amount of CPU consumed by dom0 to go up by around
> 100ms, which is quite a staggering amount given what it's
> actually doing. Python isn't the most efficient of languages, and
> we are doing a HTTP RPC between xm and xend, but this level of
> resource usage has always rather surprised me. 
> 
> I guess we could use one of the python profiling tools to find
> out where the overhead is. Since this is all 'control plane' we
> haven't made any effort to make it efficient and have
> concentrated on flexibility. It it starts becoming a bottleneck
> it may be necessary to selectively recode bits in C.
> 
> Ian
> 
> 
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
> Project Admins to receive an Apple iPod Mini FREE for your judgement on
> who ports your project to Linux PPC the best. Sponsored by IBM. 
> Deadline: Sept. 13. Go here: http://sf.net/ppc_contest.php
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel



-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM. 
Deadline: Sept. 13. Go here: http://sf.net/ppc_contest.php

Re: xm speed

[Ian Pratt]

> To do a very simple test I created a little script that does 'time xm
> list' 6 times. Here's the output:

So, real time is ~10s, total CPU consumed by dom 0 is ~1.4s.
(That's terrible!)

Is it like this even when you haven't got a domain in a bad
state? 

Of the 1.4s of CPU, consumed, 1.2s seems to being accounted to
'xm', so we can assume that 'xend' is consuming the other
0.2. What's odd about this is that _xend_ is the guy that's
actually doing most of the work -- xm just does an RPC to it.

Anyhow, it might be more informative to figure out where the
difference between real time and CPU time is coming
from. Presumably xm/xend is blocking on IO. I wander if running
an strace on xm and/or xend might be informative.

BTW: how are you accessing your root file system (local disk,
NFS) etc?

Ian







-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM. 
Deadline: Sept. 13. Go here: http://sf.net/ppc_contest.php

RE: xm speed

[James Harper]

> Is it like this even when you haven't got a domain in a bad
> state?

I think so, although I've never actually timed it before.

> Of the 1.4s of CPU, consumed, 1.2s seems to being accounted to
> 'xm', so we can assume that 'xend' is consuming the other
> 0.2. What's odd about this is that _xend_ is the guy that's
> actually doing most of the work -- xm just does an RPC to it.
> 
> Anyhow, it might be more informative to figure out where the
> difference between real time and CPU time is coming
> from. Presumably xm/xend is blocking on IO. I wander if running
> an strace on xm and/or xend might be informative.
> 
> BTW: how are you accessing your root file system (local disk,
> NFS) etc?

Root filesystem in dom0 is a local raid array, albeit a very slow one.
The disk appears to do a fair bit of chugging when I run it, although
I'm not normally near the box.

After a checkout and rebuild just now, it appears to be much faster!

Name              Id  Mem(MB)  CPU  State  Time(s)  Console
Domain-0           0      123    0  r----     81.8
gaia2              1      126    0  -b---     22.0    9601

real    0m2.254s
user    0m1.100s
sys     0m0.090s
Name              Id  Mem(MB)  CPU  State  Time(s)  Console
Domain-0           0      123    0  r----     83.9
gaia2              1      126    0  -b---     22.1    9601

real    0m2.544s
user    0m1.110s
sys     0m0.100s
Name              Id  Mem(MB)  CPU  State  Time(s)  Console
Domain-0           0      123    0  r----     86.5
gaia2              1      126    0  -b---     22.2    9601

real    0m2.465s
user    0m1.060s
sys     0m0.140s
Name              Id  Mem(MB)  CPU  State  Time(s)  Console
Domain-0           0      123    0  r----     88.6
gaia2              1      126    0  -b---     22.3    9601

real    0m2.529s
user    0m1.060s
sys     0m0.150s
Name              Id  Mem(MB)  CPU  State  Time(s)  Console
Domain-0           0      123    0  r----     91.1
gaia2              1      126    0  -b---     22.4    9601

real    0m2.371s
user    0m1.080s
sys     0m0.130s
Name              Id  Mem(MB)  CPU  State  Time(s)  Console
Domain-0           0      123    0  r----     93.3
gaia2              1      126    0  -b---     22.5    9601

real    0m2.326s
user    0m1.130s
sys     0m0.130s

2 seconds is still a bit slow considering what it is actually doing, but
better than 10.

James




-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 13. Go here: http://sf.net/ppc_contest.php

Re: xm speed

[Mike Wray]

James Harper wrote:
>>Is it like this even when you haven't got a domain in a bad
>>state?
> 
> 
> I think so, although I've never actually timed it before.
> 
> 
>>Of the 1.4s of CPU, consumed, 1.2s seems to being accounted to
>>'xm', so we can assume that 'xend' is consuming the other
>>0.2. What's odd about this is that _xend_ is the guy that's
>>actually doing most of the work -- xm just does an RPC to it.
>>
>>Anyhow, it might be more informative to figure out where the
>>difference between real time and CPU time is coming
>>from. Presumably xm/xend is blocking on IO. I wander if running
>>an strace on xm and/or xend might be informative.
>>
>>BTW: how are you accessing your root file system (local disk,
>>NFS) etc?
> 
> 
> Root filesystem in dom0 is a local raid array, albeit a very slow one.
> The disk appears to do a fair bit of chugging when I run it, although
> I'm not normally near the box.

This could be your problem - xm is going to load quite a few python
classes to make the call.

> 
> After a checkout and rebuild just now, it appears to be much faster!
> 
> Name              Id  Mem(MB)  CPU  State  Time(s)  Console
> Domain-0           0      123    0  r----     81.8
> gaia2              1      126    0  -b---     22.0    9601
> 
> real    0m2.254s
> user    0m1.100s
> sys     0m0.090s
> Name              Id  Mem(MB)  CPU  State  Time(s)  Console
> Domain-0           0      123    0  r----     83.9
> gaia2              1      126    0  -b---     22.1    9601
> 
> real    0m2.544s
> user    0m1.110s
> sys     0m0.100s
> Name              Id  Mem(MB)  CPU  State  Time(s)  Console
> Domain-0           0      123    0  r----     86.5
> gaia2              1      126    0  -b---     22.2    9601
> 
> real    0m2.465s
> user    0m1.060s
> sys     0m0.140s
> Name              Id  Mem(MB)  CPU  State  Time(s)  Console
> Domain-0           0      123    0  r----     88.6
> gaia2              1      126    0  -b---     22.3    9601
> 
> real    0m2.529s
> user    0m1.060s
> sys     0m0.150s
> Name              Id  Mem(MB)  CPU  State  Time(s)  Console
> Domain-0           0      123    0  r----     91.1
> gaia2              1      126    0  -b---     22.4    9601
> 
> real    0m2.371s
> user    0m1.080s
> sys     0m0.130s
> Name              Id  Mem(MB)  CPU  State  Time(s)  Console
> Domain-0           0      123    0  r----     93.3
> gaia2              1      126    0  -b---     22.5    9601
> 
> real    0m2.326s
> user    0m1.130s
> sys     0m0.130s
> 
> 2 seconds is still a bit slow considering what it is actually doing, but
> better than 10.
> 
> James
> 
> 

Anyway, here are some more data points. I ran xm list and compared it
with wget to see what the difference was. I only show the times once,
but they were consistently repoducible.

Timing of 'xm list':
mjw@wray-m-4 1028> time xm list
Name              Id  Mem(MB)  CPU  State  Time(s)  Console
Domain-0           0      123    0  r----     13.1

real    0m0.178s
user    0m0.160s
sys     0m0.000s

Timing of getting python to print hello:
mjw@wray-m-4 1034> time python -c 'print "Hello"'
Hello

real    0m0.016s
user    0m0.010s
sys     0m0.000s

So xm's real work takes about 0.16 s (160ms).

For a comparison let's try getting the domain list with wget:

mjw@wray-m-4 1009> echo $http_proxy
http://15.144.59.5:8088
mjw@wray-m-4 1010> unset http_proxy
mjw@wray-m-4 1014> time wget -q http://localhost:8000/xend/domain/

real    0m0.013s
user    0m0.010s
sys     0m0.000s

This takes 0.013s (13ms) so the time is spent in xm not xend.

mjw@wray-m-4 1042> uname -a
Linux wray-m-4.hpl.hp.com 2.4.27-xen0 #1 Tue Sep 14 12:00:35 BST 2004 i686 i686 
i386 GNU/Linux
mjw@wray-m-4 1043> cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 15
model           : 2
model name      : Intel(R) Xeon(TM) CPU 2.80GHz
stepping        : 9
cpu MHz         : 2791.039
cache size      : 512 KB
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 2
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov 
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe cid
bogomips        : 5570.56

Mike Wray




-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php