qemu-dm cpu utilization

qemu-dm cpu utilization

[McAfee, Tommie M]

[-- Attachment #1: Type: text/plain, Size: 1502 bytes --]


Using an 8-way ES7000 with 32GB of ram, I'm running xen at changset 11217
with dom0_mem=3092M as a kernel parameter on a x86_64 hyperviser.

It appears that qemu is adding too much overhead to
virtual machines to display graphics.
If I boot a sles9sp3 domVT in runlevel 5 with vga=0x314 for a graphicalboot, and immediately use
vncviewer simply to watch the virtual machine as it boots,
qemu-dm consumes 40% cpu as soon as vncviewer is launched and remains that way throughout this entire process  without settling down when the login screen appears.
Even If I close the console that's viewing the virtual host,  qemu-dm remains at 40-44%.


When booting the same domVT in runlevel 5 without ever attempting to connect to it with
vncviewer, qemu-dm fluctuated from 1-3% while the host was booting and then settled to 0.2% after 4-5 mins.  Assuming that the boot process was complete, I launched vncviewer and qemu-dm jumped up to a 40-44% range again.

Lastly I started the same host in runlevel 3 in pure text mode without any
vga parameters on the kernel line, connected to the virtual machine with
vncviewer, and watched the machine boot until a login prompt appeared.  At
this point, top showed qemu-dm as using 0% of the cpu.  Logging into the
virtual machine and running 'startx' however sent qemu-dm back up to a 40-
44% range.

Why is qemu-dm consuming so much of the cpu?  Why is qemu-dm still high
even when I close my vncviewer?

Tommie,
Xen Test Team
Unisys 


[-- Attachment #2: sles9sp3.hvm --]
[-- Type: application/octet-stream, Size: 734 bytes --]

#  -*- mode: python; -*-
import os, re
arch = os.uname()[4]
if re.search('64', arch):
        arch_libdir = 'lib64'
else:
        arch_libdir = 'lib'

# Kernel image file.
kernel = "/usr/lib/xen/boot/hvmloader"
builder='hvm'
memory = 1048
name = "sles9sp3x64"
vcpus=4
# enable/disable HVM guest PAE, default=0 (disabled)
pae=1
# enable/disable HVM guest ACPI, default=0 (disabled)
acpi=1
# enable/disable HVM guest APIC, default=0 (disabled)
apic=1
vif = [ 'type=ioemu, bridge=xenbr0' ]

disk = [ 'file:...................../sles9sp3x64.vmg,ioemu:hda,w' ]
device_model = '/usr/' + arch_libdir + '/xen/bin/qemu-dm'
boot='c'
sdl=0
vnc=1
# enable spawning vncviewer(only valid when vnc=1), default = 1
vncviewer=1
serial='pty'
ne2000=0


[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: qemu-dm cpu utilization

[Ian Pratt]

> It appears that qemu is adding too much overhead to
> virtual machines to display graphics.
> If I boot a sles9sp3 domVT in runlevel 5 with vga=0x314 for a
> graphicalboot, and immediately use
> vncviewer simply to watch the virtual machine as it boots,
> qemu-dm consumes 40% cpu as soon as vncviewer is launched and remains
that
> way throughout this entire process  without settling down when the
login
> screen appears.
> Even If I close the console that's viewing the virtual host,  qemu-dm
> remains at 40-44%.

That's considerably more than my experience. With a vncviewer connected
it's normal to see qemu-dm burning 5-10% CPU.

We do the majority of our HVM testing using windows as obviously the
best way of running linux is fully 'enlightened' (para-virtualized). It
would be good to test to see whether the issue that you're seeing is
sles9 specific.

Thanks,
Ian

RE: qemu-dm cpu utilization

[Puthiyaparambil, Aravindh]

Ian,

> We do the majority of our HVM testing using windows as obviously the
> best way of running linux is fully 'enlightened' (para-virtualized).
It
> would be good to test to see whether the issue that you're seeing is
> sles9 specific.

We've actually been doing most of our HVM testing with WinXP.  Using
WinXP we found that if we never ran VNC -- waited long enough that we
were sure the WinXP guests were running, and then accessed them via
remote desktop -- we did not see the high cpu utilization.  But, as soon
as we fired-up VNC, we encountered the high utilization that Tommie
described.  Tommie posted the Linux results because most of us have
better tools to investigate Linux, as opposed to Windows, problems.

Are you seeing similar issues with Windows?

Cheers,
Aravindh Puthiyaparambil 
Xen Development Team
Unisys, Tredyffrin PA

RE: qemu-dm cpu utilization

[Ian Pratt]

> > We do the majority of our HVM testing using windows as obviously the
> > best way of running linux is fully 'enlightened' (para-virtualized).
> It
> > would be good to test to see whether the issue that you're seeing is
> > sles9 specific.
> 
> We've actually been doing most of our HVM testing with WinXP.  Using
> WinXP we found that if we never ran VNC -- waited long enough that we
> were sure the WinXP guests were running, and then accessed them via
> remote desktop -- we did not see the high cpu utilization.  But, as
soon
> as we fired-up VNC, we encountered the high utilization that Tommie
> described.  Tommie posted the Linux results because most of us have
> better tools to investigate Linux, as opposed to Windows, problems.
> 
> Are you seeing similar issues with Windows?

No, at least with w2k3. Could you try and repro with w2k3?
What VNC client are you using?
Thanks,
Ian

RE: qemu-dm cpu utilization

[Puthiyaparambil, Aravindh]

> No, at least with w2k3. Could you try and repro with w2k3?

Will do.  BTW, we are seeing the same issue if we use SDL.

> What VNC client are you using?

TightVNC viewer version 1.2.9

> Thanks,
> Ian

RE: qemu-dm cpu utilization

[McAfee, Tommie M]

> No, at least with w2k3. Could you try and repro with w2k3?

I'm getting the same results with win2k3 that I get with the winXP and sles9, so it's probably not a sles9 issue.  I started the win2k3 guest and waited about 4-5 mins for it to finish booting.  At this point qemu-dm used 0% CPU.  When I launched vncviewer to display the win2k3 guest, qemu-dm jumped to 40%-45%.
 
What do you think is causing this to happen?

Tommie McAfee
Xen-Testing 



-----Original Message-----
From: Puthiyaparambil, Aravindh
Sent: Mon 8/28/2006 11:00 AM
To: Ian Pratt; McAfee, Tommie M; xen-devel@lists.xensource.com
Cc: ian.pratt@cl.cam.ac.uk
Subject: RE: [Xen-devel] qemu-dm cpu utilization
 
> No, at least with w2k3. Could you try and repro with w2k3?

Will do.  BTW, we are seeing the same issue if we use SDL.

> What VNC client are you using?

TightVNC viewer version 1.2.9

> Thanks,
> Ian

Re: qemu-dm cpu utilization

[George Dunlap]

Another pertinent question is, what size screen are you using?

>From what I've heard, rather than trap on every MMIO instruction, HVM
guests are given direct access to the virtual framebuffer.  But this
means that QEMU needs to scan the framebuffer on a regular basis to
find changes, and then do diffs to make appropriate VNC changes.  This
probably scales linearly with the number of pixels to be scanned.

Try using a lower resolution, and see what you get.

 -George

On 8/28/06, McAfee, Tommie M <Tommie.McAfee@unisys.com> wrote:
>
> > No, at least with w2k3. Could you try and repro with w2k3?
>
> I'm getting the same results with win2k3 that I get with the winXP and sles9, so it's probably not a sles9 issue.  I started the win2k3 guest and waited about 4-5 mins for it to finish booting.  At this point qemu-dm used 0% CPU.  When I launched vncviewer to display the win2k3 guest, qemu-dm jumped to 40%-45%.
>
> What do you think is causing this to happen?
>
> Tommie McAfee
> Xen-Testing
>
>
>
> -----Original Message-----
> From: Puthiyaparambil, Aravindh
> Sent: Mon 8/28/2006 11:00 AM
> To: Ian Pratt; McAfee, Tommie M; xen-devel@lists.xensource.com
> Cc: ian.pratt@cl.cam.ac.uk
> Subject: RE: [Xen-devel] qemu-dm cpu utilization
>
> > No, at least with w2k3. Could you try and repro with w2k3?
>
> Will do.  BTW, we are seeing the same issue if we use SDL.
>
> > What VNC client are you using?
>
> TightVNC viewer version 1.2.9
>
> > Thanks,
> > Ian
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

RE: qemu-dm cpu utilization

[Ian Pratt]

> Another pertinent question is, what size screen are you using?
> 
> From what I've heard, rather than trap on every MMIO 
> instruction, HVM guests are given direct access to the 
> virtual framebuffer.  But this means that QEMU needs to scan 
> the framebuffer on a regular basis to find changes, and then 
> do diffs to make appropriate VNC changes.  This probably 
> scales linearly with the number of pixels to be scanned.
> 
> Try using a lower resolution, and see what you get.

1024x768 is the 'recommended'. I don't think the emulated Cirrus card
had enough video memory to do full-colour in higher resoloutions.

Ian

RE: qemu-dm cpu utilization

[Puthiyaparambil, Aravindh]

> > Try using a lower resolution, and see what you get.
> 
> 1024x768 is the 'recommended'. I don't think the emulated Cirrus card
> had enough video memory to do full-colour in higher resoloutions.

We have been doing all our testing at 800x600 and I don't think we can
go lower than that with W2k3.

Any pointers to why this could be happening?

Thanks,
Aravindh

RE: qemu-dm cpu utilization

[Ian Pratt]

> > 1024x768 is the 'recommended'. I don't think the emulated Cirrus
card
> > had enough video memory to do full-colour in higher resoloutions.
> 
> We have been doing all our testing at 800x600 and I don't think we can
> go lower than that with W2k3.
> 
> Any pointers to why this could be happening?

Are you explicitly setting 800x600? I thought w2k3 installs defaulted to
1024x768, but maybe I'm confused.

When you disconnect the vnc console qemu should stop scanning the
framebuffer RAM. The fact that the CPU usage doesn't go down is odd.
Perhaps try attaching strace or gdb to the qemu-dm to get an idea of
what the loops is? 

Thanks,
Ian

Re: qemu-dm cpu utilization

[Anthony Liguori]

McAfee, Tommie M wrote:
> Using an 8-way ES7000 with 32GB of ram, I'm running xen at changset 11217
> with dom0_mem=3092M as a kernel parameter on a x86_64 hyperviser.
> 
> It appears that qemu is adding too much overhead to
> virtual machines to display graphics.
> If I boot a sles9sp3 domVT in runlevel 5 with vga=0x314 for a graphicalboot, and immediately use
> vncviewer simply to watch the virtual machine as it boots,
> qemu-dm consumes 40% cpu as soon as vncviewer is launched and remains that way throughout this entire process  without settling down when the login screen appears.
> Even If I close the console that's viewing the virtual host,  qemu-dm remains at 40-44%.

Almost everything in the VNC code resets when a guest disconnected 
except for the timer.  If a client disconnects, the timer routine should 
be almost a nop but it's possible that it's somehow causing qemu-dm to 
do something funky.

If you delete the timer (and set vs->timer = NULL) in 
vnc_client_io_error(), does it reduce the usage?

BTW, if you're using 800x600, I suspect that you may be falling back to 
MMIO instead of a linear framebuffer.  Try moving to 1024x768 and see if 
that helps.

Regards,

Anthony Liguori

> 
> When booting the same domVT in runlevel 5 without ever attempting to connect to it with
> vncviewer, qemu-dm fluctuated from 1-3% while the host was booting and then settled to 0.2% after 4-5 mins.  Assuming that the boot process was complete, I launched vncviewer and qemu-dm jumped up to a 40-44% range again.
> 
> Lastly I started the same host in runlevel 3 in pure text mode without any
> vga parameters on the kernel line, connected to the virtual machine with
> vncviewer, and watched the machine boot until a login prompt appeared.  At
> this point, top showed qemu-dm as using 0% of the cpu.  Logging into the
> virtual machine and running 'startx' however sent qemu-dm back up to a 40-
> 44% range.
> 
> Why is qemu-dm consuming so much of the cpu?  Why is qemu-dm still high
> even when I close my vncviewer?
> 
> Tommie,
> Xen Test Team
> Unisys 
> 
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

RE: qemu-dm cpu utilization

[McAfee, Tommie M]

 
Anthony, 
thanks for the suggestion.
I defiantly think that the resolution of 800x600
was causing some kind of problem (perhaps the linear framebuffer).  When I set it
to 1024x768 qemu-dm went down to 10%-12%.  Is this the 
normal usage that you are seeing?

Tommie McAfee
Xen-Testing 



-----Original Message-----
From: Anthony Liguori [mailto:anthony@codemonkey.ws]
Sent: Tue 8/29/2006 7:43 PM
To: McAfee, Tommie M
Subject: Re: qemu-dm cpu utilization
 
McAfee, Tommie M wrote:
> Using an 8-way ES7000 with 32GB of ram, I'm running xen at changset 11217
> with dom0_mem=3092M as a kernel parameter on a x86_64 hyperviser.
> 
> It appears that qemu is adding too much overhead to
> virtual machines to display graphics.
> If I boot a sles9sp3 domVT in runlevel 5 with vga=0x314 for a graphicalboot, and immediately use
> vncviewer simply to watch the virtual machine as it boots,
> qemu-dm consumes 40% cpu as soon as vncviewer is launched and remains that way throughout this entire process  without settling down when the login screen appears.
> Even If I close the console that's viewing the virtual host,  qemu-dm remains at 40-44%.

Almost everything in the VNC code resets when a guest disconnected 
except for the timer.  If a client disconnects, the timer routine should 
be almost a nop but it's possible that it's somehow causing qemu-dm to 
do something funky.

If you delete the timer (and set vs->timer = NULL) in 
vnc_client_io_error(), does it reduce the usage?

BTW, if you're using 800x600, I suspect that you may be falling back to 
MMIO instead of a linear framebuffer.  Try moving to 1024x768 and see if 
that helps.

Regards,

Anthony Liguori

> 
> When booting the same domVT in runlevel 5 without ever attempting to connect to it with
> vncviewer, qemu-dm fluctuated from 1-3% while the host was booting and then settled to 0.2% after 4-5 mins.  Assuming that the boot process was complete, I launched vncviewer and qemu-dm jumped up to a 40-44% range again.
> 
> Lastly I started the same host in runlevel 3 in pure text mode without any
> vga parameters on the kernel line, connected to the virtual machine with
> vncviewer, and watched the machine boot until a login prompt appeared.  At
> this point, top showed qemu-dm as using 0% of the cpu.  Logging into the
> virtual machine and running 'startx' however sent qemu-dm back up to a 40-
> 44% range.
> 
> Why is qemu-dm consuming so much of the cpu?  Why is qemu-dm still high
> even when I close my vncviewer?
> 
> Tommie,
> Xen Test Team
> Unisys 
> 
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

RE: qemu-dm cpu utilization

[McAfee, Tommie M]

Anthony,

>If you delete the timer (and set vs->timer = NULL) in 
>vnc_client_io_error(), does it reduce the usage?

setting vs->timer=NULL causes qemu-dm to segfault when I launch vncviewer
to view the virtual host.  Here is a snippet of the strace results:

ioctl(13, EVIOCGKEYCODE, 0x7fffff904ee0) = 0
select(17, [6 9 11 13 16], [], [], {0, 10000}) = 1 (in [13], left {0, 10000})
read(13, "\34\0\0\0", 4)                = 4
write(13, "\34\0\0\0", 4)               = 4
clock_gettime(CLOCK_MONOTONIC, {87584, 18585008}) = 0
clock_gettime(CLOCK_MONOTONIC, {87584, 18684008}) = 0
clock_gettime(CLOCK_MONOTONIC, {87584, 18783008}) = 0
ioctl(13, EVIOCGKEYCODE, 0x7fffff904ee0) = 0
select(17, [6 9 11 13 16], [], [], {0, 10000}) = 1 (in [13], left {0, 10000})
read(13, "\34\0\0\0", 4)                = 4
write(13, "\34\0\0\0", 4)               = 4
clock_gettime(CLOCK_MONOTONIC, {87584, 31583008}) = 0
clock_gettime(CLOCK_MONOTONIC, {87584, 31685008}) = 0
clock_gettime(CLOCK_MONOTONIC, {87584, 31785008}) = 0
clock_gettime(CLOCK_MONOTONIC, {87584, 57104008}) = 0
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
Process 17185 detached

According to strace, the timers account for a lot of qemu-dm's activity.
Is this polling a bit excessive? 

Tommie McAfee
Xen-Testing 



-----Original Message-----
From: Anthony Liguori [mailto:anthony@codemonkey.ws]
Sent: Tue 8/29/2006 7:43 PM
To: McAfee, Tommie M
Subject: Re: qemu-dm cpu utilization
 
McAfee, Tommie M wrote:
> Using an 8-way ES7000 with 32GB of ram, I'm running xen at changset 11217
> with dom0_mem=3092M as a kernel parameter on a x86_64 hyperviser.
> 
> It appears that qemu is adding too much overhead to
> virtual machines to display graphics.
> If I boot a sles9sp3 domVT in runlevel 5 with vga=0x314 for a graphicalboot, and immediately use
> vncviewer simply to watch the virtual machine as it boots,
> qemu-dm consumes 40% cpu as soon as vncviewer is launched and remains that way throughout this entire process  without settling down when the login screen appears.
> Even If I close the console that's viewing the virtual host,  qemu-dm remains at 40-44%.

Almost everything in the VNC code resets when a guest disconnected 
except for the timer.  If a client disconnects, the timer routine should 
be almost a nop but it's possible that it's somehow causing qemu-dm to 
do something funky.

If you delete the timer (and set vs->timer = NULL) in 
vnc_client_io_error(), does it reduce the usage?

BTW, if you're using 800x600, I suspect that you may be falling back to 
MMIO instead of a linear framebuffer.  Try moving to 1024x768 and see if 
that helps.

Regards,

Anthony Liguori

> 
> When booting the same domVT in runlevel 5 without ever attempting to connect to it with
> vncviewer, qemu-dm fluctuated from 1-3% while the host was booting and then settled to 0.2% after 4-5 mins.  Assuming that the boot process was complete, I launched vncviewer and qemu-dm jumped up to a 40-44% range again.
> 
> Lastly I started the same host in runlevel 3 in pure text mode without any
> vga parameters on the kernel line, connected to the virtual machine with
> vncviewer, and watched the machine boot until a login prompt appeared.  At
> this point, top showed qemu-dm as using 0% of the cpu.  Logging into the
> virtual machine and running 'startx' however sent qemu-dm back up to a 40-
> 44% range.
> 
> Why is qemu-dm consuming so much of the cpu?  Why is qemu-dm still high
> even when I close my vncviewer?
> 
> Tommie,
> Xen Test Team
> Unisys 
> 
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

RE: qemu-dm cpu utilization

[McAfee, Tommie M]

Anthony,

>You want to set it to NULL after a qemu_del_timer only in the connection
>teardown code.

Setting "ts->timer=NULL" and blocking access to this pointer in qemu_mod_timer seemed to resolve the issue.  When I close the vnc window qemu-dm drops to 0%.

Opening and closing  a connection to the guest with vncviewer doesn't seem to change the way time is being managed in the virtual machine.

Is this the right fix for this problem?

Signed-off-by: Tommie McAfee <tommie.mcafee@unisys.com>

diff -r 8273f730371b tools/ioemu/vl.c
--- a/tools/ioemu/vl.c  Tue Aug 29 12:23:11 2006 +0100
+++ b/tools/ioemu/vl.c  Wed Aug 30 13:03:01 2006 -0400
@@ -731,6 +731,8 @@ void qemu_mod_timer(QEMUTimer *ts, int64
 {
     QEMUTimer **pt, *t;

+   if(!ts)
+       return;
     qemu_del_timer(ts);

     /* add the timer in the sorted list */
diff -r 8273f730371b tools/ioemu/vnc.c
--- a/tools/ioemu/vnc.c Tue Aug 29 12:23:11 2006 +0100
+++ b/tools/ioemu/vnc.c Wed Aug 30 13:03:01 2006 -0400
@@ -626,6 +626,7 @@ static int vnc_client_io_error(VncState
        buffer_reset(&vs->input);
        buffer_reset(&vs->output);
        vs->need_update = 0;
+       vs->timer=NULL;
        return 0;
     }
     return ret;

--------------------------



-----Original Message-----
From: Anthony Liguori [mailto:aliguori@cs.utexas.edu]
Sent: Wed 8/30/2006 5:05 PM
To: McAfee, Tommie M
Cc: Anthony Liguori; xen-devel@lists.xensource.com
Subject: Re: qemu-dm cpu utilization
 
McAfee, Tommie M wrote:
> Anthony,
>
>   
>> If you delete the timer (and set vs->timer = NULL) in 
>> vnc_client_io_error(), does it reduce the usage?
>>     
>
> setting vs->timer=NULL causes qemu-dm to segfault

You want to set it to NULL after a qemu_del_timer only in the connection 
teardown code.

> According to strace, the timers account for a lot of qemu-dm's activity.
> Is this polling a bit excessive? 
>   

I don't fully understand the performance of qemu bits in qemu-dm.  
Christian could probably offer a better insight especially since I think 
he changed the normal timers a bit (at least the GUI timer).

When completely idle, qemu normally won't occupy that much CPU (often 
less than a few percent).  I'd expect it to be a little higher with qemu-dm.

Regards,

Anthony Liguori

> Tommie McAfee
> Xen-Testing 
>
>
>
> -----Original Message-----
> From: Anthony Liguori [mailto:anthony@codemonkey.ws]
> Sent: Tue 8/29/2006 7:43 PM
> To: McAfee, Tommie M
> Subject: Re: qemu-dm cpu utilization
>  
> McAfee, Tommie M wrote:
>   
>> Using an 8-way ES7000 with 32GB of ram, I'm running xen at changset 11217
>> with dom0_mem=3092M as a kernel parameter on a x86_64 hyperviser.
>>
>> It appears that qemu is adding too much overhead to
>> virtual machines to display graphics.
>> If I boot a sles9sp3 domVT in runlevel 5 with vga=0x314 for a graphicalboot, and immediately use
>> vncviewer simply to watch the virtual machine as it boots,
>> qemu-dm consumes 40% cpu as soon as vncviewer is launched and remains that way throughout this entire process  without settling down when the login screen appears.
>> Even If I close the console that's viewing the virtual host,  qemu-dm remains at 40-44%.
>>     
>
> Almost everything in the VNC code resets when a guest disconnected 
> except for the timer.  If a client disconnects, the timer routine should 
> be almost a nop but it's possible that it's somehow causing qemu-dm to 
> do something funky.
>
> If you delete the timer (and set vs->timer = NULL) in 
> vnc_client_io_error(), does it reduce the usage?
>
> BTW, if you're using 800x600, I suspect that you may be falling back to 
> MMIO instead of a linear framebuffer.  Try moving to 1024x768 and see if 
> that helps.
>
> Regards,
>
> Anthony Liguori
>
>   
>> When booting the same domVT in runlevel 5 without ever attempting to connect to it with
>> vncviewer, qemu-dm fluctuated from 1-3% while the host was booting and then settled to 0.2% after 4-5 mins.  Assuming that the boot process was complete, I launched vncviewer and qemu-dm jumped up to a 40-44% range again.
>>
>> Lastly I started the same host in runlevel 3 in pure text mode without any
>> vga parameters on the kernel line, connected to the virtual machine with
>> vncviewer, and watched the machine boot until a login prompt appeared.  At
>> this point, top showed qemu-dm as using 0% of the cpu.  Logging into the
>> virtual machine and running 'startx' however sent qemu-dm back up to a 40-
>> 44% range.
>>
>> Why is qemu-dm consuming so much of the cpu?  Why is qemu-dm still high
>> even when I close my vncviewer?
>>
>> Tommie,
>> Xen Test Team
>> Unisys 
>>
>>
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>>     
>
>