about hot plug issue

about hot plug issue

[Yu, Ping Y]

[-- Attachment #1: Type: text/plain, Size: 621 bytes --]

We found that cs 11554 increases some of timeout to fix 
the hot plug issue when starting 4 VMX simultaneously. 
Thanks for the quick reponse to nightly status report.

But we found that when in an extremely condition, like 
creating 12 VMX domains and 4 XENU domains at the same 
time, sometimes we still can meet this bug. Attached is a 
demo script to start many domains. So we just wonder changing
the timout value could not resolve this issue completely.
At the same time, we found when main_loop_wait is increased to
100 ms, windows guest's keyboard and mouse response is 
significantly slower.

Ping

[-- Attachment #2: Deleted Attachment.txt --]
[-- Type: text/plain, Size: 266 bytes --]

ÿþS\0y\0m\0a\0n\0t\0e\0c\0 \0M\0a\0i\0l\0 \0S\0e\0c\0u\0r\0i\0t\0y\0 \0r\0e\0p\0l\0a\0c\0e\0d\0 \0c\0r\0e\0a\0t\0e\0_\0a\0l\0l\0.\0s\0h\0 \0w\0i\0t\0h\0 \0t\0h\0i\0s\0 \0t\0e\0x\0t\0 \0m\0e\0s\0s\0a\0g\0e\0.\0 \0 \0T\0h\0e\0 \0o\0r\0i\0g\0i\0n\0a\0l\0 \0a\0t\0t\0a\0c\0h\0m\0e\0n\0t\0 \0t\0y\0p\0e\0 \0i\0s\0 \0p\0r\0o\0h\0i\0b\0i\0t\0e\0d\0 \0a\0n\0d\0 \0w\0a\0s\0 \0d\0e\0l\0e\0t\0e\0d\0.\0\r\0
\0

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: about hot plug issue

[Keir Fraser]

On 26/9/06 8:36 am, "Yu, Ping Y" <ping.y.yu@intel.com> wrote:

> But we found that when in an extremely condition, like
> creating 12 VMX domains and 4 XENU domains at the same
> time, sometimes we still can meet this bug. Attached is a
> demo script to start many domains. So we just wonder changing
> the timout value could not resolve this issue completely.

How long do the creation commands appear to hang before timing out? With the
new timeouts it ought to be at least a minute, which ought to be plenty of
time to create a dozen domains.

> At the same time, we found when main_loop_wait is increased to
> 100 ms, windows guest's keyboard and mouse response is
> significantly slower.

Perhaps we need a single polling daemon that can demux kb/mouse events to
all qemu instances in a way that can be select()ed on. That would avoid
unscalable polling overheads.

 -- Keir

Re: about hot plug issue

[Steven Hand]

Any extreme case is going to stress the crap out of the dom0
hotplug subsystem; it's not clear it was designed for the sort
of workloads xen puts on it :-(

BTW, your attachment was stripped, but I think we can probably
work out what it does.

cheers,

S.


----- Original Message ----- 
From: "Yu, Ping Y" <ping.y.yu@intel.com>
To: <xen-devel@lists.xensource.com>
Cc: <Christian.Limpach@xensource.com>; <sos22@cam.ac.uk>; "Steven Hand" 
<steven.hand@cl.cam.ac.uk>
Sent: Tuesday, September 26, 2006 8:36 AM
Subject: about hot plug issue


We found that cs 11554 increases some of timeout to fix
the hot plug issue when starting 4 VMX simultaneously.
Thanks for the quick reponse to nightly status report.

But we found that when in an extremely condition, like
creating 12 VMX domains and 4 XENU domains at the same
time, sometimes we still can meet this bug. Attached is a
demo script to start many domains. So we just wonder changing
the timout value could not resolve this issue completely.
At the same time, we found when main_loop_wait is increased to
100 ms, windows guest's keyboard and mouse response is
significantly slower.

Ping

RE: about hot plug issue

[Yu, Ping Y]

Yeah, in fact, the script is quite simple, and it is just 
trying to create many VMXs or XENUs at the same time.
Another issue we found is that windows guest are not adapt with 
the new timeout value.

#!/bin/bash

xm vcpu-set 0 1
sleep 5

xm cr config.vmx1 &
xm cr config.vmx2 &
xm cr config.vmx3 &
xm cr config.vmx4 &
xm cr config.vmx5 &
xm cr config.vmx6 &
xm cr config.vmx7 &
xm cr config.vmx8 &
xm cr config.vmx9 &
xm cr config.vmx10 &
xm cr config.vmx11 &
xm cr config.vmx12 &
xterm -display :1 -e "xm cr -c config.xenu1" &
xterm -display :1 -e "xm cr -c config.xenu2" &
xterm -display :1 -e "xm cr -c config.xenu3" &
xterm -display :1 -e "xm cr -c config.xenu4" &

Ping

>-----Original Message-----
>From: Steven M. Hand [mailto:smh22@hermes.cam.ac.uk] On Behalf Of Steven Hand
>Sent: 2006年9月26日 15:48
>To: Yu, Ping Y; xen-devel@lists.xensource.com
>Cc: Christian.Limpach@xensource.com; sos22@cam.ac.uk
>Subject: Re: about hot plug issue
>
>Any extreme case is going to stress the crap out of the dom0
>hotplug subsystem; it's not clear it was designed for the sort
>of workloads xen puts on it :-(
>
>BTW, your attachment was stripped, but I think we can probably
>work out what it does.
>
>cheers,
>
>S.
>
>
>----- Original Message -----
>From: "Yu, Ping Y" <ping.y.yu@intel.com>
>To: <xen-devel@lists.xensource.com>
>Cc: <Christian.Limpach@xensource.com>; <sos22@cam.ac.uk>; "Steven Hand"
><steven.hand@cl.cam.ac.uk>
>Sent: Tuesday, September 26, 2006 8:36 AM
>Subject: about hot plug issue
>
>
>We found that cs 11554 increases some of timeout to fix
>the hot plug issue when starting 4 VMX simultaneously.
>Thanks for the quick reponse to nightly status report.
>
>But we found that when in an extremely condition, like
>creating 12 VMX domains and 4 XENU domains at the same
>time, sometimes we still can meet this bug. Attached is a
>demo script to start many domains. So we just wonder changing
>the timout value could not resolve this issue completely.
>At the same time, we found when main_loop_wait is increased to
>100 ms, windows guest's keyboard and mouse response is
>significantly slower.
>
>Ping

RE: about hot plug issue

[Yu, Ping Y]

>
>How long do the creation commands appear to hang before timing out?
With the
>new timeouts it ought to be at least a minute, which ought to be plenty
of
>time to create a dozen domains.
The failure time is about 140 second.
>
>> At the same time, we found when main_loop_wait is increased to
>> 100 ms, windows guest's keyboard and mouse response is
>> significantly slower.
>
>Perhaps we need a single polling daemon that can demux kb/mouse events
to
>all qemu instances in a way that can be select()ed on. That would avoid
>unscalable polling overheads.
Good solution. Could you make a patch for this? :-)

>
> -- Keir
>
>
>
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.xensource.com
>http://lists.xensource.com/xen-devel

Re: about hot plug issue

[Keir Fraser]

On 26/9/06 09:26, "Yu, Ping Y" <ping.y.yu@intel.com> wrote:

>> How long do the creation commands appear to hang before timing out?
> With the
>> new timeouts it ought to be at least a minute, which ought to be plenty
> of
>> time to create a dozen domains.
> The failure time is about 140 second.

Well, dom0 is probably hosed for those 140 seconds and you've reached the
limit of work that can be done in single-VCPU domain 0 in that amount of
time. Also, a user is not going to want to wait an unbounded length of time
for his request to be processed -- it seems reasonable to time out, but the
error should probably be 'too busy' rather than 'unknown fatal error'. And
the timeout should be more easily user-adjustable (2 minutes is a long time
for a user to stare at an idle console).

Or we could work out how we are chewing up dom0 for 10s per created domain.
Is the total time to create 12 domains less if you create them one after
another (serially) rather than issuing the xm requests all at the same time?
I wonder if creating many domains in parallel is thrashing domain 0 somehow.

>>> At the same time, we found when main_loop_wait is increased to
>>> 100 ms, windows guest's keyboard and mouse response is
>>> significantly slower.
>> 
>> Perhaps we need a single polling daemon that can demux kb/mouse events
> to
>> all qemu instances in a way that can be select()ed on. That would avoid
>> unscalable polling overheads.
> Good solution. Could you make a patch for this? :-)

I need to discuss with other Cambridge folks. Moving to qemu stub domain and
split driver for kb/mouse input will solve this issue anyway in the longer
term.

 -- Keir

RE: about hot plug issue

[Yu, Ping Y]

>Or we could work out how we are chewing up dom0 for 10s per created
domain.
>Is the total time to create 12 domains less if you create them one
after
>another (serially) rather than issuing the xm requests all at the same
time?
>I wonder if creating many domains in parallel is thrashing domain 0
somehow.

We have tried to start 14 domains in serial, and the total time is 2
minutes 
21 seconds, and if staring in parallel, the total time is 1 minutes 51
second,
counting from "xm create" to message "Started domain domain_name)
Hope this info is useful to you. :-)
>
>>>> At the same time, we found when main_loop_wait is increased to
>>>> 100 ms, windows guest's keyboard and mouse response is
>>>> significantly slower.
>>>
>>> Perhaps we need a single polling daemon that can demux kb/mouse
events
>> to
>>> all qemu instances in a way that can be select()ed on. That would
avoid
>>> unscalable polling overheads.
>> Good solution. Could you make a patch for this? :-)
>
>I need to discuss with other Cambridge folks. Moving to qemu stub
domain and
>split driver for kb/mouse input will solve this issue anyway in the
longer
>term.
>
> -- Keir

Re: about hot plug issue

[Keir Fraser]

On 26/9/06 11:20, "Yu, Ping Y" <ping.y.yu@intel.com> wrote:

>> Or we could work out how we are chewing up dom0 for 10s per created
> domain.
>> Is the total time to create 12 domains less if you create them one
> after
>> another (serially) rather than issuing the xm requests all at the same
> time?
>> I wonder if creating many domains in parallel is thrashing domain 0
> somehow.
> 
> We have tried to start 14 domains in serial, and the total time is 2
> minutes 
> 21 seconds, and if staring in parallel, the total time is 1 minutes 51
> second,
> counting from "xm create" to message "Started domain domain_name)
> Hope this info is useful to you. :-)

Well, it means we're not thrashing dom0 by doing parallel creates. That
means the answer (if we care) is to make HVM guest creation less slow. :-)

 -- Keir