extremely long xend startup time causing /etc/init.d/xendomains not to run?

extremely long xend startup time causing /etc/init.d/xendomains not to run?

[Sean Dague]

[-- Attachment #1.1: Type: text/plain, Size: 1169 bytes --]

I've started to notice the following issue, and would like to see if anyone
else has seen it before digging too deep.

When rebooting a xen system that has auto starting domains, they mostly
don't start.

On one system that tries to start 5 DomU instances, it appears only the last
one (alphabetically) ends up starting.  On the system with only 2 DomU
instances, nothing starts.  Neither of these are very speedy machines (700 &
800 Mhz respectively), so a timing problem with /etc/init.d/xend starting is
probably exacerbated in this environment.

Sanity confirmation on this would be really useful, so if anyone else has
seen this please also post back. :)

	-Sean

-- 
__________________________________________________________________

Sean Dague                                       Mid-Hudson Valley
sean at dague dot net                            Linux Users Group
http://dague.net                                 http://mhvlug.org

There is no silver bullet.  Plus, werewolves make better neighbors
than zombies, and they tend to keep the vampire population down.
__________________________________________________________________

[-- Attachment #1.2: Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: extremely long xend startup time causing /etc/init.d/xendomains not to run?

[Anthony Liguori]

Sean Dague wrote:

>I've started to notice the following issue, and would like to see if anyone
>else has seen it before digging too deep.
>
>When rebooting a xen system that has auto starting domains, they mostly
>don't start.
>
>On one system that tries to start 5 DomU instances, it appears only the last
>one (alphabetically) ends up starting.  On the system with only 2 DomU
>instances, nothing starts.  Neither of these are very speedy machines (700 &
>800 Mhz respectively), so a timing problem with /etc/init.d/xend starting is
>probably exacerbated in this environment.
>
>Sanity confirmation on this would be really useful, so if anyone else has
>seen this please also post back. :)
>  
>
I see this a lot in QEMU.  xm is unresponsive for a bit after xend starts.

We really need to make sure that xend start does not return until xend 
is ready to receive connections.

We also have to look at making the xend startup time a bit faster.

Regards,

Anthony Liguori

>	-Sean
>
>  
>
>------------------------------------------------------------------------
>
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.xensource.com
>http://lists.xensource.com/xen-devel
>  
>

Re: extremely long xend startup time causing /etc/init.d/xendomains not to run?

[Ryan Harper]

* Sean Dague <sean@dague.net> [2005-09-14 13:55]:
> I've started to notice the following issue, and would like to see if anyone
> else has seen it before digging too deep.
> 
> When rebooting a xen system that has auto starting domains, they mostly
> don't start.
> 
> On one system that tries to start 5 DomU instances, it appears only the last
> one (alphabetically) ends up starting.  On the system with only 2 DomU
> instances, nothing starts.  Neither of these are very speedy machines (700 &
> 800 Mhz respectively), so a timing problem with /etc/init.d/xend starting is
> probably exacerbated in this environment.
> 
> Sanity confirmation on this would be really useful, so if anyone else has
> seen this please also post back. :)

I've seen this a lot when working bug#197 As the number of entries in
/var/lib/xenstored/store/domain/$uuid increase, the time for xend to get
up and running increases.  When I had over 500 stale uuid directories in
the store, it took several *minutes* for xend to full come up, and
/var/lib/xenstored was several hundred MB.

The only workaround I know is to:

/etc/init.d/xend stop
killall xenstored xenconsoled
rm -rf /var/lib/xenstored/*
/etc/init.d/xend start 

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com

Re: extremely long xend startup time causing /etc/init.d/xendomains not to run?

[Sean Dague]

[-- Attachment #1.1: Type: text/plain, Size: 2049 bytes --]

On Wed, Sep 14, 2005 at 02:13:14PM -0500, Ryan Harper wrote:
> * Sean Dague <sean@dague.net> [2005-09-14 13:55]:
> > I've started to notice the following issue, and would like to see if anyone
> > else has seen it before digging too deep.
> > 
> > When rebooting a xen system that has auto starting domains, they mostly
> > don't start.
> > 
> > On one system that tries to start 5 DomU instances, it appears only the last
> > one (alphabetically) ends up starting.  On the system with only 2 DomU
> > instances, nothing starts.  Neither of these are very speedy machines (700 &
> > 800 Mhz respectively), so a timing problem with /etc/init.d/xend starting is
> > probably exacerbated in this environment.
> > 
> > Sanity confirmation on this would be really useful, so if anyone else has
> > seen this please also post back. :)
> 
> I've seen this a lot when working bug#197 As the number of entries in
> /var/lib/xenstored/store/domain/$uuid increase, the time for xend to get
> up and running increases.  When I had over 500 stale uuid directories in
> the store, it took several *minutes* for xend to full come up, and
> /var/lib/xenstored was several hundred MB.
> 
> The only workaround I know is to:
> 
> /etc/init.d/xend stop
> killall xenstored xenconsoled
> rm -rf /var/lib/xenstored/*
> /etc/init.d/xend start 

I tried that, and nstartup still has xend in an odd state after about 15
minutes.  It looks like a few domains attempted to autostart, but never
made it.  I'll compile this into a bugzilla tonight.

	-Sean

-- 
__________________________________________________________________

Sean Dague                                       Mid-Hudson Valley
sean at dague dot net                            Linux Users Group
http://dague.net                                 http://mhvlug.org

There is no silver bullet.  Plus, werewolves make better neighbors
than zombies, and they tend to keep the vampire population down.
__________________________________________________________________

[-- Attachment #1.2: Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: [Xen-tools] Re: extremely long xend startup time causing /etc/init.d/xendomains not to run?

[Anthony Liguori]

Sean Dague wrote:

>>/etc/init.d/xend stop
>>killall xenstored xenconsoled
>>rm -rf /var/lib/xenstored/*
>>/etc/init.d/xend start 
>>    
>>
>
>I tried that, and nstartup still has xend in an odd state after about 15
>minutes.  It looks like a few domains attempted to autostart, but never
>made it.  I'll compile this into a bugzilla tonight.
>  
>
You should try with my latest patches.  Without them, I don't think 
there's much hope that the xendomains script would work on a slower system.

That doesn't mean that I think it will work, but rather that there is at 
least a chance now :-)

Regards,

Anthony Liguori

>	-Sean
>
>  
>
>------------------------------------------------------------------------
>
>_______________________________________________
>Xen-tools mailing list
>Xen-tools@lists.xensource.com
>http://lists.xensource.com/xen-tools
>

Re: [Xen-tools] Re: extremely long xend startup time causing /etc/init.d/xendomains not to run?

[Sean Dague]

[-- Attachment #1.1: Type: text/plain, Size: 1464 bytes --]

On Wed, Sep 14, 2005 at 08:44:40PM -0500, Anthony Liguori wrote:
> Sean Dague wrote:
> 
> >>/etc/init.d/xend stop
> >>killall xenstored xenconsoled
> >>rm -rf /var/lib/xenstored/*
> >>/etc/init.d/xend start 
> >>   
> >>
> >
> >I tried that, and nstartup still has xend in an odd state after about 15
> >minutes.  It looks like a few domains attempted to autostart, but never
> >made it.  I'll compile this into a bugzilla tonight.
> > 
> >
> You should try with my latest patches.  Without them, I don't think 
> there's much hope that the xendomains script would work on a slower system.
> 
> That doesn't mean that I think it will work, but rather that there is at 
> least a chance now :-)

Well, that part works now.  My 5 DomU machine now actually starts 5 DomUs on
boot! :)  Thanks Anthony!

Networking is hosed in some way still, but it's been a couple weeks since I
brought this box up to sync so that may just require some fiddling.

	-Sean

-- 
__________________________________________________________________

Sean Dague                                       Mid-Hudson Valley
sean at dague dot net                            Linux Users Group
http://dague.net                                 http://mhvlug.org

There is no silver bullet.  Plus, werewolves make better neighbors
than zombies, and they tend to keep the vampire population down.
__________________________________________________________________

[-- Attachment #1.2: Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel