A few outstanding xenstore/hotplug issues...

A few outstanding xenstore/hotplug issues...

[Matt Ayres]

Greetings!

I am still noticing a few issues related to xenstore and hotplug.

1) xenstored will periodically die.  I do have xenstore tracing enabled, 
but not much is shown.  A transaction is running one second and then 
nothing (no "FATAL ERROR" or anything). This does not happen very 
frequently, but does happen now and then. As is known, once xenstored 
dies the only way to get it running again is with a reboot. A bad effect 
lies in the fact that domains can't be shutdown properly without 
xenstored running!

2) I get the old "Cannot connect VBD" on boot-up while running 
xendomains start, I'd assume due to xenstore getting bogged down. It's 
completely random as to which domains it will do it to.  I've worked 
around this by having xendomains run from init.d and then also re-run it 
from rc.local to start all the domains that didn't start the first time. 
  Even doing this causes some domains to not start properly sometimes 
(it seems based on how many failed to start the first time).

I remember reading that xenstore is getting an overhaul in the near 
future and want to know if these are "known issues" and if I should 
bother submitting bug reports or if I should just sit tight and watch 
the lists.

Thank you,
Matt Ayres

Re: A few outstanding xenstore/hotplug issues...

[Anthony Liguori]

Matt Ayres wrote:
> Greetings!
>
> I am still noticing a few issues related to xenstore and hotplug.
>
> 1) xenstored will periodically die.  I do have xenstore tracing 
> enabled, but not much is shown.  A transaction is running one second 
> and then nothing (no "FATAL ERROR" or anything). This does not happen 
> very frequently, but does happen now and then. As is known, once 
> xenstored dies the only way to get it running again is with a reboot. 
> A bad effect lies in the fact that domains can't be shutdown properly 
> without xenstored running!

Define "die".  Does it seg fault?  Can you try to repro with ulimit -c 
unlimited and check / to see if there's a core?

If it doesn't drop a core, that is also useful to know as that would 
suggest an exit() of some sort (although no message is rather odd!).

> 2) I get the old "Cannot connect VBD" on boot-up while running 
> xendomains start, I'd assume due to xenstore getting bogged down. It's 
> completely random as to which domains it will do it to.  I've worked 
> around this by having xendomains run from init.d and then also re-run 
> it from rc.local to start all the domains that didn't start the first 
> time.  Even doing this causes some domains to not start properly 
> sometimes (it seems based on how many failed to start the first time).

What changeset are you using and do you have any custom patches going?  
Have you ever seen this without xendomains?

> I remember reading that xenstore is getting an overhaul in the near 
> future and want to know if these are "known issues" and if I should 
> bother submitting bug reports or if I should just sit tight and watch 
> the lists.

I'm not aware of any overhaul although that doesn't mean it isn't 
happening.  I'd appreciate knowing if anyone was planning on doing this 
myself.

Regards,

Anthony Liguori

> Thank you,
> Matt Ayres
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: A few outstanding xenstore/hotplug issues...

[Ewan Mellor]

On Mon, Jun 05, 2006 at 11:02:55AM -0400, Matt Ayres wrote:

> Greetings!
> 
> I am still noticing a few issues related to xenstore and hotplug.
> 
> 1) xenstored will periodically die.  I do have xenstore tracing enabled, 
> but not much is shown.  A transaction is running one second and then 
> nothing (no "FATAL ERROR" or anything). This does not happen very 
> frequently, but does happen now and then. As is known, once xenstored 
> dies the only way to get it running again is with a reboot. A bad effect 
> lies in the fact that domains can't be shutdown properly without 
> xenstored running!
> 
> 2) I get the old "Cannot connect VBD" on boot-up while running 
> xendomains start, I'd assume due to xenstore getting bogged down. It's 
> completely random as to which domains it will do it to.  I've worked 
> around this by having xendomains run from init.d and then also re-run it 
> from rc.local to start all the domains that didn't start the first time. 
>  Even doing this causes some domains to not start properly sometimes 
> (it seems based on how many failed to start the first time).
> 
> I remember reading that xenstore is getting an overhaul in the near 
> future and want to know if these are "known issues" and if I should 
> bother submitting bug reports or if I should just sit tight and watch 
> the lists.

No, there's no major effort being put into xenstore at the moment.
We're certainly interested in fixing the bugs there, but your ones have
been very hard to track down!  If you can get a core dump, then I'll
certainly have a look for you, but without that, your crash bug is so
intermittent that it's been impossible to find.

The xendomains one ought to be easier to work on, so I'll get someone to
have a look at that for you.

Cheers,

Ewan.

Re: A few outstanding xenstore/hotplug issues...

[Matt Ayres]

Ewan Mellor wrote:

>>
>> I remember reading that xenstore is getting an overhaul in the near 
>> future and want to know if these are "known issues" and if I should 
>> bother submitting bug reports or if I should just sit tight and watch 
>> the lists.
> 
> No, there's no major effort being put into xenstore at the moment.
> We're certainly interested in fixing the bugs there, but your ones have
> been very hard to track down!  If you can get a core dump, then I'll
> certainly have a look for you, but without that, your crash bug is so
> intermittent that it's been impossible to find.
> 
> The xendomains one ought to be easier to work on, so I'll get someone to
> have a look at that for you.
> 

I was confused then. I went back and realized I mis-read something in 
the thread regarding the new credit scheduler.

As far as the bugs, I'll go ahead and submit bugzilla id's for both for 
now.  The xendomains should be very easy to reproduce, simply create 
many domains (50+) and keep running xendomains start/stop and I'm 
confident you'll see it :)  I'd bet making the domain start usleep in 
/etc/sysconfig/xendomains would fix it for me as it appears it's just 
xenstore or hotplug getting backed up.  The reason I won't is because I 
don't want my servers to take twice as long to boot when the bug should 
be fixed so no band-aid is needed.

Thanks,
Matt