problem restarting xenstored

problem restarting xenstored

[Jan Beulich]

As any other daemon, I'd assume it should be possible to restart xenstored (i.e.
if it crashed or if, during development, a fix was made to it that needs to be
tested). However, as I learned after over a day of digging, things don't work
anymore after doing so: Since the watches are transient, they're lost with the
daemon going away.
Right now I specifically need this in already mentioned scenarios where I want
to not touch the base installation of a system or need parallel versions installed,
but as described above I see the general issue broader (e.g. it obviously
doesn't make sense to try to start xenstored from 'xend restart' because of
described problems).
For some reason, this wasn't a big problem when the base installation was
3.0.2-based - apparently something else prevented xenbus' event channel
handler to get triggered when the 3.0.2 xenstored loaded, and hence all the
needed watches got installed only when the proper version was loaded the
first time, giving the impression that things work as expected (until perhaps
reloading it a second time).

Any suggestions on how to overcome that would be very welcome.

Thanks, Jan

Re: problem restarting xenstored

[Ewan Mellor]

On Thu, Dec 07, 2006 at 03:21:59PM +0000, Jan Beulich wrote:

> As any other daemon, I'd assume it should be possible to restart xenstored (i.e.
> if it crashed or if, during development, a fix was made to it that needs to be
> tested). However, as I learned after over a day of digging, things don't work
> anymore after doing so: Since the watches are transient, they're lost with the
> daemon going away.
> Right now I specifically need this in already mentioned scenarios where I want
> to not touch the base installation of a system or need parallel versions installed,
> but as described above I see the general issue broader (e.g. it obviously
> doesn't make sense to try to start xenstored from 'xend restart' because of
> described problems).

xend restart doesn't restart xenstored, for this very reason of course.

I don't see why this is a problem for your "multiple installed versions of
Xen" -- surely you reboot in order to switch toolstacks?

Keir did have some ideas about making xenstored restartable -- reregistering
the watches when the daemon restarts -- but no-one's got around to doing that
work.  Are you volunteering?  ;-)

> For some reason, this wasn't a big problem when the base installation was
> 3.0.2-based - apparently something else prevented xenbus' event channel
> handler to get triggered when the 3.0.2 xenstored loaded, and hence all the
> needed watches got installed only when the proper version was loaded the
> first time, giving the impression that things work as expected (until perhaps
> reloading it a second time).

You've lost me -- what are you trying to do?  Are you trying to start multiple
versions of xenstored during the same boot?

Xenstored actually hasn't changed in any manner that would make it
backwards-incompatible, AFAIK, so you ought to be able to run a 3.0.2
toolstack with xenstored from 3.0.4, for example.

Ewan.

Re: problem restarting xenstored

[Jan Beulich]

>>> Ewan Mellor <ewan@xensource.com> 07.12.06 16:53 >>>
>On Thu, Dec 07, 2006 at 03:21:59PM +0000, Jan Beulich wrote:
>
>> As any other daemon, I'd assume it should be possible to restart xenstored (i.e.
>> if it crashed or if, during development, a fix was made to it that needs to be
>> tested). However, as I learned after over a day of digging, things don't work
>> anymore after doing so: Since the watches are transient, they're lost with the
>> daemon going away.
>> Right now I specifically need this in already mentioned scenarios where I want
>> to not touch the base installation of a system or need parallel versions installed,
>> but as described above I see the general issue broader (e.g. it obviously
>> doesn't make sense to try to start xenstored from 'xend restart' because of
>> described problems).
>
>xend restart doesn't restart xenstored, for this very reason of course.

It does, in case it isn't already running (i.e. if it crashed).

>I don't see why this is a problem for your "multiple installed versions of
>Xen" -- surely you reboot in order to switch toolstacks?

I obviously need to reboot, as the tools need to match the hypervisor.

>Keir did have some ideas about making xenstored restartable -- reregistering
>the watches when the daemon restarts -- but no-one's got around to doing that
>work.  Are you volunteering?  ;-)

No, I'm not. There's more than enough other stuff I'd like to deal with. And
I'm neither very familiar with the tools subtree, nor am I keen on touching it.

>> For some reason, this wasn't a big problem when the base installation was
>> 3.0.2-based - apparently something else prevented xenbus' event channel
>> handler to get triggered when the 3.0.2 xenstored loaded, and hence all the
>> needed watches got installed only when the proper version was loaded the
>> first time, giving the impression that things work as expected (until perhaps
>> reloading it a second time).
>
>You've lost me -- what are you trying to do?  Are you trying to start multiple
>versions of xenstored during the same boot?

Yes - as the system's base configuration (obviously, as long as the xen package
is installed) tries to start xend. xend itself fails (due to mismatching change sets),
but blktapctrl, xenstored, and xenconsoled all load successfully.

>Xenstored actually hasn't changed in any manner that would make it
>backwards-incompatible, AFAIK, so you ought to be able to run a 3.0.2
>toolstack with xenstored from 3.0.4, for example.

Obviously not - otherwise I wouldn't have this difference in behavior between
a system with 3.0.2 installed where I try to run 3.0.3 or -unstable (which
succeed the first time through) and one with 3.0.3 installed where I try to run
-unstable (and things don't work).
As pointed out before - apparently the notification to the kernel's xenbus
piece doesn't get through from 3.0.2 xenstored to a post-3.0.2
kernel/hypervisor, resulting in the watches not getting registered until an
appropriate xenstored loads. Using -unstable with a 3.0.3 xenstored, on
the other hand, results in xenbus_probe() getting invoked right away.

Jan

Re: problem restarting xenstored

[John Levon]

On Thu, Dec 07, 2006 at 03:53:42PM +0000, Ewan Mellor wrote:

> Keir did have some ideas about making xenstored restartable -- reregistering
> the watches when the daemon restarts -- but no-one's got around to doing that
> work.  Are you volunteering?  ;-)

What are those ideas?

john

Re: problem restarting xenstored

[Steven Hand]

>>You've lost me -- what are you trying to do?  Are you trying to start 
>>multiple versions of xenstored during the same boot?
>
>Yes- as the system's base configuration (obviously, as long as the xen package
>is installed) tries to start xend. xend itself fails (due to mismatching 
>change sets), but blktapctrl, xenstored, and xenconsoled all load successfully.

I've done this before by just killing the old xenstored - does this 
not work for you? 

cheers,

S.

Re: problem restarting xenstored

[Jan Beulich]

>>> Steven Hand <Steven.Hand@cl.cam.ac.uk> 07.12.06 17:24 >>>
>
>>>You've lost me -- what are you trying to do?  Are you trying to start 
>>>multiple versions of xenstored during the same boot?
>>
>>Yes- as the system's base configuration (obviously, as long as the xen package
>>is installed) tries to start xend. xend itself fails (due to mismatching 
>>change sets), but blktapctrl, xenstored, and xenconsoled all load successfully.
>
>I've done this before by just killing the old xenstored - does this 
>not work for you? 

Only if the previous one is 3.0.2 and the underlying kernel/hypervisor is at
least 3.0.3. Otherwise, I can't start any domains anymore (since due to the
lost watches no events reach the backends then anymore).

Jan

Re: problem restarting xenstored

[Anthony Liguori]

Jan Beulich wrote:
> As any other daemon, I'd assume it should be possible to restart xenstored (i.e.
> if it crashed or if, during development, a fix was made to it that needs to be
> tested). However, as I learned after over a day of digging, things don't work
> anymore after doing so: Since the watches are transient, they're lost with the
> daemon going away.
> Right now I specifically need this in already mentioned scenarios where I want
> to not touch the base installation of a system or need parallel versions installed,
> but as described above I see the general issue broader (e.g. it obviously
> doesn't make sense to try to start xenstored from 'xend restart' because of
> described problems).
> For some reason, this wasn't a big problem when the base installation was
> 3.0.2-based - apparently something else prevented xenbus' event channel
> handler to get triggered when the 3.0.2 xenstored loaded, and hence all the
> needed watches got installed only when the proper version was loaded the
> first time, giving the impression that things work as expected (until perhaps
> reloading it a second time).
> 
> Any suggestions on how to overcome that would be very welcome.

Watches have to become persistent in XenStore.  This has been a TODO 
since the very beginning that noone has ever tackled.  Currently watches 
are only stored in memory so if you restart XenStore any active watch 
will never be fired.

How much of an effect this has on your system depends on the state any 
running domains are in and what functions you use.

Regards,

Anthony Liguori

> Thanks, Jan