Xenbus behaviour

Xenbus behaviour

[Glauber de Oliveira Costa]

Hello folks,

I've noticed today that some xenbus state changes are reported twice, as
in the follow example from dmesg:

state is 4 (Connected), /local/domain/8/device/vif/4/state,
/local/domain/8/device/vif/4/stateDele
state is 4 (Connected), /local/domain/8/device/vif/4/state,
/local/domain/8/device/vif/4/stateDele

I'm not sure whether this is legal or a bug.

Any advice ?

Thanks!

-- 
Glauber de Oliveira Costa
Red Hat Inc.
"Free as in Freedom"

Re: Xenbus behaviour

[Keir Fraser]

On 13/12/06 8:11 pm, "Glauber de Oliveira Costa" <gcosta@redhat.com> wrote:

> I've noticed today that some xenbus state changes are reported twice, as
> in the follow example from dmesg:
> 
> state is 4 (Connected), /local/domain/8/device/vif/4/state,
> /local/domain/8/device/vif/4/stateDele
> state is 4 (Connected), /local/domain/8/device/vif/4/state,
> /local/domain/8/device/vif/4/stateDele
> 
> I'm not sure whether this is legal or a bug.

It's certainly legal, but a bit surprising and annoying. Only the 'state'
node is watched, so presumably it is getting written to twice?

 -- Keir

Re: Xenbus behaviour

[Glauber de Oliveira Costa]

On Thu, 2006-12-14 at 08:24 +0000, Keir Fraser wrote:
> On 13/12/06 8:11 pm, "Glauber de Oliveira Costa" <gcosta@redhat.com> wrote:
> 
> > I've noticed today that some xenbus state changes are reported twice, as
> > in the follow example from dmesg:
> > 
> > state is 4 (Connected), /local/domain/8/device/vif/4/state,
> > /local/domain/8/device/vif/4/stateDele
> > state is 4 (Connected), /local/domain/8/device/vif/4/state,
> > /local/domain/8/device/vif/4/stateDele
> > 
> > I'm not sure whether this is legal or a bug.
> 
> It's certainly legal, but a bit surprising and annoying. Only the 'state'
> node is watched, so presumably it is getting written to twice?

I'm not yet sure about the root of this problem. But from what I've
gathered so far, it seems (I'll try confirming this today) that the two
fires are actually happening. But one of them is for InitWait, and the
other one is for Closing. But when the first one is read, Closing state
has already been written in the store, overwriting the old value. (It is
not the situation that is shown in the log, tough. I'm analyzing it more
on the guest side now). Result is that guest reads Closing twice, when
it should be reading InitWait and Closing. Do you think this is a
reasonable hypothesis ? 

Moreover, if multiple triggers are indeed valid, we should maybe take
extra care about double fires in drivers. For example, this situation I'
m describing crashes xennet on domU, because when receiving the second
Closing, it tries to unregister_netdev() again, in an invalid state. 

Thanks

Re: Xenbus behaviour

[Keir Fraser]

> Moreover, if multiple triggers are indeed valid, we should maybe take
> extra care about double fires in drivers. For example, this situation I'
> m describing crashes xennet on domU, because when receiving the second
> Closing, it tries to unregister_netdev() again, in an invalid state.

The drivers do need to be robust against multiple firings so this needs
fixing.

 -- Keir

Re: Xenbus behaviour

[Ewan Mellor]

On Thu, Dec 14, 2006 at 09:56:45AM -0500, Glauber de Oliveira Costa wrote:

> On Thu, 2006-12-14 at 08:24 +0000, Keir Fraser wrote:
> > On 13/12/06 8:11 pm, "Glauber de Oliveira Costa" <gcosta@redhat.com> wrote:
> > 
> > > I've noticed today that some xenbus state changes are reported twice, as
> > > in the follow example from dmesg:
> > > 
> > > state is 4 (Connected), /local/domain/8/device/vif/4/state,
> > > /local/domain/8/device/vif/4/stateDele
> > > state is 4 (Connected), /local/domain/8/device/vif/4/state,
> > > /local/domain/8/device/vif/4/stateDele
> > > 
> > > I'm not sure whether this is legal or a bug.
> > 
> > It's certainly legal, but a bit surprising and annoying. Only the 'state'
> > node is watched, so presumably it is getting written to twice?
> 
> I'm not yet sure about the root of this problem. But from what I've
> gathered so far, it seems (I'll try confirming this today) that the two
> fires are actually happening. But one of them is for InitWait, and the
> other one is for Closing. But when the first one is read, Closing state
> has already been written in the store, overwriting the old value. (It is
> not the situation that is shown in the log, tough. I'm analyzing it more
> on the guest side now). Result is that guest reads Closing twice, when
> it should be reading InitWait and Closing. Do you think this is a
> reasonable hypothesis ? 
> 
> Moreover, if multiple triggers are indeed valid, we should maybe take
> extra care about double fires in drivers. For example, this situation I'
> m describing crashes xennet on domU, because when receiving the second
> Closing, it tries to unregister_netdev() again, in an invalid state. 

That sounds like a reasonable hypothesis, yes, and it definitely needs
fixing.  There's always going to be a race between the watch firing and the
driver coming back to read things that may change underneath.  The watch
should always mean "something might have changed, go check", and nothing more
precise than that.

Ewan.

Re: Xenbus behaviour

[Ewan Mellor]

On Wed, Dec 13, 2006 at 06:11:59PM -0200, Glauber de Oliveira Costa wrote:

> Hello folks,
> 
> I've noticed today that some xenbus state changes are reported twice, as
> in the follow example from dmesg:
> 
> state is 4 (Connected), /local/domain/8/device/vif/4/state,
> /local/domain/8/device/vif/4/stateDele
> state is 4 (Connected), /local/domain/8/device/vif/4/state,
> /local/domain/8/device/vif/4/stateDele
> 
> I'm not sure whether this is legal or a bug.

Each watch should fire once when you register it, and then once per write per
registration.  Do you have a watch on the state node specifically and then one
higher up that device tree?  If your print statement happens on every watch
regardless of what caused it, then that would do it.

The easiest way to find out is to turn on Xenstored tracing
(http://wiki.xensource.com/xenwiki/XenDebugging) as this will give you a trace
message for every single watch registration, watch firing, read and write.

Ewan.