calling vif hotplug scripts more than once

calling vif hotplug scripts more than once

[Olaf Hering]

There is a regression in libxl in its handling of hotplug scripts. Its
not clear to me how to solve it.

If in a Windows VM the network interface is disabled and reenabled in
the "Network and Sharing Center", the backend remains in state 2 and the
vif is not reconnected to the bridge.  With xend all that was done via
udev. If run_hotplug_scripts in /etc/xen/xl.conf is set to 0 it works as
well.

Looking through libxl it appears that libxl__device_nic_add ->
libxl__wait_device_connection -> device_backend_callback ->
device_hotplug does all the work. Unfortunately, device_backend_callback
unregisters the watch, so no further events will be received. It seems
device_backend_callback should call device_hotplug, and then register
yet another watch to wait for state changes so that it eventually
catches such a disable/enable event from the guest.

Since all that is generic code, it seems this use case was not
considered. It looks like a one-shot approach to me.

I'm not sure how to fix this. Should device_backend_callback call into
libxl__wait_device_connection again to wait (forever) for the next 'state==2'?


Olaf

Re: calling vif hotplug scripts more than once

[Ian Jackson]

Olaf Hering writes ("calling vif hotplug scripts more than once"):
> 
> There is a regression in libxl in its handling of hotplug scripts. Its
> not clear to me how to solve it.
> 
> If in a Windows VM the network interface is disabled and reenabled in
> the "Network and Sharing Center", the backend remains in state 2 and the
> vif is not reconnected to the bridge.  With xend all that was done via
> udev. If run_hotplug_scripts in /etc/xen/xl.conf is set to 0 it works as
> well.

Surely the problem is just that the backend isn't going back to the
operational state.  I don't see why this would need the hotplug script
to run again.

> Looking through libxl it appears that libxl__device_nic_add ->
> libxl__wait_device_connection -> device_backend_callback ->
> device_hotplug does all the work. Unfortunately, device_backend_callback
> unregisters the watch, so no further events will be received. It seems
> device_backend_callback should call device_hotplug, and then register
> yet another watch to wait for state changes so that it eventually
> catches such a disable/enable event from the guest.
> 
> Since all that is generic code, it seems this use case was not
> considered. It looks like a one-shot approach to me.

With xl, there is no daemon process which could do this waiting.

Ian.

Re: calling vif hotplug scripts more than once

[Olaf Hering]

On Tue, Jul 08, Ian Jackson wrote:

> Surely the problem is just that the backend isn't going back to the
> operational state.  I don't see why this would need the hotplug script
> to run again.

Something needs to reconnect the device to the bridge. There is also
that hotplug-status thing, which netback reads. I have to check what it
does with that.

Olaf

Re: calling vif hotplug scripts more than once

[Ian Jackson]

Olaf Hering writes ("Re: calling vif hotplug scripts more than once"):
> On Tue, Jul 08, Ian Jackson wrote:
> > Surely the problem is just that the backend isn't going back to the
> > operational state.  I don't see why this would need the hotplug script
> > to run again.
> 
> Something needs to reconnect the device to the bridge. There is also
> that hotplug-status thing, which netback reads. I have to check what it
> does with that.

Why is the device removed from the bridge ?  What does that ?

Ian.

Re: calling vif hotplug scripts more than once

[Olaf Hering]

On Tue, Jul 08, Ian Jackson wrote:

> Olaf Hering writes ("Re: calling vif hotplug scripts more than once"):
> > On Tue, Jul 08, Ian Jackson wrote:
> > > Surely the problem is just that the backend isn't going back to the
> > > operational state.  I don't see why this would need the hotplug script
> > > to run again.
> > 
> > Something needs to reconnect the device to the bridge. There is also
> > that hotplug-status thing, which netback reads. I have to check what it
> > does with that.
> 
> Why is the device removed from the bridge ?  What does that ?

Looks like netback does that, because the interface disappears.
Before "disable":

# brctl show
bridge name     bridge id               STP enabled     interfaces
br0             8000.00265523e546       no              eth0
                                                        vif1.0
                                                        vif1.0-emu
After "disable":
# brctl show
bridge name     bridge id               STP enabled     interfaces
br0             8000.00265523e546       no              eth0
                                                        vif1.0-emu

dmesg has:
[50367.178364] br0: port 2(vif1.0) entering forwarding state
[50367.178510] device vif1.0 left promiscuous mode
[50367.178516] br0: port 2(vif1.0) entering disabled state
[50367.227599] frontend_changed: backend/vif/1/0: prepare for reconnect

The last message appears when the frontend is in XenbusStateInitialising and
the backend is in XenbusStateClosed, then the backend changes to
XenbusStateInitWait.

It looks like the backend goes back into XenbusStateConnected when I write
/local/domain/0/backend/vif/1/0/hotplug-status connected, but nothing puts
vif1.0 back into br0.


Olaf

Re: calling vif hotplug scripts more than once

[Ian Campbell]

On Wed, 2014-07-09 at 09:36 +0200, Olaf Hering wrote:
> On Tue, Jul 08, Ian Jackson wrote:
> 
> > Olaf Hering writes ("Re: calling vif hotplug scripts more than once"):
> > > On Tue, Jul 08, Ian Jackson wrote:
> > > > Surely the problem is just that the backend isn't going back to the
> > > > operational state.  I don't see why this would need the hotplug script
> > > > to run again.
> > > 
> > > Something needs to reconnect the device to the bridge. There is also
> > > that hotplug-status thing, which netback reads. I have to check what it
> > > does with that.
> > 
> > Why is the device removed from the bridge ?  What does that ?
> 
> Looks like netback does that, because the interface disappears.

It's an implicit behaviour of the Linux bridge, when an interface
disappears it is removed from the bridge (this isn't true of e.g.
openvswitch, so its not universal).

Now, I *thought* that the vif backend device was supposed to remain when
the frontend disconnected and would only be removed on the final
destruction of the domain (or unplug via the toolstack).

I suppose either I'm misremembering things or it has been broken at some
point. I'm CCing David and Zoltan in case what I'm remembering is
something from the XenServer kernels (since I vaguely remember this
Windows device manage thing from way back when I used to be involved in
such things).

Ian.

Re: calling vif hotplug scripts more than once

[Paul Durrant]

> -----Original Message-----
> From: xen-devel-bounces@lists.xen.org [mailto:xen-devel-
> bounces@lists.xen.org] On Behalf Of Olaf Hering
> Sent: 09 July 2014 08:36
> To: Ian Jackson
> Cc: xen-devel@lists.xen.org; Ian Campbell; Roger Pau Monne
> Subject: Re: [Xen-devel] calling vif hotplug scripts more than once
> 
> On Tue, Jul 08, Ian Jackson wrote:
> 
> > Olaf Hering writes ("Re: calling vif hotplug scripts more than once"):
> > > On Tue, Jul 08, Ian Jackson wrote:
> > > > Surely the problem is just that the backend isn't going back to the
> > > > operational state.  I don't see why this would need the hotplug script
> > > > to run again.
> > >
> > > Something needs to reconnect the device to the bridge. There is also
> > > that hotplug-status thing, which netback reads. I have to check what it
> > > does with that.
> >
> > Why is the device removed from the bridge ?  What does that ?
> 
> Looks like netback does that, because the interface disappears.
> Before "disable":
> 

What vintage of netback are you running? This is precisely the problem I fixed with c/s 279f438e36c0a70b23b86d2090aeec50155034a9:

-----
commit 279f438e36c0a70b23b86d2090aeec50155034a9
Author: Paul Durrant <Paul.Durrant@citrix.com>
Date:   Tue Sep 17 17:46:08 2013 +0100

    xen-netback: Don't destroy the netdev until the vif is shut down

    Without this patch, if a frontend cycles through states Closing
    and Closed (which Windows frontends need to do) then the netdev
    will be destroyed and requires re-invocation of hotplug scripts
    to restore state before the frontend can move to Connected. Thus
    when udev is not in use the backend gets stuck in InitWait.

    With this patch, the netdev is left alone whilst the backend is
    still online and is only de-registered and freed just prior to
    destroying the vif (which is also nicely symmetrical with the
    netdev allocation and registration being done during probe) so
    no re-invocation of hotplug scripts is required.

    Signed-off-by: Paul Durrant <paul.durrant@citrix.com>
    Cc: David Vrabel <david.vrabel@citrix.com>
    Cc: Wei Liu <wei.liu2@citrix.com>
    Cc: Ian Campbell <ian.campbell@citrix.com>
    Acked-by: Wei Liu <wei.liu2@citrix.com>
    Signed-off-by: David S. Miller <davem@davemloft.net>
-----

  Paul

> # brctl show
> bridge name     bridge id               STP enabled     interfaces
> br0             8000.00265523e546       no              eth0
>                                                         vif1.0
>                                                         vif1.0-emu
> After "disable":
> # brctl show
> bridge name     bridge id               STP enabled     interfaces
> br0             8000.00265523e546       no              eth0
>                                                         vif1.0-emu
> 
> dmesg has:
> [50367.178364] br0: port 2(vif1.0) entering forwarding state
> [50367.178510] device vif1.0 left promiscuous mode
> [50367.178516] br0: port 2(vif1.0) entering disabled state
> [50367.227599] frontend_changed: backend/vif/1/0: prepare for reconnect
> 
> The last message appears when the frontend is in XenbusStateInitialising and
> the backend is in XenbusStateClosed, then the backend changes to
> XenbusStateInitWait.
> 
> It looks like the backend goes back into XenbusStateConnected when I write
> /local/domain/0/backend/vif/1/0/hotplug-status connected, but nothing
> puts
> vif1.0 back into br0.
> 
> 
> Olaf
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: calling vif hotplug scripts more than once

[Ian Campbell]

On Wed, 2014-07-09 at 09:59 +0100, Paul Durrant wrote:
> This is precisely the problem I fixed with c/s 279f438e36c0a70b23b86d2090aeec50155034a9:

Aha, I knew it was familiar!

Ian.

Re: calling vif hotplug scripts more than once

[Olaf Hering]

On Wed, Jul 09, Paul Durrant wrote:

> What vintage of netback are you running? This is precisely the problem
> I fixed with c/s 279f438e36c0a70b23b86d2090aeec50155034a9:

This is the SLES kernel. Perhaps this change is missing.
I will have a look. Thanks!

Olaf