Re: xl pci-detach vs xm pci-detach in Xen 4.3 (one works, the other does not)

[Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>]

On Mon, Jun 10, 2013 at 02:28:11PM +0100, George Dunlap wrote:
> On 10/06/13 14:20, Konrad Rzeszutek Wilk wrote:
> >On Mon, Jun 10, 2013 at 12:12:47PM +0100, George Dunlap wrote:
> >>create ^
> >>title it xl pci-detach failure
> >>thanks
> >>
> >>On Fri, Jun 7, 2013 at 4:45 PM, Konrad Rzeszutek Wilk
> >><konrad.wilk@oracle.com> wrote:
> >>>any thoughts? I don't know if this is a regression or not, but the
> >>>libxl errors are pointing me to the recent XSA issue - which I thought
> >>>was fixed?
> >>I think the recent issue was with xend not working.  Did we fix send
> >>and break xl?
> >Could be. It might also be that that 'xl' never did it the same way
> >as 'xend' (meaning this might be a XenBus teardown change).
> >
> >Note also that this is PV guests - and I think most of the testing
> >had been with the HVM guests with PCI passthrough. So it might be a
> >seperate issue altogether. Or that nobody tried doing PCI plug/unplug
> >in the past :-(
> 
> I certainly did before my February FOSDEM talk that included driver domains.
> 
> >
> >>In any case, this is a pretty important feature; I think we need to
> >>sort it out before release, so I'm giving it a bug id to track.
> >>
> >>Is "pci permissive" set in the global xl.conf file?
> >No. Let me of course try that.
> 
> That really should have to do with getting it working in the first
> place, not detaching it; but still...
> 
> >>Have you tried this with say, a stock Debian Wheezy kernel?
> >No. A v3.10-rc4 with Xen 4.3 latest.
> >
> >Is Debian Wheezy a 2.6.32 kernel?
> 
> No, 3.2.

I figured it out. It is the XenBus states.


The 'xm' for pci-detach would do:
 4(Connected)->7(Reconfiguring)-> 8(Reconfigured)-> 4(Connected)->5(Closing).

While 'xl' does:
 4(Connected)->7(Reconfiguring)-> 8(Reconfigured)-> 4(Connected)

That means the xen-pcifront never gets told that the connection is going to
be removed and can do its cleanup.

Without the cleanup it fails at the 3->4(Connected) change state:

[  151.403112] pcifront pci-0: publishing successful!
[  151.404313] pcifront pci-0: backend new state: 2 (old state:3)!
[  151.406160] pcifront pci-0: backend new state: 3 (old state:3)!
[  151.407287] pcifront pci-0: backend new state: 4 (old state:3)!
[  151.407463] pcifront pci-0: PCI frontend already installed!
===> [  151.407474] pcifront pci-0: 17 Error setting up PCI Frontend <=====
[  151.407680] pcifront pci-0: failed to write error node for device/pci/0 (17 Error setting up PCI Frontend) ret: -13
[  151.410502] pcifront pci-0: backend new state: 5 (old state:5)!
[  151.410509] pcifront pci-0: backend going away!

B/c the 'PCI frontend already installed' check has been hit (new in 3.7)
and would never progress further (see git commit
3d925320e9e2de162bd138bf97816bda8c3f71be - xen/pcifront: Use Xen-SWIOTLB when initting if required.)


This looks like an OK protocol change so I am inclined to say the Xen pcifront
needs a bit more checking. Sending patches for that shortly.