Re: Status of FLR in Xen 4.4

[Gordan Bobic <gordan@bobich.net>]

 On Fri, 27 Sep 2013 14:26:31 +0200, Matthias 
 <matthias.kannenberg@googlemail.com> wrote:
> Hi Gordon,
>
> I tried your patch on my dom0 kernel and I think it somehow helped in
> the sense that now I can reboot the domUs now without crashing the
> whole host, but linux domU still gets a blackscreen and windows7 domU
> only starts till black screen with (actual movable) cursor, but not
> furthor.. this might only be a coincidence, though, have to double
> check this..

 What patch? Nothing I posted to the list is fit for public
 consumption yet. You shouldn't be using it unless you really,
 REALLY know exactly what it does and know exactly what you
 are trying to achieve.

> I tried some other stuff, too:
>
> 1) after domU shutdown rebind both functions to the dom0 drivers, do 
> a
> sysfs reset and re-add to assignable devices -> crashes dom0

 My experience shows that letting dom0 drivers ever touch the hardware
 is a recipe for disaster.

> 2) after domU shutdown rebind both functions to the dom0 drivers and
> readd to assignable devices -> dom0 crashes somtime when domU using
> the devices comes up, sometimes not, but no success either way
>  3) sysfs reset of the devices within domU seems to be passed through
> dom0 (see commands in qemu-log) but no effect

 It's up to the drivers to do the sensible thing. Nvidia drivers
 handle this a little more sanely, but if the drivers cannot handle
 clobbering the device's state into a known state, you are pretty
 much fighting a losing battle.

> Also, I analysed your code and compared it to the stuff in the python
> tools of xm and it is the same approach and i don't see any obvious
> differences..

 I am starting to suspect you aren't actually talking about my code
 but somebody else's...

> Then I tried to replicate the secondary bus reset on
> command lind for testing purposes via
>
>  printf 'x40' | dd of=/sys/devices/pci0000:00/0000:00:0b.0/config 
> bs=1
> seek=$((0x3e)) count=1 conv=notrunc
>
> but I think I got some endians or offset slightly wrong because after
> that xl refuses to give the device (00:0b.0 is the bus of my
> 2-function vga card I have assigned to my domU) to the domU and later
> crashes dom0.
>
> So I'm a little lost at that point and would welcome some 
> suggestions.
>
> Does FLR reset works for any of you for vga cards?

 If you are talking about VGA cards with _proper_ FLR implementations
 on PCI level - there is no such thing. In all cases it is down to
 the domU driver to handle the card in whatever state it is. This
 works reasonably well with supported Nvidia cards (i.e.
 Quadro [K][2456]000 and Grid K[12] and equivalent modified GeForce
 cards (Fermi 4xx and Kepler 6xx/7xx series)). I never managed to
 get it working properly on any other GPUs.

 Even with Nvidia cards rebooting can lead to issues. For example,
 I have two GPUs passed to two different domUs. One is a GTX470
 modified to Q5000. The other is a GTX480 modified to Q6000. The
 domU with Q5000 always handled reboots reasonably reliably. The
 one with a Q6000 did not. I since switched the one with a Q6000
 to a QK5000 (modified GTX680), and now the reboots seem to work
 reasonably reliably, but I have found that there is still a
 crash if the monitor on the card changes between shutdown and
 restart - I'm guessing the card remembers it's state and if it
 isn't consistent when it returns, driver gets confused. I have
 other issues (see recent thread about Nvidia passthrough from
 David), but they seem to be specific to my setup.

 It's not perfect, but it's the only workable solution I have
 found.

 Gordan