Re: Re: [vt-d][xen4-rc6] Hangs on startup

[Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>]

On Mon, Mar 22, 2010 at 04:11:34PM -0600, Nadolski, Ed wrote:
> > -----Original Message-----
> > From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com]
> > Sent: Monday, March 22, 2010 2:54 PM
> > To: Nadolski, Ed
> > Cc: Łukasz Oleś; xen-devel@lists.xensource.com
> > Subject: Re: [Xen-devel] Re: [vt-d][xen4-rc6] Hangs on startup
> > 
> > On Mon, Mar 22, 2010 at 03:06:48PM -0600, Nadolski, Ed wrote:
> > >
> > >
> > > > -----Original Message-----
> > >
> > > > PCI back is to be used _only_ with PV guests - you on the other
> > > > hand are running an HVM guest.
> > > >
> > > > Per the earlier statement, I would recommend you use the 'pciback'
> > > > instead of 'pci-stub' or just not compile pciback in and see what
> > > > happens. Keep in mind: PCI back module is only needed when you want
> > to
> > > > do PV PCI passthrough, which is not what you are doing.
> > >
> > > I'm confused - does that mean pci-stub must be used for device
> > assignment to an HVM guest?  The VTdHowTo isn't clear on that.
> > 
> > Not per say.
> > 
> > xen-pciback can be used for both PV and HVM.
> 
> OK then what is the difference between pciback and xen-pciback?  I thought they were just a re-name, but there must be more than that if one supports HVM and the other doesn't.

That is is. Just a rename.
> 
> 
> > pci-stub can only be used for HVM guests.
> 
> OK, but then why would I ever want to use pci-stub, if xen-pciback already does both?

Uhh.. you probably wouldn't. But the pci-stub code has been in existence
in the Linux kernel for some time..
> 
> 
> 
> > But there seems to be a bug somewhere that when the PCI device is
> > assigned to pci-stub, pciback tries to seize it and can't find it and
> > somehow is stuck in a spin-lock.  That shouldn't be happening.
> > 
> > Right now I am trying to figure out if we remove from Lukasza system
> > pciback and only use pci-stub whether he still gets those MFN lookup
> > errors with his QLogic card. Those are, I believe, a seperate issue
> > from the pciback spinlock failure.
> 
> Yes, those sound like two different things.
> 
> I too am seeing problems with a Qlogic card in an HVM.  It looks like my card's BIOS makes an INT1A call (FIND PCI DEVICE function) that fails when the HVM is booting, and the code hangs forever.

I would think this would work as the Bochs BIOS is actually being used
with real hardware. Is it possible that some data-structure that Bochs
BIOS has for the PCI enumeration is not up-to-date with the hvmloader
sets up? 

What about the BDF numbers? Any chance of trying to make them 1-to-1 (so
what you see in Dom0 is what you would see in DomU)?

Maybe there is an option in QEMU to do that?

What about passing in the PCI device _after_ the OS has booted. You can
do that by using 'pci-attach' (or something) flags. That will mean no
BIOS initialization, but the qla2xxx driver might be OK with that.

> But AFAICT it looks like Lukasz's latest trace hasn't gotten to the point where it tries to load the PCI option ROM from the card.  I would be interested to see if his card sees that error or not, once the MFN is resolved.

<nods>
> 
> Has anyone used these Qlogic cards successfully under VT-d in a Xen HVM?  I don't know of any reason why they should not work.  I'm assuming (hoping) that they are virtualization-friendly enough to work under VT-d.

I think you and Lukasz are the first ones.