Re: Re: [vt-d][xen4-rc6] Hangs on startup

[Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>]

> >> "0000:00:1a.1">/sys/bus/pci/devices/0000:00:1a.1/driver/unbind
> >> root@debian:~# echo "0000:00:1a.1">/sys/bus/pci/drivers/pci-stub/bind
> >
> > Where did you get the documentation for those steps? I would suggest you use
> > the 'pciback' instead of 'pci-stub' module for this just to see if that
> > makes the 'pciback' later error go away.
> 
> Well, I missread http://wiki.xensource.com/xenwiki/VTdHowTo  I
> switched to pciback(I used
> http://wiki.xensource.com/xenwiki/Assign_hardware_to_DomU_with_PCIBack_as_module)
>  and this error no longer occurs.

.. snip ..
> >> (XEN) mm.c:859:d1 Error getting mfn 40004 (pfn 5555555555555555) from
> >> L1 entry 0000000040004037 for l1e_owner=1, pg_owner=1
> >> ...
> >> And guest never starts.  Perhaps this is related to the firmware which
> >> is used by qla2xxx?
> >
> > So you see this _only_ when you pass in QLA2XXX card, right? When you
> > don't pass in a PCI card, you don't see this, correct?
> 
> Exactly. This happens with both pciback and pci-stub.

You are hitting two bugs.

1). pciback doesn't release the spinlock properly, spinning forever and
causing havoc when pcistub is used for the device.

2). qemu/vt-d/bochs bios/hvmload isn't setting up the HVM PCI passthrough
properly for your QLogic card.

For the 1) there isn't much there except to find the culprit and fix it,
should be pretty easy - probably some mutex/dead-lock condition. But
there is a work-around which you found, and that is to use _only_ pciback for
HVM and PV guests.

For the 2) problem I've no idea. I haven't dug through the QEMU PCI
passthrough code that much to figure this out. Depending on your needs
we can fix this the easy or the hard way. The easy way is for you to try
PV pass-through instead of HVM pass-through. It requires taking your
kernel and initramfs image from your DomU, and referencing them direclty
in your .cfg file. There should be an example of this in /etc/xen (look
for pv-grub). Also you might need to pass in 'iommu=0' on your Xen command line. 

The hard way is to try to figure this one out. There are four pieces of
technology working here: IOMMU (VT-d chipset), the QEMU
PCI pass-through code, the hvmloader code, and the Bochs BIOS. First step is
to instrument your QEMU to print all manners of debug information and do the same with
the hvmloader (grep for 'debug').  Also adding in 'loglvl=all' on your Xen
command line is a good idea (and compiling Xen with 'debug=y'). For the 
Bochs BIOS, there are some options in it where you can tweak it to print
more details.

Oh, and also would need to instrument DomU to figure out when that
'Error getting mfn' is triggered. You can do that easily by using
'xenctx' and seeing the stack trace of the guest (And with your
System.map table pin-point what is happening).