Re: PCI Passthrough Problems/Questions

[Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>]

On Mon, Oct 25, 2010 at 10:35:08AM -0600, Nick Couchman wrote:
> This is a message I posted to xen-users without getting any good leads...I'm hoping someone on the devel list has some ideas as to what might be going on here.
> 
> I've asked a couple of these questions, already, but haven't really had
> any response, so I thought I'd try, again, and try to provide a little
> more detail.  I need to pass through a PCIe 4-port Serial card.  I've
> done this successfully multiple times with different serial cards, but
> the current one, a PCIe Perle quad-port RJ45, is causing some problems.
> I believe the problem stems from the fact that the card is a PCI card
> and that Perle just built a PCIe-PCI bridge so that they could reuse an
> existing PCI design.  The following three PCI devices are all on the
> same card:
> 
> # lspci
> ...
> 01:00.0 PCI bridge: PLX Technology, Inc. PEX8112 x1 Lane PCI
> Express-to-PCI Bridge (rev aa)
> 02:00.0 Serial controller: Perle Systems Ltd Device 0331
> 02:00.1 Bridge: Perle Systems Ltd Device 0331
> 
> I've tried various combinations of hiding 01:00.0, 02:00.0, and/or
> 02:00.1 from dom0 and passing those through.  First, if I try all three,
> I put the following in my dom0 kernel arguments:
> 
> pciback.hide=(01:00.0)(02:00.0)(02:00.1)
> 
> Before someone suggests that maybe I have my kernel arguments wrong,
> this is a SuSE kernel, so it is not pvops, and that is the correct
> syntax - I can add other PCI(e) devices on the same exact host (like a
> graphics card) and pass those through without any problem at all.  So
> syntax is correct.  After I boot with these arguments, I check the dmesg
> output for pciback messages:
> 
> # dmesg|grep pciback
> [    0.000000] Command line: root=/dev/local/dom0 resume=/dev/local/swap
> pciback.hide=(01:00.0)(02:00.0)(02:00.1) splash=silent quiet showopts
> [    0.000000] Kernel command line: root=/dev/local/dom0
> resume=/dev/local/swap pciback.hide=(01:00.0)(02:00.0)(02:00.1)
> splash=silent quiet showopts
> [    3.646529] pciback 0000:01:00.0: seizing device
> [    3.646536] pciback 0000:02:00.0: seizing device
> [    3.646540] pciback 0000:02:00.1: seizing device
> [    3.646594] pciback 0000:02:00.1: PCI INT A -> GSI 16 (level, low) ->
> IRQ 16
> [    3.646599] pciback 0000:02:00.1: PCI INT A disabled
> [    3.646637] pciback 0000:02:00.0: PCI INT A -> GSI 16 (level, low) ->
> IRQ 16
> [    3.646642] pciback 0000:02:00.0: PCI INT A disabled
> 
> So, everything points to the devices having been removed from dom0's
> visibility and placed under the control of pciback.  A check with "xm"
> confirms this:
> 
> # xm pci-list-assignable-devices
> 0000:02:00.0
> 0000:02:00.1
> 0000:01:00.0
> 
> Next, I set up the pci pass through in my HVM domU configuration.  By
> the way, VTd is enabled and working fine - xm dmesg shows it as enabled,
> and, as already mentioned, I can forward through other PCI(e) devices
> without problem.  I set the following in my domU configuration:
> 
> pci=['01:00.0','02:00.0','02:00.1']
> 
> and then I try to start the domU:
> 
> # xm start XP
> Error: pci: PCI Backend and pci-stub don't own device 0000:01:00.0
> Usage: xm start <DomainName>
> 
> ???  Ummm, actually, the device is owned by pciback, so why is Xen
> telling me it's not??  I can confirm via lspci -v:
> 
> # lspci -v
> ...
> 01:00.0 PCI bridge: PLX Technology, Inc. PEX8112 x1 Lane PCI
> Express-to-PCI Bridge (rev aa) (prog-if 00 [Normal decode])
> 	Flags: bus master, fast devsel, latency 0
> 	Memory at d0000000 (64-bit, prefetchable) [size=64K]
> 	Bus: primary=01, secondary=02, subordinate=02, sec-latency=64
> 	I/O behind bridge: 0000d000-0000dfff
> 	Memory behind bridge: fe500000-fe5fffff
> 	Capabilities: [40] Power Management version 2
> 	Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
> 	Capabilities: [60] Express PCI/PCI-X Bridge, MSI 00
> 	Capabilities: [100] Power Budgeting <?>
> 	Kernel driver in use: pciback
> 
> Any hints on this particular part of the problem would be appreciated.
> My next step was to leave this one out and just try forwarding through
> 02:00.0 and 02:00.1.  However, when I modify the domU configuration and
> remove the 01:00.0 device and try to start the domU:
> 
> # xm start XP
> Error: Failed to assign device to IOMMU
> (0000:02:00.0@100,msitranslate=1,power_mgmt=0)
> Usage: xm start <DomainName>
> 
> I've been able to find very, very little useful information on this
> particular error - most of the issues are with trying to forward through
> one or two of the devices on a card to one domU and the others to
> another or not at all - splitting the devices on a card between domains.
> I'm not really trying to do this, but I suspect that there's some sort
> of tie between the PCIe-PCI bridge device (01:00.0) and the actual
> serial card (02:00.0) that is causing Xen to need to have both those
> devices present in the same domain.  Of course, because of the problems
> I'm having forwarding the bridge through, that's not really working for
> me.

What do you see on your Xen serial output? I presume you cranked up logging:
loglevel=all guest_lvl=all iommu=verbose on your Xen command line.

Is there anything that shows up when you get the 'Failed to assign.." ?