Re: Issue with PCI-passthrough and pvops

[Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>]

On Wed, Oct 19, 2011 at 01:54:07AM +0200, Dario Faggioli wrote:
> On Mon, 2011-10-17 at 12:40 -0400, Konrad Rzeszutek Wilk wrote:
> > > Here's the thing:
> > > --
> > > # xl pci-list-assignable-devices 
> > > 0000:07:00.0
> > > 0000:07:00.1
> > > 
> > > # cat xen/VMs/Debian-squeeze.pv | grep pci=
> > > #   pci=[ '[SSSS:]BB:DD.F[,option1[,option2[...]]]', ... ]
> > > pci=[ '07:00.0' ]
> > > 
> > > # xl list
> > > Name                                        ID   Mem VCPUs	State	Time(s)
> > > Domain-0                                     0   750    16     r-----    2205.3
> > > Debian-squeeze_pv                            3   128     2     ---sc-      19.8
> > > --
> > > 
> > 
> > Do you have 'iommu=soft' in your guest config?
> >
> I do... BTW, it turned out that was an out-of-memory issue, which went
> away after increasing VM's memory (although the old amount of RAM was
> enough without PCI-passthrough and still is for HVM, but anyway...)
> 
> Now I have a pv-guest that boots but here's what the host and the guest
> are saying.
> 
> # xl list
> Name                                        ID   Mem VCPUs	State	Time(s)
> Domain-0                                     0   750    16     r-----   45807.9
> Debian-squeeze_pv                            1   512     2     r-----    2116.6
> 
> Host:
> [42515.533157] pciback 0000:07:00.0: restoring config space at offset 0xf (was 0x100, writing 0x10f)
> [42515.533194] pciback 0000:07:00.0: restoring config space at offset 0x8 (was 0xc, writing 0xd58f800c)
> [42515.533209] pciback 0000:07:00.0: restoring config space at offset 0x6 (was 0x1, writing 0xecc1)
> [42515.533224] pciback 0000:07:00.0: restoring config space at offset 0x4 (was 0xc, writing 0xd590000c)
> [42515.533290] pciback 0000:07:00.0: BAR 7: set to [mem 0xdf200000-0xdf2fffff 64bit] (PCI address [0xdf200000-0xdf2fffff])
> [42515.533302] pciback 0000:07:00.0: BAR 10: set to [mem 0xdf300000-0xdf3fffff 64bit] (PCI address [0xdf300000-0xdf3fffff])
> (XEN) [VT-D]iommu.c:1543: d0:PCIe: unmap 0000:07:00.0
> (XEN) [VT-D]iommu.c:1412: d1:PCIe: map 0000:07:00.0
> [42515.556555] xen-pciback: vpci: 0000:07:00.0: assign to virtual slot 0
> mapping kernel into physical memory
> about to get started...
> [42526.391448] pciback 0000:07:00.0: Driver tried to write to a read-only configuration space field at offset 0x168, size 2. This may be harmless, but if you have problems with your device:
> [42526.391450] 1) see permissive attribute in sysfs
> [42526.391451] 2) report problems to the xen-devel mailing list along with details of your device obtained from lspci.
> 
> Guest:
> [   19.607997] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 3.4.8-k
> [   19.608670] ixgbe: Copyright (c) 1999-2011 Intel Corporation.
> [   19.609465] ixgbe 0000:00:00.0: device not available (can't reserve [mem 0xdf300000-0xe32fffff 64bit])
> [   19.610878] ixgbe: probe of 0000:00:00.0 failed with error -22

Well, that is the problem.

> [   19.611764] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver - version 2.1.0-k
> [   19.612656] Copyright (c) 2009 - 2010 Intel Corporation.
> [   19.614144] ixgb: Intel(R) PRO/10GbE Network Driver - version 1.0.135-k2-NAPI
> [   19.614865] ixgb: Copyright (c) 1999-2008 Intel Corporation.
> 
> While in the guest, I can see the NIC with `lspci' but I can't bring it
> up. Also, trying to check in /sys/bus/pci/..., the device does not seem
> to be claimed by anyone (no driver file present).
> 
> Moreover, when trying to kill the domain, the following happens:
> # xl destroy 1
> libxl: error: libxl_pci.c:925:do_pci_remove: xc_physdev_unmap_pirq irq=40
> Aborted

Ugh, that looks like a bug.