Issue with PCI-passthrough and pvops

Issue with PCI-passthrough and pvops

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 1262 bytes --]

Hi everyone,

I'm trying to setup PCI-passthrough for a network card on a testbox.
With HVM, everything seems to work, while if I try with a pv-guest the
domain crashes!

Here's the thing:
--
# xl pci-list-assignable-devices 
0000:07:00.0
0000:07:00.1

# cat xen/VMs/Debian-squeeze.pv | grep pci=
#   pci=[ '[SSSS:]BB:DD.F[,option1[,option2[...]]]', ... ]
pci=[ '07:00.0' ]

# xl create xen/VMs/Debian-squeeze.pv
Parsing config file xen/VMs/Debian-squeeze.pv
Daemon running with PID 5588

# xl list
Name                                        ID   Mem VCPUs	State	Time(s)
Domain-0                                     0   750    16     r-----    2205.3
Debian-squeeze_pv                            3   128     2     ---sc-      19.8
--

Again, the very same config and PCI device for an HVM domain starts, and
I can see and bring up the NIC.

What am I doing wrong?

Thanks and Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
----------------------------------------------------------------------
Dario Faggioli, http://retis.sssup.it/people/faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
PhD Candidate, ReTiS Lab, Scuola Superiore Sant'Anna, Pisa  (Italy)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Issue with PCI-passthrough and pvops

[Konrad Rzeszutek Wilk]

On Mon, Oct 17, 2011 at 05:36:27PM +0200, Dario Faggioli wrote:
> Hi everyone,
> 
> I'm trying to setup PCI-passthrough for a network card on a testbox.
> With HVM, everything seems to work, while if I try with a pv-guest the
> domain crashes!
> 
> Here's the thing:
> --
> # xl pci-list-assignable-devices 
> 0000:07:00.0
> 0000:07:00.1
> 
> # cat xen/VMs/Debian-squeeze.pv | grep pci=
> #   pci=[ '[SSSS:]BB:DD.F[,option1[,option2[...]]]', ... ]
> pci=[ '07:00.0' ]
> 
> # xl create xen/VMs/Debian-squeeze.pv
> Parsing config file xen/VMs/Debian-squeeze.pv
> Daemon running with PID 5588
> 
> # xl list
> Name                                        ID   Mem VCPUs	State	Time(s)
> Domain-0                                     0   750    16     r-----    2205.3
> Debian-squeeze_pv                            3   128     2     ---sc-      19.8
> --
> 
> Again, the very same config and PCI device for an HVM domain starts, and
> I can see and bring up the NIC.
> 
> What am I doing wrong?

Do you have 'iommu=soft' in your guest config?

Re: Issue with PCI-passthrough and pvops

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 4064 bytes --]

On Mon, 2011-10-17 at 12:40 -0400, Konrad Rzeszutek Wilk wrote:
> > Here's the thing:
> > --
> > # xl pci-list-assignable-devices 
> > 0000:07:00.0
> > 0000:07:00.1
> > 
> > # cat xen/VMs/Debian-squeeze.pv | grep pci=
> > #   pci=[ '[SSSS:]BB:DD.F[,option1[,option2[...]]]', ... ]
> > pci=[ '07:00.0' ]
> > 
> > # xl list
> > Name                                        ID   Mem VCPUs	State	Time(s)
> > Domain-0                                     0   750    16     r-----    2205.3
> > Debian-squeeze_pv                            3   128     2     ---sc-      19.8
> > --
> > 
> 
> Do you have 'iommu=soft' in your guest config?
>
I do... BTW, it turned out that was an out-of-memory issue, which went
away after increasing VM's memory (although the old amount of RAM was
enough without PCI-passthrough and still is for HVM, but anyway...)

Now I have a pv-guest that boots but here's what the host and the guest
are saying.

# xl list
Name                                        ID   Mem VCPUs	State	Time(s)
Domain-0                                     0   750    16     r-----   45807.9
Debian-squeeze_pv                            1   512     2     r-----    2116.6

Host:
[42515.533157] pciback 0000:07:00.0: restoring config space at offset 0xf (was 0x100, writing 0x10f)
[42515.533194] pciback 0000:07:00.0: restoring config space at offset 0x8 (was 0xc, writing 0xd58f800c)
[42515.533209] pciback 0000:07:00.0: restoring config space at offset 0x6 (was 0x1, writing 0xecc1)
[42515.533224] pciback 0000:07:00.0: restoring config space at offset 0x4 (was 0xc, writing 0xd590000c)
[42515.533290] pciback 0000:07:00.0: BAR 7: set to [mem 0xdf200000-0xdf2fffff 64bit] (PCI address [0xdf200000-0xdf2fffff])
[42515.533302] pciback 0000:07:00.0: BAR 10: set to [mem 0xdf300000-0xdf3fffff 64bit] (PCI address [0xdf300000-0xdf3fffff])
(XEN) [VT-D]iommu.c:1543: d0:PCIe: unmap 0000:07:00.0
(XEN) [VT-D]iommu.c:1412: d1:PCIe: map 0000:07:00.0
[42515.556555] xen-pciback: vpci: 0000:07:00.0: assign to virtual slot 0
mapping kernel into physical memory
about to get started...
[42526.391448] pciback 0000:07:00.0: Driver tried to write to a read-only configuration space field at offset 0x168, size 2. This may be harmless, but if you have problems with your device:
[42526.391450] 1) see permissive attribute in sysfs
[42526.391451] 2) report problems to the xen-devel mailing list along with details of your device obtained from lspci.

Guest:
[   19.607997] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 3.4.8-k
[   19.608670] ixgbe: Copyright (c) 1999-2011 Intel Corporation.
[   19.609465] ixgbe 0000:00:00.0: device not available (can't reserve [mem 0xdf300000-0xe32fffff 64bit])
[   19.610878] ixgbe: probe of 0000:00:00.0 failed with error -22
[   19.611764] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver - version 2.1.0-k
[   19.612656] Copyright (c) 2009 - 2010 Intel Corporation.
[   19.614144] ixgb: Intel(R) PRO/10GbE Network Driver - version 1.0.135-k2-NAPI
[   19.614865] ixgb: Copyright (c) 1999-2008 Intel Corporation.

While in the guest, I can see the NIC with `lspci' but I can't bring it
up. Also, trying to check in /sys/bus/pci/..., the device does not seem
to be claimed by anyone (no driver file present).

Moreover, when trying to kill the domain, the following happens:
# xl destroy 1
libxl: error: libxl_pci.c:925:do_pci_remove: xc_physdev_unmap_pirq irq=40
Aborted
root@odyn:~# xl list
Name                                        ID   Mem VCPUs	State	Time(s)
Domain-0                                     0   750    16     r-----    1042.4
Debian-squeeze_pv

Thanks and Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
----------------------------------------------------------------------
Dario Faggioli, http://retis.sssup.it/people/faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
PhD Candidate, ReTiS Lab, Scuola Superiore Sant'Anna, Pisa  (Italy)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Issue with PCI-passthrough and pvops

[Konrad Rzeszutek Wilk]

On Wed, Oct 19, 2011 at 01:54:07AM +0200, Dario Faggioli wrote:
> On Mon, 2011-10-17 at 12:40 -0400, Konrad Rzeszutek Wilk wrote:
> > > Here's the thing:
> > > --
> > > # xl pci-list-assignable-devices 
> > > 0000:07:00.0
> > > 0000:07:00.1
> > > 
> > > # cat xen/VMs/Debian-squeeze.pv | grep pci=
> > > #   pci=[ '[SSSS:]BB:DD.F[,option1[,option2[...]]]', ... ]
> > > pci=[ '07:00.0' ]
> > > 
> > > # xl list
> > > Name                                        ID   Mem VCPUs	State	Time(s)
> > > Domain-0                                     0   750    16     r-----    2205.3
> > > Debian-squeeze_pv                            3   128     2     ---sc-      19.8
> > > --
> > > 
> > 
> > Do you have 'iommu=soft' in your guest config?
> >
> I do... BTW, it turned out that was an out-of-memory issue, which went
> away after increasing VM's memory (although the old amount of RAM was
> enough without PCI-passthrough and still is for HVM, but anyway...)
> 
> Now I have a pv-guest that boots but here's what the host and the guest
> are saying.
> 
> # xl list
> Name                                        ID   Mem VCPUs	State	Time(s)
> Domain-0                                     0   750    16     r-----   45807.9
> Debian-squeeze_pv                            1   512     2     r-----    2116.6
> 
> Host:
> [42515.533157] pciback 0000:07:00.0: restoring config space at offset 0xf (was 0x100, writing 0x10f)
> [42515.533194] pciback 0000:07:00.0: restoring config space at offset 0x8 (was 0xc, writing 0xd58f800c)
> [42515.533209] pciback 0000:07:00.0: restoring config space at offset 0x6 (was 0x1, writing 0xecc1)
> [42515.533224] pciback 0000:07:00.0: restoring config space at offset 0x4 (was 0xc, writing 0xd590000c)
> [42515.533290] pciback 0000:07:00.0: BAR 7: set to [mem 0xdf200000-0xdf2fffff 64bit] (PCI address [0xdf200000-0xdf2fffff])
> [42515.533302] pciback 0000:07:00.0: BAR 10: set to [mem 0xdf300000-0xdf3fffff 64bit] (PCI address [0xdf300000-0xdf3fffff])
> (XEN) [VT-D]iommu.c:1543: d0:PCIe: unmap 0000:07:00.0
> (XEN) [VT-D]iommu.c:1412: d1:PCIe: map 0000:07:00.0
> [42515.556555] xen-pciback: vpci: 0000:07:00.0: assign to virtual slot 0
> mapping kernel into physical memory
> about to get started...
> [42526.391448] pciback 0000:07:00.0: Driver tried to write to a read-only configuration space field at offset 0x168, size 2. This may be harmless, but if you have problems with your device:
> [42526.391450] 1) see permissive attribute in sysfs
> [42526.391451] 2) report problems to the xen-devel mailing list along with details of your device obtained from lspci.
> 
> Guest:
> [   19.607997] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 3.4.8-k
> [   19.608670] ixgbe: Copyright (c) 1999-2011 Intel Corporation.
> [   19.609465] ixgbe 0000:00:00.0: device not available (can't reserve [mem 0xdf300000-0xe32fffff 64bit])
> [   19.610878] ixgbe: probe of 0000:00:00.0 failed with error -22

Well, that is the problem.

> [   19.611764] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver - version 2.1.0-k
> [   19.612656] Copyright (c) 2009 - 2010 Intel Corporation.
> [   19.614144] ixgb: Intel(R) PRO/10GbE Network Driver - version 1.0.135-k2-NAPI
> [   19.614865] ixgb: Copyright (c) 1999-2008 Intel Corporation.
> 
> While in the guest, I can see the NIC with `lspci' but I can't bring it
> up. Also, trying to check in /sys/bus/pci/..., the device does not seem
> to be claimed by anyone (no driver file present).
> 
> Moreover, when trying to kill the domain, the following happens:
> # xl destroy 1
> libxl: error: libxl_pci.c:925:do_pci_remove: xc_physdev_unmap_pirq irq=40
> Aborted

Ugh, that looks like a bug.

Re: Issue with PCI-passthrough and pvops

[Ian Campbell]

On Wed, 2011-10-19 at 02:10 +0100, Konrad Rzeszutek Wilk wrote:
> On Wed, Oct 19, 2011 at 01:54:07AM +0200, Dario Faggioli wrote:
> > Guest:
> > [   19.607997] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 3.4.8-k
> > [   19.608670] ixgbe: Copyright (c) 1999-2011 Intel Corporation.
> > [   19.609465] ixgbe 0000:00:00.0: device not available (can't reserve [mem 0xdf300000-0xe32fffff 64bit])
> > [   19.610878] ixgbe: probe of 0000:00:00.0 failed with error -22
> 
> Well, that is the problem.

Does he need "e820_host=1" in his cfg?

Re: Issue with PCI-passthrough and pvops

[Dario Faggioli]

On Wed, Oct 19, 2011 at 9:40 AM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
>> > Guest:
>> > [   19.607997] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 3.4.8-k
>> > [   19.608670] ixgbe: Copyright (c) 1999-2011 Intel Corporation.
>> > [   19.609465] ixgbe 0000:00:00.0: device not available (can't reserve [mem 0xdf300000-0xe32fffff 64bit])
>> > [   19.610878] ixgbe: probe of 0000:00:00.0 failed with error -22
>>
>> Well, that is the problem.
>
> Does he need "e820_host=1" in his cfg?
>
Mmm... If you mean putting that line in my DomU config file
(I checked 23428:131f19c67d85, and it seems so), that is not
helping.

Thanks and regards,
Dario

PS. sorry for the webmail... Installation of new laptop going on here :-P

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
----------------------------------------------------------------------
Dario Faggioli, <http://retis.sssup.it/people/faggioli>
PhD Candidate, ReTiS Lab, Scuola Superiore Sant'Anna, Pisa  (Italy)
Senior Software Engineer, Citrix Systems R&D, Cambridge (UK)

Re: Issue with PCI-passthrough and pvops

[Konrad Rzeszutek Wilk]

On Wed, Oct 19, 2011 at 03:46:26PM +0200, Dario Faggioli wrote:
> On Wed, Oct 19, 2011 at 9:40 AM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> >> > Guest:
> >> > [   19.607997] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 3.4.8-k
> >> > [   19.608670] ixgbe: Copyright (c) 1999-2011 Intel Corporation.
> >> > [   19.609465] ixgbe 0000:00:00.0: device not available (can't reserve [mem 0xdf300000-0xe32fffff 64bit])
> >> > [   19.610878] ixgbe: probe of 0000:00:00.0 failed with error -22
> >>
> >> Well, that is the problem.
> >
> > Does he need "e820_host=1" in his cfg?
> >
> Mmm... If you mean putting that line in my DomU config file
> (I checked 23428:131f19c67d85, and it seems so), that is not
> helping.

That option is usually required if the guest has more than 2GB. As we
would end up setting an E820 that would trample over the PCI hole.
(In this case it looks like part of the PCI hole is DF300000).

Dario,

I've seen this bug before with .. Hm, some similar adapter. I know
that if pass in 'igb.max_vfs=2' and passed in the igbvf PCI cards the
guest worked just fine. It just did not like being passed in as a real device.

Otherwise, older MSI/MSI-X (non SR-IOV) cards worked fine so I never go
further in debugging this.

I would recommend you take a look at the probe function and figure out why
it can't reserve that region. And easy way to figure that out is to
boot the guest and look in /proc/iomem and see what is in the df30000-e32ffffff
region. Perhaps something else is overlapping it?

Re: Issue with PCI-passthrough and pvops

[Dario Faggioli]

On Wed, Oct 19, 2011 at 4:10 PM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:
>> >> > Guest:
>> >> > [   19.607997] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 3.4.8-k
>> >> > [   19.608670] ixgbe: Copyright (c) 1999-2011 Intel Corporation.
>> >> > [   19.609465] ixgbe 0000:00:00.0: device not available (can't reserve [mem 0xdf300000-0xe32fffff 64bit])
>> >> > [   19.610878] ixgbe: probe of 0000:00:00.0 failed with error -22
>
> Dario,
>
> I've seen this bug before with .. Hm, some similar adapter. I know
> that if pass in 'igb.max_vfs=2' and passed in the igbvf PCI cards the
> guest worked just fine. It just did not like being passed in as a real device.
>
> Otherwise, older MSI/MSI-X (non SR-IOV) cards worked fine so I never go
> further in debugging this.
>
Yeah, in fact passing other cards the box have to the very same domain
seems to just work.

> I would recommend you take a look at the probe function and figure out why
> it can't reserve that region. And easy way to figure that out is to
> boot the guest and look in /proc/iomem and see what is in the df30000-e32ffffff
> region. Perhaps something else is overlapping it?
>
I might be wrong, but it doesn't seem quite so to me:

# cat /proc/iomem
00000000-0000ffff : reserved
00010000-0009ffff : System RAM
000a0000-000fffff : reserved
  000f0000-000fffff : System ROM
00100000-1fffffff : System RAM
  01000000-01b88b4b : Kernel code
  01b88b4c-02352a7f : Kernel data
  02652000-02ee6fff : Kernel bss
20000000-bf698fff : Unusable memory
bf6af000-bf6cdfff : ACPI Tables
d58f8000-d58fbfff : 0000:00:00.0
d5900000-d597ffff : 0000:00:00.0
df200000-e71fffff : 0000:00:00.0
100000000-1000fffff : System RAM

The probe function is huge... But I'll se if I can find the time to take
a look at it in the next days...

Thanks a lot for now. :-)

Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
----------------------------------------------------------------------
Dario Faggioli, <http://retis.sssup.it/people/faggioli>
Senior Software Engineer, Citrix Systems R&D, Cambridge (UK)
PhD Candidate, ReTiS Lab, Scuola Superiore Sant'Anna, Pisa  (Italy)

Re: Issue with PCI-passthrough and pvops

[Konrad Rzeszutek Wilk]

> > I would recommend you take a look at the probe function and figure out why
> > it can't reserve that region. And easy way to figure that out is to
> > boot the guest and look in /proc/iomem and see what is in the df30000-e32ffffff
> > region. Perhaps something else is overlapping it?
> >
> I might be wrong, but it doesn't seem quite so to me:
> 
> # cat /proc/iomem
> 00000000-0000ffff : reserved
> 00010000-0009ffff : System RAM
> 000a0000-000fffff : reserved
>   000f0000-000fffff : System ROM
> 00100000-1fffffff : System RAM
>   01000000-01b88b4b : Kernel code
>   01b88b4c-02352a7f : Kernel data
>   02652000-02ee6fff : Kernel bss
> 20000000-bf698fff : Unusable memory
> bf6af000-bf6cdfff : ACPI Tables
> d58f8000-d58fbfff : 0000:00:00.0
> d5900000-d597ffff : 0000:00:00.0
> df200000-e71fffff : 0000:00:00.0
> 100000000-1000fffff : System RAM
> 
> The probe function is huge... But I'll se if I can find the time to take
> a look at it in the next days...

Hmmm. Instrumenting resources.c (__request_resource) might be the way to
figure out where it chokes on.

Re: Issue with PCI-passthrough and pvops

[Ian Campbell]

On Wed, 2011-10-19 at 00:54 +0100, Dario Faggioli wrote:
> when trying to kill the domain, the following happens:
> # xl destroy 1
> libxl: error: libxl_pci.c:925:do_pci_remove: xc_physdev_unmap_pirq irq=40
> Aborted

You most probably need 'libxl: add missing "break;" to do_pci_remove'
which I posted v3 of yesterday.

If not then "gdb xl destroy" and "bt" would be useful.

Ian.

Re: Issue with PCI-passthrough and pvops

[Dario Faggioli]

On Wed, Oct 19, 2011 at 9:39 AM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
>> # xl destroy 1
>> libxl: error: libxl_pci.c:925:do_pci_remove: xc_physdev_unmap_pirq irq=40
>> Aborted
>
> You most probably need 'libxl: add missing "break;" to do_pci_remove'
> which I posted v3 of yesterday.
>
Yep, error is gone, thanks!

Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
----------------------------------------------------------------------
Dario Faggioli, <http://retis.sssup.it/people/faggioli>
PhD Candidate, ReTiS Lab, Scuola Superiore Sant'Anna, Pisa  (Italy)
Senior Software Engineer, Citrix Systems R&D, Cambridge (UK)