PCI passthrough of USB controllers on Xen 4.8.1, Linux 4.9.29, stubdomain

PCI passthrough of USB controllers on Xen 4.8.1, Linux 4.9.29, stubdomain

[Marek Marczykowski-Górecki]

[-- Attachment #1.1: Type: text/plain, Size: 4965 bytes --]

Hi,

I'm trying to passthrough USB controllers (details below) to a HVM domain
with qemu running in stubdomain.  There are two of them (EHCI and xHCI),
neither of them works. The stubdomain is non-standard Linux
based with Qemu 2.8.0. Both dom0 and guest are running Linux 4.9.29 very
close to vanilla.  I've started debugging with EHCI. On modprobe
ehci-pci I get:

    [52453.633768] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
    [52453.634496] ehci-pci: EHCI PCI platform driver
    [52453.640088] ehci-pci 0000:00:06.0: EHCI Host Controller
    [52453.640466] ehci-pci 0000:00:06.0: new USB bus registered, assigned bus number 5
    [52462.453251] ehci-pci 0000:00:06.0: can't setup: -110
    [52462.453315] ehci-pci 0000:00:06.0: USB bus 5 deregistered
    [52462.458375] ehci-pci 0000:00:06.0: init 0000:00:06.0 fail, -110
    [52462.458421] ehci-pci: probe of 0000:00:06.0 failed with error -110

And at this time, stubdomain print this:

    [00:06.0] xen_pt_bar_read: Error: Should not read BAR through QEMU. @0x0000000000000004
    [00:06.0] xen_pt_bar_read: Error: Should not read BAR through QEMU. @0x0000000000000004
    [00:06.0] xen_pt_bar_read: Error: Should not read BAR through QEMU. @0x0000000000000004
    [00:06.0] xen_pt_bar_read: Error: Should not read BAR through QEMU. @0x0000000000000004
    [00:06.0] xen_pt_bar_read: Error: Should not read BAR through QEMU. @0x0000000000000004
    [00:06.0] xen_pt_bar_read: Error: Should not read BAR through QEMU. @0x0000000000000004
    [00:06.0] xen_pt_bar_read: Error: Should not read BAR through QEMU. @0x0000000000000004
    [00:06.0] xen_pt_bar_read: Error: Should not read BAR through QEMU. @0x0000000000000004
    [00:06.0] xen_pt_bar_read: Error: Should not read BAR through QEMU. @0x0000000000000004
    [00:06.0] xen_pt_bar_read: Error: Should not read BAR through QEMU. @0x0000000000000004

repeated a lot more times, sometimes with slightly different addresses
and/or for write.

Just to be sure, I've checked with mini-os based stubdomain, but there I
don't even see the EHCI one (only xHCI, which doesn't work the same way
as on updated stubdomain).

Regarding xHCI, the domU messages are (this one is from older kernel,
but the same happens on 4.9 too):

    [  347.822393] xhci_hcd 0000:00:05.0: xHCI Host Controller
    [  347.822764] xhci_hcd 0000:00:05.0: new USB bus registered, assigned bus number 2
    [  347.825391] xhci_hcd 0000:00:05.0: hcc params 0x200077c1 hci version 0x100 quirks 0x0004b810
    [  347.832346] usb usb2: New USB device found, idVendor=1d6b, idProduct=0002
    [  347.832482] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
    [  347.832643] usb usb2: Product: xHCI Host Controller
    [  347.832779] usb usb2: Manufacturer: Linux 4.8.13-100.fc23.x86_64 xhci-hcd
    [  347.832970] usb usb2: SerialNumber: 0000:00:05.0
    [  347.833867] hub 2-0:1.0: USB hub found
    [  347.834089] hub 2-0:1.0: 11 ports detected
    [  347.835127] xhci_hcd 0000:00:05.0: xHCI Host Controller
    [  347.836927] xhci_hcd 0000:00:05.0: new USB bus registered, assigned bus number 3
    [  347.837382] usb usb3: New USB device found, idVendor=1d6b, idProduct=0003
    [  347.837582] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
    [  347.837784] usb usb3: Product: xHCI Host Controller
    [  347.837921] usb usb3: Manufacturer: Linux 4.8.13-100.fc23.x86_64 xhci-hcd
    [  347.838159] usb usb3: SerialNumber: 0000:00:05.0
    [  347.838942] hub 3-0:1.0: USB hub found
    [  347.839158] hub 3-0:1.0: 4 ports detected
    [  363.487226] xhci_hcd 0000:00:05.0: Error while assigning device slot ID
    [  363.487368] xhci_hcd 0000:00:05.0: Max number of devices this xHCI host supports is 32.
    [  363.487508] usb usb2-port1: couldn't allocate usb_device
    [  383.967268] xhci_hcd 0000:00:05.0: Error while assigning device slot ID
    [  383.967495] xhci_hcd 0000:00:05.0: Max number of devices this xHCI host supports is 32.
    [  383.967715] usb usb2-port2: couldn't allocate usb_device

And in this case, dom0 also prints:

    [   49.155606] pciback 0000:00:14.0: Driver tried to write to a read-only configuration space field at offset 0x82, size 2. This may be harmless, but if you have problems with your device:
                   1) see permissive attribute in sysfs
                   2) report problems to the xen-devel mailing list along with details of your device obtained from lspci.
    [   66.247644] pciback 0000:00:14.0: cache line size of 64 is not supported
    [   66.247646] xen_pciback: 0000:00:14.0: cannot enable memory-write-invalidate (-22)

Enabling permissive mode doesn't change anything.

Any idea what's wrong?

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: PCI passthrough of USB controllers on Xen 4.8.1, Linux 4.9.29, stubdomain

[Jan Beulich]

>>> On 02.06.17 at 12:57, <marmarek@invisiblethingslab.com> wrote:
> And in this case, dom0 also prints:
> 
>     [   49.155606] pciback 0000:00:14.0: Driver tried to write to a 
> read-only configuration space field at offset 0x82, size 2. This may be 
> harmless, but if you have problems with your device:
>                    1) see permissive attribute in sysfs
>                    2) report problems to the xen-devel mailing list along 
> with details of your device obtained from lspci.
>     [   66.247644] pciback 0000:00:14.0: cache line size of 64 is not 
> supported
>     [   66.247646] xen_pciback: 0000:00:14.0: cannot enable 
> memory-write-invalidate (-22)
> 
> Enabling permissive mode doesn't change anything.

I doubt this - the first of the messages won't be logged in permissive
mode.

We'll also need to know what register there is at address 0x82
(possibly visible from a sufficiently verbose lspci in Dom0).

As to the latter two - lspci output may also help understand
what the issue with cache line size here is. A second source of
information may be lspci output for the device with its normal
driver loaded and attached in Dom0.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: PCI passthrough of USB controllers on Xen 4.8.1, Linux 4.9.29, stubdomain

[Marek Marczykowski-Górecki]

[-- Attachment #1.1: Type: text/plain, Size: 3414 bytes --]

On Tue, Jun 06, 2017 at 02:37:01AM -0600, Jan Beulich wrote:
> >>> On 02.06.17 at 12:57, <marmarek@invisiblethingslab.com> wrote:
> > And in this case, dom0 also prints:
> > 
> >     [   49.155606] pciback 0000:00:14.0: Driver tried to write to a 
> > read-only configuration space field at offset 0x82, size 2. This may be 
> > harmless, but if you have problems with your device:
> >                    1) see permissive attribute in sysfs
> >                    2) report problems to the xen-devel mailing list along 
> > with details of your device obtained from lspci.
> >     [   66.247644] pciback 0000:00:14.0: cache line size of 64 is not 
> > supported
> >     [   66.247646] xen_pciback: 0000:00:14.0: cannot enable 
> > memory-write-invalidate (-22)
> > 
> > Enabling permissive mode doesn't change anything.
> 
> I doubt this - the first of the messages won't be logged in permissive
> mode.

Yes, of course. But the other lines and overall effect is the same.

> We'll also need to know what register there is at address 0x82
> (possibly visible from a sufficiently verbose lspci in Dom0).
> 
> As to the latter two - lspci output may also help understand
> what the issue with cache line size here is. A second source of
> information may be lspci output for the device with its normal
> driver loaded and attached in Dom0.

Below is lspci of those two devices, in dom0, with normal driver
attached. Would lspci from domU be useful too?

[root@dom0 ~]# lspci -s 00:1d.0 -vvv
00:1d.0 USB controller: Intel Corporation Wildcat Point-LP USB EHCI Controller (rev 03) (prog-if 20 [EHCI])
	Subsystem: Intel Corporation Device 7270
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin A routed to IRQ 23
	Region 0: Memory at b221a000 (32-bit, non-prefetchable) [disabled] [size=1K]
	Capabilities: [50] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [58] Debug port: BAR=1 offset=00a0
	Capabilities: [98] PCI Advanced Features
		AFCap: TP+ FLR+
		AFCtrl: FLR-
		AFStatus: TP-
	Kernel modules: ehci_pci

[root@dom0 ~]# lspci -s 00:14.0 -vvv
00:14.0 USB controller: Intel Corporation Wildcat Point-LP USB xHCI Controller (rev 03) (prog-if 30 [XHCI])
	Subsystem: Intel Corporation Device 7270
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 170
	Region 0: Memory at b2200000 (64-bit, non-prefetchable) [size=64K]
	Capabilities: [70] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0-,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+
		Address: 00000000fee00338  Data: 0000
	Kernel driver in use: xhci_hcd
	Kernel modules: xhci_pci


-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: PCI passthrough of USB controllers on Xen 4.8.1, Linux 4.9.29, stubdomain

[Jan Beulich]

>>> On 06.06.17 at 12:41, <marmarek@invisiblethingslab.com> wrote:
> [root@dom0 ~]# lspci -s 00:14.0 -vvv
> 00:14.0 USB controller: Intel Corporation Wildcat Point-LP USB xHCI 
> Controller (rev 03) (prog-if 30 [XHCI])
> 	Subsystem: Intel Corporation Device 7270
> 	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- 
> Stepping- SERR- FastB2B- DisINTx+
> 	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> 	Latency: 0
> 	Interrupt: pin A routed to IRQ 170
> 	Region 0: Memory at b2200000 (64-bit, non-prefetchable) [size=64K]
> 	Capabilities: [70] Power Management version 2
> 		Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0-,D1-,D2-,D3hot+,D3cold+)
> 		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
> 	Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+
> 		Address: 00000000fee00338  Data: 0000

So as I did expect the field accessed is the MSI capability structure.
Such accesses shouldn't reach pciback, but instead be taken care
of by qemu. I can't really comment on the qemu side though, but
this at least makes me think the underlying cause of the problems
you see with the two controllers is the same.

Is this a regression of some sort, i.e. did the same setup work in
earlier Xen versions?

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: PCI passthrough of USB controllers on Xen 4.8.1, Linux 4.9.29, stubdomain

[Marek Marczykowski-Górecki]

[-- Attachment #1.1.1: Type: text/plain, Size: 3087 bytes --]

On Tue, Jun 06, 2017 at 04:55:20AM -0600, Jan Beulich wrote:
> >>> On 06.06.17 at 12:41, <marmarek@invisiblethingslab.com> wrote:
> > [root@dom0 ~]# lspci -s 00:14.0 -vvv
> > 00:14.0 USB controller: Intel Corporation Wildcat Point-LP USB xHCI 
> > Controller (rev 03) (prog-if 30 [XHCI])
> > 	Subsystem: Intel Corporation Device 7270
> > 	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- 
> > Stepping- SERR- FastB2B- DisINTx+
> > 	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> > 	Latency: 0
> > 	Interrupt: pin A routed to IRQ 170
> > 	Region 0: Memory at b2200000 (64-bit, non-prefetchable) [size=64K]
> > 	Capabilities: [70] Power Management version 2
> > 		Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0-,D1-,D2-,D3hot+,D3cold+)
> > 		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
> > 	Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+
> > 		Address: 00000000fee00338  Data: 0000
> 
> So as I did expect the field accessed is the MSI capability structure.
> Such accesses shouldn't reach pciback, but instead be taken care
> of by qemu. I can't really comment on the qemu side though, but
> this at least makes me think the underlying cause of the problems
> you see with the two controllers is the same.
> 
> Is this a regression of some sort, i.e. did the same setup work in
> earlier Xen versions?

This was working with pure PV, but this is probably irrelevant here.

I can't test if switching only Xen version changes anything (that would
require a lot of recompiling), but very similar setup with mini-os based
stubdom on Xen 4.6.5 and Xen 4.8.1 also doesn't work. Even with a qemu
patch disabling MSI reporting (attached). In that case, I don't get any
message from qemu, but very similar messages from xhci driver.

BTW Other patches:
https://github.com/QubesOS/qubes-vmm-xen/tree/xen-4.8/patches.misc

I've also checked Xen 4.8.1 with qemu-upstream in dom0 (instead of
Linux-based stubdom) and it also doesn't work, but with slightly
different messages:

qemu:
    [00:05.0] Write-back to unknown field 0xd8 (partially) inhibited (0x00000000)
    [00:05.0] If the device doesn't work, try enabling permissive mode
    [00:05.0] (unsafe) and if it helps report the problem to xen-devel
    [00:06.0] Write-back to unknown field 0x6c (partially) inhibited (0x00000000)
    [00:06.0] If the device doesn't work, try enabling permissive mode
    [00:06.0] (unsafe) and if it helps report the problem to xen-devel

Linux in domU:

    [   51.679188] xhci_hcd 0000:00:05.0: Error while assigning device slot ID
    [   51.679264] xhci_hcd 0000:00:05.0: Max number of devices this xHCI host supports is 32.
    [   51.679359] usb usb1-port2: couldn't allocate usb_device

(and nothing about ehci, no devices connected to it is visible)


-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[-- Attachment #1.1.2: hvmpt02-disable-msi-caps.patch --]
[-- Type: text/plain, Size: 1885 bytes --]

MiniOS + QEMU do not appear to work with either MSI or MSI-X.  Some guests
or device drivers do not gracefully handle being told a PCI device has
MSI/MSI-X and then they don't actually get it.  Disable the MSI and MSI-X
capability reporting in PCI config space, making only INTX legacy interrupts
available.

Signed-off-by: Eric Shelton <eshelton@pobox.com>

--- a/tools/qemu-xen-traditional/hw/pass-through.c	2016-10-31 12:48:42.924026468 -0400
+++ b/tools/qemu-xen-traditional/hw/pass-through.c	2016-10-31 12:50:52.953026468 -0400
@@ -874,6 +874,10 @@
         .grp_size   = 0x04,
         .size_init  = pt_reg_grp_size_init,
     },
+#ifndef CONFIG_STUBDOM
+    /* At present stubdom doesn't support MSI for passthrough, so let's not
+     * expose MSI capability to stubdom HVM guest for now.
+     */
 #ifndef __ia64__
     /* At present IA64 Xen doesn't support MSI for passthrough, so let's not
      * expose MSI capability to IA64 HVM guest for now.
@@ -886,7 +890,8 @@
         .size_init  = pt_msi_size_init,
         .emu_reg_tbl= pt_emu_reg_msi_tbl,
     },
-#endif
+#endif /* __ia64__ */
+#endif /* !CONFIG_STUBDOM */
     /* PCI-X Capabilities List Item reg group */
     {
         .grp_id     = PCI_CAP_ID_PCIX,
@@ -931,6 +936,10 @@
         .size_init  = pt_pcie_size_init,
         .emu_reg_tbl= pt_emu_reg_pcie_tbl,
     },
+#ifndef CONFIG_STUBDOM
+    /* At present stubdom doesn't support MSI for passthrough, so let's not
+     * expose MSI-X capability to stubdom HVM guest for now.
+     */
 #ifndef __ia64__
     /* At present IA64 Xen doesn't support MSI for passthrough, so let's not
      * expose MSI-X capability to IA64 HVM guest for now.
@@ -943,7 +952,8 @@
         .size_init  = pt_msix_size_init,
         .emu_reg_tbl= pt_emu_reg_msix_tbl,
     },
-#endif
+#endif /* __ia64__ */
+#endif /* !CONFIG_STUBDOM */
     {
         .grp_size = 0,
     },

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel