From mboxrd@z Thu Jan 1 00:00:00 1970 From: Konrad Rzeszutek Wilk Subject: PCI passthrough (pci-attach) to HVM guests bug (BAR64 addresses are bogus) Date: Mon, 10 Nov 2014 12:32:48 -0500 Message-ID: <20141110173248.GA13778@laptop.dumpdata.com> Mime-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Return-path: Received: from mail6.bemta3.messagelabs.com ([195.245.230.39]) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1Xnspk-0007eP-5y for xen-devel@lists.xenproject.org; Mon, 10 Nov 2014 17:32:56 +0000 Content-Disposition: inline List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: xen-devel@lists.xenproject.org, jbeulich@suse.com List-Id: xen-devel@lists.xenproject.org Hey, With Xen 4.5 (today's staging), when I boot a guest and then do pci-attach the BARs values are corrupt. For example, with this guest config: kernel=3D"hvmloader" builder=3D"hvm" serial=3D"pty" memory =3D 2048 name =3D "XTT" usb=3D1 usbdevice=3D'tablet' vcpus=3D2 vga=3D"stdvga" vif =3D [ 'mac=3D00:0f:4b:00:00:63,bridge=3Dxenbr0' ] disk=3D ['file:/root/root_image.iso,hdc:cdrom,r'] vnc=3D1 vnclisten=3D"0.0.0.0" boot =3D "dc" And with this PCI card: 5:00.0 VGA compatible controller: NVIDIA Corporation GF104GLM [Quadro 4000M= ] (rev a1) (prog-if 00 [VGA controller]) =A0=A0=A0=A0=A0=A0=A0Subsystem: Gigabyte Technology Co., Ltd Device 34fc =A0=A0=A0=A0=A0=A0=A0Flags: fast devsel, IRQ 20 =A0=A0=A0=A0=A0=A0=A0Memory at fc000000 (32-bit, non-prefetchable) [size= =3D32M] =A0=A0=A0=A0=A0=A0=A0Memory at d0000000 (64-bit, prefetchable) [size=3D128= M] =A0=A0=A0=A0=A0=A0=A0Memory at d8000000 (64-bit, prefetchable) [size=3D64M] which in dom0 is reported as: 4064] pci 0000:05:00.0: [10de:0e3b] type 00 class 0x030000 [ =A0=A022.834087] pci 0000:05:00.0: reg 0x10: [mem 0xfc000000-0xfdffffff] [ =A0=A022.834109] pci 0000:05:00.0: reg 0x14: [mem 0xd0000000-0xd7ffffff 6= 4bit pref] [ =A0=A022.834131] pci 0000:05:00.0: reg 0x1c: [mem 0xd8000000-0xdbffffff 6= 4bit pref] [ =A0=A022.834144] pci 0000:05:00.0: reg 0x24: [io =A00xc000-0xc07f] [ =A0=A022.834157] pci 0000:05:00.0: reg 0x30: [mem 0xfe000000-0xfe07ffff p= ref] When I assign said card (xl pci-attach XTT 05:00.0) the guest gives me: ... 498525] pci 0000:00:04.0: [10de:0e3b] type 00 class 0x030000 [ 152.508612] pci 0000:00:04.0: reg 0x10: [mem 0x00000000-0x01ffffff] [ 152.518320] pci 0000:00:04.0: reg 0x14: [mem 0x00000000-0x07ffffff 64bit= preff ] [ 152.529301] pci 0000:00:04.0: reg 0x1c: [mem 0x00000000-0x03ffffff 64bit= preff ] [ 152.540095] pci 0000:00:04.0: reg 0x24: [io 0x0000-0x007f] [ 152.548497] pci 0000:00:04.0: reg 0x30: [mem 0x00000000-0x0007ffff pref] [ 152.561018] vgaarb: device added: PCI:0000:00:04.0,decodes=3Dio+mem,owns= =3Dnone,llocks=3Dnone [ 152.572965] pci 0000:00:04.0: BAR 1: no space for [mem size 0x08000000 6= 4bit pref] [ 152.583917] pci 0000:00:04.0: BAR 1: failed to assign [mem size 0x080000= 00 64 bit pref] [ 152.595528] pci 0000:00:04.0: BAR 3: assigned [mem 0xf4000000-0xf7ffffff= 64bi t pref] If I boot the guest with: pci=3D["05:00.0"] it works fine: # dmesg | grep 05.0 [ 0.000000] pcpu-alloc: [0] 000 001 002 003 004 005 006 007 008 009 010 = 011 012 013 014 015 = [ 0.905008] pci 0000:00:03.0: reg 0x10: [mem 0xef000000-0xefffffff pref] [ 0.944101] pci 0000:00:05.0: [10de:0e3b] type 00 class 0x030000 [ 0.953013] pci 0000:00:05.0: reg 0x10: [mem 0xec000000-0xedffffff] [ 0.967016] pci 0000:00:05.0: reg 0x14: [mem 0xe0000000-0xe7ffffff 64bit= pref] [ 0.973016] pci 0000:00:05.0: reg 0x1c: [mem 0xe8000000-0xebffffff 64bit= pref] [ 0.981015] pci 0000:00:05.0: reg 0x24: [io 0xc200-0xc27f] [ 0.988016] pci 0000:00:05.0: reg 0x30: [mem 0xf0000000-0xf007ffff pref] [ 0.995083] vgaarb: device added: PCI:0000:00:05.0,decodes=3Dio+mem,owns= =3Dio+mem,locks=3Dnone [ 0.997000] vgaarb: bridge control possible 0000:00:05.0 [ 3.952023] nouveau [ DEVICE][0000:00:05.0] BOOT0 : 0x0c4d80a1 [ 3.952025] nouveau [ DEVICE][0000:00:05.0] Chipset: GF104 (NVC4) [ 3.952027] nouveau [ DEVICE][0000:00:05.0] Family : NVC0 [ 3.952072] nouveau [ VBIOS][0000:00:05.0] checking PRAMIN for image.= .. [ 3.952079] nouveau [ VBIOS][0000:00:05.0] ... signature not found [ 3.952080] nouveau [ VBIOS][0000:00:05.0] checking PROM for image... [ 4.096446] nouveau [ VBIOS][0000:00:05.0] ... appears to be valid [ 4.104012] nouveau [ VBIOS][0000:00:05.0] using image from PROM [ 4.111214] nouveau [ VBIOS][0000:00:05.0] BIT signature found [ 4.118054] nouveau [ VBIOS][0000:00:05.0] version 70.04.13.00.01 [ 4.125248] nouveau [ DEVINIT][0000:00:05.0] adaptor not initialised [ 4.131629] nouveau [ VBIOS][0000:00:05.0] running init tables [ 4.227827] nouveau [ PMC][0000:00:05.0] MSI interrupts enabled [ 4.234296] nouveau [ PFB][0000:00:05.0] RAM type: GDDR5 [ 4.240176] nouveau [ PFB][0000:00:05.0] RAM size: 1024 MiB [ 4.245878] nouveau [ PFB][0000:00:05.0] ZCOMP: 0 tags [ 4.255523] nouveau [ VOLT][0000:00:05.0] GPU voltage: 875000uv [ 4.291146] nouveau [ PTHERM][0000:00:05.0] FAN control: PWM [ 4.296582] nouveau [ PTHERM][0000:00:05.0] fan management: automatic [ 4.302668] nouveau [ PTHERM][0000:00:05.0] internal sensor: yes [ 4.309465] nouveau [ CLK][0000:00:05.0] 03: core 50 MHz memory 135= MHz = [ 4.317854] nouveau [ CLK][0000:00:05.0] 07: core 405 MHz memory 32= 4 MHz = [ 4.326655] nouveau [ CLK][0000:00:05.0] 0c: core 405 MHz memory 18= 00 MHz = [ 4.333742] nouveau [ CLK][0000:00:05.0] 0f: core 715 MHz memory 18= 00 MHz = [ 4.341362] nouveau [ CLK][0000:00:05.0] --: core 50 MHz memory 135= MHz = [ 4.749977] nouveau 0000:00:05.0: fb0: nouveaufb frame buffer device [ 4.756172] nouveau 0000:00:05.0: registered panic notifier [ 4.765041] [drm] Initialized nouveau 1.2.0 20120801 for 0000:00:05.0 on= minor 0 # lspci -s 00:05.0 -v 00:05.0 VGA compatible controller: nVidia Corporation Device 0e3b (rev a1) = (prog-if 00 [VGA controller]) Subsystem: Giga-byte Technology Device 34fc Physical Slot: 5 Flags: bus master, fast devsel, latency 0, IRQ 67 Memory at ec000000 (32-bit, non-prefetchable) [size=3D32M] Memory at e0000000 (64-bit, prefetchable) [size=3D128M] Memory at e8000000 (64-bit, prefetchable) [size=3D64M] I/O ports at c200 [size=3D128] Expansion ROM at f0000000 [disabled] [size=3D512K] Interesting observation: a) It does NOT make a different if I use qemu-traditinal or qemu-xen. In both cases I get the same BAR bogus numbers. b) qemu-xen logging is not that great. When I used qemu-trad I discovered: m-command: hot insert pass-through pci dev register_real_device: Assigning real physical device 05:00.0 ... register_real_device: Disable MSI translation via per device option register_real_device: Disable power management pt_iomul_init: Error: pt_iomul_init can't open file /dev/xen/pci_iomul: No = such file or directory: 0x5:0x0.0x0 pt_register_regions: IO region registered (size=3D0x02000000 base_addr=3D0x= fc000000) pt_register_regions: IO region registered (size=3D0x08000000 base_addr=3D0x= d000000c) pt_register_regions: IO region registered (size=3D0x04000000 base_addr=3D0x= d800000c) pt_register_regions: IO region registered (size=3D0x00000080 base_addr=3D0x= 0000c001) pt_register_regions: Expansion ROM registered (size=3D0x00080000 base_addr= =3D0xfe000000) pci_intx: intx=3D1 register_real_device: Real physical device 05:00.0 registered successfuly! IRQ type =3D INTx generate a sci for PHP. deassert due to disable GPE bit. ACPI:debug: write addr=3D0xb044, val=3D0x20. ACPI:debug: write addr=3D0xb045, val=3D0x0. ACPI:debug: write addr=3D0xb044, val=3D0x20. ACPI:debug: write addr=3D0xb045, val=3D0x89. ACPI:debug: write addr=3D0xb044, val=3D0x21. ACPI:debug: write addr=3D0xb045, val=3D0x89. ACPI:debug: write addr=3D0xb044, val=3D0x22. ACPI:debug: write addr=3D0xb045, val=3D0x89. ACPI:debug: write addr=3D0xb044, val=3D0x23. ACPI:debug: write addr=3D0xb045, val=3D0x89. ACPI:debug: write addr=3D0xb044, val=3D0x24. ACPI:debug: write addr=3D0xb045, val=3D0x89. ACPI:debug: write addr=3D0xb044, val=3D0x25. ACPI:debug: write addr=3D0xb045, val=3D0x89. ACPI:debug: write addr=3D0xb044, val=3D0x26. ACPI:debug: write addr=3D0xb045, val=3D0x89. ACPI:debug: write addr=3D0xb044, val=3D0x27. ACPI:debug: write addr=3D0xb045, val=3D0x89. pt_iomem_map: e_phys=3Df2000000 maddr=3Dfc000000 type=3D0 len=3D33554432 in= dex=3D0 first_map=3D1 pt_iomem_map: e_phys=3Df4000000 maddr=3Dd8000000 type=3D8 len=3D67108864 in= dex=3D3 first_map=3D1 pt_ioport_map: e_phys=3D1000 pio_base=3Dc000 len=3D128 index=3D5 first_map= =3D1 pt_msgctrl_reg_write: setup msi for dev 20 pt_msi_setup: pt_msi_setup requested pirq =3D 87 pt_msi_setup: msi mapped with pirq 57 pt_msi_update: Update msi with pirq 57 gvec 0 gflags 3057 pci_intx: intx=3D1 pt_msi_disable: Unbind msi with pirq 57, gvec 0 pt_msi_disable: Unmap msi with pirq 57 Ideas which commit id I ought to look at?