From: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
To: xen-devel@lists.xenproject.org, jbeulich@suse.com
Subject: PCI passthrough (pci-attach) to HVM guests bug (BAR64 addresses are bogus)
Date: Mon, 10 Nov 2014 12:32:48 -0500 [thread overview]
Message-ID: <20141110173248.GA13778@laptop.dumpdata.com> (raw)
Hey,
With Xen 4.5 (today's staging), when I boot a guest and then do pci-attach
the BARs values are corrupt.
For example, with this guest config:
kernel="hvmloader"
builder="hvm"
serial="pty"
memory = 2048
name = "XTT"
usb=1
usbdevice='tablet'
vcpus=2
vga="stdvga"
vif = [ 'mac=00:0f:4b:00:00:63,bridge=xenbr0' ]
disk= ['file:/root/root_image.iso,hdc:cdrom,r']
vnc=1
vnclisten="0.0.0.0"
boot = "dc"
And with this PCI card:
5:00.0 VGA compatible controller: NVIDIA Corporation GF104GLM [Quadro 4000M] (rev a1) (prog-if 00 [VGA controller])
Subsystem: Gigabyte Technology Co., Ltd Device 34fc
Flags: fast devsel, IRQ 20
Memory at fc000000 (32-bit, non-prefetchable) [size=32M]
Memory at d0000000 (64-bit, prefetchable) [size=128M]
Memory at d8000000 (64-bit, prefetchable) [size=64M]
which in dom0 is reported as:
4064] pci 0000:05:00.0: [10de:0e3b] type 00 class 0x030000
[ 22.834087] pci 0000:05:00.0: reg 0x10: [mem 0xfc000000-0xfdffffff]
[ 22.834109] pci 0000:05:00.0: reg 0x14: [mem 0xd0000000-0xd7ffffff 64bit pref]
[ 22.834131] pci 0000:05:00.0: reg 0x1c: [mem 0xd8000000-0xdbffffff 64bit pref]
[ 22.834144] pci 0000:05:00.0: reg 0x24: [io 0xc000-0xc07f]
[ 22.834157] pci 0000:05:00.0: reg 0x30: [mem 0xfe000000-0xfe07ffff pref]
When I assign said card (xl pci-attach XTT 05:00.0) the guest gives me:
... 498525] pci 0000:00:04.0: [10de:0e3b] type 00 class 0x030000
[ 152.508612] pci 0000:00:04.0: reg 0x10: [mem 0x00000000-0x01ffffff]
[ 152.518320] pci 0000:00:04.0: reg 0x14: [mem 0x00000000-0x07ffffff 64bit preff ]
[ 152.529301] pci 0000:00:04.0: reg 0x1c: [mem 0x00000000-0x03ffffff 64bit preff ]
[ 152.540095] pci 0000:00:04.0: reg 0x24: [io 0x0000-0x007f]
[ 152.548497] pci 0000:00:04.0: reg 0x30: [mem 0x00000000-0x0007ffff pref]
[ 152.561018] vgaarb: device added: PCI:0000:00:04.0,decodes=io+mem,owns=none,llocks=none
[ 152.572965] pci 0000:00:04.0: BAR 1: no space for [mem size 0x08000000 64bit pref]
[ 152.583917] pci 0000:00:04.0: BAR 1: failed to assign [mem size 0x08000000 64 bit pref]
[ 152.595528] pci 0000:00:04.0: BAR 3: assigned [mem 0xf4000000-0xf7ffffff 64bi t pref]
If I boot the guest with:
pci=["05:00.0"]
it works fine:
# dmesg | grep 05.0
[ 0.000000] pcpu-alloc: [0] 000 001 002 003 004 005 006 007 008 009 010 011 012 013 014 015
[ 0.905008] pci 0000:00:03.0: reg 0x10: [mem 0xef000000-0xefffffff pref]
[ 0.944101] pci 0000:00:05.0: [10de:0e3b] type 00 class 0x030000
[ 0.953013] pci 0000:00:05.0: reg 0x10: [mem 0xec000000-0xedffffff]
[ 0.967016] pci 0000:00:05.0: reg 0x14: [mem 0xe0000000-0xe7ffffff 64bit pref]
[ 0.973016] pci 0000:00:05.0: reg 0x1c: [mem 0xe8000000-0xebffffff 64bit pref]
[ 0.981015] pci 0000:00:05.0: reg 0x24: [io 0xc200-0xc27f]
[ 0.988016] pci 0000:00:05.0: reg 0x30: [mem 0xf0000000-0xf007ffff pref]
[ 0.995083] vgaarb: device added: PCI:0000:00:05.0,decodes=io+mem,owns=io+mem,locks=none
[ 0.997000] vgaarb: bridge control possible 0000:00:05.0
[ 3.952023] nouveau [ DEVICE][0000:00:05.0] BOOT0 : 0x0c4d80a1
[ 3.952025] nouveau [ DEVICE][0000:00:05.0] Chipset: GF104 (NVC4)
[ 3.952027] nouveau [ DEVICE][0000:00:05.0] Family : NVC0
[ 3.952072] nouveau [ VBIOS][0000:00:05.0] checking PRAMIN for image...
[ 3.952079] nouveau [ VBIOS][0000:00:05.0] ... signature not found
[ 3.952080] nouveau [ VBIOS][0000:00:05.0] checking PROM for image...
[ 4.096446] nouveau [ VBIOS][0000:00:05.0] ... appears to be valid
[ 4.104012] nouveau [ VBIOS][0000:00:05.0] using image from PROM
[ 4.111214] nouveau [ VBIOS][0000:00:05.0] BIT signature found
[ 4.118054] nouveau [ VBIOS][0000:00:05.0] version 70.04.13.00.01
[ 4.125248] nouveau [ DEVINIT][0000:00:05.0] adaptor not initialised
[ 4.131629] nouveau [ VBIOS][0000:00:05.0] running init tables
[ 4.227827] nouveau [ PMC][0000:00:05.0] MSI interrupts enabled
[ 4.234296] nouveau [ PFB][0000:00:05.0] RAM type: GDDR5
[ 4.240176] nouveau [ PFB][0000:00:05.0] RAM size: 1024 MiB
[ 4.245878] nouveau [ PFB][0000:00:05.0] ZCOMP: 0 tags
[ 4.255523] nouveau [ VOLT][0000:00:05.0] GPU voltage: 875000uv
[ 4.291146] nouveau [ PTHERM][0000:00:05.0] FAN control: PWM
[ 4.296582] nouveau [ PTHERM][0000:00:05.0] fan management: automatic
[ 4.302668] nouveau [ PTHERM][0000:00:05.0] internal sensor: yes
[ 4.309465] nouveau [ CLK][0000:00:05.0] 03: core 50 MHz memory 135 MHz
[ 4.317854] nouveau [ CLK][0000:00:05.0] 07: core 405 MHz memory 324 MHz
[ 4.326655] nouveau [ CLK][0000:00:05.0] 0c: core 405 MHz memory 1800 MHz
[ 4.333742] nouveau [ CLK][0000:00:05.0] 0f: core 715 MHz memory 1800 MHz
[ 4.341362] nouveau [ CLK][0000:00:05.0] --: core 50 MHz memory 135 MHz
[ 4.749977] nouveau 0000:00:05.0: fb0: nouveaufb frame buffer device
[ 4.756172] nouveau 0000:00:05.0: registered panic notifier
[ 4.765041] [drm] Initialized nouveau 1.2.0 20120801 for 0000:00:05.0 on minor 0
# lspci -s 00:05.0 -v
00:05.0 VGA compatible controller: nVidia Corporation Device 0e3b (rev a1) (prog-if 00 [VGA controller])
Subsystem: Giga-byte Technology Device 34fc
Physical Slot: 5
Flags: bus master, fast devsel, latency 0, IRQ 67
Memory at ec000000 (32-bit, non-prefetchable) [size=32M]
Memory at e0000000 (64-bit, prefetchable) [size=128M]
Memory at e8000000 (64-bit, prefetchable) [size=64M]
I/O ports at c200 [size=128]
Expansion ROM at f0000000 [disabled] [size=512K]
Interesting observation:
a) It does NOT make a different if I use qemu-traditinal or qemu-xen.
In both cases I get the same BAR bogus numbers.
b) qemu-xen logging is not that great. When I used qemu-trad I discovered:
m-command: hot insert pass-through pci dev
register_real_device: Assigning real physical device 05:00.0 ...
register_real_device: Disable MSI translation via per device option
register_real_device: Disable power management
pt_iomul_init: Error: pt_iomul_init can't open file /dev/xen/pci_iomul: No such file or directory: 0x5:0x0.0x0
pt_register_regions: IO region registered (size=0x02000000 base_addr=0xfc000000)
pt_register_regions: IO region registered (size=0x08000000 base_addr=0xd000000c)
pt_register_regions: IO region registered (size=0x04000000 base_addr=0xd800000c)
pt_register_regions: IO region registered (size=0x00000080 base_addr=0x0000c001)
pt_register_regions: Expansion ROM registered (size=0x00080000 base_addr=0xfe000000)
pci_intx: intx=1
register_real_device: Real physical device 05:00.0 registered successfuly!
IRQ type = INTx
generate a sci for PHP.
deassert due to disable GPE bit.
ACPI:debug: write addr=0xb044, val=0x20.
ACPI:debug: write addr=0xb045, val=0x0.
ACPI:debug: write addr=0xb044, val=0x20.
ACPI:debug: write addr=0xb045, val=0x89.
ACPI:debug: write addr=0xb044, val=0x21.
ACPI:debug: write addr=0xb045, val=0x89.
ACPI:debug: write addr=0xb044, val=0x22.
ACPI:debug: write addr=0xb045, val=0x89.
ACPI:debug: write addr=0xb044, val=0x23.
ACPI:debug: write addr=0xb045, val=0x89.
ACPI:debug: write addr=0xb044, val=0x24.
ACPI:debug: write addr=0xb045, val=0x89.
ACPI:debug: write addr=0xb044, val=0x25.
ACPI:debug: write addr=0xb045, val=0x89.
ACPI:debug: write addr=0xb044, val=0x26.
ACPI:debug: write addr=0xb045, val=0x89.
ACPI:debug: write addr=0xb044, val=0x27.
ACPI:debug: write addr=0xb045, val=0x89.
pt_iomem_map: e_phys=f2000000 maddr=fc000000 type=0 len=33554432 index=0 first_map=1
pt_iomem_map: e_phys=f4000000 maddr=d8000000 type=8 len=67108864 index=3 first_map=1
pt_ioport_map: e_phys=1000 pio_base=c000 len=128 index=5 first_map=1
pt_msgctrl_reg_write: setup msi for dev 20
pt_msi_setup: pt_msi_setup requested pirq = 87
pt_msi_setup: msi mapped with pirq 57
pt_msi_update: Update msi with pirq 57 gvec 0 gflags 3057
pci_intx: intx=1
pt_msi_disable: Unbind msi with pirq 57, gvec 0
pt_msi_disable: Unmap msi with pirq 57
Ideas which commit id I ought to look at?
next reply other threads:[~2014-11-10 17:32 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-11-10 17:32 Konrad Rzeszutek Wilk [this message]
2014-11-10 17:42 ` PCI passthrough (pci-attach) to HVM guests bug (BAR64 addresses are bogus) David Vrabel
2014-11-10 18:07 ` Konrad Rzeszutek Wilk
2014-11-10 21:32 ` Konrad Rzeszutek Wilk
2014-11-12 1:37 ` Konrad Rzeszutek Wilk
2014-11-12 9:24 ` Jan Beulich
2014-11-12 10:01 ` Malcolm Crossley
2014-11-12 10:11 ` Jan Beulich
2014-11-12 10:41 ` Malcolm Crossley
2014-11-12 15:14 ` Konrad Rzeszutek Wilk
2014-11-12 17:24 ` Jan Beulich
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20141110173248.GA13778@laptop.dumpdata.com \
--to=konrad.wilk@oracle.com \
--cc=jbeulich@suse.com \
--cc=xen-devel@lists.xenproject.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.