* [AMD iommu] pci Failed to assign device "hostdev0" : Device or resource busy
@ 2011-12-12 11:15 Andreas Hartmann
2011-12-12 18:36 ` Don Dutile
0 siblings, 1 reply; 8+ messages in thread
From: Andreas Hartmann @ 2011-12-12 11:15 UTC (permalink / raw)
To: kvm
Hello!
I've got a few questions to a problem, which already was analyzed here
sometime ago:
http://markmail.org/message/dspovwvzp3wtdrf6#query:+page:1+mid:i2oph4xwfmiknt3y+state:results
My situation is a bit different. I do have two PCI cards (a Linksys wlan
card and an intel e100 card). Each of these cards should be managed in
an own VM. I do have no problems with IRQ sharing (each device does have its own IRQ).
I'm using linux 3.0.6, kvm 0.15 and libvirt 0.9.7. Mainboard is: GA-990XA-UD3.
The problem is: both cards are behind a PCI-PCI bridge:
00:14.4 PCI bridge: ATI Technologies Inc SBx00 PCI to PCI Bridge (rev 40)
-[0000:00]-+-00.0 ATI Technologies Inc RD890 PCI to PCI bridge (external gfx0 port B)
+-00.2 ATI Technologies Inc Device 5a23
+-02.0-[01]--+-00.0 ATI Technologies Inc Device 6759
| \-00.1 ATI Technologies Inc Device aa90
+-04.0-[02]----00.0 Device 1b6f:7023
+-09.0-[03]----00.0 Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller
+-0a.0-[04]----00.0 Device 1b6f:7023
+-11.0 ATI Technologies Inc SB700/SB800 SATA Controller [AHCI mode]
+-12.0 ATI Technologies Inc SB700/SB800 USB OHCI0 Controller
+-12.2 ATI Technologies Inc SB700/SB800 USB EHCI Controller
+-13.0 ATI Technologies Inc SB700/SB800 USB OHCI0 Controller
+-13.2 ATI Technologies Inc SB700/SB800 USB EHCI Controller
+-14.0 ATI Technologies Inc SBx00 SMBus Controller
+-14.1 ATI Technologies Inc SB700/SB800 IDE Controller
+-14.2 ATI Technologies Inc SBx00 Azalia (Intel HDA)
+-14.3 ATI Technologies Inc SB700/SB800 LPC host controller
+-14.4-[05]--+-06.0 Intel Corporation 82557/8/9/0/1 Ethernet Pro 100
| \-07.0 RaLink RT2800 802.11n PCI
+-14.5 ATI Technologies Inc SB700/SB800 USB OHCI2 Controller
+-15.0-[06]--
+-16.0 ATI Technologies Inc SB700/SB800 USB OHCI0 Controller
+-16.2 ATI Technologies Inc SB700/SB800 USB EHCI Controller
+-18.0 Advanced Micro Devices [AMD] Device 1600
+-18.1 Advanced Micro Devices [AMD] Device 1601
+-18.2 Advanced Micro Devices [AMD] Device 1602
+-18.3 Advanced Micro Devices [AMD] Device 1603
+-18.4 Advanced Micro Devices [AMD] Device 1604
\-18.5 Advanced Micro Devices [AMD] Device 1605
That's what can be seen during boot:
[ 0.621901] AMD-Vi: device: 00:00.2 cap: 0040 seg: 0 flags: 3e info 1300
[ 0.621906] AMD-Vi: mmio-addr: 00000000fec30000
[ 0.622091] AMD-Vi: DEV_SELECT_RANGE_START devid: 00:00.0 flags: 00
[ 0.622095] AMD-Vi: DEV_RANGE_END devid: 00:00.2
[ 0.622097] AMD-Vi: DEV_SELECT devid: 00:02.0 flags: 00
[ 0.622100] AMD-Vi: DEV_SELECT_RANGE_START devid: 01:00.0 flags: 00
[ 0.622102] AMD-Vi: DEV_RANGE_END devid: 01:00.1
[ 0.622105] AMD-Vi: DEV_SELECT devid: 00:04.0 flags: 00
[ 0.622107] AMD-Vi: DEV_SELECT devid: 02:00.0 flags: 00
[ 0.622109] AMD-Vi: DEV_SELECT devid: 00:09.0 flags: 00
[ 0.622112] AMD-Vi: DEV_SELECT devid: 03:00.0 flags: 00
[ 0.622114] AMD-Vi: DEV_SELECT devid: 00:0a.0 flags: 00
[ 0.622117] AMD-Vi: DEV_SELECT devid: 04:00.0 flags: 00
[ 0.622119] AMD-Vi: DEV_SELECT devid: 00:11.0 flags: 00
[ 0.622122] AMD-Vi: DEV_SELECT_RANGE_START devid: 00:12.0 flags: 00
[ 0.622124] AMD-Vi: DEV_RANGE_END devid: 00:12.2
[ 0.622127] AMD-Vi: DEV_SELECT_RANGE_START devid: 00:13.0 flags: 00
[ 0.622129] AMD-Vi: DEV_RANGE_END devid: 00:13.2
[ 0.622132] AMD-Vi: DEV_SELECT devid: 00:14.0 flags: d7
[ 0.622134] AMD-Vi: DEV_SELECT devid: 00:14.1 flags: 00
[ 0.622137] AMD-Vi: DEV_SELECT devid: 00:14.2 flags: 00
[ 0.622139] AMD-Vi: DEV_SELECT devid: 00:14.3 flags: 00
[ 0.622141] AMD-Vi: DEV_SELECT devid: 00:14.4 flags: 00
[ 0.622144] AMD-Vi: DEV_ALIAS_RANGE devid: 05:00.0 flags: 00 devid_to: 00:14.4
[ 0.622147] AMD-Vi: DEV_RANGE_END devid: 05:1f.7
[ 0.622154] AMD-Vi: DEV_SELECT devid: 00:14.5 flags: 00
[ 0.622157] AMD-Vi: DEV_SELECT devid: 00:15.0 flags: 00
[ 0.622159] AMD-Vi: DEV_SELECT_RANGE_START devid: 06:00.0 flags: 00
[ 0.622162] AMD-Vi: DEV_RANGE_END devid: 06:1f.7
[ 0.622166] AMD-Vi: DEV_SELECT_RANGE_START devid: 00:16.0 flags: 00
[ 0.622169] AMD-Vi: DEV_RANGE_END devid: 00:16.2
[ 0.622176] pci 0000:00:00.2: can't derive routing for PCI INT A
[ 0.622179] pci 0000:00:00.2: PCI INT A: no GSI
[ 0.622379] pci 0000:00:00.2: irq 40 for MSI/MSI-X
[ 0.622453] AMD-Vi: Enabling IOMMU at 0000:00:00.2 cap 0x40
[ 0.681866] AMD-Vi: Lazy IO/TLB flushing enabled
[ 0.681870] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[ 0.681873] Placing 64MB software IO TLB between ffff8800cbd9d000 - ffff8800cfd9d000
[ 0.681877] software IO TLB at phys 0xcbd9d000 - 0xcfd9d000
Unbind is done like this:
# PCI bridge
echo "1002 4384" > /sys/bus/pci/drivers/pci-stub/new_id
echo "0000:00:14.4" > /sys/bus/pci/drivers/pci-stub/unbind
# e100
echo "8086 1229" > /sys/bus/pci/drivers/pci-stub/new_id
echo "0000:05:06.0" > /sys/bus/pci/drivers/pci-stub/unbind
echo "0000:05:06.0" > /sys/bus/pci/drivers/pci-stub/bind
echo "8086 1229" > /sys/bus/pci/drivers/pci-stub/remove_id
# Linksys
echo "1814 0601" > /sys/bus/pci/drivers/pci-stub/new_id
echo "0000:05:07.0" > /sys/bus/pci/drivers/pci-stub/unbind
echo "0000:05:07.0" > /sys/bus/pci/drivers/pci-stub/bind
echo "1814 0601" > /sys/bus/pci/drivers/pci-stub/remove_id
I can start a VM (and can see the devices in the VM) with both devices or just one PCI device (without the bridge). If the machine in this VM is halted, the host crashes as already described here:
http://markmail.org/message/dspovwvzp3wtdrf6
(The bridge can't be put to the VM because libvirt (I'm using libvirt) doesn't like it.)
Unfortunately, it wasn't possible to put the PCI devices to different VMs at the same time (Failed to assign device "hostdev0" : Device or resource busy) - but that's exactly what I need :-(.
Now my questions:
1. Is there any way to get the devices into different VMs? (I can't put them to another PCI slot as there are just 2 PCI slots on the board.)
2. Is there any fix to prevent the host crash - maybe in a newer version of kvm or kernel?
Thank you very much for your help,
kind regards,
Andreas Hartmann
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [AMD iommu] pci Failed to assign device "hostdev0" : Device or resource busy
2011-12-12 11:15 [AMD iommu] pci Failed to assign device "hostdev0" : Device or resource busy Andreas Hartmann
@ 2011-12-12 18:36 ` Don Dutile
2011-12-13 0:21 ` Andreas Hartmann
` (3 more replies)
0 siblings, 4 replies; 8+ messages in thread
From: Don Dutile @ 2011-12-12 18:36 UTC (permalink / raw)
To: Andreas Hartmann; +Cc: kvm
On 12/12/2011 06:15 AM, Andreas Hartmann wrote:
> Hello!
>
> I've got a few questions to a problem, which already was analyzed here
> sometime ago:
> http://markmail.org/message/dspovwvzp3wtdrf6#query:+page:1+mid:i2oph4xwfmiknt3y+state:results
>
> My situation is a bit different. I do have two PCI cards (a Linksys wlan
> card and an intel e100 card). Each of these cards should be managed in
> an own VM. I do have no problems with IRQ sharing (each device does have its own IRQ).
>
> I'm using linux 3.0.6, kvm 0.15 and libvirt 0.9.7. Mainboard is: GA-990XA-UD3.
>
>
> The problem is: both cards are behind a PCI-PCI bridge:
>
> 00:14.4 PCI bridge: ATI Technologies Inc SBx00 PCI to PCI Bridge (rev 40)
>
>
> -[0000:00]-+-00.0 ATI Technologies Inc RD890 PCI to PCI bridge (external gfx0 port B)
> +-00.2 ATI Technologies Inc Device 5a23
> +-02.0-[01]--+-00.0 ATI Technologies Inc Device 6759
> | \-00.1 ATI Technologies Inc Device aa90
> +-04.0-[02]----00.0 Device 1b6f:7023
> +-09.0-[03]----00.0 Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller
> +-0a.0-[04]----00.0 Device 1b6f:7023
> +-11.0 ATI Technologies Inc SB700/SB800 SATA Controller [AHCI mode]
> +-12.0 ATI Technologies Inc SB700/SB800 USB OHCI0 Controller
> +-12.2 ATI Technologies Inc SB700/SB800 USB EHCI Controller
> +-13.0 ATI Technologies Inc SB700/SB800 USB OHCI0 Controller
> +-13.2 ATI Technologies Inc SB700/SB800 USB EHCI Controller
> +-14.0 ATI Technologies Inc SBx00 SMBus Controller
> +-14.1 ATI Technologies Inc SB700/SB800 IDE Controller
> +-14.2 ATI Technologies Inc SBx00 Azalia (Intel HDA)
> +-14.3 ATI Technologies Inc SB700/SB800 LPC host controller
>
> +-14.4-[05]--+-06.0 Intel Corporation 82557/8/9/0/1 Ethernet Pro 100
> | \-07.0 RaLink RT2800 802.11n PCI
>
You cannot assign two devices behind a (legacy) PCI (not PCIe) bridge to two
different VMs because PCI devices dont provide a B:D.F in their transaction headers;
In this case, the PCIe-PCI bridge/switch prepends the bridge's B:D.F in front of
a transaction that is sourced by 05:06.0 or 05:07.0 .
So, the devices cannot be isolated from each other's (mem) mapping domains in the
IOMMU, thus, libvirt prevents this (security) violation from being enabled.
> +-14.5 ATI Technologies Inc SB700/SB800 USB OHCI2 Controller
> +-15.0-[06]--
> +-16.0 ATI Technologies Inc SB700/SB800 USB OHCI0 Controller
> +-16.2 ATI Technologies Inc SB700/SB800 USB EHCI Controller
> +-18.0 Advanced Micro Devices [AMD] Device 1600
> +-18.1 Advanced Micro Devices [AMD] Device 1601
> +-18.2 Advanced Micro Devices [AMD] Device 1602
> +-18.3 Advanced Micro Devices [AMD] Device 1603
> +-18.4 Advanced Micro Devices [AMD] Device 1604
> \-18.5 Advanced Micro Devices [AMD] Device 1605
>
> That's what can be seen during boot:
>
> [ 0.621901] AMD-Vi: device: 00:00.2 cap: 0040 seg: 0 flags: 3e info 1300
> [ 0.621906] AMD-Vi: mmio-addr: 00000000fec30000
> [ 0.622091] AMD-Vi: DEV_SELECT_RANGE_START devid: 00:00.0 flags: 00
> [ 0.622095] AMD-Vi: DEV_RANGE_END devid: 00:00.2
> [ 0.622097] AMD-Vi: DEV_SELECT devid: 00:02.0 flags: 00
> [ 0.622100] AMD-Vi: DEV_SELECT_RANGE_START devid: 01:00.0 flags: 00
> [ 0.622102] AMD-Vi: DEV_RANGE_END devid: 01:00.1
> [ 0.622105] AMD-Vi: DEV_SELECT devid: 00:04.0 flags: 00
> [ 0.622107] AMD-Vi: DEV_SELECT devid: 02:00.0 flags: 00
> [ 0.622109] AMD-Vi: DEV_SELECT devid: 00:09.0 flags: 00
> [ 0.622112] AMD-Vi: DEV_SELECT devid: 03:00.0 flags: 00
> [ 0.622114] AMD-Vi: DEV_SELECT devid: 00:0a.0 flags: 00
> [ 0.622117] AMD-Vi: DEV_SELECT devid: 04:00.0 flags: 00
> [ 0.622119] AMD-Vi: DEV_SELECT devid: 00:11.0 flags: 00
> [ 0.622122] AMD-Vi: DEV_SELECT_RANGE_START devid: 00:12.0 flags: 00
> [ 0.622124] AMD-Vi: DEV_RANGE_END devid: 00:12.2
> [ 0.622127] AMD-Vi: DEV_SELECT_RANGE_START devid: 00:13.0 flags: 00
> [ 0.622129] AMD-Vi: DEV_RANGE_END devid: 00:13.2
> [ 0.622132] AMD-Vi: DEV_SELECT devid: 00:14.0 flags: d7
> [ 0.622134] AMD-Vi: DEV_SELECT devid: 00:14.1 flags: 00
> [ 0.622137] AMD-Vi: DEV_SELECT devid: 00:14.2 flags: 00
> [ 0.622139] AMD-Vi: DEV_SELECT devid: 00:14.3 flags: 00
> [ 0.622141] AMD-Vi: DEV_SELECT devid: 00:14.4 flags: 00
> [ 0.622144] AMD-Vi: DEV_ALIAS_RANGE devid: 05:00.0 flags: 00 devid_to: 00:14.4
> [ 0.622147] AMD-Vi: DEV_RANGE_END devid: 05:1f.7
> [ 0.622154] AMD-Vi: DEV_SELECT devid: 00:14.5 flags: 00
> [ 0.622157] AMD-Vi: DEV_SELECT devid: 00:15.0 flags: 00
> [ 0.622159] AMD-Vi: DEV_SELECT_RANGE_START devid: 06:00.0 flags: 00
> [ 0.622162] AMD-Vi: DEV_RANGE_END devid: 06:1f.7
> [ 0.622166] AMD-Vi: DEV_SELECT_RANGE_START devid: 00:16.0 flags: 00
> [ 0.622169] AMD-Vi: DEV_RANGE_END devid: 00:16.2
> [ 0.622176] pci 0000:00:00.2: can't derive routing for PCI INT A
> [ 0.622179] pci 0000:00:00.2: PCI INT A: no GSI
> [ 0.622379] pci 0000:00:00.2: irq 40 for MSI/MSI-X
> [ 0.622453] AMD-Vi: Enabling IOMMU at 0000:00:00.2 cap 0x40
> [ 0.681866] AMD-Vi: Lazy IO/TLB flushing enabled
> [ 0.681870] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
> [ 0.681873] Placing 64MB software IO TLB between ffff8800cbd9d000 - ffff8800cfd9d000
> [ 0.681877] software IO TLB at phys 0xcbd9d000 - 0xcfd9d000
>
> Unbind is done like this:
> # PCI bridge
> echo "1002 4384"> /sys/bus/pci/drivers/pci-stub/new_id
> echo "0000:00:14.4"> /sys/bus/pci/drivers/pci-stub/unbind
>
> # e100
> echo "8086 1229"> /sys/bus/pci/drivers/pci-stub/new_id
> echo "0000:05:06.0"> /sys/bus/pci/drivers/pci-stub/unbind
> echo "0000:05:06.0"> /sys/bus/pci/drivers/pci-stub/bind
> echo "8086 1229"> /sys/bus/pci/drivers/pci-stub/remove_id
>
> # Linksys
> echo "1814 0601"> /sys/bus/pci/drivers/pci-stub/new_id
> echo "0000:05:07.0"> /sys/bus/pci/drivers/pci-stub/unbind
> echo "0000:05:07.0"> /sys/bus/pci/drivers/pci-stub/bind
> echo "1814 0601"> /sys/bus/pci/drivers/pci-stub/remove_id
>
> I can start a VM (and can see the devices in the VM) with both devices or just one PCI device (without the bridge). If the machine in this VM is halted, the host crashes as already described here:
> http://markmail.org/message/dspovwvzp3wtdrf6
>
> (The bridge can't be put to the VM because libvirt (I'm using libvirt) doesn't like it.)
>
>
> Unfortunately, it wasn't possible to put the PCI devices to different VMs at the same time (Failed to assign device "hostdev0" : Device or resource busy) - but that's exactly what I need :-(.
>
>
> Now my questions:
>
> 1. Is there any way to get the devices into different VMs? (I can't put them to another PCI slot as there are just 2 PCI slots on the board.)
Only if the two slots are behind different PCIe-PCI bridges .
> 2. Is there any fix to prevent the host crash - maybe in a newer version of kvm or kernel?
don't assign the PCI bridge; libvirt does all the proper bind/unbinding of devices
for assigned devices, so the above echo steps are unnecessary if you use libvirt.
>
>
> Thank you very much for your help,
> kind regards,
> Andreas Hartmann
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [AMD iommu] pci Failed to assign device "hostdev0" : Device or resource busy
2011-12-12 18:36 ` Don Dutile
@ 2011-12-13 0:21 ` Andreas Hartmann
2011-12-13 8:40 ` Andreas Hartmann
` (2 subsequent siblings)
3 siblings, 0 replies; 8+ messages in thread
From: Andreas Hartmann @ 2011-12-13 0:21 UTC (permalink / raw)
To: Don Dutile; +Cc: kvm
Am Mon, 12 Dec 2011 13:36:36 -0500
schrieb Don Dutile <ddutile@redhat.com>:
> On 12/12/2011 06:15 AM, Andreas Hartmann wrote:
> > Hello!
> >
> > I've got a few questions to a problem, which already was analyzed here
> > sometime ago:
> > http://markmail.org/message/dspovwvzp3wtdrf6#query:+page:1+mid:i2oph4xwfmiknt3y+state:results
> >
> > My situation is a bit different. I do have two PCI cards (a Linksys wlan
> > card and an intel e100 card). Each of these cards should be managed in
> > an own VM. I do have no problems with IRQ sharing (each device does have its own IRQ).
> >
> > I'm using linux 3.0.6, kvm 0.15 and libvirt 0.9.7. Mainboard is: GA-990XA-UD3.
> >
> >
> > The problem is: both cards are behind a PCI-PCI bridge:
> >
> > 00:14.4 PCI bridge: ATI Technologies Inc SBx00 PCI to PCI Bridge (rev 40)
> >
> >
> > -[0000:00]-+-00.0 ATI Technologies Inc RD890 PCI to PCI bridge (external gfx0 port B)
> > +-00.2 ATI Technologies Inc Device 5a23
> > +-02.0-[01]--+-00.0 ATI Technologies Inc Device 6759
> > | \-00.1 ATI Technologies Inc Device aa90
> > +-04.0-[02]----00.0 Device 1b6f:7023
> > +-09.0-[03]----00.0 Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller
> > +-0a.0-[04]----00.0 Device 1b6f:7023
> > +-11.0 ATI Technologies Inc SB700/SB800 SATA Controller [AHCI mode]
> > +-12.0 ATI Technologies Inc SB700/SB800 USB OHCI0 Controller
> > +-12.2 ATI Technologies Inc SB700/SB800 USB EHCI Controller
> > +-13.0 ATI Technologies Inc SB700/SB800 USB OHCI0 Controller
> > +-13.2 ATI Technologies Inc SB700/SB800 USB EHCI Controller
> > +-14.0 ATI Technologies Inc SBx00 SMBus Controller
> > +-14.1 ATI Technologies Inc SB700/SB800 IDE Controller
> > +-14.2 ATI Technologies Inc SBx00 Azalia (Intel HDA)
> > +-14.3 ATI Technologies Inc SB700/SB800 LPC host controller
> >
> > +-14.4-[05]--+-06.0 Intel Corporation 82557/8/9/0/1 Ethernet Pro 100
> > | \-07.0 RaLink RT2800 802.11n PCI
> >
> You cannot assign two devices behind a (legacy) PCI (not PCIe) bridge to two
> different VMs because PCI devices dont provide a B:D.F in their transaction headers;
> In this case, the PCIe-PCI bridge/switch prepends the bridge's B:D.F in front of
> a transaction that is sourced by 05:06.0 or 05:07.0 .
>
> So, the devices cannot be isolated from each other's (mem) mapping domains in the
> IOMMU, thus, libvirt prevents this (security) violation from being enabled.
Ok. If I remove the intel card and put in instead a 32 bit PCIe card
like TP-Link TG-3468, I could assign each of these two cards to
different VMs.
Is this correct?
[...]
> > 1. Is there any way to get the devices into different VMs? (I can't put them to another PCI slot as there are just 2 PCI slots on the board.)
> Only if the two slots are behind different PCIe-PCI bridges .
>
> > 2. Is there any fix to prevent the host crash - maybe in a newer version of kvm or kernel?
> don't assign the PCI bridge; libvirt does all the proper bind/unbinding of devices
> for assigned devices, so the above echo steps are unnecessary if you use libvirt.
If I just remove the intel card, I should be able to assign the wlan
card to one of my VMs. I'll try that.
Thank you for your advice!
Kind regards,
Andreas
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [AMD iommu] pci Failed to assign device "hostdev0" : Device or resource busy
2011-12-12 18:36 ` Don Dutile
2011-12-13 0:21 ` Andreas Hartmann
@ 2011-12-13 8:40 ` Andreas Hartmann
2011-12-19 20:32 ` Don Dutile
2011-12-13 9:40 ` Andreas Hartmann
2011-12-13 14:16 ` Andreas Hartmann
3 siblings, 1 reply; 8+ messages in thread
From: Andreas Hartmann @ 2011-12-13 8:40 UTC (permalink / raw)
To: Don Dutile; +Cc: kvm
Hello Don!
Am Tue, 13 Dec 2011 01:21:41 +0100
schrieb Andreas Hartmann <andihartmann@01019freenet.de>:
> Am Mon, 12 Dec 2011 13:36:36 -0500
> schrieb Don Dutile <ddutile@redhat.com>:
>
> > On 12/12/2011 06:15 AM, Andreas Hartmann wrote:
> > > Hello!
> > >
> > > I've got a few questions to a problem, which already was analyzed here
> > > sometime ago:
> > > http://markmail.org/message/dspovwvzp3wtdrf6#query:+page:1+mid:i2oph4xwfmiknt3y+state:results
> > >
> > > My situation is a bit different. I do have two PCI cards (a Linksys wlan
> > > card and an intel e100 card). Each of these cards should be managed in
> > > an own VM. I do have no problems with IRQ sharing (each device does have its own IRQ).
> > >
> > > I'm using linux 3.0.6, kvm 0.15 and libvirt 0.9.7. Mainboard is: GA-990XA-UD3.
> > >
> > >
> > > The problem is: both cards are behind a PCI-PCI bridge:
> > >
> > > 00:14.4 PCI bridge: ATI Technologies Inc SBx00 PCI to PCI Bridge (rev 40)
> > >
> > >
> > > -[0000:00]-+-00.0 ATI Technologies Inc RD890 PCI to PCI bridge (external gfx0 port B)
> > > +-00.2 ATI Technologies Inc Device 5a23
> > > +-02.0-[01]--+-00.0 ATI Technologies Inc Device 6759
> > > | \-00.1 ATI Technologies Inc Device aa90
> > > +-04.0-[02]----00.0 Device 1b6f:7023
> > > +-09.0-[03]----00.0 Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller
> > > +-0a.0-[04]----00.0 Device 1b6f:7023
> > > +-11.0 ATI Technologies Inc SB700/SB800 SATA Controller [AHCI mode]
> > > +-12.0 ATI Technologies Inc SB700/SB800 USB OHCI0 Controller
> > > +-12.2 ATI Technologies Inc SB700/SB800 USB EHCI Controller
> > > +-13.0 ATI Technologies Inc SB700/SB800 USB OHCI0 Controller
> > > +-13.2 ATI Technologies Inc SB700/SB800 USB EHCI Controller
> > > +-14.0 ATI Technologies Inc SBx00 SMBus Controller
> > > +-14.1 ATI Technologies Inc SB700/SB800 IDE Controller
> > > +-14.2 ATI Technologies Inc SBx00 Azalia (Intel HDA)
> > > +-14.3 ATI Technologies Inc SB700/SB800 LPC host controller
> > >
> > > +-14.4-[05]--+-06.0 Intel Corporation 82557/8/9/0/1 Ethernet Pro 100
> > > | \-07.0 RaLink RT2800 802.11n PCI
> > >
> > You cannot assign two devices behind a (legacy) PCI (not PCIe) bridge to two
> > different VMs because PCI devices dont provide a B:D.F in their transaction headers;
> > In this case, the PCIe-PCI bridge/switch prepends the bridge's B:D.F in front of
> > a transaction that is sourced by 05:06.0 or 05:07.0 .
> >
> > So, the devices cannot be isolated from each other's (mem) mapping domains in the
> > IOMMU, thus, libvirt prevents this (security) violation from being enabled.
>
> Ok. If I remove the intel card and put in instead a 32 bit PCIe card
> like TP-Link TG-3468, I could assign each of these two cards to
> different VMs.
>
> Is this correct?
>
> [...]
>
> > > 1. Is there any way to get the devices into different VMs? (I can't put them to another PCI slot as there are just 2 PCI slots on the board.)
> > Only if the two slots are behind different PCIe-PCI bridges .
> >
> > > 2. Is there any fix to prevent the host crash - maybe in a newer version of kvm or kernel?
> > don't assign the PCI bridge; libvirt does all the proper bind/unbinding of devices
> > for assigned devices, so the above echo steps are unnecessary if you use libvirt.
>
> If I just remove the intel card, I should be able to assign the wlan
> card to one of my VMs. I'll try that.
Meanwhile I checked this scenario: I removed the intel card and
rebooted. I got the following pci tree:
-[0000:00]-+-00.0 ATI Technologies Inc RD890 PCI to PCI bridge (external gfx0 port B)
+-00.2 ATI Technologies Inc Device 5a23
+-02.0-[01]--+-00.0 ATI Technologies Inc Device 6759
| \-00.1 ATI Technologies Inc Device aa90
+-04.0-[02]----00.0 Device 1b6f:7023
+-09.0-[03]----00.0 Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller
+-0a.0-[04]----00.0 Device 1b6f:7023
+-11.0 ATI Technologies Inc SB700/SB800 SATA Controller [AHCI mode]
+-12.0 ATI Technologies Inc SB700/SB800 USB OHCI0 Controller
+-12.2 ATI Technologies Inc SB700/SB800 USB EHCI Controller
+-13.0 ATI Technologies Inc SB700/SB800 USB OHCI0 Controller
+-13.2 ATI Technologies Inc SB700/SB800 USB EHCI Controller
+-14.0 ATI Technologies Inc SBx00 SMBus Controller
+-14.1 ATI Technologies Inc SB700/SB800 IDE Controller
+-14.2 ATI Technologies Inc SBx00 Azalia (Intel HDA)
+-14.3 ATI Technologies Inc SB700/SB800 LPC host controller
+-14.4-[05]----07.0 RaLink RT2800 802.11n PCI
+-14.5 ATI Technologies Inc SB700/SB800 USB OHCI2 Controller
+-15.0-[06]--
+-16.0 ATI Technologies Inc SB700/SB800 USB OHCI0 Controller
+-16.2 ATI Technologies Inc SB700/SB800 USB EHCI Controller
+-18.0 Advanced Micro Devices [AMD] Device 1600
+-18.1 Advanced Micro Devices [AMD] Device 1601
+-18.2 Advanced Micro Devices [AMD] Device 1602
+-18.3 Advanced Micro Devices [AMD] Device 1603
+-18.4 Advanced Micro Devices [AMD] Device 1604
\-18.5 Advanced Micro Devices [AMD] Device 1605
But this doesn't work, too: I had to manually unbind the bridge (as described in the mail before - it was
not necessary to manually unbind the device itself). On closing the VM,
the complete host crashed with this screen (excerpt):
amd_iommu_domain_destroy
iommu_domain_free
kvm_iommu_unmap_guest
kvm_arch_destroy_vm
kvm_destroy_vm
kvm_vcpu_release
__fput
flip_close
put_files_struct
do_exit
do_group_exit
get_signal_to_deliver
do_signal
do_notify_resume
int_signal
Seams to even not work if there's just one (legal) PCI device plugged
in.
Could somebody please shed some light on this problem?
Thank you,
kind regards,
Andreas Hartmann
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [AMD iommu] pci Failed to assign device "hostdev0" : Device or resource busy
2011-12-12 18:36 ` Don Dutile
2011-12-13 0:21 ` Andreas Hartmann
2011-12-13 8:40 ` Andreas Hartmann
@ 2011-12-13 9:40 ` Andreas Hartmann
2011-12-13 14:16 ` Andreas Hartmann
3 siblings, 0 replies; 8+ messages in thread
From: Andreas Hartmann @ 2011-12-13 9:40 UTC (permalink / raw)
To: Don Dutile; +Cc: kvm
Hello Don!
Some additional information about the bridge device 00:14.4 itself
(both legacy PCI cards are behind this bridge device - it's an output with both legacy devices plugged in):
00:14.4 PCI bridge: ATI Technologies Inc SBx00 PCI to PCI Bridge (rev 40) (prog-if 01 [Subtractive decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop+ ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 64
Bus: primary=00, secondary=05, subordinate=05, sec-latency=64
I/O behind bridge: 0000a000-0000afff
Memory behind bridge: fda00000-fdafffff
Prefetchable memory behind bridge: fd900000-fd9fffff
Secondary status: 66MHz- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
If the bridge is put to the VM too, libvirt exits with the following error:
error: internal error Unable to reset PCI device 0000:00:14.4: no FLR, PM reset or bus reset available
The wlan device looks like this:
05:07.0 Network controller: RaLink RT2800 802.11n PCI
Subsystem: Linksys Device 0067
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=slow >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 32 (500ns min, 1000ns max), Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 21
Region 0: Memory at fdae0000 (32-bit, non-prefetchable) [size=64K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Thank you for your advice,
kind regards,
Andreas Hartmann
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [AMD iommu] pci Failed to assign device "hostdev0" : Device or resource busy
2011-12-12 18:36 ` Don Dutile
` (2 preceding siblings ...)
2011-12-13 9:40 ` Andreas Hartmann
@ 2011-12-13 14:16 ` Andreas Hartmann
3 siblings, 0 replies; 8+ messages in thread
From: Andreas Hartmann @ 2011-12-13 14:16 UTC (permalink / raw)
To: Don Dutile; +Cc: kvm
Hello Don!
Am Tue, 13 Dec 2011 01:21:41 +0100
schrieb Andreas Hartmann <andihartmann@01019freenet.de>:
[...]
> Ok. If I remove the intel card and put in instead a 32 bit PCIe card
> like TP-Link TG-3468, I could assign each of these two cards to
> different VMs.
>
> Is this correct?
No - it isn't correct - but don't now why!
1. The legacy pci card, even if it's the only legacy one, can't be
used, because it crashes the machine after removing the VM (see other
post).
2. The TP-Link TG-3468 card crashes the host at the moment of "virsh
start VM".
03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 01)
Subsystem: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller
Flags: bus master, fast devsel, latency 0, IRQ 43
I/O ports at de00 [size=256]
Memory at fdbff000 (64-bit, non-prefetchable) [size=4K]
[virtual] Expansion ROM at fda00000 [disabled] [size=128K]
Capabilities: [40] Power Management version 2
Capabilities: [48] Vital Product Data
Capabilities: [50] MSI: Enable+ Count=1/2 Maskable- 64bit+
Capabilities: [60] Express Endpoint, MSI 00
Capabilities: [84] Vendor Specific Information: Len=4c <?>
Capabilities: [100] Advanced Error Reporting
Capabilities: [12c] Virtual Channel
Capabilities: [148] Device Serial Number f1-11-00-00-68-4c-e0-00
Capabilities: [154] Power Budgeting <?>
Kernel driver in use: r8169
04:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 06)
Subsystem: Giga-byte Technology GA-EP45-DS5 Motherboard
Flags: bus master, fast devsel, latency 0, IRQ 44
I/O ports at ce00 [size=256]
Memory at fd6ff000 (64-bit, prefetchable) [size=4K]
Memory at fd6f8000 (64-bit, prefetchable) [size=16K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [70] Express Endpoint, MSI 01
Capabilities: [b0] MSI-X: Enable- Count=4 Masked-
Capabilities: [d0] Vital Product Data
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Virtual Channel
Capabilities: [160] Device Serial Number 12-34-56-78-12-34-56-78
Kernel driver in use: r8169
First problem: I don't know, how to distinguish these two cards. They
use both the same module and they have equal PCI-ID's (but different
bus ID's).
If the desired one (03:00.0) is manually unbound with these lines
echo "10ec 8168" > /sys/bus/pci/drivers/pci-stub/new_id
echo "0000:03:00.0" > /sys/bus/pci/devices/0000\:03\:00.0/driver/unbind
echo "0000:03:00.0" > /sys/bus/pci/drivers/pci-stub/bind
both devices are unbound (this is definitely wrong) and both devices
use the same IRQ (as they were managed by r8169, they had different
IRQs):
03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 01)
Subsystem: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller
Flags: fast devsel, IRQ 17
I/O ports at de00 [size=256]
Memory at fdbff000 (64-bit, non-prefetchable) [size=4K]
[virtual] Expansion ROM at fda00000 [disabled] [size=128K]
Capabilities: [40] Power Management version 2
Capabilities: [48] Vital Product Data
Capabilities: [50] MSI: Enable- Count=1/2 Maskable- 64bit+
Capabilities: [60] Express Endpoint, MSI 00
Capabilities: [84] Vendor Specific Information: Len=4c <?>
Capabilities: [100] Advanced Error Reporting
Capabilities: [12c] Virtual Channel
Capabilities: [148] Device Serial Number f1-11-00-00-68-4c-e0-00
Capabilities: [154] Power Budgeting <?>
Kernel driver in use: pci-stub
04:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 06)
Subsystem: Giga-byte Technology GA-EP45-DS5 Motherboard
Flags: fast devsel, IRQ 17
I/O ports at ce00 [size=256]
Memory at fd6ff000 (64-bit, prefetchable) [size=4K]
Memory at fd6f8000 (64-bit, prefetchable) [size=16K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [70] Express Endpoint, MSI 01
Capabilities: [b0] MSI-X: Enable- Count=4 Masked-
Capabilities: [d0] Vital Product Data
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Virtual Channel
Capabilities: [160] Device Serial Number 12-34-56-78-12-34-56-78
Kernel driver in use: pci-stub
Second problem:
If I do a "modprobe r8169", both devices are "adressed". How can I
address just one of both?
Sorry for all of these questions :-). I really would be glad, to get an
answer!
Kind regards,
Andreas Hartmann
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [AMD iommu] pci Failed to assign device "hostdev0" : Device or resource busy
2011-12-13 8:40 ` Andreas Hartmann
@ 2011-12-19 20:32 ` Don Dutile
[not found] ` <20111220000002.0efc3ec1@dualc.maya.org>
0 siblings, 1 reply; 8+ messages in thread
From: Don Dutile @ 2011-12-19 20:32 UTC (permalink / raw)
To: Andreas Hartmann; +Cc: kvm
On 12/13/2011 03:40 AM, Andreas Hartmann wrote:
> Hello Don!
>
Andreas,
sorry for the delay, I don't follow this email list frequently (read: !daily).
> Am Tue, 13 Dec 2011 01:21:41 +0100
> schrieb Andreas Hartmann<andihartmann@01019freenet.de>:
>
>> Am Mon, 12 Dec 2011 13:36:36 -0500
>> schrieb Don Dutile<ddutile@redhat.com>:
>>
>>> On 12/12/2011 06:15 AM, Andreas Hartmann wrote:
>>>> Hello!
>>>>
>>>> I've got a few questions to a problem, which already was analyzed here
>>>> sometime ago:
>>>> http://markmail.org/message/dspovwvzp3wtdrf6#query:+page:1+mid:i2oph4xwfmiknt3y+state:results
>>>>
>>>> My situation is a bit different. I do have two PCI cards (a Linksys wlan
>>>> card and an intel e100 card). Each of these cards should be managed in
>>>> an own VM. I do have no problems with IRQ sharing (each device does have its own IRQ).
>>>>
>>>> I'm using linux 3.0.6, kvm 0.15 and libvirt 0.9.7. Mainboard is: GA-990XA-UD3.
>>>>
>>>>
>>>> The problem is: both cards are behind a PCI-PCI bridge:
>>>>
>>>> 00:14.4 PCI bridge: ATI Technologies Inc SBx00 PCI to PCI Bridge (rev 40)
>>>>
>>>>
>>>> -[0000:00]-+-00.0 ATI Technologies Inc RD890 PCI to PCI bridge (external gfx0 port B)
>>>> +-00.2 ATI Technologies Inc Device 5a23
>>>> +-02.0-[01]--+-00.0 ATI Technologies Inc Device 6759
>>>> | \-00.1 ATI Technologies Inc Device aa90
>>>> +-04.0-[02]----00.0 Device 1b6f:7023
>>>> +-09.0-[03]----00.0 Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller
>>>> +-0a.0-[04]----00.0 Device 1b6f:7023
>>>> +-11.0 ATI Technologies Inc SB700/SB800 SATA Controller [AHCI mode]
>>>> +-12.0 ATI Technologies Inc SB700/SB800 USB OHCI0 Controller
>>>> +-12.2 ATI Technologies Inc SB700/SB800 USB EHCI Controller
>>>> +-13.0 ATI Technologies Inc SB700/SB800 USB OHCI0 Controller
>>>> +-13.2 ATI Technologies Inc SB700/SB800 USB EHCI Controller
>>>> +-14.0 ATI Technologies Inc SBx00 SMBus Controller
>>>> +-14.1 ATI Technologies Inc SB700/SB800 IDE Controller
>>>> +-14.2 ATI Technologies Inc SBx00 Azalia (Intel HDA)
>>>> +-14.3 ATI Technologies Inc SB700/SB800 LPC host controller
>>>>
>>>> +-14.4-[05]--+-06.0 Intel Corporation 82557/8/9/0/1 Ethernet Pro 100
>>>> | \-07.0 RaLink RT2800 802.11n PCI
>>>>
>>> You cannot assign two devices behind a (legacy) PCI (not PCIe) bridge to two
>>> different VMs because PCI devices dont provide a B:D.F in their transaction headers;
>>> In this case, the PCIe-PCI bridge/switch prepends the bridge's B:D.F in front of
>>> a transaction that is sourced by 05:06.0 or 05:07.0 .
>>>
>>> So, the devices cannot be isolated from each other's (mem) mapping domains in the
>>> IOMMU, thus, libvirt prevents this (security) violation from being enabled.
>>
>> Ok. If I remove the intel card and put in instead a 32 bit PCIe card
>> like TP-Link TG-3468, I could assign each of these two cards to
>> different VMs.
>>
KISS: Don't try to assign legacy PCI devices; just PCIe. ;-)
Where is "TP-Link TG-3468" in lspci output below? the AMD devices in 18.0->18.5 ???
>> Is this correct?
>>
>> [...]
>>
>>>> 1. Is there any way to get the devices into different VMs? (I can't put them to another PCI slot as there are just 2 PCI slots on the board.)
>>> Only if the two slots are behind different PCIe-PCI bridges .
>>>
>>>> 2. Is there any fix to prevent the host crash - maybe in a newer version of kvm or kernel?
>>> don't assign the PCI bridge; libvirt does all the proper bind/unbinding of devices
>>> for assigned devices, so the above echo steps are unnecessary if you use libvirt.
>>
>> If I just remove the intel card, I should be able to assign the wlan
>> card to one of my VMs. I'll try that.
>
> Meanwhile I checked this scenario: I removed the intel card and
> rebooted. I got the following pci tree:
>
> -[0000:00]-+-00.0 ATI Technologies Inc RD890 PCI to PCI bridge (external gfx0 port B)
> +-00.2 ATI Technologies Inc Device 5a23
> +-02.0-[01]--+-00.0 ATI Technologies Inc Device 6759
> | \-00.1 ATI Technologies Inc Device aa90
> +-04.0-[02]----00.0 Device 1b6f:7023
> +-09.0-[03]----00.0 Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller
> +-0a.0-[04]----00.0 Device 1b6f:7023
> +-11.0 ATI Technologies Inc SB700/SB800 SATA Controller [AHCI mode]
> +-12.0 ATI Technologies Inc SB700/SB800 USB OHCI0 Controller
> +-12.2 ATI Technologies Inc SB700/SB800 USB EHCI Controller
> +-13.0 ATI Technologies Inc SB700/SB800 USB OHCI0 Controller
> +-13.2 ATI Technologies Inc SB700/SB800 USB EHCI Controller
> +-14.0 ATI Technologies Inc SBx00 SMBus Controller
> +-14.1 ATI Technologies Inc SB700/SB800 IDE Controller
> +-14.2 ATI Technologies Inc SBx00 Azalia (Intel HDA)
> +-14.3 ATI Technologies Inc SB700/SB800 LPC host controller
> +-14.4-[05]----07.0 RaLink RT2800 802.11n PCI
>
> +-14.5 ATI Technologies Inc SB700/SB800 USB OHCI2 Controller
> +-15.0-[06]--
> +-16.0 ATI Technologies Inc SB700/SB800 USB OHCI0 Controller
> +-16.2 ATI Technologies Inc SB700/SB800 USB EHCI Controller
> +-18.0 Advanced Micro Devices [AMD] Device 1600
> +-18.1 Advanced Micro Devices [AMD] Device 1601
> +-18.2 Advanced Micro Devices [AMD] Device 1602
> +-18.3 Advanced Micro Devices [AMD] Device 1603
> +-18.4 Advanced Micro Devices [AMD] Device 1604
> \-18.5 Advanced Micro Devices [AMD] Device 1605
>
> But this doesn't work, too: I had to manually unbind the bridge (as described in the mail before - it was
> not necessary to manually unbind the device itself). On closing the VM,
> the complete host crashed with this screen (excerpt):
>
You cannot assign a PCI bridge. In fact, you don't need to for device-assignment.
You just need to assign the device itself; nothing else in the host PCI hierarchy.
.... assuming the device is a PCIe device, and not a graphics card/device.
again, libvirt does all the bind/unbinding you need to do.
You want to do a 'virsh device-attach <dom-id> pci.xml
with pci.xml looking something like:
<hostdev mode='subsystem' type='pci' managed='no'>
<source>
<address bus='0' slot='XX' function='0'/>
</source>
</hostdev>
where XX is replaced by the value of the slot your PCIe device is plugged into.
> amd_iommu_domain_destroy
> iommu_domain_free
> kvm_iommu_unmap_guest
> kvm_arch_destroy_vm
> kvm_destroy_vm
> kvm_vcpu_release
> __fput
> flip_close
> put_files_struct
> do_exit
> do_group_exit
> get_signal_to_deliver
> do_signal
> do_notify_resume
> int_signal
>
> Seams to even not work if there's just one (legal) PCI device plugged
> in.
>
> Could somebody please shed some light on this problem?
>
>
> Thank you,
> kind regards,
> Andreas Hartmann
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [AMD iommu] pci Failed to assign device "hostdev0" : Device or resource busy
[not found] ` <20111220000002.0efc3ec1@dualc.maya.org>
@ 2011-12-20 8:31 ` Andreas Hartmann
0 siblings, 0 replies; 8+ messages in thread
From: Andreas Hartmann @ 2011-12-20 8:31 UTC (permalink / raw)
To: kvm
Hello kvm-list!
I sent the following text to Don with some additional detailed log
files. If somebody else need them too, I can provide them with pn.
Kind regards,
Andreas Hartmann
Andreas Hartmann schrieb:
> Hello Don,
>
> thank you for your reply!
>
> I just want to describe in short the two problems I encounter (my
> description was a bit chaotic :-) I hope it's better now :-)). I
> attached details in two files (dmesg.bz2 and lspci.bz2. It's the
> untouched raw output - no unbind or anything else had been done).
>
> General hardware setup for a quick overview:
>
> -[0000:00]-+-00.0 ATI Technologies Inc RD890 PCI to PCI bridge (external gfx0 port B)
> +-00.2 ATI Technologies Inc Device 5a23
> +-02.0-[01]--+-00.0 ATI Technologies Inc Device 6759
> | \-00.1 ATI Technologies Inc Device aa90
> +-04.0-[02]----00.0 Device 1b6f:7023
> +-05.0-[03]----00.0 Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller
> +-09.0-[04]----00.0 Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller
> +-0a.0-[05]----00.0 Device 1b6f:7023
> +-11.0 ATI Technologies Inc SB700/SB800 SATA Controller [AHCI mode]
> +-12.0 ATI Technologies Inc SB700/SB800 USB OHCI0 Controller
> +-12.2 ATI Technologies Inc SB700/SB800 USB EHCI Controller
> +-13.0 ATI Technologies Inc SB700/SB800 USB OHCI0 Controller
> +-13.2 ATI Technologies Inc SB700/SB800 USB EHCI Controller
> +-14.0 ATI Technologies Inc SBx00 SMBus Controller
> +-14.1 ATI Technologies Inc SB700/SB800 IDE Controller
> +-14.2 ATI Technologies Inc SBx00 Azalia (Intel HDA)
> +-14.3 ATI Technologies Inc SB700/SB800 LPC host controller
> +-14.4-[06]----07.0 RaLink RT2800 802.11n PCI
> +-14.5 ATI Technologies Inc SB700/SB800 USB OHCI2 Controller
> +-15.0-[07]--
> +-16.0 ATI Technologies Inc SB700/SB800 USB OHCI0 Controller
> +-16.2 ATI Technologies Inc SB700/SB800 USB EHCI Controller
> +-18.0 Advanced Micro Devices [AMD] Device 1600
> +-18.1 Advanced Micro Devices [AMD] Device 1601
> +-18.2 Advanced Micro Devices [AMD] Device 1602
> +-18.3 Advanced Micro Devices [AMD] Device 1603
> +-18.4 Advanced Micro Devices [AMD] Device 1604
> \-18.5 Advanced Micro Devices [AMD] Device 1605
>
> The relevant devices are (for problem 1):
>
> 06:07.0 Network controller: RaLink RT2800 802.11n PCI
> Subsystem: Linksys Device 0067
> Flags: bus master, slow devsel, latency 32, IRQ 21
> Memory at fd8e0000 (32-bit, non-prefetchable) [size=64K]
> Capabilities: [40] Power Management version 3
> Kernel driver in use: rt2800pci
>
> with the PCI-PCI bridge above:
>
> 00:14.4 PCI bridge: ATI Technologies Inc SBx00 PCI to PCI Bridge (rev 40) (prog-if 01 [Subtractive decode])
> Flags: bus master, VGA palette snoop, 66MHz, medium devsel, latency 64
> Bus: primary=00, secondary=06, subordinate=06, sec-latency=64
> I/O behind bridge: 00009000-00009fff
> Memory behind bridge: fd800000-fd8fffff
> Prefetchable memory behind bridge: fd700000-fd7fffff
>
>
>
> and (for problem 2):
>
> 03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 01)
> Subsystem: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller
> Flags: bus master, fast devsel, latency 0, IRQ 10
> I/O ports at de00 [size=256]
> Memory at fdbff000 (64-bit, non-prefetchable) [size=4K]
> [virtual] Expansion ROM at fda00000 [disabled] [size=128K]
> Capabilities: [40] Power Management version 2
> Capabilities: [48] Vital Product Data
> Capabilities: [50] MSI: Enable- Count=1/2 Maskable- 64bit+
> Capabilities: [60] Express Endpoint, MSI 00
> Capabilities: [84] Vendor Specific Information: Len=4c <?>
> Capabilities: [100] Advanced Error Reporting
> Capabilities: [12c] Virtual Channel
> Capabilities: [148] Device Serial Number f1-11-00-00-68-4c-e0-00
> Capabilities: [154] Power Budgeting <?>
>
> 04:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 06)
> Subsystem: Giga-byte Technology GA-EP45-DS5 Motherboard
> Flags: bus master, fast devsel, latency 0, IRQ 10
> I/O ports at ce00 [size=256]
> Memory at fd6ff000 (64-bit, prefetchable) [size=4K]
> Memory at fd6f8000 (64-bit, prefetchable) [size=16K]
> Capabilities: [40] Power Management version 3
> Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
> Capabilities: [70] Express Endpoint, MSI 01
> Capabilities: [b0] MSI-X: Enable- Count=4 Masked-
> Capabilities: [d0] Vital Product Data
> Capabilities: [100] Advanced Error Reporting
> Capabilities: [140] Virtual Channel
> Capabilities: [160] Device Serial Number 12-34-56-78-12-34-56-78
>
>
> ============================================================================================================================
>
>
> 1. problem:
>
> Wlan RaLink RT2800 802.11n is a legacy PCI device behind
> a PCI-PCI bridge and should be assigned to VM_a, but passthrough doesn't
> work because libvirt claims, that the device or resource would be busy
> (but the device is unbound correctly).
>
> There is no other device in the tree under the bridge (for details
> please see the attached log file).
>
> This is the entry in the xml-file I use:
>
> <hostdev mode='subsystem' type='pci' managed='yes'>
> <source>
> <address domain='0x0000' bus='0x06' slot='0x07' function='0x0'/>
> </source>
> </hostdev>
>
> The VM is started with "virsh start VM_a".
>
> I was able to start VM_a if the PCI-PCI bridge was unbound manually
> before VM_a was started (yes, I really know, that libvirt does the
> unbind job for me - but it doesn't work as expected :-).
> The bridge itself wasn't assigned to VM_a):
>
> echo "1002 4384" > /sys/bus/pci/drivers/pci-stub/new_id
> echo "0000:00:14.4" > /sys/bus/pci/devices/0000:00:14.4/driver/unbind
>
> Unfortunately, the host is locked up on shutdown of VM_a.
>
> That's the trace of the locked host:
>
> amd_iommu_domain_destroy
> iommu_domain_free
> kvm_iommu_unmap_guest
> kvm_arch_destroy_vm
> kvm_destroy_vm
> kvm_vcpu_release
> __fput
> flip_close
> put_files_struct
> do_exit
> do_group_exit
> get_signal_to_deliver
> do_signal
> do_notify_resume
> int_signal
>
>
> But I think this should be a valid setup according [1]:
>
> "Notes:
> VT-d spec specifies that all conventional PCI devices behind a PCIe-to
> PCI/PCI-X bridge or conventional PCI bridge can only be collectively
> assigned to the same guest. PCIe devices do not have this restriction."
>
> [1] http://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM
>
>
>
>
> 2. problem:
>
> Next I tried to assign the PCIe ethernet card, TP-Link TG-3468
> (device 03:00.0), to VM_b.
>
> This card does have exactly the same PCIID as the onboard network
> interface. If I put it (or even both ethernet network devices) to VM_b,
> the machine is locked up completely directly on start up of the "virsh
> start VM_b" command.
>
> Problem seems to be. that the kernel (-> for pci passthrough) can't
> distinguish those two devices because of their same PCIID (there were
> always both devices unbound, even if there should be just one device
> unound).
>
> The xml-entry is:
>
> <hostdev mode='subsystem' type='pci' managed='yes'>
> <source>
> <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
> </source>
> </hostdev>
>
> If I remove the additional PCIe TP-Link card, I'm able to passthrough
> the onboard ethernet card to VM_b without any problem.
>
>
>
>
> I hope I could clarify now the two problems I encounter.
>
>
> Thank you very much for your time!
> Andreas
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2011-12-20 8:34 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2011-12-12 11:15 [AMD iommu] pci Failed to assign device "hostdev0" : Device or resource busy Andreas Hartmann
2011-12-12 18:36 ` Don Dutile
2011-12-13 0:21 ` Andreas Hartmann
2011-12-13 8:40 ` Andreas Hartmann
2011-12-19 20:32 ` Don Dutile
[not found] ` <20111220000002.0efc3ec1@dualc.maya.org>
2011-12-20 8:31 ` Andreas Hartmann
2011-12-13 9:40 ` Andreas Hartmann
2011-12-13 14:16 ` Andreas Hartmann
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).