Query on IOMMU

Query on IOMMU

[Prasad Joshi]

Hello,

I am facing a problem with enabling the IOMMU.

Dec 21 15:50:57 prasad-kvm kernel: [    0.000000] Aperture pointing to e820 RAM. Ignoring.
Dec 21 15:50:57 prasad-kvm kernel: [    0.000000] Your BIOS doesn't leave a aperture memory hole
Dec 21 15:50:57 prasad-kvm kernel: [    0.000000] Please enable the IOMMU option in the BIOS setup

Dec 21 15:50:57 prasad-kvm kernel: [    2.790913] pci 0000:01:05.0: Firmware left e100 interrupts enabled; disabling
Dec 21 15:50:57 prasad-kvm kernel: [    2.791941] pci 0000:00:00.2: PCI INT A -> GSI 55 (level, low) -> IRQ 55
Dec 21 15:50:57 prasad-kvm kernel: [    2.792775] AMD-Vi: Enabling IOMMU at 0000:00:00.2 cap 0x40
Dec 21 15:50:57 prasad-kvm kernel: [    2.800989] AMD-Vi: Lazy IO/TLB flushing enabled


I have enabled IOMMU in the BIOS, but I am not sure why it is still asking to enabled IOMMU in BIOS. Do I need to worry about this?


Besides I don't see the DMAR message similar to the one mentioned on the link 
http://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM

prasad@prasad-kvm:~/KVM/kvm$ grep DMAR .config
CONFIG_DMAR=y
CONFIG_DMAR_DEFAULT_ON=y
CONFIG_DMAR_FLOPPY_WA=y
prasad@prasad-kvm:~/KVM/kvm$ dmesg | grep -i dmar
prasad@prasad-kvm:~/KVM/kvm$ 

Is there any problem with IOMMU?

Thanks and Regards,
Prasad Joshi (Student ID: 19015523)
MRes Systems Engineering

Re: Query on IOMMU

[Chris Wright]

* Prasad Joshi (P.G.Joshi@student.reading.ac.uk) wrote:
> I am facing a problem with enabling the IOMMU.
> 
> Dec 21 15:50:57 prasad-kvm kernel: [    0.000000] Aperture pointing to e820 RAM. Ignoring.
> Dec 21 15:50:57 prasad-kvm kernel: [    0.000000] Your BIOS doesn't leave a aperture memory hole
> Dec 21 15:50:57 prasad-kvm kernel: [    0.000000] Please enable the IOMMU option in the BIOS setup
> 
> Dec 21 15:50:57 prasad-kvm kernel: [    2.790913] pci 0000:01:05.0: Firmware left e100 interrupts enabled; disabling
> Dec 21 15:50:57 prasad-kvm kernel: [    2.791941] pci 0000:00:00.2: PCI INT A -> GSI 55 (level, low) -> IRQ 55
> Dec 21 15:50:57 prasad-kvm kernel: [    2.792775] AMD-Vi: Enabling IOMMU at 0000:00:00.2 cap 0x40
> Dec 21 15:50:57 prasad-kvm kernel: [    2.800989] AMD-Vi: Lazy IO/TLB flushing enabled
> 
> 
> I have enabled IOMMU in the BIOS, but I am not sure why it is still asking to enabled IOMMU in BIOS. Do I need to worry about this?

It's unfortunate wording.  It's telling you that the GART is missing,
which is fine because you have an IOMMU.

> Besides I don't see the DMAR message similar to the one mentioned on the link 
> http://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM

That wiki page is specific to Intel VT-d.  You have an AMD box with IOMMU,
so all looks fine.

Are you interested in using the IOMMU to do direct PCI device assignment
to a guest?

thanks,
-chris

RE: Query on IOMMU

[Prasad Joshi]

> From: Chris Wright [chrisw@sous-sol.org]
> Sent: 21 December 2010 16:15
> To: Prasad Joshi
> Cc: kvm@vger.kernel.org
> Subject: Re: Query on IOMMU

>> I have enabled IOMMU in the BIOS, but I am not sure why it is still asking to enabled IOMMU in BIOS. Do I need to worry about this?

> It's unfortunate wording.  It's telling you that the GART is missing,
> which is fine because you have an IOMMU.

>> Besides I don't see the DMAR message similar to the one mentioned on the link
>> http://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM

> That wiki page is specific to Intel VT-d.  You have an AMD box with IOMMU,
> so all looks fine.

Yes I am using AMD processor and ASUS motherboard. Both of them have the IOMMU support, atleast it is mentioned on the Xen VT-d

> Are you interested in using the IOMMU to do direct PCI device assignment
> to a guest?

Thanks a lot for your reply. Yes I am interested in working on GPU pass-through to Virtual Machine. But for now I am trying to pass-through a network card to VM.

root@prasad-kvm:~/VMDisks# qemu-system-x86_64 -hda Ubuntu-10.10-amd64.img -m 1024M -device pci-assign,host=01:05.0
Failed to assign device "(null)" : Device or resource busy
*** The driver 'pci-stub' is occupying your device 0000:01:05.0.
***
*** You can try the following commands to free it:
***
*** $ echo "8086 1229" > /sys/bus/pci/drivers/pci-stub/new_id
*** $ echo "0000:01:05.0" > /sys/bus/pci/drivers/pci-stub/unbind
*** $ echo "0000:01:05.0" > /sys/bus/pci/drivers/pci-stub/bind
*** $ echo "8086 1229" > /sys/bus/pci/drivers/pci-stub/remove_id
***
qemu-system-x86_64: -device pci-assign,host=01:05.0: Device 'pci-assign' could not be initialized
root@prasad-kvm:~/VMDisks# echo "8086 1229" > /sys/bus/pci/drivers/pci-stub/new_id
root@prasad-kvm:~/VMDisks# echo "0000:01:05.0" > /sys/bus/pci/drivers/pci-stub/unbind
root@prasad-kvm:~/VMDisks# echo "0000:01:05.0" > /sys/bus/pci/drivers/pci-stub/bind
root@prasad-kvm:~/VMDisks# echo "8086 1229" > /sys/bus/pci/drivers/pci-stub/remove_id
root@prasad-kvm:~/VMDisks# qemu-system-x86_64 -hda Ubuntu-10.10-amd64.img -m 1024M -device pci-assign,host=01:05.0
Failed to assign device "(null)" : Device or resource busy
*** The driver 'pci-stub' is occupying your device 0000:01:05.0.


[  605.015852] e100 0000:01:05.0: BAR 0: can't reserve [mem 0xf9cff000-0xf9cfffff]
[  605.015855] kvm_vm_ioctl_assign_device: Could not get access to device regions
[  667.410228] e100 0000:01:05.0: PCI INT A disabled
[  700.500278] pci-stub: invalid id string ""
[  707.730636] pci-stub 0000:01:05.0: claimed by stub
[  734.755491] pci-stub 0000:01:05.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
[  734.790077] pci-stub 0000:01:05.0: restoring config space at offset 0xf (was 0x38080100, writing 0x3808010b)
[  734.790095] pci-stub 0000:01:05.0: restoring config space at offset 0xc (was 0x0, writing 0xf9ce0000)
[  734.790113] pci-stub 0000:01:05.0: restoring config space at offset 0x6 (was 0x0, writing 0xf9cc0000)
[  734.790123] pci-stub 0000:01:05.0: restoring config space at offset 0x5 (was 0x1, writing 0xac01)
[  734.790132] pci-stub 0000:01:05.0: restoring config space at offset 0x4 (was 0x0, writing 0xf9cff000)
[  734.790142] pci-stub 0000:01:05.0: restoring config space at offset 0x3 (was 0x0, writing 0x4010)
[  734.790153] pci-stub 0000:01:05.0: restoring config space at offset 0x1 (was 0x2900000, writing 0x2900113)
[  735.173647] assign device 0:1:5.0 failed
[  735.173688] pci-stub 0000:01:05.0: PCI INT A disabled
[  768.850519] pci-stub 0000:01:05.0: claimed by stub
[  775.855376] pci-stub 0000:01:05.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
[  775.890080] pci-stub 0000:01:05.0: restoring config space at offset 0xf (was 0x38080100, writing 0x3808010b)
[  775.890097] pci-stub 0000:01:05.0: restoring config space at offset 0xc (was 0x0, writing 0xf9ce0000)
[  775.890115] pci-stub 0000:01:05.0: restoring config space at offset 0x6 (was 0x0, writing 0xf9cc0000)
[  775.890126] pci-stub 0000:01:05.0: restoring config space at offset 0x5 (was 0x1, writing 0xac01)
[  775.890135] pci-stub 0000:01:05.0: restoring config space at offset 0x4 (was 0x0, writing 0xf9cff000)
[  775.890144] pci-stub 0000:01:05.0: restoring config space at offset 0x3 (was 0x0, writing 0x4010)
[  775.890155] pci-stub 0000:01:05.0: restoring config space at offset 0x1 (was 0x2900000, writing 0x2900113)
[  776.275188] assign device 0:1:5.0 failed
[  776.275230] pci-stub 0000:01:05.0: PCI INT A disabled

The device was previously owned by e100, now it shows it is owned by pci_stub. Initially the pci_stub.ko module was not loaded, I had to load it manually so that these assignment commands would work.


> thanks,
> -chris

Re: Query on IOMMU

[Chris Wright]

* Prasad Joshi (P.G.Joshi@student.reading.ac.uk) wrote:
> > From: Chris Wright [chrisw@sous-sol.org]
> 
> >> I have enabled IOMMU in the BIOS, but I am not sure why it is still asking to enabled IOMMU in BIOS. Do I need to worry about this?
> 
> > It's unfortunate wording.  It's telling you that the GART is missing,
> > which is fine because you have an IOMMU.
> 
> >> Besides I don't see the DMAR message similar to the one mentioned on the link
> >> http://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM
> 
> > That wiki page is specific to Intel VT-d.  You have an AMD box with IOMMU,
> > so all looks fine.
> 
> Yes I am using AMD processor and ASUS motherboard. Both of them have the IOMMU support, atleast it is mentioned on the Xen VT-d

Looks like we need some additional info in the wiki.  Care to create an
account and add the info?

> > Are you interested in using the IOMMU to do direct PCI device assignment
> > to a guest?
> 
> Thanks a lot for your reply. Yes I am interested in working on GPU pass-through to Virtual Machine. But for now I am trying to pass-through a network card to VM.
> 
> root@prasad-kvm:~/VMDisks# qemu-system-x86_64 -hda Ubuntu-10.10-amd64.img -m 1024M -device pci-assign,host=01:05.0
> Failed to assign device "(null)" : Device or resource busy
> *** The driver 'pci-stub' is occupying your device 0000:01:05.0.
> ***
> *** You can try the following commands to free it:
> ***
> *** $ echo "8086 1229" > /sys/bus/pci/drivers/pci-stub/new_id
> *** $ echo "0000:01:05.0" > /sys/bus/pci/drivers/pci-stub/unbind
> *** $ echo "0000:01:05.0" > /sys/bus/pci/drivers/pci-stub/bind
> *** $ echo "8086 1229" > /sys/bus/pci/drivers/pci-stub/remove_id
> ***

Heh, this error is a little odd.  It's telling you the pci-stub
driver already has this device.  Then it's telling you to unbind it
from pci-stub, and bind it to pci-stub.  That error message is meant to
tell you that the real host driver (in your case e100) has the device,
unbind from it, and bind to pci-stub.

> qemu-system-x86_64: -device pci-assign,host=01:05.0: Device 'pci-assign' could not be initialized
> root@prasad-kvm:~/VMDisks# echo "8086 1229" > /sys/bus/pci/drivers/pci-stub/new_id
> root@prasad-kvm:~/VMDisks# echo "0000:01:05.0" > /sys/bus/pci/drivers/pci-stub/unbind
> root@prasad-kvm:~/VMDisks# echo "0000:01:05.0" > /sys/bus/pci/drivers/pci-stub/bind
> root@prasad-kvm:~/VMDisks# echo "8086 1229" > /sys/bus/pci/drivers/pci-stub/remove_id
> root@prasad-kvm:~/VMDisks# qemu-system-x86_64 -hda Ubuntu-10.10-amd64.img -m 1024M -device pci-assign,host=01:05.0
> Failed to assign device "(null)" : Device or resource busy
> *** The driver 'pci-stub' is occupying your device 0000:01:05.0.
> 
> 
> [  605.015852] e100 0000:01:05.0: BAR 0: can't reserve [mem 0xf9cff000-0xf9cfffff]
> [  605.015855] kvm_vm_ioctl_assign_device: Could not get access to device regions

This is what is returning -EBUSY and triggering the error message.

> [  667.410228] e100 0000:01:05.0: PCI INT A disabled
> [  700.500278] pci-stub: invalid id string ""
> [  707.730636] pci-stub 0000:01:05.0: claimed by stub
> [  734.755491] pci-stub 0000:01:05.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
> [  734.790077] pci-stub 0000:01:05.0: restoring config space at offset 0xf (was 0x38080100, writing 0x3808010b)
> [  734.790095] pci-stub 0000:01:05.0: restoring config space at offset 0xc (was 0x0, writing 0xf9ce0000)
> [  734.790113] pci-stub 0000:01:05.0: restoring config space at offset 0x6 (was 0x0, writing 0xf9cc0000)
> [  734.790123] pci-stub 0000:01:05.0: restoring config space at offset 0x5 (was 0x1, writing 0xac01)
> [  734.790132] pci-stub 0000:01:05.0: restoring config space at offset 0x4 (was 0x0, writing 0xf9cff000)
> [  734.790142] pci-stub 0000:01:05.0: restoring config space at offset 0x3 (was 0x0, writing 0x4010)
> [  734.790153] pci-stub 0000:01:05.0: restoring config space at offset 0x1 (was 0x2900000, writing 0x2900113)
> [  735.173647] assign device 0:1:5.0 failed
> [  735.173688] pci-stub 0000:01:05.0: PCI INT A disabled
> [  768.850519] pci-stub 0000:01:05.0: claimed by stub
> [  775.855376] pci-stub 0000:01:05.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
> [  775.890080] pci-stub 0000:01:05.0: restoring config space at offset 0xf (was 0x38080100, writing 0x3808010b)
> [  775.890097] pci-stub 0000:01:05.0: restoring config space at offset 0xc (was 0x0, writing 0xf9ce0000)
> [  775.890115] pci-stub 0000:01:05.0: restoring config space at offset 0x6 (was 0x0, writing 0xf9cc0000)
> [  775.890126] pci-stub 0000:01:05.0: restoring config space at offset 0x5 (was 0x1, writing 0xac01)
> [  775.890135] pci-stub 0000:01:05.0: restoring config space at offset 0x4 (was 0x0, writing 0xf9cff000)
> [  775.890144] pci-stub 0000:01:05.0: restoring config space at offset 0x3 (was 0x0, writing 0x4010)
> [  775.890155] pci-stub 0000:01:05.0: restoring config space at offset 0x1 (was 0x2900000, writing 0x2900113)
> [  776.275188] assign device 0:1:5.0 failed
> [  776.275230] pci-stub 0000:01:05.0: PCI INT A disabled
> 
> The device was previously owned by e100, now it shows it is owned by pci_stub. Initially the pci_stub.ko module was not loaded, I had to load it manually so that these assignment commands would work.

And it looks like it is not working yet.  Is that correct?

Can you start fresh (make sure the e100 is loaded and functional), and
do the following:

verify it's starting with e100
# ls -l /sys/bus/pci/devices/0000:01:05.0/driver    <-- should show e100

add device id to pci-stub driver
# modprobe pci-stub
# echo "8086 1229" > /sys/bus/pci/drivers/pci-stub/new_id

unbind e100 from device
# echo "0000:01:05.0" > /sys/bus/pci/drivers/e100/unbind

bind pci-stub to device
# echo "0000:01:05.0" > /sys/bus/pci/drivers/pci-stub/bind

remove the id from pci-stub so that subsequent probing will pick up e100
# echo "8086 1229" > /sys/bus/pci/drivers/pci-stub/remove_id

verify it worked
# ls -l /sys/bus/pci/devices/0000:01:05.0/driver    <-- should show pci-stub

Launch your VM...assigned device should show up in VM now.

thanks,
-chris

RE: Query on IOMMU

[Prasad Joshi]

> From: kvm-owner@vger.kernel.org [kvm-owner@vger.kernel.org] on behalf of Chris Wright [chrisw@sous-sol.org]
> Sent: 21 December 2010 17:12
> To: Prasad Joshi
> Cc: Chris Wright; kvm@vger.kernel.org
> Subject: Re: Query on IOMMU

>>* Prasad Joshi (P.G.Joshi@student.reading.ac.uk) wrote:
> > From: Chris Wright [chrisw@sous-sol.org]
>
> >> I have enabled IOMMU in the BIOS, but I am not sure why it is still asking to enabled IOMMU in BIOS. Do I need to worry about this?
>
> > It's unfortunate wording.  It's telling you that the GART is missing,
> > which is fine because you have an IOMMU.
>
> >> Besides I don't see the DMAR message similar to the one mentioned on the link
> >> http://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM
>
> > That wiki page is specific to Intel VT-d.  You have an AMD box with IOMMU,
> > so all looks fine.
>
> Yes I am using AMD processor and ASUS motherboard. Both of them have the IOMMU support, atleast it is mentioned on the Xen VT-d

> Looks like we need some additional info in the wiki.  Care to create an
> account and add the info?

Sure I would love to.

> > Are you interested in using the IOMMU to do direct PCI device assignment
> > to a guest?
>
> Thanks a lot for your reply. Yes I am interested in working on GPU pass-through to Virtual Machine. But for now I am trying to pass-through a network card to VM.
>
> root@prasad-kvm:~/VMDisks# qemu-system-x86_64 -hda Ubuntu-10.10-amd64.img -m 1024M -device pci-assign,host=01:05.0
> Failed to assign device "(null)" : Device or resource busy
> *** The driver 'pci-stub' is occupying your device 0000:01:05.0.
> ***
> *** You can try the following commands to free it:
> ***
> *** $ echo "8086 1229" > /sys/bus/pci/drivers/pci-stub/new_id
> *** $ echo "0000:01:05.0" > /sys/bus/pci/drivers/pci-stub/unbind
> *** $ echo "0000:01:05.0" > /sys/bus/pci/drivers/pci-stub/bind
> *** $ echo "8086 1229" > /sys/bus/pci/drivers/pci-stub/remove_id
> ***

> Heh, this error is a little odd.  It's telling you the pci-stub
> driver already has this device.  Then it's telling you to unbind it
> from pci-stub, and bind it to pci-stub.  That error message is meant to
> tell you that the real host driver (in your case e100) has the device,
> unbind from it, and bind to pci-stub.

> qemu-system-x86_64: -device pci-assign,host=01:05.0: Device 'pci-assign' could not be initialized
> root@prasad-kvm:~/VMDisks# echo "8086 1229" > /sys/bus/pci/drivers/pci-stub/new_id
> root@prasad-kvm:~/VMDisks# echo "0000:01:05.0" > /sys/bus/pci/drivers/pci-stub/unbind
> root@prasad-kvm:~/VMDisks# echo "0000:01:05.0" > /sys/bus/pci/drivers/pci-stub/bind
> root@prasad-kvm:~/VMDisks# echo "8086 1229" > /sys/bus/pci/drivers/pci-stub/remove_id
> root@prasad-kvm:~/VMDisks# qemu-system-x86_64 -hda Ubuntu-10.10-amd64.img -m 1024M -device pci-assign,host=01:05.0
> Failed to assign device "(null)" : Device or resource busy
> *** The driver 'pci-stub' is occupying your device 0000:01:05.0.
>
>
> [  605.015852] e100 0000:01:05.0: BAR 0: can't reserve [mem 0xf9cff000-0xf9cfffff]
> [  605.015855] kvm_vm_ioctl_assign_device: Could not get access to device regions

> This is what is returning -EBUSY and triggering the error message.

> The device was previously owned by e100, now it shows it is owned by pci_stub. Initially the pci_stub.ko module was not loaded, I had to load it manually so that these assignment commands would work.

> And it looks like it is not working yet.  Is that correct?

Yes this is still not working.

> Can you start fresh (make sure the e100 is loaded and functional), and
> do the following:

Better I rebooted the machine.

> verify it's starting with e100
> # ls -l /sys/bus/pci/devices/0000:01:05.0/driver    <-- should show e100

> add device id to pci-stub driver
> # modprobe pci-stub
> # echo "8086 1229" > /sys/bus/pci/drivers/pci-stub/new_id

> unbind e100 from device
> # echo "0000:01:05.0" > /sys/bus/pci/drivers/e100/unbind

> bind pci-stub to device
> # echo "0000:01:05.0" > /sys/bus/pci/drivers/pci-stub/bind

> remove the id from pci-stub so that subsequent probing will pick up e100
> # echo "8086 1229" > /sys/bus/pci/drivers/pci-stub/remove_id

> verify it worked
> # ls -l /sys/bus/pci/devices/0000:01:05.0/driver    <-- should show pci-stub

> Launch your VM...assigned device should show up in VM now.

It still fails with the same error, here is the screen shot.

root@prasad-kvm:/sys# uptime 
 17:29:11 up 2 min,  3 users,  load average: 0.93, 0.52, 0.20

root@prasad-kvm:/sys# ls -l /sys/bus/pci/devices/0000:01:05.0/driver
lrwxrwxrwx 1 root root 0 2010-12-21 17:26 /sys/bus/pci/devices/0000:01:05.0/driver -> ../../../../bus/pci/drivers/e100

root@prasad-kvm:/sys# lsmod | grep pci_stub

root@prasad-kvm:/sys# modprobe pci_stub

root@prasad-kvm:/sys# lsmod | grep pci_stub
pci_stub                1590  0 

root@prasad-kvm:/sys# echo "8086 1229" > /sys/bus/pci/drivers/pci-stub/new_id 

root@prasad-kvm:/sys# echo "0000:01:05.0" > /sys/bus/pci/drivers/e100/unbind 

root@prasad-kvm:/sys# echo "0000:01:05.0" > /sys/bus/pci/drivers/pci-stub/bind 

root@prasad-kvm:/sys# echo "8086 1229" > /sys/bus/pci/drivers/pci-stub/remove_id 

root@prasad-kvm:/sys# ls -l /sys/bus/pci/devices/0000:01:05.0/driver
lrwxrwxrwx 1 root root 0 2010-12-21 17:31 /sys/bus/pci/devices/0000:01:05.0/driver -> ../../../../bus/pci/drivers/pci-stub

root@prasad-kvm:~/VMDisks# modprobe kvm_amd

root@prasad-kvm:~/VMDisks# lsmod | grep -i kvm
kvm_amd                56416  0 
kvm                   348987  1 kvm_amd

root@prasad-kvm:~/VMDisks# qemu-system-x86_64 -hda Ubuntu-10.10-amd64.img -m 1024M -device pci-assign,host=01:05.0
Failed to assign device "(null)" : Device or resource busy
*** The driver 'pci-stub' is occupying your device 0000:01:05.0.
***
*** You can try the following commands to free it:
***
*** $ echo "8086 1229" > /sys/bus/pci/drivers/pci-stub/new_id
*** $ echo "0000:01:05.0" > /sys/bus/pci/drivers/pci-stub/unbind
*** $ echo "0000:01:05.0" > /sys/bus/pci/drivers/pci-stub/bind
*** $ echo "8086 1229" > /sys/bus/pci/drivers/pci-stub/remove_id
***
qemu-system-x86_64: -device pci-assign,host=01:05.0: Device 'pci-assign' could not be initialized
root@prasad-kvm:~/VMDisks# echo $?
1
root@prasad-kvm:~/VMDisks# 

The VM does not boot.

> thanks,
> -chris
> 

Re: Query on IOMMU

[Chris Wright]

* Prasad Joshi (P.G.Joshi@student.reading.ac.uk) wrote:
> > From: kvm-owner@vger.kernel.org [kvm-owner@vger.kernel.org] on behalf of Chris Wright [chrisw@sous-sol.org]
> > Yes I am using AMD processor and ASUS motherboard. Both of them have the IOMMU support, atleast it is mentioned on the Xen VT-d
> 
> > Looks like we need some additional info in the wiki.  Care to create an
> > account and add the info?
> 
> Sure I would love to.

Thanks, you can use the VT-d portion as an example.

The useful dmesg info will be AMD-Vi: messages, the important line
is this one:

AMD-Vi: Enabling IOMMU at ...

(and if you boot with amd_iommu_dump you'll get extra debugging info)

> > Thanks a lot for your reply. Yes I am interested in working on GPU pass-through to Virtual Machine. But for now I am trying to pass-through a network card to VM.

Great, GPU assignment has plenty of issues ;)

<snip>
> It still fails with the same error, here is the screen shot.
> 
> root@prasad-kvm:/sys# uptime 
>  17:29:11 up 2 min,  3 users,  load average: 0.93, 0.52, 0.20
> 
> root@prasad-kvm:/sys# ls -l /sys/bus/pci/devices/0000:01:05.0/driver
> lrwxrwxrwx 1 root root 0 2010-12-21 17:26 /sys/bus/pci/devices/0000:01:05.0/driver -> ../../../../bus/pci/drivers/e100
> 
> root@prasad-kvm:/sys# lsmod | grep pci_stub
> 
> root@prasad-kvm:/sys# modprobe pci_stub
> 
> root@prasad-kvm:/sys# lsmod | grep pci_stub
> pci_stub                1590  0 
> 
> root@prasad-kvm:/sys# echo "8086 1229" > /sys/bus/pci/drivers/pci-stub/new_id 
> 
> root@prasad-kvm:/sys# echo "0000:01:05.0" > /sys/bus/pci/drivers/e100/unbind 
> 
> root@prasad-kvm:/sys# echo "0000:01:05.0" > /sys/bus/pci/drivers/pci-stub/bind 
> 
> root@prasad-kvm:/sys# echo "8086 1229" > /sys/bus/pci/drivers/pci-stub/remove_id 
> 
> root@prasad-kvm:/sys# ls -l /sys/bus/pci/devices/0000:01:05.0/driver
> lrwxrwxrwx 1 root root 0 2010-12-21 17:31 /sys/bus/pci/devices/0000:01:05.0/driver -> ../../../../bus/pci/drivers/pci-stub
> 
> root@prasad-kvm:~/VMDisks# modprobe kvm_amd
> 
> root@prasad-kvm:~/VMDisks# lsmod | grep -i kvm
> kvm_amd                56416  0 
> kvm                   348987  1 kvm_amd
> 
> root@prasad-kvm:~/VMDisks# qemu-system-x86_64 -hda Ubuntu-10.10-amd64.img -m 1024M -device pci-assign,host=01:05.0
> Failed to assign device "(null)" : Device or resource busy
> *** The driver 'pci-stub' is occupying your device 0000:01:05.0.
> ***
> *** You can try the following commands to free it:
> ***
> *** $ echo "8086 1229" > /sys/bus/pci/drivers/pci-stub/new_id
> *** $ echo "0000:01:05.0" > /sys/bus/pci/drivers/pci-stub/unbind
> *** $ echo "0000:01:05.0" > /sys/bus/pci/drivers/pci-stub/bind
> *** $ echo "8086 1229" > /sys/bus/pci/drivers/pci-stub/remove_id
> ***
> qemu-system-x86_64: -device pci-assign,host=01:05.0: Device 'pci-assign' could not be initialized
> root@prasad-kvm:~/VMDisks# echo $?
> 1
> root@prasad-kvm:~/VMDisks# 
> 
> The VM does not boot.

Are you still seeing the same errors in dmesg?  Your first dmesg showed
that the e100 driver couldn't allocate BAR0:

e100 0000:01:05.0: BAR 0: can't reserve [mem 0xf9cff000-0xf9cfffff]

If the host driver can't, then kvm_vm_ioctl_assign_device() will fail as
well.  Seems as if there's a resource conflict on your machine.

Can you include a full dmesg, /proc/iomem, and lspci -vvv -xxxx?

thanks,
-chris

RE: Query on IOMMU

[Prasad Joshi]

From: kvm-owner@vger.kernel.org [kvm-owner@vger.kernel.org] on behalf of Chris Wright [chrisw@sous-sol.org]
Sent: 21 December 2010 18:33
To: Prasad Joshi
Cc: Chris Wright; kvm@vger.kernel.org
Subject: Re: Query on IOMMU

* Prasad Joshi (P.G.Joshi@student.reading.ac.uk) wrote:
> > From: kvm-owner@vger.kernel.org [kvm-owner@vger.kernel.org] on behalf of Chris Wright [chrisw@sous-sol.org]
> > Yes I am using AMD processor and ASUS motherboard. Both of them have the IOMMU support, atleast it is mentioned on the Xen VT-d
>
> > Looks like we need some additional info in the wiki.  Care to create an
> > account and add the info?
>
> Sure I would love to.

> Thanks, you can use the VT-d portion as an example.

> The useful dmesg info will be AMD-Vi: messages, the important line
> is this one:

> AMD-Vi: Enabling IOMMU at ...

> (and if you boot with amd_iommu_dump you'll get extra debugging info)

I will add this information ASAP

> > Thanks a lot for your reply. Yes I am interested in working on GPU pass-through to Virtual Machine. But for now I am trying to pass-through a network card to VM.

> Great, GPU assignment has plenty of issues ;)

<snip>
> It still fails with the same error, here is the screen shot.
>
> root@prasad-kvm:/sys# uptime
>  17:29:11 up 2 min,  3 users,  load average: 0.93, 0.52, 0.20
>
> root@prasad-kvm:/sys# ls -l /sys/bus/pci/devices/0000:01:05.0/driver
> lrwxrwxrwx 1 root root 0 2010-12-21 17:26 /sys/bus/pci/devices/0000:01:05.0/driver -> ../../../../bus/pci/drivers/e100
>
> root@prasad-kvm:/sys# lsmod | grep pci_stub
>
> root@prasad-kvm:/sys# modprobe pci_stub
>
> root@prasad-kvm:/sys# lsmod | grep pci_stub
> pci_stub                1590  0
>
> root@prasad-kvm:/sys# echo "8086 1229" > /sys/bus/pci/drivers/pci-stub/new_id
>
> root@prasad-kvm:/sys# echo "0000:01:05.0" > /sys/bus/pci/drivers/e100/unbind
>
> root@prasad-kvm:/sys# echo "0000:01:05.0" > /sys/bus/pci/drivers/pci-stub/bind
>
> root@prasad-kvm:/sys# echo "8086 1229" > /sys/bus/pci/drivers/pci-stub/remove_id
>
> root@prasad-kvm:/sys# ls -l /sys/bus/pci/devices/0000:01:05.0/driver
> lrwxrwxrwx 1 root root 0 2010-12-21 17:31 /sys/bus/pci/devices/0000:01:05.0/driver -> ../../../../bus/pci/drivers/pci-stub
>
> root@prasad-kvm:~/VMDisks# modprobe kvm_amd
>
> root@prasad-kvm:~/VMDisks# lsmod | grep -i kvm
> kvm_amd                56416  0
> kvm                   348987  1 kvm_amd
>
> root@prasad-kvm:~/VMDisks# qemu-system-x86_64 -hda Ubuntu-10.10-amd64.img -m 1024M -device pci-assign,host=01:05.0
> Failed to assign device "(null)" : Device or resource busy
> *** The driver 'pci-stub' is occupying your device 0000:01:05.0.
> ***
> *** You can try the following commands to free it:
> ***
> *** $ echo "8086 1229" > /sys/bus/pci/drivers/pci-stub/new_id
> *** $ echo "0000:01:05.0" > /sys/bus/pci/drivers/pci-stub/unbind
> *** $ echo "0000:01:05.0" > /sys/bus/pci/drivers/pci-stub/bind
> *** $ echo "8086 1229" > /sys/bus/pci/drivers/pci-stub/remove_id
> ***
> qemu-system-x86_64: -device pci-assign,host=01:05.0: Device 'pci-assign' could not be initialized
> root@prasad-kvm:~/VMDisks# echo $?
> 1
> root@prasad-kvm:~/VMDisks#
>
> The VM does not boot.

> Are you still seeing the same errors in dmesg?  Your first dmesg showed
> that the e100 driver couldn't allocate BAR0:

No I must have done something terribly wrong, the first time I ran those assignment commands.

> e100 0000:01:05.0: BAR 0: can't reserve [mem 0xf9cff000-0xf9cfffff]

> If the host driver can't, then kvm_vm_ioctl_assign_device() will fail as
> well.  Seems as if there's a resource conflict on your machine.

> Can you include a full dmesg, /proc/iomem, and lspci -vvv -xxxx?

Please note the files attached. Please ignore few messages in dmesg file, I was trying to debug the problem by adding few printks.

The following condition from __attach_device() returns the error.
static int __attach_device(struct device *dev,
               struct protection_domain *domain)
{
    ...
    if (alias_data->domain != NULL &&
        alias_data->domain != domain)
        goto out_unlock;
    ...
}

Besides when I insert the pci_stub module, it emits a messages 
[   49.197112] pci-stub: invalid id string ""
I don't know why?

Thanks and Regards,
Prasad

> thanks,
> -chris

Re: Query on IOMMU

[Chris Wright]

* Prasad Joshi (P.G.Joshi@student.reading.ac.uk) wrote:
> Besides when I insert the pci_stub module, it emits a messages 
> [   49.197112] pci-stub: invalid id string ""
> I don't know why?

It's just broken error message.  The commit b439b1d ("PCI: pci-stub: add
pci_stub.ids parameter") created that.  I looked at it very briefly a few
weeks ago and didn't see the issue.  It's cosmetic, and not related to
the failure you are seeing.

thanks,
-chris

RE: Query on IOMMU

[Prasad Joshi]

> From: Chris Wright [chrisw@sous-sol.org]
> Sent: 21 December 2010 19:29
> To: Prasad Joshi
> Cc: Chris Wright; kvm@vger.kernel.org; Tejun Heo
> Subject: Re: Query on IOMMU

> * Prasad Joshi (P.G.Joshi@student.reading.ac.uk) wrote:
>> Besides when I insert the pci_stub module, it emits a messages
>> [   49.197112] pci-stub: invalid id string ""
>> I don't know why?

> It's just broken error message.  The commit b439b1d ("PCI: pci-stub: add
> pci_stub.ids parameter") created that.  I looked at it very briefly a few
> weeks ago and didn't see the issue.  It's cosmetic, and not related to
> the failure you are seeing.

Is it okay to add a following line in section 4. unbind device from host kernel driver (example PCI device 01:00.0)

* If the PCI Stub Driver is compiled as module, then load the module using modprobe pci_stub.

When I compiled the kernel I selected it as a kernel module. As the driver was not loaded, I could not see the entries in /sys file system. I could figure that out after reading few things. It will good to add a note to mention this fact.

Let me know if I should add it or not.

> thanks,
>-chris

Re: Query on IOMMU

[Chris Wright]

* Prasad Joshi (P.G.Joshi@student.reading.ac.uk) wrote:
> > From: Chris Wright [chrisw@sous-sol.org]
> > Sent: 21 December 2010 19:29
> > To: Prasad Joshi
> > Cc: Chris Wright; kvm@vger.kernel.org; Tejun Heo
> > Subject: Re: Query on IOMMU
> 
> > * Prasad Joshi (P.G.Joshi@student.reading.ac.uk) wrote:
> >> Besides when I insert the pci_stub module, it emits a messages
> >> [   49.197112] pci-stub: invalid id string ""
> >> I don't know why?
> 
> > It's just broken error message.  The commit b439b1d ("PCI: pci-stub: add
> > pci_stub.ids parameter") created that.  I looked at it very briefly a few
> > weeks ago and didn't see the issue.  It's cosmetic, and not related to
> > the failure you are seeing.
> 
> Is it okay to add a following line in section 4. unbind device from host kernel driver (example PCI device 01:00.0)
> 
> * If the PCI Stub Driver is compiled as module, then load the module using modprobe pci_stub.
> 
> When I compiled the kernel I selected it as a kernel module. As the driver was not loaded, I could not see the entries in /sys file system. I could figure that out after reading few things. It will good to add a note to mention this fact.
> 
> Let me know if I should add it or not.

Yes, that sounds fine.

thanks,
-chris

RE: Query on IOMMU

[Prasad Joshi]

>>
>> Is it okay to add a following line in section 4. unbind device from host kernel driver (example PCI device 01:00.0)
>>
>> * If the PCI Stub Driver is compiled as module, then load the module using modprobe pci_stub.
>>
>> When I compiled the kernel I selected it as a kernel module. As the driver was not loaded, I could not see the entries in /sys file system. I > could figure that out after reading few things. It will good to add a note to mention this fact.
>>
>> Let me know if I should add it or not.
>
> Yes, that sounds fine.
>

Modified the wiki page.

Thanks and Regards,
Prasad

> thanks,
>-chris

[PATCH] pci-stub: ignore zero-length id parameters

[Tejun Heo]

pci-stub uses strsep() to separate list of ids and generates a warning
message when it fails to parse an id.  However, not specifying the
parameter results in ids set to an empty string.  strsep() happily
returns the empty string as the first token and thus triggers the
warning message spuriously.

Make the tokner ignore zero length ids.

Reported-by: Chris Wright <chrisw@sous-sol.org>
Reported-by: Prasad Joshi <P.G.Joshi@student.reading.ac.uk>
Cc: stable@kernel.org
---
 drivers/pci/pci-stub.c |    3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/pci/pci-stub.c b/drivers/pci/pci-stub.c
index f7b68ca..4ae494b 100644
--- a/drivers/pci/pci-stub.c
+++ b/drivers/pci/pci-stub.c
@@ -54,6 +54,9 @@ static int __init pci_stub_init(void)
 			subdevice = PCI_ANY_ID, class=0, class_mask=0;
 		int fields;
 
+		if (!strlen(id))
+			continue;
+
 		fields = sscanf(id, "%x:%x:%x:%x:%x:%x",
 				&vendor, &device, &subvendor, &subdevice,
 				&class, &class_mask);

Re: [PATCH] pci-stub: ignore zero-length id parameters

[Jesse Barnes]

On Wed, 22 Dec 2010 10:06:36 +0100
Tejun Heo <tj@kernel.org> wrote:

> pci-stub uses strsep() to separate list of ids and generates a warning
> message when it fails to parse an id.  However, not specifying the
> parameter results in ids set to an empty string.  strsep() happily
> returns the empty string as the first token and thus triggers the
> warning message spuriously.
> 
> Make the tokner ignore zero length ids.
> 
> Reported-by: Chris Wright <chrisw@sous-sol.org>
> Reported-by: Prasad Joshi <P.G.Joshi@student.reading.ac.uk>
> Cc: stable@kernel.org
> ---
>  drivers/pci/pci-stub.c |    3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/drivers/pci/pci-stub.c b/drivers/pci/pci-stub.c
> index f7b68ca..4ae494b 100644
> --- a/drivers/pci/pci-stub.c
> +++ b/drivers/pci/pci-stub.c
> @@ -54,6 +54,9 @@ static int __init pci_stub_init(void)
>  			subdevice = PCI_ANY_ID, class=0, class_mask=0;
>  		int fields;
>  
> +		if (!strlen(id))
> +			continue;
> +
>  		fields = sscanf(id, "%x:%x:%x:%x:%x:%x",
>  				&vendor, &device, &subvendor, &subdevice,
>  				&class, &class_mask);
> 

Applied to my linux-next branch, thanks.

-- 
Jesse Barnes, Intel Open Source Technology Center

Re: Query on IOMMU

[Chris Wright]

* Prasad Joshi (P.G.Joshi@student.reading.ac.uk) wrote:
> The following condition from __attach_device() returns the error.
> static int __attach_device(struct device *dev,
>                struct protection_domain *domain)
> {
>     ...
>     if (alias_data->domain != NULL &&
>         alias_data->domain != domain)
>         goto out_unlock;
>     ...
> }

That's the issue.  The IOMMU has a set of page tables for each DeviceID.
For most devices, the DeviceID is the same as the Bus:Dev.Func (the PCI
address) of the device.  But this does not always work.  One example is
when a device is behind a PCI-to-PCI Bridge.  In that case, the device
memory read/write requests (attempts to DMA) will appear as if they came
from the bridge.

00:14.4 PCI bridge: ATI Technologies Inc SBx00 PCI to PCI Bridge
        Bus: primary=00, secondary=01, subordinate=01, sec-latency=64

That's the bridge that sits between your e100 and the IOMMU.

thanks,
-chris

RE: Query on IOMMU

[Prasad Joshi]

I have few (may be stupid) questions on this 

> From: Chris Wright [chrisw@sous-sol.org]
> That's the issue.  The IOMMU has a set of page tables for each DeviceID.
> For most devices, the DeviceID is the same as the Bus:Dev.Func (the PCI
> address) of the device.  But this does not always work.  One example is
> when a device is behind a PCI-to-PCI Bridge.  In that case, the device
> memory read/write requests (attempts to DMA) will appear as if they came
> from the bridge.

Oh I see, I can understand this part.

> 
> 00:14.4 PCI bridge: ATI Technologies Inc SBx00 PCI to PCI Bridge
>         Bus: primary=00, secondary=01, subordinate=01, sec-latency=64
> 
> That's the bridge that sits between your e100 and the IOMMU.

Can you please explain how did you make out the device 01:05:0 is behind the bridge?
01:05.0 Ethernet controller: Intel Corporation 82557/8/9/0/1 Ethernet Pro 100 (rev 0c)

If you can explain this, I will try to find if the other network card also sits behind the bridge or not. I would like to know the same thing for the PCIe GPU card connected to my machine. If GPU card is also sitting behind the bridge then the hardware may be useless for the project. :(

Please explain how to find out this information.

> thanks,
> -chris

RE: Query on IOMMU

[Prasad Joshi]

Is the answer 

"All PCI buses located behind a PCI-PCI bridge must reside between the seondary bus number and the subordinate bus number (inclusive)."

00:14.4 PCI bridge: ATI Technologies Inc SBx00 PCI to PCI Bridge
                      Bus: primary=00, secondary=01, subordinate=01, sec-latency=64

So all the PCI devices between secondary (01) and subordinate (01) (in this case same) are behind the PCI Bridge. Correct me if I am wrong.

01:05.0 Ethernet controller: Intel Corporation 82557/8/9/0/1 Ethernet Pro 100 (rev 0c)
As Bus ID is 01 this ethernet controller is behind the PCI Bridge

03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 06)
As Bus: 03, I can assume this is not behind the PCI Bridge

But if subordinate would have been, say 03 or 04, then even this ethernet card (03:00:0) would be behind the PCI Bridge.

Am I correct?

Thanks and Regards,
Prasad

________________________________________
From: Prasad Joshi
Sent: 22 December 2010 13:16
To: Chris Wright
Cc: kvm@vger.kernel.org
Subject: RE: Query on IOMMU

I have few (may be stupid) questions on this

> From: Chris Wright [chrisw@sous-sol.org]
> That's the issue.  The IOMMU has a set of page tables for each DeviceID.
> For most devices, the DeviceID is the same as the Bus:Dev.Func (the PCI
> address) of the device.  But this does not always work.  One example is
> when a device is behind a PCI-to-PCI Bridge.  In that case, the device
> memory read/write requests (attempts to DMA) will appear as if they came
> from the bridge.

Oh I see, I can understand this part.

>
> 00:14.4 PCI bridge: ATI Technologies Inc SBx00 PCI to PCI Bridge
>         Bus: primary=00, secondary=01, subordinate=01, sec-latency=64
>
> That's the bridge that sits between your e100 and the IOMMU.

Can you please explain how did you make out the device 01:05:0 is behind the bridge?
01:05.0 Ethernet controller: Intel Corporation 82557/8/9/0/1 Ethernet Pro 100 (rev 0c)

If you can explain this, I will try to find if the other network card also sits behind the bridge or not. I would like to know the same thing for the PCIe GPU card connected to my machine. If GPU card is also sitting behind the bridge then the hardware may be useless for the project. :(

Please explain how to find out this information.

> thanks,
> -chris

Re: Query on IOMMU

[Chris Wright]

* Prasad Joshi (P.G.Joshi@student.reading.ac.uk) wrote:
> Is the answer 
> 
> "All PCI buses located behind a PCI-PCI bridge must reside between the seondary bus number and the subordinate bus number (inclusive)."
> 
> 00:14.4 PCI bridge: ATI Technologies Inc SBx00 PCI to PCI Bridge
>                       Bus: primary=00, secondary=01, subordinate=01, sec-latency=64
> 
> So all the PCI devices between secondary (01) and subordinate (01) (in this case same) are behind the PCI Bridge. Correct me if I am wrong.

That's correct.  You'll find secondary < subordinate when there's
another bridge downstream.

> 01:05.0 Ethernet controller: Intel Corporation 82557/8/9/0/1 Ethernet Pro 100 (rev 0c)
> As Bus ID is 01 this ethernet controller is behind the PCI Bridge

Yup.

> 03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 06)
> As Bus: 03, I can assume this is not behind the PCI Bridge
> 
> But if subordinate would have been, say 03 or 04, then even this ethernet card (03:00:0) would be behind the PCI Bridge.
> 
> Am I correct?

That's right.

thanks,
-chris

Re: Query on IOMMU

[Chris Wright]

* Prasad Joshi (P.G.Joshi@student.reading.ac.uk) wrote:
> I have few (may be stupid) questions on this 
> 
> > From: Chris Wright [chrisw@sous-sol.org]
> > That's the issue.  The IOMMU has a set of page tables for each DeviceID.
> > For most devices, the DeviceID is the same as the Bus:Dev.Func (the PCI
> > address) of the device.  But this does not always work.  One example is
> > when a device is behind a PCI-to-PCI Bridge.  In that case, the device
> > memory read/write requests (attempts to DMA) will appear as if they came
> > from the bridge.
> 
> Oh I see, I can understand this part.
> 
> > 
> > 00:14.4 PCI bridge: ATI Technologies Inc SBx00 PCI to PCI Bridge
> >         Bus: primary=00, secondary=01, subordinate=01, sec-latency=64
> > 
> > That's the bridge that sits between your e100 and the IOMMU.
> 
> Can you please explain how did you make out the device 01:05:0 is behind the bridge?
> 01:05.0 Ethernet controller: Intel Corporation 82557/8/9/0/1 Ethernet Pro 100 (rev 0c)

A PCI bridge has config space that states what busses are behind it.
The bridge at 00:14.4 is a bridge between bus 0 and bus 1, you can tell
from this line:

         Bus: primary=00, secondary=01, subordinate=01, sec-latency=64

There are no other devices behind that bridge (so theoretically you could
safely use it for device assignment).

> If you can explain this, I will try to find if the other network card
> also sits behind the bridge or not.

The other network interface card you have (03:00.0) is a PCIe device,
it's upstream is the PCIe port.  It should not have the aliasing issue,
and should work.

00:06.0 PCI bridge: ATI Technologies Inc RD890 PCI to PCI bridge (PCI express gpp port F)
        Bus: primary=00, secondary=03, subordinate=03, sec-latency=0

03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd.  RTL8111/8168B PCI Express Gigabit Ethernet controller

> I would like to know the same thing
> for the PCIe GPU card connected to my machine. If GPU card is also sitting
> behind the bridge then the hardware may be useless for the project. :(

The GPU is also in a PCIe port, here:

00:02.0 PCI bridge: ATI Technologies Inc RD890 PCI to PCI bridge (PCI express gpp port B)
        Bus: primary=00, secondary=06, subordinate=06, sec-latency=0

06:00.0 VGA compatible controller: nVidia Corporation G86 [Quadro NVS 290]

> Please explain how to find out this information.

Using lspci -t you can see the topology pretty easily.  Otherwise you
can sift through lspci output to find the topology.

thanks,
-chris

RE: Query on IOMMU

[Prasad Joshi]

> From: Chris Wright [chrisw@sous-sol.org]
> I would like to know the same thing
> for the PCIe GPU card connected to my machine. If GPU card is also sitting
> behind the bridge then the hardware may be useless for the project. :(

> The GPU is also in a PCIe port, here:

> 00:02.0 PCI bridge: ATI Technologies Inc RD890 PCI to PCI bridge (PCI express gpp port B)
>        Bus: primary=00, secondary=06, subordinate=06, sec-latency=0

> 06:00.0 VGA compatible controller: nVidia Corporation G86 [Quadro NVS 290]

As the secondary and subordinate are 06, it means GPU pass through won't work.

>> Please explain how to find out this information.

> Using lspci -t you can see the topology pretty easily.  Otherwise you
> can sift through lspci output to find the topology.

Thanks a lot Chris for explaining everything.

Regards,
Prasad

> thanks,
> -chris

Re: Query on IOMMU

[Chris Wright]

* Prasad Joshi (P.G.Joshi@student.reading.ac.uk) wrote:
> > From: Chris Wright [chrisw@sous-sol.org]
> > I would like to know the same thing
> > for the PCIe GPU card connected to my machine. If GPU card is also sitting
> > behind the bridge then the hardware may be useless for the project. :(
> 
> > The GPU is also in a PCIe port, here:
> 
> > 00:02.0 PCI bridge: ATI Technologies Inc RD890 PCI to PCI bridge (PCI express gpp port B)
> >        Bus: primary=00, secondary=06, subordinate=06, sec-latency=0
> 
> > 06:00.0 VGA compatible controller: nVidia Corporation G86 [Quadro NVS 290]
> 
> As the secondary and subordinate are 06, it means GPU pass through won't work.

No, it just means there are nor more bridges behind 00:02.0.  The GPU is
a PCIe device in a PCIe port (which happens to look a lot like a
bridge).  So, while GPU assignment has some tricky issues, I don't think
you'll be stopped by the IOMMU.

> >> Please explain how to find out this information.
> 
> > Using lspci -t you can see the topology pretty easily.  Otherwise you
> > can sift through lspci output to find the topology.
> 
> Thanks a lot Chris for explaining everything.

You're welcome.  Good luck with your project.

thanks,
-chris

RE: Query on IOMMU

[Prasad Joshi]

> From: Chris Wright [chrisw@sous-sol.org]
>>
>> > 00:02.0 PCI bridge: ATI Technologies Inc RD890 PCI to PCI bridge (PCI express gpp port B)
>> >        Bus: primary=00, secondary=06, subordinate=06, sec-latency=0
>>
>> > 06:00.0 VGA compatible controller: nVidia Corporation G86 [Quadro NVS 290]
>>
>> As the secondary and subordinate are 06, it means GPU pass through won't work.
>
>No, it just means there are nor more bridges behind 00:02.0.  The GPU is
>a PCIe device in a PCIe port (which happens to look a lot like a
>bridge).  So, while GPU assignment has some tricky issues, I don't think
>you'll be stopped by the IOMMU.

Great thanks Chris. Like you said lspci -t -vvv explains it very well.

>
>> >> Please explain how to find out this information.
>>
>> > Using lspci -t you can see the topology pretty easily.  Otherwise you
>> > can sift through lspci output to find the topology.
>>
>> Thanks a lot Chris for explaining everything.
>
>You're welcome.  Good luck with your project.

Thanks a lot. I will keep on posting on the mailing list whenever I need help.

>
>thanks,
>-chris

RE: Query on IOMMU

[Prasad Joshi]

> From: Prasad Joshi
>>The GPU is
>>a PCIe device in a PCIe port (which happens to look a lot like a
>>bridge).  So, while GPU assignment has some tricky issues, I don't think
>>you'll be stopped by the IOMM

Just for fun, I assigned the Nvidia Graphics Card in pass-through mode to the VM. As this is the only graphics card that I have, the host OS display was stopped. I could see the QEMU window appearing but nothing happened after that, as expected.

I will try to connect one more GPU card to see what happens.

>Great thanks Chris. Like you said lspci -t -vvv explains it very well.

>>
>>> >> Please explain how to find out this information.
>>>
>>> > Using lspci -t you can see the topology pretty easily.  Otherwise you
>>> > can sift through lspci output to find the topology.
>>>
>>> Thanks a lot Chris for explaining everything.
>>
>>You're welcome.  Good luck with your project.

>Thanks a lot. I will keep on posting on the mailing list whenever I need help.

>>
>>thanks,
>>-chris