Re: Query on IOMMU

[Chris Wright <chrisw@sous-sol.org>]

* Prasad Joshi (P.G.Joshi@student.reading.ac.uk) wrote:
> > From: Chris Wright [chrisw@sous-sol.org]
> 
> >> I have enabled IOMMU in the BIOS, but I am not sure why it is still asking to enabled IOMMU in BIOS. Do I need to worry about this?
> 
> > It's unfortunate wording.  It's telling you that the GART is missing,
> > which is fine because you have an IOMMU.
> 
> >> Besides I don't see the DMAR message similar to the one mentioned on the link
> >> http://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM
> 
> > That wiki page is specific to Intel VT-d.  You have an AMD box with IOMMU,
> > so all looks fine.
> 
> Yes I am using AMD processor and ASUS motherboard. Both of them have the IOMMU support, atleast it is mentioned on the Xen VT-d

Looks like we need some additional info in the wiki.  Care to create an
account and add the info?

> > Are you interested in using the IOMMU to do direct PCI device assignment
> > to a guest?
> 
> Thanks a lot for your reply. Yes I am interested in working on GPU pass-through to Virtual Machine. But for now I am trying to pass-through a network card to VM.
> 
> root@prasad-kvm:~/VMDisks# qemu-system-x86_64 -hda Ubuntu-10.10-amd64.img -m 1024M -device pci-assign,host=01:05.0
> Failed to assign device "(null)" : Device or resource busy
> *** The driver 'pci-stub' is occupying your device 0000:01:05.0.
> ***
> *** You can try the following commands to free it:
> ***
> *** $ echo "8086 1229" > /sys/bus/pci/drivers/pci-stub/new_id
> *** $ echo "0000:01:05.0" > /sys/bus/pci/drivers/pci-stub/unbind
> *** $ echo "0000:01:05.0" > /sys/bus/pci/drivers/pci-stub/bind
> *** $ echo "8086 1229" > /sys/bus/pci/drivers/pci-stub/remove_id
> ***

Heh, this error is a little odd.  It's telling you the pci-stub
driver already has this device.  Then it's telling you to unbind it
from pci-stub, and bind it to pci-stub.  That error message is meant to
tell you that the real host driver (in your case e100) has the device,
unbind from it, and bind to pci-stub.

> qemu-system-x86_64: -device pci-assign,host=01:05.0: Device 'pci-assign' could not be initialized
> root@prasad-kvm:~/VMDisks# echo "8086 1229" > /sys/bus/pci/drivers/pci-stub/new_id
> root@prasad-kvm:~/VMDisks# echo "0000:01:05.0" > /sys/bus/pci/drivers/pci-stub/unbind
> root@prasad-kvm:~/VMDisks# echo "0000:01:05.0" > /sys/bus/pci/drivers/pci-stub/bind
> root@prasad-kvm:~/VMDisks# echo "8086 1229" > /sys/bus/pci/drivers/pci-stub/remove_id
> root@prasad-kvm:~/VMDisks# qemu-system-x86_64 -hda Ubuntu-10.10-amd64.img -m 1024M -device pci-assign,host=01:05.0
> Failed to assign device "(null)" : Device or resource busy
> *** The driver 'pci-stub' is occupying your device 0000:01:05.0.
> 
> 
> [  605.015852] e100 0000:01:05.0: BAR 0: can't reserve [mem 0xf9cff000-0xf9cfffff]
> [  605.015855] kvm_vm_ioctl_assign_device: Could not get access to device regions

This is what is returning -EBUSY and triggering the error message.

> [  667.410228] e100 0000:01:05.0: PCI INT A disabled
> [  700.500278] pci-stub: invalid id string ""
> [  707.730636] pci-stub 0000:01:05.0: claimed by stub
> [  734.755491] pci-stub 0000:01:05.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
> [  734.790077] pci-stub 0000:01:05.0: restoring config space at offset 0xf (was 0x38080100, writing 0x3808010b)
> [  734.790095] pci-stub 0000:01:05.0: restoring config space at offset 0xc (was 0x0, writing 0xf9ce0000)
> [  734.790113] pci-stub 0000:01:05.0: restoring config space at offset 0x6 (was 0x0, writing 0xf9cc0000)
> [  734.790123] pci-stub 0000:01:05.0: restoring config space at offset 0x5 (was 0x1, writing 0xac01)
> [  734.790132] pci-stub 0000:01:05.0: restoring config space at offset 0x4 (was 0x0, writing 0xf9cff000)
> [  734.790142] pci-stub 0000:01:05.0: restoring config space at offset 0x3 (was 0x0, writing 0x4010)
> [  734.790153] pci-stub 0000:01:05.0: restoring config space at offset 0x1 (was 0x2900000, writing 0x2900113)
> [  735.173647] assign device 0:1:5.0 failed
> [  735.173688] pci-stub 0000:01:05.0: PCI INT A disabled
> [  768.850519] pci-stub 0000:01:05.0: claimed by stub
> [  775.855376] pci-stub 0000:01:05.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
> [  775.890080] pci-stub 0000:01:05.0: restoring config space at offset 0xf (was 0x38080100, writing 0x3808010b)
> [  775.890097] pci-stub 0000:01:05.0: restoring config space at offset 0xc (was 0x0, writing 0xf9ce0000)
> [  775.890115] pci-stub 0000:01:05.0: restoring config space at offset 0x6 (was 0x0, writing 0xf9cc0000)
> [  775.890126] pci-stub 0000:01:05.0: restoring config space at offset 0x5 (was 0x1, writing 0xac01)
> [  775.890135] pci-stub 0000:01:05.0: restoring config space at offset 0x4 (was 0x0, writing 0xf9cff000)
> [  775.890144] pci-stub 0000:01:05.0: restoring config space at offset 0x3 (was 0x0, writing 0x4010)
> [  775.890155] pci-stub 0000:01:05.0: restoring config space at offset 0x1 (was 0x2900000, writing 0x2900113)
> [  776.275188] assign device 0:1:5.0 failed
> [  776.275230] pci-stub 0000:01:05.0: PCI INT A disabled
> 
> The device was previously owned by e100, now it shows it is owned by pci_stub. Initially the pci_stub.ko module was not loaded, I had to load it manually so that these assignment commands would work.

And it looks like it is not working yet.  Is that correct?

Can you start fresh (make sure the e100 is loaded and functional), and
do the following:

verify it's starting with e100
# ls -l /sys/bus/pci/devices/0000:01:05.0/driver    <-- should show e100

add device id to pci-stub driver
# modprobe pci-stub
# echo "8086 1229" > /sys/bus/pci/drivers/pci-stub/new_id

unbind e100 from device
# echo "0000:01:05.0" > /sys/bus/pci/drivers/e100/unbind

bind pci-stub to device
# echo "0000:01:05.0" > /sys/bus/pci/drivers/pci-stub/bind

remove the id from pci-stub so that subsequent probing will pick up e100
# echo "8086 1229" > /sys/bus/pci/drivers/pci-stub/remove_id

verify it worked
# ls -l /sys/bus/pci/devices/0000:01:05.0/driver    <-- should show pci-stub

Launch your VM...assigned device should show up in VM now.

thanks,
-chris