Re: IOMMU and AMD 890fx

["Jens Krehbiel-Gräther" <xen@jkg-it-services.de>]

Hi!

I now have a Asus Crosshair IV for testing and this board (as I postet 
earlier) supports iommu. Now I have installed Debian lenny (5.0) and 
compiled xen 4.0.0.
Xen is running now and I have installed a hvm debian lenny to which I 
would like to bind a pci device but that doesn't work.
Perhaps I am doing something wrong and you could help me?


xen-01:~# xm dmesg |grep -i -e iommu -e amd-vi -e ivrs
(XEN) ACPI: IVRS CFD9FA00, 00C8 (r1  AMD     RD890S   202031 AMD         0)
(XEN) AMD-Vi: IOMMU 0 Enabled.
xen-01:~#

So as xm dmesg reports IOMMU is enabled. I passed the "iommu=1" 
parameter to kernel.

xen-01:~# uname -a
Linux xen-01 2.6.32-bpo.5-xen-amd64 #1 SMP Fri Jun 11 12:21:09 UTC 2010 
x86_64 GNU/Linux

xen-01:/boot# grep -i iommu config-2.6.32-bpo.5-xen-amd64
CONFIG_GART_IOMMU=y
CONFIG_CALGARY_IOMMU=y
CONFIG_CALGARY_IOMMU_ENABLED_BY_DEFAULT=y
CONFIG_AMD_IOMMU=y
# CONFIG_AMD_IOMMU_STATS is not set
CONFIG_IOMMU_HELPER=y
CONFIG_IOMMU_API=y
# CONFIG_IOMMU_DEBUG is not set
# CONFIG_IOMMU_STRESS is not set

xen-01:/boot# grep -i backend config-2.6.32-bpo.5-xen-amd64
CONFIG_XEN_BACKEND=y
CONFIG_XEN_NETDEV_BACKEND=y
CONFIG_XEN_BLKDEV_BACKEND=y
CONFIG_XEN_PCIDEV_BACKEND=y
CONFIG_XEN_PCIDEV_BACKEND_VPCI=y
# CONFIG_XEN_PCIDEV_BACKEND_PASS is not set
# CONFIG_XEN_PCIDEV_BACKEND_SLOT is not set
# CONFIG_XEN_PCIDEV_BACKEND_CONTROLLER is not set


So I think the needed kernel options are enabled.

I can hide the pci device from dom0:

lspci:
00:14.2 Audio device: ATI Technologies Inc SBx00 Azalia (Intel HDA) (rev 40)

echo -n "0000:00:14.2" > "/sys/bus/pci/drivers/HDA Intel/unbind"
echo -n "0000:00:14.2" > /sys/bus/pci/drivers/pciback/new_slot
echo -n "0000:00:14.2" > /sys/bus/pci/drivers/pciback/bind


xen-01:~# xm pci-list-assignable-devices
0000:00:14.2


Now when I try to pass through the device to the guest I get (at first 
try) this message:

xen-01:~# xm pci-attach test 0000:00:14.2 8
Error: Timed out waiting for device model action
Usage: xm pci-attach [-o|--options=<opt>] <Domain> 
<domain:bus:slot.func> [virtual slot]

Insert a new pass-through pci device.

When I try it again, I will always get this message:

xen-01:~# xm pci-attach test 0000:00:14.2 8
Error: Failed to assign device to IOMMU 
(0000:00:14.2@40,msitranslate=1,power_mgmt=0)
Usage: xm pci-attach [-o|--options=<opt>] <Domain> 
<domain:bus:slot.func> [virtual slot]

Insert a new pass-through pci device.

In /var/log/xen/xend.log there are this messages:

[2010-06-17 00:06:18 2407] DEBUG (XendDomainInfo:903) 
XendDomainInfo.pci_device_configure: ['pci', ['dev', ['slot', '0x14'], 
['domain', '0x0000'], ['key', '0000:00:14.2@8'], ['bus', '0x00'], 
['vdevfn', '0x40'], ['func', '0x2']], ['state', 'Initialising']]

[2010-06-17 00:06:18 2407] DEBUG (XendDomainInfo:675) 
XendDomainInfo.hvm_pci_device_create: {'devs': [{'slot': '0x14', 
'domain': '0x0000', 'key': '0000:00:14.2@8', 'bus': '0x00', 'vdevfn': 
'0x40', 'func': '0x2', 'uuid': 'f9a78b41-aad2-8c31-73a3-2b2162c8ebb4'}], 
'states': ['Initialising']}

[2010-06-17 00:06:18 2407] DEBUG (XendDomainInfo:801) 
XendDomainInfo.hvm_pci_device_insert_dev: {'slot': '0x14', 'domain': 
'0x0000', 'key': '0000:00:14.2@8', 'bus': '0x00', 'vdevfn': '0x40', 
'func': '0x2', 'uuid': 'f9a78b41-aad2-8c31-73a3-2b2162c8ebb4'}

[2010-06-17 00:06:18 2407] DEBUG (XendDomainInfo:822) 
XendDomainInfo.hvm_pci_device_insert_dev: 
0000:00:14.2@40,msitranslate=1,power_mgmt=0


dmesg gives this log:

[ 2177.373375] pciback 0000:00:14.2: seizing device
[ 2177.373488] xen: registering gsi 16 triggering 0 polarity 1
[ 2177.373497] xen_allocate_pirq: returning irq 16 for gsi 16
[ 2177.373519] xen: --> irq=16
[ 2177.373528] Already setup the GSI :16
[ 2177.373551] pciback 0000:00:14.2: PCI INT A -> GSI 16 (level, low) -> 
IRQ 16
[ 2177.373582] pciback 0000:00:14.2: PCI INT A disabled


and xm dmesg this:

(XEN) domctl.c:853:d0 XEN_DOMCTL_assign_device: assign device (0:14.2) 
failed


So can anyone tell me what I am doing wrong??

Hope someone could help me!

    Jens