Re: IOMMU and AMD 890fx

[Joerg Roedel <joro@8bytes.org>]

Hi Jens,

great to know. Thanks for your testing.

Joerg

On Mon, Jun 21, 2010 at 05:00:10PM +0200, Jens Krehbiel-Gräther wrote:
> Another board with working AMD IOMMU: Asus M4A89TD Pro/USB3
>
> Just got it in hands and testet successful!
>
> Jens
>
>
>
> Am 18.06.2010 16:24, schrieb Sander Eikelenboom:
>> Hi Jens,
>>
>> Thx for reporting your successful pci passthrough with an AMD IOMMU, i have put this motherboard in the xen vt-d/iommu wiki as a motherboard with the BIOS supporting the IOMMU.
>> http://wiki.xensource.com/xenwiki/VTdHowTo
>>
>> --
>> Sander
>>
>>> Hi!
>>
>>> Thanks for the tips!
>>
>>> With xen-4.0-testing.hg tree + 2.6.32-bpo.5-xen-amd64 (from debian lenny
>>> backports) it works. So the 4.0.0 release could be buggy?
>>> When I compile and install xen-4.0-testing.hg tree it works as you
>>> described.
>>> With 4.0.0 it did not work with any pci device, with xen-4.0-testing.hg
>>> tree it works with the two I tested for the moment:
>>
>>> 00:14.2 Audio device: ATI Technologies Inc SBx00 Azalia (Intel HDA) (rev 40)
>>> 03:00.0 Ethernet controller: Marvell Technology Group Ltd. Device 4381
>>> (rev 11)
>>
>>> (I just randomly take two devices).
>>> They appear in the hvm virtual machine and they are working.
>>
>>> Great, thank you very much!
>>
>>> Jens
>>
>>
>>> Am 17.06.2010 19:16, schrieb Wei Huang:
>>>> Hi Jens,
>>>>
>>>> Thanks for testing it out. I tested Xen 4.0 on my machine this morning.
>>>> The passthru did work. Here are the details:
>>>>
>>>> 1. I used Xen 4.0 (xen-4.0-testing.hg tree) + 2.6.18.8 Dom0
>>>> 2. My system is a workstation; so it doesn't have a HDA device. Here is
>>>> what I get from lspci (I skipped some of un-related devices):
>>>> ==========
>>>> ...
>>>> 00:14.0 SMBus: ATI Technologies Inc SBx00 SMBus Controller (rev 3b)
>>>> 00:14.1 IDE interface: ATI Technologies Inc SB700/SB800 IDE Controller
>>>> 00:14.3 ISA bridge: ATI Technologies Inc SB700/SB800 LPC host controller
>>>> 00:14.4 PCI bridge: ATI Technologies Inc SBx00 PCI to PCI Bridge
>>>> 00:14.5 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI2
>>>> Controller
>>>> ...
>>>> 01:00.0 VGA compatible controller: ATI Technologies Inc Device 949e
>>>> 01:00.1 Audio device: ATI Technologies Inc RV730XT Audio device [Radeon
>>>> HD 4670]
>>>> 02:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet
>>>> Controller (rev 06)
>>>> 02:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet
>>>> Controller (rev 06)
>>>> 03:00.0 Ethernet controller: Intel Corporation 82572EI Gigabit Ethernet
>>>> Controller (Copper) (rev 06)
>>>> ==========
>>>> As you can see, I have two NICs (a two-port Intel 82571EB NIC and a
>>>> single-port 82572EI NIC). I am going to passthru 82571EB NIC to a Linux
>>>> guest VM.
>>>>
>>>> 3. Here is the output from "xm dmesg". Because you also see a similar
>>>> message, I think your IOMMU is working.
>>>> (XEN) AMD-Vi: IOMMU 0 Enabled.
>>>> (XEN) I/O virtualisation enabled
>>>> (XEN) I/O virtualisation for PV guests disabled
>>>>
>>>> FYI, here is my grub entry:
>>>> ==================
>>>> title 64bit Xen-unstable on SLES11
>>>> root (hd0,5)
>>>> kernel /boot/xen.gz dom0_mem=512M console=com2 com2=115200,8n1 iommu=1
>>>> module /boot/vmlinuz-2.6.18.8 root=/dev/hdb6 resume=/dev/hdb3
>>>> module /boot/initrd-2.6.18.8
>>>> ==================
>>>>
>>>> 4. After login, I removed 82571EB from dom0
>>>>
>>>>   >  echo -n "0000:02:00.0">  /sys/bus/pci/drivers/e1000e/unbind
>>>>   >  echo -n "0000:02:00.1">  /sys/bus/pci/drivers/e1000e/unbind
>>>>   >  echo -n "0000:02:00.0">  /sys/bus/pci/drivers/pciback/new_slot
>>>>   >  echo -n "0000:02:00.1">  /sys/bus/pci/drivers/pciback/new_slot
>>>>   >  echo -n "0000:02:00.0">  /sys/bus/pci/drivers/pciback/bind
>>>>   >  echo -n "0000:02:00.1">  /sys/bus/pci/drivers/pciback/bind
>>>>
>>>> 5. In guest (ttylinux) configure file, I use the following setting.
>>>> ...
>>>> pci=[ '02:00.0', '02:00.1' ]
>>>> ...
>>>>
>>>> 6. After ttylinux booted, I saw two Intel NICs. The IP address was
>>>> obtained automatically. See the attached image file.
>>>>
>>>> Same as what Konrad has mentioned, I would recommend two things: i)
>>>> instead of passthru HDA, try other independent PCIe devices first (such
>>>> as NICs); ii) configure the passthru devices in guest configure file,
>>>> instead of use it a hot-plug device. I can send you my guest configure
>>>> file if you need it.
>>>>
>>>> PS: I am pretty confident HDA passthru work because I tested it on an
>>>> desktop system before. Here is the list of devices I have passthru to a
>>>> Win7 guest VM on that system: (1) USB keyboard and mouse; (2) HDA audio;
>>>> (3) ATI graphics.
>>>>
>>>> Thanks,
>>>> -Wei
>>>>
>>>>
>>>>
>>>> Konrad Rzeszutek Wilk wrote:
>>>>> On Thu, Jun 17, 2010 at 12:09:31AM +0200, Jens Krehbiel-Gräther wrote:
>>>>>> Hi!
>>>>>>
>>>>>> I now have a Asus Crosshair IV for testing and this board (as I
>>>>>> postet earlier) supports iommu. Now I have installed Debian lenny
>>>>>> (5.0) and compiled xen 4.0.0.
>>>>>> Xen is running now and I have installed a hvm debian lenny to which
>>>>>> I would like to bind a pci device but that doesn't work.
>>>>>> Perhaps I am doing something wrong and you could help me?
>>>>>
>>>>> Is that the _only_ PCI device you have tried to assign? Did you try to
>>>>> assign other PCI devices which have the function number being zero?
>>>>>> and xm dmesg this:
>>>>>>
>>>>>> (XEN) domctl.c:853:d0 XEN_DOMCTL_assign_device: assign device
>>>>>> (0:14.2) failed
>>>>>
>>>>> That is not very useful. Wish it gave you the return code at least.
>>>>> If you are comfortable with adding printks in the Xen hypervisor I
>>>>> would modify it a bit to see what was the 'ret' value and why
>>>>> 'intel_iommu_assign_device' function failed.
>>>>>
>>>>>> So can anyone tell me what I am doing wrong??
>>>>>
>>>>> Please also attach the full serial long, including the Xen one - and run
>>>>> it with 'loglvl=all iommu=verbose'.
>>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Xen-devel mailing list
>>>> Xen-devel@lists.xensource.com
>>>> http://lists.xensource.com/xen-devel
>>
>>
>>