From mboxrd@z Thu Jan 1 00:00:00 1970 From: Joerg Roedel Subject: Re: IOMMU and AMD 890fx Date: Mon, 21 Jun 2010 17:39:00 +0200 Message-ID: <20100621153900.GC20761@8bytes.org> References: <4C160DBF.8030701@jkg-it-services.de> <4C163371.7090809@amd.com> <4C17F2B4.7040208@web.de> <3399D2269986104B959D31B7E919313C3D10B465@SAUSEXMBP01.amd.com> <4C194B9B.10500@jkg-it-services.de> <20100617131430.GB3474@phenom.dumpdata.com> <4C1A5882.2030006@amd.com> <4C1AB15F.6060905@jkg-it-services.de> <1492414863.20100618162434@eikelenboom.it> <4C1F7E7A.7030301@jkg-it-services.de> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Return-path: Content-Disposition: inline In-Reply-To: <4C1F7E7A.7030301@jkg-it-services.de> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Jens =?iso-8859-1?Q?Krehbiel-Gr=E4ther?= Cc: Leonardo Prosperi , "xen-devel@lists.xensource.com" , Konrad Rzeszutek Wilk , "Przywara, Andre" , Wei Huang , Sander Eikelenboom , Joop Boonen , Oakley List-Id: xen-devel@lists.xenproject.org Hi Jens, great to know. Thanks for your testing. Joerg On Mon, Jun 21, 2010 at 05:00:10PM +0200, Jens Krehbiel-Gr=E4ther wrote: > Another board with working AMD IOMMU: Asus M4A89TD Pro/USB3 > > Just got it in hands and testet successful! > > Jens > > > > Am 18.06.2010 16:24, schrieb Sander Eikelenboom: >> Hi Jens, >> >> Thx for reporting your successful pci passthrough with an AMD IOMMU, i= have put this motherboard in the xen vt-d/iommu wiki as a motherboard wi= th the BIOS supporting the IOMMU. >> http://wiki.xensource.com/xenwiki/VTdHowTo >> >> -- >> Sander >> >>> Hi! >> >>> Thanks for the tips! >> >>> With xen-4.0-testing.hg tree + 2.6.32-bpo.5-xen-amd64 (from debian le= nny >>> backports) it works. So the 4.0.0 release could be buggy? >>> When I compile and install xen-4.0-testing.hg tree it works as you >>> described. >>> With 4.0.0 it did not work with any pci device, with xen-4.0-testing.= hg >>> tree it works with the two I tested for the moment: >> >>> 00:14.2 Audio device: ATI Technologies Inc SBx00 Azalia (Intel HDA) (= rev 40) >>> 03:00.0 Ethernet controller: Marvell Technology Group Ltd. Device 438= 1 >>> (rev 11) >> >>> (I just randomly take two devices). >>> They appear in the hvm virtual machine and they are working. >> >>> Great, thank you very much! >> >>> Jens >> >> >>> Am 17.06.2010 19:16, schrieb Wei Huang: >>>> Hi Jens, >>>> >>>> Thanks for testing it out. I tested Xen 4.0 on my machine this morni= ng. >>>> The passthru did work. Here are the details: >>>> >>>> 1. I used Xen 4.0 (xen-4.0-testing.hg tree) + 2.6.18.8 Dom0 >>>> 2. My system is a workstation; so it doesn't have a HDA device. Here= is >>>> what I get from lspci (I skipped some of un-related devices): >>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >>>> ... >>>> 00:14.0 SMBus: ATI Technologies Inc SBx00 SMBus Controller (rev 3b) >>>> 00:14.1 IDE interface: ATI Technologies Inc SB700/SB800 IDE Controll= er >>>> 00:14.3 ISA bridge: ATI Technologies Inc SB700/SB800 LPC host contro= ller >>>> 00:14.4 PCI bridge: ATI Technologies Inc SBx00 PCI to PCI Bridge >>>> 00:14.5 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI2 >>>> Controller >>>> ... >>>> 01:00.0 VGA compatible controller: ATI Technologies Inc Device 949e >>>> 01:00.1 Audio device: ATI Technologies Inc RV730XT Audio device [Rad= eon >>>> HD 4670] >>>> 02:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ether= net >>>> Controller (rev 06) >>>> 02:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ether= net >>>> Controller (rev 06) >>>> 03:00.0 Ethernet controller: Intel Corporation 82572EI Gigabit Ether= net >>>> Controller (Copper) (rev 06) >>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >>>> As you can see, I have two NICs (a two-port Intel 82571EB NIC and a >>>> single-port 82572EI NIC). I am going to passthru 82571EB NIC to a Li= nux >>>> guest VM. >>>> >>>> 3. Here is the output from "xm dmesg". Because you also see a simila= r >>>> message, I think your IOMMU is working. >>>> (XEN) AMD-Vi: IOMMU 0 Enabled. >>>> (XEN) I/O virtualisation enabled >>>> (XEN) I/O virtualisation for PV guests disabled >>>> >>>> FYI, here is my grub entry: >>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >>>> title 64bit Xen-unstable on SLES11 >>>> root (hd0,5) >>>> kernel /boot/xen.gz dom0_mem=3D512M console=3Dcom2 com2=3D115200,8n1= iommu=3D1 >>>> module /boot/vmlinuz-2.6.18.8 root=3D/dev/hdb6 resume=3D/dev/hdb3 >>>> module /boot/initrd-2.6.18.8 >>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >>>> >>>> 4. After login, I removed 82571EB from dom0 >>>> >>>> > echo -n "0000:02:00.0"> /sys/bus/pci/drivers/e1000e/unbind >>>> > echo -n "0000:02:00.1"> /sys/bus/pci/drivers/e1000e/unbind >>>> > echo -n "0000:02:00.0"> /sys/bus/pci/drivers/pciback/new_slot >>>> > echo -n "0000:02:00.1"> /sys/bus/pci/drivers/pciback/new_slot >>>> > echo -n "0000:02:00.0"> /sys/bus/pci/drivers/pciback/bind >>>> > echo -n "0000:02:00.1"> /sys/bus/pci/drivers/pciback/bind >>>> >>>> 5. In guest (ttylinux) configure file, I use the following setting. >>>> ... >>>> pci=3D[ '02:00.0', '02:00.1' ] >>>> ... >>>> >>>> 6. After ttylinux booted, I saw two Intel NICs. The IP address was >>>> obtained automatically. See the attached image file. >>>> >>>> Same as what Konrad has mentioned, I would recommend two things: i) >>>> instead of passthru HDA, try other independent PCIe devices first (s= uch >>>> as NICs); ii) configure the passthru devices in guest configure file= , >>>> instead of use it a hot-plug device. I can send you my guest configu= re >>>> file if you need it. >>>> >>>> PS: I am pretty confident HDA passthru work because I tested it on a= n >>>> desktop system before. Here is the list of devices I have passthru t= o a >>>> Win7 guest VM on that system: (1) USB keyboard and mouse; (2) HDA au= dio; >>>> (3) ATI graphics. >>>> >>>> Thanks, >>>> -Wei >>>> >>>> >>>> >>>> Konrad Rzeszutek Wilk wrote: >>>>> On Thu, Jun 17, 2010 at 12:09:31AM +0200, Jens Krehbiel-Gr=E4ther w= rote: >>>>>> Hi! >>>>>> >>>>>> I now have a Asus Crosshair IV for testing and this board (as I >>>>>> postet earlier) supports iommu. Now I have installed Debian lenny >>>>>> (5.0) and compiled xen 4.0.0. >>>>>> Xen is running now and I have installed a hvm debian lenny to whic= h >>>>>> I would like to bind a pci device but that doesn't work. >>>>>> Perhaps I am doing something wrong and you could help me? >>>>> >>>>> Is that the _only_ PCI device you have tried to assign? Did you try= to >>>>> assign other PCI devices which have the function number being zero? >>>>>> and xm dmesg this: >>>>>> >>>>>> (XEN) domctl.c:853:d0 XEN_DOMCTL_assign_device: assign device >>>>>> (0:14.2) failed >>>>> >>>>> That is not very useful. Wish it gave you the return code at least. >>>>> If you are comfortable with adding printks in the Xen hypervisor I >>>>> would modify it a bit to see what was the 'ret' value and why >>>>> 'intel_iommu_assign_device' function failed. >>>>> >>>>>> So can anyone tell me what I am doing wrong?? >>>>> >>>>> Please also attach the full serial long, including the Xen one - an= d run >>>>> it with 'loglvl=3Dall iommu=3Dverbose'. >>>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> Xen-devel mailing list >>>> Xen-devel@lists.xensource.com >>>> http://lists.xensource.com/xen-devel >> >> >>