Re: iommuu/vt-d issues with LSI MegaSAS (PERC5i)

[Weidong Han <weidong.han@intel.com>]

Konrad Rzeszutek Wilk wrote:
> On Wed, Jun 02, 2010 at 03:05:19PM +0800, Weidong Han wrote:
>   
>> Pasi Kärkkäinen wrote:
>>     
>>> On Wed, Jun 02, 2010 at 09:23:37AM +0800, Weidong Han wrote:
>>>   
>>>       
>>>> This PCI-x card is not suitable for assignment. It contains an invisible
>>>> device 05:08.0 (lspci cannot show it), but this invisible device won't
>>>> be mapped by VT-d because VT-d engine doesn't know this device, so you
>>>> can see the DMAR faults on it. One workaround is to hard code to map
>>>> 05:08.0 when assign 05:0e.0. BTW, PCIe LSI card don't have this problem.
>>>>
>>>>     
>>>>         
>>> Note that this problem happens when booting up dom0..
>>>   
>>>       
>> I see. all devices are assigned to dom0 in booting. It just needs to map  
>> 05:08.0 as well when map 05:0e.0 in domain_context_mapping.
>>     
>
> Hey Weidong,
>
> Thank you the explanation. I am not that familiar with the VT-D chipset,
> but it seems that this issue also appears with CardBus controllers:
> http://lkml.org/lkml/2010/5/22/69 ?
>
> For this device, the problem should also appear with the newer kernels
> without using the Hypervisor and with CONFIG_DMAR enabled, right? 
>   
yes.
> Am I to understand that the workaround you are proposing is doing
> something akin to this:
>   
Your below code only make it work for dom0 by calling lsi_hack_init. but 
if you want to assign it to guest, it still cannot work. You can 
implement a a simple temporarily workaround like this:

diff -r 2cd58d7d5db9 xen/drivers/passthrough/vtd/iommu.c
--- a/xen/drivers/passthrough/vtd/iommu.c   Tue Jun 01 20:40:34 2010 -0400
+++ b/xen/drivers/passthrough/vtd/iommu.c   Wed Jun 02 14:09:37 2010 -0400
@@ -1339,6 +1339,15 @@ static int domain_context_mapping(struct
         if ( ret )
             break;

+        /* NB. LSI workaround */
+        if ( bus == 0x05 && PCI_SLOT(devfn) == 0x0e && PCI_FUNC(devfn) 
== 0x0 )
+        {
+            ret = domain_context_mapping_one(domain, drhd->iommu,
+                                             0x05, PCI_DEVFN(8, 0));
+            if ( ret )
+                break;
+        }
+
         if ( find_upstream_bridge(&bus, &devfn, &secbus) < 1 )
             break;

>
> /*
>  *
>  * This program is free software; you can redistribute it and/or modify
>  * it under the terms of the GNU General Public License v2.0 as published by
>  * the Free Software Foundation
>  *
>  * This program is distributed in the hope that it will be useful,
>  * but WITHOUT ANY WARRANTY; without even the implied warranty of
>  * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>  * GNU General Public License for more details.
>  */
>
> #include <linux/module.h>
> #include <linux/string.h>
> #include <linux/types.h>
> #include <linux/init.h>
> #include <linux/stat.h>
> #include <linux/err.h>
> #include <linux/ctype.h>
> #include <linux/slab.h>
> #include <linux/limits.h>
> #include <linux/device.h>
> #include <linux/pci.h>
> #include <linux/device.h>
>
> #include <linux/pci.h>
>
> #include <xen/interface/xen.h>
> #include <xen/interface/physdev.h>
>
> #include <asm/xen/hypervisor.h>
> #include <asm/xen/hypercall.h>
>
> #define LSI_HACK  "0.1"
>
> MODULE_AUTHOR("Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>");
> MODULE_DESCRIPTION("lsi hack");
> MODULE_LICENSE("GPL");
> MODULE_VERSION(LSI_HACK);
>
> static int __init lsi_hack_init(void)
> {
> 	int r = 0;
>
> 	struct physdev_manage_pci manage_pci = {
>                         .bus    = 0x5,
>                         .devfn  = PCI_DEVFN(8,0),
>                 };
> 	r = HYPERVISOR_physdev_op(PHYSDEVOP_manage_pci_add,
>                         &manage_pci);
>
> 	return r;
> }
>
> static void __exit lsi_hack_exit(void)
> {
> 	int r = 0;
> 	struct physdev_manage_pci manage_pci;
>
>         manage_pci.bus = 0x5;
>         manage_pci.devfn = PCI_DEVFN(8,0);
>
>         r = HYPERVISOR_physdev_op(PHYSDEVOP_manage_pci_remove,
>                 &manage_pci);
> 	if (r)
> 		printk(KERN_ERR "%s: %d\n", __FUNCTION__, r);
> }
>
> module_init(lsi_hack_init);
> module_exit(lsi_hack_exit);
>