Re: [PATCH] Fixing ATS enabling for device assignment

Re: [PATCH] Fixing ATS enabling for device assignment

[Jan Beulich]

>>> Weidong Han  05/19/10 11:12 AM >>>
>Jan Beulich wrote:
>>>>> "Han, Weidong"  05/19/10 4:37 AM >>>
>>>>>         
>>> Currently, Xen only enables ATS in Xen booting. When a ATS capable device is assigned to guest, ATS is actually not enabled because FLR before assignment causes it to be disabled. Thus ATS cannot be used in guest. This patch enables ATS in domain_context_mapping. This ensures ATS is enabled in assignment because FLR is earlier than domain_context_mapping call. Therefore ATS can be used in guest. This patch also implements disable_ats_device to disable ATS when the device is deassigned from a domain.
>>>     
>>
>> Hmm, this looks inconsistent to me: Iirc FLR is being done from the tools, so unless it is technically impossible to do FLR after domain_context_mapping(), the patch creates a hypervisor dependency on the tools (if nothing else, a latent bug - to be introduced the moment operation ordering changes in the tools).
>>   
>It doesn't depend on FLR from tools. It just ensures ATS is enabled for 
>device assignment in guest. If ATS is already enabled, enable_ats_device 
>won't re-enable it.

Perhaps I didn't express my concern properly: It's not really the hypervisor depending on anything done by the tools, but the code you ad to the hypervisor will trun out pointless if the tools move FLR past the point where domain_context_mapping() gets executed. It would seem more clean to me if e.g. the tools re-enabled ATS after doing FLR if it was enabled before.

>> Further, if ATS gets enabled at boot time, why would you want to disable it after de-assignment?
>>   
>ATS enabled devices will be added to ats_devices list. disabling ATS 
>will remove a device from that list and avoid leak. If ATS is enabled on 
>a VF, and it's destroyed by removing PF driver, disable_ats_device will 
>be invoked to remove it from the list.

That's fine, but doesn't answer my question.

Jan

Re: [PATCH] Fixing ATS enabling for device assignment

[Weidong Han]

Jan Beulich wrote:
>>>> Weidong Han  05/19/10 11:12 AM >>>
>>>>         
>> Jan Beulich wrote:
>>     
>>>>>> "Han, Weidong"  05/19/10 4:37 AM >>>
>>>>>>         
>>>>>>             
>>>> Currently, Xen only enables ATS in Xen booting. When a ATS capable device is assigned to guest, ATS is actually not enabled because FLR before assignment causes it to be disabled. Thus ATS cannot be used in guest. This patch enables ATS in domain_context_mapping. This ensures ATS is enabled in assignment because FLR is earlier than domain_context_mapping call. Therefore ATS can be used in guest. This patch also implements disable_ats_device to disable ATS when the device is deassigned from a domain.
>>>>     
>>>>         
>>> Hmm, this looks inconsistent to me: Iirc FLR is being done from the tools, so unless it is technically impossible to do FLR after domain_context_mapping(), the patch creates a hypervisor dependency on the tools (if nothing else, a latent bug - to be introduced the moment operation ordering changes in the tools).
>>>   
>>>       
>> It doesn't depend on FLR from tools. It just ensures ATS is enabled for 
>> device assignment in guest. If ATS is already enabled, enable_ats_device 
>> won't re-enable it.
>>     
>
> Perhaps I didn't express my concern properly: It's not really the hypervisor depending on anything done by the tools, but the code you ad to the hypervisor will trun out pointless if the tools move FLR past the point where domain_context_mapping() gets executed.
Yes, it's a problem in your case. But FLR is required to be earlier than 
domain_context_mapping for device assignment.
>  It would seem more clean to me if e.g. the tools re-enabled ATS after doing FLR if it was enabled before.I
>   
Sounds good. But i'm not sure if it's suitable or easy to maintain 
capability enabling/disabling in control panel, I feel current device 
handling logic (hotplug, assign/deassign, PV and hvm) in control panel 
is a bit of complex.
>   
>>> Further, if ATS gets enabled at boot time, why would you want to disable it after de-assignment?
>>>   
>>>       
>> ATS enabled devices will be added to ats_devices list. disabling ATS 
>> will remove a device from that list and avoid leak. If ATS is enabled on 
>> a VF, and it's destroyed by removing PF driver, disable_ats_device will 
>> be invoked to remove it from the list.
>>     
>
> That's fine, but doesn't answer my question.
>   
if not disable it after de-assignment, enable_ats_device will add the 
device multiple times to ats_devices list.

Regards,
Weidong
> Jan
>
>   

Re: [PATCH] Fixing ATS enabling for device assignment

[Jan Beulich]

>>> "Han, Weidong"  05/19/10 4:37 AM >>>
>Currently, Xen only enables ATS in Xen booting. When a ATS capable device is assigned to guest, ATS is actually not enabled because FLR before assignment causes it to be disabled. Thus ATS cannot be used in guest. This patch enables ATS in domain_context_mapping. This ensures ATS is enabled in assignment because FLR is earlier than domain_context_mapping call. Therefore ATS can be used in guest. This patch also implements disable_ats_device to disable ATS when the device is deassigned from a domain.

Hmm, this looks inconsistent to me: Iirc FLR is being done from the tools, so unless it is technically impossible to do FLR after domain_context_mapping(), the patch creates a hypervisor dependency on the tools (if nothing else, a latent bug - to be introduced the moment operation ordering changes in the tools).

Further, if ATS gets enabled at boot time, why would you want to disable it after de-assignment?

Finally (only partially related) - why are {en,dis}able_ats_device() architecture specific? PCI config space accesses really aren't I would think.

Jan

Re: [PATCH] Fixing ATS enabling for device assignment

[Weidong Han]

Jan Beulich wrote:
>>>> "Han, Weidong"  05/19/10 4:37 AM >>>
>>>>         
>> Currently, Xen only enables ATS in Xen booting. When a ATS capable device is assigned to guest, ATS is actually not enabled because FLR before assignment causes it to be disabled. Thus ATS cannot be used in guest. This patch enables ATS in domain_context_mapping. This ensures ATS is enabled in assignment because FLR is earlier than domain_context_mapping call. Therefore ATS can be used in guest. This patch also implements disable_ats_device to disable ATS when the device is deassigned from a domain.
>>     
>
> Hmm, this looks inconsistent to me: Iirc FLR is being done from the tools, so unless it is technically impossible to do FLR after domain_context_mapping(), the patch creates a hypervisor dependency on the tools (if nothing else, a latent bug - to be introduced the moment operation ordering changes in the tools).
>   
It doesn't depend on FLR from tools. It just ensures ATS is enabled for 
device assignment in guest. If ATS is already enabled, enable_ats_device 
won't re-enable it.
> Further, if ATS gets enabled at boot time, why would you want to disable it after de-assignment?
>   
ATS enabled devices will be added to ats_devices list. disabling ATS 
will remove a device from that list and avoid leak. If ATS is enabled on 
a VF, and it's destroyed by removing PF driver, disable_ats_device will 
be invoked to remove it from the list.

Regards,
Weidong
> Finally (only partially related) - why are {en,dis}able_ats_device() architecture specific? PCI config space accesses really aren't I would think
>   
> Jan
>
>   

[PATCH] Fixing ATS enabling for device assignment

[Han, Weidong]

[-- Attachment #1: Type: text/plain, Size: 560 bytes --]

Currently, Xen only enables ATS in Xen booting. When a ATS capable device is assigned to guest, ATS is actually not enabled because FLR before assignment causes it to be disabled. Thus ATS cannot be used in guest. This patch enables ATS in domain_context_mapping. This ensures ATS is enabled in assignment because FLR is earlier than domain_context_mapping call. Therefore ATS can be used in guest. This patch also implements disable_ats_device to disable ATS when the device is deassigned from a domain.

Signed-off-by: Weidong Han <weidong.han@intel.com>

[-- Attachment #2: ats-fix.patch --]
[-- Type: application/octet-stream, Size: 4319 bytes --]

diff -r a167ea374f26 xen/drivers/passthrough/vtd/ia64/ats.c
--- a/xen/drivers/passthrough/vtd/ia64/ats.c	Thu Apr 29 19:38:52 2010 +0100
+++ b/xen/drivers/passthrough/vtd/ia64/ats.c	Mon May 17 17:12:46 2010 +0800
@@ -47,6 +47,11 @@ int enable_ats_device(int seg, int bus, 
     return 0;
 }
 
+int disable_ats_device(int seg, int bus, int devfn)
+{
+    return 0;
+}
+
 int dev_invalidate_iotlb(struct iommu *iommu, u16 did,
     u64 addr, unsigned int size_order, u64 type)
 {
diff -r a167ea374f26 xen/drivers/passthrough/vtd/iommu.c
--- a/xen/drivers/passthrough/vtd/iommu.c	Thu Apr 29 19:38:52 2010 +0100
+++ b/xen/drivers/passthrough/vtd/iommu.c	Tue May 18 13:41:12 2010 +0800
@@ -1325,6 +1325,9 @@ static int domain_context_mapping(struct
             dprintk(VTDPREFIX, "d%d:PCIe: map bdf = %x:%x.%x\n",
                     domain->domain_id, bus, PCI_SLOT(devfn), PCI_FUNC(devfn));
         ret = domain_context_mapping_one(domain, drhd->iommu, bus, devfn);
+        if ( !ret && ats_device(0, bus, devfn) )
+            enable_ats_device(0, bus, devfn);
+
         break;
 
     case DEV_TYPE_PCI:
@@ -1454,6 +1457,9 @@ static int domain_context_unmap(struct d
             dprintk(VTDPREFIX, "d%d:PCIe: unmap bdf = %x:%x.%x\n",
                     domain->domain_id, bus, PCI_SLOT(devfn), PCI_FUNC(devfn));
         ret = domain_context_unmap_one(domain, iommu, bus, devfn);
+        if ( !ret && ats_device(0, bus, devfn) )
+            disable_ats_device(0, bus, devfn);
+
         break;
 
     case DEV_TYPE_PCI:
@@ -1772,8 +1778,6 @@ static void setup_dom0_devices(struct do
             list_add(&pdev->domain_list, &d->arch.pdev_list);
             domain_context_mapping(d, pdev->bus, pdev->devfn);
             pci_enable_acs(pdev);
-            if ( ats_device(0, pdev->bus, pdev->devfn) )
-                enable_ats_device(0, pdev->bus, pdev->devfn);
         }
     }
     spin_unlock(&pcidevs_lock);
diff -r a167ea374f26 xen/drivers/passthrough/vtd/x86/ats.c
--- a/xen/drivers/passthrough/vtd/x86/ats.c	Thu Apr 29 19:38:52 2010 +0100
+++ b/xen/drivers/passthrough/vtd/x86/ats.c	Tue May 18 13:44:22 2010 +0800
@@ -92,6 +92,9 @@ int ats_device(int seg, int bus, int dev
 
     pdev = pci_get_pdev(bus, devfn);
     drhd = acpi_find_matched_drhd_unit(pdev);
+    if ( !drhd )
+        return 0;
+
     if ( !ecap_queued_inval(drhd->iommu->ecap) ||
          !ecap_dev_iotlb(drhd->iommu->ecap) )
         return 0;
@@ -144,6 +147,9 @@ int enable_ats_device(int seg, int bus, 
 
     value = pci_conf_read16(bus, PCI_SLOT(devfn),
                             PCI_FUNC(devfn), pos + ATS_REG_CTL);
+    if ( value & ATS_ENABLE )
+        return 0;
+
     value |= ATS_ENABLE;
     pci_conf_write16(bus, PCI_SLOT(devfn), PCI_FUNC(devfn),
                      pos + ATS_REG_CTL, value);
@@ -153,9 +159,49 @@ int enable_ats_device(int seg, int bus, 
     pdev->devfn = devfn;
     pdev->ats_queue_depth = queue_depth;
     list_add(&(pdev->list), &ats_devices);
+    if ( iommu_verbose )
+        dprintk(XENLOG_INFO VTDPREFIX, "%x:%x.%x: ATS is enabled\n",
+                bus, PCI_SLOT(devfn), PCI_FUNC(devfn));
 
     return pos;
 }
+
+int disable_ats_device(int seg, int bus, int devfn)
+{
+    struct list_head *pdev_list, *tmp;
+    struct pci_ats_dev *pdev;
+    u32 value;
+    int pos;
+
+    pos = pci_find_ext_capability(seg, bus, devfn, PCI_EXT_CAP_ID_ATS);
+    if ( !pos )
+        return 0;
+
+    /* BUGBUG: add back seg when multi-seg platform support is enabled */
+    value = pci_conf_read16(bus, PCI_SLOT(devfn),
+                            PCI_FUNC(devfn), pos + ATS_REG_CTL);
+    value &= ~ATS_ENABLE;
+    pci_conf_write16(bus, PCI_SLOT(devfn), PCI_FUNC(devfn),
+                     pos + ATS_REG_CTL, value);
+
+    list_for_each_safe( pdev_list, tmp, &ats_devices )
+    {
+        pdev = list_entry(pdev_list, struct pci_ats_dev, list);
+        if ( pdev->bus == bus && pdev->devfn == devfn )
+        {
+            list_del(&pdev->list);
+            xfree(pdev);
+            break;
+        }
+    }
+
+    if ( iommu_verbose )
+        dprintk(XENLOG_INFO VTDPREFIX, "%x:%x.%x: ATS is disabled\n",
+                bus, PCI_SLOT(devfn), PCI_FUNC(devfn));
+
+    return 0;
+}
+
 
 static int device_in_domain(struct iommu *iommu, struct pci_ats_dev *pdev, u16 did)
 {

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel