co-assignment needs for PCI devices

co-assignment needs for PCI devices

[Jan Beulich]

c/s 18046 adds the requirement to co-assign PCI devices behind bridges/
PCIe functions on the same device when the (perhaps individual) device/
function intended to be assigned doesn't support FLR and doesn't sit on
bus 0. For non-IOMMU environments (which are insecure anyway) this
could be considered a regression, since passing through individual devices/
functions didn't know such a restriction earlier.

Also, being not really knowledgeable about all the differences between
PCIe and PCI, I'd appreciate some clarification on why on PCIe it is
possible and correct to do a secondary bus reset when targeting just
the (PCIe) functions of a single device - to me this seems to imply that
there's a one-device-per-non-root-bus restriction there.

Thanks, Jan

RE: co-assignment needs for PCI devices

[Cui, Dexuan]

Jan Beulich wrote:
> c/s 18046 adds the requirement to co-assign PCI devices behind
> bridges/ PCIe functions on the same device when the (perhaps
> individual) device/ function intended to be assigned doesn't support
> FLR and doesn't sit on bus 0. For non-IOMMU environments (which are
> insecure anyway) this 
> could be considered a regression, since passing through individual
> devices/ functions didn't know such a restriction earlier.
Yes, this may not proper for the non-IOMMU case. I planned to fix it, but haven't done that yet... I'd appreciate anybody who can help to do that. :-)
We need to consider IOMMU case, non-IOMMU case and 'hybrid case'(namely, the xen parameter iommu=on,no-pv).

> Also, being not really knowledgeable about all the differences between
> PCIe and PCI, I'd appreciate some clarification on why on PCIe it is
> possible and correct to do a secondary bus reset when targeting just
> the (PCIe) functions of a single device - to me this seems to imply
> that there's a one-device-per-non-root-bus restriction there.
According to VT-d spec (section 3.6.1: Identifying Origination of DMA Requests), PCI devices behind the same uppermost PCI/PCI-X bridge must be co-assigned to the same guest. PCIe devices have not such a constraint.

FLR (Functional Level Reset) is used to quiesce an assigned device when we destroy a guest or we detach the device from the guest. If a PCIe device has no standard FLR capability, we'll try to use SecondaryBusReset as a way to do FLR. Unluckily,  SecondaryBusReset resets all the devices behind the bridge, so for a multi-function PCIe device without FLR capability, we require they have to be co-assigned to the same guest --  certainly this is only for iommu case and for non-iommu case this requirement may be not proper and we shoud fix it...

Thanks,
-- Dexuan

RE: co-assignment needs for PCI devices

[Jan Beulich]

>>> "Cui, Dexuan" <dexuan.cui@intel.com> 31.03.09 09:48 >>>
>Jan Beulich wrote:
>> Also, being not really knowledgeable about all the differences between
>> PCIe and PCI, I'd appreciate some clarification on why on PCIe it is
>> possible and correct to do a secondary bus reset when targeting just
>> the (PCIe) functions of a single device - to me this seems to imply
>> that there's a one-device-per-non-root-bus restriction there.
>According to VT-d spec (section 3.6.1: Identifying Origination of DMA Requests), PCI devices behind the same uppermost PCI/PCI-X
>bridge must be co-assigned to the same guest. PCIe devices have not such a constraint.
>
>FLR (Functional Level Reset) is used to quiesce an assigned device when we destroy a guest or we detach the device from the guest.
>If a PCIe device has no standard FLR capability, we'll try to use SecondaryBusReset as a way to do FLR. Unluckily,
>SecondaryBusReset resets all the devices behind the bridge, so for a multi-function PCIe device without FLR capability, we require
>they have to be co-assigned to the same guest --  certainly this is only for iommu case and for non-iommu case this requirement may
>be not proper and we shoud fix it...

But - for the sake of my education, if you forgive - you didn't answer whether there is a one-device-per-secondary-bus restriction on PCIe...

Thanks, Jan

RE: co-assignment needs for PCI devices

[Cui, Dexuan]

Jan Beulich wrote:
>>>> "Cui, Dexuan" <dexuan.cui@intel.com> 31.03.09 09:48 >>>
>> Jan Beulich wrote:
>>> Also, being not really knowledgeable about all the differences
>>> between PCIe and PCI, I'd appreciate some clarification on why on
>>> PCIe it is possible and correct to do a secondary bus reset when
>>> targeting just the (PCIe) functions of a single device - to me this
>>> seems to imply that there's a one-device-per-non-root-bus
>>> restriction there. 
>> According to VT-d spec (section 3.6.1: Identifying Origination of
>> DMA Requests), PCI devices behind the same uppermost PCI/PCI-X
>> bridge must be co-assigned to the same guest. PCIe devices have not
>> such a constraint.  
>> 
>> FLR (Functional Level Reset) is used to quiesce an assigned device
>> when we destroy a guest or we detach the device from the guest. If a
>> PCIe device has no standard FLR capability, we'll try to use
>> SecondaryBusReset as a way to do FLR. Unluckily, SecondaryBusReset
>> resets all the devices behind the bridge, so for a multi-function
>> PCIe device without FLR capability, we require they have to be
>> co-assigned to the same guest --  certainly this is only for iommu
>> case and for non-iommu case this requirement may be not proper and
>> we shoud fix it...    
> 
> But - for the sake of my education, if you forgive - you didn't
> answer whether there is a one-device-per-secondary-bus restriction on
> PCIe...  

My knowledge to the question is yes, and the "one-device" can be a single-function device or a multi-function device.

Thanks,
-- Dexuan

RE: co-assignment needs for PCI devices

[Ross Philipson]

Thanks for the questions and answers regarding the below topics :) I was just working on related functionality and was arriving at some of the same questions (specifically the last one regarding PCIe one-device-per-secondary-bus).

I had another question/comment related to this topic and the xend code that is locating co-assigned PCI devices. It seems the routine find_the_uppermost_pci_bridge() is meant to find the uppermost bridge device that bridges *TO* a PCI/PCI-X bus. Per the logic this makes sense that it would be any non-PCIe bridge but also a PCIe-to-PCI/PCI-X bridge which is tested for earlier when the device type is determined. So far this all seems to make sense. But in the routine above, the logic seems like it would actually go past what I would think of as the topmost PCI/PCI-X bridge and find the next parent (note that if dev_type == DEV_TYPE_PCIe_BRIDGE, the loop ends but that device is taken to be the target bridge). Is this correct or is there an issue in this logic?

Thanks
Ross

-----Original Message-----
From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Cui, Dexuan
Sent: Tuesday, March 31, 2009 4:11 AM
To: Jan Beulich
Cc: xen-devel@lists.xensource.com
Subject: [Xen-devel] RE: co-assignment needs for PCI devices

Jan Beulich wrote:
>>>> "Cui, Dexuan" <dexuan.cui@intel.com> 31.03.09 09:48 >>>
>> Jan Beulich wrote:
>>> Also, being not really knowledgeable about all the differences
>>> between PCIe and PCI, I'd appreciate some clarification on why on
>>> PCIe it is possible and correct to do a secondary bus reset when
>>> targeting just the (PCIe) functions of a single device - to me this
>>> seems to imply that there's a one-device-per-non-root-bus
>>> restriction there. 
>> According to VT-d spec (section 3.6.1: Identifying Origination of
>> DMA Requests), PCI devices behind the same uppermost PCI/PCI-X
>> bridge must be co-assigned to the same guest. PCIe devices have not
>> such a constraint.  
>> 
>> FLR (Functional Level Reset) is used to quiesce an assigned device
>> when we destroy a guest or we detach the device from the guest. If a
>> PCIe device has no standard FLR capability, we'll try to use
>> SecondaryBusReset as a way to do FLR. Unluckily, SecondaryBusReset
>> resets all the devices behind the bridge, so for a multi-function
>> PCIe device without FLR capability, we require they have to be
>> co-assigned to the same guest --  certainly this is only for iommu
>> case and for non-iommu case this requirement may be not proper and
>> we shoud fix it...    
> 
> But - for the sake of my education, if you forgive - you didn't
> answer whether there is a one-device-per-secondary-bus restriction on
> PCIe...  

My knowledge to the question is yes, and the "one-device" can be a single-function device or a multi-function device.

Thanks,
-- Dexuan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: co-assignment needs for PCI devices

[Cui, Dexuan]

[-- Attachment #1: Type: text/plain, Size: 3596 bytes --]

Hi Ross,
find_the_uppermost_pci_bridge() is used to find the uppermost PCIe-to-PCI/PCI-X bridge.
In the loop of the function, when dev_parent.dev_type == DEV_TYPE_PCIe_BRIDGE and the loop ends, dev.dev_type should be DEV_TYPE_PCI_BRIDGE.
Looks we have no issue here.

Thanks,
-- Dexuan



-----Original Message-----
From: Ross Philipson [mailto:Ross.Philipson@citrix.com] 
Sent: 2009年3月31日 20:22
To: Cui, Dexuan; Jan Beulich
Cc: xen-devel@lists.xensource.com
Subject: RE: co-assignment needs for PCI devices

Thanks for the questions and answers regarding the below topics :) I was just working on related functionality and was arriving at some of the same questions (specifically the last one regarding PCIe one-device-per-secondary-bus).

I had another question/comment related to this topic and the xend code that is locating co-assigned PCI devices. It seems the routine find_the_uppermost_pci_bridge() is meant to find the uppermost bridge device that bridges *TO* a PCI/PCI-X bus. Per the logic this makes sense that it would be any non-PCIe bridge but also a PCIe-to-PCI/PCI-X bridge which is tested for earlier when the device type is determined. So far this all seems to make sense. But in the routine above, the logic seems like it would actually go past what I would think of as the topmost PCI/PCI-X bridge and find the next parent (note that if dev_type == DEV_TYPE_PCIe_BRIDGE, the loop ends but that device is taken to be the target bridge). Is this correct or is there an issue in this logic?

Thanks
Ross

-----Original Message-----
From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Cui, Dexuan
Sent: Tuesday, March 31, 2009 4:11 AM
To: Jan Beulich
Cc: xen-devel@lists.xensource.com
Subject: [Xen-devel] RE: co-assignment needs for PCI devices

Jan Beulich wrote:
>>>> "Cui, Dexuan" <dexuan.cui@intel.com> 31.03.09 09:48 >>>
>> Jan Beulich wrote:
>>> Also, being not really knowledgeable about all the differences
>>> between PCIe and PCI, I'd appreciate some clarification on why on
>>> PCIe it is possible and correct to do a secondary bus reset when
>>> targeting just the (PCIe) functions of a single device - to me this
>>> seems to imply that there's a one-device-per-non-root-bus
>>> restriction there. 
>> According to VT-d spec (section 3.6.1: Identifying Origination of
>> DMA Requests), PCI devices behind the same uppermost PCI/PCI-X
>> bridge must be co-assigned to the same guest. PCIe devices have not
>> such a constraint.  
>> 
>> FLR (Functional Level Reset) is used to quiesce an assigned device
>> when we destroy a guest or we detach the device from the guest. If a
>> PCIe device has no standard FLR capability, we'll try to use
>> SecondaryBusReset as a way to do FLR. Unluckily, SecondaryBusReset
>> resets all the devices behind the bridge, so for a multi-function
>> PCIe device without FLR capability, we require they have to be
>> co-assigned to the same guest --  certainly this is only for iommu
>> case and for non-iommu case this requirement may be not proper and
>> we shoud fix it...    
> 
> But - for the sake of my education, if you forgive - you didn't
> answer whether there is a one-device-per-secondary-bus restriction on
> PCIe...  

My knowledge to the question is yes, and the "one-device" can be a single-function device or a multi-function device.

Thanks,
-- Dexuan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: co-assignment needs for PCI devices

[Ross Philipson]

Ah I see now. I was looking at the trivial case where the first parent was DEV_TYPE_PCIe_BRIDGE but this would never really happen given the code leading up to the call to find_the_uppermost_pci_bridge(). Got it thanks.

Ross

-----Original Message-----
From: Cui, Dexuan [mailto:dexuan.cui@intel.com] 
Sent: Wednesday, April 01, 2009 4:20 AM
To: Ross Philipson; Jan Beulich
Cc: xen-devel@lists.xensource.com
Subject: RE: co-assignment needs for PCI devices

Hi Ross,
find_the_uppermost_pci_bridge() is used to find the uppermost PCIe-to-PCI/PCI-X bridge.
In the loop of the function, when dev_parent.dev_type == DEV_TYPE_PCIe_BRIDGE and the loop ends, dev.dev_type should be DEV_TYPE_PCI_BRIDGE.
Looks we have no issue here.

Thanks,
-- Dexuan



-----Original Message-----
From: Ross Philipson [mailto:Ross.Philipson@citrix.com] 
Sent: 2009年3月31日 20:22
To: Cui, Dexuan; Jan Beulich
Cc: xen-devel@lists.xensource.com
Subject: RE: co-assignment needs for PCI devices

Thanks for the questions and answers regarding the below topics :) I was just working on related functionality and was arriving at some of the same questions (specifically the last one regarding PCIe one-device-per-secondary-bus).

I had another question/comment related to this topic and the xend code that is locating co-assigned PCI devices. It seems the routine find_the_uppermost_pci_bridge() is meant to find the uppermost bridge device that bridges *TO* a PCI/PCI-X bus. Per the logic this makes sense that it would be any non-PCIe bridge but also a PCIe-to-PCI/PCI-X bridge which is tested for earlier when the device type is determined. So far this all seems to make sense. But in the routine above, the logic seems like it would actually go past what I would think of as the topmost PCI/PCI-X bridge and find the next parent (note that if dev_type == DEV_TYPE_PCIe_BRIDGE, the loop ends but that device is taken to be the target bridge). Is this correct or is there an issue in this logic?


Thanks
Ross

-----Original Message-----
From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Cui, Dexuan
Sent: Tuesday, March 31, 2009 4:11 AM
To: Jan Beulich
Cc: xen-devel@lists.xensource.com
Subject: [Xen-devel] RE: co-assignment needs for PCI devices

Jan Beulich wrote:
>>>> "Cui, Dexuan" <dexuan.cui@intel.com> 31.03.09 09:48 >>>
>> Jan Beulich wrote:
>>> Also, being not really knowledgeable about all the differences
>>> between PCIe and PCI, I'd appreciate some clarification on why on
>>> PCIe it is possible and correct to do a secondary bus reset when
>>> targeting just the (PCIe) functions of a single device - to me this
>>> seems to imply that there's a one-device-per-non-root-bus
>>> restriction there. 
>> According to VT-d spec (section 3.6.1: Identifying Origination of
>> DMA Requests), PCI devices behind the same uppermost PCI/PCI-X
>> bridge must be co-assigned to the same guest. PCIe devices have not
>> such a constraint.  
>> 
>> FLR (Functional Level Reset) is used to quiesce an assigned device
>> when we destroy a guest or we detach the device from the guest. If a
>> PCIe device has no standard FLR capability, we'll try to use
>> SecondaryBusReset as a way to do FLR. Unluckily, SecondaryBusReset
>> resets all the devices behind the bridge, so for a multi-function
>> PCIe device without FLR capability, we require they have to be
>> co-assigned to the same guest --  certainly this is only for iommu
>> case and for non-iommu case this requirement may be not proper and
>> we shoud fix it...    
> 
> But - for the sake of my education, if you forgive - you didn't
> answer whether there is a one-device-per-secondary-bus restriction on
> PCIe...  

My knowledge to the question is yes, and the "one-device" can be a single-function device or a multi-function device.

Thanks,
-- Dexuan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel