Determining iommu groups in Xen?

Determining iommu groups in Xen?

[Peter Kay]

This should be a simple question, but I can't find the answer : how are iommu groups determined/found in Xen?

I've used KVM before, and the use of the vfio framework makes it easy to find the iommu groups. Unless a 'will never be approved' patch is applied to the Linux kernel, it is impossible to pass through only one device to a KVM VM out of a group due to the lack of ACS (iommu protection between devices on the same bus segment).

Xen does not seem to enforce this - why not, especially as it can cause security and stability issues?

PK
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: Determining iommu groups in Xen?

[Andrew Cooper]

On 28/08/14 14:26, Peter Kay wrote:
> This should be a simple question, but I can't find the answer : how are iommu groups determined/found in Xen?
>
> I've used KVM before, and the use of the vfio framework makes it easy to find the iommu groups. Unless a 'will never be approved' patch is applied to the Linux kernel, it is impossible to pass through only one device to a KVM VM out of a group due to the lack of ACS (iommu protection between devices on the same bus segment).
>
> Xen does not seem to enforce this - why not, especially as it can cause security and stability issues?
>
> PK

>From memory, ACS is present to fix an interaction issue between PCI
Passthrough and peer-to-peer dma translations in the PCIe spec.  ACS
instructions a bridge/switch to forward the transaction to the upstream
port for translation by the IOMMU instead of resolving it privately as a
peer-to-peer transaction.

Xen uses the domain identifier to determine which iommu context to apply
to different dma requests.  Without ACS, it is not safe (or indeed
functional in general) to pass through different devices behind a PCIe
switch to different domains.  The problem is that if two different
domains program different devices with overlapping guest physical
addresses, the peer-to-peer option in PCIe causes the dma traffic to be
bounced between the two devices, rather than ending up being translated
into separate domains memory ranges.

Unfortunately, a lot of 1st era PCIe switches after ACS was specified
have errata, caused by an ambiguity in the spec, which means that
despite claiming ACS support, they don't function correctly.  In some
cases, there are workarounds, but in other cases there are not.

It is sad to say that passing through multiple devices behind a switch
to multiple domains doesn't work very well in a lot of cases.

Certainly within XenServer, we state that customers using PCIPassthrough
must trust the guest administrators, which 'fixes' the security aspect
of things from the point of view of malicious guests.

~Andrew

Re: Determining iommu groups in Xen?

[Peter Kay]

 

On 28 August 2014 14:54:01 BST, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
>On 28/08/14 14:26, Peter Kay wrote:
>> This should be a simple question, but I can't find the answer : how
>are iommu groups determined/found in Xen?
...snip...
>From memory, ACS is present to fix an interaction issue between PCI
>Passthrough and peer-to-peer dma translations in the PCIe spec.  ACS
>instructions a bridge/switch to forward the transaction to the upstream
>port for translation by the IOMMU instead of resolving it privately as
>a peer-to-peer transaction.

Yes, precisely.

>Unfortunately, a lot of 1st era PCIe switches after ACS was specified
>have errata, caused by an ambiguity in the spec, which means that
>despite claiming ACS support, they don't function correctly.  

Yes. KVM has a list of quirks from Intel specifying which chipsets actually work.

>Certainly within XenServer, we state that customers using
>PCIPassthrough must trust the guest administrators, which
>'fixes' the security aspect of things from the point of view
> of malicious guests.
>
>~Andrew
Fair enough; possibly not ideal but it's an administrator function with calculated risk. A warning might be nice, though.

The more important question is : how do I determine the iommu groups? Do I need to write some code? It isn't possible to discover from the motherboard manual (it lies) and furthermore, as expansion cards are added and removed the iommu groups change slightly as do the PCI IDs (which is also annoying as it's then necessary to edit all the domU config files plus the pciback hide parameters. KVM is just as broken here).

I currently have an issue with the system spontaneously rebooting under both 4.4 and -unstable as soon as I pass through the entire USB controller (or the half on one set or other of PCI IDs). I don't know if it's a bug or something daft is happening with iommu groups or similar.

On the bright side, USB host device passthrough with a virtual USB controller appears to work well under Windows, Linux and FreeBSD. KVM doesn't fare so well under *BSD.

PK
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: Determining iommu groups in Xen?

[Andrew Cooper]

On 28/08/14 17:48, Peter Kay wrote:
>  
>
> On 28 August 2014 14:54:01 BST, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
>> On 28/08/14 14:26, Peter Kay wrote:
>>> This should be a simple question, but I can't find the answer : how
>> are iommu groups determined/found in Xen?
> ...snip...
> >From memory, ACS is present to fix an interaction issue between PCI
>> Passthrough and peer-to-peer dma translations in the PCIe spec.  ACS
>> instructions a bridge/switch to forward the transaction to the upstream
>> port for translation by the IOMMU instead of resolving it privately as
>> a peer-to-peer transaction.
> Yes, precisely.
>
>> Unfortunately, a lot of 1st era PCIe switches after ACS was specified
>> have errata, caused by an ambiguity in the spec, which means that
>> despite claiming ACS support, they don't function correctly.  
> Yes. KVM has a list of quirks from Intel specifying which chipsets actually work.
>
>> Certainly within XenServer, we state that customers using
>> PCIPassthrough must trust the guest administrators, which
>> 'fixes' the security aspect of things from the point of view
>> of malicious guests.
>>
>> ~Andrew
> Fair enough; possibly not ideal but it's an administrator function with calculated risk. A warning might be nice, though.

I am not aware of a single server platform which doesn't have a single
erraturm which breaks the end-to-end security or functionality of PCI
Passthrough.  I would love to be proved wrong in this regard.

>
> The more important question is : how do I determine the iommu groups? Do I need to write some code? It isn't possible to discover from the motherboard manual (it lies) and furthermore, as expansion cards are added and removed the iommu groups change slightly as do the PCI IDs (which is also annoying as it's then necessary to edit all the domU config files plus the pciback hide parameters. KVM is just as broken here).

I am confused as to what exactly you mean by iommu groups in this
context.  My initial guess of the iommu context identifiers for HAP/EPT
tables was clearly wrong.

>
> I currently have an issue with the system spontaneously rebooting under both 4.4 and -unstable as soon as I pass through the entire USB controller (or the half on one set or other of PCI IDs). I don't know if it's a bug or something daft is happening with iommu groups or similar.

Almost certainly to do with the (lack of correct) RMRR support.  There
is a patch series on-list attempting to remedy this problem.

~Andrew

>
> On the bright side, USB host device passthrough with a virtual USB controller appears to work well under Windows, Linux and FreeBSD. KVM doesn't fare so well under *BSD.
>
> PK

Re: Determining iommu groups in Xen?

[Peter Kay]

On 28 August 2014 18:13:07 BST, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
>On 28/08/14 17:48, Peter Kay wrote:
>> Fair enough; possibly not ideal but it's an administrator function
>with calculated risk. A warning might be nice, though.
>
>I am not aware of a single server platform which doesn't have a single
>erraturm which breaks the end-to-end security or functionality of PCI
>Passthrough.  I would love to be proved wrong in this regard.

I will try and find the PCI quirk posts for KVM.

>I am confused as to what exactly you mean by iommu groups in this
>context.  My initial guess of the iommu context identifiers for HAP/EPT
>tables was clearly wrong.
An iommu group, as far as I'm aware, is the group of devices that are not protected from each other. In KVM, you must pass through the entire group to a VM at once, unless a 'don't go crying to me if it stomps over your memory space or worse' patch is applied to the kennel claiming that everything is fine.


>Almost certainly to do with the (lack of correct) RMRR support.  There
>is a patch series on-list attempting to remedy this problem.
Thank you, I'll search for this. It does pass through reliably in KVM, which is good, because their virtual USB sucks. I'm reasonably certain that my S3210SHLC motherboard is quite solid, provided you accept some of the interesting design decisions. (Well, apart from leaving the X38 audio chipset inaccessible, but present enough to confuse the Linux kernel to hang on boot occasionally)

PK
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: Determining iommu groups in Xen?

[Andrew Cooper]

On 28/08/14 18:53, Peter Kay wrote:
>
> On 28 August 2014 18:13:07 BST, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
>> On 28/08/14 17:48, Peter Kay wrote:
>>> Fair enough; possibly not ideal but it's an administrator function
>> with calculated risk. A warning might be nice, though.
>>
>> I am not aware of a single server platform which doesn't have a single
>> erraturm which breaks the end-to-end security or functionality of PCI
>> Passthrough.  I would love to be proved wrong in this regard.
> I will try and find the PCI quirk posts for KVM.
>
>> I am confused as to what exactly you mean by iommu groups in this
>> context.  My initial guess of the iommu context identifiers for HAP/EPT
>> tables was clearly wrong.
> An iommu group, as far as I'm aware, is the group of devices that are not protected from each other. In KVM, you must pass through the entire group to a VM at once, unless a 'don't go crying to me if it stomps over your memory space or worse' patch is applied to the kennel claiming that everything is fine.

I have googled the term in the meantime, and it is what I initially thought.

All PCI devices passed though to the same domain share the same single
"iommu group" per Kernel/KVM terminology.  There is not currently any
support for multiple iommu contexts within a single VM.

~Andrew

>
>
>> Almost certainly to do with the (lack of correct) RMRR support.  There
>> is a patch series on-list attempting to remedy this problem.
> Thank you, I'll search for this. It does pass through reliably in KVM, which is good, because their virtual USB sucks. I'm reasonably certain that my S3210SHLC motherboard is quite solid, provided you accept some of the interesting design decisions. (Well, apart from leaving the X38 audio chipset inaccessible, but present enough to confuse the Linux kernel to hang on boot occasionally)
>
> PK

Re: Determining iommu groups in Xen?

[Peter Kay]

 

On 28 August 2014 19:02:47 BST, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
>On 28/08/14 18:53, Peter Kay wrote:
>>
>> On 28 August 2014 18:13:07 BST, Andrew Cooper
><andrew.cooper3@citrix.com> wrote:

>> An iommu group, as far as I'm aware, is the group of devices that are
>not protected from each other. In KVM, you must pass through the entire
>group to a VM at once, unless a 'don't go crying to me if it stomps
>over your memory space or worse' patch is applied to the kennel
>claiming that everything is fine.
>
>I have googled the term in the meantime, and it is what I initially
>thought.
>
>All PCI devices passed though to the same domain share the same single
>"iommu group" per Kernel/KVM terminology.  There is not currently any
>support for multiple iommu contexts within a single VM.
>
>~Andrew
No, it's not that. The iommu group relates to the root port above the devices.

I can say from experience it's perfectly acceptable to have multiple iommu groups assigned to one VM in both Xen and KVM. Usually I have a PCIe graphics card (attached to a PCIe root so in its own group) and three PCI cards on a PCI segment which IIRC is attached to a PCIe to PCI bridge via a PCIe root, in another group. They're all ultimately hanging off the ICH9R PCH, but that shouldn't matter.

I'll see if I can find a more definitive explanation in a bit

PK
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: Determining iommu groups in Xen?

[Peter Kay]

[-- Attachment #1.1: Type: text/plain, Size: 2261 bytes --]

On 28 August 2014 19:45, Peter Kay <syllopsium@syllopsium.co.uk> wrote:

>
>
> On 28 August 2014 19:02:47 BST, Andrew Cooper <andrew.cooper3@citrix.com>
> wrote:
> >On 28/08/14 18:53, Peter Kay wrote:
> >>
> >> On 28 August 2014 18:13:07 BST, Andrew Cooper
> ><andrew.cooper3@citrix.com> wrote:
>
> >> An iommu group, as far as I'm aware, is the group of devices that are
> >not protected from each other. In KVM, you must pass through the entire
> >group to a VM at once, unless a 'don't go crying to me if it stomps
> >over your memory space or worse' patch is applied to the kennel
> >claiming that everything is fine.
> >
> >I have googled the term in the meantime, and it is what I initially
> >thought.
> >
> >All PCI devices passed though to the same domain share the same single
> >"iommu group" per Kernel/KVM terminology.  There is not currently any
> >support for multiple iommu contexts within a single VM.
> >
> >~Andrew
>

See  http://lxr.free-electrons.com/source/drivers/iommu/iommu.c  and
intel-iommu.c (or amd-iommu.c). It is based on the ACS capability of the
upstream device. See in particular intel_iommu_add_device()

>From  https://www.kernel.org/doc/Documentation/vfio.txt

'Therefore, while for the most part an IOMMU may have device level
granularity, any system is susceptible to reduced granularity.  The
IOMMU API therefore supports a notion of IOMMU groups.  A group is
a set of devices which is isolatable from all other devices in the
system.  Groups are therefore the unit of ownership used by VFIO'

So far as reliable quirks go for ACS protection, see drivers/pci/quirks.c
static const u16 pci_quirk_intel_pch_acs_ids[] and Red Hat bugzilla 1037684

I'll have to do some more testing to see if lspci -t is a reasonable
indication of iommu groups or if I can write some code to figure them out.

Obviously returning the information from the Linux source is ultimately not
really a good idea(*), because the dom0 may not be Linux. It is in my case,
because NetBSD is (unfortunately) not yet functional enough for my needs
and I don't want to use Solaris derived OS, but that doesn't help everyone
else.

(*) Assuming it's possible at all, as the Linux dom0 is running on top of
Xen and therefore is restricted in some ways.

PK

[-- Attachment #1.2: Type: text/html, Size: 3461 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Determining iommu groups in Xen?

[Andrew Cooper]

[-- Attachment #1.1: Type: text/plain, Size: 3299 bytes --]

On 29/08/2014 01:35, Peter Kay wrote:
>
>
> On 28 August 2014 19:45, Peter Kay <syllopsium@syllopsium.co.uk
> <mailto:syllopsium@syllopsium.co.uk>> wrote:
>
>
>
>     On 28 August 2014 19:02:47 BST, Andrew Cooper
>     <andrew.cooper3@citrix.com <mailto:andrew.cooper3@citrix.com>> wrote:
>     >On 28/08/14 18:53, Peter Kay wrote:
>     >>
>     >> On 28 August 2014 18:13:07 BST, Andrew Cooper
>     ><andrew.cooper3@citrix.com <mailto:andrew.cooper3@citrix.com>> wrote:
>
>     >> An iommu group, as far as I'm aware, is the group of devices
>     that are
>     >not protected from each other. In KVM, you must pass through the
>     entire
>     >group to a VM at once, unless a 'don't go crying to me if it stomps
>     >over your memory space or worse' patch is applied to the kennel
>     >claiming that everything is fine.
>     >
>     >I have googled the term in the meantime, and it is what I initially
>     >thought.
>     >
>     >All PCI devices passed though to the same domain share the same
>     single
>     >"iommu group" per Kernel/KVM terminology.  There is not currently any
>     >support for multiple iommu contexts within a single VM.
>     >
>     >~Andrew
>
>  
> See  http://lxr.free-electrons.com/source/drivers/iommu/iommu.c  and
> intel-iommu.c (or amd-iommu.c). It is based on the ACS capability of
> the upstream device. See in particular intel_iommu_add_device()
>
> From  https://www.kernel.org/doc/Documentation/vfio.txt
>
> 'Therefore, while for the most part an IOMMU may have device level
> granularity, any system is susceptible to reduced granularity.  The
> IOMMU API therefore supports a notion of IOMMU groups.  A group is
> a set of devices which is isolatable from all other devices in the
> system.  Groups are therefore the unit of ownership used by VFIO'
>
> So far as reliable quirks go for ACS protection, see
> drivers/pci/quirks.c static const u16 pci_quirk_intel_pch_acs_ids[]
> and Red Hat bugzilla 1037684
>
> I'll have to do some more testing to see if lspci -t is a reasonable
> indication of iommu groups or if I can write some code to figure them out.
>
> Obviously returning the information from the Linux source is
> ultimately not really a good idea(*), because the dom0 may not be
> Linux. It is in my case, because NetBSD is (unfortunately) not yet
> functional enough for my needs and I don't want to use Solaris derived
> OS, but that doesn't help everyone else.
>
> (*) Assuming it's possible at all, as the Linux dom0 is running on top
> of Xen and therefore is restricted in some ways.
>
> PK

Ah right.  I see now.  The IOMMU groups are kernel/errata logic applied
to the system which impose restrictions as to which devices cannot
safely/functionally be split apart.

There is absolutely nothing like this in Xen, or dom0 (as dom0 is
unaware of IOMMUs in general).  If I recall correctly, it does feature
on the wishlist of the XenServer team of which I am am member, pending
some copious quantites of free time.  I know for certain that the libxl
and Xapi toolstacks do not have logic like this, leaving all passthrough
setup in the manual hands of the host administrator.

Konrad: Probably an item for the 4.6 wishlist/featurelist.  It will
probably mix well with the other IO-NUMA stuff which has been deferred.

~Andrew

[-- Attachment #1.2: Type: text/html, Size: 6270 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Determining iommu groups in Xen?

[Konrad Rzeszutek Wilk]

On Fri, Aug 29, 2014 at 09:27:14AM +0100, Andrew Cooper wrote:
> On 29/08/2014 01:35, Peter Kay wrote:
> >
> >
> > On 28 August 2014 19:45, Peter Kay <syllopsium@syllopsium.co.uk
> > <mailto:syllopsium@syllopsium.co.uk>> wrote:
> >
> >
> >
> >     On 28 August 2014 19:02:47 BST, Andrew Cooper
> >     <andrew.cooper3@citrix.com <mailto:andrew.cooper3@citrix.com>> wrote:
> >     >On 28/08/14 18:53, Peter Kay wrote:
> >     >>
> >     >> On 28 August 2014 18:13:07 BST, Andrew Cooper
> >     ><andrew.cooper3@citrix.com <mailto:andrew.cooper3@citrix.com>> wrote:
> >
> >     >> An iommu group, as far as I'm aware, is the group of devices
> >     that are
> >     >not protected from each other. In KVM, you must pass through the
> >     entire
> >     >group to a VM at once, unless a 'don't go crying to me if it stomps
> >     >over your memory space or worse' patch is applied to the kennel
> >     >claiming that everything is fine.
> >     >
> >     >I have googled the term in the meantime, and it is what I initially
> >     >thought.
> >     >
> >     >All PCI devices passed though to the same domain share the same
> >     single
> >     >"iommu group" per Kernel/KVM terminology.  There is not currently any
> >     >support for multiple iommu contexts within a single VM.
> >     >
> >     >~Andrew
> >
> >  
> > See  http://lxr.free-electrons.com/source/drivers/iommu/iommu.c  and
> > intel-iommu.c (or amd-iommu.c). It is based on the ACS capability of
> > the upstream device. See in particular intel_iommu_add_device()
> >
> > From  https://www.kernel.org/doc/Documentation/vfio.txt
> >
> > 'Therefore, while for the most part an IOMMU may have device level
> > granularity, any system is susceptible to reduced granularity.  The
> > IOMMU API therefore supports a notion of IOMMU groups.  A group is
> > a set of devices which is isolatable from all other devices in the
> > system.  Groups are therefore the unit of ownership used by VFIO'
> >
> > So far as reliable quirks go for ACS protection, see
> > drivers/pci/quirks.c static const u16 pci_quirk_intel_pch_acs_ids[]
> > and Red Hat bugzilla 1037684
> >
> > I'll have to do some more testing to see if lspci -t is a reasonable
> > indication of iommu groups or if I can write some code to figure them out.
> >
> > Obviously returning the information from the Linux source is
> > ultimately not really a good idea(*), because the dom0 may not be
> > Linux. It is in my case, because NetBSD is (unfortunately) not yet
> > functional enough for my needs and I don't want to use Solaris derived
> > OS, but that doesn't help everyone else.
> >
> > (*) Assuming it's possible at all, as the Linux dom0 is running on top
> > of Xen and therefore is restricted in some ways.
> >
> > PK
> 
> Ah right.  I see now.  The IOMMU groups are kernel/errata logic applied
> to the system which impose restrictions as to which devices cannot
> safely/functionally be split apart.
> 
> There is absolutely nothing like this in Xen, or dom0 (as dom0 is
> unaware of IOMMUs in general).  If I recall correctly, it does feature
> on the wishlist of the XenServer team of which I am am member, pending
> some copious quantites of free time.  I know for certain that the libxl
> and Xapi toolstacks do not have logic like this, leaving all passthrough
> setup in the manual hands of the host administrator.
> 
> Konrad: Probably an item for the 4.6 wishlist/featurelist.  It will
> probably mix well with the other IO-NUMA stuff which has been deferred.

Done.