Re: [RFC] Independent use of IOMMU groups

[Alex Williamson <alex.williamson-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>]

On Fri, 2015-11-27 at 16:39 +0100, Joerg Roedel wrote:
> Hi Alex,
> 
> On Fri, Nov 06, 2015 at 08:35:40AM -0700, Alex Williamson wrote:
> > VFIO is really built on iommu groups, so making a vfio group independent
> > of iommu groups is a difficult proposition.
> 
> I have been thinking about the relation between vfio device groups and
> iommu-groups lately, because at least for PCI the iommu-grouping is too
> coarse grained. I ran into this with the default-domain approach I am
> working on.
> 
> Grouping devices together that have different request-ids (multifunction
> and acs based grouping) only makes sense when the device is controlled
> by an untrusted piece of software, in our case userspace or a KVM guest.
> The device drivers in Linux are trusted, and this coarse grained
> grouping becomes problematic, because it forces more devices into a
> single domain, which can become a bottleneck for DMA-API allocations.
> 
> I have been thinking about moving the multi-function and acs grouping
> into vfio code, meaning that a vfio-group contains more than one
> iommu-group. The problem with this is that iommu-groups are exposed
> in sysfs and thus became a userspace ABI.
> 
> So the vfio-group code might need changes anyway which could solve the
> above problem too, no? I am just not sure yet what the best way is to
> solve it.

Hi Joerg,

That's a hard one.  As you say, iommu groups are really a userspace ABI
and tightly integrated into the mapping of vfio groups, so I don't
really think we have much flexibility in re-defining an iommu group.
The original intent with putting the grouping logic in the iommu drivers
and core code was that vfio isn't smart enough to be able to determine
both the iommu visibility and topology based isolation for any given
architecture.  I think that's still true.  We don't really want to
enable vfio on platforms that haven't given the issue sufficient
consideration to enable the iommu API for devices.

On the other hand, it doesn't make a whole lot of sense for native
kernel drivers to care about topology based isolation.  It would be
preferable to fully isolate a device, but we do manage the IOVA address
space for native drivers, so unintentional peer-to-peer shouldn't really
be a possibility.

It makes sense to me to think about an iommu group as a set of one or
more iommu granules, where each granule is the granularity of the iommu
visibility.  A granule probably also has a relation to DMA aliases.  So
perhaps the granule encompasses all DMA related visibility issues and
the group is an overlay which takes topology into account when the IOVA
space is defined by the user (and malicious DMA needs to be considered
as well).

So maybe the first step is to create that dividing line and figure out
what granules look like and how we can more explicitly expose groups on
top of them.  Easier said than done, I'm sure.  Thanks,

Alex

PS - sorry for the delay, was off on holiday.