Re: Plan for /dev/ioasid RFC v2

[Jason Gunthorpe <jgg@nvidia.com>]

On Fri, Jun 18, 2021 at 04:57:40PM +0000, Tian, Kevin wrote:
> > From: Jason Gunthorpe <jgg@nvidia.com>
> > Sent: Friday, June 18, 2021 8:20 AM
> > 
> > On Thu, Jun 17, 2021 at 03:14:52PM -0600, Alex Williamson wrote:
> > 
> > > I've referred to this as a limitation of type1, that we can't put
> > > devices within the same group into different address spaces, such as
> > > behind separate vRoot-Ports in a vIOMMU config, but really, who cares?
> > > As isolation support improves we see fewer multi-device groups, this
> > > scenario becomes the exception.  Buy better hardware to use the devices
> > > independently.
> > 
> > This is basically my thinking too, but my conclusion is that we should
> > not continue to make groups central to the API.
> > 
> > As I've explained to David this is actually causing functional
> > problems and mess - and I don't see a clean way to keep groups central
> > but still have the device in control of what is happening. We need
> > this device <-> iommu connection to be direct to robustly model all
> > the things that are in the RFC.
> > 
> > To keep groups central someone needs to sketch out how to solve
> > today's mdev SW page table and mdev PASID issues in a clean
> > way. Device centric is my suggestion on how to make it clean, but I
> > haven't heard an alternative??
> > 
> > So, I view the purpose of this discussion to scope out what a
> > device-centric world looks like and then if we can securely fit in the
> > legacy non-isolated world on top of that clean future oriented
> > API. Then decide if it is work worth doing or not.
> > 
> > To my mind it looks like it is not so bad, granted not every detail is
> > clear, and no code has be sketched, but I don't see a big scary
> > blocker emerging. An extra ioctl or two, some special logic that
> > activates for >1 device groups that looks a lot like VFIO's current
> > logic..
> > 
> > At some level I would be perfectly fine if we made the group FD part
> > of the API for >1 device groups - except that complexifies every user
> > space implementation to deal with that. It doesn't feel like a good
> > trade off.
> > 
> 
> Would it be an acceptable tradeoff by leaving >1 device groups 
> supported only via legacy VFIO (which is anyway kept for backward 
> compatibility), if we think such scenario is being deprecated over 
> time (thus little value to add new features on it)? Then all new 
> sub-systems including vdpa and new vfio only support singleton 
> device group via /dev/iommu...

That might just be a great idea - userspace has to support those APIs
anyhow, if it can be made trivially obvious to use this fallback even
though /dev/iommu is available it is a great place to start. It also
means PASID/etc are naturally blocked off.

Maybe years down the road we will want to harmonize them, so I would
still sketch it out enough to be confident it could be implemented..

Jason