Re: [virtio-comment] Live Migration of Virtio Virtual Function

[Jason Gunthorpe <jgg@nvidia.com>]

On Tue, Aug 24, 2021 at 10:41:54AM +0800, Jason Wang wrote:

> > migration exposed to the guest ? No.
> 
> Can you explain why?

For the SRIOV case migration is a privileged operation of the
hypervisor. The guest must not be allowed to interact with it in any
way otherwise the hypervisor migration could be attacked from the
guest and this has definite security implications.

In practice this means that nothing related to migration can be
located on the MMIO pages/queues/etc of the VF. The reasons for this
are a bit complicated and has to do with the limitations of IO
isolation with VFIO - eg you can't reliably split a single PCI BDF
into hypervisor/guest security domains without PASID.

We recently revisited this concept again with a HNS vfio driver. IIRC
Intel messed it up in their mdev driver too.

> > >>> Let's open another thread for this if you wish, it has nothing related
> > >>> to the spec but how it is implemented in Linux. If you search the
> > >>> archive, something similar to "vfio_virtio_pci" has been proposed
> > >>> several years before by Intel. The idea has been rejected, and we have
> > >>> leveraged Linux vDPA bus for virtio-pci devices.

That was largely because Intel was proposing to use mdevs to create an
entire VDPA subsystem hidden inside VFIO.

We've invested in a pure VFIO solution which should be merged soon:

https://lore.kernel.org/kvm/20210819161914.7ad2e80e.alex.williamson@redhat.com/

It does not rely on mdevs. It is not trying to recreate VDPA. Instead
the HW provides a fully functional virto VF and the solution uses
normal SRIOV approaches.

You can contrast this with the two virtio-net solutions mlx5 will
support:

- One is the existing hypervisor assisted VDPA solution where the mlx5
  driver does HW accelerated queue processing.

- The other one is a full PCI VF that provides a virtio-net function
  without any hypervisor assistance. In this case we will have a VFIO
  migration driver as above that to provide SRIOV VF live migration.

I see in this thread that these two things are becoming quite
confused. They are very different, have different security postures
and use different parts of the hypervisor stack, and intended for
quite different use cases.

> Your proposal works only for PCI with SR-IOV. And I want to leverage
> it to be useful for other platforms or transport. That's all my
> motivation.

I've read most of the emails here I still don't see what the use case
is for this beyond PCI SRIOV.

In a general sense it requires virtio to specify how PASID works. No
matter what we must create a split secure/guest world where DMAs from
each world are uniquely tagged. In the pure PCI world this means
either using PF/VF or VF/PASID.

In general PASID still has a long road to go before it is working in
Linux:

https://lore.kernel.org/kvm/BN9PR11MB5433B1E4AE5B0480369F97178C189@BN9PR11MB5433.namprd11.prod.outlook.com/

So, IMHO, it make sense to focus on the PF/VF definition for spec
purposes.

I agree it would be good spec design to have a general concept of a
secure and guest world and specific sections that defines how it works
for different scenarios, but that seems like a language remark and not
one about the design. For instance the admin queue Max is adding is
clearly part of the secure world and putting it on the PF is the only
option for the SRIOV mode.

Jason