RE: [PATCH v11 05/11] vfio/migration: Block multiple devices migration

[Shameerali Kolothum Thodi via <qemu-devel@nongnu.org>]

> -----Original Message-----
> From: Alex Williamson [mailto:alex.williamson@redhat.com]
> Sent: 16 May 2023 15:28
> To: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>
> Cc: Jason Gunthorpe <jgg@nvidia.com>; Avihai Horon <avihaih@nvidia.com>;
> qemu-devel@nongnu.org; Juan Quintela <quintela@redhat.com>; Dr. David
> Alan Gilbert <dgilbert@redhat.com>; Michael S. Tsirkin <mst@redhat.com>;
> Cornelia Huck <cohuck@redhat.com>; Paolo Bonzini
> <pbonzini@redhat.com>; Vladimir Sementsov-Ogievskiy
> <vsementsov@yandex-team.ru>; Cédric Le Goater <clg@redhat.com>; Yishai
> Hadas <yishaih@nvidia.com>; Maor Gottlieb <maorg@nvidia.com>; Kirti
> Wankhede <kwankhede@nvidia.com>; Tarun Gupta <targupta@nvidia.com>;
> Joao Martins <joao.m.martins@oracle.com>
> Subject: Re: [PATCH v11 05/11] vfio/migration: Block multiple devices
> migration
> 
> On Tue, 16 May 2023 13:57:22 +0000
> Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>
> wrote:
> 
> > > -----Original Message-----
> > > From: Jason Gunthorpe [mailto:jgg@nvidia.com]
> > > Sent: 16 May 2023 13:00
> > > To: Shameerali Kolothum Thodi
> <shameerali.kolothum.thodi@huawei.com>
> > > Cc: Avihai Horon <avihaih@nvidia.com>; qemu-devel@nongnu.org; Alex
> > > Williamson <alex.williamson@redhat.com>; Juan Quintela
> > > <quintela@redhat.com>; Dr. David Alan Gilbert <dgilbert@redhat.com>;
> > > Michael S. Tsirkin <mst@redhat.com>; Cornelia Huck
> <cohuck@redhat.com>;
> > > Paolo Bonzini <pbonzini@redhat.com>; Vladimir Sementsov-Ogievskiy
> > > <vsementsov@yandex-team.ru>; Cédric Le Goater <clg@redhat.com>;
> Yishai
> > > Hadas <yishaih@nvidia.com>; Maor Gottlieb <maorg@nvidia.com>; Kirti
> > > Wankhede <kwankhede@nvidia.com>; Tarun Gupta
> <targupta@nvidia.com>;
> > > Joao Martins <joao.m.martins@oracle.com>
> > > Subject: Re: [PATCH v11 05/11] vfio/migration: Block multiple devices
> > > migration
> > >
> > > On Tue, May 16, 2023 at 10:03:54AM +0000, Shameerali Kolothum Thodi
> > > wrote:
> > >
> > > > > Currently VFIO migration doesn't implement some kind of
> intermediate
> > > > > quiescent state in which P2P DMAs are quiesced before stopping or
> > > > > running the device. This can cause problems in multi-device migration
> > > > > where the devices are doing P2P DMAs, since the devices are not
> stopped
> > > > > together at the same time.
> > > > >
> > > > > Until such support is added, block migration of multiple devices.
> > > >
> > > > Missed this one. Currently this blocks even if the attached devices are
> not
> > > > capable of P2P DMAs. eg; HiSilicon ACC devices. These are integrated
> end
> > > point
> > > > devices without any P2P capability between them. Is it Ok to check for
> > > > VFIO_MIGRATION_P2P flag and allow if the devices are not supporting
> > > that?
> > >
> > > Lacking VFIO_MIGRATION_P2P doesn't mean the device is incapable of
> > > P2P, it means the migration can't support P2P.
> > >
> > > We'd need some kind of new flag to check and such devices should be
> > > blocked from creating P2P mappings. Basically we don't currently
> > > fully support devices that are incapable of P2P operations.
> >
> > Ok. I will take a look.
> >
> > > What happens on your platform if a guest tries to do P2P? Does the
> > > platform crash?
> >
> > I am not sure. Since the devices are behind SMMU, I was under the
> assumption
> > that we do have the guarantee of isolation here(grouping). Or this is
> something
> > we are worried only during migration?
> 
> Grouping doesn't guarantee that mappings cannot be created through the
> SMMU between devices.  When we refer to devices being isolated between
> groups, that only excludes internal P2P between devices, for example
> across the internal link between functions with implementation specific
> routing.  For group isolation, the guarantee is that DMA is always
> routed upstream, not that the ultimate target cannot be another device.
> To guarantee lack of P2P the SMMU would need to reject non-memory
> translation targets.  Thanks,

Ok. Got it. So it depends on what SMMU does for that mapping and is not
related to migration per se and has the potential to crash the system if 
SMMU go ahead with that memory access. Isn't it a more generic problem
then when we have multiple devices attached to the VM? I need to check if 
there is anything in SMMU spec that forbids this access.

Thanks,
Shameer