Re: [Intel-wired-lan] [PATCH iwl-next V2 10/15] ice: save and restore TX queue head

From: Peter Xu <peterx@redhat.com>
To: Jason Gunthorpe <jgg@nvidia.com>
Cc: "Tian, Kevin" <kevin.tian@intel.com>,
	"Liu, Yi L" <yi.l.liu@intel.com>,
	"intel-wired-lan@lists.osuosl.org"
	<intel-wired-lan@lists.osuosl.org>,
	"Burra, Phani R" <phani.r.burra@intel.com>
Subject: Re: [Intel-wired-lan] [PATCH iwl-next V2 10/15] ice: save and restore TX queue head
Date: Mon, 17 Jul 2023 17:43:34 -0400	[thread overview]
Message-ID: <ZLW2BgWI+yx7qKaf@x1n> (raw)
In-Reply-To: <ZKwpm1k+1hjN7oyE@nvidia.com>

On Mon, Jul 10, 2023 at 12:54:03PM -0300, Jason Gunthorpe wrote:
> On Tue, Jul 04, 2023 at 01:59:49PM -0400, Peter Xu wrote:
> 
> > > This sounds counter-intuitive even for emulated devices. vIOMMU is
> > > involved only when the device wants to access guest memory. But
> > > by definition the RESUMING path should just restore the device to
> > > the point where it is stopped. Why would such restore create a
> > > dependency on vIOMMU state?
> > 
> > For why QEMU migrates vIOMMU earlier than PCI generic devices, I can try to
> > repeat here: IIRC it's because some device needs dma translations during
> > post_load(), I forgot which device and why, but it can make some sense to
> > me if recovering states of the pci device may need help from a translation.
> 
> Well, it has nothing to do with VFIO given the timeframe, and if
> internal qemu devices are doing weird things like this it also
> suggests they have problems with P2P as well.
> 
> The VFIO definition we have is that a device may not do any DMA while in
> resuming/resuming_p2p, it is only allowed to initiate DMA once it
> fully reaches running.
> 
> Obviously we have to setup the vIOMMU before going to running.
> 
> The internal qemu devices must follow this definition as well or they
> are not P2P compatible.
> 
> About the only exception I could think is if some internal devices was
> caching translations for some purpose, but it really should setup that
> caching at entry to running..

Yes it's something like that which is a pure translation request, rather
than doing a real DMA, as far as I remember.  It does sound fine to
postpone that into a vm state change handler to me, but worth checking when
someone really works on it, and also whether it's feasible - e.g. hopefully
only a few devices need to work that around.

It may boil down to why we need to avoid migrating vIOMMU before other
devices (which I am still unsure about..), and which way is easier.

Thanks,

-- 
Peter Xu

_______________________________________________
Intel-wired-lan mailing list
Intel-wired-lan@osuosl.org
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan