Re: [RFC PATCH 13/15] iommufd: Persist iommu domains for live update

[Jason Gunthorpe <jgg@ziepe.ca>]

On Thu, Oct 09, 2025 at 09:28:44PM -0400, Pasha Tatashin wrote:
> On Thu, Oct 2, 2025 at 1:37 PM Jason Gunthorpe <jgg@ziepe.ca> wrote:
> >
> > On Thu, Oct 02, 2025 at 10:03:05AM -0700, Samiullah Khawaja wrote:
> > > > I think the simplest thing is the domain exists forever until
> > > > userspace attaches an iommufd, takes ownership of it and frees it.
> > > > Nothing to do with finish.
> > >
> > > Hmm.. I think this is tricky. There needs to be a way to clean up and
> > > discard the old state if the userspace doesn't need it.
> >
> > Why?
> >
> > Isn't "userspace doesn't need it" some extermely weird unused corner
> > case?
> 
> It might be a corner case, but at cloud scale, even rare cases happen.
> For example, if four VMs are resumed and one crashes while retrieving
> half of its resources, we can't simply reboot the machine because of
> that. We must have a way to recover the machine to a normal state,
> even if some resources are not reclaimed. I would say that finish must
> be properly backward-ordered, but we still should release resources
> that are not reclaimed during finish, as well as those that were
> reclaimed but later closed.

Sure, but as I said, userspace should deal with most of this, and I
think we should lean into the worst error flows end up "leaking"
resources. They are not actually leaked, the luo still holds them and
userspace could still try again later to restore and free them. They
will get cleaned up on the next kexec, and kexec to recover from a
partially failed kexec is not an unreasonable plan...

This means think carefully about the userspace restore sequence so it
is more reliable. Like don't restore the memfd as the first thing :)

Only if there are real measurements that this is not sufficent would I
think about teaching the kernel to do a non-restore flow where it
directly destroys the object in a way that cannot fail. Eg the memfd
can directly free the page list instead of allocating an xarray. This
is alot more complex error path code to add to the kernel so lets not
do it without a strong justification.

You also can't do it until something sequences the vfio and iommufd
parts to unfreeze the memfd, this is very complicated error flows as
well.

Jason