Re: [PATCH V1 vfio 6/6] vfio/mlx5: Add REINIT support to VFIO_MIG_GET_PRECOPY_INFO

[Peter Xu <peterx@redhat.com>]

Hi, Yishai,

Please feel free to treat my comments as pure questions only.

On Tue, Mar 10, 2026 at 06:40:06PM +0200, Yishai Hadas wrote:
> When userspace opts into VFIO_DEVICE_FEATURE_MIG_PRECOPY_INFOv2, the
> driver may report the VFIO_PRECOPY_INFO_REINIT output flag in response
> to the VFIO_MIG_GET_PRECOPY_INFO ioctl, along with a new initial_bytes
> value.

Does it also mean that VFIO_PRECOPY_INFO_REINIT is almost only a hint that
can be deduced by the userspace too, if it remembers the last time fetch of
initial_bytes?

It definitely sounds a bit weird when some initial_* data can actually
change, because it's not "initial_" anymore.

Another question is, if initial_bytes reached zero, could it be boosted
again to be non-zero?

I don't see what stops it from happening, if the "we get some fresh new
critical data" seem to be able to happen anytime..  but if so, I wonder if
it's a problem to QEMU: when initial_bytes reported to 0 at least _once_ it
means it's possible src QEMU decides to switchover.  Then looks like it
beats the purpose of "don't switchover until we flush the critical data"
whole idea.

Is there a way the HW can report and confidentally say no further critical
data will be generated?

> 
> The presence of the VFIO_PRECOPY_INFO_REINIT flag indicates to the
> caller that new initial data is available in the migration stream.
> 
> If the firmware reports a new initial-data chunk, any previously dirty
> bytes in memory are treated as initial bytes, since the caller must read
> both sets before reaching the end of the initial-data region.

This is unfortunate.  I believe it's a limtation because of the current
single fd streaming protocol, so HW can only append things because it's
kind of a pipeline.

One thing to mention is, I recall VFIO migration suffers from a major
bottleneck on read() of the VFIO FD, it means this streaming whole design
is also causing other perf issues.

Have you or anyone thought about making it not a stream anymore?  Take
example of RAM blocks: it is pagesize accessible, with that we can do a lot
more, e.g. we don't need to streamline pages, we can send pages in whatever
order.  Meanwhile, we can send pages concurrently because they're not
streamlined too.

I wonder if VFIO FDs can provide something like that too, as a start it
doesn't need to be as fine granule, maybe at least instead of using one
stream it can provide two streams, one for initial_bytes (or, I really
think this should be called "critical data" or something similar, if it
represents that rather than "some initial states", not anymore), another
one for dirty.  Then at least when you attach new critical data you don't
need to flush dirty queue too.

If to extend it a bit more, then we can also make e.g. dirty queue to be
multiple FDs, so that userspace can read() in multiple threads, speeding up
the switchover phase.

I had a vague memory that there's sometimes kernel big locks to block it,
but from interfacing POV it sounds always better to avoid using one fd to
stream everything.

Thanks,

> 
> In this case, the driver issues a new SAVE command to fetch the data and
> prepare it for a subsequent read() from userspace.

-- 
Peter Xu