Re: [PATCH v2 28/29] migration: Add direct-io parameter

[Fabiano Rosas <farosas@suse.de>]

Daniel P. Berrangé <berrange@redhat.com> writes:

> On Tue, Oct 31, 2023 at 12:52:41PM -0300, Fabiano Rosas wrote:
>> Daniel P. Berrangé <berrange@redhat.com> writes:
>> >
>> > I guess I'm not seeing the problem still.  A single FD is passed across
>> > from libvirt, but QEMU is free to turn that into *many* FDs for its
>> > internal use, using dup() and then setting O_DIRECT on as many/few of
>> > the dup()d FDs as its wants to.
>> 
>> The problem is that duplicated FDs share the file status flags. If we
>> set O_DIRECT on the multifd channels and the main thread happens to do
>> an unaligned write with qemu_file_put* then the filesystem will fail
>> that write.
>
> Doh, I had forgotten that sharing.
>
> Do we have any synchronization between multifd  channels and the main
> thread ?  eg does the main thread wait for RAM sending completion
> before carrying on writing other non-RAM data ?

We do have, but the issue with that approach is that there are no rules
for adding data into the stream. Anyone could add a qemu_put_* call
right in the middle of the section for whatever reason.

That is almost a separate matter due to our current compatibility model
being based on capabilities rather than resilience of the stream
format. So extraneous data in the stream always causes the migration to
break.

But with the O_DIRECT situation we'd be adding another aspect to
this. Not only changing the code requires syncing capabilities (as it
does today), but it would also require knowing which parts of the stream
can be interrupted by new data and which cannot.

So while it would probably work, it's also a little fragile. If QEMU
were given 2 FDs or given access to the file, then only the multifd
channels would get O_DIRECT and they are guaranteed to not have
extraneous unaligned data showing up.