Re: [PATCH v2 18/18] migration/ram: Add direct-io support to precopy file migration

[Jim Fehlig <jfehlig@suse.com>]

On 6/7/24 12:42 PM, Fabiano Rosas wrote:
> Peter Xu <peterx@redhat.com> writes:
> 
>> On Thu, May 23, 2024 at 04:05:48PM -0300, Fabiano Rosas wrote:
>>> We've recently added support for direct-io with multifd, which brings
>>> performance benefits, but creates a non-uniform user interface by
>>> coupling direct-io with the multifd capability. This means that users
>>> cannot keep the direct-io flag enabled while disabling multifd.
>>>
>>> Libvirt in particular already has support for direct-io and parallel
>>> migration separately from each other, so it would be a regression to
>>> now require both options together. It's relatively simple for QEMU to
>>> add support for direct-io migration without multifd, so let's do this
>>> in order to keep both options decoupled.
>>>
>>> We cannot simply enable the O_DIRECT flag, however, because not all IO
>>> performed by the migration thread satisfies the alignment requirements
>>> of O_DIRECT. There are many small read & writes that add headers and
>>> synchronization flags to the stream, which at the moment are required
>>> to always be present.
>>>
>>> Fortunately, due to fixed-ram migration there is a discernible moment
>>> where only RAM pages are written to the migration file. Enable
>>> direct-io during that moment.
>>>
>>> Signed-off-by: Fabiano Rosas <farosas@suse.de>
>>
>> Is anyone going to consume this?  How's the performance?
> 
> I don't think we have a pre-determined consumer for this. This came up
> in an internal discussion about making the interface simpler for libvirt
> and in a thread on the libvirt mailing list[1] about using O_DIRECT to
> keep the snapshot data out of the caches to avoid impacting the rest of
> the system. (I could have described this better in the commit message,
> sorry).
> 
> Quoting Daniel:
> 
>    "Note the reason for using O_DIRECT is *not* to make saving / restoring
>     the guest VM faster. Rather it is to ensure that saving/restoring a VM
>     does not trash the host I/O / buffer cache, which will negatively impact
>     performance of all the *other* concurrently running VMs."
> 
> 1- https://lore.kernel.org/r/87sez86ztq.fsf@suse.de
> 
> About performance, a quick test on a stopped 30G guest, shows
> mapped-ram=on direct-io=on it's 12% slower than mapped-ram=on
> direct-io=off.
> 
>>
>> It doesn't look super fast to me if we need to enable/disable dio in each
>> loop.. then it's a matter of whether we should bother, or would it be
>> easier that we simply require multifd when direct-io=on.
> 
> AIUI, the issue here that users are already allowed to specify in
> libvirt the equivalent to direct-io and multifd independent of each
> other (bypass-cache, parallel). To start requiring both together now in
> some situations would be a regression. I confess I don't know libvirt
> code to know whether this can be worked around somehow, but as I said,
> it's a relatively simple change from the QEMU side.

Currently, libvirt does not support --parallel with virDomainSave* and 
virDomainRestore* APIs. I'll work on that after getting support for mapped-ram 
merged. --parallel is supported in virDomainMigrate* APIs, but obviously those 
APIs don't accept --bypass-cache.

Regards,
Jim

> 
> Another option which would be for libvirt to keep using multifd, but
> make it 1 channel only if --parallel is not specified. That might be
> enough to solve the interface issues. Of course, it's a different code
> altogether than the usual precopy code that gets executed when
> multifd=off, I don't know whether that could be an issue somehow.