towards a workable O_DIRECT outmigration to a file

[Nikolay Borisov <nborisov@suse.com>]

Hello,

I'm currently looking into implementing a 'file:' uri for migration save 
in qemu. Ideally the solution will be O_DIRECT compatible. I'm aware of 
the branch https://gitlab.com/berrange/qemu/-/tree/mig-file. In the 
process of brainstorming how a solution would like the a couple of 
questions transpired that I think warrant wider discussion in the community.

First, implementing a solution which is self-contained within qemu would 
be easy enough( famous last words) but the gist is one  has to only care 
about the format within qemu. However, I'm being told that what libvirt 
does is prepend its own custom header to the resulting saved file, then 
slipstreams the migration stream from qemu. Now with the solution that I 
envision I intend to keep all write-related logic inside qemu, this 
means there's no way to incorporate the logic of libvirt. The reason I'd 
like to keep the write process within qemu is to avoid an extra copy of 
data between the two processes (qemu outging migration and libvirt), 
with the current fd approach qemu is passed an fd, data is copied 
between qemu/libvirt and finally the libvirt_iohelper writes the data. 
So the question which remains to be answered is how would libvirt make 
use of this new functionality in qemu? I was thinking something along 
the lines of :

1. Qemu writes its migration stream to a file, ideally on a filesystem 
which supports reflink - xfs/btrfs

2. Libvirt writes it's header to a separate file
2.1 Reflinks the qemu's stream right after its header
2.2 Writes its trailer

3. Unlink() qemu's file, now only libvirt's file remains on-disk.

I wouldn't call this solution hacky though it definitely leaves some 
bitter aftertaste.


Another solution would be to extend the 'fd:' protocol to allow multiple 
descriptors (for multifd) support to be passed in. The reason dup() 
can't be used is because in order for multifd to be supported it's 
required to be able to write to multiple, non-overlapping regions of the 
file. And duplicated fd's share their offsets etc. But that really seems 
more or less hacky. Alternatively it's possible that pwrite() are used 
to write to non-overlapping regions in the file. Any feedback is welcomed.


Regards,
Nikolay