Re: vhost-user (virtio-fs) migration: back end state

[Hanna Czenczek <hreitz@redhat.com>]

On 06.02.23 22:02, Juan Quintela wrote:
> Stefan Hajnoczi <stefanha@gmail.com> wrote:
>> On Mon, 6 Feb 2023 at 07:36, Hanna Czenczek <hreitz@redhat.com> wrote:
>>> Hi Stefan,
>>>
>>> For true virtio-fs migration, we need to migrate the daemon’s (back
>>> end’s) state somehow.  I’m addressing you because you had a talk on this
>>> topic at KVM Forum 2021. :)
>>>
>>> As far as I understood your talk, the only standardized way to migrate a
>>> vhost-user back end’s state is via dbus-vmstate.  I believe that
>>> interface is unsuitable for our use case, because we will need to
>>> migrate more than 1 MB of state.  Now, that 1 MB limit has supposedly
>>> been chosen arbitrarily, but the introducing commit’s message says that
>>> it’s based on the idea that the data must be supplied basically
>>> immediately anyway (due to both dbus and qemu migration requirements),
>>> and I don’t think we can meet that requirement.
>> Yes, dbus-vmstate is the available today. It's independent of
>> vhost-user and VIRTIO.
> Once that we are here:
> - typical size of your starte (either vhost-user or whatever)

Difficult to say, completely depends on the use case.  When identifying 
files by path and organizing them in a tree structure, probably ~48 
bytes per indexed file, plus, say, 16 bytes per open file.

So for a small shared filesystem, the state can be very small, but we’ll 
also have to prepare for cases where it is in the range of several MB.

The main problem isn’t size but that (when identifying files by path) 
we’ll probably want to construct the paths when migrating, which won’t 
be done instantaneously.

> - what are the posibilities that you can enter the iterative stage
>    negotiation (i.e. that you can create a dirty bitmap about your state)

Very good.  We should know when parts of the state (information about a 
specific indexed or open file) changes.  (Exceptions apply, but they 
mostly come down to whether indexed files are identified by path or file 
handle, which is a choice the user will probably need to make.  Either 
one comes with caveats.)

[...]

>>> Of course, we could use a channel that completely bypasses qemu, but I
>>> think we’d like to avoid that if possible.  First, this would require
>>> adding functionality to virtiofsd to configure this channel.  Second,
>>> not storing the state in the central VM state means that migrating to
>>> file doesn’t work (well, we could migrate to a dedicated state file,
>>> but...).
> How much is migration to file used in practice?
> I would like to have some information here.
> It could be necessary probably to be able to encrypt it.  And that is a
> (different) whole can of worms.

I don’t think virtio-fs state needs to be encrypted any more than any 
other state.  It’ll basically just map FUSE inode IDs to a file in the 
shared directory, either via path or file handle; and then also track 
open(2) flags for opened files.  (At least that’s what’s currently on my 
radar.)  That information should actually be replicated in the guest, 
too (because it too will have mapped the filesystem paths to FUSE inode 
IDs), so isn’t more security relevant than guest memory.

Hanna