Re: vhost-user (virtio-fs) migration: back end state

[Juan Quintela <quintela@redhat.com>]

Hanna Czenczek <hreitz@redhat.com> wrote:
> On 06.02.23 22:02, Juan Quintela wrote:
>> Stefan Hajnoczi <stefanha@gmail.com> wrote:
>>> On Mon, 6 Feb 2023 at 07:36, Hanna Czenczek <hreitz@redhat.com> wrote:
>>>> Hi Stefan,
>>>>
>>>> For true virtio-fs migration, we need to migrate the daemon’s (back
>>>> end’s) state somehow.  I’m addressing you because you had a talk on this
>>>> topic at KVM Forum 2021. :)
>>>>
>>>> As far as I understood your talk, the only standardized way to migrate a
>>>> vhost-user back end’s state is via dbus-vmstate.  I believe that
>>>> interface is unsuitable for our use case, because we will need to
>>>> migrate more than 1 MB of state.  Now, that 1 MB limit has supposedly
>>>> been chosen arbitrarily, but the introducing commit’s message says that
>>>> it’s based on the idea that the data must be supplied basically
>>>> immediately anyway (due to both dbus and qemu migration requirements),
>>>> and I don’t think we can meet that requirement.
>>> Yes, dbus-vmstate is the available today. It's independent of
>>> vhost-user and VIRTIO.
>> Once that we are here:
>> - typical size of your starte (either vhost-user or whatever)
>
> Difficult to say, completely depends on the use case.  When
> identifying files by path and organizing them in a tree structure,
> probably ~48 bytes per indexed file, plus, say, 16 bytes per open
> file.
>
> So for a small shared filesystem, the state can be very small, but
> we’ll also have to prepare for cases where it is in the range of
> several MB.

That is not two bad.  Anything below a few tens megabytes is easy to
manage.  Anything in the hundred of megabytes or more really need thought.

> The main problem isn’t size but that (when identifying files by path)
> we’ll probably want to construct the paths when migrating, which won’t
> be done instantaneously.
>
>> - what are the posibilities that you can enter the iterative stage
>>    negotiation (i.e. that you can create a dirty bitmap about your state)
>
> Very good.  We should know when parts of the state (information about
> a specific indexed or open file) changes.  (Exceptions apply, but they
> mostly come down to whether indexed files are identified by path or
> file handle, which is a choice the user will probably need to make. 
> Either one comes with caveats.)

That is good.

>> How much is migration to file used in practice?
>> I would like to have some information here.
>> It could be necessary probably to be able to encrypt it.  And that is a
>> (different) whole can of worms.
>
> I don’t think virtio-fs state needs to be encrypted any more than any
> other state.  It’ll basically just map FUSE inode IDs to a file in the
> shared directory, either via path or file handle; and then also track
> open(2) flags for opened files.  (At least that’s what’s currently on
> my radar.)  That information should actually be replicated in the
> guest, too (because it too will have mapped the filesystem paths to
> FUSE inode IDs), so isn’t more security relevant than guest memory.

Oh, that was not about virtio-fs at all.

Is because you talked about file migration.

Right now, we need to use exec migration to do this.  but it is clearly
suboptimal.  Basically we just do a normal migration, but that means
that we have a lot of duplicated pages on the destination.

But we can do better.  Just create a file that is as big as the memory,
and write everypage in its own place.  So loading is going to be really
fast.  (yes holes on RAM is a different issue, but we can ignore that
for now).
And the other thing is that we have to really encrypt it somehow, so I
guess that a block cipher should work, but encryption is not my field of
expertise at all.

In vhost-user-fs case, I fully agree with you that if you are
"exporting" part of the local filesystem, encryption don't buy you
anything.

Later, Juan.