Live migration support for virtio-fs

Live migration support for virtio-fs

[Ge, Xiongzi]

Thanks Dr Gilbert, Vivek, Stefan, Greg!
I put together the discussion into this thread and CC qemu-devel@nongnu.org. 

Problem:
Current Virtio-FS does not support live migration.  Even when the virtiofs directory is not mounted on the guest, VM cannot do live migration. Any suggestions/interest If we want to make this work so we can still have the live migration feature for other purpose on that VM? 

If we already have a shared file system like Ceph, does this make it different? 

Thanks,
Xiongzi

---------
Stefan's reply:
The virtio-fs device holds a bunch of state, like the ino_map, dirp_map,
and the fd_map in the virtiofsd C implementation. That is the FUSE
session state that needs to be migrated in order to resume seamlessly
(without unmounting and mounting again).

If the backend is a distributed file system then it my have APIs that
make migration easier. If it's possible to re-attach to open files from
another host then that is perfect. But on the flipside if there are no
APIs for doing that then it might be impossible to reliably live
migration because some state cannot be transferred between hosts.

Max Reitz is laying the foundation for live migration by working on
open_by_handle_at(2) support. This will probably be needed in order to
transfer open files from one host to another.

It should be possible to enable live migration when the filesystem is
not mounted. That might be a good first step to enabling live migration.

------------
From Greg Kurz:
Live migration of virtio-fs is still at the early discussion stage AFAICT.
It might take time before we have something working. But in the meantime,
it seems abusive to block migration if we have a guarantee that the device
isn't servicing requests. FWIW virtio-9p only blocks migration when the
shared directory is mounted on the guest.


On 3/23/21, 11:01 AM, "Dr. David Alan Gilbert" <dgilbert@redhat.com> wrote:

    * Vivek Goyal (vgoyal@redhat.com) wrote:
    > On Tue, Mar 23, 2021 at 10:29:09AM +0000, Dr. David Alan Gilbert wrote:
    > > * Ge, Xiongzi (Xiongzi.Ge@netapp.com) wrote:
    > > > Hello Vivek, Dr. Gilbert, and the virtio-fs team,
    > >
    > > Hi Xiongzi,
    > >
    > > > It seems that virtio-fs does not support live migration. Once a vm
    > > > is configured with virtio-fs, live migration cannot be performed even
    > > > it is not mounted on the guest.
    > >
    > > Right.
    > >
    > > > Is there any progress for this?
    > >
    > > Max Reitz is looking at some parts of it; in particular storing file
    > > handles that can later be reopened.
    > > But there are a bunch of other parts we've not looked at yet either
    > > (like the dirty page marking around all syscalls).
    > >
    > > > If the
    > > > shared directory is from a distributed file system like Ceph, would it
    > > > be easier than the general case to be implemented?
    > >
    > > Maybe; there are some tricky semantics problems; for example, lets
    > > imagine that you open the file   'a/b/c'  on the source, and sometime
    > > after you open it, 'b' gets renamed to 'd';  when you resume on the
    > > destination you need to make sure you know how to get to that file.
    > > Depending on the filesystem semantics you might need to make that work
    > > even if 'a/b/c' had been deleted but you still had it open.
    >
    > Hi Dave,
    >
    > I am assuming that with file handles, renaming of file probably is not
    > a problem. open_by_handle_at() will still be able to find it.

    Right; although again with soemthing like CEPH you might not even need
    that if you had a virtiofs daemon that spoke direct to CEPH, you might
    be dealing with a CEPH filehandle.

    > I think real problem (as you pointed out later) is unliked file which is
    > still in use by the virtiofsd.

    Dave

    > Thanks
    > Vivek
    --
    Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Virtio-fs] Live migration support for virtio-fs

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2187 bytes --]

On Wed, Apr 07, 2021 at 06:05:29PM +0000, Ge, Xiongzi wrote:
> Thanks Dr Gilbert, Vivek, Stefan, Greg!
> I put together the discussion into this thread and CC qemu-devel@nongnu.org. 
> 
> Problem:
> Current Virtio-FS does not support live migration.  Even when the virtiofs directory is not mounted on the guest, VM cannot do live migration. Any suggestions/interest If we want to make this work so we can still have the live migration feature for other purpose on that VM? 
> 
> If we already have a shared file system like Ceph, does this make it different? 

(Marc-André: I CCed you regarding D-Bus VMState, see below.)

During the last virtio-fs call on Wednesday we discussed how to allow
live migration when the file system is not mounted. This means live
migration of a mounted file system still wouldn't be supported, but at
least it would be possible to migrate when the file system is not
mounted.

The easiest way to do that is to hot unplug the vhost-user-fs-pci device
so that QEMU can live migrate the VM and then hot plug the
vhost-user-fs-pci device again after migration has completed. This
requires either manually performing the hot plug operations or some
automation to do this around live migration.

A smoother approach would be for QEMU to allow live migration when the
file system is not mounted. Unfortunately it turns out this is actually
rather involved because virtiofsd runs as a separate process from QEMU
and the live migration interface for vhost-user devices is currently
pretty basic
(https://qemu.readthedocs.io/en/latest/interop/dbus-vmstate.html). There
is currently no way for virtiofsd to communicate with QEMU to allow/deny
live migration depending on whether virtiofsd currently has an active
FUSE session (mount). It might be possible to extend the D-Bus VMState
interface and implement support for "migration blockers" as they are
known in QEMU, but I wonder whether it's not better to just work on
full virtio-fs live migration instead. Either task is complex and I
wouldn't recommend it as a first contribution to virtiofsd or QEMU.

Can you make use of the hotplug approach to live migration?

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Virtio-fs] Live migration support for virtio-fs

[Ge, Xiongzi]

Thanks, Stefan!
It seems that libvirt does not support device-attach for filesystem type device, like virtiofs. Are there any other ways to do this hot-unplugging? 

Xiongzi

On 4/8/21, 12:21 PM, "Stefan Hajnoczi" <stefanha@redhat.com> wrote:

    On Wed, Apr 07, 2021 at 06:05:29PM +0000, Ge, Xiongzi wrote:
    > Thanks Dr Gilbert, Vivek, Stefan, Greg!
    > I put together the discussion into this thread and CC qemu-devel@nongnu.org. 
    > 
    > Problem:
    > Current Virtio-FS does not support live migration.  Even when the virtiofs directory is not mounted on the guest, VM cannot do live migration. Any suggestions/interest If we want to make this work so we can still have the live migration feature for other purpose on that VM? 
    > 
    > If we already have a shared file system like Ceph, does this make it different? 

    (Marc-André: I CCed you regarding D-Bus VMState, see below.)

    During the last virtio-fs call on Wednesday we discussed how to allow
    live migration when the file system is not mounted. This means live
    migration of a mounted file system still wouldn't be supported, but at
    least it would be possible to migrate when the file system is not
    mounted.

    The easiest way to do that is to hot unplug the vhost-user-fs-pci device
    so that QEMU can live migrate the VM and then hot plug the
    vhost-user-fs-pci device again after migration has completed. This
    requires either manually performing the hot plug operations or some
    automation to do this around live migration.

    A smoother approach would be for QEMU to allow live migration when the
    file system is not mounted. Unfortunately it turns out this is actually
    rather involved because virtiofsd runs as a separate process from QEMU
    and the live migration interface for vhost-user devices is currently
    pretty basic
    (https://qemu.readthedocs.io/en/latest/interop/dbus-vmstate.html). There
    is currently no way for virtiofsd to communicate with QEMU to allow/deny
    live migration depending on whether virtiofsd currently has an active
    FUSE session (mount). It might be possible to extend the D-Bus VMState
    interface and implement support for "migration blockers" as they are
    known in QEMU, but I wonder whether it's not better to just work on
    full virtio-fs live migration instead. Either task is complex and I
    wouldn't recommend it as a first contribution to virtiofsd or QEMU.

    Can you make use of the hotplug approach to live migration?

    Stefan

Re: [Virtio-fs] Live migration support for virtio-fs

[Vivek Goyal]

Hi Xiongzi,

Try qemu monitor to hot device add/remove that works. I think libvirt
support is yet to be added.

One issue I faced was that I had to add a pcie root to move virtiofs
device off pcie.0 and then hot add remove functionality works.

So I had to add pcie root port by adding following to qemu command
line.

-device pcie-root-port,id=root1,slot=0

And now you can hot add a device using device_add.

 (qemu) device_add vhost-user-fs-pci,bus=root1,id=myvirtiofs,chardev=char0,tag=myfs,queue-size=1024

- To hot remove device, issue from qemu monitor.

  (qemu) device_del <device_id>
  ex. device_del myvirtiofs

Thanks
Vivek

On Mon, May 03, 2021 at 12:50:56PM +0000, Ge, Xiongzi wrote:
> Thanks, Stefan!
> It seems that libvirt does not support device-attach for filesystem type device, like virtiofs. Are there any other ways to do this hot-unplugging? 
> 
> Xiongzi
> 
> On 4/8/21, 12:21 PM, "Stefan Hajnoczi" <stefanha@redhat.com> wrote:
> 
>     On Wed, Apr 07, 2021 at 06:05:29PM +0000, Ge, Xiongzi wrote:
>     > Thanks Dr Gilbert, Vivek, Stefan, Greg!
>     > I put together the discussion into this thread and CC qemu-devel@nongnu.org. 
>     > 
>     > Problem:
>     > Current Virtio-FS does not support live migration.  Even when the virtiofs directory is not mounted on the guest, VM cannot do live migration. Any suggestions/interest If we want to make this work so we can still have the live migration feature for other purpose on that VM? 
>     > 
>     > If we already have a shared file system like Ceph, does this make it different? 
> 
>     (Marc-André: I CCed you regarding D-Bus VMState, see below.)
> 
>     During the last virtio-fs call on Wednesday we discussed how to allow
>     live migration when the file system is not mounted. This means live
>     migration of a mounted file system still wouldn't be supported, but at
>     least it would be possible to migrate when the file system is not
>     mounted.
> 
>     The easiest way to do that is to hot unplug the vhost-user-fs-pci device
>     so that QEMU can live migrate the VM and then hot plug the
>     vhost-user-fs-pci device again after migration has completed. This
>     requires either manually performing the hot plug operations or some
>     automation to do this around live migration.
> 
>     A smoother approach would be for QEMU to allow live migration when the
>     file system is not mounted. Unfortunately it turns out this is actually
>     rather involved because virtiofsd runs as a separate process from QEMU
>     and the live migration interface for vhost-user devices is currently
>     pretty basic
>     (https://qemu.readthedocs.io/en/latest/interop/dbus-vmstate.html). There
>     is currently no way for virtiofsd to communicate with QEMU to allow/deny
>     live migration depending on whether virtiofsd currently has an active
>     FUSE session (mount). It might be possible to extend the D-Bus VMState
>     interface and implement support for "migration blockers" as they are
>     known in QEMU, but I wonder whether it's not better to just work on
>     full virtio-fs live migration instead. Either task is complex and I
>     wouldn't recommend it as a first contribution to virtiofsd or QEMU.
> 
>     Can you make use of the hotplug approach to live migration?
> 
>     Stefan
>