* [Virtio-fs] Live migration support for virtio-fs
@ 2021-03-23 1:35 Ge, Xiongzi
2021-03-23 9:02 ` Greg Kurz
` (2 more replies)
0 siblings, 3 replies; 14+ messages in thread
From: Ge, Xiongzi @ 2021-03-23 1:35 UTC (permalink / raw)
To: Vivek Goyal; +Cc: virtio-fs@redhat.com
Hello Vivek, Dr. Gilbert, and the virtio-fs team,
It seems that virtio-fs does not support live migration. Once a vm is configured with virtio-fs, live migration cannot be performed even it is not mounted on the guest. Is there any progress for this? If the shared directory is from a distributed file system like Ceph, would it be easier than the general case to be implemented?
Thanks,
Xiongzi
^ permalink raw reply [flat|nested] 14+ messages in thread* Re: [Virtio-fs] Live migration support for virtio-fs 2021-03-23 1:35 [Virtio-fs] Live migration support for virtio-fs Ge, Xiongzi @ 2021-03-23 9:02 ` Greg Kurz 2021-03-23 10:29 ` Dr. David Alan Gilbert 2021-03-23 13:33 ` Stefan Hajnoczi 2 siblings, 0 replies; 14+ messages in thread From: Greg Kurz @ 2021-03-23 9:02 UTC (permalink / raw) To: Ge, Xiongzi; +Cc: virtio-fs@redhat.com, Vivek Goyal On Tue, 23 Mar 2021 01:35:58 +0000 "Ge, Xiongzi" <Xiongzi.Ge@netapp.com> wrote: > Hello Vivek, Dr. Gilbert, and the virtio-fs team, > > It seems that virtio-fs does not support live migration. Once a vm is > configured with virtio-fs, live migration cannot be performed even it > is not mounted on the guest. Is there any progress for this? If the Live migration of virtio-fs is still at the early discussion stage AFAICT. It might take time before we have something working. But in the meantime, it seems abusive to block migration if we have a guarantee that the device isn't servicing requests. FWIW virtio-9p only blocks migration when the shared directory is mounted on the guest. > shared directory is from a distributed file system like Ceph, would it > be easier than the general case to be implemented? > > Thanks, > Xiongzi > > > > _______________________________________________ > Virtio-fs mailing list > Virtio-fs@redhat.com > https://listman.redhat.com/mailman/listinfo/virtio-fs > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [Virtio-fs] Live migration support for virtio-fs 2021-03-23 1:35 [Virtio-fs] Live migration support for virtio-fs Ge, Xiongzi 2021-03-23 9:02 ` Greg Kurz @ 2021-03-23 10:29 ` Dr. David Alan Gilbert 2021-03-23 14:53 ` Vivek Goyal 2021-03-23 13:33 ` Stefan Hajnoczi 2 siblings, 1 reply; 14+ messages in thread From: Dr. David Alan Gilbert @ 2021-03-23 10:29 UTC (permalink / raw) To: Ge, Xiongzi; +Cc: virtio-fs@redhat.com, Vivek Goyal * Ge, Xiongzi (Xiongzi.Ge@netapp.com) wrote: > Hello Vivek, Dr. Gilbert, and the virtio-fs team, Hi Xiongzi, > It seems that virtio-fs does not support live migration. Once a vm > is configured with virtio-fs, live migration cannot be performed even > it is not mounted on the guest. Right. > Is there any progress for this? Max Reitz is looking at some parts of it; in particular storing file handles that can later be reopened. But there are a bunch of other parts we've not looked at yet either (like the dirty page marking around all syscalls). > If the > shared directory is from a distributed file system like Ceph, would it > be easier than the general case to be implemented? Maybe; there are some tricky semantics problems; for example, lets imagine that you open the file 'a/b/c' on the source, and sometime after you open it, 'b' gets renamed to 'd'; when you resume on the destination you need to make sure you know how to get to that file. Depending on the filesystem semantics you might need to make that work even if 'a/b/c' had been deleted but you still had it open. Dave > > Thanks, > Xiongzi > > -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [Virtio-fs] Live migration support for virtio-fs 2021-03-23 10:29 ` Dr. David Alan Gilbert @ 2021-03-23 14:53 ` Vivek Goyal 2021-03-23 15:00 ` Dr. David Alan Gilbert 0 siblings, 1 reply; 14+ messages in thread From: Vivek Goyal @ 2021-03-23 14:53 UTC (permalink / raw) To: Dr. David Alan Gilbert; +Cc: virtio-fs@redhat.com, Ge, Xiongzi On Tue, Mar 23, 2021 at 10:29:09AM +0000, Dr. David Alan Gilbert wrote: > * Ge, Xiongzi (Xiongzi.Ge@netapp.com) wrote: > > Hello Vivek, Dr. Gilbert, and the virtio-fs team, > > Hi Xiongzi, > > > It seems that virtio-fs does not support live migration. Once a vm > > is configured with virtio-fs, live migration cannot be performed even > > it is not mounted on the guest. > > Right. > > > Is there any progress for this? > > Max Reitz is looking at some parts of it; in particular storing file > handles that can later be reopened. > But there are a bunch of other parts we've not looked at yet either > (like the dirty page marking around all syscalls). > > > If the > > shared directory is from a distributed file system like Ceph, would it > > be easier than the general case to be implemented? > > Maybe; there are some tricky semantics problems; for example, lets > imagine that you open the file 'a/b/c' on the source, and sometime > after you open it, 'b' gets renamed to 'd'; when you resume on the > destination you need to make sure you know how to get to that file. > Depending on the filesystem semantics you might need to make that work > even if 'a/b/c' had been deleted but you still had it open. Hi Dave, I am assuming that with file handles, renaming of file probably is not a problem. open_by_handle_at() will still be able to find it. I think real problem (as you pointed out later) is unliked file which is still in use by the virtiofsd. Thanks Vivek ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [Virtio-fs] Live migration support for virtio-fs 2021-03-23 14:53 ` Vivek Goyal @ 2021-03-23 15:00 ` Dr. David Alan Gilbert 2021-04-07 18:05 ` Ge, Xiongzi 0 siblings, 1 reply; 14+ messages in thread From: Dr. David Alan Gilbert @ 2021-03-23 15:00 UTC (permalink / raw) To: Vivek Goyal; +Cc: virtio-fs@redhat.com, Ge, Xiongzi * Vivek Goyal (vgoyal@redhat.com) wrote: > On Tue, Mar 23, 2021 at 10:29:09AM +0000, Dr. David Alan Gilbert wrote: > > * Ge, Xiongzi (Xiongzi.Ge@netapp.com) wrote: > > > Hello Vivek, Dr. Gilbert, and the virtio-fs team, > > > > Hi Xiongzi, > > > > > It seems that virtio-fs does not support live migration. Once a vm > > > is configured with virtio-fs, live migration cannot be performed even > > > it is not mounted on the guest. > > > > Right. > > > > > Is there any progress for this? > > > > Max Reitz is looking at some parts of it; in particular storing file > > handles that can later be reopened. > > But there are a bunch of other parts we've not looked at yet either > > (like the dirty page marking around all syscalls). > > > > > If the > > > shared directory is from a distributed file system like Ceph, would it > > > be easier than the general case to be implemented? > > > > Maybe; there are some tricky semantics problems; for example, lets > > imagine that you open the file 'a/b/c' on the source, and sometime > > after you open it, 'b' gets renamed to 'd'; when you resume on the > > destination you need to make sure you know how to get to that file. > > Depending on the filesystem semantics you might need to make that work > > even if 'a/b/c' had been deleted but you still had it open. > > Hi Dave, > > I am assuming that with file handles, renaming of file probably is not > a problem. open_by_handle_at() will still be able to find it. Right; although again with soemthing like CEPH you might not even need that if you had a virtiofs daemon that spoke direct to CEPH, you might be dealing with a CEPH filehandle. > I think real problem (as you pointed out later) is unliked file which is > still in use by the virtiofsd. Dave > Thanks > Vivek -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK ^ permalink raw reply [flat|nested] 14+ messages in thread
* [Virtio-fs] Live migration support for virtio-fs 2021-03-23 15:00 ` Dr. David Alan Gilbert @ 2021-04-07 18:05 ` Ge, Xiongzi 0 siblings, 0 replies; 14+ messages in thread From: Ge, Xiongzi @ 2021-04-07 18:05 UTC (permalink / raw) To: Dr. David Alan Gilbert, Vivek Goyal Cc: virtio-fs@redhat.com, qemu-devel@nongnu.org Thanks Dr Gilbert, Vivek, Stefan, Greg! I put together the discussion into this thread and CC qemu-devel@nongnu.org. Problem: Current Virtio-FS does not support live migration. Even when the virtiofs directory is not mounted on the guest, VM cannot do live migration. Any suggestions/interest If we want to make this work so we can still have the live migration feature for other purpose on that VM? If we already have a shared file system like Ceph, does this make it different? Thanks, Xiongzi --------- Stefan's reply: The virtio-fs device holds a bunch of state, like the ino_map, dirp_map, and the fd_map in the virtiofsd C implementation. That is the FUSE session state that needs to be migrated in order to resume seamlessly (without unmounting and mounting again). If the backend is a distributed file system then it my have APIs that make migration easier. If it's possible to re-attach to open files from another host then that is perfect. But on the flipside if there are no APIs for doing that then it might be impossible to reliably live migration because some state cannot be transferred between hosts. Max Reitz is laying the foundation for live migration by working on open_by_handle_at(2) support. This will probably be needed in order to transfer open files from one host to another. It should be possible to enable live migration when the filesystem is not mounted. That might be a good first step to enabling live migration. ------------ >From Greg Kurz: Live migration of virtio-fs is still at the early discussion stage AFAICT. It might take time before we have something working. But in the meantime, it seems abusive to block migration if we have a guarantee that the device isn't servicing requests. FWIW virtio-9p only blocks migration when the shared directory is mounted on the guest. On 3/23/21, 11:01 AM, "Dr. David Alan Gilbert" <dgilbert@redhat.com> wrote: * Vivek Goyal (vgoyal@redhat.com) wrote: > On Tue, Mar 23, 2021 at 10:29:09AM +0000, Dr. David Alan Gilbert wrote: > > * Ge, Xiongzi (Xiongzi.Ge@netapp.com) wrote: > > > Hello Vivek, Dr. Gilbert, and the virtio-fs team, > > > > Hi Xiongzi, > > > > > It seems that virtio-fs does not support live migration. Once a vm > > > is configured with virtio-fs, live migration cannot be performed even > > > it is not mounted on the guest. > > > > Right. > > > > > Is there any progress for this? > > > > Max Reitz is looking at some parts of it; in particular storing file > > handles that can later be reopened. > > But there are a bunch of other parts we've not looked at yet either > > (like the dirty page marking around all syscalls). > > > > > If the > > > shared directory is from a distributed file system like Ceph, would it > > > be easier than the general case to be implemented? > > > > Maybe; there are some tricky semantics problems; for example, lets > > imagine that you open the file 'a/b/c' on the source, and sometime > > after you open it, 'b' gets renamed to 'd'; when you resume on the > > destination you need to make sure you know how to get to that file. > > Depending on the filesystem semantics you might need to make that work > > even if 'a/b/c' had been deleted but you still had it open. > > Hi Dave, > > I am assuming that with file handles, renaming of file probably is not > a problem. open_by_handle_at() will still be able to find it. Right; although again with soemthing like CEPH you might not even need that if you had a virtiofs daemon that spoke direct to CEPH, you might be dealing with a CEPH filehandle. > I think real problem (as you pointed out later) is unliked file which is > still in use by the virtiofsd. Dave > Thanks > Vivek -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK ^ permalink raw reply [flat|nested] 14+ messages in thread
* Live migration support for virtio-fs @ 2021-04-07 18:05 ` Ge, Xiongzi 0 siblings, 0 replies; 14+ messages in thread From: Ge, Xiongzi @ 2021-04-07 18:05 UTC (permalink / raw) To: Dr. David Alan Gilbert, Vivek Goyal Cc: virtio-fs@redhat.com, qemu-devel@nongnu.org Thanks Dr Gilbert, Vivek, Stefan, Greg! I put together the discussion into this thread and CC qemu-devel@nongnu.org. Problem: Current Virtio-FS does not support live migration. Even when the virtiofs directory is not mounted on the guest, VM cannot do live migration. Any suggestions/interest If we want to make this work so we can still have the live migration feature for other purpose on that VM? If we already have a shared file system like Ceph, does this make it different? Thanks, Xiongzi --------- Stefan's reply: The virtio-fs device holds a bunch of state, like the ino_map, dirp_map, and the fd_map in the virtiofsd C implementation. That is the FUSE session state that needs to be migrated in order to resume seamlessly (without unmounting and mounting again). If the backend is a distributed file system then it my have APIs that make migration easier. If it's possible to re-attach to open files from another host then that is perfect. But on the flipside if there are no APIs for doing that then it might be impossible to reliably live migration because some state cannot be transferred between hosts. Max Reitz is laying the foundation for live migration by working on open_by_handle_at(2) support. This will probably be needed in order to transfer open files from one host to another. It should be possible to enable live migration when the filesystem is not mounted. That might be a good first step to enabling live migration. ------------ From Greg Kurz: Live migration of virtio-fs is still at the early discussion stage AFAICT. It might take time before we have something working. But in the meantime, it seems abusive to block migration if we have a guarantee that the device isn't servicing requests. FWIW virtio-9p only blocks migration when the shared directory is mounted on the guest. On 3/23/21, 11:01 AM, "Dr. David Alan Gilbert" <dgilbert@redhat.com> wrote: * Vivek Goyal (vgoyal@redhat.com) wrote: > On Tue, Mar 23, 2021 at 10:29:09AM +0000, Dr. David Alan Gilbert wrote: > > * Ge, Xiongzi (Xiongzi.Ge@netapp.com) wrote: > > > Hello Vivek, Dr. Gilbert, and the virtio-fs team, > > > > Hi Xiongzi, > > > > > It seems that virtio-fs does not support live migration. Once a vm > > > is configured with virtio-fs, live migration cannot be performed even > > > it is not mounted on the guest. > > > > Right. > > > > > Is there any progress for this? > > > > Max Reitz is looking at some parts of it; in particular storing file > > handles that can later be reopened. > > But there are a bunch of other parts we've not looked at yet either > > (like the dirty page marking around all syscalls). > > > > > If the > > > shared directory is from a distributed file system like Ceph, would it > > > be easier than the general case to be implemented? > > > > Maybe; there are some tricky semantics problems; for example, lets > > imagine that you open the file 'a/b/c' on the source, and sometime > > after you open it, 'b' gets renamed to 'd'; when you resume on the > > destination you need to make sure you know how to get to that file. > > Depending on the filesystem semantics you might need to make that work > > even if 'a/b/c' had been deleted but you still had it open. > > Hi Dave, > > I am assuming that with file handles, renaming of file probably is not > a problem. open_by_handle_at() will still be able to find it. Right; although again with soemthing like CEPH you might not even need that if you had a virtiofs daemon that spoke direct to CEPH, you might be dealing with a CEPH filehandle. > I think real problem (as you pointed out later) is unliked file which is > still in use by the virtiofsd. Dave > Thanks > Vivek -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [Virtio-fs] Live migration support for virtio-fs 2021-04-07 18:05 ` Ge, Xiongzi @ 2021-04-08 16:20 ` Stefan Hajnoczi -1 siblings, 0 replies; 14+ messages in thread From: Stefan Hajnoczi @ 2021-04-08 16:20 UTC (permalink / raw) To: Ge, Xiongzi Cc: virtio-fs@redhat.com, Marc-André Lureau, Vivek Goyal, qemu-devel@nongnu.org [-- Attachment #1: Type: text/plain, Size: 2187 bytes --] On Wed, Apr 07, 2021 at 06:05:29PM +0000, Ge, Xiongzi wrote: > Thanks Dr Gilbert, Vivek, Stefan, Greg! > I put together the discussion into this thread and CC qemu-devel@nongnu.org. > > Problem: > Current Virtio-FS does not support live migration. Even when the virtiofs directory is not mounted on the guest, VM cannot do live migration. Any suggestions/interest If we want to make this work so we can still have the live migration feature for other purpose on that VM? > > If we already have a shared file system like Ceph, does this make it different? (Marc-André: I CCed you regarding D-Bus VMState, see below.) During the last virtio-fs call on Wednesday we discussed how to allow live migration when the file system is not mounted. This means live migration of a mounted file system still wouldn't be supported, but at least it would be possible to migrate when the file system is not mounted. The easiest way to do that is to hot unplug the vhost-user-fs-pci device so that QEMU can live migrate the VM and then hot plug the vhost-user-fs-pci device again after migration has completed. This requires either manually performing the hot plug operations or some automation to do this around live migration. A smoother approach would be for QEMU to allow live migration when the file system is not mounted. Unfortunately it turns out this is actually rather involved because virtiofsd runs as a separate process from QEMU and the live migration interface for vhost-user devices is currently pretty basic (https://qemu.readthedocs.io/en/latest/interop/dbus-vmstate.html). There is currently no way for virtiofsd to communicate with QEMU to allow/deny live migration depending on whether virtiofsd currently has an active FUSE session (mount). It might be possible to extend the D-Bus VMState interface and implement support for "migration blockers" as they are known in QEMU, but I wonder whether it's not better to just work on full virtio-fs live migration instead. Either task is complex and I wouldn't recommend it as a first contribution to virtiofsd or QEMU. Can you make use of the hotplug approach to live migration? Stefan [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [Virtio-fs] Live migration support for virtio-fs @ 2021-04-08 16:20 ` Stefan Hajnoczi 0 siblings, 0 replies; 14+ messages in thread From: Stefan Hajnoczi @ 2021-04-08 16:20 UTC (permalink / raw) To: Ge, Xiongzi Cc: virtio-fs@redhat.com, Marc-André Lureau, Dr. David Alan Gilbert, Vivek Goyal, qemu-devel@nongnu.org [-- Attachment #1: Type: text/plain, Size: 2187 bytes --] On Wed, Apr 07, 2021 at 06:05:29PM +0000, Ge, Xiongzi wrote: > Thanks Dr Gilbert, Vivek, Stefan, Greg! > I put together the discussion into this thread and CC qemu-devel@nongnu.org. > > Problem: > Current Virtio-FS does not support live migration. Even when the virtiofs directory is not mounted on the guest, VM cannot do live migration. Any suggestions/interest If we want to make this work so we can still have the live migration feature for other purpose on that VM? > > If we already have a shared file system like Ceph, does this make it different? (Marc-André: I CCed you regarding D-Bus VMState, see below.) During the last virtio-fs call on Wednesday we discussed how to allow live migration when the file system is not mounted. This means live migration of a mounted file system still wouldn't be supported, but at least it would be possible to migrate when the file system is not mounted. The easiest way to do that is to hot unplug the vhost-user-fs-pci device so that QEMU can live migrate the VM and then hot plug the vhost-user-fs-pci device again after migration has completed. This requires either manually performing the hot plug operations or some automation to do this around live migration. A smoother approach would be for QEMU to allow live migration when the file system is not mounted. Unfortunately it turns out this is actually rather involved because virtiofsd runs as a separate process from QEMU and the live migration interface for vhost-user devices is currently pretty basic (https://qemu.readthedocs.io/en/latest/interop/dbus-vmstate.html). There is currently no way for virtiofsd to communicate with QEMU to allow/deny live migration depending on whether virtiofsd currently has an active FUSE session (mount). It might be possible to extend the D-Bus VMState interface and implement support for "migration blockers" as they are known in QEMU, but I wonder whether it's not better to just work on full virtio-fs live migration instead. Either task is complex and I wouldn't recommend it as a first contribution to virtiofsd or QEMU. Can you make use of the hotplug approach to live migration? Stefan [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [Virtio-fs] Live migration support for virtio-fs 2021-04-08 16:20 ` Stefan Hajnoczi @ 2021-05-03 12:50 ` Ge, Xiongzi -1 siblings, 0 replies; 14+ messages in thread From: Ge, Xiongzi @ 2021-05-03 12:50 UTC (permalink / raw) To: Stefan Hajnoczi Cc: virtio-fs@redhat.com, Marc-André Lureau, Vivek Goyal, qemu-devel@nongnu.org Thanks, Stefan! It seems that libvirt does not support device-attach for filesystem type device, like virtiofs. Are there any other ways to do this hot-unplugging? Xiongzi On 4/8/21, 12:21 PM, "Stefan Hajnoczi" <stefanha@redhat.com> wrote: On Wed, Apr 07, 2021 at 06:05:29PM +0000, Ge, Xiongzi wrote: > Thanks Dr Gilbert, Vivek, Stefan, Greg! > I put together the discussion into this thread and CC qemu-devel@nongnu.org. > > Problem: > Current Virtio-FS does not support live migration. Even when the virtiofs directory is not mounted on the guest, VM cannot do live migration. Any suggestions/interest If we want to make this work so we can still have the live migration feature for other purpose on that VM? > > If we already have a shared file system like Ceph, does this make it different? (Marc-André: I CCed you regarding D-Bus VMState, see below.) During the last virtio-fs call on Wednesday we discussed how to allow live migration when the file system is not mounted. This means live migration of a mounted file system still wouldn't be supported, but at least it would be possible to migrate when the file system is not mounted. The easiest way to do that is to hot unplug the vhost-user-fs-pci device so that QEMU can live migrate the VM and then hot plug the vhost-user-fs-pci device again after migration has completed. This requires either manually performing the hot plug operations or some automation to do this around live migration. A smoother approach would be for QEMU to allow live migration when the file system is not mounted. Unfortunately it turns out this is actually rather involved because virtiofsd runs as a separate process from QEMU and the live migration interface for vhost-user devices is currently pretty basic (https://qemu.readthedocs.io/en/latest/interop/dbus-vmstate.html). There is currently no way for virtiofsd to communicate with QEMU to allow/deny live migration depending on whether virtiofsd currently has an active FUSE session (mount). It might be possible to extend the D-Bus VMState interface and implement support for "migration blockers" as they are known in QEMU, but I wonder whether it's not better to just work on full virtio-fs live migration instead. Either task is complex and I wouldn't recommend it as a first contribution to virtiofsd or QEMU. Can you make use of the hotplug approach to live migration? Stefan ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [Virtio-fs] Live migration support for virtio-fs @ 2021-05-03 12:50 ` Ge, Xiongzi 0 siblings, 0 replies; 14+ messages in thread From: Ge, Xiongzi @ 2021-05-03 12:50 UTC (permalink / raw) To: Stefan Hajnoczi Cc: virtio-fs@redhat.com, Marc-André Lureau, Dr. David Alan Gilbert, Vivek Goyal, qemu-devel@nongnu.org Thanks, Stefan! It seems that libvirt does not support device-attach for filesystem type device, like virtiofs. Are there any other ways to do this hot-unplugging? Xiongzi On 4/8/21, 12:21 PM, "Stefan Hajnoczi" <stefanha@redhat.com> wrote: On Wed, Apr 07, 2021 at 06:05:29PM +0000, Ge, Xiongzi wrote: > Thanks Dr Gilbert, Vivek, Stefan, Greg! > I put together the discussion into this thread and CC qemu-devel@nongnu.org. > > Problem: > Current Virtio-FS does not support live migration. Even when the virtiofs directory is not mounted on the guest, VM cannot do live migration. Any suggestions/interest If we want to make this work so we can still have the live migration feature for other purpose on that VM? > > If we already have a shared file system like Ceph, does this make it different? (Marc-André: I CCed you regarding D-Bus VMState, see below.) During the last virtio-fs call on Wednesday we discussed how to allow live migration when the file system is not mounted. This means live migration of a mounted file system still wouldn't be supported, but at least it would be possible to migrate when the file system is not mounted. The easiest way to do that is to hot unplug the vhost-user-fs-pci device so that QEMU can live migrate the VM and then hot plug the vhost-user-fs-pci device again after migration has completed. This requires either manually performing the hot plug operations or some automation to do this around live migration. A smoother approach would be for QEMU to allow live migration when the file system is not mounted. Unfortunately it turns out this is actually rather involved because virtiofsd runs as a separate process from QEMU and the live migration interface for vhost-user devices is currently pretty basic (https://qemu.readthedocs.io/en/latest/interop/dbus-vmstate.html). There is currently no way for virtiofsd to communicate with QEMU to allow/deny live migration depending on whether virtiofsd currently has an active FUSE session (mount). It might be possible to extend the D-Bus VMState interface and implement support for "migration blockers" as they are known in QEMU, but I wonder whether it's not better to just work on full virtio-fs live migration instead. Either task is complex and I wouldn't recommend it as a first contribution to virtiofsd or QEMU. Can you make use of the hotplug approach to live migration? Stefan ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [Virtio-fs] Live migration support for virtio-fs 2021-05-03 12:50 ` Ge, Xiongzi @ 2021-05-03 13:07 ` Vivek Goyal -1 siblings, 0 replies; 14+ messages in thread From: Vivek Goyal @ 2021-05-03 13:07 UTC (permalink / raw) To: Ge, Xiongzi Cc: virtio-fs@redhat.com, Marc-André Lureau, qemu-devel@nongnu.org Hi Xiongzi, Try qemu monitor to hot device add/remove that works. I think libvirt support is yet to be added. One issue I faced was that I had to add a pcie root to move virtiofs device off pcie.0 and then hot add remove functionality works. So I had to add pcie root port by adding following to qemu command line. -device pcie-root-port,id=root1,slot=0 And now you can hot add a device using device_add. (qemu) device_add vhost-user-fs-pci,bus=root1,id=myvirtiofs,chardev=char0,tag=myfs,queue-size=1024 - To hot remove device, issue from qemu monitor. (qemu) device_del <device_id> ex. device_del myvirtiofs Thanks Vivek On Mon, May 03, 2021 at 12:50:56PM +0000, Ge, Xiongzi wrote: > Thanks, Stefan! > It seems that libvirt does not support device-attach for filesystem type device, like virtiofs. Are there any other ways to do this hot-unplugging? > > Xiongzi > > On 4/8/21, 12:21 PM, "Stefan Hajnoczi" <stefanha@redhat.com> wrote: > > On Wed, Apr 07, 2021 at 06:05:29PM +0000, Ge, Xiongzi wrote: > > Thanks Dr Gilbert, Vivek, Stefan, Greg! > > I put together the discussion into this thread and CC qemu-devel@nongnu.org. > > > > Problem: > > Current Virtio-FS does not support live migration. Even when the virtiofs directory is not mounted on the guest, VM cannot do live migration. Any suggestions/interest If we want to make this work so we can still have the live migration feature for other purpose on that VM? > > > > If we already have a shared file system like Ceph, does this make it different? > > (Marc-André: I CCed you regarding D-Bus VMState, see below.) > > During the last virtio-fs call on Wednesday we discussed how to allow > live migration when the file system is not mounted. This means live > migration of a mounted file system still wouldn't be supported, but at > least it would be possible to migrate when the file system is not > mounted. > > The easiest way to do that is to hot unplug the vhost-user-fs-pci device > so that QEMU can live migrate the VM and then hot plug the > vhost-user-fs-pci device again after migration has completed. This > requires either manually performing the hot plug operations or some > automation to do this around live migration. > > A smoother approach would be for QEMU to allow live migration when the > file system is not mounted. Unfortunately it turns out this is actually > rather involved because virtiofsd runs as a separate process from QEMU > and the live migration interface for vhost-user devices is currently > pretty basic > (https://qemu.readthedocs.io/en/latest/interop/dbus-vmstate.html). There > is currently no way for virtiofsd to communicate with QEMU to allow/deny > live migration depending on whether virtiofsd currently has an active > FUSE session (mount). It might be possible to extend the D-Bus VMState > interface and implement support for "migration blockers" as they are > known in QEMU, but I wonder whether it's not better to just work on > full virtio-fs live migration instead. Either task is complex and I > wouldn't recommend it as a first contribution to virtiofsd or QEMU. > > Can you make use of the hotplug approach to live migration? > > Stefan > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [Virtio-fs] Live migration support for virtio-fs @ 2021-05-03 13:07 ` Vivek Goyal 0 siblings, 0 replies; 14+ messages in thread From: Vivek Goyal @ 2021-05-03 13:07 UTC (permalink / raw) To: Ge, Xiongzi Cc: virtio-fs@redhat.com, Marc-André Lureau, Dr. David Alan Gilbert, Stefan Hajnoczi, qemu-devel@nongnu.org Hi Xiongzi, Try qemu monitor to hot device add/remove that works. I think libvirt support is yet to be added. One issue I faced was that I had to add a pcie root to move virtiofs device off pcie.0 and then hot add remove functionality works. So I had to add pcie root port by adding following to qemu command line. -device pcie-root-port,id=root1,slot=0 And now you can hot add a device using device_add. (qemu) device_add vhost-user-fs-pci,bus=root1,id=myvirtiofs,chardev=char0,tag=myfs,queue-size=1024 - To hot remove device, issue from qemu monitor. (qemu) device_del <device_id> ex. device_del myvirtiofs Thanks Vivek On Mon, May 03, 2021 at 12:50:56PM +0000, Ge, Xiongzi wrote: > Thanks, Stefan! > It seems that libvirt does not support device-attach for filesystem type device, like virtiofs. Are there any other ways to do this hot-unplugging? > > Xiongzi > > On 4/8/21, 12:21 PM, "Stefan Hajnoczi" <stefanha@redhat.com> wrote: > > On Wed, Apr 07, 2021 at 06:05:29PM +0000, Ge, Xiongzi wrote: > > Thanks Dr Gilbert, Vivek, Stefan, Greg! > > I put together the discussion into this thread and CC qemu-devel@nongnu.org. > > > > Problem: > > Current Virtio-FS does not support live migration. Even when the virtiofs directory is not mounted on the guest, VM cannot do live migration. Any suggestions/interest If we want to make this work so we can still have the live migration feature for other purpose on that VM? > > > > If we already have a shared file system like Ceph, does this make it different? > > (Marc-André: I CCed you regarding D-Bus VMState, see below.) > > During the last virtio-fs call on Wednesday we discussed how to allow > live migration when the file system is not mounted. This means live > migration of a mounted file system still wouldn't be supported, but at > least it would be possible to migrate when the file system is not > mounted. > > The easiest way to do that is to hot unplug the vhost-user-fs-pci device > so that QEMU can live migrate the VM and then hot plug the > vhost-user-fs-pci device again after migration has completed. This > requires either manually performing the hot plug operations or some > automation to do this around live migration. > > A smoother approach would be for QEMU to allow live migration when the > file system is not mounted. Unfortunately it turns out this is actually > rather involved because virtiofsd runs as a separate process from QEMU > and the live migration interface for vhost-user devices is currently > pretty basic > (https://qemu.readthedocs.io/en/latest/interop/dbus-vmstate.html). There > is currently no way for virtiofsd to communicate with QEMU to allow/deny > live migration depending on whether virtiofsd currently has an active > FUSE session (mount). It might be possible to extend the D-Bus VMState > interface and implement support for "migration blockers" as they are > known in QEMU, but I wonder whether it's not better to just work on > full virtio-fs live migration instead. Either task is complex and I > wouldn't recommend it as a first contribution to virtiofsd or QEMU. > > Can you make use of the hotplug approach to live migration? > > Stefan > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [Virtio-fs] Live migration support for virtio-fs 2021-03-23 1:35 [Virtio-fs] Live migration support for virtio-fs Ge, Xiongzi 2021-03-23 9:02 ` Greg Kurz 2021-03-23 10:29 ` Dr. David Alan Gilbert @ 2021-03-23 13:33 ` Stefan Hajnoczi 2 siblings, 0 replies; 14+ messages in thread From: Stefan Hajnoczi @ 2021-03-23 13:33 UTC (permalink / raw) To: Ge, Xiongzi; +Cc: virtio-fs@redhat.com, Vivek Goyal, mreitz [-- Attachment #1: Type: text/plain, Size: 1351 bytes --] On Tue, Mar 23, 2021 at 01:35:58AM +0000, Ge, Xiongzi wrote: > It seems that virtio-fs does not support live migration. Once a vm is configured with virtio-fs, live migration cannot be performed even it is not mounted on the guest. Is there any progress for this? If the shared directory is from a distributed file system like Ceph, would it be easier than the general case to be implemented? The virtio-fs device holds a bunch of state, like the ino_map, dirp_map, and the fd_map in the virtiofsd C implementation. That is the FUSE session state that needs to be migrated in order to resume seamlessly (without unmounting and mounting again). If the backend is a distributed file system then it my have APIs that make migration easier. If it's possible to re-attach to open files from another host then that is perfect. But on the flipside if there are no APIs for doing that then it might be impossible to reliably live migration because some state cannot be transferred between hosts. Max Reitz is laying the foundation for live migration by working on open_by_handle_at(2) support. This will probably be needed in order to transfer open files from one host to another. It should be possible to enable live migration when the filesystem is not mounted. That might be a good first step to enabling live migration. Stefan [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2021-05-03 13:15 UTC | newest] Thread overview: 14+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2021-03-23 1:35 [Virtio-fs] Live migration support for virtio-fs Ge, Xiongzi 2021-03-23 9:02 ` Greg Kurz 2021-03-23 10:29 ` Dr. David Alan Gilbert 2021-03-23 14:53 ` Vivek Goyal 2021-03-23 15:00 ` Dr. David Alan Gilbert 2021-04-07 18:05 ` Ge, Xiongzi 2021-04-07 18:05 ` Ge, Xiongzi 2021-04-08 16:20 ` [Virtio-fs] " Stefan Hajnoczi 2021-04-08 16:20 ` Stefan Hajnoczi 2021-05-03 12:50 ` Ge, Xiongzi 2021-05-03 12:50 ` Ge, Xiongzi 2021-05-03 13:07 ` Vivek Goyal 2021-05-03 13:07 ` Vivek Goyal 2021-03-23 13:33 ` Stefan Hajnoczi
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.