On Mon, Apr 20, 2026 at 04:49:04PM +0200, Stefano Garzarella wrote: > Thanks for starting the discussion here, let me add also Hanna, German, and > Stefan in CC that can help us. > > On Fri, Apr 10, 2026 at 07:12:05PM -0700, Jorge E. Moreira wrote: > > Migration of back-end state happens while the device is suspended (i.e > > all vrings are stopped). To resume normal operation on the destination, > > the vrings need to be started again with a kick (either a write on the > > FD or the VHOST_USER_VRING_KICK in-band message if negotiated). While > > It's true that in the spec we have: > "Each ring is initialized in a stopped and disabled state. The back-end > must start a ring upon receiving a kick (that is, detecting that file > descriptor is readable) on the descriptor specified by > VHOST_USER_SET_VRING_KICK or receiving the in-band message > VHOST_USER_VRING_KICK if negotiated, and stop a ring upon receiving > VHOST_USER_GET_VRING_BASE." > > But IMO this applies when a driver is not yet loaded. > When we are migrating, the driver could be already loaded. So, in the new > device running in the destination, IMO we should consider the ring already > started or add some messages to tell to the device: "hey, the device was > already started, this is a migration and it's completed". > > Sending a kick from the frontend, seems more an hack here. > > That said, for example, in subprojects/libvhost-user/libvhost-user.c IIUC > the virtqueue is started when the SET_VRING_KICK is handled by > vu_set_vring_kick_exec(), but not sure how compliant it is. > > > these notifications are typically sent by the driver, it has no reason > > to send them in the destination if it already sent them in the source as > > the driver is unaware that a migration took place. Therefore it should > > be the responsibility of the vhost-user front-end to ensure these vrings > > are started. This is particularly necessary for queues where data only > > flows from device to driver, such as those used by the vsock and input > > devices. > > Exactly, so IMO we should not use the kick, but maybe add something new or > clarify what to do after the migration. > > For example in the "Migrating back-end state" we have: > "Migrating device state involves transferring the state from one > back-end, called the source, to another back-end, called the destination. > After migration, the destination transparently resumes operation without > requiring the driver to re-initialize the device at the VIRTIO level." > > So, IMO we can use the VHOST_USER_SET_DEVICE_STATE_FD channel exactly to > inform the new device about the state: "there isn't any state to transfer, > but I notify you that the device was already initialized, so the vrings can > be started". > > > > > This behavior is already used by some qemu vhost-user front-ends (e.g > > vhost-user-blk) and by front-ends implemented on other VMMs(e.g CrosVm). > > I looked at vhost-user-blk frontend, but I don't see it. I mean I see the > code around the comment "/* Kick right away to begin processing requests > already in vring */" but that one IIUC was introduced more to fix devices > violating specs, so not sure it's a good example to follow: > > commit 110b9463d5c820120c8311db79f55a64c9d81ebe > Author: Yongji Xie > Date: Wed Jun 6 21:24:48 2018 +0800 > > vhost-user-blk: start vhost when guest kicks > Some old guests (before commit 7a11370e5: "virtio_blk: enable VQs early") > kick virtqueue before setting VIRTIO_CONFIG_S_DRIVER_OK. This violates > the virtio spec. But virtio 1.0 transitional devices support this behaviour. > So we should start vhost when guest kicks in this case. > Signed-off-by: Yongji Xie > Signed-off-by: Chai Wen > Signed-off-by: Ni Xun > Reviewed-by: Stefan Hajnoczi > Reviewed-by: Michael S. Tsirkin > Signed-off-by: Michael S. Tsirkin > > > > Adding it to the vhost-user documentation makes it explicit that this > > strategy is permitted and suggest it to vhost-user front-end authors. > > Explicitly documenting it is necessary because vring kicks appear > > designed to originate in the driver, so having some originate in the > > front-end can be counterintuitive and cause developers to waste time > > looking for other alternatives or face pushback during code review. > > As I pointed out in our discussion in > https://github.com/rust-vmm/vhost-device/pull/936 > IMO we should use some in-band messages and not relaying on kicks that > should be used only by the driver to notify the device about new available > buffers. > > That said, I agree that we need to clarify in the specifications exactly > what the backend and frontend should do after a migration to start vrings if > there is no need to exchange a state. > > > Any other opinion? IMO no protocol changes are needed but the vhost-user spec should be tweaked. Hanna worked on device state migration and can confirm/deny what I'm about to describe. QEMU's libvhost-user implementation already starts the ring when VHOST_USER_SET_VRING_KICK is received. This makes more sense than waiting for the kick fd since a back-end that uses polling (peeking at the vring in memory) shouldn't need to monitor the kick fd. See below though about races between the kick fd and vhost-user protocol messages. All of this boils down to the ring state machine. libvhost-user's behavior is: 1. Virtqueues are started by VHOST_USER_SET_VRING_KICK. 2. Virtqueues are stopped by VHOST_USER_GET_VRING_BASE. 3. Virtqueues are enabled/disabled by VHOST_USER_SET_VRING_ENABLE. The same sequence of vhost-user protocol messages that is used to start/stop a device locally (e.g. pause and resume a VM) is the same that can be used during migration. The ring state machine already exists and needs to be used when migrating device state. The following changes to the vhost-user spec would make this clearer: 1. Mention that virtqueues are started by VHOST_USER_SET_VRING_KICK. Note that monitoring the kick fd may be used to avoid races between the kick fd and vhost-user protocol messages, so in practice back-end implementors may still want to start the virtqueue when the kick fd becomes readable. 2. Add a clarification to "Migrating back-end state" that the device must be suspended (see _suspended_device_state) when VHOST_USER_SET_DEVICE_STATE_FD is sent and device state is transferred. This is already implicit in "Device state transfer parameters", but it's not obvious when reading the "Migrating back-end state" section. Stefan