Re: [PATCH] vhost-user.rst: Explicitly allow front-end to write to kick FDs

[Stefan Hajnoczi <stefanha@redhat.com>]

[-- Attachment #1: Type: text/plain, Size: 6859 bytes --]

On Mon, Apr 20, 2026 at 04:49:04PM +0200, Stefano Garzarella wrote:
> Thanks for starting the discussion here, let me add also Hanna, German, and
> Stefan in CC that can help us.
> 
> On Fri, Apr 10, 2026 at 07:12:05PM -0700, Jorge E. Moreira wrote:
> > Migration of back-end state happens while the device is suspended (i.e
> > all vrings are stopped). To resume normal operation on the destination,
> > the vrings need to be started again with a kick (either a write on the
> > FD or the VHOST_USER_VRING_KICK in-band message if negotiated). While
> 
> It's true that in the spec we have:
>   "Each ring is initialized in a stopped and disabled state. The   back-end
> must start a ring upon receiving a kick (that is, detecting   that file
> descriptor is readable) on the descriptor specified by
> VHOST_USER_SET_VRING_KICK or receiving the in-band message
> VHOST_USER_VRING_KICK if negotiated, and stop a ring upon receiving
> VHOST_USER_GET_VRING_BASE."
> 
> But IMO this applies when a driver is not yet loaded.
> When we are migrating, the driver could be already loaded. So, in the new
> device running in the destination, IMO we should consider the ring already
> started or add some messages to tell to the device: "hey, the device was
> already started, this is a migration and it's completed".
> 
> Sending a kick from the frontend, seems more an hack here.
> 
> That said, for example, in subprojects/libvhost-user/libvhost-user.c IIUC
> the virtqueue is started when the SET_VRING_KICK is handled by
> vu_set_vring_kick_exec(), but not sure how compliant it is.
> 
> > these notifications are typically sent by the driver, it has no reason
> > to send them in the destination if it already sent them in the source as
> > the driver is unaware that a migration took place. Therefore it should
> > be the responsibility of the vhost-user front-end to ensure these vrings
> > are started. This is particularly necessary for queues where data only
> > flows from device to driver, such as those used by the vsock and input
> > devices.
> 
> Exactly, so IMO we should not use the kick, but maybe add something new or
> clarify what to do after the migration.
> 
> For example in the "Migrating back-end state" we have:
>   "Migrating device state involves transferring the state from one
> back-end, called the source, to another back-end, called the   destination.
> After migration, the destination transparently resumes   operation without
> requiring the driver to re-initialize the device at   the VIRTIO level."
> 
> So, IMO we can use the VHOST_USER_SET_DEVICE_STATE_FD channel exactly to
> inform the new device about the state: "there isn't any state to transfer,
> but I notify you that the device was already initialized, so the vrings can
> be started".
> 
> > 
> > This behavior is already used by some qemu vhost-user front-ends (e.g
> > vhost-user-blk) and by front-ends implemented on other VMMs(e.g CrosVm).
> 
> I looked at vhost-user-blk frontend, but I don't see it. I mean I see the
> code around the comment "/* Kick right away to begin processing requests
> already in vring */" but that one IIUC was introduced more to fix devices
> violating specs, so not sure it's a good example to follow:
> 
> commit 110b9463d5c820120c8311db79f55a64c9d81ebe
> Author: Yongji Xie <elohimes@gmail.com>
> Date:   Wed Jun 6 21:24:48 2018 +0800
> 
>     vhost-user-blk: start vhost when guest kicks
>     Some old guests (before commit 7a11370e5: "virtio_blk: enable VQs early")
>     kick virtqueue before setting VIRTIO_CONFIG_S_DRIVER_OK. This violates
>     the virtio spec. But virtio 1.0 transitional devices support this behaviour.
>     So we should start vhost when guest kicks in this case.
>     Signed-off-by: Yongji Xie <xieyongji@baidu.com>
>     Signed-off-by: Chai Wen <chaiwen@baidu.com>
>     Signed-off-by: Ni Xun <nixun@baidu.com>
>     Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
>     Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
>     Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> 
> 
> > Adding it to the vhost-user documentation makes it explicit that this
> > strategy is permitted and suggest it to vhost-user front-end authors.
> > Explicitly documenting it is necessary because vring kicks appear
> > designed to originate in the driver, so having some originate in the
> > front-end can be counterintuitive and cause developers to waste time
> > looking for other alternatives or face pushback during code review.
> 
> As I pointed out in our discussion in
> https://github.com/rust-vmm/vhost-device/pull/936
> IMO we should use some in-band messages and not relaying on kicks that
> should be used only by the driver to notify the device about new available
> buffers.
> 
> That said, I agree that we need to clarify in the specifications exactly
> what the backend and frontend should do after a migration to start vrings if
> there is no need to exchange a state.
> 
> 
> Any other opinion?

IMO no protocol changes are needed but the vhost-user spec should be
tweaked. Hanna worked on device state migration and can confirm/deny
what I'm about to describe.

QEMU's libvhost-user implementation already starts the ring when
VHOST_USER_SET_VRING_KICK is received. This makes more sense than
waiting for the kick fd since a back-end that uses polling (peeking at
the vring in memory) shouldn't need to monitor the kick fd. See below
though about races between the kick fd and vhost-user protocol messages.

All of this boils down to the ring state machine. libvhost-user's
behavior is:
1. Virtqueues are started by VHOST_USER_SET_VRING_KICK.
2. Virtqueues are stopped by VHOST_USER_GET_VRING_BASE.
3. Virtqueues are enabled/disabled by VHOST_USER_SET_VRING_ENABLE.

The same sequence of vhost-user protocol messages that is used to
start/stop a device locally (e.g. pause and resume a VM) is the same
that can be used during migration. The ring state machine already exists
and needs to be used when migrating device state.

The following changes to the vhost-user spec would make this clearer:

1. Mention that virtqueues are started by VHOST_USER_SET_VRING_KICK.
   Note that monitoring the kick fd may be used to avoid races between
   the kick fd and vhost-user protocol messages, so in practice back-end
   implementors may still want to start the virtqueue when the kick fd
   becomes readable.

2. Add a clarification to "Migrating back-end state" that the device
   must be suspended (see _suspended_device_state) when
   VHOST_USER_SET_DEVICE_STATE_FD is sent and device state is
   transferred. This is already implicit in "Device state transfer
   parameters", but it's not obvious when reading the "Migrating
   back-end state" section.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]