[PATCH] vhost-user.rst: Explicitly allow front-end to write to kick FDs

[PATCH] vhost-user.rst: Explicitly allow front-end to write to kick FDs

[Jorge E. Moreira]

Migration of back-end state happens while the device is suspended (i.e
all vrings are stopped). To resume normal operation on the destination,
the vrings need to be started again with a kick (either a write on the
FD or the VHOST_USER_VRING_KICK in-band message if negotiated). While
these notifications are typically sent by the driver, it has no reason
to send them in the destination if it already sent them in the source as
the driver is unaware that a migration took place. Therefore it should
be the responsibility of the vhost-user front-end to ensure these vrings
are started. This is particularly necessary for queues where data only
flows from device to driver, such as those used by the vsock and input
devices.

This behavior is already used by some qemu vhost-user front-ends (e.g
vhost-user-blk) and by front-ends implemented on other VMMs(e.g CrosVm).
Adding it to the vhost-user documentation makes it explicit that this
strategy is permitted and suggest it to vhost-user front-end authors.
Explicitly documenting it is necessary because vring kicks appear
designed to originate in the driver, so having some originate in the
front-end can be counterintuitive and cause developers to waste time
looking for other alternatives or face pushback during code review.

Signed-off-by: Jorge E. Moreira <jemoreira@google.com>
---
 docs/interop/vhost-user.rst | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/docs/interop/vhost-user.rst b/docs/interop/vhost-user.rst
index 137c9f3669..ad5aba3430 100644
--- a/docs/interop/vhost-user.rst
+++ b/docs/interop/vhost-user.rst
@@ -656,7 +656,10 @@ destination, following the usual protocol for establishing a connection
 to a vhost-user back-end: This includes, for example, setting up memory
 mappings and kick and call FDs as necessary, negotiating protocol
 features, or setting the initial vring base indices (to the same value
-as on the source side, so that operation can resume).
+as on the source side, so that operation can resume). The vhost-user front-end
+may also write to the kick FDs of vrings containing unused buffers or send
+``VHOST_USER_VRING_KICK`` if negotiated to start those vrings in the destination
+since the driver likely already kicked them in the source and won't do it again.
 
 Both on the source and on the destination side, after the respective
 front-end has seen all data transferred (when the transfer FD has been
-- 
2.53.0.1213.gd9a14994de-goog

Re: [PATCH] vhost-user.rst: Explicitly allow front-end to write to kick FDs

[Stefano Garzarella]

Thanks for starting the discussion here, let me add also Hanna, German, 
and Stefan in CC that can help us.

On Fri, Apr 10, 2026 at 07:12:05PM -0700, Jorge E. Moreira wrote:
>Migration of back-end state happens while the device is suspended (i.e
>all vrings are stopped). To resume normal operation on the destination,
>the vrings need to be started again with a kick (either a write on the
>FD or the VHOST_USER_VRING_KICK in-band message if negotiated). While

It's true that in the spec we have:
   "Each ring is initialized in a stopped and disabled state. The 
   back-end must start a ring upon receiving a kick (that is, detecting 
   that file descriptor is readable) on the descriptor specified by 
   VHOST_USER_SET_VRING_KICK or receiving the in-band message 
   VHOST_USER_VRING_KICK if negotiated, and stop a ring upon receiving 
   VHOST_USER_GET_VRING_BASE."

But IMO this applies when a driver is not yet loaded.
When we are migrating, the driver could be already loaded. So, in the 
new device running in the destination, IMO we should consider the ring 
already started or add some messages to tell to the device: "hey, the 
device was already started, this is a migration and it's completed".

Sending a kick from the frontend, seems more an hack here.

That said, for example, in subprojects/libvhost-user/libvhost-user.c 
IIUC the virtqueue is started when the SET_VRING_KICK is handled by 
vu_set_vring_kick_exec(), but not sure how compliant it is.

>these notifications are typically sent by the driver, it has no reason
>to send them in the destination if it already sent them in the source as
>the driver is unaware that a migration took place. Therefore it should
>be the responsibility of the vhost-user front-end to ensure these vrings
>are started. This is particularly necessary for queues where data only
>flows from device to driver, such as those used by the vsock and input
>devices.

Exactly, so IMO we should not use the kick, but maybe add something new 
or clarify what to do after the migration.

For example in the "Migrating back-end state" we have:
   "Migrating device state involves transferring the state from one 
   back-end, called the source, to another back-end, called the 
   destination. After migration, the destination transparently resumes 
   operation without requiring the driver to re-initialize the device at 
   the VIRTIO level."

So, IMO we can use the VHOST_USER_SET_DEVICE_STATE_FD channel exactly to 
inform the new device about the state: "there isn't any state to 
transfer, but I notify you that the device was already initialized, so 
the vrings can be started".

>
>This behavior is already used by some qemu vhost-user front-ends (e.g
>vhost-user-blk) and by front-ends implemented on other VMMs(e.g CrosVm).

I looked at vhost-user-blk frontend, but I don't see it. I mean I see 
the code around the comment "/* Kick right away to begin processing 
requests already in vring */" but that one IIUC was introduced more to 
fix devices violating specs, so not sure it's a good example to follow:

commit 110b9463d5c820120c8311db79f55a64c9d81ebe
Author: Yongji Xie <elohimes@gmail.com>
Date:   Wed Jun 6 21:24:48 2018 +0800

     vhost-user-blk: start vhost when guest kicks
     
     Some old guests (before commit 7a11370e5: "virtio_blk: enable VQs early")
     kick virtqueue before setting VIRTIO_CONFIG_S_DRIVER_OK. This violates
     the virtio spec. But virtio 1.0 transitional devices support this behaviour.
     So we should start vhost when guest kicks in this case.
     
     Signed-off-by: Yongji Xie <xieyongji@baidu.com>
     Signed-off-by: Chai Wen <chaiwen@baidu.com>
     Signed-off-by: Ni Xun <nixun@baidu.com>
     Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
     Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
     Signed-off-by: Michael S. Tsirkin <mst@redhat.com>


>Adding it to the vhost-user documentation makes it explicit that this
>strategy is permitted and suggest it to vhost-user front-end authors.
>Explicitly documenting it is necessary because vring kicks appear
>designed to originate in the driver, so having some originate in the
>front-end can be counterintuitive and cause developers to waste time
>looking for other alternatives or face pushback during code review.

As I pointed out in our discussion in 
https://github.com/rust-vmm/vhost-device/pull/936
IMO we should use some in-band messages and not relaying on kicks that 
should be used only by the driver to notify the device about new 
available buffers.

That said, I agree that we need to clarify in the specifications exactly 
what the backend and frontend should do after a migration to start 
vrings if there is no need to exchange a state.


Any other opinion?

Thanks,
Stefano

>
>Signed-off-by: Jorge E. Moreira <jemoreira@google.com>
>---
> docs/interop/vhost-user.rst | 5 ++++-
> 1 file changed, 4 insertions(+), 1 deletion(-)
>
>diff --git a/docs/interop/vhost-user.rst b/docs/interop/vhost-user.rst
>index 137c9f3669..ad5aba3430 100644
>--- a/docs/interop/vhost-user.rst
>+++ b/docs/interop/vhost-user.rst
>@@ -656,7 +656,10 @@ destination, following the usual protocol for establishing a connection
> to a vhost-user back-end: This includes, for example, setting up memory
> mappings and kick and call FDs as necessary, negotiating protocol
> features, or setting the initial vring base indices (to the same value
>-as on the source side, so that operation can resume).
>+as on the source side, so that operation can resume). The vhost-user front-end
>+may also write to the kick FDs of vrings containing unused buffers or send
>+``VHOST_USER_VRING_KICK`` if negotiated to start those vrings in the destination
>+since the driver likely already kicked them in the source and won't do it again.
>
> Both on the source and on the destination side, after the respective
> front-end has seen all data transferred (when the transfer FD has been
>-- 
>2.53.0.1213.gd9a14994de-goog
>

Re: [PATCH] vhost-user.rst: Explicitly allow front-end to write to kick FDs

[Stefano Garzarella]

On Mon, 20 Apr 2026 at 16:49, Stefano Garzarella <sgarzare@redhat.com> wrote:
>
> Thanks for starting the discussion here, let me add also Hanna, German,
> and Stefan in CC that can help us.
>
> On Fri, Apr 10, 2026 at 07:12:05PM -0700, Jorge E. Moreira wrote:
> >Migration of back-end state happens while the device is suspended (i.e
> >all vrings are stopped). To resume normal operation on the destination,
> >the vrings need to be started again with a kick (either a write on the
> >FD or the VHOST_USER_VRING_KICK in-band message if negotiated). While
>
> It's true that in the spec we have:
>    "Each ring is initialized in a stopped and disabled state. The
>    back-end must start a ring upon receiving a kick (that is, detecting
>    that file descriptor is readable) on the descriptor specified by
>    VHOST_USER_SET_VRING_KICK or receiving the in-band message
>    VHOST_USER_VRING_KICK if negotiated, and stop a ring upon receiving
>    VHOST_USER_GET_VRING_BASE."
>
> But IMO this applies when a driver is not yet loaded.
> When we are migrating, the driver could be already loaded. So, in the
> new device running in the destination, IMO we should consider the ring
> already started or add some messages to tell to the device: "hey, the
> device was already started, this is a migration and it's completed".
>
> Sending a kick from the frontend, seems more an hack here.
>
> That said, for example, in subprojects/libvhost-user/libvhost-user.c
> IIUC the virtqueue is started when the SET_VRING_KICK is handled by
> vu_set_vring_kick_exec(), but not sure how compliant it is.
>
> >these notifications are typically sent by the driver, it has no reason
> >to send them in the destination if it already sent them in the source as
> >the driver is unaware that a migration took place. Therefore it should
> >be the responsibility of the vhost-user front-end to ensure these vrings
> >are started. This is particularly necessary for queues where data only
> >flows from device to driver, such as those used by the vsock and input
> >devices.
>
> Exactly, so IMO we should not use the kick, but maybe add something new
> or clarify what to do after the migration.
>
> For example in the "Migrating back-end state" we have:
>    "Migrating device state involves transferring the state from one
>    back-end, called the source, to another back-end, called the
>    destination. After migration, the destination transparently resumes
>    operation without requiring the driver to re-initialize the device at
>    the VIRTIO level."
>
> So, IMO we can use the VHOST_USER_SET_DEVICE_STATE_FD channel exactly to
> inform the new device about the state: "there isn't any state to
> transfer, but I notify you that the device was already initialized, so
> the vrings can be started".
>
> >
> >This behavior is already used by some qemu vhost-user front-ends (e.g
> >vhost-user-blk) and by front-ends implemented on other VMMs(e.g CrosVm).
>
> I looked at vhost-user-blk frontend, but I don't see it. I mean I see
> the code around the comment "/* Kick right away to begin processing
> requests already in vring */" but that one IIUC was introduced more to
> fix devices violating specs, so not sure it's a good example to follow:
>
> commit 110b9463d5c820120c8311db79f55a64c9d81ebe
> Author: Yongji Xie <elohimes@gmail.com>
> Date:   Wed Jun 6 21:24:48 2018 +0800
>
>      vhost-user-blk: start vhost when guest kicks
>
>      Some old guests (before commit 7a11370e5: "virtio_blk: enable VQs early")
>      kick virtqueue before setting VIRTIO_CONFIG_S_DRIVER_OK. This violates
>      the virtio spec. But virtio 1.0 transitional devices support this behaviour.
>      So we should start vhost when guest kicks in this case.
>
>      Signed-off-by: Yongji Xie <xieyongji@baidu.com>
>      Signed-off-by: Chai Wen <chaiwen@baidu.com>
>      Signed-off-by: Ni Xun <nixun@baidu.com>
>      Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
>      Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
>      Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
>
>
> >Adding it to the vhost-user documentation makes it explicit that this
> >strategy is permitted and suggest it to vhost-user front-end authors.
> >Explicitly documenting it is necessary because vring kicks appear
> >designed to originate in the driver, so having some originate in the
> >front-end can be counterintuitive and cause developers to waste time
> >looking for other alternatives or face pushback during code review.
>
> As I pointed out in our discussion in
> https://github.com/rust-vmm/vhost-device/pull/936
> IMO we should use some in-band messages and not relaying on kicks that
> should be used only by the driver to notify the device about new
> available buffers.
>
> That said, I agree that we need to clarify in the specifications exactly
> what the backend and frontend should do after a migration to start
> vrings if there is no need to exchange a state.
>
>
> Any other opinion?
>
> Thanks,
> Stefano
>
> >
> >Signed-off-by: Jorge E. Moreira <jemoreira@google.com>
> >---
> > docs/interop/vhost-user.rst | 5 ++++-
> > 1 file changed, 4 insertions(+), 1 deletion(-)
> >
> >diff --git a/docs/interop/vhost-user.rst b/docs/interop/vhost-user.rst
> >index 137c9f3669..ad5aba3430 100644
> >--- a/docs/interop/vhost-user.rst
> >+++ b/docs/interop/vhost-user.rst
> >@@ -656,7 +656,10 @@ destination, following the usual protocol for establishing a connection
> > to a vhost-user back-end: This includes, for example, setting up memory
> > mappings and kick and call FDs as necessary, negotiating protocol
> > features, or setting the initial vring base indices (to the same value
> >-as on the source side, so that operation can resume).
> >+as on the source side, so that operation can resume). The vhost-user front-end
> >+may also write to the kick FDs of vrings containing unused buffers or send
> >+``VHOST_USER_VRING_KICK`` if negotiated to start those vrings in the destination
> >+since the driver likely already kicked them in the source and won't do it again.

After discussing this with Hanna, we came to the conclusion that your
idea of injecting the kick is the least invasive option for now and
complies with the spec (even though I still don’t think it’s a nice
thing to do).

So it’s fine to continue in this direction, but I might add these
words more in the "Migration" section than here, since we’re talking
about an optional state migration here. For example after "No further
update must be done before rings are restarted."
Or in the "Ring states" section, where we can clarify how to restart a
ring after a migration. Or in both :-)

WDYT?

Then instead of "may" I'd use "should". And I would refer to the fact
that migration is transparent to the driver, so the front-end should
kick all initialized vrings to comply with what we described in the
"Ring states" section.

Thanks,
Stefano

Re: [PATCH] vhost-user.rst: Explicitly allow front-end to write to kick FDs

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 6859 bytes --]

On Mon, Apr 20, 2026 at 04:49:04PM +0200, Stefano Garzarella wrote:
> Thanks for starting the discussion here, let me add also Hanna, German, and
> Stefan in CC that can help us.
> 
> On Fri, Apr 10, 2026 at 07:12:05PM -0700, Jorge E. Moreira wrote:
> > Migration of back-end state happens while the device is suspended (i.e
> > all vrings are stopped). To resume normal operation on the destination,
> > the vrings need to be started again with a kick (either a write on the
> > FD or the VHOST_USER_VRING_KICK in-band message if negotiated). While
> 
> It's true that in the spec we have:
>   "Each ring is initialized in a stopped and disabled state. The   back-end
> must start a ring upon receiving a kick (that is, detecting   that file
> descriptor is readable) on the descriptor specified by
> VHOST_USER_SET_VRING_KICK or receiving the in-band message
> VHOST_USER_VRING_KICK if negotiated, and stop a ring upon receiving
> VHOST_USER_GET_VRING_BASE."
> 
> But IMO this applies when a driver is not yet loaded.
> When we are migrating, the driver could be already loaded. So, in the new
> device running in the destination, IMO we should consider the ring already
> started or add some messages to tell to the device: "hey, the device was
> already started, this is a migration and it's completed".
> 
> Sending a kick from the frontend, seems more an hack here.
> 
> That said, for example, in subprojects/libvhost-user/libvhost-user.c IIUC
> the virtqueue is started when the SET_VRING_KICK is handled by
> vu_set_vring_kick_exec(), but not sure how compliant it is.
> 
> > these notifications are typically sent by the driver, it has no reason
> > to send them in the destination if it already sent them in the source as
> > the driver is unaware that a migration took place. Therefore it should
> > be the responsibility of the vhost-user front-end to ensure these vrings
> > are started. This is particularly necessary for queues where data only
> > flows from device to driver, such as those used by the vsock and input
> > devices.
> 
> Exactly, so IMO we should not use the kick, but maybe add something new or
> clarify what to do after the migration.
> 
> For example in the "Migrating back-end state" we have:
>   "Migrating device state involves transferring the state from one
> back-end, called the source, to another back-end, called the   destination.
> After migration, the destination transparently resumes   operation without
> requiring the driver to re-initialize the device at   the VIRTIO level."
> 
> So, IMO we can use the VHOST_USER_SET_DEVICE_STATE_FD channel exactly to
> inform the new device about the state: "there isn't any state to transfer,
> but I notify you that the device was already initialized, so the vrings can
> be started".
> 
> > 
> > This behavior is already used by some qemu vhost-user front-ends (e.g
> > vhost-user-blk) and by front-ends implemented on other VMMs(e.g CrosVm).
> 
> I looked at vhost-user-blk frontend, but I don't see it. I mean I see the
> code around the comment "/* Kick right away to begin processing requests
> already in vring */" but that one IIUC was introduced more to fix devices
> violating specs, so not sure it's a good example to follow:
> 
> commit 110b9463d5c820120c8311db79f55a64c9d81ebe
> Author: Yongji Xie <elohimes@gmail.com>
> Date:   Wed Jun 6 21:24:48 2018 +0800
> 
>     vhost-user-blk: start vhost when guest kicks
>     Some old guests (before commit 7a11370e5: "virtio_blk: enable VQs early")
>     kick virtqueue before setting VIRTIO_CONFIG_S_DRIVER_OK. This violates
>     the virtio spec. But virtio 1.0 transitional devices support this behaviour.
>     So we should start vhost when guest kicks in this case.
>     Signed-off-by: Yongji Xie <xieyongji@baidu.com>
>     Signed-off-by: Chai Wen <chaiwen@baidu.com>
>     Signed-off-by: Ni Xun <nixun@baidu.com>
>     Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
>     Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
>     Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> 
> 
> > Adding it to the vhost-user documentation makes it explicit that this
> > strategy is permitted and suggest it to vhost-user front-end authors.
> > Explicitly documenting it is necessary because vring kicks appear
> > designed to originate in the driver, so having some originate in the
> > front-end can be counterintuitive and cause developers to waste time
> > looking for other alternatives or face pushback during code review.
> 
> As I pointed out in our discussion in
> https://github.com/rust-vmm/vhost-device/pull/936
> IMO we should use some in-band messages and not relaying on kicks that
> should be used only by the driver to notify the device about new available
> buffers.
> 
> That said, I agree that we need to clarify in the specifications exactly
> what the backend and frontend should do after a migration to start vrings if
> there is no need to exchange a state.
> 
> 
> Any other opinion?

IMO no protocol changes are needed but the vhost-user spec should be
tweaked. Hanna worked on device state migration and can confirm/deny
what I'm about to describe.

QEMU's libvhost-user implementation already starts the ring when
VHOST_USER_SET_VRING_KICK is received. This makes more sense than
waiting for the kick fd since a back-end that uses polling (peeking at
the vring in memory) shouldn't need to monitor the kick fd. See below
though about races between the kick fd and vhost-user protocol messages.

All of this boils down to the ring state machine. libvhost-user's
behavior is:
1. Virtqueues are started by VHOST_USER_SET_VRING_KICK.
2. Virtqueues are stopped by VHOST_USER_GET_VRING_BASE.
3. Virtqueues are enabled/disabled by VHOST_USER_SET_VRING_ENABLE.

The same sequence of vhost-user protocol messages that is used to
start/stop a device locally (e.g. pause and resume a VM) is the same
that can be used during migration. The ring state machine already exists
and needs to be used when migrating device state.

The following changes to the vhost-user spec would make this clearer:

1. Mention that virtqueues are started by VHOST_USER_SET_VRING_KICK.
   Note that monitoring the kick fd may be used to avoid races between
   the kick fd and vhost-user protocol messages, so in practice back-end
   implementors may still want to start the virtqueue when the kick fd
   becomes readable.

2. Add a clarification to "Migrating back-end state" that the device
   must be suspended (see _suspended_device_state) when
   VHOST_USER_SET_DEVICE_STATE_FD is sent and device state is
   transferred. This is already implicit in "Device state transfer
   parameters", but it's not obvious when reading the "Migrating
   back-end state" section.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH] vhost-user.rst: Explicitly allow front-end to write to kick FDs

[Jorge Moreira]

> So it’s fine to continue in this direction, but I might add these
words more in the "Migration" section than here, since we’re talking
about an optional state migration here. For example after "No further
update must be done before rings are restarted."
Or in the "Ring states" section, where we can clarify how to restart a
ring after a migration. Or in both :-)

I admit I don't know what that LOG_FD is for or what a "userfault-fd"
is, so I don't know how they relate to vring state. The paragraph I
modified discusses which part of the back-end state is in fact handled
by the front-end, which somewhat matches the "vring-can/should-kick"
idea. That being said, I don't have a strong preference for where this
should go. Repeating it in "Ring states" makes sense to me.


> QEMU's libvhost-user implementation already starts the ring when
VHOST_USER_SET_VRING_KICK is received. This makes more sense than
waiting for the kick fd since a back-end that uses polling (peeking at
the vring in memory) shouldn't need to monitor the kick fd. See below
though about races between the kick fd and vhost-user protocol messages.

Starting rings when SET_VRING_KICK is received technically violates
the vhost-user protocol as defined in the spec. QEMU may support
back-ends behaving this way, but other VMMs may not. Changing the spec
accordingly could potentially cause other implementations not
depending on QEMU's libvhost-user to become non-compliant, I prefer to
avoid this.

> Note that monitoring the kick fd may be used to avoid races between
> the kick fd and vhost-user protocol messages, so in practice back-end
> implementors may still want to start the virtqueue when the kick fd
> becomes readable

Implementations having to do in practice what the spec says today is
another reason not to change the it.

While starting the vrings on SET_VRING_KICK could solve the state
machine issue, it still won't notify the back-end that buffers are
ready (the driver won't do this). Non-polling back-ends depend on this
kick, especially for queues where data flows only from the driver to
the back-end. Most implementations likely attempt to read from the
queue only after receiving the kick.

------
Thanks for the feedback! I'll wait for consensus before making further
modifications.


On Mon, Apr 20, 2026 at 11:18 AM Stefan Hajnoczi <stefanha@redhat.com> wrote:
>
> On Mon, Apr 20, 2026 at 04:49:04PM +0200, Stefano Garzarella wrote:
> > Thanks for starting the discussion here, let me add also Hanna, German, and
> > Stefan in CC that can help us.
> >
> > On Fri, Apr 10, 2026 at 07:12:05PM -0700, Jorge E. Moreira wrote:
> > > Migration of back-end state happens while the device is suspended (i.e
> > > all vrings are stopped). To resume normal operation on the destination,
> > > the vrings need to be started again with a kick (either a write on the
> > > FD or the VHOST_USER_VRING_KICK in-band message if negotiated). While
> >
> > It's true that in the spec we have:
> >   "Each ring is initialized in a stopped and disabled state. The   back-end
> > must start a ring upon receiving a kick (that is, detecting   that file
> > descriptor is readable) on the descriptor specified by
> > VHOST_USER_SET_VRING_KICK or receiving the in-band message
> > VHOST_USER_VRING_KICK if negotiated, and stop a ring upon receiving
> > VHOST_USER_GET_VRING_BASE."
> >
> > But IMO this applies when a driver is not yet loaded.
> > When we are migrating, the driver could be already loaded. So, in the new
> > device running in the destination, IMO we should consider the ring already
> > started or add some messages to tell to the device: "hey, the device was
> > already started, this is a migration and it's completed".
> >
> > Sending a kick from the frontend, seems more an hack here.
> >
> > That said, for example, in subprojects/libvhost-user/libvhost-user.c IIUC
> > the virtqueue is started when the SET_VRING_KICK is handled by
> > vu_set_vring_kick_exec(), but not sure how compliant it is.
> >
> > > these notifications are typically sent by the driver, it has no reason
> > > to send them in the destination if it already sent them in the source as
> > > the driver is unaware that a migration took place. Therefore it should
> > > be the responsibility of the vhost-user front-end to ensure these vrings
> > > are started. This is particularly necessary for queues where data only
> > > flows from device to driver, such as those used by the vsock and input
> > > devices.
> >
> > Exactly, so IMO we should not use the kick, but maybe add something new or
> > clarify what to do after the migration.
> >
> > For example in the "Migrating back-end state" we have:
> >   "Migrating device state involves transferring the state from one
> > back-end, called the source, to another back-end, called the   destination.
> > After migration, the destination transparently resumes   operation without
> > requiring the driver to re-initialize the device at   the VIRTIO level."
> >
> > So, IMO we can use the VHOST_USER_SET_DEVICE_STATE_FD channel exactly to
> > inform the new device about the state: "there isn't any state to transfer,
> > but I notify you that the device was already initialized, so the vrings can
> > be started".
> >
> > >
> > > This behavior is already used by some qemu vhost-user front-ends (e.g
> > > vhost-user-blk) and by front-ends implemented on other VMMs(e.g CrosVm).
> >
> > I looked at vhost-user-blk frontend, but I don't see it. I mean I see the
> > code around the comment "/* Kick right away to begin processing requests
> > already in vring */" but that one IIUC was introduced more to fix devices
> > violating specs, so not sure it's a good example to follow:
> >
> > commit 110b9463d5c820120c8311db79f55a64c9d81ebe
> > Author: Yongji Xie <elohimes@gmail.com>
> > Date:   Wed Jun 6 21:24:48 2018 +0800
> >
> >     vhost-user-blk: start vhost when guest kicks
> >     Some old guests (before commit 7a11370e5: "virtio_blk: enable VQs early")
> >     kick virtqueue before setting VIRTIO_CONFIG_S_DRIVER_OK. This violates
> >     the virtio spec. But virtio 1.0 transitional devices support this behaviour.
> >     So we should start vhost when guest kicks in this case.
> >     Signed-off-by: Yongji Xie <xieyongji@baidu.com>
> >     Signed-off-by: Chai Wen <chaiwen@baidu.com>
> >     Signed-off-by: Ni Xun <nixun@baidu.com>
> >     Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
> >     Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
> >     Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> >
> >
> > > Adding it to the vhost-user documentation makes it explicit that this
> > > strategy is permitted and suggest it to vhost-user front-end authors.
> > > Explicitly documenting it is necessary because vring kicks appear
> > > designed to originate in the driver, so having some originate in the
> > > front-end can be counterintuitive and cause developers to waste time
> > > looking for other alternatives or face pushback during code review.
> >
> > As I pointed out in our discussion in
> > https://github.com/rust-vmm/vhost-device/pull/936
> > IMO we should use some in-band messages and not relaying on kicks that
> > should be used only by the driver to notify the device about new available
> > buffers.
> >
> > That said, I agree that we need to clarify in the specifications exactly
> > what the backend and frontend should do after a migration to start vrings if
> > there is no need to exchange a state.
> >
> >
> > Any other opinion?
>
> IMO no protocol changes are needed but the vhost-user spec should be
> tweaked. Hanna worked on device state migration and can confirm/deny
> what I'm about to describe.
>
> QEMU's libvhost-user implementation already starts the ring when
> VHOST_USER_SET_VRING_KICK is received. This makes more sense than
> waiting for the kick fd since a back-end that uses polling (peeking at
> the vring in memory) shouldn't need to monitor the kick fd. See below
> though about races between the kick fd and vhost-user protocol messages.
>
> All of this boils down to the ring state machine. libvhost-user's
> behavior is:
> 1. Virtqueues are started by VHOST_USER_SET_VRING_KICK.
> 2. Virtqueues are stopped by VHOST_USER_GET_VRING_BASE.
> 3. Virtqueues are enabled/disabled by VHOST_USER_SET_VRING_ENABLE.
>
> The same sequence of vhost-user protocol messages that is used to
> start/stop a device locally (e.g. pause and resume a VM) is the same
> that can be used during migration. The ring state machine already exists
> and needs to be used when migrating device state.
>
> The following changes to the vhost-user spec would make this clearer:
>
> 1. Mention that virtqueues are started by VHOST_USER_SET_VRING_KICK.
>    Note that monitoring the kick fd may be used to avoid races between
>    the kick fd and vhost-user protocol messages, so in practice back-end
>    implementors may still want to start the virtqueue when the kick fd
>    becomes readable.
>
> 2. Add a clarification to "Migrating back-end state" that the device
>    must be suspended (see _suspended_device_state) when
>    VHOST_USER_SET_DEVICE_STATE_FD is sent and device state is
>    transferred. This is already implicit in "Device state transfer
>    parameters", but it's not obvious when reading the "Migrating
>    back-end state" section.
>
> Stefan

Re: [PATCH] vhost-user.rst: Explicitly allow front-end to write to kick FDs

[Stefano Garzarella]

On Tue, 21 Apr 2026 at 02:48, Jorge Moreira <jemoreira@google.com> wrote:
>
> > So it’s fine to continue in this direction, but I might add these
> words more in the "Migration" section than here, since we’re talking
> about an optional state migration here. For example after "No further
> update must be done before rings are restarted."
> Or in the "Ring states" section, where we can clarify how to restart a
> ring after a migration. Or in both :-)
>
> I admit I don't know what that LOG_FD is for or what a "userfault-fd"
> is, so I don't know how they relate to vring state. The paragraph I
> modified discusses which part of the back-end state is in fact handled
> by the front-end, which somewhat matches the "vring-can/should-kick"
> idea. That being said, I don't have a strong preference for where this
> should go. Repeating it in "Ring states" makes sense to me.

I didn't express myself clearly, but my point is that discussing this
in the "Migrating back-end state" section is misleading. We should
address it first because it has nothing to do with state; it's also
needed for devices that don't require any state transfer.

>
>
> > QEMU's libvhost-user implementation already starts the ring when
> VHOST_USER_SET_VRING_KICK is received. This makes more sense than
> waiting for the kick fd since a back-end that uses polling (peeking at
> the vring in memory) shouldn't need to monitor the kick fd. See below
> though about races between the kick fd and vhost-user protocol messages.

Just a note, we do the same in the rust-vmm vhost-user-backend crate as well.

> Starting rings when SET_VRING_KICK is received technically violates
> the vhost-user protocol as defined in the spec. QEMU may support
> back-ends behaving this way, but other VMMs may not. Changing the spec
> accordingly could potentially cause other implementations not
> depending on QEMU's libvhost-user to become non-compliant, I prefer to
> avoid this.

This is a good point and essentially what changed my mind.

However, this doesn't change anything for those frontends that also
send the kick (I don't know which ones do this, but QEMU doesn't, and
I don't think CH does either, since it uses the rust-vmm crates); they
continue to work just the same.

As for the backends, we’ve seen that both libvhost-user in QEMU and
vhost-user-backend in rust-vmm do exactly this (start vrings once
SET_VRING_KICK is received), so perhaps clarifying in this way isn't
too risky.

Do you know other backends we should check?

>
> > Note that monitoring the kick fd may be used to avoid races between
> > the kick fd and vhost-user protocol messages, so in practice back-end
> > implementors may still want to start the virtqueue when the kick fd
> > becomes readable
>
> Implementations having to do in practice what the spec says today is
> another reason not to change the it.
>
> While starting the vrings on SET_VRING_KICK could solve the state
> machine issue, it still won't notify the back-end that buffers are
> ready (the driver won't do this). Non-polling back-ends depend on this
> kick, especially for queues where data flows only from the driver to
> the back-end. Most implementations likely attempt to read from the
> queue only after receiving the kick.

Once the vrings are set, the VMM and the device have exactly the same
view of the vring state, so IMO the kick is just some extra step (I'm
fine to add it just because the spec wasn't clear). The device can
start in the same way without it.

>
> ------
> Thanks for the feedback! I'll wait for consensus before making further
> modifications.

Thanks!
Stefano

>
>
> On Mon, Apr 20, 2026 at 11:18 AM Stefan Hajnoczi <stefanha@redhat.com> wrote:
> >
> > On Mon, Apr 20, 2026 at 04:49:04PM +0200, Stefano Garzarella wrote:
> > > Thanks for starting the discussion here, let me add also Hanna, German, and
> > > Stefan in CC that can help us.
> > >
> > > On Fri, Apr 10, 2026 at 07:12:05PM -0700, Jorge E. Moreira wrote:
> > > > Migration of back-end state happens while the device is suspended (i.e
> > > > all vrings are stopped). To resume normal operation on the destination,
> > > > the vrings need to be started again with a kick (either a write on the
> > > > FD or the VHOST_USER_VRING_KICK in-band message if negotiated). While
> > >
> > > It's true that in the spec we have:
> > >   "Each ring is initialized in a stopped and disabled state. The   back-end
> > > must start a ring upon receiving a kick (that is, detecting   that file
> > > descriptor is readable) on the descriptor specified by
> > > VHOST_USER_SET_VRING_KICK or receiving the in-band message
> > > VHOST_USER_VRING_KICK if negotiated, and stop a ring upon receiving
> > > VHOST_USER_GET_VRING_BASE."
> > >
> > > But IMO this applies when a driver is not yet loaded.
> > > When we are migrating, the driver could be already loaded. So, in the new
> > > device running in the destination, IMO we should consider the ring already
> > > started or add some messages to tell to the device: "hey, the device was
> > > already started, this is a migration and it's completed".
> > >
> > > Sending a kick from the frontend, seems more an hack here.
> > >
> > > That said, for example, in subprojects/libvhost-user/libvhost-user.c IIUC
> > > the virtqueue is started when the SET_VRING_KICK is handled by
> > > vu_set_vring_kick_exec(), but not sure how compliant it is.
> > >
> > > > these notifications are typically sent by the driver, it has no reason
> > > > to send them in the destination if it already sent them in the source as
> > > > the driver is unaware that a migration took place. Therefore it should
> > > > be the responsibility of the vhost-user front-end to ensure these vrings
> > > > are started. This is particularly necessary for queues where data only
> > > > flows from device to driver, such as those used by the vsock and input
> > > > devices.
> > >
> > > Exactly, so IMO we should not use the kick, but maybe add something new or
> > > clarify what to do after the migration.
> > >
> > > For example in the "Migrating back-end state" we have:
> > >   "Migrating device state involves transferring the state from one
> > > back-end, called the source, to another back-end, called the   destination.
> > > After migration, the destination transparently resumes   operation without
> > > requiring the driver to re-initialize the device at   the VIRTIO level."
> > >
> > > So, IMO we can use the VHOST_USER_SET_DEVICE_STATE_FD channel exactly to
> > > inform the new device about the state: "there isn't any state to transfer,
> > > but I notify you that the device was already initialized, so the vrings can
> > > be started".
> > >
> > > >
> > > > This behavior is already used by some qemu vhost-user front-ends (e.g
> > > > vhost-user-blk) and by front-ends implemented on other VMMs(e.g CrosVm).
> > >
> > > I looked at vhost-user-blk frontend, but I don't see it. I mean I see the
> > > code around the comment "/* Kick right away to begin processing requests
> > > already in vring */" but that one IIUC was introduced more to fix devices
> > > violating specs, so not sure it's a good example to follow:
> > >
> > > commit 110b9463d5c820120c8311db79f55a64c9d81ebe
> > > Author: Yongji Xie <elohimes@gmail.com>
> > > Date:   Wed Jun 6 21:24:48 2018 +0800
> > >
> > >     vhost-user-blk: start vhost when guest kicks
> > >     Some old guests (before commit 7a11370e5: "virtio_blk: enable VQs early")
> > >     kick virtqueue before setting VIRTIO_CONFIG_S_DRIVER_OK. This violates
> > >     the virtio spec. But virtio 1.0 transitional devices support this behaviour.
> > >     So we should start vhost when guest kicks in this case.
> > >     Signed-off-by: Yongji Xie <xieyongji@baidu.com>
> > >     Signed-off-by: Chai Wen <chaiwen@baidu.com>
> > >     Signed-off-by: Ni Xun <nixun@baidu.com>
> > >     Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
> > >     Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
> > >     Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> > >
> > >
> > > > Adding it to the vhost-user documentation makes it explicit that this
> > > > strategy is permitted and suggest it to vhost-user front-end authors.
> > > > Explicitly documenting it is necessary because vring kicks appear
> > > > designed to originate in the driver, so having some originate in the
> > > > front-end can be counterintuitive and cause developers to waste time
> > > > looking for other alternatives or face pushback during code review.
> > >
> > > As I pointed out in our discussion in
> > > https://github.com/rust-vmm/vhost-device/pull/936
> > > IMO we should use some in-band messages and not relaying on kicks that
> > > should be used only by the driver to notify the device about new available
> > > buffers.
> > >
> > > That said, I agree that we need to clarify in the specifications exactly
> > > what the backend and frontend should do after a migration to start vrings if
> > > there is no need to exchange a state.
> > >
> > >
> > > Any other opinion?
> >
> > IMO no protocol changes are needed but the vhost-user spec should be
> > tweaked. Hanna worked on device state migration and can confirm/deny
> > what I'm about to describe.
> >
> > QEMU's libvhost-user implementation already starts the ring when
> > VHOST_USER_SET_VRING_KICK is received. This makes more sense than
> > waiting for the kick fd since a back-end that uses polling (peeking at
> > the vring in memory) shouldn't need to monitor the kick fd. See below
> > though about races between the kick fd and vhost-user protocol messages.
> >
> > All of this boils down to the ring state machine. libvhost-user's
> > behavior is:
> > 1. Virtqueues are started by VHOST_USER_SET_VRING_KICK.
> > 2. Virtqueues are stopped by VHOST_USER_GET_VRING_BASE.
> > 3. Virtqueues are enabled/disabled by VHOST_USER_SET_VRING_ENABLE.
> >
> > The same sequence of vhost-user protocol messages that is used to
> > start/stop a device locally (e.g. pause and resume a VM) is the same
> > that can be used during migration. The ring state machine already exists
> > and needs to be used when migrating device state.
> >
> > The following changes to the vhost-user spec would make this clearer:
> >
> > 1. Mention that virtqueues are started by VHOST_USER_SET_VRING_KICK.
> >    Note that monitoring the kick fd may be used to avoid races between
> >    the kick fd and vhost-user protocol messages, so in practice back-end
> >    implementors may still want to start the virtqueue when the kick fd
> >    becomes readable.
> >
> > 2. Add a clarification to "Migrating back-end state" that the device
> >    must be suspended (see _suspended_device_state) when
> >    VHOST_USER_SET_DEVICE_STATE_FD is sent and device state is
> >    transferred. This is already implicit in "Device state transfer
> >    parameters", but it's not obvious when reading the "Migrating
> >    back-end state" section.
> >
> > Stefan
>

Re: [PATCH] vhost-user.rst: Explicitly allow front-end to write to kick FDs

[Jorge Moreira]

> Do you know other backends we should check?

CrosVm's: https://github.com/google/crosvm/tree/main/devices/src/virtio/vhost_user_backend


On Tue, Apr 21, 2026 at 12:55 AM Stefano Garzarella <sgarzare@redhat.com> wrote:
>
> On Tue, 21 Apr 2026 at 02:48, Jorge Moreira <jemoreira@google.com> wrote:
> >
> > > So it’s fine to continue in this direction, but I might add these
> > words more in the "Migration" section than here, since we’re talking
> > about an optional state migration here. For example after "No further
> > update must be done before rings are restarted."
> > Or in the "Ring states" section, where we can clarify how to restart a
> > ring after a migration. Or in both :-)
> >
> > I admit I don't know what that LOG_FD is for or what a "userfault-fd"
> > is, so I don't know how they relate to vring state. The paragraph I
> > modified discusses which part of the back-end state is in fact handled
> > by the front-end, which somewhat matches the "vring-can/should-kick"
> > idea. That being said, I don't have a strong preference for where this
> > should go. Repeating it in "Ring states" makes sense to me.
>
> I didn't express myself clearly, but my point is that discussing this
> in the "Migrating back-end state" section is misleading. We should
> address it first because it has nothing to do with state; it's also
> needed for devices that don't require any state transfer.
>
> >
> >
> > > QEMU's libvhost-user implementation already starts the ring when
> > VHOST_USER_SET_VRING_KICK is received. This makes more sense than
> > waiting for the kick fd since a back-end that uses polling (peeking at
> > the vring in memory) shouldn't need to monitor the kick fd. See below
> > though about races between the kick fd and vhost-user protocol messages.
>
> Just a note, we do the same in the rust-vmm vhost-user-backend crate as well.
>
> > Starting rings when SET_VRING_KICK is received technically violates
> > the vhost-user protocol as defined in the spec. QEMU may support
> > back-ends behaving this way, but other VMMs may not. Changing the spec
> > accordingly could potentially cause other implementations not
> > depending on QEMU's libvhost-user to become non-compliant, I prefer to
> > avoid this.
>
> This is a good point and essentially what changed my mind.
>
> However, this doesn't change anything for those frontends that also
> send the kick (I don't know which ones do this, but QEMU doesn't, and
> I don't think CH does either, since it uses the rust-vmm crates); they
> continue to work just the same.
>
> As for the backends, we’ve seen that both libvhost-user in QEMU and
> vhost-user-backend in rust-vmm do exactly this (start vrings once
> SET_VRING_KICK is received), so perhaps clarifying in this way isn't
> too risky.
>
> Do you know other backends we should check?
>
> >
> > > Note that monitoring the kick fd may be used to avoid races between
> > > the kick fd and vhost-user protocol messages, so in practice back-end
> > > implementors may still want to start the virtqueue when the kick fd
> > > becomes readable
> >
> > Implementations having to do in practice what the spec says today is
> > another reason not to change the it.
> >
> > While starting the vrings on SET_VRING_KICK could solve the state
> > machine issue, it still won't notify the back-end that buffers are
> > ready (the driver won't do this). Non-polling back-ends depend on this
> > kick, especially for queues where data flows only from the driver to
> > the back-end. Most implementations likely attempt to read from the
> > queue only after receiving the kick.
>
> Once the vrings are set, the VMM and the device have exactly the same
> view of the vring state, so IMO the kick is just some extra step (I'm
> fine to add it just because the spec wasn't clear). The device can
> start in the same way without it.
>
> >
> > ------
> > Thanks for the feedback! I'll wait for consensus before making further
> > modifications.
>
> Thanks!
> Stefano
>
> >
> >
> > On Mon, Apr 20, 2026 at 11:18 AM Stefan Hajnoczi <stefanha@redhat.com> wrote:
> > >
> > > On Mon, Apr 20, 2026 at 04:49:04PM +0200, Stefano Garzarella wrote:
> > > > Thanks for starting the discussion here, let me add also Hanna, German, and
> > > > Stefan in CC that can help us.
> > > >
> > > > On Fri, Apr 10, 2026 at 07:12:05PM -0700, Jorge E. Moreira wrote:
> > > > > Migration of back-end state happens while the device is suspended (i.e
> > > > > all vrings are stopped). To resume normal operation on the destination,
> > > > > the vrings need to be started again with a kick (either a write on the
> > > > > FD or the VHOST_USER_VRING_KICK in-band message if negotiated). While
> > > >
> > > > It's true that in the spec we have:
> > > >   "Each ring is initialized in a stopped and disabled state. The   back-end
> > > > must start a ring upon receiving a kick (that is, detecting   that file
> > > > descriptor is readable) on the descriptor specified by
> > > > VHOST_USER_SET_VRING_KICK or receiving the in-band message
> > > > VHOST_USER_VRING_KICK if negotiated, and stop a ring upon receiving
> > > > VHOST_USER_GET_VRING_BASE."
> > > >
> > > > But IMO this applies when a driver is not yet loaded.
> > > > When we are migrating, the driver could be already loaded. So, in the new
> > > > device running in the destination, IMO we should consider the ring already
> > > > started or add some messages to tell to the device: "hey, the device was
> > > > already started, this is a migration and it's completed".
> > > >
> > > > Sending a kick from the frontend, seems more an hack here.
> > > >
> > > > That said, for example, in subprojects/libvhost-user/libvhost-user.c IIUC
> > > > the virtqueue is started when the SET_VRING_KICK is handled by
> > > > vu_set_vring_kick_exec(), but not sure how compliant it is.
> > > >
> > > > > these notifications are typically sent by the driver, it has no reason
> > > > > to send them in the destination if it already sent them in the source as
> > > > > the driver is unaware that a migration took place. Therefore it should
> > > > > be the responsibility of the vhost-user front-end to ensure these vrings
> > > > > are started. This is particularly necessary for queues where data only
> > > > > flows from device to driver, such as those used by the vsock and input
> > > > > devices.
> > > >
> > > > Exactly, so IMO we should not use the kick, but maybe add something new or
> > > > clarify what to do after the migration.
> > > >
> > > > For example in the "Migrating back-end state" we have:
> > > >   "Migrating device state involves transferring the state from one
> > > > back-end, called the source, to another back-end, called the   destination.
> > > > After migration, the destination transparently resumes   operation without
> > > > requiring the driver to re-initialize the device at   the VIRTIO level."
> > > >
> > > > So, IMO we can use the VHOST_USER_SET_DEVICE_STATE_FD channel exactly to
> > > > inform the new device about the state: "there isn't any state to transfer,
> > > > but I notify you that the device was already initialized, so the vrings can
> > > > be started".
> > > >
> > > > >
> > > > > This behavior is already used by some qemu vhost-user front-ends (e.g
> > > > > vhost-user-blk) and by front-ends implemented on other VMMs(e.g CrosVm).
> > > >
> > > > I looked at vhost-user-blk frontend, but I don't see it. I mean I see the
> > > > code around the comment "/* Kick right away to begin processing requests
> > > > already in vring */" but that one IIUC was introduced more to fix devices
> > > > violating specs, so not sure it's a good example to follow:
> > > >
> > > > commit 110b9463d5c820120c8311db79f55a64c9d81ebe
> > > > Author: Yongji Xie <elohimes@gmail.com>
> > > > Date:   Wed Jun 6 21:24:48 2018 +0800
> > > >
> > > >     vhost-user-blk: start vhost when guest kicks
> > > >     Some old guests (before commit 7a11370e5: "virtio_blk: enable VQs early")
> > > >     kick virtqueue before setting VIRTIO_CONFIG_S_DRIVER_OK. This violates
> > > >     the virtio spec. But virtio 1.0 transitional devices support this behaviour.
> > > >     So we should start vhost when guest kicks in this case.
> > > >     Signed-off-by: Yongji Xie <xieyongji@baidu.com>
> > > >     Signed-off-by: Chai Wen <chaiwen@baidu.com>
> > > >     Signed-off-by: Ni Xun <nixun@baidu.com>
> > > >     Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
> > > >     Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
> > > >     Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> > > >
> > > >
> > > > > Adding it to the vhost-user documentation makes it explicit that this
> > > > > strategy is permitted and suggest it to vhost-user front-end authors.
> > > > > Explicitly documenting it is necessary because vring kicks appear
> > > > > designed to originate in the driver, so having some originate in the
> > > > > front-end can be counterintuitive and cause developers to waste time
> > > > > looking for other alternatives or face pushback during code review.
> > > >
> > > > As I pointed out in our discussion in
> > > > https://github.com/rust-vmm/vhost-device/pull/936
> > > > IMO we should use some in-band messages and not relaying on kicks that
> > > > should be used only by the driver to notify the device about new available
> > > > buffers.
> > > >
> > > > That said, I agree that we need to clarify in the specifications exactly
> > > > what the backend and frontend should do after a migration to start vrings if
> > > > there is no need to exchange a state.
> > > >
> > > >
> > > > Any other opinion?
> > >
> > > IMO no protocol changes are needed but the vhost-user spec should be
> > > tweaked. Hanna worked on device state migration and can confirm/deny
> > > what I'm about to describe.
> > >
> > > QEMU's libvhost-user implementation already starts the ring when
> > > VHOST_USER_SET_VRING_KICK is received. This makes more sense than
> > > waiting for the kick fd since a back-end that uses polling (peeking at
> > > the vring in memory) shouldn't need to monitor the kick fd. See below
> > > though about races between the kick fd and vhost-user protocol messages.
> > >
> > > All of this boils down to the ring state machine. libvhost-user's
> > > behavior is:
> > > 1. Virtqueues are started by VHOST_USER_SET_VRING_KICK.
> > > 2. Virtqueues are stopped by VHOST_USER_GET_VRING_BASE.
> > > 3. Virtqueues are enabled/disabled by VHOST_USER_SET_VRING_ENABLE.
> > >
> > > The same sequence of vhost-user protocol messages that is used to
> > > start/stop a device locally (e.g. pause and resume a VM) is the same
> > > that can be used during migration. The ring state machine already exists
> > > and needs to be used when migrating device state.
> > >
> > > The following changes to the vhost-user spec would make this clearer:
> > >
> > > 1. Mention that virtqueues are started by VHOST_USER_SET_VRING_KICK.
> > >    Note that monitoring the kick fd may be used to avoid races between
> > >    the kick fd and vhost-user protocol messages, so in practice back-end
> > >    implementors may still want to start the virtqueue when the kick fd
> > >    becomes readable.
> > >
> > > 2. Add a clarification to "Migrating back-end state" that the device
> > >    must be suspended (see _suspended_device_state) when
> > >    VHOST_USER_SET_DEVICE_STATE_FD is sent and device state is
> > >    transferred. This is already implicit in "Device state transfer
> > >    parameters", but it's not obvious when reading the "Migrating
> > >    back-end state" section.
> > >
> > > Stefan
> >
>

Re: [PATCH] vhost-user.rst: Explicitly allow front-end to write to kick FDs

[Stefano Garzarella]

On Tue, 21 Apr 2026 at 18:06, Jorge Moreira <jemoreira@google.com> wrote:
>
> > Do you know other backends we should check?
>
> CrosVm's: https://github.com/google/crosvm/tree/main/devices/src/virtio/vhost_user_backend

Which do exactly the same
https://github.com/google/crosvm/blob/b49e336de56d6353ae06245de0c7fabec3db2d5e/devices/src/virtio/vhost_user_backend/handler.rs#L596

No?


>
>
> On Tue, Apr 21, 2026 at 12:55 AM Stefano Garzarella <sgarzare@redhat.com> wrote:
> >
> > On Tue, 21 Apr 2026 at 02:48, Jorge Moreira <jemoreira@google.com> wrote:
> > >
> > > > So it’s fine to continue in this direction, but I might add these
> > > words more in the "Migration" section than here, since we’re talking
> > > about an optional state migration here. For example after "No further
> > > update must be done before rings are restarted."
> > > Or in the "Ring states" section, where we can clarify how to restart a
> > > ring after a migration. Or in both :-)
> > >
> > > I admit I don't know what that LOG_FD is for or what a "userfault-fd"
> > > is, so I don't know how they relate to vring state. The paragraph I
> > > modified discusses which part of the back-end state is in fact handled
> > > by the front-end, which somewhat matches the "vring-can/should-kick"
> > > idea. That being said, I don't have a strong preference for where this
> > > should go. Repeating it in "Ring states" makes sense to me.
> >
> > I didn't express myself clearly, but my point is that discussing this
> > in the "Migrating back-end state" section is misleading. We should
> > address it first because it has nothing to do with state; it's also
> > needed for devices that don't require any state transfer.
> >
> > >
> > >
> > > > QEMU's libvhost-user implementation already starts the ring when
> > > VHOST_USER_SET_VRING_KICK is received. This makes more sense than
> > > waiting for the kick fd since a back-end that uses polling (peeking at
> > > the vring in memory) shouldn't need to monitor the kick fd. See below
> > > though about races between the kick fd and vhost-user protocol messages.
> >
> > Just a note, we do the same in the rust-vmm vhost-user-backend crate as well.
> >
> > > Starting rings when SET_VRING_KICK is received technically violates
> > > the vhost-user protocol as defined in the spec. QEMU may support
> > > back-ends behaving this way, but other VMMs may not. Changing the spec
> > > accordingly could potentially cause other implementations not
> > > depending on QEMU's libvhost-user to become non-compliant, I prefer to
> > > avoid this.
> >
> > This is a good point and essentially what changed my mind.
> >
> > However, this doesn't change anything for those frontends that also
> > send the kick (I don't know which ones do this, but QEMU doesn't, and
> > I don't think CH does either, since it uses the rust-vmm crates); they
> > continue to work just the same.
> >
> > As for the backends, we’ve seen that both libvhost-user in QEMU and
> > vhost-user-backend in rust-vmm do exactly this (start vrings once
> > SET_VRING_KICK is received), so perhaps clarifying in this way isn't
> > too risky.
> >
> > Do you know other backends we should check?
> >
> > >
> > > > Note that monitoring the kick fd may be used to avoid races between
> > > > the kick fd and vhost-user protocol messages, so in practice back-end
> > > > implementors may still want to start the virtqueue when the kick fd
> > > > becomes readable
> > >
> > > Implementations having to do in practice what the spec says today is
> > > another reason not to change the it.
> > >
> > > While starting the vrings on SET_VRING_KICK could solve the state
> > > machine issue, it still won't notify the back-end that buffers are
> > > ready (the driver won't do this). Non-polling back-ends depend on this
> > > kick, especially for queues where data flows only from the driver to
> > > the back-end. Most implementations likely attempt to read from the
> > > queue only after receiving the kick.
> >
> > Once the vrings are set, the VMM and the device have exactly the same
> > view of the vring state, so IMO the kick is just some extra step (I'm
> > fine to add it just because the spec wasn't clear). The device can
> > start in the same way without it.
> >
> > >
> > > ------
> > > Thanks for the feedback! I'll wait for consensus before making further
> > > modifications.
> >
> > Thanks!
> > Stefano
> >
> > >
> > >
> > > On Mon, Apr 20, 2026 at 11:18 AM Stefan Hajnoczi <stefanha@redhat.com> wrote:
> > > >
> > > > On Mon, Apr 20, 2026 at 04:49:04PM +0200, Stefano Garzarella wrote:
> > > > > Thanks for starting the discussion here, let me add also Hanna, German, and
> > > > > Stefan in CC that can help us.
> > > > >
> > > > > On Fri, Apr 10, 2026 at 07:12:05PM -0700, Jorge E. Moreira wrote:
> > > > > > Migration of back-end state happens while the device is suspended (i.e
> > > > > > all vrings are stopped). To resume normal operation on the destination,
> > > > > > the vrings need to be started again with a kick (either a write on the
> > > > > > FD or the VHOST_USER_VRING_KICK in-band message if negotiated). While
> > > > >
> > > > > It's true that in the spec we have:
> > > > >   "Each ring is initialized in a stopped and disabled state. The   back-end
> > > > > must start a ring upon receiving a kick (that is, detecting   that file
> > > > > descriptor is readable) on the descriptor specified by
> > > > > VHOST_USER_SET_VRING_KICK or receiving the in-band message
> > > > > VHOST_USER_VRING_KICK if negotiated, and stop a ring upon receiving
> > > > > VHOST_USER_GET_VRING_BASE."
> > > > >
> > > > > But IMO this applies when a driver is not yet loaded.
> > > > > When we are migrating, the driver could be already loaded. So, in the new
> > > > > device running in the destination, IMO we should consider the ring already
> > > > > started or add some messages to tell to the device: "hey, the device was
> > > > > already started, this is a migration and it's completed".
> > > > >
> > > > > Sending a kick from the frontend, seems more an hack here.
> > > > >
> > > > > That said, for example, in subprojects/libvhost-user/libvhost-user.c IIUC
> > > > > the virtqueue is started when the SET_VRING_KICK is handled by
> > > > > vu_set_vring_kick_exec(), but not sure how compliant it is.
> > > > >
> > > > > > these notifications are typically sent by the driver, it has no reason
> > > > > > to send them in the destination if it already sent them in the source as
> > > > > > the driver is unaware that a migration took place. Therefore it should
> > > > > > be the responsibility of the vhost-user front-end to ensure these vrings
> > > > > > are started. This is particularly necessary for queues where data only
> > > > > > flows from device to driver, such as those used by the vsock and input
> > > > > > devices.
> > > > >
> > > > > Exactly, so IMO we should not use the kick, but maybe add something new or
> > > > > clarify what to do after the migration.
> > > > >
> > > > > For example in the "Migrating back-end state" we have:
> > > > >   "Migrating device state involves transferring the state from one
> > > > > back-end, called the source, to another back-end, called the   destination.
> > > > > After migration, the destination transparently resumes   operation without
> > > > > requiring the driver to re-initialize the device at   the VIRTIO level."
> > > > >
> > > > > So, IMO we can use the VHOST_USER_SET_DEVICE_STATE_FD channel exactly to
> > > > > inform the new device about the state: "there isn't any state to transfer,
> > > > > but I notify you that the device was already initialized, so the vrings can
> > > > > be started".
> > > > >
> > > > > >
> > > > > > This behavior is already used by some qemu vhost-user front-ends (e.g
> > > > > > vhost-user-blk) and by front-ends implemented on other VMMs(e.g CrosVm).
> > > > >
> > > > > I looked at vhost-user-blk frontend, but I don't see it. I mean I see the
> > > > > code around the comment "/* Kick right away to begin processing requests
> > > > > already in vring */" but that one IIUC was introduced more to fix devices
> > > > > violating specs, so not sure it's a good example to follow:
> > > > >
> > > > > commit 110b9463d5c820120c8311db79f55a64c9d81ebe
> > > > > Author: Yongji Xie <elohimes@gmail.com>
> > > > > Date:   Wed Jun 6 21:24:48 2018 +0800
> > > > >
> > > > >     vhost-user-blk: start vhost when guest kicks
> > > > >     Some old guests (before commit 7a11370e5: "virtio_blk: enable VQs early")
> > > > >     kick virtqueue before setting VIRTIO_CONFIG_S_DRIVER_OK. This violates
> > > > >     the virtio spec. But virtio 1.0 transitional devices support this behaviour.
> > > > >     So we should start vhost when guest kicks in this case.
> > > > >     Signed-off-by: Yongji Xie <xieyongji@baidu.com>
> > > > >     Signed-off-by: Chai Wen <chaiwen@baidu.com>
> > > > >     Signed-off-by: Ni Xun <nixun@baidu.com>
> > > > >     Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
> > > > >     Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
> > > > >     Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> > > > >
> > > > >
> > > > > > Adding it to the vhost-user documentation makes it explicit that this
> > > > > > strategy is permitted and suggest it to vhost-user front-end authors.
> > > > > > Explicitly documenting it is necessary because vring kicks appear
> > > > > > designed to originate in the driver, so having some originate in the
> > > > > > front-end can be counterintuitive and cause developers to waste time
> > > > > > looking for other alternatives or face pushback during code review.
> > > > >
> > > > > As I pointed out in our discussion in
> > > > > https://github.com/rust-vmm/vhost-device/pull/936
> > > > > IMO we should use some in-band messages and not relaying on kicks that
> > > > > should be used only by the driver to notify the device about new available
> > > > > buffers.
> > > > >
> > > > > That said, I agree that we need to clarify in the specifications exactly
> > > > > what the backend and frontend should do after a migration to start vrings if
> > > > > there is no need to exchange a state.
> > > > >
> > > > >
> > > > > Any other opinion?
> > > >
> > > > IMO no protocol changes are needed but the vhost-user spec should be
> > > > tweaked. Hanna worked on device state migration and can confirm/deny
> > > > what I'm about to describe.
> > > >
> > > > QEMU's libvhost-user implementation already starts the ring when
> > > > VHOST_USER_SET_VRING_KICK is received. This makes more sense than
> > > > waiting for the kick fd since a back-end that uses polling (peeking at
> > > > the vring in memory) shouldn't need to monitor the kick fd. See below
> > > > though about races between the kick fd and vhost-user protocol messages.
> > > >
> > > > All of this boils down to the ring state machine. libvhost-user's
> > > > behavior is:
> > > > 1. Virtqueues are started by VHOST_USER_SET_VRING_KICK.
> > > > 2. Virtqueues are stopped by VHOST_USER_GET_VRING_BASE.
> > > > 3. Virtqueues are enabled/disabled by VHOST_USER_SET_VRING_ENABLE.
> > > >
> > > > The same sequence of vhost-user protocol messages that is used to
> > > > start/stop a device locally (e.g. pause and resume a VM) is the same
> > > > that can be used during migration. The ring state machine already exists
> > > > and needs to be used when migrating device state.
> > > >
> > > > The following changes to the vhost-user spec would make this clearer:
> > > >
> > > > 1. Mention that virtqueues are started by VHOST_USER_SET_VRING_KICK.
> > > >    Note that monitoring the kick fd may be used to avoid races between
> > > >    the kick fd and vhost-user protocol messages, so in practice back-end
> > > >    implementors may still want to start the virtqueue when the kick fd
> > > >    becomes readable.
> > > >
> > > > 2. Add a clarification to "Migrating back-end state" that the device
> > > >    must be suspended (see _suspended_device_state) when
> > > >    VHOST_USER_SET_DEVICE_STATE_FD is sent and device state is
> > > >    transferred. This is already implicit in "Device state transfer
> > > >    parameters", but it's not obvious when reading the "Migrating
> > > >    back-end state" section.
> > > >
> > > > Stefan
> > >
> >
>

Re: [PATCH] vhost-user.rst: Explicitly allow front-end to write to kick FDs

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 995 bytes --]

On Tue, Apr 21, 2026 at 06:49:32PM +0200, Stefano Garzarella wrote:
> On Tue, 21 Apr 2026 at 18:06, Jorge Moreira <jemoreira@google.com> wrote:
> >
> > > Do you know other backends we should check?
> >
> > CrosVm's: https://github.com/google/crosvm/tree/main/devices/src/virtio/vhost_user_backend
> 
> Which do exactly the same
> https://github.com/google/crosvm/blob/b49e336de56d6353ae06245de0c7fabec3db2d5e/devices/src/virtio/vhost_user_backend/handler.rs#L596
> 
> No?

To round this out, I looked at DPDK, one of the most popular vhost-user
back-end implementations.

The DPDK back-end starts/stops the rings in the vhost-user message
handler rather than upon reading the kick fd:
https://github.com/DPDK/dpdk/blob/main/lib/vhost/vhost_user.c#L3274

I think it would be nice to update the spec since it seems
implementations aren't actually waiting for the kick fd. However, it is
not central to the original discussion, so feel free to skip it if you
want.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH] vhost-user.rst: Explicitly allow front-end to write to kick FDs

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 8676 bytes --]

On Mon, Apr 20, 2026 at 05:57:40PM +0200, Stefano Garzarella wrote:
> On Mon, 20 Apr 2026 at 16:49, Stefano Garzarella <sgarzare@redhat.com> wrote:
> >
> > Thanks for starting the discussion here, let me add also Hanna, German,
> > and Stefan in CC that can help us.
> >
> > On Fri, Apr 10, 2026 at 07:12:05PM -0700, Jorge E. Moreira wrote:
> > >Migration of back-end state happens while the device is suspended (i.e
> > >all vrings are stopped). To resume normal operation on the destination,
> > >the vrings need to be started again with a kick (either a write on the
> > >FD or the VHOST_USER_VRING_KICK in-band message if negotiated). While
> >
> > It's true that in the spec we have:
> >    "Each ring is initialized in a stopped and disabled state. The
> >    back-end must start a ring upon receiving a kick (that is, detecting
> >    that file descriptor is readable) on the descriptor specified by
> >    VHOST_USER_SET_VRING_KICK or receiving the in-band message
> >    VHOST_USER_VRING_KICK if negotiated, and stop a ring upon receiving
> >    VHOST_USER_GET_VRING_BASE."
> >
> > But IMO this applies when a driver is not yet loaded.
> > When we are migrating, the driver could be already loaded. So, in the
> > new device running in the destination, IMO we should consider the ring
> > already started or add some messages to tell to the device: "hey, the
> > device was already started, this is a migration and it's completed".
> >
> > Sending a kick from the frontend, seems more an hack here.
> >
> > That said, for example, in subprojects/libvhost-user/libvhost-user.c
> > IIUC the virtqueue is started when the SET_VRING_KICK is handled by
> > vu_set_vring_kick_exec(), but not sure how compliant it is.
> >
> > >these notifications are typically sent by the driver, it has no reason
> > >to send them in the destination if it already sent them in the source as
> > >the driver is unaware that a migration took place. Therefore it should
> > >be the responsibility of the vhost-user front-end to ensure these vrings
> > >are started. This is particularly necessary for queues where data only
> > >flows from device to driver, such as those used by the vsock and input
> > >devices.
> >
> > Exactly, so IMO we should not use the kick, but maybe add something new
> > or clarify what to do after the migration.
> >
> > For example in the "Migrating back-end state" we have:
> >    "Migrating device state involves transferring the state from one
> >    back-end, called the source, to another back-end, called the
> >    destination. After migration, the destination transparently resumes
> >    operation without requiring the driver to re-initialize the device at
> >    the VIRTIO level."
> >
> > So, IMO we can use the VHOST_USER_SET_DEVICE_STATE_FD channel exactly to
> > inform the new device about the state: "there isn't any state to
> > transfer, but I notify you that the device was already initialized, so
> > the vrings can be started".
> >
> > >
> > >This behavior is already used by some qemu vhost-user front-ends (e.g
> > >vhost-user-blk) and by front-ends implemented on other VMMs(e.g CrosVm).
> >
> > I looked at vhost-user-blk frontend, but I don't see it. I mean I see
> > the code around the comment "/* Kick right away to begin processing
> > requests already in vring */" but that one IIUC was introduced more to
> > fix devices violating specs, so not sure it's a good example to follow:
> >
> > commit 110b9463d5c820120c8311db79f55a64c9d81ebe
> > Author: Yongji Xie <elohimes@gmail.com>
> > Date:   Wed Jun 6 21:24:48 2018 +0800
> >
> >      vhost-user-blk: start vhost when guest kicks
> >
> >      Some old guests (before commit 7a11370e5: "virtio_blk: enable VQs early")
> >      kick virtqueue before setting VIRTIO_CONFIG_S_DRIVER_OK. This violates
> >      the virtio spec. But virtio 1.0 transitional devices support this behaviour.
> >      So we should start vhost when guest kicks in this case.
> >
> >      Signed-off-by: Yongji Xie <xieyongji@baidu.com>
> >      Signed-off-by: Chai Wen <chaiwen@baidu.com>
> >      Signed-off-by: Ni Xun <nixun@baidu.com>
> >      Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
> >      Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
> >      Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> >
> >
> > >Adding it to the vhost-user documentation makes it explicit that this
> > >strategy is permitted and suggest it to vhost-user front-end authors.
> > >Explicitly documenting it is necessary because vring kicks appear
> > >designed to originate in the driver, so having some originate in the
> > >front-end can be counterintuitive and cause developers to waste time
> > >looking for other alternatives or face pushback during code review.
> >
> > As I pointed out in our discussion in
> > https://github.com/rust-vmm/vhost-device/pull/936
> > IMO we should use some in-band messages and not relaying on kicks that
> > should be used only by the driver to notify the device about new
> > available buffers.
> >
> > That said, I agree that we need to clarify in the specifications exactly
> > what the backend and frontend should do after a migration to start
> > vrings if there is no need to exchange a state.
> >
> >
> > Any other opinion?
> >
> > Thanks,
> > Stefano
> >
> > >
> > >Signed-off-by: Jorge E. Moreira <jemoreira@google.com>
> > >---
> > > docs/interop/vhost-user.rst | 5 ++++-
> > > 1 file changed, 4 insertions(+), 1 deletion(-)
> > >
> > >diff --git a/docs/interop/vhost-user.rst b/docs/interop/vhost-user.rst
> > >index 137c9f3669..ad5aba3430 100644
> > >--- a/docs/interop/vhost-user.rst
> > >+++ b/docs/interop/vhost-user.rst
> > >@@ -656,7 +656,10 @@ destination, following the usual protocol for establishing a connection
> > > to a vhost-user back-end: This includes, for example, setting up memory
> > > mappings and kick and call FDs as necessary, negotiating protocol
> > > features, or setting the initial vring base indices (to the same value
> > >-as on the source side, so that operation can resume).
> > >+as on the source side, so that operation can resume). The vhost-user front-end
> > >+may also write to the kick FDs of vrings containing unused buffers or send
> > >+``VHOST_USER_VRING_KICK`` if negotiated to start those vrings in the destination
> > >+since the driver likely already kicked them in the source and won't do it again.
> 
> After discussing this with Hanna, we came to the conclusion that your
> idea of injecting the kick is the least invasive option for now and
> complies with the spec (even though I still don’t think it’s a nice
> thing to do).
> 
> So it’s fine to continue in this direction, but I might add these
> words more in the "Migration" section than here, since we’re talking
> about an optional state migration here. For example after "No further
> update must be done before rings are restarted."
> Or in the "Ring states" section, where we can clarify how to restart a
> ring after a migration. Or in both :-)
> 
> WDYT?
> 
> Then instead of "may" I'd use "should". And I would refer to the fact
> that migration is transparent to the driver, so the front-end should
> kick all initialized vrings to comply with what we described in the
> "Ring states" section.

I don't agree. Here is my thinking about how to solve this:

Device state migration uses the ring state machine to suspend the device
on the source host and start rings on the destination host after loading
state.

Starting rings is not specific to migration, it is covered by the ring
state machine that's also used when pausing and resuming a VM locally,
for example. We need to fix the ring state machine section in the spec
rather than making changes to the migration or device state fd sections.

Given that there are vhost-user back-ends like DPDK that do not rely on
the kick fd (they do not use it when running in poll mode), injecting a
kick cannot be necessary. Adding anything to the spec that requires the
kick fd is at best a no-op and at worst could break those back-ends.

The spec must be updated to say that rings are started by
VHOST_USER_SET_KICK, which is what implementations already do today.

If you want to add a note that front-ends may send a kick after
VHOST_USER_SET_KICK completes to ensure that rings are started according
to the old spec wording, then that is fine, but it should be clear that
kicks are not the primary mechanism for starting rings since it would be
dangerous to rely on that (it breaks poll-mode back-ends).

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH] vhost-user.rst: Explicitly allow front-end to write to kick FDs

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 517 bytes --]

On Mon, Apr 20, 2026 at 05:48:13PM -0700, Jorge Moreira wrote:
> While starting the vrings on SET_VRING_KICK could solve the state
> machine issue, it still won't notify the back-end that buffers are
> ready (the driver won't do this). Non-polling back-ends depend on this
> kick, especially for queues where data flows only from the driver to
> the back-end. Most implementations likely attempt to read from the
> queue only after receiving the kick.

This is an interesting question to clarify in the spec.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH] vhost-user.rst: Explicitly allow front-end to write to kick FDs

[Jorge Moreira]

On Tue, Apr 21, 2026 at 2:12 PM Stefan Hajnoczi <stefanha@redhat.com> wrote:
>
> On Mon, Apr 20, 2026 at 05:48:13PM -0700, Jorge Moreira wrote:
> > While starting the vrings on SET_VRING_KICK could solve the state
> > machine issue, it still won't notify the back-end that buffers are
> > ready (the driver won't do this). Non-polling back-ends depend on this
> > kick, especially for queues where data flows only from the driver to
> > the back-end. Most implementations likely attempt to read from the
> > queue only after receiving the kick.
>
> This is an interesting question to clarify in the spec.
>
> Stefan

This is the question that interests me most, to be honest. I'd rather
have the discussion about when to activate the vrings in a different
thread and keep this one focused on whether the front-end should send
the kick or if the back-end is expected to check if there are "new"
buffers in the vring after restore.

Re: [PATCH] vhost-user.rst: Explicitly allow front-end to write to kick FDs

[Stefano Garzarella]

On Tue, 21 Apr 2026 at 22:52, Stefan Hajnoczi <stefanha@redhat.com> wrote:
>
> On Mon, Apr 20, 2026 at 05:57:40PM +0200, Stefano Garzarella wrote:
> > On Mon, 20 Apr 2026 at 16:49, Stefano Garzarella <sgarzare@redhat.com> wrote:
> > >
> > > Thanks for starting the discussion here, let me add also Hanna, German,
> > > and Stefan in CC that can help us.
> > >
> > > On Fri, Apr 10, 2026 at 07:12:05PM -0700, Jorge E. Moreira wrote:
> > > >Migration of back-end state happens while the device is suspended (i.e
> > > >all vrings are stopped). To resume normal operation on the destination,
> > > >the vrings need to be started again with a kick (either a write on the
> > > >FD or the VHOST_USER_VRING_KICK in-band message if negotiated). While
> > >
> > > It's true that in the spec we have:
> > >    "Each ring is initialized in a stopped and disabled state. The
> > >    back-end must start a ring upon receiving a kick (that is, detecting
> > >    that file descriptor is readable) on the descriptor specified by
> > >    VHOST_USER_SET_VRING_KICK or receiving the in-band message
> > >    VHOST_USER_VRING_KICK if negotiated, and stop a ring upon receiving
> > >    VHOST_USER_GET_VRING_BASE."
> > >
> > > But IMO this applies when a driver is not yet loaded.
> > > When we are migrating, the driver could be already loaded. So, in the
> > > new device running in the destination, IMO we should consider the ring
> > > already started or add some messages to tell to the device: "hey, the
> > > device was already started, this is a migration and it's completed".
> > >
> > > Sending a kick from the frontend, seems more an hack here.
> > >
> > > That said, for example, in subprojects/libvhost-user/libvhost-user.c
> > > IIUC the virtqueue is started when the SET_VRING_KICK is handled by
> > > vu_set_vring_kick_exec(), but not sure how compliant it is.
> > >
> > > >these notifications are typically sent by the driver, it has no reason
> > > >to send them in the destination if it already sent them in the source as
> > > >the driver is unaware that a migration took place. Therefore it should
> > > >be the responsibility of the vhost-user front-end to ensure these vrings
> > > >are started. This is particularly necessary for queues where data only
> > > >flows from device to driver, such as those used by the vsock and input
> > > >devices.
> > >
> > > Exactly, so IMO we should not use the kick, but maybe add something new
> > > or clarify what to do after the migration.
> > >
> > > For example in the "Migrating back-end state" we have:
> > >    "Migrating device state involves transferring the state from one
> > >    back-end, called the source, to another back-end, called the
> > >    destination. After migration, the destination transparently resumes
> > >    operation without requiring the driver to re-initialize the device at
> > >    the VIRTIO level."
> > >
> > > So, IMO we can use the VHOST_USER_SET_DEVICE_STATE_FD channel exactly to
> > > inform the new device about the state: "there isn't any state to
> > > transfer, but I notify you that the device was already initialized, so
> > > the vrings can be started".
> > >
> > > >
> > > >This behavior is already used by some qemu vhost-user front-ends (e.g
> > > >vhost-user-blk) and by front-ends implemented on other VMMs(e.g CrosVm).
> > >
> > > I looked at vhost-user-blk frontend, but I don't see it. I mean I see
> > > the code around the comment "/* Kick right away to begin processing
> > > requests already in vring */" but that one IIUC was introduced more to
> > > fix devices violating specs, so not sure it's a good example to follow:
> > >
> > > commit 110b9463d5c820120c8311db79f55a64c9d81ebe
> > > Author: Yongji Xie <elohimes@gmail.com>
> > > Date:   Wed Jun 6 21:24:48 2018 +0800
> > >
> > >      vhost-user-blk: start vhost when guest kicks
> > >
> > >      Some old guests (before commit 7a11370e5: "virtio_blk: enable VQs early")
> > >      kick virtqueue before setting VIRTIO_CONFIG_S_DRIVER_OK. This violates
> > >      the virtio spec. But virtio 1.0 transitional devices support this behaviour.
> > >      So we should start vhost when guest kicks in this case.
> > >
> > >      Signed-off-by: Yongji Xie <xieyongji@baidu.com>
> > >      Signed-off-by: Chai Wen <chaiwen@baidu.com>
> > >      Signed-off-by: Ni Xun <nixun@baidu.com>
> > >      Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
> > >      Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
> > >      Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> > >
> > >
> > > >Adding it to the vhost-user documentation makes it explicit that this
> > > >strategy is permitted and suggest it to vhost-user front-end authors.
> > > >Explicitly documenting it is necessary because vring kicks appear
> > > >designed to originate in the driver, so having some originate in the
> > > >front-end can be counterintuitive and cause developers to waste time
> > > >looking for other alternatives or face pushback during code review.
> > >
> > > As I pointed out in our discussion in
> > > https://github.com/rust-vmm/vhost-device/pull/936
> > > IMO we should use some in-band messages and not relaying on kicks that
> > > should be used only by the driver to notify the device about new
> > > available buffers.
> > >
> > > That said, I agree that we need to clarify in the specifications exactly
> > > what the backend and frontend should do after a migration to start
> > > vrings if there is no need to exchange a state.
> > >
> > >
> > > Any other opinion?
> > >
> > > Thanks,
> > > Stefano
> > >
> > > >
> > > >Signed-off-by: Jorge E. Moreira <jemoreira@google.com>
> > > >---
> > > > docs/interop/vhost-user.rst | 5 ++++-
> > > > 1 file changed, 4 insertions(+), 1 deletion(-)
> > > >
> > > >diff --git a/docs/interop/vhost-user.rst b/docs/interop/vhost-user.rst
> > > >index 137c9f3669..ad5aba3430 100644
> > > >--- a/docs/interop/vhost-user.rst
> > > >+++ b/docs/interop/vhost-user.rst
> > > >@@ -656,7 +656,10 @@ destination, following the usual protocol for establishing a connection
> > > > to a vhost-user back-end: This includes, for example, setting up memory
> > > > mappings and kick and call FDs as necessary, negotiating protocol
> > > > features, or setting the initial vring base indices (to the same value
> > > >-as on the source side, so that operation can resume).
> > > >+as on the source side, so that operation can resume). The vhost-user front-end
> > > >+may also write to the kick FDs of vrings containing unused buffers or send
> > > >+``VHOST_USER_VRING_KICK`` if negotiated to start those vrings in the destination
> > > >+since the driver likely already kicked them in the source and won't do it again.
> >
> > After discussing this with Hanna, we came to the conclusion that your
> > idea of injecting the kick is the least invasive option for now and
> > complies with the spec (even though I still don’t think it’s a nice
> > thing to do).
> >
> > So it’s fine to continue in this direction, but I might add these
> > words more in the "Migration" section than here, since we’re talking
> > about an optional state migration here. For example after "No further
> > update must be done before rings are restarted."
> > Or in the "Ring states" section, where we can clarify how to restart a
> > ring after a migration. Or in both :-)
> >
> > WDYT?
> >
> > Then instead of "may" I'd use "should". And I would refer to the fact
> > that migration is transparent to the driver, so the front-end should
> > kick all initialized vrings to comply with what we described in the
> > "Ring states" section.
>
> I don't agree. Here is my thinking about how to solve this:
>
> Device state migration uses the ring state machine to suspend the device
> on the source host and start rings on the destination host after loading
> state.
>
> Starting rings is not specific to migration, it is covered by the ring
> state machine that's also used when pausing and resuming a VM locally,
> for example. We need to fix the ring state machine section in the spec
> rather than making changes to the migration or device state fd sections.
>
> Given that there are vhost-user back-ends like DPDK that do not rely on
> the kick fd (they do not use it when running in poll mode), injecting a
> kick cannot be necessary. Adding anything to the spec that requires the
> kick fd is at best a no-op and at worst could break those back-ends.
>
> The spec must be updated to say that rings are started by
> VHOST_USER_SET_KICK, which is what implementations already do today.

If we update the spec with this, I agree with you. I also think it's
the right thing to do. Even though I would have preferred a specific
message to get things started, which would require a new feature,
etc., I agree that’s too much.

My reasoning was based on the fact that we always needed to inject a
kick to start the rings, which is essentially what we currently state
in the specification. Not sure why all the implementation fortunately
didn't follow that, maybe because they got inspired by libvhost-user.

BTW, I think the main issue to clarify is what the device should do
when the vrings are configured, but the driver has already been
initialized (which is usually the case after migration).

If the driver hasn't been initialized, everything is clear.

>
> If you want to add a note that front-ends may send a kick after
> VHOST_USER_SET_KICK completes to ensure that rings are started according
> to the old spec wording, then that is fine, but it should be clear that
> kicks are not the primary mechanism for starting rings since it would be
> dangerous to rely on that (it breaks poll-mode back-ends).

Agree, after clarifying VHOST_USER_SET_KICK.

Thanks,
Stefano

Re: [PATCH] vhost-user.rst: Explicitly allow front-end to write to kick FDs

[Stefano Garzarella]

On Wed, 22 Apr 2026 at 03:16, Jorge Moreira <jemoreira@google.com> wrote:
>
> On Tue, Apr 21, 2026 at 2:12 PM Stefan Hajnoczi <stefanha@redhat.com> wrote:
> >
> > On Mon, Apr 20, 2026 at 05:48:13PM -0700, Jorge Moreira wrote:
> > > While starting the vrings on SET_VRING_KICK could solve the state
> > > machine issue, it still won't notify the back-end that buffers are
> > > ready (the driver won't do this). Non-polling back-ends depend on this
> > > kick, especially for queues where data flows only from the driver to
> > > the back-end. Most implementations likely attempt to read from the
> > > queue only after receiving the kick.
> >
> > This is an interesting question to clarify in the spec.

Yep, which is in part related to what I wrote in the other reply:
"I think the main issue to clarify is what the device should do
when the vrings are configured, but the driver has already been
initialized (which is usually the case after migration)."

> >
> > Stefan
>
> This is the question that interests me most, to be honest. I'd rather
> have the discussion about when to activate the vrings in a different
> thread and keep this one focused on whether the front-end should send
> the kick or if the back-end is expected to check if there are "new"
> buffers in the vring after restore.
>

IMO we don't need anything from the VMM. When the device receives
SET_VRING_KICK, it can check if the vring already contains buffers
(and this is the part we might need to clarify) and wake-up the other
threads (or always wake-ups them, as crosvm does IIUC, and let them
perform this check).
After sending the SET_VRING_KICK message to the device, the VMM has
the exact same knowledge of the vring state as the device, therefore,
it's still unclear to me why we need to inject that kick.

Stefano

Re: [PATCH] vhost-user.rst: Explicitly allow front-end to write to kick FDs

[Jorge Moreira]

On Wed, Apr 22, 2026 at 1:32 AM Stefano Garzarella <sgarzare@redhat.com> wrote:
>
> On Wed, 22 Apr 2026 at 03:16, Jorge Moreira <jemoreira@google.com> wrote:
> >
> > On Tue, Apr 21, 2026 at 2:12 PM Stefan Hajnoczi <stefanha@redhat.com> wrote:
> > >
> > > On Mon, Apr 20, 2026 at 05:48:13PM -0700, Jorge Moreira wrote:
> > > > While starting the vrings on SET_VRING_KICK could solve the state
> > > > machine issue, it still won't notify the back-end that buffers are
> > > > ready (the driver won't do this). Non-polling back-ends depend on this
> > > > kick, especially for queues where data flows only from the driver to
> > > > the back-end. Most implementations likely attempt to read from the
> > > > queue only after receiving the kick.
> > >
> > > This is an interesting question to clarify in the spec.
>
> Yep, which is in part related to what I wrote in the other reply:
> "I think the main issue to clarify is what the device should do
> when the vrings are configured, but the driver has already been
> initialized (which is usually the case after migration)."
>
> > >
> > > Stefan
> >
> > This is the question that interests me most, to be honest. I'd rather
> > have the discussion about when to activate the vrings in a different
> > thread and keep this one focused on whether the front-end should send
> > the kick or if the back-end is expected to check if there are "new"
> > buffers in the vring after restore.
> >
>
> IMO we don't need anything from the VMM. When the device receives
> SET_VRING_KICK, it can check if the vring already contains buffers
> (and this is the part we might need to clarify) and wake-up the other
> threads (or always wake-ups them, as crosvm does IIUC, and let them
> perform this check).
> After sending the SET_VRING_KICK message to the device, the VMM has
> the exact same knowledge of the vring state as the device, therefore,
> it's still unclear to me why we need to inject that kick.
>
> Stefano
>

Is it possible to activate a vring after it has been deactivated with
VHOST_USER_GET_VRING_BASE? If yes, does the front-end need to send the
kick file descriptor again with VHOST_USER_SET_VRING_KICK to
reactivate it?

Re: [PATCH] vhost-user.rst: Explicitly allow front-end to write to kick FDs

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 3633 bytes --]

On Wed, Apr 22, 2026 at 12:20:52PM -0700, Jorge Moreira wrote:
> On Wed, Apr 22, 2026 at 1:32 AM Stefano Garzarella <sgarzare@redhat.com> wrote:
> >
> > On Wed, 22 Apr 2026 at 03:16, Jorge Moreira <jemoreira@google.com> wrote:
> > >
> > > On Tue, Apr 21, 2026 at 2:12 PM Stefan Hajnoczi <stefanha@redhat.com> wrote:
> > > >
> > > > On Mon, Apr 20, 2026 at 05:48:13PM -0700, Jorge Moreira wrote:
> > > > > While starting the vrings on SET_VRING_KICK could solve the state
> > > > > machine issue, it still won't notify the back-end that buffers are
> > > > > ready (the driver won't do this). Non-polling back-ends depend on this
> > > > > kick, especially for queues where data flows only from the driver to
> > > > > the back-end. Most implementations likely attempt to read from the
> > > > > queue only after receiving the kick.
> > > >
> > > > This is an interesting question to clarify in the spec.
> >
> > Yep, which is in part related to what I wrote in the other reply:
> > "I think the main issue to clarify is what the device should do
> > when the vrings are configured, but the driver has already been
> > initialized (which is usually the case after migration)."
> >
> > > >
> > > > Stefan
> > >
> > > This is the question that interests me most, to be honest. I'd rather
> > > have the discussion about when to activate the vrings in a different
> > > thread and keep this one focused on whether the front-end should send
> > > the kick or if the back-end is expected to check if there are "new"
> > > buffers in the vring after restore.
> > >
> >
> > IMO we don't need anything from the VMM. When the device receives
> > SET_VRING_KICK, it can check if the vring already contains buffers
> > (and this is the part we might need to clarify) and wake-up the other
> > threads (or always wake-ups them, as crosvm does IIUC, and let them
> > perform this check).
> > After sending the SET_VRING_KICK message to the device, the VMM has
> > the exact same knowledge of the vring state as the device, therefore,
> > it's still unclear to me why we need to inject that kick.
> >
> > Stefano
> >
> 
> Is it possible to activate a vring after it has been deactivated with
> VHOST_USER_GET_VRING_BASE? If yes, does the front-end need to send the
> kick file descriptor again with VHOST_USER_SET_VRING_KICK to
> reactivate it?

Hi Jorge and Stefano,
Yes, VHOST_USER_GET_VRING_BASE -> VHOST_USER_SET_VRING_KICK occurs when
a VM is paused and then resumed.

You can stress test this by driving I/O using iperf (virtio-net) or fio
(virtio-blk) inside the guest and sending 'stop'/'cont' commands to
QEMU's monitor.

Here is QEMU's code for starting (including re-starting) rings:
https://gitlab.com/qemu-project/qemu/-/blob/master/hw/virtio/vhost.c?ref_type=heads#L1341

QEMU does not inject a kick. The back-end must check the rings itself.

I'm not sure that all vhost-user back-ends actually check the rings. I
think back-ends should do it, but we should also update the spec with an
front-end implementation note recommending injecting a kick after
VHOST_USER_SET_VRING_KICK completes in order to maximize compatibility
with implementations that follow the current spec wording. And at the
same time I think the spec should also be changed to say that
VHOST_USER_SET_VRING_KICK starts the ring and back-ends SHOULD check the
vring upon processing the message.

That seems like it would clean up the issues without introducing
compatibility issues or making existing implementations non-compliant
with the updated spec.

What do you think?

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH] vhost-user.rst: Explicitly allow front-end to write to kick FDs

[Jorge Moreira]

On Mon, Apr 27, 2026 at 3:45 PM Stefan Hajnoczi <stefanha@redhat.com> wrote:
>
> On Wed, Apr 22, 2026 at 12:20:52PM -0700, Jorge Moreira wrote:
> > On Wed, Apr 22, 2026 at 1:32 AM Stefano Garzarella <sgarzare@redhat.com> wrote:
> > >
> > > On Wed, 22 Apr 2026 at 03:16, Jorge Moreira <jemoreira@google.com> wrote:
> > > >
> > > > On Tue, Apr 21, 2026 at 2:12 PM Stefan Hajnoczi <stefanha@redhat.com> wrote:
> > > > >
> > > > > On Mon, Apr 20, 2026 at 05:48:13PM -0700, Jorge Moreira wrote:
> > > > > > While starting the vrings on SET_VRING_KICK could solve the state
> > > > > > machine issue, it still won't notify the back-end that buffers are
> > > > > > ready (the driver won't do this). Non-polling back-ends depend on this
> > > > > > kick, especially for queues where data flows only from the driver to
> > > > > > the back-end. Most implementations likely attempt to read from the
> > > > > > queue only after receiving the kick.
> > > > >
> > > > > This is an interesting question to clarify in the spec.
> > >
> > > Yep, which is in part related to what I wrote in the other reply:
> > > "I think the main issue to clarify is what the device should do
> > > when the vrings are configured, but the driver has already been
> > > initialized (which is usually the case after migration)."
> > >
> > > > >
> > > > > Stefan
> > > >
> > > > This is the question that interests me most, to be honest. I'd rather
> > > > have the discussion about when to activate the vrings in a different
> > > > thread and keep this one focused on whether the front-end should send
> > > > the kick or if the back-end is expected to check if there are "new"
> > > > buffers in the vring after restore.
> > > >
> > >
> > > IMO we don't need anything from the VMM. When the device receives
> > > SET_VRING_KICK, it can check if the vring already contains buffers
> > > (and this is the part we might need to clarify) and wake-up the other
> > > threads (or always wake-ups them, as crosvm does IIUC, and let them
> > > perform this check).
> > > After sending the SET_VRING_KICK message to the device, the VMM has
> > > the exact same knowledge of the vring state as the device, therefore,
> > > it's still unclear to me why we need to inject that kick.
> > >
> > > Stefano
> > >
> >
> > Is it possible to activate a vring after it has been deactivated with
> > VHOST_USER_GET_VRING_BASE? If yes, does the front-end need to send the
> > kick file descriptor again with VHOST_USER_SET_VRING_KICK to
> > reactivate it?
>
> Hi Jorge and Stefano,
> Yes, VHOST_USER_GET_VRING_BASE -> VHOST_USER_SET_VRING_KICK occurs when
> a VM is paused and then resumed.
>
> You can stress test this by driving I/O using iperf (virtio-net) or fio
> (virtio-blk) inside the guest and sending 'stop'/'cont' commands to
> QEMU's monitor.
>
> Here is QEMU's code for starting (including re-starting) rings:
> https://gitlab.com/qemu-project/qemu/-/blob/master/hw/virtio/vhost.c?ref_type=heads#L1341
>
> QEMU does not inject a kick. The back-end must check the rings itself.
>
> I'm not sure that all vhost-user back-ends actually check the rings. I
> think back-ends should do it, but we should also update the spec with an
> front-end implementation note recommending injecting a kick after
> VHOST_USER_SET_VRING_KICK completes in order to maximize compatibility
> with implementations that follow the current spec wording. And at the
> same time I think the spec should also be changed to say that
> VHOST_USER_SET_VRING_KICK starts the ring and back-ends SHOULD check the
> vring upon processing the message.
>
> That seems like it would clean up the issues without introducing
> compatibility issues or making existing implementations non-compliant
> with the updated spec.
>
> What do you think?

Sounds good to me

>
> Stefan

Re: [PATCH] vhost-user.rst: Explicitly allow front-end to write to kick FDs

[Michael S. Tsirkin]

On Mon, Apr 27, 2026 at 06:45:45PM -0400, Stefan Hajnoczi wrote:
> On Wed, Apr 22, 2026 at 12:20:52PM -0700, Jorge Moreira wrote:
> > On Wed, Apr 22, 2026 at 1:32 AM Stefano Garzarella <sgarzare@redhat.com> wrote:
> > >
> > > On Wed, 22 Apr 2026 at 03:16, Jorge Moreira <jemoreira@google.com> wrote:
> > > >
> > > > On Tue, Apr 21, 2026 at 2:12 PM Stefan Hajnoczi <stefanha@redhat.com> wrote:
> > > > >
> > > > > On Mon, Apr 20, 2026 at 05:48:13PM -0700, Jorge Moreira wrote:
> > > > > > While starting the vrings on SET_VRING_KICK could solve the state
> > > > > > machine issue, it still won't notify the back-end that buffers are
> > > > > > ready (the driver won't do this). Non-polling back-ends depend on this
> > > > > > kick, especially for queues where data flows only from the driver to
> > > > > > the back-end. Most implementations likely attempt to read from the
> > > > > > queue only after receiving the kick.
> > > > >
> > > > > This is an interesting question to clarify in the spec.
> > >
> > > Yep, which is in part related to what I wrote in the other reply:
> > > "I think the main issue to clarify is what the device should do
> > > when the vrings are configured, but the driver has already been
> > > initialized (which is usually the case after migration)."
> > >
> > > > >
> > > > > Stefan
> > > >
> > > > This is the question that interests me most, to be honest. I'd rather
> > > > have the discussion about when to activate the vrings in a different
> > > > thread and keep this one focused on whether the front-end should send
> > > > the kick or if the back-end is expected to check if there are "new"
> > > > buffers in the vring after restore.
> > > >
> > >
> > > IMO we don't need anything from the VMM. When the device receives
> > > SET_VRING_KICK, it can check if the vring already contains buffers
> > > (and this is the part we might need to clarify) and wake-up the other
> > > threads (or always wake-ups them, as crosvm does IIUC, and let them
> > > perform this check).
> > > After sending the SET_VRING_KICK message to the device, the VMM has
> > > the exact same knowledge of the vring state as the device, therefore,
> > > it's still unclear to me why we need to inject that kick.
> > >
> > > Stefano
> > >
> > 
> > Is it possible to activate a vring after it has been deactivated with
> > VHOST_USER_GET_VRING_BASE? If yes, does the front-end need to send the
> > kick file descriptor again with VHOST_USER_SET_VRING_KICK to
> > reactivate it?
> 
> Hi Jorge and Stefano,
> Yes, VHOST_USER_GET_VRING_BASE -> VHOST_USER_SET_VRING_KICK occurs when
> a VM is paused and then resumed.
> 
> You can stress test this by driving I/O using iperf (virtio-net) or fio
> (virtio-blk) inside the guest and sending 'stop'/'cont' commands to
> QEMU's monitor.
> 
> Here is QEMU's code for starting (including re-starting) rings:
> https://gitlab.com/qemu-project/qemu/-/blob/master/hw/virtio/vhost.c?ref_type=heads#L1341
> 
> QEMU does not inject a kick. The back-end must check the rings itself.


I would worry that there could be buggy drivers that expect buffers not
consumed until the 1st kick.

A safer thing to do would be for source to record the fact there
was a kick after reset and for destination to then inject/check.


> I'm not sure that all vhost-user back-ends actually check the rings. I
> think back-ends should do it, but we should also update the spec with an
> front-end implementation note recommending injecting a kick after
> VHOST_USER_SET_VRING_KICK completes in order to maximize compatibility
> with implementations that follow the current spec wording. And at the
> same time I think the spec should also be changed to say that
> VHOST_USER_SET_VRING_KICK starts the ring and back-ends SHOULD check the
> vring upon processing the message.
> 
> That seems like it would clean up the issues without introducing
> compatibility issues or making existing implementations non-compliant
> with the updated spec.
> 
> What do you think?
> 
> Stefan

Re: [PATCH] vhost-user.rst: Explicitly allow front-end to write to kick FDs

[Stefano Garzarella]

On Mon, Apr 27, 2026 at 03:48:44PM -0700, Jorge Moreira wrote:
>On Mon, Apr 27, 2026 at 3:45 PM Stefan Hajnoczi <stefanha@redhat.com> wrote:
>>
>> On Wed, Apr 22, 2026 at 12:20:52PM -0700, Jorge Moreira wrote:
>> > On Wed, Apr 22, 2026 at 1:32 AM Stefano Garzarella <sgarzare@redhat.com> wrote:
>> > >
>> > > On Wed, 22 Apr 2026 at 03:16, Jorge Moreira <jemoreira@google.com> wrote:
>> > > >
>> > > > On Tue, Apr 21, 2026 at 2:12 PM Stefan Hajnoczi <stefanha@redhat.com> wrote:
>> > > > >
>> > > > > On Mon, Apr 20, 2026 at 05:48:13PM -0700, Jorge Moreira wrote:
>> > > > > > While starting the vrings on SET_VRING_KICK could solve the state
>> > > > > > machine issue, it still won't notify the back-end that buffers are
>> > > > > > ready (the driver won't do this). Non-polling back-ends depend on this
>> > > > > > kick, especially for queues where data flows only from the driver to
>> > > > > > the back-end. Most implementations likely attempt to read from the
>> > > > > > queue only after receiving the kick.
>> > > > >
>> > > > > This is an interesting question to clarify in the spec.
>> > >
>> > > Yep, which is in part related to what I wrote in the other reply:
>> > > "I think the main issue to clarify is what the device should do
>> > > when the vrings are configured, but the driver has already been
>> > > initialized (which is usually the case after migration)."
>> > >
>> > > > >
>> > > > > Stefan
>> > > >
>> > > > This is the question that interests me most, to be honest. I'd rather
>> > > > have the discussion about when to activate the vrings in a different
>> > > > thread and keep this one focused on whether the front-end should send
>> > > > the kick or if the back-end is expected to check if there are "new"
>> > > > buffers in the vring after restore.
>> > > >
>> > >
>> > > IMO we don't need anything from the VMM. When the device receives
>> > > SET_VRING_KICK, it can check if the vring already contains buffers
>> > > (and this is the part we might need to clarify) and wake-up the other
>> > > threads (or always wake-ups them, as crosvm does IIUC, and let them
>> > > perform this check).
>> > > After sending the SET_VRING_KICK message to the device, the VMM has
>> > > the exact same knowledge of the vring state as the device, therefore,
>> > > it's still unclear to me why we need to inject that kick.
>> > >
>> > > Stefano
>> > >
>> >
>> > Is it possible to activate a vring after it has been deactivated with
>> > VHOST_USER_GET_VRING_BASE? If yes, does the front-end need to send the
>> > kick file descriptor again with VHOST_USER_SET_VRING_KICK to
>> > reactivate it?
>>
>> Hi Jorge and Stefano,
>> Yes, VHOST_USER_GET_VRING_BASE -> VHOST_USER_SET_VRING_KICK occurs when
>> a VM is paused and then resumed.
>>
>> You can stress test this by driving I/O using iperf (virtio-net) or fio
>> (virtio-blk) inside the guest and sending 'stop'/'cont' commands to
>> QEMU's monitor.
>>
>> Here is QEMU's code for starting (including re-starting) rings:
>> https://gitlab.com/qemu-project/qemu/-/blob/master/hw/virtio/vhost.c?ref_type=heads#L1341
>>
>> QEMU does not inject a kick. The back-end must check the rings itself.
>>
>> I'm not sure that all vhost-user back-ends actually check the rings. I
>> think back-ends should do it, but we should also update the spec with an
>> front-end implementation note recommending injecting a kick after
>> VHOST_USER_SET_VRING_KICK completes in order to maximize compatibility

Okay, but since, as we've seen, no frontend currently implements this, 
we need to make it clear that a backend shouldn't necessarily expect the 
kick injected from every frontend, but should support it in some way 
since some of them can inject it.

IMHO especially new backend implementations shouldn't rely on the kick 
injection.

So, to summarize:
- the frontend should also send a kick to restart the queues
- the backend should restart the queues after VHOST_USER_SET_VRING_KICK, 
   but it might also receive a kick

>> with implementations that follow the current spec wording. And at the
>> same time I think the spec should also be changed to say that
>> VHOST_USER_SET_VRING_KICK starts the ring and back-ends SHOULD check the
>> vring upon processing the message.

Yep, I think we are aligned.

>>
>> That seems like it would clean up the issues without introducing
>> compatibility issues or making existing implementations non-compliant
>> with the updated spec.
>>
>> What do you think?

LGTM!

>
>Sounds good to me
>

@Jorge do you want to propose this change?

Thanks,
Stefano

Re: [PATCH] vhost-user.rst: Explicitly allow front-end to write to kick FDs

[Jorge Moreira]

On Tue, Apr 28, 2026 at 7:33 AM Stefano Garzarella <sgarzare@redhat.com> wrote:
>
> On Mon, Apr 27, 2026 at 03:48:44PM -0700, Jorge Moreira wrote:
> >On Mon, Apr 27, 2026 at 3:45 PM Stefan Hajnoczi <stefanha@redhat.com> wrote:
> >>
> >> On Wed, Apr 22, 2026 at 12:20:52PM -0700, Jorge Moreira wrote:
> >> > On Wed, Apr 22, 2026 at 1:32 AM Stefano Garzarella <sgarzare@redhat.com> wrote:
> >> > >
> >> > > On Wed, 22 Apr 2026 at 03:16, Jorge Moreira <jemoreira@google.com> wrote:
> >> > > >
> >> > > > On Tue, Apr 21, 2026 at 2:12 PM Stefan Hajnoczi <stefanha@redhat.com> wrote:
> >> > > > >
> >> > > > > On Mon, Apr 20, 2026 at 05:48:13PM -0700, Jorge Moreira wrote:
> >> > > > > > While starting the vrings on SET_VRING_KICK could solve the state
> >> > > > > > machine issue, it still won't notify the back-end that buffers are
> >> > > > > > ready (the driver won't do this). Non-polling back-ends depend on this
> >> > > > > > kick, especially for queues where data flows only from the driver to
> >> > > > > > the back-end. Most implementations likely attempt to read from the
> >> > > > > > queue only after receiving the kick.
> >> > > > >
> >> > > > > This is an interesting question to clarify in the spec.
> >> > >
> >> > > Yep, which is in part related to what I wrote in the other reply:
> >> > > "I think the main issue to clarify is what the device should do
> >> > > when the vrings are configured, but the driver has already been
> >> > > initialized (which is usually the case after migration)."
> >> > >
> >> > > > >
> >> > > > > Stefan
> >> > > >
> >> > > > This is the question that interests me most, to be honest. I'd rather
> >> > > > have the discussion about when to activate the vrings in a different
> >> > > > thread and keep this one focused on whether the front-end should send
> >> > > > the kick or if the back-end is expected to check if there are "new"
> >> > > > buffers in the vring after restore.
> >> > > >
> >> > >
> >> > > IMO we don't need anything from the VMM. When the device receives
> >> > > SET_VRING_KICK, it can check if the vring already contains buffers
> >> > > (and this is the part we might need to clarify) and wake-up the other
> >> > > threads (or always wake-ups them, as crosvm does IIUC, and let them
> >> > > perform this check).
> >> > > After sending the SET_VRING_KICK message to the device, the VMM has
> >> > > the exact same knowledge of the vring state as the device, therefore,
> >> > > it's still unclear to me why we need to inject that kick.
> >> > >
> >> > > Stefano
> >> > >
> >> >
> >> > Is it possible to activate a vring after it has been deactivated with
> >> > VHOST_USER_GET_VRING_BASE? If yes, does the front-end need to send the
> >> > kick file descriptor again with VHOST_USER_SET_VRING_KICK to
> >> > reactivate it?
> >>
> >> Hi Jorge and Stefano,
> >> Yes, VHOST_USER_GET_VRING_BASE -> VHOST_USER_SET_VRING_KICK occurs when
> >> a VM is paused and then resumed.
> >>
> >> You can stress test this by driving I/O using iperf (virtio-net) or fio
> >> (virtio-blk) inside the guest and sending 'stop'/'cont' commands to
> >> QEMU's monitor.
> >>
> >> Here is QEMU's code for starting (including re-starting) rings:
> >> https://gitlab.com/qemu-project/qemu/-/blob/master/hw/virtio/vhost.c?ref_type=heads#L1341
> >>
> >> QEMU does not inject a kick. The back-end must check the rings itself.
> >>
> >> I'm not sure that all vhost-user back-ends actually check the rings. I
> >> think back-ends should do it, but we should also update the spec with an
> >> front-end implementation note recommending injecting a kick after
> >> VHOST_USER_SET_VRING_KICK completes in order to maximize compatibility
>
> Okay, but since, as we've seen, no frontend currently implements this,
> we need to make it clear that a backend shouldn't necessarily expect the
> kick injected from every frontend, but should support it in some way
> since some of them can inject it.
>
> IMHO especially new backend implementations shouldn't rely on the kick
> injection.
>
> So, to summarize:
> - the frontend should also send a kick to restart the queues
> - the backend should restart the queues after VHOST_USER_SET_VRING_KICK,
>    but it might also receive a kick
>
> >> with implementations that follow the current spec wording. And at the
> >> same time I think the spec should also be changed to say that
> >> VHOST_USER_SET_VRING_KICK starts the ring and back-ends SHOULD check the
> >> vring upon processing the message.
>
> Yep, I think we are aligned.
>
> >>
> >> That seems like it would clean up the issues without introducing
> >> compatibility issues or making existing implementations non-compliant
> >> with the updated spec.
> >>
> >> What do you think?
>
> LGTM!
>
> >
> >Sounds good to me
> >
>
> @Jorge do you want to propose this change?

You have a much better idea of where and how this should be written,
it will save us a few rounds of review if one of you makes that
change.

>
> Thanks,
> Stefano
>

Re: [PATCH] vhost-user.rst: Explicitly allow front-end to write to kick FDs

[Stefano Garzarella]

On Tue, Apr 28, 2026 at 10:19:30AM -0700, Jorge Moreira wrote:
>On Tue, Apr 28, 2026 at 7:33 AM Stefano Garzarella <sgarzare@redhat.com> wrote:
>>
>> On Mon, Apr 27, 2026 at 03:48:44PM -0700, Jorge Moreira wrote:
>> >On Mon, Apr 27, 2026 at 3:45 PM Stefan Hajnoczi <stefanha@redhat.com> wrote:
>> >>
>> >> On Wed, Apr 22, 2026 at 12:20:52PM -0700, Jorge Moreira wrote:
>> >> > On Wed, Apr 22, 2026 at 1:32 AM Stefano Garzarella <sgarzare@redhat.com> wrote:
>> >> > >
>> >> > > On Wed, 22 Apr 2026 at 03:16, Jorge Moreira <jemoreira@google.com> wrote:
>> >> > > >
>> >> > > > On Tue, Apr 21, 2026 at 2:12 PM Stefan Hajnoczi <stefanha@redhat.com> wrote:
>> >> > > > >
>> >> > > > > On Mon, Apr 20, 2026 at 05:48:13PM -0700, Jorge Moreira wrote:
>> >> > > > > > While starting the vrings on SET_VRING_KICK could solve the state
>> >> > > > > > machine issue, it still won't notify the back-end that buffers are
>> >> > > > > > ready (the driver won't do this). Non-polling back-ends depend on this
>> >> > > > > > kick, especially for queues where data flows only from the driver to
>> >> > > > > > the back-end. Most implementations likely attempt to read from the
>> >> > > > > > queue only after receiving the kick.
>> >> > > > >
>> >> > > > > This is an interesting question to clarify in the spec.
>> >> > >
>> >> > > Yep, which is in part related to what I wrote in the other reply:
>> >> > > "I think the main issue to clarify is what the device should do
>> >> > > when the vrings are configured, but the driver has already been
>> >> > > initialized (which is usually the case after migration)."
>> >> > >
>> >> > > > >
>> >> > > > > Stefan
>> >> > > >
>> >> > > > This is the question that interests me most, to be honest. I'd rather
>> >> > > > have the discussion about when to activate the vrings in a different
>> >> > > > thread and keep this one focused on whether the front-end should send
>> >> > > > the kick or if the back-end is expected to check if there are "new"
>> >> > > > buffers in the vring after restore.
>> >> > > >
>> >> > >
>> >> > > IMO we don't need anything from the VMM. When the device receives
>> >> > > SET_VRING_KICK, it can check if the vring already contains buffers
>> >> > > (and this is the part we might need to clarify) and wake-up the other
>> >> > > threads (or always wake-ups them, as crosvm does IIUC, and let them
>> >> > > perform this check).
>> >> > > After sending the SET_VRING_KICK message to the device, the VMM has
>> >> > > the exact same knowledge of the vring state as the device, therefore,
>> >> > > it's still unclear to me why we need to inject that kick.
>> >> > >
>> >> > > Stefano
>> >> > >
>> >> >
>> >> > Is it possible to activate a vring after it has been deactivated with
>> >> > VHOST_USER_GET_VRING_BASE? If yes, does the front-end need to send the
>> >> > kick file descriptor again with VHOST_USER_SET_VRING_KICK to
>> >> > reactivate it?
>> >>
>> >> Hi Jorge and Stefano,
>> >> Yes, VHOST_USER_GET_VRING_BASE -> VHOST_USER_SET_VRING_KICK occurs when
>> >> a VM is paused and then resumed.
>> >>
>> >> You can stress test this by driving I/O using iperf (virtio-net) or fio
>> >> (virtio-blk) inside the guest and sending 'stop'/'cont' commands to
>> >> QEMU's monitor.
>> >>
>> >> Here is QEMU's code for starting (including re-starting) rings:
>> >> https://gitlab.com/qemu-project/qemu/-/blob/master/hw/virtio/vhost.c?ref_type=heads#L1341
>> >>
>> >> QEMU does not inject a kick. The back-end must check the rings itself.
>> >>
>> >> I'm not sure that all vhost-user back-ends actually check the rings. I
>> >> think back-ends should do it, but we should also update the spec with an
>> >> front-end implementation note recommending injecting a kick after
>> >> VHOST_USER_SET_VRING_KICK completes in order to maximize compatibility
>>
>> Okay, but since, as we've seen, no frontend currently implements this,
>> we need to make it clear that a backend shouldn't necessarily expect the
>> kick injected from every frontend, but should support it in some way
>> since some of them can inject it.
>>
>> IMHO especially new backend implementations shouldn't rely on the kick
>> injection.
>>
>> So, to summarize:
>> - the frontend should also send a kick to restart the queues
>> - the backend should restart the queues after VHOST_USER_SET_VRING_KICK,
>>    but it might also receive a kick
>>
>> >> with implementations that follow the current spec wording. And at the
>> >> same time I think the spec should also be changed to say that
>> >> VHOST_USER_SET_VRING_KICK starts the ring and back-ends SHOULD check the
>> >> vring upon processing the message.
>>
>> Yep, I think we are aligned.
>>
>> >>
>> >> That seems like it would clean up the issues without introducing
>> >> compatibility issues or making existing implementations non-compliant
>> >> with the updated spec.
>> >>
>> >> What do you think?
>>
>> LGTM!
>>
>> >
>> >Sounds good to me
>> >
>>
>> @Jorge do you want to propose this change?
>
>You have a much better idea of where and how this should be written,
>it will save us a few rounds of review if one of you makes that
>change.

Okay, I can try, but next week.

Thanks,
Stefano

Re: [PATCH] vhost-user.rst: Explicitly allow front-end to write to kick FDs

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 4637 bytes --]

On Tue, Apr 28, 2026 at 02:59:58AM -0400, Michael S. Tsirkin wrote:
> On Mon, Apr 27, 2026 at 06:45:45PM -0400, Stefan Hajnoczi wrote:
> > On Wed, Apr 22, 2026 at 12:20:52PM -0700, Jorge Moreira wrote:
> > > On Wed, Apr 22, 2026 at 1:32 AM Stefano Garzarella <sgarzare@redhat.com> wrote:
> > > >
> > > > On Wed, 22 Apr 2026 at 03:16, Jorge Moreira <jemoreira@google.com> wrote:
> > > > >
> > > > > On Tue, Apr 21, 2026 at 2:12 PM Stefan Hajnoczi <stefanha@redhat.com> wrote:
> > > > > >
> > > > > > On Mon, Apr 20, 2026 at 05:48:13PM -0700, Jorge Moreira wrote:
> > > > > > > While starting the vrings on SET_VRING_KICK could solve the state
> > > > > > > machine issue, it still won't notify the back-end that buffers are
> > > > > > > ready (the driver won't do this). Non-polling back-ends depend on this
> > > > > > > kick, especially for queues where data flows only from the driver to
> > > > > > > the back-end. Most implementations likely attempt to read from the
> > > > > > > queue only after receiving the kick.
> > > > > >
> > > > > > This is an interesting question to clarify in the spec.
> > > >
> > > > Yep, which is in part related to what I wrote in the other reply:
> > > > "I think the main issue to clarify is what the device should do
> > > > when the vrings are configured, but the driver has already been
> > > > initialized (which is usually the case after migration)."
> > > >
> > > > > >
> > > > > > Stefan
> > > > >
> > > > > This is the question that interests me most, to be honest. I'd rather
> > > > > have the discussion about when to activate the vrings in a different
> > > > > thread and keep this one focused on whether the front-end should send
> > > > > the kick or if the back-end is expected to check if there are "new"
> > > > > buffers in the vring after restore.
> > > > >
> > > >
> > > > IMO we don't need anything from the VMM. When the device receives
> > > > SET_VRING_KICK, it can check if the vring already contains buffers
> > > > (and this is the part we might need to clarify) and wake-up the other
> > > > threads (or always wake-ups them, as crosvm does IIUC, and let them
> > > > perform this check).
> > > > After sending the SET_VRING_KICK message to the device, the VMM has
> > > > the exact same knowledge of the vring state as the device, therefore,
> > > > it's still unclear to me why we need to inject that kick.
> > > >
> > > > Stefano
> > > >
> > > 
> > > Is it possible to activate a vring after it has been deactivated with
> > > VHOST_USER_GET_VRING_BASE? If yes, does the front-end need to send the
> > > kick file descriptor again with VHOST_USER_SET_VRING_KICK to
> > > reactivate it?
> > 
> > Hi Jorge and Stefano,
> > Yes, VHOST_USER_GET_VRING_BASE -> VHOST_USER_SET_VRING_KICK occurs when
> > a VM is paused and then resumed.
> > 
> > You can stress test this by driving I/O using iperf (virtio-net) or fio
> > (virtio-blk) inside the guest and sending 'stop'/'cont' commands to
> > QEMU's monitor.
> > 
> > Here is QEMU's code for starting (including re-starting) rings:
> > https://gitlab.com/qemu-project/qemu/-/blob/master/hw/virtio/vhost.c?ref_type=heads#L1341
> > 
> > QEMU does not inject a kick. The back-end must check the rings itself.
> 
> 
> I would worry that there could be buggy drivers that expect buffers not
> consumed until the 1st kick.

This describes the behavior today. I'm not sure whether you are
commenting on the paragraph I wrote below suggesting how to change
things or whether you are commenting on the current state?

> A safer thing to do would be for source to record the fact there
> was a kick after reset and for destination to then inject/check.

Can you rephrase that, it's unclear to me what "source" and
"destination" mean?

Thanks,
Stefan

> > I'm not sure that all vhost-user back-ends actually check the rings. I
> > think back-ends should do it, but we should also update the spec with an
> > front-end implementation note recommending injecting a kick after
> > VHOST_USER_SET_VRING_KICK completes in order to maximize compatibility
> > with implementations that follow the current spec wording. And at the
> > same time I think the spec should also be changed to say that
> > VHOST_USER_SET_VRING_KICK starts the ring and back-ends SHOULD check the
> > vring upon processing the message.
> > 
> > That seems like it would clean up the issues without introducing
> > compatibility issues or making existing implementations non-compliant
> > with the updated spec.
> > 
> > What do you think?
> > 
> > Stefan
> 
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH] vhost-user.rst: Explicitly allow front-end to write to kick FDs

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 5055 bytes --]

On Tue, Apr 28, 2026 at 04:33:23PM +0200, Stefano Garzarella wrote:
> On Mon, Apr 27, 2026 at 03:48:44PM -0700, Jorge Moreira wrote:
> > On Mon, Apr 27, 2026 at 3:45 PM Stefan Hajnoczi <stefanha@redhat.com> wrote:
> > > 
> > > On Wed, Apr 22, 2026 at 12:20:52PM -0700, Jorge Moreira wrote:
> > > > On Wed, Apr 22, 2026 at 1:32 AM Stefano Garzarella <sgarzare@redhat.com> wrote:
> > > > >
> > > > > On Wed, 22 Apr 2026 at 03:16, Jorge Moreira <jemoreira@google.com> wrote:
> > > > > >
> > > > > > On Tue, Apr 21, 2026 at 2:12 PM Stefan Hajnoczi <stefanha@redhat.com> wrote:
> > > > > > >
> > > > > > > On Mon, Apr 20, 2026 at 05:48:13PM -0700, Jorge Moreira wrote:
> > > > > > > > While starting the vrings on SET_VRING_KICK could solve the state
> > > > > > > > machine issue, it still won't notify the back-end that buffers are
> > > > > > > > ready (the driver won't do this). Non-polling back-ends depend on this
> > > > > > > > kick, especially for queues where data flows only from the driver to
> > > > > > > > the back-end. Most implementations likely attempt to read from the
> > > > > > > > queue only after receiving the kick.
> > > > > > >
> > > > > > > This is an interesting question to clarify in the spec.
> > > > >
> > > > > Yep, which is in part related to what I wrote in the other reply:
> > > > > "I think the main issue to clarify is what the device should do
> > > > > when the vrings are configured, but the driver has already been
> > > > > initialized (which is usually the case after migration)."
> > > > >
> > > > > > >
> > > > > > > Stefan
> > > > > >
> > > > > > This is the question that interests me most, to be honest. I'd rather
> > > > > > have the discussion about when to activate the vrings in a different
> > > > > > thread and keep this one focused on whether the front-end should send
> > > > > > the kick or if the back-end is expected to check if there are "new"
> > > > > > buffers in the vring after restore.
> > > > > >
> > > > >
> > > > > IMO we don't need anything from the VMM. When the device receives
> > > > > SET_VRING_KICK, it can check if the vring already contains buffers
> > > > > (and this is the part we might need to clarify) and wake-up the other
> > > > > threads (or always wake-ups them, as crosvm does IIUC, and let them
> > > > > perform this check).
> > > > > After sending the SET_VRING_KICK message to the device, the VMM has
> > > > > the exact same knowledge of the vring state as the device, therefore,
> > > > > it's still unclear to me why we need to inject that kick.
> > > > >
> > > > > Stefano
> > > > >
> > > >
> > > > Is it possible to activate a vring after it has been deactivated with
> > > > VHOST_USER_GET_VRING_BASE? If yes, does the front-end need to send the
> > > > kick file descriptor again with VHOST_USER_SET_VRING_KICK to
> > > > reactivate it?
> > > 
> > > Hi Jorge and Stefano,
> > > Yes, VHOST_USER_GET_VRING_BASE -> VHOST_USER_SET_VRING_KICK occurs when
> > > a VM is paused and then resumed.
> > > 
> > > You can stress test this by driving I/O using iperf (virtio-net) or fio
> > > (virtio-blk) inside the guest and sending 'stop'/'cont' commands to
> > > QEMU's monitor.
> > > 
> > > Here is QEMU's code for starting (including re-starting) rings:
> > > https://gitlab.com/qemu-project/qemu/-/blob/master/hw/virtio/vhost.c?ref_type=heads#L1341
> > > 
> > > QEMU does not inject a kick. The back-end must check the rings itself.
> > > 
> > > I'm not sure that all vhost-user back-ends actually check the rings. I
> > > think back-ends should do it, but we should also update the spec with an
> > > front-end implementation note recommending injecting a kick after
> > > VHOST_USER_SET_VRING_KICK completes in order to maximize compatibility
> 
> Okay, but since, as we've seen, no frontend currently implements this, we
> need to make it clear that a backend shouldn't necessarily expect the kick
> injected from every frontend, but should support it in some way since some
> of them can inject it.
> 
> IMHO especially new backend implementations shouldn't rely on the kick
> injection.
> 
> So, to summarize:
> - the frontend should also send a kick to restart the queues
> - the backend should restart the queues after VHOST_USER_SET_VRING_KICK,
> but it might also receive a kick

Yes, agreed.

> 
> > > with implementations that follow the current spec wording. And at the
> > > same time I think the spec should also be changed to say that
> > > VHOST_USER_SET_VRING_KICK starts the ring and back-ends SHOULD check the
> > > vring upon processing the message.
> 
> Yep, I think we are aligned.
> 
> > > 
> > > That seems like it would clean up the issues without introducing
> > > compatibility issues or making existing implementations non-compliant
> > > with the updated spec.
> > > 
> > > What do you think?
> 
> LGTM!
> 
> > 
> > Sounds good to me
> > 
> 
> @Jorge do you want to propose this change?
> 
> Thanks,
> Stefano
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH] vhost-user.rst: Explicitly allow front-end to write to kick FDs

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 5682 bytes --]

On Tue, Apr 28, 2026 at 10:19:30AM -0700, Jorge Moreira wrote:
> On Tue, Apr 28, 2026 at 7:33 AM Stefano Garzarella <sgarzare@redhat.com> wrote:
> >
> > On Mon, Apr 27, 2026 at 03:48:44PM -0700, Jorge Moreira wrote:
> > >On Mon, Apr 27, 2026 at 3:45 PM Stefan Hajnoczi <stefanha@redhat.com> wrote:
> > >>
> > >> On Wed, Apr 22, 2026 at 12:20:52PM -0700, Jorge Moreira wrote:
> > >> > On Wed, Apr 22, 2026 at 1:32 AM Stefano Garzarella <sgarzare@redhat.com> wrote:
> > >> > >
> > >> > > On Wed, 22 Apr 2026 at 03:16, Jorge Moreira <jemoreira@google.com> wrote:
> > >> > > >
> > >> > > > On Tue, Apr 21, 2026 at 2:12 PM Stefan Hajnoczi <stefanha@redhat.com> wrote:
> > >> > > > >
> > >> > > > > On Mon, Apr 20, 2026 at 05:48:13PM -0700, Jorge Moreira wrote:
> > >> > > > > > While starting the vrings on SET_VRING_KICK could solve the state
> > >> > > > > > machine issue, it still won't notify the back-end that buffers are
> > >> > > > > > ready (the driver won't do this). Non-polling back-ends depend on this
> > >> > > > > > kick, especially for queues where data flows only from the driver to
> > >> > > > > > the back-end. Most implementations likely attempt to read from the
> > >> > > > > > queue only after receiving the kick.
> > >> > > > >
> > >> > > > > This is an interesting question to clarify in the spec.
> > >> > >
> > >> > > Yep, which is in part related to what I wrote in the other reply:
> > >> > > "I think the main issue to clarify is what the device should do
> > >> > > when the vrings are configured, but the driver has already been
> > >> > > initialized (which is usually the case after migration)."
> > >> > >
> > >> > > > >
> > >> > > > > Stefan
> > >> > > >
> > >> > > > This is the question that interests me most, to be honest. I'd rather
> > >> > > > have the discussion about when to activate the vrings in a different
> > >> > > > thread and keep this one focused on whether the front-end should send
> > >> > > > the kick or if the back-end is expected to check if there are "new"
> > >> > > > buffers in the vring after restore.
> > >> > > >
> > >> > >
> > >> > > IMO we don't need anything from the VMM. When the device receives
> > >> > > SET_VRING_KICK, it can check if the vring already contains buffers
> > >> > > (and this is the part we might need to clarify) and wake-up the other
> > >> > > threads (or always wake-ups them, as crosvm does IIUC, and let them
> > >> > > perform this check).
> > >> > > After sending the SET_VRING_KICK message to the device, the VMM has
> > >> > > the exact same knowledge of the vring state as the device, therefore,
> > >> > > it's still unclear to me why we need to inject that kick.
> > >> > >
> > >> > > Stefano
> > >> > >
> > >> >
> > >> > Is it possible to activate a vring after it has been deactivated with
> > >> > VHOST_USER_GET_VRING_BASE? If yes, does the front-end need to send the
> > >> > kick file descriptor again with VHOST_USER_SET_VRING_KICK to
> > >> > reactivate it?
> > >>
> > >> Hi Jorge and Stefano,
> > >> Yes, VHOST_USER_GET_VRING_BASE -> VHOST_USER_SET_VRING_KICK occurs when
> > >> a VM is paused and then resumed.
> > >>
> > >> You can stress test this by driving I/O using iperf (virtio-net) or fio
> > >> (virtio-blk) inside the guest and sending 'stop'/'cont' commands to
> > >> QEMU's monitor.
> > >>
> > >> Here is QEMU's code for starting (including re-starting) rings:
> > >> https://gitlab.com/qemu-project/qemu/-/blob/master/hw/virtio/vhost.c?ref_type=heads#L1341
> > >>
> > >> QEMU does not inject a kick. The back-end must check the rings itself.
> > >>
> > >> I'm not sure that all vhost-user back-ends actually check the rings. I
> > >> think back-ends should do it, but we should also update the spec with an
> > >> front-end implementation note recommending injecting a kick after
> > >> VHOST_USER_SET_VRING_KICK completes in order to maximize compatibility
> >
> > Okay, but since, as we've seen, no frontend currently implements this,
> > we need to make it clear that a backend shouldn't necessarily expect the
> > kick injected from every frontend, but should support it in some way
> > since some of them can inject it.
> >
> > IMHO especially new backend implementations shouldn't rely on the kick
> > injection.
> >
> > So, to summarize:
> > - the frontend should also send a kick to restart the queues
> > - the backend should restart the queues after VHOST_USER_SET_VRING_KICK,
> >    but it might also receive a kick
> >
> > >> with implementations that follow the current spec wording. And at the
> > >> same time I think the spec should also be changed to say that
> > >> VHOST_USER_SET_VRING_KICK starts the ring and back-ends SHOULD check the
> > >> vring upon processing the message.
> >
> > Yep, I think we are aligned.
> >
> > >>
> > >> That seems like it would clean up the issues without introducing
> > >> compatibility issues or making existing implementations non-compliant
> > >> with the updated spec.
> > >>
> > >> What do you think?
> >
> > LGTM!
> >
> > >
> > >Sounds good to me
> > >
> >
> > @Jorge do you want to propose this change?
> 
> You have a much better idea of where and how this should be written,
> it will save us a few rounds of review if one of you makes that
> change.

I can give it a shot and will CC you.

Also, I wanted to apologize for not being very forthcoming in this
discussion. I had an intuition about this issue but lacked the time to
research and double-check the details. As a result, I didn't respond
to all your points in detail. Sorry if it was frustrating.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]