[PATCH v2 0/3] vhost-user.rst: clarify when rings are started

[PATCH v2 0/3] vhost-user.rst: clarify when rings are started

[Stefan Hajnoczi]

Jorge Moreira <jemoreira@google.com> pointed out that the ring state machine is
underspecified. This patch series updates the spec to clarify how the ring
state machine works, makes libvhost-user spec-compliant, and improves the
compatibility of QEMU's vhost-user front-end.

Stefan Hajnoczi (3):
  vhost-user.rst: clarify when rings are started
  libvhost-user: look for available vq buffers upon SET_VRING_KICK
  vhost-user: inject kick after SET_VRING_KICK

 docs/interop/vhost-user.rst               | 26 +++++++++++++++++------
 hw/virtio/vhost-user.c                    | 24 ++++++++++++++++++++-
 subprojects/libvhost-user/libvhost-user.c | 10 ++++-----
 3 files changed, 48 insertions(+), 12 deletions(-)

-- 
2.54.0

[PATCH v2 1/3] vhost-user.rst: clarify when rings are started

[Stefan Hajnoczi]

Jorge Moreira <jemoreira@google.com> pointed out that the ring state
machine is underspecified. In the discussion that followed, we
discovered that the spec says one thing and implementations do something
else. This patch updates the spec to reflect how things are actually
implemented across widely-used front-ends and back-ends including QEMU,
crosvm, rust-vmm, and DPDK. Do this while taking care not to make any
other existing implementations non-compliant by changing the sepc.

The spec says rings are started when a kick is received but the
implementations actually start rings when VHOST_USER_SET_VRING_KICK is
received.

Reconcile this as follows:
- Clarify that a ring can be stopped and then started again. The
  back-end must resume processing available requests when the ring is
  restarted.
- Update the spec to say rings are started when
  VHOST_USER_SET_VRING_KICK is received.
- Ensure compatibility by saying front-ends SHOULD inject a kick in case
  the back-end strictly implemented the old spec.
- Avoid future back-end dependencies on injected kicks by saying that
  back-ends SHOULD NOT expect a kick to start rings.

This way implementors have clarity on how things work while still
allowing compatibility for existing implementations.

Reported-by: Jorge Moreira <jemoreira@google.com>
Cc: "Michael S . Tsirkin" <mst@redhat.com>
Cc: Stefano Garzarella <sgarzare@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 docs/interop/vhost-user.rst | 26 ++++++++++++++++++++------
 1 file changed, 20 insertions(+), 6 deletions(-)

diff --git a/docs/interop/vhost-user.rst b/docs/interop/vhost-user.rst
index 137c9f3669..6ced95e080 100644
--- a/docs/interop/vhost-user.rst
+++ b/docs/interop/vhost-user.rst
@@ -462,12 +462,26 @@ Rings have two independent states: started/stopped, and enabled/disabled.
 * started and enabled: The back-end must process the ring normally, i.e.
   process all requests and execute them.
 
-Each ring is initialized in a stopped and disabled state.  The back-end
-must start a ring upon receiving a kick (that is, detecting that file
-descriptor is readable) on the descriptor specified by
-``VHOST_USER_SET_VRING_KICK`` or receiving the in-band message
-``VHOST_USER_VRING_KICK`` if negotiated, and stop a ring upon receiving
-``VHOST_USER_GET_VRING_BASE``.
+Each ring is initialized in a stopped and disabled state.  Rings are started
+with ``VHOST_USER_SET_VRING_KICK`` (or ``VHOST_USER_VRING_KICK`` if
+``VHOST_USER_PROTOCOL_F_INBAND_NOTIFICATIONS`` is negotiated) and stopped with
+``VHOST_USER_GET_VRING_BASE``.  A stopped ring enters the started state again
+with ``VHOST_USER_SET_VRING_KICK`` (or ``VHOST_USER_VRING_KICK`` if
+``VHOST_USER_PROTOCOL_F_INBAND_NOTIFICATIONS`` is negotiated) and the back-end
+resumes processing requests.
+
+Note that previous versions of this specification stated that rings start when
+the back-end receives a kick (that is, detecting that file descriptor is
+readable) on the descriptor specified by ``VHOST_USER_SET_VRING_KICK`` or
+receiving the in-band message ``VHOST_USER_VRING_KICK`` if negotiated.
+Widely-used front-ends and back-ends did not implement this behavior and it
+complicates poll mode back-ends that do not rely on the kick file descriptor.
+
+For compatibility with back-ends that implemented the start on kick behavior,
+front-ends SHOULD inject a kick after ``VHOST_USER_SET_VRING_KICK``.  This
+ensures that the back-end processes any available requests in the ring.
+Back-ends SHOULD NOT rely on receiving a kick after
+``VHOST_USER_SET_VRING_KICK``.
 
 Rings can be enabled or disabled by ``VHOST_USER_SET_VRING_ENABLE``.
 
-- 
2.54.0

[PATCH v2 2/3] libvhost-user: look for available vq buffers upon SET_VRING_KICK

[Stefan Hajnoczi]

When a vring is started the back-end must look for available vq buffers
and process them. This scenario can happen if the back-end is stopped
with unprocessed available buffers and then started again.

The inflight I/O tracking code already did this, but it should also be
done when inflight I/O tracking is not enabled. Move the code.

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 subprojects/libvhost-user/libvhost-user.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/subprojects/libvhost-user/libvhost-user.c b/subprojects/libvhost-user/libvhost-user.c
index 9c630c2170..2e286ea6d9 100644
--- a/subprojects/libvhost-user/libvhost-user.c
+++ b/subprojects/libvhost-user/libvhost-user.c
@@ -1390,11 +1390,6 @@ vu_check_queue_inflights(VuDev *dev, VuVirtq *vq)
         vq->counter = vq->resubmit_list[0].counter + 1;
     }
 
-    /* in case of I/O hang after reconnecting */
-    if (eventfd_write(vq->kick_fd, 1)) {
-        return -1;
-    }
-
     return 0;
 }
 
@@ -1436,6 +1431,11 @@ vu_set_vring_kick_exec(VuDev *dev, VhostUserMsg *vmsg)
         vu_panic(dev, "Failed to check inflights for vq: %d\n", index);
     }
 
+    /* Inject a kick to look for available vq buffers */
+    if (eventfd_write(dev->vq[index].kick_fd, 1)) {
+        return -1;
+    }
+
     return false;
 }
 
-- 
2.54.0

[PATCH v2 3/3] vhost-user: inject kick after SET_VRING_KICK

[Stefan Hajnoczi]

The vhost-user specification was updated to say that front-ends should
inject a kick after SET_VRING_KICK in case the back-end implements the
old spec wording which said vrings start when a kick is received. Do
this in QEMU's front-end.

An example scenario where this behavior helps: the back-end fails to
check if the vring has available buffers when SET_VRING_KICK is received
and the front-end stopped and then restarted the vring. In the case the
back-end may not notice the available buffers unless the front-end
injects a kick.

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 hw/virtio/vhost-user.c | 24 +++++++++++++++++++++++-
 1 file changed, 23 insertions(+), 1 deletion(-)

diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c
index a8907cca74..7ef4105473 100644
--- a/hw/virtio/vhost-user.c
+++ b/hw/virtio/vhost-user.c
@@ -1365,7 +1365,29 @@ static int vhost_set_vring_file(struct vhost_dev *dev,
 static int vhost_user_set_vring_kick(struct vhost_dev *dev,
                                      struct vhost_vring_file *file)
 {
-    return vhost_set_vring_file(dev, VHOST_USER_SET_VRING_KICK, file);
+    int ret = vhost_set_vring_file(dev, VHOST_USER_SET_VRING_KICK, file);
+    if (ret < 0) {
+        return ret;
+    }
+
+    /*
+     * Inject a kick in case the back-end only starts vring processing upon
+     * receiving a kick. The spec suggests this to improve compatibility.
+     */
+    if (file->fd != -1) {
+        uint64_t val = 1;
+        ssize_t nwritten;
+
+        do {
+            nwritten = write(file->fd, &val, sizeof(val));
+        } while (nwritten < 0 && errno == EINTR);
+
+        if (nwritten < 0 && errno != EAGAIN /* back-end can already read */) {
+            return -errno;
+        }
+    }
+
+    return 0;
 }
 
 static int vhost_user_set_vring_call(struct vhost_dev *dev,
-- 
2.54.0

Re: [PATCH v2 2/3] libvhost-user: look for available vq buffers upon SET_VRING_KICK

[Stefano Garzarella]

On Tue, May 19, 2026 at 03:15:47PM -0400, Stefan Hajnoczi wrote:
>When a vring is started the back-end must look for available vq buffers
>and process them. This scenario can happen if the back-end is stopped
>with unprocessed available buffers and then started again.
>
>The inflight I/O tracking code already did this, but it should also be
>done when inflight I/O tracking is not enabled. Move the code.
>
>Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
>---
> subprojects/libvhost-user/libvhost-user.c | 10 +++++-----
> 1 file changed, 5 insertions(+), 5 deletions(-)
>
>diff --git a/subprojects/libvhost-user/libvhost-user.c b/subprojects/libvhost-user/libvhost-user.c
>index 9c630c2170..2e286ea6d9 100644
>--- a/subprojects/libvhost-user/libvhost-user.c
>+++ b/subprojects/libvhost-user/libvhost-user.c
>@@ -1390,11 +1390,6 @@ vu_check_queue_inflights(VuDev *dev, VuVirtq *vq)
>         vq->counter = vq->resubmit_list[0].counter + 1;
>     }
>
>-    /* in case of I/O hang after reconnecting */
>-    if (eventfd_write(vq->kick_fd, 1)) {
>-        return -1;
>-    }
>-
>     return 0;
> }
>
>@@ -1436,6 +1431,11 @@ vu_set_vring_kick_exec(VuDev *dev, VhostUserMsg *vmsg)
>         vu_panic(dev, "Failed to check inflights for vq: %d\n", index);
>     }
>
>+    /* Inject a kick to look for available vq buffers */
>+    if (eventfd_write(dev->vq[index].kick_fd, 1)) {

Should we check if kick_fd is not -1 like we already do in other places 
in this function? (mainly when `nofd` is true)

>+        return -1;

This function returns boolean, -1 is `true` and IIUC means that we ask 
for a reply which is not the case here. We should fix it.

Thanks,
Stefano

>+    }
>+

>     return false;
> }
>
>-- 
>2.54.0
>

Re: [PATCH v2 1/3] vhost-user.rst: clarify when rings are started

[Stefano Garzarella]

On Tue, May 19, 2026 at 03:15:46PM -0400, Stefan Hajnoczi wrote:
>Jorge Moreira <jemoreira@google.com> pointed out that the ring state
>machine is underspecified. In the discussion that followed, we
>discovered that the spec says one thing and implementations do something
>else. This patch updates the spec to reflect how things are actually
>implemented across widely-used front-ends and back-ends including QEMU,
>crosvm, rust-vmm, and DPDK. Do this while taking care not to make any
>other existing implementations non-compliant by changing the sepc.

s/sepc/spec

>
>The spec says rings are started when a kick is received but the
>implementations actually start rings when VHOST_USER_SET_VRING_KICK is
>received.
>
>Reconcile this as follows:
>- Clarify that a ring can be stopped and then started again. The
>  back-end must resume processing available requests when the ring is
>  restarted.
>- Update the spec to say rings are started when
>  VHOST_USER_SET_VRING_KICK is received.
>- Ensure compatibility by saying front-ends SHOULD inject a kick in case
>  the back-end strictly implemented the old spec.
>- Avoid future back-end dependencies on injected kicks by saying that
>  back-ends SHOULD NOT expect a kick to start rings.
>
>This way implementors have clarity on how things work while still
>allowing compatibility for existing implementations.
>
>Reported-by: Jorge Moreira <jemoreira@google.com>
>Cc: "Michael S . Tsirkin" <mst@redhat.com>
>Cc: Stefano Garzarella <sgarzare@redhat.com>
>Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
>---
> docs/interop/vhost-user.rst | 26 ++++++++++++++++++++------
> 1 file changed, 20 insertions(+), 6 deletions(-)

Reviewed-by: Stefano Garzarella <sgarzare@redhat.com>

Re: [PATCH v2 3/3] vhost-user: inject kick after SET_VRING_KICK

[Stefano Garzarella]

On Tue, May 19, 2026 at 03:15:48PM -0400, Stefan Hajnoczi wrote:
>The vhost-user specification was updated to say that front-ends should
>inject a kick after SET_VRING_KICK in case the back-end implements the
>old spec wording which said vrings start when a kick is received. Do
>this in QEMU's front-end.
>
>An example scenario where this behavior helps: the back-end fails to
>check if the vring has available buffers when SET_VRING_KICK is received
>and the front-end stopped and then restarted the vring. In the case the
>back-end may not notice the available buffers unless the front-end
>injects a kick.
>
>Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
>---
> hw/virtio/vhost-user.c | 24 +++++++++++++++++++++++-
> 1 file changed, 23 insertions(+), 1 deletion(-)

Reviewed-by: Stefano Garzarella <sgarzare@redhat.com>

Re: [PATCH v2 2/3] libvhost-user: look for available vq buffers upon SET_VRING_KICK

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1985 bytes --]

On Wed, Jun 03, 2026 at 11:03:18AM +0200, Stefano Garzarella wrote:
> On Tue, May 19, 2026 at 03:15:47PM -0400, Stefan Hajnoczi wrote:
> > When a vring is started the back-end must look for available vq buffers
> > and process them. This scenario can happen if the back-end is stopped
> > with unprocessed available buffers and then started again.
> > 
> > The inflight I/O tracking code already did this, but it should also be
> > done when inflight I/O tracking is not enabled. Move the code.
> > 
> > Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> > ---
> > subprojects/libvhost-user/libvhost-user.c | 10 +++++-----
> > 1 file changed, 5 insertions(+), 5 deletions(-)
> > 
> > diff --git a/subprojects/libvhost-user/libvhost-user.c b/subprojects/libvhost-user/libvhost-user.c
> > index 9c630c2170..2e286ea6d9 100644
> > --- a/subprojects/libvhost-user/libvhost-user.c
> > +++ b/subprojects/libvhost-user/libvhost-user.c
> > @@ -1390,11 +1390,6 @@ vu_check_queue_inflights(VuDev *dev, VuVirtq *vq)
> >         vq->counter = vq->resubmit_list[0].counter + 1;
> >     }
> > 
> > -    /* in case of I/O hang after reconnecting */
> > -    if (eventfd_write(vq->kick_fd, 1)) {
> > -        return -1;
> > -    }
> > -
> >     return 0;
> > }
> > 
> > @@ -1436,6 +1431,11 @@ vu_set_vring_kick_exec(VuDev *dev, VhostUserMsg *vmsg)
> >         vu_panic(dev, "Failed to check inflights for vq: %d\n", index);
> >     }
> > 
> > +    /* Inject a kick to look for available vq buffers */
> > +    if (eventfd_write(dev->vq[index].kick_fd, 1)) {
> 
> Should we check if kick_fd is not -1 like we already do in other places in
> this function? (mainly when `nofd` is true)
>
> > +        return -1;
> 
> This function returns boolean, -1 is `true` and IIUC means that we ask for a
> reply which is not the case here. We should fix it.

Thanks for catch these things. I'm going to redo this patch. Not sure
what I was thinking :).

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH v2 1/3] vhost-user.rst: clarify when rings are started

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 697 bytes --]

On Wed, Jun 03, 2026 at 11:04:42AM +0200, Stefano Garzarella wrote:
> On Tue, May 19, 2026 at 03:15:46PM -0400, Stefan Hajnoczi wrote:
> > Jorge Moreira <jemoreira@google.com> pointed out that the ring state
> > machine is underspecified. In the discussion that followed, we
> > discovered that the spec says one thing and implementations do something
> > else. This patch updates the spec to reflect how things are actually
> > implemented across widely-used front-ends and back-ends including QEMU,
> > crosvm, rust-vmm, and DPDK. Do this while taking care not to make any
> > other existing implementations non-compliant by changing the sepc.
> 
> s/sepc/spec

Will fix.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]