[PATCH v3 0/3] Add queue ready message to VDUSE

[PATCH v3 0/3] Add queue ready message to VDUSE

[Eugenio Pérez]

This series introduces a new VDUSE message for VDUSE userland instance
to detect when a VirtQueue (VQ) is enabled, replacing the polling.

VirtIO net devices' dataplane is started after the control virtqueue so
QEMU can apply the configuration in the destination of a Live Migration.
Without this feature, the VDUSE instance must poll the VQs to check when
(and if) a VQ has been enabled.

This series also implements VDUSE feature flags allowing the VDUSE
devices to opt-in to the VQ ready message.  Devices that opt-in to this
feature will receive explicit notifications when a VQ is ready. Devices
that do not set this flag remain unaffected, ensuring backward
compatibility without indefinitely incrementing API versions.

The VDUSE features is a 64 bit bitmap for simplicity, the same way as
vhost and vhost-net started.  It can be extended as a flexible array of
bits when we reach so many features, but it seems unlikely at this
point.

Error cases tested:
* Call VDUSE_GET_FEATURES without get the API VERSION (so API == 0) and
  with API VERSION set to 1 with VDUSE_SET_API_VERSION (-EINVAL returned
  from VDUSE_GET_FEATURES ioctl).
* Try to create a device with config->features different than 0 without
  fetch the api version (so API == 0) and with API_VERSION set to 1
  (-EINVAL returned from VDUSE_CREATE_DEV ioctl).
* Test regular initialization of single queue devices with V2 with and
  without VDUSE_F_QUEUE_READY set in device->config.
* Test expected behavior when VDUSE userland instance returns
  VDUSE_REQ_RESULT_FAILED from VDUSE_SET_VQ_READY message.
* Repeat all the tests with multiqueue devices, by reverting
  56e71885b0349 ("vduse: Temporarily fail if control queue feature requested").

This series depends on
https://lore.kernel.org/lkml/20260119143306.1818855-1-eperezma@redhat.com/

v3:
* Remove API_VERSION bump to 2
* Add comment about struct vduse_dev_config:vduse_features is only valid
  if VDUSE_GET_FEATURES success.

v2:
* Fix comment of vduse_dev_request.vq_ready
* Set vq_ready before sending the message to the VDUSE userland
  instance, avoiding the need for SMP sync after receiving the message.
* Return -EINVAL if control ioctl called with version < 2, so userland
  visible reply is kept (Jason).

Eugenio Pérez (3):
  vduse: store control device pointer
  vduse: add VDUSE_GET_FEATURES ioctl
  vduse: add F_QUEUE_READY feature

 drivers/vdpa/vdpa_user/vduse_dev.c | 41 +++++++++++++++++++++++++++---
 include/uapi/linux/vduse.h         | 25 +++++++++++++++++-
 2 files changed, 61 insertions(+), 5 deletions(-)

-- 
2.53.0

[PATCH v3 1/3] vduse: store control device pointer

[Eugenio Pérez]

This helps log the errors in next patches.  The alternative is to
perform a linear search for it with class_find_device_by_devt(class, devt),
as device_destroy do for cleaning.

Signed-off-by: Eugenio Pérez <eperezma@redhat.com>
---
 drivers/vdpa/vdpa_user/vduse_dev.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/drivers/vdpa/vdpa_user/vduse_dev.c b/drivers/vdpa/vdpa_user/vduse_dev.c
index 405d59610f76..d1da7c15d98b 100644
--- a/drivers/vdpa/vdpa_user/vduse_dev.c
+++ b/drivers/vdpa/vdpa_user/vduse_dev.c
@@ -164,6 +164,7 @@ static DEFINE_IDR(vduse_idr);
 
 static dev_t vduse_major;
 static struct cdev vduse_ctrl_cdev;
+static const struct device *vduse_ctrl_dev;
 static struct cdev vduse_cdev;
 static struct workqueue_struct *vduse_irq_wq;
 static struct workqueue_struct *vduse_irq_bound_wq;
@@ -2396,7 +2397,6 @@ static void vduse_mgmtdev_exit(void)
 static int vduse_init(void)
 {
 	int ret;
-	struct device *dev;
 
 	ret = class_register(&vduse_class);
 	if (ret)
@@ -2413,9 +2413,9 @@ static int vduse_init(void)
 	if (ret)
 		goto err_ctrl_cdev;
 
-	dev = device_create(&vduse_class, NULL, vduse_major, NULL, "control");
-	if (IS_ERR(dev)) {
-		ret = PTR_ERR(dev);
+	vduse_ctrl_dev = device_create(&vduse_class, NULL, vduse_major, NULL, "control");
+	if (IS_ERR(vduse_ctrl_dev)) {
+		ret = PTR_ERR(vduse_ctrl_dev);
 		goto err_device;
 	}
 
-- 
2.53.0

[PATCH v3 2/3] vduse: add VDUSE_GET_FEATURES ioctl

[Eugenio Pérez]

Add an ioctl to allow VDUSE instances to query the available features
supported by the kernel module.

Signed-off-by: Eugenio Pérez <eperezma@redhat.com>
---
A simple u64 bitmap is used for feature flags. While a flexible array
could support indefinite expansion, 64 bits is sufficient for the
foreseeable future and simplifies the implementation.

Also, dev_dbg is used for logging rather than dev_err as these can be
triggered from userspace.
---
v3:
* Remove check for API_VERSION < 2
* Add comment about struct vduse_dev_config:vduse_features is only valid
  if VDUSE_GET_FEATURES success.

v2:
* return -EINVAL if ioctl called with version < 2, so userland visible
  reply is kept (Jason).
---
 drivers/vdpa/vdpa_user/vduse_dev.c | 7 +++++++
 include/uapi/linux/vduse.h         | 7 ++++++-
 2 files changed, 13 insertions(+), 1 deletion(-)

diff --git a/drivers/vdpa/vdpa_user/vduse_dev.c b/drivers/vdpa/vdpa_user/vduse_dev.c
index d1da7c15d98b..17e0358d3a68 100644
--- a/drivers/vdpa/vdpa_user/vduse_dev.c
+++ b/drivers/vdpa/vdpa_user/vduse_dev.c
@@ -52,6 +52,9 @@
 
 #define IRQ_UNBOUND -1
 
+/* Supported VDUSE features */
+static const uint64_t vduse_features;
+
 /*
  * VDUSE instance have not asked the vduse API version, so assume 0.
  *
@@ -2207,6 +2210,10 @@ static long vduse_ioctl(struct file *file, unsigned int cmd,
 		ret = vduse_destroy_dev(name);
 		break;
 	}
+	case VDUSE_GET_FEATURES:
+		ret = put_user(vduse_features, (u64 __user *)argp);
+		break;
+
 	default:
 		ret = -EINVAL;
 		break;
diff --git a/include/uapi/linux/vduse.h b/include/uapi/linux/vduse.h
index 361eea511c21..e9b5f32a452d 100644
--- a/include/uapi/linux/vduse.h
+++ b/include/uapi/linux/vduse.h
@@ -33,6 +33,7 @@
  * @vq_align: the allocation alignment of virtqueue's metadata
  * @ngroups: number of vq groups that VDUSE device declares
  * @nas: number of address spaces that VDUSE device declares
+ * @vduse_features: VDUSE features
  * @reserved: for future use, needs to be initialized to zero
  * @config_size: the size of the configuration space
  * @config: the buffer of the configuration space
@@ -49,7 +50,8 @@ struct vduse_dev_config {
 	__u32 vq_align;
 	__u32 ngroups; /* if VDUSE_API_VERSION >= 1 */
 	__u32 nas; /* if VDUSE_API_VERSION >= 1 */
-	__u32 reserved[11];
+	__u64 vduse_features; /* if VDUSE_GET_FEATURES is not EINVAL */
+	__u32 reserved[9];
 	__u32 config_size;
 	__u8 config[];
 };
@@ -63,6 +65,9 @@ struct vduse_dev_config {
  */
 #define VDUSE_DESTROY_DEV	_IOW(VDUSE_BASE, 0x03, char[VDUSE_NAME_MAX])
 
+/* Get the VDUSE supported features */
+#define VDUSE_GET_FEATURES	_IOR(VDUSE_BASE, 0x04, __u64)
+
 /* The ioctls for VDUSE device (/dev/vduse/$NAME) */
 
 /**
-- 
2.53.0

Re: [PATCH v3 2/3] vduse: add VDUSE_GET_FEATURES ioctl

[Jason Wang]

On Wed, Mar 11, 2026 at 3:08 AM Eugenio Pérez <eperezma@redhat.com> wrote:
>
> Add an ioctl to allow VDUSE instances to query the available features
> supported by the kernel module.
>
> Signed-off-by: Eugenio Pérez <eperezma@redhat.com>
> ---
> A simple u64 bitmap is used for feature flags. While a flexible array
> could support indefinite expansion, 64 bits is sufficient for the
> foreseeable future and simplifies the implementation.
>
> Also, dev_dbg is used for logging rather than dev_err as these can be
> triggered from userspace.
> ---
> v3:
> * Remove check for API_VERSION < 2
> * Add comment about struct vduse_dev_config:vduse_features is only valid
>   if VDUSE_GET_FEATURES success.
>
> v2:
> * return -EINVAL if ioctl called with version < 2, so userland visible
>   reply is kept (Jason).
> ---
>  drivers/vdpa/vdpa_user/vduse_dev.c | 7 +++++++
>  include/uapi/linux/vduse.h         | 7 ++++++-
>  2 files changed, 13 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/vdpa/vdpa_user/vduse_dev.c b/drivers/vdpa/vdpa_user/vduse_dev.c
> index d1da7c15d98b..17e0358d3a68 100644
> --- a/drivers/vdpa/vdpa_user/vduse_dev.c
> +++ b/drivers/vdpa/vdpa_user/vduse_dev.c
> @@ -52,6 +52,9 @@
>
>  #define IRQ_UNBOUND -1
>
> +/* Supported VDUSE features */
> +static const uint64_t vduse_features;
> +
>  /*
>   * VDUSE instance have not asked the vduse API version, so assume 0.
>   *
> @@ -2207,6 +2210,10 @@ static long vduse_ioctl(struct file *file, unsigned int cmd,
>                 ret = vduse_destroy_dev(name);
>                 break;
>         }
> +       case VDUSE_GET_FEATURES:
> +               ret = put_user(vduse_features, (u64 __user *)argp);
> +               break;
> +
>         default:
>                 ret = -EINVAL;
>                 break;
> diff --git a/include/uapi/linux/vduse.h b/include/uapi/linux/vduse.h
> index 361eea511c21..e9b5f32a452d 100644
> --- a/include/uapi/linux/vduse.h
> +++ b/include/uapi/linux/vduse.h
> @@ -33,6 +33,7 @@
>   * @vq_align: the allocation alignment of virtqueue's metadata
>   * @ngroups: number of vq groups that VDUSE device declares
>   * @nas: number of address spaces that VDUSE device declares
> + * @vduse_features: VDUSE features
>   * @reserved: for future use, needs to be initialized to zero
>   * @config_size: the size of the configuration space
>   * @config: the buffer of the configuration space
> @@ -49,7 +50,8 @@ struct vduse_dev_config {
>         __u32 vq_align;
>         __u32 ngroups; /* if VDUSE_API_VERSION >= 1 */
>         __u32 nas; /* if VDUSE_API_VERSION >= 1 */
> -       __u32 reserved[11];
> +       __u64 vduse_features; /* if VDUSE_GET_FEATURES is not EINVAL */
> +       __u32 reserved[9];

If we go with VDUSE_GET_FEATURES, I'd perfer to go with
VDUSE_SET_FEATURES intead of this.

But we can hear from others.

>         __u32 config_size;
>         __u8 config[];
>  };
> @@ -63,6 +65,9 @@ struct vduse_dev_config {
>   */
>  #define VDUSE_DESTROY_DEV      _IOW(VDUSE_BASE, 0x03, char[VDUSE_NAME_MAX])
>
> +/* Get the VDUSE supported features */
> +#define VDUSE_GET_FEATURES     _IOR(VDUSE_BASE, 0x04, __u64)
> +
>  /* The ioctls for VDUSE device (/dev/vduse/$NAME) */
>
>  /**
> --
> 2.53.0
>

Thanks

Re: [PATCH v3 2/3] vduse: add VDUSE_GET_FEATURES ioctl

[Eugenio Perez Martin]

On Thu, Mar 12, 2026 at 4:56 AM Jason Wang <jasowang@redhat.com> wrote:
>
> On Wed, Mar 11, 2026 at 3:08 AM Eugenio Pérez <eperezma@redhat.com> wrote:
> >
> > Add an ioctl to allow VDUSE instances to query the available features
> > supported by the kernel module.
> >
> > Signed-off-by: Eugenio Pérez <eperezma@redhat.com>
> > ---
> > A simple u64 bitmap is used for feature flags. While a flexible array
> > could support indefinite expansion, 64 bits is sufficient for the
> > foreseeable future and simplifies the implementation.
> >
> > Also, dev_dbg is used for logging rather than dev_err as these can be
> > triggered from userspace.
> > ---
> > v3:
> > * Remove check for API_VERSION < 2
> > * Add comment about struct vduse_dev_config:vduse_features is only valid
> >   if VDUSE_GET_FEATURES success.
> >
> > v2:
> > * return -EINVAL if ioctl called with version < 2, so userland visible
> >   reply is kept (Jason).
> > ---
> >  drivers/vdpa/vdpa_user/vduse_dev.c | 7 +++++++
> >  include/uapi/linux/vduse.h         | 7 ++++++-
> >  2 files changed, 13 insertions(+), 1 deletion(-)
> >
> > diff --git a/drivers/vdpa/vdpa_user/vduse_dev.c b/drivers/vdpa/vdpa_user/vduse_dev.c
> > index d1da7c15d98b..17e0358d3a68 100644
> > --- a/drivers/vdpa/vdpa_user/vduse_dev.c
> > +++ b/drivers/vdpa/vdpa_user/vduse_dev.c
> > @@ -52,6 +52,9 @@
> >
> >  #define IRQ_UNBOUND -1
> >
> > +/* Supported VDUSE features */
> > +static const uint64_t vduse_features;
> > +
> >  /*
> >   * VDUSE instance have not asked the vduse API version, so assume 0.
> >   *
> > @@ -2207,6 +2210,10 @@ static long vduse_ioctl(struct file *file, unsigned int cmd,
> >                 ret = vduse_destroy_dev(name);
> >                 break;
> >         }
> > +       case VDUSE_GET_FEATURES:
> > +               ret = put_user(vduse_features, (u64 __user *)argp);
> > +               break;
> > +
> >         default:
> >                 ret = -EINVAL;
> >                 break;
> > diff --git a/include/uapi/linux/vduse.h b/include/uapi/linux/vduse.h
> > index 361eea511c21..e9b5f32a452d 100644
> > --- a/include/uapi/linux/vduse.h
> > +++ b/include/uapi/linux/vduse.h
> > @@ -33,6 +33,7 @@
> >   * @vq_align: the allocation alignment of virtqueue's metadata
> >   * @ngroups: number of vq groups that VDUSE device declares
> >   * @nas: number of address spaces that VDUSE device declares
> > + * @vduse_features: VDUSE features
> >   * @reserved: for future use, needs to be initialized to zero
> >   * @config_size: the size of the configuration space
> >   * @config: the buffer of the configuration space
> > @@ -49,7 +50,8 @@ struct vduse_dev_config {
> >         __u32 vq_align;
> >         __u32 ngroups; /* if VDUSE_API_VERSION >= 1 */
> >         __u32 nas; /* if VDUSE_API_VERSION >= 1 */
> > -       __u32 reserved[11];
> > +       __u64 vduse_features; /* if VDUSE_GET_FEATURES is not EINVAL */
> > +       __u32 reserved[9];
>
> If we go with VDUSE_GET_FEATURES, I'd perfer to go with
> VDUSE_SET_FEATURES intead of this.
>

I didn't realize the lack of symmetry, but that approach grows and
complicates the possible states of struct vduse_control and the
vduse_ioctl code for little benefit in my opinion. Making it atomic in
VDUSE_CREATE_DEV ioctl helps centralize all the potential errors.
Also, from previous experience with vhost_vdpa, the ioctls slow down
the device creation, which could affect things like VDUSE adoption for
containers etc.

On the other hand, it would make it easier to tell if the device
couldn't be created because of invalid features set. This is highly
unlikely, as the VDUSE device should retrieve the features by calling
VDUSE_GET_FEATURES earlier.

I don't think it is worth it, but if you think we should go that way
I'm ok with the change.

> But we can hear from others.
>
> >         __u32 config_size;
> >         __u8 config[];
> >  };
> > @@ -63,6 +65,9 @@ struct vduse_dev_config {
> >   */
> >  #define VDUSE_DESTROY_DEV      _IOW(VDUSE_BASE, 0x03, char[VDUSE_NAME_MAX])
> >
> > +/* Get the VDUSE supported features */
> > +#define VDUSE_GET_FEATURES     _IOR(VDUSE_BASE, 0x04, __u64)
> > +
> >  /* The ioctls for VDUSE device (/dev/vduse/$NAME) */
> >
> >  /**
> > --
> > 2.53.0
> >
>
> Thanks
>

Re: [PATCH v3 2/3] vduse: add VDUSE_GET_FEATURES ioctl

[Jason Wang]

On Thu, Mar 12, 2026 at 3:12 PM Eugenio Perez Martin
<eperezma@redhat.com> wrote:
>
> On Thu, Mar 12, 2026 at 4:56 AM Jason Wang <jasowang@redhat.com> wrote:
> >
> > On Wed, Mar 11, 2026 at 3:08 AM Eugenio Pérez <eperezma@redhat.com> wrote:
> > >
> > > Add an ioctl to allow VDUSE instances to query the available features
> > > supported by the kernel module.
> > >
> > > Signed-off-by: Eugenio Pérez <eperezma@redhat.com>
> > > ---
> > > A simple u64 bitmap is used for feature flags. While a flexible array
> > > could support indefinite expansion, 64 bits is sufficient for the
> > > foreseeable future and simplifies the implementation.
> > >
> > > Also, dev_dbg is used for logging rather than dev_err as these can be
> > > triggered from userspace.
> > > ---
> > > v3:
> > > * Remove check for API_VERSION < 2
> > > * Add comment about struct vduse_dev_config:vduse_features is only valid
> > >   if VDUSE_GET_FEATURES success.
> > >
> > > v2:
> > > * return -EINVAL if ioctl called with version < 2, so userland visible
> > >   reply is kept (Jason).
> > > ---
> > >  drivers/vdpa/vdpa_user/vduse_dev.c | 7 +++++++
> > >  include/uapi/linux/vduse.h         | 7 ++++++-
> > >  2 files changed, 13 insertions(+), 1 deletion(-)
> > >
> > > diff --git a/drivers/vdpa/vdpa_user/vduse_dev.c b/drivers/vdpa/vdpa_user/vduse_dev.c
> > > index d1da7c15d98b..17e0358d3a68 100644
> > > --- a/drivers/vdpa/vdpa_user/vduse_dev.c
> > > +++ b/drivers/vdpa/vdpa_user/vduse_dev.c
> > > @@ -52,6 +52,9 @@
> > >
> > >  #define IRQ_UNBOUND -1
> > >
> > > +/* Supported VDUSE features */
> > > +static const uint64_t vduse_features;
> > > +
> > >  /*
> > >   * VDUSE instance have not asked the vduse API version, so assume 0.
> > >   *
> > > @@ -2207,6 +2210,10 @@ static long vduse_ioctl(struct file *file, unsigned int cmd,
> > >                 ret = vduse_destroy_dev(name);
> > >                 break;
> > >         }
> > > +       case VDUSE_GET_FEATURES:
> > > +               ret = put_user(vduse_features, (u64 __user *)argp);
> > > +               break;
> > > +
> > >         default:
> > >                 ret = -EINVAL;
> > >                 break;
> > > diff --git a/include/uapi/linux/vduse.h b/include/uapi/linux/vduse.h
> > > index 361eea511c21..e9b5f32a452d 100644
> > > --- a/include/uapi/linux/vduse.h
> > > +++ b/include/uapi/linux/vduse.h
> > > @@ -33,6 +33,7 @@
> > >   * @vq_align: the allocation alignment of virtqueue's metadata
> > >   * @ngroups: number of vq groups that VDUSE device declares
> > >   * @nas: number of address spaces that VDUSE device declares
> > > + * @vduse_features: VDUSE features
> > >   * @reserved: for future use, needs to be initialized to zero
> > >   * @config_size: the size of the configuration space
> > >   * @config: the buffer of the configuration space
> > > @@ -49,7 +50,8 @@ struct vduse_dev_config {
> > >         __u32 vq_align;
> > >         __u32 ngroups; /* if VDUSE_API_VERSION >= 1 */
> > >         __u32 nas; /* if VDUSE_API_VERSION >= 1 */
> > > -       __u32 reserved[11];
> > > +       __u64 vduse_features; /* if VDUSE_GET_FEATURES is not EINVAL */
> > > +       __u32 reserved[9];
> >
> > If we go with VDUSE_GET_FEATURES, I'd perfer to go with
> > VDUSE_SET_FEATURES intead of this.
> >
>
> I didn't realize the lack of symmetry, but that approach grows and
> complicates the possible states of struct vduse_control and the
> vduse_ioctl code for little benefit in my opinion. Making it atomic in
> VDUSE_CREATE_DEV ioctl helps centralize all the potential errors.
> Also, from previous experience with vhost_vdpa, the ioctls slow down
> the device creation, which could affect things like VDUSE adoption for
> containers etc.

Maybe, but if we really care about the time spent on device creation,
uAPI needs redesigning. Or at least more feature in the future
wouldn't slow down further.

>
> On the other hand, it would make it easier to tell if the device
> couldn't be created because of invalid features set. This is highly
> unlikely, as the VDUSE device should retrieve the features by calling
> VDUSE_GET_FEATURES earlier.
>
> I don't think it is worth it, but if you think we should go that way
> I'm ok with the change.

It depends on the intialziation flow. After getting features, I'd
expect it's something like this

A:

1) GET_API_VERSION
2) SET_API_VERSION
3) GET_FEATURES
4) SET_FEATURES
5) CREATE_DEV

Or B in this patch:

1) GET_API_VERSION
2) SET_API_VERSION
3) GET_FEATURES
4) CREATE_DEV

I think it's better to stick the symmetry as API_VERSION (technically,
API could be set via config as well?)

BTW, I found VDUSE_SET_API_VERSION could be set multiple times (even
after the device is create), do we need to fix this?

Thanks

>
> > But we can hear from others.
> >
> > >         __u32 config_size;
> > >         __u8 config[];
> > >  };
> > > @@ -63,6 +65,9 @@ struct vduse_dev_config {
> > >   */
> > >  #define VDUSE_DESTROY_DEV      _IOW(VDUSE_BASE, 0x03, char[VDUSE_NAME_MAX])
> > >
> > > +/* Get the VDUSE supported features */
> > > +#define VDUSE_GET_FEATURES     _IOR(VDUSE_BASE, 0x04, __u64)
> > > +
> > >  /* The ioctls for VDUSE device (/dev/vduse/$NAME) */
> > >
> > >  /**
> > > --
> > > 2.53.0
> > >
> >
> > Thanks
> >
>

Re: [PATCH v3 2/3] vduse: add VDUSE_GET_FEATURES ioctl

[Eugenio Perez Martin]

On Fri, Mar 13, 2026 at 6:56 AM Jason Wang <jasowang@redhat.com> wrote:
>
> On Thu, Mar 12, 2026 at 3:12 PM Eugenio Perez Martin
> <eperezma@redhat.com> wrote:
> >
> > On Thu, Mar 12, 2026 at 4:56 AM Jason Wang <jasowang@redhat.com> wrote:
> > >
> > > On Wed, Mar 11, 2026 at 3:08 AM Eugenio Pérez <eperezma@redhat.com> wrote:
> > > >
> > > > Add an ioctl to allow VDUSE instances to query the available features
> > > > supported by the kernel module.
> > > >
> > > > Signed-off-by: Eugenio Pérez <eperezma@redhat.com>
> > > > ---
> > > > A simple u64 bitmap is used for feature flags. While a flexible array
> > > > could support indefinite expansion, 64 bits is sufficient for the
> > > > foreseeable future and simplifies the implementation.
> > > >
> > > > Also, dev_dbg is used for logging rather than dev_err as these can be
> > > > triggered from userspace.
> > > > ---
> > > > v3:
> > > > * Remove check for API_VERSION < 2
> > > > * Add comment about struct vduse_dev_config:vduse_features is only valid
> > > >   if VDUSE_GET_FEATURES success.
> > > >
> > > > v2:
> > > > * return -EINVAL if ioctl called with version < 2, so userland visible
> > > >   reply is kept (Jason).
> > > > ---
> > > >  drivers/vdpa/vdpa_user/vduse_dev.c | 7 +++++++
> > > >  include/uapi/linux/vduse.h         | 7 ++++++-
> > > >  2 files changed, 13 insertions(+), 1 deletion(-)
> > > >
> > > > diff --git a/drivers/vdpa/vdpa_user/vduse_dev.c b/drivers/vdpa/vdpa_user/vduse_dev.c
> > > > index d1da7c15d98b..17e0358d3a68 100644
> > > > --- a/drivers/vdpa/vdpa_user/vduse_dev.c
> > > > +++ b/drivers/vdpa/vdpa_user/vduse_dev.c
> > > > @@ -52,6 +52,9 @@
> > > >
> > > >  #define IRQ_UNBOUND -1
> > > >
> > > > +/* Supported VDUSE features */
> > > > +static const uint64_t vduse_features;
> > > > +
> > > >  /*
> > > >   * VDUSE instance have not asked the vduse API version, so assume 0.
> > > >   *
> > > > @@ -2207,6 +2210,10 @@ static long vduse_ioctl(struct file *file, unsigned int cmd,
> > > >                 ret = vduse_destroy_dev(name);
> > > >                 break;
> > > >         }
> > > > +       case VDUSE_GET_FEATURES:
> > > > +               ret = put_user(vduse_features, (u64 __user *)argp);
> > > > +               break;
> > > > +
> > > >         default:
> > > >                 ret = -EINVAL;
> > > >                 break;
> > > > diff --git a/include/uapi/linux/vduse.h b/include/uapi/linux/vduse.h
> > > > index 361eea511c21..e9b5f32a452d 100644
> > > > --- a/include/uapi/linux/vduse.h
> > > > +++ b/include/uapi/linux/vduse.h
> > > > @@ -33,6 +33,7 @@
> > > >   * @vq_align: the allocation alignment of virtqueue's metadata
> > > >   * @ngroups: number of vq groups that VDUSE device declares
> > > >   * @nas: number of address spaces that VDUSE device declares
> > > > + * @vduse_features: VDUSE features
> > > >   * @reserved: for future use, needs to be initialized to zero
> > > >   * @config_size: the size of the configuration space
> > > >   * @config: the buffer of the configuration space
> > > > @@ -49,7 +50,8 @@ struct vduse_dev_config {
> > > >         __u32 vq_align;
> > > >         __u32 ngroups; /* if VDUSE_API_VERSION >= 1 */
> > > >         __u32 nas; /* if VDUSE_API_VERSION >= 1 */
> > > > -       __u32 reserved[11];
> > > > +       __u64 vduse_features; /* if VDUSE_GET_FEATURES is not EINVAL */
> > > > +       __u32 reserved[9];
> > >
> > > If we go with VDUSE_GET_FEATURES, I'd perfer to go with
> > > VDUSE_SET_FEATURES intead of this.
> > >
> >
> > I didn't realize the lack of symmetry, but that approach grows and
> > complicates the possible states of struct vduse_control and the
> > vduse_ioctl code for little benefit in my opinion. Making it atomic in
> > VDUSE_CREATE_DEV ioctl helps centralize all the potential errors.
> > Also, from previous experience with vhost_vdpa, the ioctls slow down
> > the device creation, which could affect things like VDUSE adoption for
> > containers etc.
>
> Maybe, but if we really care about the time spent on device creation,
> uAPI needs redesigning. Or at least more feature in the future
> wouldn't slow down further.
>
> >
> > On the other hand, it would make it easier to tell if the device
> > couldn't be created because of invalid features set. This is highly
> > unlikely, as the VDUSE device should retrieve the features by calling
> > VDUSE_GET_FEATURES earlier.
> >
> > I don't think it is worth it, but if you think we should go that way
> > I'm ok with the change.
>
> It depends on the intialziation flow. After getting features, I'd
> expect it's something like this
>
> A:
>
> 1) GET_API_VERSION
> 2) SET_API_VERSION
> 3) GET_FEATURES
> 4) SET_FEATURES
> 5) CREATE_DEV
>
> Or B in this patch:
>
> 1) GET_API_VERSION
> 2) SET_API_VERSION
> 3) GET_FEATURES
> 4) CREATE_DEV
>
> I think it's better to stick the symmetry as API_VERSION (technically,
> API could be set via config as well?)
>

Yes, I'd have chosen to set the API_VERSION via config too.

> BTW, I found VDUSE_SET_API_VERSION could be set multiple times (even
> after the device is create), do we need to fix this?
>

It shouldn't be a problem, each device has its own copy of
api_version. It seems clear to me that the API setting only affects
devices created after the change, but I can add a guard after the
first device is created.

> Thanks
>
> >
> > > But we can hear from others.
> > >
> > > >         __u32 config_size;
> > > >         __u8 config[];
> > > >  };
> > > > @@ -63,6 +65,9 @@ struct vduse_dev_config {
> > > >   */
> > > >  #define VDUSE_DESTROY_DEV      _IOW(VDUSE_BASE, 0x03, char[VDUSE_NAME_MAX])
> > > >
> > > > +/* Get the VDUSE supported features */
> > > > +#define VDUSE_GET_FEATURES     _IOR(VDUSE_BASE, 0x04, __u64)
> > > > +
> > > >  /* The ioctls for VDUSE device (/dev/vduse/$NAME) */
> > > >
> > > >  /**
> > > > --
> > > > 2.53.0
> > > >
> > >
> > > Thanks
> > >
> >
>

[PATCH v3 3/3] vduse: add F_QUEUE_READY feature

[Eugenio Pérez]

Add the VDUSE_F_QUEUE_READY feature flag. This allows the kernel module
to explicitly signal userspace when a specific virtqueue has been
enabled.

In scenarios like Live Migration of VirtIO net devices, the dataplane
starts after the control virtqueue allowing QEMU to apply configuration
in the destination device.

Signed-off-by: Eugenio Pérez <eperezma@redhat.com>
---
v2:
* Fix comment of vduse_dev_request.vq_ready
* Set vq_ready before sending the message to the VDUSE userland
  instance, avoiding the need for SMP sync after receiving the message.
---
 drivers/vdpa/vdpa_user/vduse_dev.c | 28 +++++++++++++++++++++++++++-
 include/uapi/linux/vduse.h         | 18 ++++++++++++++++++
 2 files changed, 45 insertions(+), 1 deletion(-)

diff --git a/drivers/vdpa/vdpa_user/vduse_dev.c b/drivers/vdpa/vdpa_user/vduse_dev.c
index 17e0358d3a68..4f642b95a7cb 100644
--- a/drivers/vdpa/vdpa_user/vduse_dev.c
+++ b/drivers/vdpa/vdpa_user/vduse_dev.c
@@ -9,6 +9,7 @@
  */
 
 #include "linux/virtio_net.h"
+#include <linux/bits.h>
 #include <linux/cleanup.h>
 #include <linux/init.h>
 #include <linux/module.h>
@@ -53,7 +54,7 @@
 #define IRQ_UNBOUND -1
 
 /* Supported VDUSE features */
-static const uint64_t vduse_features;
+static const uint64_t vduse_features = BIT_U64(VDUSE_F_QUEUE_READY);
 
 /*
  * VDUSE instance have not asked the vduse API version, so assume 0.
@@ -120,6 +121,7 @@ struct vduse_dev {
 	char *name;
 	struct mutex lock;
 	spinlock_t msg_lock;
+	u64 vduse_features;
 	u64 msg_unique;
 	u32 msg_timeout;
 	wait_queue_head_t waitq;
@@ -601,8 +603,29 @@ static void vduse_vdpa_set_vq_ready(struct vdpa_device *vdpa,
 {
 	struct vduse_dev *dev = vdpa_to_vduse(vdpa);
 	struct vduse_virtqueue *vq = dev->vqs[idx];
+	struct vduse_dev_msg msg = { 0 };
+	int r;
 
 	vq->ready = ready;
+
+	if (!(dev->vduse_features & BIT_U64(VDUSE_F_QUEUE_READY)))
+		return;
+
+	msg.req.type = VDUSE_SET_VQ_READY;
+	msg.req.vq_ready.num = idx;
+	msg.req.vq_ready.ready = !!ready;
+
+	r = vduse_dev_msg_sync(dev, &msg);
+
+	if (r < 0) {
+		dev_dbg(&vdpa->dev, "device refuses to set vq %u ready %u",
+			idx, ready);
+
+		/* We can't do better than break the device in this case */
+		spin_lock(&dev->msg_lock);
+		vduse_dev_broken(dev);
+		spin_unlock(&dev->msg_lock);
+	}
 }
 
 static bool vduse_vdpa_get_vq_ready(struct vdpa_device *vdpa, u16 idx)
@@ -2078,6 +2101,9 @@ static int vduse_create_dev(struct vduse_dev_config *config,
 	dev->device_features = config->features;
 	dev->device_id = config->device_id;
 	dev->vendor_id = config->vendor_id;
+	dev->vduse_features = config->vduse_features;
+	dev_dbg(vduse_ctrl_dev, "Creating device %s with features 0x%llx",
+		config->name, config->vduse_features);
 
 	dev->nas = (dev->api_version < VDUSE_API_VERSION_1) ? 1 : config->nas;
 	dev->as = kcalloc(dev->nas, sizeof(dev->as[0]), GFP_KERNEL);
diff --git a/include/uapi/linux/vduse.h b/include/uapi/linux/vduse.h
index e9b5f32a452d..7324faea5df4 100644
--- a/include/uapi/linux/vduse.h
+++ b/include/uapi/linux/vduse.h
@@ -14,6 +14,9 @@
 
 #define VDUSE_API_VERSION_1	1
 
+/* The VDUSE instance expects a request for vq ready */
+#define VDUSE_F_QUEUE_READY	0
+
 /*
  * Get the version of VDUSE API that kernel supported (VDUSE_API_VERSION).
  * This is used for future extension.
@@ -330,6 +333,7 @@ enum vduse_req_type {
 	VDUSE_SET_STATUS,
 	VDUSE_UPDATE_IOTLB,
 	VDUSE_SET_VQ_GROUP_ASID,
+	VDUSE_SET_VQ_READY,
 };
 
 /**
@@ -377,6 +381,15 @@ struct vduse_iova_range_v2 {
 	__u32 padding;
 };
 
+/**
+ * struct vduse_vq_ready - Virtqueue ready request message
+ * @num: Virtqueue number
+ */
+struct vduse_vq_ready {
+	__u32 num;
+	__u32 ready;
+};
+
 /**
  * struct vduse_dev_request - control request
  * @type: request type
@@ -387,6 +400,7 @@ struct vduse_iova_range_v2 {
  * @iova: IOVA range for updating
  * @iova_v2: IOVA range for updating if API_VERSION >= 1
  * @vq_group_asid: ASID of a virtqueue group
+ * @vq_ready: Virtqueue ready request
  * @padding: padding
  *
  * Structure used by read(2) on /dev/vduse/$NAME.
@@ -404,6 +418,10 @@ struct vduse_dev_request {
 		 */
 		struct vduse_iova_range_v2 iova_v2;
 		struct vduse_vq_group_asid vq_group_asid;
+
+		/* Only if VDUSE_F_QUEUE_READY is negotiated */
+		struct vduse_vq_ready vq_ready;
+
 		__u32 padding[32];
 	};
 };
-- 
2.53.0

Re: [PATCH v3 3/3] vduse: add F_QUEUE_READY feature

[Jason Wang]

On Wed, Mar 11, 2026 at 3:08 AM Eugenio Pérez <eperezma@redhat.com> wrote:
>
> Add the VDUSE_F_QUEUE_READY feature flag. This allows the kernel module
> to explicitly signal userspace when a specific virtqueue has been
> enabled.
>
> In scenarios like Live Migration of VirtIO net devices, the dataplane
> starts after the control virtqueue allowing QEMU to apply configuration
> in the destination device.
>
> Signed-off-by: Eugenio Pérez <eperezma@redhat.com>
> ---
> v2:
> * Fix comment of vduse_dev_request.vq_ready
> * Set vq_ready before sending the message to the VDUSE userland
>   instance, avoiding the need for SMP sync after receiving the message.
> ---
>  drivers/vdpa/vdpa_user/vduse_dev.c | 28 +++++++++++++++++++++++++++-
>  include/uapi/linux/vduse.h         | 18 ++++++++++++++++++
>  2 files changed, 45 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/vdpa/vdpa_user/vduse_dev.c b/drivers/vdpa/vdpa_user/vduse_dev.c
> index 17e0358d3a68..4f642b95a7cb 100644
> --- a/drivers/vdpa/vdpa_user/vduse_dev.c
> +++ b/drivers/vdpa/vdpa_user/vduse_dev.c
> @@ -9,6 +9,7 @@
>   */
>
>  #include "linux/virtio_net.h"
> +#include <linux/bits.h>
>  #include <linux/cleanup.h>
>  #include <linux/init.h>
>  #include <linux/module.h>
> @@ -53,7 +54,7 @@
>  #define IRQ_UNBOUND -1
>
>  /* Supported VDUSE features */
> -static const uint64_t vduse_features;
> +static const uint64_t vduse_features = BIT_U64(VDUSE_F_QUEUE_READY);
>
>  /*
>   * VDUSE instance have not asked the vduse API version, so assume 0.
> @@ -120,6 +121,7 @@ struct vduse_dev {
>         char *name;
>         struct mutex lock;
>         spinlock_t msg_lock;
> +       u64 vduse_features;
>         u64 msg_unique;
>         u32 msg_timeout;
>         wait_queue_head_t waitq;
> @@ -601,8 +603,29 @@ static void vduse_vdpa_set_vq_ready(struct vdpa_device *vdpa,
>  {
>         struct vduse_dev *dev = vdpa_to_vduse(vdpa);
>         struct vduse_virtqueue *vq = dev->vqs[idx];
> +       struct vduse_dev_msg msg = { 0 };
> +       int r;
>
>         vq->ready = ready;
> +
> +       if (!(dev->vduse_features & BIT_U64(VDUSE_F_QUEUE_READY)))
> +               return;
> +
> +       msg.req.type = VDUSE_SET_VQ_READY;
> +       msg.req.vq_ready.num = idx;
> +       msg.req.vq_ready.ready = !!ready;
> +
> +       r = vduse_dev_msg_sync(dev, &msg);
> +
> +       if (r < 0) {
> +               dev_dbg(&vdpa->dev, "device refuses to set vq %u ready %u",
> +                       idx, ready);
> +
> +               /* We can't do better than break the device in this case */

It's better to explain why we can't depend on vduse_dev_msg_sync() here.

For example it did:

        if (unlikely(dev->broken))
                return -EIO;

        init_waitqueue_head(&msg->waitq);
        spin_lock(&dev->msg_lock);
        if (unlikely(dev->broken)) {
                spin_unlock(&dev->msg_lock);
                return -EIO;
        }

and

        if (!msg->completed) {
                list_del(&msg->list);
                msg->resp.result = VDUSE_REQ_RESULT_FAILED;
                /* Mark the device as malfunction when there is a timeout */
                if (!ret)
                        vduse_dev_broken(dev);
        }

> +               spin_lock(&dev->msg_lock);
> +               vduse_dev_broken(dev);
> +               spin_unlock(&dev->msg_lock);
> +       }
>  }
>
>  static bool vduse_vdpa_get_vq_ready(struct vdpa_device *vdpa, u16 idx)
> @@ -2078,6 +2101,9 @@ static int vduse_create_dev(struct vduse_dev_config *config,
>         dev->device_features = config->features;
>         dev->device_id = config->device_id;
>         dev->vendor_id = config->vendor_id;
> +       dev->vduse_features = config->vduse_features;
> +       dev_dbg(vduse_ctrl_dev, "Creating device %s with features 0x%llx",
> +               config->name, config->vduse_features);
>
>         dev->nas = (dev->api_version < VDUSE_API_VERSION_1) ? 1 : config->nas;
>         dev->as = kcalloc(dev->nas, sizeof(dev->as[0]), GFP_KERNEL);
> diff --git a/include/uapi/linux/vduse.h b/include/uapi/linux/vduse.h
> index e9b5f32a452d..7324faea5df4 100644
> --- a/include/uapi/linux/vduse.h
> +++ b/include/uapi/linux/vduse.h
> @@ -14,6 +14,9 @@
>
>  #define VDUSE_API_VERSION_1    1
>
> +/* The VDUSE instance expects a request for vq ready */
> +#define VDUSE_F_QUEUE_READY    0
> +
>  /*
>   * Get the version of VDUSE API that kernel supported (VDUSE_API_VERSION).
>   * This is used for future extension.
> @@ -330,6 +333,7 @@ enum vduse_req_type {
>         VDUSE_SET_STATUS,
>         VDUSE_UPDATE_IOTLB,
>         VDUSE_SET_VQ_GROUP_ASID,
> +       VDUSE_SET_VQ_READY,
>  };
>
>  /**
> @@ -377,6 +381,15 @@ struct vduse_iova_range_v2 {
>         __u32 padding;
>  };
>
> +/**
> + * struct vduse_vq_ready - Virtqueue ready request message
> + * @num: Virtqueue number
> + */
> +struct vduse_vq_ready {
> +       __u32 num;
> +       __u32 ready;
> +};
> +
>  /**
>   * struct vduse_dev_request - control request
>   * @type: request type
> @@ -387,6 +400,7 @@ struct vduse_iova_range_v2 {
>   * @iova: IOVA range for updating
>   * @iova_v2: IOVA range for updating if API_VERSION >= 1
>   * @vq_group_asid: ASID of a virtqueue group
> + * @vq_ready: Virtqueue ready request
>   * @padding: padding
>   *
>   * Structure used by read(2) on /dev/vduse/$NAME.
> @@ -404,6 +418,10 @@ struct vduse_dev_request {
>                  */
>                 struct vduse_iova_range_v2 iova_v2;
>                 struct vduse_vq_group_asid vq_group_asid;
> +
> +               /* Only if VDUSE_F_QUEUE_READY is negotiated */
> +               struct vduse_vq_ready vq_ready;
> +
>                 __u32 padding[32];
>         };
>  };
> --
> 2.53.0
>

Thanks

Re: [PATCH v3 3/3] vduse: add F_QUEUE_READY feature

[Eugenio Perez Martin]

On Thu, Mar 12, 2026 at 4:58 AM Jason Wang <jasowang@redhat.com> wrote:
>
> On Wed, Mar 11, 2026 at 3:08 AM Eugenio Pérez <eperezma@redhat.com> wrote:
> >
> > Add the VDUSE_F_QUEUE_READY feature flag. This allows the kernel module
> > to explicitly signal userspace when a specific virtqueue has been
> > enabled.
> >
> > In scenarios like Live Migration of VirtIO net devices, the dataplane
> > starts after the control virtqueue allowing QEMU to apply configuration
> > in the destination device.
> >
> > Signed-off-by: Eugenio Pérez <eperezma@redhat.com>
> > ---
> > v2:
> > * Fix comment of vduse_dev_request.vq_ready
> > * Set vq_ready before sending the message to the VDUSE userland
> >   instance, avoiding the need for SMP sync after receiving the message.
> > ---
> >  drivers/vdpa/vdpa_user/vduse_dev.c | 28 +++++++++++++++++++++++++++-
> >  include/uapi/linux/vduse.h         | 18 ++++++++++++++++++
> >  2 files changed, 45 insertions(+), 1 deletion(-)
> >
> > diff --git a/drivers/vdpa/vdpa_user/vduse_dev.c b/drivers/vdpa/vdpa_user/vduse_dev.c
> > index 17e0358d3a68..4f642b95a7cb 100644
> > --- a/drivers/vdpa/vdpa_user/vduse_dev.c
> > +++ b/drivers/vdpa/vdpa_user/vduse_dev.c
> > @@ -9,6 +9,7 @@
> >   */
> >
> >  #include "linux/virtio_net.h"
> > +#include <linux/bits.h>
> >  #include <linux/cleanup.h>
> >  #include <linux/init.h>
> >  #include <linux/module.h>
> > @@ -53,7 +54,7 @@
> >  #define IRQ_UNBOUND -1
> >
> >  /* Supported VDUSE features */
> > -static const uint64_t vduse_features;
> > +static const uint64_t vduse_features = BIT_U64(VDUSE_F_QUEUE_READY);
> >
> >  /*
> >   * VDUSE instance have not asked the vduse API version, so assume 0.
> > @@ -120,6 +121,7 @@ struct vduse_dev {
> >         char *name;
> >         struct mutex lock;
> >         spinlock_t msg_lock;
> > +       u64 vduse_features;
> >         u64 msg_unique;
> >         u32 msg_timeout;
> >         wait_queue_head_t waitq;
> > @@ -601,8 +603,29 @@ static void vduse_vdpa_set_vq_ready(struct vdpa_device *vdpa,
> >  {
> >         struct vduse_dev *dev = vdpa_to_vduse(vdpa);
> >         struct vduse_virtqueue *vq = dev->vqs[idx];
> > +       struct vduse_dev_msg msg = { 0 };
> > +       int r;
> >
> >         vq->ready = ready;
> > +
> > +       if (!(dev->vduse_features & BIT_U64(VDUSE_F_QUEUE_READY)))
> > +               return;
> > +
> > +       msg.req.type = VDUSE_SET_VQ_READY;
> > +       msg.req.vq_ready.num = idx;
> > +       msg.req.vq_ready.ready = !!ready;
> > +
> > +       r = vduse_dev_msg_sync(dev, &msg);
> > +
> > +       if (r < 0) {
> > +               dev_dbg(&vdpa->dev, "device refuses to set vq %u ready %u",
> > +                       idx, ready);
> > +
> > +               /* We can't do better than break the device in this case */
>
> It's better to explain why we can't depend on vduse_dev_msg_sync() here.
>
> For example it did:
>
>         if (unlikely(dev->broken))
>                 return -EIO;
>
>         init_waitqueue_head(&msg->waitq);
>         spin_lock(&dev->msg_lock);
>         if (unlikely(dev->broken)) {
>                 spin_unlock(&dev->msg_lock);
>                 return -EIO;
>         }
>
> and
>
>         if (!msg->completed) {
>                 list_del(&msg->list);
>                 msg->resp.result = VDUSE_REQ_RESULT_FAILED;
>                 /* Mark the device as malfunction when there is a timeout */
>                 if (!ret)
>                         vduse_dev_broken(dev);
>         }
>

I'm not sure I follow you here.

We can't do better than breaking the device because the function
returns no error state, and the caller does not expect an error code.
Do you mean we can't depend on vduse_dev_msg_sync to call
vduse_dev_broken(dev) by itself?

> > +               spin_lock(&dev->msg_lock);
> > +               vduse_dev_broken(dev);
> > +               spin_unlock(&dev->msg_lock);
> > +       }
> >  }
> >
> >  static bool vduse_vdpa_get_vq_ready(struct vdpa_device *vdpa, u16 idx)
> > @@ -2078,6 +2101,9 @@ static int vduse_create_dev(struct vduse_dev_config *config,
> >         dev->device_features = config->features;
> >         dev->device_id = config->device_id;
> >         dev->vendor_id = config->vendor_id;
> > +       dev->vduse_features = config->vduse_features;
> > +       dev_dbg(vduse_ctrl_dev, "Creating device %s with features 0x%llx",
> > +               config->name, config->vduse_features);
> >
> >         dev->nas = (dev->api_version < VDUSE_API_VERSION_1) ? 1 : config->nas;
> >         dev->as = kcalloc(dev->nas, sizeof(dev->as[0]), GFP_KERNEL);
> > diff --git a/include/uapi/linux/vduse.h b/include/uapi/linux/vduse.h
> > index e9b5f32a452d..7324faea5df4 100644
> > --- a/include/uapi/linux/vduse.h
> > +++ b/include/uapi/linux/vduse.h
> > @@ -14,6 +14,9 @@
> >
> >  #define VDUSE_API_VERSION_1    1
> >
> > +/* The VDUSE instance expects a request for vq ready */
> > +#define VDUSE_F_QUEUE_READY    0
> > +
> >  /*
> >   * Get the version of VDUSE API that kernel supported (VDUSE_API_VERSION).
> >   * This is used for future extension.
> > @@ -330,6 +333,7 @@ enum vduse_req_type {
> >         VDUSE_SET_STATUS,
> >         VDUSE_UPDATE_IOTLB,
> >         VDUSE_SET_VQ_GROUP_ASID,
> > +       VDUSE_SET_VQ_READY,
> >  };
> >
> >  /**
> > @@ -377,6 +381,15 @@ struct vduse_iova_range_v2 {
> >         __u32 padding;
> >  };
> >
> > +/**
> > + * struct vduse_vq_ready - Virtqueue ready request message
> > + * @num: Virtqueue number
> > + */
> > +struct vduse_vq_ready {
> > +       __u32 num;
> > +       __u32 ready;
> > +};
> > +
> >  /**
> >   * struct vduse_dev_request - control request
> >   * @type: request type
> > @@ -387,6 +400,7 @@ struct vduse_iova_range_v2 {
> >   * @iova: IOVA range for updating
> >   * @iova_v2: IOVA range for updating if API_VERSION >= 1
> >   * @vq_group_asid: ASID of a virtqueue group
> > + * @vq_ready: Virtqueue ready request
> >   * @padding: padding
> >   *
> >   * Structure used by read(2) on /dev/vduse/$NAME.
> > @@ -404,6 +418,10 @@ struct vduse_dev_request {
> >                  */
> >                 struct vduse_iova_range_v2 iova_v2;
> >                 struct vduse_vq_group_asid vq_group_asid;
> > +
> > +               /* Only if VDUSE_F_QUEUE_READY is negotiated */
> > +               struct vduse_vq_ready vq_ready;
> > +
> >                 __u32 padding[32];
> >         };
> >  };
> > --
> > 2.53.0
> >
>
> Thanks
>

Re: [PATCH v3 3/3] vduse: add F_QUEUE_READY feature

[Jason Wang]

On Thu, Mar 12, 2026 at 2:24 PM Eugenio Perez Martin
<eperezma@redhat.com> wrote:
>
> On Thu, Mar 12, 2026 at 4:58 AM Jason Wang <jasowang@redhat.com> wrote:
> >
> > On Wed, Mar 11, 2026 at 3:08 AM Eugenio Pérez <eperezma@redhat.com> wrote:
> > >
> > > Add the VDUSE_F_QUEUE_READY feature flag. This allows the kernel module
> > > to explicitly signal userspace when a specific virtqueue has been
> > > enabled.
> > >
> > > In scenarios like Live Migration of VirtIO net devices, the dataplane
> > > starts after the control virtqueue allowing QEMU to apply configuration
> > > in the destination device.
> > >
> > > Signed-off-by: Eugenio Pérez <eperezma@redhat.com>
> > > ---
> > > v2:
> > > * Fix comment of vduse_dev_request.vq_ready
> > > * Set vq_ready before sending the message to the VDUSE userland
> > >   instance, avoiding the need for SMP sync after receiving the message.
> > > ---
> > >  drivers/vdpa/vdpa_user/vduse_dev.c | 28 +++++++++++++++++++++++++++-
> > >  include/uapi/linux/vduse.h         | 18 ++++++++++++++++++
> > >  2 files changed, 45 insertions(+), 1 deletion(-)
> > >
> > > diff --git a/drivers/vdpa/vdpa_user/vduse_dev.c b/drivers/vdpa/vdpa_user/vduse_dev.c
> > > index 17e0358d3a68..4f642b95a7cb 100644
> > > --- a/drivers/vdpa/vdpa_user/vduse_dev.c
> > > +++ b/drivers/vdpa/vdpa_user/vduse_dev.c
> > > @@ -9,6 +9,7 @@
> > >   */
> > >
> > >  #include "linux/virtio_net.h"
> > > +#include <linux/bits.h>
> > >  #include <linux/cleanup.h>
> > >  #include <linux/init.h>
> > >  #include <linux/module.h>
> > > @@ -53,7 +54,7 @@
> > >  #define IRQ_UNBOUND -1
> > >
> > >  /* Supported VDUSE features */
> > > -static const uint64_t vduse_features;
> > > +static const uint64_t vduse_features = BIT_U64(VDUSE_F_QUEUE_READY);
> > >
> > >  /*
> > >   * VDUSE instance have not asked the vduse API version, so assume 0.
> > > @@ -120,6 +121,7 @@ struct vduse_dev {
> > >         char *name;
> > >         struct mutex lock;
> > >         spinlock_t msg_lock;
> > > +       u64 vduse_features;
> > >         u64 msg_unique;
> > >         u32 msg_timeout;
> > >         wait_queue_head_t waitq;
> > > @@ -601,8 +603,29 @@ static void vduse_vdpa_set_vq_ready(struct vdpa_device *vdpa,
> > >  {
> > >         struct vduse_dev *dev = vdpa_to_vduse(vdpa);
> > >         struct vduse_virtqueue *vq = dev->vqs[idx];
> > > +       struct vduse_dev_msg msg = { 0 };
> > > +       int r;
> > >
> > >         vq->ready = ready;
> > > +
> > > +       if (!(dev->vduse_features & BIT_U64(VDUSE_F_QUEUE_READY)))
> > > +               return;
> > > +
> > > +       msg.req.type = VDUSE_SET_VQ_READY;
> > > +       msg.req.vq_ready.num = idx;
> > > +       msg.req.vq_ready.ready = !!ready;
> > > +
> > > +       r = vduse_dev_msg_sync(dev, &msg);
> > > +
> > > +       if (r < 0) {
> > > +               dev_dbg(&vdpa->dev, "device refuses to set vq %u ready %u",
> > > +                       idx, ready);
> > > +
> > > +               /* We can't do better than break the device in this case */
> >
> > It's better to explain why we can't depend on vduse_dev_msg_sync() here.
> >
> > For example it did:
> >
> >         if (unlikely(dev->broken))
> >                 return -EIO;
> >
> >         init_waitqueue_head(&msg->waitq);
> >         spin_lock(&dev->msg_lock);
> >         if (unlikely(dev->broken)) {
> >                 spin_unlock(&dev->msg_lock);
> >                 return -EIO;
> >         }
> >
> > and
> >
> >         if (!msg->completed) {
> >                 list_del(&msg->list);
> >                 msg->resp.result = VDUSE_REQ_RESULT_FAILED;
> >                 /* Mark the device as malfunction when there is a timeout */
> >                 if (!ret)
> >                         vduse_dev_broken(dev);
> >         }
> >
>
> I'm not sure I follow you here.
>
> We can't do better than breaking the device because the function
> returns no error state, and the caller does not expect an error code.
> Do you mean we can't depend on vduse_dev_msg_sync to call
> vduse_dev_broken(dev) by itself?

I think I meant, reset seems to be more heavyweight than suspend.

So if reset can fail, I don't see reason ot break device only for
suspend failure.

Thanks

>
> > > +               spin_lock(&dev->msg_lock);
> > > +               vduse_dev_broken(dev);
> > > +               spin_unlock(&dev->msg_lock);
> > > +       }
> > >  }
> > >
> > >  static bool vduse_vdpa_get_vq_ready(struct vdpa_device *vdpa, u16 idx)
> > > @@ -2078,6 +2101,9 @@ static int vduse_create_dev(struct vduse_dev_config *config,
> > >         dev->device_features = config->features;
> > >         dev->device_id = config->device_id;
> > >         dev->vendor_id = config->vendor_id;
> > > +       dev->vduse_features = config->vduse_features;
> > > +       dev_dbg(vduse_ctrl_dev, "Creating device %s with features 0x%llx",
> > > +               config->name, config->vduse_features);
> > >
> > >         dev->nas = (dev->api_version < VDUSE_API_VERSION_1) ? 1 : config->nas;
> > >         dev->as = kcalloc(dev->nas, sizeof(dev->as[0]), GFP_KERNEL);
> > > diff --git a/include/uapi/linux/vduse.h b/include/uapi/linux/vduse.h
> > > index e9b5f32a452d..7324faea5df4 100644
> > > --- a/include/uapi/linux/vduse.h
> > > +++ b/include/uapi/linux/vduse.h
> > > @@ -14,6 +14,9 @@
> > >
> > >  #define VDUSE_API_VERSION_1    1
> > >
> > > +/* The VDUSE instance expects a request for vq ready */
> > > +#define VDUSE_F_QUEUE_READY    0
> > > +
> > >  /*
> > >   * Get the version of VDUSE API that kernel supported (VDUSE_API_VERSION).
> > >   * This is used for future extension.
> > > @@ -330,6 +333,7 @@ enum vduse_req_type {
> > >         VDUSE_SET_STATUS,
> > >         VDUSE_UPDATE_IOTLB,
> > >         VDUSE_SET_VQ_GROUP_ASID,
> > > +       VDUSE_SET_VQ_READY,
> > >  };
> > >
> > >  /**
> > > @@ -377,6 +381,15 @@ struct vduse_iova_range_v2 {
> > >         __u32 padding;
> > >  };
> > >
> > > +/**
> > > + * struct vduse_vq_ready - Virtqueue ready request message
> > > + * @num: Virtqueue number
> > > + */
> > > +struct vduse_vq_ready {
> > > +       __u32 num;
> > > +       __u32 ready;
> > > +};
> > > +
> > >  /**
> > >   * struct vduse_dev_request - control request
> > >   * @type: request type
> > > @@ -387,6 +400,7 @@ struct vduse_iova_range_v2 {
> > >   * @iova: IOVA range for updating
> > >   * @iova_v2: IOVA range for updating if API_VERSION >= 1
> > >   * @vq_group_asid: ASID of a virtqueue group
> > > + * @vq_ready: Virtqueue ready request
> > >   * @padding: padding
> > >   *
> > >   * Structure used by read(2) on /dev/vduse/$NAME.
> > > @@ -404,6 +418,10 @@ struct vduse_dev_request {
> > >                  */
> > >                 struct vduse_iova_range_v2 iova_v2;
> > >                 struct vduse_vq_group_asid vq_group_asid;
> > > +
> > > +               /* Only if VDUSE_F_QUEUE_READY is negotiated */
> > > +               struct vduse_vq_ready vq_ready;
> > > +
> > >                 __u32 padding[32];
> > >         };
> > >  };
> > > --
> > > 2.53.0
> > >
> >
> > Thanks
> >
>

Re: [PATCH v3 3/3] vduse: add F_QUEUE_READY feature

[Eugenio Perez Martin]

On Fri, Mar 13, 2026 at 7:05 AM Jason Wang <jasowang@redhat.com> wrote:
>
> On Thu, Mar 12, 2026 at 2:24 PM Eugenio Perez Martin
> <eperezma@redhat.com> wrote:
> >
> > On Thu, Mar 12, 2026 at 4:58 AM Jason Wang <jasowang@redhat.com> wrote:
> > >
> > > On Wed, Mar 11, 2026 at 3:08 AM Eugenio Pérez <eperezma@redhat.com> wrote:
> > > >
> > > > Add the VDUSE_F_QUEUE_READY feature flag. This allows the kernel module
> > > > to explicitly signal userspace when a specific virtqueue has been
> > > > enabled.
> > > >
> > > > In scenarios like Live Migration of VirtIO net devices, the dataplane
> > > > starts after the control virtqueue allowing QEMU to apply configuration
> > > > in the destination device.
> > > >
> > > > Signed-off-by: Eugenio Pérez <eperezma@redhat.com>
> > > > ---
> > > > v2:
> > > > * Fix comment of vduse_dev_request.vq_ready
> > > > * Set vq_ready before sending the message to the VDUSE userland
> > > >   instance, avoiding the need for SMP sync after receiving the message.
> > > > ---
> > > >  drivers/vdpa/vdpa_user/vduse_dev.c | 28 +++++++++++++++++++++++++++-
> > > >  include/uapi/linux/vduse.h         | 18 ++++++++++++++++++
> > > >  2 files changed, 45 insertions(+), 1 deletion(-)
> > > >
> > > > diff --git a/drivers/vdpa/vdpa_user/vduse_dev.c b/drivers/vdpa/vdpa_user/vduse_dev.c
> > > > index 17e0358d3a68..4f642b95a7cb 100644
> > > > --- a/drivers/vdpa/vdpa_user/vduse_dev.c
> > > > +++ b/drivers/vdpa/vdpa_user/vduse_dev.c
> > > > @@ -9,6 +9,7 @@
> > > >   */
> > > >
> > > >  #include "linux/virtio_net.h"
> > > > +#include <linux/bits.h>
> > > >  #include <linux/cleanup.h>
> > > >  #include <linux/init.h>
> > > >  #include <linux/module.h>
> > > > @@ -53,7 +54,7 @@
> > > >  #define IRQ_UNBOUND -1
> > > >
> > > >  /* Supported VDUSE features */
> > > > -static const uint64_t vduse_features;
> > > > +static const uint64_t vduse_features = BIT_U64(VDUSE_F_QUEUE_READY);
> > > >
> > > >  /*
> > > >   * VDUSE instance have not asked the vduse API version, so assume 0.
> > > > @@ -120,6 +121,7 @@ struct vduse_dev {
> > > >         char *name;
> > > >         struct mutex lock;
> > > >         spinlock_t msg_lock;
> > > > +       u64 vduse_features;
> > > >         u64 msg_unique;
> > > >         u32 msg_timeout;
> > > >         wait_queue_head_t waitq;
> > > > @@ -601,8 +603,29 @@ static void vduse_vdpa_set_vq_ready(struct vdpa_device *vdpa,
> > > >  {
> > > >         struct vduse_dev *dev = vdpa_to_vduse(vdpa);
> > > >         struct vduse_virtqueue *vq = dev->vqs[idx];
> > > > +       struct vduse_dev_msg msg = { 0 };
> > > > +       int r;
> > > >
> > > >         vq->ready = ready;
> > > > +
> > > > +       if (!(dev->vduse_features & BIT_U64(VDUSE_F_QUEUE_READY)))
> > > > +               return;
> > > > +
> > > > +       msg.req.type = VDUSE_SET_VQ_READY;
> > > > +       msg.req.vq_ready.num = idx;
> > > > +       msg.req.vq_ready.ready = !!ready;
> > > > +
> > > > +       r = vduse_dev_msg_sync(dev, &msg);
> > > > +
> > > > +       if (r < 0) {
> > > > +               dev_dbg(&vdpa->dev, "device refuses to set vq %u ready %u",
> > > > +                       idx, ready);
> > > > +
> > > > +               /* We can't do better than break the device in this case */
> > >
> > > It's better to explain why we can't depend on vduse_dev_msg_sync() here.
> > >
> > > For example it did:
> > >
> > >         if (unlikely(dev->broken))
> > >                 return -EIO;
> > >
> > >         init_waitqueue_head(&msg->waitq);
> > >         spin_lock(&dev->msg_lock);
> > >         if (unlikely(dev->broken)) {
> > >                 spin_unlock(&dev->msg_lock);
> > >                 return -EIO;
> > >         }
> > >
> > > and
> > >
> > >         if (!msg->completed) {
> > >                 list_del(&msg->list);
> > >                 msg->resp.result = VDUSE_REQ_RESULT_FAILED;
> > >                 /* Mark the device as malfunction when there is a timeout */
> > >                 if (!ret)
> > >                         vduse_dev_broken(dev);
> > >         }
> > >
> >
> > I'm not sure I follow you here.
> >
> > We can't do better than breaking the device because the function
> > returns no error state, and the caller does not expect an error code.
> > Do you mean we can't depend on vduse_dev_msg_sync to call
> > vduse_dev_broken(dev) by itself?
>
> I think I meant, reset seems to be more heavyweight than suspend.
>
> So if reset can fail, I don't see reason ot break device only for
> suspend failure.
>

Sorry I still don't get you.

This series does not implement suspend at all. It doesn't modify the
VDUSE device reset or the virtio reset behavior. It only implements
the vq ready message for the device. If the device returns an error
from that operation, what is your proposal for when the driver sends
new messages like resume?

> Thanks
>
> >
> > > > +               spin_lock(&dev->msg_lock);
> > > > +               vduse_dev_broken(dev);
> > > > +               spin_unlock(&dev->msg_lock);
> > > > +       }
> > > >  }
> > > >
> > > >  static bool vduse_vdpa_get_vq_ready(struct vdpa_device *vdpa, u16 idx)
> > > > @@ -2078,6 +2101,9 @@ static int vduse_create_dev(struct vduse_dev_config *config,
> > > >         dev->device_features = config->features;
> > > >         dev->device_id = config->device_id;
> > > >         dev->vendor_id = config->vendor_id;
> > > > +       dev->vduse_features = config->vduse_features;
> > > > +       dev_dbg(vduse_ctrl_dev, "Creating device %s with features 0x%llx",
> > > > +               config->name, config->vduse_features);
> > > >
> > > >         dev->nas = (dev->api_version < VDUSE_API_VERSION_1) ? 1 : config->nas;
> > > >         dev->as = kcalloc(dev->nas, sizeof(dev->as[0]), GFP_KERNEL);
> > > > diff --git a/include/uapi/linux/vduse.h b/include/uapi/linux/vduse.h
> > > > index e9b5f32a452d..7324faea5df4 100644
> > > > --- a/include/uapi/linux/vduse.h
> > > > +++ b/include/uapi/linux/vduse.h
> > > > @@ -14,6 +14,9 @@
> > > >
> > > >  #define VDUSE_API_VERSION_1    1
> > > >
> > > > +/* The VDUSE instance expects a request for vq ready */
> > > > +#define VDUSE_F_QUEUE_READY    0
> > > > +
> > > >  /*
> > > >   * Get the version of VDUSE API that kernel supported (VDUSE_API_VERSION).
> > > >   * This is used for future extension.
> > > > @@ -330,6 +333,7 @@ enum vduse_req_type {
> > > >         VDUSE_SET_STATUS,
> > > >         VDUSE_UPDATE_IOTLB,
> > > >         VDUSE_SET_VQ_GROUP_ASID,
> > > > +       VDUSE_SET_VQ_READY,
> > > >  };
> > > >
> > > >  /**
> > > > @@ -377,6 +381,15 @@ struct vduse_iova_range_v2 {
> > > >         __u32 padding;
> > > >  };
> > > >
> > > > +/**
> > > > + * struct vduse_vq_ready - Virtqueue ready request message
> > > > + * @num: Virtqueue number
> > > > + */
> > > > +struct vduse_vq_ready {
> > > > +       __u32 num;
> > > > +       __u32 ready;
> > > > +};
> > > > +
> > > >  /**
> > > >   * struct vduse_dev_request - control request
> > > >   * @type: request type
> > > > @@ -387,6 +400,7 @@ struct vduse_iova_range_v2 {
> > > >   * @iova: IOVA range for updating
> > > >   * @iova_v2: IOVA range for updating if API_VERSION >= 1
> > > >   * @vq_group_asid: ASID of a virtqueue group
> > > > + * @vq_ready: Virtqueue ready request
> > > >   * @padding: padding
> > > >   *
> > > >   * Structure used by read(2) on /dev/vduse/$NAME.
> > > > @@ -404,6 +418,10 @@ struct vduse_dev_request {
> > > >                  */
> > > >                 struct vduse_iova_range_v2 iova_v2;
> > > >                 struct vduse_vq_group_asid vq_group_asid;
> > > > +
> > > > +               /* Only if VDUSE_F_QUEUE_READY is negotiated */
> > > > +               struct vduse_vq_ready vq_ready;
> > > > +
> > > >                 __u32 padding[32];
> > > >         };
> > > >  };
> > > > --
> > > > 2.53.0
> > > >
> > >
> > > Thanks
> > >
> >
>

Re: [PATCH v3 3/3] vduse: add F_QUEUE_READY feature

[Eugenio Perez Martin]

On Fri, Mar 13, 2026 at 8:08 AM Eugenio Perez Martin
<eperezma@redhat.com> wrote:
>
> On Fri, Mar 13, 2026 at 7:05 AM Jason Wang <jasowang@redhat.com> wrote:
> >
> > On Thu, Mar 12, 2026 at 2:24 PM Eugenio Perez Martin
> > <eperezma@redhat.com> wrote:
> > >
> > > On Thu, Mar 12, 2026 at 4:58 AM Jason Wang <jasowang@redhat.com> wrote:
> > > >
> > > > On Wed, Mar 11, 2026 at 3:08 AM Eugenio Pérez <eperezma@redhat.com> wrote:
> > > > >
> > > > > Add the VDUSE_F_QUEUE_READY feature flag. This allows the kernel module
> > > > > to explicitly signal userspace when a specific virtqueue has been
> > > > > enabled.
> > > > >
> > > > > In scenarios like Live Migration of VirtIO net devices, the dataplane
> > > > > starts after the control virtqueue allowing QEMU to apply configuration
> > > > > in the destination device.
> > > > >
> > > > > Signed-off-by: Eugenio Pérez <eperezma@redhat.com>
> > > > > ---
> > > > > v2:
> > > > > * Fix comment of vduse_dev_request.vq_ready
> > > > > * Set vq_ready before sending the message to the VDUSE userland
> > > > >   instance, avoiding the need for SMP sync after receiving the message.
> > > > > ---
> > > > >  drivers/vdpa/vdpa_user/vduse_dev.c | 28 +++++++++++++++++++++++++++-
> > > > >  include/uapi/linux/vduse.h         | 18 ++++++++++++++++++
> > > > >  2 files changed, 45 insertions(+), 1 deletion(-)
> > > > >
> > > > > diff --git a/drivers/vdpa/vdpa_user/vduse_dev.c b/drivers/vdpa/vdpa_user/vduse_dev.c
> > > > > index 17e0358d3a68..4f642b95a7cb 100644
> > > > > --- a/drivers/vdpa/vdpa_user/vduse_dev.c
> > > > > +++ b/drivers/vdpa/vdpa_user/vduse_dev.c
> > > > > @@ -9,6 +9,7 @@
> > > > >   */
> > > > >
> > > > >  #include "linux/virtio_net.h"
> > > > > +#include <linux/bits.h>
> > > > >  #include <linux/cleanup.h>
> > > > >  #include <linux/init.h>
> > > > >  #include <linux/module.h>
> > > > > @@ -53,7 +54,7 @@
> > > > >  #define IRQ_UNBOUND -1
> > > > >
> > > > >  /* Supported VDUSE features */
> > > > > -static const uint64_t vduse_features;
> > > > > +static const uint64_t vduse_features = BIT_U64(VDUSE_F_QUEUE_READY);
> > > > >
> > > > >  /*
> > > > >   * VDUSE instance have not asked the vduse API version, so assume 0.
> > > > > @@ -120,6 +121,7 @@ struct vduse_dev {
> > > > >         char *name;
> > > > >         struct mutex lock;
> > > > >         spinlock_t msg_lock;
> > > > > +       u64 vduse_features;
> > > > >         u64 msg_unique;
> > > > >         u32 msg_timeout;
> > > > >         wait_queue_head_t waitq;
> > > > > @@ -601,8 +603,29 @@ static void vduse_vdpa_set_vq_ready(struct vdpa_device *vdpa,
> > > > >  {
> > > > >         struct vduse_dev *dev = vdpa_to_vduse(vdpa);
> > > > >         struct vduse_virtqueue *vq = dev->vqs[idx];
> > > > > +       struct vduse_dev_msg msg = { 0 };
> > > > > +       int r;
> > > > >
> > > > >         vq->ready = ready;
> > > > > +
> > > > > +       if (!(dev->vduse_features & BIT_U64(VDUSE_F_QUEUE_READY)))
> > > > > +               return;
> > > > > +
> > > > > +       msg.req.type = VDUSE_SET_VQ_READY;
> > > > > +       msg.req.vq_ready.num = idx;
> > > > > +       msg.req.vq_ready.ready = !!ready;
> > > > > +
> > > > > +       r = vduse_dev_msg_sync(dev, &msg);
> > > > > +
> > > > > +       if (r < 0) {
> > > > > +               dev_dbg(&vdpa->dev, "device refuses to set vq %u ready %u",
> > > > > +                       idx, ready);
> > > > > +
> > > > > +               /* We can't do better than break the device in this case */
> > > >
> > > > It's better to explain why we can't depend on vduse_dev_msg_sync() here.
> > > >
> > > > For example it did:
> > > >
> > > >         if (unlikely(dev->broken))
> > > >                 return -EIO;
> > > >
> > > >         init_waitqueue_head(&msg->waitq);
> > > >         spin_lock(&dev->msg_lock);
> > > >         if (unlikely(dev->broken)) {
> > > >                 spin_unlock(&dev->msg_lock);
> > > >                 return -EIO;
> > > >         }
> > > >
> > > > and
> > > >
> > > >         if (!msg->completed) {
> > > >                 list_del(&msg->list);
> > > >                 msg->resp.result = VDUSE_REQ_RESULT_FAILED;
> > > >                 /* Mark the device as malfunction when there is a timeout */
> > > >                 if (!ret)
> > > >                         vduse_dev_broken(dev);
> > > >         }
> > > >
> > >
> > > I'm not sure I follow you here.
> > >
> > > We can't do better than breaking the device because the function
> > > returns no error state, and the caller does not expect an error code.
> > > Do you mean we can't depend on vduse_dev_msg_sync to call
> > > vduse_dev_broken(dev) by itself?
> >
> > I think I meant, reset seems to be more heavyweight than suspend.
> >
> > So if reset can fail, I don't see reason ot break device only for
> > suspend failure.
> >
>
> Sorry I still don't get you.
>
> This series does not implement suspend at all. It doesn't modify the
> VDUSE device reset or the virtio reset behavior. It only implements
> the vq ready message for the device. If the device returns an error
> from that operation, what is your proposal for when the driver sends
> new messages like resume?
>

Friendly ping.

Re: [PATCH v3 3/3] vduse: add F_QUEUE_READY feature

[Michael S. Tsirkin]

On Tue, Mar 24, 2026 at 03:01:47PM +0100, Eugenio Perez Martin wrote:
> On Fri, Mar 13, 2026 at 8:08 AM Eugenio Perez Martin
> <eperezma@redhat.com> wrote:
> >
> > On Fri, Mar 13, 2026 at 7:05 AM Jason Wang <jasowang@redhat.com> wrote:
> > >
> > > On Thu, Mar 12, 2026 at 2:24 PM Eugenio Perez Martin
> > > <eperezma@redhat.com> wrote:
> > > >
> > > > On Thu, Mar 12, 2026 at 4:58 AM Jason Wang <jasowang@redhat.com> wrote:
> > > > >
> > > > > On Wed, Mar 11, 2026 at 3:08 AM Eugenio Pérez <eperezma@redhat.com> wrote:
> > > > > >
> > > > > > Add the VDUSE_F_QUEUE_READY feature flag. This allows the kernel module
> > > > > > to explicitly signal userspace when a specific virtqueue has been
> > > > > > enabled.
> > > > > >
> > > > > > In scenarios like Live Migration of VirtIO net devices, the dataplane
> > > > > > starts after the control virtqueue allowing QEMU to apply configuration
> > > > > > in the destination device.
> > > > > >
> > > > > > Signed-off-by: Eugenio Pérez <eperezma@redhat.com>
> > > > > > ---
> > > > > > v2:
> > > > > > * Fix comment of vduse_dev_request.vq_ready
> > > > > > * Set vq_ready before sending the message to the VDUSE userland
> > > > > >   instance, avoiding the need for SMP sync after receiving the message.
> > > > > > ---
> > > > > >  drivers/vdpa/vdpa_user/vduse_dev.c | 28 +++++++++++++++++++++++++++-
> > > > > >  include/uapi/linux/vduse.h         | 18 ++++++++++++++++++
> > > > > >  2 files changed, 45 insertions(+), 1 deletion(-)
> > > > > >
> > > > > > diff --git a/drivers/vdpa/vdpa_user/vduse_dev.c b/drivers/vdpa/vdpa_user/vduse_dev.c
> > > > > > index 17e0358d3a68..4f642b95a7cb 100644
> > > > > > --- a/drivers/vdpa/vdpa_user/vduse_dev.c
> > > > > > +++ b/drivers/vdpa/vdpa_user/vduse_dev.c
> > > > > > @@ -9,6 +9,7 @@
> > > > > >   */
> > > > > >
> > > > > >  #include "linux/virtio_net.h"
> > > > > > +#include <linux/bits.h>
> > > > > >  #include <linux/cleanup.h>
> > > > > >  #include <linux/init.h>
> > > > > >  #include <linux/module.h>
> > > > > > @@ -53,7 +54,7 @@
> > > > > >  #define IRQ_UNBOUND -1
> > > > > >
> > > > > >  /* Supported VDUSE features */
> > > > > > -static const uint64_t vduse_features;
> > > > > > +static const uint64_t vduse_features = BIT_U64(VDUSE_F_QUEUE_READY);
> > > > > >
> > > > > >  /*
> > > > > >   * VDUSE instance have not asked the vduse API version, so assume 0.
> > > > > > @@ -120,6 +121,7 @@ struct vduse_dev {
> > > > > >         char *name;
> > > > > >         struct mutex lock;
> > > > > >         spinlock_t msg_lock;
> > > > > > +       u64 vduse_features;
> > > > > >         u64 msg_unique;
> > > > > >         u32 msg_timeout;
> > > > > >         wait_queue_head_t waitq;
> > > > > > @@ -601,8 +603,29 @@ static void vduse_vdpa_set_vq_ready(struct vdpa_device *vdpa,
> > > > > >  {
> > > > > >         struct vduse_dev *dev = vdpa_to_vduse(vdpa);
> > > > > >         struct vduse_virtqueue *vq = dev->vqs[idx];
> > > > > > +       struct vduse_dev_msg msg = { 0 };
> > > > > > +       int r;
> > > > > >
> > > > > >         vq->ready = ready;
> > > > > > +
> > > > > > +       if (!(dev->vduse_features & BIT_U64(VDUSE_F_QUEUE_READY)))
> > > > > > +               return;
> > > > > > +
> > > > > > +       msg.req.type = VDUSE_SET_VQ_READY;
> > > > > > +       msg.req.vq_ready.num = idx;
> > > > > > +       msg.req.vq_ready.ready = !!ready;
> > > > > > +
> > > > > > +       r = vduse_dev_msg_sync(dev, &msg);
> > > > > > +
> > > > > > +       if (r < 0) {
> > > > > > +               dev_dbg(&vdpa->dev, "device refuses to set vq %u ready %u",
> > > > > > +                       idx, ready);
> > > > > > +
> > > > > > +               /* We can't do better than break the device in this case */
> > > > >
> > > > > It's better to explain why we can't depend on vduse_dev_msg_sync() here.
> > > > >
> > > > > For example it did:
> > > > >
> > > > >         if (unlikely(dev->broken))
> > > > >                 return -EIO;
> > > > >
> > > > >         init_waitqueue_head(&msg->waitq);
> > > > >         spin_lock(&dev->msg_lock);
> > > > >         if (unlikely(dev->broken)) {
> > > > >                 spin_unlock(&dev->msg_lock);
> > > > >                 return -EIO;
> > > > >         }
> > > > >
> > > > > and
> > > > >
> > > > >         if (!msg->completed) {
> > > > >                 list_del(&msg->list);
> > > > >                 msg->resp.result = VDUSE_REQ_RESULT_FAILED;
> > > > >                 /* Mark the device as malfunction when there is a timeout */
> > > > >                 if (!ret)
> > > > >                         vduse_dev_broken(dev);
> > > > >         }
> > > > >
> > > >
> > > > I'm not sure I follow you here.
> > > >
> > > > We can't do better than breaking the device because the function
> > > > returns no error state, and the caller does not expect an error code.
> > > > Do you mean we can't depend on vduse_dev_msg_sync to call
> > > > vduse_dev_broken(dev) by itself?
> > >
> > > I think I meant, reset seems to be more heavyweight than suspend.
> > >
> > > So if reset can fail, I don't see reason ot break device only for
> > > suspend failure.
> > >
> >
> > Sorry I still don't get you.
> >
> > This series does not implement suspend at all. It doesn't modify the
> > VDUSE device reset or the virtio reset behavior. It only implements
> > the vq ready message for the device. If the device returns an error
> > from that operation, what is your proposal for when the driver sends
> > new messages like resume?
> >
> 
> Friendly ping.

Jason, more comments? If this is to go in it has to go into linux next.

-- 
MST

Re: [PATCH v3 3/3] vduse: add F_QUEUE_READY feature

[Jason Wang]

On Tue, Mar 24, 2026 at 11:24 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Tue, Mar 24, 2026 at 03:01:47PM +0100, Eugenio Perez Martin wrote:
> > On Fri, Mar 13, 2026 at 8:08 AM Eugenio Perez Martin
> > <eperezma@redhat.com> wrote:
> > >
> > > On Fri, Mar 13, 2026 at 7:05 AM Jason Wang <jasowang@redhat.com> wrote:
> > > >
> > > > On Thu, Mar 12, 2026 at 2:24 PM Eugenio Perez Martin
> > > > <eperezma@redhat.com> wrote:
> > > > >
> > > > > On Thu, Mar 12, 2026 at 4:58 AM Jason Wang <jasowang@redhat.com> wrote:
> > > > > >
> > > > > > On Wed, Mar 11, 2026 at 3:08 AM Eugenio Pérez <eperezma@redhat.com> wrote:
> > > > > > >
> > > > > > > Add the VDUSE_F_QUEUE_READY feature flag. This allows the kernel module
> > > > > > > to explicitly signal userspace when a specific virtqueue has been
> > > > > > > enabled.
> > > > > > >
> > > > > > > In scenarios like Live Migration of VirtIO net devices, the dataplane
> > > > > > > starts after the control virtqueue allowing QEMU to apply configuration
> > > > > > > in the destination device.
> > > > > > >
> > > > > > > Signed-off-by: Eugenio Pérez <eperezma@redhat.com>
> > > > > > > ---
> > > > > > > v2:
> > > > > > > * Fix comment of vduse_dev_request.vq_ready
> > > > > > > * Set vq_ready before sending the message to the VDUSE userland
> > > > > > >   instance, avoiding the need for SMP sync after receiving the message.
> > > > > > > ---
> > > > > > >  drivers/vdpa/vdpa_user/vduse_dev.c | 28 +++++++++++++++++++++++++++-
> > > > > > >  include/uapi/linux/vduse.h         | 18 ++++++++++++++++++
> > > > > > >  2 files changed, 45 insertions(+), 1 deletion(-)
> > > > > > >
> > > > > > > diff --git a/drivers/vdpa/vdpa_user/vduse_dev.c b/drivers/vdpa/vdpa_user/vduse_dev.c
> > > > > > > index 17e0358d3a68..4f642b95a7cb 100644
> > > > > > > --- a/drivers/vdpa/vdpa_user/vduse_dev.c
> > > > > > > +++ b/drivers/vdpa/vdpa_user/vduse_dev.c
> > > > > > > @@ -9,6 +9,7 @@
> > > > > > >   */
> > > > > > >
> > > > > > >  #include "linux/virtio_net.h"
> > > > > > > +#include <linux/bits.h>
> > > > > > >  #include <linux/cleanup.h>
> > > > > > >  #include <linux/init.h>
> > > > > > >  #include <linux/module.h>
> > > > > > > @@ -53,7 +54,7 @@
> > > > > > >  #define IRQ_UNBOUND -1
> > > > > > >
> > > > > > >  /* Supported VDUSE features */
> > > > > > > -static const uint64_t vduse_features;
> > > > > > > +static const uint64_t vduse_features = BIT_U64(VDUSE_F_QUEUE_READY);
> > > > > > >
> > > > > > >  /*
> > > > > > >   * VDUSE instance have not asked the vduse API version, so assume 0.
> > > > > > > @@ -120,6 +121,7 @@ struct vduse_dev {
> > > > > > >         char *name;
> > > > > > >         struct mutex lock;
> > > > > > >         spinlock_t msg_lock;
> > > > > > > +       u64 vduse_features;
> > > > > > >         u64 msg_unique;
> > > > > > >         u32 msg_timeout;
> > > > > > >         wait_queue_head_t waitq;
> > > > > > > @@ -601,8 +603,29 @@ static void vduse_vdpa_set_vq_ready(struct vdpa_device *vdpa,
> > > > > > >  {
> > > > > > >         struct vduse_dev *dev = vdpa_to_vduse(vdpa);
> > > > > > >         struct vduse_virtqueue *vq = dev->vqs[idx];
> > > > > > > +       struct vduse_dev_msg msg = { 0 };
> > > > > > > +       int r;
> > > > > > >
> > > > > > >         vq->ready = ready;
> > > > > > > +
> > > > > > > +       if (!(dev->vduse_features & BIT_U64(VDUSE_F_QUEUE_READY)))
> > > > > > > +               return;
> > > > > > > +
> > > > > > > +       msg.req.type = VDUSE_SET_VQ_READY;
> > > > > > > +       msg.req.vq_ready.num = idx;
> > > > > > > +       msg.req.vq_ready.ready = !!ready;
> > > > > > > +
> > > > > > > +       r = vduse_dev_msg_sync(dev, &msg);
> > > > > > > +
> > > > > > > +       if (r < 0) {
> > > > > > > +               dev_dbg(&vdpa->dev, "device refuses to set vq %u ready %u",
> > > > > > > +                       idx, ready);
> > > > > > > +
> > > > > > > +               /* We can't do better than break the device in this case */
> > > > > >
> > > > > > It's better to explain why we can't depend on vduse_dev_msg_sync() here.
> > > > > >
> > > > > > For example it did:
> > > > > >
> > > > > >         if (unlikely(dev->broken))
> > > > > >                 return -EIO;
> > > > > >
> > > > > >         init_waitqueue_head(&msg->waitq);
> > > > > >         spin_lock(&dev->msg_lock);
> > > > > >         if (unlikely(dev->broken)) {
> > > > > >                 spin_unlock(&dev->msg_lock);
> > > > > >                 return -EIO;
> > > > > >         }
> > > > > >
> > > > > > and
> > > > > >
> > > > > >         if (!msg->completed) {
> > > > > >                 list_del(&msg->list);
> > > > > >                 msg->resp.result = VDUSE_REQ_RESULT_FAILED;
> > > > > >                 /* Mark the device as malfunction when there is a timeout */
> > > > > >                 if (!ret)
> > > > > >                         vduse_dev_broken(dev);
> > > > > >         }
> > > > > >
> > > > >
> > > > > I'm not sure I follow you here.
> > > > >
> > > > > We can't do better than breaking the device because the function
> > > > > returns no error state, and the caller does not expect an error code.
> > > > > Do you mean we can't depend on vduse_dev_msg_sync to call
> > > > > vduse_dev_broken(dev) by itself?
> > > >
> > > > I think I meant, reset seems to be more heavyweight than suspend.
> > > >
> > > > So if reset can fail, I don't see reason ot break device only for
> > > > suspend failure.
> > > >
> > >
> > > Sorry I still don't get you.
> > >
> > > This series does not implement suspend at all. It doesn't modify the
> > > VDUSE device reset or the virtio reset behavior. It only implements
> > > the vq ready message for the device. If the device returns an error
> > > from that operation, what is your proposal for when the driver sends
> > > new messages like resume?
> > >
> >
> > Friendly ping.
>
> Jason, more comments? If this is to go in it has to go into linux next.

A typo, basically I meant that reset is more heavyweight than queue
ready. If we decide to check the response for queue ready, we need to
check the reset as well.

But I'm fine if you think we can start from this.

Thanks

>
> --
> MST
>

Re: [PATCH v3 3/3] vduse: add F_QUEUE_READY feature

[Eugenio Perez Martin]

On Thu, Mar 26, 2026 at 3:44 AM Jason Wang <jasowang@redhat.com> wrote:
>
> On Tue, Mar 24, 2026 at 11:24 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Tue, Mar 24, 2026 at 03:01:47PM +0100, Eugenio Perez Martin wrote:
> > > On Fri, Mar 13, 2026 at 8:08 AM Eugenio Perez Martin
> > > <eperezma@redhat.com> wrote:
> > > >
> > > > On Fri, Mar 13, 2026 at 7:05 AM Jason Wang <jasowang@redhat.com> wrote:
> > > > >
> > > > > On Thu, Mar 12, 2026 at 2:24 PM Eugenio Perez Martin
> > > > > <eperezma@redhat.com> wrote:
> > > > > >
> > > > > > On Thu, Mar 12, 2026 at 4:58 AM Jason Wang <jasowang@redhat.com> wrote:
> > > > > > >
> > > > > > > On Wed, Mar 11, 2026 at 3:08 AM Eugenio Pérez <eperezma@redhat.com> wrote:
> > > > > > > >
> > > > > > > > Add the VDUSE_F_QUEUE_READY feature flag. This allows the kernel module
> > > > > > > > to explicitly signal userspace when a specific virtqueue has been
> > > > > > > > enabled.
> > > > > > > >
> > > > > > > > In scenarios like Live Migration of VirtIO net devices, the dataplane
> > > > > > > > starts after the control virtqueue allowing QEMU to apply configuration
> > > > > > > > in the destination device.
> > > > > > > >
> > > > > > > > Signed-off-by: Eugenio Pérez <eperezma@redhat.com>
> > > > > > > > ---
> > > > > > > > v2:
> > > > > > > > * Fix comment of vduse_dev_request.vq_ready
> > > > > > > > * Set vq_ready before sending the message to the VDUSE userland
> > > > > > > >   instance, avoiding the need for SMP sync after receiving the message.
> > > > > > > > ---
> > > > > > > >  drivers/vdpa/vdpa_user/vduse_dev.c | 28 +++++++++++++++++++++++++++-
> > > > > > > >  include/uapi/linux/vduse.h         | 18 ++++++++++++++++++
> > > > > > > >  2 files changed, 45 insertions(+), 1 deletion(-)
> > > > > > > >
> > > > > > > > diff --git a/drivers/vdpa/vdpa_user/vduse_dev.c b/drivers/vdpa/vdpa_user/vduse_dev.c
> > > > > > > > index 17e0358d3a68..4f642b95a7cb 100644
> > > > > > > > --- a/drivers/vdpa/vdpa_user/vduse_dev.c
> > > > > > > > +++ b/drivers/vdpa/vdpa_user/vduse_dev.c
> > > > > > > > @@ -9,6 +9,7 @@
> > > > > > > >   */
> > > > > > > >
> > > > > > > >  #include "linux/virtio_net.h"
> > > > > > > > +#include <linux/bits.h>
> > > > > > > >  #include <linux/cleanup.h>
> > > > > > > >  #include <linux/init.h>
> > > > > > > >  #include <linux/module.h>
> > > > > > > > @@ -53,7 +54,7 @@
> > > > > > > >  #define IRQ_UNBOUND -1
> > > > > > > >
> > > > > > > >  /* Supported VDUSE features */
> > > > > > > > -static const uint64_t vduse_features;
> > > > > > > > +static const uint64_t vduse_features = BIT_U64(VDUSE_F_QUEUE_READY);
> > > > > > > >
> > > > > > > >  /*
> > > > > > > >   * VDUSE instance have not asked the vduse API version, so assume 0.
> > > > > > > > @@ -120,6 +121,7 @@ struct vduse_dev {
> > > > > > > >         char *name;
> > > > > > > >         struct mutex lock;
> > > > > > > >         spinlock_t msg_lock;
> > > > > > > > +       u64 vduse_features;
> > > > > > > >         u64 msg_unique;
> > > > > > > >         u32 msg_timeout;
> > > > > > > >         wait_queue_head_t waitq;
> > > > > > > > @@ -601,8 +603,29 @@ static void vduse_vdpa_set_vq_ready(struct vdpa_device *vdpa,
> > > > > > > >  {
> > > > > > > >         struct vduse_dev *dev = vdpa_to_vduse(vdpa);
> > > > > > > >         struct vduse_virtqueue *vq = dev->vqs[idx];
> > > > > > > > +       struct vduse_dev_msg msg = { 0 };
> > > > > > > > +       int r;
> > > > > > > >
> > > > > > > >         vq->ready = ready;
> > > > > > > > +
> > > > > > > > +       if (!(dev->vduse_features & BIT_U64(VDUSE_F_QUEUE_READY)))
> > > > > > > > +               return;
> > > > > > > > +
> > > > > > > > +       msg.req.type = VDUSE_SET_VQ_READY;
> > > > > > > > +       msg.req.vq_ready.num = idx;
> > > > > > > > +       msg.req.vq_ready.ready = !!ready;
> > > > > > > > +
> > > > > > > > +       r = vduse_dev_msg_sync(dev, &msg);
> > > > > > > > +
> > > > > > > > +       if (r < 0) {
> > > > > > > > +               dev_dbg(&vdpa->dev, "device refuses to set vq %u ready %u",
> > > > > > > > +                       idx, ready);
> > > > > > > > +
> > > > > > > > +               /* We can't do better than break the device in this case */
> > > > > > >
> > > > > > > It's better to explain why we can't depend on vduse_dev_msg_sync() here.
> > > > > > >
> > > > > > > For example it did:
> > > > > > >
> > > > > > >         if (unlikely(dev->broken))
> > > > > > >                 return -EIO;
> > > > > > >
> > > > > > >         init_waitqueue_head(&msg->waitq);
> > > > > > >         spin_lock(&dev->msg_lock);
> > > > > > >         if (unlikely(dev->broken)) {
> > > > > > >                 spin_unlock(&dev->msg_lock);
> > > > > > >                 return -EIO;
> > > > > > >         }
> > > > > > >
> > > > > > > and
> > > > > > >
> > > > > > >         if (!msg->completed) {
> > > > > > >                 list_del(&msg->list);
> > > > > > >                 msg->resp.result = VDUSE_REQ_RESULT_FAILED;
> > > > > > >                 /* Mark the device as malfunction when there is a timeout */
> > > > > > >                 if (!ret)
> > > > > > >                         vduse_dev_broken(dev);
> > > > > > >         }
> > > > > > >
> > > > > >
> > > > > > I'm not sure I follow you here.
> > > > > >
> > > > > > We can't do better than breaking the device because the function
> > > > > > returns no error state, and the caller does not expect an error code.
> > > > > > Do you mean we can't depend on vduse_dev_msg_sync to call
> > > > > > vduse_dev_broken(dev) by itself?
> > > > >
> > > > > I think I meant, reset seems to be more heavyweight than suspend.
> > > > >
> > > > > So if reset can fail, I don't see reason ot break device only for
> > > > > suspend failure.
> > > > >
> > > >
> > > > Sorry I still don't get you.
> > > >
> > > > This series does not implement suspend at all. It doesn't modify the
> > > > VDUSE device reset or the virtio reset behavior. It only implements
> > > > the vq ready message for the device. If the device returns an error
> > > > from that operation, what is your proposal for when the driver sends
> > > > new messages like resume?
> > > >
> > >
> > > Friendly ping.
> >
> > Jason, more comments? If this is to go in it has to go into linux next.
>
> A typo, basically I meant that reset is more heavyweight than queue
> ready. If we decide to check the response for queue ready, we need to
> check the reset as well.
>
> But I'm fine if you think we can start from this.
>

If you mean something like this:
diff --git a/drivers/vdpa/vdpa_user/vduse_dev.c
b/drivers/vdpa/vdpa_user/vduse_dev.c
index 6202f6902fcd..23b850023417 100644
--- a/drivers/vdpa/vdpa_user/vduse_dev.c
+++ b/drivers/vdpa/vdpa_user/vduse_dev.c
@@ -816,7 +816,8 @@ static int vduse_vdpa_reset(struct vdpa_device *vdpa)
        struct vduse_dev *dev = vdpa_to_vduse(vdpa);
        int ret = vduse_dev_set_status(dev, 0);

-       vduse_dev_reset(dev);
+       if (ret == 0)
+               vduse_dev_reset(dev);

        return ret;
 }

---

I totally agree. But in my opinion, it's outside this series' scope. I
can send a patch on top.