* Re: [virtio-dev] [RFC PATCH v2] docs/interop: define PROBE feature for vhost-user VirtIO devices
@ 2023-09-05 9:34 ` Alex Bennée
0 siblings, 0 replies; 27+ messages in thread
From: Alex Bennée @ 2023-09-05 9:34 UTC (permalink / raw)
To: Albert Esteve
Cc: slp, mst, marcandre.lureau, stefanha, viresh.kumar, sgarzare,
takahiro.akashi, erik.schilling, manos.pitsidianakis,
mathieu.poirier, qemu-devel, virtio-dev, virtio-comment
Albert Esteve <aesteve@redhat.com> writes:
> This looks great! Thanks for this proposal.
>
> On Fri, Sep 1, 2023 at 1:00 PM Alex Bennée <alex.bennee@linaro.org> wrote:
>
> Currently QEMU has to know some details about the VirtIO device
> supported by a vhost-user daemon to be able to setup the guest. This
> makes it hard for QEMU to add support for additional vhost-user
> daemons without adding specific stubs for each additional VirtIO
> device.
>
> This patch suggests a new feature flag (VHOST_USER_PROTOCOL_F_PROBE)
> which the back-end can advertise which allows a probe message to be
> sent to get all the details QEMU needs to know in one message.
>
> Together with the existing features VHOST_USER_PROTOCOL_F_STATUS and
> VHOST_USER_PROTOCOL_F_CONFIG we can create "standalone" vhost-user
> daemons which are capable of handling all aspects of the VirtIO
> transactions with only a generic stub on the QEMU side. These daemons
> can also be used without QEMU in situations where there isn't a full
> VMM managing their setup.
>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>
> ---
> v2
> - dropped F_STANDALONE in favour of F_PROBE
> - split probe details across several messages
> - probe messages don't automatically imply a standalone daemon
> - add wording where probe details interact (F_MQ/F_CONFIG)
> - define VMM and make clear QEMU is only one of many potential VMMs
> - reword commit message
> ---
> docs/interop/vhost-user.rst | 90 ++++++++++++++++++++++++++++++++-----
> hw/virtio/vhost-user.c | 8 ++++
> 2 files changed, 88 insertions(+), 10 deletions(-)
>
> diff --git a/docs/interop/vhost-user.rst b/docs/interop/vhost-user.rst
> index 5a070adbc1..ba3b5e07b7 100644
> --- a/docs/interop/vhost-user.rst
> +++ b/docs/interop/vhost-user.rst
> @@ -7,6 +7,7 @@ Vhost-user Protocol
> ..
> Copyright 2014 Virtual Open Systems Sarl.
> Copyright 2019 Intel Corporation
> + Copyright 2023 Linaro Ltd
> Licence: This work is licensed under the terms of the GNU GPL,
> version 2 or later. See the COPYING file in the top-level
> directory.
> @@ -27,17 +28,31 @@ The protocol defines 2 sides of the communication, *front-end* and
> *back-end*. The *front-end* is the application that shares its virtqueues, in
> our case QEMU. The *back-end* is the consumer of the virtqueues.
>
> -In the current implementation QEMU is the *front-end*, and the *back-end*
> -is the external process consuming the virtio queues, for example a
> -software Ethernet switch running in user space, such as Snabbswitch,
> -or a block device back-end processing read & write to a virtual
> -disk. In order to facilitate interoperability between various back-end
> -implementations, it is recommended to follow the :ref:`Backend program
> -conventions <backend_conventions>`.
> +In the current implementation a Virtual Machine Manager (VMM) such as
> +QEMU is the *front-end*, and the *back-end* is the external process
> +consuming the virtio queues, for example a software Ethernet switch
> +running in user space, such as Snabbswitch, or a block device back-end
> +processing read & write to a virtual disk. In order to facilitate
> +interoperability between various back-end implementations, it is
> +recommended to follow the :ref:`Backend program conventions
> +<backend_conventions>`.
>
> The *front-end* and *back-end* can be either a client (i.e. connecting) or
> server (listening) in the socket communication.
>
> +Probing device details
> +----------------------
> +
> +Traditionally the vhost-user daemon *back-end* shares configuration
> +responsibilities with the VMM *front-end* which needs to know certain
> +key bits of information about the device. This means the VMM needs to
> +define at least a minimal stub for each VirtIO device it wants to
> +support. If the daemon supports the right set of protocol features the
> +VMM can probe the daemon for the information it needs to setup the
> +device. See :ref:`Probing features for standalone daemons
> +<probing_features>` for more details.
> +
> +
> Support for platforms other than Linux
> --------------------------------------
>
> @@ -316,6 +331,7 @@ replies. Here is a list of the ones that do:
> * ``VHOST_USER_GET_VRING_BASE``
> * ``VHOST_USER_SET_LOG_BASE`` (if ``VHOST_USER_PROTOCOL_F_LOG_SHMFD``)
> * ``VHOST_USER_GET_INFLIGHT_FD`` (if ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD``)
> +* ``VHOST_USER_GET_BACKEND_SPECS`` (if ``VHOST_USER_PROTOCOL_F_STANDALONE``)
>
> .. seealso::
>
> @@ -396,9 +412,10 @@ must support changing some configuration aspects on the fly.
> Multiple queue support
> ----------------------
>
> -Many devices have a fixed number of virtqueues. In this case the front-end
> -already knows the number of available virtqueues without communicating with the
> -back-end.
> +Many devices have a fixed number of virtqueues. In this case the
> +*front-end* usually already knows the number of available virtqueues
> +without communicating with the back-end. For standalone daemons this
> +number can be can be probed with the ``VHOST_USER_GET_MIN_VQ`` message.
>
> Some devices do not have a fixed number of virtqueues. Instead the maximum
> number of virtqueues is chosen by the back-end. The number can depend on host
> @@ -885,6 +902,23 @@ Protocol features
> #define VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS 15
> #define VHOST_USER_PROTOCOL_F_STATUS 16
> #define VHOST_USER_PROTOCOL_F_XEN_MMAP 17
> + #define VHOST_USER_PROTOCOL_F_PROBE 18
> +
> +.. _probing_features:
> +
> +Probing features for standalone daemons
> +---------------------------------------
> +
> +The protocol feature ``VHOST_USER_PROTOCOL_F_PROBE`` enables a number
> +of additional messages which allow the *front-end* to probe details
> +about the VirtIO device from the *back-end*. However for a *back-end*
> +to be described as standalone it must also support:
> +
> + * ``VHOST_USER_PROTOCOL_F_STATUS``
> + * ``VHOST_USER_PROTOCOL_F_CONFIG`` (if there is a config space)
> +
> +which are required to ensure the *back-end* daemon can operate
> +without the *front-end* managing some aspects of its configuration.
>
> Front-end message types
> -----------------------
> @@ -1440,6 +1474,42 @@ Front-end message types
> query the back-end for its device status as defined in the Virtio
> specification.
>
> +``VHOST_USER_GET_DEVICE_ID``
> + :id: 41
> + :request payload: N/A
> + :reply payload: ``u32``
> +
> + When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
> + successfully negotiated, this message is submitted by the front-end
> + to query what VirtIO device the back-end support. This is intended
> + to remove the need for the front-end to know ahead of time what the
> + VirtIO device the backend emulates is.
> +
> +``VHOST_USER_GET_CONFIG_SIZE``
> + :id: 42
> + :request payload: N/A
> + :reply payload: ``u32``
> +
> + When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
> + successfully negotiated, this message is submitted by the front-end
> + to query the size of the VirtIO device's config space. This is
> + intended to remove the need for the front-end to know ahead of time
> + what the size is. Replying with 0 when
> + ``VHOST_USER_PROTOCOL_F_CONFIG`` has been negotiated would indicate
> + an bug.
> +
> +``VHOST_USER_GET_MIN_VQ``
> + :id: 43
> + :request payload: N/A
> + :reply payload: ``u32``
> +
> + When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
> + successfully negotiated, this message is submitted by the front-end to
> + query minimum number of VQ's required to support the device. A
> + device may support more than this number of VQ's if it advertises
> + the ``VHOST_USER_PROTOCOL_F_MQ`` protocol feature. Reporting a
> + number greater than the result of ``VHOST_USER_GET_QUEUE_NUM`` would
> + indicate a bug.
>
> Maybe I lack some background, but not sure what min_vq is here?
There will be a minimum number of queues you need to support the device.
For example the virtio-sound spec specifies you need four queues:
control, event, tx, rx
> This looks like quering the number of VQs the backend requires/uses.
> Which, in case of MQ, it may be bigger (which is where I assume comes the `min`
> part, if we consider `VHOST_USER_GET_QUEUE_NUM` the `max`).
The MQ extension is currently used by networking but in theory any
device could attempt to parallelism by extending the number of virt
queues needed. So for net you get:
receiveq1
transmitq1
optional controlq
So VHOST_USER_GET_MIN_VQ would report 2 or 3 (if VIRTIO_NET_F_CTRL_VQ is
negotiated). However VHOST_USER_GET_QUEUE_NUM is only usable if
VIRTIO_NET_F_MQ has been negotiated and could report more.
>
> Couldn't we reuse the `VHOST_USER_GET_QUEUE_NUM` type for this?
>
> Back-end message types
> ----------------------
> diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c
> index 8dcf049d42..4d433cdf2b 100644
> --- a/hw/virtio/vhost-user.c
> +++ b/hw/virtio/vhost-user.c
> @@ -202,6 +202,13 @@ typedef struct VhostUserInflight {
> uint16_t queue_size;
> } VhostUserInflight;
>
> +typedef struct VhostUserBackendSpecs {
> + uint32_t device_id;
> + uint32_t config_size;
> + uint32_t min_vqs;
> + uint32_t max_vqs;
> +} VhostUserBackendSpecs;
> +
> typedef struct {
> VhostUserRequest request;
>
> @@ -226,6 +233,7 @@ typedef union {
> VhostUserCryptoSession session;
> VhostUserVringArea area;
> VhostUserInflight inflight;
> + VhostUserBackendSpecs specs;
Oops these snuck in, I shall clean them up
> } VhostUserPayload;
>
> typedef struct VhostUserMsg {
> --
> 2.39.2
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
--
Alex Bennée
Virtualisation Tech Lead @ Linaro
^ permalink raw reply [flat|nested] 27+ messages in thread* Re: [virtio-dev] [RFC PATCH v2] docs/interop: define PROBE feature for vhost-user VirtIO devices
@ 2023-09-05 9:34 ` Alex Bennée
0 siblings, 0 replies; 27+ messages in thread
From: Alex Bennée @ 2023-09-05 9:34 UTC (permalink / raw)
To: Albert Esteve
Cc: slp, mst, marcandre.lureau, stefanha, viresh.kumar, sgarzare,
takahiro.akashi, erik.schilling, manos.pitsidianakis,
mathieu.poirier, qemu-devel, virtio-dev, virtio-comment
Albert Esteve <aesteve@redhat.com> writes:
> This looks great! Thanks for this proposal.
>
> On Fri, Sep 1, 2023 at 1:00 PM Alex Bennée <alex.bennee@linaro.org> wrote:
>
> Currently QEMU has to know some details about the VirtIO device
> supported by a vhost-user daemon to be able to setup the guest. This
> makes it hard for QEMU to add support for additional vhost-user
> daemons without adding specific stubs for each additional VirtIO
> device.
>
> This patch suggests a new feature flag (VHOST_USER_PROTOCOL_F_PROBE)
> which the back-end can advertise which allows a probe message to be
> sent to get all the details QEMU needs to know in one message.
>
> Together with the existing features VHOST_USER_PROTOCOL_F_STATUS and
> VHOST_USER_PROTOCOL_F_CONFIG we can create "standalone" vhost-user
> daemons which are capable of handling all aspects of the VirtIO
> transactions with only a generic stub on the QEMU side. These daemons
> can also be used without QEMU in situations where there isn't a full
> VMM managing their setup.
>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>
> ---
> v2
> - dropped F_STANDALONE in favour of F_PROBE
> - split probe details across several messages
> - probe messages don't automatically imply a standalone daemon
> - add wording where probe details interact (F_MQ/F_CONFIG)
> - define VMM and make clear QEMU is only one of many potential VMMs
> - reword commit message
> ---
> docs/interop/vhost-user.rst | 90 ++++++++++++++++++++++++++++++++-----
> hw/virtio/vhost-user.c | 8 ++++
> 2 files changed, 88 insertions(+), 10 deletions(-)
>
> diff --git a/docs/interop/vhost-user.rst b/docs/interop/vhost-user.rst
> index 5a070adbc1..ba3b5e07b7 100644
> --- a/docs/interop/vhost-user.rst
> +++ b/docs/interop/vhost-user.rst
> @@ -7,6 +7,7 @@ Vhost-user Protocol
> ..
> Copyright 2014 Virtual Open Systems Sarl.
> Copyright 2019 Intel Corporation
> + Copyright 2023 Linaro Ltd
> Licence: This work is licensed under the terms of the GNU GPL,
> version 2 or later. See the COPYING file in the top-level
> directory.
> @@ -27,17 +28,31 @@ The protocol defines 2 sides of the communication, *front-end* and
> *back-end*. The *front-end* is the application that shares its virtqueues, in
> our case QEMU. The *back-end* is the consumer of the virtqueues.
>
> -In the current implementation QEMU is the *front-end*, and the *back-end*
> -is the external process consuming the virtio queues, for example a
> -software Ethernet switch running in user space, such as Snabbswitch,
> -or a block device back-end processing read & write to a virtual
> -disk. In order to facilitate interoperability between various back-end
> -implementations, it is recommended to follow the :ref:`Backend program
> -conventions <backend_conventions>`.
> +In the current implementation a Virtual Machine Manager (VMM) such as
> +QEMU is the *front-end*, and the *back-end* is the external process
> +consuming the virtio queues, for example a software Ethernet switch
> +running in user space, such as Snabbswitch, or a block device back-end
> +processing read & write to a virtual disk. In order to facilitate
> +interoperability between various back-end implementations, it is
> +recommended to follow the :ref:`Backend program conventions
> +<backend_conventions>`.
>
> The *front-end* and *back-end* can be either a client (i.e. connecting) or
> server (listening) in the socket communication.
>
> +Probing device details
> +----------------------
> +
> +Traditionally the vhost-user daemon *back-end* shares configuration
> +responsibilities with the VMM *front-end* which needs to know certain
> +key bits of information about the device. This means the VMM needs to
> +define at least a minimal stub for each VirtIO device it wants to
> +support. If the daemon supports the right set of protocol features the
> +VMM can probe the daemon for the information it needs to setup the
> +device. See :ref:`Probing features for standalone daemons
> +<probing_features>` for more details.
> +
> +
> Support for platforms other than Linux
> --------------------------------------
>
> @@ -316,6 +331,7 @@ replies. Here is a list of the ones that do:
> * ``VHOST_USER_GET_VRING_BASE``
> * ``VHOST_USER_SET_LOG_BASE`` (if ``VHOST_USER_PROTOCOL_F_LOG_SHMFD``)
> * ``VHOST_USER_GET_INFLIGHT_FD`` (if ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD``)
> +* ``VHOST_USER_GET_BACKEND_SPECS`` (if ``VHOST_USER_PROTOCOL_F_STANDALONE``)
>
> .. seealso::
>
> @@ -396,9 +412,10 @@ must support changing some configuration aspects on the fly.
> Multiple queue support
> ----------------------
>
> -Many devices have a fixed number of virtqueues. In this case the front-end
> -already knows the number of available virtqueues without communicating with the
> -back-end.
> +Many devices have a fixed number of virtqueues. In this case the
> +*front-end* usually already knows the number of available virtqueues
> +without communicating with the back-end. For standalone daemons this
> +number can be can be probed with the ``VHOST_USER_GET_MIN_VQ`` message.
>
> Some devices do not have a fixed number of virtqueues. Instead the maximum
> number of virtqueues is chosen by the back-end. The number can depend on host
> @@ -885,6 +902,23 @@ Protocol features
> #define VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS 15
> #define VHOST_USER_PROTOCOL_F_STATUS 16
> #define VHOST_USER_PROTOCOL_F_XEN_MMAP 17
> + #define VHOST_USER_PROTOCOL_F_PROBE 18
> +
> +.. _probing_features:
> +
> +Probing features for standalone daemons
> +---------------------------------------
> +
> +The protocol feature ``VHOST_USER_PROTOCOL_F_PROBE`` enables a number
> +of additional messages which allow the *front-end* to probe details
> +about the VirtIO device from the *back-end*. However for a *back-end*
> +to be described as standalone it must also support:
> +
> + * ``VHOST_USER_PROTOCOL_F_STATUS``
> + * ``VHOST_USER_PROTOCOL_F_CONFIG`` (if there is a config space)
> +
> +which are required to ensure the *back-end* daemon can operate
> +without the *front-end* managing some aspects of its configuration.
>
> Front-end message types
> -----------------------
> @@ -1440,6 +1474,42 @@ Front-end message types
> query the back-end for its device status as defined in the Virtio
> specification.
>
> +``VHOST_USER_GET_DEVICE_ID``
> + :id: 41
> + :request payload: N/A
> + :reply payload: ``u32``
> +
> + When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
> + successfully negotiated, this message is submitted by the front-end
> + to query what VirtIO device the back-end support. This is intended
> + to remove the need for the front-end to know ahead of time what the
> + VirtIO device the backend emulates is.
> +
> +``VHOST_USER_GET_CONFIG_SIZE``
> + :id: 42
> + :request payload: N/A
> + :reply payload: ``u32``
> +
> + When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
> + successfully negotiated, this message is submitted by the front-end
> + to query the size of the VirtIO device's config space. This is
> + intended to remove the need for the front-end to know ahead of time
> + what the size is. Replying with 0 when
> + ``VHOST_USER_PROTOCOL_F_CONFIG`` has been negotiated would indicate
> + an bug.
> +
> +``VHOST_USER_GET_MIN_VQ``
> + :id: 43
> + :request payload: N/A
> + :reply payload: ``u32``
> +
> + When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
> + successfully negotiated, this message is submitted by the front-end to
> + query minimum number of VQ's required to support the device. A
> + device may support more than this number of VQ's if it advertises
> + the ``VHOST_USER_PROTOCOL_F_MQ`` protocol feature. Reporting a
> + number greater than the result of ``VHOST_USER_GET_QUEUE_NUM`` would
> + indicate a bug.
>
> Maybe I lack some background, but not sure what min_vq is here?
There will be a minimum number of queues you need to support the device.
For example the virtio-sound spec specifies you need four queues:
control, event, tx, rx
> This looks like quering the number of VQs the backend requires/uses.
> Which, in case of MQ, it may be bigger (which is where I assume comes the `min`
> part, if we consider `VHOST_USER_GET_QUEUE_NUM` the `max`).
The MQ extension is currently used by networking but in theory any
device could attempt to parallelism by extending the number of virt
queues needed. So for net you get:
receiveq1
transmitq1
optional controlq
So VHOST_USER_GET_MIN_VQ would report 2 or 3 (if VIRTIO_NET_F_CTRL_VQ is
negotiated). However VHOST_USER_GET_QUEUE_NUM is only usable if
VIRTIO_NET_F_MQ has been negotiated and could report more.
>
> Couldn't we reuse the `VHOST_USER_GET_QUEUE_NUM` type for this?
>
> Back-end message types
> ----------------------
> diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c
> index 8dcf049d42..4d433cdf2b 100644
> --- a/hw/virtio/vhost-user.c
> +++ b/hw/virtio/vhost-user.c
> @@ -202,6 +202,13 @@ typedef struct VhostUserInflight {
> uint16_t queue_size;
> } VhostUserInflight;
>
> +typedef struct VhostUserBackendSpecs {
> + uint32_t device_id;
> + uint32_t config_size;
> + uint32_t min_vqs;
> + uint32_t max_vqs;
> +} VhostUserBackendSpecs;
> +
> typedef struct {
> VhostUserRequest request;
>
> @@ -226,6 +233,7 @@ typedef union {
> VhostUserCryptoSession session;
> VhostUserVringArea area;
> VhostUserInflight inflight;
> + VhostUserBackendSpecs specs;
Oops these snuck in, I shall clean them up
> } VhostUserPayload;
>
> typedef struct VhostUserMsg {
> --
> 2.39.2
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
--
Alex Bennée
Virtualisation Tech Lead @ Linaro
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 27+ messages in thread* Re: [virtio-dev] [RFC PATCH v2] docs/interop: define PROBE feature for vhost-user VirtIO devices
2023-09-05 9:34 ` Alex Bennée
(?)
(?)
@ 2023-09-05 10:02 ` Albert Esteve
-1 siblings, 0 replies; 27+ messages in thread
From: Albert Esteve @ 2023-09-05 10:02 UTC (permalink / raw)
To: Alex Bennée
Cc: slp, mst, marcandre.lureau, stefanha, viresh.kumar, sgarzare,
takahiro.akashi, erik.schilling, manos.pitsidianakis,
mathieu.poirier, qemu-devel, virtio-dev, virtio-comment
[-- Attachment #1: Type: text/plain, Size: 11216 bytes --]
On Tue, Sep 5, 2023 at 11:43 AM Alex Bennée <alex.bennee@linaro.org> wrote:
>
> Albert Esteve <aesteve@redhat.com> writes:
>
> > This looks great! Thanks for this proposal.
> >
> > On Fri, Sep 1, 2023 at 1:00 PM Alex Bennée <alex.bennee@linaro.org>
> wrote:
> >
> > Currently QEMU has to know some details about the VirtIO device
> > supported by a vhost-user daemon to be able to setup the guest. This
> > makes it hard for QEMU to add support for additional vhost-user
> > daemons without adding specific stubs for each additional VirtIO
> > device.
> >
> > This patch suggests a new feature flag (VHOST_USER_PROTOCOL_F_PROBE)
> > which the back-end can advertise which allows a probe message to be
> > sent to get all the details QEMU needs to know in one message.
> >
> > Together with the existing features VHOST_USER_PROTOCOL_F_STATUS and
> > VHOST_USER_PROTOCOL_F_CONFIG we can create "standalone" vhost-user
> > daemons which are capable of handling all aspects of the VirtIO
> > transactions with only a generic stub on the QEMU side. These daemons
> > can also be used without QEMU in situations where there isn't a full
> > VMM managing their setup.
> >
> > Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> >
> > ---
> > v2
> > - dropped F_STANDALONE in favour of F_PROBE
> > - split probe details across several messages
> > - probe messages don't automatically imply a standalone daemon
> > - add wording where probe details interact (F_MQ/F_CONFIG)
> > - define VMM and make clear QEMU is only one of many potential VMMs
> > - reword commit message
> > ---
> > docs/interop/vhost-user.rst | 90 ++++++++++++++++++++++++++++++++-----
> > hw/virtio/vhost-user.c | 8 ++++
> > 2 files changed, 88 insertions(+), 10 deletions(-)
> >
> > diff --git a/docs/interop/vhost-user.rst b/docs/interop/vhost-user.rst
> > index 5a070adbc1..ba3b5e07b7 100644
> > --- a/docs/interop/vhost-user.rst
> > +++ b/docs/interop/vhost-user.rst
> > @@ -7,6 +7,7 @@ Vhost-user Protocol
> > ..
> > Copyright 2014 Virtual Open Systems Sarl.
> > Copyright 2019 Intel Corporation
> > + Copyright 2023 Linaro Ltd
> > Licence: This work is licensed under the terms of the GNU GPL,
> > version 2 or later. See the COPYING file in the top-level
> > directory.
> > @@ -27,17 +28,31 @@ The protocol defines 2 sides of the communication,
> *front-end* and
> > *back-end*. The *front-end* is the application that shares its
> virtqueues, in
> > our case QEMU. The *back-end* is the consumer of the virtqueues.
> >
> > -In the current implementation QEMU is the *front-end*, and the
> *back-end*
> > -is the external process consuming the virtio queues, for example a
> > -software Ethernet switch running in user space, such as Snabbswitch,
> > -or a block device back-end processing read & write to a virtual
> > -disk. In order to facilitate interoperability between various back-end
> > -implementations, it is recommended to follow the :ref:`Backend program
> > -conventions <backend_conventions>`.
> > +In the current implementation a Virtual Machine Manager (VMM) such as
> > +QEMU is the *front-end*, and the *back-end* is the external process
> > +consuming the virtio queues, for example a software Ethernet switch
> > +running in user space, such as Snabbswitch, or a block device back-end
> > +processing read & write to a virtual disk. In order to facilitate
> > +interoperability between various back-end implementations, it is
> > +recommended to follow the :ref:`Backend program conventions
> > +<backend_conventions>`.
> >
> > The *front-end* and *back-end* can be either a client (i.e.
> connecting) or
> > server (listening) in the socket communication.
> >
> > +Probing device details
> > +----------------------
> > +
> > +Traditionally the vhost-user daemon *back-end* shares configuration
> > +responsibilities with the VMM *front-end* which needs to know certain
> > +key bits of information about the device. This means the VMM needs to
> > +define at least a minimal stub for each VirtIO device it wants to
> > +support. If the daemon supports the right set of protocol features the
> > +VMM can probe the daemon for the information it needs to setup the
> > +device. See :ref:`Probing features for standalone daemons
> > +<probing_features>` for more details.
> > +
> > +
> > Support for platforms other than Linux
> > --------------------------------------
> >
> > @@ -316,6 +331,7 @@ replies. Here is a list of the ones that do:
> > * ``VHOST_USER_GET_VRING_BASE``
> > * ``VHOST_USER_SET_LOG_BASE`` (if ``VHOST_USER_PROTOCOL_F_LOG_SHMFD``)
> > * ``VHOST_USER_GET_INFLIGHT_FD`` (if
> ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD``)
> > +* ``VHOST_USER_GET_BACKEND_SPECS`` (if
> ``VHOST_USER_PROTOCOL_F_STANDALONE``)
> >
> > .. seealso::
> >
> > @@ -396,9 +412,10 @@ must support changing some configuration aspects
> on the fly.
> > Multiple queue support
> > ----------------------
> >
> > -Many devices have a fixed number of virtqueues. In this case the
> front-end
> > -already knows the number of available virtqueues without communicating
> with the
> > -back-end.
> > +Many devices have a fixed number of virtqueues. In this case the
> > +*front-end* usually already knows the number of available virtqueues
> > +without communicating with the back-end. For standalone daemons this
> > +number can be can be probed with the ``VHOST_USER_GET_MIN_VQ`` message.
> >
> > Some devices do not have a fixed number of virtqueues. Instead the
> maximum
> > number of virtqueues is chosen by the back-end. The number can depend
> on host
> > @@ -885,6 +902,23 @@ Protocol features
> > #define VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS 15
> > #define VHOST_USER_PROTOCOL_F_STATUS 16
> > #define VHOST_USER_PROTOCOL_F_XEN_MMAP 17
> > + #define VHOST_USER_PROTOCOL_F_PROBE 18
> > +
> > +.. _probing_features:
> > +
> > +Probing features for standalone daemons
> > +---------------------------------------
> > +
> > +The protocol feature ``VHOST_USER_PROTOCOL_F_PROBE`` enables a number
> > +of additional messages which allow the *front-end* to probe details
> > +about the VirtIO device from the *back-end*. However for a *back-end*
> > +to be described as standalone it must also support:
> > +
> > + * ``VHOST_USER_PROTOCOL_F_STATUS``
> > + * ``VHOST_USER_PROTOCOL_F_CONFIG`` (if there is a config space)
> > +
> > +which are required to ensure the *back-end* daemon can operate
> > +without the *front-end* managing some aspects of its configuration.
> >
> > Front-end message types
> > -----------------------
> > @@ -1440,6 +1474,42 @@ Front-end message types
> > query the back-end for its device status as defined in the Virtio
> > specification.
> >
> > +``VHOST_USER_GET_DEVICE_ID``
> > + :id: 41
> > + :request payload: N/A
> > + :reply payload: ``u32``
> > +
> > + When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
> > + successfully negotiated, this message is submitted by the front-end
> > + to query what VirtIO device the back-end support. This is intended
> > + to remove the need for the front-end to know ahead of time what the
> > + VirtIO device the backend emulates is.
> > +
> > +``VHOST_USER_GET_CONFIG_SIZE``
> > + :id: 42
> > + :request payload: N/A
> > + :reply payload: ``u32``
> > +
> > + When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
> > + successfully negotiated, this message is submitted by the front-end
> > + to query the size of the VirtIO device's config space. This is
> > + intended to remove the need for the front-end to know ahead of time
> > + what the size is. Replying with 0 when
> > + ``VHOST_USER_PROTOCOL_F_CONFIG`` has been negotiated would indicate
> > + an bug.
> > +
> > +``VHOST_USER_GET_MIN_VQ``
> > + :id: 43
> > + :request payload: N/A
> > + :reply payload: ``u32``
> > +
> > + When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
> > + successfully negotiated, this message is submitted by the front-end
> to
> > + query minimum number of VQ's required to support the device. A
> > + device may support more than this number of VQ's if it advertises
> > + the ``VHOST_USER_PROTOCOL_F_MQ`` protocol feature. Reporting a
> > + number greater than the result of ``VHOST_USER_GET_QUEUE_NUM`` would
> > + indicate a bug.
> >
> > Maybe I lack some background, but not sure what min_vq is here?
>
> There will be a minimum number of queues you need to support the device.
> For example the virtio-sound spec specifies you need four queues:
> control, event, tx, rx
>
> > This looks like quering the number of VQs the backend requires/uses.
> > Which, in case of MQ, it may be bigger (which is where I assume comes
> the `min`
> > part, if we consider `VHOST_USER_GET_QUEUE_NUM` the `max`).
>
> The MQ extension is currently used by networking but in theory any
> device could attempt to parallelism by extending the number of virt
> queues needed. So for net you get:
>
> receiveq1
> transmitq1
> optional controlq
>
> So VHOST_USER_GET_MIN_VQ would report 2 or 3 (if VIRTIO_NET_F_CTRL_VQ is
> negotiated). However VHOST_USER_GET_QUEUE_NUM is only usable if
> VIRTIO_NET_F_MQ has been negotiated and could report more.
>
>
Ah I see, I understand it better now. It is a pity that we cannot
multipurpose
the GET_QUEUE_NUM request.
FWIW:
Acked-by: Albert Esteve <aesteve@redhat.com>
> >
> > Couldn't we reuse the `VHOST_USER_GET_QUEUE_NUM` type for this?
> >
> > Back-end message types
> > ----------------------
> > diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c
> > index 8dcf049d42..4d433cdf2b 100644
> > --- a/hw/virtio/vhost-user.c
> > +++ b/hw/virtio/vhost-user.c
> > @@ -202,6 +202,13 @@ typedef struct VhostUserInflight {
> > uint16_t queue_size;
> > } VhostUserInflight;
> >
> > +typedef struct VhostUserBackendSpecs {
> > + uint32_t device_id;
> > + uint32_t config_size;
> > + uint32_t min_vqs;
> > + uint32_t max_vqs;
> > +} VhostUserBackendSpecs;
> > +
> > typedef struct {
> > VhostUserRequest request;
> >
> > @@ -226,6 +233,7 @@ typedef union {
> > VhostUserCryptoSession session;
> > VhostUserVringArea area;
> > VhostUserInflight inflight;
> > + VhostUserBackendSpecs specs;
>
> Oops these snuck in, I shall clean them up
>
> > } VhostUserPayload;
> >
> > typedef struct VhostUserMsg {
> > --
> > 2.39.2
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> > For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
>
>
> --
> Alex Bennée
> Virtualisation Tech Lead @ Linaro
>
>
[-- Attachment #2: Type: text/html, Size: 13714 bytes --]
^ permalink raw reply [flat|nested] 27+ messages in thread* [virtio-comment] Re: [virtio-dev] [RFC PATCH v2] docs/interop: define PROBE feature for vhost-user VirtIO devices
2023-09-05 9:34 ` Alex Bennée
@ 2023-09-07 19:29 ` Stefan Hajnoczi
-1 siblings, 0 replies; 27+ messages in thread
From: Stefan Hajnoczi @ 2023-09-07 19:29 UTC (permalink / raw)
To: Alex Bennée
Cc: Albert Esteve, slp, mst, marcandre.lureau, viresh.kumar, sgarzare,
takahiro.akashi, erik.schilling, manos.pitsidianakis,
mathieu.poirier, qemu-devel, virtio-dev, virtio-comment
[-- Attachment #1: Type: text/plain, Size: 10072 bytes --]
On Tue, Sep 05, 2023 at 10:34:11AM +0100, Alex Bennée wrote:
>
> Albert Esteve <aesteve@redhat.com> writes:
>
> > This looks great! Thanks for this proposal.
> >
> > On Fri, Sep 1, 2023 at 1:00 PM Alex Bennée <alex.bennee@linaro.org> wrote:
> >
> > Currently QEMU has to know some details about the VirtIO device
> > supported by a vhost-user daemon to be able to setup the guest. This
> > makes it hard for QEMU to add support for additional vhost-user
> > daemons without adding specific stubs for each additional VirtIO
> > device.
> >
> > This patch suggests a new feature flag (VHOST_USER_PROTOCOL_F_PROBE)
> > which the back-end can advertise which allows a probe message to be
> > sent to get all the details QEMU needs to know in one message.
> >
> > Together with the existing features VHOST_USER_PROTOCOL_F_STATUS and
> > VHOST_USER_PROTOCOL_F_CONFIG we can create "standalone" vhost-user
> > daemons which are capable of handling all aspects of the VirtIO
> > transactions with only a generic stub on the QEMU side. These daemons
> > can also be used without QEMU in situations where there isn't a full
> > VMM managing their setup.
> >
> > Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> >
> > ---
> > v2
> > - dropped F_STANDALONE in favour of F_PROBE
> > - split probe details across several messages
> > - probe messages don't automatically imply a standalone daemon
> > - add wording where probe details interact (F_MQ/F_CONFIG)
> > - define VMM and make clear QEMU is only one of many potential VMMs
> > - reword commit message
> > ---
> > docs/interop/vhost-user.rst | 90 ++++++++++++++++++++++++++++++++-----
> > hw/virtio/vhost-user.c | 8 ++++
> > 2 files changed, 88 insertions(+), 10 deletions(-)
> >
> > diff --git a/docs/interop/vhost-user.rst b/docs/interop/vhost-user.rst
> > index 5a070adbc1..ba3b5e07b7 100644
> > --- a/docs/interop/vhost-user.rst
> > +++ b/docs/interop/vhost-user.rst
> > @@ -7,6 +7,7 @@ Vhost-user Protocol
> > ..
> > Copyright 2014 Virtual Open Systems Sarl.
> > Copyright 2019 Intel Corporation
> > + Copyright 2023 Linaro Ltd
> > Licence: This work is licensed under the terms of the GNU GPL,
> > version 2 or later. See the COPYING file in the top-level
> > directory.
> > @@ -27,17 +28,31 @@ The protocol defines 2 sides of the communication, *front-end* and
> > *back-end*. The *front-end* is the application that shares its virtqueues, in
> > our case QEMU. The *back-end* is the consumer of the virtqueues.
> >
> > -In the current implementation QEMU is the *front-end*, and the *back-end*
> > -is the external process consuming the virtio queues, for example a
> > -software Ethernet switch running in user space, such as Snabbswitch,
> > -or a block device back-end processing read & write to a virtual
> > -disk. In order to facilitate interoperability between various back-end
> > -implementations, it is recommended to follow the :ref:`Backend program
> > -conventions <backend_conventions>`.
> > +In the current implementation a Virtual Machine Manager (VMM) such as
> > +QEMU is the *front-end*, and the *back-end* is the external process
> > +consuming the virtio queues, for example a software Ethernet switch
> > +running in user space, such as Snabbswitch, or a block device back-end
> > +processing read & write to a virtual disk. In order to facilitate
> > +interoperability between various back-end implementations, it is
> > +recommended to follow the :ref:`Backend program conventions
> > +<backend_conventions>`.
> >
> > The *front-end* and *back-end* can be either a client (i.e. connecting) or
> > server (listening) in the socket communication.
> >
> > +Probing device details
> > +----------------------
> > +
> > +Traditionally the vhost-user daemon *back-end* shares configuration
> > +responsibilities with the VMM *front-end* which needs to know certain
> > +key bits of information about the device. This means the VMM needs to
> > +define at least a minimal stub for each VirtIO device it wants to
> > +support. If the daemon supports the right set of protocol features the
> > +VMM can probe the daemon for the information it needs to setup the
> > +device. See :ref:`Probing features for standalone daemons
> > +<probing_features>` for more details.
> > +
> > +
> > Support for platforms other than Linux
> > --------------------------------------
> >
> > @@ -316,6 +331,7 @@ replies. Here is a list of the ones that do:
> > * ``VHOST_USER_GET_VRING_BASE``
> > * ``VHOST_USER_SET_LOG_BASE`` (if ``VHOST_USER_PROTOCOL_F_LOG_SHMFD``)
> > * ``VHOST_USER_GET_INFLIGHT_FD`` (if ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD``)
> > +* ``VHOST_USER_GET_BACKEND_SPECS`` (if ``VHOST_USER_PROTOCOL_F_STANDALONE``)
> >
> > .. seealso::
> >
> > @@ -396,9 +412,10 @@ must support changing some configuration aspects on the fly.
> > Multiple queue support
> > ----------------------
> >
> > -Many devices have a fixed number of virtqueues. In this case the front-end
> > -already knows the number of available virtqueues without communicating with the
> > -back-end.
> > +Many devices have a fixed number of virtqueues. In this case the
> > +*front-end* usually already knows the number of available virtqueues
> > +without communicating with the back-end. For standalone daemons this
> > +number can be can be probed with the ``VHOST_USER_GET_MIN_VQ`` message.
> >
> > Some devices do not have a fixed number of virtqueues. Instead the maximum
> > number of virtqueues is chosen by the back-end. The number can depend on host
> > @@ -885,6 +902,23 @@ Protocol features
> > #define VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS 15
> > #define VHOST_USER_PROTOCOL_F_STATUS 16
> > #define VHOST_USER_PROTOCOL_F_XEN_MMAP 17
> > + #define VHOST_USER_PROTOCOL_F_PROBE 18
> > +
> > +.. _probing_features:
> > +
> > +Probing features for standalone daemons
> > +---------------------------------------
> > +
> > +The protocol feature ``VHOST_USER_PROTOCOL_F_PROBE`` enables a number
> > +of additional messages which allow the *front-end* to probe details
> > +about the VirtIO device from the *back-end*. However for a *back-end*
> > +to be described as standalone it must also support:
> > +
> > + * ``VHOST_USER_PROTOCOL_F_STATUS``
> > + * ``VHOST_USER_PROTOCOL_F_CONFIG`` (if there is a config space)
> > +
> > +which are required to ensure the *back-end* daemon can operate
> > +without the *front-end* managing some aspects of its configuration.
> >
> > Front-end message types
> > -----------------------
> > @@ -1440,6 +1474,42 @@ Front-end message types
> > query the back-end for its device status as defined in the Virtio
> > specification.
> >
> > +``VHOST_USER_GET_DEVICE_ID``
> > + :id: 41
> > + :request payload: N/A
> > + :reply payload: ``u32``
> > +
> > + When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
> > + successfully negotiated, this message is submitted by the front-end
> > + to query what VirtIO device the back-end support. This is intended
> > + to remove the need for the front-end to know ahead of time what the
> > + VirtIO device the backend emulates is.
> > +
> > +``VHOST_USER_GET_CONFIG_SIZE``
> > + :id: 42
> > + :request payload: N/A
> > + :reply payload: ``u32``
> > +
> > + When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
> > + successfully negotiated, this message is submitted by the front-end
> > + to query the size of the VirtIO device's config space. This is
> > + intended to remove the need for the front-end to know ahead of time
> > + what the size is. Replying with 0 when
> > + ``VHOST_USER_PROTOCOL_F_CONFIG`` has been negotiated would indicate
> > + an bug.
> > +
> > +``VHOST_USER_GET_MIN_VQ``
> > + :id: 43
> > + :request payload: N/A
> > + :reply payload: ``u32``
> > +
> > + When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
> > + successfully negotiated, this message is submitted by the front-end to
> > + query minimum number of VQ's required to support the device. A
> > + device may support more than this number of VQ's if it advertises
> > + the ``VHOST_USER_PROTOCOL_F_MQ`` protocol feature. Reporting a
> > + number greater than the result of ``VHOST_USER_GET_QUEUE_NUM`` would
> > + indicate a bug.
> >
> > Maybe I lack some background, but not sure what min_vq is here?
>
> There will be a minimum number of queues you need to support the device.
> For example the virtio-sound spec specifies you need four queues:
> control, event, tx, rx
I don't understand why the front-end needs to know that? The backend
already reports the number of queues and not all of them need to be
initialized by the driver.
>
> > This looks like quering the number of VQs the backend requires/uses.
> > Which, in case of MQ, it may be bigger (which is where I assume comes the `min`
> > part, if we consider `VHOST_USER_GET_QUEUE_NUM` the `max`).
>
> The MQ extension is currently used by networking but in theory any
> device could attempt to parallelism by extending the number of virt
> queues needed. So for net you get:
>
> receiveq1
> transmitq1
> optional controlq
>
> So VHOST_USER_GET_MIN_VQ would report 2 or 3 (if VIRTIO_NET_F_CTRL_VQ is
> negotiated).
I'm confused. VHOST_USER_GET_MIN_VQ comes before VIRTIO Feature Bit
negotiation (VIRTIO_NET_F_CTRL_VQ).
> However VHOST_USER_GET_QUEUE_NUM is only usable if
> VIRTIO_NET_F_MQ has been negotiated and could report more.
I don't understand. This patch adds a new feature and it can require
VHOST_USER_PROTOCOL_F_MQ. There are no existing back-ends that require
backwards compatibility.
Stefan
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [virtio-dev] [RFC PATCH v2] docs/interop: define PROBE feature for vhost-user VirtIO devices
@ 2023-09-07 19:29 ` Stefan Hajnoczi
0 siblings, 0 replies; 27+ messages in thread
From: Stefan Hajnoczi @ 2023-09-07 19:29 UTC (permalink / raw)
To: Alex Bennée
Cc: Albert Esteve, slp, mst, marcandre.lureau, viresh.kumar, sgarzare,
takahiro.akashi, erik.schilling, manos.pitsidianakis,
mathieu.poirier, qemu-devel, virtio-dev, virtio-comment
[-- Attachment #1: Type: text/plain, Size: 10072 bytes --]
On Tue, Sep 05, 2023 at 10:34:11AM +0100, Alex Bennée wrote:
>
> Albert Esteve <aesteve@redhat.com> writes:
>
> > This looks great! Thanks for this proposal.
> >
> > On Fri, Sep 1, 2023 at 1:00 PM Alex Bennée <alex.bennee@linaro.org> wrote:
> >
> > Currently QEMU has to know some details about the VirtIO device
> > supported by a vhost-user daemon to be able to setup the guest. This
> > makes it hard for QEMU to add support for additional vhost-user
> > daemons without adding specific stubs for each additional VirtIO
> > device.
> >
> > This patch suggests a new feature flag (VHOST_USER_PROTOCOL_F_PROBE)
> > which the back-end can advertise which allows a probe message to be
> > sent to get all the details QEMU needs to know in one message.
> >
> > Together with the existing features VHOST_USER_PROTOCOL_F_STATUS and
> > VHOST_USER_PROTOCOL_F_CONFIG we can create "standalone" vhost-user
> > daemons which are capable of handling all aspects of the VirtIO
> > transactions with only a generic stub on the QEMU side. These daemons
> > can also be used without QEMU in situations where there isn't a full
> > VMM managing their setup.
> >
> > Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> >
> > ---
> > v2
> > - dropped F_STANDALONE in favour of F_PROBE
> > - split probe details across several messages
> > - probe messages don't automatically imply a standalone daemon
> > - add wording where probe details interact (F_MQ/F_CONFIG)
> > - define VMM and make clear QEMU is only one of many potential VMMs
> > - reword commit message
> > ---
> > docs/interop/vhost-user.rst | 90 ++++++++++++++++++++++++++++++++-----
> > hw/virtio/vhost-user.c | 8 ++++
> > 2 files changed, 88 insertions(+), 10 deletions(-)
> >
> > diff --git a/docs/interop/vhost-user.rst b/docs/interop/vhost-user.rst
> > index 5a070adbc1..ba3b5e07b7 100644
> > --- a/docs/interop/vhost-user.rst
> > +++ b/docs/interop/vhost-user.rst
> > @@ -7,6 +7,7 @@ Vhost-user Protocol
> > ..
> > Copyright 2014 Virtual Open Systems Sarl.
> > Copyright 2019 Intel Corporation
> > + Copyright 2023 Linaro Ltd
> > Licence: This work is licensed under the terms of the GNU GPL,
> > version 2 or later. See the COPYING file in the top-level
> > directory.
> > @@ -27,17 +28,31 @@ The protocol defines 2 sides of the communication, *front-end* and
> > *back-end*. The *front-end* is the application that shares its virtqueues, in
> > our case QEMU. The *back-end* is the consumer of the virtqueues.
> >
> > -In the current implementation QEMU is the *front-end*, and the *back-end*
> > -is the external process consuming the virtio queues, for example a
> > -software Ethernet switch running in user space, such as Snabbswitch,
> > -or a block device back-end processing read & write to a virtual
> > -disk. In order to facilitate interoperability between various back-end
> > -implementations, it is recommended to follow the :ref:`Backend program
> > -conventions <backend_conventions>`.
> > +In the current implementation a Virtual Machine Manager (VMM) such as
> > +QEMU is the *front-end*, and the *back-end* is the external process
> > +consuming the virtio queues, for example a software Ethernet switch
> > +running in user space, such as Snabbswitch, or a block device back-end
> > +processing read & write to a virtual disk. In order to facilitate
> > +interoperability between various back-end implementations, it is
> > +recommended to follow the :ref:`Backend program conventions
> > +<backend_conventions>`.
> >
> > The *front-end* and *back-end* can be either a client (i.e. connecting) or
> > server (listening) in the socket communication.
> >
> > +Probing device details
> > +----------------------
> > +
> > +Traditionally the vhost-user daemon *back-end* shares configuration
> > +responsibilities with the VMM *front-end* which needs to know certain
> > +key bits of information about the device. This means the VMM needs to
> > +define at least a minimal stub for each VirtIO device it wants to
> > +support. If the daemon supports the right set of protocol features the
> > +VMM can probe the daemon for the information it needs to setup the
> > +device. See :ref:`Probing features for standalone daemons
> > +<probing_features>` for more details.
> > +
> > +
> > Support for platforms other than Linux
> > --------------------------------------
> >
> > @@ -316,6 +331,7 @@ replies. Here is a list of the ones that do:
> > * ``VHOST_USER_GET_VRING_BASE``
> > * ``VHOST_USER_SET_LOG_BASE`` (if ``VHOST_USER_PROTOCOL_F_LOG_SHMFD``)
> > * ``VHOST_USER_GET_INFLIGHT_FD`` (if ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD``)
> > +* ``VHOST_USER_GET_BACKEND_SPECS`` (if ``VHOST_USER_PROTOCOL_F_STANDALONE``)
> >
> > .. seealso::
> >
> > @@ -396,9 +412,10 @@ must support changing some configuration aspects on the fly.
> > Multiple queue support
> > ----------------------
> >
> > -Many devices have a fixed number of virtqueues. In this case the front-end
> > -already knows the number of available virtqueues without communicating with the
> > -back-end.
> > +Many devices have a fixed number of virtqueues. In this case the
> > +*front-end* usually already knows the number of available virtqueues
> > +without communicating with the back-end. For standalone daemons this
> > +number can be can be probed with the ``VHOST_USER_GET_MIN_VQ`` message.
> >
> > Some devices do not have a fixed number of virtqueues. Instead the maximum
> > number of virtqueues is chosen by the back-end. The number can depend on host
> > @@ -885,6 +902,23 @@ Protocol features
> > #define VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS 15
> > #define VHOST_USER_PROTOCOL_F_STATUS 16
> > #define VHOST_USER_PROTOCOL_F_XEN_MMAP 17
> > + #define VHOST_USER_PROTOCOL_F_PROBE 18
> > +
> > +.. _probing_features:
> > +
> > +Probing features for standalone daemons
> > +---------------------------------------
> > +
> > +The protocol feature ``VHOST_USER_PROTOCOL_F_PROBE`` enables a number
> > +of additional messages which allow the *front-end* to probe details
> > +about the VirtIO device from the *back-end*. However for a *back-end*
> > +to be described as standalone it must also support:
> > +
> > + * ``VHOST_USER_PROTOCOL_F_STATUS``
> > + * ``VHOST_USER_PROTOCOL_F_CONFIG`` (if there is a config space)
> > +
> > +which are required to ensure the *back-end* daemon can operate
> > +without the *front-end* managing some aspects of its configuration.
> >
> > Front-end message types
> > -----------------------
> > @@ -1440,6 +1474,42 @@ Front-end message types
> > query the back-end for its device status as defined in the Virtio
> > specification.
> >
> > +``VHOST_USER_GET_DEVICE_ID``
> > + :id: 41
> > + :request payload: N/A
> > + :reply payload: ``u32``
> > +
> > + When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
> > + successfully negotiated, this message is submitted by the front-end
> > + to query what VirtIO device the back-end support. This is intended
> > + to remove the need for the front-end to know ahead of time what the
> > + VirtIO device the backend emulates is.
> > +
> > +``VHOST_USER_GET_CONFIG_SIZE``
> > + :id: 42
> > + :request payload: N/A
> > + :reply payload: ``u32``
> > +
> > + When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
> > + successfully negotiated, this message is submitted by the front-end
> > + to query the size of the VirtIO device's config space. This is
> > + intended to remove the need for the front-end to know ahead of time
> > + what the size is. Replying with 0 when
> > + ``VHOST_USER_PROTOCOL_F_CONFIG`` has been negotiated would indicate
> > + an bug.
> > +
> > +``VHOST_USER_GET_MIN_VQ``
> > + :id: 43
> > + :request payload: N/A
> > + :reply payload: ``u32``
> > +
> > + When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
> > + successfully negotiated, this message is submitted by the front-end to
> > + query minimum number of VQ's required to support the device. A
> > + device may support more than this number of VQ's if it advertises
> > + the ``VHOST_USER_PROTOCOL_F_MQ`` protocol feature. Reporting a
> > + number greater than the result of ``VHOST_USER_GET_QUEUE_NUM`` would
> > + indicate a bug.
> >
> > Maybe I lack some background, but not sure what min_vq is here?
>
> There will be a minimum number of queues you need to support the device.
> For example the virtio-sound spec specifies you need four queues:
> control, event, tx, rx
I don't understand why the front-end needs to know that? The backend
already reports the number of queues and not all of them need to be
initialized by the driver.
>
> > This looks like quering the number of VQs the backend requires/uses.
> > Which, in case of MQ, it may be bigger (which is where I assume comes the `min`
> > part, if we consider `VHOST_USER_GET_QUEUE_NUM` the `max`).
>
> The MQ extension is currently used by networking but in theory any
> device could attempt to parallelism by extending the number of virt
> queues needed. So for net you get:
>
> receiveq1
> transmitq1
> optional controlq
>
> So VHOST_USER_GET_MIN_VQ would report 2 or 3 (if VIRTIO_NET_F_CTRL_VQ is
> negotiated).
I'm confused. VHOST_USER_GET_MIN_VQ comes before VIRTIO Feature Bit
negotiation (VIRTIO_NET_F_CTRL_VQ).
> However VHOST_USER_GET_QUEUE_NUM is only usable if
> VIRTIO_NET_F_MQ has been negotiated and could report more.
I don't understand. This patch adds a new feature and it can require
VHOST_USER_PROTOCOL_F_MQ. There are no existing back-ends that require
backwards compatibility.
Stefan
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 27+ messages in thread
* [virtio-comment] Re: [virtio-dev] [RFC PATCH v2] docs/interop: define PROBE feature for vhost-user VirtIO devices
2023-09-07 19:29 ` Stefan Hajnoczi
(?)
@ 2023-09-08 6:41 ` Alex Bennée
-1 siblings, 0 replies; 27+ messages in thread
From: Alex Bennée @ 2023-09-08 6:41 UTC (permalink / raw)
To: Stefan Hajnoczi
Cc: Albert Esteve, slp, mst, marcandre.lureau, viresh.kumar, sgarzare,
takahiro.akashi, erik.schilling, manos.pitsidianakis,
mathieu.poirier, qemu-devel, virtio-dev, virtio-comment
Stefan Hajnoczi <stefanha@redhat.com> writes:
> On Tue, Sep 05, 2023 at 10:34:11AM +0100, Alex Bennée wrote:
>>
>> Albert Esteve <aesteve@redhat.com> writes:
>>
>> > This looks great! Thanks for this proposal.
>> >
>> > On Fri, Sep 1, 2023 at 1:00 PM Alex Bennée <alex.bennee@linaro.org> wrote:
>> >
>> > Currently QEMU has to know some details about the VirtIO device
>> > supported by a vhost-user daemon to be able to setup the guest. This
>> > makes it hard for QEMU to add support for additional vhost-user
>> > daemons without adding specific stubs for each additional VirtIO
>> > device.
>> >
>> > This patch suggests a new feature flag (VHOST_USER_PROTOCOL_F_PROBE)
>> > which the back-end can advertise which allows a probe message to be
>> > sent to get all the details QEMU needs to know in one message.
>> >
>> > Together with the existing features VHOST_USER_PROTOCOL_F_STATUS and
>> > VHOST_USER_PROTOCOL_F_CONFIG we can create "standalone" vhost-user
>> > daemons which are capable of handling all aspects of the VirtIO
>> > transactions with only a generic stub on the QEMU side. These daemons
>> > can also be used without QEMU in situations where there isn't a full
>> > VMM managing their setup.
>> >
>> > Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>> >
>> > ---
>> > v2
>> > - dropped F_STANDALONE in favour of F_PROBE
>> > - split probe details across several messages
>> > - probe messages don't automatically imply a standalone daemon
>> > - add wording where probe details interact (F_MQ/F_CONFIG)
>> > - define VMM and make clear QEMU is only one of many potential VMMs
>> > - reword commit message
>> > ---
>> > docs/interop/vhost-user.rst | 90 ++++++++++++++++++++++++++++++++-----
>> > hw/virtio/vhost-user.c | 8 ++++
>> > 2 files changed, 88 insertions(+), 10 deletions(-)
>> >
>> > diff --git a/docs/interop/vhost-user.rst b/docs/interop/vhost-user.rst
>> > index 5a070adbc1..ba3b5e07b7 100644
>> > --- a/docs/interop/vhost-user.rst
>> > +++ b/docs/interop/vhost-user.rst
>> > @@ -7,6 +7,7 @@ Vhost-user Protocol
>> > ..
>> > Copyright 2014 Virtual Open Systems Sarl.
>> > Copyright 2019 Intel Corporation
>> > + Copyright 2023 Linaro Ltd
>> > Licence: This work is licensed under the terms of the GNU GPL,
>> > version 2 or later. See the COPYING file in the top-level
>> > directory.
>> > @@ -27,17 +28,31 @@ The protocol defines 2 sides of the communication, *front-end* and
>> > *back-end*. The *front-end* is the application that shares its virtqueues, in
>> > our case QEMU. The *back-end* is the consumer of the virtqueues.
>> >
>> > -In the current implementation QEMU is the *front-end*, and the *back-end*
>> > -is the external process consuming the virtio queues, for example a
>> > -software Ethernet switch running in user space, such as Snabbswitch,
>> > -or a block device back-end processing read & write to a virtual
>> > -disk. In order to facilitate interoperability between various back-end
>> > -implementations, it is recommended to follow the :ref:`Backend program
>> > -conventions <backend_conventions>`.
>> > +In the current implementation a Virtual Machine Manager (VMM) such as
>> > +QEMU is the *front-end*, and the *back-end* is the external process
>> > +consuming the virtio queues, for example a software Ethernet switch
>> > +running in user space, such as Snabbswitch, or a block device back-end
>> > +processing read & write to a virtual disk. In order to facilitate
>> > +interoperability between various back-end implementations, it is
>> > +recommended to follow the :ref:`Backend program conventions
>> > +<backend_conventions>`.
>> >
>> > The *front-end* and *back-end* can be either a client (i.e. connecting) or
>> > server (listening) in the socket communication.
>> >
>> > +Probing device details
>> > +----------------------
>> > +
>> > +Traditionally the vhost-user daemon *back-end* shares configuration
>> > +responsibilities with the VMM *front-end* which needs to know certain
>> > +key bits of information about the device. This means the VMM needs to
>> > +define at least a minimal stub for each VirtIO device it wants to
>> > +support. If the daemon supports the right set of protocol features the
>> > +VMM can probe the daemon for the information it needs to setup the
>> > +device. See :ref:`Probing features for standalone daemons
>> > +<probing_features>` for more details.
>> > +
>> > +
>> > Support for platforms other than Linux
>> > --------------------------------------
>> >
>> > @@ -316,6 +331,7 @@ replies. Here is a list of the ones that do:
>> > * ``VHOST_USER_GET_VRING_BASE``
>> > * ``VHOST_USER_SET_LOG_BASE`` (if ``VHOST_USER_PROTOCOL_F_LOG_SHMFD``)
>> > * ``VHOST_USER_GET_INFLIGHT_FD`` (if ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD``)
>> > +* ``VHOST_USER_GET_BACKEND_SPECS`` (if ``VHOST_USER_PROTOCOL_F_STANDALONE``)
>> >
>> > .. seealso::
>> >
>> > @@ -396,9 +412,10 @@ must support changing some configuration aspects on the fly.
>> > Multiple queue support
>> > ----------------------
>> >
>> > -Many devices have a fixed number of virtqueues. In this case the front-end
>> > -already knows the number of available virtqueues without communicating with the
>> > -back-end.
>> > +Many devices have a fixed number of virtqueues. In this case the
>> > +*front-end* usually already knows the number of available virtqueues
>> > +without communicating with the back-end. For standalone daemons this
>> > +number can be can be probed with the ``VHOST_USER_GET_MIN_VQ`` message.
>> >
>> > Some devices do not have a fixed number of virtqueues. Instead the maximum
>> > number of virtqueues is chosen by the back-end. The number can depend on host
>> > @@ -885,6 +902,23 @@ Protocol features
>> > #define VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS 15
>> > #define VHOST_USER_PROTOCOL_F_STATUS 16
>> > #define VHOST_USER_PROTOCOL_F_XEN_MMAP 17
>> > + #define VHOST_USER_PROTOCOL_F_PROBE 18
>> > +
>> > +.. _probing_features:
>> > +
>> > +Probing features for standalone daemons
>> > +---------------------------------------
>> > +
>> > +The protocol feature ``VHOST_USER_PROTOCOL_F_PROBE`` enables a number
>> > +of additional messages which allow the *front-end* to probe details
>> > +about the VirtIO device from the *back-end*. However for a *back-end*
>> > +to be described as standalone it must also support:
>> > +
>> > + * ``VHOST_USER_PROTOCOL_F_STATUS``
>> > + * ``VHOST_USER_PROTOCOL_F_CONFIG`` (if there is a config space)
>> > +
>> > +which are required to ensure the *back-end* daemon can operate
>> > +without the *front-end* managing some aspects of its configuration.
>> >
>> > Front-end message types
>> > -----------------------
>> > @@ -1440,6 +1474,42 @@ Front-end message types
>> > query the back-end for its device status as defined in the Virtio
>> > specification.
>> >
>> > +``VHOST_USER_GET_DEVICE_ID``
>> > + :id: 41
>> > + :request payload: N/A
>> > + :reply payload: ``u32``
>> > +
>> > + When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
>> > + successfully negotiated, this message is submitted by the front-end
>> > + to query what VirtIO device the back-end support. This is intended
>> > + to remove the need for the front-end to know ahead of time what the
>> > + VirtIO device the backend emulates is.
>> > +
>> > +``VHOST_USER_GET_CONFIG_SIZE``
>> > + :id: 42
>> > + :request payload: N/A
>> > + :reply payload: ``u32``
>> > +
>> > + When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
>> > + successfully negotiated, this message is submitted by the front-end
>> > + to query the size of the VirtIO device's config space. This is
>> > + intended to remove the need for the front-end to know ahead of time
>> > + what the size is. Replying with 0 when
>> > + ``VHOST_USER_PROTOCOL_F_CONFIG`` has been negotiated would indicate
>> > + an bug.
>> > +
>> > +``VHOST_USER_GET_MIN_VQ``
>> > + :id: 43
>> > + :request payload: N/A
>> > + :reply payload: ``u32``
>> > +
>> > + When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
>> > + successfully negotiated, this message is submitted by the front-end to
>> > + query minimum number of VQ's required to support the device. A
>> > + device may support more than this number of VQ's if it advertises
>> > + the ``VHOST_USER_PROTOCOL_F_MQ`` protocol feature. Reporting a
>> > + number greater than the result of ``VHOST_USER_GET_QUEUE_NUM`` would
>> > + indicate a bug.
>> >
>> > Maybe I lack some background, but not sure what min_vq is here?
>>
>> There will be a minimum number of queues you need to support the device.
>> For example the virtio-sound spec specifies you need four queues:
>> control, event, tx, rx
>
> I don't understand why the front-end needs to know that? The backend
> already reports the number of queues and not all of them need to be
> initialized by the driver.
But how many don't need to be initialised? We can't just skip:
/* Allocate queues */
vub->vqs = g_ptr_array_sized_new(vub->num_vqs);
for (int i = 0; i < vub->num_vqs; i++) {
g_ptr_array_add(vub->vqs,
virtio_add_queue(vdev, vub->vq_size, vub_handle_output));
}
Or are you saying just require probe-able backends to support
VHOST_USER_PROTOCOL_F_MQ and have it always report the minimmum number
of queues if it is not a MQ capable device?
>> > This looks like quering the number of VQs the backend requires/uses.
>> > Which, in case of MQ, it may be bigger (which is where I assume comes the `min`
>> > part, if we consider `VHOST_USER_GET_QUEUE_NUM` the `max`).
>>
>> The MQ extension is currently used by networking but in theory any
>> device could attempt to parallelism by extending the number of virt
>> queues needed. So for net you get:
>>
>> receiveq1
>> transmitq1
>> optional controlq
>>
>> So VHOST_USER_GET_MIN_VQ would report 2 or 3 (if VIRTIO_NET_F_CTRL_VQ is
>> negotiated).
>
> I'm confused. VHOST_USER_GET_MIN_VQ comes before VIRTIO Feature Bit
> negotiation (VIRTIO_NET_F_CTRL_VQ).
>
>> However VHOST_USER_GET_QUEUE_NUM is only usable if
>> VIRTIO_NET_F_MQ has been negotiated and could report more.
>
> I don't understand. This patch adds a new feature and it can require
> VHOST_USER_PROTOCOL_F_MQ. There are no existing back-ends that require
> backwards compatibility.
>
> Stefan
--
Alex Bennée
Virtualisation Tech Lead @ Linaro
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 27+ messages in thread* Re: [virtio-dev] [RFC PATCH v2] docs/interop: define PROBE feature for vhost-user VirtIO devices
@ 2023-09-08 6:41 ` Alex Bennée
0 siblings, 0 replies; 27+ messages in thread
From: Alex Bennée @ 2023-09-08 6:41 UTC (permalink / raw)
To: Stefan Hajnoczi
Cc: Albert Esteve, slp, mst, marcandre.lureau, viresh.kumar, sgarzare,
takahiro.akashi, erik.schilling, manos.pitsidianakis,
mathieu.poirier, qemu-devel, virtio-dev, virtio-comment
Stefan Hajnoczi <stefanha@redhat.com> writes:
> On Tue, Sep 05, 2023 at 10:34:11AM +0100, Alex Bennée wrote:
>>
>> Albert Esteve <aesteve@redhat.com> writes:
>>
>> > This looks great! Thanks for this proposal.
>> >
>> > On Fri, Sep 1, 2023 at 1:00 PM Alex Bennée <alex.bennee@linaro.org> wrote:
>> >
>> > Currently QEMU has to know some details about the VirtIO device
>> > supported by a vhost-user daemon to be able to setup the guest. This
>> > makes it hard for QEMU to add support for additional vhost-user
>> > daemons without adding specific stubs for each additional VirtIO
>> > device.
>> >
>> > This patch suggests a new feature flag (VHOST_USER_PROTOCOL_F_PROBE)
>> > which the back-end can advertise which allows a probe message to be
>> > sent to get all the details QEMU needs to know in one message.
>> >
>> > Together with the existing features VHOST_USER_PROTOCOL_F_STATUS and
>> > VHOST_USER_PROTOCOL_F_CONFIG we can create "standalone" vhost-user
>> > daemons which are capable of handling all aspects of the VirtIO
>> > transactions with only a generic stub on the QEMU side. These daemons
>> > can also be used without QEMU in situations where there isn't a full
>> > VMM managing their setup.
>> >
>> > Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>> >
>> > ---
>> > v2
>> > - dropped F_STANDALONE in favour of F_PROBE
>> > - split probe details across several messages
>> > - probe messages don't automatically imply a standalone daemon
>> > - add wording where probe details interact (F_MQ/F_CONFIG)
>> > - define VMM and make clear QEMU is only one of many potential VMMs
>> > - reword commit message
>> > ---
>> > docs/interop/vhost-user.rst | 90 ++++++++++++++++++++++++++++++++-----
>> > hw/virtio/vhost-user.c | 8 ++++
>> > 2 files changed, 88 insertions(+), 10 deletions(-)
>> >
>> > diff --git a/docs/interop/vhost-user.rst b/docs/interop/vhost-user.rst
>> > index 5a070adbc1..ba3b5e07b7 100644
>> > --- a/docs/interop/vhost-user.rst
>> > +++ b/docs/interop/vhost-user.rst
>> > @@ -7,6 +7,7 @@ Vhost-user Protocol
>> > ..
>> > Copyright 2014 Virtual Open Systems Sarl.
>> > Copyright 2019 Intel Corporation
>> > + Copyright 2023 Linaro Ltd
>> > Licence: This work is licensed under the terms of the GNU GPL,
>> > version 2 or later. See the COPYING file in the top-level
>> > directory.
>> > @@ -27,17 +28,31 @@ The protocol defines 2 sides of the communication, *front-end* and
>> > *back-end*. The *front-end* is the application that shares its virtqueues, in
>> > our case QEMU. The *back-end* is the consumer of the virtqueues.
>> >
>> > -In the current implementation QEMU is the *front-end*, and the *back-end*
>> > -is the external process consuming the virtio queues, for example a
>> > -software Ethernet switch running in user space, such as Snabbswitch,
>> > -or a block device back-end processing read & write to a virtual
>> > -disk. In order to facilitate interoperability between various back-end
>> > -implementations, it is recommended to follow the :ref:`Backend program
>> > -conventions <backend_conventions>`.
>> > +In the current implementation a Virtual Machine Manager (VMM) such as
>> > +QEMU is the *front-end*, and the *back-end* is the external process
>> > +consuming the virtio queues, for example a software Ethernet switch
>> > +running in user space, such as Snabbswitch, or a block device back-end
>> > +processing read & write to a virtual disk. In order to facilitate
>> > +interoperability between various back-end implementations, it is
>> > +recommended to follow the :ref:`Backend program conventions
>> > +<backend_conventions>`.
>> >
>> > The *front-end* and *back-end* can be either a client (i.e. connecting) or
>> > server (listening) in the socket communication.
>> >
>> > +Probing device details
>> > +----------------------
>> > +
>> > +Traditionally the vhost-user daemon *back-end* shares configuration
>> > +responsibilities with the VMM *front-end* which needs to know certain
>> > +key bits of information about the device. This means the VMM needs to
>> > +define at least a minimal stub for each VirtIO device it wants to
>> > +support. If the daemon supports the right set of protocol features the
>> > +VMM can probe the daemon for the information it needs to setup the
>> > +device. See :ref:`Probing features for standalone daemons
>> > +<probing_features>` for more details.
>> > +
>> > +
>> > Support for platforms other than Linux
>> > --------------------------------------
>> >
>> > @@ -316,6 +331,7 @@ replies. Here is a list of the ones that do:
>> > * ``VHOST_USER_GET_VRING_BASE``
>> > * ``VHOST_USER_SET_LOG_BASE`` (if ``VHOST_USER_PROTOCOL_F_LOG_SHMFD``)
>> > * ``VHOST_USER_GET_INFLIGHT_FD`` (if ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD``)
>> > +* ``VHOST_USER_GET_BACKEND_SPECS`` (if ``VHOST_USER_PROTOCOL_F_STANDALONE``)
>> >
>> > .. seealso::
>> >
>> > @@ -396,9 +412,10 @@ must support changing some configuration aspects on the fly.
>> > Multiple queue support
>> > ----------------------
>> >
>> > -Many devices have a fixed number of virtqueues. In this case the front-end
>> > -already knows the number of available virtqueues without communicating with the
>> > -back-end.
>> > +Many devices have a fixed number of virtqueues. In this case the
>> > +*front-end* usually already knows the number of available virtqueues
>> > +without communicating with the back-end. For standalone daemons this
>> > +number can be can be probed with the ``VHOST_USER_GET_MIN_VQ`` message.
>> >
>> > Some devices do not have a fixed number of virtqueues. Instead the maximum
>> > number of virtqueues is chosen by the back-end. The number can depend on host
>> > @@ -885,6 +902,23 @@ Protocol features
>> > #define VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS 15
>> > #define VHOST_USER_PROTOCOL_F_STATUS 16
>> > #define VHOST_USER_PROTOCOL_F_XEN_MMAP 17
>> > + #define VHOST_USER_PROTOCOL_F_PROBE 18
>> > +
>> > +.. _probing_features:
>> > +
>> > +Probing features for standalone daemons
>> > +---------------------------------------
>> > +
>> > +The protocol feature ``VHOST_USER_PROTOCOL_F_PROBE`` enables a number
>> > +of additional messages which allow the *front-end* to probe details
>> > +about the VirtIO device from the *back-end*. However for a *back-end*
>> > +to be described as standalone it must also support:
>> > +
>> > + * ``VHOST_USER_PROTOCOL_F_STATUS``
>> > + * ``VHOST_USER_PROTOCOL_F_CONFIG`` (if there is a config space)
>> > +
>> > +which are required to ensure the *back-end* daemon can operate
>> > +without the *front-end* managing some aspects of its configuration.
>> >
>> > Front-end message types
>> > -----------------------
>> > @@ -1440,6 +1474,42 @@ Front-end message types
>> > query the back-end for its device status as defined in the Virtio
>> > specification.
>> >
>> > +``VHOST_USER_GET_DEVICE_ID``
>> > + :id: 41
>> > + :request payload: N/A
>> > + :reply payload: ``u32``
>> > +
>> > + When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
>> > + successfully negotiated, this message is submitted by the front-end
>> > + to query what VirtIO device the back-end support. This is intended
>> > + to remove the need for the front-end to know ahead of time what the
>> > + VirtIO device the backend emulates is.
>> > +
>> > +``VHOST_USER_GET_CONFIG_SIZE``
>> > + :id: 42
>> > + :request payload: N/A
>> > + :reply payload: ``u32``
>> > +
>> > + When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
>> > + successfully negotiated, this message is submitted by the front-end
>> > + to query the size of the VirtIO device's config space. This is
>> > + intended to remove the need for the front-end to know ahead of time
>> > + what the size is. Replying with 0 when
>> > + ``VHOST_USER_PROTOCOL_F_CONFIG`` has been negotiated would indicate
>> > + an bug.
>> > +
>> > +``VHOST_USER_GET_MIN_VQ``
>> > + :id: 43
>> > + :request payload: N/A
>> > + :reply payload: ``u32``
>> > +
>> > + When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
>> > + successfully negotiated, this message is submitted by the front-end to
>> > + query minimum number of VQ's required to support the device. A
>> > + device may support more than this number of VQ's if it advertises
>> > + the ``VHOST_USER_PROTOCOL_F_MQ`` protocol feature. Reporting a
>> > + number greater than the result of ``VHOST_USER_GET_QUEUE_NUM`` would
>> > + indicate a bug.
>> >
>> > Maybe I lack some background, but not sure what min_vq is here?
>>
>> There will be a minimum number of queues you need to support the device.
>> For example the virtio-sound spec specifies you need four queues:
>> control, event, tx, rx
>
> I don't understand why the front-end needs to know that? The backend
> already reports the number of queues and not all of them need to be
> initialized by the driver.
But how many don't need to be initialised? We can't just skip:
/* Allocate queues */
vub->vqs = g_ptr_array_sized_new(vub->num_vqs);
for (int i = 0; i < vub->num_vqs; i++) {
g_ptr_array_add(vub->vqs,
virtio_add_queue(vdev, vub->vq_size, vub_handle_output));
}
Or are you saying just require probe-able backends to support
VHOST_USER_PROTOCOL_F_MQ and have it always report the minimmum number
of queues if it is not a MQ capable device?
>> > This looks like quering the number of VQs the backend requires/uses.
>> > Which, in case of MQ, it may be bigger (which is where I assume comes the `min`
>> > part, if we consider `VHOST_USER_GET_QUEUE_NUM` the `max`).
>>
>> The MQ extension is currently used by networking but in theory any
>> device could attempt to parallelism by extending the number of virt
>> queues needed. So for net you get:
>>
>> receiveq1
>> transmitq1
>> optional controlq
>>
>> So VHOST_USER_GET_MIN_VQ would report 2 or 3 (if VIRTIO_NET_F_CTRL_VQ is
>> negotiated).
>
> I'm confused. VHOST_USER_GET_MIN_VQ comes before VIRTIO Feature Bit
> negotiation (VIRTIO_NET_F_CTRL_VQ).
>
>> However VHOST_USER_GET_QUEUE_NUM is only usable if
>> VIRTIO_NET_F_MQ has been negotiated and could report more.
>
> I don't understand. This patch adds a new feature and it can require
> VHOST_USER_PROTOCOL_F_MQ. There are no existing back-ends that require
> backwards compatibility.
>
> Stefan
--
Alex Bennée
Virtualisation Tech Lead @ Linaro
^ permalink raw reply [flat|nested] 27+ messages in thread* Re: [virtio-dev] [RFC PATCH v2] docs/interop: define PROBE feature for vhost-user VirtIO devices
@ 2023-09-08 6:41 ` Alex Bennée
0 siblings, 0 replies; 27+ messages in thread
From: Alex Bennée @ 2023-09-08 6:41 UTC (permalink / raw)
To: Stefan Hajnoczi
Cc: Albert Esteve, slp, mst, marcandre.lureau, viresh.kumar, sgarzare,
takahiro.akashi, erik.schilling, manos.pitsidianakis,
mathieu.poirier, qemu-devel, virtio-dev, virtio-comment
Stefan Hajnoczi <stefanha@redhat.com> writes:
> On Tue, Sep 05, 2023 at 10:34:11AM +0100, Alex Bennée wrote:
>>
>> Albert Esteve <aesteve@redhat.com> writes:
>>
>> > This looks great! Thanks for this proposal.
>> >
>> > On Fri, Sep 1, 2023 at 1:00 PM Alex Bennée <alex.bennee@linaro.org> wrote:
>> >
>> > Currently QEMU has to know some details about the VirtIO device
>> > supported by a vhost-user daemon to be able to setup the guest. This
>> > makes it hard for QEMU to add support for additional vhost-user
>> > daemons without adding specific stubs for each additional VirtIO
>> > device.
>> >
>> > This patch suggests a new feature flag (VHOST_USER_PROTOCOL_F_PROBE)
>> > which the back-end can advertise which allows a probe message to be
>> > sent to get all the details QEMU needs to know in one message.
>> >
>> > Together with the existing features VHOST_USER_PROTOCOL_F_STATUS and
>> > VHOST_USER_PROTOCOL_F_CONFIG we can create "standalone" vhost-user
>> > daemons which are capable of handling all aspects of the VirtIO
>> > transactions with only a generic stub on the QEMU side. These daemons
>> > can also be used without QEMU in situations where there isn't a full
>> > VMM managing their setup.
>> >
>> > Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>> >
>> > ---
>> > v2
>> > - dropped F_STANDALONE in favour of F_PROBE
>> > - split probe details across several messages
>> > - probe messages don't automatically imply a standalone daemon
>> > - add wording where probe details interact (F_MQ/F_CONFIG)
>> > - define VMM and make clear QEMU is only one of many potential VMMs
>> > - reword commit message
>> > ---
>> > docs/interop/vhost-user.rst | 90 ++++++++++++++++++++++++++++++++-----
>> > hw/virtio/vhost-user.c | 8 ++++
>> > 2 files changed, 88 insertions(+), 10 deletions(-)
>> >
>> > diff --git a/docs/interop/vhost-user.rst b/docs/interop/vhost-user.rst
>> > index 5a070adbc1..ba3b5e07b7 100644
>> > --- a/docs/interop/vhost-user.rst
>> > +++ b/docs/interop/vhost-user.rst
>> > @@ -7,6 +7,7 @@ Vhost-user Protocol
>> > ..
>> > Copyright 2014 Virtual Open Systems Sarl.
>> > Copyright 2019 Intel Corporation
>> > + Copyright 2023 Linaro Ltd
>> > Licence: This work is licensed under the terms of the GNU GPL,
>> > version 2 or later. See the COPYING file in the top-level
>> > directory.
>> > @@ -27,17 +28,31 @@ The protocol defines 2 sides of the communication, *front-end* and
>> > *back-end*. The *front-end* is the application that shares its virtqueues, in
>> > our case QEMU. The *back-end* is the consumer of the virtqueues.
>> >
>> > -In the current implementation QEMU is the *front-end*, and the *back-end*
>> > -is the external process consuming the virtio queues, for example a
>> > -software Ethernet switch running in user space, such as Snabbswitch,
>> > -or a block device back-end processing read & write to a virtual
>> > -disk. In order to facilitate interoperability between various back-end
>> > -implementations, it is recommended to follow the :ref:`Backend program
>> > -conventions <backend_conventions>`.
>> > +In the current implementation a Virtual Machine Manager (VMM) such as
>> > +QEMU is the *front-end*, and the *back-end* is the external process
>> > +consuming the virtio queues, for example a software Ethernet switch
>> > +running in user space, such as Snabbswitch, or a block device back-end
>> > +processing read & write to a virtual disk. In order to facilitate
>> > +interoperability between various back-end implementations, it is
>> > +recommended to follow the :ref:`Backend program conventions
>> > +<backend_conventions>`.
>> >
>> > The *front-end* and *back-end* can be either a client (i.e. connecting) or
>> > server (listening) in the socket communication.
>> >
>> > +Probing device details
>> > +----------------------
>> > +
>> > +Traditionally the vhost-user daemon *back-end* shares configuration
>> > +responsibilities with the VMM *front-end* which needs to know certain
>> > +key bits of information about the device. This means the VMM needs to
>> > +define at least a minimal stub for each VirtIO device it wants to
>> > +support. If the daemon supports the right set of protocol features the
>> > +VMM can probe the daemon for the information it needs to setup the
>> > +device. See :ref:`Probing features for standalone daemons
>> > +<probing_features>` for more details.
>> > +
>> > +
>> > Support for platforms other than Linux
>> > --------------------------------------
>> >
>> > @@ -316,6 +331,7 @@ replies. Here is a list of the ones that do:
>> > * ``VHOST_USER_GET_VRING_BASE``
>> > * ``VHOST_USER_SET_LOG_BASE`` (if ``VHOST_USER_PROTOCOL_F_LOG_SHMFD``)
>> > * ``VHOST_USER_GET_INFLIGHT_FD`` (if ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD``)
>> > +* ``VHOST_USER_GET_BACKEND_SPECS`` (if ``VHOST_USER_PROTOCOL_F_STANDALONE``)
>> >
>> > .. seealso::
>> >
>> > @@ -396,9 +412,10 @@ must support changing some configuration aspects on the fly.
>> > Multiple queue support
>> > ----------------------
>> >
>> > -Many devices have a fixed number of virtqueues. In this case the front-end
>> > -already knows the number of available virtqueues without communicating with the
>> > -back-end.
>> > +Many devices have a fixed number of virtqueues. In this case the
>> > +*front-end* usually already knows the number of available virtqueues
>> > +without communicating with the back-end. For standalone daemons this
>> > +number can be can be probed with the ``VHOST_USER_GET_MIN_VQ`` message.
>> >
>> > Some devices do not have a fixed number of virtqueues. Instead the maximum
>> > number of virtqueues is chosen by the back-end. The number can depend on host
>> > @@ -885,6 +902,23 @@ Protocol features
>> > #define VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS 15
>> > #define VHOST_USER_PROTOCOL_F_STATUS 16
>> > #define VHOST_USER_PROTOCOL_F_XEN_MMAP 17
>> > + #define VHOST_USER_PROTOCOL_F_PROBE 18
>> > +
>> > +.. _probing_features:
>> > +
>> > +Probing features for standalone daemons
>> > +---------------------------------------
>> > +
>> > +The protocol feature ``VHOST_USER_PROTOCOL_F_PROBE`` enables a number
>> > +of additional messages which allow the *front-end* to probe details
>> > +about the VirtIO device from the *back-end*. However for a *back-end*
>> > +to be described as standalone it must also support:
>> > +
>> > + * ``VHOST_USER_PROTOCOL_F_STATUS``
>> > + * ``VHOST_USER_PROTOCOL_F_CONFIG`` (if there is a config space)
>> > +
>> > +which are required to ensure the *back-end* daemon can operate
>> > +without the *front-end* managing some aspects of its configuration.
>> >
>> > Front-end message types
>> > -----------------------
>> > @@ -1440,6 +1474,42 @@ Front-end message types
>> > query the back-end for its device status as defined in the Virtio
>> > specification.
>> >
>> > +``VHOST_USER_GET_DEVICE_ID``
>> > + :id: 41
>> > + :request payload: N/A
>> > + :reply payload: ``u32``
>> > +
>> > + When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
>> > + successfully negotiated, this message is submitted by the front-end
>> > + to query what VirtIO device the back-end support. This is intended
>> > + to remove the need for the front-end to know ahead of time what the
>> > + VirtIO device the backend emulates is.
>> > +
>> > +``VHOST_USER_GET_CONFIG_SIZE``
>> > + :id: 42
>> > + :request payload: N/A
>> > + :reply payload: ``u32``
>> > +
>> > + When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
>> > + successfully negotiated, this message is submitted by the front-end
>> > + to query the size of the VirtIO device's config space. This is
>> > + intended to remove the need for the front-end to know ahead of time
>> > + what the size is. Replying with 0 when
>> > + ``VHOST_USER_PROTOCOL_F_CONFIG`` has been negotiated would indicate
>> > + an bug.
>> > +
>> > +``VHOST_USER_GET_MIN_VQ``
>> > + :id: 43
>> > + :request payload: N/A
>> > + :reply payload: ``u32``
>> > +
>> > + When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
>> > + successfully negotiated, this message is submitted by the front-end to
>> > + query minimum number of VQ's required to support the device. A
>> > + device may support more than this number of VQ's if it advertises
>> > + the ``VHOST_USER_PROTOCOL_F_MQ`` protocol feature. Reporting a
>> > + number greater than the result of ``VHOST_USER_GET_QUEUE_NUM`` would
>> > + indicate a bug.
>> >
>> > Maybe I lack some background, but not sure what min_vq is here?
>>
>> There will be a minimum number of queues you need to support the device.
>> For example the virtio-sound spec specifies you need four queues:
>> control, event, tx, rx
>
> I don't understand why the front-end needs to know that? The backend
> already reports the number of queues and not all of them need to be
> initialized by the driver.
But how many don't need to be initialised? We can't just skip:
/* Allocate queues */
vub->vqs = g_ptr_array_sized_new(vub->num_vqs);
for (int i = 0; i < vub->num_vqs; i++) {
g_ptr_array_add(vub->vqs,
virtio_add_queue(vdev, vub->vq_size, vub_handle_output));
}
Or are you saying just require probe-able backends to support
VHOST_USER_PROTOCOL_F_MQ and have it always report the minimmum number
of queues if it is not a MQ capable device?
>> > This looks like quering the number of VQs the backend requires/uses.
>> > Which, in case of MQ, it may be bigger (which is where I assume comes the `min`
>> > part, if we consider `VHOST_USER_GET_QUEUE_NUM` the `max`).
>>
>> The MQ extension is currently used by networking but in theory any
>> device could attempt to parallelism by extending the number of virt
>> queues needed. So for net you get:
>>
>> receiveq1
>> transmitq1
>> optional controlq
>>
>> So VHOST_USER_GET_MIN_VQ would report 2 or 3 (if VIRTIO_NET_F_CTRL_VQ is
>> negotiated).
>
> I'm confused. VHOST_USER_GET_MIN_VQ comes before VIRTIO Feature Bit
> negotiation (VIRTIO_NET_F_CTRL_VQ).
>
>> However VHOST_USER_GET_QUEUE_NUM is only usable if
>> VIRTIO_NET_F_MQ has been negotiated and could report more.
>
> I don't understand. This patch adds a new feature and it can require
> VHOST_USER_PROTOCOL_F_MQ. There are no existing back-ends that require
> backwards compatibility.
>
> Stefan
--
Alex Bennée
Virtualisation Tech Lead @ Linaro
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 27+ messages in thread* Re: [virtio-dev] [RFC PATCH v2] docs/interop: define PROBE feature for vhost-user VirtIO devices
2023-09-08 6:41 ` Alex Bennée
(?)
(?)
@ 2023-09-08 10:11 ` Stefan Hajnoczi
2023-09-08 11:59 ` Alex Bennée
-1 siblings, 1 reply; 27+ messages in thread
From: Stefan Hajnoczi @ 2023-09-08 10:11 UTC (permalink / raw)
To: Alex Bennée
Cc: Stefan Hajnoczi, Albert Esteve, slp, mst, marcandre.lureau,
viresh.kumar, sgarzare, takahiro.akashi, erik.schilling,
manos.pitsidianakis, mathieu.poirier, qemu-devel, virtio-dev,
virtio-comment
On Fri, 8 Sept 2023 at 02:43, Alex Bennée <alex.bennee@linaro.org> wrote:
>
>
> Stefan Hajnoczi <stefanha@redhat.com> writes:
>
> > On Tue, Sep 05, 2023 at 10:34:11AM +0100, Alex Bennée wrote:
> >>
> >> Albert Esteve <aesteve@redhat.com> writes:
> >>
> >> > This looks great! Thanks for this proposal.
> >> >
> >> > On Fri, Sep 1, 2023 at 1:00 PM Alex Bennée <alex.bennee@linaro.org> wrote:
> >> >
> >> > Currently QEMU has to know some details about the VirtIO device
> >> > supported by a vhost-user daemon to be able to setup the guest. This
> >> > makes it hard for QEMU to add support for additional vhost-user
> >> > daemons without adding specific stubs for each additional VirtIO
> >> > device.
> >> >
> >> > This patch suggests a new feature flag (VHOST_USER_PROTOCOL_F_PROBE)
> >> > which the back-end can advertise which allows a probe message to be
> >> > sent to get all the details QEMU needs to know in one message.
> >> >
> >> > Together with the existing features VHOST_USER_PROTOCOL_F_STATUS and
> >> > VHOST_USER_PROTOCOL_F_CONFIG we can create "standalone" vhost-user
> >> > daemons which are capable of handling all aspects of the VirtIO
> >> > transactions with only a generic stub on the QEMU side. These daemons
> >> > can also be used without QEMU in situations where there isn't a full
> >> > VMM managing their setup.
> >> >
> >> > Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> >> >
> >> > ---
> >> > v2
> >> > - dropped F_STANDALONE in favour of F_PROBE
> >> > - split probe details across several messages
> >> > - probe messages don't automatically imply a standalone daemon
> >> > - add wording where probe details interact (F_MQ/F_CONFIG)
> >> > - define VMM and make clear QEMU is only one of many potential VMMs
> >> > - reword commit message
> >> > ---
> >> > docs/interop/vhost-user.rst | 90 ++++++++++++++++++++++++++++++++-----
> >> > hw/virtio/vhost-user.c | 8 ++++
> >> > 2 files changed, 88 insertions(+), 10 deletions(-)
> >> >
> >> > diff --git a/docs/interop/vhost-user.rst b/docs/interop/vhost-user.rst
> >> > index 5a070adbc1..ba3b5e07b7 100644
> >> > --- a/docs/interop/vhost-user.rst
> >> > +++ b/docs/interop/vhost-user.rst
> >> > @@ -7,6 +7,7 @@ Vhost-user Protocol
> >> > ..
> >> > Copyright 2014 Virtual Open Systems Sarl.
> >> > Copyright 2019 Intel Corporation
> >> > + Copyright 2023 Linaro Ltd
> >> > Licence: This work is licensed under the terms of the GNU GPL,
> >> > version 2 or later. See the COPYING file in the top-level
> >> > directory.
> >> > @@ -27,17 +28,31 @@ The protocol defines 2 sides of the communication, *front-end* and
> >> > *back-end*. The *front-end* is the application that shares its virtqueues, in
> >> > our case QEMU. The *back-end* is the consumer of the virtqueues.
> >> >
> >> > -In the current implementation QEMU is the *front-end*, and the *back-end*
> >> > -is the external process consuming the virtio queues, for example a
> >> > -software Ethernet switch running in user space, such as Snabbswitch,
> >> > -or a block device back-end processing read & write to a virtual
> >> > -disk. In order to facilitate interoperability between various back-end
> >> > -implementations, it is recommended to follow the :ref:`Backend program
> >> > -conventions <backend_conventions>`.
> >> > +In the current implementation a Virtual Machine Manager (VMM) such as
> >> > +QEMU is the *front-end*, and the *back-end* is the external process
> >> > +consuming the virtio queues, for example a software Ethernet switch
> >> > +running in user space, such as Snabbswitch, or a block device back-end
> >> > +processing read & write to a virtual disk. In order to facilitate
> >> > +interoperability between various back-end implementations, it is
> >> > +recommended to follow the :ref:`Backend program conventions
> >> > +<backend_conventions>`.
> >> >
> >> > The *front-end* and *back-end* can be either a client (i.e. connecting) or
> >> > server (listening) in the socket communication.
> >> >
> >> > +Probing device details
> >> > +----------------------
> >> > +
> >> > +Traditionally the vhost-user daemon *back-end* shares configuration
> >> > +responsibilities with the VMM *front-end* which needs to know certain
> >> > +key bits of information about the device. This means the VMM needs to
> >> > +define at least a minimal stub for each VirtIO device it wants to
> >> > +support. If the daemon supports the right set of protocol features the
> >> > +VMM can probe the daemon for the information it needs to setup the
> >> > +device. See :ref:`Probing features for standalone daemons
> >> > +<probing_features>` for more details.
> >> > +
> >> > +
> >> > Support for platforms other than Linux
> >> > --------------------------------------
> >> >
> >> > @@ -316,6 +331,7 @@ replies. Here is a list of the ones that do:
> >> > * ``VHOST_USER_GET_VRING_BASE``
> >> > * ``VHOST_USER_SET_LOG_BASE`` (if ``VHOST_USER_PROTOCOL_F_LOG_SHMFD``)
> >> > * ``VHOST_USER_GET_INFLIGHT_FD`` (if ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD``)
> >> > +* ``VHOST_USER_GET_BACKEND_SPECS`` (if ``VHOST_USER_PROTOCOL_F_STANDALONE``)
> >> >
> >> > .. seealso::
> >> >
> >> > @@ -396,9 +412,10 @@ must support changing some configuration aspects on the fly.
> >> > Multiple queue support
> >> > ----------------------
> >> >
> >> > -Many devices have a fixed number of virtqueues. In this case the front-end
> >> > -already knows the number of available virtqueues without communicating with the
> >> > -back-end.
> >> > +Many devices have a fixed number of virtqueues. In this case the
> >> > +*front-end* usually already knows the number of available virtqueues
> >> > +without communicating with the back-end. For standalone daemons this
> >> > +number can be can be probed with the ``VHOST_USER_GET_MIN_VQ`` message.
> >> >
> >> > Some devices do not have a fixed number of virtqueues. Instead the maximum
> >> > number of virtqueues is chosen by the back-end. The number can depend on host
> >> > @@ -885,6 +902,23 @@ Protocol features
> >> > #define VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS 15
> >> > #define VHOST_USER_PROTOCOL_F_STATUS 16
> >> > #define VHOST_USER_PROTOCOL_F_XEN_MMAP 17
> >> > + #define VHOST_USER_PROTOCOL_F_PROBE 18
> >> > +
> >> > +.. _probing_features:
> >> > +
> >> > +Probing features for standalone daemons
> >> > +---------------------------------------
> >> > +
> >> > +The protocol feature ``VHOST_USER_PROTOCOL_F_PROBE`` enables a number
> >> > +of additional messages which allow the *front-end* to probe details
> >> > +about the VirtIO device from the *back-end*. However for a *back-end*
> >> > +to be described as standalone it must also support:
> >> > +
> >> > + * ``VHOST_USER_PROTOCOL_F_STATUS``
> >> > + * ``VHOST_USER_PROTOCOL_F_CONFIG`` (if there is a config space)
> >> > +
> >> > +which are required to ensure the *back-end* daemon can operate
> >> > +without the *front-end* managing some aspects of its configuration.
> >> >
> >> > Front-end message types
> >> > -----------------------
> >> > @@ -1440,6 +1474,42 @@ Front-end message types
> >> > query the back-end for its device status as defined in the Virtio
> >> > specification.
> >> >
> >> > +``VHOST_USER_GET_DEVICE_ID``
> >> > + :id: 41
> >> > + :request payload: N/A
> >> > + :reply payload: ``u32``
> >> > +
> >> > + When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
> >> > + successfully negotiated, this message is submitted by the front-end
> >> > + to query what VirtIO device the back-end support. This is intended
> >> > + to remove the need for the front-end to know ahead of time what the
> >> > + VirtIO device the backend emulates is.
> >> > +
> >> > +``VHOST_USER_GET_CONFIG_SIZE``
> >> > + :id: 42
> >> > + :request payload: N/A
> >> > + :reply payload: ``u32``
> >> > +
> >> > + When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
> >> > + successfully negotiated, this message is submitted by the front-end
> >> > + to query the size of the VirtIO device's config space. This is
> >> > + intended to remove the need for the front-end to know ahead of time
> >> > + what the size is. Replying with 0 when
> >> > + ``VHOST_USER_PROTOCOL_F_CONFIG`` has been negotiated would indicate
> >> > + an bug.
> >> > +
> >> > +``VHOST_USER_GET_MIN_VQ``
> >> > + :id: 43
> >> > + :request payload: N/A
> >> > + :reply payload: ``u32``
> >> > +
> >> > + When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
> >> > + successfully negotiated, this message is submitted by the front-end to
> >> > + query minimum number of VQ's required to support the device. A
> >> > + device may support more than this number of VQ's if it advertises
> >> > + the ``VHOST_USER_PROTOCOL_F_MQ`` protocol feature. Reporting a
> >> > + number greater than the result of ``VHOST_USER_GET_QUEUE_NUM`` would
> >> > + indicate a bug.
> >> >
> >> > Maybe I lack some background, but not sure what min_vq is here?
> >>
> >> There will be a minimum number of queues you need to support the device.
> >> For example the virtio-sound spec specifies you need four queues:
> >> control, event, tx, rx
> >
> > I don't understand why the front-end needs to know that? The backend
> > already reports the number of queues and not all of them need to be
> > initialized by the driver.
>
> But how many don't need to be initialised? We can't just skip:
>
> /* Allocate queues */
> vub->vqs = g_ptr_array_sized_new(vub->num_vqs);
> for (int i = 0; i < vub->num_vqs; i++) {
> g_ptr_array_add(vub->vqs,
> virtio_add_queue(vdev, vub->vq_size, vub_handle_output));
> }
>
> Or are you saying just require probe-able backends to support
> VHOST_USER_PROTOCOL_F_MQ and have it always report the minimmum number
> of queues if it is not a MQ capable device?
The front-end should prepare to allow the maximum number of virtqueues
returned by VHOST_USER_GET_QUEUE_NUM (VHOST_USER_PROTOCOL_F_MQ).
VIRTIO Transports have a way to query the maximum number of queues but
not a way to query the minimum number of queues. Why is the minimum
necessary?
Stefan
>
> >> > This looks like quering the number of VQs the backend requires/uses.
> >> > Which, in case of MQ, it may be bigger (which is where I assume comes the `min`
> >> > part, if we consider `VHOST_USER_GET_QUEUE_NUM` the `max`).
> >>
> >> The MQ extension is currently used by networking but in theory any
> >> device could attempt to parallelism by extending the number of virt
> >> queues needed. So for net you get:
> >>
> >> receiveq1
> >> transmitq1
> >> optional controlq
> >>
> >> So VHOST_USER_GET_MIN_VQ would report 2 or 3 (if VIRTIO_NET_F_CTRL_VQ is
> >> negotiated).
> >
> > I'm confused. VHOST_USER_GET_MIN_VQ comes before VIRTIO Feature Bit
> > negotiation (VIRTIO_NET_F_CTRL_VQ).
> >
> >> However VHOST_USER_GET_QUEUE_NUM is only usable if
> >> VIRTIO_NET_F_MQ has been negotiated and could report more.
> >
> > I don't understand. This patch adds a new feature and it can require
> > VHOST_USER_PROTOCOL_F_MQ. There are no existing back-ends that require
> > backwards compatibility.
> >
> > Stefan
>
>
> --
> Alex Bennée
> Virtualisation Tech Lead @ Linaro
>
^ permalink raw reply [flat|nested] 27+ messages in thread* [virtio-comment] Re: [virtio-dev] [RFC PATCH v2] docs/interop: define PROBE feature for vhost-user VirtIO devices
2023-09-08 10:11 ` Stefan Hajnoczi
2023-09-08 11:59 ` Alex Bennée
@ 2023-09-08 11:59 ` Alex Bennée
0 siblings, 0 replies; 27+ messages in thread
From: Alex Bennée @ 2023-09-08 11:59 UTC (permalink / raw)
To: Stefan Hajnoczi
Cc: Stefan Hajnoczi, Albert Esteve, slp, mst, marcandre.lureau,
viresh.kumar, sgarzare, takahiro.akashi, erik.schilling,
manos.pitsidianakis, mathieu.poirier, qemu-devel, virtio-dev,
virtio-comment
Stefan Hajnoczi <stefanha@gmail.com> writes:
> On Fri, 8 Sept 2023 at 02:43, Alex Bennée <alex.bennee@linaro.org> wrote:
>>
>>
>> Stefan Hajnoczi <stefanha@redhat.com> writes:
>>
>> > On Tue, Sep 05, 2023 at 10:34:11AM +0100, Alex Bennée wrote:
>> >>
>> >> Albert Esteve <aesteve@redhat.com> writes:
>> >>
>> >> > This looks great! Thanks for this proposal.
>> >> >
>> >> > On Fri, Sep 1, 2023 at 1:00 PM Alex Bennée <alex.bennee@linaro.org> wrote:
>> >> >
>> >> > Currently QEMU has to know some details about the VirtIO device
>> >> > supported by a vhost-user daemon to be able to setup the guest. This
>> >> > makes it hard for QEMU to add support for additional vhost-user
>> >> > daemons without adding specific stubs for each additional VirtIO
>> >> > device.
>> >> >
>> >> > This patch suggests a new feature flag (VHOST_USER_PROTOCOL_F_PROBE)
>> >> > which the back-end can advertise which allows a probe message to be
>> >> > sent to get all the details QEMU needs to know in one message.
>> >> >
>> >> > Together with the existing features VHOST_USER_PROTOCOL_F_STATUS and
>> >> > VHOST_USER_PROTOCOL_F_CONFIG we can create "standalone" vhost-user
>> >> > daemons which are capable of handling all aspects of the VirtIO
>> >> > transactions with only a generic stub on the QEMU side. These daemons
>> >> > can also be used without QEMU in situations where there isn't a full
>> >> > VMM managing their setup.
>> >> >
>> >> > Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>> >> >
>> >> > ---
>> >> > v2
>> >> > - dropped F_STANDALONE in favour of F_PROBE
>> >> > - split probe details across several messages
>> >> > - probe messages don't automatically imply a standalone daemon
>> >> > - add wording where probe details interact (F_MQ/F_CONFIG)
>> >> > - define VMM and make clear QEMU is only one of many potential VMMs
>> >> > - reword commit message
>> >> > ---
>> >> > docs/interop/vhost-user.rst | 90 ++++++++++++++++++++++++++++++++-----
>> >> > hw/virtio/vhost-user.c | 8 ++++
>> >> > 2 files changed, 88 insertions(+), 10 deletions(-)
>> >> >
>> >> > diff --git a/docs/interop/vhost-user.rst b/docs/interop/vhost-user.rst
>> >> > index 5a070adbc1..ba3b5e07b7 100644
>> >> > --- a/docs/interop/vhost-user.rst
>> >> > +++ b/docs/interop/vhost-user.rst
>> >> > @@ -7,6 +7,7 @@ Vhost-user Protocol
>> >> > ..
>> >> > Copyright 2014 Virtual Open Systems Sarl.
>> >> > Copyright 2019 Intel Corporation
>> >> > + Copyright 2023 Linaro Ltd
>> >> > Licence: This work is licensed under the terms of the GNU GPL,
>> >> > version 2 or later. See the COPYING file in the top-level
>> >> > directory.
>> >> > @@ -27,17 +28,31 @@ The protocol defines 2 sides of the communication, *front-end* and
>> >> > *back-end*. The *front-end* is the application that shares its virtqueues, in
>> >> > our case QEMU. The *back-end* is the consumer of the virtqueues.
>> >> >
>> >> > -In the current implementation QEMU is the *front-end*, and the *back-end*
>> >> > -is the external process consuming the virtio queues, for example a
>> >> > -software Ethernet switch running in user space, such as Snabbswitch,
>> >> > -or a block device back-end processing read & write to a virtual
>> >> > -disk. In order to facilitate interoperability between various back-end
>> >> > -implementations, it is recommended to follow the :ref:`Backend program
>> >> > -conventions <backend_conventions>`.
>> >> > +In the current implementation a Virtual Machine Manager (VMM) such as
>> >> > +QEMU is the *front-end*, and the *back-end* is the external process
>> >> > +consuming the virtio queues, for example a software Ethernet switch
>> >> > +running in user space, such as Snabbswitch, or a block device back-end
>> >> > +processing read & write to a virtual disk. In order to facilitate
>> >> > +interoperability between various back-end implementations, it is
>> >> > +recommended to follow the :ref:`Backend program conventions
>> >> > +<backend_conventions>`.
>> >> >
>> >> > The *front-end* and *back-end* can be either a client (i.e. connecting) or
>> >> > server (listening) in the socket communication.
>> >> >
>> >> > +Probing device details
>> >> > +----------------------
>> >> > +
>> >> > +Traditionally the vhost-user daemon *back-end* shares configuration
>> >> > +responsibilities with the VMM *front-end* which needs to know certain
>> >> > +key bits of information about the device. This means the VMM needs to
>> >> > +define at least a minimal stub for each VirtIO device it wants to
>> >> > +support. If the daemon supports the right set of protocol features the
>> >> > +VMM can probe the daemon for the information it needs to setup the
>> >> > +device. See :ref:`Probing features for standalone daemons
>> >> > +<probing_features>` for more details.
>> >> > +
>> >> > +
>> >> > Support for platforms other than Linux
>> >> > --------------------------------------
>> >> >
>> >> > @@ -316,6 +331,7 @@ replies. Here is a list of the ones that do:
>> >> > * ``VHOST_USER_GET_VRING_BASE``
>> >> > * ``VHOST_USER_SET_LOG_BASE`` (if ``VHOST_USER_PROTOCOL_F_LOG_SHMFD``)
>> >> > * ``VHOST_USER_GET_INFLIGHT_FD`` (if ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD``)
>> >> > +* ``VHOST_USER_GET_BACKEND_SPECS`` (if ``VHOST_USER_PROTOCOL_F_STANDALONE``)
>> >> >
>> >> > .. seealso::
>> >> >
>> >> > @@ -396,9 +412,10 @@ must support changing some configuration aspects on the fly.
>> >> > Multiple queue support
>> >> > ----------------------
>> >> >
>> >> > -Many devices have a fixed number of virtqueues. In this case the front-end
>> >> > -already knows the number of available virtqueues without communicating with the
>> >> > -back-end.
>> >> > +Many devices have a fixed number of virtqueues. In this case the
>> >> > +*front-end* usually already knows the number of available virtqueues
>> >> > +without communicating with the back-end. For standalone daemons this
>> >> > +number can be can be probed with the ``VHOST_USER_GET_MIN_VQ`` message.
>> >> >
>> >> > Some devices do not have a fixed number of virtqueues. Instead the maximum
>> >> > number of virtqueues is chosen by the back-end. The number can depend on host
>> >> > @@ -885,6 +902,23 @@ Protocol features
>> >> > #define VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS 15
>> >> > #define VHOST_USER_PROTOCOL_F_STATUS 16
>> >> > #define VHOST_USER_PROTOCOL_F_XEN_MMAP 17
>> >> > + #define VHOST_USER_PROTOCOL_F_PROBE 18
>> >> > +
>> >> > +.. _probing_features:
>> >> > +
>> >> > +Probing features for standalone daemons
>> >> > +---------------------------------------
>> >> > +
>> >> > +The protocol feature ``VHOST_USER_PROTOCOL_F_PROBE`` enables a number
>> >> > +of additional messages which allow the *front-end* to probe details
>> >> > +about the VirtIO device from the *back-end*. However for a *back-end*
>> >> > +to be described as standalone it must also support:
>> >> > +
>> >> > + * ``VHOST_USER_PROTOCOL_F_STATUS``
>> >> > + * ``VHOST_USER_PROTOCOL_F_CONFIG`` (if there is a config space)
>> >> > +
>> >> > +which are required to ensure the *back-end* daemon can operate
>> >> > +without the *front-end* managing some aspects of its configuration.
>> >> >
>> >> > Front-end message types
>> >> > -----------------------
>> >> > @@ -1440,6 +1474,42 @@ Front-end message types
>> >> > query the back-end for its device status as defined in the Virtio
>> >> > specification.
>> >> >
>> >> > +``VHOST_USER_GET_DEVICE_ID``
>> >> > + :id: 41
>> >> > + :request payload: N/A
>> >> > + :reply payload: ``u32``
>> >> > +
>> >> > + When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
>> >> > + successfully negotiated, this message is submitted by the front-end
>> >> > + to query what VirtIO device the back-end support. This is intended
>> >> > + to remove the need for the front-end to know ahead of time what the
>> >> > + VirtIO device the backend emulates is.
>> >> > +
>> >> > +``VHOST_USER_GET_CONFIG_SIZE``
>> >> > + :id: 42
>> >> > + :request payload: N/A
>> >> > + :reply payload: ``u32``
>> >> > +
>> >> > + When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
>> >> > + successfully negotiated, this message is submitted by the front-end
>> >> > + to query the size of the VirtIO device's config space. This is
>> >> > + intended to remove the need for the front-end to know ahead of time
>> >> > + what the size is. Replying with 0 when
>> >> > + ``VHOST_USER_PROTOCOL_F_CONFIG`` has been negotiated would indicate
>> >> > + an bug.
>> >> > +
>> >> > +``VHOST_USER_GET_MIN_VQ``
>> >> > + :id: 43
>> >> > + :request payload: N/A
>> >> > + :reply payload: ``u32``
>> >> > +
>> >> > + When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
>> >> > + successfully negotiated, this message is submitted by the front-end to
>> >> > + query minimum number of VQ's required to support the device. A
>> >> > + device may support more than this number of VQ's if it advertises
>> >> > + the ``VHOST_USER_PROTOCOL_F_MQ`` protocol feature. Reporting a
>> >> > + number greater than the result of ``VHOST_USER_GET_QUEUE_NUM`` would
>> >> > + indicate a bug.
>> >> >
>> >> > Maybe I lack some background, but not sure what min_vq is here?
>> >>
>> >> There will be a minimum number of queues you need to support the device.
>> >> For example the virtio-sound spec specifies you need four queues:
>> >> control, event, tx, rx
>> >
>> > I don't understand why the front-end needs to know that? The backend
>> > already reports the number of queues and not all of them need to be
>> > initialized by the driver.
>>
>> But how many don't need to be initialised? We can't just skip:
>>
>> /* Allocate queues */
>> vub->vqs = g_ptr_array_sized_new(vub->num_vqs);
>> for (int i = 0; i < vub->num_vqs; i++) {
>> g_ptr_array_add(vub->vqs,
>> virtio_add_queue(vdev, vub->vq_size, vub_handle_output));
>> }
>>
>> Or are you saying just require probe-able backends to support
>> VHOST_USER_PROTOCOL_F_MQ and have it always report the minimmum number
>> of queues if it is not a MQ capable device?
>
> The front-end should prepare to allow the maximum number of virtqueues
> returned by VHOST_USER_GET_QUEUE_NUM (VHOST_USER_PROTOCOL_F_MQ).
>
> VIRTIO Transports have a way to query the maximum number of queues but
> not a way to query the minimum number of queues. Why is the minimum
> necessary?
It seems excessive to automatically create the maximum number of VQs. I
guess for backends that don't support the MQ feature (i.e. a variable
number of VQs) we could just say VHOST_USER_GET_QUEUE_NUM == min. But
now we are overloading a different message originally added for
something else.
--
Alex Bennée
Virtualisation Tech Lead @ Linaro
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 27+ messages in thread* Re: [virtio-dev] [RFC PATCH v2] docs/interop: define PROBE feature for vhost-user VirtIO devices
@ 2023-09-08 11:59 ` Alex Bennée
0 siblings, 0 replies; 27+ messages in thread
From: Alex Bennée @ 2023-09-08 11:59 UTC (permalink / raw)
To: Stefan Hajnoczi
Cc: Stefan Hajnoczi, Albert Esteve, slp, mst, marcandre.lureau,
viresh.kumar, sgarzare, takahiro.akashi, erik.schilling,
manos.pitsidianakis, mathieu.poirier, qemu-devel, virtio-dev,
virtio-comment
Stefan Hajnoczi <stefanha@gmail.com> writes:
> On Fri, 8 Sept 2023 at 02:43, Alex Bennée <alex.bennee@linaro.org> wrote:
>>
>>
>> Stefan Hajnoczi <stefanha@redhat.com> writes:
>>
>> > On Tue, Sep 05, 2023 at 10:34:11AM +0100, Alex Bennée wrote:
>> >>
>> >> Albert Esteve <aesteve@redhat.com> writes:
>> >>
>> >> > This looks great! Thanks for this proposal.
>> >> >
>> >> > On Fri, Sep 1, 2023 at 1:00 PM Alex Bennée <alex.bennee@linaro.org> wrote:
>> >> >
>> >> > Currently QEMU has to know some details about the VirtIO device
>> >> > supported by a vhost-user daemon to be able to setup the guest. This
>> >> > makes it hard for QEMU to add support for additional vhost-user
>> >> > daemons without adding specific stubs for each additional VirtIO
>> >> > device.
>> >> >
>> >> > This patch suggests a new feature flag (VHOST_USER_PROTOCOL_F_PROBE)
>> >> > which the back-end can advertise which allows a probe message to be
>> >> > sent to get all the details QEMU needs to know in one message.
>> >> >
>> >> > Together with the existing features VHOST_USER_PROTOCOL_F_STATUS and
>> >> > VHOST_USER_PROTOCOL_F_CONFIG we can create "standalone" vhost-user
>> >> > daemons which are capable of handling all aspects of the VirtIO
>> >> > transactions with only a generic stub on the QEMU side. These daemons
>> >> > can also be used without QEMU in situations where there isn't a full
>> >> > VMM managing their setup.
>> >> >
>> >> > Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>> >> >
>> >> > ---
>> >> > v2
>> >> > - dropped F_STANDALONE in favour of F_PROBE
>> >> > - split probe details across several messages
>> >> > - probe messages don't automatically imply a standalone daemon
>> >> > - add wording where probe details interact (F_MQ/F_CONFIG)
>> >> > - define VMM and make clear QEMU is only one of many potential VMMs
>> >> > - reword commit message
>> >> > ---
>> >> > docs/interop/vhost-user.rst | 90 ++++++++++++++++++++++++++++++++-----
>> >> > hw/virtio/vhost-user.c | 8 ++++
>> >> > 2 files changed, 88 insertions(+), 10 deletions(-)
>> >> >
>> >> > diff --git a/docs/interop/vhost-user.rst b/docs/interop/vhost-user.rst
>> >> > index 5a070adbc1..ba3b5e07b7 100644
>> >> > --- a/docs/interop/vhost-user.rst
>> >> > +++ b/docs/interop/vhost-user.rst
>> >> > @@ -7,6 +7,7 @@ Vhost-user Protocol
>> >> > ..
>> >> > Copyright 2014 Virtual Open Systems Sarl.
>> >> > Copyright 2019 Intel Corporation
>> >> > + Copyright 2023 Linaro Ltd
>> >> > Licence: This work is licensed under the terms of the GNU GPL,
>> >> > version 2 or later. See the COPYING file in the top-level
>> >> > directory.
>> >> > @@ -27,17 +28,31 @@ The protocol defines 2 sides of the communication, *front-end* and
>> >> > *back-end*. The *front-end* is the application that shares its virtqueues, in
>> >> > our case QEMU. The *back-end* is the consumer of the virtqueues.
>> >> >
>> >> > -In the current implementation QEMU is the *front-end*, and the *back-end*
>> >> > -is the external process consuming the virtio queues, for example a
>> >> > -software Ethernet switch running in user space, such as Snabbswitch,
>> >> > -or a block device back-end processing read & write to a virtual
>> >> > -disk. In order to facilitate interoperability between various back-end
>> >> > -implementations, it is recommended to follow the :ref:`Backend program
>> >> > -conventions <backend_conventions>`.
>> >> > +In the current implementation a Virtual Machine Manager (VMM) such as
>> >> > +QEMU is the *front-end*, and the *back-end* is the external process
>> >> > +consuming the virtio queues, for example a software Ethernet switch
>> >> > +running in user space, such as Snabbswitch, or a block device back-end
>> >> > +processing read & write to a virtual disk. In order to facilitate
>> >> > +interoperability between various back-end implementations, it is
>> >> > +recommended to follow the :ref:`Backend program conventions
>> >> > +<backend_conventions>`.
>> >> >
>> >> > The *front-end* and *back-end* can be either a client (i.e. connecting) or
>> >> > server (listening) in the socket communication.
>> >> >
>> >> > +Probing device details
>> >> > +----------------------
>> >> > +
>> >> > +Traditionally the vhost-user daemon *back-end* shares configuration
>> >> > +responsibilities with the VMM *front-end* which needs to know certain
>> >> > +key bits of information about the device. This means the VMM needs to
>> >> > +define at least a minimal stub for each VirtIO device it wants to
>> >> > +support. If the daemon supports the right set of protocol features the
>> >> > +VMM can probe the daemon for the information it needs to setup the
>> >> > +device. See :ref:`Probing features for standalone daemons
>> >> > +<probing_features>` for more details.
>> >> > +
>> >> > +
>> >> > Support for platforms other than Linux
>> >> > --------------------------------------
>> >> >
>> >> > @@ -316,6 +331,7 @@ replies. Here is a list of the ones that do:
>> >> > * ``VHOST_USER_GET_VRING_BASE``
>> >> > * ``VHOST_USER_SET_LOG_BASE`` (if ``VHOST_USER_PROTOCOL_F_LOG_SHMFD``)
>> >> > * ``VHOST_USER_GET_INFLIGHT_FD`` (if ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD``)
>> >> > +* ``VHOST_USER_GET_BACKEND_SPECS`` (if ``VHOST_USER_PROTOCOL_F_STANDALONE``)
>> >> >
>> >> > .. seealso::
>> >> >
>> >> > @@ -396,9 +412,10 @@ must support changing some configuration aspects on the fly.
>> >> > Multiple queue support
>> >> > ----------------------
>> >> >
>> >> > -Many devices have a fixed number of virtqueues. In this case the front-end
>> >> > -already knows the number of available virtqueues without communicating with the
>> >> > -back-end.
>> >> > +Many devices have a fixed number of virtqueues. In this case the
>> >> > +*front-end* usually already knows the number of available virtqueues
>> >> > +without communicating with the back-end. For standalone daemons this
>> >> > +number can be can be probed with the ``VHOST_USER_GET_MIN_VQ`` message.
>> >> >
>> >> > Some devices do not have a fixed number of virtqueues. Instead the maximum
>> >> > number of virtqueues is chosen by the back-end. The number can depend on host
>> >> > @@ -885,6 +902,23 @@ Protocol features
>> >> > #define VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS 15
>> >> > #define VHOST_USER_PROTOCOL_F_STATUS 16
>> >> > #define VHOST_USER_PROTOCOL_F_XEN_MMAP 17
>> >> > + #define VHOST_USER_PROTOCOL_F_PROBE 18
>> >> > +
>> >> > +.. _probing_features:
>> >> > +
>> >> > +Probing features for standalone daemons
>> >> > +---------------------------------------
>> >> > +
>> >> > +The protocol feature ``VHOST_USER_PROTOCOL_F_PROBE`` enables a number
>> >> > +of additional messages which allow the *front-end* to probe details
>> >> > +about the VirtIO device from the *back-end*. However for a *back-end*
>> >> > +to be described as standalone it must also support:
>> >> > +
>> >> > + * ``VHOST_USER_PROTOCOL_F_STATUS``
>> >> > + * ``VHOST_USER_PROTOCOL_F_CONFIG`` (if there is a config space)
>> >> > +
>> >> > +which are required to ensure the *back-end* daemon can operate
>> >> > +without the *front-end* managing some aspects of its configuration.
>> >> >
>> >> > Front-end message types
>> >> > -----------------------
>> >> > @@ -1440,6 +1474,42 @@ Front-end message types
>> >> > query the back-end for its device status as defined in the Virtio
>> >> > specification.
>> >> >
>> >> > +``VHOST_USER_GET_DEVICE_ID``
>> >> > + :id: 41
>> >> > + :request payload: N/A
>> >> > + :reply payload: ``u32``
>> >> > +
>> >> > + When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
>> >> > + successfully negotiated, this message is submitted by the front-end
>> >> > + to query what VirtIO device the back-end support. This is intended
>> >> > + to remove the need for the front-end to know ahead of time what the
>> >> > + VirtIO device the backend emulates is.
>> >> > +
>> >> > +``VHOST_USER_GET_CONFIG_SIZE``
>> >> > + :id: 42
>> >> > + :request payload: N/A
>> >> > + :reply payload: ``u32``
>> >> > +
>> >> > + When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
>> >> > + successfully negotiated, this message is submitted by the front-end
>> >> > + to query the size of the VirtIO device's config space. This is
>> >> > + intended to remove the need for the front-end to know ahead of time
>> >> > + what the size is. Replying with 0 when
>> >> > + ``VHOST_USER_PROTOCOL_F_CONFIG`` has been negotiated would indicate
>> >> > + an bug.
>> >> > +
>> >> > +``VHOST_USER_GET_MIN_VQ``
>> >> > + :id: 43
>> >> > + :request payload: N/A
>> >> > + :reply payload: ``u32``
>> >> > +
>> >> > + When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
>> >> > + successfully negotiated, this message is submitted by the front-end to
>> >> > + query minimum number of VQ's required to support the device. A
>> >> > + device may support more than this number of VQ's if it advertises
>> >> > + the ``VHOST_USER_PROTOCOL_F_MQ`` protocol feature. Reporting a
>> >> > + number greater than the result of ``VHOST_USER_GET_QUEUE_NUM`` would
>> >> > + indicate a bug.
>> >> >
>> >> > Maybe I lack some background, but not sure what min_vq is here?
>> >>
>> >> There will be a minimum number of queues you need to support the device.
>> >> For example the virtio-sound spec specifies you need four queues:
>> >> control, event, tx, rx
>> >
>> > I don't understand why the front-end needs to know that? The backend
>> > already reports the number of queues and not all of them need to be
>> > initialized by the driver.
>>
>> But how many don't need to be initialised? We can't just skip:
>>
>> /* Allocate queues */
>> vub->vqs = g_ptr_array_sized_new(vub->num_vqs);
>> for (int i = 0; i < vub->num_vqs; i++) {
>> g_ptr_array_add(vub->vqs,
>> virtio_add_queue(vdev, vub->vq_size, vub_handle_output));
>> }
>>
>> Or are you saying just require probe-able backends to support
>> VHOST_USER_PROTOCOL_F_MQ and have it always report the minimmum number
>> of queues if it is not a MQ capable device?
>
> The front-end should prepare to allow the maximum number of virtqueues
> returned by VHOST_USER_GET_QUEUE_NUM (VHOST_USER_PROTOCOL_F_MQ).
>
> VIRTIO Transports have a way to query the maximum number of queues but
> not a way to query the minimum number of queues. Why is the minimum
> necessary?
It seems excessive to automatically create the maximum number of VQs. I
guess for backends that don't support the MQ feature (i.e. a variable
number of VQs) we could just say VHOST_USER_GET_QUEUE_NUM == min. But
now we are overloading a different message originally added for
something else.
--
Alex Bennée
Virtualisation Tech Lead @ Linaro
^ permalink raw reply [flat|nested] 27+ messages in thread* Re: [virtio-dev] [RFC PATCH v2] docs/interop: define PROBE feature for vhost-user VirtIO devices
@ 2023-09-08 11:59 ` Alex Bennée
0 siblings, 0 replies; 27+ messages in thread
From: Alex Bennée @ 2023-09-08 11:59 UTC (permalink / raw)
To: Stefan Hajnoczi
Cc: Stefan Hajnoczi, Albert Esteve, slp, mst, marcandre.lureau,
viresh.kumar, sgarzare, takahiro.akashi, erik.schilling,
manos.pitsidianakis, mathieu.poirier, qemu-devel, virtio-dev,
virtio-comment
Stefan Hajnoczi <stefanha@gmail.com> writes:
> On Fri, 8 Sept 2023 at 02:43, Alex Bennée <alex.bennee@linaro.org> wrote:
>>
>>
>> Stefan Hajnoczi <stefanha@redhat.com> writes:
>>
>> > On Tue, Sep 05, 2023 at 10:34:11AM +0100, Alex Bennée wrote:
>> >>
>> >> Albert Esteve <aesteve@redhat.com> writes:
>> >>
>> >> > This looks great! Thanks for this proposal.
>> >> >
>> >> > On Fri, Sep 1, 2023 at 1:00 PM Alex Bennée <alex.bennee@linaro.org> wrote:
>> >> >
>> >> > Currently QEMU has to know some details about the VirtIO device
>> >> > supported by a vhost-user daemon to be able to setup the guest. This
>> >> > makes it hard for QEMU to add support for additional vhost-user
>> >> > daemons without adding specific stubs for each additional VirtIO
>> >> > device.
>> >> >
>> >> > This patch suggests a new feature flag (VHOST_USER_PROTOCOL_F_PROBE)
>> >> > which the back-end can advertise which allows a probe message to be
>> >> > sent to get all the details QEMU needs to know in one message.
>> >> >
>> >> > Together with the existing features VHOST_USER_PROTOCOL_F_STATUS and
>> >> > VHOST_USER_PROTOCOL_F_CONFIG we can create "standalone" vhost-user
>> >> > daemons which are capable of handling all aspects of the VirtIO
>> >> > transactions with only a generic stub on the QEMU side. These daemons
>> >> > can also be used without QEMU in situations where there isn't a full
>> >> > VMM managing their setup.
>> >> >
>> >> > Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>> >> >
>> >> > ---
>> >> > v2
>> >> > - dropped F_STANDALONE in favour of F_PROBE
>> >> > - split probe details across several messages
>> >> > - probe messages don't automatically imply a standalone daemon
>> >> > - add wording where probe details interact (F_MQ/F_CONFIG)
>> >> > - define VMM and make clear QEMU is only one of many potential VMMs
>> >> > - reword commit message
>> >> > ---
>> >> > docs/interop/vhost-user.rst | 90 ++++++++++++++++++++++++++++++++-----
>> >> > hw/virtio/vhost-user.c | 8 ++++
>> >> > 2 files changed, 88 insertions(+), 10 deletions(-)
>> >> >
>> >> > diff --git a/docs/interop/vhost-user.rst b/docs/interop/vhost-user.rst
>> >> > index 5a070adbc1..ba3b5e07b7 100644
>> >> > --- a/docs/interop/vhost-user.rst
>> >> > +++ b/docs/interop/vhost-user.rst
>> >> > @@ -7,6 +7,7 @@ Vhost-user Protocol
>> >> > ..
>> >> > Copyright 2014 Virtual Open Systems Sarl.
>> >> > Copyright 2019 Intel Corporation
>> >> > + Copyright 2023 Linaro Ltd
>> >> > Licence: This work is licensed under the terms of the GNU GPL,
>> >> > version 2 or later. See the COPYING file in the top-level
>> >> > directory.
>> >> > @@ -27,17 +28,31 @@ The protocol defines 2 sides of the communication, *front-end* and
>> >> > *back-end*. The *front-end* is the application that shares its virtqueues, in
>> >> > our case QEMU. The *back-end* is the consumer of the virtqueues.
>> >> >
>> >> > -In the current implementation QEMU is the *front-end*, and the *back-end*
>> >> > -is the external process consuming the virtio queues, for example a
>> >> > -software Ethernet switch running in user space, such as Snabbswitch,
>> >> > -or a block device back-end processing read & write to a virtual
>> >> > -disk. In order to facilitate interoperability between various back-end
>> >> > -implementations, it is recommended to follow the :ref:`Backend program
>> >> > -conventions <backend_conventions>`.
>> >> > +In the current implementation a Virtual Machine Manager (VMM) such as
>> >> > +QEMU is the *front-end*, and the *back-end* is the external process
>> >> > +consuming the virtio queues, for example a software Ethernet switch
>> >> > +running in user space, such as Snabbswitch, or a block device back-end
>> >> > +processing read & write to a virtual disk. In order to facilitate
>> >> > +interoperability between various back-end implementations, it is
>> >> > +recommended to follow the :ref:`Backend program conventions
>> >> > +<backend_conventions>`.
>> >> >
>> >> > The *front-end* and *back-end* can be either a client (i.e. connecting) or
>> >> > server (listening) in the socket communication.
>> >> >
>> >> > +Probing device details
>> >> > +----------------------
>> >> > +
>> >> > +Traditionally the vhost-user daemon *back-end* shares configuration
>> >> > +responsibilities with the VMM *front-end* which needs to know certain
>> >> > +key bits of information about the device. This means the VMM needs to
>> >> > +define at least a minimal stub for each VirtIO device it wants to
>> >> > +support. If the daemon supports the right set of protocol features the
>> >> > +VMM can probe the daemon for the information it needs to setup the
>> >> > +device. See :ref:`Probing features for standalone daemons
>> >> > +<probing_features>` for more details.
>> >> > +
>> >> > +
>> >> > Support for platforms other than Linux
>> >> > --------------------------------------
>> >> >
>> >> > @@ -316,6 +331,7 @@ replies. Here is a list of the ones that do:
>> >> > * ``VHOST_USER_GET_VRING_BASE``
>> >> > * ``VHOST_USER_SET_LOG_BASE`` (if ``VHOST_USER_PROTOCOL_F_LOG_SHMFD``)
>> >> > * ``VHOST_USER_GET_INFLIGHT_FD`` (if ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD``)
>> >> > +* ``VHOST_USER_GET_BACKEND_SPECS`` (if ``VHOST_USER_PROTOCOL_F_STANDALONE``)
>> >> >
>> >> > .. seealso::
>> >> >
>> >> > @@ -396,9 +412,10 @@ must support changing some configuration aspects on the fly.
>> >> > Multiple queue support
>> >> > ----------------------
>> >> >
>> >> > -Many devices have a fixed number of virtqueues. In this case the front-end
>> >> > -already knows the number of available virtqueues without communicating with the
>> >> > -back-end.
>> >> > +Many devices have a fixed number of virtqueues. In this case the
>> >> > +*front-end* usually already knows the number of available virtqueues
>> >> > +without communicating with the back-end. For standalone daemons this
>> >> > +number can be can be probed with the ``VHOST_USER_GET_MIN_VQ`` message.
>> >> >
>> >> > Some devices do not have a fixed number of virtqueues. Instead the maximum
>> >> > number of virtqueues is chosen by the back-end. The number can depend on host
>> >> > @@ -885,6 +902,23 @@ Protocol features
>> >> > #define VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS 15
>> >> > #define VHOST_USER_PROTOCOL_F_STATUS 16
>> >> > #define VHOST_USER_PROTOCOL_F_XEN_MMAP 17
>> >> > + #define VHOST_USER_PROTOCOL_F_PROBE 18
>> >> > +
>> >> > +.. _probing_features:
>> >> > +
>> >> > +Probing features for standalone daemons
>> >> > +---------------------------------------
>> >> > +
>> >> > +The protocol feature ``VHOST_USER_PROTOCOL_F_PROBE`` enables a number
>> >> > +of additional messages which allow the *front-end* to probe details
>> >> > +about the VirtIO device from the *back-end*. However for a *back-end*
>> >> > +to be described as standalone it must also support:
>> >> > +
>> >> > + * ``VHOST_USER_PROTOCOL_F_STATUS``
>> >> > + * ``VHOST_USER_PROTOCOL_F_CONFIG`` (if there is a config space)
>> >> > +
>> >> > +which are required to ensure the *back-end* daemon can operate
>> >> > +without the *front-end* managing some aspects of its configuration.
>> >> >
>> >> > Front-end message types
>> >> > -----------------------
>> >> > @@ -1440,6 +1474,42 @@ Front-end message types
>> >> > query the back-end for its device status as defined in the Virtio
>> >> > specification.
>> >> >
>> >> > +``VHOST_USER_GET_DEVICE_ID``
>> >> > + :id: 41
>> >> > + :request payload: N/A
>> >> > + :reply payload: ``u32``
>> >> > +
>> >> > + When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
>> >> > + successfully negotiated, this message is submitted by the front-end
>> >> > + to query what VirtIO device the back-end support. This is intended
>> >> > + to remove the need for the front-end to know ahead of time what the
>> >> > + VirtIO device the backend emulates is.
>> >> > +
>> >> > +``VHOST_USER_GET_CONFIG_SIZE``
>> >> > + :id: 42
>> >> > + :request payload: N/A
>> >> > + :reply payload: ``u32``
>> >> > +
>> >> > + When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
>> >> > + successfully negotiated, this message is submitted by the front-end
>> >> > + to query the size of the VirtIO device's config space. This is
>> >> > + intended to remove the need for the front-end to know ahead of time
>> >> > + what the size is. Replying with 0 when
>> >> > + ``VHOST_USER_PROTOCOL_F_CONFIG`` has been negotiated would indicate
>> >> > + an bug.
>> >> > +
>> >> > +``VHOST_USER_GET_MIN_VQ``
>> >> > + :id: 43
>> >> > + :request payload: N/A
>> >> > + :reply payload: ``u32``
>> >> > +
>> >> > + When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
>> >> > + successfully negotiated, this message is submitted by the front-end to
>> >> > + query minimum number of VQ's required to support the device. A
>> >> > + device may support more than this number of VQ's if it advertises
>> >> > + the ``VHOST_USER_PROTOCOL_F_MQ`` protocol feature. Reporting a
>> >> > + number greater than the result of ``VHOST_USER_GET_QUEUE_NUM`` would
>> >> > + indicate a bug.
>> >> >
>> >> > Maybe I lack some background, but not sure what min_vq is here?
>> >>
>> >> There will be a minimum number of queues you need to support the device.
>> >> For example the virtio-sound spec specifies you need four queues:
>> >> control, event, tx, rx
>> >
>> > I don't understand why the front-end needs to know that? The backend
>> > already reports the number of queues and not all of them need to be
>> > initialized by the driver.
>>
>> But how many don't need to be initialised? We can't just skip:
>>
>> /* Allocate queues */
>> vub->vqs = g_ptr_array_sized_new(vub->num_vqs);
>> for (int i = 0; i < vub->num_vqs; i++) {
>> g_ptr_array_add(vub->vqs,
>> virtio_add_queue(vdev, vub->vq_size, vub_handle_output));
>> }
>>
>> Or are you saying just require probe-able backends to support
>> VHOST_USER_PROTOCOL_F_MQ and have it always report the minimmum number
>> of queues if it is not a MQ capable device?
>
> The front-end should prepare to allow the maximum number of virtqueues
> returned by VHOST_USER_GET_QUEUE_NUM (VHOST_USER_PROTOCOL_F_MQ).
>
> VIRTIO Transports have a way to query the maximum number of queues but
> not a way to query the minimum number of queues. Why is the minimum
> necessary?
It seems excessive to automatically create the maximum number of VQs. I
guess for backends that don't support the MQ feature (i.e. a variable
number of VQs) we could just say VHOST_USER_GET_QUEUE_NUM == min. But
now we are overloading a different message originally added for
something else.
--
Alex Bennée
Virtualisation Tech Lead @ Linaro
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 27+ messages in thread* Re: [virtio-dev] [RFC PATCH v2] docs/interop: define PROBE feature for vhost-user VirtIO devices
2023-09-08 11:59 ` Alex Bennée
(?)
(?)
@ 2023-09-08 12:38 ` Stefan Hajnoczi
-1 siblings, 0 replies; 27+ messages in thread
From: Stefan Hajnoczi @ 2023-09-08 12:38 UTC (permalink / raw)
To: Alex Bennée
Cc: Stefan Hajnoczi, Albert Esteve, slp, mst, marcandre.lureau,
viresh.kumar, sgarzare, takahiro.akashi, erik.schilling,
manos.pitsidianakis, mathieu.poirier, qemu-devel, virtio-dev,
virtio-comment
A QEMU built-in VIRTIO device will also call virtio_add_queue() for
the maximum number of virtqueues.
I'm not sure what the concern is about adding as few virtqueues as possible?
If the front-end's implementation is inefficient, then it should be
optimized so that untouched virtqueues don't consume resources. I
don't see the need to try to add a special message to vhost-user to
try to reduce the number of virtqueues.
Stefan
On Fri, 8 Sept 2023 at 08:03, Alex Bennée <alex.bennee@linaro.org> wrote:
>
>
> Stefan Hajnoczi <stefanha@gmail.com> writes:
>
> > On Fri, 8 Sept 2023 at 02:43, Alex Bennée <alex.bennee@linaro.org> wrote:
> >>
> >>
> >> Stefan Hajnoczi <stefanha@redhat.com> writes:
> >>
> >> > On Tue, Sep 05, 2023 at 10:34:11AM +0100, Alex Bennée wrote:
> >> >>
> >> >> Albert Esteve <aesteve@redhat.com> writes:
> >> >>
> >> >> > This looks great! Thanks for this proposal.
> >> >> >
> >> >> > On Fri, Sep 1, 2023 at 1:00 PM Alex Bennée <alex.bennee@linaro.org> wrote:
> >> >> >
> >> >> > Currently QEMU has to know some details about the VirtIO device
> >> >> > supported by a vhost-user daemon to be able to setup the guest. This
> >> >> > makes it hard for QEMU to add support for additional vhost-user
> >> >> > daemons without adding specific stubs for each additional VirtIO
> >> >> > device.
> >> >> >
> >> >> > This patch suggests a new feature flag (VHOST_USER_PROTOCOL_F_PROBE)
> >> >> > which the back-end can advertise which allows a probe message to be
> >> >> > sent to get all the details QEMU needs to know in one message.
> >> >> >
> >> >> > Together with the existing features VHOST_USER_PROTOCOL_F_STATUS and
> >> >> > VHOST_USER_PROTOCOL_F_CONFIG we can create "standalone" vhost-user
> >> >> > daemons which are capable of handling all aspects of the VirtIO
> >> >> > transactions with only a generic stub on the QEMU side. These daemons
> >> >> > can also be used without QEMU in situations where there isn't a full
> >> >> > VMM managing their setup.
> >> >> >
> >> >> > Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> >> >> >
> >> >> > ---
> >> >> > v2
> >> >> > - dropped F_STANDALONE in favour of F_PROBE
> >> >> > - split probe details across several messages
> >> >> > - probe messages don't automatically imply a standalone daemon
> >> >> > - add wording where probe details interact (F_MQ/F_CONFIG)
> >> >> > - define VMM and make clear QEMU is only one of many potential VMMs
> >> >> > - reword commit message
> >> >> > ---
> >> >> > docs/interop/vhost-user.rst | 90 ++++++++++++++++++++++++++++++++-----
> >> >> > hw/virtio/vhost-user.c | 8 ++++
> >> >> > 2 files changed, 88 insertions(+), 10 deletions(-)
> >> >> >
> >> >> > diff --git a/docs/interop/vhost-user.rst b/docs/interop/vhost-user.rst
> >> >> > index 5a070adbc1..ba3b5e07b7 100644
> >> >> > --- a/docs/interop/vhost-user.rst
> >> >> > +++ b/docs/interop/vhost-user.rst
> >> >> > @@ -7,6 +7,7 @@ Vhost-user Protocol
> >> >> > ..
> >> >> > Copyright 2014 Virtual Open Systems Sarl.
> >> >> > Copyright 2019 Intel Corporation
> >> >> > + Copyright 2023 Linaro Ltd
> >> >> > Licence: This work is licensed under the terms of the GNU GPL,
> >> >> > version 2 or later. See the COPYING file in the top-level
> >> >> > directory.
> >> >> > @@ -27,17 +28,31 @@ The protocol defines 2 sides of the communication, *front-end* and
> >> >> > *back-end*. The *front-end* is the application that shares its virtqueues, in
> >> >> > our case QEMU. The *back-end* is the consumer of the virtqueues.
> >> >> >
> >> >> > -In the current implementation QEMU is the *front-end*, and the *back-end*
> >> >> > -is the external process consuming the virtio queues, for example a
> >> >> > -software Ethernet switch running in user space, such as Snabbswitch,
> >> >> > -or a block device back-end processing read & write to a virtual
> >> >> > -disk. In order to facilitate interoperability between various back-end
> >> >> > -implementations, it is recommended to follow the :ref:`Backend program
> >> >> > -conventions <backend_conventions>`.
> >> >> > +In the current implementation a Virtual Machine Manager (VMM) such as
> >> >> > +QEMU is the *front-end*, and the *back-end* is the external process
> >> >> > +consuming the virtio queues, for example a software Ethernet switch
> >> >> > +running in user space, such as Snabbswitch, or a block device back-end
> >> >> > +processing read & write to a virtual disk. In order to facilitate
> >> >> > +interoperability between various back-end implementations, it is
> >> >> > +recommended to follow the :ref:`Backend program conventions
> >> >> > +<backend_conventions>`.
> >> >> >
> >> >> > The *front-end* and *back-end* can be either a client (i.e. connecting) or
> >> >> > server (listening) in the socket communication.
> >> >> >
> >> >> > +Probing device details
> >> >> > +----------------------
> >> >> > +
> >> >> > +Traditionally the vhost-user daemon *back-end* shares configuration
> >> >> > +responsibilities with the VMM *front-end* which needs to know certain
> >> >> > +key bits of information about the device. This means the VMM needs to
> >> >> > +define at least a minimal stub for each VirtIO device it wants to
> >> >> > +support. If the daemon supports the right set of protocol features the
> >> >> > +VMM can probe the daemon for the information it needs to setup the
> >> >> > +device. See :ref:`Probing features for standalone daemons
> >> >> > +<probing_features>` for more details.
> >> >> > +
> >> >> > +
> >> >> > Support for platforms other than Linux
> >> >> > --------------------------------------
> >> >> >
> >> >> > @@ -316,6 +331,7 @@ replies. Here is a list of the ones that do:
> >> >> > * ``VHOST_USER_GET_VRING_BASE``
> >> >> > * ``VHOST_USER_SET_LOG_BASE`` (if ``VHOST_USER_PROTOCOL_F_LOG_SHMFD``)
> >> >> > * ``VHOST_USER_GET_INFLIGHT_FD`` (if ``VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD``)
> >> >> > +* ``VHOST_USER_GET_BACKEND_SPECS`` (if ``VHOST_USER_PROTOCOL_F_STANDALONE``)
> >> >> >
> >> >> > .. seealso::
> >> >> >
> >> >> > @@ -396,9 +412,10 @@ must support changing some configuration aspects on the fly.
> >> >> > Multiple queue support
> >> >> > ----------------------
> >> >> >
> >> >> > -Many devices have a fixed number of virtqueues. In this case the front-end
> >> >> > -already knows the number of available virtqueues without communicating with the
> >> >> > -back-end.
> >> >> > +Many devices have a fixed number of virtqueues. In this case the
> >> >> > +*front-end* usually already knows the number of available virtqueues
> >> >> > +without communicating with the back-end. For standalone daemons this
> >> >> > +number can be can be probed with the ``VHOST_USER_GET_MIN_VQ`` message.
> >> >> >
> >> >> > Some devices do not have a fixed number of virtqueues. Instead the maximum
> >> >> > number of virtqueues is chosen by the back-end. The number can depend on host
> >> >> > @@ -885,6 +902,23 @@ Protocol features
> >> >> > #define VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS 15
> >> >> > #define VHOST_USER_PROTOCOL_F_STATUS 16
> >> >> > #define VHOST_USER_PROTOCOL_F_XEN_MMAP 17
> >> >> > + #define VHOST_USER_PROTOCOL_F_PROBE 18
> >> >> > +
> >> >> > +.. _probing_features:
> >> >> > +
> >> >> > +Probing features for standalone daemons
> >> >> > +---------------------------------------
> >> >> > +
> >> >> > +The protocol feature ``VHOST_USER_PROTOCOL_F_PROBE`` enables a number
> >> >> > +of additional messages which allow the *front-end* to probe details
> >> >> > +about the VirtIO device from the *back-end*. However for a *back-end*
> >> >> > +to be described as standalone it must also support:
> >> >> > +
> >> >> > + * ``VHOST_USER_PROTOCOL_F_STATUS``
> >> >> > + * ``VHOST_USER_PROTOCOL_F_CONFIG`` (if there is a config space)
> >> >> > +
> >> >> > +which are required to ensure the *back-end* daemon can operate
> >> >> > +without the *front-end* managing some aspects of its configuration.
> >> >> >
> >> >> > Front-end message types
> >> >> > -----------------------
> >> >> > @@ -1440,6 +1474,42 @@ Front-end message types
> >> >> > query the back-end for its device status as defined in the Virtio
> >> >> > specification.
> >> >> >
> >> >> > +``VHOST_USER_GET_DEVICE_ID``
> >> >> > + :id: 41
> >> >> > + :request payload: N/A
> >> >> > + :reply payload: ``u32``
> >> >> > +
> >> >> > + When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
> >> >> > + successfully negotiated, this message is submitted by the front-end
> >> >> > + to query what VirtIO device the back-end support. This is intended
> >> >> > + to remove the need for the front-end to know ahead of time what the
> >> >> > + VirtIO device the backend emulates is.
> >> >> > +
> >> >> > +``VHOST_USER_GET_CONFIG_SIZE``
> >> >> > + :id: 42
> >> >> > + :request payload: N/A
> >> >> > + :reply payload: ``u32``
> >> >> > +
> >> >> > + When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
> >> >> > + successfully negotiated, this message is submitted by the front-end
> >> >> > + to query the size of the VirtIO device's config space. This is
> >> >> > + intended to remove the need for the front-end to know ahead of time
> >> >> > + what the size is. Replying with 0 when
> >> >> > + ``VHOST_USER_PROTOCOL_F_CONFIG`` has been negotiated would indicate
> >> >> > + an bug.
> >> >> > +
> >> >> > +``VHOST_USER_GET_MIN_VQ``
> >> >> > + :id: 43
> >> >> > + :request payload: N/A
> >> >> > + :reply payload: ``u32``
> >> >> > +
> >> >> > + When the ``VHOST_USER_PROTOCOL_F_PROBE`` protocol feature has been
> >> >> > + successfully negotiated, this message is submitted by the front-end to
> >> >> > + query minimum number of VQ's required to support the device. A
> >> >> > + device may support more than this number of VQ's if it advertises
> >> >> > + the ``VHOST_USER_PROTOCOL_F_MQ`` protocol feature. Reporting a
> >> >> > + number greater than the result of ``VHOST_USER_GET_QUEUE_NUM`` would
> >> >> > + indicate a bug.
> >> >> >
> >> >> > Maybe I lack some background, but not sure what min_vq is here?
> >> >>
> >> >> There will be a minimum number of queues you need to support the device.
> >> >> For example the virtio-sound spec specifies you need four queues:
> >> >> control, event, tx, rx
> >> >
> >> > I don't understand why the front-end needs to know that? The backend
> >> > already reports the number of queues and not all of them need to be
> >> > initialized by the driver.
> >>
> >> But how many don't need to be initialised? We can't just skip:
> >>
> >> /* Allocate queues */
> >> vub->vqs = g_ptr_array_sized_new(vub->num_vqs);
> >> for (int i = 0; i < vub->num_vqs; i++) {
> >> g_ptr_array_add(vub->vqs,
> >> virtio_add_queue(vdev, vub->vq_size, vub_handle_output));
> >> }
> >>
> >> Or are you saying just require probe-able backends to support
> >> VHOST_USER_PROTOCOL_F_MQ and have it always report the minimmum number
> >> of queues if it is not a MQ capable device?
> >
> > The front-end should prepare to allow the maximum number of virtqueues
> > returned by VHOST_USER_GET_QUEUE_NUM (VHOST_USER_PROTOCOL_F_MQ).
> >
> > VIRTIO Transports have a way to query the maximum number of queues but
> > not a way to query the minimum number of queues. Why is the minimum
> > necessary?
>
> It seems excessive to automatically create the maximum number of VQs. I
> guess for backends that don't support the MQ feature (i.e. a variable
> number of VQs) we could just say VHOST_USER_GET_QUEUE_NUM == min. But
> now we are overloading a different message originally added for
> something else.
>
> --
> Alex Bennée
> Virtualisation Tech Lead @ Linaro
^ permalink raw reply [flat|nested] 27+ messages in thread