[virtio-comment] [PATCH V3 RESEND 0/4] Introduce virtio transport virtqueue

[virtio-comment] [PATCH V3 RESEND 0/4] Introduce virtio transport virtqueue

[Zhu Lingshan]

This series introduces virtio transport virtqueue as a new
transport layer for virtio devices.

This series defines the concepts of virtio management device
and virtio managed devices.

This series also defines a commands set over the transport virtqueue,
including:
The command to query available resources of the management device
The commands to create / destroy the managed devices.
The commands to config the managed devices.
The commands to config virtqueues of the managed devices.

Changes from V2:
1)Drop Feature bit VIRTIO_F_TRANSPT_VQ_MDEV (Michael)
2)Extend the feature bits to 128 bits as required in section 2.2 Feature Bits (Michael)
3)Explicit describes the feature bit VIRTIO_F_TRANSPT_VQ should be negotiated with the management device for the commands (Michael)
4)Removed MSI mask / unmask commands (Michael)
5)Add commands to get MSI entries of vqs and config space (Michael)
6)Add a description on the behaviors when disable a MSI (Michael)
7)Removed the section “Management device discovery” (Michael)
8)Remove pre-created managed devices, it is totally depending on the HW implementation (Michael)
9)Detailed descriptions for VIRITO DEVICE ID in the creation command (Michael)
10)Better device requirements for the device config space generation (Michael)
11)Re-use struct transportq_ctrl_packed_vq_state{} for both packed vq state set/get (Michael)

Changes form V1:
1)Add a command to set per-vq asid(Jason)
2)Add a command to reset a virtqueue(Jason)
3)Add support for packed virtqueue state(Jason)
4)Allow nested managed devices(Jason)
5)Provides device feature bits in the provisioning(Jason)
6)Removed most of the error codes, only left OK and ERR(Jason)
7)“virtual devices” is renamed to “managed devices”, to be more general(Jason)

Zhu Lingshan (4):
  Introduce virito transport virtqueue
  Introduce the commands set of the transport vq
  Describe the process to present a managed device
  Add transport vq number for virtio blk and net

 content.tex      | 760 +++++++++++++++++++++++++++++++++++++++++++++++
 introduction.tex |   3 +
 2 files changed, 763 insertions(+)

-- 
2.35.3


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

[virtio-comment] [PATCH V3 RESEND 1/4] Introduce virito transport virtqueue

[Zhu Lingshan]

This commit introduces transport virtqueue as a new transport
layer for virtio devices. And the format of the commands
through the transport virtqueue is defined as well.

We also give examples for the management devices and the
managed devices.

Signed-off-by: Jason Wang <jasowang@redhat.com>
Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com>
---
 content.tex      | 81 ++++++++++++++++++++++++++++++++++++++++++++++++
 introduction.tex |  3 ++
 2 files changed, 84 insertions(+)

diff --git a/content.tex b/content.tex
index e863709..c747d21 100644
--- a/content.tex
+++ b/content.tex
@@ -2895,6 +2895,87 @@ \subsubsection{Resetting Devices}\label{sec:Virtio Transport Options / Virtio ov
 MAY also choose to verify reset completion by reading \field{device status} via
 CCW_CMD_READ_STATUS and checking whether it is 0 afterwards.
 
+\section{Virtio Over Transport Virtqueue}\label{sec:Virtio Transport Options Virtio Over Transport Virtqueue}
+
+In some cases, it is challenging to implement a virtio device in a transport specific method.
+
+One example is that a physical device may try to present multiple managed (maybe virtual) devices
+with limited transport specific resources.
+
+Another example is to implement managed devices which are transport independent.
+
+In those cases, a transport virtqueue could be used as the transport layer to
+implement virtio managed device.
+
+\subsection{Basic Concepts}\label{sec:Virtio Transport Options / Virtio over Transport Virtqueue / Basic Concepts}
+
+Feature bit VIRTIO_F_TRANSPT_VQ indicates that a device offers a transport virtqueue,
+and the managed devices are created, destroyed and configured through the transport virtqueue.
+
+\subsubsection{The Management Device}\label{sec:Virtio Transport Options / Virtio over Transport Virtqueue / Basic Concepts / The Management Device}
+
+A device that offers feature bit VIRTIO_F_TRANSPT_VQ and a transport virtqueue is a management device.
+It processes the commands through the transport virtqueue commands.
+
+For example, a PCIe device with a transport virtqueue is a management device.
+
+\devicenormative{\subsubsection}{The Management Device}{Virtio Transport Options / Virtio Over Transport Virtqueue / Basic Concepts / The management Device}
+
+The management device MUST offer feature bit VIRTIO_F_TRANSPT_VQ and a transport virtqueue as a device specific virtqueue.
+
+\subsubsection{The Managed Device}\label{sec:Virtio Transport Options / Virtio over Transport Virtqueue / Basic Concepts / The Managed Device}
+
+A managed device is a kind of device that is created through a transport virtqueue,
+and utilizes the transport virtqueue as its transport layer.
+
+An example of managed devices is a Scalable I/O Virtualization \hyperref[intro:SIOV]{[SIOV]} device which is created and managed
+through a transport virtqueue of a management device.
+
+\subsubsection{Managed Devices Dscovery}\label{sec:Virtio Transport Options /Virtio Over Transport Virtqueue / Managed Devices Discovery}
+
+Managed devices are created and discovered through a transport virtqueue.
+
+\subsubsection{Managed Devices Interrupts}\label{sec:Virtio Transport Options /Virtio Over Transport Virtqueue / Managed Devices Interrupts}
+
+The managed devices utilize MSI (Message Signaled Interrupts) to send interrupts, there are no legacy interrupts.
+
+A MSI entry is  composed of a 32-bit data payload and a 64-bit address.
+
+\devicenormative{\subsubsection}{The Managed Device}{sec:Virtio Transport Options /Virtio Over Transport Virtqueue / The Managed Device}
+
+The managed device MUST not share its MSI entries with another manged device or with the management device.
+
+\subsection{Format of Commands through Transport Virtqueue}\label{sec:Virtio Transport Options / Virtio over Transport Virtqueue / Format of Commands through Transport Virtqueue}
+
+All transport virtqueue commands are of the following form:
+
+\begin{lstlisting}
+struct virtio_transportq_ctrl {
+        u64 device_id;
+        u16 class;
+        u16 command;
+        u8 command-out-data[];
+        u32 ack;
+        u8 command-in-data[];
+};
+
+/* ack values */
+#define VIRTIO_TRANSPTQ_OK     0
+#define VIRTIO_TRANSPTQ_ERR    255
+\end{lstlisting}
+
+The \field{device_id}, \field{class}, \field{command} and
+\field{command-out-data} are set by the management device driver,
+and the device sets \field{ack} and \field{command-in-data}.
+
+\field{device_id} is a UUID to identify a virtio managed device,
+which defined in \hyperref[intro:rfc4122]{RFC4122}. The management device allocates this UUID,
+it should be unique in the management device context.
+
+The \field{device_id} value 0 is used to identify the management device itself.
+
+\field{class} is an identifier of a set of commands with similar  purposes.
+
 \chapter{Device Types}\label{sec:Device Types}
 
 On top of the queues, config space and feature negotiation facilities
diff --git a/introduction.tex b/introduction.tex
index a9491cf..94a1b51 100644
--- a/introduction.tex
+++ b/introduction.tex
@@ -80,6 +80,9 @@ \section{Normative References}\label{sec:Normative References}
 	\phantomsection\label{intro:SCMI}\textbf{[SCMI]} &
 	Arm System Control and Management Interface, DEN0056,
 	\newline\url{https://developer.arm.com/docs/den0056/c}, version C and any future revisions\\
+	\phantomsection\label{intro:SIOV}\textbf{[SIOV]} &
+	Scalable I/O Virtualization,
+	\newline\url{https://www.opencompute.org/documents/ocp-scalable-io-virtualization-technical-specification-revision-1-v1-2-pdf}\\
 
 \end{longtable}
 
-- 
2.35.3


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] [PATCH V3 RESEND 1/4] Introduce virito transport virtqueue

[Jason Wang]

On Fri, Aug 5, 2022 at 6:02 PM Zhu Lingshan <lingshan.zhu@intel.com> wrote:
>
> This commit introduces transport virtqueue as a new transport
> layer for virtio devices. And the format of the commands
> through the transport virtqueue is defined as well.
>
> We also give examples for the management devices and the
> managed devices.
>
> Signed-off-by: Jason Wang <jasowang@redhat.com>
> Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com>
> ---
>  content.tex      | 81 ++++++++++++++++++++++++++++++++++++++++++++++++
>  introduction.tex |  3 ++
>  2 files changed, 84 insertions(+)
>
> diff --git a/content.tex b/content.tex
> index e863709..c747d21 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -2895,6 +2895,87 @@ \subsubsection{Resetting Devices}\label{sec:Virtio Transport Options / Virtio ov
>  MAY also choose to verify reset completion by reading \field{device status} via
>  CCW_CMD_READ_STATUS and checking whether it is 0 afterwards.
>
> +\section{Virtio Over Transport Virtqueue}\label{sec:Virtio Transport Options Virtio Over Transport Virtqueue}
> +
> +In some cases, it is challenging to implement a virtio device in a transport specific method.
> +
> +One example is that a physical device may try to present multiple managed (maybe virtual) devices
> +with limited transport specific resources.

We haven't introduced the concept of managed devices, so I'd suggest
to simply say

"try to present multiple virtio devices ..."

> +
> +Another example is to implement managed devices which are transport independent.

Ditto, we'd better not talk managed/management before we introduce the
concept of them.

> +
> +In those cases, a transport virtqueue could be used as the transport layer to
> +implement virtio managed device.

I'd suggest introducing the concept of management/managed device first.

> +
> +\subsection{Basic Concepts}\label{sec:Virtio Transport Options / Virtio over Transport Virtqueue / Basic Concepts}
> +
> +Feature bit VIRTIO_F_TRANSPT_VQ indicates that a device offers a transport virtqueue,
> +and the managed devices are created, destroyed and configured through the transport virtqueue.
> +
> +\subsubsection{The Management Device}\label{sec:Virtio Transport Options / Virtio over Transport Virtqueue / Basic Concepts / The Management Device}
> +
> +A device that offers feature bit VIRTIO_F_TRANSPT_VQ and a transport virtqueue is a management device.
> +It processes the commands through the transport virtqueue commands.

I think we need to be verbose here, e.g what did those transport
virtqueue commands do? What's the relationship between management
device and managed device?

> +
> +For example, a PCIe device with a transport virtqueue is a management device.
> +
> +\devicenormative{\subsubsection}{The Management Device}{Virtio Transport Options / Virtio Over Transport Virtqueue / Basic Concepts / The management Device}
> +
> +The management device MUST offer feature bit VIRTIO_F_TRANSPT_VQ and a transport virtqueue as a device specific virtqueue.

I don't get the meaning of this. Each type of virtqueue must be probed
in a device specific way (e.g the virtqueues subsection in the spec)?

> +
> +\subsubsection{The Managed Device}\label{sec:Virtio Transport Options / Virtio over Transport Virtqueue / Basic Concepts / The Managed Device}
> +
> +A managed device is a kind of device that is created through a transport virtqueue,
> +and utilizes the transport virtqueue as its transport layer.

Do we need to say e.g the relationship with the management device?

And we need to define managed devices more accurately, E.g it should
be isolated from other managed devices so it can be assigned to a
guest independently. For isolation, it probably means
interrupt/DMA/P2P (PCI example). Or maybe we can have an example of
managed device that is implemented via PCI?

And spec said:

"Virtio can use various different buses, thus the standard is split
into virtio general and bus-specific sections."

We need to tweak this to say virtio can use transport virtqueue at least?

> +
> +An example of managed devices is a Scalable I/O Virtualization \hyperref[intro:SIOV]{[SIOV]} device which is created and managed

Let's try to use SIOV terminology here, I guess it should be "VDEV"?

> +through a transport virtqueue of a management device.
> +
> +\subsubsection{Managed Devices Dscovery}\label{sec:Virtio Transport Options /Virtio Over Transport Virtqueue / Managed Devices Discovery}
> +
> +Managed devices are created and discovered through a transport virtqueue.
> +
> +\subsubsection{Managed Devices Interrupts}\label{sec:Virtio Transport Options /Virtio Over Transport Virtqueue / Managed Devices Interrupts}
> +
> +The managed devices utilize MSI (Message Signaled Interrupts) to send interrupts,

We can remove "to send interrupts".

> there are no legacy interrupts.
> +
> +A MSI entry is  composed of a 32-bit data payload and a 64-bit address.

We probably need to be more verbose here, what's the use of payload and address?

> +
> +\devicenormative{\subsubsection}{The Managed Device}{sec:Virtio Transport Options /Virtio Over Transport Virtqueue / The Managed Device}
> +
> +The managed device MUST not share its MSI entries with another manged device or with the management device.

Not sure why this is needed since this would be obvious in the
subsection of MSI storage.

> +
> +\subsection{Format of Commands through Transport Virtqueue}\label{sec:Virtio Transport Options / Virtio over Transport Virtqueue / Format of Commands through Transport Virtqueue}
> +
> +All transport virtqueue commands are of the following form:
> +
> +\begin{lstlisting}
> +struct virtio_transportq_ctrl {
> +        u64 device_id;
> +        u16 class;
> +        u16 command;
> +        u8 command-out-data[];
> +        u32 ack;
> +        u8 command-in-data[];
> +};
> +
> +/* ack values */
> +#define VIRTIO_TRANSPTQ_OK     0
> +#define VIRTIO_TRANSPTQ_ERR    255

Any reason starting from 255?

> +\end{lstlisting}
> +
> +The \field{device_id}, \field{class}, \field{command} and
> +\field{command-out-data} are set by the management device driver,
> +and the device sets \field{ack} and \field{command-in-data}.

"management device" actually?

> +
> +\field{device_id} is a UUID to identify a virtio managed device,

It's not a UUID actually, it would be sufficient to be unique among
the managed devices which belong to a single management device.

> +which defined in \hyperref[intro:rfc4122]{RFC4122}. The management device allocates this UUID,
> +it should be unique in the management device context.
> +
> +The \field{device_id} value 0 is used to identify the management device itself.
> +
> +\field{class} is an identifier of a set of commands with similar  purposes.
> +
>  \chapter{Device Types}\label{sec:Device Types}
>
>  On top of the queues, config space and feature negotiation facilities
> diff --git a/introduction.tex b/introduction.tex
> index a9491cf..94a1b51 100644
> --- a/introduction.tex
> +++ b/introduction.tex
> @@ -80,6 +80,9 @@ \section{Normative References}\label{sec:Normative References}
>         \phantomsection\label{intro:SCMI}\textbf{[SCMI]} &
>         Arm System Control and Management Interface, DEN0056,
>         \newline\url{https://developer.arm.com/docs/den0056/c}, version C and any future revisions\\
> +       \phantomsection\label{intro:SIOV}\textbf{[SIOV]} &
> +       Scalable I/O Virtualization,
> +       \newline\url{https://www.opencompute.org/documents/ocp-scalable-io-virtualization-technical-specification-revision-1-v1-2-pdf}\\
>
>  \end{longtable}
>
> --
> 2.35.3
>
>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>

Re: [virtio-comment] [PATCH V3 RESEND 1/4] Introduce virito transport virtqueue

[Zhu, Lingshan]

On 8/8/2022 5:12 PM, Jason Wang wrote:
> On Fri, Aug 5, 2022 at 6:02 PM Zhu Lingshan <lingshan.zhu@intel.com> wrote:
>> This commit introduces transport virtqueue as a new transport
>> layer for virtio devices. And the format of the commands
>> through the transport virtqueue is defined as well.
>>
>> We also give examples for the management devices and the
>> managed devices.
>>
>> Signed-off-by: Jason Wang <jasowang@redhat.com>
>> Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com>
>> ---
>>   content.tex      | 81 ++++++++++++++++++++++++++++++++++++++++++++++++
>>   introduction.tex |  3 ++
>>   2 files changed, 84 insertions(+)
>>
>> diff --git a/content.tex b/content.tex
>> index e863709..c747d21 100644
>> --- a/content.tex
>> +++ b/content.tex
>> @@ -2895,6 +2895,87 @@ \subsubsection{Resetting Devices}\label{sec:Virtio Transport Options / Virtio ov
>>   MAY also choose to verify reset completion by reading \field{device status} via
>>   CCW_CMD_READ_STATUS and checking whether it is 0 afterwards.
>>
>> +\section{Virtio Over Transport Virtqueue}\label{sec:Virtio Transport Options Virtio Over Transport Virtqueue}
>> +
>> +In some cases, it is challenging to implement a virtio device in a transport specific method.
>> +
>> +One example is that a physical device may try to present multiple managed (maybe virtual) devices
>> +with limited transport specific resources.
> We haven't introduced the concept of managed devices, so I'd suggest
> to simply say
>
> "try to present multiple virtio devices ..."
will fix it
>
>> +
>> +Another example is to implement managed devices which are transport independent.
> Ditto, we'd better not talk managed/management before we introduce the
> concept of them.
OK
>
>> +
>> +In those cases, a transport virtqueue could be used as the transport layer to
>> +implement virtio managed device.
> I'd suggest introducing the concept of management/managed device first.
this is the first paragraph, maybe too early to introduce management / 
managed devices before the feature bit,
I will avoid talking about management/managed devices before define 
them(right below this paragraph).
>
>> +
>> +\subsection{Basic Concepts}\label{sec:Virtio Transport Options / Virtio over Transport Virtqueue / Basic Concepts}
>> +
>> +Feature bit VIRTIO_F_TRANSPT_VQ indicates that a device offers a transport virtqueue,
>> +and the managed devices are created, destroyed and configured through the transport virtqueue.
>> +
>> +\subsubsection{The Management Device}\label{sec:Virtio Transport Options / Virtio over Transport Virtqueue / Basic Concepts / The Management Device}
>> +
>> +A device that offers feature bit VIRTIO_F_TRANSPT_VQ and a transport virtqueue is a management device.
>> +It processes the commands through the transport virtqueue commands.
> I think we need to be verbose here, e.g what did those transport
> virtqueue commands do? What's the relationship between management
> device and managed device?
I will add an overview of the transport virtqueue commands here, and a 
description of the relationship in the "managed device" section.
>
>> +
>> +For example, a PCIe device with a transport virtqueue is a management device.
>> +
>> +\devicenormative{\subsubsection}{The Management Device}{Virtio Transport Options / Virtio Over Transport Virtqueue / Basic Concepts / The management Device}
>> +
>> +The management device MUST offer feature bit VIRTIO_F_TRANSPT_VQ and a transport virtqueue as a device specific virtqueue.
> I don't get the meaning of this. Each type of virtqueue must be probed
> in a device specific way (e.g the virtqueues subsection in the spec)?
I will remove "as a device specific virtqueue."
>
>> +
>> +\subsubsection{The Managed Device}\label{sec:Virtio Transport Options / Virtio over Transport Virtqueue / Basic Concepts / The Managed Device}
>> +
>> +A managed device is a kind of device that is created through a transport virtqueue,
>> +and utilizes the transport virtqueue as its transport layer.
> Do we need to say e.g the relationship with the management device?
I will emphasize the relationship with the management device here, says
the managed device is created/destroyed/ and configured by the management
device through the transport vq
>
> And we need to define managed devices more accurately, E.g it should
> be isolated from other managed devices so it can be assigned to a
> guest independently. For isolation, it probably means
> interrupt/DMA/P2P (PCI example). Or maybe we can have an example of
> managed device that is implemented via PCI?
OK, we can add these in the device requirements of managed devices
>
> And spec said:
>
> "Virtio can use various different buses, thus the standard is split
> into virtio general and bus-specific sections."
>
> We need to tweak this to say virtio can use transport virtqueue at least?
I will add: virtio uses transport virtqueue to implement managed devices.
>
>> +
>> +An example of managed devices is a Scalable I/O Virtualization \hyperref[intro:SIOV]{[SIOV]} device which is created and managed
> Let's try to use SIOV terminology here, I guess it should be "VDEV"?
Yes, it is VDEV
>
>> +through a transport virtqueue of a management device.
>> +
>> +\subsubsection{Managed Devices Dscovery}\label{sec:Virtio Transport Options /Virtio Over Transport Virtqueue / Managed Devices Discovery}
>> +
>> +Managed devices are created and discovered through a transport virtqueue.
>> +
>> +\subsubsection{Managed Devices Interrupts}\label{sec:Virtio Transport Options /Virtio Over Transport Virtqueue / Managed Devices Interrupts}
>> +
>> +The managed devices utilize MSI (Message Signaled Interrupts) to send interrupts,
> We can remove "to send interrupts".
I think at least we need "for interrupts", or this is not a completed 
sentence.
>
>> there are no legacy interrupts.
>> +
>> +A MSI entry is  composed of a 32-bit data payload and a 64-bit address.
> We probably need to be more verbose here, what's the use of payload and address?
I think this is the MSI concept, do we need to explain it here? The 
concepts, e.g., payload initialization
can be architecture specific
>
>> +
>> +\devicenormative{\subsubsection}{The Managed Device}{sec:Virtio Transport Options /Virtio Over Transport Virtqueue / The Managed Device}
>> +
>> +The managed device MUST not share its MSI entries with another manged device or with the management device.
> Not sure why this is needed since this would be obvious in the
> subsection of MSI storage.
we did not describe this in the MSI commands section yet, we need this 
because the interrupt
source should be isolated, and SIOV spec says this.
>
>> +
>> +\subsection{Format of Commands through Transport Virtqueue}\label{sec:Virtio Transport Options / Virtio over Transport Virtqueue / Format of Commands through Transport Virtqueue}
>> +
>> +All transport virtqueue commands are of the following form:
>> +
>> +\begin{lstlisting}
>> +struct virtio_transportq_ctrl {
>> +        u64 device_id;
>> +        u16 class;
>> +        u16 command;
>> +        u8 command-out-data[];
>> +        u32 ack;
>> +        u8 command-in-data[];
>> +};
>> +
>> +/* ack values */
>> +#define VIRTIO_TRANSPTQ_OK     0
>> +#define VIRTIO_TRANSPTQ_ERR    255
> Any reason starting from 255?
actually it is ending with 255, remember we have other error codes like 
EINVAL and EAGAIN
between 0 and 255. I think we can define ERR as 1 since we have removed 
other codes.
>
>> +\end{lstlisting}
>> +
>> +The \field{device_id}, \field{class}, \field{command} and
>> +\field{command-out-data} are set by the management device driver,
>> +and the device sets \field{ack} and \field{command-in-data}.
> "management device" actually?
Not sure, it can be the managed device, so I only use "the device".
How about "the management device or the managed device"?
And replace "the management device driver" with "the management device 
driver or the managed device driver"
>
>> +
>> +\field{device_id} is a UUID to identify a virtio managed device,
> It's not a UUID actually, it would be sufficient to be unique among
> the managed devices which belong to a single management device.
Yes is should be a unique device ID, I think UUID is perfectly meet this
requirement so that we don't need to define the rules and concepts of 
the ID
>
>> +which defined in \hyperref[intro:rfc4122]{RFC4122}. The management device allocates this UUID,
>> +it should be unique in the management device context.
>> +
>> +The \field{device_id} value 0 is used to identify the management device itself.
>> +
>> +\field{class} is an identifier of a set of commands with similar  purposes.
>> +
>>   \chapter{Device Types}\label{sec:Device Types}
>>
>>   On top of the queues, config space and feature negotiation facilities
>> diff --git a/introduction.tex b/introduction.tex
>> index a9491cf..94a1b51 100644
>> --- a/introduction.tex
>> +++ b/introduction.tex
>> @@ -80,6 +80,9 @@ \section{Normative References}\label{sec:Normative References}
>>          \phantomsection\label{intro:SCMI}\textbf{[SCMI]} &
>>          Arm System Control and Management Interface, DEN0056,
>>          \newline\url{https://developer.arm.com/docs/den0056/c}, version C and any future revisions\\
>> +       \phantomsection\label{intro:SIOV}\textbf{[SIOV]} &
>> +       Scalable I/O Virtualization,
>> +       \newline\url{https://www.opencompute.org/documents/ocp-scalable-io-virtualization-technical-specification-revision-1-v1-2-pdf}\\
>>
>>   \end{longtable}
>>
>> --
>> 2.35.3
>>
>>
>> This publicly archived list offers a means to provide input to the
>> OASIS Virtual I/O Device (VIRTIO) TC.
>>
>> In order to verify user consent to the Feedback License terms and
>> to minimize spam in the list archive, subscription is required
>> before posting.
>>
>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>> List help: virtio-comment-help@lists.oasis-open.org
>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>> Committee: https://www.oasis-open.org/committees/virtio/
>> Join OASIS: https://www.oasis-open.org/join/
>>

Re: [virtio-comment] [PATCH V3 RESEND 1/4] Introduce virito transport virtqueue

[Michael S. Tsirkin]

On Tue, Aug 09, 2022 at 04:36:43PM +0800, Zhu, Lingshan wrote:
> > > +
> > > +A device that offers feature bit VIRTIO_F_TRANSPT_VQ and a transport virtqueue is a management device.
> > > +It processes the commands through the transport virtqueue commands.
> > I think we need to be verbose here, e.g what did those transport
> > virtqueue commands do? What's the relationship between management
> > device and managed device?
> I will add an overview of the transport virtqueue commands here, and a
> description of the relationship in the "managed device" section.

Transport is fine but management of devices is clearly something
nvidia's patches do. So I think it's best to reuse the concept of device
groups for this, from Max's patchset. Let's not replicate that work at
least. I promised Max to help a bit with wording so I'll soon send a
revision of his patchset, the generic part about device group
from which you should be able to reuse.

-- 
MST

Re: [virtio-comment] [PATCH V3 RESEND 1/4] Introduce virito transport virtqueue

[Zhu, Lingshan]

On 8/9/2022 5:12 PM, Michael S. Tsirkin wrote:
> On Tue, Aug 09, 2022 at 04:36:43PM +0800, Zhu, Lingshan wrote:
>>>> +
>>>> +A device that offers feature bit VIRTIO_F_TRANSPT_VQ and a transport virtqueue is a management device.
>>>> +It processes the commands through the transport virtqueue commands.
>>> I think we need to be verbose here, e.g what did those transport
>>> virtqueue commands do? What's the relationship between management
>>> device and managed device?
>> I will add an overview of the transport virtqueue commands here, and a
>> description of the relationship in the "managed device" section.
> Transport is fine but management of devices is clearly something
> nvidia's patches do. So I think it's best to reuse the concept of device
> groups for this, from Max's patchset. Let's not replicate that work at
> least. I promised Max to help a bit with wording so I'll soon send a
> revision of his patchset, the generic part about device group
> from which you should be able to reuse.
Of course, it is possible to add the device groups in this series for sure.

What I don't understand is: what kind of commands against a device 
group? Destroy all?
And normally a parent device only support one kind of devices, like a 
SRIOV capable
virtio-net PF only supports virtio-net VFs on it, and maybe there will 
not be
a device support both SIOV and SRIOV, it is complex in the HW 
implementation.

Thanks,
Zhu Lingshan

Re: [virtio-comment] [PATCH V3 RESEND 1/4] Introduce virito transport virtqueue

[Jason Wang]

On Tue, Aug 9, 2022 at 5:19 PM Zhu, Lingshan <lingshan.zhu@intel.com> wrote:
>
>
>
> On 8/9/2022 5:12 PM, Michael S. Tsirkin wrote:
> > On Tue, Aug 09, 2022 at 04:36:43PM +0800, Zhu, Lingshan wrote:
> >>>> +
> >>>> +A device that offers feature bit VIRTIO_F_TRANSPT_VQ and a transport virtqueue is a management device.
> >>>> +It processes the commands through the transport virtqueue commands.
> >>> I think we need to be verbose here, e.g what did those transport
> >>> virtqueue commands do? What's the relationship between management
> >>> device and managed device?
> >> I will add an overview of the transport virtqueue commands here, and a
> >> description of the relationship in the "managed device" section.
> > Transport is fine but management of devices is clearly something
> > nvidia's patches do. So I think it's best to reuse the concept of device
> > groups for this, from Max's patchset. Let's not replicate that work at
> > least. I promised Max to help a bit with wording so I'll soon send a
> > revision of his patchset, the generic part about device group
> > from which you should be able to reuse.
> Of course, it is possible to add the device groups in this series for sure.
>
> What I don't understand is: what kind of commands against a device
> group? Destroy all?
> And normally a parent device only support one kind of devices, like a
> SRIOV capable
> virtio-net PF only supports virtio-net VFs on it, and maybe there will
> not be
> a device support both SIOV and SRIOV, it is complex in the HW
> implementation.

For having backward compatibility? (E.g for platforms that don't support SIOV)

Thanks

>
> Thanks,
> Zhu Lingshan
>

Re: [virtio-comment] [PATCH V3 RESEND 1/4] Introduce virito transport virtqueue

[Zhu, Lingshan]

On 8/9/2022 5:21 PM, Jason Wang wrote:
> On Tue, Aug 9, 2022 at 5:19 PM Zhu, Lingshan <lingshan.zhu@intel.com> wrote:
>>
>>
>> On 8/9/2022 5:12 PM, Michael S. Tsirkin wrote:
>>> On Tue, Aug 09, 2022 at 04:36:43PM +0800, Zhu, Lingshan wrote:
>>>>>> +
>>>>>> +A device that offers feature bit VIRTIO_F_TRANSPT_VQ and a transport virtqueue is a management device.
>>>>>> +It processes the commands through the transport virtqueue commands.
>>>>> I think we need to be verbose here, e.g what did those transport
>>>>> virtqueue commands do? What's the relationship between management
>>>>> device and managed device?
>>>> I will add an overview of the transport virtqueue commands here, and a
>>>> description of the relationship in the "managed device" section.
>>> Transport is fine but management of devices is clearly something
>>> nvidia's patches do. So I think it's best to reuse the concept of device
>>> groups for this, from Max's patchset. Let's not replicate that work at
>>> least. I promised Max to help a bit with wording so I'll soon send a
>>> revision of his patchset, the generic part about device group
>>> from which you should be able to reuse.
>> Of course, it is possible to add the device groups in this series for sure.
>>
>> What I don't understand is: what kind of commands against a device
>> group? Destroy all?
>> And normally a parent device only support one kind of devices, like a
>> SRIOV capable
>> virtio-net PF only supports virtio-net VFs on it, and maybe there will
>> not be
>> a device support both SIOV and SRIOV, it is complex in the HW
>> implementation.
> For having backward compatibility? (E.g for platforms that don't support SIOV)
>
> Thanks
If not a device model like SIOV which does not have a dedicated physical 
transport layer,
the question would be: Why do we need a side channel for a VF? This 
looks conflicts with
VF provisioning and virtio-spec definitions. E.g, why allow changing MSI 
in the flight
outside the guest control?

Thanks,
Zhu Lingshan
>
>> Thanks,
>> Zhu Lingshan
>>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] [PATCH V3 RESEND 1/4] Introduce virito transport virtqueue

[Jason Wang]

On Tue, Aug 9, 2022 at 5:28 PM Zhu, Lingshan <lingshan.zhu@intel.com> wrote:
>
>
>
> On 8/9/2022 5:21 PM, Jason Wang wrote:
> > On Tue, Aug 9, 2022 at 5:19 PM Zhu, Lingshan <lingshan.zhu@intel.com> wrote:
> >>
> >>
> >> On 8/9/2022 5:12 PM, Michael S. Tsirkin wrote:
> >>> On Tue, Aug 09, 2022 at 04:36:43PM +0800, Zhu, Lingshan wrote:
> >>>>>> +
> >>>>>> +A device that offers feature bit VIRTIO_F_TRANSPT_VQ and a transport virtqueue is a management device.
> >>>>>> +It processes the commands through the transport virtqueue commands.
> >>>>> I think we need to be verbose here, e.g what did those transport
> >>>>> virtqueue commands do? What's the relationship between management
> >>>>> device and managed device?
> >>>> I will add an overview of the transport virtqueue commands here, and a
> >>>> description of the relationship in the "managed device" section.
> >>> Transport is fine but management of devices is clearly something
> >>> nvidia's patches do. So I think it's best to reuse the concept of device
> >>> groups for this, from Max's patchset. Let's not replicate that work at
> >>> least. I promised Max to help a bit with wording so I'll soon send a
> >>> revision of his patchset, the generic part about device group
> >>> from which you should be able to reuse.
> >> Of course, it is possible to add the device groups in this series for sure.
> >>
> >> What I don't understand is: what kind of commands against a device
> >> group? Destroy all?
> >> And normally a parent device only support one kind of devices, like a
> >> SRIOV capable
> >> virtio-net PF only supports virtio-net VFs on it, and maybe there will
> >> not be
> >> a device support both SIOV and SRIOV, it is complex in the HW
> >> implementation.
> > For having backward compatibility? (E.g for platforms that don't support SIOV)
> >
> > Thanks
> If not a device model like SIOV which does not have a dedicated physical
> transport layer,
> the question would be: Why do we need a side channel for a VF? This
> looks conflicts with
> VF provisioning and virtio-spec definitions. E.g, why allow changing MSI
> in the flight
> outside the guest control?

Just to clarify my points. I meant that vendors may choose to
implement both SRIOV and SIOV in the hardware. And I believe this is
what most vendor will do.

But it doesn't mean the control path needs to be shared.

Thanks

>
> Thanks,
> Zhu Lingshan
> >
> >> Thanks,
> >> Zhu Lingshan
> >>
>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] [PATCH V3 RESEND 1/4] Introduce virito transport virtqueue

[Michael S. Tsirkin]

On Tue, Aug 09, 2022 at 05:31:32PM +0800, Jason Wang wrote:
> On Tue, Aug 9, 2022 at 5:28 PM Zhu, Lingshan <lingshan.zhu@intel.com> wrote:
> >
> >
> >
> > On 8/9/2022 5:21 PM, Jason Wang wrote:
> > > On Tue, Aug 9, 2022 at 5:19 PM Zhu, Lingshan <lingshan.zhu@intel.com> wrote:
> > >>
> > >>
> > >> On 8/9/2022 5:12 PM, Michael S. Tsirkin wrote:
> > >>> On Tue, Aug 09, 2022 at 04:36:43PM +0800, Zhu, Lingshan wrote:
> > >>>>>> +
> > >>>>>> +A device that offers feature bit VIRTIO_F_TRANSPT_VQ and a transport virtqueue is a management device.
> > >>>>>> +It processes the commands through the transport virtqueue commands.
> > >>>>> I think we need to be verbose here, e.g what did those transport
> > >>>>> virtqueue commands do? What's the relationship between management
> > >>>>> device and managed device?
> > >>>> I will add an overview of the transport virtqueue commands here, and a
> > >>>> description of the relationship in the "managed device" section.
> > >>> Transport is fine but management of devices is clearly something
> > >>> nvidia's patches do. So I think it's best to reuse the concept of device
> > >>> groups for this, from Max's patchset. Let's not replicate that work at
> > >>> least. I promised Max to help a bit with wording so I'll soon send a
> > >>> revision of his patchset, the generic part about device group
> > >>> from which you should be able to reuse.
> > >> Of course, it is possible to add the device groups in this series for sure.
> > >>
> > >> What I don't understand is: what kind of commands against a device
> > >> group? Destroy all?
> > >> And normally a parent device only support one kind of devices, like a
> > >> SRIOV capable
> > >> virtio-net PF only supports virtio-net VFs on it, and maybe there will
> > >> not be
> > >> a device support both SIOV and SRIOV, it is complex in the HW
> > >> implementation.
> > > For having backward compatibility? (E.g for platforms that don't support SIOV)
> > >
> > > Thanks
> > If not a device model like SIOV which does not have a dedicated physical
> > transport layer,
> > the question would be: Why do we need a side channel for a VF? This
> > looks conflicts with
> > VF provisioning and virtio-spec definitions. E.g, why allow changing MSI
> > in the flight
> > outside the guest control?
> 
> Just to clarify my points. I meant that vendors may choose to
> implement both SRIOV and SIOV in the hardware. And I believe this is
> what most vendor will do.
> 
> But it doesn't mean the control path needs to be shared.
> 
> Thanks

Okay. But I feel some general structure and terminology should be shared, given
the similarity.


> >
> > Thanks,
> > Zhu Lingshan
> > >
> > >> Thanks,
> > >> Zhu Lingshan
> > >>
> >

Re: [virtio-comment] [PATCH V3 RESEND 1/4] Introduce virito transport virtqueue

[Jason Wang]

On Tue, Aug 9, 2022 at 5:35 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Tue, Aug 09, 2022 at 05:31:32PM +0800, Jason Wang wrote:
> > On Tue, Aug 9, 2022 at 5:28 PM Zhu, Lingshan <lingshan.zhu@intel.com> wrote:
> > >
> > >
> > >
> > > On 8/9/2022 5:21 PM, Jason Wang wrote:
> > > > On Tue, Aug 9, 2022 at 5:19 PM Zhu, Lingshan <lingshan.zhu@intel.com> wrote:
> > > >>
> > > >>
> > > >> On 8/9/2022 5:12 PM, Michael S. Tsirkin wrote:
> > > >>> On Tue, Aug 09, 2022 at 04:36:43PM +0800, Zhu, Lingshan wrote:
> > > >>>>>> +
> > > >>>>>> +A device that offers feature bit VIRTIO_F_TRANSPT_VQ and a transport virtqueue is a management device.
> > > >>>>>> +It processes the commands through the transport virtqueue commands.
> > > >>>>> I think we need to be verbose here, e.g what did those transport
> > > >>>>> virtqueue commands do? What's the relationship between management
> > > >>>>> device and managed device?
> > > >>>> I will add an overview of the transport virtqueue commands here, and a
> > > >>>> description of the relationship in the "managed device" section.
> > > >>> Transport is fine but management of devices is clearly something
> > > >>> nvidia's patches do. So I think it's best to reuse the concept of device
> > > >>> groups for this, from Max's patchset. Let's not replicate that work at
> > > >>> least. I promised Max to help a bit with wording so I'll soon send a
> > > >>> revision of his patchset, the generic part about device group
> > > >>> from which you should be able to reuse.
> > > >> Of course, it is possible to add the device groups in this series for sure.
> > > >>
> > > >> What I don't understand is: what kind of commands against a device
> > > >> group? Destroy all?
> > > >> And normally a parent device only support one kind of devices, like a
> > > >> SRIOV capable
> > > >> virtio-net PF only supports virtio-net VFs on it, and maybe there will
> > > >> not be
> > > >> a device support both SIOV and SRIOV, it is complex in the HW
> > > >> implementation.
> > > > For having backward compatibility? (E.g for platforms that don't support SIOV)
> > > >
> > > > Thanks
> > > If not a device model like SIOV which does not have a dedicated physical
> > > transport layer,
> > > the question would be: Why do we need a side channel for a VF? This
> > > looks conflicts with
> > > VF provisioning and virtio-spec definitions. E.g, why allow changing MSI
> > > in the flight
> > > outside the guest control?
> >
> > Just to clarify my points. I meant that vendors may choose to
> > implement both SRIOV and SIOV in the hardware. And I believe this is
> > what most vendor will do.
> >
> > But it doesn't mean the control path needs to be shared.
> >
> > Thanks
>
> Okay. But I feel some general structure and terminology should be shared, given
> the similarity.

Yes, I agree.

Thanks

>
>
> > >
> > > Thanks,
> > > Zhu Lingshan
> > > >
> > > >> Thanks,
> > > >> Zhu Lingshan
> > > >>
> > >
>

Re: [virtio-comment] [PATCH V3 RESEND 1/4] Introduce virito transport virtqueue

[Zhu, Lingshan]

On 8/9/2022 5:37 PM, Jason Wang wrote:
> On Tue, Aug 9, 2022 at 5:35 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>> On Tue, Aug 09, 2022 at 05:31:32PM +0800, Jason Wang wrote:
>>> On Tue, Aug 9, 2022 at 5:28 PM Zhu, Lingshan <lingshan.zhu@intel.com> wrote:
>>>>
>>>>
>>>> On 8/9/2022 5:21 PM, Jason Wang wrote:
>>>>> On Tue, Aug 9, 2022 at 5:19 PM Zhu, Lingshan <lingshan.zhu@intel.com> wrote:
>>>>>>
>>>>>> On 8/9/2022 5:12 PM, Michael S. Tsirkin wrote:
>>>>>>> On Tue, Aug 09, 2022 at 04:36:43PM +0800, Zhu, Lingshan wrote:
>>>>>>>>>> +
>>>>>>>>>> +A device that offers feature bit VIRTIO_F_TRANSPT_VQ and a transport virtqueue is a management device.
>>>>>>>>>> +It processes the commands through the transport virtqueue commands.
>>>>>>>>> I think we need to be verbose here, e.g what did those transport
>>>>>>>>> virtqueue commands do? What's the relationship between management
>>>>>>>>> device and managed device?
>>>>>>>> I will add an overview of the transport virtqueue commands here, and a
>>>>>>>> description of the relationship in the "managed device" section.
>>>>>>> Transport is fine but management of devices is clearly something
>>>>>>> nvidia's patches do. So I think it's best to reuse the concept of device
>>>>>>> groups for this, from Max's patchset. Let's not replicate that work at
>>>>>>> least. I promised Max to help a bit with wording so I'll soon send a
>>>>>>> revision of his patchset, the generic part about device group
>>>>>>> from which you should be able to reuse.
>>>>>> Of course, it is possible to add the device groups in this series for sure.
>>>>>>
>>>>>> What I don't understand is: what kind of commands against a device
>>>>>> group? Destroy all?
>>>>>> And normally a parent device only support one kind of devices, like a
>>>>>> SRIOV capable
>>>>>> virtio-net PF only supports virtio-net VFs on it, and maybe there will
>>>>>> not be
>>>>>> a device support both SIOV and SRIOV, it is complex in the HW
>>>>>> implementation.
>>>>> For having backward compatibility? (E.g for platforms that don't support SIOV)
>>>>>
>>>>> Thanks
>>>> If not a device model like SIOV which does not have a dedicated physical
>>>> transport layer,
>>>> the question would be: Why do we need a side channel for a VF? This
>>>> looks conflicts with
>>>> VF provisioning and virtio-spec definitions. E.g, why allow changing MSI
>>>> in the flight
>>>> outside the guest control?
>>> Just to clarify my points. I meant that vendors may choose to
>>> implement both SRIOV and SIOV in the hardware. And I believe this is
>>> what most vendor will do.
>>>
>>> But it doesn't mean the control path needs to be shared.
>>>
>>> Thanks
>> Okay. But I feel some general structure and terminology should be shared, given
>> the similarity.
> Yes, I agree.
Yes, it looks we all want to control sub-devices from the parent device

Thanks,
Zhu Lingshan
>
> Thanks
>
>>
>>>> Thanks,
>>>> Zhu Lingshan
>>>>>> Thanks,
>>>>>> Zhu Lingshan
>>>>>>
>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] [PATCH V3 RESEND 1/4] Introduce virito transport virtqueue

[Michael S. Tsirkin]

On Tue, Aug 09, 2022 at 05:28:26PM +0800, Zhu, Lingshan wrote:
> 
> 
> On 8/9/2022 5:21 PM, Jason Wang wrote:
> > On Tue, Aug 9, 2022 at 5:19 PM Zhu, Lingshan <lingshan.zhu@intel.com> wrote:
> > > 
> > > 
> > > On 8/9/2022 5:12 PM, Michael S. Tsirkin wrote:
> > > > On Tue, Aug 09, 2022 at 04:36:43PM +0800, Zhu, Lingshan wrote:
> > > > > > > +
> > > > > > > +A device that offers feature bit VIRTIO_F_TRANSPT_VQ and a transport virtqueue is a management device.
> > > > > > > +It processes the commands through the transport virtqueue commands.
> > > > > > I think we need to be verbose here, e.g what did those transport
> > > > > > virtqueue commands do? What's the relationship between management
> > > > > > device and managed device?
> > > > > I will add an overview of the transport virtqueue commands here, and a
> > > > > description of the relationship in the "managed device" section.
> > > > Transport is fine but management of devices is clearly something
> > > > nvidia's patches do. So I think it's best to reuse the concept of device
> > > > groups for this, from Max's patchset. Let's not replicate that work at
> > > > least. I promised Max to help a bit with wording so I'll soon send a
> > > > revision of his patchset, the generic part about device group
> > > > from which you should be able to reuse.
> > > Of course, it is possible to add the device groups in this series for sure.
> > > 
> > > What I don't understand is: what kind of commands against a device
> > > group? Destroy all?
> > > And normally a parent device only support one kind of devices, like a
> > > SRIOV capable
> > > virtio-net PF only supports virtio-net VFs on it, and maybe there will
> > > not be
> > > a device support both SIOV and SRIOV, it is complex in the HW
> > > implementation.
> > For having backward compatibility? (E.g for platforms that don't support SIOV)
> > 
> > Thanks
> If not a device model like SIOV which does not have a dedicated physical
> transport layer,
> the question would be: Why do we need a side channel for a VF? This looks
> conflicts with
> VF provisioning and virtio-spec definitions. E.g, why allow changing MSI in
> the flight
> outside the guest control?
> 
> Thanks,
> Zhu Lingshan

MSI is a weird one, seeing that it's an nvidia specific extension to the pci spec.
A better example in the latest version is controlling device features
or config space for example.
That part is somewhat similar to what you guys have with SIOV.

-- 
MST

Re: [virtio-comment] [PATCH V3 RESEND 1/4] Introduce virito transport virtqueue

[Michael S. Tsirkin]

On Tue, Aug 09, 2022 at 05:19:39PM +0800, Zhu, Lingshan wrote:
> 
> 
> On 8/9/2022 5:12 PM, Michael S. Tsirkin wrote:
> > On Tue, Aug 09, 2022 at 04:36:43PM +0800, Zhu, Lingshan wrote:
> > > > > +
> > > > > +A device that offers feature bit VIRTIO_F_TRANSPT_VQ and a transport virtqueue is a management device.
> > > > > +It processes the commands through the transport virtqueue commands.
> > > > I think we need to be verbose here, e.g what did those transport
> > > > virtqueue commands do? What's the relationship between management
> > > > device and managed device?
> > > I will add an overview of the transport virtqueue commands here, and a
> > > description of the relationship in the "managed device" section.
> > Transport is fine but management of devices is clearly something
> > nvidia's patches do. So I think it's best to reuse the concept of device
> > groups for this, from Max's patchset. Let's not replicate that work at
> > least. I promised Max to help a bit with wording so I'll soon send a
> > revision of his patchset, the generic part about device group
> > from which you should be able to reuse.
> Of course, it is possible to add the device groups in this series for sure.
> 
> What I don't understand is: what kind of commands against a device group?
> Destroy all?
> And normally a parent device only support one kind of devices, like a SRIOV
> capable
> virtio-net PF only supports virtio-net VFs on it, and maybe there will not
> be
> a device support both SIOV and SRIOV, it is complex in the HW
> implementation.
> 
> Thanks,
> Zhu Lingshan

Yes, I understand that. I think at this point it's okay to not
require reuse of same commands for SIOV and SR-IOV. There clearly
is little interest from the two groups in sharing the VQ
for two purposes.

However, I think the descriptive section defining the concept of device
group, group member, member id and whatever we come up with for the
device that controls the group are all common ground and we should not
use distict terminology.

This has no hardware impact at all but helps users get oriented
more easily.

And I think a generic command structure with group type, member id
etc can also be shared. Your formats are close enough for that
to be possible.

-- 
MST

Re: [virtio-comment] [PATCH V3 RESEND 1/4] Introduce virito transport virtqueue

[Zhu, Lingshan]

On 8/9/2022 5:29 PM, Michael S. Tsirkin wrote:
> On Tue, Aug 09, 2022 at 05:19:39PM +0800, Zhu, Lingshan wrote:
>>
>> On 8/9/2022 5:12 PM, Michael S. Tsirkin wrote:
>>> On Tue, Aug 09, 2022 at 04:36:43PM +0800, Zhu, Lingshan wrote:
>>>>>> +
>>>>>> +A device that offers feature bit VIRTIO_F_TRANSPT_VQ and a transport virtqueue is a management device.
>>>>>> +It processes the commands through the transport virtqueue commands.
>>>>> I think we need to be verbose here, e.g what did those transport
>>>>> virtqueue commands do? What's the relationship between management
>>>>> device and managed device?
>>>> I will add an overview of the transport virtqueue commands here, and a
>>>> description of the relationship in the "managed device" section.
>>> Transport is fine but management of devices is clearly something
>>> nvidia's patches do. So I think it's best to reuse the concept of device
>>> groups for this, from Max's patchset. Let's not replicate that work at
>>> least. I promised Max to help a bit with wording so I'll soon send a
>>> revision of his patchset, the generic part about device group
>>> from which you should be able to reuse.
>> Of course, it is possible to add the device groups in this series for sure.
>>
>> What I don't understand is: what kind of commands against a device group?
>> Destroy all?
>> And normally a parent device only support one kind of devices, like a SRIOV
>> capable
>> virtio-net PF only supports virtio-net VFs on it, and maybe there will not
>> be
>> a device support both SIOV and SRIOV, it is complex in the HW
>> implementation.
>>
>> Thanks,
>> Zhu Lingshan
> Yes, I understand that. I think at this point it's okay to not
> require reuse of same commands for SIOV and SR-IOV. There clearly
> is little interest from the two groups in sharing the VQ
> for two purposes.
>
> However, I think the descriptive section defining the concept of device
> group, group member, member id and whatever we come up with for the
> device that controls the group are all common ground and we should not
> use distict terminology.
>
> This has no hardware impact at all but helps users get oriented
> more easily.
>
> And I think a generic command structure with group type, member id
> etc can also be shared. Your formats are close enough for that
> to be possible.
I still don't understand what problems the device group can address,
can you help name some? And if we have different types of devices
over the transport vq, why we need to group them if we don't have
any commands against the group?

Like UUID == 0x000001 for a SIOV device, UUID == 0x000002 for a SRIOV 
device.

Thanks,
Zhu Lingshan
>

Re: [virtio-comment] [PATCH V3 RESEND 1/4] Introduce virito transport virtqueue

[Michael S. Tsirkin]

On Tue, Aug 09, 2022 at 05:43:02PM +0800, Zhu, Lingshan wrote:
> 
> 
> On 8/9/2022 5:29 PM, Michael S. Tsirkin wrote:
> > On Tue, Aug 09, 2022 at 05:19:39PM +0800, Zhu, Lingshan wrote:
> > > 
> > > On 8/9/2022 5:12 PM, Michael S. Tsirkin wrote:
> > > > On Tue, Aug 09, 2022 at 04:36:43PM +0800, Zhu, Lingshan wrote:
> > > > > > > +
> > > > > > > +A device that offers feature bit VIRTIO_F_TRANSPT_VQ and a transport virtqueue is a management device.
> > > > > > > +It processes the commands through the transport virtqueue commands.
> > > > > > I think we need to be verbose here, e.g what did those transport
> > > > > > virtqueue commands do? What's the relationship between management
> > > > > > device and managed device?
> > > > > I will add an overview of the transport virtqueue commands here, and a
> > > > > description of the relationship in the "managed device" section.
> > > > Transport is fine but management of devices is clearly something
> > > > nvidia's patches do. So I think it's best to reuse the concept of device
> > > > groups for this, from Max's patchset. Let's not replicate that work at
> > > > least. I promised Max to help a bit with wording so I'll soon send a
> > > > revision of his patchset, the generic part about device group
> > > > from which you should be able to reuse.
> > > Of course, it is possible to add the device groups in this series for sure.
> > > 
> > > What I don't understand is: what kind of commands against a device group?
> > > Destroy all?
> > > And normally a parent device only support one kind of devices, like a SRIOV
> > > capable
> > > virtio-net PF only supports virtio-net VFs on it, and maybe there will not
> > > be
> > > a device support both SIOV and SRIOV, it is complex in the HW
> > > implementation.
> > > 
> > > Thanks,
> > > Zhu Lingshan
> > Yes, I understand that. I think at this point it's okay to not
> > require reuse of same commands for SIOV and SR-IOV. There clearly
> > is little interest from the two groups in sharing the VQ
> > for two purposes.
> > 
> > However, I think the descriptive section defining the concept of device
> > group, group member, member id and whatever we come up with for the
> > device that controls the group are all common ground and we should not
> > use distict terminology.
> > 
> > This has no hardware impact at all but helps users get oriented
> > more easily.
> > 
> > And I think a generic command structure with group type, member id
> > etc can also be shared. Your formats are close enough for that
> > to be possible.
> I still don't understand what problems the device group can address,
> can you help name some?

Just to use common terminology, so we don't need to come
up with all these words like managed device management device
and to make users learn it.

> And if we have different types of devices
> over the transport vq, why we need to group them if we don't have
> any commands against the group?

You have a group of devices de facto, whatever transport vq manages.


> Like UUID == 0x000001 for a SIOV device, UUID == 0x000002 for a SRIOV
> device.

I'm not sure what you mean here, and 0x000001 is clearly not a UUID.

> 
> Thanks,
> Zhu Lingshan

Re: [virtio-comment] [PATCH V3 RESEND 1/4] Introduce virito transport virtqueue

[Jason Wang]

On Tue, Aug 9, 2022 at 5:12 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Tue, Aug 09, 2022 at 04:36:43PM +0800, Zhu, Lingshan wrote:
> > > > +
> > > > +A device that offers feature bit VIRTIO_F_TRANSPT_VQ and a transport virtqueue is a management device.
> > > > +It processes the commands through the transport virtqueue commands.
> > > I think we need to be verbose here, e.g what did those transport
> > > virtqueue commands do? What's the relationship between management
> > > device and managed device?
> > I will add an overview of the transport virtqueue commands here, and a
> > description of the relationship in the "managed device" section.
>
> Transport is fine but management of devices is clearly something
> nvidia's patches do. So I think it's best to reuse the concept of device
> groups for this, from Max's patchset. Let's not replicate that work at
> least. I promised Max to help a bit with wording so I'll soon send a
> revision of his patchset, the generic part about device group
> from which you should be able to reuse.

I wonder if "parent/child" makes any more sense than "management/managed" here?

Thanks

>
> --
> MST
>

Re: [virtio-comment] [PATCH V3 RESEND 1/4] Introduce virito transport virtqueue

[Michael S. Tsirkin]

On Tue, Aug 09, 2022 at 05:28:33PM +0800, Jason Wang wrote:
> On Tue, Aug 9, 2022 at 5:12 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Tue, Aug 09, 2022 at 04:36:43PM +0800, Zhu, Lingshan wrote:
> > > > > +
> > > > > +A device that offers feature bit VIRTIO_F_TRANSPT_VQ and a transport virtqueue is a management device.
> > > > > +It processes the commands through the transport virtqueue commands.
> > > > I think we need to be verbose here, e.g what did those transport
> > > > virtqueue commands do? What's the relationship between management
> > > > device and managed device?
> > > I will add an overview of the transport virtqueue commands here, and a
> > > description of the relationship in the "managed device" section.
> >
> > Transport is fine but management of devices is clearly something
> > nvidia's patches do. So I think it's best to reuse the concept of device
> > groups for this, from Max's patchset. Let's not replicate that work at
> > least. I promised Max to help a bit with wording so I'll soon send a
> > revision of his patchset, the generic part about device group
> > from which you should be able to reuse.
> 
> I wonder if "parent/child" makes any more sense than "management/managed" here?
> 
> Thanks

I suggest reusing Max's proposal.
There we have a group, a group type, we have a group member.
All that is left is a name for a device that owns the vq
through which the commands are sent (PF for the SRIOV group).
Group parent might be ok, sure. Group owner as another option?


> >
> > --
> > MST
> >
> 
> 
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
> 
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
> 
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] [PATCH V3 RESEND 1/4] Introduce virito transport virtqueue

[Jason Wang]

On Wed, Aug 10, 2022 at 5:04 AM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Tue, Aug 09, 2022 at 05:28:33PM +0800, Jason Wang wrote:
> > On Tue, Aug 9, 2022 at 5:12 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > >
> > > On Tue, Aug 09, 2022 at 04:36:43PM +0800, Zhu, Lingshan wrote:
> > > > > > +
> > > > > > +A device that offers feature bit VIRTIO_F_TRANSPT_VQ and a transport virtqueue is a management device.
> > > > > > +It processes the commands through the transport virtqueue commands.
> > > > > I think we need to be verbose here, e.g what did those transport
> > > > > virtqueue commands do? What's the relationship between management
> > > > > device and managed device?
> > > > I will add an overview of the transport virtqueue commands here, and a
> > > > description of the relationship in the "managed device" section.
> > >
> > > Transport is fine but management of devices is clearly something
> > > nvidia's patches do. So I think it's best to reuse the concept of device
> > > groups for this, from Max's patchset. Let's not replicate that work at
> > > least. I promised Max to help a bit with wording so I'll soon send a
> > > revision of his patchset, the generic part about device group
> > > from which you should be able to reuse.
> >
> > I wonder if "parent/child" makes any more sense than "management/managed" here?
> >
> > Thanks
>
> I suggest reusing Max's proposal.
> There we have a group, a group type, we have a group member.

For group type, I will have the question if it can be implied in the
virtqueue type.

> All that is left is a name for a device that owns the vq
> through which the commands are sent (PF for the SRIOV group).
> Group parent might be ok, sure. Group owner as another option?

Either should be fine.

Thanks

>
>
> > >
> > > --
> > > MST
> > >
> >
> >
> > This publicly archived list offers a means to provide input to the
> > OASIS Virtual I/O Device (VIRTIO) TC.
> >
> > In order to verify user consent to the Feedback License terms and
> > to minimize spam in the list archive, subscription is required
> > before posting.
> >
> > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > List help: virtio-comment-help@lists.oasis-open.org
> > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > Committee: https://www.oasis-open.org/committees/virtio/
> > Join OASIS: https://www.oasis-open.org/join/
>

Re: [virtio-comment] [PATCH V3 RESEND 1/4] Introduce virito transport virtqueue

[Michael S. Tsirkin]

On Wed, Aug 10, 2022 at 03:41:48PM +0800, Jason Wang wrote:
> On Wed, Aug 10, 2022 at 5:04 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Tue, Aug 09, 2022 at 05:28:33PM +0800, Jason Wang wrote:
> > > On Tue, Aug 9, 2022 at 5:12 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > >
> > > > On Tue, Aug 09, 2022 at 04:36:43PM +0800, Zhu, Lingshan wrote:
> > > > > > > +
> > > > > > > +A device that offers feature bit VIRTIO_F_TRANSPT_VQ and a transport virtqueue is a management device.
> > > > > > > +It processes the commands through the transport virtqueue commands.
> > > > > > I think we need to be verbose here, e.g what did those transport
> > > > > > virtqueue commands do? What's the relationship between management
> > > > > > device and managed device?
> > > > > I will add an overview of the transport virtqueue commands here, and a
> > > > > description of the relationship in the "managed device" section.
> > > >
> > > > Transport is fine but management of devices is clearly something
> > > > nvidia's patches do. So I think it's best to reuse the concept of device
> > > > groups for this, from Max's patchset. Let's not replicate that work at
> > > > least. I promised Max to help a bit with wording so I'll soon send a
> > > > revision of his patchset, the generic part about device group
> > > > from which you should be able to reuse.
> > >
> > > I wonder if "parent/child" makes any more sense than "management/managed" here?
> > >
> > > Thanks
> >
> > I suggest reusing Max's proposal.
> > There we have a group, a group type, we have a group member.
> 
> For group type, I will have the question if it can be implied in the
> virtqueue type.

Not really. For example we will want LM to work for both VFs and SFs.
Migration commands do not belong on the transport VQ since migration is
not part of transport.

> > All that is left is a name for a device that owns the vq
> > through which the commands are sent (PF for the SRIOV group).
> > Group parent might be ok, sure. Group owner as another option?
> 
> Either should be fine.
> 
> Thanks
> 
> >
> >
> > > >
> > > > --
> > > > MST
> > > >
> > >
> > >
> > > This publicly archived list offers a means to provide input to the
> > > OASIS Virtual I/O Device (VIRTIO) TC.
> > >
> > > In order to verify user consent to the Feedback License terms and
> > > to minimize spam in the list archive, subscription is required
> > > before posting.
> > >
> > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > List help: virtio-comment-help@lists.oasis-open.org
> > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > Committee: https://www.oasis-open.org/committees/virtio/
> > > Join OASIS: https://www.oasis-open.org/join/
> >

[virtio-comment] [PATCH V3 RESEND 2/4] Introduce the commands set of the transport vq

[Zhu Lingshan]

This commit introduces the commands set of the
transport virtqueue, including:

The command to query available resources of the management device
The commands to create / destroy the managed devices.
The commands to config the managed devices.
The commands to config virtqueues of the managed devices.

Signed-off-by: Jason Wang <jasowang@redhat.com>
Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com>
---
 content.tex | 636 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 636 insertions(+)

diff --git a/content.tex b/content.tex
index c747d21..190f3a0 100644
--- a/content.tex
+++ b/content.tex
@@ -2975,6 +2975,642 @@ \subsection{Format of Commands through Transport Virtqueue}\label{sec:Virtio Tra
 The \field{device_id} value 0 is used to identify the management device itself.
 
 \field{class} is an identifier of a set of commands with similar  purposes.
+ 
+\subsection{Available Resource of the Management Device}\label{sec:Virtio Transport Options / Virtio Over Transport Virtqueue / Available Resource of the Management Device}
+
+When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
+statistical information of available resources on the management device could be fetched
+by the following command:
+
+\begin{lstlisting}
+#define VIRTIO_TRANSPTQ_CTRL_AVAIL_RES    1 (class)
+ #define VIRTIO_TRANSPTQ_CTRL_AVAIL_RES_GET     0 (command)
+\end{lstlisting}
+
+VIRTIO_TRANSPTQ_CTRL_AVAIL_RES_GET command is used to get the statistical information of the available
+resource of a management device. There is no command-out-data for VIRTIO_TRANSPTQ_CTRL_AVAIL_RES_GET
+command. The management device fills command-in-data in the following format:
+
+\begin{lstlisting}
+struct virtio_mgmt_dev_avail_res {
+        /* The number of management device total remaining available free
+         * virtqueues for the managed devices
+         */
+        u16 num_avail_vqs;
+        /* The minimal number of virtqueues for a managed device */
+        u16 min_dev_vqs;
+        /* The maximum number of virtqueues for a managed device */
+        u16 max_dev_vqs;
+        /* The number of managed devices that can be created with min_vqs virtqueues */
+        u16 num_avail_dev;
+};
+\end{lstlisting}
+
+\field{max_avail_dev} MAY not equal to \field{num_avail_vqs} divided by \field{min_vqs}
+due to the limitations of of other resources.
+
+\devicenormative{\subsubsection}{Available Resource of the Management Device}{Virtio Transport Options /
+Virtio Over Transport Virtqueue / Available Resource of the Management Device}
+
+The management device MUST fail VIRTIO_TRANSPTQ_CTRL_AVAIL_RES_GET command if \field{device_id} is not 0
+
+\drivernormative{\subsubsection}{Available Resource of the Management Device}{Virtio Transport Options /
+Virtio Over Transport Virtqueue / Available Resource of the Management Device}
+
+The management device driver MUST use 0 as \field{device_id} for
+VIRTIO_TRANSPTQ_CTRL_AVAIL_RES_GET command.
+
+\subsection{Create and Destroy Managed Devices}\label{sec:Virtio Transport Options / Virtio Over Transport Virtqueue / Create and Destroy Managed Devices}
+
+When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
+managed devices must be created and destroyed through the transport virtqueue.
+
+\begin{lstlisting}
+#define VIRTIO_TRANSPORTQ_CTRL_DEV    2
+ #define VIRTIO_TRANSPORTQ_CTRL_DEV_CREATE        0
+ #define VIRTIO_TRANSPORTQ_CTRL_DEV_DESTROY       1
+
+struct virtio_transportq_ctrl_dev_attribute {
+       u64 device_features[2];
+       u32 virtio_device_id;
+       u8 dev_config[];
+};
+\end{lstlisting}
+
+The VIRTIO_TRANSPORTQ_CTRL_DEV_CREATE command is used to create a managed device.
+
+As described in struct virtio_transportq_ctrl_dev_attribute, the command-out-data for VIRTIO_TRANSPTQ_CTRL_DEV_CREATE includes:
+\begin{itemize*}
+\item \field{device_features}: the 128-bit device feature bits (defined in section \ref{sec:Basic Facilities of a Virtio Device / Feature Bits})
+of the managed device.
+\item \field{virtio_device_id}: the virtio device id defined in Chapter \ref{sec:Device Types}.
+\item \field{dev_config}: the device type specific configurations. E.g., for a virtio-net device,
+it is struct virtio_net_config in section \ref{sec:Device Types / Network Device / Device configuration layout};
+for a virtio-block device, it is struct virtio_blk_config in section \ref{sec:Device Types / Block Device / Device configuration layout}
+\end{itemize*}
+
+When succeed, the device returns a 64 bits UUID as an unique identifier of the created managed device in command-in-data.
+
+The VIRTIO_TRANSPORTQ_CTRL_DEV_DESTROY command is used to destroy a
+managed device which is identified by its 64 bits UUID
+\field{device_id}. There's no command-in-data nor command-out-data for
+VIRTIO_TRANSPTQ_CTRL_DEV_DESTROY command.
+
+\devicenormative{\subsubsection}{Create and Destroy Managed Devices}{Virtio Transport Options / Virtio Over Transport Virtqueue / Create and Destroy Managed Devices}
+
+The management device MUST fail VIRTIO_TRANSPORTQ_CTRL_DEV_CREATE command
+if \field{device_id} is not 0.
+
+The management device MUST fail VIRTIO_TRANSPORTQ_CTRL_DEV CREATE command
+if \field{device_features} exceeds the features that can be provided from the management device.
+
+The management device MUST fail VIRTO_TRANSPORTQ_CTRL_DEV_DESTROY command
+if \field{device_id} is 0.
+
+\drivernormative{\subsubsection}{Create and Destroy Managed Devices}{Virtio Transport Options / Virtio Over Transport Virtqueue / Create and Destroy Managed Devices}
+
+The management device driver MUST use 0 as \field{device_id} for
+TRANSPORTQ_CTRL_DEV_CREATE command.
+
+\subsection{Features Negotiation}\label{sec:Virtio Transport Options / Virtio
+Over Transport Virtqueue / Features Negotiation}
+
+When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
+the features negotiation of managed devices is done by the
+following commands:
+
+\begin{lstlisting}
+#define VIRTIO_TRANSPTQ_CTRL_FEAT   3
+ #define VIRTIO_TRANSPTQ_CTRL_FEAT_DEVICE_GET        0
+ #define VIRTIO_TRANSPTQ_CTRL_FEAT_DRIVER_SET        1
+ #define VIRTIO_TRANSPTQ_CTRL_FEAT_DRIVER_GET        2
+\end{lstlisting}
+
+The VIRTIO_TRANSPTQ_CTRL_FEAT_DEVICE_GET command is used to get the features offered
+by a managed device. The command-in-data is the 128-bit feature bits
+(defined in section \ref{sec:Basic Facilities of a Virtio Device / Feature Bits}).
+There is no command-out-data. 
+
+The VIRTIO_TRANSPTQ_CTRL_FEAT_DRIVER_SET command is for the driver to accept feature
+bits offered by the managed device. The command-out-data is the 128-bit feature bits
+passed to the managed device. There is no command-in-data.
+
+The VIRTIO_TRANSPTQ_CTRL_FEAT_DRIVER_GET command is used to get the features accepted
+by both the managed device driver and the managed device. The command-in-data is the
+128-bit feature bits. There is no command-out-data.
+
+\devicenormative{\subsubsection}{Features Negotiation}{Virtio Transport Options /
+Virtio Over Transport Virtqueue / Features Negotiation}
+
+The management device MUST fail VIRTIO_TRANSPTQ_F_CTRL_FEAT class commands if
+\field{device_id} is 0.
+
+\subsection{Device Status}\label{sec:Virtio Transport Options / Virtio Over
+Transport Virtqueue / Device Status}
+
+When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
+the device status of a managed device can be accessed by the following
+commands:
+
+\begin{lstlisting}
+#define VIRTIO_TRANSPTQ_CTRL_STATUS    4
+ #define VIRTIO_TRANSPTQ_CTRL_STATUS_GET        0
+ #define VIRTIO_TRANSPTQ_CTRL_STATUS_SET        1
+\end{lstlisting}
+
+The VIRTIO_TRANSPTQ_CTRL_STATUS_GET command is used to get the device status of
+a managed device. The command-in-data is the 8-bit status
+returned from the device. There's no command-out-data for this
+command.
+
+The VIRTIO_TRANSPTQ_CTRL_STATUS_SET command is used to set the device status of
+a managed device. The command-out-data is the 8-bit status
+to set to the device. There's no command-in-data for this command.
+
+\devicenormative{\subsubsection}{Device Status}{Virtio Transport Options / Virtio
+Over Transport Virtqueue / Device Status}
+
+The management device MUST reset the managed device when 0
+is set via VIRTIO_TRANSPTQ_CTRL_STATUS_SET, the success of this
+command demonstrates the success of the reset.
+
+The management device MUST present 0 through
+VIRTIO_TRANSPTQ_CTRL_STATUS_GET once the reset is successfully done.
+
+The management device MUST fail the device status access if
+\field{device_id} is 0.
+
+\drivernormative{\subsubsection}{Device Status}{Virtio Transport Options / Virtio
+Over Transport Virtqueue / Device Status}
+
+After writing 0 via VIRTIO_TRANSPTQ_CTRL_STATUS_SET, the driver MUST wait
+for the success of the command before re-initializing the device.
+
+\subsection{Device Generation}\label{sec:Virtio Transport Options / Virtio Over Transport Virtqueue / Device Generation}
+
+When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
+the device generation count can be read from the following commands:
+
+\begin{lstlisting}
+#define VIRTIO_TRANSPTQ_CTRL_GENERATION    5
+ #define VIRTIO_TRANSPTQ_CTRL_GENERATION_GET        0
+\end{lstlisting}
+
+The VIRTIO_TRANSPTQ_CTRL_GENERATION_GET command is used to get the device configuration generation count
+of the managed device. The command-in-data is the one byte device
+generation returned from the device. There's no command-out-data for
+this command.
+
+\devicenormative{\subsubsection}{Device Generation}{Virtio Transport Options / Virtio Over Transport Virtqueue / Device Generation}
+
+The managed device MUST present a changed generation count after the driver
+has read any device-specific configuration values if the values have been changed
+during the last read.
+
+The management device MUST fail the device generation access if \field{device_id} is 0.
+
+\subsection{Device Specific Configuration}\label{sec:Virtio Transport Options /
+Virtio Over Transport Virtqueue / Device Specific Configuration}
+
+When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
+the device config space contents of a managed device can be accessed through
+the following commands:
+
+\begin{lstlisting}
+#define VIRTIO_TRANSPTQ_CTRL_CONFIG    6
+  #define VIRTIO_TRANSPTQ_CTRL_CONFIG_GET    0
+  #define VIRTIO_TRANSPTQ_CTRL_CONFIG_SET    1
+
+struct virtio_transportq_ctrl_dev_config_get {
+       u32 offset;
+       u32 size;
+};
+
+struct virtio_transportq_ctrl_dev_config_set {
+       u32 offset;
+       u32 size;
+       u8  data[];
+};
+\end{lstlisting}
+
+The VIRTIO_TRANSPTQ_CTRL_CONFIG_GET command is used to read data from the
+device configuration space.  The command-out-data is the \field{offset}
+from the start of the config space and the \field{size}
+of the data (as described in struct virtio_transportq_ctrl_dev_config_get).
+The command-in-data is an array of the data that read from the config space.
+
+The VIRTIO_TRANSPTQ_CTRL_CONFIG_SET command is used to write data to the device
+configuration space. The command-out-data contains the
+\field{offset} from the start of the config space, the \field{size} of the data and
+the \field{data} to be written (as described in struct virtio_transportq_ctrl_dev_config_set).
+There's no command-in-data for this command.
+
+\devicenormative{\subsubsection}{Device Specific Configuration}{Virtio Transport
+Options / Virtio Over Transport Virtqueue / Device Specific Configuration}
+
+The management device MUST fail the device configuration space access
+if the driver access the range which is outside the config space.
+
+The management device MUST fail the device configuration space access
+if \field{device_id} is 0.
+
+\subsection{MSI Configuration}\label{sec:Virtio Transport Options / Virtio Over
+Transport Virtqueue / MSI Configuration}
+
+When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
+the MSI entries of a managed device can be accessed  through the following command:
+
+\begin{lstlisting}
+#define VIRTIO_TRANSPTQ_CTRL_MSI    7
+ #define VIRTIO_TRANSPTQ_CTRL_MSI_VQ_GET        0
+ #define VIRTIO_TRANSPTQ_CTRL_MSI_VQ_SET        1
+ #define VIRTIO_TRANSPTQ_CTRL_MSI_VQ_ENABLE     2
+ #define VIRTIO_TRANSPTQ_CTRL_MSI_CONFIG_GET    3
+ #define VIRTIO_TRANSPTQ_CTRL_MSI_CONFIG_WET    4
+ #define VIRTIO_TRANSPTQ_CTRL_MSI_CONFIG_ENABLE 5
+
+struct virtio_transportq_ctrl_msi_vq_config {
+       u16 queue_index;
+       u64 address;
+       u32 data;
+       u8 padding[2];
+};
+
+struct virtio_transportq_ctrl_msi_vq_enable {
+       u16 queue_index;
+       u8 enable;
+       u8 padding[5];
+};
+
+struct virtio_transportq_ctrl_msi_config {
+       u64 address;
+       u32 data;
+       u8 padding[4];
+};
+\end{lstlisting}
+
+The VIRTIO_TRANSPTQ_CTRL_MSI_VQ_GET command is used to read the MSI entry of a
+specific virtqueue. The command-out-data is the queue index. The command-in-data contains
+\field{queue_index}, the \field{address} and \field{data}
+(as described in struct virtio_transportq_ctrl_msi_vq_config).
+
+
+The VIRTIO_TRANSPTQ_CTRL_MSI_VQ_SET command is used to set the MSI entry for a
+specific virtqueue. The command-out-data is the \field{queue_index},
+\field{address} and \field{data} (as described in struct virtio_transportq_ctrl_msi_vq_config).
+There is no command-in-data.
+
+The VIRTIO_TRANSPTQ_CTRL_MSI_VQ_ENABLE command is used to enable or disable
+the MSI interrupt for a specific virtqueue. The command-out-data is the
+\field{queue_index} and a byte \field{enable} which representing ENABLE or DISABLE the MSI.
+(as described in struct virtio_transportq_ctrl_msi_vq_enable).
+
+\begin{lstlisting}
+#define VIRTIO_TRANSPTQ_CTRL_MSI_VQ_ENABLE     1
+#define VIRTIO_TRANSPTQ_CTRL_MSI_VQ_DISABLE    0
+\end{lstlisting}
+
+The VIRTIO_TRANSPTQ_CTRL_MSI_CONFIG_GET command is used to read the MSI entry
+of the config interrupt. The command-in-data contains the \field{address} and
+\field{data} (as described in struct virtio_transportq_ctrl_msi_config).
+There is no command-out-data.
+
+The VIRTIO_TRANSPTQ_CTRL_MSI_CONFIG_SET command is used to set the MSI entry
+for the config interrupt. The command-out-data is the \field{address} and
+\field{data} (as described in struct virtio_transportq_ctrl_msi_config).
+There is no command-in-data.
+
+The VIRTIO_TRANSPTQ_CTRL_MSI_CONFIG_ENABLE command is used to enable or disable
+the MSI interrupt for config space. The command-out-data is a byte which representing
+ENABLE or DISABLE the config MSI.
+There is no command-in-data
+
+\begin{lstlisting}
+#define VIRTIO_TRANSPTQ_CTRL_MSI_CFG_ENABLE     1
+#define VIRTIO_TRANSPTQ_CTRL_MSI_CFG_DISABLE    0
+\end{lstlisting}
+
+\devicenormative{\subsubsection}{MSI Configuration}{Virtio Transport Options / Virtio
+ver Transport Virtqueue / MSI Configuration}
+
+The managed device MUST disable the MSI interrupts for both virtqueues and config space
+upon a reset.
+
+Once a MSI entry is disabled, the managed device MUST not send any interrupts
+by this MSI entry.
+
+\drivernormative{\subsubsection}{MSI Configuration}{Virtio Transport Options / Virtio
+Over Transport Virtqueue / MSI Configuration}
+
+The driver MUST allocate transport or platform specific MSI entries
+for both virtqueues and config space if it wants to use interrupts.
+
+The driver MAY choose to disable the MSI if polling mode is used.
+
+\subsection{Virtqueue Address}\label{sec:Virtio Transport Options / Virtio Over
+Transport Virtqueue / Virtqueue Address}
+
+When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
+the addresses of a specific virtqueue are accessed through the following command:
+
+\begin{lstlisting}
+#define VIRTIO_TRANSPTQ_CTRL_VQ_ADDR    8
+ #define VIRTIO_TRANSPTQ_CTRL_VQ_ADDR_SET       1
+
+struct virtio_transportq_ctrl_vq_addr {
+       u16 queue_index;
+       u64 descriptor_area;
+       u64 device_area;
+       u64 driver_area;
+       u8 padding[6];
+};
+\end{lstlisting}
+
+The VIRTIO_TRANSPTQ_CTRL_VQ_ADDR_SET command is used to set the addresses of a specified
+virtqueue. The command-out-data contains the \field{queue_index}, the addresses of \field{device_area},
+\field{descriptor_area} and \field{driver_area} (as described in struct
+virtio_transportq_ctrl_vq_addr). There's no command-in-data.
+
+\devicenormative{\subsubsection}{Virtqueue Address}{Virtio Transport Options / Virtio
+Over Transport Virtqueue / Virtqueeu Address}
+
+The management device MUST fail the commands of class
+VIRTIO_TRANSPTQ_CTRL_VQ_ADDR if \field{device_id} is 0.
+
+The management device MUST fail the commands of class
+VIRTIO_TRANSPTQ_CTRL_VQ_ADDR if \field{queue_index} is out-of-range invalid.
+
+\subsection{Virtqueue Status}\label{sec:Virtio Transport Options / Virtio Over
+Transport Virtqueue / Virtqueue Status}
+
+When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
+virtqueue status is accessed through the following command:
+
+\begin{lstlisting}
+#define VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE    9
+ #define VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_GET       0
+ #define VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_SET       1
+
+struct virtio_transportq_ctrl_vq_status_set {
+       u16 queue_index;
+       u8 status;
+       u8 padding[5];
+
+#define VIRTIO_TRANSPTQ_VQ_ENABLE     1
+#define VIRTIO_TRANSPTQ_VQ_DISABLE    0
+
+};
+
+\end{lstlisting}
+
+The VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_GET is used to get the status of a
+specific virtqueue. The command-out-data is the queue
+index. The command-in-data is the virtqueue status.
+
+The VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_SET command is used to set the status of a
+specific virtqueue. The command-out-data is the \field{queue_index} and the
+\field{status} representing ENABLE or DISABLE that is set to the virtqueue
+(as described in struct virtio_transportq_ctrl_vq_status_set).
+There's no command-in-data.
+
+\devicenormative{\subsubsection}{Virtqueue Status}{Virtio Transport Options / Virtio
+Over Transport Virtqueue / Virtqueue Status}
+
+When disabled, the managed device MUST stop processing requests from
+this virtqueue.
+
+The management device MUST present a 0 via
+VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_GET upon a reset of the managed device.
+
+The management device MUST fail the virtqueue status access if
+\field{device_id} is 0.
+
+The management device MUST fail the virtqueue status access if
+the queue_index is out-of-range invalid.
+
+
+\drivernormative{\subsubsection}{Virtqueue Status}{Virtio Transport Options / Virtio
+Over Transport Virtqueue / Virtqueue Status}
+
+The driver MUST configure other virtqueue fields before enabling
+the virtqueue with VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_SET.
+
+\subsection{Virtqueue Size}\label{sec:Virtio Transport Options / Virtio Over
+Transport Virtqueue / Virtqueue Size}
+
+When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
+virtqueue size is accessed through the following command:
+
+\begin{lstlisting}
+#define VIRTIO_TRANSPTQ_CTRL_VQ_SIZE    10
+ #define VIRTIO_TRANSPTQ_CTRL_VQ_SIZE_GET       0
+ #define VIRTIO_TRANSPTQ_CTRL_VQ_SIZE_SET       1
+
+struct virtio_transportq_ctrl_vq_size_set {
+       u16 queue_index;
+       u16 size;
+       u8 padding[4];
+};
+\end{lstlisting}
+
+The VIRTIO_TRANSPTQ_CTRL_VQ_SIZE_GET command is used to get the virtqueue
+size. On reset, the maximum queue size supported by the device is
+returned. The command-out-data is the queue index. The
+command-in-data is an 16-bit queue size.
+
+The VIRTIO_TRANSPTQ_CTRL_VQ_SIZE_SET command is used to set the virtqueue
+size. The command-out-data is the \field{queue_index} and the \field{size}
+of the virtqueue (as described in struct virtio_transportq_ctrl_vq_size_set).
+There's no command-in-data.
+
+\devicenormative{\subsubsection}{Virtqueue Status}{Virtio Transport Options / Virtio
+Over Transport Virtqueue / Virtqueue Size}
+
+The management device MUST fail the virtqueue size access if
+\field{device_id} is 0.
+
+The management device MUST fail the virtqueue size access if
+the queue index is out-of-range invalid.
+
+\subsection{Virtqueue Notification}\label{sec:Virtio Transport Options / Virtio
+Over Transport Virtqueue / Virtqueue Notification}
+
+When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
+the virtqueue notification area information can be get through the following commands:
+
+\begin{lstlisting}
+#define VIRTIO_TRANSPTQ_CTRL_VQ_NOTIFY    11
+ #define VIRTIO_TRANSPTQ_CTRL_VQ_NOTIFY_GET          0
+
+struct virtio_transportq_ctrl_vq_notification_area {
+       u64 address;
+       u64 size;
+};
+\end{lstlisting}
+
+The VIRTIO_TRANSPTQ_CTRL_VQ_NOTIFY_GET is used to get the transport
+specific address area that can be used to notify a virtqueue. The
+command-out-data is an u16 of the queue index. The command-in-data
+contains the \field{address} and the \field{size}
+of the notification area (as described in struct virtio_transportq_ctrl_vq_notification_area).
+
+\devicenormative{\subsubsection}{Virtqueue Notification}{Virtio Transport Options /
+Virtio Over Transport Virtqueue / Virtqueue Notification}
+
+The management device MUST fail the virtqueue notification area information
+access if \field{device_id} is 0.
+
+The management device MUST fail the virtqueue notification area information
+access if the queue index is out-of-range invalid.
+
+\drivernormative{\subsubsection}{Virtqueue Notification}{Virtio Transport Options /
+Virtio Over Transport Virtqueue / Virtqueue Notification}
+
+The driver MAY choose to notify the virtqueue by writing the queue
+index at address \field{address} which is fetched from the
+VIRTIO_TRANSPTQ_CTRL_VQ_NOTIFY_GET command.
+
+\subsection{Virtqueue State}\label{sec:Virtio Transport Options / Virtio
+Over Transport Virtqueue / Virtqueue State}
+
+When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
+the virtqueue state is accessed through the following commands:
+
+\begin{lstlisting}
+#define VIRTIO_TRANSPTQ_CTRL_VQ_STATE            12
+ #define VIRTIO_TRANSPTQ_CTRL_VQ_STATE_GET       0
+ #define VIRTIO_TRANSPTQ_CTRL_VQ_STATE_SET       1
+\end{lstlisting}
+
+\subsubsection{Split Virtqueue State}\label{sec:Virtio Transport Options / Virtio
+Over Transport Virtqueue / Virtqueue State / Split Virtqueue State}
+
+\begin{lstlisting}
+struct virtio_transportq_ctrl_split_vq_state_set {
+    u16 queue_index;
+    u16 avail_index;
+    u8 padding[4];
+};
+\end{lstlisting}
+
+The VIRTIO_TRANSPTQ_CTRL_VQ_STATE_GET command can be  used to get the state of a
+split virtqueue. The command-out-data is the queue index, the command-in-data is the available index of the virtqueue.
+
+The VIRTIO_TRANSPTQ_CTRL_VQ_STATE_SET command can be used to set the state of a
+split virtqueue. The command-out-data contains the \field{queue_index} and the
+available index (\field{avail_index}) for the virtqueue (as described in struct virtio_transportq_ctrl_split_vq_state_set).
+There is no command-in-data.
+
+\subsubsection{Packed Virtqueue State}\label{sec:Virtio Transport Options / Virtio
+Over Transport Virtqueue / Virtqueue State / Packed Virtqueue State}
+
+\begin{lstlisting}
+struct virtio_transportq_ctrl_packed_vq_state {
+    u16 queue_index;
+    u16 avail_counter;
+    u16 avail_index;
+    u16 used_counter;
+    u16 used_index;
+    u8 padding[6];
+};
+\end{lstlisting}
+
+The state of a packed virtqueue includes :\\
+\field{avail_counter}: last driver ring wrap counter observed by device.\\
+\field{avail_index}: virtqueue available index.\\
+\field{used_counter}: device ring wrap counter.\\
+\field{used_index}: virtqueue used index.
+
+The VIRTIO_TRANSPTQ_CTRL_VQ_STATE_GET command can be used to get the state of a packed virtqueue.
+The command-out-data is the queue index, the command-in-data contains the \field{queue_index},
+\field{avail_counter}, \field{avail_index}, \field{used_counter} and \field{used_index} of the virtqueue
+(as described in transportq_ctrl_packed_vq_state).
+
+The VIRTIO_TRANSPTQ_CTRL_VQ_STATE_SET command can be used to set the state of a packed virtqueue.
+The command-out-data contains the \field{queue_index}, \field{avail_counter}, \field{avail_index},
+\field{used_counter} and \field{used_index} for the virtqueue
+(as described in transportq_ctrl_packed_vq_state).
+There is no command-in-data.
+
+\devicenormative{\subsubsection}{Virtqueue State}{Virtio Transport Options /
+Virtio Over Transport Virtqueue / Virtqueue State}
+
+The management device MUST fail the virtqueue index access if \field{device_id} is 0.
+
+The management device MUST fail the virtqueue index access if \field{queue_index} is out-of-range invalid.
+
+\subsection{Virtqueue ASID}\label{sec:Virtio Transport Options / Virtio Over
+Transport Virtqueue / Virtqueue ASID}
+
+When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
+the address space id of a virtqueue could be set through the following command:
+
+\begin{lstlisting}
+#define VIRTIO_TRANSPTQ_CTRL_VQ_ASID        13
+ #define VIRTIO_TRANSPTQ_CTRL_VQ_ASID_SET   1
+
+struct virtio_transportq_ctrl_vq_asid_set {
+       u16 queue_index;
+       u32 asid;
+       u8 padding[2];
+};
+\end{lstlisting}
+
+The VIRTIO_TRANSPTQ_CTRL_VQ_ASID_SET command is used to set the address space id(\field{asid})
+of a virtqueue.
+
+The address space id is an identifier of a memory space which is used to convey the address space targeted by the
+memory accesses, and to distinguish memory accesses performed by different virtqueues
+
+One example of the address space id is PASID (Process Address Space Identifier) which is
+defined in \hyperref[intro:PCIe]{[PCIe]} specification.
+
+The command-out-data of VIRTIO_TRANSPTQ_CTRL_VQ_ASID_SET command is the \field{queue_index}
+and the address space id (\field{asid}).
+(as described in struct virtio_transportq_ctrl_vq_asid_set).
+
+\devicenormative{\subsubsection}{Virtqueue ASID}{Virtio Transport Options /
+Virtio Over Transport Virtqueue / Virtqueue ASID}
+
+Once a virtqueue has been set an asid, it MUST perform any memory accesses with the asid.
+
+The management device MUST fail the virtqueue index access if \field{device_id} is 0.
+
+The management device MUST fail the virtqueue index access if \field{queue_index} is out-of-range invalid.
+
+\subsection{Virtqueue Reset}\label{sec:Virtio Transport Options / Virtio Over
+Transport Virtqueue / Virtqueue Reset}
+
+When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
+a virtqueue can be reset through the following command:
+
+\begin{lstlisting}
+#define VIRTIO_TRANSPTQ_CTRL_VQ_RESET        14
+ #define VIRTIO_TRANSPTQ_CTRL_VQ_DO_RESET    1
+\end{lstlisting}
+
+The VIRTIO_TRANSPTQ_CTRL_VQ_DO_RESET command is used to reset a virtqueue.
+The command-out-data is the virtqueue index, there is no command-in-data.
+
+\devicenormative{\subsubsection}{Virtqueue Reset}{Virtio Transport Options /
+Virtio Over Transport Virtqueue / Virtqueue Reset}
+
+The management device MUST fail VIRTIO_TRANSPTQ_CTRL_VQ_RESET if
+\field{device_id} is 0.
+
+The management device MUST fail VIRTIO_TRANSPTQ_CTRL_VQ_RESET if
+the virtqueue index is out-of-range  invalid.
+
+The managed device MUST stop consuming the descriptors in the virtqueue once reset it.
+
+The managed device MUST present default states of a virtqueue after reset it.
+
+\drivernormative{\subsubsection}{Virtqueue Reset}{Virtio Transport Options /
+Virtio Over Transport Virtqueue / Virtqueue Reset}
+
+After reseting a virtqueue, the driver MUST wait until the reset is successfully
+done (by verifying virtio_transportq_ctrl.ack == VIRTIO_TRANSPTQ_OK)
+before re-enabling it.
 
 \chapter{Device Types}\label{sec:Device Types}
 
-- 
2.35.3


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] [PATCH V3 RESEND 2/4] Introduce the commands set of the transport vq

[Jason Wang]

On Fri, Aug 5, 2022 at 6:02 PM Zhu Lingshan <lingshan.zhu@intel.com> wrote:
>
> This commit introduces the commands set of the
> transport virtqueue, including:
>
> The command to query available resources of the management device
> The commands to create / destroy the managed devices.
> The commands to config the managed devices.
> The commands to config virtqueues of the managed devices.
>
> Signed-off-by: Jason Wang <jasowang@redhat.com>
> Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com>
> ---
>  content.tex | 636 ++++++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 636 insertions(+)
>
> diff --git a/content.tex b/content.tex
> index c747d21..190f3a0 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -2975,6 +2975,642 @@ \subsection{Format of Commands through Transport Virtqueue}\label{sec:Virtio Tra
>  The \field{device_id} value 0 is used to identify the management device itself.
>
>  \field{class} is an identifier of a set of commands with similar  purposes.
> +
> +\subsection{Available Resource of the Management Device}\label{sec:Virtio Transport Options / Virtio Over Transport Virtqueue / Available Resource of the Management Device}
> +
> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
> +statistical information of available resources on the management device could be fetched
> +by the following command:
> +
> +\begin{lstlisting}
> +#define VIRTIO_TRANSPTQ_CTRL_AVAIL_RES    1 (class)
> + #define VIRTIO_TRANSPTQ_CTRL_AVAIL_RES_GET     0 (command)
> +\end{lstlisting}
> +
> +VIRTIO_TRANSPTQ_CTRL_AVAIL_RES_GET command is used to get the statistical information of the available
> +resource of a management device. There is no command-out-data for VIRTIO_TRANSPTQ_CTRL_AVAIL_RES_GET
> +command. The management device fills command-in-data in the following format:
> +
> +\begin{lstlisting}
> +struct virtio_mgmt_dev_avail_res {
> +        /* The number of management device total remaining available free
> +         * virtqueues for the managed devices
> +         */

It's hard to phrase this sentence.

> +        u16 num_avail_vqs;
> +        /* The minimal number of virtqueues for a managed device */
> +        u16 min_dev_vqs;
> +        /* The maximum number of virtqueues for a managed device */

Nit the min/max_dev_vqs are not the resources, so we'd better rename
the virtio_mgmt_dev_avail_res.

> +        u16 max_dev_vqs;
> +        /* The number of managed devices that can be created with min_vqs virtqueues */

Is the part "with min_vqs virtqueues" a must? My understanding of the
above is a hint that the managed device is loosely coupled with the
virtqueue resources.

So it would be possible that num_avail_dev = 0 but num_avail_vqs not?

Btw, "num" prefix is probably not a must.

> +        u16 num_avail_dev;

To be consistent, let's use "num_avail_devs"?

> +};
> +\end{lstlisting}
> +
> +\field{max_avail_dev} MAY not equal to \field{num_avail_vqs} divided by \field{min_vqs}
> +due to the limitations of of other resources.

two "of".

Keywords like "MAY" should go to normatives part.

> +
> +\devicenormative{\subsubsection}{Available Resource of the Management Device}{Virtio Transport Options /
> +Virtio Over Transport Virtqueue / Available Resource of the Management Device}
> +
> +The management device MUST fail VIRTIO_TRANSPTQ_CTRL_AVAIL_RES_GET command if \field{device_id} is not 0
> +
> +\drivernormative{\subsubsection}{Available Resource of the Management Device}{Virtio Transport Options /
> +Virtio Over Transport Virtqueue / Available Resource of the Management Device}
> +
> +The management device driver MUST use 0 as \field{device_id} for
> +VIRTIO_TRANSPTQ_CTRL_AVAIL_RES_GET command.
> +
> +\subsection{Create and Destroy Managed Devices}\label{sec:Virtio Transport Options / Virtio Over Transport Virtqueue / Create and Destroy Managed Devices}
> +
> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
> +managed devices must be created and destroyed through the transport virtqueue.
> +
> +\begin{lstlisting}
> +#define VIRTIO_TRANSPORTQ_CTRL_DEV    2
> + #define VIRTIO_TRANSPORTQ_CTRL_DEV_CREATE        0
> + #define VIRTIO_TRANSPORTQ_CTRL_DEV_DESTROY       1
> +
> +struct virtio_transportq_ctrl_dev_attribute {
> +       u64 device_features[2];

Let's reserve more bits for features here as we are about to reach 64.

> +       u32 virtio_device_id;
> +       u8 dev_config[];

To be consistent, let's use one of "device" or "dev" in the same structure.

> +};
> +\end{lstlisting}
> +
> +The VIRTIO_TRANSPORTQ_CTRL_DEV_CREATE command is used to create a managed device.
> +
> +As described in struct virtio_transportq_ctrl_dev_attribute, the command-out-data for VIRTIO_TRANSPTQ_CTRL_DEV_CREATE includes:
> +\begin{itemize*}
> +\item \field{device_features}: the 128-bit device feature bits (defined in section \ref{sec:Basic Facilities of a Virtio Device / Feature Bits})
> +of the managed device.
> +\item \field{virtio_device_id}: the virtio device id defined in Chapter \ref{sec:Device Types}.
> +\item \field{dev_config}: the device type specific configurations. E.g., for a virtio-net device,
> +it is struct virtio_net_config in section \ref{sec:Device Types / Network Device / Device configuration layout};
> +for a virtio-block device, it is struct virtio_blk_config in section \ref{sec:Device Types / Block Device / Device configuration layout}
> +\end{itemize*}
> +
> +When succeed, the device returns a 64 bits UUID as an unique identifier of the created managed device in command-in-data.

We don't need to be universally unique here.

> +
> +The VIRTIO_TRANSPORTQ_CTRL_DEV_DESTROY command is used to destroy a
> +managed device which is identified by its 64 bits UUID
> +\field{device_id}. There's no command-in-data nor command-out-data for
> +VIRTIO_TRANSPTQ_CTRL_DEV_DESTROY command.
> +
> +\devicenormative{\subsubsection}{Create and Destroy Managed Devices}{Virtio Transport Options / Virtio Over Transport Virtqueue / Create and Destroy Managed Devices}
> +
> +The management device MUST fail VIRTIO_TRANSPORTQ_CTRL_DEV_CREATE command
> +if \field{device_id} is not 0.
> +
> +The management device MUST fail VIRTIO_TRANSPORTQ_CTRL_DEV CREATE command
> +if \field{device_features} exceeds the features that can be provided from the management device.

So PCI has le32 device_feature_select. How to make sure it can match
the capacity of device_features[] above?

> +
> +The management device MUST fail VIRTO_TRANSPORTQ_CTRL_DEV_DESTROY command
> +if \field{device_id} is 0.
> +
> +\drivernormative{\subsubsection}{Create and Destroy Managed Devices}{Virtio Transport Options / Virtio Over Transport Virtqueue / Create and Destroy Managed Devices}
> +
> +The management device driver MUST use 0 as \field{device_id} for
> +TRANSPORTQ_CTRL_DEV_CREATE command.
> +
> +\subsection{Features Negotiation}\label{sec:Virtio Transport Options / Virtio
> +Over Transport Virtqueue / Features Negotiation}
> +
> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
> +the features negotiation of managed devices is done by the
> +following commands:
> +
> +\begin{lstlisting}
> +#define VIRTIO_TRANSPTQ_CTRL_FEAT   3
> + #define VIRTIO_TRANSPTQ_CTRL_FEAT_DEVICE_GET        0
> + #define VIRTIO_TRANSPTQ_CTRL_FEAT_DRIVER_SET        1
> + #define VIRTIO_TRANSPTQ_CTRL_FEAT_DRIVER_GET        2
> +\end{lstlisting}
> +
> +The VIRTIO_TRANSPTQ_CTRL_FEAT_DEVICE_GET command is used to get the features offered
> +by a managed device. The command-in-data is the 128-bit feature bits
> +(defined in section \ref{sec:Basic Facilities of a Virtio Device / Feature Bits}).

Need to make sure the capacity of the feature bit matches what the
transport of the management device can provide.

> +There is no command-out-data.
> +
> +The VIRTIO_TRANSPTQ_CTRL_FEAT_DRIVER_SET command is for the driver to accept feature
> +bits offered by the managed device. The command-out-data is the 128-bit feature bits
> +passed to the managed device. There is no command-in-data.
> +
> +The VIRTIO_TRANSPTQ_CTRL_FEAT_DRIVER_GET command is used to get the features accepted
> +by both the managed device driver and the managed device. The command-in-data is the
> +128-bit feature bits. There is no command-out-data.
> +
> +\devicenormative{\subsubsection}{Features Negotiation}{Virtio Transport Options /
> +Virtio Over Transport Virtqueue / Features Negotiation}
> +
> +The management device MUST fail VIRTIO_TRANSPTQ_F_CTRL_FEAT class commands if
> +\field{device_id} is 0.
> +
> +\subsection{Device Status}\label{sec:Virtio Transport Options / Virtio Over
> +Transport Virtqueue / Device Status}
> +
> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
> +the device status of a managed device can be accessed by the following
> +commands:
> +
> +\begin{lstlisting}
> +#define VIRTIO_TRANSPTQ_CTRL_STATUS    4
> + #define VIRTIO_TRANSPTQ_CTRL_STATUS_GET        0
> + #define VIRTIO_TRANSPTQ_CTRL_STATUS_SET        1
> +\end{lstlisting}
> +
> +The VIRTIO_TRANSPTQ_CTRL_STATUS_GET command is used to get the device status of
> +a managed device. The command-in-data is the 8-bit status
> +returned from the device. There's no command-out-data for this
> +command.
> +
> +The VIRTIO_TRANSPTQ_CTRL_STATUS_SET command is used to set the device status of
> +a managed device. The command-out-data is the 8-bit status
> +to set to the device. There's no command-in-data for this command.
> +
> +\devicenormative{\subsubsection}{Device Status}{Virtio Transport Options / Virtio
> +Over Transport Virtqueue / Device Status}
> +
> +The management device MUST reset the managed device when 0
> +is set via VIRTIO_TRANSPTQ_CTRL_STATUS_SET, the success of this
> +command demonstrates the success of the reset.
> +
> +The management device MUST present 0 through
> +VIRTIO_TRANSPTQ_CTRL_STATUS_GET once the reset is successfully done.
> +
> +The management device MUST fail the device status access if
> +\field{device_id} is 0.
> +
> +\drivernormative{\subsubsection}{Device Status}{Virtio Transport Options / Virtio
> +Over Transport Virtqueue / Device Status}
> +
> +After writing 0 via VIRTIO_TRANSPTQ_CTRL_STATUS_SET, the driver MUST wait
> +for the success of the command before re-initializing the device.
> +
> +\subsection{Device Generation}\label{sec:Virtio Transport Options / Virtio Over Transport Virtqueue / Device Generation}
> +
> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,

Btw, since we're under the transport options (which implies the
transport virtqueue), maybe we can drop the above for each command.

> +the device generation count can be read from the following commands:
> +
> +\begin{lstlisting}
> +#define VIRTIO_TRANSPTQ_CTRL_GENERATION    5
> + #define VIRTIO_TRANSPTQ_CTRL_GENERATION_GET        0
> +\end{lstlisting}
> +
> +The VIRTIO_TRANSPTQ_CTRL_GENERATION_GET command is used to get the device configuration generation count
> +of the managed device. The command-in-data is the one byte device
> +generation returned from the device. There's no command-out-data for
> +this command.
> +
> +\devicenormative{\subsubsection}{Device Generation}{Virtio Transport Options / Virtio Over Transport Virtqueue / Device Generation}
> +
> +The managed device MUST present a changed generation count after the driver
> +has read any device-specific configuration values if the values have been changed
> +during the last read.
> +
> +The management device MUST fail the device generation access if \field{device_id} is 0.
> +
> +\subsection{Device Specific Configuration}\label{sec:Virtio Transport Options /
> +Virtio Over Transport Virtqueue / Device Specific Configuration}
> +
> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
> +the device config space contents of a managed device can be accessed through
> +the following commands:
> +
> +\begin{lstlisting}
> +#define VIRTIO_TRANSPTQ_CTRL_CONFIG    6
> +  #define VIRTIO_TRANSPTQ_CTRL_CONFIG_GET    0
> +  #define VIRTIO_TRANSPTQ_CTRL_CONFIG_SET    1
> +
> +struct virtio_transportq_ctrl_dev_config_get {
> +       u32 offset;
> +       u32 size;
> +};
> +
> +struct virtio_transportq_ctrl_dev_config_set {
> +       u32 offset;
> +       u32 size;
> +       u8  data[];
> +};
> +\end{lstlisting}
> +
> +The VIRTIO_TRANSPTQ_CTRL_CONFIG_GET command is used to read data from the
> +device configuration space.  The command-out-data is the \field{offset}
> +from the start of the config space and the \field{size}
> +of the data (as described in struct virtio_transportq_ctrl_dev_config_get).
> +The command-in-data is an array of the data that read from the config space.
> +
> +The VIRTIO_TRANSPTQ_CTRL_CONFIG_SET command is used to write data to the device
> +configuration space. The command-out-data contains the
> +\field{offset} from the start of the config space, the \field{size} of the data and
> +the \field{data} to be written (as described in struct virtio_transportq_ctrl_dev_config_set).
> +There's no command-in-data for this command.
> +
> +\devicenormative{\subsubsection}{Device Specific Configuration}{Virtio Transport
> +Options / Virtio Over Transport Virtqueue / Device Specific Configuration}
> +
> +The management device MUST fail the device configuration space access
> +if the driver access the range which is outside the config space.

"accesses"

> +
> +The management device MUST fail the device configuration space access
> +if \field{device_id} is 0.
> +
> +\subsection{MSI Configuration}\label{sec:Virtio Transport Options / Virtio Over
> +Transport Virtqueue / MSI Configuration}
> +
> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
> +the MSI entries of a managed device can be accessed  through the following command:

Rethink of this, it implies a per virtqueue MSI storage which I'm not
sure is expensive or not. Do we need an indirection layer as PCI did?

> +
> +\begin{lstlisting}
> +#define VIRTIO_TRANSPTQ_CTRL_MSI    7
> + #define VIRTIO_TRANSPTQ_CTRL_MSI_VQ_GET        0
> + #define VIRTIO_TRANSPTQ_CTRL_MSI_VQ_SET        1
> + #define VIRTIO_TRANSPTQ_CTRL_MSI_VQ_ENABLE     2
> + #define VIRTIO_TRANSPTQ_CTRL_MSI_CONFIG_GET    3
> + #define VIRTIO_TRANSPTQ_CTRL_MSI_CONFIG_WET    4

I think you mean CONFIG_SET here.

> + #define VIRTIO_TRANSPTQ_CTRL_MSI_CONFIG_ENABLE 5
> +
> +struct virtio_transportq_ctrl_msi_vq_config {
> +       u16 queue_index;
> +       u64 address;
> +       u32 data;
> +       u8 padding[2];
> +};
> +
> +struct virtio_transportq_ctrl_msi_vq_enable {
> +       u16 queue_index;
> +       u8 enable;
> +       u8 padding[5];
> +};
> +
> +struct virtio_transportq_ctrl_msi_config {
> +       u64 address;
> +       u32 data;
> +       u8 padding[4];
> +};
> +\end{lstlisting}
> +
> +The VIRTIO_TRANSPTQ_CTRL_MSI_VQ_GET command is used to read the MSI entry of a
> +specific virtqueue. The command-out-data is the queue index. The command-in-data contains
> +\field{queue_index}, the \field{address} and \field{data}
> +(as described in struct virtio_transportq_ctrl_msi_vq_config).

Let's split or reuse the virtio_transportq_ctrl_msi_config since
virtio_transportq_ctrl_msi_vq_config will not be used a as single
buffer.

> +
> +
> +The VIRTIO_TRANSPTQ_CTRL_MSI_VQ_SET command is used to set the MSI entry for a
> +specific virtqueue. The command-out-data is the \field{queue_index},
> +\field{address} and \field{data} (as described in struct virtio_transportq_ctrl_msi_vq_config).
> +There is no command-in-data.
> +
> +The VIRTIO_TRANSPTQ_CTRL_MSI_VQ_ENABLE command is used to enable or disable
> +the MSI interrupt for a specific virtqueue. The command-out-data is the
> +\field{queue_index} and a byte \field{enable} which representing ENABLE or DISABLE the MSI.

Should be "represents".

Btw, is there a chance that we may lose an interrupt?

> +(as described in struct virtio_transportq_ctrl_msi_vq_enable).
> +
> +\begin{lstlisting}
> +#define VIRTIO_TRANSPTQ_CTRL_MSI_VQ_ENABLE     1
> +#define VIRTIO_TRANSPTQ_CTRL_MSI_VQ_DISABLE    0
> +\end{lstlisting}
> +
> +The VIRTIO_TRANSPTQ_CTRL_MSI_CONFIG_GET command is used to read the MSI entry
> +of the config interrupt. The command-in-data contains the \field{address} and
> +\field{data} (as described in struct virtio_transportq_ctrl_msi_config).
> +There is no command-out-data.
> +
> +The VIRTIO_TRANSPTQ_CTRL_MSI_CONFIG_SET command is used to set the MSI entry
> +for the config interrupt. The command-out-data is the \field{address} and
> +\field{data} (as described in struct virtio_transportq_ctrl_msi_config).
> +There is no command-in-data.
> +
> +The VIRTIO_TRANSPTQ_CTRL_MSI_CONFIG_ENABLE command is used to enable or disable
> +the MSI interrupt for config space. The command-out-data is a byte which representing
> +ENABLE or DISABLE the config MSI.
> +There is no command-in-data
> +
> +\begin{lstlisting}
> +#define VIRTIO_TRANSPTQ_CTRL_MSI_CFG_ENABLE     1
> +#define VIRTIO_TRANSPTQ_CTRL_MSI_CFG_DISABLE    0
> +\end{lstlisting}
> +
> +\devicenormative{\subsubsection}{MSI Configuration}{Virtio Transport Options / Virtio
> +ver Transport Virtqueue / MSI Configuration}
> +
> +The managed device MUST disable the MSI interrupts for both virtqueues and config space
> +upon a reset.
> +
> +Once a MSI entry is disabled, the managed device MUST not send any interrupts
> +by this MSI entry.

MSI is stored based on virtqueue and config space, so I'm not quite
sure "MSI entry" is accurate here.

> +
> +\drivernormative{\subsubsection}{MSI Configuration}{Virtio Transport Options / Virtio
> +Over Transport Virtqueue / MSI Configuration}
> +
> +The driver MUST allocate transport or platform specific MSI entries
> +for both virtqueues and config space if it wants to use interrupts.
> +
> +The driver MAY choose to disable the MSI if polling mode is used.
> +
> +\subsection{Virtqueue Address}\label{sec:Virtio Transport Options / Virtio Over
> +Transport Virtqueue / Virtqueue Address}
> +
> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
> +the addresses of a specific virtqueue are accessed through the following command:
> +
> +\begin{lstlisting}
> +#define VIRTIO_TRANSPTQ_CTRL_VQ_ADDR    8
> + #define VIRTIO_TRANSPTQ_CTRL_VQ_ADDR_SET       1
> +
> +struct virtio_transportq_ctrl_vq_addr {
> +       u16 queue_index;
> +       u64 descriptor_area;
> +       u64 device_area;
> +       u64 driver_area;
> +       u8 padding[6];
> +};
> +\end{lstlisting}
> +
> +The VIRTIO_TRANSPTQ_CTRL_VQ_ADDR_SET command is used to set the addresses of a specified
> +virtqueue. The command-out-data contains the \field{queue_index}, the addresses of \field{device_area},
> +\field{descriptor_area} and \field{driver_area} (as described in struct
> +virtio_transportq_ctrl_vq_addr). There's no command-in-data.
> +
> +\devicenormative{\subsubsection}{Virtqueue Address}{Virtio Transport Options / Virtio
> +Over Transport Virtqueue / Virtqueeu Address}
> +
> +The management device MUST fail the commands of class
> +VIRTIO_TRANSPTQ_CTRL_VQ_ADDR if \field{device_id} is 0.
> +
> +The management device MUST fail the commands of class
> +VIRTIO_TRANSPTQ_CTRL_VQ_ADDR if \field{queue_index} is out-of-range invalid.
> +
> +\subsection{Virtqueue Status}\label{sec:Virtio Transport Options / Virtio Over
> +Transport Virtqueue / Virtqueue Status}
> +
> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
> +virtqueue status is accessed through the following command:
> +
> +\begin{lstlisting}
> +#define VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE    9
> + #define VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_GET       0
> + #define VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_SET       1
> +
> +struct virtio_transportq_ctrl_vq_status_set {
> +       u16 queue_index;
> +       u8 status;
> +       u8 padding[5];
> +
> +#define VIRTIO_TRANSPTQ_VQ_ENABLE     1
> +#define VIRTIO_TRANSPTQ_VQ_DISABLE    0
> +
> +};
> +
> +\end{lstlisting}
> +
> +The VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_GET is used to get the status of a
> +specific virtqueue. The command-out-data is the queue
> +index. The command-in-data is the virtqueue status.
> +
> +The VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_SET command is used to set the status of a
> +specific virtqueue. The command-out-data is the \field{queue_index} and the
> +\field{status} representing ENABLE or DISABLE that is set to the virtqueue
> +(as described in struct virtio_transportq_ctrl_vq_status_set).
> +There's no command-in-data.
> +
> +\devicenormative{\subsubsection}{Virtqueue Status}{Virtio Transport Options / Virtio
> +Over Transport Virtqueue / Virtqueue Status}
> +
> +When disabled, the managed device MUST stop processing requests from
> +this virtqueue.

Is this a hint that the driver can write VIRTIO_TRANSPTQ_VQ_DISABLE to
status? If yes, what's the difference between this and virtqueue
reset?

> +
> +The management device MUST present a 0 via
> +VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_GET upon a reset of the managed device.
> +
> +The management device MUST fail the virtqueue status access if
> +\field{device_id} is 0.
> +
> +The management device MUST fail the virtqueue status access if
> +the queue_index is out-of-range invalid.
> +
> +
> +\drivernormative{\subsubsection}{Virtqueue Status}{Virtio Transport Options / Virtio
> +Over Transport Virtqueue / Virtqueue Status}
> +
> +The driver MUST configure other virtqueue fields before enabling
> +the virtqueue with VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_SET.
> +
> +\subsection{Virtqueue Size}\label{sec:Virtio Transport Options / Virtio Over
> +Transport Virtqueue / Virtqueue Size}
> +
> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
> +virtqueue size is accessed through the following command:
> +
> +\begin{lstlisting}
> +#define VIRTIO_TRANSPTQ_CTRL_VQ_SIZE    10
> + #define VIRTIO_TRANSPTQ_CTRL_VQ_SIZE_GET       0
> + #define VIRTIO_TRANSPTQ_CTRL_VQ_SIZE_SET       1
> +
> +struct virtio_transportq_ctrl_vq_size_set {
> +       u16 queue_index;
> +       u16 size;
> +       u8 padding[4];
> +};
> +\end{lstlisting}
> +
> +The VIRTIO_TRANSPTQ_CTRL_VQ_SIZE_GET command is used to get the virtqueue
> +size. On reset, the maximum queue size supported by the device is
> +returned. The command-out-data is the queue index. The
> +command-in-data is an 16-bit queue size.
> +
> +The VIRTIO_TRANSPTQ_CTRL_VQ_SIZE_SET command is used to set the virtqueue
> +size. The command-out-data is the \field{queue_index} and the \field{size}
> +of the virtqueue (as described in struct virtio_transportq_ctrl_vq_size_set).
> +There's no command-in-data.
> +
> +\devicenormative{\subsubsection}{Virtqueue Status}{Virtio Transport Options / Virtio
> +Over Transport Virtqueue / Virtqueue Size}
> +
> +The management device MUST fail the virtqueue size access if
> +\field{device_id} is 0.
> +
> +The management device MUST fail the virtqueue size access if
> +the queue index is out-of-range invalid.
> +
> +\subsection{Virtqueue Notification}\label{sec:Virtio Transport Options / Virtio
> +Over Transport Virtqueue / Virtqueue Notification}
> +
> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
> +the virtqueue notification area information can be get through the following commands:
> +
> +\begin{lstlisting}
> +#define VIRTIO_TRANSPTQ_CTRL_VQ_NOTIFY    11
> + #define VIRTIO_TRANSPTQ_CTRL_VQ_NOTIFY_GET          0
> +
> +struct virtio_transportq_ctrl_vq_notification_area {
> +       u64 address;
> +       u64 size;
> +};
> +\end{lstlisting}
> +
> +The VIRTIO_TRANSPTQ_CTRL_VQ_NOTIFY_GET is used to get the transport
> +specific address area that can be used to notify a virtqueue. The
> +command-out-data is an u16 of the queue index. The command-in-data
> +contains the \field{address} and the \field{size}
> +of the notification area (as described in struct virtio_transportq_ctrl_vq_notification_area).
> +
> +\devicenormative{\subsubsection}{Virtqueue Notification}{Virtio Transport Options /
> +Virtio Over Transport Virtqueue / Virtqueue Notification}
> +
> +The management device MUST fail the virtqueue notification area information
> +access if \field{device_id} is 0.
> +
> +The management device MUST fail the virtqueue notification area information
> +access if the queue index is out-of-range invalid.

We probably need to say the management device must reserve sufficient
transport specific resources as notification area or we need to have a
fallback of using a dedicated command to notify.

> +
> +\drivernormative{\subsubsection}{Virtqueue Notification}{Virtio Transport Options /
> +Virtio Over Transport Virtqueue / Virtqueue Notification}
> +
> +The driver MAY choose to notify the virtqueue by writing the queue
> +index at address \field{address} which is fetched from the
> +VIRTIO_TRANSPTQ_CTRL_VQ_NOTIFY_GET command.
> +
> +\subsection{Virtqueue State}\label{sec:Virtio Transport Options / Virtio
> +Over Transport Virtqueue / Virtqueue State}
> +
> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
> +the virtqueue state is accessed through the following commands:
> +
> +\begin{lstlisting}
> +#define VIRTIO_TRANSPTQ_CTRL_VQ_STATE            12
> + #define VIRTIO_TRANSPTQ_CTRL_VQ_STATE_GET       0
> + #define VIRTIO_TRANSPTQ_CTRL_VQ_STATE_SET       1
> +\end{lstlisting}
> +
> +\subsubsection{Split Virtqueue State}\label{sec:Virtio Transport Options / Virtio
> +Over Transport Virtqueue / Virtqueue State / Split Virtqueue State}
> +
> +\begin{lstlisting}
> +struct virtio_transportq_ctrl_split_vq_state_set {
> +    u16 queue_index;
> +    u16 avail_index;
> +    u8 padding[4];
> +};
> +\end{lstlisting}
> +
> +The VIRTIO_TRANSPTQ_CTRL_VQ_STATE_GET command can be  used to get the state of a
> +split virtqueue. The command-out-data is the queue index, the command-in-data is the available index of the virtqueue.

We need to be accurate for "available index". E.g we have one for in
the shared memory, so do you mean last_avail_idx?

> +
> +The VIRTIO_TRANSPTQ_CTRL_VQ_STATE_SET command can be used to set the state of a
> +split virtqueue. The command-out-data contains the \field{queue_index} and the
> +available index (\field{avail_index}) for the virtqueue (as described in struct virtio_transportq_ctrl_split_vq_state_set).
> +There is no command-in-data.
> +
> +\subsubsection{Packed Virtqueue State}\label{sec:Virtio Transport Options / Virtio
> +Over Transport Virtqueue / Virtqueue State / Packed Virtqueue State}
> +
> +\begin{lstlisting}
> +struct virtio_transportq_ctrl_packed_vq_state {
> +    u16 queue_index;
> +    u16 avail_counter;
> +    u16 avail_index;
> +    u16 used_counter;
> +    u16 used_index;

Let's simply reuse the structure in the kernel headers.

Using u16 for wrap_couter seems unnecessary etc.

> +    u8 padding[6];
> +};
> +\end{lstlisting}
> +
> +The state of a packed virtqueue includes :\\
> +\field{avail_counter}: last driver ring wrap counter observed by device.\\
> +\field{avail_index}: virtqueue available index.\\
> +\field{used_counter}: device ring wrap counter.\\
> +\field{used_index}: virtqueue used index.
> +
> +The VIRTIO_TRANSPTQ_CTRL_VQ_STATE_GET command can be used to get the state of a packed virtqueue.
> +The command-out-data is the queue index, the command-in-data contains the \field{queue_index},
> +\field{avail_counter}, \field{avail_index}, \field{used_counter} and \field{used_index} of the virtqueue
> +(as described in transportq_ctrl_packed_vq_state).
> +
> +The VIRTIO_TRANSPTQ_CTRL_VQ_STATE_SET command can be used to set the state of a packed virtqueue.
> +The command-out-data contains the \field{queue_index}, \field{avail_counter}, \field{avail_index},
> +\field{used_counter} and \field{used_index} for the virtqueue
> +(as described in transportq_ctrl_packed_vq_state).
> +There is no command-in-data.
> +
> +\devicenormative{\subsubsection}{Virtqueue State}{Virtio Transport Options /
> +Virtio Over Transport Virtqueue / Virtqueue State}
> +
> +The management device MUST fail the virtqueue index access if \field{device_id} is 0.
> +
> +The management device MUST fail the virtqueue index access if \field{queue_index} is out-of-range invalid.
> +
> +\subsection{Virtqueue ASID}\label{sec:Virtio Transport Options / Virtio Over
> +Transport Virtqueue / Virtqueue ASID}
> +
> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
> +the address space id of a virtqueue could be set through the following command:
> +
> +\begin{lstlisting}
> +#define VIRTIO_TRANSPTQ_CTRL_VQ_ASID        13
> + #define VIRTIO_TRANSPTQ_CTRL_VQ_ASID_SET   1
> +
> +struct virtio_transportq_ctrl_vq_asid_set {
> +       u16 queue_index;
> +       u32 asid;
> +       u8 padding[2];

I'd reserve more here. E.g in the future each virtqueue may have a
secondary ASID,

> +};
> +\end{lstlisting}
> +
> +The VIRTIO_TRANSPTQ_CTRL_VQ_ASID_SET command is used to set the address space id(\field{asid})
> +of a virtqueue.
> +
> +The address space id is an identifier

Let's add "transport specific" before "identifier"

> of a memory space which is used to convey the address space targeted by the
> +memory accesses

We need to say the ASID is for DMA not here at least.

>, and to distinguish memory accesses performed by different virtqueues

This may confuse the readers since a single ASID could be shared by virtqueues.

> +
> +One example of the address space id is PASID (Process Address Space Identifier) which is
> +defined in \hyperref[intro:PCIe]{[PCIe]} specification.
> +
> +The command-out-data of VIRTIO_TRANSPTQ_CTRL_VQ_ASID_SET command is the \field{queue_index}
> +and the address space id (\field{asid}).
> +(as described in struct virtio_transportq_ctrl_vq_asid_set).
> +
> +\devicenormative{\subsubsection}{Virtqueue ASID}{Virtio Transport Options /
> +Virtio Over Transport Virtqueue / Virtqueue ASID}
> +
> +Once a virtqueue has been set an asid, it MUST perform any memory accesses with the asid.

Let's limit "memory accesses" to "DMA". And we need to say, this is
true only if the transport specific feature is enabled e.g via the PCI
PF PASID extended capability.

> +
> +The management device MUST fail the virtqueue index access if \field{device_id} is 0.
> +
> +The management device MUST fail the virtqueue index access if \field{queue_index} is out-of-range invalid.

What if the ASID is out of the range of PASID?

> +
> +\subsection{Virtqueue Reset}\label{sec:Virtio Transport Options / Virtio Over
> +Transport Virtqueue / Virtqueue Reset}
> +
> +When VIRTIO_F_TRANSPT_VQ

And reset is negotiated.

> is negotiated with the management device,
> +a virtqueue can be reset through the following command:
> +
> +\begin{lstlisting}
> +#define VIRTIO_TRANSPTQ_CTRL_VQ_RESET        14
> + #define VIRTIO_TRANSPTQ_CTRL_VQ_DO_RESET    1

RESET_RESET should be fine.

> +\end{lstlisting}
> +
> +The VIRTIO_TRANSPTQ_CTRL_VQ_DO_RESET command is used to reset a virtqueue.
> +The command-out-data is the virtqueue index, there is no command-in-data.
> +
> +\devicenormative{\subsubsection}{Virtqueue Reset}{Virtio Transport Options /
> +Virtio Over Transport Virtqueue / Virtqueue Reset}
> +
> +The management device MUST fail VIRTIO_TRANSPTQ_CTRL_VQ_RESET if
> +\field{device_id} is 0.
> +
> +The management device MUST fail VIRTIO_TRANSPTQ_CTRL_VQ_RESET if
> +the virtqueue index is out-of-range  invalid.
> +
> +The managed device MUST stop consuming the descriptors in the virtqueue once reset it.
> +
> +The managed device MUST present default states of a virtqueue after reset it.
> +

We need to "dup" the normatives for PCI queue reset here or consider a
place to unify.

Thanks

> +\drivernormative{\subsubsection}{Virtqueue Reset}{Virtio Transport Options /
> +Virtio Over Transport Virtqueue / Virtqueue Reset}
> +
> +After reseting a virtqueue, the driver MUST wait until the reset is successfully
> +done (by verifying virtio_transportq_ctrl.ack == VIRTIO_TRANSPTQ_OK)
> +before re-enabling it.
>
>  \chapter{Device Types}\label{sec:Device Types}
>
> --
> 2.35.3
>
>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>

Re: [virtio-comment] [PATCH V3 RESEND 2/4] Introduce the commands set of the transport vq

[Zhu, Lingshan]

On 8/8/2022 6:04 PM, Jason Wang wrote:
> On Fri, Aug 5, 2022 at 6:02 PM Zhu Lingshan <lingshan.zhu@intel.com> wrote:
>> This commit introduces the commands set of the
>> transport virtqueue, including:
>>
>> The command to query available resources of the management device
>> The commands to create / destroy the managed devices.
>> The commands to config the managed devices.
>> The commands to config virtqueues of the managed devices.
>>
>> Signed-off-by: Jason Wang <jasowang@redhat.com>
>> Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com>
>> ---
>>   content.tex | 636 ++++++++++++++++++++++++++++++++++++++++++++++++++++
>>   1 file changed, 636 insertions(+)
>>
>> diff --git a/content.tex b/content.tex
>> index c747d21..190f3a0 100644
>> --- a/content.tex
>> +++ b/content.tex
>> @@ -2975,6 +2975,642 @@ \subsection{Format of Commands through Transport Virtqueue}\label{sec:Virtio Tra
>>   The \field{device_id} value 0 is used to identify the management device itself.
>>
>>   \field{class} is an identifier of a set of commands with similar  purposes.
>> +
>> +\subsection{Available Resource of the Management Device}\label{sec:Virtio Transport Options / Virtio Over Transport Virtqueue / Available Resource of the Management Device}
>> +
>> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
>> +statistical information of available resources on the management device could be fetched
>> +by the following command:
>> +
>> +\begin{lstlisting}
>> +#define VIRTIO_TRANSPTQ_CTRL_AVAIL_RES    1 (class)
>> + #define VIRTIO_TRANSPTQ_CTRL_AVAIL_RES_GET     0 (command)
>> +\end{lstlisting}
>> +
>> +VIRTIO_TRANSPTQ_CTRL_AVAIL_RES_GET command is used to get the statistical information of the available
>> +resource of a management device. There is no command-out-data for VIRTIO_TRANSPTQ_CTRL_AVAIL_RES_GET
>> +command. The management device fills command-in-data in the following format:
>> +
>> +\begin{lstlisting}
>> +struct virtio_mgmt_dev_avail_res {
>> +        /* The number of management device total remaining available free
>> +         * virtqueues for the managed devices
>> +         */
> It's hard to phrase this sentence.
How about "The number of remaining available virtqueues for new managed 
devices"
>
>> +        u16 num_avail_vqs;
>> +        /* The minimal number of virtqueues for a managed device */
>> +        u16 min_dev_vqs;
>> +        /* The maximum number of virtqueues for a managed device */
> Nit the min/max_dev_vqs are not the resources, so we'd better rename
> the virtio_mgmt_dev_avail_res.
OK, I will use struct virito_mgmt_dev_info{} in the next version, and 
the command and its description
will change as well.
>
>> +        u16 max_dev_vqs;
>> +        /* The number of managed devices that can be created with min_vqs virtqueues */
> Is the part "with min_vqs virtqueues" a must? My understanding of the
> above is a hint that the managed device is loosely coupled with the
> virtqueue resources.
Yes, the number of the number of total managed devices can be limited by
other resources, like the filters.
>
> So it would be possible that num_avail_dev = 0 but num_avail_vqs not?
>
> Btw, "num" prefix is probably not a must.
OK
>
>> +        u16 num_avail_dev;
> To be consistent, let's use "num_avail_devs"?
I will remove all num_ prefix, and it should be avail_devs
>
>> +};
>> +\end{lstlisting}
>> +
>> +\field{max_avail_dev} MAY not equal to \field{num_avail_vqs} divided by \field{min_vqs}
>> +due to the limitations of of other resources.
> two "of".
will fix
>
> Keywords like "MAY" should go to normatives part.
OK, so I will only use "may", so it is a normal description.
>
>> +
>> +\devicenormative{\subsubsection}{Available Resource of the Management Device}{Virtio Transport Options /
>> +Virtio Over Transport Virtqueue / Available Resource of the Management Device}
>> +
>> +The management device MUST fail VIRTIO_TRANSPTQ_CTRL_AVAIL_RES_GET command if \field{device_id} is not 0
>> +
>> +\drivernormative{\subsubsection}{Available Resource of the Management Device}{Virtio Transport Options /
>> +Virtio Over Transport Virtqueue / Available Resource of the Management Device}
>> +
>> +The management device driver MUST use 0 as \field{device_id} for
>> +VIRTIO_TRANSPTQ_CTRL_AVAIL_RES_GET command.
>> +
>> +\subsection{Create and Destroy Managed Devices}\label{sec:Virtio Transport Options / Virtio Over Transport Virtqueue / Create and Destroy Managed Devices}
>> +
>> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
>> +managed devices must be created and destroyed through the transport virtqueue.
>> +
>> +\begin{lstlisting}
>> +#define VIRTIO_TRANSPORTQ_CTRL_DEV    2
>> + #define VIRTIO_TRANSPORTQ_CTRL_DEV_CREATE        0
>> + #define VIRTIO_TRANSPORTQ_CTRL_DEV_DESTROY       1
>> +
>> +struct virtio_transportq_ctrl_dev_attribute {
>> +       u64 device_features[2];
> Let's reserve more bits for features here as we are about to reach 64.
OK, it looks 256bytes may be enough.
>
>> +       u32 virtio_device_id;
>> +       u8 dev_config[];
> To be consistent, let's use one of "device" or "dev" in the same structure.
sure, device_config[]
>
>> +};
>> +\end{lstlisting}
>> +
>> +The VIRTIO_TRANSPORTQ_CTRL_DEV_CREATE command is used to create a managed device.
>> +
>> +As described in struct virtio_transportq_ctrl_dev_attribute, the command-out-data for VIRTIO_TRANSPTQ_CTRL_DEV_CREATE includes:
>> +\begin{itemize*}
>> +\item \field{device_features}: the 128-bit device feature bits (defined in section \ref{sec:Basic Facilities of a Virtio Device / Feature Bits})
>> +of the managed device.
>> +\item \field{virtio_device_id}: the virtio device id defined in Chapter \ref{sec:Device Types}.
>> +\item \field{dev_config}: the device type specific configurations. E.g., for a virtio-net device,
>> +it is struct virtio_net_config in section \ref{sec:Device Types / Network Device / Device configuration layout};
>> +for a virtio-block device, it is struct virtio_blk_config in section \ref{sec:Device Types / Block Device / Device configuration layout}
>> +\end{itemize*}
>> +
>> +When succeed, the device returns a 64 bits UUID as an unique identifier of the created managed device in command-in-data.
> We don't need to be universally unique here.
Why? I think it should be unique to identify a device, or are you 
suggesting to remove "as an unique identifier",
just UUID is enough?
>
>> +
>> +The VIRTIO_TRANSPORTQ_CTRL_DEV_DESTROY command is used to destroy a
>> +managed device which is identified by its 64 bits UUID
>> +\field{device_id}. There's no command-in-data nor command-out-data for
>> +VIRTIO_TRANSPTQ_CTRL_DEV_DESTROY command.
>> +
>> +\devicenormative{\subsubsection}{Create and Destroy Managed Devices}{Virtio Transport Options / Virtio Over Transport Virtqueue / Create and Destroy Managed Devices}
>> +
>> +The management device MUST fail VIRTIO_TRANSPORTQ_CTRL_DEV_CREATE command
>> +if \field{device_id} is not 0.
>> +
>> +The management device MUST fail VIRTIO_TRANSPORTQ_CTRL_DEV CREATE command
>> +if \field{device_features} exceeds the features that can be provided from the management device.
> So PCI has le32 device_feature_select. How to make sure it can match
> the capacity of device_features[] above?
we present all feature bits in device_features[], the spec says the 
feature bits are valid up to bit 127,
128 and above are reserved. So as you suggest, we can use u64 
device_features[4], for 128-bit feature bits
and another 128 bits for further extensions.
>
>> +
>> +The management device MUST fail VIRTO_TRANSPORTQ_CTRL_DEV_DESTROY command
>> +if \field{device_id} is 0.
>> +
>> +\drivernormative{\subsubsection}{Create and Destroy Managed Devices}{Virtio Transport Options / Virtio Over Transport Virtqueue / Create and Destroy Managed Devices}
>> +
>> +The management device driver MUST use 0 as \field{device_id} for
>> +TRANSPORTQ_CTRL_DEV_CREATE command.
>> +
>> +\subsection{Features Negotiation}\label{sec:Virtio Transport Options / Virtio
>> +Over Transport Virtqueue / Features Negotiation}
>> +
>> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
>> +the features negotiation of managed devices is done by the
>> +following commands:
>> +
>> +\begin{lstlisting}
>> +#define VIRTIO_TRANSPTQ_CTRL_FEAT   3
>> + #define VIRTIO_TRANSPTQ_CTRL_FEAT_DEVICE_GET        0
>> + #define VIRTIO_TRANSPTQ_CTRL_FEAT_DRIVER_SET        1
>> + #define VIRTIO_TRANSPTQ_CTRL_FEAT_DRIVER_GET        2
>> +\end{lstlisting}
>> +
>> +The VIRTIO_TRANSPTQ_CTRL_FEAT_DEVICE_GET command is used to get the features offered
>> +by a managed device. The command-in-data is the 128-bit feature bits
>> +(defined in section \ref{sec:Basic Facilities of a Virtio Device / Feature Bits}).
> Need to make sure the capacity of the feature bit matches what the
> transport of the management device can provide.
I will provide a structure for the features, so both device and driver 
would be aligned on
the size of the feature bits, like

struct virtio_transportq_ctrl_dev_features {
     u64 features[4];
};
>
>> +There is no command-out-data.
>> +
>> +The VIRTIO_TRANSPTQ_CTRL_FEAT_DRIVER_SET command is for the driver to accept feature
>> +bits offered by the managed device. The command-out-data is the 128-bit feature bits
>> +passed to the managed device. There is no command-in-data.
>> +
>> +The VIRTIO_TRANSPTQ_CTRL_FEAT_DRIVER_GET command is used to get the features accepted
>> +by both the managed device driver and the managed device. The command-in-data is the
>> +128-bit feature bits. There is no command-out-data.
>> +
>> +\devicenormative{\subsubsection}{Features Negotiation}{Virtio Transport Options /
>> +Virtio Over Transport Virtqueue / Features Negotiation}
>> +
>> +The management device MUST fail VIRTIO_TRANSPTQ_F_CTRL_FEAT class commands if
>> +\field{device_id} is 0.
>> +
>> +\subsection{Device Status}\label{sec:Virtio Transport Options / Virtio Over
>> +Transport Virtqueue / Device Status}
>> +
>> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
>> +the device status of a managed device can be accessed by the following
>> +commands:
>> +
>> +\begin{lstlisting}
>> +#define VIRTIO_TRANSPTQ_CTRL_STATUS    4
>> + #define VIRTIO_TRANSPTQ_CTRL_STATUS_GET        0
>> + #define VIRTIO_TRANSPTQ_CTRL_STATUS_SET        1
>> +\end{lstlisting}
>> +
>> +The VIRTIO_TRANSPTQ_CTRL_STATUS_GET command is used to get the device status of
>> +a managed device. The command-in-data is the 8-bit status
>> +returned from the device. There's no command-out-data for this
>> +command.
>> +
>> +The VIRTIO_TRANSPTQ_CTRL_STATUS_SET command is used to set the device status of
>> +a managed device. The command-out-data is the 8-bit status
>> +to set to the device. There's no command-in-data for this command.
>> +
>> +\devicenormative{\subsubsection}{Device Status}{Virtio Transport Options / Virtio
>> +Over Transport Virtqueue / Device Status}
>> +
>> +The management device MUST reset the managed device when 0
>> +is set via VIRTIO_TRANSPTQ_CTRL_STATUS_SET, the success of this
>> +command demonstrates the success of the reset.
>> +
>> +The management device MUST present 0 through
>> +VIRTIO_TRANSPTQ_CTRL_STATUS_GET once the reset is successfully done.
>> +
>> +The management device MUST fail the device status access if
>> +\field{device_id} is 0.
>> +
>> +\drivernormative{\subsubsection}{Device Status}{Virtio Transport Options / Virtio
>> +Over Transport Virtqueue / Device Status}
>> +
>> +After writing 0 via VIRTIO_TRANSPTQ_CTRL_STATUS_SET, the driver MUST wait
>> +for the success of the command before re-initializing the device.
>> +
>> +\subsection{Device Generation}\label{sec:Virtio Transport Options / Virtio Over Transport Virtqueue / Device Generation}
>> +
>> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
> Btw, since we're under the transport options (which implies the
> transport virtqueue), maybe we can drop the above for each command.
OK, can do, and I will describe this in the Basic Concept section.
>
>> +the device generation count can be read from the following commands:
>> +
>> +\begin{lstlisting}
>> +#define VIRTIO_TRANSPTQ_CTRL_GENERATION    5
>> + #define VIRTIO_TRANSPTQ_CTRL_GENERATION_GET        0
>> +\end{lstlisting}
>> +
>> +The VIRTIO_TRANSPTQ_CTRL_GENERATION_GET command is used to get the device configuration generation count
>> +of the managed device. The command-in-data is the one byte device
>> +generation returned from the device. There's no command-out-data for
>> +this command.
>> +
>> +\devicenormative{\subsubsection}{Device Generation}{Virtio Transport Options / Virtio Over Transport Virtqueue / Device Generation}
>> +
>> +The managed device MUST present a changed generation count after the driver
>> +has read any device-specific configuration values if the values have been changed
>> +during the last read.
>> +
>> +The management device MUST fail the device generation access if \field{device_id} is 0.
>> +
>> +\subsection{Device Specific Configuration}\label{sec:Virtio Transport Options /
>> +Virtio Over Transport Virtqueue / Device Specific Configuration}
>> +
>> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
>> +the device config space contents of a managed device can be accessed through
>> +the following commands:
>> +
>> +\begin{lstlisting}
>> +#define VIRTIO_TRANSPTQ_CTRL_CONFIG    6
>> +  #define VIRTIO_TRANSPTQ_CTRL_CONFIG_GET    0
>> +  #define VIRTIO_TRANSPTQ_CTRL_CONFIG_SET    1
>> +
>> +struct virtio_transportq_ctrl_dev_config_get {
>> +       u32 offset;
>> +       u32 size;
>> +};
>> +
>> +struct virtio_transportq_ctrl_dev_config_set {
>> +       u32 offset;
>> +       u32 size;
>> +       u8  data[];
>> +};
>> +\end{lstlisting}
>> +
>> +The VIRTIO_TRANSPTQ_CTRL_CONFIG_GET command is used to read data from the
>> +device configuration space.  The command-out-data is the \field{offset}
>> +from the start of the config space and the \field{size}
>> +of the data (as described in struct virtio_transportq_ctrl_dev_config_get).
>> +The command-in-data is an array of the data that read from the config space.
>> +
>> +The VIRTIO_TRANSPTQ_CTRL_CONFIG_SET command is used to write data to the device
>> +configuration space. The command-out-data contains the
>> +\field{offset} from the start of the config space, the \field{size} of the data and
>> +the \field{data} to be written (as described in struct virtio_transportq_ctrl_dev_config_set).
>> +There's no command-in-data for this command.
>> +
>> +\devicenormative{\subsubsection}{Device Specific Configuration}{Virtio Transport
>> +Options / Virtio Over Transport Virtqueue / Device Specific Configuration}
>> +
>> +The management device MUST fail the device configuration space access
>> +if the driver access the range which is outside the config space.
> "accesses"
yes
>
>> +
>> +The management device MUST fail the device configuration space access
>> +if \field{device_id} is 0.
>> +
>> +\subsection{MSI Configuration}\label{sec:Virtio Transport Options / Virtio Over
>> +Transport Virtqueue / MSI Configuration}
>> +
>> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
>> +the MSI entries of a managed device can be accessed  through the following command:
> Rethink of this, it implies a per virtqueue MSI storage which I'm not
> sure is expensive or not. Do we need an indirection layer as PCI did?
I think we need this for performance, or are you suggesting to add a
device scope MSI?
>
>> +
>> +\begin{lstlisting}
>> +#define VIRTIO_TRANSPTQ_CTRL_MSI    7
>> + #define VIRTIO_TRANSPTQ_CTRL_MSI_VQ_GET        0
>> + #define VIRTIO_TRANSPTQ_CTRL_MSI_VQ_SET        1
>> + #define VIRTIO_TRANSPTQ_CTRL_MSI_VQ_ENABLE     2
>> + #define VIRTIO_TRANSPTQ_CTRL_MSI_CONFIG_GET    3
>> + #define VIRTIO_TRANSPTQ_CTRL_MSI_CONFIG_WET    4
> I think you mean CONFIG_SET here.
yes
>
>> + #define VIRTIO_TRANSPTQ_CTRL_MSI_CONFIG_ENABLE 5
>> +
>> +struct virtio_transportq_ctrl_msi_vq_config {
>> +       u16 queue_index;
>> +       u64 address;
>> +       u32 data;
>> +       u8 padding[2];
>> +};
>> +
>> +struct virtio_transportq_ctrl_msi_vq_enable {
>> +       u16 queue_index;
>> +       u8 enable;
>> +       u8 padding[5];
>> +};
>> +
>> +struct virtio_transportq_ctrl_msi_config {
>> +       u64 address;
>> +       u32 data;
>> +       u8 padding[4];
>> +};
>> +\end{lstlisting}
>> +
>> +The VIRTIO_TRANSPTQ_CTRL_MSI_VQ_GET command is used to read the MSI entry of a
>> +specific virtqueue. The command-out-data is the queue index. The command-in-data contains
>> +\field{queue_index}, the \field{address} and \field{data}
>> +(as described in struct virtio_transportq_ctrl_msi_vq_config).
> Let's split or reuse the virtio_transportq_ctrl_msi_config since
> virtio_transportq_ctrl_msi_vq_config will not be used a as single
> buffer.
I don't get it, if we re-use the struct virtio_transportq_ctrl_msi_config,
it would be like:

struct virtio_transportq_ctrl_msi {
     u64 addr;
     u32 data;
};

struct virtio_transportq_ctrl_msi_vq_config {
     u16 queue_index;
     struct virtio_transportq_ctrl_msi;
};

it is still two structs and a struct in another struct.
>
>> +
>> +
>> +The VIRTIO_TRANSPTQ_CTRL_MSI_VQ_SET command is used to set the MSI entry for a
>> +specific virtqueue. The command-out-data is the \field{queue_index},
>> +\field{address} and \field{data} (as described in struct virtio_transportq_ctrl_msi_vq_config).
>> +There is no command-in-data.
>> +
>> +The VIRTIO_TRANSPTQ_CTRL_MSI_VQ_ENABLE command is used to enable or disable
>> +the MSI interrupt for a specific virtqueue. The command-out-data is the
>> +\field{queue_index} and a byte \field{enable} which representing ENABLE or DISABLE the MSI.
> Should be "represents".
OK
>
> Btw, is there a chance that we may lose an interrupt?
No, I think we won't lose any interrupts, because once disabled, it can 
not send new interrupts,
but the last interrupt send by the MSI entry is still there, it's in-band.
>
>> +(as described in struct virtio_transportq_ctrl_msi_vq_enable).
>> +
>> +\begin{lstlisting}
>> +#define VIRTIO_TRANSPTQ_CTRL_MSI_VQ_ENABLE     1
>> +#define VIRTIO_TRANSPTQ_CTRL_MSI_VQ_DISABLE    0
>> +\end{lstlisting}
>> +
>> +The VIRTIO_TRANSPTQ_CTRL_MSI_CONFIG_GET command is used to read the MSI entry
>> +of the config interrupt. The command-in-data contains the \field{address} and
>> +\field{data} (as described in struct virtio_transportq_ctrl_msi_config).
>> +There is no command-out-data.
>> +
>> +The VIRTIO_TRANSPTQ_CTRL_MSI_CONFIG_SET command is used to set the MSI entry
>> +for the config interrupt. The command-out-data is the \field{address} and
>> +\field{data} (as described in struct virtio_transportq_ctrl_msi_config).
>> +There is no command-in-data.
>> +
>> +The VIRTIO_TRANSPTQ_CTRL_MSI_CONFIG_ENABLE command is used to enable or disable
>> +the MSI interrupt for config space. The command-out-data is a byte which representing
>> +ENABLE or DISABLE the config MSI.
>> +There is no command-in-data
>> +
>> +\begin{lstlisting}
>> +#define VIRTIO_TRANSPTQ_CTRL_MSI_CFG_ENABLE     1
>> +#define VIRTIO_TRANSPTQ_CTRL_MSI_CFG_DISABLE    0
>> +\end{lstlisting}
>> +
>> +\devicenormative{\subsubsection}{MSI Configuration}{Virtio Transport Options / Virtio
>> +ver Transport Virtqueue / MSI Configuration}
>> +
>> +The managed device MUST disable the MSI interrupts for both virtqueues and config space
>> +upon a reset.
>> +
>> +Once a MSI entry is disabled, the managed device MUST not send any interrupts
>> +by this MSI entry.
> MSI is stored based on virtqueue and config space, so I'm not quite
> sure "MSI entry" is accurate here.
I will replace all "MSI entry" with "MSI vector"
>
>> +
>> +\drivernormative{\subsubsection}{MSI Configuration}{Virtio Transport Options / Virtio
>> +Over Transport Virtqueue / MSI Configuration}
>> +
>> +The driver MUST allocate transport or platform specific MSI entries
>> +for both virtqueues and config space if it wants to use interrupts.
>> +
>> +The driver MAY choose to disable the MSI if polling mode is used.
>> +
>> +\subsection{Virtqueue Address}\label{sec:Virtio Transport Options / Virtio Over
>> +Transport Virtqueue / Virtqueue Address}
>> +
>> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
>> +the addresses of a specific virtqueue are accessed through the following command:
>> +
>> +\begin{lstlisting}
>> +#define VIRTIO_TRANSPTQ_CTRL_VQ_ADDR    8
>> + #define VIRTIO_TRANSPTQ_CTRL_VQ_ADDR_SET       1
>> +
>> +struct virtio_transportq_ctrl_vq_addr {
>> +       u16 queue_index;
>> +       u64 descriptor_area;
>> +       u64 device_area;
>> +       u64 driver_area;
>> +       u8 padding[6];
>> +};
>> +\end{lstlisting}
>> +
>> +The VIRTIO_TRANSPTQ_CTRL_VQ_ADDR_SET command is used to set the addresses of a specified
>> +virtqueue. The command-out-data contains the \field{queue_index}, the addresses of \field{device_area},
>> +\field{descriptor_area} and \field{driver_area} (as described in struct
>> +virtio_transportq_ctrl_vq_addr). There's no command-in-data.
>> +
>> +\devicenormative{\subsubsection}{Virtqueue Address}{Virtio Transport Options / Virtio
>> +Over Transport Virtqueue / Virtqueeu Address}
>> +
>> +The management device MUST fail the commands of class
>> +VIRTIO_TRANSPTQ_CTRL_VQ_ADDR if \field{device_id} is 0.
>> +
>> +The management device MUST fail the commands of class
>> +VIRTIO_TRANSPTQ_CTRL_VQ_ADDR if \field{queue_index} is out-of-range invalid.
>> +
>> +\subsection{Virtqueue Status}\label{sec:Virtio Transport Options / Virtio Over
>> +Transport Virtqueue / Virtqueue Status}
>> +
>> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
>> +virtqueue status is accessed through the following command:
>> +
>> +\begin{lstlisting}
>> +#define VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE    9
>> + #define VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_GET       0
>> + #define VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_SET       1
>> +
>> +struct virtio_transportq_ctrl_vq_status_set {
>> +       u16 queue_index;
>> +       u8 status;
>> +       u8 padding[5];
>> +
>> +#define VIRTIO_TRANSPTQ_VQ_ENABLE     1
>> +#define VIRTIO_TRANSPTQ_VQ_DISABLE    0
>> +
>> +};
>> +
>> +\end{lstlisting}
>> +
>> +The VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_GET is used to get the status of a
>> +specific virtqueue. The command-out-data is the queue
>> +index. The command-in-data is the virtqueue status.
>> +
>> +The VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_SET command is used to set the status of a
>> +specific virtqueue. The command-out-data is the \field{queue_index} and the
>> +\field{status} representing ENABLE or DISABLE that is set to the virtqueue
>> +(as described in struct virtio_transportq_ctrl_vq_status_set).
>> +There's no command-in-data.
>> +
>> +\devicenormative{\subsubsection}{Virtqueue Status}{Virtio Transport Options / Virtio
>> +Over Transport Virtqueue / Virtqueue Status}
>> +
>> +When disabled, the managed device MUST stop processing requests from
>> +this virtqueue.
> Is this a hint that the driver can write VIRTIO_TRANSPTQ_VQ_DISABLE to
> status? If yes, what's the difference between this and virtqueue
> reset?
I think disabling a queue is like "set queue_enable = 0", once disabled,
the device should not reset the queue configurations like the indexes 
and addresses,
it just freeze the queue.

When reset a queue, like the spec says, the device needs to reset all queue
configs.
>
>> +
>> +The management device MUST present a 0 via
>> +VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_GET upon a reset of the managed device.
>> +
>> +The management device MUST fail the virtqueue status access if
>> +\field{device_id} is 0.
>> +
>> +The management device MUST fail the virtqueue status access if
>> +the queue_index is out-of-range invalid.
>> +
>> +
>> +\drivernormative{\subsubsection}{Virtqueue Status}{Virtio Transport Options / Virtio
>> +Over Transport Virtqueue / Virtqueue Status}
>> +
>> +The driver MUST configure other virtqueue fields before enabling
>> +the virtqueue with VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_SET.
>> +
>> +\subsection{Virtqueue Size}\label{sec:Virtio Transport Options / Virtio Over
>> +Transport Virtqueue / Virtqueue Size}
>> +
>> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
>> +virtqueue size is accessed through the following command:
>> +
>> +\begin{lstlisting}
>> +#define VIRTIO_TRANSPTQ_CTRL_VQ_SIZE    10
>> + #define VIRTIO_TRANSPTQ_CTRL_VQ_SIZE_GET       0
>> + #define VIRTIO_TRANSPTQ_CTRL_VQ_SIZE_SET       1
>> +
>> +struct virtio_transportq_ctrl_vq_size_set {
>> +       u16 queue_index;
>> +       u16 size;
>> +       u8 padding[4];
>> +};
>> +\end{lstlisting}
>> +
>> +The VIRTIO_TRANSPTQ_CTRL_VQ_SIZE_GET command is used to get the virtqueue
>> +size. On reset, the maximum queue size supported by the device is
>> +returned. The command-out-data is the queue index. The
>> +command-in-data is an 16-bit queue size.
>> +
>> +The VIRTIO_TRANSPTQ_CTRL_VQ_SIZE_SET command is used to set the virtqueue
>> +size. The command-out-data is the \field{queue_index} and the \field{size}
>> +of the virtqueue (as described in struct virtio_transportq_ctrl_vq_size_set).
>> +There's no command-in-data.
>> +
>> +\devicenormative{\subsubsection}{Virtqueue Status}{Virtio Transport Options / Virtio
>> +Over Transport Virtqueue / Virtqueue Size}
>> +
>> +The management device MUST fail the virtqueue size access if
>> +\field{device_id} is 0.
>> +
>> +The management device MUST fail the virtqueue size access if
>> +the queue index is out-of-range invalid.
>> +
>> +\subsection{Virtqueue Notification}\label{sec:Virtio Transport Options / Virtio
>> +Over Transport Virtqueue / Virtqueue Notification}
>> +
>> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
>> +the virtqueue notification area information can be get through the following commands:
>> +
>> +\begin{lstlisting}
>> +#define VIRTIO_TRANSPTQ_CTRL_VQ_NOTIFY    11
>> + #define VIRTIO_TRANSPTQ_CTRL_VQ_NOTIFY_GET          0
>> +
>> +struct virtio_transportq_ctrl_vq_notification_area {
>> +       u64 address;
>> +       u64 size;
>> +};
>> +\end{lstlisting}
>> +
>> +The VIRTIO_TRANSPTQ_CTRL_VQ_NOTIFY_GET is used to get the transport
>> +specific address area that can be used to notify a virtqueue. The
>> +command-out-data is an u16 of the queue index. The command-in-data
>> +contains the \field{address} and the \field{size}
>> +of the notification area (as described in struct virtio_transportq_ctrl_vq_notification_area).
>> +
>> +\devicenormative{\subsubsection}{Virtqueue Notification}{Virtio Transport Options /
>> +Virtio Over Transport Virtqueue / Virtqueue Notification}
>> +
>> +The management device MUST fail the virtqueue notification area information
>> +access if \field{device_id} is 0.
>> +
>> +The management device MUST fail the virtqueue notification area information
>> +access if the queue index is out-of-range invalid.
> We probably need to say the management device must reserve sufficient
> transport specific resources as notification area or we need to have a
> fallback of using a dedicated command to notify.
OK, will add this
>
>> +
>> +\drivernormative{\subsubsection}{Virtqueue Notification}{Virtio Transport Options /
>> +Virtio Over Transport Virtqueue / Virtqueue Notification}
>> +
>> +The driver MAY choose to notify the virtqueue by writing the queue
>> +index at address \field{address} which is fetched from the
>> +VIRTIO_TRANSPTQ_CTRL_VQ_NOTIFY_GET command.
>> +
>> +\subsection{Virtqueue State}\label{sec:Virtio Transport Options / Virtio
>> +Over Transport Virtqueue / Virtqueue State}
>> +
>> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
>> +the virtqueue state is accessed through the following commands:
>> +
>> +\begin{lstlisting}
>> +#define VIRTIO_TRANSPTQ_CTRL_VQ_STATE            12
>> + #define VIRTIO_TRANSPTQ_CTRL_VQ_STATE_GET       0
>> + #define VIRTIO_TRANSPTQ_CTRL_VQ_STATE_SET       1
>> +\end{lstlisting}
>> +
>> +\subsubsection{Split Virtqueue State}\label{sec:Virtio Transport Options / Virtio
>> +Over Transport Virtqueue / Virtqueue State / Split Virtqueue State}
>> +
>> +\begin{lstlisting}
>> +struct virtio_transportq_ctrl_split_vq_state_set {
>> +    u16 queue_index;
>> +    u16 avail_index;
>> +    u8 padding[4];
>> +};
>> +\end{lstlisting}
>> +
>> +The VIRTIO_TRANSPTQ_CTRL_VQ_STATE_GET command can be  used to get the state of a
>> +split virtqueue. The command-out-data is the queue index, the command-in-data is the available index of the virtqueue.
> We need to be accurate for "available index". E.g we have one for in
> the shared memory, so do you mean last_avail_idx?
actually there is no definition of "last available index" in the spec, 
so how about we say:
on-device available index?
>
>> +
>> +The VIRTIO_TRANSPTQ_CTRL_VQ_STATE_SET command can be used to set the state of a
>> +split virtqueue. The command-out-data contains the \field{queue_index} and the
>> +available index (\field{avail_index}) for the virtqueue (as described in struct virtio_transportq_ctrl_split_vq_state_set).
>> +There is no command-in-data.
>> +
>> +\subsubsection{Packed Virtqueue State}\label{sec:Virtio Transport Options / Virtio
>> +Over Transport Virtqueue / Virtqueue State / Packed Virtqueue State}
>> +
>> +\begin{lstlisting}
>> +struct virtio_transportq_ctrl_packed_vq_state {
>> +    u16 queue_index;
>> +    u16 avail_counter;
>> +    u16 avail_index;
>> +    u16 used_counter;
>> +    u16 used_index;
> Let's simply reuse the structure in the kernel headers.
>
> Using u16 for wrap_couter seems unnecessary etc.
OK, the bit operations are not often.
>
>> +    u8 padding[6];
>> +};
>> +\end{lstlisting}
>> +
>> +The state of a packed virtqueue includes :\\
>> +\field{avail_counter}: last driver ring wrap counter observed by device.\\
>> +\field{avail_index}: virtqueue available index.\\
>> +\field{used_counter}: device ring wrap counter.\\
>> +\field{used_index}: virtqueue used index.
>> +
>> +The VIRTIO_TRANSPTQ_CTRL_VQ_STATE_GET command can be used to get the state of a packed virtqueue.
>> +The command-out-data is the queue index, the command-in-data contains the \field{queue_index},
>> +\field{avail_counter}, \field{avail_index}, \field{used_counter} and \field{used_index} of the virtqueue
>> +(as described in transportq_ctrl_packed_vq_state).
>> +
>> +The VIRTIO_TRANSPTQ_CTRL_VQ_STATE_SET command can be used to set the state of a packed virtqueue.
>> +The command-out-data contains the \field{queue_index}, \field{avail_counter}, \field{avail_index},
>> +\field{used_counter} and \field{used_index} for the virtqueue
>> +(as described in transportq_ctrl_packed_vq_state).
>> +There is no command-in-data.
>> +
>> +\devicenormative{\subsubsection}{Virtqueue State}{Virtio Transport Options /
>> +Virtio Over Transport Virtqueue / Virtqueue State}
>> +
>> +The management device MUST fail the virtqueue index access if \field{device_id} is 0.
>> +
>> +The management device MUST fail the virtqueue index access if \field{queue_index} is out-of-range invalid.
>> +
>> +\subsection{Virtqueue ASID}\label{sec:Virtio Transport Options / Virtio Over
>> +Transport Virtqueue / Virtqueue ASID}
>> +
>> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
>> +the address space id of a virtqueue could be set through the following command:
>> +
>> +\begin{lstlisting}
>> +#define VIRTIO_TRANSPTQ_CTRL_VQ_ASID        13
>> + #define VIRTIO_TRANSPTQ_CTRL_VQ_ASID_SET   1
>> +
>> +struct virtio_transportq_ctrl_vq_asid_set {
>> +       u16 queue_index;
>> +       u32 asid;
>> +       u8 padding[2];
> I'd reserve more here. E.g in the future each virtqueue may have a
> secondary ASID,
like asid[2]? But why? For a device, there may needs a secondary ASID 
because it
may need to access different memory region on behalf of different DMA 
sources.
But for a virtqueue, it only has one purpose, to access the queue 
buffer, maybe it
doesn't need a secondary asid
>
>> +};
>> +\end{lstlisting}
>> +
>> +The VIRTIO_TRANSPTQ_CTRL_VQ_ASID_SET command is used to set the address space id(\field{asid})
>> +of a virtqueue.
>> +
>> +The address space id is an identifier
> Let's add "transport specific" before "identifier"
OK
>
>> of a memory space which is used to convey the address space targeted by the
>> +memory accesses
> We need to say the ASID is for DMA not here at least.
I don't get it, we need to explain what ASID is used for after
we introduce it.
>
>> , and to distinguish memory accesses performed by different virtqueues
> This may confuse the readers since a single ASID could be shared by virtqueues.
so it should be "by different virtqueues or virtqueue groups"
>
>> +
>> +One example of the address space id is PASID (Process Address Space Identifier) which is
>> +defined in \hyperref[intro:PCIe]{[PCIe]} specification.
>> +
>> +The command-out-data of VIRTIO_TRANSPTQ_CTRL_VQ_ASID_SET command is the \field{queue_index}
>> +and the address space id (\field{asid}).
>> +(as described in struct virtio_transportq_ctrl_vq_asid_set).
>> +
>> +\devicenormative{\subsubsection}{Virtqueue ASID}{Virtio Transport Options /
>> +Virtio Over Transport Virtqueue / Virtqueue ASID}
>> +
>> +Once a virtqueue has been set an asid, it MUST perform any memory accesses with the asid.
> Let's limit "memory accesses" to "DMA". And we need to say, this is
> true only if the transport specific feature is enabled e.g via the PCI
> PF PASID extended capability.
OK, I will use "DMA memory access" in the next version. I am not sure 
whether we should
introduce another transport as a dependency. I will add a device 
requirement: the HW implementation should fail this
command if there is no ASID support
>
>> +
>> +The management device MUST fail the virtqueue index access if \field{device_id} is 0.
>> +
>> +The management device MUST fail the virtqueue index access if \field{queue_index} is out-of-range invalid.
> What if the ASID is out of the range of PASID?
I think we can just call kernel API to allocate an ASID, I remember it 
is ioasid_alloc(), and we need to specify the max asid and min asid there.
For a vendor driver, it knows the transport of management device, so it 
can specify a correct max_asid.
>
>> +
>> +\subsection{Virtqueue Reset}\label{sec:Virtio Transport Options / Virtio Over
>> +Transport Virtqueue / Virtqueue Reset}
>> +
>> +When VIRTIO_F_TRANSPT_VQ
> And reset is negotiated.
OK
>
>> is negotiated with the management device,
>> +a virtqueue can be reset through the following command:
>> +
>> +\begin{lstlisting}
>> +#define VIRTIO_TRANSPTQ_CTRL_VQ_RESET        14
>> + #define VIRTIO_TRANSPTQ_CTRL_VQ_DO_RESET    1
> RESET_RESET should be fine.
OK
>
>> +\end{lstlisting}
>> +
>> +The VIRTIO_TRANSPTQ_CTRL_VQ_DO_RESET command is used to reset a virtqueue.
>> +The command-out-data is the virtqueue index, there is no command-in-data.
>> +
>> +\devicenormative{\subsubsection}{Virtqueue Reset}{Virtio Transport Options /
>> +Virtio Over Transport Virtqueue / Virtqueue Reset}
>> +
>> +The management device MUST fail VIRTIO_TRANSPTQ_CTRL_VQ_RESET if
>> +\field{device_id} is 0.
>> +
>> +The management device MUST fail VIRTIO_TRANSPTQ_CTRL_VQ_RESET if
>> +the virtqueue index is out-of-range  invalid.
>> +
>> +The managed device MUST stop consuming the descriptors in the virtqueue once reset it.
>> +
>> +The managed device MUST present default states of a virtqueue after reset it.
>> +
> We need to "dup" the normatives for PCI queue reset here or consider a
> place to unify.
sure

Thanks for review
>
> Thanks
>
>> +\drivernormative{\subsubsection}{Virtqueue Reset}{Virtio Transport Options /
>> +Virtio Over Transport Virtqueue / Virtqueue Reset}
>> +
>> +After reseting a virtqueue, the driver MUST wait until the reset is successfully
>> +done (by verifying virtio_transportq_ctrl.ack == VIRTIO_TRANSPTQ_OK)
>> +before re-enabling it.
>>
>>   \chapter{Device Types}\label{sec:Device Types}
>>
>> --
>> 2.35.3
>>
>>
>> This publicly archived list offers a means to provide input to the
>> OASIS Virtual I/O Device (VIRTIO) TC.
>>
>> In order to verify user consent to the Feedback License terms and
>> to minimize spam in the list archive, subscription is required
>> before posting.
>>
>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>> List help: virtio-comment-help@lists.oasis-open.org
>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>> Committee: https://www.oasis-open.org/committees/virtio/
>> Join OASIS: https://www.oasis-open.org/join/
>>
>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>



This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] [PATCH V3 RESEND 2/4] Introduce the commands set of the transport vq

[Jason Wang]

On Tue, Aug 9, 2022 at 9:09 PM Zhu, Lingshan <lingshan.zhu@intel.com> wrote:
>
>
>
> On 8/8/2022 6:04 PM, Jason Wang wrote:
> > On Fri, Aug 5, 2022 at 6:02 PM Zhu Lingshan <lingshan.zhu@intel.com> wrote:
> >> This commit introduces the commands set of the
> >> transport virtqueue, including:
> >>
> >> The command to query available resources of the management device
> >> The commands to create / destroy the managed devices.
> >> The commands to config the managed devices.
> >> The commands to config virtqueues of the managed devices.
> >>
> >> Signed-off-by: Jason Wang <jasowang@redhat.com>
> >> Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com>
> >> ---
> >>   content.tex | 636 ++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>   1 file changed, 636 insertions(+)
> >>
> >> diff --git a/content.tex b/content.tex
> >> index c747d21..190f3a0 100644
> >> --- a/content.tex
> >> +++ b/content.tex
> >> @@ -2975,6 +2975,642 @@ \subsection{Format of Commands through Transport Virtqueue}\label{sec:Virtio Tra
> >>   The \field{device_id} value 0 is used to identify the management device itself.
> >>
> >>   \field{class} is an identifier of a set of commands with similar  purposes.
> >> +
> >> +\subsection{Available Resource of the Management Device}\label{sec:Virtio Transport Options / Virtio Over Transport Virtqueue / Available Resource of the Management Device}
> >> +
> >> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
> >> +statistical information of available resources on the management device could be fetched
> >> +by the following command:
> >> +
> >> +\begin{lstlisting}
> >> +#define VIRTIO_TRANSPTQ_CTRL_AVAIL_RES    1 (class)
> >> + #define VIRTIO_TRANSPTQ_CTRL_AVAIL_RES_GET     0 (command)
> >> +\end{lstlisting}
> >> +
> >> +VIRTIO_TRANSPTQ_CTRL_AVAIL_RES_GET command is used to get the statistical information of the available
> >> +resource of a management device. There is no command-out-data for VIRTIO_TRANSPTQ_CTRL_AVAIL_RES_GET
> >> +command. The management device fills command-in-data in the following format:
> >> +
> >> +\begin{lstlisting}
> >> +struct virtio_mgmt_dev_avail_res {
> >> +        /* The number of management device total remaining available free
> >> +         * virtqueues for the managed devices
> >> +         */
> > It's hard to phrase this sentence.
> How about "The number of remaining available virtqueues for new managed
> devices"
> >
> >> +        u16 num_avail_vqs;
> >> +        /* The minimal number of virtqueues for a managed device */
> >> +        u16 min_dev_vqs;
> >> +        /* The maximum number of virtqueues for a managed device */
> > Nit the min/max_dev_vqs are not the resources, so we'd better rename
> > the virtio_mgmt_dev_avail_res.
> OK, I will use struct virito_mgmt_dev_info{} in the next version, and
> the command and its description
> will change as well.
> >
> >> +        u16 max_dev_vqs;
> >> +        /* The number of managed devices that can be created with min_vqs virtqueues */
> > Is the part "with min_vqs virtqueues" a must? My understanding of the
> > above is a hint that the managed device is loosely coupled with the
> > virtqueue resources.
> Yes, the number of the number of total managed devices can be limited by
> other resources, like the filters.
> >
> > So it would be possible that num_avail_dev = 0 but num_avail_vqs not?
> >
> > Btw, "num" prefix is probably not a must.
> OK
> >
> >> +        u16 num_avail_dev;
> > To be consistent, let's use "num_avail_devs"?
> I will remove all num_ prefix, and it should be avail_devs
> >
> >> +};
> >> +\end{lstlisting}
> >> +
> >> +\field{max_avail_dev} MAY not equal to \field{num_avail_vqs} divided by \field{min_vqs}
> >> +due to the limitations of of other resources.
> > two "of".
> will fix
> >
> > Keywords like "MAY" should go to normatives part.
> OK, so I will only use "may", so it is a normal description.
> >
> >> +
> >> +\devicenormative{\subsubsection}{Available Resource of the Management Device}{Virtio Transport Options /
> >> +Virtio Over Transport Virtqueue / Available Resource of the Management Device}
> >> +
> >> +The management device MUST fail VIRTIO_TRANSPTQ_CTRL_AVAIL_RES_GET command if \field{device_id} is not 0
> >> +
> >> +\drivernormative{\subsubsection}{Available Resource of the Management Device}{Virtio Transport Options /
> >> +Virtio Over Transport Virtqueue / Available Resource of the Management Device}
> >> +
> >> +The management device driver MUST use 0 as \field{device_id} for
> >> +VIRTIO_TRANSPTQ_CTRL_AVAIL_RES_GET command.
> >> +
> >> +\subsection{Create and Destroy Managed Devices}\label{sec:Virtio Transport Options / Virtio Over Transport Virtqueue / Create and Destroy Managed Devices}
> >> +
> >> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
> >> +managed devices must be created and destroyed through the transport virtqueue.
> >> +
> >> +\begin{lstlisting}
> >> +#define VIRTIO_TRANSPORTQ_CTRL_DEV    2
> >> + #define VIRTIO_TRANSPORTQ_CTRL_DEV_CREATE        0
> >> + #define VIRTIO_TRANSPORTQ_CTRL_DEV_DESTROY       1
> >> +
> >> +struct virtio_transportq_ctrl_dev_attribute {
> >> +       u64 device_features[2];
> > Let's reserve more bits for features here as we are about to reach 64.
> OK, it looks 256bytes may be enough.

It's better to match the parent/management device, but maybe we can
start from something that is fine for the next 5 years. So I'd rather
go with even more than this like device_feautres[10].


> >
> >> +       u32 virtio_device_id;
> >> +       u8 dev_config[];
> > To be consistent, let's use one of "device" or "dev" in the same structure.
> sure, device_config[]
> >
> >> +};
> >> +\end{lstlisting}
> >> +
> >> +The VIRTIO_TRANSPORTQ_CTRL_DEV_CREATE command is used to create a managed device.
> >> +
> >> +As described in struct virtio_transportq_ctrl_dev_attribute, the command-out-data for VIRTIO_TRANSPTQ_CTRL_DEV_CREATE includes:
> >> +\begin{itemize*}
> >> +\item \field{device_features}: the 128-bit device feature bits (defined in section \ref{sec:Basic Facilities of a Virtio Device / Feature Bits})
> >> +of the managed device.
> >> +\item \field{virtio_device_id}: the virtio device id defined in Chapter \ref{sec:Device Types}.
> >> +\item \field{dev_config}: the device type specific configurations. E.g., for a virtio-net device,
> >> +it is struct virtio_net_config in section \ref{sec:Device Types / Network Device / Device configuration layout};
> >> +for a virtio-block device, it is struct virtio_blk_config in section \ref{sec:Device Types / Block Device / Device configuration layout}
> >> +\end{itemize*}
> >> +
> >> +When succeed, the device returns a 64 bits UUID as an unique identifier of the created managed device in command-in-data.
> > We don't need to be universally unique here.
> Why? I think it should be unique to identify a device, or are you
> suggesting to remove "as an unique identifier",
> just UUID is enough?

UUID means it's unique in the universal. If it means ID for managed
device in PF1 must differ from ID for managed device in PF2. This is
not needed.

So if we want to be flexible, we can allow the ID to be allocated by
the driver as the original RFC did. The driver is free to allocate
anything it want (UID or UUID).

> >
> >> +
> >> +The VIRTIO_TRANSPORTQ_CTRL_DEV_DESTROY command is used to destroy a
> >> +managed device which is identified by its 64 bits UUID
> >> +\field{device_id}. There's no command-in-data nor command-out-data for
> >> +VIRTIO_TRANSPTQ_CTRL_DEV_DESTROY command.
> >> +
> >> +\devicenormative{\subsubsection}{Create and Destroy Managed Devices}{Virtio Transport Options / Virtio Over Transport Virtqueue / Create and Destroy Managed Devices}
> >> +
> >> +The management device MUST fail VIRTIO_TRANSPORTQ_CTRL_DEV_CREATE command
> >> +if \field{device_id} is not 0.
> >> +
> >> +The management device MUST fail VIRTIO_TRANSPORTQ_CTRL_DEV CREATE command
> >> +if \field{device_features} exceeds the features that can be provided from the management device.
> > So PCI has le32 device_feature_select. How to make sure it can match
> > the capacity of device_features[] above?
> we present all feature bits in device_features[], the spec says the
> feature bits are valid up to bit 127,
> 128 and above are reserved. So as you suggest, we can use u64
> device_features[4], for 128-bit feature bits
> and another 128 bits for further extensions.
> >
> >> +
> >> +The management device MUST fail VIRTO_TRANSPORTQ_CTRL_DEV_DESTROY command
> >> +if \field{device_id} is 0.
> >> +
> >> +\drivernormative{\subsubsection}{Create and Destroy Managed Devices}{Virtio Transport Options / Virtio Over Transport Virtqueue / Create and Destroy Managed Devices}
> >> +
> >> +The management device driver MUST use 0 as \field{device_id} for
> >> +TRANSPORTQ_CTRL_DEV_CREATE command.
> >> +
> >> +\subsection{Features Negotiation}\label{sec:Virtio Transport Options / Virtio
> >> +Over Transport Virtqueue / Features Negotiation}
> >> +
> >> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
> >> +the features negotiation of managed devices is done by the
> >> +following commands:
> >> +
> >> +\begin{lstlisting}
> >> +#define VIRTIO_TRANSPTQ_CTRL_FEAT   3
> >> + #define VIRTIO_TRANSPTQ_CTRL_FEAT_DEVICE_GET        0
> >> + #define VIRTIO_TRANSPTQ_CTRL_FEAT_DRIVER_SET        1
> >> + #define VIRTIO_TRANSPTQ_CTRL_FEAT_DRIVER_GET        2
> >> +\end{lstlisting}
> >> +
> >> +The VIRTIO_TRANSPTQ_CTRL_FEAT_DEVICE_GET command is used to get the features offered
> >> +by a managed device. The command-in-data is the 128-bit feature bits
> >> +(defined in section \ref{sec:Basic Facilities of a Virtio Device / Feature Bits}).
> > Need to make sure the capacity of the feature bit matches what the
> > transport of the management device can provide.
> I will provide a structure for the features, so both device and driver
> would be aligned on
> the size of the feature bits, like
>
> struct virtio_transportq_ctrl_dev_features {
>      u64 features[4];
> };
> >
> >> +There is no command-out-data.
> >> +
> >> +The VIRTIO_TRANSPTQ_CTRL_FEAT_DRIVER_SET command is for the driver to accept feature
> >> +bits offered by the managed device. The command-out-data is the 128-bit feature bits
> >> +passed to the managed device. There is no command-in-data.
> >> +
> >> +The VIRTIO_TRANSPTQ_CTRL_FEAT_DRIVER_GET command is used to get the features accepted
> >> +by both the managed device driver and the managed device. The command-in-data is the
> >> +128-bit feature bits. There is no command-out-data.
> >> +
> >> +\devicenormative{\subsubsection}{Features Negotiation}{Virtio Transport Options /
> >> +Virtio Over Transport Virtqueue / Features Negotiation}
> >> +
> >> +The management device MUST fail VIRTIO_TRANSPTQ_F_CTRL_FEAT class commands if
> >> +\field{device_id} is 0.
> >> +
> >> +\subsection{Device Status}\label{sec:Virtio Transport Options / Virtio Over
> >> +Transport Virtqueue / Device Status}
> >> +
> >> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
> >> +the device status of a managed device can be accessed by the following
> >> +commands:
> >> +
> >> +\begin{lstlisting}
> >> +#define VIRTIO_TRANSPTQ_CTRL_STATUS    4
> >> + #define VIRTIO_TRANSPTQ_CTRL_STATUS_GET        0
> >> + #define VIRTIO_TRANSPTQ_CTRL_STATUS_SET        1
> >> +\end{lstlisting}
> >> +
> >> +The VIRTIO_TRANSPTQ_CTRL_STATUS_GET command is used to get the device status of
> >> +a managed device. The command-in-data is the 8-bit status
> >> +returned from the device. There's no command-out-data for this
> >> +command.
> >> +
> >> +The VIRTIO_TRANSPTQ_CTRL_STATUS_SET command is used to set the device status of
> >> +a managed device. The command-out-data is the 8-bit status
> >> +to set to the device. There's no command-in-data for this command.
> >> +
> >> +\devicenormative{\subsubsection}{Device Status}{Virtio Transport Options / Virtio
> >> +Over Transport Virtqueue / Device Status}
> >> +
> >> +The management device MUST reset the managed device when 0
> >> +is set via VIRTIO_TRANSPTQ_CTRL_STATUS_SET, the success of this
> >> +command demonstrates the success of the reset.
> >> +
> >> +The management device MUST present 0 through
> >> +VIRTIO_TRANSPTQ_CTRL_STATUS_GET once the reset is successfully done.
> >> +
> >> +The management device MUST fail the device status access if
> >> +\field{device_id} is 0.
> >> +
> >> +\drivernormative{\subsubsection}{Device Status}{Virtio Transport Options / Virtio
> >> +Over Transport Virtqueue / Device Status}
> >> +
> >> +After writing 0 via VIRTIO_TRANSPTQ_CTRL_STATUS_SET, the driver MUST wait
> >> +for the success of the command before re-initializing the device.
> >> +
> >> +\subsection{Device Generation}\label{sec:Virtio Transport Options / Virtio Over Transport Virtqueue / Device Generation}
> >> +
> >> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
> > Btw, since we're under the transport options (which implies the
> > transport virtqueue), maybe we can drop the above for each command.
> OK, can do, and I will describe this in the Basic Concept section.
> >
> >> +the device generation count can be read from the following commands:
> >> +
> >> +\begin{lstlisting}
> >> +#define VIRTIO_TRANSPTQ_CTRL_GENERATION    5
> >> + #define VIRTIO_TRANSPTQ_CTRL_GENERATION_GET        0
> >> +\end{lstlisting}
> >> +
> >> +The VIRTIO_TRANSPTQ_CTRL_GENERATION_GET command is used to get the device configuration generation count
> >> +of the managed device. The command-in-data is the one byte device
> >> +generation returned from the device. There's no command-out-data for
> >> +this command.
> >> +
> >> +\devicenormative{\subsubsection}{Device Generation}{Virtio Transport Options / Virtio Over Transport Virtqueue / Device Generation}
> >> +
> >> +The managed device MUST present a changed generation count after the driver
> >> +has read any device-specific configuration values if the values have been changed
> >> +during the last read.
> >> +
> >> +The management device MUST fail the device generation access if \field{device_id} is 0.
> >> +
> >> +\subsection{Device Specific Configuration}\label{sec:Virtio Transport Options /
> >> +Virtio Over Transport Virtqueue / Device Specific Configuration}
> >> +
> >> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
> >> +the device config space contents of a managed device can be accessed through
> >> +the following commands:
> >> +
> >> +\begin{lstlisting}
> >> +#define VIRTIO_TRANSPTQ_CTRL_CONFIG    6
> >> +  #define VIRTIO_TRANSPTQ_CTRL_CONFIG_GET    0
> >> +  #define VIRTIO_TRANSPTQ_CTRL_CONFIG_SET    1
> >> +
> >> +struct virtio_transportq_ctrl_dev_config_get {
> >> +       u32 offset;
> >> +       u32 size;
> >> +};
> >> +
> >> +struct virtio_transportq_ctrl_dev_config_set {
> >> +       u32 offset;
> >> +       u32 size;
> >> +       u8  data[];
> >> +};
> >> +\end{lstlisting}
> >> +
> >> +The VIRTIO_TRANSPTQ_CTRL_CONFIG_GET command is used to read data from the
> >> +device configuration space.  The command-out-data is the \field{offset}
> >> +from the start of the config space and the \field{size}
> >> +of the data (as described in struct virtio_transportq_ctrl_dev_config_get).
> >> +The command-in-data is an array of the data that read from the config space.
> >> +
> >> +The VIRTIO_TRANSPTQ_CTRL_CONFIG_SET command is used to write data to the device
> >> +configuration space. The command-out-data contains the
> >> +\field{offset} from the start of the config space, the \field{size} of the data and
> >> +the \field{data} to be written (as described in struct virtio_transportq_ctrl_dev_config_set).
> >> +There's no command-in-data for this command.
> >> +
> >> +\devicenormative{\subsubsection}{Device Specific Configuration}{Virtio Transport
> >> +Options / Virtio Over Transport Virtqueue / Device Specific Configuration}
> >> +
> >> +The management device MUST fail the device configuration space access
> >> +if the driver access the range which is outside the config space.
> > "accesses"
> yes
> >
> >> +
> >> +The management device MUST fail the device configuration space access
> >> +if \field{device_id} is 0.
> >> +
> >> +\subsection{MSI Configuration}\label{sec:Virtio Transport Options / Virtio Over
> >> +Transport Virtqueue / MSI Configuration}
> >> +
> >> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
> >> +the MSI entries of a managed device can be accessed  through the following command:
> > Rethink of this, it implies a per virtqueue MSI storage which I'm not
> > sure is expensive or not. Do we need an indirection layer as PCI did?
> I think we need this for performance, or are you suggesting to add a
> device scope MSI?

I meant having MSI vectors and let the virtqueue refer to the MSI vectors.

Current design means for a 1024 virtqueues device to store 1024 MSI entries.
With the indirection of the MSI vectors array, the device is free to
have 1 to 1024 MSI entries.

> >
> >> +
> >> +\begin{lstlisting}
> >> +#define VIRTIO_TRANSPTQ_CTRL_MSI    7
> >> + #define VIRTIO_TRANSPTQ_CTRL_MSI_VQ_GET        0
> >> + #define VIRTIO_TRANSPTQ_CTRL_MSI_VQ_SET        1
> >> + #define VIRTIO_TRANSPTQ_CTRL_MSI_VQ_ENABLE     2
> >> + #define VIRTIO_TRANSPTQ_CTRL_MSI_CONFIG_GET    3
> >> + #define VIRTIO_TRANSPTQ_CTRL_MSI_CONFIG_WET    4
> > I think you mean CONFIG_SET here.
> yes
> >
> >> + #define VIRTIO_TRANSPTQ_CTRL_MSI_CONFIG_ENABLE 5
> >> +
> >> +struct virtio_transportq_ctrl_msi_vq_config {
> >> +       u16 queue_index;
> >> +       u64 address;
> >> +       u32 data;
> >> +       u8 padding[2];
> >> +};
> >> +
> >> +struct virtio_transportq_ctrl_msi_vq_enable {
> >> +       u16 queue_index;
> >> +       u8 enable;
> >> +       u8 padding[5];
> >> +};
> >> +
> >> +struct virtio_transportq_ctrl_msi_config {
> >> +       u64 address;
> >> +       u32 data;
> >> +       u8 padding[4];
> >> +};
> >> +\end{lstlisting}
> >> +
> >> +The VIRTIO_TRANSPTQ_CTRL_MSI_VQ_GET command is used to read the MSI entry of a
> >> +specific virtqueue. The command-out-data is the queue index. The command-in-data contains
> >> +\field{queue_index}, the \field{address} and \field{data}
> >> +(as described in struct virtio_transportq_ctrl_msi_vq_config).
> > Let's split or reuse the virtio_transportq_ctrl_msi_config since
> > virtio_transportq_ctrl_msi_vq_config will not be used a as single
> > buffer.
> I don't get it, if we re-use the struct virtio_transportq_ctrl_msi_config,
> it would be like:
>
> struct virtio_transportq_ctrl_msi {
>      u64 addr;
>      u32 data;
> };
>
> struct virtio_transportq_ctrl_msi_vq_config {
>      u16 queue_index;

What I meant is, for GET. queue_index is out buffer, and re-use
virtio_transportq_ctrl_msi is in buffer.

>      struct virtio_transportq_ctrl_msi;
> };
>
> it is still two structs and a struct in another struct.
> >
> >> +
> >> +
> >> +The VIRTIO_TRANSPTQ_CTRL_MSI_VQ_SET command is used to set the MSI entry for a
> >> +specific virtqueue. The command-out-data is the \field{queue_index},
> >> +\field{address} and \field{data} (as described in struct virtio_transportq_ctrl_msi_vq_config).
> >> +There is no command-in-data.
> >> +
> >> +The VIRTIO_TRANSPTQ_CTRL_MSI_VQ_ENABLE command is used to enable or disable
> >> +the MSI interrupt for a specific virtqueue. The command-out-data is the
> >> +\field{queue_index} and a byte \field{enable} which representing ENABLE or DISABLE the MSI.
> > Should be "represents".
> OK
> >
> > Btw, is there a chance that we may lose an interrupt?
> No, I think we won't lose any interrupts, because once disabled, it can
> not send new interrupts,
> but the last interrupt send by the MSI entry is still there, it's in-band.

So consider driver is changing affinity what it did is:

disable
write MSI
enable

What if there's an interrupt coming in the middle? Does this mean the
device needs to implement the pending bit internally?

FYI we had similar discussion in the past:

https://lists.oasis-open.org/archives/virtio-dev/202002/msg00027.html

> >
> >> +(as described in struct virtio_transportq_ctrl_msi_vq_enable).
> >> +
> >> +\begin{lstlisting}
> >> +#define VIRTIO_TRANSPTQ_CTRL_MSI_VQ_ENABLE     1
> >> +#define VIRTIO_TRANSPTQ_CTRL_MSI_VQ_DISABLE    0
> >> +\end{lstlisting}
> >> +
> >> +The VIRTIO_TRANSPTQ_CTRL_MSI_CONFIG_GET command is used to read the MSI entry
> >> +of the config interrupt. The command-in-data contains the \field{address} and
> >> +\field{data} (as described in struct virtio_transportq_ctrl_msi_config).
> >> +There is no command-out-data.
> >> +
> >> +The VIRTIO_TRANSPTQ_CTRL_MSI_CONFIG_SET command is used to set the MSI entry
> >> +for the config interrupt. The command-out-data is the \field{address} and
> >> +\field{data} (as described in struct virtio_transportq_ctrl_msi_config).
> >> +There is no command-in-data.
> >> +
> >> +The VIRTIO_TRANSPTQ_CTRL_MSI_CONFIG_ENABLE command is used to enable or disable
> >> +the MSI interrupt for config space. The command-out-data is a byte which representing
> >> +ENABLE or DISABLE the config MSI.
> >> +There is no command-in-data
> >> +
> >> +\begin{lstlisting}
> >> +#define VIRTIO_TRANSPTQ_CTRL_MSI_CFG_ENABLE     1
> >> +#define VIRTIO_TRANSPTQ_CTRL_MSI_CFG_DISABLE    0
> >> +\end{lstlisting}
> >> +
> >> +\devicenormative{\subsubsection}{MSI Configuration}{Virtio Transport Options / Virtio
> >> +ver Transport Virtqueue / MSI Configuration}
> >> +
> >> +The managed device MUST disable the MSI interrupts for both virtqueues and config space
> >> +upon a reset.
> >> +
> >> +Once a MSI entry is disabled, the managed device MUST not send any interrupts
> >> +by this MSI entry.
> > MSI is stored based on virtqueue and config space, so I'm not quite
> > sure "MSI entry" is accurate here.
> I will replace all "MSI entry" with "MSI vector"
> >
> >> +
> >> +\drivernormative{\subsubsection}{MSI Configuration}{Virtio Transport Options / Virtio
> >> +Over Transport Virtqueue / MSI Configuration}
> >> +
> >> +The driver MUST allocate transport or platform specific MSI entries
> >> +for both virtqueues and config space if it wants to use interrupts.
> >> +
> >> +The driver MAY choose to disable the MSI if polling mode is used.
> >> +
> >> +\subsection{Virtqueue Address}\label{sec:Virtio Transport Options / Virtio Over
> >> +Transport Virtqueue / Virtqueue Address}
> >> +
> >> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
> >> +the addresses of a specific virtqueue are accessed through the following command:
> >> +
> >> +\begin{lstlisting}
> >> +#define VIRTIO_TRANSPTQ_CTRL_VQ_ADDR    8
> >> + #define VIRTIO_TRANSPTQ_CTRL_VQ_ADDR_SET       1
> >> +
> >> +struct virtio_transportq_ctrl_vq_addr {
> >> +       u16 queue_index;
> >> +       u64 descriptor_area;
> >> +       u64 device_area;
> >> +       u64 driver_area;
> >> +       u8 padding[6];
> >> +};
> >> +\end{lstlisting}
> >> +
> >> +The VIRTIO_TRANSPTQ_CTRL_VQ_ADDR_SET command is used to set the addresses of a specified
> >> +virtqueue. The command-out-data contains the \field{queue_index}, the addresses of \field{device_area},
> >> +\field{descriptor_area} and \field{driver_area} (as described in struct
> >> +virtio_transportq_ctrl_vq_addr). There's no command-in-data.
> >> +
> >> +\devicenormative{\subsubsection}{Virtqueue Address}{Virtio Transport Options / Virtio
> >> +Over Transport Virtqueue / Virtqueeu Address}
> >> +
> >> +The management device MUST fail the commands of class
> >> +VIRTIO_TRANSPTQ_CTRL_VQ_ADDR if \field{device_id} is 0.
> >> +
> >> +The management device MUST fail the commands of class
> >> +VIRTIO_TRANSPTQ_CTRL_VQ_ADDR if \field{queue_index} is out-of-range invalid.
> >> +
> >> +\subsection{Virtqueue Status}\label{sec:Virtio Transport Options / Virtio Over
> >> +Transport Virtqueue / Virtqueue Status}
> >> +
> >> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
> >> +virtqueue status is accessed through the following command:
> >> +
> >> +\begin{lstlisting}
> >> +#define VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE    9
> >> + #define VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_GET       0
> >> + #define VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_SET       1
> >> +
> >> +struct virtio_transportq_ctrl_vq_status_set {
> >> +       u16 queue_index;
> >> +       u8 status;
> >> +       u8 padding[5];
> >> +
> >> +#define VIRTIO_TRANSPTQ_VQ_ENABLE     1
> >> +#define VIRTIO_TRANSPTQ_VQ_DISABLE    0
> >> +
> >> +};
> >> +
> >> +\end{lstlisting}
> >> +
> >> +The VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_GET is used to get the status of a
> >> +specific virtqueue. The command-out-data is the queue
> >> +index. The command-in-data is the virtqueue status.
> >> +
> >> +The VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_SET command is used to set the status of a
> >> +specific virtqueue. The command-out-data is the \field{queue_index} and the
> >> +\field{status} representing ENABLE or DISABLE that is set to the virtqueue
> >> +(as described in struct virtio_transportq_ctrl_vq_status_set).
> >> +There's no command-in-data.
> >> +
> >> +\devicenormative{\subsubsection}{Virtqueue Status}{Virtio Transport Options / Virtio
> >> +Over Transport Virtqueue / Virtqueue Status}
> >> +
> >> +When disabled, the managed device MUST stop processing requests from
> >> +this virtqueue.
> > Is this a hint that the driver can write VIRTIO_TRANSPTQ_VQ_DISABLE to
> > status? If yes, what's the difference between this and virtqueue
> > reset?
> I think disabling a queue is like "set queue_enable = 0", once disabled,
> the device should not reset the queue configurations like the indexes
> and addresses,
> it just freeze the queue.

Then you need to define what "freeze" means. To reduce the changset,
I'd suggest disallowing such writing as what PCI did.

If needed, we can re-enable it in the future.

>
> When reset a queue, like the spec says, the device needs to reset all queue
> configs.
> >
> >> +
> >> +The management device MUST present a 0 via
> >> +VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_GET upon a reset of the managed device.
> >> +
> >> +The management device MUST fail the virtqueue status access if
> >> +\field{device_id} is 0.
> >> +
> >> +The management device MUST fail the virtqueue status access if
> >> +the queue_index is out-of-range invalid.
> >> +
> >> +
> >> +\drivernormative{\subsubsection}{Virtqueue Status}{Virtio Transport Options / Virtio
> >> +Over Transport Virtqueue / Virtqueue Status}
> >> +
> >> +The driver MUST configure other virtqueue fields before enabling
> >> +the virtqueue with VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_SET.
> >> +
> >> +\subsection{Virtqueue Size}\label{sec:Virtio Transport Options / Virtio Over
> >> +Transport Virtqueue / Virtqueue Size}
> >> +
> >> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
> >> +virtqueue size is accessed through the following command:
> >> +
> >> +\begin{lstlisting}
> >> +#define VIRTIO_TRANSPTQ_CTRL_VQ_SIZE    10
> >> + #define VIRTIO_TRANSPTQ_CTRL_VQ_SIZE_GET       0
> >> + #define VIRTIO_TRANSPTQ_CTRL_VQ_SIZE_SET       1
> >> +
> >> +struct virtio_transportq_ctrl_vq_size_set {
> >> +       u16 queue_index;
> >> +       u16 size;
> >> +       u8 padding[4];
> >> +};
> >> +\end{lstlisting}
> >> +
> >> +The VIRTIO_TRANSPTQ_CTRL_VQ_SIZE_GET command is used to get the virtqueue
> >> +size. On reset, the maximum queue size supported by the device is
> >> +returned. The command-out-data is the queue index. The
> >> +command-in-data is an 16-bit queue size.
> >> +
> >> +The VIRTIO_TRANSPTQ_CTRL_VQ_SIZE_SET command is used to set the virtqueue
> >> +size. The command-out-data is the \field{queue_index} and the \field{size}
> >> +of the virtqueue (as described in struct virtio_transportq_ctrl_vq_size_set).
> >> +There's no command-in-data.
> >> +
> >> +\devicenormative{\subsubsection}{Virtqueue Status}{Virtio Transport Options / Virtio
> >> +Over Transport Virtqueue / Virtqueue Size}
> >> +
> >> +The management device MUST fail the virtqueue size access if
> >> +\field{device_id} is 0.
> >> +
> >> +The management device MUST fail the virtqueue size access if
> >> +the queue index is out-of-range invalid.
> >> +
> >> +\subsection{Virtqueue Notification}\label{sec:Virtio Transport Options / Virtio
> >> +Over Transport Virtqueue / Virtqueue Notification}
> >> +
> >> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
> >> +the virtqueue notification area information can be get through the following commands:
> >> +
> >> +\begin{lstlisting}
> >> +#define VIRTIO_TRANSPTQ_CTRL_VQ_NOTIFY    11
> >> + #define VIRTIO_TRANSPTQ_CTRL_VQ_NOTIFY_GET          0
> >> +
> >> +struct virtio_transportq_ctrl_vq_notification_area {
> >> +       u64 address;
> >> +       u64 size;
> >> +};
> >> +\end{lstlisting}
> >> +
> >> +The VIRTIO_TRANSPTQ_CTRL_VQ_NOTIFY_GET is used to get the transport
> >> +specific address area that can be used to notify a virtqueue. The
> >> +command-out-data is an u16 of the queue index. The command-in-data
> >> +contains the \field{address} and the \field{size}
> >> +of the notification area (as described in struct virtio_transportq_ctrl_vq_notification_area).
> >> +
> >> +\devicenormative{\subsubsection}{Virtqueue Notification}{Virtio Transport Options /
> >> +Virtio Over Transport Virtqueue / Virtqueue Notification}
> >> +
> >> +The management device MUST fail the virtqueue notification area information
> >> +access if \field{device_id} is 0.
> >> +
> >> +The management device MUST fail the virtqueue notification area information
> >> +access if the queue index is out-of-range invalid.
> > We probably need to say the management device must reserve sufficient
> > transport specific resources as notification area or we need to have a
> > fallback of using a dedicated command to notify.
> OK, will add this
> >
> >> +
> >> +\drivernormative{\subsubsection}{Virtqueue Notification}{Virtio Transport Options /
> >> +Virtio Over Transport Virtqueue / Virtqueue Notification}
> >> +
> >> +The driver MAY choose to notify the virtqueue by writing the queue
> >> +index at address \field{address} which is fetched from the
> >> +VIRTIO_TRANSPTQ_CTRL_VQ_NOTIFY_GET command.
> >> +
> >> +\subsection{Virtqueue State}\label{sec:Virtio Transport Options / Virtio
> >> +Over Transport Virtqueue / Virtqueue State}
> >> +
> >> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
> >> +the virtqueue state is accessed through the following commands:
> >> +
> >> +\begin{lstlisting}
> >> +#define VIRTIO_TRANSPTQ_CTRL_VQ_STATE            12
> >> + #define VIRTIO_TRANSPTQ_CTRL_VQ_STATE_GET       0
> >> + #define VIRTIO_TRANSPTQ_CTRL_VQ_STATE_SET       1
> >> +\end{lstlisting}
> >> +
> >> +\subsubsection{Split Virtqueue State}\label{sec:Virtio Transport Options / Virtio
> >> +Over Transport Virtqueue / Virtqueue State / Split Virtqueue State}
> >> +
> >> +\begin{lstlisting}
> >> +struct virtio_transportq_ctrl_split_vq_state_set {
> >> +    u16 queue_index;
> >> +    u16 avail_index;
> >> +    u8 padding[4];
> >> +};
> >> +\end{lstlisting}
> >> +
> >> +The VIRTIO_TRANSPTQ_CTRL_VQ_STATE_GET command can be  used to get the state of a
> >> +split virtqueue. The command-out-data is the queue index, the command-in-data is the available index of the virtqueue.
> > We need to be accurate for "available index". E.g we have one for in
> > the shared memory, so do you mean last_avail_idx?
> actually there is no definition of "last available index" in the spec,
> so how about we say:
> on-device available index?

Any should be fine but need to explain what does it mean.

> >
> >> +
> >> +The VIRTIO_TRANSPTQ_CTRL_VQ_STATE_SET command can be used to set the state of a
> >> +split virtqueue. The command-out-data contains the \field{queue_index} and the
> >> +available index (\field{avail_index}) for the virtqueue (as described in struct virtio_transportq_ctrl_split_vq_state_set).
> >> +There is no command-in-data.
> >> +
> >> +\subsubsection{Packed Virtqueue State}\label{sec:Virtio Transport Options / Virtio
> >> +Over Transport Virtqueue / Virtqueue State / Packed Virtqueue State}
> >> +
> >> +\begin{lstlisting}
> >> +struct virtio_transportq_ctrl_packed_vq_state {
> >> +    u16 queue_index;
> >> +    u16 avail_counter;
> >> +    u16 avail_index;
> >> +    u16 used_counter;
> >> +    u16 used_index;
> > Let's simply reuse the structure in the kernel headers.
> >
> > Using u16 for wrap_couter seems unnecessary etc.
> OK, the bit operations are not often.

It should be very common in the hardware interface.

> >
> >> +    u8 padding[6];
> >> +};
> >> +\end{lstlisting}
> >> +
> >> +The state of a packed virtqueue includes :\\
> >> +\field{avail_counter}: last driver ring wrap counter observed by device.\\
> >> +\field{avail_index}: virtqueue available index.\\
> >> +\field{used_counter}: device ring wrap counter.\\
> >> +\field{used_index}: virtqueue used index.
> >> +
> >> +The VIRTIO_TRANSPTQ_CTRL_VQ_STATE_GET command can be used to get the state of a packed virtqueue.
> >> +The command-out-data is the queue index, the command-in-data contains the \field{queue_index},
> >> +\field{avail_counter}, \field{avail_index}, \field{used_counter} and \field{used_index} of the virtqueue
> >> +(as described in transportq_ctrl_packed_vq_state).
> >> +
> >> +The VIRTIO_TRANSPTQ_CTRL_VQ_STATE_SET command can be used to set the state of a packed virtqueue.
> >> +The command-out-data contains the \field{queue_index}, \field{avail_counter}, \field{avail_index},
> >> +\field{used_counter} and \field{used_index} for the virtqueue
> >> +(as described in transportq_ctrl_packed_vq_state).
> >> +There is no command-in-data.
> >> +
> >> +\devicenormative{\subsubsection}{Virtqueue State}{Virtio Transport Options /
> >> +Virtio Over Transport Virtqueue / Virtqueue State}
> >> +
> >> +The management device MUST fail the virtqueue index access if \field{device_id} is 0.
> >> +
> >> +The management device MUST fail the virtqueue index access if \field{queue_index} is out-of-range invalid.
> >> +
> >> +\subsection{Virtqueue ASID}\label{sec:Virtio Transport Options / Virtio Over
> >> +Transport Virtqueue / Virtqueue ASID}
> >> +
> >> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
> >> +the address space id of a virtqueue could be set through the following command:
> >> +
> >> +\begin{lstlisting}
> >> +#define VIRTIO_TRANSPTQ_CTRL_VQ_ASID        13
> >> + #define VIRTIO_TRANSPTQ_CTRL_VQ_ASID_SET   1
> >> +
> >> +struct virtio_transportq_ctrl_vq_asid_set {
> >> +       u16 queue_index;
> >> +       u32 asid;
> >> +       u8 padding[2];
> > I'd reserve more here. E.g in the future each virtqueue may have a
> > secondary ASID,
> like asid[2]? But why? For a device, there may needs a secondary ASID
> because it
> may need to access different memory region on behalf of different DMA
> sources.
> But for a virtqueue, it only has one purpose, to access the queue
> buffer, maybe it
> doesn't need a secondary asid

E.g for transparent shadow virtqueue.

> >
> >> +};
> >> +\end{lstlisting}
> >> +
> >> +The VIRTIO_TRANSPTQ_CTRL_VQ_ASID_SET command is used to set the address space id(\field{asid})
> >> +of a virtqueue.
> >> +
> >> +The address space id is an identifier
> > Let's add "transport specific" before "identifier"
> OK
> >
> >> of a memory space which is used to convey the address space targeted by the
> >> +memory accesses
> > We need to say the ASID is for DMA not here at least.

I meant you need to explain how address space is used. (e.g it it used
only for DMA).

> I don't get it, we need to explain what ASID is used for after
> we introduce it.
> >
> >> , and to distinguish memory accesses performed by different virtqueues
> > This may confuse the readers since a single ASID could be shared by virtqueues.
> so it should be "by different virtqueues or virtqueue groups"

I think we can refer to the explanation of transport specific ASID here.

Thanks

Re: [virtio-comment] [PATCH V3 RESEND 2/4] Introduce the commands set of the transport vq

[Zhu, Lingshan]

On 8/10/2022 9:56 AM, Jason Wang wrote:
> On Tue, Aug 9, 2022 at 9:09 PM Zhu, Lingshan <lingshan.zhu@intel.com> wrote:
>>
>>
>> On 8/8/2022 6:04 PM, Jason Wang wrote:
>>> On Fri, Aug 5, 2022 at 6:02 PM Zhu Lingshan <lingshan.zhu@intel.com> wrote:
>>>> This commit introduces the commands set of the
>>>> transport virtqueue, including:
>>>>
>>>> The command to query available resources of the management device
>>>> The commands to create / destroy the managed devices.
>>>> The commands to config the managed devices.
>>>> The commands to config virtqueues of the managed devices.
>>>>
>>>> Signed-off-by: Jason Wang <jasowang@redhat.com>
>>>> Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com>
>>>> ---
>>>>    content.tex | 636 ++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>    1 file changed, 636 insertions(+)
>>>>
>>>> diff --git a/content.tex b/content.tex
>>>> index c747d21..190f3a0 100644
>>>> --- a/content.tex
>>>> +++ b/content.tex
>>>> @@ -2975,6 +2975,642 @@ \subsection{Format of Commands through Transport Virtqueue}\label{sec:Virtio Tra
>>>>    The \field{device_id} value 0 is used to identify the management device itself.
>>>>
>>>>    \field{class} is an identifier of a set of commands with similar  purposes.
>>>> +
>>>> +\subsection{Available Resource of the Management Device}\label{sec:Virtio Transport Options / Virtio Over Transport Virtqueue / Available Resource of the Management Device}
>>>> +
>>>> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
>>>> +statistical information of available resources on the management device could be fetched
>>>> +by the following command:
>>>> +
>>>> +\begin{lstlisting}
>>>> +#define VIRTIO_TRANSPTQ_CTRL_AVAIL_RES    1 (class)
>>>> + #define VIRTIO_TRANSPTQ_CTRL_AVAIL_RES_GET     0 (command)
>>>> +\end{lstlisting}
>>>> +
>>>> +VIRTIO_TRANSPTQ_CTRL_AVAIL_RES_GET command is used to get the statistical information of the available
>>>> +resource of a management device. There is no command-out-data for VIRTIO_TRANSPTQ_CTRL_AVAIL_RES_GET
>>>> +command. The management device fills command-in-data in the following format:
>>>> +
>>>> +\begin{lstlisting}
>>>> +struct virtio_mgmt_dev_avail_res {
>>>> +        /* The number of management device total remaining available free
>>>> +         * virtqueues for the managed devices
>>>> +         */
>>> It's hard to phrase this sentence.
>> How about "The number of remaining available virtqueues for new managed
>> devices"
>>>> +        u16 num_avail_vqs;
>>>> +        /* The minimal number of virtqueues for a managed device */
>>>> +        u16 min_dev_vqs;
>>>> +        /* The maximum number of virtqueues for a managed device */
>>> Nit the min/max_dev_vqs are not the resources, so we'd better rename
>>> the virtio_mgmt_dev_avail_res.
>> OK, I will use struct virito_mgmt_dev_info{} in the next version, and
>> the command and its description
>> will change as well.
>>>> +        u16 max_dev_vqs;
>>>> +        /* The number of managed devices that can be created with min_vqs virtqueues */
>>> Is the part "with min_vqs virtqueues" a must? My understanding of the
>>> above is a hint that the managed device is loosely coupled with the
>>> virtqueue resources.
>> Yes, the number of the number of total managed devices can be limited by
>> other resources, like the filters.
>>> So it would be possible that num_avail_dev = 0 but num_avail_vqs not?
>>>
>>> Btw, "num" prefix is probably not a must.
>> OK
>>>> +        u16 num_avail_dev;
>>> To be consistent, let's use "num_avail_devs"?
>> I will remove all num_ prefix, and it should be avail_devs
>>>> +};
>>>> +\end{lstlisting}
>>>> +
>>>> +\field{max_avail_dev} MAY not equal to \field{num_avail_vqs} divided by \field{min_vqs}
>>>> +due to the limitations of of other resources.
>>> two "of".
>> will fix
>>> Keywords like "MAY" should go to normatives part.
>> OK, so I will only use "may", so it is a normal description.
>>>> +
>>>> +\devicenormative{\subsubsection}{Available Resource of the Management Device}{Virtio Transport Options /
>>>> +Virtio Over Transport Virtqueue / Available Resource of the Management Device}
>>>> +
>>>> +The management device MUST fail VIRTIO_TRANSPTQ_CTRL_AVAIL_RES_GET command if \field{device_id} is not 0
>>>> +
>>>> +\drivernormative{\subsubsection}{Available Resource of the Management Device}{Virtio Transport Options /
>>>> +Virtio Over Transport Virtqueue / Available Resource of the Management Device}
>>>> +
>>>> +The management device driver MUST use 0 as \field{device_id} for
>>>> +VIRTIO_TRANSPTQ_CTRL_AVAIL_RES_GET command.
>>>> +
>>>> +\subsection{Create and Destroy Managed Devices}\label{sec:Virtio Transport Options / Virtio Over Transport Virtqueue / Create and Destroy Managed Devices}
>>>> +
>>>> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
>>>> +managed devices must be created and destroyed through the transport virtqueue.
>>>> +
>>>> +\begin{lstlisting}
>>>> +#define VIRTIO_TRANSPORTQ_CTRL_DEV    2
>>>> + #define VIRTIO_TRANSPORTQ_CTRL_DEV_CREATE        0
>>>> + #define VIRTIO_TRANSPORTQ_CTRL_DEV_DESTROY       1
>>>> +
>>>> +struct virtio_transportq_ctrl_dev_attribute {
>>>> +       u64 device_features[2];
>>> Let's reserve more bits for features here as we are about to reach 64.
>> OK, it looks 256bytes may be enough.
> It's better to match the parent/management device, but maybe we can
> start from something that is fine for the next 5 years. So I'd rather
> go with even more than this like device_feautres[10].
The management device implementation can be various, I will make it 
device_features[12]
>
>
>>>> +       u32 virtio_device_id;
>>>> +       u8 dev_config[];
>>> To be consistent, let's use one of "device" or "dev" in the same structure.
>> sure, device_config[]
>>>> +};
>>>> +\end{lstlisting}
>>>> +
>>>> +The VIRTIO_TRANSPORTQ_CTRL_DEV_CREATE command is used to create a managed device.
>>>> +
>>>> +As described in struct virtio_transportq_ctrl_dev_attribute, the command-out-data for VIRTIO_TRANSPTQ_CTRL_DEV_CREATE includes:
>>>> +\begin{itemize*}
>>>> +\item \field{device_features}: the 128-bit device feature bits (defined in section \ref{sec:Basic Facilities of a Virtio Device / Feature Bits})
>>>> +of the managed device.
>>>> +\item \field{virtio_device_id}: the virtio device id defined in Chapter \ref{sec:Device Types}.
>>>> +\item \field{dev_config}: the device type specific configurations. E.g., for a virtio-net device,
>>>> +it is struct virtio_net_config in section \ref{sec:Device Types / Network Device / Device configuration layout};
>>>> +for a virtio-block device, it is struct virtio_blk_config in section \ref{sec:Device Types / Block Device / Device configuration layout}
>>>> +\end{itemize*}
>>>> +
>>>> +When succeed, the device returns a 64 bits UUID as an unique identifier of the created managed device in command-in-data.
>>> We don't need to be universally unique here.
>> Why? I think it should be unique to identify a device, or are you
>> suggesting to remove "as an unique identifier",
>> just UUID is enough?
> UUID means it's unique in the universal. If it means ID for managed
> device in PF1 must differ from ID for managed device in PF2. This is
> not needed.
>
> So if we want to be flexible, we can allow the ID to be allocated by
> the driver as the original RFC did. The driver is free to allocate
> anything it want (UID or UUID).
I agree this ID only needs to be unique in the "management device scope",
maybe a PF scope. So I think this ID can be allocated by the management 
device
as this patch proposed. I will remove this UUID requirement, change it to be
a 64bit management-device-allocated ID, so the management device can
choose its own algorithm to generate this manged device ID
>
>>>> +
>>>> +The VIRTIO_TRANSPORTQ_CTRL_DEV_DESTROY command is used to destroy a
>>>> +managed device which is identified by its 64 bits UUID
>>>> +\field{device_id}. There's no command-in-data nor command-out-data for
>>>> +VIRTIO_TRANSPTQ_CTRL_DEV_DESTROY command.
>>>> +
>>>> +\devicenormative{\subsubsection}{Create and Destroy Managed Devices}{Virtio Transport Options / Virtio Over Transport Virtqueue / Create and Destroy Managed Devices}
>>>> +
>>>> +The management device MUST fail VIRTIO_TRANSPORTQ_CTRL_DEV_CREATE command
>>>> +if \field{device_id} is not 0.
>>>> +
>>>> +The management device MUST fail VIRTIO_TRANSPORTQ_CTRL_DEV CREATE command
>>>> +if \field{device_features} exceeds the features that can be provided from the management device.
>>> So PCI has le32 device_feature_select. How to make sure it can match
>>> the capacity of device_features[] above?
>> we present all feature bits in device_features[], the spec says the
>> feature bits are valid up to bit 127,
>> 128 and above are reserved. So as you suggest, we can use u64
>> device_features[4], for 128-bit feature bits
>> and another 128 bits for further extensions.
>>>> +
>>>> +The management device MUST fail VIRTO_TRANSPORTQ_CTRL_DEV_DESTROY command
>>>> +if \field{device_id} is 0.
>>>> +
>>>> +\drivernormative{\subsubsection}{Create and Destroy Managed Devices}{Virtio Transport Options / Virtio Over Transport Virtqueue / Create and Destroy Managed Devices}
>>>> +
>>>> +The management device driver MUST use 0 as \field{device_id} for
>>>> +TRANSPORTQ_CTRL_DEV_CREATE command.
>>>> +
>>>> +\subsection{Features Negotiation}\label{sec:Virtio Transport Options / Virtio
>>>> +Over Transport Virtqueue / Features Negotiation}
>>>> +
>>>> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
>>>> +the features negotiation of managed devices is done by the
>>>> +following commands:
>>>> +
>>>> +\begin{lstlisting}
>>>> +#define VIRTIO_TRANSPTQ_CTRL_FEAT   3
>>>> + #define VIRTIO_TRANSPTQ_CTRL_FEAT_DEVICE_GET        0
>>>> + #define VIRTIO_TRANSPTQ_CTRL_FEAT_DRIVER_SET        1
>>>> + #define VIRTIO_TRANSPTQ_CTRL_FEAT_DRIVER_GET        2
>>>> +\end{lstlisting}
>>>> +
>>>> +The VIRTIO_TRANSPTQ_CTRL_FEAT_DEVICE_GET command is used to get the features offered
>>>> +by a managed device. The command-in-data is the 128-bit feature bits
>>>> +(defined in section \ref{sec:Basic Facilities of a Virtio Device / Feature Bits}).
>>> Need to make sure the capacity of the feature bit matches what the
>>> transport of the management device can provide.
>> I will provide a structure for the features, so both device and driver
>> would be aligned on
>> the size of the feature bits, like
>>
>> struct virtio_transportq_ctrl_dev_features {
>>       u64 features[4];
>> };
>>>> +There is no command-out-data.
>>>> +
>>>> +The VIRTIO_TRANSPTQ_CTRL_FEAT_DRIVER_SET command is for the driver to accept feature
>>>> +bits offered by the managed device. The command-out-data is the 128-bit feature bits
>>>> +passed to the managed device. There is no command-in-data.
>>>> +
>>>> +The VIRTIO_TRANSPTQ_CTRL_FEAT_DRIVER_GET command is used to get the features accepted
>>>> +by both the managed device driver and the managed device. The command-in-data is the
>>>> +128-bit feature bits. There is no command-out-data.
>>>> +
>>>> +\devicenormative{\subsubsection}{Features Negotiation}{Virtio Transport Options /
>>>> +Virtio Over Transport Virtqueue / Features Negotiation}
>>>> +
>>>> +The management device MUST fail VIRTIO_TRANSPTQ_F_CTRL_FEAT class commands if
>>>> +\field{device_id} is 0.
>>>> +
>>>> +\subsection{Device Status}\label{sec:Virtio Transport Options / Virtio Over
>>>> +Transport Virtqueue / Device Status}
>>>> +
>>>> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
>>>> +the device status of a managed device can be accessed by the following
>>>> +commands:
>>>> +
>>>> +\begin{lstlisting}
>>>> +#define VIRTIO_TRANSPTQ_CTRL_STATUS    4
>>>> + #define VIRTIO_TRANSPTQ_CTRL_STATUS_GET        0
>>>> + #define VIRTIO_TRANSPTQ_CTRL_STATUS_SET        1
>>>> +\end{lstlisting}
>>>> +
>>>> +The VIRTIO_TRANSPTQ_CTRL_STATUS_GET command is used to get the device status of
>>>> +a managed device. The command-in-data is the 8-bit status
>>>> +returned from the device. There's no command-out-data for this
>>>> +command.
>>>> +
>>>> +The VIRTIO_TRANSPTQ_CTRL_STATUS_SET command is used to set the device status of
>>>> +a managed device. The command-out-data is the 8-bit status
>>>> +to set to the device. There's no command-in-data for this command.
>>>> +
>>>> +\devicenormative{\subsubsection}{Device Status}{Virtio Transport Options / Virtio
>>>> +Over Transport Virtqueue / Device Status}
>>>> +
>>>> +The management device MUST reset the managed device when 0
>>>> +is set via VIRTIO_TRANSPTQ_CTRL_STATUS_SET, the success of this
>>>> +command demonstrates the success of the reset.
>>>> +
>>>> +The management device MUST present 0 through
>>>> +VIRTIO_TRANSPTQ_CTRL_STATUS_GET once the reset is successfully done.
>>>> +
>>>> +The management device MUST fail the device status access if
>>>> +\field{device_id} is 0.
>>>> +
>>>> +\drivernormative{\subsubsection}{Device Status}{Virtio Transport Options / Virtio
>>>> +Over Transport Virtqueue / Device Status}
>>>> +
>>>> +After writing 0 via VIRTIO_TRANSPTQ_CTRL_STATUS_SET, the driver MUST wait
>>>> +for the success of the command before re-initializing the device.
>>>> +
>>>> +\subsection{Device Generation}\label{sec:Virtio Transport Options / Virtio Over Transport Virtqueue / Device Generation}
>>>> +
>>>> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
>>> Btw, since we're under the transport options (which implies the
>>> transport virtqueue), maybe we can drop the above for each command.
>> OK, can do, and I will describe this in the Basic Concept section.
>>>> +the device generation count can be read from the following commands:
>>>> +
>>>> +\begin{lstlisting}
>>>> +#define VIRTIO_TRANSPTQ_CTRL_GENERATION    5
>>>> + #define VIRTIO_TRANSPTQ_CTRL_GENERATION_GET        0
>>>> +\end{lstlisting}
>>>> +
>>>> +The VIRTIO_TRANSPTQ_CTRL_GENERATION_GET command is used to get the device configuration generation count
>>>> +of the managed device. The command-in-data is the one byte device
>>>> +generation returned from the device. There's no command-out-data for
>>>> +this command.
>>>> +
>>>> +\devicenormative{\subsubsection}{Device Generation}{Virtio Transport Options / Virtio Over Transport Virtqueue / Device Generation}
>>>> +
>>>> +The managed device MUST present a changed generation count after the driver
>>>> +has read any device-specific configuration values if the values have been changed
>>>> +during the last read.
>>>> +
>>>> +The management device MUST fail the device generation access if \field{device_id} is 0.
>>>> +
>>>> +\subsection{Device Specific Configuration}\label{sec:Virtio Transport Options /
>>>> +Virtio Over Transport Virtqueue / Device Specific Configuration}
>>>> +
>>>> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
>>>> +the device config space contents of a managed device can be accessed through
>>>> +the following commands:
>>>> +
>>>> +\begin{lstlisting}
>>>> +#define VIRTIO_TRANSPTQ_CTRL_CONFIG    6
>>>> +  #define VIRTIO_TRANSPTQ_CTRL_CONFIG_GET    0
>>>> +  #define VIRTIO_TRANSPTQ_CTRL_CONFIG_SET    1
>>>> +
>>>> +struct virtio_transportq_ctrl_dev_config_get {
>>>> +       u32 offset;
>>>> +       u32 size;
>>>> +};
>>>> +
>>>> +struct virtio_transportq_ctrl_dev_config_set {
>>>> +       u32 offset;
>>>> +       u32 size;
>>>> +       u8  data[];
>>>> +};
>>>> +\end{lstlisting}
>>>> +
>>>> +The VIRTIO_TRANSPTQ_CTRL_CONFIG_GET command is used to read data from the
>>>> +device configuration space.  The command-out-data is the \field{offset}
>>>> +from the start of the config space and the \field{size}
>>>> +of the data (as described in struct virtio_transportq_ctrl_dev_config_get).
>>>> +The command-in-data is an array of the data that read from the config space.
>>>> +
>>>> +The VIRTIO_TRANSPTQ_CTRL_CONFIG_SET command is used to write data to the device
>>>> +configuration space. The command-out-data contains the
>>>> +\field{offset} from the start of the config space, the \field{size} of the data and
>>>> +the \field{data} to be written (as described in struct virtio_transportq_ctrl_dev_config_set).
>>>> +There's no command-in-data for this command.
>>>> +
>>>> +\devicenormative{\subsubsection}{Device Specific Configuration}{Virtio Transport
>>>> +Options / Virtio Over Transport Virtqueue / Device Specific Configuration}
>>>> +
>>>> +The management device MUST fail the device configuration space access
>>>> +if the driver access the range which is outside the config space.
>>> "accesses"
>> yes
>>>> +
>>>> +The management device MUST fail the device configuration space access
>>>> +if \field{device_id} is 0.
>>>> +
>>>> +\subsection{MSI Configuration}\label{sec:Virtio Transport Options / Virtio Over
>>>> +Transport Virtqueue / MSI Configuration}
>>>> +
>>>> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
>>>> +the MSI entries of a managed device can be accessed  through the following command:
>>> Rethink of this, it implies a per virtqueue MSI storage which I'm not
>>> sure is expensive or not. Do we need an indirection layer as PCI did?
>> I think we need this for performance, or are you suggesting to add a
>> device scope MSI?
> I meant having MSI vectors and let the virtqueue refer to the MSI vectors.
>
> Current design means for a 1024 virtqueues device to store 1024 MSI entries.
> With the indirection of the MSI vectors array, the device is free to
> have 1 to 1024 MSI entries.
I am not sure I get your points, how can the queues decide the MSI 
entries, it should
be the driver and the platform set the vq's MSI. The driver has to 
decide/control and handle
the MSI interrupts.

IMHO there are two options:
1)every virtqueue stores its own MSI vector, per-vq MSI vector can get 
better performance
for sure. But if there are limited MSI resources, the driver may decide 
to share MSI vectors
among the vqs(1k queues, 1 to 1024 MSI entries), and there should be 
proper interrupt handlers in the driver.

2)add a device scope MSI entry(only for vqs, not config interrupt, 
because we don't want to look into config space every time receive a vq 
interrupt),
all interrupts from the device are through this MSI entry,
this is optional, can help save resource. But this is complex, like 
first set a device scope MSI for all vqs, then set a MSI for vq1, in 
this case,
we need vq1 use its own MSI to send interrupts, shadows the device MSI. 
This is complex, maybe not worthy, I think the platform(x86, arm) has enough
MSI resource and it is just one vector per vq.
>
>>>> +
>>>> +\begin{lstlisting}
>>>> +#define VIRTIO_TRANSPTQ_CTRL_MSI    7
>>>> + #define VIRTIO_TRANSPTQ_CTRL_MSI_VQ_GET        0
>>>> + #define VIRTIO_TRANSPTQ_CTRL_MSI_VQ_SET        1
>>>> + #define VIRTIO_TRANSPTQ_CTRL_MSI_VQ_ENABLE     2
>>>> + #define VIRTIO_TRANSPTQ_CTRL_MSI_CONFIG_GET    3
>>>> + #define VIRTIO_TRANSPTQ_CTRL_MSI_CONFIG_WET    4
>>> I think you mean CONFIG_SET here.
>> yes
>>>> + #define VIRTIO_TRANSPTQ_CTRL_MSI_CONFIG_ENABLE 5
>>>> +
>>>> +struct virtio_transportq_ctrl_msi_vq_config {
>>>> +       u16 queue_index;
>>>> +       u64 address;
>>>> +       u32 data;
>>>> +       u8 padding[2];
>>>> +};
>>>> +
>>>> +struct virtio_transportq_ctrl_msi_vq_enable {
>>>> +       u16 queue_index;
>>>> +       u8 enable;
>>>> +       u8 padding[5];
>>>> +};
>>>> +
>>>> +struct virtio_transportq_ctrl_msi_config {
>>>> +       u64 address;
>>>> +       u32 data;
>>>> +       u8 padding[4];
>>>> +};
>>>> +\end{lstlisting}
>>>> +
>>>> +The VIRTIO_TRANSPTQ_CTRL_MSI_VQ_GET command is used to read the MSI entry of a
>>>> +specific virtqueue. The command-out-data is the queue index. The command-in-data contains
>>>> +\field{queue_index}, the \field{address} and \field{data}
>>>> +(as described in struct virtio_transportq_ctrl_msi_vq_config).
>>> Let's split or reuse the virtio_transportq_ctrl_msi_config since
>>> virtio_transportq_ctrl_msi_vq_config will not be used a as single
>>> buffer.
>> I don't get it, if we re-use the struct virtio_transportq_ctrl_msi_config,
>> it would be like:
>>
>> struct virtio_transportq_ctrl_msi {
>>       u64 addr;
>>       u32 data;
>> };
>>
>> struct virtio_transportq_ctrl_msi_vq_config {
>>       u16 queue_index;
> What I meant is, for GET. queue_index is out buffer, and re-use
> virtio_transportq_ctrl_msi is in buffer.
OK, for get we can reuse virtio_transportq_ctrl_msi_config, and we need
to rename this struct to virto_transport_ctrl_msi, to be generic
>
>>       struct virtio_transportq_ctrl_msi;
>> };
>>
>> it is still two structs and a struct in another struct.
>>>> +
>>>> +
>>>> +The VIRTIO_TRANSPTQ_CTRL_MSI_VQ_SET command is used to set the MSI entry for a
>>>> +specific virtqueue. The command-out-data is the \field{queue_index},
>>>> +\field{address} and \field{data} (as described in struct virtio_transportq_ctrl_msi_vq_config).
>>>> +There is no command-in-data.
>>>> +
>>>> +The VIRTIO_TRANSPTQ_CTRL_MSI_VQ_ENABLE command is used to enable or disable
>>>> +the MSI interrupt for a specific virtqueue. The command-out-data is the
>>>> +\field{queue_index} and a byte \field{enable} which representing ENABLE or DISABLE the MSI.
>>> Should be "represents".
>> OK
>>> Btw, is there a chance that we may lose an interrupt?
>> No, I think we won't lose any interrupts, because once disabled, it can
>> not send new interrupts,
>> but the last interrupt send by the MSI entry is still there, it's in-band.
> So consider driver is changing affinity what it did is:
>
> disable
> write MSI
> enable
>
> What if there's an interrupt coming in the middle? Does this mean the
> device needs to implement the pending bit internally?
Please correct me if I misunderstand anything. MSI interrupts are 
in-band DMA,
so if we don't lose the DMA package, we won't lose the interrupt.

Changing affinity only affecting which process handles the interrupt. I 
think
we can take an example of auto-affinity, if a processor is too busy, the 
platform
may offload some interrupts to other processors, this also changes the 
affinity,
we don't lose any interrupts.

I am not sure whether we need to disable/enable the interrupts for 
change the affinity,
its in-band DMA, maybe we don't need this

Thanks,
Zhu Lingshan
>
> FYI we had similar discussion in the past:
>
> https://lists.oasis-open.org/archives/virtio-dev/202002/msg00027.html
>
>>>> +(as described in struct virtio_transportq_ctrl_msi_vq_enable).
>>>> +
>>>> +\begin{lstlisting}
>>>> +#define VIRTIO_TRANSPTQ_CTRL_MSI_VQ_ENABLE     1
>>>> +#define VIRTIO_TRANSPTQ_CTRL_MSI_VQ_DISABLE    0
>>>> +\end{lstlisting}
>>>> +
>>>> +The VIRTIO_TRANSPTQ_CTRL_MSI_CONFIG_GET command is used to read the MSI entry
>>>> +of the config interrupt. The command-in-data contains the \field{address} and
>>>> +\field{data} (as described in struct virtio_transportq_ctrl_msi_config).
>>>> +There is no command-out-data.
>>>> +
>>>> +The VIRTIO_TRANSPTQ_CTRL_MSI_CONFIG_SET command is used to set the MSI entry
>>>> +for the config interrupt. The command-out-data is the \field{address} and
>>>> +\field{data} (as described in struct virtio_transportq_ctrl_msi_config).
>>>> +There is no command-in-data.
>>>> +
>>>> +The VIRTIO_TRANSPTQ_CTRL_MSI_CONFIG_ENABLE command is used to enable or disable
>>>> +the MSI interrupt for config space. The command-out-data is a byte which representing
>>>> +ENABLE or DISABLE the config MSI.
>>>> +There is no command-in-data
>>>> +
>>>> +\begin{lstlisting}
>>>> +#define VIRTIO_TRANSPTQ_CTRL_MSI_CFG_ENABLE     1
>>>> +#define VIRTIO_TRANSPTQ_CTRL_MSI_CFG_DISABLE    0
>>>> +\end{lstlisting}
>>>> +
>>>> +\devicenormative{\subsubsection}{MSI Configuration}{Virtio Transport Options / Virtio
>>>> +ver Transport Virtqueue / MSI Configuration}
>>>> +
>>>> +The managed device MUST disable the MSI interrupts for both virtqueues and config space
>>>> +upon a reset.
>>>> +
>>>> +Once a MSI entry is disabled, the managed device MUST not send any interrupts
>>>> +by this MSI entry.
>>> MSI is stored based on virtqueue and config space, so I'm not quite
>>> sure "MSI entry" is accurate here.
>> I will replace all "MSI entry" with "MSI vector"
>>>> +
>>>> +\drivernormative{\subsubsection}{MSI Configuration}{Virtio Transport Options / Virtio
>>>> +Over Transport Virtqueue / MSI Configuration}
>>>> +
>>>> +The driver MUST allocate transport or platform specific MSI entries
>>>> +for both virtqueues and config space if it wants to use interrupts.
>>>> +
>>>> +The driver MAY choose to disable the MSI if polling mode is used.
>>>> +
>>>> +\subsection{Virtqueue Address}\label{sec:Virtio Transport Options / Virtio Over
>>>> +Transport Virtqueue / Virtqueue Address}
>>>> +
>>>> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
>>>> +the addresses of a specific virtqueue are accessed through the following command:
>>>> +
>>>> +\begin{lstlisting}
>>>> +#define VIRTIO_TRANSPTQ_CTRL_VQ_ADDR    8
>>>> + #define VIRTIO_TRANSPTQ_CTRL_VQ_ADDR_SET       1
>>>> +
>>>> +struct virtio_transportq_ctrl_vq_addr {
>>>> +       u16 queue_index;
>>>> +       u64 descriptor_area;
>>>> +       u64 device_area;
>>>> +       u64 driver_area;
>>>> +       u8 padding[6];
>>>> +};
>>>> +\end{lstlisting}
>>>> +
>>>> +The VIRTIO_TRANSPTQ_CTRL_VQ_ADDR_SET command is used to set the addresses of a specified
>>>> +virtqueue. The command-out-data contains the \field{queue_index}, the addresses of \field{device_area},
>>>> +\field{descriptor_area} and \field{driver_area} (as described in struct
>>>> +virtio_transportq_ctrl_vq_addr). There's no command-in-data.
>>>> +
>>>> +\devicenormative{\subsubsection}{Virtqueue Address}{Virtio Transport Options / Virtio
>>>> +Over Transport Virtqueue / Virtqueeu Address}
>>>> +
>>>> +The management device MUST fail the commands of class
>>>> +VIRTIO_TRANSPTQ_CTRL_VQ_ADDR if \field{device_id} is 0.
>>>> +
>>>> +The management device MUST fail the commands of class
>>>> +VIRTIO_TRANSPTQ_CTRL_VQ_ADDR if \field{queue_index} is out-of-range invalid.
>>>> +
>>>> +\subsection{Virtqueue Status}\label{sec:Virtio Transport Options / Virtio Over
>>>> +Transport Virtqueue / Virtqueue Status}
>>>> +
>>>> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
>>>> +virtqueue status is accessed through the following command:
>>>> +
>>>> +\begin{lstlisting}
>>>> +#define VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE    9
>>>> + #define VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_GET       0
>>>> + #define VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_SET       1
>>>> +
>>>> +struct virtio_transportq_ctrl_vq_status_set {
>>>> +       u16 queue_index;
>>>> +       u8 status;
>>>> +       u8 padding[5];
>>>> +
>>>> +#define VIRTIO_TRANSPTQ_VQ_ENABLE     1
>>>> +#define VIRTIO_TRANSPTQ_VQ_DISABLE    0
>>>> +
>>>> +};
>>>> +
>>>> +\end{lstlisting}
>>>> +
>>>> +The VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_GET is used to get the status of a
>>>> +specific virtqueue. The command-out-data is the queue
>>>> +index. The command-in-data is the virtqueue status.
>>>> +
>>>> +The VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_SET command is used to set the status of a
>>>> +specific virtqueue. The command-out-data is the \field{queue_index} and the
>>>> +\field{status} representing ENABLE or DISABLE that is set to the virtqueue
>>>> +(as described in struct virtio_transportq_ctrl_vq_status_set).
>>>> +There's no command-in-data.
>>>> +
>>>> +\devicenormative{\subsubsection}{Virtqueue Status}{Virtio Transport Options / Virtio
>>>> +Over Transport Virtqueue / Virtqueue Status}
>>>> +
>>>> +When disabled, the managed device MUST stop processing requests from
>>>> +this virtqueue.
>>> Is this a hint that the driver can write VIRTIO_TRANSPTQ_VQ_DISABLE to
>>> status? If yes, what's the difference between this and virtqueue
>>> reset?
>> I think disabling a queue is like "set queue_enable = 0", once disabled,
>> the device should not reset the queue configurations like the indexes
>> and addresses,
>> it just freeze the queue.
> Then you need to define what "freeze" means. To reduce the changset,
> I'd suggest disallowing such writing as what PCI did.
>
> If needed, we can re-enable it in the future.
>
>> When reset a queue, like the spec says, the device needs to reset all queue
>> configs.
>>>> +
>>>> +The management device MUST present a 0 via
>>>> +VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_GET upon a reset of the managed device.
>>>> +
>>>> +The management device MUST fail the virtqueue status access if
>>>> +\field{device_id} is 0.
>>>> +
>>>> +The management device MUST fail the virtqueue status access if
>>>> +the queue_index is out-of-range invalid.
>>>> +
>>>> +
>>>> +\drivernormative{\subsubsection}{Virtqueue Status}{Virtio Transport Options / Virtio
>>>> +Over Transport Virtqueue / Virtqueue Status}
>>>> +
>>>> +The driver MUST configure other virtqueue fields before enabling
>>>> +the virtqueue with VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_SET.
>>>> +
>>>> +\subsection{Virtqueue Size}\label{sec:Virtio Transport Options / Virtio Over
>>>> +Transport Virtqueue / Virtqueue Size}
>>>> +
>>>> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
>>>> +virtqueue size is accessed through the following command:
>>>> +
>>>> +\begin{lstlisting}
>>>> +#define VIRTIO_TRANSPTQ_CTRL_VQ_SIZE    10
>>>> + #define VIRTIO_TRANSPTQ_CTRL_VQ_SIZE_GET       0
>>>> + #define VIRTIO_TRANSPTQ_CTRL_VQ_SIZE_SET       1
>>>> +
>>>> +struct virtio_transportq_ctrl_vq_size_set {
>>>> +       u16 queue_index;
>>>> +       u16 size;
>>>> +       u8 padding[4];
>>>> +};
>>>> +\end{lstlisting}
>>>> +
>>>> +The VIRTIO_TRANSPTQ_CTRL_VQ_SIZE_GET command is used to get the virtqueue
>>>> +size. On reset, the maximum queue size supported by the device is
>>>> +returned. The command-out-data is the queue index. The
>>>> +command-in-data is an 16-bit queue size.
>>>> +
>>>> +The VIRTIO_TRANSPTQ_CTRL_VQ_SIZE_SET command is used to set the virtqueue
>>>> +size. The command-out-data is the \field{queue_index} and the \field{size}
>>>> +of the virtqueue (as described in struct virtio_transportq_ctrl_vq_size_set).
>>>> +There's no command-in-data.
>>>> +
>>>> +\devicenormative{\subsubsection}{Virtqueue Status}{Virtio Transport Options / Virtio
>>>> +Over Transport Virtqueue / Virtqueue Size}
>>>> +
>>>> +The management device MUST fail the virtqueue size access if
>>>> +\field{device_id} is 0.
>>>> +
>>>> +The management device MUST fail the virtqueue size access if
>>>> +the queue index is out-of-range invalid.
>>>> +
>>>> +\subsection{Virtqueue Notification}\label{sec:Virtio Transport Options / Virtio
>>>> +Over Transport Virtqueue / Virtqueue Notification}
>>>> +
>>>> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
>>>> +the virtqueue notification area information can be get through the following commands:
>>>> +
>>>> +\begin{lstlisting}
>>>> +#define VIRTIO_TRANSPTQ_CTRL_VQ_NOTIFY    11
>>>> + #define VIRTIO_TRANSPTQ_CTRL_VQ_NOTIFY_GET          0
>>>> +
>>>> +struct virtio_transportq_ctrl_vq_notification_area {
>>>> +       u64 address;
>>>> +       u64 size;
>>>> +};
>>>> +\end{lstlisting}
>>>> +
>>>> +The VIRTIO_TRANSPTQ_CTRL_VQ_NOTIFY_GET is used to get the transport
>>>> +specific address area that can be used to notify a virtqueue. The
>>>> +command-out-data is an u16 of the queue index. The command-in-data
>>>> +contains the \field{address} and the \field{size}
>>>> +of the notification area (as described in struct virtio_transportq_ctrl_vq_notification_area).
>>>> +
>>>> +\devicenormative{\subsubsection}{Virtqueue Notification}{Virtio Transport Options /
>>>> +Virtio Over Transport Virtqueue / Virtqueue Notification}
>>>> +
>>>> +The management device MUST fail the virtqueue notification area information
>>>> +access if \field{device_id} is 0.
>>>> +
>>>> +The management device MUST fail the virtqueue notification area information
>>>> +access if the queue index is out-of-range invalid.
>>> We probably need to say the management device must reserve sufficient
>>> transport specific resources as notification area or we need to have a
>>> fallback of using a dedicated command to notify.
>> OK, will add this
>>>> +
>>>> +\drivernormative{\subsubsection}{Virtqueue Notification}{Virtio Transport Options /
>>>> +Virtio Over Transport Virtqueue / Virtqueue Notification}
>>>> +
>>>> +The driver MAY choose to notify the virtqueue by writing the queue
>>>> +index at address \field{address} which is fetched from the
>>>> +VIRTIO_TRANSPTQ_CTRL_VQ_NOTIFY_GET command.
>>>> +
>>>> +\subsection{Virtqueue State}\label{sec:Virtio Transport Options / Virtio
>>>> +Over Transport Virtqueue / Virtqueue State}
>>>> +
>>>> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
>>>> +the virtqueue state is accessed through the following commands:
>>>> +
>>>> +\begin{lstlisting}
>>>> +#define VIRTIO_TRANSPTQ_CTRL_VQ_STATE            12
>>>> + #define VIRTIO_TRANSPTQ_CTRL_VQ_STATE_GET       0
>>>> + #define VIRTIO_TRANSPTQ_CTRL_VQ_STATE_SET       1
>>>> +\end{lstlisting}
>>>> +
>>>> +\subsubsection{Split Virtqueue State}\label{sec:Virtio Transport Options / Virtio
>>>> +Over Transport Virtqueue / Virtqueue State / Split Virtqueue State}
>>>> +
>>>> +\begin{lstlisting}
>>>> +struct virtio_transportq_ctrl_split_vq_state_set {
>>>> +    u16 queue_index;
>>>> +    u16 avail_index;
>>>> +    u8 padding[4];
>>>> +};
>>>> +\end{lstlisting}
>>>> +
>>>> +The VIRTIO_TRANSPTQ_CTRL_VQ_STATE_GET command can be  used to get the state of a
>>>> +split virtqueue. The command-out-data is the queue index, the command-in-data is the available index of the virtqueue.
>>> We need to be accurate for "available index". E.g we have one for in
>>> the shared memory, so do you mean last_avail_idx?
>> actually there is no definition of "last available index" in the spec,
>> so how about we say:
>> on-device available index?
> Any should be fine but need to explain what does it mean.
>
>>>> +
>>>> +The VIRTIO_TRANSPTQ_CTRL_VQ_STATE_SET command can be used to set the state of a
>>>> +split virtqueue. The command-out-data contains the \field{queue_index} and the
>>>> +available index (\field{avail_index}) for the virtqueue (as described in struct virtio_transportq_ctrl_split_vq_state_set).
>>>> +There is no command-in-data.
>>>> +
>>>> +\subsubsection{Packed Virtqueue State}\label{sec:Virtio Transport Options / Virtio
>>>> +Over Transport Virtqueue / Virtqueue State / Packed Virtqueue State}
>>>> +
>>>> +\begin{lstlisting}
>>>> +struct virtio_transportq_ctrl_packed_vq_state {
>>>> +    u16 queue_index;
>>>> +    u16 avail_counter;
>>>> +    u16 avail_index;
>>>> +    u16 used_counter;
>>>> +    u16 used_index;
>>> Let's simply reuse the structure in the kernel headers.
>>>
>>> Using u16 for wrap_couter seems unnecessary etc.
>> OK, the bit operations are not often.
> It should be very common in the hardware interface.
>
>>>> +    u8 padding[6];
>>>> +};
>>>> +\end{lstlisting}
>>>> +
>>>> +The state of a packed virtqueue includes :\\
>>>> +\field{avail_counter}: last driver ring wrap counter observed by device.\\
>>>> +\field{avail_index}: virtqueue available index.\\
>>>> +\field{used_counter}: device ring wrap counter.\\
>>>> +\field{used_index}: virtqueue used index.
>>>> +
>>>> +The VIRTIO_TRANSPTQ_CTRL_VQ_STATE_GET command can be used to get the state of a packed virtqueue.
>>>> +The command-out-data is the queue index, the command-in-data contains the \field{queue_index},
>>>> +\field{avail_counter}, \field{avail_index}, \field{used_counter} and \field{used_index} of the virtqueue
>>>> +(as described in transportq_ctrl_packed_vq_state).
>>>> +
>>>> +The VIRTIO_TRANSPTQ_CTRL_VQ_STATE_SET command can be used to set the state of a packed virtqueue.
>>>> +The command-out-data contains the \field{queue_index}, \field{avail_counter}, \field{avail_index},
>>>> +\field{used_counter} and \field{used_index} for the virtqueue
>>>> +(as described in transportq_ctrl_packed_vq_state).
>>>> +There is no command-in-data.
>>>> +
>>>> +\devicenormative{\subsubsection}{Virtqueue State}{Virtio Transport Options /
>>>> +Virtio Over Transport Virtqueue / Virtqueue State}
>>>> +
>>>> +The management device MUST fail the virtqueue index access if \field{device_id} is 0.
>>>> +
>>>> +The management device MUST fail the virtqueue index access if \field{queue_index} is out-of-range invalid.
>>>> +
>>>> +\subsection{Virtqueue ASID}\label{sec:Virtio Transport Options / Virtio Over
>>>> +Transport Virtqueue / Virtqueue ASID}
>>>> +
>>>> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
>>>> +the address space id of a virtqueue could be set through the following command:
>>>> +
>>>> +\begin{lstlisting}
>>>> +#define VIRTIO_TRANSPTQ_CTRL_VQ_ASID        13
>>>> + #define VIRTIO_TRANSPTQ_CTRL_VQ_ASID_SET   1
>>>> +
>>>> +struct virtio_transportq_ctrl_vq_asid_set {
>>>> +       u16 queue_index;
>>>> +       u32 asid;
>>>> +       u8 padding[2];
>>> I'd reserve more here. E.g in the future each virtqueue may have a
>>> secondary ASID,
>> like asid[2]? But why? For a device, there may needs a secondary ASID
>> because it
>> may need to access different memory region on behalf of different DMA
>> sources.
>> But for a virtqueue, it only has one purpose, to access the queue
>> buffer, maybe it
>> doesn't need a secondary asid
> E.g for transparent shadow virtqueue.
>
>>>> +};
>>>> +\end{lstlisting}
>>>> +
>>>> +The VIRTIO_TRANSPTQ_CTRL_VQ_ASID_SET command is used to set the address space id(\field{asid})
>>>> +of a virtqueue.
>>>> +
>>>> +The address space id is an identifier
>>> Let's add "transport specific" before "identifier"
>> OK
>>>> of a memory space which is used to convey the address space targeted by the
>>>> +memory accesses
>>> We need to say the ASID is for DMA not here at least.
> I meant you need to explain how address space is used. (e.g it it used
> only for DMA).
>
>> I don't get it, we need to explain what ASID is used for after
>> we introduce it.
>>>> , and to distinguish memory accesses performed by different virtqueues
>>> This may confuse the readers since a single ASID could be shared by virtqueues.
>> so it should be "by different virtqueues or virtqueue groups"
> I think we can refer to the explanation of transport specific ASID here.
>
> Thanks
>

Re: [virtio-comment] [PATCH V3 RESEND 2/4] Introduce the commands set of the transport vq

[Michael S. Tsirkin]

On Wed, Aug 10, 2022 at 04:49:25PM +0800, Zhu, Lingshan wrote:
> > I meant having MSI vectors and let the virtqueue refer to the MSI vectors.
> > 
> > Current design means for a 1024 virtqueues device to store 1024 MSI entries.
> > With the indirection of the MSI vectors array, the device is free to
> > have 1 to 1024 MSI entries.
> I am not sure I get your points, how can the queues decide the MSI entries,
> it should
> be the driver and the platform set the vq's MSI. The driver has to
> decide/control and handle
> the MSI interrupts.
> 
> IMHO there are two options:
> 1)every virtqueue stores its own MSI vector, per-vq MSI vector can get
> better performance
> for sure. But if there are limited MSI resources, the driver may decide to
> share MSI vectors
> among the vqs(1k queues, 1 to 1024 MSI entries), and there should be proper
> interrupt handlers in the driver.
> 
> 2)add a device scope MSI entry(only for vqs, not config interrupt, because
> we don't want to look into config space every time receive a vq interrupt),
> all interrupts from the device are through this MSI entry,
> this is optional, can help save resource. But this is complex, like first
> set a device scope MSI for all vqs, then set a MSI for vq1, in this case,
> we need vq1 use its own MSI to send interrupts, shadows the device MSI. This
> is complex, maybe not worthy, I think the platform(x86, arm) has enough
> MSI resource and it is just one vector per vq.

Consider a PCI device with subfunctions for example. It maintains
MSI-X vectors in an MSI-X table. vq has to select a vector from
that table. Which one?
1- vq can map to a vector number (this is what PCI)
2- vq can just imply vector number e.g. if vector number == vq number
3- vq can include a copy of vector itself

3 will force us to come up with a way to mask vectors in the transport,
not nice.

-- 
MST

Re: [virtio-comment] [PATCH V3 RESEND 2/4] Introduce the commands set of the transport vq

[Zhu, Lingshan]

On 8/10/2022 8:58 PM, Michael S. Tsirkin wrote:
> On Wed, Aug 10, 2022 at 04:49:25PM +0800, Zhu, Lingshan wrote:
>>> I meant having MSI vectors and let the virtqueue refer to the MSI vectors.
>>>
>>> Current design means for a 1024 virtqueues device to store 1024 MSI entries.
>>> With the indirection of the MSI vectors array, the device is free to
>>> have 1 to 1024 MSI entries.
>> I am not sure I get your points, how can the queues decide the MSI entries,
>> it should
>> be the driver and the platform set the vq's MSI. The driver has to
>> decide/control and handle
>> the MSI interrupts.
>>
>> IMHO there are two options:
>> 1)every virtqueue stores its own MSI vector, per-vq MSI vector can get
>> better performance
>> for sure. But if there are limited MSI resources, the driver may decide to
>> share MSI vectors
>> among the vqs(1k queues, 1 to 1024 MSI entries), and there should be proper
>> interrupt handlers in the driver.
>>
>> 2)add a device scope MSI entry(only for vqs, not config interrupt, because
>> we don't want to look into config space every time receive a vq interrupt),
>> all interrupts from the device are through this MSI entry,
>> this is optional, can help save resource. But this is complex, like first
>> set a device scope MSI for all vqs, then set a MSI for vq1, in this case,
>> we need vq1 use its own MSI to send interrupts, shadows the device MSI. This
>> is complex, maybe not worthy, I think the platform(x86, arm) has enough
>> MSI resource and it is just one vector per vq.
> Consider a PCI device with subfunctions for example. It maintains
> MSI-X vectors in an MSI-X table. vq has to select a vector from
> that table. Which one?
> 1- vq can map to a vector number (this is what PCI)
> 2- vq can just imply vector number e.g. if vector number == vq number
> 3- vq can include a copy of vector itself
>
> 3 will force us to come up with a way to mask vectors in the transport,
> not nice.
I am not sure for SIOV SF devices, whether the parent PCI device needs 
to store all
MSI information in its MSI table.

Because these MSI vectors are not for the PCI device to send interrupts, 
they are for
the SFs, this means we even don't need the PCI AIPs.

Every SF's vq should be able to store its own MSI vector. For this 
transport vq,
I think it provides a way to set a vector for the vq, the vector may not 
be a copy of the PCI
vector, nor map to any PCI vectors.

Thanks
>

Re: [virtio-comment] [PATCH V3 RESEND 2/4] Introduce the commands set of the transport vq

[Michael S. Tsirkin]

On Tue, Aug 16, 2022 at 01:55:24PM +0800, Zhu, Lingshan wrote:
> 
> 
> On 8/10/2022 8:58 PM, Michael S. Tsirkin wrote:
> > On Wed, Aug 10, 2022 at 04:49:25PM +0800, Zhu, Lingshan wrote:
> > > > I meant having MSI vectors and let the virtqueue refer to the MSI vectors.
> > > > 
> > > > Current design means for a 1024 virtqueues device to store 1024 MSI entries.
> > > > With the indirection of the MSI vectors array, the device is free to
> > > > have 1 to 1024 MSI entries.
> > > I am not sure I get your points, how can the queues decide the MSI entries,
> > > it should
> > > be the driver and the platform set the vq's MSI. The driver has to
> > > decide/control and handle
> > > the MSI interrupts.
> > > 
> > > IMHO there are two options:
> > > 1)every virtqueue stores its own MSI vector, per-vq MSI vector can get
> > > better performance
> > > for sure. But if there are limited MSI resources, the driver may decide to
> > > share MSI vectors
> > > among the vqs(1k queues, 1 to 1024 MSI entries), and there should be proper
> > > interrupt handlers in the driver.
> > > 
> > > 2)add a device scope MSI entry(only for vqs, not config interrupt, because
> > > we don't want to look into config space every time receive a vq interrupt),
> > > all interrupts from the device are through this MSI entry,
> > > this is optional, can help save resource. But this is complex, like first
> > > set a device scope MSI for all vqs, then set a MSI for vq1, in this case,
> > > we need vq1 use its own MSI to send interrupts, shadows the device MSI. This
> > > is complex, maybe not worthy, I think the platform(x86, arm) has enough
> > > MSI resource and it is just one vector per vq.
> > Consider a PCI device with subfunctions for example. It maintains
> > MSI-X vectors in an MSI-X table. vq has to select a vector from
> > that table. Which one?
> > 1- vq can map to a vector number (this is what PCI)
> > 2- vq can just imply vector number e.g. if vector number == vq number
> > 3- vq can include a copy of vector itself
> > 
> > 3 will force us to come up with a way to mask vectors in the transport,
> > not nice.
> I am not sure for SIOV SF devices, whether the parent PCI device needs to
> store all
> MSI information in its MSI table.
>
> Because these MSI vectors are not for the PCI device to send interrupts,
> they are for
> the SFs, this means we even don't need the PCI AIPs.
> 
> Every SF's vq should be able to store its own MSI vector. For this transport
> vq,
> I think it provides a way to set a vector for the vq, the vector may not be
> a copy of the PCI
> vector, nor map to any PCI vectors.
> 
> Thanks

Doing this will break things like VFIO passthrough of the parent.
So I would say it should at least be optional.

And may I then suggest you peruse the relevant spec and add the
requisite ton of functionality, then go read at least Linux source to
see whether doing things like masking interrupt vectors using a command
that then needs to block waiting for an interrupt is even practical to
integrate in existing OS-es.

I mean you can do all this but you better be prepared to then
reimplement all of functionality in the MSI-X spec.
It was not put there on a whim.

-- 
MST

Re: [virtio-comment] [PATCH V3 RESEND 2/4] Introduce the commands set of the transport vq

[Zhu, Lingshan]

On 8/16/2022 2:21 PM, Michael S. Tsirkin wrote:
> On Tue, Aug 16, 2022 at 01:55:24PM +0800, Zhu, Lingshan wrote:
>>
>> On 8/10/2022 8:58 PM, Michael S. Tsirkin wrote:
>>> On Wed, Aug 10, 2022 at 04:49:25PM +0800, Zhu, Lingshan wrote:
>>>>> I meant having MSI vectors and let the virtqueue refer to the MSI vectors.
>>>>>
>>>>> Current design means for a 1024 virtqueues device to store 1024 MSI entries.
>>>>> With the indirection of the MSI vectors array, the device is free to
>>>>> have 1 to 1024 MSI entries.
>>>> I am not sure I get your points, how can the queues decide the MSI entries,
>>>> it should
>>>> be the driver and the platform set the vq's MSI. The driver has to
>>>> decide/control and handle
>>>> the MSI interrupts.
>>>>
>>>> IMHO there are two options:
>>>> 1)every virtqueue stores its own MSI vector, per-vq MSI vector can get
>>>> better performance
>>>> for sure. But if there are limited MSI resources, the driver may decide to
>>>> share MSI vectors
>>>> among the vqs(1k queues, 1 to 1024 MSI entries), and there should be proper
>>>> interrupt handlers in the driver.
>>>>
>>>> 2)add a device scope MSI entry(only for vqs, not config interrupt, because
>>>> we don't want to look into config space every time receive a vq interrupt),
>>>> all interrupts from the device are through this MSI entry,
>>>> this is optional, can help save resource. But this is complex, like first
>>>> set a device scope MSI for all vqs, then set a MSI for vq1, in this case,
>>>> we need vq1 use its own MSI to send interrupts, shadows the device MSI. This
>>>> is complex, maybe not worthy, I think the platform(x86, arm) has enough
>>>> MSI resource and it is just one vector per vq.
>>> Consider a PCI device with subfunctions for example. It maintains
>>> MSI-X vectors in an MSI-X table. vq has to select a vector from
>>> that table. Which one?
>>> 1- vq can map to a vector number (this is what PCI)
>>> 2- vq can just imply vector number e.g. if vector number == vq number
>>> 3- vq can include a copy of vector itself
>>>
>>> 3 will force us to come up with a way to mask vectors in the transport,
>>> not nice.
>> I am not sure for SIOV SF devices, whether the parent PCI device needs to
>> store all
>> MSI information in its MSI table.
>>
>> Because these MSI vectors are not for the PCI device to send interrupts,
>> they are for
>> the SFs, this means we even don't need the PCI AIPs.
>>
>> Every SF's vq should be able to store its own MSI vector. For this transport
>> vq,
>> I think it provides a way to set a vector for the vq, the vector may not be
>> a copy of the PCI
>> vector, nor map to any PCI vectors.
>>
>> Thanks
> Doing this will break things like VFIO passthrough of the parent.
> So I would say it should at least be optional.
Would you please help explain how this can break VFIO passthrough the 
parent device?

As SIOV spec said, a SF may use MSI-X, but it is just one case. A vq can 
use general
MSI(not limited to PCI) to send interrupts, in this case,
the vq's MSI vectors are not in the PCI MSI-X table.

>
> And may I then suggest you peruse the relevant spec and add the
> requisite ton of functionality, then go read at least Linux source to
> see whether doing things like masking interrupt vectors using a command
> that then needs to block waiting for an interrupt is even practical to
> integrate in existing OS-es.
>
> I mean you can do all this but you better be prepared to then
> reimplement all of functionality in the MSI-X spec.
> It was not put there on a whim.
we provided commands to unmask / mask MSI in the last version.
I can bring them back.

I will add a command to read the pending interrupt bit of a vq

Thanks,
Zhu Lingshan

Re: [virtio-comment] [PATCH V3 RESEND 2/4] Introduce the commands set of the transport vq

[Michael S. Tsirkin]

On Wed, Aug 10, 2022 at 09:56:51AM +0800, Jason Wang wrote:
> On Tue, Aug 9, 2022 at 9:09 PM Zhu, Lingshan <lingshan.zhu@intel.com> wrote:
> >
> >
> >
> > On 8/8/2022 6:04 PM, Jason Wang wrote:
> > > On Fri, Aug 5, 2022 at 6:02 PM Zhu Lingshan <lingshan.zhu@intel.com> wrote:
> > >> This commit introduces the commands set of the
> > >> transport virtqueue, including:
> > >>
> > >> The command to query available resources of the management device
> > >> The commands to create / destroy the managed devices.
> > >> The commands to config the managed devices.
> > >> The commands to config virtqueues of the managed devices.
> > >>
> > >> Signed-off-by: Jason Wang <jasowang@redhat.com>
> > >> Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com>
> > >> ---
> > >>   content.tex | 636 ++++++++++++++++++++++++++++++++++++++++++++++++++++
> > >>   1 file changed, 636 insertions(+)
> > >>
> > >> diff --git a/content.tex b/content.tex
> > >> index c747d21..190f3a0 100644
> > >> --- a/content.tex
> > >> +++ b/content.tex
> > >> @@ -2975,6 +2975,642 @@ \subsection{Format of Commands through Transport Virtqueue}\label{sec:Virtio Tra
> > >>   The \field{device_id} value 0 is used to identify the management device itself.
> > >>
> > >>   \field{class} is an identifier of a set of commands with similar  purposes.
> > >> +
> > >> +\subsection{Available Resource of the Management Device}\label{sec:Virtio Transport Options / Virtio Over Transport Virtqueue / Available Resource of the Management Device}
> > >> +
> > >> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
> > >> +statistical information of available resources on the management device could be fetched
> > >> +by the following command:
> > >> +
> > >> +\begin{lstlisting}
> > >> +#define VIRTIO_TRANSPTQ_CTRL_AVAIL_RES    1 (class)
> > >> + #define VIRTIO_TRANSPTQ_CTRL_AVAIL_RES_GET     0 (command)
> > >> +\end{lstlisting}
> > >> +
> > >> +VIRTIO_TRANSPTQ_CTRL_AVAIL_RES_GET command is used to get the statistical information of the available
> > >> +resource of a management device. There is no command-out-data for VIRTIO_TRANSPTQ_CTRL_AVAIL_RES_GET
> > >> +command. The management device fills command-in-data in the following format:
> > >> +
> > >> +\begin{lstlisting}
> > >> +struct virtio_mgmt_dev_avail_res {
> > >> +        /* The number of management device total remaining available free
> > >> +         * virtqueues for the managed devices
> > >> +         */
> > > It's hard to phrase this sentence.
> > How about "The number of remaining available virtqueues for new managed
> > devices"
> > >
> > >> +        u16 num_avail_vqs;
> > >> +        /* The minimal number of virtqueues for a managed device */
> > >> +        u16 min_dev_vqs;
> > >> +        /* The maximum number of virtqueues for a managed device */
> > > Nit the min/max_dev_vqs are not the resources, so we'd better rename
> > > the virtio_mgmt_dev_avail_res.
> > OK, I will use struct virito_mgmt_dev_info{} in the next version, and
> > the command and its description
> > will change as well.
> > >
> > >> +        u16 max_dev_vqs;
> > >> +        /* The number of managed devices that can be created with min_vqs virtqueues */
> > > Is the part "with min_vqs virtqueues" a must? My understanding of the
> > > above is a hint that the managed device is loosely coupled with the
> > > virtqueue resources.
> > Yes, the number of the number of total managed devices can be limited by
> > other resources, like the filters.
> > >
> > > So it would be possible that num_avail_dev = 0 but num_avail_vqs not?
> > >
> > > Btw, "num" prefix is probably not a must.
> > OK
> > >
> > >> +        u16 num_avail_dev;
> > > To be consistent, let's use "num_avail_devs"?
> > I will remove all num_ prefix, and it should be avail_devs
> > >
> > >> +};
> > >> +\end{lstlisting}
> > >> +
> > >> +\field{max_avail_dev} MAY not equal to \field{num_avail_vqs} divided by \field{min_vqs}
> > >> +due to the limitations of of other resources.
> > > two "of".
> > will fix
> > >
> > > Keywords like "MAY" should go to normatives part.
> > OK, so I will only use "may", so it is a normal description.
> > >
> > >> +
> > >> +\devicenormative{\subsubsection}{Available Resource of the Management Device}{Virtio Transport Options /
> > >> +Virtio Over Transport Virtqueue / Available Resource of the Management Device}
> > >> +
> > >> +The management device MUST fail VIRTIO_TRANSPTQ_CTRL_AVAIL_RES_GET command if \field{device_id} is not 0
> > >> +
> > >> +\drivernormative{\subsubsection}{Available Resource of the Management Device}{Virtio Transport Options /
> > >> +Virtio Over Transport Virtqueue / Available Resource of the Management Device}
> > >> +
> > >> +The management device driver MUST use 0 as \field{device_id} for
> > >> +VIRTIO_TRANSPTQ_CTRL_AVAIL_RES_GET command.
> > >> +
> > >> +\subsection{Create and Destroy Managed Devices}\label{sec:Virtio Transport Options / Virtio Over Transport Virtqueue / Create and Destroy Managed Devices}
> > >> +
> > >> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
> > >> +managed devices must be created and destroyed through the transport virtqueue.
> > >> +
> > >> +\begin{lstlisting}
> > >> +#define VIRTIO_TRANSPORTQ_CTRL_DEV    2
> > >> + #define VIRTIO_TRANSPORTQ_CTRL_DEV_CREATE        0
> > >> + #define VIRTIO_TRANSPORTQ_CTRL_DEV_DESTROY       1
> > >> +
> > >> +struct virtio_transportq_ctrl_dev_attribute {
> > >> +       u64 device_features[2];
> > > Let's reserve more bits for features here as we are about to reach 64.
> > OK, it looks 256bytes may be enough.
> 
> It's better to match the parent/management device, but maybe we can
> start from something that is fine for the next 5 years. So I'd rather
> go with even more than this like device_feautres[10].

Let's just specify the size. BTW should be
u8 device_features_dwords;
u32 device_features[N];


> 
> > >
> > >> +       u32 virtio_device_id;

Starting with virtio_device_id seems cleaner.

Do we also want a vendor id? Or just assume same vendor as parent?

> > >> +       u8 dev_config[];
> > > To be consistent, let's use one of "device" or "dev" in the same structure.
> > sure, device_config[]
> > >
> > >> +};
> > >> +\end{lstlisting}
> > >> +
> > >> +The VIRTIO_TRANSPORTQ_CTRL_DEV_CREATE command is used to create a managed device.
> > >> +
> > >> +As described in struct virtio_transportq_ctrl_dev_attribute, the command-out-data for VIRTIO_TRANSPTQ_CTRL_DEV_CREATE includes:
> > >> +\begin{itemize*}
> > >> +\item \field{device_features}: the 128-bit device feature bits (defined in section \ref{sec:Basic Facilities of a Virtio Device / Feature Bits})
> > >> +of the managed device.
> > >> +\item \field{virtio_device_id}: the virtio device id defined in Chapter \ref{sec:Device Types}.
> > >> +\item \field{dev_config}: the device type specific configurations. E.g., for a virtio-net device,
> > >> +it is struct virtio_net_config in section \ref{sec:Device Types / Network Device / Device configuration layout};
> > >> +for a virtio-block device, it is struct virtio_blk_config in section \ref{sec:Device Types / Block Device / Device configuration layout}
> > >> +\end{itemize*}
> > >> +
> > >> +When succeed, the device returns a 64 bits UUID as an unique identifier of the created managed device in command-in-data.
> > > We don't need to be universally unique here.
> > Why? I think it should be unique to identify a device, or are you
> > suggesting to remove "as an unique identifier",
> > just UUID is enough?
> 
> UUID means it's unique in the universal. If it means ID for managed
> device in PF1 must differ from ID for managed device in PF2. This is
> not needed.
> 
> So if we want to be flexible, we can allow the ID to be allocated by
> the driver as the original RFC did. The driver is free to allocate
> anything it want (UID or UUID).
> 
> > >
> > >> +
> > >> +The VIRTIO_TRANSPORTQ_CTRL_DEV_DESTROY command is used to destroy a
> > >> +managed device which is identified by its 64 bits UUID
> > >> +\field{device_id}. There's no command-in-data nor command-out-data for
> > >> +VIRTIO_TRANSPTQ_CTRL_DEV_DESTROY command.
> > >> +
> > >> +\devicenormative{\subsubsection}{Create and Destroy Managed Devices}{Virtio Transport Options / Virtio Over Transport Virtqueue / Create and Destroy Managed Devices}
> > >> +
> > >> +The management device MUST fail VIRTIO_TRANSPORTQ_CTRL_DEV_CREATE command
> > >> +if \field{device_id} is not 0.
> > >> +
> > >> +The management device MUST fail VIRTIO_TRANSPORTQ_CTRL_DEV CREATE command
> > >> +if \field{device_features} exceeds the features that can be provided from the management device.
> > > So PCI has le32 device_feature_select. How to make sure it can match
> > > the capacity of device_features[] above?
> > we present all feature bits in device_features[], the spec says the
> > feature bits are valid up to bit 127,
> > 128 and above are reserved. So as you suggest, we can use u64
> > device_features[4], for 128-bit feature bits
> > and another 128 bits for further extensions.
> > >
> > >> +
> > >> +The management device MUST fail VIRTO_TRANSPORTQ_CTRL_DEV_DESTROY command
> > >> +if \field{device_id} is 0.
> > >> +
> > >> +\drivernormative{\subsubsection}{Create and Destroy Managed Devices}{Virtio Transport Options / Virtio Over Transport Virtqueue / Create and Destroy Managed Devices}
> > >> +
> > >> +The management device driver MUST use 0 as \field{device_id} for
> > >> +TRANSPORTQ_CTRL_DEV_CREATE command.
> > >> +
> > >> +\subsection{Features Negotiation}\label{sec:Virtio Transport Options / Virtio
> > >> +Over Transport Virtqueue / Features Negotiation}
> > >> +
> > >> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
> > >> +the features negotiation of managed devices is done by the
> > >> +following commands:
> > >> +
> > >> +\begin{lstlisting}
> > >> +#define VIRTIO_TRANSPTQ_CTRL_FEAT   3
> > >> + #define VIRTIO_TRANSPTQ_CTRL_FEAT_DEVICE_GET        0
> > >> + #define VIRTIO_TRANSPTQ_CTRL_FEAT_DRIVER_SET        1
> > >> + #define VIRTIO_TRANSPTQ_CTRL_FEAT_DRIVER_GET        2
> > >> +\end{lstlisting}
> > >> +
> > >> +The VIRTIO_TRANSPTQ_CTRL_FEAT_DEVICE_GET command is used to get the features offered
> > >> +by a managed device. The command-in-data is the 128-bit feature bits
> > >> +(defined in section \ref{sec:Basic Facilities of a Virtio Device / Feature Bits}).
> > > Need to make sure the capacity of the feature bit matches what the
> > > transport of the management device can provide.
> > I will provide a structure for the features, so both device and driver
> > would be aligned on
> > the size of the feature bits, like
> >
> > struct virtio_transportq_ctrl_dev_features {
> >      u64 features[4];
> > };
> > >
> > >> +There is no command-out-data.
> > >> +
> > >> +The VIRTIO_TRANSPTQ_CTRL_FEAT_DRIVER_SET command is for the driver to accept feature
> > >> +bits offered by the managed device. The command-out-data is the 128-bit feature bits
> > >> +passed to the managed device. There is no command-in-data.
> > >> +
> > >> +The VIRTIO_TRANSPTQ_CTRL_FEAT_DRIVER_GET command is used to get the features accepted
> > >> +by both the managed device driver and the managed device. The command-in-data is the
> > >> +128-bit feature bits. There is no command-out-data.
> > >> +
> > >> +\devicenormative{\subsubsection}{Features Negotiation}{Virtio Transport Options /
> > >> +Virtio Over Transport Virtqueue / Features Negotiation}
> > >> +
> > >> +The management device MUST fail VIRTIO_TRANSPTQ_F_CTRL_FEAT class commands if
> > >> +\field{device_id} is 0.
> > >> +
> > >> +\subsection{Device Status}\label{sec:Virtio Transport Options / Virtio Over
> > >> +Transport Virtqueue / Device Status}
> > >> +
> > >> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
> > >> +the device status of a managed device can be accessed by the following
> > >> +commands:
> > >> +
> > >> +\begin{lstlisting}
> > >> +#define VIRTIO_TRANSPTQ_CTRL_STATUS    4
> > >> + #define VIRTIO_TRANSPTQ_CTRL_STATUS_GET        0
> > >> + #define VIRTIO_TRANSPTQ_CTRL_STATUS_SET        1
> > >> +\end{lstlisting}
> > >> +
> > >> +The VIRTIO_TRANSPTQ_CTRL_STATUS_GET command is used to get the device status of
> > >> +a managed device. The command-in-data is the 8-bit status
> > >> +returned from the device. There's no command-out-data for this
> > >> +command.
> > >> +
> > >> +The VIRTIO_TRANSPTQ_CTRL_STATUS_SET command is used to set the device status of
> > >> +a managed device. The command-out-data is the 8-bit status
> > >> +to set to the device. There's no command-in-data for this command.
> > >> +
> > >> +\devicenormative{\subsubsection}{Device Status}{Virtio Transport Options / Virtio
> > >> +Over Transport Virtqueue / Device Status}
> > >> +
> > >> +The management device MUST reset the managed device when 0
> > >> +is set via VIRTIO_TRANSPTQ_CTRL_STATUS_SET, the success of this
> > >> +command demonstrates the success of the reset.
> > >> +
> > >> +The management device MUST present 0 through
> > >> +VIRTIO_TRANSPTQ_CTRL_STATUS_GET once the reset is successfully done.
> > >> +
> > >> +The management device MUST fail the device status access if
> > >> +\field{device_id} is 0.
> > >> +
> > >> +\drivernormative{\subsubsection}{Device Status}{Virtio Transport Options / Virtio
> > >> +Over Transport Virtqueue / Device Status}
> > >> +
> > >> +After writing 0 via VIRTIO_TRANSPTQ_CTRL_STATUS_SET, the driver MUST wait
> > >> +for the success of the command before re-initializing the device.
> > >> +
> > >> +\subsection{Device Generation}\label{sec:Virtio Transport Options / Virtio Over Transport Virtqueue / Device Generation}
> > >> +
> > >> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
> > > Btw, since we're under the transport options (which implies the
> > > transport virtqueue), maybe we can drop the above for each command.
> > OK, can do, and I will describe this in the Basic Concept section.
> > >
> > >> +the device generation count can be read from the following commands:
> > >> +
> > >> +\begin{lstlisting}
> > >> +#define VIRTIO_TRANSPTQ_CTRL_GENERATION    5
> > >> + #define VIRTIO_TRANSPTQ_CTRL_GENERATION_GET        0
> > >> +\end{lstlisting}
> > >> +
> > >> +The VIRTIO_TRANSPTQ_CTRL_GENERATION_GET command is used to get the device configuration generation count
> > >> +of the managed device. The command-in-data is the one byte device
> > >> +generation returned from the device. There's no command-out-data for
> > >> +this command.
> > >> +
> > >> +\devicenormative{\subsubsection}{Device Generation}{Virtio Transport Options / Virtio Over Transport Virtqueue / Device Generation}
> > >> +
> > >> +The managed device MUST present a changed generation count after the driver
> > >> +has read any device-specific configuration values if the values have been changed
> > >> +during the last read.
> > >> +
> > >> +The management device MUST fail the device generation access if \field{device_id} is 0.
> > >> +
> > >> +\subsection{Device Specific Configuration}\label{sec:Virtio Transport Options /
> > >> +Virtio Over Transport Virtqueue / Device Specific Configuration}
> > >> +
> > >> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
> > >> +the device config space contents of a managed device can be accessed through
> > >> +the following commands:
> > >> +
> > >> +\begin{lstlisting}
> > >> +#define VIRTIO_TRANSPTQ_CTRL_CONFIG    6
> > >> +  #define VIRTIO_TRANSPTQ_CTRL_CONFIG_GET    0
> > >> +  #define VIRTIO_TRANSPTQ_CTRL_CONFIG_SET    1
> > >> +
> > >> +struct virtio_transportq_ctrl_dev_config_get {
> > >> +       u32 offset;
> > >> +       u32 size;
> > >> +};
> > >> +
> > >> +struct virtio_transportq_ctrl_dev_config_set {
> > >> +       u32 offset;
> > >> +       u32 size;
> > >> +       u8  data[];
> > >> +};
> > >> +\end{lstlisting}
> > >> +
> > >> +The VIRTIO_TRANSPTQ_CTRL_CONFIG_GET command is used to read data from the
> > >> +device configuration space.  The command-out-data is the \field{offset}
> > >> +from the start of the config space and the \field{size}
> > >> +of the data (as described in struct virtio_transportq_ctrl_dev_config_get).
> > >> +The command-in-data is an array of the data that read from the config space.
> > >> +
> > >> +The VIRTIO_TRANSPTQ_CTRL_CONFIG_SET command is used to write data to the device
> > >> +configuration space. The command-out-data contains the
> > >> +\field{offset} from the start of the config space, the \field{size} of the data and
> > >> +the \field{data} to be written (as described in struct virtio_transportq_ctrl_dev_config_set).
> > >> +There's no command-in-data for this command.
> > >> +
> > >> +\devicenormative{\subsubsection}{Device Specific Configuration}{Virtio Transport
> > >> +Options / Virtio Over Transport Virtqueue / Device Specific Configuration}
> > >> +
> > >> +The management device MUST fail the device configuration space access
> > >> +if the driver access the range which is outside the config space.
> > > "accesses"
> > yes
> > >
> > >> +
> > >> +The management device MUST fail the device configuration space access
> > >> +if \field{device_id} is 0.
> > >> +
> > >> +\subsection{MSI Configuration}\label{sec:Virtio Transport Options / Virtio Over
> > >> +Transport Virtqueue / MSI Configuration}
> > >> +
> > >> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
> > >> +the MSI entries of a managed device can be accessed  through the following command:
> > > Rethink of this, it implies a per virtqueue MSI storage which I'm not
> > > sure is expensive or not. Do we need an indirection layer as PCI did?
> > I think we need this for performance, or are you suggesting to add a
> > device scope MSI?
> 
> I meant having MSI vectors and let the virtqueue refer to the MSI vectors.
> 
> Current design means for a 1024 virtqueues device to store 1024 MSI entries.
> With the indirection of the MSI vectors array, the device is free to
> have 1 to 1024 MSI entries.

BTW we need a way to mask and change vectors too.
This happens under all kind of locks I am not sure
how practical it is to do this through a command.
Pls explore the linux code for this.


> > >
> > >> +
> > >> +\begin{lstlisting}
> > >> +#define VIRTIO_TRANSPTQ_CTRL_MSI    7
> > >> + #define VIRTIO_TRANSPTQ_CTRL_MSI_VQ_GET        0
> > >> + #define VIRTIO_TRANSPTQ_CTRL_MSI_VQ_SET        1
> > >> + #define VIRTIO_TRANSPTQ_CTRL_MSI_VQ_ENABLE     2
> > >> + #define VIRTIO_TRANSPTQ_CTRL_MSI_CONFIG_GET    3
> > >> + #define VIRTIO_TRANSPTQ_CTRL_MSI_CONFIG_WET    4
> > > I think you mean CONFIG_SET here.
> > yes
> > >
> > >> + #define VIRTIO_TRANSPTQ_CTRL_MSI_CONFIG_ENABLE 5
> > >> +
> > >> +struct virtio_transportq_ctrl_msi_vq_config {
> > >> +       u16 queue_index;
> > >> +       u64 address;
> > >> +       u32 data;
> > >> +       u8 padding[2];
> > >> +};
> > >> +
> > >> +struct virtio_transportq_ctrl_msi_vq_enable {
> > >> +       u16 queue_index;
> > >> +       u8 enable;
> > >> +       u8 padding[5];
> > >> +};
> > >> +
> > >> +struct virtio_transportq_ctrl_msi_config {
> > >> +       u64 address;
> > >> +       u32 data;
> > >> +       u8 padding[4];
> > >> +};
> > >> +\end{lstlisting}
> > >> +
> > >> +The VIRTIO_TRANSPTQ_CTRL_MSI_VQ_GET command is used to read the MSI entry of a
> > >> +specific virtqueue. The command-out-data is the queue index. The command-in-data contains
> > >> +\field{queue_index}, the \field{address} and \field{data}
> > >> +(as described in struct virtio_transportq_ctrl_msi_vq_config).
> > > Let's split or reuse the virtio_transportq_ctrl_msi_config since
> > > virtio_transportq_ctrl_msi_vq_config will not be used a as single
> > > buffer.
> > I don't get it, if we re-use the struct virtio_transportq_ctrl_msi_config,
> > it would be like:
> >
> > struct virtio_transportq_ctrl_msi {
> >      u64 addr;
> >      u32 data;
> > };
> >
> > struct virtio_transportq_ctrl_msi_vq_config {
> >      u16 queue_index;
> 
> What I meant is, for GET. queue_index is out buffer, and re-use
> virtio_transportq_ctrl_msi is in buffer.
> 
> >      struct virtio_transportq_ctrl_msi;
> > };
> >
> > it is still two structs and a struct in another struct.
> > >
> > >> +
> > >> +
> > >> +The VIRTIO_TRANSPTQ_CTRL_MSI_VQ_SET command is used to set the MSI entry for a
> > >> +specific virtqueue. The command-out-data is the \field{queue_index},
> > >> +\field{address} and \field{data} (as described in struct virtio_transportq_ctrl_msi_vq_config).
> > >> +There is no command-in-data.
> > >> +
> > >> +The VIRTIO_TRANSPTQ_CTRL_MSI_VQ_ENABLE command is used to enable or disable
> > >> +the MSI interrupt for a specific virtqueue. The command-out-data is the
> > >> +\field{queue_index} and a byte \field{enable} which representing ENABLE or DISABLE the MSI.
> > > Should be "represents".
> > OK
> > >
> > > Btw, is there a chance that we may lose an interrupt?
> > No, I think we won't lose any interrupts, because once disabled, it can
> > not send new interrupts,
> > but the last interrupt send by the MSI entry is still there, it's in-band.
> 
> So consider driver is changing affinity what it did is:
> 
> disable
> write MSI
> enable
> 
> What if there's an interrupt coming in the middle? Does this mean the
> device needs to implement the pending bit internally?

I suspect so.

> FYI we had similar discussion in the past:
> 
> https://lists.oasis-open.org/archives/virtio-dev/202002/msg00027.html

Yes MSI for MMIO got blocked on the same set of issues.


> > >
> > >> +(as described in struct virtio_transportq_ctrl_msi_vq_enable).
> > >> +
> > >> +\begin{lstlisting}
> > >> +#define VIRTIO_TRANSPTQ_CTRL_MSI_VQ_ENABLE     1
> > >> +#define VIRTIO_TRANSPTQ_CTRL_MSI_VQ_DISABLE    0
> > >> +\end{lstlisting}
> > >> +
> > >> +The VIRTIO_TRANSPTQ_CTRL_MSI_CONFIG_GET command is used to read the MSI entry
> > >> +of the config interrupt. The command-in-data contains the \field{address} and
> > >> +\field{data} (as described in struct virtio_transportq_ctrl_msi_config).
> > >> +There is no command-out-data.
> > >> +
> > >> +The VIRTIO_TRANSPTQ_CTRL_MSI_CONFIG_SET command is used to set the MSI entry
> > >> +for the config interrupt. The command-out-data is the \field{address} and
> > >> +\field{data} (as described in struct virtio_transportq_ctrl_msi_config).
> > >> +There is no command-in-data.
> > >> +
> > >> +The VIRTIO_TRANSPTQ_CTRL_MSI_CONFIG_ENABLE command is used to enable or disable
> > >> +the MSI interrupt for config space. The command-out-data is a byte which representing
> > >> +ENABLE or DISABLE the config MSI.
> > >> +There is no command-in-data
> > >> +
> > >> +\begin{lstlisting}
> > >> +#define VIRTIO_TRANSPTQ_CTRL_MSI_CFG_ENABLE     1
> > >> +#define VIRTIO_TRANSPTQ_CTRL_MSI_CFG_DISABLE    0
> > >> +\end{lstlisting}
> > >> +
> > >> +\devicenormative{\subsubsection}{MSI Configuration}{Virtio Transport Options / Virtio
> > >> +ver Transport Virtqueue / MSI Configuration}
> > >> +
> > >> +The managed device MUST disable the MSI interrupts for both virtqueues and config space
> > >> +upon a reset.
> > >> +
> > >> +Once a MSI entry is disabled, the managed device MUST not send any interrupts
> > >> +by this MSI entry.
> > > MSI is stored based on virtqueue and config space, so I'm not quite
> > > sure "MSI entry" is accurate here.
> > I will replace all "MSI entry" with "MSI vector"
> > >
> > >> +
> > >> +\drivernormative{\subsubsection}{MSI Configuration}{Virtio Transport Options / Virtio
> > >> +Over Transport Virtqueue / MSI Configuration}
> > >> +
> > >> +The driver MUST allocate transport or platform specific MSI entries
> > >> +for both virtqueues and config space if it wants to use interrupts.
> > >> +
> > >> +The driver MAY choose to disable the MSI if polling mode is used.
> > >> +
> > >> +\subsection{Virtqueue Address}\label{sec:Virtio Transport Options / Virtio Over
> > >> +Transport Virtqueue / Virtqueue Address}
> > >> +
> > >> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
> > >> +the addresses of a specific virtqueue are accessed through the following command:
> > >> +
> > >> +\begin{lstlisting}
> > >> +#define VIRTIO_TRANSPTQ_CTRL_VQ_ADDR    8
> > >> + #define VIRTIO_TRANSPTQ_CTRL_VQ_ADDR_SET       1
> > >> +
> > >> +struct virtio_transportq_ctrl_vq_addr {
> > >> +       u16 queue_index;
> > >> +       u64 descriptor_area;
> > >> +       u64 device_area;
> > >> +       u64 driver_area;
> > >> +       u8 padding[6];
> > >> +};
> > >> +\end{lstlisting}
> > >> +
> > >> +The VIRTIO_TRANSPTQ_CTRL_VQ_ADDR_SET command is used to set the addresses of a specified
> > >> +virtqueue. The command-out-data contains the \field{queue_index}, the addresses of \field{device_area},
> > >> +\field{descriptor_area} and \field{driver_area} (as described in struct
> > >> +virtio_transportq_ctrl_vq_addr). There's no command-in-data.
> > >> +
> > >> +\devicenormative{\subsubsection}{Virtqueue Address}{Virtio Transport Options / Virtio
> > >> +Over Transport Virtqueue / Virtqueeu Address}
> > >> +
> > >> +The management device MUST fail the commands of class
> > >> +VIRTIO_TRANSPTQ_CTRL_VQ_ADDR if \field{device_id} is 0.
> > >> +
> > >> +The management device MUST fail the commands of class
> > >> +VIRTIO_TRANSPTQ_CTRL_VQ_ADDR if \field{queue_index} is out-of-range invalid.
> > >> +
> > >> +\subsection{Virtqueue Status}\label{sec:Virtio Transport Options / Virtio Over
> > >> +Transport Virtqueue / Virtqueue Status}
> > >> +
> > >> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
> > >> +virtqueue status is accessed through the following command:
> > >> +
> > >> +\begin{lstlisting}
> > >> +#define VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE    9
> > >> + #define VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_GET       0
> > >> + #define VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_SET       1
> > >> +
> > >> +struct virtio_transportq_ctrl_vq_status_set {
> > >> +       u16 queue_index;
> > >> +       u8 status;
> > >> +       u8 padding[5];
> > >> +
> > >> +#define VIRTIO_TRANSPTQ_VQ_ENABLE     1
> > >> +#define VIRTIO_TRANSPTQ_VQ_DISABLE    0
> > >> +
> > >> +};
> > >> +
> > >> +\end{lstlisting}
> > >> +
> > >> +The VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_GET is used to get the status of a
> > >> +specific virtqueue. The command-out-data is the queue
> > >> +index. The command-in-data is the virtqueue status.
> > >> +
> > >> +The VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_SET command is used to set the status of a
> > >> +specific virtqueue. The command-out-data is the \field{queue_index} and the
> > >> +\field{status} representing ENABLE or DISABLE that is set to the virtqueue
> > >> +(as described in struct virtio_transportq_ctrl_vq_status_set).
> > >> +There's no command-in-data.
> > >> +
> > >> +\devicenormative{\subsubsection}{Virtqueue Status}{Virtio Transport Options / Virtio
> > >> +Over Transport Virtqueue / Virtqueue Status}
> > >> +
> > >> +When disabled, the managed device MUST stop processing requests from
> > >> +this virtqueue.
> > > Is this a hint that the driver can write VIRTIO_TRANSPTQ_VQ_DISABLE to
> > > status? If yes, what's the difference between this and virtqueue
> > > reset?
> > I think disabling a queue is like "set queue_enable = 0", once disabled,
> > the device should not reset the queue configurations like the indexes
> > and addresses,
> > it just freeze the queue.
> 
> Then you need to define what "freeze" means. To reduce the changset,
> I'd suggest disallowing such writing as what PCI did.
> 
> If needed, we can re-enable it in the future.
> 
> >
> > When reset a queue, like the spec says, the device needs to reset all queue
> > configs.
> > >
> > >> +
> > >> +The management device MUST present a 0 via
> > >> +VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_GET upon a reset of the managed device.
> > >> +
> > >> +The management device MUST fail the virtqueue status access if
> > >> +\field{device_id} is 0.
> > >> +
> > >> +The management device MUST fail the virtqueue status access if
> > >> +the queue_index is out-of-range invalid.
> > >> +
> > >> +
> > >> +\drivernormative{\subsubsection}{Virtqueue Status}{Virtio Transport Options / Virtio
> > >> +Over Transport Virtqueue / Virtqueue Status}
> > >> +
> > >> +The driver MUST configure other virtqueue fields before enabling
> > >> +the virtqueue with VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_SET.
> > >> +
> > >> +\subsection{Virtqueue Size}\label{sec:Virtio Transport Options / Virtio Over
> > >> +Transport Virtqueue / Virtqueue Size}
> > >> +
> > >> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
> > >> +virtqueue size is accessed through the following command:
> > >> +
> > >> +\begin{lstlisting}
> > >> +#define VIRTIO_TRANSPTQ_CTRL_VQ_SIZE    10
> > >> + #define VIRTIO_TRANSPTQ_CTRL_VQ_SIZE_GET       0
> > >> + #define VIRTIO_TRANSPTQ_CTRL_VQ_SIZE_SET       1
> > >> +
> > >> +struct virtio_transportq_ctrl_vq_size_set {
> > >> +       u16 queue_index;
> > >> +       u16 size;
> > >> +       u8 padding[4];
> > >> +};
> > >> +\end{lstlisting}
> > >> +
> > >> +The VIRTIO_TRANSPTQ_CTRL_VQ_SIZE_GET command is used to get the virtqueue
> > >> +size. On reset, the maximum queue size supported by the device is
> > >> +returned. The command-out-data is the queue index. The
> > >> +command-in-data is an 16-bit queue size.
> > >> +
> > >> +The VIRTIO_TRANSPTQ_CTRL_VQ_SIZE_SET command is used to set the virtqueue
> > >> +size. The command-out-data is the \field{queue_index} and the \field{size}
> > >> +of the virtqueue (as described in struct virtio_transportq_ctrl_vq_size_set).
> > >> +There's no command-in-data.
> > >> +
> > >> +\devicenormative{\subsubsection}{Virtqueue Status}{Virtio Transport Options / Virtio
> > >> +Over Transport Virtqueue / Virtqueue Size}
> > >> +
> > >> +The management device MUST fail the virtqueue size access if
> > >> +\field{device_id} is 0.
> > >> +
> > >> +The management device MUST fail the virtqueue size access if
> > >> +the queue index is out-of-range invalid.
> > >> +
> > >> +\subsection{Virtqueue Notification}\label{sec:Virtio Transport Options / Virtio
> > >> +Over Transport Virtqueue / Virtqueue Notification}
> > >> +
> > >> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
> > >> +the virtqueue notification area information can be get through the following commands:
> > >> +
> > >> +\begin{lstlisting}
> > >> +#define VIRTIO_TRANSPTQ_CTRL_VQ_NOTIFY    11
> > >> + #define VIRTIO_TRANSPTQ_CTRL_VQ_NOTIFY_GET          0
> > >> +
> > >> +struct virtio_transportq_ctrl_vq_notification_area {
> > >> +       u64 address;
> > >> +       u64 size;
> > >> +};
> > >> +\end{lstlisting}
> > >> +
> > >> +The VIRTIO_TRANSPTQ_CTRL_VQ_NOTIFY_GET is used to get the transport
> > >> +specific address area that can be used to notify a virtqueue. The
> > >> +command-out-data is an u16 of the queue index. The command-in-data
> > >> +contains the \field{address} and the \field{size}
> > >> +of the notification area (as described in struct virtio_transportq_ctrl_vq_notification_area).
> > >> +
> > >> +\devicenormative{\subsubsection}{Virtqueue Notification}{Virtio Transport Options /
> > >> +Virtio Over Transport Virtqueue / Virtqueue Notification}
> > >> +
> > >> +The management device MUST fail the virtqueue notification area information
> > >> +access if \field{device_id} is 0.
> > >> +
> > >> +The management device MUST fail the virtqueue notification area information
> > >> +access if the queue index is out-of-range invalid.
> > > We probably need to say the management device must reserve sufficient
> > > transport specific resources as notification area or we need to have a
> > > fallback of using a dedicated command to notify.
> > OK, will add this
> > >
> > >> +
> > >> +\drivernormative{\subsubsection}{Virtqueue Notification}{Virtio Transport Options /
> > >> +Virtio Over Transport Virtqueue / Virtqueue Notification}
> > >> +
> > >> +The driver MAY choose to notify the virtqueue by writing the queue
> > >> +index at address \field{address} which is fetched from the
> > >> +VIRTIO_TRANSPTQ_CTRL_VQ_NOTIFY_GET command.
> > >> +
> > >> +\subsection{Virtqueue State}\label{sec:Virtio Transport Options / Virtio
> > >> +Over Transport Virtqueue / Virtqueue State}
> > >> +
> > >> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
> > >> +the virtqueue state is accessed through the following commands:
> > >> +
> > >> +\begin{lstlisting}
> > >> +#define VIRTIO_TRANSPTQ_CTRL_VQ_STATE            12
> > >> + #define VIRTIO_TRANSPTQ_CTRL_VQ_STATE_GET       0
> > >> + #define VIRTIO_TRANSPTQ_CTRL_VQ_STATE_SET       1
> > >> +\end{lstlisting}
> > >> +
> > >> +\subsubsection{Split Virtqueue State}\label{sec:Virtio Transport Options / Virtio
> > >> +Over Transport Virtqueue / Virtqueue State / Split Virtqueue State}
> > >> +
> > >> +\begin{lstlisting}
> > >> +struct virtio_transportq_ctrl_split_vq_state_set {
> > >> +    u16 queue_index;
> > >> +    u16 avail_index;
> > >> +    u8 padding[4];
> > >> +};
> > >> +\end{lstlisting}
> > >> +
> > >> +The VIRTIO_TRANSPTQ_CTRL_VQ_STATE_GET command can be  used to get the state of a
> > >> +split virtqueue. The command-out-data is the queue index, the command-in-data is the available index of the virtqueue.
> > > We need to be accurate for "available index". E.g we have one for in
> > > the shared memory, so do you mean last_avail_idx?
> > actually there is no definition of "last available index" in the spec,
> > so how about we say:
> > on-device available index?
> 
> Any should be fine but need to explain what does it mean.
> 
> > >
> > >> +
> > >> +The VIRTIO_TRANSPTQ_CTRL_VQ_STATE_SET command can be used to set the state of a
> > >> +split virtqueue. The command-out-data contains the \field{queue_index} and the
> > >> +available index (\field{avail_index}) for the virtqueue (as described in struct virtio_transportq_ctrl_split_vq_state_set).
> > >> +There is no command-in-data.
> > >> +
> > >> +\subsubsection{Packed Virtqueue State}\label{sec:Virtio Transport Options / Virtio
> > >> +Over Transport Virtqueue / Virtqueue State / Packed Virtqueue State}
> > >> +
> > >> +\begin{lstlisting}
> > >> +struct virtio_transportq_ctrl_packed_vq_state {
> > >> +    u16 queue_index;
> > >> +    u16 avail_counter;
> > >> +    u16 avail_index;
> > >> +    u16 used_counter;
> > >> +    u16 used_index;
> > > Let's simply reuse the structure in the kernel headers.
> > >
> > > Using u16 for wrap_couter seems unnecessary etc.
> > OK, the bit operations are not often.
> 
> It should be very common in the hardware interface.
> 
> > >
> > >> +    u8 padding[6];
> > >> +};
> > >> +\end{lstlisting}
> > >> +
> > >> +The state of a packed virtqueue includes :\\
> > >> +\field{avail_counter}: last driver ring wrap counter observed by device.\\
> > >> +\field{avail_index}: virtqueue available index.\\
> > >> +\field{used_counter}: device ring wrap counter.\\
> > >> +\field{used_index}: virtqueue used index.
> > >> +
> > >> +The VIRTIO_TRANSPTQ_CTRL_VQ_STATE_GET command can be used to get the state of a packed virtqueue.
> > >> +The command-out-data is the queue index, the command-in-data contains the \field{queue_index},
> > >> +\field{avail_counter}, \field{avail_index}, \field{used_counter} and \field{used_index} of the virtqueue
> > >> +(as described in transportq_ctrl_packed_vq_state).
> > >> +
> > >> +The VIRTIO_TRANSPTQ_CTRL_VQ_STATE_SET command can be used to set the state of a packed virtqueue.
> > >> +The command-out-data contains the \field{queue_index}, \field{avail_counter}, \field{avail_index},
> > >> +\field{used_counter} and \field{used_index} for the virtqueue
> > >> +(as described in transportq_ctrl_packed_vq_state).
> > >> +There is no command-in-data.
> > >> +
> > >> +\devicenormative{\subsubsection}{Virtqueue State}{Virtio Transport Options /
> > >> +Virtio Over Transport Virtqueue / Virtqueue State}
> > >> +
> > >> +The management device MUST fail the virtqueue index access if \field{device_id} is 0.
> > >> +
> > >> +The management device MUST fail the virtqueue index access if \field{queue_index} is out-of-range invalid.
> > >> +
> > >> +\subsection{Virtqueue ASID}\label{sec:Virtio Transport Options / Virtio Over
> > >> +Transport Virtqueue / Virtqueue ASID}
> > >> +
> > >> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
> > >> +the address space id of a virtqueue could be set through the following command:
> > >> +
> > >> +\begin{lstlisting}
> > >> +#define VIRTIO_TRANSPTQ_CTRL_VQ_ASID        13
> > >> + #define VIRTIO_TRANSPTQ_CTRL_VQ_ASID_SET   1
> > >> +
> > >> +struct virtio_transportq_ctrl_vq_asid_set {
> > >> +       u16 queue_index;
> > >> +       u32 asid;
> > >> +       u8 padding[2];
> > > I'd reserve more here. E.g in the future each virtqueue may have a
> > > secondary ASID,
> > like asid[2]? But why? For a device, there may needs a secondary ASID
> > because it
> > may need to access different memory region on behalf of different DMA
> > sources.
> > But for a virtqueue, it only has one purpose, to access the queue
> > buffer, maybe it
> > doesn't need a secondary asid
> 
> E.g for transparent shadow virtqueue.
> 
> > >
> > >> +};
> > >> +\end{lstlisting}
> > >> +
> > >> +The VIRTIO_TRANSPTQ_CTRL_VQ_ASID_SET command is used to set the address space id(\field{asid})
> > >> +of a virtqueue.
> > >> +
> > >> +The address space id is an identifier
> > > Let's add "transport specific" before "identifier"
> > OK
> > >
> > >> of a memory space which is used to convey the address space targeted by the
> > >> +memory accesses
> > > We need to say the ASID is for DMA not here at least.
> 
> I meant you need to explain how address space is used. (e.g it it used
> only for DMA).
> 
> > I don't get it, we need to explain what ASID is used for after
> > we introduce it.
> > >
> > >> , and to distinguish memory accesses performed by different virtqueues
> > > This may confuse the readers since a single ASID could be shared by virtqueues.
> > so it should be "by different virtqueues or virtqueue groups"
> 
> I think we can refer to the explanation of transport specific ASID here.
> 
> Thanks

Re: [virtio-comment] [PATCH V3 RESEND 2/4] Introduce the commands set of the transport vq

[Zhu, Lingshan]

On 8/10/2022 5:34 PM, Michael S. Tsirkin wrote:
> On Wed, Aug 10, 2022 at 09:56:51AM +0800, Jason Wang wrote:
>> On Tue, Aug 9, 2022 at 9:09 PM Zhu, Lingshan <lingshan.zhu@intel.com> wrote:
>>>
>>>
>>> On 8/8/2022 6:04 PM, Jason Wang wrote:
>>>> On Fri, Aug 5, 2022 at 6:02 PM Zhu Lingshan <lingshan.zhu@intel.com> wrote:
>>>>> This commit introduces the commands set of the
>>>>> transport virtqueue, including:
>>>>>
>>>>> The command to query available resources of the management device
>>>>> The commands to create / destroy the managed devices.
>>>>> The commands to config the managed devices.
>>>>> The commands to config virtqueues of the managed devices.
>>>>>
>>>>> Signed-off-by: Jason Wang <jasowang@redhat.com>
>>>>> Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com>
>>>>> ---
>>>>>    content.tex | 636 ++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>    1 file changed, 636 insertions(+)
>>>>>
>>>>> diff --git a/content.tex b/content.tex
>>>>> index c747d21..190f3a0 100644
>>>>> --- a/content.tex
>>>>> +++ b/content.tex
>>>>> @@ -2975,6 +2975,642 @@ \subsection{Format of Commands through Transport Virtqueue}\label{sec:Virtio Tra
>>>>>    The \field{device_id} value 0 is used to identify the management device itself.
>>>>>
>>>>>    \field{class} is an identifier of a set of commands with similar  purposes.
>>>>> +
>>>>> +\subsection{Available Resource of the Management Device}\label{sec:Virtio Transport Options / Virtio Over Transport Virtqueue / Available Resource of the Management Device}
>>>>> +
>>>>> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
>>>>> +statistical information of available resources on the management device could be fetched
>>>>> +by the following command:
>>>>> +
>>>>> +\begin{lstlisting}
>>>>> +#define VIRTIO_TRANSPTQ_CTRL_AVAIL_RES    1 (class)
>>>>> + #define VIRTIO_TRANSPTQ_CTRL_AVAIL_RES_GET     0 (command)
>>>>> +\end{lstlisting}
>>>>> +
>>>>> +VIRTIO_TRANSPTQ_CTRL_AVAIL_RES_GET command is used to get the statistical information of the available
>>>>> +resource of a management device. There is no command-out-data for VIRTIO_TRANSPTQ_CTRL_AVAIL_RES_GET
>>>>> +command. The management device fills command-in-data in the following format:
>>>>> +
>>>>> +\begin{lstlisting}
>>>>> +struct virtio_mgmt_dev_avail_res {
>>>>> +        /* The number of management device total remaining available free
>>>>> +         * virtqueues for the managed devices
>>>>> +         */
>>>> It's hard to phrase this sentence.
>>> How about "The number of remaining available virtqueues for new managed
>>> devices"
>>>>> +        u16 num_avail_vqs;
>>>>> +        /* The minimal number of virtqueues for a managed device */
>>>>> +        u16 min_dev_vqs;
>>>>> +        /* The maximum number of virtqueues for a managed device */
>>>> Nit the min/max_dev_vqs are not the resources, so we'd better rename
>>>> the virtio_mgmt_dev_avail_res.
>>> OK, I will use struct virito_mgmt_dev_info{} in the next version, and
>>> the command and its description
>>> will change as well.
>>>>> +        u16 max_dev_vqs;
>>>>> +        /* The number of managed devices that can be created with min_vqs virtqueues */
>>>> Is the part "with min_vqs virtqueues" a must? My understanding of the
>>>> above is a hint that the managed device is loosely coupled with the
>>>> virtqueue resources.
>>> Yes, the number of the number of total managed devices can be limited by
>>> other resources, like the filters.
>>>> So it would be possible that num_avail_dev = 0 but num_avail_vqs not?
>>>>
>>>> Btw, "num" prefix is probably not a must.
>>> OK
>>>>> +        u16 num_avail_dev;
>>>> To be consistent, let's use "num_avail_devs"?
>>> I will remove all num_ prefix, and it should be avail_devs
>>>>> +};
>>>>> +\end{lstlisting}
>>>>> +
>>>>> +\field{max_avail_dev} MAY not equal to \field{num_avail_vqs} divided by \field{min_vqs}
>>>>> +due to the limitations of of other resources.
>>>> two "of".
>>> will fix
>>>> Keywords like "MAY" should go to normatives part.
>>> OK, so I will only use "may", so it is a normal description.
>>>>> +
>>>>> +\devicenormative{\subsubsection}{Available Resource of the Management Device}{Virtio Transport Options /
>>>>> +Virtio Over Transport Virtqueue / Available Resource of the Management Device}
>>>>> +
>>>>> +The management device MUST fail VIRTIO_TRANSPTQ_CTRL_AVAIL_RES_GET command if \field{device_id} is not 0
>>>>> +
>>>>> +\drivernormative{\subsubsection}{Available Resource of the Management Device}{Virtio Transport Options /
>>>>> +Virtio Over Transport Virtqueue / Available Resource of the Management Device}
>>>>> +
>>>>> +The management device driver MUST use 0 as \field{device_id} for
>>>>> +VIRTIO_TRANSPTQ_CTRL_AVAIL_RES_GET command.
>>>>> +
>>>>> +\subsection{Create and Destroy Managed Devices}\label{sec:Virtio Transport Options / Virtio Over Transport Virtqueue / Create and Destroy Managed Devices}
>>>>> +
>>>>> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
>>>>> +managed devices must be created and destroyed through the transport virtqueue.
>>>>> +
>>>>> +\begin{lstlisting}
>>>>> +#define VIRTIO_TRANSPORTQ_CTRL_DEV    2
>>>>> + #define VIRTIO_TRANSPORTQ_CTRL_DEV_CREATE        0
>>>>> + #define VIRTIO_TRANSPORTQ_CTRL_DEV_DESTROY       1
>>>>> +
>>>>> +struct virtio_transportq_ctrl_dev_attribute {
>>>>> +       u64 device_features[2];
>>>> Let's reserve more bits for features here as we are about to reach 64.
>>> OK, it looks 256bytes may be enough.
>> It's better to match the parent/management device, but maybe we can
>> start from something that is fine for the next 5 years. So I'd rather
>> go with even more than this like device_feautres[10].
> Let's just specify the size. BTW should be
> u8 device_features_dwords;
> u32 device_features[N];
OK
>
>
>>>>> +       u32 virtio_device_id;
> Starting with virtio_device_id seems cleaner.
>
> Do we also want a vendor id? Or just assume same vendor as parent?
I think we need a vendor id, I will add this.
the vendor should be the same vendor as the parent by default.
And maybe we also need sub vendor id and sub device id, like what
PCI does.
>
>>>>> +       u8 dev_config[];
>>>> To be consistent, let's use one of "device" or "dev" in the same structure.
>>> sure, device_config[]
>>>>> +};
>>>>> +\end{lstlisting}
>>>>> +
>>>>> +The VIRTIO_TRANSPORTQ_CTRL_DEV_CREATE command is used to create a managed device.
>>>>> +
>>>>> +As described in struct virtio_transportq_ctrl_dev_attribute, the command-out-data for VIRTIO_TRANSPTQ_CTRL_DEV_CREATE includes:
>>>>> +\begin{itemize*}
>>>>> +\item \field{device_features}: the 128-bit device feature bits (defined in section \ref{sec:Basic Facilities of a Virtio Device / Feature Bits})
>>>>> +of the managed device.
>>>>> +\item \field{virtio_device_id}: the virtio device id defined in Chapter \ref{sec:Device Types}.
>>>>> +\item \field{dev_config}: the device type specific configurations. E.g., for a virtio-net device,
>>>>> +it is struct virtio_net_config in section \ref{sec:Device Types / Network Device / Device configuration layout};
>>>>> +for a virtio-block device, it is struct virtio_blk_config in section \ref{sec:Device Types / Block Device / Device configuration layout}
>>>>> +\end{itemize*}
>>>>> +
>>>>> +When succeed, the device returns a 64 bits UUID as an unique identifier of the created managed device in command-in-data.
>>>> We don't need to be universally unique here.
>>> Why? I think it should be unique to identify a device, or are you
>>> suggesting to remove "as an unique identifier",
>>> just UUID is enough?
>> UUID means it's unique in the universal. If it means ID for managed
>> device in PF1 must differ from ID for managed device in PF2. This is
>> not needed.
>>
>> So if we want to be flexible, we can allow the ID to be allocated by
>> the driver as the original RFC did. The driver is free to allocate
>> anything it want (UID or UUID).
>>
>>>>> +
>>>>> +The VIRTIO_TRANSPORTQ_CTRL_DEV_DESTROY command is used to destroy a
>>>>> +managed device which is identified by its 64 bits UUID
>>>>> +\field{device_id}. There's no command-in-data nor command-out-data for
>>>>> +VIRTIO_TRANSPTQ_CTRL_DEV_DESTROY command.
>>>>> +
>>>>> +\devicenormative{\subsubsection}{Create and Destroy Managed Devices}{Virtio Transport Options / Virtio Over Transport Virtqueue / Create and Destroy Managed Devices}
>>>>> +
>>>>> +The management device MUST fail VIRTIO_TRANSPORTQ_CTRL_DEV_CREATE command
>>>>> +if \field{device_id} is not 0.
>>>>> +
>>>>> +The management device MUST fail VIRTIO_TRANSPORTQ_CTRL_DEV CREATE command
>>>>> +if \field{device_features} exceeds the features that can be provided from the management device.
>>>> So PCI has le32 device_feature_select. How to make sure it can match
>>>> the capacity of device_features[] above?
>>> we present all feature bits in device_features[], the spec says the
>>> feature bits are valid up to bit 127,
>>> 128 and above are reserved. So as you suggest, we can use u64
>>> device_features[4], for 128-bit feature bits
>>> and another 128 bits for further extensions.
>>>>> +
>>>>> +The management device MUST fail VIRTO_TRANSPORTQ_CTRL_DEV_DESTROY command
>>>>> +if \field{device_id} is 0.
>>>>> +
>>>>> +\drivernormative{\subsubsection}{Create and Destroy Managed Devices}{Virtio Transport Options / Virtio Over Transport Virtqueue / Create and Destroy Managed Devices}
>>>>> +
>>>>> +The management device driver MUST use 0 as \field{device_id} for
>>>>> +TRANSPORTQ_CTRL_DEV_CREATE command.
>>>>> +
>>>>> +\subsection{Features Negotiation}\label{sec:Virtio Transport Options / Virtio
>>>>> +Over Transport Virtqueue / Features Negotiation}
>>>>> +
>>>>> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
>>>>> +the features negotiation of managed devices is done by the
>>>>> +following commands:
>>>>> +
>>>>> +\begin{lstlisting}
>>>>> +#define VIRTIO_TRANSPTQ_CTRL_FEAT   3
>>>>> + #define VIRTIO_TRANSPTQ_CTRL_FEAT_DEVICE_GET        0
>>>>> + #define VIRTIO_TRANSPTQ_CTRL_FEAT_DRIVER_SET        1
>>>>> + #define VIRTIO_TRANSPTQ_CTRL_FEAT_DRIVER_GET        2
>>>>> +\end{lstlisting}
>>>>> +
>>>>> +The VIRTIO_TRANSPTQ_CTRL_FEAT_DEVICE_GET command is used to get the features offered
>>>>> +by a managed device. The command-in-data is the 128-bit feature bits
>>>>> +(defined in section \ref{sec:Basic Facilities of a Virtio Device / Feature Bits}).
>>>> Need to make sure the capacity of the feature bit matches what the
>>>> transport of the management device can provide.
>>> I will provide a structure for the features, so both device and driver
>>> would be aligned on
>>> the size of the feature bits, like
>>>
>>> struct virtio_transportq_ctrl_dev_features {
>>>       u64 features[4];
>>> };
>>>>> +There is no command-out-data.
>>>>> +
>>>>> +The VIRTIO_TRANSPTQ_CTRL_FEAT_DRIVER_SET command is for the driver to accept feature
>>>>> +bits offered by the managed device. The command-out-data is the 128-bit feature bits
>>>>> +passed to the managed device. There is no command-in-data.
>>>>> +
>>>>> +The VIRTIO_TRANSPTQ_CTRL_FEAT_DRIVER_GET command is used to get the features accepted
>>>>> +by both the managed device driver and the managed device. The command-in-data is the
>>>>> +128-bit feature bits. There is no command-out-data.
>>>>> +
>>>>> +\devicenormative{\subsubsection}{Features Negotiation}{Virtio Transport Options /
>>>>> +Virtio Over Transport Virtqueue / Features Negotiation}
>>>>> +
>>>>> +The management device MUST fail VIRTIO_TRANSPTQ_F_CTRL_FEAT class commands if
>>>>> +\field{device_id} is 0.
>>>>> +
>>>>> +\subsection{Device Status}\label{sec:Virtio Transport Options / Virtio Over
>>>>> +Transport Virtqueue / Device Status}
>>>>> +
>>>>> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
>>>>> +the device status of a managed device can be accessed by the following
>>>>> +commands:
>>>>> +
>>>>> +\begin{lstlisting}
>>>>> +#define VIRTIO_TRANSPTQ_CTRL_STATUS    4
>>>>> + #define VIRTIO_TRANSPTQ_CTRL_STATUS_GET        0
>>>>> + #define VIRTIO_TRANSPTQ_CTRL_STATUS_SET        1
>>>>> +\end{lstlisting}
>>>>> +
>>>>> +The VIRTIO_TRANSPTQ_CTRL_STATUS_GET command is used to get the device status of
>>>>> +a managed device. The command-in-data is the 8-bit status
>>>>> +returned from the device. There's no command-out-data for this
>>>>> +command.
>>>>> +
>>>>> +The VIRTIO_TRANSPTQ_CTRL_STATUS_SET command is used to set the device status of
>>>>> +a managed device. The command-out-data is the 8-bit status
>>>>> +to set to the device. There's no command-in-data for this command.
>>>>> +
>>>>> +\devicenormative{\subsubsection}{Device Status}{Virtio Transport Options / Virtio
>>>>> +Over Transport Virtqueue / Device Status}
>>>>> +
>>>>> +The management device MUST reset the managed device when 0
>>>>> +is set via VIRTIO_TRANSPTQ_CTRL_STATUS_SET, the success of this
>>>>> +command demonstrates the success of the reset.
>>>>> +
>>>>> +The management device MUST present 0 through
>>>>> +VIRTIO_TRANSPTQ_CTRL_STATUS_GET once the reset is successfully done.
>>>>> +
>>>>> +The management device MUST fail the device status access if
>>>>> +\field{device_id} is 0.
>>>>> +
>>>>> +\drivernormative{\subsubsection}{Device Status}{Virtio Transport Options / Virtio
>>>>> +Over Transport Virtqueue / Device Status}
>>>>> +
>>>>> +After writing 0 via VIRTIO_TRANSPTQ_CTRL_STATUS_SET, the driver MUST wait
>>>>> +for the success of the command before re-initializing the device.
>>>>> +
>>>>> +\subsection{Device Generation}\label{sec:Virtio Transport Options / Virtio Over Transport Virtqueue / Device Generation}
>>>>> +
>>>>> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
>>>> Btw, since we're under the transport options (which implies the
>>>> transport virtqueue), maybe we can drop the above for each command.
>>> OK, can do, and I will describe this in the Basic Concept section.
>>>>> +the device generation count can be read from the following commands:
>>>>> +
>>>>> +\begin{lstlisting}
>>>>> +#define VIRTIO_TRANSPTQ_CTRL_GENERATION    5
>>>>> + #define VIRTIO_TRANSPTQ_CTRL_GENERATION_GET        0
>>>>> +\end{lstlisting}
>>>>> +
>>>>> +The VIRTIO_TRANSPTQ_CTRL_GENERATION_GET command is used to get the device configuration generation count
>>>>> +of the managed device. The command-in-data is the one byte device
>>>>> +generation returned from the device. There's no command-out-data for
>>>>> +this command.
>>>>> +
>>>>> +\devicenormative{\subsubsection}{Device Generation}{Virtio Transport Options / Virtio Over Transport Virtqueue / Device Generation}
>>>>> +
>>>>> +The managed device MUST present a changed generation count after the driver
>>>>> +has read any device-specific configuration values if the values have been changed
>>>>> +during the last read.
>>>>> +
>>>>> +The management device MUST fail the device generation access if \field{device_id} is 0.
>>>>> +
>>>>> +\subsection{Device Specific Configuration}\label{sec:Virtio Transport Options /
>>>>> +Virtio Over Transport Virtqueue / Device Specific Configuration}
>>>>> +
>>>>> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
>>>>> +the device config space contents of a managed device can be accessed through
>>>>> +the following commands:
>>>>> +
>>>>> +\begin{lstlisting}
>>>>> +#define VIRTIO_TRANSPTQ_CTRL_CONFIG    6
>>>>> +  #define VIRTIO_TRANSPTQ_CTRL_CONFIG_GET    0
>>>>> +  #define VIRTIO_TRANSPTQ_CTRL_CONFIG_SET    1
>>>>> +
>>>>> +struct virtio_transportq_ctrl_dev_config_get {
>>>>> +       u32 offset;
>>>>> +       u32 size;
>>>>> +};
>>>>> +
>>>>> +struct virtio_transportq_ctrl_dev_config_set {
>>>>> +       u32 offset;
>>>>> +       u32 size;
>>>>> +       u8  data[];
>>>>> +};
>>>>> +\end{lstlisting}
>>>>> +
>>>>> +The VIRTIO_TRANSPTQ_CTRL_CONFIG_GET command is used to read data from the
>>>>> +device configuration space.  The command-out-data is the \field{offset}
>>>>> +from the start of the config space and the \field{size}
>>>>> +of the data (as described in struct virtio_transportq_ctrl_dev_config_get).
>>>>> +The command-in-data is an array of the data that read from the config space.
>>>>> +
>>>>> +The VIRTIO_TRANSPTQ_CTRL_CONFIG_SET command is used to write data to the device
>>>>> +configuration space. The command-out-data contains the
>>>>> +\field{offset} from the start of the config space, the \field{size} of the data and
>>>>> +the \field{data} to be written (as described in struct virtio_transportq_ctrl_dev_config_set).
>>>>> +There's no command-in-data for this command.
>>>>> +
>>>>> +\devicenormative{\subsubsection}{Device Specific Configuration}{Virtio Transport
>>>>> +Options / Virtio Over Transport Virtqueue / Device Specific Configuration}
>>>>> +
>>>>> +The management device MUST fail the device configuration space access
>>>>> +if the driver access the range which is outside the config space.
>>>> "accesses"
>>> yes
>>>>> +
>>>>> +The management device MUST fail the device configuration space access
>>>>> +if \field{device_id} is 0.
>>>>> +
>>>>> +\subsection{MSI Configuration}\label{sec:Virtio Transport Options / Virtio Over
>>>>> +Transport Virtqueue / MSI Configuration}
>>>>> +
>>>>> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
>>>>> +the MSI entries of a managed device can be accessed  through the following command:
>>>> Rethink of this, it implies a per virtqueue MSI storage which I'm not
>>>> sure is expensive or not. Do we need an indirection layer as PCI did?
>>> I think we need this for performance, or are you suggesting to add a
>>> device scope MSI?
>> I meant having MSI vectors and let the virtqueue refer to the MSI vectors.
>>
>> Current design means for a 1024 virtqueues device to store 1024 MSI entries.
>> With the indirection of the MSI vectors array, the device is free to
>> have 1 to 1024 MSI entries.
> BTW we need a way to mask and change vectors too.
> This happens under all kind of locks I am not sure
> how practical it is to do this through a command.
> Pls explore the linux code for this.
There was a unmask/mask command in the last version,
I can add them back, and I will also introduce
a command to read the pending bit of a vq,
for compatibility.

I think changing a vector can be covered by VIRTIO_TRANSPTQ_CTRL_MSI_VQ_SET
>
>
>>>>> +
>>>>> +\begin{lstlisting}
>>>>> +#define VIRTIO_TRANSPTQ_CTRL_MSI    7
>>>>> + #define VIRTIO_TRANSPTQ_CTRL_MSI_VQ_GET        0
>>>>> + #define VIRTIO_TRANSPTQ_CTRL_MSI_VQ_SET        1
>>>>> + #define VIRTIO_TRANSPTQ_CTRL_MSI_VQ_ENABLE     2
>>>>> + #define VIRTIO_TRANSPTQ_CTRL_MSI_CONFIG_GET    3
>>>>> + #define VIRTIO_TRANSPTQ_CTRL_MSI_CONFIG_WET    4
>>>> I think you mean CONFIG_SET here.
>>> yes
>>>>> + #define VIRTIO_TRANSPTQ_CTRL_MSI_CONFIG_ENABLE 5
>>>>> +
>>>>> +struct virtio_transportq_ctrl_msi_vq_config {
>>>>> +       u16 queue_index;
>>>>> +       u64 address;
>>>>> +       u32 data;
>>>>> +       u8 padding[2];
>>>>> +};
>>>>> +
>>>>> +struct virtio_transportq_ctrl_msi_vq_enable {
>>>>> +       u16 queue_index;
>>>>> +       u8 enable;
>>>>> +       u8 padding[5];
>>>>> +};
>>>>> +
>>>>> +struct virtio_transportq_ctrl_msi_config {
>>>>> +       u64 address;
>>>>> +       u32 data;
>>>>> +       u8 padding[4];
>>>>> +};
>>>>> +\end{lstlisting}
>>>>> +
>>>>> +The VIRTIO_TRANSPTQ_CTRL_MSI_VQ_GET command is used to read the MSI entry of a
>>>>> +specific virtqueue. The command-out-data is the queue index. The command-in-data contains
>>>>> +\field{queue_index}, the \field{address} and \field{data}
>>>>> +(as described in struct virtio_transportq_ctrl_msi_vq_config).
>>>> Let's split or reuse the virtio_transportq_ctrl_msi_config since
>>>> virtio_transportq_ctrl_msi_vq_config will not be used a as single
>>>> buffer.
>>> I don't get it, if we re-use the struct virtio_transportq_ctrl_msi_config,
>>> it would be like:
>>>
>>> struct virtio_transportq_ctrl_msi {
>>>       u64 addr;
>>>       u32 data;
>>> };
>>>
>>> struct virtio_transportq_ctrl_msi_vq_config {
>>>       u16 queue_index;
>> What I meant is, for GET. queue_index is out buffer, and re-use
>> virtio_transportq_ctrl_msi is in buffer.
>>
>>>       struct virtio_transportq_ctrl_msi;
>>> };
>>>
>>> it is still two structs and a struct in another struct.
>>>>> +
>>>>> +
>>>>> +The VIRTIO_TRANSPTQ_CTRL_MSI_VQ_SET command is used to set the MSI entry for a
>>>>> +specific virtqueue. The command-out-data is the \field{queue_index},
>>>>> +\field{address} and \field{data} (as described in struct virtio_transportq_ctrl_msi_vq_config).
>>>>> +There is no command-in-data.
>>>>> +
>>>>> +The VIRTIO_TRANSPTQ_CTRL_MSI_VQ_ENABLE command is used to enable or disable
>>>>> +the MSI interrupt for a specific virtqueue. The command-out-data is the
>>>>> +\field{queue_index} and a byte \field{enable} which representing ENABLE or DISABLE the MSI.
>>>> Should be "represents".
>>> OK
>>>> Btw, is there a chance that we may lose an interrupt?
>>> No, I think we won't lose any interrupts, because once disabled, it can
>>> not send new interrupts,
>>> but the last interrupt send by the MSI entry is still there, it's in-band.
>> So consider driver is changing affinity what it did is:
>>
>> disable
>> write MSI
>> enable
>>
>> What if there's an interrupt coming in the middle? Does this mean the
>> device needs to implement the pending bit internally?
> I suspect so.
I am not sure, MSI interrupts are in-band DMA, so once send out, it will 
be routed
to a processor anyway(the processor is tagged by the destination ID in 
the MSI address).
So normally it can not be lost.

Once a MSI is disabled, I think it can not send interrupts anymore

Thanks
>
>> FYI we had similar discussion in the past:
>>
>> https://lists.oasis-open.org/archives/virtio-dev/202002/msg00027.html
> Yes MSI for MMIO got blocked on the same set of issues.
>
>
>>>>> +(as described in struct virtio_transportq_ctrl_msi_vq_enable).
>>>>> +
>>>>> +\begin{lstlisting}
>>>>> +#define VIRTIO_TRANSPTQ_CTRL_MSI_VQ_ENABLE     1
>>>>> +#define VIRTIO_TRANSPTQ_CTRL_MSI_VQ_DISABLE    0
>>>>> +\end{lstlisting}
>>>>> +
>>>>> +The VIRTIO_TRANSPTQ_CTRL_MSI_CONFIG_GET command is used to read the MSI entry
>>>>> +of the config interrupt. The command-in-data contains the \field{address} and
>>>>> +\field{data} (as described in struct virtio_transportq_ctrl_msi_config).
>>>>> +There is no command-out-data.
>>>>> +
>>>>> +The VIRTIO_TRANSPTQ_CTRL_MSI_CONFIG_SET command is used to set the MSI entry
>>>>> +for the config interrupt. The command-out-data is the \field{address} and
>>>>> +\field{data} (as described in struct virtio_transportq_ctrl_msi_config).
>>>>> +There is no command-in-data.
>>>>> +
>>>>> +The VIRTIO_TRANSPTQ_CTRL_MSI_CONFIG_ENABLE command is used to enable or disable
>>>>> +the MSI interrupt for config space. The command-out-data is a byte which representing
>>>>> +ENABLE or DISABLE the config MSI.
>>>>> +There is no command-in-data
>>>>> +
>>>>> +\begin{lstlisting}
>>>>> +#define VIRTIO_TRANSPTQ_CTRL_MSI_CFG_ENABLE     1
>>>>> +#define VIRTIO_TRANSPTQ_CTRL_MSI_CFG_DISABLE    0
>>>>> +\end{lstlisting}
>>>>> +
>>>>> +\devicenormative{\subsubsection}{MSI Configuration}{Virtio Transport Options / Virtio
>>>>> +ver Transport Virtqueue / MSI Configuration}
>>>>> +
>>>>> +The managed device MUST disable the MSI interrupts for both virtqueues and config space
>>>>> +upon a reset.
>>>>> +
>>>>> +Once a MSI entry is disabled, the managed device MUST not send any interrupts
>>>>> +by this MSI entry.
>>>> MSI is stored based on virtqueue and config space, so I'm not quite
>>>> sure "MSI entry" is accurate here.
>>> I will replace all "MSI entry" with "MSI vector"
>>>>> +
>>>>> +\drivernormative{\subsubsection}{MSI Configuration}{Virtio Transport Options / Virtio
>>>>> +Over Transport Virtqueue / MSI Configuration}
>>>>> +
>>>>> +The driver MUST allocate transport or platform specific MSI entries
>>>>> +for both virtqueues and config space if it wants to use interrupts.
>>>>> +
>>>>> +The driver MAY choose to disable the MSI if polling mode is used.
>>>>> +
>>>>> +\subsection{Virtqueue Address}\label{sec:Virtio Transport Options / Virtio Over
>>>>> +Transport Virtqueue / Virtqueue Address}
>>>>> +
>>>>> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
>>>>> +the addresses of a specific virtqueue are accessed through the following command:
>>>>> +
>>>>> +\begin{lstlisting}
>>>>> +#define VIRTIO_TRANSPTQ_CTRL_VQ_ADDR    8
>>>>> + #define VIRTIO_TRANSPTQ_CTRL_VQ_ADDR_SET       1
>>>>> +
>>>>> +struct virtio_transportq_ctrl_vq_addr {
>>>>> +       u16 queue_index;
>>>>> +       u64 descriptor_area;
>>>>> +       u64 device_area;
>>>>> +       u64 driver_area;
>>>>> +       u8 padding[6];
>>>>> +};
>>>>> +\end{lstlisting}
>>>>> +
>>>>> +The VIRTIO_TRANSPTQ_CTRL_VQ_ADDR_SET command is used to set the addresses of a specified
>>>>> +virtqueue. The command-out-data contains the \field{queue_index}, the addresses of \field{device_area},
>>>>> +\field{descriptor_area} and \field{driver_area} (as described in struct
>>>>> +virtio_transportq_ctrl_vq_addr). There's no command-in-data.
>>>>> +
>>>>> +\devicenormative{\subsubsection}{Virtqueue Address}{Virtio Transport Options / Virtio
>>>>> +Over Transport Virtqueue / Virtqueeu Address}
>>>>> +
>>>>> +The management device MUST fail the commands of class
>>>>> +VIRTIO_TRANSPTQ_CTRL_VQ_ADDR if \field{device_id} is 0.
>>>>> +
>>>>> +The management device MUST fail the commands of class
>>>>> +VIRTIO_TRANSPTQ_CTRL_VQ_ADDR if \field{queue_index} is out-of-range invalid.
>>>>> +
>>>>> +\subsection{Virtqueue Status}\label{sec:Virtio Transport Options / Virtio Over
>>>>> +Transport Virtqueue / Virtqueue Status}
>>>>> +
>>>>> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
>>>>> +virtqueue status is accessed through the following command:
>>>>> +
>>>>> +\begin{lstlisting}
>>>>> +#define VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE    9
>>>>> + #define VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_GET       0
>>>>> + #define VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_SET       1
>>>>> +
>>>>> +struct virtio_transportq_ctrl_vq_status_set {
>>>>> +       u16 queue_index;
>>>>> +       u8 status;
>>>>> +       u8 padding[5];
>>>>> +
>>>>> +#define VIRTIO_TRANSPTQ_VQ_ENABLE     1
>>>>> +#define VIRTIO_TRANSPTQ_VQ_DISABLE    0
>>>>> +
>>>>> +};
>>>>> +
>>>>> +\end{lstlisting}
>>>>> +
>>>>> +The VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_GET is used to get the status of a
>>>>> +specific virtqueue. The command-out-data is the queue
>>>>> +index. The command-in-data is the virtqueue status.
>>>>> +
>>>>> +The VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_SET command is used to set the status of a
>>>>> +specific virtqueue. The command-out-data is the \field{queue_index} and the
>>>>> +\field{status} representing ENABLE or DISABLE that is set to the virtqueue
>>>>> +(as described in struct virtio_transportq_ctrl_vq_status_set).
>>>>> +There's no command-in-data.
>>>>> +
>>>>> +\devicenormative{\subsubsection}{Virtqueue Status}{Virtio Transport Options / Virtio
>>>>> +Over Transport Virtqueue / Virtqueue Status}
>>>>> +
>>>>> +When disabled, the managed device MUST stop processing requests from
>>>>> +this virtqueue.
>>>> Is this a hint that the driver can write VIRTIO_TRANSPTQ_VQ_DISABLE to
>>>> status? If yes, what's the difference between this and virtqueue
>>>> reset?
>>> I think disabling a queue is like "set queue_enable = 0", once disabled,
>>> the device should not reset the queue configurations like the indexes
>>> and addresses,
>>> it just freeze the queue.
>> Then you need to define what "freeze" means. To reduce the changset,
>> I'd suggest disallowing such writing as what PCI did.
>>
>> If needed, we can re-enable it in the future.
>>
>>> When reset a queue, like the spec says, the device needs to reset all queue
>>> configs.
>>>>> +
>>>>> +The management device MUST present a 0 via
>>>>> +VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_GET upon a reset of the managed device.
>>>>> +
>>>>> +The management device MUST fail the virtqueue status access if
>>>>> +\field{device_id} is 0.
>>>>> +
>>>>> +The management device MUST fail the virtqueue status access if
>>>>> +the queue_index is out-of-range invalid.
>>>>> +
>>>>> +
>>>>> +\drivernormative{\subsubsection}{Virtqueue Status}{Virtio Transport Options / Virtio
>>>>> +Over Transport Virtqueue / Virtqueue Status}
>>>>> +
>>>>> +The driver MUST configure other virtqueue fields before enabling
>>>>> +the virtqueue with VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_SET.
>>>>> +
>>>>> +\subsection{Virtqueue Size}\label{sec:Virtio Transport Options / Virtio Over
>>>>> +Transport Virtqueue / Virtqueue Size}
>>>>> +
>>>>> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
>>>>> +virtqueue size is accessed through the following command:
>>>>> +
>>>>> +\begin{lstlisting}
>>>>> +#define VIRTIO_TRANSPTQ_CTRL_VQ_SIZE    10
>>>>> + #define VIRTIO_TRANSPTQ_CTRL_VQ_SIZE_GET       0
>>>>> + #define VIRTIO_TRANSPTQ_CTRL_VQ_SIZE_SET       1
>>>>> +
>>>>> +struct virtio_transportq_ctrl_vq_size_set {
>>>>> +       u16 queue_index;
>>>>> +       u16 size;
>>>>> +       u8 padding[4];
>>>>> +};
>>>>> +\end{lstlisting}
>>>>> +
>>>>> +The VIRTIO_TRANSPTQ_CTRL_VQ_SIZE_GET command is used to get the virtqueue
>>>>> +size. On reset, the maximum queue size supported by the device is
>>>>> +returned. The command-out-data is the queue index. The
>>>>> +command-in-data is an 16-bit queue size.
>>>>> +
>>>>> +The VIRTIO_TRANSPTQ_CTRL_VQ_SIZE_SET command is used to set the virtqueue
>>>>> +size. The command-out-data is the \field{queue_index} and the \field{size}
>>>>> +of the virtqueue (as described in struct virtio_transportq_ctrl_vq_size_set).
>>>>> +There's no command-in-data.
>>>>> +
>>>>> +\devicenormative{\subsubsection}{Virtqueue Status}{Virtio Transport Options / Virtio
>>>>> +Over Transport Virtqueue / Virtqueue Size}
>>>>> +
>>>>> +The management device MUST fail the virtqueue size access if
>>>>> +\field{device_id} is 0.
>>>>> +
>>>>> +The management device MUST fail the virtqueue size access if
>>>>> +the queue index is out-of-range invalid.
>>>>> +
>>>>> +\subsection{Virtqueue Notification}\label{sec:Virtio Transport Options / Virtio
>>>>> +Over Transport Virtqueue / Virtqueue Notification}
>>>>> +
>>>>> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
>>>>> +the virtqueue notification area information can be get through the following commands:
>>>>> +
>>>>> +\begin{lstlisting}
>>>>> +#define VIRTIO_TRANSPTQ_CTRL_VQ_NOTIFY    11
>>>>> + #define VIRTIO_TRANSPTQ_CTRL_VQ_NOTIFY_GET          0
>>>>> +
>>>>> +struct virtio_transportq_ctrl_vq_notification_area {
>>>>> +       u64 address;
>>>>> +       u64 size;
>>>>> +};
>>>>> +\end{lstlisting}
>>>>> +
>>>>> +The VIRTIO_TRANSPTQ_CTRL_VQ_NOTIFY_GET is used to get the transport
>>>>> +specific address area that can be used to notify a virtqueue. The
>>>>> +command-out-data is an u16 of the queue index. The command-in-data
>>>>> +contains the \field{address} and the \field{size}
>>>>> +of the notification area (as described in struct virtio_transportq_ctrl_vq_notification_area).
>>>>> +
>>>>> +\devicenormative{\subsubsection}{Virtqueue Notification}{Virtio Transport Options /
>>>>> +Virtio Over Transport Virtqueue / Virtqueue Notification}
>>>>> +
>>>>> +The management device MUST fail the virtqueue notification area information
>>>>> +access if \field{device_id} is 0.
>>>>> +
>>>>> +The management device MUST fail the virtqueue notification area information
>>>>> +access if the queue index is out-of-range invalid.
>>>> We probably need to say the management device must reserve sufficient
>>>> transport specific resources as notification area or we need to have a
>>>> fallback of using a dedicated command to notify.
>>> OK, will add this
>>>>> +
>>>>> +\drivernormative{\subsubsection}{Virtqueue Notification}{Virtio Transport Options /
>>>>> +Virtio Over Transport Virtqueue / Virtqueue Notification}
>>>>> +
>>>>> +The driver MAY choose to notify the virtqueue by writing the queue
>>>>> +index at address \field{address} which is fetched from the
>>>>> +VIRTIO_TRANSPTQ_CTRL_VQ_NOTIFY_GET command.
>>>>> +
>>>>> +\subsection{Virtqueue State}\label{sec:Virtio Transport Options / Virtio
>>>>> +Over Transport Virtqueue / Virtqueue State}
>>>>> +
>>>>> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
>>>>> +the virtqueue state is accessed through the following commands:
>>>>> +
>>>>> +\begin{lstlisting}
>>>>> +#define VIRTIO_TRANSPTQ_CTRL_VQ_STATE            12
>>>>> + #define VIRTIO_TRANSPTQ_CTRL_VQ_STATE_GET       0
>>>>> + #define VIRTIO_TRANSPTQ_CTRL_VQ_STATE_SET       1
>>>>> +\end{lstlisting}
>>>>> +
>>>>> +\subsubsection{Split Virtqueue State}\label{sec:Virtio Transport Options / Virtio
>>>>> +Over Transport Virtqueue / Virtqueue State / Split Virtqueue State}
>>>>> +
>>>>> +\begin{lstlisting}
>>>>> +struct virtio_transportq_ctrl_split_vq_state_set {
>>>>> +    u16 queue_index;
>>>>> +    u16 avail_index;
>>>>> +    u8 padding[4];
>>>>> +};
>>>>> +\end{lstlisting}
>>>>> +
>>>>> +The VIRTIO_TRANSPTQ_CTRL_VQ_STATE_GET command can be  used to get the state of a
>>>>> +split virtqueue. The command-out-data is the queue index, the command-in-data is the available index of the virtqueue.
>>>> We need to be accurate for "available index". E.g we have one for in
>>>> the shared memory, so do you mean last_avail_idx?
>>> actually there is no definition of "last available index" in the spec,
>>> so how about we say:
>>> on-device available index?
>> Any should be fine but need to explain what does it mean.
>>
>>>>> +
>>>>> +The VIRTIO_TRANSPTQ_CTRL_VQ_STATE_SET command can be used to set the state of a
>>>>> +split virtqueue. The command-out-data contains the \field{queue_index} and the
>>>>> +available index (\field{avail_index}) for the virtqueue (as described in struct virtio_transportq_ctrl_split_vq_state_set).
>>>>> +There is no command-in-data.
>>>>> +
>>>>> +\subsubsection{Packed Virtqueue State}\label{sec:Virtio Transport Options / Virtio
>>>>> +Over Transport Virtqueue / Virtqueue State / Packed Virtqueue State}
>>>>> +
>>>>> +\begin{lstlisting}
>>>>> +struct virtio_transportq_ctrl_packed_vq_state {
>>>>> +    u16 queue_index;
>>>>> +    u16 avail_counter;
>>>>> +    u16 avail_index;
>>>>> +    u16 used_counter;
>>>>> +    u16 used_index;
>>>> Let's simply reuse the structure in the kernel headers.
>>>>
>>>> Using u16 for wrap_couter seems unnecessary etc.
>>> OK, the bit operations are not often.
>> It should be very common in the hardware interface.
>>
>>>>> +    u8 padding[6];
>>>>> +};
>>>>> +\end{lstlisting}
>>>>> +
>>>>> +The state of a packed virtqueue includes :\\
>>>>> +\field{avail_counter}: last driver ring wrap counter observed by device.\\
>>>>> +\field{avail_index}: virtqueue available index.\\
>>>>> +\field{used_counter}: device ring wrap counter.\\
>>>>> +\field{used_index}: virtqueue used index.
>>>>> +
>>>>> +The VIRTIO_TRANSPTQ_CTRL_VQ_STATE_GET command can be used to get the state of a packed virtqueue.
>>>>> +The command-out-data is the queue index, the command-in-data contains the \field{queue_index},
>>>>> +\field{avail_counter}, \field{avail_index}, \field{used_counter} and \field{used_index} of the virtqueue
>>>>> +(as described in transportq_ctrl_packed_vq_state).
>>>>> +
>>>>> +The VIRTIO_TRANSPTQ_CTRL_VQ_STATE_SET command can be used to set the state of a packed virtqueue.
>>>>> +The command-out-data contains the \field{queue_index}, \field{avail_counter}, \field{avail_index},
>>>>> +\field{used_counter} and \field{used_index} for the virtqueue
>>>>> +(as described in transportq_ctrl_packed_vq_state).
>>>>> +There is no command-in-data.
>>>>> +
>>>>> +\devicenormative{\subsubsection}{Virtqueue State}{Virtio Transport Options /
>>>>> +Virtio Over Transport Virtqueue / Virtqueue State}
>>>>> +
>>>>> +The management device MUST fail the virtqueue index access if \field{device_id} is 0.
>>>>> +
>>>>> +The management device MUST fail the virtqueue index access if \field{queue_index} is out-of-range invalid.
>>>>> +
>>>>> +\subsection{Virtqueue ASID}\label{sec:Virtio Transport Options / Virtio Over
>>>>> +Transport Virtqueue / Virtqueue ASID}
>>>>> +
>>>>> +When VIRTIO_F_TRANSPT_VQ is negotiated with the management device,
>>>>> +the address space id of a virtqueue could be set through the following command:
>>>>> +
>>>>> +\begin{lstlisting}
>>>>> +#define VIRTIO_TRANSPTQ_CTRL_VQ_ASID        13
>>>>> + #define VIRTIO_TRANSPTQ_CTRL_VQ_ASID_SET   1
>>>>> +
>>>>> +struct virtio_transportq_ctrl_vq_asid_set {
>>>>> +       u16 queue_index;
>>>>> +       u32 asid;
>>>>> +       u8 padding[2];
>>>> I'd reserve more here. E.g in the future each virtqueue may have a
>>>> secondary ASID,
>>> like asid[2]? But why? For a device, there may needs a secondary ASID
>>> because it
>>> may need to access different memory region on behalf of different DMA
>>> sources.
>>> But for a virtqueue, it only has one purpose, to access the queue
>>> buffer, maybe it
>>> doesn't need a secondary asid
>> E.g for transparent shadow virtqueue.
>>
>>>>> +};
>>>>> +\end{lstlisting}
>>>>> +
>>>>> +The VIRTIO_TRANSPTQ_CTRL_VQ_ASID_SET command is used to set the address space id(\field{asid})
>>>>> +of a virtqueue.
>>>>> +
>>>>> +The address space id is an identifier
>>>> Let's add "transport specific" before "identifier"
>>> OK
>>>>> of a memory space which is used to convey the address space targeted by the
>>>>> +memory accesses
>>>> We need to say the ASID is for DMA not here at least.
>> I meant you need to explain how address space is used. (e.g it it used
>> only for DMA).
>>
>>> I don't get it, we need to explain what ASID is used for after
>>> we introduce it.
>>>>> , and to distinguish memory accesses performed by different virtqueues
>>>> This may confuse the readers since a single ASID could be shared by virtqueues.
>>> so it should be "by different virtqueues or virtqueue groups"
>> I think we can refer to the explanation of transport specific ASID here.
>>
>> Thanks



This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

[virtio-comment] [PATCH V3 RESEND 3/4] Describe the process to present a managed device

[Zhu Lingshan]

This commit describes the process to present a manage
device, including device creation, config and virtqueue config

Signed-off-by: Jason Wang <jasowang@redhat.com>
Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com>
---
 content.tex | 37 +++++++++++++++++++++++++++++++++++++
 1 file changed, 37 insertions(+)

diff --git a/content.tex b/content.tex
index 190f3a0..be632d1 100644
--- a/content.tex
+++ b/content.tex
@@ -3612,6 +3612,43 @@ \subsection{Virtqueue Reset}\label{sec:Virtio Transport Options / Virtio Over
 done (by verifying virtio_transportq_ctrl.ack == VIRTIO_TRANSPTQ_OK)
 before re-enabling it.
 
+\subsection{Presenting Managed Devices}\label{sec:Virtio Transport Options /
+Virtio Over Transport Virtqueue / Presenting Managed Device}
+
+If VIRTIO_F_TRANSPT_VQ is negotiated with the management device.
+The presenting of the managed devices requires cooperation between
+the management device driver, the transport virtqueue and the managed device driver.
+
+The management device driver typically takes the following steps to create a
+managed device:
+
+\begin{enumerate}
+\item Determine the virtio device id and device specific configuration.
+\item Create the managed devices by VIRTIO_TRANSPORTQ_CTRL_DEV_CREATE command.
+\item Optionally, configure the MSI entries by VIRTIO_TRANSPTQ_CTRL_MSI class commands
+\item The managed device driver probes the managed device.
+\end{enumerate}
+
+The management device driver will then use the transport virtqueue to
+process the requests from the managed device driver.
+
+Managed device setup performed by the managde device driver:
+\begin{itemize*}
+\item Features negotiation by VIRTIO_TRANSPTQ_CTRL_FEAT class commands.
+\item Device status management by VIRTIO_TRANSPTQ_CTRL_STATUS class commands.
+\item Device configuration space access by VIRTIO_TRANSPTQ_CTRL_CONFIG class commands.
+\end{itemize*}
+
+Virtqueues setup performed by the managed driver:
+\begin{itemize*}
+\item Set the virtqueue address by VIRTIO_TRANSPTQ_CTRL_VQ_ADDR_SET command.
+\item Set the virtqueue size by VIRTIO_TRANSPTQ_CTRL_VQ_SIZE class commands.
+\item Get the virtqueue notification area by VIRTIO_TRANSPTQ_CTRL_VQ_NOTIFY_GET command.
+\item Set the virtqueue state by VIRTIO_TRANSPTQ_CTRL_VQ_STATE_SET command.
+\item Optionally, set virtqueue asid by VIRTIO_TRANSPTQ_CTRL_VQ_ASID_SET command.
+\item Enable the virtqueue by VIRTIO_TRANSPTQ_CTRL_VQ_ENABLE_SET command.
+\end{itemize*}
+
 \chapter{Device Types}\label{sec:Device Types}
 
 On top of the queues, config space and feature negotiation facilities
-- 
2.35.3


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

[virtio-comment] [PATCH V3 RESEND 4/4] Add transport vq number for virtio blk and net

[Zhu Lingshan]

This commit add the queue number of the transport virtqueue
for virtio_blk and virtio_net

Signed-off-by: Jason Wang <jasowang@redhat.com>
Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com>
---
 content.tex | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/content.tex b/content.tex
index be632d1..ba129d4 100644
--- a/content.tex
+++ b/content.tex
@@ -3777,6 +3777,7 @@ \subsection{Virtqueues}\label{sec:Device Types / Network Device / Virtqueues}
 \item[2(N-1)] receiveqN
 \item[2(N-1)+1] transmitqN
 \item[2N] controlq
+\item[2N+1] transportq
 \end{description}
 
  N=1 if neither VIRTIO_NET_F_MQ nor VIRTIO_NET_F_RSS are negotiated, otherwise N is set by
@@ -3784,6 +3785,8 @@ \subsection{Virtqueues}\label{sec:Device Types / Network Device / Virtqueues}
 
  controlq only exists if VIRTIO_NET_F_CTRL_VQ set.
 
+ transportq only exists if VIRTIO_F_TRANSPT_VQ set
+
 \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits}
 
 \begin{description}
@@ -5363,11 +5366,14 @@ \subsection{Virtqueues}\label{sec:Device Types / Block Device / Virtqueues}
 \item[0] requestq1
 \item[\ldots]
 \item[N-1] requestqN
+\item[N] transportq
 \end{description}
 
  N=1 if VIRTIO_BLK_F_MQ is not negotiated, otherwise N is set by
  \field{num_queues}.
 
+ transportq only exists if VIRTIO_F_TRANSPT_VQ set.
+
 \subsection{Feature bits}\label{sec:Device Types / Block Device / Feature bits}
 
 \begin{description}
-- 
2.35.3


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/