* [virtio-comment] [PATCH v4 1/7] transport-fabrics: introduce Virtio Over Fabrics overview
2023-06-26 7:25 [virtio-comment] [PATCH v4 0/7] Introduce Virtio Over Fabrics zhenwei pi
@ 2023-06-26 7:25 ` zhenwei pi
2023-07-29 23:23 ` Raphael Norwitz
2023-11-15 14:20 ` [virtio-comment] " Stefan Hajnoczi
2023-06-26 7:25 ` [virtio-comment] [PATCH v4 2/7] transport-fabrics: introduce Virtio-oF Qualified Name zhenwei pi
` (6 subsequent siblings)
7 siblings, 2 replies; 33+ messages in thread
From: zhenwei pi @ 2023-06-26 7:25 UTC (permalink / raw)
To: stefanha, parav, mst, jasowang
Cc: virtio-comment, houp, helei.sig11, zhouhuaping.san, xinhao.kong,
zhenwei pi
In the past years, virtio supports lots of device specifications by
PCI/MMIO/CCW. These devices work fine in the virtualization environment.
Introduce Virtio Over Fabrics transport to support "network attached
peripheral devices". With this transport, Many Virtio based devices
work over fabrics. Note that the balloon device may not make sense,
shared memory regions won't work.
Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
---
content.tex | 1 +
transport-fabrics.tex | 32 ++++++++++++++++++++++++++++++++
2 files changed, 33 insertions(+)
create mode 100644 transport-fabrics.tex
diff --git a/content.tex b/content.tex
index d2ab9eb..bbbd79e 100644
--- a/content.tex
+++ b/content.tex
@@ -637,6 +637,7 @@ \chapter{Virtio Transport Options}\label{sec:Virtio Transport Options}
\input{transport-pci.tex}
\input{transport-mmio.tex}
\input{transport-ccw.tex}
+\input{transport-fabrics.tex}
\chapter{Device Types}\label{sec:Device Types}
diff --git a/transport-fabrics.tex b/transport-fabrics.tex
new file mode 100644
index 0000000..d10be2a
--- /dev/null
+++ b/transport-fabrics.tex
@@ -0,0 +1,32 @@
+\section{Virtio Over Fabrics}\label{sec:Virtio Transport Options / Virtio Over Fabrics}
+
+Virtio Over Fabrics (Virtio-oF) enables operations over fabrics that rely
+primarily on message passing.
+
+Virtio-oF uses a reliable connection to transmit data. The reliable
+connection facilitates communication between entities playing the following roles:
+
+\begin{itemize}
+\item A Virtio-oF initiator functions as a Virtio-oF client.
+The Virtio-oF initiator sends commands and associated data from the driver
+to the Virtio-oF target.
+\item A Virtio-oF target functions as a Virtio-oF server.
+The Virtio-oF target forwards commands to the device and sends completions
+and associated data back to the Virtio-oF initiator.
+\end{itemize}
+
+Virtio-oF has the following features:
+
+\begin{itemize}
+\item A Virtio-oF target is allowed to be connected by 0 or more Virtio-oF initiators.
+\item A Virtio-oF initiator is allowed to connect to a single Virtio-oF target only.
+A Virtio-oF device instance is a virtio device that the Virtio-oF initiator is
+accessing through the Virtio-oF target.
+\item There is a one-to-one mapping between the Virtio-oF queue and the reliable connection.
+\item There is one, and only one, Virtio-oF control queue for a Virtio-oF device instance.
+The Virtio-oF control queue is used to execute control commands,
+for example, to get the Virtio Device ID.
+\item There is a one-to-one mapping between virtqueue and Virtio-oF virtqueue
+which executes the bulk data transport on virtio devices.
+\item The arrival of data on the Virtio-oF queue indicates that a notification has arrived.
+\end{itemize}
--
2.25.1
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply related [flat|nested] 33+ messages in thread* Re: [virtio-comment] [PATCH v4 1/7] transport-fabrics: introduce Virtio Over Fabrics overview
2023-06-26 7:25 ` [virtio-comment] [PATCH v4 1/7] transport-fabrics: introduce Virtio Over Fabrics overview zhenwei pi
@ 2023-07-29 23:23 ` Raphael Norwitz
2023-07-31 4:44 ` zhenwei pi
2023-11-15 14:20 ` [virtio-comment] " Stefan Hajnoczi
1 sibling, 1 reply; 33+ messages in thread
From: Raphael Norwitz @ 2023-07-29 23:23 UTC (permalink / raw)
To: zhenwei pi
Cc: Stefan Hajnoczi, parav@nvidia.com, Michael S. Tsirkin, Jason Wang,
virtio-comment@lists.oasis-open.org, houp@yusur.tech,
helei.sig11@bytedance.com, zhouhuaping.san@bytedance.com,
xinhao.kong@duke.edu
Hey Zhenwei,
Interesting proposal!
I have taken a pass through the series and have some thoughts/feedback. I see potential value in a way to desegregate device processing from the hypervisor, while re-using a lot of the existing code and well defined abstractions in the virtio ecosystem.
For storage in particular it may have some performance impact by cutting out virtio processing from the hot path, but IMO there are better ways to achieve that working out of the box today. Your “eager buffers” are a nice touch for the write path, but come with their own costs I’ll describe later.
With that, I do see some limitations.
By design, this protocol gives the guest kernel direct access to the target. That in turn becomes a potential attack surface if a VM is compromised, whereas a virtio-over-{PCI/CCW/MMIO} abstraction shields the target from direct access.
While migration should work cleanly for a TCP fabric, have you thought about how migration will work with RDMA? In theory, as VFIO migration matures, it’s possible the guest kernel and user space stacks could be hardened to deal with hardware being ripped out from under it, but that may be a bit futuristic.
Also could you ever effectively migrate a VM from a TCP host to an RDMA accelerated one? The guest could have some smart logic to figure out “now my NIC doesn’t support RDMA, let me switch to TCP”. Today if you use RDMA acceleration behind any of the existing transports the guest does not have to worry about this and it can safely migrate anywhere.
Similarly what happens if you need to change the target IP or if one target goes down? Is there any way to handle that without making changes inside the guest? I think you should describe “redirection” behavior like iSCSI has, but I’ll describe more in the relevant patch.
For the storage use-case in general, I’m struggling to understand the advantages of virtio-over-fabrics over either the kernel NVMEoF or iSER initiator. It seems like it fundamentally boils down to a similar paradigm, the kernel submits IOVs to the target directly and it DMAs back and forth as required. In my experience, those who want to squeeze every last op out of their VMs skip virtio and do that anyways. For NVMeOF, the only limitation is that you need to use NVMe, but given how virtualization-friendly and ubiquitous it’s becoming for storage I don’t think that’s bad. iSER has some challenges but they are definitely surmountable.
Again, I like the “eager buffers", but if you’re thinking about storage performance, why not rather try introduce that into NVMEoF or iSER?
Other than storage, what devices will benefit from fabrics acceleration enough to compromise the ergonomics of the existing virtio transports?
In summary, I’d like to see:
[1] More discussion of the kinds of devices which will benefit from Virtio Over Fabrics.
[2] A comprehensive description of what the dataflow should look like. From having read the spec I get the gist but an example flow would be helpful for reference. Maybe just in the cover letter?
[3] If the goal is storage performance, a discussion of how this protocol provides performance benefits over and above the alternatives.
> On Jun 26, 2023, at 3:25 AM, zhenwei pi <pizhenwei@bytedance.com> wrote:
>
> In the past years, virtio supports lots of device specifications by
> PCI/MMIO/CCW. These devices work fine in the virtualization environment.
>
> Introduce Virtio Over Fabrics transport to support "network attached
> peripheral devices". With this transport, Many Virtio based devices
> work over fabrics. Note that the balloon device may not make sense,
Is it just balloon/shared memory devices that don’t make sense? Does virtio-net make sense?
Virtio-oF as described has fundamental drawbacks for SCSI in that a single virtio-scsi controller VQ may service many different iSCSI targets and your proposed protocol has one connection per VQ. Connections are likely to be made to different hosts for a single controller, so you’d have to create some abstraction to allow for multiple connections per VQ, or (more likely) rework the virtio-scsi device definition specifically for Virtio-oF.
> shared memory regions won't work.
>
> Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
> ---
> content.tex | 1 +
> transport-fabrics.tex | 32 ++++++++++++++++++++++++++++++++
> 2 files changed, 33 insertions(+)
> create mode 100644 transport-fabrics.tex
>
> diff --git a/content.tex b/content.tex
> index d2ab9eb..bbbd79e 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -637,6 +637,7 @@ \chapter{Virtio Transport Options}\label{sec:Virtio Transport Options}
> \input{transport-pci.tex}
> \input{transport-mmio.tex}
> \input{transport-ccw.tex}
> +\input{transport-fabrics.tex}
>
> \chapter{Device Types}\label{sec:Device Types}
>
> diff --git a/transport-fabrics.tex b/transport-fabrics.tex
> new file mode 100644
> index 0000000..d10be2a
> --- /dev/null
> +++ b/transport-fabrics.tex
> @@ -0,0 +1,32 @@
> +\section{Virtio Over Fabrics}\label{sec:Virtio Transport Options / Virtio Over Fabrics}
> +
> +Virtio Over Fabrics (Virtio-oF) enables operations over fabrics that rely
> +primarily on message passing.
> +
> +Virtio-oF uses a reliable connection to transmit data. The reliable
> +connection facilitates communication between entities playing the following roles:
> +
> +\begin{itemize}
> +\item A Virtio-oF initiator functions as a Virtio-oF client.
> +The Virtio-oF initiator sends commands and associated data from the driver
> +to the Virtio-oF target.
> +\item A Virtio-oF target functions as a Virtio-oF server.
> +The Virtio-oF target forwards commands to the device and sends completions
> +and associated data back to the Virtio-oF initiator.
> +\end{itemize}
> +
> +Virtio-oF has the following features:
> +
> +\begin{itemize}
> +\item A Virtio-oF target is allowed to be connected by 0 or more Virtio-oF initiators.
> +\item A Virtio-oF initiator is allowed to connect to a single Virtio-oF target only.
> +A Virtio-oF device instance is a virtio device that the Virtio-oF initiator is
> +accessing through the Virtio-oF target.
> +\item There is a one-to-one mapping between the Virtio-oF queue and the reliable connection.
Again - there are drawbacks for some device types (i.e. SCSI). Maybe soften the language to suggest that it could be extended in the future?
> +\item There is one, and only one, Virtio-oF control queue for a Virtio-oF device instance.
Any particular reason for that?
> +The Virtio-oF control queue is used to execute control commands,
> +for example, to get the Virtio Device ID.
> +\item There is a one-to-one mapping between virtqueue and Virtio-oF virtqueue
> +which executes the bulk data transport on virtio devices.
> +\item The arrival of data on the Virtio-oF queue indicates that a notification has arrived.
> +\end{itemize}
> --
> 2.25.1
>
>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>
>
^ permalink raw reply [flat|nested] 33+ messages in thread* Re: Re: [virtio-comment] [PATCH v4 1/7] transport-fabrics: introduce Virtio Over Fabrics overview
2023-07-29 23:23 ` Raphael Norwitz
@ 2023-07-31 4:44 ` zhenwei pi
2023-08-01 0:20 ` Raphael Norwitz
0 siblings, 1 reply; 33+ messages in thread
From: zhenwei pi @ 2023-07-31 4:44 UTC (permalink / raw)
To: Raphael Norwitz
Cc: Stefan Hajnoczi, parav@nvidia.com, Michael S. Tsirkin, Jason Wang,
virtio-comment@lists.oasis-open.org, houp@yusur.tech,
zhouhuaping.san@bytedance.com, xinhao.kong@duke.edu
On 7/30/23 07:23, Raphael Norwitz wrote:
> Hey Zhenwei,
>
> Interesting proposal!
>
> I have taken a pass through the series and have some thoughts/feedback. I see potential value in a way to desegregate device processing from the hypervisor, while re-using a lot of the existing code and well defined abstractions in the virtio ecosystem.
>
Hi Raphael,
From the point of my view, Virtio-oF will support several scenarios:
[1] host OS/containers. The host kernel connects the target directly,
then the host OS emulate a virtio device.
[2] hypervisor. A hypervisor handles request from guest, then translate
the request into Virtio-oF, forward requests to target.
[3] vhost-user. Like hypervisor, but has a higher performance.
[4] virtio hardware acceleration on DPU/smartNIC.
> For storage in particular it may have some performance impact by cutting out virtio processing from the hot path, but IMO there are better ways to achieve that working out of the box today. Your “eager buffers” are a nice touch for the write path, but come with their own costs I’ll describe later.
>
> With that, I do see some limitations.
>
> By design, this protocol gives the guest kernel direct access to the target. That in turn becomes a potential attack surface if a VM is compromised, whereas a virtio-over-{PCI/CCW/MMIO} abstraction shields the target from direct access.
>
The eager buffer is WRITE-only MR to the initiator, would you please
give me more hints about 'potential attack'?
> While migration should work cleanly for a TCP fabric, have you thought about how migration will work with RDMA? In theory, as VFIO migration matures, it’s possible the guest kernel and user space stacks could be hardened to deal with hardware being ripped out from under it, but that may be a bit futuristic.
>
Do you mean the VM migration in scenario[2] and scenario[3]? as far as I
can see, draining the inflight requests would be fine on migration.
> Also could you ever effectively migrate a VM from a TCP host to an RDMA accelerated one? The guest could have some smart logic to figure out “now my NIC doesn’t support RDMA, let me switch to TCP”. Today if you use RDMA acceleration behind any of the existing transports the guest does not have to worry about this and it can safely migrate anywhere.
>
Frankly, I don't think this should be part of 'Virtio-oF specification'.
From the point of my view, the specification defines the transport
binding only, the uplayer plays 'smart-select' game.
> Similarly what happens if you need to change the target IP or if one target goes down? Is there any way to handle that without making changes inside the guest? I think you should describe “redirection” behavior like iSCSI has, but I’ll describe more in the relevant patch.
>
Let's continue this discussion in
'[PATCH v4 4/7] transport-fabrics: introduce command set'
> For the storage use-case in general, I’m struggling to understand the advantages of virtio-over-fabrics over either the kernel NVMEoF or iSER initiator. It seems like it fundamentally boils down to a similar paradigm, the kernel submits IOVs to the target directly and it DMAs back and forth as required. In my experience, those who want to squeeze every last op out of their VMs skip virtio and do that anyways. For NVMeOF, the only limitation is that you need to use NVMe, but given how virtualization-friendly and ubiquitous it’s becoming for storage I don’t think that’s bad. iSER has some challenges but they are definitely surmountable.
>
> Again, I like the “eager buffers", but if you’re thinking about storage performance, why not rather try introduce that into NVMEoF or iSER?
>
> Other than storage, what devices will benefit from fabrics acceleration enough to compromise the ergonomics of the existing virtio transports?
>
> In summary, I’d like to see:
>
> [1] More discussion of the kinds of devices which will benefit from Virtio Over Fabrics.
About 2022 Q1, I developed virtio-crypto akcipher service(currently RSA
offload supported only), this allows HTTPS performance(Nginx+openssl) up
to ~300% in a guest.
In our product environment, we use a lot of QAT cards as a
akcipher-service-pool, and Crypto Over Fabrics is used between QEMU and
server.
Then the VM has no dependence on host server, and support VM migration.
On the other hand, I have a plan to develop GPU Over Fabrics for the
virtual desktop. So I have to develop a GPU Over Fabrics protocol...
Once the Virtio-Camera or Virtio-Video gets ready, Camera Over Fabrics
is needed for camera redirect ...
Rather than XXX Over Fabrics again and again, I'd like to re-use Virtio
devices, a single Virtio-oF is enough. This is the background and
motivation about Virtio-oF.
> [2] A comprehensive description of what the dataflow should look like. From having read the spec I get the gist but an example flow would be helpful for reference. Maybe just in the cover letter?
>
> [3] If the goal is storage performance, a discussion of how this protocol provides performance benefits over and above the alternatives.
>
As the above description, the goal is *NOT* storage performance.
Frankly, I don't think there is essential difference between
NVMe-oF(multi queues supported) and Virtio-oF blk(multi queues
supported), but different with iSCSI/iSER(single queue supported). The
format of command(SCSI/NVMe/Virtio) hardly affects the storage performance.
>> On Jun 26, 2023, at 3:25 AM, zhenwei pi <pizhenwei@bytedance.com> wrote:
>>
>> In the past years, virtio supports lots of device specifications by
>> PCI/MMIO/CCW. These devices work fine in the virtualization environment.
>>
>> Introduce Virtio Over Fabrics transport to support "network attached
>> peripheral devices". With this transport, Many Virtio based devices
>> work over fabrics. Note that the balloon device may not make sense,
>
> Is it just balloon/shared memory devices that don’t make sense? Does virtio-net make sense?
>
> Virtio-oF as described has fundamental drawbacks for SCSI in that a single virtio-scsi controller VQ may service many different iSCSI targets and your proposed protocol has one connection per VQ. Connections are likely to be made to different hosts for a single controller, so you’d have to create some abstraction to allow for multiple connections per VQ, or (more likely) rework the virtio-scsi device definition specifically for Virtio-oF.
>
>
>> shared memory regions won't work.
>>
>> Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
>> ---
>> content.tex | 1 +
>> transport-fabrics.tex | 32 ++++++++++++++++++++++++++++++++
>> 2 files changed, 33 insertions(+)
>> create mode 100644 transport-fabrics.tex
>>
>> diff --git a/content.tex b/content.tex
>> index d2ab9eb..bbbd79e 100644
>> --- a/content.tex
>> +++ b/content.tex
>> @@ -637,6 +637,7 @@ \chapter{Virtio Transport Options}\label{sec:Virtio Transport Options}
>> \input{transport-pci.tex}
>> \input{transport-mmio.tex}
>> \input{transport-ccw.tex}
>> +\input{transport-fabrics.tex}
>>
>> \chapter{Device Types}\label{sec:Device Types}
>>
>> diff --git a/transport-fabrics.tex b/transport-fabrics.tex
>> new file mode 100644
>> index 0000000..d10be2a
>> --- /dev/null
>> +++ b/transport-fabrics.tex
>> @@ -0,0 +1,32 @@
>> +\section{Virtio Over Fabrics}\label{sec:Virtio Transport Options / Virtio Over Fabrics}
>> +
>> +Virtio Over Fabrics (Virtio-oF) enables operations over fabrics that rely
>> +primarily on message passing.
>> +
>> +Virtio-oF uses a reliable connection to transmit data. The reliable
>> +connection facilitates communication between entities playing the following roles:
>> +
>> +\begin{itemize}
>> +\item A Virtio-oF initiator functions as a Virtio-oF client.
>> +The Virtio-oF initiator sends commands and associated data from the driver
>> +to the Virtio-oF target.
>> +\item A Virtio-oF target functions as a Virtio-oF server.
>> +The Virtio-oF target forwards commands to the device and sends completions
>> +and associated data back to the Virtio-oF initiator.
>> +\end{itemize}
>> +
>> +Virtio-oF has the following features:
>> +
>> +\begin{itemize}
>> +\item A Virtio-oF target is allowed to be connected by 0 or more Virtio-oF initiators.
>> +\item A Virtio-oF initiator is allowed to connect to a single Virtio-oF target only.
>> +A Virtio-oF device instance is a virtio device that the Virtio-oF initiator is
>> +accessing through the Virtio-oF target.
>> +\item There is a one-to-one mapping between the Virtio-oF queue and the reliable connection.
>
> Again - there are drawbacks for some device types (i.e. SCSI). Maybe soften the language to suggest that it could be extended in the future?
>
>> +\item There is one, and only one, Virtio-oF control queue for a Virtio-oF device instance.
>
> Any particular reason for that?
>
--
zhenwei pi
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 33+ messages in thread* Re: [virtio-comment] [PATCH v4 1/7] transport-fabrics: introduce Virtio Over Fabrics overview
2023-07-31 4:44 ` zhenwei pi
@ 2023-08-01 0:20 ` Raphael Norwitz
0 siblings, 0 replies; 33+ messages in thread
From: Raphael Norwitz @ 2023-08-01 0:20 UTC (permalink / raw)
To: zhenwei pi
Cc: Stefan Hajnoczi, parav@nvidia.com, Michael S. Tsirkin, Jason Wang,
virtio-comment@lists.oasis-open.org, houp@yusur.tech,
zhouhuaping.san@bytedance.com, xinhao.kong@duke.edu
Some quick responses. I’ll get back to you on patches 3 and 5 in a while.
> On Jul 31, 2023, at 12:44 AM, zhenwei pi <pizhenwei@bytedance.com> wrote:
>
>
>
> On 7/30/23 07:23, Raphael Norwitz wrote:
>> Hey Zhenwei,
>> Interesting proposal!
>> I have taken a pass through the series and have some thoughts/feedback. I see potential value in a way to desegregate device processing from the hypervisor, while re-using a lot of the existing code and well defined abstractions in the virtio ecosystem.
>
> Hi Raphael,
>
> From the point of my view, Virtio-oF will support several scenarios:
> [1] host OS/containers. The host kernel connects the target directly, then the host OS emulate a virtio device.
> [2] hypervisor. A hypervisor handles request from guest, then translate the request into Virtio-oF, forward requests to target.
> [3] vhost-user. Like hypervisor, but has a higher performance.
> [4] virtio hardware acceleration on DPU/smartNIC.
The only difference between 2 and 3 being emulation in user space vs the kernel?
>
>> For storage in particular it may have some performance impact by cutting out virtio processing from the hot path, but IMO there are better ways to achieve that working out of the box today. Your “eager buffers” are a nice touch for the write path, but come with their own costs I’ll describe later.
>> With that, I do see some limitations.
>> By design, this protocol gives the guest kernel direct access to the target. That in turn becomes a potential attack surface if a VM is compromised, whereas a virtio-over-{PCI/CCW/MMIO} abstraction shields the target from direct access.
>
> The eager buffer is WRITE-only MR to the initiator, would you please give me more hints about 'potential attack'?
I was imagining something like scenario [1] where an untrusted guest was given access to a Virtio-OF target, like in-guest iSCSI or NVMEoF.
I’m not saying it necessarily creates vulnerabilities, just that if you use it for storage by giving the guest direct access to the target it has security implications which virtio over PCI doesn’t have. It is certainly no worse than in-guest iSCSI or NVMEoF.
>
>> While migration should work cleanly for a TCP fabric, have you thought about how migration will work with RDMA? In theory, as VFIO migration matures, it’s possible the guest kernel and user space stacks could be hardened to deal with hardware being ripped out from under it, but that may be a bit futuristic.
>
> Do you mean the VM migration in scenario[2] and scenario[3]? as far as I can see, draining the inflight requests would be fine on migration.
I was talking about scenario [1]-like cases. Correct - migration concerns go away with your scenarios [2] and [3]. Probably [4] too.
>
>> Also could you ever effectively migrate a VM from a TCP host to an RDMA accelerated one? The guest could have some smart logic to figure out “now my NIC doesn’t support RDMA, let me switch to TCP”. Today if you use RDMA acceleration behind any of the existing transports the guest does not have to worry about this and it can safely migrate anywhere.
>
> Frankly, I don't think this should be part of 'Virtio-oF specification'. From the point of my view, the specification defines the transport binding only, the uplayer plays 'smart-select' game.
Agreed - it should not be a part of the spec.
I was just trying to point out tradeoffs and complications I imagined with an eye towards storage.
>
>> Similarly what happens if you need to change the target IP or if one target goes down? Is there any way to handle that without making changes inside the guest? I think you should describe “redirection” behavior like iSCSI has, but I’ll describe more in the relevant patch.
>
> Let's continue this discussion in
> '[PATCH v4 4/7] transport-fabrics: introduce command set'
>
>> For the storage use-case in general, I’m struggling to understand the advantages of virtio-over-fabrics over either the kernel NVMEoF or iSER initiator. It seems like it fundamentally boils down to a similar paradigm, the kernel submits IOVs to the target directly and it DMAs back and forth as required. In my experience, those who want to squeeze every last op out of their VMs skip virtio and do that anyways. For NVMeOF, the only limitation is that you need to use NVMe, but given how virtualization-friendly and ubiquitous it’s becoming for storage I don’t think that’s bad. iSER has some challenges but they are definitely surmountable.
>> Again, I like the “eager buffers", but if you’re thinking about storage performance, why not rather try introduce that into NVMEoF or iSER?
>> Other than storage, what devices will benefit from fabrics acceleration enough to compromise the ergonomics of the existing virtio transports?
>> In summary, I’d like to see:
>> [1] More discussion of the kinds of devices which will benefit from Virtio Over Fabrics.
>
> About 2022 Q1, I developed virtio-crypto akcipher service(currently RSA offload supported only), this allows HTTPS performance(Nginx+openssl) up to ~300% in a guest.
>
> In our product environment, we use a lot of QAT cards as a akcipher-service-pool, and Crypto Over Fabrics is used between QEMU and server.
> Then the VM has no dependence on host server, and support VM migration.
>
> On the other hand, I have a plan to develop GPU Over Fabrics for the virtual desktop. So I have to develop a GPU Over Fabrics protocol...
>
> Once the Virtio-Camera or Virtio-Video gets ready, Camera Over Fabrics is needed for camera redirect ...
>
> Rather than XXX Over Fabrics again and again, I'd like to re-use Virtio devices, a single Virtio-oF is enough. This is the background and motivation about Virtio-oF.
>
All sound like very compelling use cases. That clears up my main concern.
>> [2] A comprehensive description of what the dataflow should look like. From having read the spec I get the gist but an example flow would be helpful for reference. Maybe just in the cover letter?
>> [3] If the goal is storage performance, a discussion of how this protocol provides performance benefits over and above the alternatives.
>
> As the above description, the goal is *NOT* storage performance.
Thanks for making that clear. I had thought this was intended to be a storage protocol.
> Frankly, I don't think there is essential difference between NVMe-oF(multi queues supported) and Virtio-oF blk(multi queues supported), but different with iSCSI/iSER(single queue supported). The format of command(SCSI/NVMe/Virtio) hardly affects the storage performance.
>
Great - we’re on the same page as far as storage is concerned.
>>> On Jun 26, 2023, at 3:25 AM, zhenwei pi <pizhenwei@bytedance.com> wrote:
>>>
>>> In the past years, virtio supports lots of device specifications by
>>> PCI/MMIO/CCW. These devices work fine in the virtualization environment.
>>>
>>> Introduce Virtio Over Fabrics transport to support "network attached
>>> peripheral devices". With this transport, Many Virtio based devices
>>> work over fabrics. Note that the balloon device may not make sense,
>> Is it just balloon/shared memory devices that don’t make sense? Does virtio-net make sense?
>> Virtio-oF as described has fundamental drawbacks for SCSI in that a single virtio-scsi controller VQ may service many different iSCSI targets and your proposed protocol has one connection per VQ. Connections are likely to be made to different hosts for a single controller, so you’d have to create some abstraction to allow for multiple connections per VQ, or (more likely) rework the virtio-scsi device definition specifically for Virtio-oF.
>>> shared memory regions won't work.
>>>
>>> Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
>>> ---
>>> content.tex | 1 +
>>> transport-fabrics.tex | 32 ++++++++++++++++++++++++++++++++
>>> 2 files changed, 33 insertions(+)
>>> create mode 100644 transport-fabrics.tex
>>>
>>> diff --git a/content.tex b/content.tex
>>> index d2ab9eb..bbbd79e 100644
>>> --- a/content.tex
>>> +++ b/content.tex
>>> @@ -637,6 +637,7 @@ \chapter{Virtio Transport Options}\label{sec:Virtio Transport Options}
>>> \input{transport-pci.tex}
>>> \input{transport-mmio.tex}
>>> \input{transport-ccw.tex}
>>> +\input{transport-fabrics.tex}
>>>
>>> \chapter{Device Types}\label{sec:Device Types}
>>>
>>> diff --git a/transport-fabrics.tex b/transport-fabrics.tex
>>> new file mode 100644
>>> index 0000000..d10be2a
>>> --- /dev/null
>>> +++ b/transport-fabrics.tex
>>> @@ -0,0 +1,32 @@
>>> +\section{Virtio Over Fabrics}\label{sec:Virtio Transport Options / Virtio Over Fabrics}
>>> +
>>> +Virtio Over Fabrics (Virtio-oF) enables operations over fabrics that rely
>>> +primarily on message passing.
>>> +
>>> +Virtio-oF uses a reliable connection to transmit data. The reliable
>>> +connection facilitates communication between entities playing the following roles:
>>> +
>>> +\begin{itemize}
>>> +\item A Virtio-oF initiator functions as a Virtio-oF client.
>>> +The Virtio-oF initiator sends commands and associated data from the driver
>>> +to the Virtio-oF target.
>>> +\item A Virtio-oF target functions as a Virtio-oF server.
>>> +The Virtio-oF target forwards commands to the device and sends completions
>>> +and associated data back to the Virtio-oF initiator.
>>> +\end{itemize}
>>> +
>>> +Virtio-oF has the following features:
>>> +
>>> +\begin{itemize}
>>> +\item A Virtio-oF target is allowed to be connected by 0 or more Virtio-oF initiators.
>>> +\item A Virtio-oF initiator is allowed to connect to a single Virtio-oF target only.
>>> +A Virtio-oF device instance is a virtio device that the Virtio-oF initiator is
>>> +accessing through the Virtio-oF target.
>>> +\item There is a one-to-one mapping between the Virtio-oF queue and the reliable connection.
>> Again - there are drawbacks for some device types (i.e. SCSI). Maybe soften the language to suggest that it could be extended in the future?
>>> +\item There is one, and only one, Virtio-oF control queue for a Virtio-oF device instance.
>> Any particular reason for that?
>
>
> --
> zhenwei pi
^ permalink raw reply [flat|nested] 33+ messages in thread
* [virtio-comment] Re: [PATCH v4 1/7] transport-fabrics: introduce Virtio Over Fabrics overview
2023-06-26 7:25 ` [virtio-comment] [PATCH v4 1/7] transport-fabrics: introduce Virtio Over Fabrics overview zhenwei pi
2023-07-29 23:23 ` Raphael Norwitz
@ 2023-11-15 14:20 ` Stefan Hajnoczi
1 sibling, 0 replies; 33+ messages in thread
From: Stefan Hajnoczi @ 2023-11-15 14:20 UTC (permalink / raw)
To: zhenwei pi
Cc: parav, mst, jasowang, virtio-comment, houp, helei.sig11,
zhouhuaping.san, xinhao.kong
[-- Attachment #1: Type: text/plain, Size: 2656 bytes --]
On Mon, Jun 26, 2023 at 03:25:45PM +0800, zhenwei pi wrote:
> In the past years, virtio supports lots of device specifications by
> PCI/MMIO/CCW. These devices work fine in the virtualization environment.
>
> Introduce Virtio Over Fabrics transport to support "network attached
> peripheral devices". With this transport, Many Virtio based devices
> work over fabrics. Note that the balloon device may not make sense,
> shared memory regions won't work.
>
> Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
> ---
> content.tex | 1 +
> transport-fabrics.tex | 32 ++++++++++++++++++++++++++++++++
> 2 files changed, 33 insertions(+)
> create mode 100644 transport-fabrics.tex
>
> diff --git a/content.tex b/content.tex
> index d2ab9eb..bbbd79e 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -637,6 +637,7 @@ \chapter{Virtio Transport Options}\label{sec:Virtio Transport Options}
> \input{transport-pci.tex}
> \input{transport-mmio.tex}
> \input{transport-ccw.tex}
> +\input{transport-fabrics.tex}
>
> \chapter{Device Types}\label{sec:Device Types}
>
> diff --git a/transport-fabrics.tex b/transport-fabrics.tex
> new file mode 100644
> index 0000000..d10be2a
> --- /dev/null
> +++ b/transport-fabrics.tex
> @@ -0,0 +1,32 @@
> +\section{Virtio Over Fabrics}\label{sec:Virtio Transport Options / Virtio Over Fabrics}
> +
> +Virtio Over Fabrics (Virtio-oF) enables operations over fabrics that rely
> +primarily on message passing.
> +
> +Virtio-oF uses a reliable connection to transmit data. The reliable
> +connection facilitates communication between entities playing the following roles:
> +
> +\begin{itemize}
> +\item A Virtio-oF initiator functions as a Virtio-oF client.
> +The Virtio-oF initiator sends commands and associated data from the driver
> +to the Virtio-oF target.
> +\item A Virtio-oF target functions as a Virtio-oF server.
> +The Virtio-oF target forwards commands to the device and sends completions
> +and associated data back to the Virtio-oF initiator.
> +\end{itemize}
> +
> +Virtio-oF has the following features:
> +
> +\begin{itemize}
> +\item A Virtio-oF target is allowed to be connected by 0 or more Virtio-oF initiators.
> +\item A Virtio-oF initiator is allowed to connect to a single Virtio-oF target only.
> +A Virtio-oF device instance is a virtio device that the Virtio-oF initiator is
> +accessing through the Virtio-oF target.
Let's make sure the protocol is extensible so that multi-path or
persistent sessions (support for reconnect) are possible in the future.
They don't need to be there from the beginning.
Stefan
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 33+ messages in thread
* [virtio-comment] [PATCH v4 2/7] transport-fabrics: introduce Virtio-oF Qualified Name
2023-06-26 7:25 [virtio-comment] [PATCH v4 0/7] Introduce Virtio Over Fabrics zhenwei pi
2023-06-26 7:25 ` [virtio-comment] [PATCH v4 1/7] transport-fabrics: introduce Virtio Over Fabrics overview zhenwei pi
@ 2023-06-26 7:25 ` zhenwei pi
2023-07-29 23:24 ` Raphael Norwitz
2023-06-26 7:25 ` [virtio-comment] [PATCH v4 3/7] transport-fabrics: introduce Virtio-oF Protocol Data Unit zhenwei pi
` (5 subsequent siblings)
7 siblings, 1 reply; 33+ messages in thread
From: zhenwei pi @ 2023-06-26 7:25 UTC (permalink / raw)
To: stefanha, parav, mst, jasowang
Cc: virtio-comment, houp, helei.sig11, zhouhuaping.san, xinhao.kong,
zhenwei pi
Add VQN section. The VQN is a little different from iSCSI/NVMe-oF on
style limitation. Because iSCSI/NVMe-of is storage specific protocol,
the full string IQN(for iSCSI/iSER) and NQN(for NVMe-oF) represents
a "storage access address". However, Virtio Over Fabrics works as
transport layer rather than device layer, a URL style string is better
to Virtio Over Fabrics. For example:
virtio-rdma://192.168.1.100:8549/blk-device/nvme-pool/849a39ad-8d7b-4a7a-adb6-e7407ace532c
virtio-tcpip://192.168.1.110/blk-device/hdd-pool/238151a7-acd7-4621-bbdf-382ddbccb6a1
virtio-tcpip://192.168.1.100:8549/crypto-device/25307f22-e5a8-4ea2-b7ca-79f5c3bebc3c
...
A hunam readable VQN is helpful to maintain/debug/distinguish.
Note that this section only defines the "VQN" schema.
For a process, the following two are both fine:
~# ./foo --full-url virtio-rdma://192.168.1.100:8549/blk-device/nvme-pool/849a39ad-8d7b-4a7a-adb6-e7407ace532c
~# ./foo --transport rdma --address 192.168.1.100 --port 8549
--tvqn blk-device/hdd-pool/238151a7-acd7-4621-bbdf-382ddbccb6a1
Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
---
transport-fabrics.tex | 15 +++++++++++++++
1 file changed, 15 insertions(+)
diff --git a/transport-fabrics.tex b/transport-fabrics.tex
index d10be2a..54d7558 100644
--- a/transport-fabrics.tex
+++ b/transport-fabrics.tex
@@ -30,3 +30,18 @@ \section{Virtio Over Fabrics}\label{sec:Virtio Transport Options / Virtio Over F
which executes the bulk data transport on virtio devices.
\item The arrival of data on the Virtio-oF queue indicates that a notification has arrived.
\end{itemize}
+
+
+\subsection{Virtio-oF Qualified Name}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Virtio-oF Qualified Name}
+Virtio-oF Qualified Names (VQNs) are used to uniquely describe a Virtio-oF initiator
+or a Virtio-oF target for identification.
+
+A VQN is encoded as a string of Unicode characters with the following properties:
+
+\begin{itemize}
+\item The encoding is UTF-8 (refer to RFC 3629).
+\item The characters dash('-'), dot ('.') and slash('/') are used in formatting.
+\item The string is NUL terminated.
+\item The maximum name is 256 bytes in length, including the NUL character.
+\item There is no strict style limitation.
+\end{itemize}
--
2.25.1
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply related [flat|nested] 33+ messages in thread* Re: [virtio-comment] [PATCH v4 2/7] transport-fabrics: introduce Virtio-oF Qualified Name
2023-06-26 7:25 ` [virtio-comment] [PATCH v4 2/7] transport-fabrics: introduce Virtio-oF Qualified Name zhenwei pi
@ 2023-07-29 23:24 ` Raphael Norwitz
0 siblings, 0 replies; 33+ messages in thread
From: Raphael Norwitz @ 2023-07-29 23:24 UTC (permalink / raw)
To: zhenwei pi
Cc: Stefan Hajnoczi, parav@nvidia.com, Michael S. Tsirkin, Jason Wang,
virtio-comment@lists.oasis-open.org, houp@yusur.tech,
helei.sig11@bytedance.com, zhouhuaping.san@bytedance.com,
xinhao.kong@duke.edu
> On Jun 26, 2023, at 3:25 AM, zhenwei pi <pizhenwei@bytedance.com> wrote:
>
> Add VQN section. The VQN is a little different from iSCSI/NVMe-oF on
> style limitation. Because iSCSI/NVMe-of is storage specific protocol,
> the full string IQN(for iSCSI/iSER) and NQN(for NVMe-oF) represents
> a "storage access address". However, Virtio Over Fabrics works as
> transport layer rather than device layer, a URL style string is better
> to Virtio Over Fabrics. For example:
> virtio-rdma://192.168.1.100:8549/blk-device/nvme-pool/849a39ad-8d7b-4a7a-adb6-e7407ace532c
> virtio-tcpip://192.168.1.110/blk-device/hdd-pool/238151a7-acd7-4621-bbdf-382ddbccb6a1
> virtio-tcpip://192.168.1.100:8549/crypto-device/25307f22-e5a8-4ea2-b7ca-79f5c3bebc3c
> ...
>
> A hunam readable VQN is helpful to maintain/debug/distinguish.
s/hunam/human
> Note that this section only defines the "VQN" schema.
> For a process, the following two are both fine:
> ~# ./foo --full-url virtio-rdma://192.168.1.100:8549/blk-device/nvme-pool/849a39ad-8d7b-4a7a-adb6-e7407ace532c
> ~# ./foo --transport rdma --address 192.168.1.100 --port 8549
> --tvqn blk-device/hdd-pool/238151a7-acd7-4621-bbdf-382ddbccb6a1
>
> Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
> ---
> transport-fabrics.tex | 15 +++++++++++++++
> 1 file changed, 15 insertions(+)
>
> diff --git a/transport-fabrics.tex b/transport-fabrics.tex
> index d10be2a..54d7558 100644
> --- a/transport-fabrics.tex
> +++ b/transport-fabrics.tex
> @@ -30,3 +30,18 @@ \section{Virtio Over Fabrics}\label{sec:Virtio Transport Options / Virtio Over F
> which executes the bulk data transport on virtio devices.
> \item The arrival of data on the Virtio-oF queue indicates that a notification has arrived.
> \end{itemize}
> +
> +
> +\subsection{Virtio-oF Qualified Name}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Virtio-oF Qualified Name}
> +Virtio-oF Qualified Names (VQNs) are used to uniquely describe a Virtio-oF initiator
> +or a Virtio-oF target for identification.
> +
> +A VQN is encoded as a string of Unicode characters with the following properties:
> +
> +\begin{itemize}
> +\item The encoding is UTF-8 (refer to RFC 3629).
> +\item The characters dash('-'), dot ('.') and slash('/') are used in formatting.
> +\item The string is NUL terminated.
> +\item The maximum name is 256 bytes in length, including the NUL character.
> +\item There is no strict style limitation.
> +\end{itemize}
> --
> 2.25.1
>
>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>
>
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 33+ messages in thread
* [virtio-comment] [PATCH v4 3/7] transport-fabrics: introduce Virtio-oF Protocol Data Unit
2023-06-26 7:25 [virtio-comment] [PATCH v4 0/7] Introduce Virtio Over Fabrics zhenwei pi
2023-06-26 7:25 ` [virtio-comment] [PATCH v4 1/7] transport-fabrics: introduce Virtio Over Fabrics overview zhenwei pi
2023-06-26 7:25 ` [virtio-comment] [PATCH v4 2/7] transport-fabrics: introduce Virtio-oF Qualified Name zhenwei pi
@ 2023-06-26 7:25 ` zhenwei pi
2023-07-29 23:25 ` Raphael Norwitz
2023-06-26 7:25 ` [virtio-comment] [PATCH v4 4/7] transport-fabrics: introduce command set zhenwei pi
` (4 subsequent siblings)
7 siblings, 1 reply; 33+ messages in thread
From: zhenwei pi @ 2023-06-26 7:25 UTC (permalink / raw)
To: stefanha, parav, mst, jasowang
Cc: virtio-comment, houp, helei.sig11, zhouhuaping.san, xinhao.kong,
zhenwei pi
Introduce Virtio-oF PDU for Virtio-oF queue, add Stream Data Transfers
and Keyed Data Transfers mechanism.
Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
---
transport-fabrics.tex | 68 +++++++++++++++++++++++++++++++++++++++++++
1 file changed, 68 insertions(+)
diff --git a/transport-fabrics.tex b/transport-fabrics.tex
index 54d7558..d562b2c 100644
--- a/transport-fabrics.tex
+++ b/transport-fabrics.tex
@@ -45,3 +45,71 @@ \subsection{Virtio-oF Qualified Name}\label{sec:Virtio Transport Options / Virti
\item The maximum name is 256 bytes in length, including the NUL character.
\item There is no strict style limitation.
\end{itemize}
+
+
+\subsection{Protocol Data Unit}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Protocol Data Unit}
+This section defines Virtio-oF Protocol Data Unit (PDU) for both the Virtio-oF control queue and Virtio-oF virtqueue.
+
+Virtio-oF PDU is a unit of information exchanged between a Virtio-oF initiator and a Virtio-oF target:
+\begin{itemize}
+\item A request Virtio-oF PDU from Virtio-oF initiator to Virtio-oF target contains command and associated data, if present.
+\item A response Virtio-oF PDU from Virtio-oF target to Virtio-oF initiator contains completion and associated data, if present.
+\end{itemize}
+
+\subsubsection{Stream Data Transfers}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Protocol Data Unit/ Stream Data Transfers}
+Stream-based (i.e. TCP/IP) Virtio-oF queue transfers Virtio-oF PDUs in a streaming fashion.
+
+The layout in a stream:
+\begin{lstlisting}
+PDUx and PDUz contain a command only, PDUy contains a command and data:
+
+ +----+ +---------+ +----+
+ |PDUx| | PDUy | |PDUz|
+ ... +----+ ... +----+----+ ... +----+ ...
+ |CMD | |CMD |Data| |CMD |
+ +----+ +---------+ +----+
+
+PDUl contains completion only, PDUm and PDUn contain completion and data:
+
+ +----+ +---------+ +---------+
+ |PDUl| | PDUm | | PDUn |
+ ... +----+ ... +----+----+ ... +----+----+ ...
+ |COMP| |COMP|Data| |COMP|Data|
+ +----+ +---------+ +---------+
+\end{lstlisting}
+
+\subsubsection{Keyed Data Transfers}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Protocol Data Unit/ Keyed Data Transfers}
+Message-based (i.e. RDMA) Virtio-oF queue transfers Virtio-oF PDUs in a message fashion, and uses the following structure to describe the remote data:
+
+\begin{lstlisting}
+struct virtio_of_keyed_desc {
+ /* the remote address of data */
+ le64 addr;
+ /* the length of data */
+ le32 length;
+ /* the key to access the remote data */
+ le32 key;
+};
+\end{lstlisting}
+
+A Virtio-oF control queue supports 1 keyed descriptor, and a Virtio-oF virtqueue supports 1 or more keyed descriptors.
+
+The PDUs of messages:
+\begin{lstlisting}
+PDUx contains a command only, PDUy contains a command and 1 descriptor,
+and PDUz contains a command and k - j + 1 descriptors.
+
+ +----+ +---------+ +--------------------+
+ |PDUx| | PDUy | | PDUz |
+ ... +----+ ... +----+----+ ... +----+-----+---+-----+ ...
+ |CMD | |CMD |DESC| |CMD |DESCj|...|DESCk|
+ +----+ +---------+ +----------+---+-----+
+
+PDUl, PDUm, and PDUn contain completion only.
+
+ +----+ +----+ +----+
+ |PDUl| |PDUm| |PDUn|
+ ... +----+ ... +----+ ... +----+ ...
+ |COMP| |COMP| |COMP|
+ +----+ +----+ +----+
+\end{lstlisting}
--
2.25.1
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply related [flat|nested] 33+ messages in thread* Re: [virtio-comment] [PATCH v4 3/7] transport-fabrics: introduce Virtio-oF Protocol Data Unit
2023-06-26 7:25 ` [virtio-comment] [PATCH v4 3/7] transport-fabrics: introduce Virtio-oF Protocol Data Unit zhenwei pi
@ 2023-07-29 23:25 ` Raphael Norwitz
2023-07-31 2:28 ` zhenwei pi
0 siblings, 1 reply; 33+ messages in thread
From: Raphael Norwitz @ 2023-07-29 23:25 UTC (permalink / raw)
To: zhenwei pi
Cc: Stefan Hajnoczi, parav@nvidia.com, Michael S. Tsirkin,
jasowang@redhat.com, virtio-comment@lists.oasis-open.org,
houp@yusur.tech, helei.sig11@bytedance.com,
zhouhuaping.san@bytedance.com, xinhao.kong@duke.edu
I’d suggest making some tweaks to save memory on the target side for RDMA.
> On Jun 26, 2023, at 3:25 AM, zhenwei pi <pizhenwei@bytedance.com> wrote:
>
> Introduce Virtio-oF PDU for Virtio-oF queue, add Stream Data Transfers
> and Keyed Data Transfers mechanism.
>
> Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
> ---
> transport-fabrics.tex | 68 +++++++++++++++++++++++++++++++++++++++++++
> 1 file changed, 68 insertions(+)
>
> diff --git a/transport-fabrics.tex b/transport-fabrics.tex
> index 54d7558..d562b2c 100644
> --- a/transport-fabrics.tex
> +++ b/transport-fabrics.tex
> @@ -45,3 +45,71 @@ \subsection{Virtio-oF Qualified Name}\label{sec:Virtio Transport Options / Virti
> \item The maximum name is 256 bytes in length, including the NUL character.
> \item There is no strict style limitation.
> \end{itemize}
> +
> +
> +\subsection{Protocol Data Unit}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Protocol Data Unit}
> +This section defines Virtio-oF Protocol Data Unit (PDU) for both the Virtio-oF control queue and Virtio-oF virtqueue.
> +
> +Virtio-oF PDU is a unit of information exchanged between a Virtio-oF initiator and a Virtio-oF target:
> +\begin{itemize}
> +\item A request Virtio-oF PDU from Virtio-oF initiator to Virtio-oF target contains command and associated data, if present.
> +\item A response Virtio-oF PDU from Virtio-oF target to Virtio-oF initiator contains completion and associated data, if present.
> +\end{itemize}
> +
> +\subsubsection{Stream Data Transfers}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Protocol Data Unit/ Stream Data Transfers}
> +Stream-based (i.e. TCP/IP) Virtio-oF queue transfers Virtio-oF PDUs in a streaming fashion.
> +
> +The layout in a stream:
> +\begin{lstlisting}
> +PDUx and PDUz contain a command only, PDUy contains a command and data:
> +
> + +----+ +---------+ +----+
> + |PDUx| | PDUy | |PDUz|
> + ... +----+ ... +----+----+ ... +----+ ...
> + |CMD | |CMD |Data| |CMD |
> + +----+ +---------+ +----+
> +
> +PDUl contains completion only, PDUm and PDUn contain completion and data:
> +
> + +----+ +---------+ +---------+
> + |PDUl| | PDUm | | PDUn |
> + ... +----+ ... +----+----+ ... +----+----+ ...
> + |COMP| |COMP|Data| |COMP|Data|
> + +----+ +---------+ +---------+
> +\end{lstlisting}
> +
> +\subsubsection{Keyed Data Transfers}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Protocol Data Unit/ Keyed Data Transfers}
> +Message-based (i.e. RDMA) Virtio-oF queue transfers Virtio-oF PDUs in a message fashion, and uses the following structure to describe the remote data:
> +
> +\begin{lstlisting}
> +struct virtio_of_keyed_desc {
> + /* the remote address of data */
> + le64 addr;
> + /* the length of data */
> + le32 length;
> + /* the key to access the remote data */
> + le32 key;
> +};
> +\end{lstlisting}
> +
> +A Virtio-oF control queue supports 1 keyed descriptor, and a Virtio-oF virtqueue supports 1 or more keyed descriptors.
> +
> +The PDUs of messages:
> +\begin{lstlisting}
> +PDUx contains a command only, PDUy contains a command and 1 descriptor,
> +and PDUz contains a command and k - j + 1 descriptors.
> +
> + +----+ +---------+ +--------------------+
> + |PDUx| | PDUy | | PDUz |
> + ... +----+ ... +----+----+ ... +----+-----+---+-----+ ...
> + |CMD | |CMD |DESC| |CMD |DESCj|...|DESCk|
> + +----+ +---------+ +----------+---+-----+
On the target side you have to prepost (MAX_SG_LEN * sizeof(virtio_of_keyed_desc) + sizeof(virtio_command_vq)) * max outstanding for each connection.
Assuming a 128 max outstanding and MAX_SG_LEN=1024 (QEMU’s IOV_INTERNAL) that puts you at (1024 * 8 + 13) * 128 ~= 1MB per connection. If a target is servicing thousands of connections that adds up.
I’d suggest adding a protocol message to cap the MAX_SG_LEN and communicate the value to the target.
Would also be great to save space in virtio_of_keyed_desc, but I don’t think there is a way.
> +
> +PDUl, PDUm, and PDUn contain completion only.
> +
> + +----+ +----+ +----+
> + |PDUl| |PDUm| |PDUn|
> + ... +----+ ... +----+ ... +----+ ...
> + |COMP| |COMP| |COMP|
> + +----+ +----+ +----+
> +\end{lstlisting}
> --
> 2.25.1
>
>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>
>
^ permalink raw reply [flat|nested] 33+ messages in thread* Re: Re: [virtio-comment] [PATCH v4 3/7] transport-fabrics: introduce Virtio-oF Protocol Data Unit
2023-07-29 23:25 ` Raphael Norwitz
@ 2023-07-31 2:28 ` zhenwei pi
0 siblings, 0 replies; 33+ messages in thread
From: zhenwei pi @ 2023-07-31 2:28 UTC (permalink / raw)
To: Raphael Norwitz
Cc: Stefan Hajnoczi, parav@nvidia.com, Michael S. Tsirkin,
jasowang@redhat.com, virtio-comment@lists.oasis-open.org,
houp@yusur.tech, zhouhuaping.san@bytedance.com,
xinhao.kong@duke.edu
On 7/30/23 07:25, Raphael Norwitz wrote:
> I’d suggest making some tweaks to save memory on the target side for RDMA.
>
>> On Jun 26, 2023, at 3:25 AM, zhenwei pi <pizhenwei@bytedance.com> wrote:
>>
>> Introduce Virtio-oF PDU for Virtio-oF queue, add Stream Data Transfers
>> and Keyed Data Transfers mechanism.
>>
>> Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
>> ---
>> transport-fabrics.tex | 68 +++++++++++++++++++++++++++++++++++++++++++
>> 1 file changed, 68 insertions(+)
>>
>> diff --git a/transport-fabrics.tex b/transport-fabrics.tex
>> index 54d7558..d562b2c 100644
>> --- a/transport-fabrics.tex
>> +++ b/transport-fabrics.tex
>> @@ -45,3 +45,71 @@ \subsection{Virtio-oF Qualified Name}\label{sec:Virtio Transport Options / Virti
>> \item The maximum name is 256 bytes in length, including the NUL character.
>> \item There is no strict style limitation.
>> \end{itemize}
>> +
>> +
>> +\subsection{Protocol Data Unit}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Protocol Data Unit}
>> +This section defines Virtio-oF Protocol Data Unit (PDU) for both the Virtio-oF control queue and Virtio-oF virtqueue.
>> +
>> +Virtio-oF PDU is a unit of information exchanged between a Virtio-oF initiator and a Virtio-oF target:
>> +\begin{itemize}
>> +\item A request Virtio-oF PDU from Virtio-oF initiator to Virtio-oF target contains command and associated data, if present.
>> +\item A response Virtio-oF PDU from Virtio-oF target to Virtio-oF initiator contains completion and associated data, if present.
>> +\end{itemize}
>> +
>> +\subsubsection{Stream Data Transfers}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Protocol Data Unit/ Stream Data Transfers}
>> +Stream-based (i.e. TCP/IP) Virtio-oF queue transfers Virtio-oF PDUs in a streaming fashion.
>> +
>> +The layout in a stream:
>> +\begin{lstlisting}
>> +PDUx and PDUz contain a command only, PDUy contains a command and data:
>> +
>> + +----+ +---------+ +----+
>> + |PDUx| | PDUy | |PDUz|
>> + ... +----+ ... +----+----+ ... +----+ ...
>> + |CMD | |CMD |Data| |CMD |
>> + +----+ +---------+ +----+
>> +
>> +PDUl contains completion only, PDUm and PDUn contain completion and data:
>> +
>> + +----+ +---------+ +---------+
>> + |PDUl| | PDUm | | PDUn |
>> + ... +----+ ... +----+----+ ... +----+----+ ...
>> + |COMP| |COMP|Data| |COMP|Data|
>> + +----+ +---------+ +---------+
>> +\end{lstlisting}
>> +
>> +\subsubsection{Keyed Data Transfers}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Protocol Data Unit/ Keyed Data Transfers}
>> +Message-based (i.e. RDMA) Virtio-oF queue transfers Virtio-oF PDUs in a message fashion, and uses the following structure to describe the remote data:
>> +
>> +\begin{lstlisting}
>> +struct virtio_of_keyed_desc {
>> + /* the remote address of data */
>> + le64 addr;
>> + /* the length of data */
>> + le32 length;
>> + /* the key to access the remote data */
>> + le32 key;
>> +};
>> +\end{lstlisting}
>> +
>> +A Virtio-oF control queue supports 1 keyed descriptor, and a Virtio-oF virtqueue supports 1 or more keyed descriptors.
>> +
>> +The PDUs of messages:
>> +\begin{lstlisting}
>> +PDUx contains a command only, PDUy contains a command and 1 descriptor,
>> +and PDUz contains a command and k - j + 1 descriptors.
>> +
>> + +----+ +---------+ +--------------------+
>> + |PDUx| | PDUy | | PDUz |
>> + ... +----+ ... +----+----+ ... +----+-----+---+-----+ ...
>> + |CMD | |CMD |DESC| |CMD |DESCj|...|DESCk|
>> + +----+ +---------+ +----------+---+-----+
>
> On the target side you have to prepost (MAX_SG_LEN * sizeof(virtio_of_keyed_desc) + sizeof(virtio_command_vq)) * max outstanding for each connection.
>
> Assuming a 128 max outstanding and MAX_SG_LEN=1024 (QEMU’s IOV_INTERNAL) that puts you at (1024 * 8 + 13) * 128 ~= 1MB per connection. If a target is servicing thousands of connections that adds up.
>
> I’d suggest adding a protocol message to cap the MAX_SG_LEN and communicate the value to the target.
>
> Would also be great to save space in virtio_of_keyed_desc, but I don’t think there is a way.
>
Hi,
I have to say that the virtio_of_keyed_desc is not 1:1 mapping to VQ desc.
MAX_SG_LEN is supposed at least 1(by default): then 1 memory region is
used for both WRITE and READ direction. Example of virtio-blk write
128k, 1 virtio_of_keyed_desc describes a MR of 16(OUT: virtio-blk
header) + 128K(OUT: blk data) + 1(IN: u8 status)
Or MAX_SG_LEN is 2, then we can separate OUT vq buffers and IN vq
buffers into 2 MR described by 2 virtio_of_keyed_desc.
'Get Keyed Number Descriptors Command' (from [PATCH 4/7]
transport-fabrics: introduce command set) is used to get the 'MAX_SG_LEN'.
>> +
>> +PDUl, PDUm, and PDUn contain completion only.
>> +
>> + +----+ +----+ +----+
>> + |PDUl| |PDUm| |PDUn|
>> + ... +----+ ... +----+ ... +----+ ...
>> + |COMP| |COMP| |COMP|
>> + +----+ +----+ +----+
>> +\end{lstlisting}
>> --
>> 2.25.1
>>
>>
>> This publicly archived list offers a means to provide input to the
>> OASIS Virtual I/O Device (VIRTIO) TC.
>>
>> In order to verify user consent to the Feedback License terms and
>> to minimize spam in the list archive, subscription is required
>> before posting.
>>
>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>> List help: virtio-comment-help@lists.oasis-open.org
>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>> Committee: https://www.oasis-open.org/committees/virtio/
>> Join OASIS: https://www.oasis-open.org/join/
>>
>>
>
--
zhenwei pi
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 33+ messages in thread
* [virtio-comment] [PATCH v4 4/7] transport-fabrics: introduce command set
2023-06-26 7:25 [virtio-comment] [PATCH v4 0/7] Introduce Virtio Over Fabrics zhenwei pi
` (2 preceding siblings ...)
2023-06-26 7:25 ` [virtio-comment] [PATCH v4 3/7] transport-fabrics: introduce Virtio-oF Protocol Data Unit zhenwei pi
@ 2023-06-26 7:25 ` zhenwei pi
2023-06-27 9:04 ` [virtio-comment] " Jason Wang
2023-07-29 23:28 ` [virtio-comment] " Raphael Norwitz
2023-06-26 7:25 ` [virtio-comment] [PATCH v4 5/7] transport-fabrics: introduce transport binding zhenwei pi
` (3 subsequent siblings)
7 siblings, 2 replies; 33+ messages in thread
From: zhenwei pi @ 2023-06-26 7:25 UTC (permalink / raw)
To: stefanha, parav, mst, jasowang
Cc: virtio-comment, houp, helei.sig11, zhouhuaping.san, xinhao.kong,
zhenwei pi
Introduce command set for Virtio-oF which includes:
- command id: a tag to distinguish in-flight commands
- status: indicate the result of a Virtio-oF command
- opcodes: the Virtio-oF transport layer commands use 0x0000-0x0fff,
and the device layer commands use 0x1000-0xffff.
get/set status/feature/ config use consecutive number.
Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
---
transport-fabrics.tex | 616 ++++++++++++++++++++++++++++++++++++++++++
1 file changed, 616 insertions(+)
diff --git a/transport-fabrics.tex b/transport-fabrics.tex
index d562b2c..3fd9e8e 100644
--- a/transport-fabrics.tex
+++ b/transport-fabrics.tex
@@ -113,3 +113,619 @@ \subsubsection{Keyed Data Transfers}\label{sec:Virtio Transport Options / Virtio
|COMP| |COMP| |COMP|
+----+ +----+ +----+
\end{lstlisting}
+
+
+\subsection{Commands Definition}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition}
+This section defines data structures, opcodes, and status for Virtio-oF.
+A Virtio-oF command is fixed to 16 bytes, and a Virtio-oF completion is fixed to 16 bytes.
+Note that the reserved fields of Virtio-oF commands and completions are filled with zero.
+
+\subsubsection{Command ID}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Command ID}
+The Command ID (le16 type) is used to uniquely describe a Virtio-oF PDU for identification.
+Generally, the Virtio-oF initiator allocates a Command ID that is unique for all in-flight commands,
+and the Virtio-oF target specifies the same Command ID for completion.
+
+Command IDs 0xff00 - 0xffff are reserved for the Virtio-oF control queue asynchronous events.
+The reserved Command IDs for the Virtio-oF control queue are as follows:
+
+\begin{tabular}{ |l|l| }
+\hline
+Command ID & Description \\
+\hline \hline
+0xfffe & Config change. Causes the initiator to generate a configuration change notification \\
+\hline
+0xffff & Keepalive. The initiator ignores this event \\
+\hline
+0xff00 - 0xfffd & Reserved \\
+\hline
+\end{tabular}
+
+\subsubsection{Status}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Status}
+The Status (le16 type) of a Virtio-oF completion is used to indicate the result of a Virtio-oF command in detail.
+The status values are defined as follows:
+
+\begin{lstlisting}
+/* command executed successfully */
+#define VIRTIO_OF_STATUS_SUCCESS 0x0000
+
+/* unrecognized command, or disabled command due to unsupported feature */
+#define VIRTIO_OF_STATUS_ENOCMD 0x0001
+/* in-flight commands exceeded queue size */
+#define VIRTIO_OF_STATUS_ECMDQUOT 0x0002
+
+/* no such target specified by TVQN */
+#define VIRTIO_OF_STATUS_ENOTGT 0x1001
+/* failed to create Virtio-oF device instance */
+#define VIRTIO_OF_STATUS_ENODEV 0x1002
+/* rejected due to access control */
+#define VIRTIO_OF_STATUS_EACLREJECTED 0x1003
+/* bad Virtio-oF device instance ID */
+#define VIRTIO_OF_STATUS_EBADDEV 0x1010
+/* mismatched TVQN or IVQN from Virtio-oF device instance */
+#define VIRTIO_OF_STATUS_EBADVQN 0x1011
+/* Virtio-oF virtqueue index exceeded */
+#define VIRTIO_OF_STATUS_EQUEUEQUOT 0x1020
+/* Virtio-oF virtqueue size exceeded */
+#define VIRTIO_OF_STATUS_EQSIZEQUOT 0x1021
+
+/* unsupported Virtio-oF feature */
+#define VIRTIO_OF_STATUS_EFEATURE 0x2000
+/* unsupported Virtio-oF device instance status */
+#define VIRTIO_OF_STATUS_ESTATUS 0x2010
+/* unsupported Virtio-oF device instance feature */
+#define VIRTIO_OF_STATUS_EDEVFEATURE 0x2020
+/* unsupported offset of device configuration space */
+#define VIRTIO_OF_STATUS_ECONFOFF 0x2030
+/* unsupported bytes of device configuration space */
+#define VIRTIO_OF_STATUS_ECONFBYTES 0x2031
+/* failed to read device-readable virtqueue buffers */
+#define VIRTIO_OF_STATUS_EOUTVQBUF 0x20f0
+/* failed to write device-writable virtqueue buffers */
+#define VIRTIO_OF_STATUS_EINVQBUF 0x20f1
+\end{lstlisting}
+
+\subsubsection{Opcodes}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes}
+Opcodes (u16 type) of Virtio-oF Commands are as follows:
+
+\begin{lstlisting}
+#define virtio_of_op_connect 0x0000
+#define virtio_of_op_disconnect 0x0001
+#define virtio_of_op_keepalive 0x0002
+#define virtio_of_op_get_feature 0x0004
+#define virtio_of_op_set_feature 0x0005
+#define virtio_of_op_get_keyed_num_descs 0x0100
+#define virtio_of_op_vq 0x0fff
+#define virtio_of_op_get_vendor_id 0x1000
+#define virtio_of_op_get_device_id 0x1001
+#define virtio_of_op_reset_device 0x1003
+#define virtio_of_op_get_status 0x1004
+#define virtio_of_op_set_status 0x1005
+#define virtio_of_op_get_device_feature 0x1006
+#define virtio_of_op_set_driver_feature 0x1009
+#define virtio_of_op_get_vq_size 0x100a
+#define virtio_of_op_get_config 0x100c
+#define virtio_of_op_set_config 0x100d
+\end{lstlisting}
+
+\paragraph{Connect Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Connect Command}
+
+The Connect Command is used to establish a Virtio-oF queue for both the Virtio-oF control queue and Virtio-oF virtqueue, it is always the first command to execute on a Virtio-oF queue.
+
+The structure of the Connect Command structure is as follows:
+\begin{lstlisting}
+struct virtio_of_command_connect {
+ /* opcode is virtio_of_op_connect */
+ le16 opcode;
+ /* unique ID for all in-flight commands */
+ le16 command_id;
+ /* Virtio-oF device instance ID. 0xffff is reserved for the Virtio-oF control queue */
+ le16 device_instance_id;
+ /* used by Virtio-oF virtqueue only, equal to virtqueue index */
+ le16 vq_index;
+ /* the length of the Connect Body */
+ le32 length;
+#define VIRTIO_OF_CONTROL_QUEUE_SIZE 32
+ /* the size of a Virtio-oF queue, 0 means the maximum queue size supported */
+ le16 queue_size;
+ u8 reserved[2];
+};
+\end{lstlisting}
+
+The structure of the Connect Body is as follows:
+\begin{lstlisting}
+struct virtio_of_connect {
+ /* Virtio-oF initiator VQN */
+ u8 ivqn[256];
+ /* Virtio-oF target VQN */
+ u8 tvqn[256];
+ u8 reserved[512];
+};
+\end{lstlisting}
+
+The structure of the Connect Completion is as follows:
+\begin{lstlisting}
+struct virtio_of_completion_connect {
+ le16 status;
+ le16 command_id;
+ /* Virtio-oF device instance ID */
+ le16 device_instance_id;
+ u8 reserved[10];
+};
+\end{lstlisting}
+
+The Connect Command, Body, and Completion have the following usages:
+\begin{itemize}
+\item The Virtio-oF initiator specifies \texttt{device_instance_id} of 0xffff in the Connect command
+and Virtio-oF initiator/target VQN in the Connect Body to establish the Virtio-oF control queue first.
+\item The Virtio-oF target allocates any available ID for a newly created Virtio-oF device instance,
+and specifies the \texttt{device_instance_id} of the new Virtio-oF device instance ID in the Connect Completion.
+\item The Virtio-oF initiator specifies the Virtio-oF device instance ID and \texttt{vq_index} to establish Virtio-oF virtqueues one by one.
+The Connect Body is optional for the Virtio-oF virtqueues, once a Virtio-oF virtqueue issues a Connect Command without Connect Body, the Virtio-oF virtqueue uses the same Connect Body as the Virtio-oF control queue.
+\end{itemize}
+
+\paragraph{Disconnect Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Disconnect Command}
+
+The Disconnect Command is used to gracefully disconnect both the Virtio-oF control queue and Virtio-oF virtqueue. All in-flight commands are completed upon receipt of the Disconnect Completion.
+
+The structure of the Disconnect Command structure is as follows:
+\begin{lstlisting}
+struct virtio_of_command_disconnect {
+ /* opcode is virtio_of_op_disconnect */
+ le16 opcode;
+ /* unique ID for all in-flight commands */
+ le16 command_id;
+ u8 reserved[12];
+};
+\end{lstlisting}
+
+The structure of the Disconnect Completion is as follows:
+\begin{lstlisting}
+struct virtio_of_completion_disconnect {
+ le16 status;
+ le16 command_id;
+ u8 reserved[12];
+};
+\end{lstlisting}
+
+The Disconnect Command and Completion have the following usages:
+\begin{itemize}
+\item The Virtio-oF initiator should disconnect the Virtio-oF virtqueues before the Virtio-oF control queue.
+\item Once the Virtio-oF control queue is disconnected, the Virtio-oF device instance and associated resources should be destroyed.
+\item If the underlying transport connection of a Virtio-oF queue is disconnected without issuing the Disconnect Command, the in-flight commands are not guaranteed to execute successfully.
+\end{itemize}
+
+\paragraph{Keepalive Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Keepalive Command}
+The Keepalive Command is used as a health check mechanism to detect when a Virtio-oF device instance becomes unavailable, it is used for the Virtio-oF control queue only.
+Once the Virtio-oF device instance becomes unavailable, the initiator should disconnect all the Virtio-oF queues and release associated resources.
+
+The structure of the Keepalive Command structure is as follows:
+\begin{lstlisting}
+struct virtio_of_command_keepalive {
+ /* opcode is virtio_of_op_keepalive */
+ le16 opcode;
+ /* unique ID for all in-flight commands */
+ le16 command_id;
+ u8 reserved[12];
+};
+\end{lstlisting}
+
+The structure of the Keepalive Completion is as follows:
+\begin{lstlisting}
+struct virtio_of_completion_keepalive {
+ le16 status;
+ le16 command_id;
+ u8 reserved[12];
+};
+\end{lstlisting}
+
+\paragraph{Get Feature Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Get Feature Command}
+The Get Feature Command is used to get feature bits of a Virtio-oF device instance for the Virtio-oF control queue only.
+
+The structure of the Get Feature Command structure is as follows:
+\begin{lstlisting}
+struct virtio_of_command_get_feature {
+ /* opcode is virtio_of_op_get_feature */
+ le16 opcode;
+ /* unique ID for all in-flight commands */
+ le16 command_id;
+ /* value 0x0 selects feature bits 0 to 63, 0x1 selects feature bits 64 to 127, etc */
+ le32 feature_select;
+ u8 reserved[8];
+};
+\end{lstlisting}
+
+The structure of the Get Feature Completion is as follows:
+\begin{lstlisting}
+struct virtio_of_completion_get_feature {
+ le16 status;
+ le16 command_id;
+ u8 reserved[4];
+ /* feature bits */
+ le64 feature;
+};
+\end{lstlisting}
+
+The feature bits of a Virtio-oF device instance are as follows:
+\begin{lstlisting}
+/* support virtio_of_op_get_keyed_num_descs to get the maximum number of keyed descriptors */
+#define VIRTIO_OF_F_KEYED_NUM_DESCS 0
+\end{lstlisting}
+
+\paragraph{Set Feature Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Set Feature Command}
+The Set Feature Command is used to set the feature bits of a Virtio-oF device instance for the Virtio-oF control queue only.
+
+The structure of the Set Feature Command structure is as follows:
+\begin{lstlisting}
+struct virtio_of_command_set_feature {
+ /* opcode is virtio_of_op_set_feature */
+ le16 opcode;
+ /* unique ID for all in-flight commands */
+ le16 command_id;
+ /* value 0x0 selects feature bits 0 to 63, 0x1 selects feature bits 64 to 127, etc */
+ le32 feature_select;
+ /* feature bits */
+ le64 feature;
+};
+\end{lstlisting}
+
+The structure of the Set Feature Completion is as follows:
+\begin{lstlisting}
+struct virtio_of_completion_set_feature {
+ le16 status;
+ le16 command_id;
+ u8 reserved[12];
+};
+\end{lstlisting}
+
+Note: it is recommended that the
+\nameref{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Get Feature Command}
+and Set Feature Command should be executed upon the establishment of the Virtio-oF control queue for feature negotiation.
+However, they are allowed to be executed during the Virtio-oF control queue lifecycle.
+
+\paragraph{Get Keyed Number Descriptors Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Get Keyed Number Descriptors Command}
+The Get Keyed Number Descriptors Command is used to get the maximum number of keyed descriptors of Virtio-oF virtqueues, it is executed on the Virtio-oF control queue.
+
+The structure of the Get Keyed Number Descriptors Command structure is as follows:
+\begin{lstlisting}
+struct virtio_of_command_get_keyed_num_descs {
+ /* opcode is virtio_of_op_get_keyed_num_descs */
+ le16 opcode;
+ /* unique ID for all in-flight commands */
+ le16 command_id;
+ u8 reserved[12];
+};
+\end{lstlisting}
+
+The structure of the Get Keyed Number Descriptors Completion is as follows:
+\begin{lstlisting}
+struct virtio_of_completion_get_keyed_num_descs {
+ le16 status;
+ le16 command_id;
+ /* the maximum number of keyed descriptors */
+ u8 descs;
+ u8 reserved[11];
+};
+\end{lstlisting}
+
+\paragraph{VQ Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / VQ Command}
+The VQ Command is used to transmit virtqueue buffers for Virtio-oF virtqueue only.
+
+The structure of the VQ Command structure is as follows:
+\begin{lstlisting}
+struct virtio_of_command_vq {
+ /* opcode is virtio_of_op_vq */
+ le16 opcode;
+ /* unique ID for all in-flight commands */
+ le16 command_id;
+ u8 reserved[4];
+ /* the length of device-readable virtqueue buffers */
+ le32 out_length;
+ /* the length of device-writable virtqueue buffers */
+ le32 in_length;
+};
+\end{lstlisting}
+
+The structure of the VQ Completion is as follows:
+\begin{lstlisting}
+struct virtio_of_completion_vq {
+ le16 status;
+ le16 command_id;
+ /* the length of virtqueue buffers from Virtio-oF target */
+ le32 length;
+ u8 reserved[8];
+};
+\end{lstlisting}
+
+For \nameref{sec:Virtio Transport Options / Virtio Over Fabrics / Protocol Data Unit/ Stream Data Transfers},
+\texttt{out_length} of virtio_of_command_vq describes the following bytes in a request Virtio-oF PDU,
+and \texttt{length} of virtio_of_completion_vq describes the following bytes in a response Virtio-oF PDU.
+
+For \nameref{sec:Virtio Transport Options / Virtio Over Fabrics / Protocol Data Unit/ Keyed Data Transfers},
+there are 1 or more keyed descriptors (virtio_of_keyed_desc type) in a request Virtio-oF PDU to describe the virtqueue buffers of \texttt{out_length + in_length} bytes,
+and \texttt{length} of virtio_of_completion_vq describes the number of bytes written into the device writable portion of the buffer described by the keyed descriptors.
+
+\paragraph{Get Vendor ID Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Get Vendor ID Command}
+The Get Vendor ID Command is used to get the Virtio Vendor ID of a Virtio-oF device instance for the Virtio-oF control queue only.
+
+The structure of the Get Vendor ID Command structure is as follows:
+\begin{lstlisting}
+struct virtio_of_command_get_vendor_id {
+ /* opcode is virtio_of_op_get_vendor_id */
+ le16 opcode;
+ /* unique ID for all in-flight commands */
+ le16 command_id;
+ u8 reserved[12];
+};
+\end{lstlisting}
+
+The structure of the Get Vendor ID Completion is as follows:
+\begin{lstlisting}
+struct virtio_of_completion_get_vendor_id {
+ le16 status;
+ le16 command_id;
+ /* Virtio Vendor ID */
+ le32 vendor_id;
+ u8 reserved[8];
+};
+\end{lstlisting}
+
+\paragraph{Get Device ID Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Get Device ID Command}
+The Get Device ID Command is used to get the Virtio Device ID of a Virtio-oF device instance for the Virtio-oF control queue only.
+
+The structure of the Get Device ID Command structure is as follows:
+\begin{lstlisting}
+struct virtio_of_command_get_device_id {
+ /* opcode is virtio_of_op_get_device_id */
+ le16 opcode;
+ /* unique ID for all in-flight commands */
+ le16 command_id;
+ u8 reserved[12];
+};
+\end{lstlisting}
+
+The structure of the Get Device ID Completion is as follows:
+\begin{lstlisting}
+struct virtio_of_completion_get_device_id {
+ le16 status;
+ le16 command_id;
+ /* Virtio Device ID */
+ le32 device_id;
+ u8 reserved[8];
+};
+\end{lstlisting}
+
+\paragraph{Reset Device Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Reset Device Command}
+The Reset Device Command is used to reset a Virtio-oF device instance for the Virtio-oF control queue only.
+
+The structure of the Reset Device Command structure is as follows:
+\begin{lstlisting}
+struct virtio_of_command_reset_device {
+ /* opcode is virtio_of_op_get_device_id */
+ le16 opcode;
+ /* unique ID for all in-flight commands */
+ le16 command_id;
+ u8 reserved[12];
+};
+\end{lstlisting}
+
+The structure of the Reset Device Completion is as follows:
+\begin{lstlisting}
+struct virtio_of_completion_reset_device {
+ le16 status;
+ le16 command_id;
+ u8 reserved[12];
+};
+\end{lstlisting}
+
+\paragraph{Get Status Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Get Status Command}
+The Get Status Command is used to get the status of a Virtio-oF device instance for the Virtio-oF control queue only.
+
+The structure of the Get Status Command structure is as follows:
+\begin{lstlisting}
+struct virtio_of_command_get_status {
+ /* opcode is virtio_of_op_get_status */
+ le16 opcode;
+ /* unique ID for all in-flight commands */
+ le16 command_id;
+ u8 reserved[12];
+};
+\end{lstlisting}
+
+The structure of the Get Status Completion is as follows:
+\begin{lstlisting}
+struct virtio_of_completion_get_status {
+ le16 status;
+ le16 command_id;
+ /* status of the Virtio-oF device instance */
+ le32 dev_status;
+ u8 reserved[8];
+};
+\end{lstlisting}
+
+\paragraph{Set Status Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Set Status Command}
+The Set Status Command is used to set the status of a Virtio-oF device instance for the Virtio-oF control queue only.
+
+The structure of the Set Status Command structure is as follows:
+\begin{lstlisting}
+struct virtio_of_command_set_status {
+ /* opcode is virtio_of_op_set_status */
+ le16 opcode;
+ /* unique ID for all in-flight commands */
+ le16 command_id;
+ /* status of the Virtio-oF device instance */
+ le32 status;
+ u8 reserved[8];
+};
+\end{lstlisting}
+
+The structure of the Set Status Completion is as follows:
+\begin{lstlisting}
+struct virtio_of_completion_set_status {
+ le16 status;
+ le16 command_id;
+ u8 reserved[12];
+};
+\end{lstlisting}
+
+\paragraph{Get Device Feature Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Get Device Feature Command}
+The Get Device Feature Command is used to get virtio device feature bits of a Virtio-oF device instance for the Virtio-oF control queue only.
+
+The structure of the Get Device Feature Command structure is as follows:
+\begin{lstlisting}
+struct virtio_of_command_get_device_feature {
+ /* opcode is virtio_of_op_get_device_feature */
+ le16 opcode;
+ /* unique ID for all in-flight commands */
+ le16 command_id;
+ /* value 0x0 selects feature bits 0 to 63, 0x1 selects feature bits 64 to 127, etc */
+ le32 feature_select;
+ u8 reserved[8];
+};
+\end{lstlisting}
+
+The structure of the Get Device Feature Completion is as follows:
+\begin{lstlisting}
+struct virtio_of_completion_get_device_feature {
+ le16 status;
+ le16 command_id;
+ u8 reserved[4];
+ /* feature bits */
+ le64 feature;
+};
+\end{lstlisting}
+
+\paragraph{Set Driver Feature Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Set Driver Feature Command}
+The Set Driver Feature Command is used to set virtio driver feature bits of a Virtio-oF device instance for the Virtio-oF control queue only.
+
+The structure of the Set Driver Feature Command structure is as follows:
+\begin{lstlisting}
+struct virtio_of_command_set_driver_feature {
+ /* opcode is virtio_of_op_set_driver_feature */
+ le16 opcode;
+ /* unique ID for all in-flight commands */
+ le16 command_id;
+ /* value 0x0 selects feature bits 0 to 63, 0x1 selects feature bits 64 to 127, etc */
+ le32 feature_select;
+ /* feature bits */
+ le64 feature;
+};
+\end{lstlisting}
+
+The structure of the Set Driver Feature Completion is as follows:
+\begin{lstlisting}
+struct virtio_of_completion_set_driver_feature {
+ le16 status;
+ le16 command_id;
+ u8 reserved[12];
+};
+\end{lstlisting}
+
+\paragraph{Get VQ Size Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Get VQ Size Command}
+The Get VQ Size Command is used to get the maximum queue size of a Virtio-oF virtqueue for the Virtio-oF control queue only.
+
+The structure of the Get VQ Size Command structure is as follows:
+\begin{lstlisting}
+struct virtio_of_command_get_vq_size {
+ /* opcode is virtio_of_op_get_vq_size */
+ le16 opcode;
+ /* unique ID for all in-flight commands */
+ le16 command_id;
+ /* the Virtio-oF virtqueue index */
+ le16 vq_index;
+ u8 reserved[10];
+};
+\end{lstlisting}
+
+The structure of the Get VQ Size Completion is as follows:
+\begin{lstlisting}
+struct virtio_of_completion_get_vq_size {
+ le16 status;
+ le16 command_id;
+ /* size of Virtio-oF virtqueues */
+ le16 size;
+ u8 reserved[10];
+};
+\end{lstlisting}
+
+Note: the Virtio-oF control queue has a fixed queue size, the Virtio-oF initiator can specify a smaller Virtio-oF virtqueue size by
+\nameref{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Connect Command}.
+
+\paragraph{Get Config Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Get Config Command}
+The Get Config Command is used to get the device configuration space of a Virtio-oF device instance for the Virtio-oF control queue only.
+
+The structure of the Get Config Command structure is as follows:
+\begin{lstlisting}
+struct virtio_of_command_get_config {
+ /* opcode is virtio_of_op_get_config */
+ le16 opcode;
+ /* unique ID for all in-flight commands */
+ le16 command_id;
+ /* offset of device configuration space */
+ le16 offset;
+ /* bytes to get, available values: 1, 2, 4, 8 */
+ u8 bytes;
+ u8 reserved[9];
+};
+\end{lstlisting}
+
+The structure of the Get Config Completion is as follows:
+\begin{lstlisting}
+struct virtio_of_completion_get_config {
+ le16 status;
+ le16 command_id;
+ /* generation count for the device configuration space */
+ le32 generation;
+ /* v is u8, u16, u32, or u64, then: v = (typeof v)le64_to_cpu(config) */
+ le64 config;
+};
+\end{lstlisting}
+
+\paragraph{Set Config Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Set Config Command}
+The Set Config Command is used to set the device configuration space of a Virtio-oF device instance for the Virtio-oF control queue only.
+
+The structure of the Set Config Command structure is as follows:
+\begin{lstlisting}
+struct virtio_of_command_set_config {
+ /* opcode is virtio_of_op_set_config */
+ le16 opcode;
+ /* unique ID for all in-flight commands */
+ le16 command_id;
+ /* offset of device configuration space */
+ le16 offset;
+ /* bytes to get, available values: 1, 2, 4, 8 */
+ u8 bytes;
+ u8 reserved;
+ /* v is u8, u16, u32, or u64, then: config = cpu_to_le64((u64)v) */
+ le64 config;
+};
+\end{lstlisting}
+
+The structure of the Set Config Completion is as follows:
+\begin{lstlisting}
+struct virtio_of_completion_set_config {
+ le16 status;
+ le16 command_id;
+ u8 reserved[12];
+};
+\end{lstlisting}
+
+\paragraph{Config Change Completion}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Config Change Completion}
+A Config Change Completion is used as a configuration change notification of a Virtio-oF device instance for the Virtio-oF control queue only.
+Note: only a single outstanding Config Change Completion is allowed, the configuration change notification is suppressed until the Virtio-oF initiator issues a
+\nameref{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Get Config Command}.
+
+The structure of the Config Change Completion is as follows:
+\begin{lstlisting}
+struct virtio_of_completion_config_change {
+ le16 status;
+ /* command_id is 0xfffe */
+ le16 command_id;
+ /* generation count for the device configuration space */
+ le32 generation;
+ u8 reserved[8];
+};
+\end{lstlisting}
+
+\paragraph{Keepalive Completion}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Keepalive Completion}
+A Keepalive Completion is used as a health check mechanism to detect when a Virtio-oF initiator becomes unavailable, it is used by a Virtio-oF device instance for the Virtio-oF control queue only.
+Once the Virtio-oF initiator becomes unavailable, the Virtio-oF target should destroy the Virtio-oF device instance and associated resources.
--
2.25.1
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply related [flat|nested] 33+ messages in thread* [virtio-comment] Re: [PATCH v4 4/7] transport-fabrics: introduce command set
2023-06-26 7:25 ` [virtio-comment] [PATCH v4 4/7] transport-fabrics: introduce command set zhenwei pi
@ 2023-06-27 9:04 ` Jason Wang
2023-06-28 1:34 ` [virtio-comment] " zhenwei pi
2023-07-29 23:28 ` [virtio-comment] " Raphael Norwitz
1 sibling, 1 reply; 33+ messages in thread
From: Jason Wang @ 2023-06-27 9:04 UTC (permalink / raw)
To: zhenwei pi
Cc: stefanha, parav, mst, virtio-comment, houp, helei.sig11,
zhouhuaping.san, xinhao.kong, Zhu Lingshan
On Mon, Jun 26, 2023 at 3:28 PM zhenwei pi <pizhenwei@bytedance.com> wrote:
>
> Introduce command set for Virtio-oF which includes:
> - command id: a tag to distinguish in-flight commands
> - status: indicate the result of a Virtio-oF command
> - opcodes: the Virtio-oF transport layer commands use 0x0000-0x0fff,
> and the device layer commands use 0x1000-0xffff.
> get/set status/feature/ config use consecutive number.
>
> Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
> ---
> transport-fabrics.tex | 616 ++++++++++++++++++++++++++++++++++++++++++
> 1 file changed, 616 insertions(+)
>
> diff --git a/transport-fabrics.tex b/transport-fabrics.tex
> index d562b2c..3fd9e8e 100644
> --- a/transport-fabrics.tex
> +++ b/transport-fabrics.tex
> @@ -113,3 +113,619 @@ \subsubsection{Keyed Data Transfers}\label{sec:Virtio Transport Options / Virtio
> |COMP| |COMP| |COMP|
> +----+ +----+ +----+
> \end{lstlisting}
> +
> +
> +\subsection{Commands Definition}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition}
> +This section defines data structures, opcodes, and status for Virtio-oF.
> +A Virtio-oF command is fixed to 16 bytes, and a Virtio-oF completion is fixed to 16 bytes.
> +Note that the reserved fields of Virtio-oF commands and completions are filled with zero.
> +
> +\subsubsection{Command ID}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Command ID}
> +The Command ID (le16 type) is used to uniquely describe a Virtio-oF PDU for identification.
> +Generally, the Virtio-oF initiator allocates a Command ID that is unique for all in-flight commands,
> +and the Virtio-oF target specifies the same Command ID for completion.
> +
> +Command IDs 0xff00 - 0xffff are reserved for the Virtio-oF control queue asynchronous events.
> +The reserved Command IDs for the Virtio-oF control queue are as follows:
> +
> +\begin{tabular}{ |l|l| }
> +\hline
> +Command ID & Description \\
> +\hline \hline
> +0xfffe & Config change. Causes the initiator to generate a configuration change notification \\
> +\hline
> +0xffff & Keepalive. The initiator ignores this event \\
> +\hline
> +0xff00 - 0xfffd & Reserved \\
> +\hline
> +\end{tabular}
> +
> +\subsubsection{Status}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Status}
> +The Status (le16 type) of a Virtio-oF completion is used to indicate the result of a Virtio-oF command in detail.
> +The status values are defined as follows:
> +
> +\begin{lstlisting}
> +/* command executed successfully */
> +#define VIRTIO_OF_STATUS_SUCCESS 0x0000
> +
> +/* unrecognized command, or disabled command due to unsupported feature */
> +#define VIRTIO_OF_STATUS_ENOCMD 0x0001
> +/* in-flight commands exceeded queue size */
> +#define VIRTIO_OF_STATUS_ECMDQUOT 0x0002
> +
> +/* no such target specified by TVQN */
> +#define VIRTIO_OF_STATUS_ENOTGT 0x1001
> +/* failed to create Virtio-oF device instance */
> +#define VIRTIO_OF_STATUS_ENODEV 0x1002
> +/* rejected due to access control */
> +#define VIRTIO_OF_STATUS_EACLREJECTED 0x1003
> +/* bad Virtio-oF device instance ID */
> +#define VIRTIO_OF_STATUS_EBADDEV 0x1010
> +/* mismatched TVQN or IVQN from Virtio-oF device instance */
> +#define VIRTIO_OF_STATUS_EBADVQN 0x1011
> +/* Virtio-oF virtqueue index exceeded */
> +#define VIRTIO_OF_STATUS_EQUEUEQUOT 0x1020
> +/* Virtio-oF virtqueue size exceeded */
> +#define VIRTIO_OF_STATUS_EQSIZEQUOT 0x1021
> +
> +/* unsupported Virtio-oF feature */
> +#define VIRTIO_OF_STATUS_EFEATURE 0x2000
> +/* unsupported Virtio-oF device instance status */
> +#define VIRTIO_OF_STATUS_ESTATUS 0x2010
> +/* unsupported Virtio-oF device instance feature */
> +#define VIRTIO_OF_STATUS_EDEVFEATURE 0x2020
> +/* unsupported offset of device configuration space */
> +#define VIRTIO_OF_STATUS_ECONFOFF 0x2030
> +/* unsupported bytes of device configuration space */
> +#define VIRTIO_OF_STATUS_ECONFBYTES 0x2031
> +/* failed to read device-readable virtqueue buffers */
> +#define VIRTIO_OF_STATUS_EOUTVQBUF 0x20f0
> +/* failed to write device-writable virtqueue buffers */
> +#define VIRTIO_OF_STATUS_EINVQBUF 0x20f1
> +\end{lstlisting}
> +
> +\subsubsection{Opcodes}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes}
> +Opcodes (u16 type) of Virtio-oF Commands are as follows:
> +
> +\begin{lstlisting}
> +#define virtio_of_op_connect 0x0000
> +#define virtio_of_op_disconnect 0x0001
> +#define virtio_of_op_keepalive 0x0002
> +#define virtio_of_op_get_feature 0x0004
> +#define virtio_of_op_set_feature 0x0005
> +#define virtio_of_op_get_keyed_num_descs 0x0100
> +#define virtio_of_op_vq 0x0fff
> +#define virtio_of_op_get_vendor_id 0x1000
> +#define virtio_of_op_get_device_id 0x1001
> +#define virtio_of_op_reset_device 0x1003
> +#define virtio_of_op_get_status 0x1004
> +#define virtio_of_op_set_status 0x1005
> +#define virtio_of_op_get_device_feature 0x1006
> +#define virtio_of_op_set_driver_feature 0x1009
> +#define virtio_of_op_get_vq_size 0x100a
> +#define virtio_of_op_get_config 0x100c
> +#define virtio_of_op_set_config 0x100d
We had a proposal of transport virtqueue that allows using a
virtqueue[1] for configuring a device. And Ling shan will rebase it on
top of the admin commands.
That is to say, admin commands could be used for transport which will
be duplicated with the commands you invented here.
I wonder if the virtio-OF can just reuse the admin commands (have a
quick glance and I didn't find any blocker for this).
Thanks
[1] https://lists.oasis-open.org/archives/virtio-comment/202208/msg00003.html
> +\end{lstlisting}
> +
> +\paragraph{Connect Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Connect Command}
> +
> +The Connect Command is used to establish a Virtio-oF queue for both the Virtio-oF control queue and Virtio-oF virtqueue, it is always the first command to execute on a Virtio-oF queue.
> +
> +The structure of the Connect Command structure is as follows:
> +\begin{lstlisting}
> +struct virtio_of_command_connect {
> + /* opcode is virtio_of_op_connect */
> + le16 opcode;
> + /* unique ID for all in-flight commands */
> + le16 command_id;
> + /* Virtio-oF device instance ID. 0xffff is reserved for the Virtio-oF control queue */
> + le16 device_instance_id;
> + /* used by Virtio-oF virtqueue only, equal to virtqueue index */
> + le16 vq_index;
> + /* the length of the Connect Body */
> + le32 length;
> +#define VIRTIO_OF_CONTROL_QUEUE_SIZE 32
> + /* the size of a Virtio-oF queue, 0 means the maximum queue size supported */
> + le16 queue_size;
> + u8 reserved[2];
> +};
> +\end{lstlisting}
> +
> +The structure of the Connect Body is as follows:
> +\begin{lstlisting}
> +struct virtio_of_connect {
> + /* Virtio-oF initiator VQN */
> + u8 ivqn[256];
> + /* Virtio-oF target VQN */
> + u8 tvqn[256];
> + u8 reserved[512];
> +};
> +\end{lstlisting}
> +
> +The structure of the Connect Completion is as follows:
> +\begin{lstlisting}
> +struct virtio_of_completion_connect {
> + le16 status;
> + le16 command_id;
> + /* Virtio-oF device instance ID */
> + le16 device_instance_id;
> + u8 reserved[10];
> +};
> +\end{lstlisting}
> +
> +The Connect Command, Body, and Completion have the following usages:
> +\begin{itemize}
> +\item The Virtio-oF initiator specifies \texttt{device_instance_id} of 0xffff in the Connect command
> +and Virtio-oF initiator/target VQN in the Connect Body to establish the Virtio-oF control queue first.
> +\item The Virtio-oF target allocates any available ID for a newly created Virtio-oF device instance,
> +and specifies the \texttt{device_instance_id} of the new Virtio-oF device instance ID in the Connect Completion.
> +\item The Virtio-oF initiator specifies the Virtio-oF device instance ID and \texttt{vq_index} to establish Virtio-oF virtqueues one by one.
> +The Connect Body is optional for the Virtio-oF virtqueues, once a Virtio-oF virtqueue issues a Connect Command without Connect Body, the Virtio-oF virtqueue uses the same Connect Body as the Virtio-oF control queue.
> +\end{itemize}
> +
> +\paragraph{Disconnect Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Disconnect Command}
> +
> +The Disconnect Command is used to gracefully disconnect both the Virtio-oF control queue and Virtio-oF virtqueue. All in-flight commands are completed upon receipt of the Disconnect Completion.
> +
> +The structure of the Disconnect Command structure is as follows:
> +\begin{lstlisting}
> +struct virtio_of_command_disconnect {
> + /* opcode is virtio_of_op_disconnect */
> + le16 opcode;
> + /* unique ID for all in-flight commands */
> + le16 command_id;
> + u8 reserved[12];
> +};
> +\end{lstlisting}
> +
> +The structure of the Disconnect Completion is as follows:
> +\begin{lstlisting}
> +struct virtio_of_completion_disconnect {
> + le16 status;
> + le16 command_id;
> + u8 reserved[12];
> +};
> +\end{lstlisting}
> +
> +The Disconnect Command and Completion have the following usages:
> +\begin{itemize}
> +\item The Virtio-oF initiator should disconnect the Virtio-oF virtqueues before the Virtio-oF control queue.
> +\item Once the Virtio-oF control queue is disconnected, the Virtio-oF device instance and associated resources should be destroyed.
> +\item If the underlying transport connection of a Virtio-oF queue is disconnected without issuing the Disconnect Command, the in-flight commands are not guaranteed to execute successfully.
> +\end{itemize}
> +
> +\paragraph{Keepalive Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Keepalive Command}
> +The Keepalive Command is used as a health check mechanism to detect when a Virtio-oF device instance becomes unavailable, it is used for the Virtio-oF control queue only.
> +Once the Virtio-oF device instance becomes unavailable, the initiator should disconnect all the Virtio-oF queues and release associated resources.
> +
> +The structure of the Keepalive Command structure is as follows:
> +\begin{lstlisting}
> +struct virtio_of_command_keepalive {
> + /* opcode is virtio_of_op_keepalive */
> + le16 opcode;
> + /* unique ID for all in-flight commands */
> + le16 command_id;
> + u8 reserved[12];
> +};
> +\end{lstlisting}
> +
> +The structure of the Keepalive Completion is as follows:
> +\begin{lstlisting}
> +struct virtio_of_completion_keepalive {
> + le16 status;
> + le16 command_id;
> + u8 reserved[12];
> +};
> +\end{lstlisting}
> +
> +\paragraph{Get Feature Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Get Feature Command}
> +The Get Feature Command is used to get feature bits of a Virtio-oF device instance for the Virtio-oF control queue only.
> +
> +The structure of the Get Feature Command structure is as follows:
> +\begin{lstlisting}
> +struct virtio_of_command_get_feature {
> + /* opcode is virtio_of_op_get_feature */
> + le16 opcode;
> + /* unique ID for all in-flight commands */
> + le16 command_id;
> + /* value 0x0 selects feature bits 0 to 63, 0x1 selects feature bits 64 to 127, etc */
> + le32 feature_select;
> + u8 reserved[8];
> +};
> +\end{lstlisting}
> +
> +The structure of the Get Feature Completion is as follows:
> +\begin{lstlisting}
> +struct virtio_of_completion_get_feature {
> + le16 status;
> + le16 command_id;
> + u8 reserved[4];
> + /* feature bits */
> + le64 feature;
> +};
> +\end{lstlisting}
> +
> +The feature bits of a Virtio-oF device instance are as follows:
> +\begin{lstlisting}
> +/* support virtio_of_op_get_keyed_num_descs to get the maximum number of keyed descriptors */
> +#define VIRTIO_OF_F_KEYED_NUM_DESCS 0
> +\end{lstlisting}
> +
> +\paragraph{Set Feature Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Set Feature Command}
> +The Set Feature Command is used to set the feature bits of a Virtio-oF device instance for the Virtio-oF control queue only.
> +
> +The structure of the Set Feature Command structure is as follows:
> +\begin{lstlisting}
> +struct virtio_of_command_set_feature {
> + /* opcode is virtio_of_op_set_feature */
> + le16 opcode;
> + /* unique ID for all in-flight commands */
> + le16 command_id;
> + /* value 0x0 selects feature bits 0 to 63, 0x1 selects feature bits 64 to 127, etc */
> + le32 feature_select;
> + /* feature bits */
> + le64 feature;
> +};
> +\end{lstlisting}
> +
> +The structure of the Set Feature Completion is as follows:
> +\begin{lstlisting}
> +struct virtio_of_completion_set_feature {
> + le16 status;
> + le16 command_id;
> + u8 reserved[12];
> +};
> +\end{lstlisting}
> +
> +Note: it is recommended that the
> +\nameref{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Get Feature Command}
> +and Set Feature Command should be executed upon the establishment of the Virtio-oF control queue for feature negotiation.
> +However, they are allowed to be executed during the Virtio-oF control queue lifecycle.
> +
> +\paragraph{Get Keyed Number Descriptors Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Get Keyed Number Descriptors Command}
> +The Get Keyed Number Descriptors Command is used to get the maximum number of keyed descriptors of Virtio-oF virtqueues, it is executed on the Virtio-oF control queue.
> +
> +The structure of the Get Keyed Number Descriptors Command structure is as follows:
> +\begin{lstlisting}
> +struct virtio_of_command_get_keyed_num_descs {
> + /* opcode is virtio_of_op_get_keyed_num_descs */
> + le16 opcode;
> + /* unique ID for all in-flight commands */
> + le16 command_id;
> + u8 reserved[12];
> +};
> +\end{lstlisting}
> +
> +The structure of the Get Keyed Number Descriptors Completion is as follows:
> +\begin{lstlisting}
> +struct virtio_of_completion_get_keyed_num_descs {
> + le16 status;
> + le16 command_id;
> + /* the maximum number of keyed descriptors */
> + u8 descs;
> + u8 reserved[11];
> +};
> +\end{lstlisting}
> +
> +\paragraph{VQ Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / VQ Command}
> +The VQ Command is used to transmit virtqueue buffers for Virtio-oF virtqueue only.
> +
> +The structure of the VQ Command structure is as follows:
> +\begin{lstlisting}
> +struct virtio_of_command_vq {
> + /* opcode is virtio_of_op_vq */
> + le16 opcode;
> + /* unique ID for all in-flight commands */
> + le16 command_id;
> + u8 reserved[4];
> + /* the length of device-readable virtqueue buffers */
> + le32 out_length;
> + /* the length of device-writable virtqueue buffers */
> + le32 in_length;
> +};
> +\end{lstlisting}
> +
> +The structure of the VQ Completion is as follows:
> +\begin{lstlisting}
> +struct virtio_of_completion_vq {
> + le16 status;
> + le16 command_id;
> + /* the length of virtqueue buffers from Virtio-oF target */
> + le32 length;
> + u8 reserved[8];
> +};
> +\end{lstlisting}
> +
> +For \nameref{sec:Virtio Transport Options / Virtio Over Fabrics / Protocol Data Unit/ Stream Data Transfers},
> +\texttt{out_length} of virtio_of_command_vq describes the following bytes in a request Virtio-oF PDU,
> +and \texttt{length} of virtio_of_completion_vq describes the following bytes in a response Virtio-oF PDU.
> +
> +For \nameref{sec:Virtio Transport Options / Virtio Over Fabrics / Protocol Data Unit/ Keyed Data Transfers},
> +there are 1 or more keyed descriptors (virtio_of_keyed_desc type) in a request Virtio-oF PDU to describe the virtqueue buffers of \texttt{out_length + in_length} bytes,
> +and \texttt{length} of virtio_of_completion_vq describes the number of bytes written into the device writable portion of the buffer described by the keyed descriptors.
> +
> +\paragraph{Get Vendor ID Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Get Vendor ID Command}
> +The Get Vendor ID Command is used to get the Virtio Vendor ID of a Virtio-oF device instance for the Virtio-oF control queue only.
> +
> +The structure of the Get Vendor ID Command structure is as follows:
> +\begin{lstlisting}
> +struct virtio_of_command_get_vendor_id {
> + /* opcode is virtio_of_op_get_vendor_id */
> + le16 opcode;
> + /* unique ID for all in-flight commands */
> + le16 command_id;
> + u8 reserved[12];
> +};
> +\end{lstlisting}
> +
> +The structure of the Get Vendor ID Completion is as follows:
> +\begin{lstlisting}
> +struct virtio_of_completion_get_vendor_id {
> + le16 status;
> + le16 command_id;
> + /* Virtio Vendor ID */
> + le32 vendor_id;
> + u8 reserved[8];
> +};
> +\end{lstlisting}
> +
> +\paragraph{Get Device ID Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Get Device ID Command}
> +The Get Device ID Command is used to get the Virtio Device ID of a Virtio-oF device instance for the Virtio-oF control queue only.
> +
> +The structure of the Get Device ID Command structure is as follows:
> +\begin{lstlisting}
> +struct virtio_of_command_get_device_id {
> + /* opcode is virtio_of_op_get_device_id */
> + le16 opcode;
> + /* unique ID for all in-flight commands */
> + le16 command_id;
> + u8 reserved[12];
> +};
> +\end{lstlisting}
> +
> +The structure of the Get Device ID Completion is as follows:
> +\begin{lstlisting}
> +struct virtio_of_completion_get_device_id {
> + le16 status;
> + le16 command_id;
> + /* Virtio Device ID */
> + le32 device_id;
> + u8 reserved[8];
> +};
> +\end{lstlisting}
> +
> +\paragraph{Reset Device Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Reset Device Command}
> +The Reset Device Command is used to reset a Virtio-oF device instance for the Virtio-oF control queue only.
> +
> +The structure of the Reset Device Command structure is as follows:
> +\begin{lstlisting}
> +struct virtio_of_command_reset_device {
> + /* opcode is virtio_of_op_get_device_id */
> + le16 opcode;
> + /* unique ID for all in-flight commands */
> + le16 command_id;
> + u8 reserved[12];
> +};
> +\end{lstlisting}
> +
> +The structure of the Reset Device Completion is as follows:
> +\begin{lstlisting}
> +struct virtio_of_completion_reset_device {
> + le16 status;
> + le16 command_id;
> + u8 reserved[12];
> +};
> +\end{lstlisting}
> +
> +\paragraph{Get Status Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Get Status Command}
> +The Get Status Command is used to get the status of a Virtio-oF device instance for the Virtio-oF control queue only.
> +
> +The structure of the Get Status Command structure is as follows:
> +\begin{lstlisting}
> +struct virtio_of_command_get_status {
> + /* opcode is virtio_of_op_get_status */
> + le16 opcode;
> + /* unique ID for all in-flight commands */
> + le16 command_id;
> + u8 reserved[12];
> +};
> +\end{lstlisting}
> +
> +The structure of the Get Status Completion is as follows:
> +\begin{lstlisting}
> +struct virtio_of_completion_get_status {
> + le16 status;
> + le16 command_id;
> + /* status of the Virtio-oF device instance */
> + le32 dev_status;
> + u8 reserved[8];
> +};
> +\end{lstlisting}
> +
> +\paragraph{Set Status Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Set Status Command}
> +The Set Status Command is used to set the status of a Virtio-oF device instance for the Virtio-oF control queue only.
> +
> +The structure of the Set Status Command structure is as follows:
> +\begin{lstlisting}
> +struct virtio_of_command_set_status {
> + /* opcode is virtio_of_op_set_status */
> + le16 opcode;
> + /* unique ID for all in-flight commands */
> + le16 command_id;
> + /* status of the Virtio-oF device instance */
> + le32 status;
> + u8 reserved[8];
> +};
> +\end{lstlisting}
> +
> +The structure of the Set Status Completion is as follows:
> +\begin{lstlisting}
> +struct virtio_of_completion_set_status {
> + le16 status;
> + le16 command_id;
> + u8 reserved[12];
> +};
> +\end{lstlisting}
> +
> +\paragraph{Get Device Feature Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Get Device Feature Command}
> +The Get Device Feature Command is used to get virtio device feature bits of a Virtio-oF device instance for the Virtio-oF control queue only.
> +
> +The structure of the Get Device Feature Command structure is as follows:
> +\begin{lstlisting}
> +struct virtio_of_command_get_device_feature {
> + /* opcode is virtio_of_op_get_device_feature */
> + le16 opcode;
> + /* unique ID for all in-flight commands */
> + le16 command_id;
> + /* value 0x0 selects feature bits 0 to 63, 0x1 selects feature bits 64 to 127, etc */
> + le32 feature_select;
> + u8 reserved[8];
> +};
> +\end{lstlisting}
> +
> +The structure of the Get Device Feature Completion is as follows:
> +\begin{lstlisting}
> +struct virtio_of_completion_get_device_feature {
> + le16 status;
> + le16 command_id;
> + u8 reserved[4];
> + /* feature bits */
> + le64 feature;
> +};
> +\end{lstlisting}
> +
> +\paragraph{Set Driver Feature Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Set Driver Feature Command}
> +The Set Driver Feature Command is used to set virtio driver feature bits of a Virtio-oF device instance for the Virtio-oF control queue only.
> +
> +The structure of the Set Driver Feature Command structure is as follows:
> +\begin{lstlisting}
> +struct virtio_of_command_set_driver_feature {
> + /* opcode is virtio_of_op_set_driver_feature */
> + le16 opcode;
> + /* unique ID for all in-flight commands */
> + le16 command_id;
> + /* value 0x0 selects feature bits 0 to 63, 0x1 selects feature bits 64 to 127, etc */
> + le32 feature_select;
> + /* feature bits */
> + le64 feature;
> +};
> +\end{lstlisting}
> +
> +The structure of the Set Driver Feature Completion is as follows:
> +\begin{lstlisting}
> +struct virtio_of_completion_set_driver_feature {
> + le16 status;
> + le16 command_id;
> + u8 reserved[12];
> +};
> +\end{lstlisting}
> +
> +\paragraph{Get VQ Size Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Get VQ Size Command}
> +The Get VQ Size Command is used to get the maximum queue size of a Virtio-oF virtqueue for the Virtio-oF control queue only.
> +
> +The structure of the Get VQ Size Command structure is as follows:
> +\begin{lstlisting}
> +struct virtio_of_command_get_vq_size {
> + /* opcode is virtio_of_op_get_vq_size */
> + le16 opcode;
> + /* unique ID for all in-flight commands */
> + le16 command_id;
> + /* the Virtio-oF virtqueue index */
> + le16 vq_index;
> + u8 reserved[10];
> +};
> +\end{lstlisting}
> +
> +The structure of the Get VQ Size Completion is as follows:
> +\begin{lstlisting}
> +struct virtio_of_completion_get_vq_size {
> + le16 status;
> + le16 command_id;
> + /* size of Virtio-oF virtqueues */
> + le16 size;
> + u8 reserved[10];
> +};
> +\end{lstlisting}
> +
> +Note: the Virtio-oF control queue has a fixed queue size, the Virtio-oF initiator can specify a smaller Virtio-oF virtqueue size by
> +\nameref{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Connect Command}.
> +
> +\paragraph{Get Config Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Get Config Command}
> +The Get Config Command is used to get the device configuration space of a Virtio-oF device instance for the Virtio-oF control queue only.
> +
> +The structure of the Get Config Command structure is as follows:
> +\begin{lstlisting}
> +struct virtio_of_command_get_config {
> + /* opcode is virtio_of_op_get_config */
> + le16 opcode;
> + /* unique ID for all in-flight commands */
> + le16 command_id;
> + /* offset of device configuration space */
> + le16 offset;
> + /* bytes to get, available values: 1, 2, 4, 8 */
> + u8 bytes;
> + u8 reserved[9];
> +};
> +\end{lstlisting}
> +
> +The structure of the Get Config Completion is as follows:
> +\begin{lstlisting}
> +struct virtio_of_completion_get_config {
> + le16 status;
> + le16 command_id;
> + /* generation count for the device configuration space */
> + le32 generation;
> + /* v is u8, u16, u32, or u64, then: v = (typeof v)le64_to_cpu(config) */
> + le64 config;
> +};
> +\end{lstlisting}
> +
> +\paragraph{Set Config Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Set Config Command}
> +The Set Config Command is used to set the device configuration space of a Virtio-oF device instance for the Virtio-oF control queue only.
> +
> +The structure of the Set Config Command structure is as follows:
> +\begin{lstlisting}
> +struct virtio_of_command_set_config {
> + /* opcode is virtio_of_op_set_config */
> + le16 opcode;
> + /* unique ID for all in-flight commands */
> + le16 command_id;
> + /* offset of device configuration space */
> + le16 offset;
> + /* bytes to get, available values: 1, 2, 4, 8 */
> + u8 bytes;
> + u8 reserved;
> + /* v is u8, u16, u32, or u64, then: config = cpu_to_le64((u64)v) */
> + le64 config;
> +};
> +\end{lstlisting}
> +
> +The structure of the Set Config Completion is as follows:
> +\begin{lstlisting}
> +struct virtio_of_completion_set_config {
> + le16 status;
> + le16 command_id;
> + u8 reserved[12];
> +};
> +\end{lstlisting}
> +
> +\paragraph{Config Change Completion}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Config Change Completion}
> +A Config Change Completion is used as a configuration change notification of a Virtio-oF device instance for the Virtio-oF control queue only.
> +Note: only a single outstanding Config Change Completion is allowed, the configuration change notification is suppressed until the Virtio-oF initiator issues a
> +\nameref{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Get Config Command}.
> +
> +The structure of the Config Change Completion is as follows:
> +\begin{lstlisting}
> +struct virtio_of_completion_config_change {
> + le16 status;
> + /* command_id is 0xfffe */
> + le16 command_id;
> + /* generation count for the device configuration space */
> + le32 generation;
> + u8 reserved[8];
> +};
> +\end{lstlisting}
> +
> +\paragraph{Keepalive Completion}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Keepalive Completion}
> +A Keepalive Completion is used as a health check mechanism to detect when a Virtio-oF initiator becomes unavailable, it is used by a Virtio-oF device instance for the Virtio-oF control queue only.
> +Once the Virtio-oF initiator becomes unavailable, the Virtio-oF target should destroy the Virtio-oF device instance and associated resources.
> --
> 2.25.1
>
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 33+ messages in thread* [virtio-comment] Re: Re: [PATCH v4 4/7] transport-fabrics: introduce command set
2023-06-27 9:04 ` [virtio-comment] " Jason Wang
@ 2023-06-28 1:34 ` zhenwei pi
2023-06-28 2:56 ` Jason Wang
0 siblings, 1 reply; 33+ messages in thread
From: zhenwei pi @ 2023-06-28 1:34 UTC (permalink / raw)
To: Jason Wang
Cc: stefanha, parav, mst, virtio-comment, houp, helei.sig11,
zhouhuaping.san, xinhao.kong, Zhu Lingshan
On 6/27/23 17:04, Jason Wang wrote:
> On Mon, Jun 26, 2023 at 3:28 PM zhenwei pi <pizhenwei@bytedance.com> wrote:
>>
>> Introduce command set for Virtio-oF which includes:
>> - command id: a tag to distinguish in-flight commands
>> - status: indicate the result of a Virtio-oF command
>> - opcodes: the Virtio-oF transport layer commands use 0x0000-0x0fff,
>> and the device layer commands use 0x1000-0xffff.
>> get/set status/feature/ config use consecutive number.
>>
>> Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
>> ---
>> transport-fabrics.tex | 616 ++++++++++++++++++++++++++++++++++++++++++
>> 1 file changed, 616 insertions(+)
>>
>> diff --git a/transport-fabrics.tex b/transport-fabrics.tex
>> index d562b2c..3fd9e8e 100644
>> --- a/transport-fabrics.tex
>> +++ b/transport-fabrics.tex
>> @@ -113,3 +113,619 @@ \subsubsection{Keyed Data Transfers}\label{sec:Virtio Transport Options / Virtio
>> |COMP| |COMP| |COMP|
>> +----+ +----+ +----+
>> \end{lstlisting}
>> +
>> +
>> +\subsection{Commands Definition}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition}
>> +This section defines data structures, opcodes, and status for Virtio-oF.
>> +A Virtio-oF command is fixed to 16 bytes, and a Virtio-oF completion is fixed to 16 bytes.
>> +Note that the reserved fields of Virtio-oF commands and completions are filled with zero.
>> +
>> +\subsubsection{Command ID}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Command ID}
>> +The Command ID (le16 type) is used to uniquely describe a Virtio-oF PDU for identification.
>> +Generally, the Virtio-oF initiator allocates a Command ID that is unique for all in-flight commands,
>> +and the Virtio-oF target specifies the same Command ID for completion.
>> +
>> +Command IDs 0xff00 - 0xffff are reserved for the Virtio-oF control queue asynchronous events.
>> +The reserved Command IDs for the Virtio-oF control queue are as follows:
>> +
>> +\begin{tabular}{ |l|l| }
>> +\hline
>> +Command ID & Description \\
>> +\hline \hline
>> +0xfffe & Config change. Causes the initiator to generate a configuration change notification \\
>> +\hline
>> +0xffff & Keepalive. The initiator ignores this event \\
>> +\hline
>> +0xff00 - 0xfffd & Reserved \\
>> +\hline
>> +\end{tabular}
>> +
>> +\subsubsection{Status}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Status}
>> +The Status (le16 type) of a Virtio-oF completion is used to indicate the result of a Virtio-oF command in detail.
>> +The status values are defined as follows:
>> +
>> +\begin{lstlisting}
>> +/* command executed successfully */
>> +#define VIRTIO_OF_STATUS_SUCCESS 0x0000
>> +
>> +/* unrecognized command, or disabled command due to unsupported feature */
>> +#define VIRTIO_OF_STATUS_ENOCMD 0x0001
>> +/* in-flight commands exceeded queue size */
>> +#define VIRTIO_OF_STATUS_ECMDQUOT 0x0002
>> +
>> +/* no such target specified by TVQN */
>> +#define VIRTIO_OF_STATUS_ENOTGT 0x1001
>> +/* failed to create Virtio-oF device instance */
>> +#define VIRTIO_OF_STATUS_ENODEV 0x1002
>> +/* rejected due to access control */
>> +#define VIRTIO_OF_STATUS_EACLREJECTED 0x1003
>> +/* bad Virtio-oF device instance ID */
>> +#define VIRTIO_OF_STATUS_EBADDEV 0x1010
>> +/* mismatched TVQN or IVQN from Virtio-oF device instance */
>> +#define VIRTIO_OF_STATUS_EBADVQN 0x1011
>> +/* Virtio-oF virtqueue index exceeded */
>> +#define VIRTIO_OF_STATUS_EQUEUEQUOT 0x1020
>> +/* Virtio-oF virtqueue size exceeded */
>> +#define VIRTIO_OF_STATUS_EQSIZEQUOT 0x1021
>> +
>> +/* unsupported Virtio-oF feature */
>> +#define VIRTIO_OF_STATUS_EFEATURE 0x2000
>> +/* unsupported Virtio-oF device instance status */
>> +#define VIRTIO_OF_STATUS_ESTATUS 0x2010
>> +/* unsupported Virtio-oF device instance feature */
>> +#define VIRTIO_OF_STATUS_EDEVFEATURE 0x2020
>> +/* unsupported offset of device configuration space */
>> +#define VIRTIO_OF_STATUS_ECONFOFF 0x2030
>> +/* unsupported bytes of device configuration space */
>> +#define VIRTIO_OF_STATUS_ECONFBYTES 0x2031
>> +/* failed to read device-readable virtqueue buffers */
>> +#define VIRTIO_OF_STATUS_EOUTVQBUF 0x20f0
>> +/* failed to write device-writable virtqueue buffers */
>> +#define VIRTIO_OF_STATUS_EINVQBUF 0x20f1
>> +\end{lstlisting}
>> +
>> +\subsubsection{Opcodes}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes}
>> +Opcodes (u16 type) of Virtio-oF Commands are as follows:
>> +
>> +\begin{lstlisting}
>> +#define virtio_of_op_connect 0x0000
>> +#define virtio_of_op_disconnect 0x0001
>> +#define virtio_of_op_keepalive 0x0002
>> +#define virtio_of_op_get_feature 0x0004
>> +#define virtio_of_op_set_feature 0x0005
>> +#define virtio_of_op_get_keyed_num_descs 0x0100
>> +#define virtio_of_op_vq 0x0fff
>> +#define virtio_of_op_get_vendor_id 0x1000
>> +#define virtio_of_op_get_device_id 0x1001
>> +#define virtio_of_op_reset_device 0x1003
>> +#define virtio_of_op_get_status 0x1004
>> +#define virtio_of_op_set_status 0x1005
>> +#define virtio_of_op_get_device_feature 0x1006
>> +#define virtio_of_op_set_driver_feature 0x1009
>> +#define virtio_of_op_get_vq_size 0x100a
>> +#define virtio_of_op_get_config 0x100c
>> +#define virtio_of_op_set_config 0x100d
>
> We had a proposal of transport virtqueue that allows using a
> virtqueue[1] for configuring a device. And Ling shan will rebase it on
> top of the admin commands.
>
> That is to say, admin commands could be used for transport which will
> be duplicated with the commands you invented here.
>
> I wonder if the virtio-OF can just reuse the admin commands (have a
> quick glance and I didn't find any blocker for this).
>
> Thanks
>
> [1] https://lists.oasis-open.org/archives/virtio-comment/202208/msg00003.html
>
From my understand, the virtio device related commands looks
similar(set/get status/features, reset device ...), but not same:
1, the admin commands are executed upon the admin virtqueue, it uses
virtqueue based transfer.
The virtio-of commands are executed upon a virtio-of queue(established
reliable connection), not a virtqueue. A virtio-of device could support
admin commands in theory.
I suppose a virtio-of device with admin vq supported like:
+---------------------+
| +-------------+ |
| |control queue| |
| +-------------+ |
+---------------------+
| +-------------+ |
| | virtqueue0 | |
| +-------------+ |
| | virtqueue1 | |
| +-------------+ |
| | virtqueue ..| |
| +-------------+ |
| | admin vq | |
| +-------------+ |
+---------------------+
The virtio-of commands ares used by both virtio-of control queue and
virtio-of virtqueues(include admin vq).
The admin commands are used by admin vq only.
2, the virtio-of commands are used over fabrics, they are defined in a
fixed size, the admin commands are described by virtqueue descriptors.
3, the virtio-of commands need a command_id field to distinguish each
other for command depth, but the admin commands don't need command_id
field(supported by virtqueue descriptor heads).
4, connect/disconnect are fabrics specific, stream/keyed transfer
related features/commands are fabrics specific too.
interrupt related commands are not used by virtio-of.
As far as I can see, they are quite different, defining them separately
would be fine.
--
zhenwei pi
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 33+ messages in thread* [virtio-comment] Re: Re: [PATCH v4 4/7] transport-fabrics: introduce command set
2023-06-28 1:34 ` [virtio-comment] " zhenwei pi
@ 2023-06-28 2:56 ` Jason Wang
2023-06-28 6:53 ` [virtio-comment] " zhenwei pi
0 siblings, 1 reply; 33+ messages in thread
From: Jason Wang @ 2023-06-28 2:56 UTC (permalink / raw)
To: zhenwei pi
Cc: stefanha, parav, mst, virtio-comment, houp, helei.sig11,
zhouhuaping.san, xinhao.kong, Zhu Lingshan
On Wed, Jun 28, 2023 at 9:37 AM zhenwei pi <pizhenwei@bytedance.com> wrote:
>
>
>
> On 6/27/23 17:04, Jason Wang wrote:
> > On Mon, Jun 26, 2023 at 3:28 PM zhenwei pi <pizhenwei@bytedance.com> wrote:
> >>
> >> Introduce command set for Virtio-oF which includes:
> >> - command id: a tag to distinguish in-flight commands
> >> - status: indicate the result of a Virtio-oF command
> >> - opcodes: the Virtio-oF transport layer commands use 0x0000-0x0fff,
> >> and the device layer commands use 0x1000-0xffff.
> >> get/set status/feature/ config use consecutive number.
> >>
> >> Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
> >> ---
> >> transport-fabrics.tex | 616 ++++++++++++++++++++++++++++++++++++++++++
> >> 1 file changed, 616 insertions(+)
> >>
> >> diff --git a/transport-fabrics.tex b/transport-fabrics.tex
> >> index d562b2c..3fd9e8e 100644
> >> --- a/transport-fabrics.tex
> >> +++ b/transport-fabrics.tex
> >> @@ -113,3 +113,619 @@ \subsubsection{Keyed Data Transfers}\label{sec:Virtio Transport Options / Virtio
> >> |COMP| |COMP| |COMP|
> >> +----+ +----+ +----+
> >> \end{lstlisting}
> >> +
> >> +
> >> +\subsection{Commands Definition}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition}
> >> +This section defines data structures, opcodes, and status for Virtio-oF.
> >> +A Virtio-oF command is fixed to 16 bytes, and a Virtio-oF completion is fixed to 16 bytes.
> >> +Note that the reserved fields of Virtio-oF commands and completions are filled with zero.
> >> +
> >> +\subsubsection{Command ID}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Command ID}
> >> +The Command ID (le16 type) is used to uniquely describe a Virtio-oF PDU for identification.
> >> +Generally, the Virtio-oF initiator allocates a Command ID that is unique for all in-flight commands,
> >> +and the Virtio-oF target specifies the same Command ID for completion.
> >> +
> >> +Command IDs 0xff00 - 0xffff are reserved for the Virtio-oF control queue asynchronous events.
> >> +The reserved Command IDs for the Virtio-oF control queue are as follows:
> >> +
> >> +\begin{tabular}{ |l|l| }
> >> +\hline
> >> +Command ID & Description \\
> >> +\hline \hline
> >> +0xfffe & Config change. Causes the initiator to generate a configuration change notification \\
> >> +\hline
> >> +0xffff & Keepalive. The initiator ignores this event \\
> >> +\hline
> >> +0xff00 - 0xfffd & Reserved \\
> >> +\hline
> >> +\end{tabular}
> >> +
> >> +\subsubsection{Status}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Status}
> >> +The Status (le16 type) of a Virtio-oF completion is used to indicate the result of a Virtio-oF command in detail.
> >> +The status values are defined as follows:
> >> +
> >> +\begin{lstlisting}
> >> +/* command executed successfully */
> >> +#define VIRTIO_OF_STATUS_SUCCESS 0x0000
> >> +
> >> +/* unrecognized command, or disabled command due to unsupported feature */
> >> +#define VIRTIO_OF_STATUS_ENOCMD 0x0001
> >> +/* in-flight commands exceeded queue size */
> >> +#define VIRTIO_OF_STATUS_ECMDQUOT 0x0002
> >> +
> >> +/* no such target specified by TVQN */
> >> +#define VIRTIO_OF_STATUS_ENOTGT 0x1001
> >> +/* failed to create Virtio-oF device instance */
> >> +#define VIRTIO_OF_STATUS_ENODEV 0x1002
> >> +/* rejected due to access control */
> >> +#define VIRTIO_OF_STATUS_EACLREJECTED 0x1003
> >> +/* bad Virtio-oF device instance ID */
> >> +#define VIRTIO_OF_STATUS_EBADDEV 0x1010
> >> +/* mismatched TVQN or IVQN from Virtio-oF device instance */
> >> +#define VIRTIO_OF_STATUS_EBADVQN 0x1011
> >> +/* Virtio-oF virtqueue index exceeded */
> >> +#define VIRTIO_OF_STATUS_EQUEUEQUOT 0x1020
> >> +/* Virtio-oF virtqueue size exceeded */
> >> +#define VIRTIO_OF_STATUS_EQSIZEQUOT 0x1021
> >> +
> >> +/* unsupported Virtio-oF feature */
> >> +#define VIRTIO_OF_STATUS_EFEATURE 0x2000
> >> +/* unsupported Virtio-oF device instance status */
> >> +#define VIRTIO_OF_STATUS_ESTATUS 0x2010
> >> +/* unsupported Virtio-oF device instance feature */
> >> +#define VIRTIO_OF_STATUS_EDEVFEATURE 0x2020
> >> +/* unsupported offset of device configuration space */
> >> +#define VIRTIO_OF_STATUS_ECONFOFF 0x2030
> >> +/* unsupported bytes of device configuration space */
> >> +#define VIRTIO_OF_STATUS_ECONFBYTES 0x2031
> >> +/* failed to read device-readable virtqueue buffers */
> >> +#define VIRTIO_OF_STATUS_EOUTVQBUF 0x20f0
> >> +/* failed to write device-writable virtqueue buffers */
> >> +#define VIRTIO_OF_STATUS_EINVQBUF 0x20f1
> >> +\end{lstlisting}
> >> +
> >> +\subsubsection{Opcodes}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes}
> >> +Opcodes (u16 type) of Virtio-oF Commands are as follows:
> >> +
> >> +\begin{lstlisting}
> >> +#define virtio_of_op_connect 0x0000
> >> +#define virtio_of_op_disconnect 0x0001
> >> +#define virtio_of_op_keepalive 0x0002
> >> +#define virtio_of_op_get_feature 0x0004
> >> +#define virtio_of_op_set_feature 0x0005
> >> +#define virtio_of_op_get_keyed_num_descs 0x0100
> >> +#define virtio_of_op_vq 0x0fff
> >> +#define virtio_of_op_get_vendor_id 0x1000
> >> +#define virtio_of_op_get_device_id 0x1001
> >> +#define virtio_of_op_reset_device 0x1003
> >> +#define virtio_of_op_get_status 0x1004
> >> +#define virtio_of_op_set_status 0x1005
> >> +#define virtio_of_op_get_device_feature 0x1006
> >> +#define virtio_of_op_set_driver_feature 0x1009
> >> +#define virtio_of_op_get_vq_size 0x100a
> >> +#define virtio_of_op_get_config 0x100c
> >> +#define virtio_of_op_set_config 0x100d
> >
> > We had a proposal of transport virtqueue that allows using a
> > virtqueue[1] for configuring a device. And Ling shan will rebase it on
> > top of the admin commands.
> >
> > That is to say, admin commands could be used for transport which will
> > be duplicated with the commands you invented here.
> >
> > I wonder if the virtio-OF can just reuse the admin commands (have a
> > quick glance and I didn't find any blocker for this).
> >
> > Thanks
> >
> > [1] https://lists.oasis-open.org/archives/virtio-comment/202208/msg00003.html
> >
>
> From my understand, the virtio device related commands looks
> similar(set/get status/features, reset device ...), but not same:
> 1, the admin commands are executed upon the admin virtqueue, it uses
> virtqueue based transfer.
Note that the actual commands are independent of admin virtqueue. It
is defined separately as admin commands and it would be supported by
the MMIO in the future.
> The virtio-of commands are executed upon a virtio-of queue(established
> reliable connection), not a virtqueue. A virtio-of device could support
> admin commands in theory.
> I suppose a virtio-of device with admin vq supported like:
> +---------------------+
> | +-------------+ |
> | |control queue| |
> | +-------------+ |
> +---------------------+
> | +-------------+ |
> | | virtqueue0 | |
> | +-------------+ |
> | | virtqueue1 | |
> | +-------------+ |
> | | virtqueue ..| |
> | +-------------+ |
> | | admin vq | |
> | +-------------+ |
> +---------------------+
>
> The virtio-of commands ares used by both virtio-of control queue and
> virtio-of virtqueues(include admin vq).
> The admin commands are used by admin vq only.
>
> 2, the virtio-of commands are used over fabrics, they are defined in a
> fixed size,
Any issue you see if it is variable?
> the admin commands are described by virtqueue descriptors.
See above, it's not necessarily carried via descriptors.
>
> 3, the virtio-of commands need a command_id field to distinguish each
> other for command depth, but the admin commands don't need command_id
> field(supported by virtqueue descriptor heads).
I'm not sure I get this, could you give me an example?
>
> 4, connect/disconnect are fabrics specific, stream/keyed transfer
> related features/commands are fabrics specific too.
> interrupt related commands are not used by virtio-of.
So we had:
struct virtio_admin_cmd {
/* Device-readable part */
le16 opcode;
/*
* 1 - SR-IOV
* 2-65535 - reserved
*/
le16 group_type;
...
};
The of specific command could have a dedicated group in this case.
Thanks
>
> As far as I can see, they are quite different, defining them separately
> would be fine.
>
> --
> zhenwei pi
>
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 33+ messages in thread* [virtio-comment] Re: Re: Re: [PATCH v4 4/7] transport-fabrics: introduce command set
2023-06-28 2:56 ` Jason Wang
@ 2023-06-28 6:53 ` zhenwei pi
0 siblings, 0 replies; 33+ messages in thread
From: zhenwei pi @ 2023-06-28 6:53 UTC (permalink / raw)
To: Jason Wang
Cc: stefanha, parav, mst, virtio-comment, houp, helei.sig11,
zhouhuaping.san, xinhao.kong, Zhu Lingshan
On 6/28/23 10:56, Jason Wang wrote:
> On Wed, Jun 28, 2023 at 9:37 AM zhenwei pi <pizhenwei@bytedance.com> wrote:
>>
>>
>>
>> On 6/27/23 17:04, Jason Wang wrote:
>>> On Mon, Jun 26, 2023 at 3:28 PM zhenwei pi <pizhenwei@bytedance.com> wrote:
>>>>
>>>> Introduce command set for Virtio-oF which includes:
>>>> - command id: a tag to distinguish in-flight commands
>>>> - status: indicate the result of a Virtio-oF command
>>>> - opcodes: the Virtio-oF transport layer commands use 0x0000-0x0fff,
[snip]
>>>> +\begin{lstlisting}
>>>> +#define virtio_of_op_connect 0x0000
>>>> +#define virtio_of_op_disconnect 0x0001
>>>> +#define virtio_of_op_keepalive 0x0002
>>>> +#define virtio_of_op_get_feature 0x0004
>>>> +#define virtio_of_op_set_feature 0x0005
>>>> +#define virtio_of_op_get_keyed_num_descs 0x0100
>>>> +#define virtio_of_op_vq 0x0fff
>>>> +#define virtio_of_op_get_vendor_id 0x1000
>>>> +#define virtio_of_op_get_device_id 0x1001
>>>> +#define virtio_of_op_reset_device 0x1003
>>>> +#define virtio_of_op_get_status 0x1004
>>>> +#define virtio_of_op_set_status 0x1005
>>>> +#define virtio_of_op_get_device_feature 0x1006
>>>> +#define virtio_of_op_set_driver_feature 0x1009
>>>> +#define virtio_of_op_get_vq_size 0x100a
>>>> +#define virtio_of_op_get_config 0x100c
>>>> +#define virtio_of_op_set_config 0x100d
>>>
>>> We had a proposal of transport virtqueue that allows using a
>>> virtqueue[1] for configuring a device. And Ling shan will rebase it on
>>> top of the admin commands.
>>>
>>> That is to say, admin commands could be used for transport which will
>>> be duplicated with the commands you invented here.
>>>
>>> I wonder if the virtio-OF can just reuse the admin commands (have a
>>> quick glance and I didn't find any blocker for this).
>>>
>>> Thanks
>>>
>>> [1] https://lists.oasis-open.org/archives/virtio-comment/202208/msg00003.html
>>>
>>
>> From my understand, the virtio device related commands looks
>> similar(set/get status/features, reset device ...), but not same:
>> 1, the admin commands are executed upon the admin virtqueue, it uses
>> virtqueue based transfer.
>
> Note that the actual commands are independent of admin virtqueue. It
> is defined separately as admin commands and it would be supported by
> the MMIO in the future.
>
>> The virtio-of commands are executed upon a virtio-of queue(established
>> reliable connection), not a virtqueue. A virtio-of device could support
>> admin commands in theory.
>> I suppose a virtio-of device with admin vq supported like:
>> +---------------------+
>> | +-------------+ |
>> | |control queue| |
>> | +-------------+ |
>> +---------------------+
>> | +-------------+ |
>> | | virtqueue0 | |
>> | +-------------+ |
>> | | virtqueue1 | |
>> | +-------------+ |
>> | | virtqueue ..| |
>> | +-------------+ |
>> | | admin vq | |
>> | +-------------+ |
>> +---------------------+
>>
>> The virtio-of commands ares used by both virtio-of control queue and
>> virtio-of virtqueues(include admin vq).
>> The admin commands are used by admin vq only.
>>
>> 2, the virtio-of commands are used over fabrics, they are defined in a
>> fixed size,
>
> Any issue you see if it is variable?
>
For stream transfer, a command of variable length is difficult to decode.
For keyed transfer, the receiver side must know the size of commands,
then post recv large enough memory. And unaligned length hurts the
performance.
>> the admin commands are described by virtqueue descriptors.
>
> See above, it's not necessarily carried via descriptors.
>
>>
>> 3, the virtio-of commands need a command_id field to distinguish each
>> other for command depth, but the admin commands don't need command_id
>> field(supported by virtqueue descriptor heads).
>
> I'm not sure I get this, could you give me an example?
>
Please see subsubsection{Command ID} in this patch.
>>
>> 4, connect/disconnect are fabrics specific, stream/keyed transfer
>> related features/commands are fabrics specific too.
>> interrupt related commands are not used by virtio-of.
>
> So we had:
>
> struct virtio_admin_cmd {
> /* Device-readable part */
> le16 opcode;
> /*
> * 1 - SR-IOV
> * 2-65535 - reserved
> */
> le16 group_type;
> ...
> };
>
> The of specific command could have a dedicated group in this case.
>
> Thanks
>
The command set to virtio-of looks like:
[Common configuration structure layout] to Virtio PCI,
[MMIO Device Register Layout] to Virtio MMIO,
[Channel Commands for Virtio] to Virtio Channel I/O.
By the way, [Channel Commands for Virtio] is already defined, is it
possible to reuse it?
>>
>> As far as I can see, they are quite different, defining them separately
>> would be fine.
>>
>> --
>> zhenwei pi
>>
>
--
zhenwei pi
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [virtio-comment] [PATCH v4 4/7] transport-fabrics: introduce command set
2023-06-26 7:25 ` [virtio-comment] [PATCH v4 4/7] transport-fabrics: introduce command set zhenwei pi
2023-06-27 9:04 ` [virtio-comment] " Jason Wang
@ 2023-07-29 23:28 ` Raphael Norwitz
2023-07-31 2:34 ` zhenwei pi
1 sibling, 1 reply; 33+ messages in thread
From: Raphael Norwitz @ 2023-07-29 23:28 UTC (permalink / raw)
To: zhenwei pi
Cc: Stefan Hajnoczi, parav@nvidia.com, Michael S. Tsirkin, Jason Wang,
virtio-comment@lists.oasis-open.org, houp@yusur.tech,
helei.sig11@bytedance.com, zhouhuaping.san@bytedance.com,
xinhao.kong@duke.edu
As noted in my high level feedback I think the protocol should have redirection semantics.
> On Jun 26, 2023, at 3:25 AM, zhenwei pi <pizhenwei@bytedance.com> wrote:
>
> Introduce command set for Virtio-oF which includes:
> - command id: a tag to distinguish in-flight commands
> - status: indicate the result of a Virtio-oF command
> - opcodes: the Virtio-oF transport layer commands use 0x0000-0x0fff,
> and the device layer commands use 0x1000-0xffff.
> get/set status/feature/ config use consecutive number.
>
> Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
> ---
> transport-fabrics.tex | 616 ++++++++++++++++++++++++++++++++++++++++++
> 1 file changed, 616 insertions(+)
>
> diff --git a/transport-fabrics.tex b/transport-fabrics.tex
> index d562b2c..3fd9e8e 100644
> --- a/transport-fabrics.tex
> +++ b/transport-fabrics.tex
> @@ -113,3 +113,619 @@ \subsubsection{Keyed Data Transfers}\label{sec:Virtio Transport Options / Virtio
> |COMP| |COMP| |COMP|
> +----+ +----+ +----+
> \end{lstlisting}
> +
> +
> +\subsection{Commands Definition}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition}
> +This section defines data structures, opcodes, and status for Virtio-oF.
> +A Virtio-oF command is fixed to 16 bytes, and a Virtio-oF completion is fixed to 16 bytes.
> +Note that the reserved fields of Virtio-oF commands and completions are filled with zero.
> +
> +\subsubsection{Command ID}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Command ID}
> +The Command ID (le16 type) is used to uniquely describe a Virtio-oF PDU for identification.
> +Generally, the Virtio-oF initiator allocates a Command ID that is unique for all in-flight commands,
> +and the Virtio-oF target specifies the same Command ID for completion.
> +
> +Command IDs 0xff00 - 0xffff are reserved for the Virtio-oF control queue asynchronous events.
> +The reserved Command IDs for the Virtio-oF control queue are as follows:
> +
> +\begin{tabular}{ |l|l| }
> +\hline
> +Command ID & Description \\
> +\hline \hline
> +0xfffe & Config change. Causes the initiator to generate a configuration change notification \\
> +\hline
> +0xffff & Keepalive. The initiator ignores this event \\
> +\hline
> +0xff00 - 0xfffd & Reserved \\
> +\hline
> +\end{tabular}
> +
> +\subsubsection{Status}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Status}
> +The Status (le16 type) of a Virtio-oF completion is used to indicate the result of a Virtio-oF command in detail.
> +The status values are defined as follows:
> +
> +\begin{lstlisting}
> +/* command executed successfully */
> +#define VIRTIO_OF_STATUS_SUCCESS 0x0000
> +
> +/* unrecognized command, or disabled command due to unsupported feature */
> +#define VIRTIO_OF_STATUS_ENOCMD 0x0001
> +/* in-flight commands exceeded queue size */
> +#define VIRTIO_OF_STATUS_ECMDQUOT 0x0002
> +
> +/* no such target specified by TVQN */
> +#define VIRTIO_OF_STATUS_ENOTGT 0x1001
> +/* failed to create Virtio-oF device instance */
> +#define VIRTIO_OF_STATUS_ENODEV 0x1002
> +/* rejected due to access control */
> +#define VIRTIO_OF_STATUS_EACLREJECTED 0x1003
> +/* bad Virtio-oF device instance ID */
> +#define VIRTIO_OF_STATUS_EBADDEV 0x1010
> +/* mismatched TVQN or IVQN from Virtio-oF device instance */
> +#define VIRTIO_OF_STATUS_EBADVQN 0x1011
> +/* Virtio-oF virtqueue index exceeded */
> +#define VIRTIO_OF_STATUS_EQUEUEQUOT 0x1020
> +/* Virtio-oF virtqueue size exceeded */
> +#define VIRTIO_OF_STATUS_EQSIZEQUOT 0x1021
> +
> +/* unsupported Virtio-oF feature */
> +#define VIRTIO_OF_STATUS_EFEATURE 0x2000
> +/* unsupported Virtio-oF device instance status */
> +#define VIRTIO_OF_STATUS_ESTATUS 0x2010
> +/* unsupported Virtio-oF device instance feature */
> +#define VIRTIO_OF_STATUS_EDEVFEATURE 0x2020
> +/* unsupported offset of device configuration space */
> +#define VIRTIO_OF_STATUS_ECONFOFF 0x2030
> +/* unsupported bytes of device configuration space */
> +#define VIRTIO_OF_STATUS_ECONFBYTES 0x2031
> +/* failed to read device-readable virtqueue buffers */
> +#define VIRTIO_OF_STATUS_EOUTVQBUF 0x20f0
> +/* failed to write device-writable virtqueue buffers */
> +#define VIRTIO_OF_STATUS_EINVQBUF 0x20f1
> +\end{lstlisting}
> +
> +\subsubsection{Opcodes}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes}
> +Opcodes (u16 type) of Virtio-oF Commands are as follows:
> +
> +\begin{lstlisting}
> +#define virtio_of_op_connect 0x0000
> +#define virtio_of_op_disconnect 0x0001
> +#define virtio_of_op_keepalive 0x0002
> +#define virtio_of_op_get_feature 0x0004
> +#define virtio_of_op_set_feature 0x0005
> +#define virtio_of_op_get_keyed_num_descs 0x0100
> +#define virtio_of_op_vq 0x0fff
> +#define virtio_of_op_get_vendor_id 0x1000
> +#define virtio_of_op_get_device_id 0x1001
> +#define virtio_of_op_reset_device 0x1003
> +#define virtio_of_op_get_status 0x1004
> +#define virtio_of_op_set_status 0x1005
> +#define virtio_of_op_get_device_feature 0x1006
> +#define virtio_of_op_set_driver_feature 0x1009
> +#define virtio_of_op_get_vq_size 0x100a
> +#define virtio_of_op_get_config 0x100c
> +#define virtio_of_op_set_config 0x100d
> +\end{lstlisting}
> +
> +\paragraph{Connect Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Connect Command}
> +
This would be a good place to introduce some kind of target-side redirect mechanism to support redirecting to another IP.
Probably just another op/status code and message from the target for redirection?
> +The Connect Command is used to establish a Virtio-oF queue for both the Virtio-oF control queue and Virtio-oF virtqueue, it is always the first command to execute on a Virtio-oF queue.
> +
> +The structure of the Connect Command structure is as follows:
> +\begin{lstlisting}
> +struct virtio_of_command_connect {
> + /* opcode is virtio_of_op_connect */
> + le16 opcode;
> + /* unique ID for all in-flight commands */
> + le16 command_id;
> + /* Virtio-oF device instance ID. 0xffff is reserved for the Virtio-oF control queue */
> + le16 device_instance_id;
> + /* used by Virtio-oF virtqueue only, equal to virtqueue index */
> + le16 vq_index;
> + /* the length of the Connect Body */
> + le32 length;
> +#define VIRTIO_OF_CONTROL_QUEUE_SIZE 32
> + /* the size of a Virtio-oF queue, 0 means the maximum queue size supported */
> + le16 queue_size;
> + u8 reserved[2];
> +};
> +\end{lstlisting}
> +
> +The structure of the Connect Body is as follows:
> +\begin{lstlisting}
> +struct virtio_of_connect {
> + /* Virtio-oF initiator VQN */
> + u8 ivqn[256];
> + /* Virtio-oF target VQN */
> + u8 tvqn[256];
> + u8 reserved[512];
> +};
> +\end{lstlisting}
> +
> +The structure of the Connect Completion is as follows:
> +\begin{lstlisting}
> +struct virtio_of_completion_connect {
> + le16 status;
> + le16 command_id;
> + /* Virtio-oF device instance ID */
> + le16 device_instance_id;
> + u8 reserved[10];
> +};
> +\end{lstlisting}
> +
> +The Connect Command, Body, and Completion have the following usages:
> +\begin{itemize}
> +\item The Virtio-oF initiator specifies \texttt{device_instance_id} of 0xffff in the Connect command
> +and Virtio-oF initiator/target VQN in the Connect Body to establish the Virtio-oF control queue first.
> +\item The Virtio-oF target allocates any available ID for a newly created Virtio-oF device instance,
> +and specifies the \texttt{device_instance_id} of the new Virtio-oF device instance ID in the Connect Completion.
> +\item The Virtio-oF initiator specifies the Virtio-oF device instance ID and \texttt{vq_index} to establish Virtio-oF virtqueues one by one.
> +The Connect Body is optional for the Virtio-oF virtqueues, once a Virtio-oF virtqueue issues a Connect Command without Connect Body, the Virtio-oF virtqueue uses the same Connect Body as the Virtio-oF control queue.
> +\end{itemize}
> +
> +\paragraph{Disconnect Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Disconnect Command}
> +
> +The Disconnect Command is used to gracefully disconnect both the Virtio-oF control queue and Virtio-oF virtqueue. All in-flight commands are completed upon receipt of the Disconnect Completion.
> +
> +The structure of the Disconnect Command structure is as follows:
> +\begin{lstlisting}
> +struct virtio_of_command_disconnect {
> + /* opcode is virtio_of_op_disconnect */
> + le16 opcode;
> + /* unique ID for all in-flight commands */
> + le16 command_id;
> + u8 reserved[12];
> +};
> +\end{lstlisting}
> +
> +The structure of the Disconnect Completion is as follows:
> +\begin{lstlisting}
> +struct virtio_of_completion_disconnect {
> + le16 status;
> + le16 command_id;
> + u8 reserved[12];
> +};
> +\end{lstlisting}
> +
> +The Disconnect Command and Completion have the following usages:
> +\begin{itemize}
> +\item The Virtio-oF initiator should disconnect the Virtio-oF virtqueues before the Virtio-oF control queue.
> +\item Once the Virtio-oF control queue is disconnected, the Virtio-oF device instance and associated resources should be destroyed.
> +\item If the underlying transport connection of a Virtio-oF queue is disconnected without issuing the Disconnect Command, the in-flight commands are not guaranteed to execute successfully.
> +\end{itemize}
> +
> +\paragraph{Keepalive Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Keepalive Command}
> +The Keepalive Command is used as a health check mechanism to detect when a Virtio-oF device instance becomes unavailable, it is used for the Virtio-oF control queue only.
> +Once the Virtio-oF device instance becomes unavailable, the initiator should disconnect all the Virtio-oF queues and release associated resources.
> +
> +The structure of the Keepalive Command structure is as follows:
> +\begin{lstlisting}
> +struct virtio_of_command_keepalive {
> + /* opcode is virtio_of_op_keepalive */
> + le16 opcode;
> + /* unique ID for all in-flight commands */
> + le16 command_id;
> + u8 reserved[12];
> +};
> +\end{lstlisting}
> +
> +The structure of the Keepalive Completion is as follows:
> +\begin{lstlisting}
> +struct virtio_of_completion_keepalive {
> + le16 status;
> + le16 command_id;
> + u8 reserved[12];
> +};
> +\end{lstlisting}
> +
> +\paragraph{Get Feature Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Get Feature Command}
> +The Get Feature Command is used to get feature bits of a Virtio-oF device instance for the Virtio-oF control queue only.
> +
> +The structure of the Get Feature Command structure is as follows:
> +\begin{lstlisting}
> +struct virtio_of_command_get_feature {
> + /* opcode is virtio_of_op_get_feature */
> + le16 opcode;
> + /* unique ID for all in-flight commands */
> + le16 command_id;
> + /* value 0x0 selects feature bits 0 to 63, 0x1 selects feature bits 64 to 127, etc */
> + le32 feature_select;
> + u8 reserved[8];
> +};
> +\end{lstlisting}
> +
> +The structure of the Get Feature Completion is as follows:
> +\begin{lstlisting}
> +struct virtio_of_completion_get_feature {
> + le16 status;
> + le16 command_id;
> + u8 reserved[4];
> + /* feature bits */
> + le64 feature;
> +};
> +\end{lstlisting}
> +
> +The feature bits of a Virtio-oF device instance are as follows:
> +\begin{lstlisting}
> +/* support virtio_of_op_get_keyed_num_descs to get the maximum number of keyed descriptors */
> +#define VIRTIO_OF_F_KEYED_NUM_DESCS 0
> +\end{lstlisting}
> +
> +\paragraph{Set Feature Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Set Feature Command}
> +The Set Feature Command is used to set the feature bits of a Virtio-oF device instance for the Virtio-oF control queue only.
> +
> +The structure of the Set Feature Command structure is as follows:
> +\begin{lstlisting}
> +struct virtio_of_command_set_feature {
> + /* opcode is virtio_of_op_set_feature */
> + le16 opcode;
> + /* unique ID for all in-flight commands */
> + le16 command_id;
> + /* value 0x0 selects feature bits 0 to 63, 0x1 selects feature bits 64 to 127, etc */
> + le32 feature_select;
> + /* feature bits */
> + le64 feature;
> +};
> +\end{lstlisting}
> +
> +The structure of the Set Feature Completion is as follows:
> +\begin{lstlisting}
> +struct virtio_of_completion_set_feature {
> + le16 status;
> + le16 command_id;
> + u8 reserved[12];
> +};
> +\end{lstlisting}
> +
> +Note: it is recommended that the
> +\nameref{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Get Feature Command}
> +and Set Feature Command should be executed upon the establishment of the Virtio-oF control queue for feature negotiation.
> +However, they are allowed to be executed during the Virtio-oF control queue lifecycle.
> +
> +\paragraph{Get Keyed Number Descriptors Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Get Keyed Number Descriptors Command}
> +The Get Keyed Number Descriptors Command is used to get the maximum number of keyed descriptors of Virtio-oF virtqueues, it is executed on the Virtio-oF control queue.
> +
> +The structure of the Get Keyed Number Descriptors Command structure is as follows:
> +\begin{lstlisting}
> +struct virtio_of_command_get_keyed_num_descs {
> + /* opcode is virtio_of_op_get_keyed_num_descs */
> + le16 opcode;
> + /* unique ID for all in-flight commands */
> + le16 command_id;
> + u8 reserved[12];
> +};
> +\end{lstlisting}
> +
> +The structure of the Get Keyed Number Descriptors Completion is as follows:
> +\begin{lstlisting}
> +struct virtio_of_completion_get_keyed_num_descs {
> + le16 status;
> + le16 command_id;
> + /* the maximum number of keyed descriptors */
> + u8 descs;
> + u8 reserved[11];
> +};
> +\end{lstlisting}
> +
> +\paragraph{VQ Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / VQ Command}
> +The VQ Command is used to transmit virtqueue buffers for Virtio-oF virtqueue only.
> +
> +The structure of the VQ Command structure is as follows:
> +\begin{lstlisting}
> +struct virtio_of_command_vq {
> + /* opcode is virtio_of_op_vq */
> + le16 opcode;
> + /* unique ID for all in-flight commands */
> + le16 command_id;
> + u8 reserved[4];
> + /* the length of device-readable virtqueue buffers */
> + le32 out_length;
> + /* the length of device-writable virtqueue buffers */
> + le32 in_length;
> +};
> +\end{lstlisting}
> +
> +The structure of the VQ Completion is as follows:
> +\begin{lstlisting}
> +struct virtio_of_completion_vq {
> + le16 status;
> + le16 command_id;
> + /* the length of virtqueue buffers from Virtio-oF target */
> + le32 length;
> + u8 reserved[8];
> +};
> +\end{lstlisting}
> +
> +For \nameref{sec:Virtio Transport Options / Virtio Over Fabrics / Protocol Data Unit/ Stream Data Transfers},
> +\texttt{out_length} of virtio_of_command_vq describes the following bytes in a request Virtio-oF PDU,
> +and \texttt{length} of virtio_of_completion_vq describes the following bytes in a response Virtio-oF PDU.
> +
> +For \nameref{sec:Virtio Transport Options / Virtio Over Fabrics / Protocol Data Unit/ Keyed Data Transfers},
> +there are 1 or more keyed descriptors (virtio_of_keyed_desc type) in a request Virtio-oF PDU to describe the virtqueue buffers of \texttt{out_length + in_length} bytes,
> +and \texttt{length} of virtio_of_completion_vq describes the number of bytes written into the device writable portion of the buffer described by the keyed descriptors.
> +
> +\paragraph{Get Vendor ID Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Get Vendor ID Command}
> +The Get Vendor ID Command is used to get the Virtio Vendor ID of a Virtio-oF device instance for the Virtio-oF control queue only.
> +
> +The structure of the Get Vendor ID Command structure is as follows:
> +\begin{lstlisting}
> +struct virtio_of_command_get_vendor_id {
> + /* opcode is virtio_of_op_get_vendor_id */
> + le16 opcode;
> + /* unique ID for all in-flight commands */
> + le16 command_id;
> + u8 reserved[12];
> +};
> +\end{lstlisting}
> +
> +The structure of the Get Vendor ID Completion is as follows:
> +\begin{lstlisting}
> +struct virtio_of_completion_get_vendor_id {
> + le16 status;
> + le16 command_id;
> + /* Virtio Vendor ID */
> + le32 vendor_id;
> + u8 reserved[8];
> +};
> +\end{lstlisting}
> +
> +\paragraph{Get Device ID Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Get Device ID Command}
> +The Get Device ID Command is used to get the Virtio Device ID of a Virtio-oF device instance for the Virtio-oF control queue only.
> +
> +The structure of the Get Device ID Command structure is as follows:
> +\begin{lstlisting}
> +struct virtio_of_command_get_device_id {
> + /* opcode is virtio_of_op_get_device_id */
> + le16 opcode;
> + /* unique ID for all in-flight commands */
> + le16 command_id;
> + u8 reserved[12];
> +};
> +\end{lstlisting}
> +
> +The structure of the Get Device ID Completion is as follows:
> +\begin{lstlisting}
> +struct virtio_of_completion_get_device_id {
> + le16 status;
> + le16 command_id;
> + /* Virtio Device ID */
> + le32 device_id;
> + u8 reserved[8];
> +};
> +\end{lstlisting}
> +
> +\paragraph{Reset Device Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Reset Device Command}
> +The Reset Device Command is used to reset a Virtio-oF device instance for the Virtio-oF control queue only.
> +
> +The structure of the Reset Device Command structure is as follows:
> +\begin{lstlisting}
> +struct virtio_of_command_reset_device {
> + /* opcode is virtio_of_op_get_device_id */
> + le16 opcode;
> + /* unique ID for all in-flight commands */
> + le16 command_id;
> + u8 reserved[12];
> +};
> +\end{lstlisting}
> +
> +The structure of the Reset Device Completion is as follows:
> +\begin{lstlisting}
> +struct virtio_of_completion_reset_device {
> + le16 status;
> + le16 command_id;
> + u8 reserved[12];
> +};
> +\end{lstlisting}
> +
> +\paragraph{Get Status Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Get Status Command}
> +The Get Status Command is used to get the status of a Virtio-oF device instance for the Virtio-oF control queue only.
> +
> +The structure of the Get Status Command structure is as follows:
> +\begin{lstlisting}
> +struct virtio_of_command_get_status {
> + /* opcode is virtio_of_op_get_status */
> + le16 opcode;
> + /* unique ID for all in-flight commands */
> + le16 command_id;
> + u8 reserved[12];
> +};
> +\end{lstlisting}
> +
> +The structure of the Get Status Completion is as follows:
> +\begin{lstlisting}
> +struct virtio_of_completion_get_status {
> + le16 status;
> + le16 command_id;
> + /* status of the Virtio-oF device instance */
> + le32 dev_status;
> + u8 reserved[8];
> +};
> +\end{lstlisting}
> +
> +\paragraph{Set Status Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Set Status Command}
> +The Set Status Command is used to set the status of a Virtio-oF device instance for the Virtio-oF control queue only.
> +
> +The structure of the Set Status Command structure is as follows:
> +\begin{lstlisting}
> +struct virtio_of_command_set_status {
> + /* opcode is virtio_of_op_set_status */
> + le16 opcode;
> + /* unique ID for all in-flight commands */
> + le16 command_id;
> + /* status of the Virtio-oF device instance */
> + le32 status;
> + u8 reserved[8];
> +};
> +\end{lstlisting}
> +
> +The structure of the Set Status Completion is as follows:
> +\begin{lstlisting}
> +struct virtio_of_completion_set_status {
> + le16 status;
> + le16 command_id;
> + u8 reserved[12];
> +};
> +\end{lstlisting}
> +
> +\paragraph{Get Device Feature Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Get Device Feature Command}
> +The Get Device Feature Command is used to get virtio device feature bits of a Virtio-oF device instance for the Virtio-oF control queue only.
> +
> +The structure of the Get Device Feature Command structure is as follows:
> +\begin{lstlisting}
> +struct virtio_of_command_get_device_feature {
> + /* opcode is virtio_of_op_get_device_feature */
> + le16 opcode;
> + /* unique ID for all in-flight commands */
> + le16 command_id;
> + /* value 0x0 selects feature bits 0 to 63, 0x1 selects feature bits 64 to 127, etc */
> + le32 feature_select;
> + u8 reserved[8];
> +};
> +\end{lstlisting}
> +
> +The structure of the Get Device Feature Completion is as follows:
> +\begin{lstlisting}
> +struct virtio_of_completion_get_device_feature {
> + le16 status;
> + le16 command_id;
> + u8 reserved[4];
> + /* feature bits */
> + le64 feature;
> +};
> +\end{lstlisting}
> +
> +\paragraph{Set Driver Feature Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Set Driver Feature Command}
> +The Set Driver Feature Command is used to set virtio driver feature bits of a Virtio-oF device instance for the Virtio-oF control queue only.
> +
> +The structure of the Set Driver Feature Command structure is as follows:
> +\begin{lstlisting}
> +struct virtio_of_command_set_driver_feature {
> + /* opcode is virtio_of_op_set_driver_feature */
> + le16 opcode;
> + /* unique ID for all in-flight commands */
> + le16 command_id;
> + /* value 0x0 selects feature bits 0 to 63, 0x1 selects feature bits 64 to 127, etc */
> + le32 feature_select;
> + /* feature bits */
> + le64 feature;
> +};
> +\end{lstlisting}
> +
> +The structure of the Set Driver Feature Completion is as follows:
> +\begin{lstlisting}
> +struct virtio_of_completion_set_driver_feature {
> + le16 status;
> + le16 command_id;
> + u8 reserved[12];
> +};
> +\end{lstlisting}
> +
> +\paragraph{Get VQ Size Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Get VQ Size Command}
> +The Get VQ Size Command is used to get the maximum queue size of a Virtio-oF virtqueue for the Virtio-oF control queue only.
> +
> +The structure of the Get VQ Size Command structure is as follows:
> +\begin{lstlisting}
> +struct virtio_of_command_get_vq_size {
> + /* opcode is virtio_of_op_get_vq_size */
> + le16 opcode;
> + /* unique ID for all in-flight commands */
> + le16 command_id;
> + /* the Virtio-oF virtqueue index */
> + le16 vq_index;
> + u8 reserved[10];
> +};
> +\end{lstlisting}
> +
> +The structure of the Get VQ Size Completion is as follows:
> +\begin{lstlisting}
> +struct virtio_of_completion_get_vq_size {
> + le16 status;
> + le16 command_id;
> + /* size of Virtio-oF virtqueues */
> + le16 size;
> + u8 reserved[10];
> +};
> +\end{lstlisting}
> +
> +Note: the Virtio-oF control queue has a fixed queue size, the Virtio-oF initiator can specify a smaller Virtio-oF virtqueue size by
> +\nameref{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Connect Command}.
> +
> +\paragraph{Get Config Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Get Config Command}
> +The Get Config Command is used to get the device configuration space of a Virtio-oF device instance for the Virtio-oF control queue only.
> +
> +The structure of the Get Config Command structure is as follows:
> +\begin{lstlisting}
> +struct virtio_of_command_get_config {
> + /* opcode is virtio_of_op_get_config */
> + le16 opcode;
> + /* unique ID for all in-flight commands */
> + le16 command_id;
> + /* offset of device configuration space */
> + le16 offset;
> + /* bytes to get, available values: 1, 2, 4, 8 */
> + u8 bytes;
> + u8 reserved[9];
> +};
> +\end{lstlisting}
> +
> +The structure of the Get Config Completion is as follows:
> +\begin{lstlisting}
> +struct virtio_of_completion_get_config {
> + le16 status;
> + le16 command_id;
> + /* generation count for the device configuration space */
> + le32 generation;
> + /* v is u8, u16, u32, or u64, then: v = (typeof v)le64_to_cpu(config) */
> + le64 config;
> +};
> +\end{lstlisting}
> +
> +\paragraph{Set Config Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Set Config Command}
> +The Set Config Command is used to set the device configuration space of a Virtio-oF device instance for the Virtio-oF control queue only.
> +
> +The structure of the Set Config Command structure is as follows:
> +\begin{lstlisting}
> +struct virtio_of_command_set_config {
> + /* opcode is virtio_of_op_set_config */
> + le16 opcode;
> + /* unique ID for all in-flight commands */
> + le16 command_id;
> + /* offset of device configuration space */
> + le16 offset;
> + /* bytes to get, available values: 1, 2, 4, 8 */
> + u8 bytes;
> + u8 reserved;
> + /* v is u8, u16, u32, or u64, then: config = cpu_to_le64((u64)v) */
> + le64 config;
> +};
> +\end{lstlisting}
> +
> +The structure of the Set Config Completion is as follows:
> +\begin{lstlisting}
> +struct virtio_of_completion_set_config {
> + le16 status;
> + le16 command_id;
> + u8 reserved[12];
> +};
> +\end{lstlisting}
> +
> +\paragraph{Config Change Completion}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Config Change Completion}
> +A Config Change Completion is used as a configuration change notification of a Virtio-oF device instance for the Virtio-oF control queue only.
> +Note: only a single outstanding Config Change Completion is allowed, the configuration change notification is suppressed until the Virtio-oF initiator issues a
> +\nameref{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Get Config Command}.
> +
> +The structure of the Config Change Completion is as follows:
> +\begin{lstlisting}
> +struct virtio_of_completion_config_change {
> + le16 status;
> + /* command_id is 0xfffe */
> + le16 command_id;
> + /* generation count for the device configuration space */
> + le32 generation;
> + u8 reserved[8];
> +};
> +\end{lstlisting}
> +
> +\paragraph{Keepalive Completion}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Keepalive Completion}
> +A Keepalive Completion is used as a health check mechanism to detect when a Virtio-oF initiator becomes unavailable, it is used by a Virtio-oF device instance for the Virtio-oF control queue only.
> +Once the Virtio-oF initiator becomes unavailable, the Virtio-oF target should destroy the Virtio-oF device instance and associated resources.
> --
> 2.25.1
>
>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>
>
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 33+ messages in thread* Re: Re: [virtio-comment] [PATCH v4 4/7] transport-fabrics: introduce command set
2023-07-29 23:28 ` [virtio-comment] " Raphael Norwitz
@ 2023-07-31 2:34 ` zhenwei pi
2023-08-01 0:25 ` Raphael Norwitz
0 siblings, 1 reply; 33+ messages in thread
From: zhenwei pi @ 2023-07-31 2:34 UTC (permalink / raw)
To: Raphael Norwitz
Cc: Stefan Hajnoczi, parav@nvidia.com, Michael S. Tsirkin, Jason Wang,
virtio-comment@lists.oasis-open.org, houp@yusur.tech,
zhouhuaping.san@bytedance.com, xinhao.kong@duke.edu
On 7/30/23 07:28, Raphael Norwitz wrote:
> As noted in my high level feedback I think the protocol should have redirection semantics.
>
[snip]
>> +\end{lstlisting}
>> +
>> +\paragraph{Connect Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Connect Command}
>> +
>
> This would be a good place to introduce some kind of target-side redirect mechanism to support redirecting to another IP.
>
> Probably just another op/status code and message from the target for redirection?
>
Redirect mechanism is not used widely in my practice. Instead, multi
path mechanism is used a lot. Would you please share your experience on
this?
--
zhenwei pi
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 33+ messages in thread* Re: [virtio-comment] [PATCH v4 4/7] transport-fabrics: introduce command set
2023-07-31 2:34 ` zhenwei pi
@ 2023-08-01 0:25 ` Raphael Norwitz
2023-08-01 2:00 ` zhenwei pi
0 siblings, 1 reply; 33+ messages in thread
From: Raphael Norwitz @ 2023-08-01 0:25 UTC (permalink / raw)
To: zhenwei pi
Cc: Stefan Hajnoczi, parav@nvidia.com, Michael S. Tsirkin, Jason Wang,
virtio-comment@lists.oasis-open.org, houp@yusur.tech,
zhouhuaping.san@bytedance.com, xinhao.kong@duke.edu
> On Jul 30, 2023, at 10:34 PM, zhenwei pi <pizhenwei@bytedance.com> wrote:
>
>
>
> On 7/30/23 07:28, Raphael Norwitz wrote:
>> As noted in my high level feedback I think the protocol should have redirection semantics.
> [snip]
>>> +\end{lstlisting}
>>> +
>>> +\paragraph{Connect Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Connect Command}
>>> +
>> This would be a good place to introduce some kind of target-side redirect mechanism to support redirecting to another IP.
>> Probably just another op/status code and message from the target for redirection?
>
> Redirect mechanism is not used widely in my practice. Instead, multi path mechanism is used a lot. Would you please share your experience on this?
>
Agreed - multipath is more common in iSCSI implementations. How would you make multipath work for Virtio-o-F since you only have one TCP or RDMA connection per VQ?
What concern do you have with redirection-based iSCSI implementations? It’s conceptually quite trivial. We have published documentation on our implementation here: https://www.nutanixbible.com/5b-book-of-ahv-how-it-works.html .
I imagine Virtio-oF implementations could do something similar to avoid reliance on a fixed IP.
> --
> zhenwei pi
^ permalink raw reply [flat|nested] 33+ messages in thread* Re: Re: [virtio-comment] [PATCH v4 4/7] transport-fabrics: introduce command set
2023-08-01 0:25 ` Raphael Norwitz
@ 2023-08-01 2:00 ` zhenwei pi
0 siblings, 0 replies; 33+ messages in thread
From: zhenwei pi @ 2023-08-01 2:00 UTC (permalink / raw)
To: Raphael Norwitz
Cc: Stefan Hajnoczi, parav@nvidia.com, Michael S. Tsirkin, Jason Wang,
virtio-comment@lists.oasis-open.org, houp@yusur.tech,
zhouhuaping.san@bytedance.com, xinhao.kong@duke.edu
On 8/1/23 08:25, Raphael Norwitz wrote:
>
>
>> On Jul 30, 2023, at 10:34 PM, zhenwei pi <pizhenwei@bytedance.com> wrote:
>>
>>
>>
>> On 7/30/23 07:28, Raphael Norwitz wrote:
>>> As noted in my high level feedback I think the protocol should have redirection semantics.
>> [snip]
>>>> +\end{lstlisting}
>>>> +
>>>> +\paragraph{Connect Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Connect Command}
>>>> +
>>> This would be a good place to introduce some kind of target-side redirect mechanism to support redirecting to another IP.
>>> Probably just another op/status code and message from the target for redirection?
>>
>> Redirect mechanism is not used widely in my practice. Instead, multi path mechanism is used a lot. Would you please share your experience on this?
>>
>
> Agreed - multipath is more common in iSCSI implementations. How would you make multipath work for Virtio-o-F since you only have one TCP or RDMA connection per VQ?
>
> What concern do you have with redirection-based iSCSI implementations? It’s conceptually quite trivial. We have published documentation on our implementation here: https://www.nutanixbible.com/5b-book-of-ahv-how-it-works.html .
>
> I imagine Virtio-oF implementations could do something similar to avoid reliance on a fixed IP.
>
>> --
>> zhenwei pi
>
Hi,
I'm not against redirect mechanism(like iSCSI implementations), but this
leads complexity(especially to a draft version).
The alternative implementations in my mind of the first step:
- configure Virtio-oF IPs to DNS server. The initiator resolves the IP
on connecting/re-connecting. (this looks like iSCSI redirection - use
DNS server instead of iSCSI redirection server, but no additional
specification definition is needed)
- Or, configure IP list to a initiator, the initiator select
one(RR/random...) on connecting/re-connecting.
--
zhenwei pi
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 33+ messages in thread
* [virtio-comment] [PATCH v4 5/7] transport-fabrics: introduce transport binding
2023-06-26 7:25 [virtio-comment] [PATCH v4 0/7] Introduce Virtio Over Fabrics zhenwei pi
` (3 preceding siblings ...)
2023-06-26 7:25 ` [virtio-comment] [PATCH v4 4/7] transport-fabrics: introduce command set zhenwei pi
@ 2023-06-26 7:25 ` zhenwei pi
2023-07-29 23:28 ` Raphael Norwitz
2023-06-26 7:25 ` [virtio-comment] [PATCH v4 6/7] transport-fabrics: add device initialization zhenwei pi
` (2 subsequent siblings)
7 siblings, 1 reply; 33+ messages in thread
From: zhenwei pi @ 2023-06-26 7:25 UTC (permalink / raw)
To: stefanha, parav, mst, jasowang
Cc: virtio-comment, houp, helei.sig11, zhouhuaping.san, xinhao.kong,
zhenwei pi
Add TCP/IP, TLS TCP/IP, RDMA transports.
Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
---
transport-fabrics.tex | 26 ++++++++++++++++++++++++++
1 file changed, 26 insertions(+)
diff --git a/transport-fabrics.tex b/transport-fabrics.tex
index 3fd9e8e..75e4855 100644
--- a/transport-fabrics.tex
+++ b/transport-fabrics.tex
@@ -729,3 +729,29 @@ \subsubsection{Opcodes}\label{sec:Virtio Transport Options / Virtio Over Fabrics
\paragraph{Keepalive Completion}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Keepalive Completion}
A Keepalive Completion is used as a health check mechanism to detect when a Virtio-oF initiator becomes unavailable, it is used by a Virtio-oF device instance for the Virtio-oF control queue only.
Once the Virtio-oF initiator becomes unavailable, the Virtio-oF target should destroy the Virtio-oF device instance and associated resources.
+
+
+\subsection{Transport Binding}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Transport Binding}
+
+\subsubsection{TCP/IP}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Transport Binding / TCP IP}
+Virtio-oF supports both TCP/IPv4 and TCP/IPv6, it uses
+\nameref{sec:Virtio Transport Options / Virtio Over Fabrics / Protocol Data Unit/ Stream Data Transfers}.
+A TCP/IP based Virtio-oF target uses port 8549 (CRC-16/ARC of "Virtio") by default.
+
+\subsubsection{TLS TCP/IP}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Transport Binding / TLS TCP IP}
+Virtio-oF supports both TLS v1.2 and TLS v1.3 over the underlying TCP/IPv4 and TCP/IPv6, it uses
+\nameref{sec:Virtio Transport Options / Virtio Over Fabrics / Protocol Data Unit/ Stream Data Transfers}.
+A TLS TCP/IP based Virtio-oF target uses port 8549 (CRC-16/ARC of "Virtio") by default.
+
+\subsubsection{RDMA}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Transport Binding / RDMA}
+Virtio-oF supports RDMA, it uses
+\nameref{sec:Virtio Transport Options / Virtio Over Fabrics / Protocol Data Unit/ Keyed Data Transfers}.
+A RoCEv2 based Virtio-oF target uses UDP port 8549 (CRC-16/ARC of "Virtio") by default, and a iWARP based Virtio-oF target uses TCP port 8549 by default.
+
+RDMA uses the following operations:
+\begin{itemize}
+\item The Virtio-oF initiator sends a PDU by RDMA SEND.
+\item The Virtio-oF target reads the remote data by RDMA SEND.
+\item The Virtio-oF target writes the remote data by RDMA WRITE.
+\item The Virtio-oF target sends a PDU by RDMA SEND.
+\end{itemize}
--
2.25.1
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply related [flat|nested] 33+ messages in thread* Re: [virtio-comment] [PATCH v4 5/7] transport-fabrics: introduce transport binding
2023-06-26 7:25 ` [virtio-comment] [PATCH v4 5/7] transport-fabrics: introduce transport binding zhenwei pi
@ 2023-07-29 23:28 ` Raphael Norwitz
2023-07-31 2:36 ` zhenwei pi
0 siblings, 1 reply; 33+ messages in thread
From: Raphael Norwitz @ 2023-07-29 23:28 UTC (permalink / raw)
To: zhenwei pi
Cc: Stefan Hajnoczi, parav@nvidia.com, Michael S. Tsirkin, Jason Wang,
virtio-comment@lists.oasis-open.org, houp@yusur.tech,
helei.sig11@bytedance.com, zhouhuaping.san@bytedance.com,
xinhao.kong@duke.edu
> On Jun 26, 2023, at 3:25 AM, zhenwei pi <pizhenwei@bytedance.com> wrote:
>
> Add TCP/IP, TLS TCP/IP, RDMA transports.
>
> Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
> ---
> transport-fabrics.tex | 26 ++++++++++++++++++++++++++
> 1 file changed, 26 insertions(+)
>
> diff --git a/transport-fabrics.tex b/transport-fabrics.tex
> index 3fd9e8e..75e4855 100644
> --- a/transport-fabrics.tex
> +++ b/transport-fabrics.tex
> @@ -729,3 +729,29 @@ \subsubsection{Opcodes}\label{sec:Virtio Transport Options / Virtio Over Fabrics
> \paragraph{Keepalive Completion}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Keepalive Completion}
> A Keepalive Completion is used as a health check mechanism to detect when a Virtio-oF initiator becomes unavailable, it is used by a Virtio-oF device instance for the Virtio-oF control queue only.
> Once the Virtio-oF initiator becomes unavailable, the Virtio-oF target should destroy the Virtio-oF device instance and associated resources.
> +
> +
> +\subsection{Transport Binding}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Transport Binding}
> +
> +\subsubsection{TCP/IP}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Transport Binding / TCP IP}
> +Virtio-oF supports both TCP/IPv4 and TCP/IPv6, it uses
> +\nameref{sec:Virtio Transport Options / Virtio Over Fabrics / Protocol Data Unit/ Stream Data Transfers}.
> +A TCP/IP based Virtio-oF target uses port 8549 (CRC-16/ARC of "Virtio") by default.
> +
> +\subsubsection{TLS TCP/IP}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Transport Binding / TLS TCP IP}
> +Virtio-oF supports both TLS v1.2 and TLS v1.3 over the underlying TCP/IPv4 and TCP/IPv6, it uses
> +\nameref{sec:Virtio Transport Options / Virtio Over Fabrics / Protocol Data Unit/ Stream Data Transfers}.
> +A TLS TCP/IP based Virtio-oF target uses port 8549 (CRC-16/ARC of "Virtio") by default.
> +
> +\subsubsection{RDMA}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Transport Binding / RDMA}
> +Virtio-oF supports RDMA, it uses
> +\nameref{sec:Virtio Transport Options / Virtio Over Fabrics / Protocol Data Unit/ Keyed Data Transfers}.
> +A RoCEv2 based Virtio-oF target uses UDP port 8549 (CRC-16/ARC of "Virtio") by default, and a iWARP based Virtio-oF target uses TCP port 8549 by default.
> +
> +RDMA uses the following operations:
> +\begin{itemize}
> +\item The Virtio-oF initiator sends a PDU by RDMA SEND.
What about with “eager buffers”? It probably uses RDMA write for those?
> +\item The Virtio-oF target reads the remote data by RDMA SEND.
I assume you meant the target reads with RDMA read?
> +\item The Virtio-oF target writes the remote data by RDMA WRITE.
> +\item The Virtio-oF target sends a PDU by RDMA SEND.
> +\end{itemize}
> --
> 2.25.1
>
>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>
>
^ permalink raw reply [flat|nested] 33+ messages in thread* Re: Re: [virtio-comment] [PATCH v4 5/7] transport-fabrics: introduce transport binding
2023-07-29 23:28 ` Raphael Norwitz
@ 2023-07-31 2:36 ` zhenwei pi
0 siblings, 0 replies; 33+ messages in thread
From: zhenwei pi @ 2023-07-31 2:36 UTC (permalink / raw)
To: Raphael Norwitz
Cc: Stefan Hajnoczi, parav@nvidia.com, Michael S. Tsirkin, Jason Wang,
virtio-comment@lists.oasis-open.org, houp@yusur.tech,
zhouhuaping.san@bytedance.com, xinhao.kong@duke.edu
On 7/30/23 07:28, Raphael Norwitz wrote:
>
>
>> On Jun 26, 2023, at 3:25 AM, zhenwei pi <pizhenwei@bytedance.com> wrote:
>>
>> Add TCP/IP, TLS TCP/IP, RDMA transports.
>>
>> Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
>> ---
>> transport-fabrics.tex | 26 ++++++++++++++++++++++++++
>> 1 file changed, 26 insertions(+)
>>
>> diff --git a/transport-fabrics.tex b/transport-fabrics.tex
>> index 3fd9e8e..75e4855 100644
>> --- a/transport-fabrics.tex
>> +++ b/transport-fabrics.tex
>> @@ -729,3 +729,29 @@ \subsubsection{Opcodes}\label{sec:Virtio Transport Options / Virtio Over Fabrics
>> \paragraph{Keepalive Completion}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Keepalive Completion}
>> A Keepalive Completion is used as a health check mechanism to detect when a Virtio-oF initiator becomes unavailable, it is used by a Virtio-oF device instance for the Virtio-oF control queue only.
>> Once the Virtio-oF initiator becomes unavailable, the Virtio-oF target should destroy the Virtio-oF device instance and associated resources.
>> +
>> +
>> +\subsection{Transport Binding}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Transport Binding}
>> +
>> +\subsubsection{TCP/IP}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Transport Binding / TCP IP}
>> +Virtio-oF supports both TCP/IPv4 and TCP/IPv6, it uses
>> +\nameref{sec:Virtio Transport Options / Virtio Over Fabrics / Protocol Data Unit/ Stream Data Transfers}.
>> +A TCP/IP based Virtio-oF target uses port 8549 (CRC-16/ARC of "Virtio") by default.
>> +
>> +\subsubsection{TLS TCP/IP}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Transport Binding / TLS TCP IP}
>> +Virtio-oF supports both TLS v1.2 and TLS v1.3 over the underlying TCP/IPv4 and TCP/IPv6, it uses
>> +\nameref{sec:Virtio Transport Options / Virtio Over Fabrics / Protocol Data Unit/ Stream Data Transfers}.
>> +A TLS TCP/IP based Virtio-oF target uses port 8549 (CRC-16/ARC of "Virtio") by default.
>> +
>> +\subsubsection{RDMA}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Transport Binding / RDMA}
>> +Virtio-oF supports RDMA, it uses
>> +\nameref{sec:Virtio Transport Options / Virtio Over Fabrics / Protocol Data Unit/ Keyed Data Transfers}.
>> +A RoCEv2 based Virtio-oF target uses UDP port 8549 (CRC-16/ARC of "Virtio") by default, and a iWARP based Virtio-oF target uses TCP port 8549 by default.
>> +
>> +RDMA uses the following operations:
>> +\begin{itemize}
>> +\item The Virtio-oF initiator sends a PDU by RDMA SEND.
>
> What about with “eager buffers”? It probably uses RDMA write for those?
>
Yes, the target side exposes a MR, the initiator side use RDMA WRITE to
access it.
>> +\item The Virtio-oF target reads the remote data by RDMA SEND.
>
> I assume you meant the target reads with RDMA read?
>
Thanks, I'll fix this.
>> +\item The Virtio-oF target writes the remote data by RDMA WRITE.
>> +\item The Virtio-oF target sends a PDU by RDMA SEND.
>> +\end{itemize}
>> --
>> 2.25.1
>>
>>
>> This publicly archived list offers a means to provide input to the
>> OASIS Virtual I/O Device (VIRTIO) TC.
>>
>> In order to verify user consent to the Feedback License terms and
>> to minimize spam in the list archive, subscription is required
>> before posting.
>>
>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>> List help: virtio-comment-help@lists.oasis-open.org
>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>> Committee: https://www.oasis-open.org/committees/virtio/
>> Join OASIS: https://www.oasis-open.org/join/
>>
>>
>
--
zhenwei pi
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 33+ messages in thread
* [virtio-comment] [PATCH v4 6/7] transport-fabrics: add device initialization
2023-06-26 7:25 [virtio-comment] [PATCH v4 0/7] Introduce Virtio Over Fabrics zhenwei pi
` (4 preceding siblings ...)
2023-06-26 7:25 ` [virtio-comment] [PATCH v4 5/7] transport-fabrics: introduce transport binding zhenwei pi
@ 2023-06-26 7:25 ` zhenwei pi
2023-07-29 23:29 ` Raphael Norwitz
2023-06-26 7:25 ` [virtio-comment] [PATCH v4 7/7] transport-fabrics: introduce keyed eager buffers zhenwei pi
2023-07-19 7:07 ` [virtio-comment] PING: [PATCH v4 0/7] Introduce Virtio Over Fabrics zhenwei pi
7 siblings, 1 reply; 33+ messages in thread
From: zhenwei pi @ 2023-06-26 7:25 UTC (permalink / raw)
To: stefanha, parav, mst, jasowang
Cc: virtio-comment, houp, helei.sig11, zhouhuaping.san, xinhao.kong,
zhenwei pi
Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
---
transport-fabrics.tex | 33 +++++++++++++++++++++++++++++++++
1 file changed, 33 insertions(+)
diff --git a/transport-fabrics.tex b/transport-fabrics.tex
index 75e4855..f7f29cf 100644
--- a/transport-fabrics.tex
+++ b/transport-fabrics.tex
@@ -755,3 +755,36 @@ \subsubsection{RDMA}\label{sec:Virtio Transport Options / Virtio Over Fabrics /
\item The Virtio-oF target writes the remote data by RDMA WRITE.
\item The Virtio-oF target sends a PDU by RDMA SEND.
\end{itemize}
+
+
+\subsection{Device Initialization}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Device Initialization}
+\begin{enumerate}
+\item The Virtio-oF control queue must be established first by connecting
+from the Virtio-oF initiator to the Virtio-oF target and issuing a
+\nameref{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Connect Command}
+to create a Virtio-oF device instance.
+\item Optionally, The Virtio-oF initiator issues a
+\nameref{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Get Feature Command}
+to discover the features offered by the Virtio-oF device instance.
+After feature negotiation, the Virtio-oF initiator also issues a
+\nameref{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Set Feature Command}
+to finalize features.
+\item The Virtio-oF initiator issues a
+\nameref{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Get Device ID Command}
+and
+\nameref{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Get Vendor ID Command}
+for virtio driver.
+\item During virtio device initialization, the Virtio-oF initiator issues a
+\nameref{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Get Status Command}
+and
+\nameref{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Set Status Command}.
+\item Optionally, The Virtio-oF initiator issues a
+\nameref{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Get VQ Size Command}
+to get the size of a Virtio-oF virtqueue.
+\item The Virtio-oF initiator issues a
+\nameref{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Connect Command}
+to establish a Virtio-oF virtqueue.
+\item The Virtio-oF initiator creates Virtio-oF virtqueues one by one, then issues a
+\nameref{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / VQ Command}
+to supply virtqueue buffers to the Virtio-oF device instance on Virtio-oF virtqueues.
+\end{enumerate}
--
2.25.1
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply related [flat|nested] 33+ messages in thread* Re: [virtio-comment] [PATCH v4 6/7] transport-fabrics: add device initialization
2023-06-26 7:25 ` [virtio-comment] [PATCH v4 6/7] transport-fabrics: add device initialization zhenwei pi
@ 2023-07-29 23:29 ` Raphael Norwitz
0 siblings, 0 replies; 33+ messages in thread
From: Raphael Norwitz @ 2023-07-29 23:29 UTC (permalink / raw)
To: zhenwei pi
Cc: Stefan Hajnoczi, parav@nvidia.com, Michael S. Tsirkin,
jasowang@redhat.com, virtio-comment@lists.oasis-open.org,
houp@yusur.tech, helei.sig11@bytedance.com,
zhouhuaping.san@bytedance.com, xinhao.kong@duke.edu
> On Jun 26, 2023, at 3:25 AM, zhenwei pi <pizhenwei@bytedance.com> wrote:
>
> Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
> ---
> transport-fabrics.tex | 33 +++++++++++++++++++++++++++++++++
> 1 file changed, 33 insertions(+)
>
> diff --git a/transport-fabrics.tex b/transport-fabrics.tex
> index 75e4855..f7f29cf 100644
> --- a/transport-fabrics.tex
> +++ b/transport-fabrics.tex
> @@ -755,3 +755,36 @@ \subsubsection{RDMA}\label{sec:Virtio Transport Options / Virtio Over Fabrics /
> \item The Virtio-oF target writes the remote data by RDMA WRITE.
> \item The Virtio-oF target sends a PDU by RDMA SEND.
> \end{itemize}
> +
> +
> +\subsection{Device Initialization}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Device Initialization}
> +\begin{enumerate}
> +\item The Virtio-oF control queue must be established first by connecting
> +from the Virtio-oF initiator to the Virtio-oF target and issuing a
> +\nameref{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Connect Command}
> +to create a Virtio-oF device instance.
> +\item Optionally, The Virtio-oF initiator issues a
> +\nameref{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Get Feature Command}
> +to discover the features offered by the Virtio-oF device instance.
> +After feature negotiation, the Virtio-oF initiator also issues a
> +\nameref{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Set Feature Command}
> +to finalize features.
> +\item The Virtio-oF initiator issues a
> +\nameref{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Get Device ID Command}
> +and
> +\nameref{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Get Vendor ID Command}
> +for virtio driver.
> +\item During virtio device initialization, the Virtio-oF initiator issues a
> +\nameref{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Get Status Command}
> +and
> +\nameref{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Set Status Command}.
> +\item Optionally, The Virtio-oF initiator issues a
> +\nameref{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Get VQ Size Command}
> +to get the size of a Virtio-oF virtqueue.
This may be a good place to add a Get/Set request max scatter gather length?
> +\item The Virtio-oF initiator issues a
> +\nameref{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Connect Command}
> +to establish a Virtio-oF virtqueue.
> +\item The Virtio-oF initiator creates Virtio-oF virtqueues one by one, then issues a
> +\nameref{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / VQ Command}
> +to supply virtqueue buffers to the Virtio-oF device instance on Virtio-oF virtqueues.
> +\end{enumerate}
> --
> 2.25.1
>
>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>
>
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 33+ messages in thread
* [virtio-comment] [PATCH v4 7/7] transport-fabrics: introduce keyed eager buffers
2023-06-26 7:25 [virtio-comment] [PATCH v4 0/7] Introduce Virtio Over Fabrics zhenwei pi
` (5 preceding siblings ...)
2023-06-26 7:25 ` [virtio-comment] [PATCH v4 6/7] transport-fabrics: add device initialization zhenwei pi
@ 2023-06-26 7:25 ` zhenwei pi
2023-07-29 23:30 ` Raphael Norwitz
2023-07-19 7:07 ` [virtio-comment] PING: [PATCH v4 0/7] Introduce Virtio Over Fabrics zhenwei pi
7 siblings, 1 reply; 33+ messages in thread
From: zhenwei pi @ 2023-06-26 7:25 UTC (permalink / raw)
To: stefanha, parav, mst, jasowang
Cc: virtio-comment, houp, helei.sig11, zhouhuaping.san, xinhao.kong,
zhenwei pi
The Virtio-oF device instance can expose keyed eager buffers, the Virtio-oF
initiator can write device-readable virtqueue buffers and send VQ command
in a single RTT on the fabric, this makes better performance.
Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
---
transport-fabrics.tex | 81 +++++++++++++++++++++++++++++++++++++++++++
1 file changed, 81 insertions(+)
diff --git a/transport-fabrics.tex b/transport-fabrics.tex
index f7f29cf..b0ccfb1 100644
--- a/transport-fabrics.tex
+++ b/transport-fabrics.tex
@@ -194,6 +194,8 @@ \subsubsection{Opcodes}\label{sec:Virtio Transport Options / Virtio Over Fabrics
#define virtio_of_op_get_feature 0x0004
#define virtio_of_op_set_feature 0x0005
#define virtio_of_op_get_keyed_num_descs 0x0100
+#define virtio_of_op_get_keyed_eager_buf 0x0102
+#define virtio_of_op_vq_keyed_eager_buf 0x0ffe
#define virtio_of_op_vq 0x0fff
#define virtio_of_op_get_vendor_id 0x1000
#define virtio_of_op_get_device_id 0x1001
@@ -349,6 +351,9 @@ \subsubsection{Opcodes}\label{sec:Virtio Transport Options / Virtio Over Fabrics
\begin{lstlisting}
/* support virtio_of_op_get_keyed_num_descs to get the maximum number of keyed descriptors */
#define VIRTIO_OF_F_KEYED_NUM_DESCS 0
+
+/* support virtio_of_op_get_keyed_eager_buf to get the eager buffers of the Virtio-oF virtqueues */
+#define VIRTIO_OF_F_KEYED_EAGER_BUF 1
\end{lstlisting}
\paragraph{Set Feature Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Set Feature Command}
@@ -407,6 +412,82 @@ \subsubsection{Opcodes}\label{sec:Virtio Transport Options / Virtio Over Fabrics
};
\end{lstlisting}
+\paragraph{Get Keyed Eager Buffers Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Get Keyed Eager Buffers Command}
+A Virtio-oF virtqueue may expose 0 or more keyed buffers that allow the Virtio-oF initiator to write directly to the Virtio-oF device instance.
+The Keyed Eager Buffers mechanism reduces round-trip time on the fabrics.
+The Get Keyed Eager Buffers Command is used to get the eager buffers of a Virtio-oF virtqueue,
+and it is executed on the Virtio-oF control queue.
+
+The structure of the Get Keyed Deager Buffers Command structure is as follows:
+\begin{lstlisting}
+struct virtio_of_command_get_keyed_eager_buf {
+ /* opcode is virtio_of_op_get_keyed_eager_buf */
+ le16 opcode;
+ /* unique ID for all in-flight commands */
+ le16 command_id;
+ /* the Virtio-oF virtqueue index */
+ le16 vq_index;
+ /* the maximum number of eager buffer descriptors to obtain */
+ u8 descs;
+ u8 reserved[9];
+};
+\end{lstlisting}
+
+The Get Keyed Eager Buffers Body is a contiguous memory of \texttt{descs} elements (virtio_of_keyed_desc type).
+
+The structure of the Get Keyed Eager Buffers Completion is as follows:
+\begin{lstlisting}
+struct virtio_of_completion_get_keyed_eager_buf {
+ le16 status;
+ le16 command_id;
+ /* the number of eager buffer descriptors */
+ u8 descs;
+ u8 reserved[11];
+};
+\end{lstlisting}
+
+Note: multiple keyed eager buffers compose a logical memory range sequentially.
+To locate a position within the range, use the offset and length values as follows:
+\begin{lstlisting}
+ +-----------+-----------+ +-----------+
+ |addr0, len0|addr1, len1| ... |addrN, lenN|
+ +-----------+-----------+ +-----------+
+ | |
+ \---offset, length---/
+\end{lstlisting}
+
+\paragraph{VQ Keyed Eager Buffers Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / VQ Keyed Eager Buffers Command}
+The Virtio-oF initiator must write device-readable virtqueue buffers into the Virtio-oF virtqueue eager buffers,
+then issue The VQ Keyed Eager Buffers Command to transmit virtqueue buffers.
+This command is executed on the Virtio-oF virtqueue only.
+
+The structure of the VQ Keyed Eager Buffers Command structure is as follows:
+\begin{lstlisting}
+struct virtio_of_command_vq_keyed_eager_buf {
+ /* opcode is virtio_of_op_vq_keyed_eager_buf */
+ le16 opcode;
+ /* unique ID for all in-flight commands */
+ le16 command_id;
+ /* the offset of Virtio-oF virtqueue eager buffers */
+ le32 out_offset;
+ /* the length of device-readable virtqueue buffers */
+ le32 out_length;
+ /* the length of device-writable virtqueue buffers */
+ le32 in_length;
+};
+\end{lstlisting}
+
+The structure of the VQ Keyed Eager Buffers Completion is as follows:
+\begin{lstlisting}
+struct virtio_of_completion_vq_keyed_eager_buf {
+ le16 status;
+ le16 command_id;
+ /* the length of virtqueue buffers from Virtio-oF target */
+ le32 length;
+ u8 reserved[8];
+};
+\end{lstlisting}
+
\paragraph{VQ Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / VQ Command}
The VQ Command is used to transmit virtqueue buffers for Virtio-oF virtqueue only.
--
2.25.1
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply related [flat|nested] 33+ messages in thread* Re: [virtio-comment] [PATCH v4 7/7] transport-fabrics: introduce keyed eager buffers
2023-06-26 7:25 ` [virtio-comment] [PATCH v4 7/7] transport-fabrics: introduce keyed eager buffers zhenwei pi
@ 2023-07-29 23:30 ` Raphael Norwitz
2023-07-31 3:21 ` zhenwei pi
0 siblings, 1 reply; 33+ messages in thread
From: Raphael Norwitz @ 2023-07-29 23:30 UTC (permalink / raw)
To: zhenwei pi
Cc: Stefan Hajnoczi, parav@nvidia.com, Michael S. Tsirkin, Jason Wang,
virtio-comment@lists.oasis-open.org, houp@yusur.tech,
helei.sig11@bytedance.com, zhouhuaping.san@bytedance.com,
xinhao.kong@duke.edu
I do think this is a very valuable addition to the protocol but it introduces complication.
The fundamental drawback is that it’s really expensive to allocate and pre-post enough memory in the target. In theory, a guest could submit <VQ size> <max IO size> IOs on all VQ connections. Assuming a 128 VQ size and 1MB IOs, that’s 128 MB per connection. That may be great for some use cases but doesn’t scale.
With the proposed design I’d worry about memory fragmentation. Maybe it would be better to have eager buffer regions for different IO size buckets per VQ? While slightly less memory efficient it would be much easier to manage.
Again, if this is just for storage device write path, why not try extend iSER or NVMF with this functionality?
> On Jun 26, 2023, at 3:25 AM, zhenwei pi <pizhenwei@bytedance.com> wrote:
>
> The Virtio-oF device instance can expose keyed eager buffers, the Virtio-oF
> initiator can write device-readable virtqueue buffers and send VQ command
> in a single RTT on the fabric, this makes better performance.
>
> Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
> ---
> transport-fabrics.tex | 81 +++++++++++++++++++++++++++++++++++++++++++
> 1 file changed, 81 insertions(+)
>
> diff --git a/transport-fabrics.tex b/transport-fabrics.tex
> index f7f29cf..b0ccfb1 100644
> --- a/transport-fabrics.tex
> +++ b/transport-fabrics.tex
> @@ -194,6 +194,8 @@ \subsubsection{Opcodes}\label{sec:Virtio Transport Options / Virtio Over Fabrics
> #define virtio_of_op_get_feature 0x0004
> #define virtio_of_op_set_feature 0x0005
> #define virtio_of_op_get_keyed_num_descs 0x0100
> +#define virtio_of_op_get_keyed_eager_buf 0x0102
> +#define virtio_of_op_vq_keyed_eager_buf 0x0ffe
> #define virtio_of_op_vq 0x0fff
> #define virtio_of_op_get_vendor_id 0x1000
> #define virtio_of_op_get_device_id 0x1001
> @@ -349,6 +351,9 @@ \subsubsection{Opcodes}\label{sec:Virtio Transport Options / Virtio Over Fabrics
> \begin{lstlisting}
> /* support virtio_of_op_get_keyed_num_descs to get the maximum number of keyed descriptors */
> #define VIRTIO_OF_F_KEYED_NUM_DESCS 0
> +
> +/* support virtio_of_op_get_keyed_eager_buf to get the eager buffers of the Virtio-oF virtqueues */
> +#define VIRTIO_OF_F_KEYED_EAGER_BUF 1
> \end{lstlisting}
>
> \paragraph{Set Feature Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Set Feature Command}
> @@ -407,6 +412,82 @@ \subsubsection{Opcodes}\label{sec:Virtio Transport Options / Virtio Over Fabrics
> };
> \end{lstlisting}
>
> +\paragraph{Get Keyed Eager Buffers Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Get Keyed Eager Buffers Command}
> +A Virtio-oF virtqueue may expose 0 or more keyed buffers that allow the Virtio-oF initiator to write directly to the Virtio-oF device instance.
> +The Keyed Eager Buffers mechanism reduces round-trip time on the fabrics.
> +The Get Keyed Eager Buffers Command is used to get the eager buffers of a Virtio-oF virtqueue,
> +and it is executed on the Virtio-oF control queue.
> +
> +The structure of the Get Keyed Deager Buffers Command structure is as follows:
s/Deager/Eager
> +\begin{lstlisting}
> +struct virtio_of_command_get_keyed_eager_buf {
> + /* opcode is virtio_of_op_get_keyed_eager_buf */
> + le16 opcode;
> + /* unique ID for all in-flight commands */
> + le16 command_id;
> + /* the Virtio-oF virtqueue index */
> + le16 vq_index;
> + /* the maximum number of eager buffer descriptors to obtain */
> + u8 descs;
> + u8 reserved[9];
> +};
> +\end{lstlisting}
> +
> +The Get Keyed Eager Buffers Body is a contiguous memory of \texttt{descs} elements (virtio_of_keyed_desc type).
> +
> +The structure of the Get Keyed Eager Buffers Completion is as follows:
> +\begin{lstlisting}
Assuming you are going to have the initiator RDMA write to this memory, shouldn’t this include the MR key?
> +struct virtio_of_completion_get_keyed_eager_buf {
> + le16 status;
> + le16 command_id;
> + /* the number of eager buffer descriptors */
> + u8 descs;
> + u8 reserved[11];
> +};
> +\end{lstlisting}
> +
> +Note: multiple keyed eager buffers compose a logical memory range sequentially.
> +To locate a position within the range, use the offset and length values as follows:
> +\begin{lstlisting}
> + +-----------+-----------+ +-----------+
> + |addr0, len0|addr1, len1| ... |addrN, lenN|
> + +-----------+-----------+ +-----------+
> + | |
> + \---offset, length---/
> +\end{lstlisting}
> +
> +\paragraph{VQ Keyed Eager Buffers Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / VQ Keyed Eager Buffers Command}
> +The Virtio-oF initiator must write device-readable virtqueue buffers into the Virtio-oF virtqueue eager buffers,
> +then issue The VQ Keyed Eager Buffers Command to transmit virtqueue buffers.
> +This command is executed on the Virtio-oF virtqueue only.
> +
> +The structure of the VQ Keyed Eager Buffers Command structure is as follows:
> +\begin{lstlisting}
> +struct virtio_of_command_vq_keyed_eager_buf {
> + /* opcode is virtio_of_op_vq_keyed_eager_buf */
> + le16 opcode;
> + /* unique ID for all in-flight commands */
> + le16 command_id;
> + /* the offset of Virtio-oF virtqueue eager buffers */
> + le32 out_offset;
> + /* the length of device-readable virtqueue buffers */
> + le32 out_length;
> + /* the length of device-writable virtqueue buffers */
> + le32 in_length;
> +};
> +\end{lstlisting}
> +
> +The structure of the VQ Keyed Eager Buffers Completion is as follows:
> +\begin{lstlisting}
> +struct virtio_of_completion_vq_keyed_eager_buf {
> + le16 status;
> + le16 command_id;
> + /* the length of virtqueue buffers from Virtio-oF target */
> + le32 length;
> + u8 reserved[8];
> +};
> +\end{lstlisting}
> +
> \paragraph{VQ Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / VQ Command}
> The VQ Command is used to transmit virtqueue buffers for Virtio-oF virtqueue only.
>
> --
> 2.25.1
>
>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>
>
^ permalink raw reply [flat|nested] 33+ messages in thread* Re: Re: [virtio-comment] [PATCH v4 7/7] transport-fabrics: introduce keyed eager buffers
2023-07-29 23:30 ` Raphael Norwitz
@ 2023-07-31 3:21 ` zhenwei pi
0 siblings, 0 replies; 33+ messages in thread
From: zhenwei pi @ 2023-07-31 3:21 UTC (permalink / raw)
To: Raphael Norwitz
Cc: Stefan Hajnoczi, parav@nvidia.com, Michael S. Tsirkin, Jason Wang,
virtio-comment@lists.oasis-open.org, houp@yusur.tech,
zhouhuaping.san@bytedance.com, xinhao.kong@duke.edu
On 7/30/23 07:30, Raphael Norwitz wrote:
> I do think this is a very valuable addition to the protocol but it introduces complication.
>
> The fundamental drawback is that it’s really expensive to allocate and pre-post enough memory in the target. In theory, a guest could submit <VQ size> <max IO size> IOs on all VQ connections. Assuming a 128 VQ size and 1MB IOs, that’s 128 MB per connection. That may be great for some use cases but doesn’t scale.
>
> With the proposed design I’d worry about memory fragmentation. Maybe it would be better to have eager buffer regions for different IO size buckets per VQ? While slightly less memory efficient it would be much easier to manage.
>
Typically, 1 eager buffer is used for 1 virtio-of queue. For example, a
target exposes 1 eager buffer of 1M. The initiator side managers 1M as
buddy(maybe list, or other data structures).
The initiator tries to find a suitable region from eager buffer:
1, on success, the initiator writes the eager buffer, and sends command
in 1 RTT.
2, on failure, the initiator prepares a MR (for RDMA READ from target)
and sends command. The the target handles command and IN buffer in 2
RTT. (if no eager buffer supported, always select this path)
> Again, if this is just for storage device write path, why not try extend iSER or NVMF with this functionality?
>
Let's discuss this in overview.
>> On Jun 26, 2023, at 3:25 AM, zhenwei pi <pizhenwei@bytedance.com> wrote:
>>
>> The Virtio-oF device instance can expose keyed eager buffers, the Virtio-oF
>> initiator can write device-readable virtqueue buffers and send VQ command
>> in a single RTT on the fabric, this makes better performance.
>>
>> Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
>> ---
>> transport-fabrics.tex | 81 +++++++++++++++++++++++++++++++++++++++++++
>> 1 file changed, 81 insertions(+)
>>
>> diff --git a/transport-fabrics.tex b/transport-fabrics.tex
>> index f7f29cf..b0ccfb1 100644
>> --- a/transport-fabrics.tex
>> +++ b/transport-fabrics.tex
>> @@ -194,6 +194,8 @@ \subsubsection{Opcodes}\label{sec:Virtio Transport Options / Virtio Over Fabrics
>> #define virtio_of_op_get_feature 0x0004
>> #define virtio_of_op_set_feature 0x0005
>> #define virtio_of_op_get_keyed_num_descs 0x0100
>> +#define virtio_of_op_get_keyed_eager_buf 0x0102
>> +#define virtio_of_op_vq_keyed_eager_buf 0x0ffe
>> #define virtio_of_op_vq 0x0fff
>> #define virtio_of_op_get_vendor_id 0x1000
>> #define virtio_of_op_get_device_id 0x1001
>> @@ -349,6 +351,9 @@ \subsubsection{Opcodes}\label{sec:Virtio Transport Options / Virtio Over Fabrics
>> \begin{lstlisting}
>> /* support virtio_of_op_get_keyed_num_descs to get the maximum number of keyed descriptors */
>> #define VIRTIO_OF_F_KEYED_NUM_DESCS 0
>> +
>> +/* support virtio_of_op_get_keyed_eager_buf to get the eager buffers of the Virtio-oF virtqueues */
>> +#define VIRTIO_OF_F_KEYED_EAGER_BUF 1
>> \end{lstlisting}
>>
>> \paragraph{Set Feature Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Set Feature Command}
>> @@ -407,6 +412,82 @@ \subsubsection{Opcodes}\label{sec:Virtio Transport Options / Virtio Over Fabrics
>> };
>> \end{lstlisting}
>>
>> +\paragraph{Get Keyed Eager Buffers Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / Get Keyed Eager Buffers Command}
>> +A Virtio-oF virtqueue may expose 0 or more keyed buffers that allow the Virtio-oF initiator to write directly to the Virtio-oF device instance.
>> +The Keyed Eager Buffers mechanism reduces round-trip time on the fabrics.
>> +The Get Keyed Eager Buffers Command is used to get the eager buffers of a Virtio-oF virtqueue,
>> +and it is executed on the Virtio-oF control queue.
>> +
>> +The structure of the Get Keyed Deager Buffers Command structure is as follows:
>
> s/Deager/Eager
>
>> +\begin{lstlisting}
>> +struct virtio_of_command_get_keyed_eager_buf {
>> + /* opcode is virtio_of_op_get_keyed_eager_buf */
>> + le16 opcode;
>> + /* unique ID for all in-flight commands */
>> + le16 command_id;
>> + /* the Virtio-oF virtqueue index */
>> + le16 vq_index;
>> + /* the maximum number of eager buffer descriptors to obtain */
>> + u8 descs;
>> + u8 reserved[9];
>> +};
>> +\end{lstlisting}
>> +
>> +The Get Keyed Eager Buffers Body is a contiguous memory of \texttt{descs} elements (virtio_of_keyed_desc type).
>> +
>> +The structure of the Get Keyed Eager Buffers Completion is as follows:
>> +\begin{lstlisting}
>
> Assuming you are going to have the initiator RDMA write to this memory, shouldn’t this include the MR key?
>
>> +struct virtio_of_completion_get_keyed_eager_buf {
>> + le16 status;
>> + le16 command_id;
>> + /* the number of eager buffer descriptors */
>> + u8 descs;
>> + u8 reserved[11];
>> +};
>> +\end{lstlisting}
>> +
>> +Note: multiple keyed eager buffers compose a logical memory range sequentially.
>> +To locate a position within the range, use the offset and length values as follows:
>> +\begin{lstlisting}
>> + +-----------+-----------+ +-----------+
>> + |addr0, len0|addr1, len1| ... |addrN, lenN|
>> + +-----------+-----------+ +-----------+
>> + | |
>> + \---offset, length---/
>> +\end{lstlisting}
>> +
>> +\paragraph{VQ Keyed Eager Buffers Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / VQ Keyed Eager Buffers Command}
>> +The Virtio-oF initiator must write device-readable virtqueue buffers into the Virtio-oF virtqueue eager buffers,
>> +then issue The VQ Keyed Eager Buffers Command to transmit virtqueue buffers.
>> +This command is executed on the Virtio-oF virtqueue only.
>> +
>> +The structure of the VQ Keyed Eager Buffers Command structure is as follows:
>> +\begin{lstlisting}
>> +struct virtio_of_command_vq_keyed_eager_buf {
>> + /* opcode is virtio_of_op_vq_keyed_eager_buf */
>> + le16 opcode;
>> + /* unique ID for all in-flight commands */
>> + le16 command_id;
>> + /* the offset of Virtio-oF virtqueue eager buffers */
>> + le32 out_offset;
>> + /* the length of device-readable virtqueue buffers */
>> + le32 out_length;
>> + /* the length of device-writable virtqueue buffers */
>> + le32 in_length;
>> +};
>> +\end{lstlisting}
>> +
>> +The structure of the VQ Keyed Eager Buffers Completion is as follows:
>> +\begin{lstlisting}
>> +struct virtio_of_completion_vq_keyed_eager_buf {
>> + le16 status;
>> + le16 command_id;
>> + /* the length of virtqueue buffers from Virtio-oF target */
>> + le32 length;
>> + u8 reserved[8];
>> +};
>> +\end{lstlisting}
>> +
>> \paragraph{VQ Command}\label{sec:Virtio Transport Options / Virtio Over Fabrics / Commands Definition / Opcodes / VQ Command}
>> The VQ Command is used to transmit virtqueue buffers for Virtio-oF virtqueue only.
>>
>> --
>> 2.25.1
>>
>>
>> This publicly archived list offers a means to provide input to the
>> OASIS Virtual I/O Device (VIRTIO) TC.
>>
>> In order to verify user consent to the Feedback License terms and
>> to minimize spam in the list archive, subscription is required
>> before posting.
>>
>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>> List help: virtio-comment-help@lists.oasis-open.org
>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>> Committee: https://www.oasis-open.org/committees/virtio/
>> Join OASIS: https://www.oasis-open.org/join/
>>
>>
>
--
zhenwei pi
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 33+ messages in thread
* [virtio-comment] PING: [PATCH v4 0/7] Introduce Virtio Over Fabrics
2023-06-26 7:25 [virtio-comment] [PATCH v4 0/7] Introduce Virtio Over Fabrics zhenwei pi
` (6 preceding siblings ...)
2023-06-26 7:25 ` [virtio-comment] [PATCH v4 7/7] transport-fabrics: introduce keyed eager buffers zhenwei pi
@ 2023-07-19 7:07 ` zhenwei pi
2023-07-19 12:44 ` [virtio-comment] " Parav Pandit
7 siblings, 1 reply; 33+ messages in thread
From: zhenwei pi @ 2023-07-19 7:07 UTC (permalink / raw)
To: stefanha, parav, mst, jasowang
Cc: virtio-comment, houp, helei.sig11, zhouhuaping.san, xinhao.kong
Hi Stefan & Parav
Ping!
On 6/26/23 15:25, zhenwei pi wrote:
> Hi Stefan & Parav
>
> There is minor flaws in v3 series, please skip v3, review the v4
> instead. Thanks!
>
> v3 -> v4:
> [PATCH v3 4/7] transport-fabrics: introduce command set
> - typo fixes: u16 -> le16, u64 -> le64
> - fix unaligned struct virtio_of_completion_get_keyed_num_descs
> - drop duplicated keepalive completion structure
>
> v2 -> v3:
> [PATCH v2 01/11] transport-fabrics: introduce Virtio Over Fabrics
> - Drop referring to PCI.
> - Drop undefined concept 'vring'.
> - Add 'Virtio-oF'(short for 'Virtio Over Fabrics'), and use this instead.
> - Add mapping between target and initiator.
> - Add Virtio-oF queue mapping.
> - Fix grammar flaws.
>
> [PATCH v2 02/11] transport-fabrics: introduce Virtio Qualified Name
> - Use 'NUL' instead of 'NULL'.
> - Add 'including the NUL character' in maximum name.
>
> [PATCH v2 03/11] transport-fabircs: introduce Segment Descriptor
> [PATCH v2 04/11] transport-fabrics: introduce Stream Transmission
> [PATCH v2 05/11] transport-fabrics: introduce Keyed Transmission
> - Merge three patches into one.
> - Rename this section to 'Virtio-oF Protocol Data Unit' instead of 'Transmission Protocol'.
> - Remove undefined 'opcode' reference.
> - Drop common 'struct virtio_of_vring_desc'
> - For stream data transfers, use a single command/completion instead of descriptors.
> - For keyed data transport, use a single descriptor instead of multi descriptors. Also support keyed descriptor count probe.
>
> [PATCH v2 06/11] transport-fabrics: introduce command set
> [PATCH v2 07/11] transport-fabrics: introduce opcodes
> [PATCH v2 08/11] transport-fabrics: introduce status of completion
> - Merge three patches into one.
> - Drop common data structures definition, define data structure for each command.
> - Drop virtio_of_op_get_num_queues, the Virtio-oF device instance would response error status on invalid virtqueue.
> - Allow Virtio-oF virtqueue to use empty connect body.
> - Allow Virtio-oF virtqueue to specify queue size on connecting.
> - Drop errno like status, define detailed status.
> - Other minor fixes.
>
> [PATCH v2 09/11] transport-fabrics: add TCP&RDMA binding
> - Rename TCP to TCP/IP to avoid ambiguity.
> - Add TLS TCP/IP transport.
> - Add RDMA opcode usage.
>
> [PATCH v2 10/11] transport-fabrics: add device initialization
> - Drop PCI reference, describe device initialization instead.
> - Fix grammar flaws.
>
> [PATCH v2 11/11] transport-fabrics: support inline data for keyed
> - Use eager buffer instead.
>
>
> v1 -> v2:
> - Suggested by Parav, split a large patch into several small patches.
> - Small changes for VQN, add "There is no strict style limitation".
> - Move *bytes* field limitation from get/set config opcode section to
> Config Command.
>
>
> v1:
> Introduce Virtio-oF specification, include:
> - overview
> - Virtio Qualified Name
> - Segment Descriptor definition
> - Buffer Mapping definition: Stream Transmission and Keyed Transmission
> - Command set definition
> - opcode definition
> - status definition
> - transport binding: TCP and RDMA
> - device initialization
>
> Previous discussion:
> https://lists.oasis-open.org/archives/virtio-comment/202304/msg00442.html
>
> zhenwei pi (7):
> transport-fabrics: introduce Virtio Over Fabrics overview
> transport-fabrics: introduce Virtio-oF Qualified Name
> transport-fabrics: introduce Virtio-oF Protocol Data Unit
> transport-fabrics: introduce command set
> transport-fabrics: introduce transport binding
> transport-fabrics: add device initialization
> transport-fabrics: introduce keyed eager buffers
>
> content.tex | 1 +
> transport-fabrics.tex | 871 ++++++++++++++++++++++++++++++++++++++++++
> 2 files changed, 872 insertions(+)
> create mode 100644 transport-fabrics.tex
>
--
zhenwei pi
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 33+ messages in thread* [virtio-comment] RE: PING: [PATCH v4 0/7] Introduce Virtio Over Fabrics
2023-07-19 7:07 ` [virtio-comment] PING: [PATCH v4 0/7] Introduce Virtio Over Fabrics zhenwei pi
@ 2023-07-19 12:44 ` Parav Pandit
2023-07-19 12:48 ` [virtio-comment] " Michael S. Tsirkin
0 siblings, 1 reply; 33+ messages in thread
From: Parav Pandit @ 2023-07-19 12:44 UTC (permalink / raw)
To: zhenwei pi, stefanha@redhat.com, mst@redhat.com,
jasowang@redhat.com
Cc: virtio-comment@lists.oasis-open.org, houp@yusur.tech,
helei.sig11@bytedance.com, zhouhuaping.san@bytedance.com,
xinhao.kong@duke.edu
> From: zhenwei pi <pizhenwei@bytedance.com>
> Sent: Wednesday, July 19, 2023 3:07 AM
>
> Hi Stefan & Parav
>
> Ping!
>
This task is not in the top items for 1.4 for virtio, so had slow progress.
I will spend some time this month.
^ permalink raw reply [flat|nested] 33+ messages in thread
* [virtio-comment] Re: PING: [PATCH v4 0/7] Introduce Virtio Over Fabrics
2023-07-19 12:44 ` [virtio-comment] " Parav Pandit
@ 2023-07-19 12:48 ` Michael S. Tsirkin
2023-07-20 2:32 ` [virtio-comment] " zhenwei pi
0 siblings, 1 reply; 33+ messages in thread
From: Michael S. Tsirkin @ 2023-07-19 12:48 UTC (permalink / raw)
To: Parav Pandit
Cc: zhenwei pi, stefanha@redhat.com, jasowang@redhat.com,
virtio-comment@lists.oasis-open.org, houp@yusur.tech,
helei.sig11@bytedance.com, zhouhuaping.san@bytedance.com,
xinhao.kong@duke.edu
On Wed, Jul 19, 2023 at 12:44:52PM +0000, Parav Pandit wrote:
>
>
> > From: zhenwei pi <pizhenwei@bytedance.com>
> > Sent: Wednesday, July 19, 2023 3:07 AM
> >
> > Hi Stefan & Parav
> >
> > Ping!
> >
>
> This task is not in the top items for 1.4 for virtio, so had slow progress.
> I will spend some time this month.
maybe it should be :)
zhenwei if you are going to do something as significant as adding a new
transport you really should join the TC as a member I think.
--
MST
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 33+ messages in thread
* [virtio-comment] Re: Re: PING: [PATCH v4 0/7] Introduce Virtio Over Fabrics
2023-07-19 12:48 ` [virtio-comment] " Michael S. Tsirkin
@ 2023-07-20 2:32 ` zhenwei pi
2023-07-20 2:34 ` [virtio-comment] " Parav Pandit
0 siblings, 1 reply; 33+ messages in thread
From: zhenwei pi @ 2023-07-20 2:32 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: stefanha@redhat.com, Parav Pandit, jasowang@redhat.com,
virtio-comment@lists.oasis-open.org, houp@yusur.tech,
zhouhuaping.san@bytedance.com, xinhao.kong@duke.edu
On 7/19/23 20:48, Michael S. Tsirkin wrote:
> On Wed, Jul 19, 2023 at 12:44:52PM +0000, Parav Pandit wrote:
>>
>>
>>> From: zhenwei pi <pizhenwei@bytedance.com>
>>> Sent: Wednesday, July 19, 2023 3:07 AM
>>>
>>> Hi Stefan & Parav
>>>
>>> Ping!
>>>
>>
>> This task is not in the top items for 1.4 for virtio, so had slow progress.
>> I will spend some time this month.
>
> maybe it should be :)
> zhenwei if you are going to do something as significant as adding a new
> transport you really should join the TC as a member I think.
>
Hi Michael
I'm excited to join the TC as a member, I have already subscribed the
virtio-comment mailing list, should I do anything else?
--
zhenwei pi
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 33+ messages in thread
* [virtio-comment] RE: Re: PING: [PATCH v4 0/7] Introduce Virtio Over Fabrics
2023-07-20 2:32 ` [virtio-comment] " zhenwei pi
@ 2023-07-20 2:34 ` Parav Pandit
0 siblings, 0 replies; 33+ messages in thread
From: Parav Pandit @ 2023-07-20 2:34 UTC (permalink / raw)
To: zhenwei pi, Michael S. Tsirkin
Cc: stefanha@redhat.com, jasowang@redhat.com,
virtio-comment@lists.oasis-open.org, houp@yusur.tech,
zhouhuaping.san@bytedance.com, xinhao.kong@duke.edu
> From: zhenwei pi <pizhenwei@bytedance.com>
> Sent: Wednesday, July 19, 2023 10:32 PM
>
> On 7/19/23 20:48, Michael S. Tsirkin wrote:
> > On Wed, Jul 19, 2023 at 12:44:52PM +0000, Parav Pandit wrote:
> >>
> >>
> >>> From: zhenwei pi <pizhenwei@bytedance.com>
> >>> Sent: Wednesday, July 19, 2023 3:07 AM
> >>>
> >>> Hi Stefan & Parav
> >>>
> >>> Ping!
> >>>
> >>
> >> This task is not in the top items for 1.4 for virtio, so had slow progress.
> >> I will spend some time this month.
> >
> > maybe it should be :)
> > zhenwei if you are going to do something as significant as adding a
> > new transport you really should join the TC as a member I think.
> >
>
> Hi Michael
>
> I'm excited to join the TC as a member, I have already subscribed the virtio-
> comment mailing list, should I do anything else?
Michael will have more details to explain,
In mean time, please explore following steps to join OASIS followed by virtio-tc at [1].
[1] https://www.oasis-open.org/join-a-tc/
^ permalink raw reply [flat|nested] 33+ messages in thread