[virtio-dev] [RFC] virtio-net: support access and control the member devices

[virtio-dev] [RFC] virtio-net: support access and control the member devices

[Xuan Zhuo]

Now we have the concept of the device group. This allows us to
support sr-iov and siov more conveniently.

The following description uses pf, vf as an example. The scene with sf
should be similar.

According to my understanding, the management at the level of virtqueue
should be handled by admin queue. And the net related management is
still managed by cvq of pf. So this patch works with the cvq of the pf.

The problem I am trying to solve is when there are multiple vf
virtio-net devices under one pf.
It is necessary for administrators to query the status and information
of each vf and configure mac.

Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
---
 device-types/net/description.tex        | 72 +++++++++++++++++++++++--
 device-types/net/device-conformance.tex |  1 +
 2 files changed, 70 insertions(+), 3 deletions(-)

diff --git a/device-types/net/description.tex b/device-types/net/description.tex
index 76585b0..1494a18 100644
--- a/device-types/net/description.tex
+++ b/device-types/net/description.tex
@@ -88,6 +88,9 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits
 \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control
     channel.
 
+\item[VIRTIO_NET_F_ACCESS_MEMBER_DEVICE(51)] Device can access and control the
+    member devices.
+
 \item[VIRTIO_NET_F_HASH_TUNNEL(51)] Device supports inner header hash for encapsulated packets.
 
 \item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports virtqueue notification coalescing.
@@ -1156,6 +1159,7 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
         u8 command;
         u8 command-specific-data[];
         u8 ack;
+        u8 command-specific-data-reply[];
 };
 
 /* ack values */
@@ -1164,9 +1168,11 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
 \end{lstlisting}
 
 The \field{class}, \field{command} and command-specific-data are set by the
-driver, and the device sets the \field{ack} byte. There is little it can
-do except issue a diagnostic if \field{ack} is not
-VIRTIO_NET_OK.
+driver, and the device sets the \field{ack} byte and optionally
+\field{command-specific-data-reply}. There is little the driver can
+do except issue a diagnostic if \field{ack} is not VIRTIO_NET_OK.
+
+The command VIRTIO_NET_CTRL_MEMBER_GET_INFO contains \field{command-specific-data-reply}.
 
 \paragraph{Packet Receive Filtering}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Packet Receive Filtering}
 \label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Setting Promiscuous Mode}%old label for latexdiff
@@ -1805,6 +1811,66 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
 
 Upon reset, a device MUST initialize all coalescing parameters to 0.
 
+\paragraph{Member Device}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Member Info}
+
+If a deivce is the own deivce inside one device group and the feature
+VIRTIO_NET_F_ACCESS_MEMBER_DEVICE is negotiated, the own driver can obtain
+the info and change the configuration of the member devices of this device
+group.
+
+\begin{lstlisting}
+struct virtio_net_ctrl_member {
+        le64 group_member_id;
+};
+
+struct virtio_net_ctrl_member_info_reply {
+        u8 mac[6];
+        le16 status;
+        le16 max_virtqueue_pairs;
+        le16 mtu;
+        le32 speed;
+        u8 duplex;
+        u8 driver_ok;
+        le16 current_virtqueue_pairs;
+};
+
+struct virtio_net_ctrl_member_set_mac {
+        le64 group_member_id;
+        u8 mac[6];
+};
+
+#define VIRTIO_NET_CTRL_MEMBER 7
+ #define VIRTIO_NET_CTRL_MEMBER_GET_INFO  0
+ #define VIRTIO_NET_CTRL_MEMBER_SET_MAC   1
+\end{lstlisting}
+
+\devicenormative{\subparagraph}{Member Device}{Device Types / Network Device / Device Operation / Control Virtqueue / Member Device}
+
+The device only offers the feature VIRTIO_NET_F_ACCESS_MEMBER_DEVICE when it is
+the own device.
+
+If the \field{group_member_id} is not within the range of the member device id,
+the own device MUST respond with VIRTIO_NET_ERR.
+
+struct virtio_net_ctrl_member_info_reply is used to reply the command
+VIRTIO_NET_CTRL_MEMBER_GET_INFO via \field{command-specific-data-reply}.
+
+If the member device specified by the \field{group_member_id} is in the
+DRIVER_OK status, the own device MUST set the \field{driver_ok} to 1 and set the
+\field{current_virtqueue_pairs} to the number of virtqueue pairs that are being
+used by the member device. Otherwise, both \field{driver_ok} and
+\field{current_virtqueue_pairs} MUST be set to zero.
+
+All other fields of the struct virtio_net_ctrl_member_info_reply MUST be equal
+to the values in the struct virtio_net_config of the member device.
+
+If the member device specified by the \field{group_member_id} supports
+VIRTIO_NET_F_MAC and its status is not DRIVER_OK, the own device MUST reply to
+VIRTIO_NET_CTRL_MEMBER_SET_MAC with VIRTIO_NET_OK, and the member device MUST
+offer the \field{mac} to the member driver via the struct virtio_net_config.
+Otherwise, the own device MUST reply to the command
+VIRTIO_NET_CTRL_MEMBER_SET_MAC with VIRTIO_NET_ERR.
+
 \subsubsection{Legacy Interface: Framing Requirements}\label{sec:Device
 Types / Network Device / Legacy Interface: Framing Requirements}
 
diff --git a/device-types/net/device-conformance.tex b/device-types/net/device-conformance.tex
index f88f48b..9af3293 100644
--- a/device-types/net/device-conformance.tex
+++ b/device-types/net/device-conformance.tex
@@ -15,4 +15,5 @@
 \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Receive-side scaling (RSS) / RSS processing}
 \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
 \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Inner Header Hash}
+\item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Member Device}
 \end{itemize}
-- 
2.32.0.3.g01195cf9f


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [RFC] virtio-net: support access and control the member devices

[Michael S. Tsirkin]

On Thu, Aug 03, 2023 at 04:31:50PM +0800, Xuan Zhuo wrote:
> Now we have the concept of the device group. This allows us to
> support sr-iov and siov more conveniently.
> 
> The following description uses pf, vf as an example. The scene with sf
> should be similar.
> 
> According to my understanding, the management at the level of virtqueue
> should be handled by admin queue. And the net related management is
> still managed by cvq of pf. So this patch works with the cvq of the pf.
> 
> The problem I am trying to solve is when there are multiple vf
> virtio-net devices under one pf.
> It is necessary for administrators to query the status and information
> of each vf and configure mac.
> 
> Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>


I think we should use admin queue for this, and not cvq.

> ---
>  device-types/net/description.tex        | 72 +++++++++++++++++++++++--
>  device-types/net/device-conformance.tex |  1 +
>  2 files changed, 70 insertions(+), 3 deletions(-)
> 
> diff --git a/device-types/net/description.tex b/device-types/net/description.tex
> index 76585b0..1494a18 100644
> --- a/device-types/net/description.tex
> +++ b/device-types/net/description.tex
> @@ -88,6 +88,9 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits
>  \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control
>      channel.
>  
> +\item[VIRTIO_NET_F_ACCESS_MEMBER_DEVICE(51)] Device can access and control the
> +    member devices.
> +
>  \item[VIRTIO_NET_F_HASH_TUNNEL(51)] Device supports inner header hash for encapsulated packets.
>  
>  \item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports virtqueue notification coalescing.
> @@ -1156,6 +1159,7 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
>          u8 command;
>          u8 command-specific-data[];
>          u8 ack;
> +        u8 command-specific-data-reply[];
>  };
>  
>  /* ack values */
> @@ -1164,9 +1168,11 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
>  \end{lstlisting}
>  
>  The \field{class}, \field{command} and command-specific-data are set by the
> -driver, and the device sets the \field{ack} byte. There is little it can
> -do except issue a diagnostic if \field{ack} is not
> -VIRTIO_NET_OK.
> +driver, and the device sets the \field{ack} byte and optionally
> +\field{command-specific-data-reply}. There is little the driver can
> +do except issue a diagnostic if \field{ack} is not VIRTIO_NET_OK.
> +
> +The command VIRTIO_NET_CTRL_MEMBER_GET_INFO contains \field{command-specific-data-reply}.
>  
>  \paragraph{Packet Receive Filtering}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Packet Receive Filtering}
>  \label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Setting Promiscuous Mode}%old label for latexdiff
> @@ -1805,6 +1811,66 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
>  
>  Upon reset, a device MUST initialize all coalescing parameters to 0.
>  
> +\paragraph{Member Device}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Member Info}
> +
> +If a deivce is the own deivce inside one device group and the feature
> +VIRTIO_NET_F_ACCESS_MEMBER_DEVICE is negotiated, the own driver can obtain
> +the info and change the configuration of the member devices of this device
> +group.
> +
> +\begin{lstlisting}
> +struct virtio_net_ctrl_member {
> +        le64 group_member_id;
> +};
> +
> +struct virtio_net_ctrl_member_info_reply {
> +        u8 mac[6];
> +        le16 status;
> +        le16 max_virtqueue_pairs;
> +        le16 mtu;
> +        le32 speed;
> +        u8 duplex;
> +        u8 driver_ok;
> +        le16 current_virtqueue_pairs;
> +};
> +
> +struct virtio_net_ctrl_member_set_mac {
> +        le64 group_member_id;
> +        u8 mac[6];
> +};
> +
> +#define VIRTIO_NET_CTRL_MEMBER 7
> + #define VIRTIO_NET_CTRL_MEMBER_GET_INFO  0
> + #define VIRTIO_NET_CTRL_MEMBER_SET_MAC   1
> +\end{lstlisting}
> +
> +\devicenormative{\subparagraph}{Member Device}{Device Types / Network Device / Device Operation / Control Virtqueue / Member Device}
> +
> +The device only offers the feature VIRTIO_NET_F_ACCESS_MEMBER_DEVICE when it is
> +the own device.
> +
> +If the \field{group_member_id} is not within the range of the member device id,
> +the own device MUST respond with VIRTIO_NET_ERR.
> +
> +struct virtio_net_ctrl_member_info_reply is used to reply the command
> +VIRTIO_NET_CTRL_MEMBER_GET_INFO via \field{command-specific-data-reply}.
> +
> +If the member device specified by the \field{group_member_id} is in the
> +DRIVER_OK status, the own device MUST set the \field{driver_ok} to 1 and set the
> +\field{current_virtqueue_pairs} to the number of virtqueue pairs that are being
> +used by the member device. Otherwise, both \field{driver_ok} and
> +\field{current_virtqueue_pairs} MUST be set to zero.
> +
> +All other fields of the struct virtio_net_ctrl_member_info_reply MUST be equal
> +to the values in the struct virtio_net_config of the member device.
> +
> +If the member device specified by the \field{group_member_id} supports
> +VIRTIO_NET_F_MAC and its status is not DRIVER_OK, the own device MUST reply to
> +VIRTIO_NET_CTRL_MEMBER_SET_MAC with VIRTIO_NET_OK, and the member device MUST
> +offer the \field{mac} to the member driver via the struct virtio_net_config.
> +Otherwise, the own device MUST reply to the command
> +VIRTIO_NET_CTRL_MEMBER_SET_MAC with VIRTIO_NET_ERR.
> +
>  \subsubsection{Legacy Interface: Framing Requirements}\label{sec:Device
>  Types / Network Device / Legacy Interface: Framing Requirements}
>  
> diff --git a/device-types/net/device-conformance.tex b/device-types/net/device-conformance.tex
> index f88f48b..9af3293 100644
> --- a/device-types/net/device-conformance.tex
> +++ b/device-types/net/device-conformance.tex
> @@ -15,4 +15,5 @@
>  \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Receive-side scaling (RSS) / RSS processing}
>  \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
>  \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Inner Header Hash}
> +\item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Member Device}
>  \end{itemize}
> -- 
> 2.32.0.3.g01195cf9f


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] RE: [RFC] virtio-net: support access and control the member devices

[Parav Pandit]

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Thursday, August 3, 2023 3:14 PM
> 
> On Thu, Aug 03, 2023 at 04:31:50PM +0800, Xuan Zhuo wrote:
> > Now we have the concept of the device group. This allows us to support
> > sr-iov and siov more conveniently.
> >
> > The following description uses pf, vf as an example. The scene with sf
> > should be similar.
> >
> > According to my understanding, the management at the level of
> > virtqueue should be handled by admin queue. And the net related
> > management is still managed by cvq of pf. So this patch works with the cvq of
> the pf.
> >
> > The problem I am trying to solve is when there are multiple vf
> > virtio-net devices under one pf.
> > It is necessary for administrators to query the status and information
> > of each vf and configure mac.
> >
> > Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> 
> 
> I think we should use admin queue for this, and not cvq.
> 
+1


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] RE: [RFC] virtio-net: support access and control the member devices

[Parav Pandit]

> From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> Sent: Thursday, August 3, 2023 2:02 PM
> To: virtio-dev@lists.oasis-open.org
> Cc: jasowang@redhat.com; Michael S. Tsirkin <mst@redhat.com>; Parav Pandit
> <parav@nvidia.com>; virtio-comment@lists.oasis-open.org
> Subject: [RFC] virtio-net: support access and control the member devices
> 
> Now we have the concept of the device group. This allows us to support sr-iov
> and siov more conveniently.
> 
> The following description uses pf, vf as an example. The scene with sf should be
> similar.
> 
> According to my understanding, the management at the level of virtqueue
> should be handled by admin queue. And the net related management is still
> managed by cvq of pf. So this patch works with the cvq of the pf.
>
Admin command is preferred
 
> The problem I am trying to solve is when there are multiple vf virtio-net devices
> under one pf.
> It is necessary for administrators to query the status and information of each vf
> and configure mac.
This patch enables to configure also, please add it in the commit log.

> 
> Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> ---
>  device-types/net/description.tex        | 72 +++++++++++++++++++++++--
>  device-types/net/device-conformance.tex |  1 +
>  2 files changed, 70 insertions(+), 3 deletions(-)
> 
> diff --git a/device-types/net/description.tex b/device-types/net/description.tex
> index 76585b0..1494a18 100644
> --- a/device-types/net/description.tex
> +++ b/device-types/net/description.tex
> @@ -88,6 +88,9 @@ \subsection{Feature bits}\label{sec:Device Types /
> Network Device / Feature bits  \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set
> MAC address through control
>      channel.
> 
> +\item[VIRTIO_NET_F_ACCESS_MEMBER_DEVICE(51)] Device can access and
> control the
> +    member devices.
> +
Once admin command is used, patch will be simpler without new feature bit.

>  \item[VIRTIO_NET_F_HASH_TUNNEL(51)] Device supports inner header hash
> for encapsulated packets.
> 
>  \item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports virtqueue
> notification coalescing.
> @@ -1156,6 +1159,7 @@ \subsubsection{Control Virtqueue}\label{sec:Device
> Types / Network Device / Devi
>          u8 command;
>          u8 command-specific-data[];
>          u8 ack;
> +        u8 command-specific-data-reply[];
>  };
> 
>  /* ack values */
> @@ -1164,9 +1168,11 @@ \subsubsection{Control Virtqueue}\label{sec:Device
> Types / Network Device / Devi  \end{lstlisting}
> 
>  The \field{class}, \field{command} and command-specific-data are set by the -
> driver, and the device sets the \field{ack} byte. There is little it can -do except
> issue a diagnostic if \field{ack} is not -VIRTIO_NET_OK.
> +driver, and the device sets the \field{ack} byte and optionally
> +\field{command-specific-data-reply}. There is little the driver can do
> +except issue a diagnostic if \field{ack} is not VIRTIO_NET_OK.
> +
> +The command VIRTIO_NET_CTRL_MEMBER_GET_INFO contains
> \field{command-specific-data-reply}.
> 
>  \paragraph{Packet Receive Filtering}\label{sec:Device Types / Network Device
> / Device Operation / Control Virtqueue / Packet Receive Filtering}
> \label{sec:Device Types / Network Device / Device Operation / Control
> Virtqueue / Setting Promiscuous Mode}%old label for latexdiff @@ -1805,6
> +1811,66 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types /
> Network Device / Devi
> 
>  Upon reset, a device MUST initialize all coalescing parameters to 0.
> 
> +\paragraph{Member Device}\label{sec:Device Types / Network Device /
> +Device Operation / Control Virtqueue / Member Info}
> +
> +If a deivce is the own deivce inside one device group and the feature
> +VIRTIO_NET_F_ACCESS_MEMBER_DEVICE is negotiated, the own driver can
> +obtain the info and change the configuration of the member devices of
> +this device group.
> +
> +\begin{lstlisting}
> +struct virtio_net_ctrl_member {
> +        le64 group_member_id;
> +};
> +
> +struct virtio_net_ctrl_member_info_reply {
We need a bitmap to indicate which fields are valid.
We did a nice work of it in vdpa tool to describe both, the fields and bitmap.

Also, when you change to admin command, please change suffix to from _reply to _result to align to its terminology.

> +        u8 mac[6];
> +        le16 status;
> +        le16 max_virtqueue_pairs;
> +        le16 mtu;
> +        le32 speed;
> +        u8 duplex;

We should above split constant config space fields and below run time fields meant for debug purpose.
We are working on the member device migration proposal and below fields belong to "device context" should be available through such separate command.
But if you do it before, please cover the whole device state and we will be able to reuse for multiple use-cases.

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [RFC] virtio-net: support access and control the member devices

[Xuan Zhuo]

On Thu, 3 Aug 2023 05:44:23 -0400, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> On Thu, Aug 03, 2023 at 04:31:50PM +0800, Xuan Zhuo wrote:
> > Now we have the concept of the device group. This allows us to
> > support sr-iov and siov more conveniently.
> >
> > The following description uses pf, vf as an example. The scene with sf
> > should be similar.
> >
> > According to my understanding, the management at the level of virtqueue
> > should be handled by admin queue. And the net related management is
> > still managed by cvq of pf. So this patch works with the cvq of the pf.
> >
> > The problem I am trying to solve is when there are multiple vf
> > virtio-net devices under one pf.
> > It is necessary for administrators to query the status and information
> > of each vf and configure mac.
> >
> > Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
>
>
> I think we should use admin queue for this, and not cvq.


To me the cvq of pf is ok, because the we configure for the net.

Do you mean we introduce a new admin queue command
VIRTIO_ADMIN_CMD_VIRTIO_NET_GET_INFO?

Then we will have too much commands for net, block.....

Thanks.

>
> > ---
> >  device-types/net/description.tex        | 72 +++++++++++++++++++++++--
> >  device-types/net/device-conformance.tex |  1 +
> >  2 files changed, 70 insertions(+), 3 deletions(-)
> >
> > diff --git a/device-types/net/description.tex b/device-types/net/description.tex
> > index 76585b0..1494a18 100644
> > --- a/device-types/net/description.tex
> > +++ b/device-types/net/description.tex
> > @@ -88,6 +88,9 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits
> >  \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control
> >      channel.
> >
> > +\item[VIRTIO_NET_F_ACCESS_MEMBER_DEVICE(51)] Device can access and control the
> > +    member devices.
> > +
> >  \item[VIRTIO_NET_F_HASH_TUNNEL(51)] Device supports inner header hash for encapsulated packets.
> >
> >  \item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports virtqueue notification coalescing.
> > @@ -1156,6 +1159,7 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> >          u8 command;
> >          u8 command-specific-data[];
> >          u8 ack;
> > +        u8 command-specific-data-reply[];
> >  };
> >
> >  /* ack values */
> > @@ -1164,9 +1168,11 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> >  \end{lstlisting}
> >
> >  The \field{class}, \field{command} and command-specific-data are set by the
> > -driver, and the device sets the \field{ack} byte. There is little it can
> > -do except issue a diagnostic if \field{ack} is not
> > -VIRTIO_NET_OK.
> > +driver, and the device sets the \field{ack} byte and optionally
> > +\field{command-specific-data-reply}. There is little the driver can
> > +do except issue a diagnostic if \field{ack} is not VIRTIO_NET_OK.
> > +
> > +The command VIRTIO_NET_CTRL_MEMBER_GET_INFO contains \field{command-specific-data-reply}.
> >
> >  \paragraph{Packet Receive Filtering}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Packet Receive Filtering}
> >  \label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Setting Promiscuous Mode}%old label for latexdiff
> > @@ -1805,6 +1811,66 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> >
> >  Upon reset, a device MUST initialize all coalescing parameters to 0.
> >
> > +\paragraph{Member Device}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Member Info}
> > +
> > +If a deivce is the own deivce inside one device group and the feature
> > +VIRTIO_NET_F_ACCESS_MEMBER_DEVICE is negotiated, the own driver can obtain
> > +the info and change the configuration of the member devices of this device
> > +group.
> > +
> > +\begin{lstlisting}
> > +struct virtio_net_ctrl_member {
> > +        le64 group_member_id;
> > +};
> > +
> > +struct virtio_net_ctrl_member_info_reply {
> > +        u8 mac[6];
> > +        le16 status;
> > +        le16 max_virtqueue_pairs;
> > +        le16 mtu;
> > +        le32 speed;
> > +        u8 duplex;
> > +        u8 driver_ok;
> > +        le16 current_virtqueue_pairs;
> > +};
> > +
> > +struct virtio_net_ctrl_member_set_mac {
> > +        le64 group_member_id;
> > +        u8 mac[6];
> > +};
> > +
> > +#define VIRTIO_NET_CTRL_MEMBER 7
> > + #define VIRTIO_NET_CTRL_MEMBER_GET_INFO  0
> > + #define VIRTIO_NET_CTRL_MEMBER_SET_MAC   1
> > +\end{lstlisting}
> > +
> > +\devicenormative{\subparagraph}{Member Device}{Device Types / Network Device / Device Operation / Control Virtqueue / Member Device}
> > +
> > +The device only offers the feature VIRTIO_NET_F_ACCESS_MEMBER_DEVICE when it is
> > +the own device.
> > +
> > +If the \field{group_member_id} is not within the range of the member device id,
> > +the own device MUST respond with VIRTIO_NET_ERR.
> > +
> > +struct virtio_net_ctrl_member_info_reply is used to reply the command
> > +VIRTIO_NET_CTRL_MEMBER_GET_INFO via \field{command-specific-data-reply}.
> > +
> > +If the member device specified by the \field{group_member_id} is in the
> > +DRIVER_OK status, the own device MUST set the \field{driver_ok} to 1 and set the
> > +\field{current_virtqueue_pairs} to the number of virtqueue pairs that are being
> > +used by the member device. Otherwise, both \field{driver_ok} and
> > +\field{current_virtqueue_pairs} MUST be set to zero.
> > +
> > +All other fields of the struct virtio_net_ctrl_member_info_reply MUST be equal
> > +to the values in the struct virtio_net_config of the member device.
> > +
> > +If the member device specified by the \field{group_member_id} supports
> > +VIRTIO_NET_F_MAC and its status is not DRIVER_OK, the own device MUST reply to
> > +VIRTIO_NET_CTRL_MEMBER_SET_MAC with VIRTIO_NET_OK, and the member device MUST
> > +offer the \field{mac} to the member driver via the struct virtio_net_config.
> > +Otherwise, the own device MUST reply to the command
> > +VIRTIO_NET_CTRL_MEMBER_SET_MAC with VIRTIO_NET_ERR.
> > +
> >  \subsubsection{Legacy Interface: Framing Requirements}\label{sec:Device
> >  Types / Network Device / Legacy Interface: Framing Requirements}
> >
> > diff --git a/device-types/net/device-conformance.tex b/device-types/net/device-conformance.tex
> > index f88f48b..9af3293 100644
> > --- a/device-types/net/device-conformance.tex
> > +++ b/device-types/net/device-conformance.tex
> > @@ -15,4 +15,5 @@
> >  \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Receive-side scaling (RSS) / RSS processing}
> >  \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
> >  \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Inner Header Hash}
> > +\item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Member Device}
> >  \end{itemize}
> > --
> > 2.32.0.3.g01195cf9f
>

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] RE: [RFC] virtio-net: support access and control the member devices

[Parav Pandit]

> From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> Sent: Thursday, August 3, 2023 4:22 PM
> 
> Do you mean we introduce a new admin queue command
> VIRTIO_ADMIN_CMD_VIRTIO_NET_GET_INFO?
> 
> Then we will have too much commands for net, block.....

VIRTIO_ADMIN_CMD_DEV_CTX_GET

One command that gives device context information in TLV (type length value) format.
One of field of the device context is device config space.
Device will return the information depending what device it is, net/blk/console each returns different info.

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: RE: [RFC] virtio-net: support access and control the member devices

[Xuan Zhuo]

On Thu, 3 Aug 2023 10:59:53 +0000, Parav Pandit <parav@nvidia.com> wrote:
>
> > From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > Sent: Thursday, August 3, 2023 4:22 PM
> >
> > Do you mean we introduce a new admin queue command
> > VIRTIO_ADMIN_CMD_VIRTIO_NET_GET_INFO?
> >
> > Then we will have too much commands for net, block.....
>
> VIRTIO_ADMIN_CMD_DEV_CTX_GET
>
> One command that gives device context information in TLV (type length value) format.
> One of field of the device context is device config space.
> Device will return the information depending what device it is, net/blk/console each returns different info.


Ok, I got.

That is ok for me about the device context information.

But how about the configuration of the mac? By admin queue?

Thanks.

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: RE: [RFC] virtio-net: support access and control the member devices

[Xuan Zhuo]

On Thu, 3 Aug 2023 10:23:32 +0000, Parav Pandit <parav@nvidia.com> wrote:
>
> > From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > Sent: Thursday, August 3, 2023 2:02 PM
> > To: virtio-dev@lists.oasis-open.org
> > Cc: jasowang@redhat.com; Michael S. Tsirkin <mst@redhat.com>; Parav Pandit
> > <parav@nvidia.com>; virtio-comment@lists.oasis-open.org
> > Subject: [RFC] virtio-net: support access and control the member devices
> >
> > Now we have the concept of the device group. This allows us to support sr-iov
> > and siov more conveniently.
> >
> > The following description uses pf, vf as an example. The scene with sf should be
> > similar.
> >
> > According to my understanding, the management at the level of virtqueue
> > should be handled by admin queue. And the net related management is still
> > managed by cvq of pf. So this patch works with the cvq of the pf.
> >
> Admin command is preferred
>
> > The problem I am trying to solve is when there are multiple vf virtio-net devices
> > under one pf.
> > It is necessary for administrators to query the status and information of each vf
> > and configure mac.
> This patch enables to configure also, please add it in the commit log.
>
> >
> > Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > ---
> >  device-types/net/description.tex        | 72 +++++++++++++++++++++++--
> >  device-types/net/device-conformance.tex |  1 +
> >  2 files changed, 70 insertions(+), 3 deletions(-)
> >
> > diff --git a/device-types/net/description.tex b/device-types/net/description.tex
> > index 76585b0..1494a18 100644
> > --- a/device-types/net/description.tex
> > +++ b/device-types/net/description.tex
> > @@ -88,6 +88,9 @@ \subsection{Feature bits}\label{sec:Device Types /
> > Network Device / Feature bits  \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set
> > MAC address through control
> >      channel.
> >
> > +\item[VIRTIO_NET_F_ACCESS_MEMBER_DEVICE(51)] Device can access and
> > control the
> > +    member devices.
> > +
> Once admin command is used, patch will be simpler without new feature bit.
>
> >  \item[VIRTIO_NET_F_HASH_TUNNEL(51)] Device supports inner header hash
> > for encapsulated packets.
> >
> >  \item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports virtqueue
> > notification coalescing.
> > @@ -1156,6 +1159,7 @@ \subsubsection{Control Virtqueue}\label{sec:Device
> > Types / Network Device / Devi
> >          u8 command;
> >          u8 command-specific-data[];
> >          u8 ack;
> > +        u8 command-specific-data-reply[];
> >  };
> >
> >  /* ack values */
> > @@ -1164,9 +1168,11 @@ \subsubsection{Control Virtqueue}\label{sec:Device
> > Types / Network Device / Devi  \end{lstlisting}
> >
> >  The \field{class}, \field{command} and command-specific-data are set by the -
> > driver, and the device sets the \field{ack} byte. There is little it can -do except
> > issue a diagnostic if \field{ack} is not -VIRTIO_NET_OK.
> > +driver, and the device sets the \field{ack} byte and optionally
> > +\field{command-specific-data-reply}. There is little the driver can do
> > +except issue a diagnostic if \field{ack} is not VIRTIO_NET_OK.
> > +
> > +The command VIRTIO_NET_CTRL_MEMBER_GET_INFO contains
> > \field{command-specific-data-reply}.
> >
> >  \paragraph{Packet Receive Filtering}\label{sec:Device Types / Network Device
> > / Device Operation / Control Virtqueue / Packet Receive Filtering}
> > \label{sec:Device Types / Network Device / Device Operation / Control
> > Virtqueue / Setting Promiscuous Mode}%old label for latexdiff @@ -1805,6
> > +1811,66 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types /
> > Network Device / Devi
> >
> >  Upon reset, a device MUST initialize all coalescing parameters to 0.
> >
> > +\paragraph{Member Device}\label{sec:Device Types / Network Device /
> > +Device Operation / Control Virtqueue / Member Info}
> > +
> > +If a deivce is the own deivce inside one device group and the feature
> > +VIRTIO_NET_F_ACCESS_MEMBER_DEVICE is negotiated, the own driver can
> > +obtain the info and change the configuration of the member devices of
> > +this device group.
> > +
> > +\begin{lstlisting}
> > +struct virtio_net_ctrl_member {
> > +        le64 group_member_id;
> > +};
> > +
> > +struct virtio_net_ctrl_member_info_reply {
> We need a bitmap to indicate which fields are valid.
> We did a nice work of it in vdpa tool to describe both, the fields and bitmap.
>
> Also, when you change to admin command, please change suffix to from _reply to _result to align to its terminology.


OK.


>
> > +        u8 mac[6];
> > +        le16 status;
> > +        le16 max_virtqueue_pairs;
> > +        le16 mtu;
> > +        le32 speed;
> > +        u8 duplex;
>
> We should above split constant config space fields and below run time fields meant for debug purpose.
> We are working on the member device migration proposal and below fields belong to "device context" should be available through such separate command.
> But if you do it before, please cover the whole device state and we will be able to reuse for multiple use-cases.


I am not sure I got fully.

I will try.

Thanks.

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] RE: RE: [RFC] virtio-net: support access and control the member devices

[Parav Pandit]

> From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> Sent: Thursday, August 3, 2023 4:36 PM


> 
> OK.
> 
> 
> >
> > > +        u8 mac[6];
> > > +        le16 status;
> > > +        le16 max_virtqueue_pairs;
> > > +        le16 mtu;
> > > +        le32 speed;
> > > +        u8 duplex;
> >
> > We should above split constant config space fields and below run time fields
> meant for debug purpose.
> > We are working on the member device migration proposal and below fields
> belong to "device context" should be available through such separate command.
> > But if you do it before, please cover the whole device state and we will be
> able to reuse for multiple use-cases.
> 
> 
> I am not sure I got fully.
> 
> I will try.

Something like below.

struct virtio_field {
	le32 type;	/* contains the value of enum virtio_dev_field_type */ 
	le32 size;	/* size of the data field in bytes */
	u8 data[]; 
};

enum virtio_dev_field_type {
        VIRTIO_DEV_PCI_COMMON_CFG, 	/*  struct virtio_dev_common_cfg */ 
        VIRTIO_DEV_CFG_SPACE,	/* config space fields, for net  struct virtio_net_config etc */ 
        VIRTIO_DEV_QUEUE_CFG,	/* struct virtio_dev_q_cfg */ 
};

In your use case, it is VIRTIO_DEV_CFG_SPACE, just blob of data bytes.

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: RE: RE: [RFC] virtio-net: support access and control the member devices

[Xuan Zhuo]

On Thu, 3 Aug 2023 11:13:28 +0000, Parav Pandit <parav@nvidia.com> wrote:
>
> > From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > Sent: Thursday, August 3, 2023 4:36 PM
>
>
> >
> > OK.
> >
> >
> > >
> > > > +        u8 mac[6];
> > > > +        le16 status;
> > > > +        le16 max_virtqueue_pairs;
> > > > +        le16 mtu;
> > > > +        le32 speed;
> > > > +        u8 duplex;
> > >
> > > We should above split constant config space fields and below run time fields
> > meant for debug purpose.
> > > We are working on the member device migration proposal and below fields
> > belong to "device context" should be available through such separate command.
> > > But if you do it before, please cover the whole device state and we will be
> > able to reuse for multiple use-cases.
> >
> >
> > I am not sure I got fully.
> >
> > I will try.
>
> Something like below.
>
> struct virtio_field {
> 	le32 type;	/* contains the value of enum virtio_dev_field_type */
> 	le32 size;	/* size of the data field in bytes */
> 	u8 data[];
> };
>
> enum virtio_dev_field_type {
>         VIRTIO_DEV_PCI_COMMON_CFG, 	/*  struct virtio_dev_common_cfg */
>         VIRTIO_DEV_CFG_SPACE,	/* config space fields, for net  struct virtio_net_config etc */
>         VIRTIO_DEV_QUEUE_CFG,	/* struct virtio_dev_q_cfg */
> };
>
> In your use case, it is VIRTIO_DEV_CFG_SPACE, just blob of data bytes.


That is ok for me.

My question is what is ctx? How do we decide what to put in ctx? How do we
define ctx?

Thanks.



---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] RE: RE: [RFC] virtio-net: support access and control the member devices

[Parav Pandit]

> From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> Sent: Thursday, August 3, 2023 4:34 PM

> > One command that gives device context information in TLV (type length
> value) format.
> > One of field of the device context is device config space.
> > Device will return the information depending what device it is,
> net/blk/console each returns different info.
> 
> 
> Ok, I got.
> 
> That is ok for me about the device context information.
> 
> But how about the configuration of the mac? By admin queue?
> 
In cloud env, we find them that increasingly users want to avoid such config from the compute side (PF).
But some users may find it useful from the PF admin command as well.
Dual thoughts.


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: RE: RE: [RFC] virtio-net: support access and control the member devices

[Xuan Zhuo]

On Thu, 3 Aug 2023 11:15:34 +0000, Parav Pandit <parav@nvidia.com> wrote:
>
> > From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > Sent: Thursday, August 3, 2023 4:34 PM
>
> > > One command that gives device context information in TLV (type length
> > value) format.
> > > One of field of the device context is device config space.
> > > Device will return the information depending what device it is,
> > net/blk/console each returns different info.
> >
> >
> > Ok, I got.
> >
> > That is ok for me about the device context information.
> >
> > But how about the configuration of the mac? By admin queue?
> >
> In cloud env, we find them that increasingly users want to avoid such config from the compute side (PF).
> But some users may find it useful from the PF admin command as well.
> Dual thoughts.

In some of our cases this was required.
I think this should be done by cvq.
But getting the ctx of the device is high priority, we can start from here.

Thanks.

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] RE: RE: RE: [RFC] virtio-net: support access and control the member devices

[Parav Pandit]

> From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> Sent: Thursday, August 3, 2023 4:46 PM
> 
> On Thu, 3 Aug 2023 11:13:28 +0000, Parav Pandit <parav@nvidia.com> wrote:
> >
> > > From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > > Sent: Thursday, August 3, 2023 4:36 PM
> >
> >
> > >
> > > OK.
> > >
> > >
> > > >
> > > > > +        u8 mac[6];
> > > > > +        le16 status;
> > > > > +        le16 max_virtqueue_pairs;
> > > > > +        le16 mtu;
> > > > > +        le32 speed;
> > > > > +        u8 duplex;
> > > >
> > > > We should above split constant config space fields and below run
> > > > time fields
> > > meant for debug purpose.
> > > > We are working on the member device migration proposal and below
> > > > fields
> > > belong to "device context" should be available through such separate
> command.
> > > > But if you do it before, please cover the whole device state and
> > > > we will be
> > > able to reuse for multiple use-cases.
> > >
> > >
> > > I am not sure I got fully.
> > >
> > > I will try.
> >
> > Something like below.
> >
> > struct virtio_field {
> > 	le32 type;	/* contains the value of enum virtio_dev_field_type */
> > 	le32 size;	/* size of the data field in bytes */
> > 	u8 data[];
> > };
> >
> > enum virtio_dev_field_type {
> >         VIRTIO_DEV_PCI_COMMON_CFG, 	/*  struct
> virtio_dev_common_cfg */
> >         VIRTIO_DEV_CFG_SPACE,	/* config space fields, for net  struct
> virtio_net_config etc */
> >         VIRTIO_DEV_QUEUE_CFG,	/* struct virtio_dev_q_cfg */
> > };
> >
> > In your use case, it is VIRTIO_DEV_CFG_SPACE, just blob of data bytes.
> 
> 
> That is ok for me.
> 
> My question is what is ctx? How do we decide what to put in ctx? How do we
> define ctx?

In the live migration use case flow, ctx is collection of all the device information.
I only listed 3 above for simplicity of our discussion.
In specific config space query, like your example, 
The owner driver asks for specific piece of the context to query by supplying the bitmap for above enum.

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] RE: RE: RE: [RFC] virtio-net: support access and control the member devices

[Parav Pandit]

> From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> Sent: Thursday, August 3, 2023 4:48 PM
> 
> In some of our cases this was required.
> I think this should be done by cvq.

Admin command is what is suitable for device management, including features and configspace fields setup.
You can see in your patch that the whole group management infra needs to be built on cvq.

In future, there can be a group management device which may not be the parent PF.
When virtio matures there, same admin command will work on such group management device.
(for example for PF).

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: RE: RE: RE: [RFC] virtio-net: support access and control the member devices

[Xuan Zhuo]

On Thu, 3 Aug 2023 11:21:08 +0000, Parav Pandit <parav@nvidia.com> wrote:
>
> > From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > Sent: Thursday, August 3, 2023 4:46 PM
> >
> > On Thu, 3 Aug 2023 11:13:28 +0000, Parav Pandit <parav@nvidia.com> wrote:
> > >
> > > > From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > > > Sent: Thursday, August 3, 2023 4:36 PM
> > >
> > >
> > > >
> > > > OK.
> > > >
> > > >
> > > > >
> > > > > > +        u8 mac[6];
> > > > > > +        le16 status;
> > > > > > +        le16 max_virtqueue_pairs;
> > > > > > +        le16 mtu;
> > > > > > +        le32 speed;
> > > > > > +        u8 duplex;
> > > > >
> > > > > We should above split constant config space fields and below run
> > > > > time fields
> > > > meant for debug purpose.
> > > > > We are working on the member device migration proposal and below
> > > > > fields
> > > > belong to "device context" should be available through such separate
> > command.
> > > > > But if you do it before, please cover the whole device state and
> > > > > we will be
> > > > able to reuse for multiple use-cases.
> > > >
> > > >
> > > > I am not sure I got fully.
> > > >
> > > > I will try.
> > >
> > > Something like below.
> > >
> > > struct virtio_field {
> > > 	le32 type;	/* contains the value of enum virtio_dev_field_type */
> > > 	le32 size;	/* size of the data field in bytes */
> > > 	u8 data[];
> > > };
> > >
> > > enum virtio_dev_field_type {
> > >         VIRTIO_DEV_PCI_COMMON_CFG, 	/*  struct
> > virtio_dev_common_cfg */
> > >         VIRTIO_DEV_CFG_SPACE,	/* config space fields, for net  struct
> > virtio_net_config etc */
> > >         VIRTIO_DEV_QUEUE_CFG,	/* struct virtio_dev_q_cfg */
> > > };
> > >
> > > In your use case, it is VIRTIO_DEV_CFG_SPACE, just blob of data bytes.
> >
> >
> > That is ok for me.
> >
> > My question is what is ctx? How do we decide what to put in ctx? How do we
> > define ctx?
>
> In the live migration use case flow, ctx is collection of all the device information.
> I only listed 3 above for simplicity of our discussion.
> In specific config space query, like your example,
> The owner driver asks for specific piece of the context to query by supplying the bitmap for above enum.


OK. I see.

Let we start from this.

Thanks.


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: RE: RE: RE: [RFC] virtio-net: support access and control the member devices

[Xuan Zhuo]

On Thu, 3 Aug 2023 11:25:31 +0000, Parav Pandit <parav@nvidia.com> wrote:
>
> > From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > Sent: Thursday, August 3, 2023 4:48 PM
> >
> > In some of our cases this was required.
> > I think this should be done by cvq.
>
> Admin command is what is suitable for device management, including features and configspace fields setup.

I agree.

But some configuration such as mac, ip(eswitch rule) for vfs that are suitable for
the admin queue?

Thanks.


> You can see in your patch that the whole group management infra needs to be built on cvq.
>
> In future, there can be a group management device which may not be the parent PF.
> When virtio matures there, same admin command will work on such group management device.
> (for example for PF).

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] RE: RE: RE: RE: [RFC] virtio-net: support access and control the member devices

[Parav Pandit]

> From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> Sent: Thursday, August 3, 2023 5:00 PM


> > Admin command is what is suitable for device management, including
> features and configspace fields setup.
> 
> I agree.
> 
> But some configuration such as mac, ip(eswitch rule) for vfs that are suitable
> for the admin queue?
Mac, why not.

Eswith rules should be programmed through the flow vq, that we are working with Heng, Satananda and David.
So, a PF or a switching device (new virtio device type) will have the flowvq as well.


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] RE: RE: RE: RE: [RFC] virtio-net: support access and control the member devices

[Xuan Zhuo]

On Thu, 3 Aug 2023 11:39:20 +0000, Parav Pandit <parav@nvidia.com> wrote:
>
> > From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > Sent: Thursday, August 3, 2023 5:00 PM
>
>
> > > Admin command is what is suitable for device management, including
> > features and configspace fields setup.
> >
> > I agree.
> >
> > But some configuration such as mac, ip(eswitch rule) for vfs that are suitable
> > for the admin queue?
> Mac, why not.

Mac is the concept of the virtio-net.
How do abstract it? Or a admin queue command VIRTIO_ADMIN_CMD_NET_MAC_SET?

>
> Eswith rules should be programmed through the flow vq, that we are working with Heng, Satananda and David.

I know you are doing for the n-tuple receive flow filters.

Here I mean we want to limit the ip of the VF. In the cloud case, the vm is
untrust, we should force the vm that it just can use the specified ip.

Then can we configure the ip of the vfs via the admin queue?

Thanks.



> So, a PF or a switching device (new virtio device type) will have the flowvq as well.
>
>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] RE: [virtio-comment] RE: RE: RE: RE: [RFC] virtio-net: support access and control the member devices

[Parav Pandit]

> From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> Sent: Thursday, August 3, 2023 5:11 PM
> 
> On Thu, 3 Aug 2023 11:39:20 +0000, Parav Pandit <parav@nvidia.com> wrote:
> >
> > > From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > > Sent: Thursday, August 3, 2023 5:00 PM
> >
> >
> > > > Admin command is what is suitable for device management, including
> > > features and configspace fields setup.
> > >
> > > I agree.
> > >
> > > But some configuration such as mac, ip(eswitch rule) for vfs that
> > > are suitable for the admin queue?
> > Mac, why not.
> 
> Mac is the concept of the virtio-net.
> How do abstract it? Or a admin queue command
> VIRTIO_ADMIN_CMD_NET_MAC_SET?
> 
I don't think we should abstract it.
A one command that can set one or multiple fields of the config space is fine.

> >
> > Eswith rules should be programmed through the flow vq, that we are working
> with Heng, Satananda and David.
> 
> I know you are doing for the n-tuple receive flow filters.
> 
> Here I mean we want to limit the ip of the VF. In the cloud case, the vm is
> untrust, we should force the vm that it just can use the specified ip.
>
That is fine, 
 It is just that the flow filter commands may be different for ingress and egress direction of the VF.
The command that we are adding does not have concept of switch ports, because there is no switch.

For eswitch case, you would define more commands that will have switch ports...

> Then can we configure the ip of the vfs via the admin queue?

It should be through the flow filter queue so that it can be performant like nic tx/rxq which has dedicated role.
For switch use case command formats or fields needs extension.


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] RE: [virtio-comment] RE: RE: RE: RE: [RFC] virtio-net: support access and control the member devices

[Satananda Burla]

> -----Original Message-----
> From: virtio-dev@lists.oasis-open.org <virtio-dev@lists.oasis-open.org>
> On Behalf Of Parav Pandit
> Sent: Thursday, August 3, 2023 6:16 AM
> To: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> Cc: virtio-dev@lists.oasis-open.org; jasowang@redhat.com; virtio-
> comment@lists.oasis-open.org; Michael S. Tsirkin <mst@redhat.com>
> Subject: [EXT] [virtio-dev] RE: [virtio-comment] RE: RE: RE: RE: [RFC]
> virtio-net: support access and control the member devices
> 
> External Email
> 
> ----------------------------------------------------------------------
> 
> 
> > From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > Sent: Thursday, August 3, 2023 5:11 PM
> >
> > On Thu, 3 Aug 2023 11:39:20 +0000, Parav Pandit <parav@nvidia.com>
> wrote:
> > >
> > > > From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > > > Sent: Thursday, August 3, 2023 5:00 PM
> > >
> > >
> > > > > Admin command is what is suitable for device management,
> including
> > > > features and configspace fields setup.
> > > >
> > > > I agree.
> > > >
> > > > But some configuration such as mac, ip(eswitch rule) for vfs that
> > > > are suitable for the admin queue?
> > > Mac, why not.
> >
> > Mac is the concept of the virtio-net.
> > How do abstract it? Or a admin queue command
> > VIRTIO_ADMIN_CMD_NET_MAC_SET?
> >
> I don't think we should abstract it.
> A one command that can set one or multiple fields of the config space is
> fine.
> 
> > >
> > > Eswith rules should be programmed through the flow vq, that we are
> working
> > with Heng, Satananda and David.
> >
> > I know you are doing for the n-tuple receive flow filters.
> >
> > Here I mean we want to limit the ip of the VF. In the cloud case, the
> vm is
> > untrust, we should force the vm that it just can use the specified ip.
> >
> That is fine,
>  It is just that the flow filter commands may be different for ingress
> and egress direction of the VF.
> The command that we are adding does not have concept of switch ports,
> because there is no switch.
> 
> For eswitch case, you would define more commands that will have switch
> ports...
> 
> > Then can we configure the ip of the vfs via the admin queue?
> 
> It should be through the flow filter queue so that it can be performant
> like nic tx/rxq which has dedicated role.
> For switch use case command formats or fields needs extension.
IMO, kernel already prescribes using representors for mac/vlan  programming
for SRIOV/SIOV eswitch devices, member devices may have to support representor
interfaces.
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] RE: RE: RE: RE: [RFC] virtio-net: support access and control the member devices

[Jason Wang]

On Fri, Aug 4, 2023 at 1:03 AM Satananda Burla <sburla@marvell.com> wrote:
>
>
>
> > -----Original Message-----
> > From: virtio-dev@lists.oasis-open.org <virtio-dev@lists.oasis-open.org>
> > On Behalf Of Parav Pandit
> > Sent: Thursday, August 3, 2023 6:16 AM
> > To: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > Cc: virtio-dev@lists.oasis-open.org; jasowang@redhat.com; virtio-
> > comment@lists.oasis-open.org; Michael S. Tsirkin <mst@redhat.com>
> > Subject: [EXT] [virtio-dev] RE: [virtio-comment] RE: RE: RE: RE: [RFC]
> > virtio-net: support access and control the member devices
> >
> > External Email
> >
> > ----------------------------------------------------------------------
> >
> >
> > > From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > > Sent: Thursday, August 3, 2023 5:11 PM
> > >
> > > On Thu, 3 Aug 2023 11:39:20 +0000, Parav Pandit <parav@nvidia.com>
> > wrote:
> > > >
> > > > > From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > > > > Sent: Thursday, August 3, 2023 5:00 PM
> > > >
> > > >
> > > > > > Admin command is what is suitable for device management,
> > including
> > > > > features and configspace fields setup.
> > > > >
> > > > > I agree.
> > > > >
> > > > > But some configuration such as mac, ip(eswitch rule) for vfs that
> > > > > are suitable for the admin queue?
> > > > Mac, why not.
> > >
> > > Mac is the concept of the virtio-net.
> > > How do abstract it? Or a admin queue command
> > > VIRTIO_ADMIN_CMD_NET_MAC_SET?
> > >
> > I don't think we should abstract it.
> > A one command that can set one or multiple fields of the config space is
> > fine.
> >
> > > >
> > > > Eswith rules should be programmed through the flow vq, that we are
> > working
> > > with Heng, Satananda and David.
> > >
> > > I know you are doing for the n-tuple receive flow filters.
> > >
> > > Here I mean we want to limit the ip of the VF. In the cloud case, the
> > vm is
> > > untrust, we should force the vm that it just can use the specified ip.
> > >
> > That is fine,
> >  It is just that the flow filter commands may be different for ingress
> > and egress direction of the VF.
> > The command that we are adding does not have concept of switch ports,
> > because there is no switch.
> >
> > For eswitch case, you would define more commands that will have switch
> > ports...
> >
> > > Then can we configure the ip of the vfs via the admin queue?
> >
> > It should be through the flow filter queue so that it can be performant
> > like nic tx/rxq which has dedicated role.
> > For switch use case command formats or fields needs extension.
> IMO, kernel already prescribes using representors for mac/vlan  programming
> for SRIOV/SIOV eswitch devices, member devices may have to support representor
> interfaces.

I second this.

Thanks

> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> > For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
>
>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [RFC] virtio-net: support access and control the member devices

[Jason Wang]

On Thu, Aug 3, 2023 at 5:44 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Thu, Aug 03, 2023 at 04:31:50PM +0800, Xuan Zhuo wrote:
> > Now we have the concept of the device group. This allows us to
> > support sr-iov and siov more conveniently.
> >
> > The following description uses pf, vf as an example. The scene with sf
> > should be similar.
> >
> > According to my understanding, the management at the level of virtqueue
> > should be handled by admin queue. And the net related management is
> > still managed by cvq of pf. So this patch works with the cvq of the pf.
> >
> > The problem I am trying to solve is when there are multiple vf
> > virtio-net devices under one pf.
> > It is necessary for administrators to query the status and information
> > of each vf and configure mac.
> >
> > Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
>
>
> I think we should use admin queue for this, and not cvq.

I think we probably need some split here.

E.g the fields belonging to configuration space might go with a more
general interface like admin command/queue.

The net specific ones need to go through a net specific interface.
(cvq of PF or reps)

Thanks


>
> > ---
> >  device-types/net/description.tex        | 72 +++++++++++++++++++++++--
> >  device-types/net/device-conformance.tex |  1 +
> >  2 files changed, 70 insertions(+), 3 deletions(-)
> >
> > diff --git a/device-types/net/description.tex b/device-types/net/description.tex
> > index 76585b0..1494a18 100644
> > --- a/device-types/net/description.tex
> > +++ b/device-types/net/description.tex
> > @@ -88,6 +88,9 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits
> >  \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control
> >      channel.
> >
> > +\item[VIRTIO_NET_F_ACCESS_MEMBER_DEVICE(51)] Device can access and control the
> > +    member devices.
> > +
> >  \item[VIRTIO_NET_F_HASH_TUNNEL(51)] Device supports inner header hash for encapsulated packets.
> >
> >  \item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports virtqueue notification coalescing.
> > @@ -1156,6 +1159,7 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> >          u8 command;
> >          u8 command-specific-data[];
> >          u8 ack;
> > +        u8 command-specific-data-reply[];
> >  };
> >
> >  /* ack values */
> > @@ -1164,9 +1168,11 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> >  \end{lstlisting}
> >
> >  The \field{class}, \field{command} and command-specific-data are set by the
> > -driver, and the device sets the \field{ack} byte. There is little it can
> > -do except issue a diagnostic if \field{ack} is not
> > -VIRTIO_NET_OK.
> > +driver, and the device sets the \field{ack} byte and optionally
> > +\field{command-specific-data-reply}. There is little the driver can
> > +do except issue a diagnostic if \field{ack} is not VIRTIO_NET_OK.
> > +
> > +The command VIRTIO_NET_CTRL_MEMBER_GET_INFO contains \field{command-specific-data-reply}.
> >
> >  \paragraph{Packet Receive Filtering}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Packet Receive Filtering}
> >  \label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Setting Promiscuous Mode}%old label for latexdiff
> > @@ -1805,6 +1811,66 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> >
> >  Upon reset, a device MUST initialize all coalescing parameters to 0.
> >
> > +\paragraph{Member Device}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Member Info}
> > +
> > +If a deivce is the own deivce inside one device group and the feature
> > +VIRTIO_NET_F_ACCESS_MEMBER_DEVICE is negotiated, the own driver can obtain
> > +the info and change the configuration of the member devices of this device
> > +group.
> > +
> > +\begin{lstlisting}
> > +struct virtio_net_ctrl_member {
> > +        le64 group_member_id;
> > +};
> > +
> > +struct virtio_net_ctrl_member_info_reply {
> > +        u8 mac[6];
> > +        le16 status;
> > +        le16 max_virtqueue_pairs;
> > +        le16 mtu;
> > +        le32 speed;
> > +        u8 duplex;
> > +        u8 driver_ok;
> > +        le16 current_virtqueue_pairs;
> > +};
> > +
> > +struct virtio_net_ctrl_member_set_mac {
> > +        le64 group_member_id;
> > +        u8 mac[6];
> > +};
> > +
> > +#define VIRTIO_NET_CTRL_MEMBER 7
> > + #define VIRTIO_NET_CTRL_MEMBER_GET_INFO  0
> > + #define VIRTIO_NET_CTRL_MEMBER_SET_MAC   1
> > +\end{lstlisting}
> > +
> > +\devicenormative{\subparagraph}{Member Device}{Device Types / Network Device / Device Operation / Control Virtqueue / Member Device}
> > +
> > +The device only offers the feature VIRTIO_NET_F_ACCESS_MEMBER_DEVICE when it is
> > +the own device.
> > +
> > +If the \field{group_member_id} is not within the range of the member device id,
> > +the own device MUST respond with VIRTIO_NET_ERR.
> > +
> > +struct virtio_net_ctrl_member_info_reply is used to reply the command
> > +VIRTIO_NET_CTRL_MEMBER_GET_INFO via \field{command-specific-data-reply}.
> > +
> > +If the member device specified by the \field{group_member_id} is in the
> > +DRIVER_OK status, the own device MUST set the \field{driver_ok} to 1 and set the
> > +\field{current_virtqueue_pairs} to the number of virtqueue pairs that are being
> > +used by the member device. Otherwise, both \field{driver_ok} and
> > +\field{current_virtqueue_pairs} MUST be set to zero.
> > +
> > +All other fields of the struct virtio_net_ctrl_member_info_reply MUST be equal
> > +to the values in the struct virtio_net_config of the member device.
> > +
> > +If the member device specified by the \field{group_member_id} supports
> > +VIRTIO_NET_F_MAC and its status is not DRIVER_OK, the own device MUST reply to
> > +VIRTIO_NET_CTRL_MEMBER_SET_MAC with VIRTIO_NET_OK, and the member device MUST
> > +offer the \field{mac} to the member driver via the struct virtio_net_config.
> > +Otherwise, the own device MUST reply to the command
> > +VIRTIO_NET_CTRL_MEMBER_SET_MAC with VIRTIO_NET_ERR.
> > +
> >  \subsubsection{Legacy Interface: Framing Requirements}\label{sec:Device
> >  Types / Network Device / Legacy Interface: Framing Requirements}
> >
> > diff --git a/device-types/net/device-conformance.tex b/device-types/net/device-conformance.tex
> > index f88f48b..9af3293 100644
> > --- a/device-types/net/device-conformance.tex
> > +++ b/device-types/net/device-conformance.tex
> > @@ -15,4 +15,5 @@
> >  \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Receive-side scaling (RSS) / RSS processing}
> >  \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
> >  \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Inner Header Hash}
> > +\item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Member Device}
> >  \end{itemize}
> > --
> > 2.32.0.3.g01195cf9f
>


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] RE: [RFC] virtio-net: support access and control the member devices

[Parav Pandit]

> From: Jason Wang <jasowang@redhat.com>
> Sent: Friday, August 4, 2023 9:02 AM

> I think we probably need some split here.
> 
> E.g the fields belonging to configuration space might go with a more general
> interface like admin command/queue.
> 
> The net specific ones need to go through a net specific interface.
> (cvq of PF or reps)

I don’t see the need of such split.
Why do you think that?

[virtio-dev] RE: [virtio-comment] RE: RE: RE: RE: [RFC] virtio-net: support access and control the member devices

[Parav Pandit]

> From: Satananda Burla <sburla@marvell.com>
> Sent: Thursday, August 3, 2023 10:33 PM


> IMO, kernel already prescribes using representors for mac/vlan  programming
> for SRIOV/SIOV eswitch devices, member devices may have to support
> representor interfaces.

Member device's representor switch port is on the owner (switch) device.
And flow filter entries are programmed on this owner (switch) device's flow vq.
OS level representor is also on this owner (switch) device.

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: RE: [virtio-comment] RE: RE: RE: RE: [RFC] virtio-net: support access and control the member devices

[Xuan Zhuo]

On Thu, 3 Aug 2023 13:15:59 +0000, Parav Pandit <parav@nvidia.com> wrote:
>
>
> > From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > Sent: Thursday, August 3, 2023 5:11 PM
> >
> > On Thu, 3 Aug 2023 11:39:20 +0000, Parav Pandit <parav@nvidia.com> wrote:
> > >
> > > > From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > > > Sent: Thursday, August 3, 2023 5:00 PM
> > >
> > >
> > > > > Admin command is what is suitable for device management, including
> > > > features and configspace fields setup.
> > > >
> > > > I agree.
> > > >
> > > > But some configuration such as mac, ip(eswitch rule) for vfs that
> > > > are suitable for the admin queue?
> > > Mac, why not.
> >
> > Mac is the concept of the virtio-net.
> > How do abstract it? Or a admin queue command
> > VIRTIO_ADMIN_CMD_NET_MAC_SET?
> >
> I don't think we should abstract it.
> A one command that can set one or multiple fields of the config space is fine.


So, you mean we will have many commands like:

VIRTIO_ADMIN_CMD_NET_MAC_SET
VIRTIO_ADMIN_CMD_NET_MTU_SET
VIRTIO_ADMIN_CMD_NET_SPEED_SET
VIRTIO_ADMIN_CMD_BLK_XXX1
VIRTIO_ADMIN_CMD_BLK_XXX2
.....

Or a generic command with the offset and size of the field of the config space?


>
> > >
> > > Eswith rules should be programmed through the flow vq, that we are working
> > with Heng, Satananda and David.
> >
> > I know you are doing for the n-tuple receive flow filters.
> >
> > Here I mean we want to limit the ip of the VF. In the cloud case, the vm is
> > untrust, we should force the vm that it just can use the specified ip.
> >
> That is fine,
>  It is just that the flow filter commands may be different for ingress and egress direction of the VF.
> The command that we are adding does not have concept of switch ports, because there is no switch.
>
> For eswitch case, you would define more commands that will have switch ports...

Yes, for this requirement, I will introduce the concept of simple switch into virtio-net.
Then we can configure the ip of each port of the switch.

>
> > Then can we configure the ip of the vfs via the admin queue?
>
> It should be through the flow filter queue so that it can be performant like nic tx/rxq which has dedicated role.

Why flow filter queue? I think the cvq is ok.

In our case, the ip of the port is configured by the admin on the setup of the
device.

And that is configured to the switch. I think the flow filter is working inside
the deivce.

Thanks.


> For switch use case command formats or fields needs extension.
>


















---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] RE: RE: [virtio-comment] RE: RE: RE: RE: [RFC] virtio-net: support access and control the member devices

[Parav Pandit]

> From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> Sent: Friday, August 4, 2023 11:49 AM

> So, you mean we will have many commands like:
> 
> VIRTIO_ADMIN_CMD_NET_MAC_SET
> VIRTIO_ADMIN_CMD_NET_MTU_SET
> VIRTIO_ADMIN_CMD_NET_SPEED_SET
> VIRTIO_ADMIN_CMD_BLK_XXX1
> VIRTIO_ADMIN_CMD_BLK_XXX2
> .....
> 
> Or a generic command with the offset and size of the field of the config space?
>
Mix of both. One command per field is too much.
Going forward we may not have single boxed config space as many of the config is coming through the cvq logically separate per functionality.

Hence, command should be one per device type, that can set one or more attributes of the device.
Attribute may be residing in the config space or otherwise.

VIRTIO_ADMIN_CMD_NET_SET
Input: bitmap (bit per field of config space or otherwise)
Input: field matches to bitmap

VIRTIO_ADMIN_CMD_BLK_SET

> > It should be through the flow filter queue so that it can be performant like nic
> tx/rxq which has dedicated role.
> 
> Why flow filter queue? I think the cvq is ok.
>
For any sensible use of the switch, beyond few filter commands needs a high perf queue.
 
> In our case, the ip of the port is configured by the admin on the setup of the
> device.
> 
> And that is configured to the switch. I think the flow filter is working inside the
> deivce.
> 
I lost you on your last two points in terminology of admin, device etc.

We have
1. owner device
2. member device
3. Switch is typically on a owner device, having switch port for each member device.

a. Owner device programs switch rules for ports.
b. Owner device also may program flow filter rules of its own for RQ for RSS etc.
c. Member device programs follow filter rules of rss and more.

Flow filters are generic framework applies to all 3 use cases.

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: RE: RE: [virtio-comment] RE: RE: RE: RE: [RFC] virtio-net: support access and control the member devices

[Xuan Zhuo]

On Fri, 4 Aug 2023 06:37:37 +0000, Parav Pandit <parav@nvidia.com> wrote:
>
>
> > From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > Sent: Friday, August 4, 2023 11:49 AM
>
> > So, you mean we will have many commands like:
> >
> > VIRTIO_ADMIN_CMD_NET_MAC_SET
> > VIRTIO_ADMIN_CMD_NET_MTU_SET
> > VIRTIO_ADMIN_CMD_NET_SPEED_SET
> > VIRTIO_ADMIN_CMD_BLK_XXX1
> > VIRTIO_ADMIN_CMD_BLK_XXX2
> > .....
> >
> > Or a generic command with the offset and size of the field of the config space?
> >
> Mix of both. One command per field is too much.
> Going forward we may not have single boxed config space as many of the config is coming through the cvq logically separate per functionality.
>
> Hence, command should be one per device type, that can set one or more attributes of the device.
> Attribute may be residing in the config space or otherwise.
>
> VIRTIO_ADMIN_CMD_NET_SET
> Input: bitmap (bit per field of config space or otherwise)
> Input: field matches to bitmap
>
> VIRTIO_ADMIN_CMD_BLK_SET


Great! Sounds good.

>
> > > It should be through the flow filter queue so that it can be performant like nic
> > tx/rxq which has dedicated role.
> >
> > Why flow filter queue? I think the cvq is ok.
> >
> For any sensible use of the switch, beyond few filter commands needs a high perf queue.
>
> > In our case, the ip of the port is configured by the admin on the setup of the
> > device.
> >
> > And that is configured to the switch. I think the flow filter is working inside the
> > deivce.
> >
> I lost you on your last two points in terminology of admin, device etc.
>
> We have
> 1. owner device
> 2. member device
> 3. Switch is typically on a owner device, having switch port for each member device.
>
> a. Owner device programs switch rules for ports.
> b. Owner device also may program flow filter rules of its own for RQ for RSS etc.
> c. Member device programs follow filter rules of rss and more.
>
> Flow filters are generic framework applies to all 3 use cases.

I know the gap. For us, the switch is under all devices(including the owner device).
So we need to control the switch separately.

In your structure, the owner device works as the switch. It has port to connect
to the vf devices.


Thanks.



















---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] RE: RE: RE: [virtio-comment] RE: RE: RE: RE: [RFC] virtio-net: support access and control the member devices

[Parav Pandit]

> From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> Sent: Friday, August 4, 2023 12:24 PM

> I know the gap. For us, the switch is under all devices(including the owner
> device).
> So we need to control the switch separately.
>
Right. Same for other devices too, including ours.
Hence, it is also located outside of the owner device (on a different system).

If the switch is not a virtio device, does it really need to be exposed as virtio object?

> In your structure, the owner device works as the switch. It has port to connect
> to the vf devices.
> 
Right. I just put owner device as switch for simplicity of the discussion.

At this point to have port for owner device requires creating a dedicated switching object, to be located sometimes side by side inside the owner, sometimes outside.
All of these cases to be crafted, please rethink if this is _really_ needed as virtio object or not.

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: RE: RE: RE: [virtio-comment] RE: RE: RE: RE: [RFC] virtio-net: support access and control the member devices

[Xuan Zhuo]

On Fri, 4 Aug 2023 07:26:01 +0000, Parav Pandit <parav@nvidia.com> wrote:
>
>
> > From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > Sent: Friday, August 4, 2023 12:24 PM
>
> > I know the gap. For us, the switch is under all devices(including the owner
> > device).
> > So we need to control the switch separately.
> >
> Right. Same for other devices too, including ours.
> Hence, it is also located outside of the owner device (on a different system).
>
> If the switch is not a virtio device, does it really need to be exposed as virtio object?
>
> > In your structure, the owner device works as the switch. It has port to connect
> > to the vf devices.
> >
> Right. I just put owner device as switch for simplicity of the discussion.
>
> At this point to have port for owner device requires creating a dedicated switching object, to be located sometimes side by side inside the owner, sometimes outside.
> All of these cases to be crafted, please rethink if this is _really_ needed as virtio object or not.


YES.

We can hear others.

@Jason @Michael


Thanks

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: RE: RE: RE: [virtio-comment] RE: RE: RE: RE: [RFC] virtio-net: support access and control the member devices

[Michael S. Tsirkin]

On Fri, Aug 04, 2023 at 03:44:57PM +0800, Xuan Zhuo wrote:
> On Fri, 4 Aug 2023 07:26:01 +0000, Parav Pandit <parav@nvidia.com> wrote:
> >
> >
> > > From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > > Sent: Friday, August 4, 2023 12:24 PM
> >
> > > I know the gap. For us, the switch is under all devices(including the owner
> > > device).
> > > So we need to control the switch separately.
> > >
> > Right. Same for other devices too, including ours.
> > Hence, it is also located outside of the owner device (on a different system).
> >
> > If the switch is not a virtio device, does it really need to be exposed as virtio object?
> >
> > > In your structure, the owner device works as the switch. It has port to connect
> > > to the vf devices.
> > >
> > Right. I just put owner device as switch for simplicity of the discussion.
> >
> > At this point to have port for owner device requires creating a dedicated switching object, to be located sometimes side by side inside the owner, sometimes outside.
> > All of these cases to be crafted, please rethink if this is _really_ needed as virtio object or not.
> 
> 
> YES.
> 
> We can hear others.
> 
> @Jason @Michael
> 
> 
> Thanks


This is so abstract, hard to have any position as I'm not sure what
we are discussing. If some virtio devices have an integrated switch
then ability to control the switch through virtio seems useful.

Re:queues - it's not by chance that we have multiple admin queues.
So driver can dedicate one queue to filtering commands if
that's felt to be important.


-- 
MST


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] RE: RE: RE: RE: [virtio-comment] RE: RE: RE: RE: [RFC] virtio-net: support access and control the member devices

[Parav Pandit]

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Friday, August 4, 2023 4:03 PM

> > >
> > > At this point to have port for owner device requires creating a dedicated
> switching object, to be located sometimes side by side inside the owner,
> sometimes outside.
> > > All of these cases to be crafted, please rethink if this is _really_ needed as
> virtio object or not.
> >

> >
> > YES.
> >
> > We can hear others.
> >
> > @Jason @Michael
> >
> >
> > Thanks
> 
> 
> This is so abstract, hard to have any position as I'm not sure what we are
> discussing. If some virtio devices have an integrated switch then ability to
> control the switch through virtio seems useful.
>
True, for us, at this point we do not have plan to expose virtio switch device because users are not blocked on it.

> Re:queues - it's not by chance that we have multiple admin queues.
> So driver can dedicate one queue to filtering commands if that's felt to be
> important.
> 
Admin queue currently do not send non admin command of the device.
Would you propose admin queue for something else also for rtc or console or cryto device and indicate its role so device can understand what is coming to it.

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: RE: RE: RE: [virtio-comment] RE: RE: RE: RE: [RFC] virtio-net: support access and control the member devices

[Michael S. Tsirkin]

On Fri, Aug 04, 2023 at 12:49:04PM +0000, Parav Pandit wrote:
> 
> 
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Friday, August 4, 2023 4:03 PM
> 
> > > >
> > > > At this point to have port for owner device requires creating a dedicated
> > switching object, to be located sometimes side by side inside the owner,
> > sometimes outside.
> > > > All of these cases to be crafted, please rethink if this is _really_ needed as
> > virtio object or not.
> > >
> 
> > >
> > > YES.
> > >
> > > We can hear others.
> > >
> > > @Jason @Michael
> > >
> > >
> > > Thanks
> > 
> > 
> > This is so abstract, hard to have any position as I'm not sure what we are
> > discussing. If some virtio devices have an integrated switch then ability to
> > control the switch through virtio seems useful.
> >
> True, for us, at this point we do not have plan to expose virtio switch device because users are not blocked on it.

If someone does though I see no reason not to - we have a variety
of devices, switch would be just another one. The motivation would
be the usual one for virtio - so there's one generic driver.

> > Re:queues - it's not by chance that we have multiple admin queues.
> > So driver can dedicate one queue to filtering commands if that's felt to be
> > important.
> > 
> Admin queue currently do not send non admin command of the device.
> Would you propose admin queue for something else also for rtc or console or cryto device and indicate its role so device can understand what is coming to it.

Well, I guess this switch has group members as its "ports" so
maybe that makes it like an admin command. Again it's all up
in the air too much to be able to judge.

-- 
MST


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] RE: RE: RE: RE: [virtio-comment] RE: RE: RE: RE: [RFC] virtio-net: support access and control the member devices

[Parav Pandit]

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Friday, August 4, 2023 7:37 PM

> > True, for us, at this point we do not have plan to expose virtio switch device
> because users are not blocked on it.
> 
> If someone does though I see no reason not to - we have a variety of devices,
> switch would be just another one. The motivation would be the usual one for
> virtio - so there's one generic driver.
>
Sure.
 
> > > Re:queues - it's not by chance that we have multiple admin queues.
> > > So driver can dedicate one queue to filtering commands if that's
> > > felt to be important.
> > >
> > Admin queue currently do not send non admin command of the device.
> > Would you propose admin queue for something else also for rtc or console or
> cryto device and indicate its role so device can understand what is coming to it.
> 
> Well, I guess this switch has group members as its "ports" so maybe that makes
> it like an admin command. Again it's all up in the air too much to be able to
> judge.

Flowvq is needed for non switch device too.
It has advanced to design stage now, its sad that you are no following it.
Anyways, flow operations is not a admin command.
I don't find admin queue suitable for self unless we modify its definitions a lot.

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: RE: RE: RE: [virtio-comment] RE: RE: RE: RE: [RFC] virtio-net: support access and control the member devices

[Michael S. Tsirkin]

On Fri, Aug 04, 2023 at 02:12:41PM +0000, Parav Pandit wrote:
> 
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Friday, August 4, 2023 7:37 PM
> 
> > > True, for us, at this point we do not have plan to expose virtio switch device
> > because users are not blocked on it.
> > 
> > If someone does though I see no reason not to - we have a variety of devices,
> > switch would be just another one. The motivation would be the usual one for
> > virtio - so there's one generic driver.
> >
> Sure.
>  
> > > > Re:queues - it's not by chance that we have multiple admin queues.
> > > > So driver can dedicate one queue to filtering commands if that's
> > > > felt to be important.
> > > >
> > > Admin queue currently do not send non admin command of the device.
> > > Would you propose admin queue for something else also for rtc or console or
> > cryto device and indicate its role so device can understand what is coming to it.
> > 
> > Well, I guess this switch has group members as its "ports" so maybe that makes
> > it like an admin command. Again it's all up in the air too much to be able to
> > judge.
> 
> Flowvq is needed for non switch device too.
> It has advanced to design stage now, its sad that you are no following it.

Frankly I didn't even understand that's what this is talking about.

> Anyways, flow operations is not a admin command.
> I don't find admin queue suitable for self unless we modify its definitions a lot.


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: RE: RE: RE: RE: [virtio-comment] RE: RE: RE: RE: [RFC] virtio-net: support access and control the member devices

[Xuan Zhuo]

On Fri, 4 Aug 2023 12:49:04 +0000, Parav Pandit <parav@nvidia.com> wrote:
>
>
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Friday, August 4, 2023 4:03 PM
>
> > > >
> > > > At this point to have port for owner device requires creating a dedicated
> > switching object, to be located sometimes side by side inside the owner,
> > sometimes outside.
> > > > All of these cases to be crafted, please rethink if this is _really_ needed as
> > virtio object or not.
> > >
>
> > >
> > > YES.
> > >
> > > We can hear others.
> > >
> > > @Jason @Michael
> > >
> > >
> > > Thanks
> >
> >
> > This is so abstract, hard to have any position as I'm not sure what we are
> > discussing. If some virtio devices have an integrated switch then ability to
> > control the switch through virtio seems useful.
> >
> True, for us, at this point we do not have plan to expose virtio switch device because users are not blocked on it.


Also for us.

But we need to limit the ip of every member device. That is useful for cloud.
Because the user of each VM is untrustworthy. We must limit the ip traffic of
every member device.

We have two choose:

1. add feature to device by cvq of pf(or admin queue?), that can limit the ip(receive and transmit).
2. add feature to switch, it can limit the ip for every port. If we choose this
   way, I will try introduce the simple switch concept to the virtio-net.
   Because except this we have not more requirement for the switch. So we donot
   plan to introduce a complex switch.

Thanks.


>
> > Re:queues - it's not by chance that we have multiple admin queues.
> > So driver can dedicate one queue to filtering commands if that's felt to be
> > important.
> >
> Admin queue currently do not send non admin command of the device.
> Would you propose admin queue for something else also for rtc or console or cryto device and indicate its role so device can understand what is coming to it.

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: RE: RE: RE: RE: [virtio-comment] RE: RE: RE: RE: [RFC] virtio-net: support access and control the member devices

[Jason Wang]

On Mon, Aug 7, 2023 at 2:13 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
>
> On Fri, 4 Aug 2023 12:49:04 +0000, Parav Pandit <parav@nvidia.com> wrote:
> >
> >
> > > From: Michael S. Tsirkin <mst@redhat.com>
> > > Sent: Friday, August 4, 2023 4:03 PM
> >
> > > > >
> > > > > At this point to have port for owner device requires creating a dedicated
> > > switching object, to be located sometimes side by side inside the owner,
> > > sometimes outside.
> > > > > All of these cases to be crafted, please rethink if this is _really_ needed as
> > > virtio object or not.
> > > >
> >
> > > >
> > > > YES.
> > > >
> > > > We can hear others.
> > > >
> > > > @Jason @Michael
> > > >
> > > >
> > > > Thanks
> > >
> > >
> > > This is so abstract, hard to have any position as I'm not sure what we are
> > > discussing. If some virtio devices have an integrated switch then ability to
> > > control the switch through virtio seems useful.
> > >
> > True, for us, at this point we do not have plan to expose virtio switch device because users are not blocked on it.
>
>
> Also for us.
>
> But we need to limit the ip of every member device.

This has been discussed somehow before we need probably more like:
spoof check and trust which are already supported by iproute2:

https://lists.oasis-open.org/archives/virtio-comment/202101/msg00047.html

> That is useful for cloud.
> Because the user of each VM is untrustworthy. We must limit the ip traffic of
> every member device.
>
> We have two choose:
>
> 1. add feature to device by cvq of pf(or admin queue?), that can limit the ip(receive and transmit).
> 2. add feature to switch, it can limit the ip for every port. If we choose this
>    way, I will try introduce the simple switch concept to the virtio-net.
>    Because except this we have not more requirement for the switch. So we donot
>    plan to introduce a complex switch.

This requirement (IP limitation) sounds more like a filter feature
which seems not directly related to switch.

Thanks

>
> Thanks.
>
>
> >
> > > Re:queues - it's not by chance that we have multiple admin queues.
> > > So driver can dedicate one queue to filtering commands if that's felt to be
> > > important.
> > >
> > Admin queue currently do not send non admin command of the device.
> > Would you propose admin queue for something else also for rtc or console or cryto device and indicate its role so device can understand what is coming to it.
>


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [RFC] virtio-net: support access and control the member devices

[Jason Wang]

On Fri, Aug 4, 2023 at 12:05 PM Parav Pandit <parav@nvidia.com> wrote:
>
>
>
> > From: Jason Wang <jasowang@redhat.com>
> > Sent: Friday, August 4, 2023 9:02 AM
>
> > I think we probably need some split here.
> >
> > E.g the fields belonging to configuration space might go with a more general
> > interface like admin command/queue.
> >
> > The net specific ones need to go through a net specific interface.
> > (cvq of PF or reps)
>
> I don’t see the need of such split.
> Why do you think that?

It's simply because part of the fields belong to config space.

Thanks


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: RE: RE: RE: RE: [virtio-comment] RE: RE: RE: RE: [RFC] virtio-net: support access and control the member devices

[Xuan Zhuo]

On Tue, 8 Aug 2023 10:18:15 +0800, Jason Wang <jasowang@redhat.com> wrote:
> On Mon, Aug 7, 2023 at 2:13 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> >
> > On Fri, 4 Aug 2023 12:49:04 +0000, Parav Pandit <parav@nvidia.com> wrote:
> > >
> > >
> > > > From: Michael S. Tsirkin <mst@redhat.com>
> > > > Sent: Friday, August 4, 2023 4:03 PM
> > >
> > > > > >
> > > > > > At this point to have port for owner device requires creating a dedicated
> > > > switching object, to be located sometimes side by side inside the owner,
> > > > sometimes outside.
> > > > > > All of these cases to be crafted, please rethink if this is _really_ needed as
> > > > virtio object or not.
> > > > >
> > >
> > > > >
> > > > > YES.
> > > > >
> > > > > We can hear others.
> > > > >
> > > > > @Jason @Michael
> > > > >
> > > > >
> > > > > Thanks
> > > >
> > > >
> > > > This is so abstract, hard to have any position as I'm not sure what we are
> > > > discussing. If some virtio devices have an integrated switch then ability to
> > > > control the switch through virtio seems useful.
> > > >
> > > True, for us, at this point we do not have plan to expose virtio switch device because users are not blocked on it.
> >
> >
> > Also for us.
> >
> > But we need to limit the ip of every member device.
>
> This has been discussed somehow before we need probably more like:
> spoof check and trust which are already supported by iproute2:
>
> https://lists.oasis-open.org/archives/virtio-comment/202101/msg00047.html


Sorry, I do not think that the above patch is match for us.

In our case, one VM user may use the IP of another VM user to send/receive
packets.

So if the above idea is useful to our case, cloud you explain more?

>
> > That is useful for cloud.
> > Because the user of each VM is untrustworthy. We must limit the ip traffic of
> > every member device.
> >
> > We have two choose:
> >
> > 1. add feature to device by cvq of pf(or admin queue?), that can limit the ip(receive and transmit).
> > 2. add feature to switch, it can limit the ip for every port. If we choose this
> >    way, I will try introduce the simple switch concept to the virtio-net.
> >    Because except this we have not more requirement for the switch. So we donot
> >    plan to introduce a complex switch.
>
> This requirement (IP limitation) sounds more like a filter feature
> which seems not directly related to switch.

Yes, this can be a filter feature. But we also need to limit the tx traffic.

For receive traffic, many net cards support that the flowdirector(or RFF) can
steer the traffic to the VF.

https://dpdk.readthedocs.io/en/v17.11/howto/flow_bifurcation.html

But for us, that is just work for the receive traffic,
we also need to limit the tx traffic.

So we want to introduce as the feature of the device.

Thanks



>
> Thanks
>
> >
> > Thanks.
> >
> >
> > >
> > > > Re:queues - it's not by chance that we have multiple admin queues.
> > > > So driver can dedicate one queue to filtering commands if that's felt to be
> > > > important.
> > > >
> > > Admin queue currently do not send non admin command of the device.
> > > Would you propose admin queue for something else also for rtc or console or cryto device and indicate its role so device can understand what is coming to it.
> >
>

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: RE: RE: RE: RE: [virtio-comment] RE: RE: RE: RE: [RFC] virtio-net: support access and control the member devices

[Jason Wang]

On Tue, Aug 8, 2023 at 10:46 AM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
>
> On Tue, 8 Aug 2023 10:18:15 +0800, Jason Wang <jasowang@redhat.com> wrote:
> > On Mon, Aug 7, 2023 at 2:13 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> > >
> > > On Fri, 4 Aug 2023 12:49:04 +0000, Parav Pandit <parav@nvidia.com> wrote:
> > > >
> > > >
> > > > > From: Michael S. Tsirkin <mst@redhat.com>
> > > > > Sent: Friday, August 4, 2023 4:03 PM
> > > >
> > > > > > >
> > > > > > > At this point to have port for owner device requires creating a dedicated
> > > > > switching object, to be located sometimes side by side inside the owner,
> > > > > sometimes outside.
> > > > > > > All of these cases to be crafted, please rethink if this is _really_ needed as
> > > > > virtio object or not.
> > > > > >
> > > >
> > > > > >
> > > > > > YES.
> > > > > >
> > > > > > We can hear others.
> > > > > >
> > > > > > @Jason @Michael
> > > > > >
> > > > > >
> > > > > > Thanks
> > > > >
> > > > >
> > > > > This is so abstract, hard to have any position as I'm not sure what we are
> > > > > discussing. If some virtio devices have an integrated switch then ability to
> > > > > control the switch through virtio seems useful.
> > > > >
> > > > True, for us, at this point we do not have plan to expose virtio switch device because users are not blocked on it.
> > >
> > >
> > > Also for us.
> > >
> > > But we need to limit the ip of every member device.
> >
> > This has been discussed somehow before we need probably more like:
> > spoof check and trust which are already supported by iproute2:
> >
> > https://lists.oasis-open.org/archives/virtio-comment/202101/msg00047.html
>
>
> Sorry, I do not think that the above patch is match for us.
>
> In our case, one VM user may use the IP of another VM user to send/receive
> packets.
>
> So if the above idea is useful to our case, cloud you explain more?
>
> >
> > > That is useful for cloud.
> > > Because the user of each VM is untrustworthy. We must limit the ip traffic of
> > > every member device.
> > >
> > > We have two choose:
> > >
> > > 1. add feature to device by cvq of pf(or admin queue?), that can limit the ip(receive and transmit).
> > > 2. add feature to switch, it can limit the ip for every port. If we choose this
> > >    way, I will try introduce the simple switch concept to the virtio-net.
> > >    Because except this we have not more requirement for the switch. So we donot
> > >    plan to introduce a complex switch.
> >
> > This requirement (IP limitation) sounds more like a filter feature
> > which seems not directly related to switch.
>
> Yes, this can be a filter feature. But we also need to limit the tx traffic.
>
> For receive traffic, many net cards support that the flowdirector(or RFF) can
> steer the traffic to the VF.
>
> https://dpdk.readthedocs.io/en/v17.11/howto/flow_bifurcation.html
>
> But for us, that is just work for the receive traffic,
> we also need to limit the tx traffic.
>
> So we want to introduce as the feature of the device.

Yes, but even for TX, would it be better to filter the IP as early as
possible in the TX path other than depend on the switch to do that?

Thanks

>
> Thanks
>
>
>
> >
> > Thanks
> >
> > >
> > > Thanks.
> > >
> > >
> > > >
> > > > > Re:queues - it's not by chance that we have multiple admin queues.
> > > > > So driver can dedicate one queue to filtering commands if that's felt to be
> > > > > important.
> > > > >
> > > > Admin queue currently do not send non admin command of the device.
> > > > Would you propose admin queue for something else also for rtc or console or cryto device and indicate its role so device can understand what is coming to it.
> > >
> >
>


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: RE: RE: RE: RE: [virtio-comment] RE: RE: RE: RE: [RFC] virtio-net: support access and control the member devices

[Xuan Zhuo]

On Tue, 8 Aug 2023 11:09:03 +0800, Jason Wang <jasowang@redhat.com> wrote:
> On Tue, Aug 8, 2023 at 10:46 AM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> >
> > On Tue, 8 Aug 2023 10:18:15 +0800, Jason Wang <jasowang@redhat.com> wrote:
> > > On Mon, Aug 7, 2023 at 2:13 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> > > >
> > > > On Fri, 4 Aug 2023 12:49:04 +0000, Parav Pandit <parav@nvidia.com> wrote:
> > > > >
> > > > >
> > > > > > From: Michael S. Tsirkin <mst@redhat.com>
> > > > > > Sent: Friday, August 4, 2023 4:03 PM
> > > > >
> > > > > > > >
> > > > > > > > At this point to have port for owner device requires creating a dedicated
> > > > > > switching object, to be located sometimes side by side inside the owner,
> > > > > > sometimes outside.
> > > > > > > > All of these cases to be crafted, please rethink if this is _really_ needed as
> > > > > > virtio object or not.
> > > > > > >
> > > > >
> > > > > > >
> > > > > > > YES.
> > > > > > >
> > > > > > > We can hear others.
> > > > > > >
> > > > > > > @Jason @Michael
> > > > > > >
> > > > > > >
> > > > > > > Thanks
> > > > > >
> > > > > >
> > > > > > This is so abstract, hard to have any position as I'm not sure what we are
> > > > > > discussing. If some virtio devices have an integrated switch then ability to
> > > > > > control the switch through virtio seems useful.
> > > > > >
> > > > > True, for us, at this point we do not have plan to expose virtio switch device because users are not blocked on it.
> > > >
> > > >
> > > > Also for us.
> > > >
> > > > But we need to limit the ip of every member device.
> > >
> > > This has been discussed somehow before we need probably more like:
> > > spoof check and trust which are already supported by iproute2:
> > >
> > > https://lists.oasis-open.org/archives/virtio-comment/202101/msg00047.html
> >
> >
> > Sorry, I do not think that the above patch is match for us.
> >
> > In our case, one VM user may use the IP of another VM user to send/receive
> > packets.
> >
> > So if the above idea is useful to our case, cloud you explain more?
> >
> > >
> > > > That is useful for cloud.
> > > > Because the user of each VM is untrustworthy. We must limit the ip traffic of
> > > > every member device.
> > > >
> > > > We have two choose:
> > > >
> > > > 1. add feature to device by cvq of pf(or admin queue?), that can limit the ip(receive and transmit).
> > > > 2. add feature to switch, it can limit the ip for every port. If we choose this
> > > >    way, I will try introduce the simple switch concept to the virtio-net.
> > > >    Because except this we have not more requirement for the switch. So we donot
> > > >    plan to introduce a complex switch.
> > >
> > > This requirement (IP limitation) sounds more like a filter feature
> > > which seems not directly related to switch.
> >
> > Yes, this can be a filter feature. But we also need to limit the tx traffic.
> >
> > For receive traffic, many net cards support that the flowdirector(or RFF) can
> > steer the traffic to the VF.
> >
> > https://dpdk.readthedocs.io/en/v17.11/howto/flow_bifurcation.html
> >
> > But for us, that is just work for the receive traffic,
> > we also need to limit the tx traffic.
> >
> > So we want to introduce as the feature of the device.
>
> Yes, but even for TX, would it be better to filter the IP as early as
> possible in the TX path other than depend on the switch to do that?

Yes, if we do this in the device. Then work has nothing to do with the switch.

Thanks.

>
> Thanks
>
> >
> > Thanks
> >
> >
> >
> > >
> > > Thanks
> > >
> > > >
> > > > Thanks.
> > > >
> > > >
> > > > >
> > > > > > Re:queues - it's not by chance that we have multiple admin queues.
> > > > > > So driver can dedicate one queue to filtering commands if that's felt to be
> > > > > > important.
> > > > > >
> > > > > Admin queue currently do not send non admin command of the device.
> > > > > Would you propose admin queue for something else also for rtc or console or cryto device and indicate its role so device can understand what is coming to it.
> > > >
> > >
> >
>

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] RE: RE: RE: RE: RE: [virtio-comment] RE: RE: RE: RE: [RFC] virtio-net: support access and control the member devices

[Parav Pandit]

> From: Jason Wang <jasowang@redhat.com>
> Sent: Tuesday, August 8, 2023 8:39 AM

> 
> Yes, but even for TX, would it be better to filter the IP as early as possible in the
> TX path other than depend on the switch to do that?

The idea is to introduce filters on the new virtio switch object for tx and rx both.
A virtio switch object can be part of a existing virtio device or a new virtio device type in itself.

Xuan,
As we discussed, since the owner device packets also needs to be filtered, potentially outside of the owner device itself,

Do you see the need to introduce virtio switch object now, or can it wait?

[virtio-dev] Re: RE: RE: RE: RE: [virtio-comment] RE: RE: RE: RE: [RFC] virtio-net: support access and control the member devices

[Jason Wang]

On Tue, Aug 8, 2023 at 11:16 AM Parav Pandit <parav@nvidia.com> wrote:
>
>
>
> > From: Jason Wang <jasowang@redhat.com>
> > Sent: Tuesday, August 8, 2023 8:39 AM
>
> >
> > Yes, but even for TX, would it be better to filter the IP as early as possible in the
> > TX path other than depend on the switch to do that?
>
> The idea is to introduce filters on the new virtio switch object for tx and rx both.

It can be done in this way for sure but the question is why it must be
done in this way. A drawback of using switch is that it introduces
dependencies.

> A virtio switch object can be part of a existing virtio device or a new virtio device type in itself.

That's fine.

>
> Xuan,
> As we discussed, since the owner device packets also needs to be filtered, potentially outside of the owner device itself,

This seems the admin request out of the scope of virtio.

Thanks

>
> Do you see the need to introduce virtio switch object now, or can it wait?


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: RE: RE: RE: RE: RE: [virtio-comment] RE: RE: RE: RE: [RFC] virtio-net: support access and control the member devices

[Xuan Zhuo]

On Tue, 8 Aug 2023 03:16:44 +0000, Parav Pandit <parav@nvidia.com> wrote:
>
>
> > From: Jason Wang <jasowang@redhat.com>
> > Sent: Tuesday, August 8, 2023 8:39 AM
>
> >
> > Yes, but even for TX, would it be better to filter the IP as early as possible in the
> > TX path other than depend on the switch to do that?
>
> The idea is to introduce filters on the new virtio switch object for tx and rx both.

Filters for tx?

Are there other use cases?


> A virtio switch object can be part of a existing virtio device or a new virtio device type in itself.
>
> Xuan,
> As we discussed, since the owner device packets also needs to be filtered, potentially outside of the owner device itself,
>
> Do you see the need to introduce virtio switch object now, or can it wait?

For virtio switch, I am ok.

But for me, I just have this requirement that needs the switch.
So if we do this in the device, then that is not need for me.

Now, you think we should introduce the rx/tx filter to the virtio switch.

@Jason @Michael how do you think?

Thanks.




---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] RE: RE: RE: RE: RE: RE: [virtio-comment] RE: RE: RE: RE: [RFC] virtio-net: support access and control the member devices

[Parav Pandit]

> From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> Sent: Tuesday, August 8, 2023 9:31 AM
> 
> On Tue, 8 Aug 2023 03:16:44 +0000, Parav Pandit <parav@nvidia.com> wrote:
> >
> >
> > > From: Jason Wang <jasowang@redhat.com>
> > > Sent: Tuesday, August 8, 2023 8:39 AM
> >
> > >
> > > Yes, but even for TX, would it be better to filter the IP as early
> > > as possible in the TX path other than depend on the switch to do that?
> >
> > The idea is to introduce filters on the new virtio switch object for tx and rx
> both.
> 
> Filters for tx?
>
Yes. the regular switchdev model that exists in Linux kernel for several years now.
And similar in another hypervisor.
 
> Are there other use cases?
> 

> 
> > A virtio switch object can be part of a existing virtio device or a new virtio
> device type in itself.
> >
> > Xuan,
> > As we discussed, since the owner device packets also needs to be
> > filtered, potentially outside of the owner device itself,
> >
> > Do you see the need to introduce virtio switch object now, or can it wait?
> 
> For virtio switch, I am ok.
> 
I didn't follow your answer "I am ok".
Do you see the need now or can it wait?

> But for me, I just have this requirement that needs the switch.

> So if we do this in the device, then that is not need for me.
> 
Device meaning outside of the virtio domain, for example dpu or something else, yes?

> Now, you think we should introduce the rx/tx filter to the virtio switch.
>
I am asking to _not_ introduce currently.

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: RE: RE: RE: RE: RE: RE: [virtio-comment] RE: RE: RE: RE: [RFC] virtio-net: support access and control the member devices

[Xuan Zhuo]

On Tue, 8 Aug 2023 04:11:03 +0000, Parav Pandit <parav@nvidia.com> wrote:
>
>
> > From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > Sent: Tuesday, August 8, 2023 9:31 AM
> >
> > On Tue, 8 Aug 2023 03:16:44 +0000, Parav Pandit <parav@nvidia.com> wrote:
> > >
> > >
> > > > From: Jason Wang <jasowang@redhat.com>
> > > > Sent: Tuesday, August 8, 2023 8:39 AM
> > >
> > > >
> > > > Yes, but even for TX, would it be better to filter the IP as early
> > > > as possible in the TX path other than depend on the switch to do that?
> > >
> > > The idea is to introduce filters on the new virtio switch object for tx and rx
> > both.
> >
> > Filters for tx?
> >
> Yes. the regular switchdev model that exists in Linux kernel for several years now.
> And similar in another hypervisor.
>
> > Are there other use cases?
> >
>
> >
> > > A virtio switch object can be part of a existing virtio device or a new virtio
> > device type in itself.
> > >
> > > Xuan,
> > > As we discussed, since the owner device packets also needs to be
> > > filtered, potentially outside of the owner device itself,
> > >
> > > Do you see the need to introduce virtio switch object now, or can it wait?
> >
> > For virtio switch, I am ok.
> >
> I didn't follow your answer "I am ok".
> Do you see the need now or can it wait?
>
> > But for me, I just have this requirement that needs the switch.
>
> > So if we do this in the device, then that is not need for me.
> >
> Device meaning outside of the virtio domain, for example dpu or something else, yes?

NO.

I mean the virtio-net device.


>
> > Now, you think we should introduce the rx/tx filter to the virtio switch.
> >
> I am asking to _not_ introduce currently.

I don't care whether swtich is introduced or not. Because we have no
dependencies on it.

My requirement is to implement ip restriction which is very important for cloud
scenarios.

I think the gap is, I don't care how it's implemented. Because for me, it can be
supported by switch rx, tx filter or virtio-net device. I'd love to hear
everyone's opinions.

Thanks.



---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] RE: RE: RE: RE: RE: [virtio-comment] RE: RE: RE: RE: [RFC] virtio-net: support access and control the member devices

[Parav Pandit]

> From: Jason Wang <jasowang@redhat.com>
> Sent: Tuesday, August 8, 2023 9:21 AM

> > The idea is to introduce filters on the new virtio switch object for tx and rx
> both.
> 
> It can be done in this way for sure but the question is why it must be done in
> this way. 
This option because it is in use by very big and mature eco system of multiple sw stacks, kernel subsystem, drivers, and nics for several years now.

> A drawback of using switch is that it introduces dependencies.
>
Feature is not a dependency. :)
 
> > A virtio switch object can be part of a existing virtio device or a new virtio
> device type in itself.
> 
> That's fine.
> 
> >
> > Xuan,
> > As we discussed, since the owner device packets also needs to be
> > filtered, potentially outside of the owner device itself,
> 
> This seems the admin request out of the scope of virtio.
>
Not really, it could be virto switch device that manage PF also.
At that point, there may be two functions, PF and switching PF, switching PF filters the traffic of the PF.

Anyways, I am just not finding it useful enough at current point in time for us as far mature alternatives exist that users are comfortable with.
I would like to listen to Xuan if they really have use case for having switching PF as virtio object or not.

[virtio-dev] RE: RE: RE: RE: RE: RE: [virtio-comment] RE: RE: RE: RE: [RFC] virtio-net: support access and control the member devices

[Satananda Burla]

> -----Original Message-----
> From: virtio-dev@lists.oasis-open.org <virtio-dev@lists.oasis-open.org>
> On Behalf Of Parav Pandit
> Sent: Monday, August 7, 2023 9:11 PM
> To: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> Cc: virtio-dev@lists.oasis-open.org; virtio-comment@lists.oasis-
> open.org; Michael S. Tsirkin <mst@redhat.com>; Jason Wang
> <jasowang@redhat.com>
> Subject: [EXT] [virtio-dev] RE: RE: RE: RE: RE: RE: [virtio-comment] RE:
> RE: RE: RE: [RFC] virtio-net: support access and control the member
> devices
> 
> External Email
> 
> ----------------------------------------------------------------------
> 
> 
> > From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > Sent: Tuesday, August 8, 2023 9:31 AM
> >
> > On Tue, 8 Aug 2023 03:16:44 +0000, Parav Pandit <parav@nvidia.com>
> wrote:
> > >
> > >
> > > > From: Jason Wang <jasowang@redhat.com>
> > > > Sent: Tuesday, August 8, 2023 8:39 AM
> > >
> > > >
> > > > Yes, but even for TX, would it be better to filter the IP as early
> > > > as possible in the TX path other than depend on the switch to do
> that?
> > >
> > > The idea is to introduce filters on the new virtio switch object for
> tx and rx
> > both.
> >
> > Filters for tx?
> >
> Yes. the regular switchdev model that exists in Linux kernel for several
> years now.
This is what representors would be used for. The Tx of representee is the
RX of representor. Hence one can install RX L4/L3/L2 filters of choice
on representor using tc flower, offload to device if need be, 
these would then be applied on the tx path of representee. Similarly the
owner device can also have rx filters installed to direct traffic to the
the representee.
> And similar in another hypervisor.
> 
> > Are there other use cases?
> >
> 
> >
> > > A virtio switch object can be part of a existing virtio device or a
> new virtio
> > device type in itself.
> > >
> > > Xuan,
> > > As we discussed, since the owner device packets also needs to be
> > > filtered, potentially outside of the owner device itself,
> > >
> > > Do you see the need to introduce virtio switch object now, or can it
> wait?
> >
> > For virtio switch, I am ok.
> >
> I didn't follow your answer "I am ok".
> Do you see the need now or can it wait?
> 
> > But for me, I just have this requirement that needs the switch.
> 
> > So if we do this in the device, then that is not need for me.
> >
> Device meaning outside of the virtio domain, for example dpu or
> something else, yes?
> 
> > Now, you think we should introduce the rx/tx filter to the virtio
> switch.
> >
> I am asking to _not_ introduce currently.
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: RE: RE: RE: RE: [virtio-comment] RE: RE: RE: RE: [RFC] virtio-net: support access and control the member devices

[Jason Wang]

On Tue, Aug 8, 2023 at 12:17 PM Parav Pandit <parav@nvidia.com> wrote:
>
>
> > From: Jason Wang <jasowang@redhat.com>
> > Sent: Tuesday, August 8, 2023 9:21 AM
>
> > > The idea is to introduce filters on the new virtio switch object for tx and rx
> > both.
> >
> > It can be done in this way for sure but the question is why it must be done in
> > this way.
> This option because it is in use by very big and mature eco system of multiple sw stacks, kernel subsystem, drivers, and nics for several years now.
>
> > A drawback of using switch is that it introduces dependencies.
> >
> Feature is not a dependency. :)

Well, I meant you need a switch in order to let the IP filter work then.

>
> > > A virtio switch object can be part of a existing virtio device or a new virtio
> > device type in itself.
> >
> > That's fine.
> >
> > >
> > > Xuan,
> > > As we discussed, since the owner device packets also needs to be
> > > filtered, potentially outside of the owner device itself,
> >
> > This seems the admin request out of the scope of virtio.
> >
> Not really, it could be virto switch device that manage PF also.
> At that point, there may be two functions, PF and switching PF, switching PF filters the traffic of the PF.

That's fine. But such filtering needs to be done in a switch specific
way not via the admin command/virtqueue.

Thanks

>
> Anyways, I am just not finding it useful enough at current point in time for us as far mature alternatives exist that users are comfortable with.
> I would like to listen to Xuan if they really have use case for having switching PF as virtio object or not.
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] RE: RE: RE: RE: RE: RE: RE: [virtio-comment] RE: RE: RE: RE: [RFC] virtio-net: support access and control the member devices

[Parav Pandit]

> From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> Sent: Tuesday, August 8, 2023 9:47 AM

> I don't care whether swtich is introduced or not. Because we have no
> dependencies on it.
> 
> My requirement is to implement ip restriction which is very important for cloud
> scenarios.
>
These restrictions are needed for PF and VFs both, so can this be implemented outside of the PF and VF in your environment?
 
> I think the gap is, I don't care how it's implemented. Because for me, it can be
> supported by switch rx, tx filter or virtio-net device. I'd love to hear everyone's
> opinions.
> 
One proven model is to have virtio-net device to have switch object that exposes flow filters capability, if at all this is needed in your env.


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] RE: RE: RE: RE: RE: [virtio-comment] RE: RE: RE: RE: [RFC] virtio-net: support access and control the member devices

[Parav Pandit]

> From: Jason Wang <jasowang@redhat.com>
> Sent: Tuesday, August 8, 2023 10:03 AM

> > This option because it is in use by very big and mature eco system of multiple
> sw stacks, kernel subsystem, drivers, and nics for several years now.
> >
> > > A drawback of using switch is that it introduces dependencies.
> > >
> > Feature is not a dependency. :)
> 
> Well, I meant you need a switch in order to let the IP filter work then.
>
Ok.
 
> >
> > > > A virtio switch object can be part of a existing virtio device or
> > > > a new virtio
> > > device type in itself.
> > >
> > > That's fine.
> > >
> > > >
> > > > Xuan,
> > > > As we discussed, since the owner device packets also needs to be
> > > > filtered, potentially outside of the owner device itself,
> > >
> > > This seems the admin request out of the scope of virtio.
> > >
> > Not really, it could be virto switch device that manage PF also.
> > At that point, there may be two functions, PF and switching PF, switching PF
> filters the traffic of the PF.
> 
> That's fine. But such filtering needs to be done in a switch specific way not via
> the admin command/virtqueue.

A switch object needs a generic flow filter vq(s) to meet the high rate needed.
Several of us have worked through the flow filter vq for few several weeks on bi-weekly basis and over public mailing lists.

We can differ the design discussion once we have clarity on requirements. :)

[virtio-dev] Re: RE: RE: RE: RE: [virtio-comment] RE: RE: RE: RE: [RFC] virtio-net: support access and control the member devices

[Jason Wang]

On Tue, Aug 8, 2023 at 12:38 PM Parav Pandit <parav@nvidia.com> wrote:
>
>
> > From: Jason Wang <jasowang@redhat.com>
> > Sent: Tuesday, August 8, 2023 10:03 AM
>
> > > This option because it is in use by very big and mature eco system of multiple
> > sw stacks, kernel subsystem, drivers, and nics for several years now.
> > >
> > > > A drawback of using switch is that it introduces dependencies.
> > > >
> > > Feature is not a dependency. :)
> >
> > Well, I meant you need a switch in order to let the IP filter work then.
> >
> Ok.
>
> > >
> > > > > A virtio switch object can be part of a existing virtio device or
> > > > > a new virtio
> > > > device type in itself.
> > > >
> > > > That's fine.
> > > >
> > > > >
> > > > > Xuan,
> > > > > As we discussed, since the owner device packets also needs to be
> > > > > filtered, potentially outside of the owner device itself,
> > > >
> > > > This seems the admin request out of the scope of virtio.
> > > >
> > > Not really, it could be virto switch device that manage PF also.
> > > At that point, there may be two functions, PF and switching PF, switching PF
> > filters the traffic of the PF.
> >
> > That's fine. But such filtering needs to be done in a switch specific way not via
> > the admin command/virtqueue.
>
> A switch object needs a generic flow filter vq(s) to meet the high rate needed.
> Several of us have worked through the flow filter vq for few several weeks on bi-weekly basis and over public mailing lists.
>
> We can differ the design discussion once we have clarity on requirements. :)

It can be done via flow filter or not for sure. We need more inputs on
the requirement. Maybe a new thread is more suitable.

Thanks


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: RE: RE: RE: RE: RE: RE: RE: [virtio-comment] RE: RE: RE: RE: [RFC] virtio-net: support access and control the member devices

[Xuan Zhuo]

On Tue, 8 Aug 2023 04:34:18 +0000, Parav Pandit <parav@nvidia.com> wrote:
>
> > From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > Sent: Tuesday, August 8, 2023 9:47 AM
>
> > I don't care whether swtich is introduced or not. Because we have no
> > dependencies on it.
> >
> > My requirement is to implement ip restriction which is very important for cloud
> > scenarios.
> >
> These restrictions are needed for PF and VFs both, so can this be implemented outside of the PF and VF in your environment?


I see.

The point is that we want to config by the virtio admin queue (or cvq).
So we want introduce this feature to the virtio.
We will expose this function to the users.
We want the user uses a standard way to config the ip restrictions.

Thanks.

>
> > I think the gap is, I don't care how it's implemented. Because for me, it can be
> > supported by switch rx, tx filter or virtio-net device. I'd love to hear everyone's
> > opinions.
> >
> One proven model is to have virtio-net device to have switch object that exposes flow filters capability, if at all this is needed in your env.
>



---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: RE: RE: RE: RE: [virtio-comment] RE: RE: RE: RE: [RFC] virtio-net: support access and control the member devices

[Xuan Zhuo]

On Tue, 8 Aug 2023 13:21:23 +0800, Jason Wang <jasowang@redhat.com> wrote:
> On Tue, Aug 8, 2023 at 12:38 PM Parav Pandit <parav@nvidia.com> wrote:
> >
> >
> > > From: Jason Wang <jasowang@redhat.com>
> > > Sent: Tuesday, August 8, 2023 10:03 AM
> >
> > > > This option because it is in use by very big and mature eco system of multiple
> > > sw stacks, kernel subsystem, drivers, and nics for several years now.
> > > >
> > > > > A drawback of using switch is that it introduces dependencies.
> > > > >
> > > > Feature is not a dependency. :)
> > >
> > > Well, I meant you need a switch in order to let the IP filter work then.
> > >
> > Ok.
> >
> > > >
> > > > > > A virtio switch object can be part of a existing virtio device or
> > > > > > a new virtio
> > > > > device type in itself.
> > > > >
> > > > > That's fine.
> > > > >
> > > > > >
> > > > > > Xuan,
> > > > > > As we discussed, since the owner device packets also needs to be
> > > > > > filtered, potentially outside of the owner device itself,
> > > > >
> > > > > This seems the admin request out of the scope of virtio.
> > > > >
> > > > Not really, it could be virto switch device that manage PF also.
> > > > At that point, there may be two functions, PF and switching PF, switching PF
> > > filters the traffic of the PF.
> > >
> > > That's fine. But such filtering needs to be done in a switch specific way not via
> > > the admin command/virtqueue.
> >
> > A switch object needs a generic flow filter vq(s) to meet the high rate needed.
> > Several of us have worked through the flow filter vq for few several weeks on bi-weekly basis and over public mailing lists.
> >
> > We can differ the design discussion once we have clarity on requirements. :)
>
> It can be done via flow filter or not for sure. We need more inputs on
> the requirement. Maybe a new thread is more suitable.

YES.

I will post a summary to a new thread.

Thanks.


>
> Thanks
>

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: RE: RE: RE: RE: RE: [virtio-comment] RE: RE: RE: RE: [RFC] virtio-net: support access and control the member devices

[Xuan Zhuo]

On Tue, 8 Aug 2023 04:16:59 +0000, Parav Pandit <parav@nvidia.com> wrote:
>
> > From: Jason Wang <jasowang@redhat.com>
> > Sent: Tuesday, August 8, 2023 9:21 AM
>
> > > The idea is to introduce filters on the new virtio switch object for tx and rx
> > both.
> >
> > It can be done in this way for sure but the question is why it must be done in
> > this way.
> This option because it is in use by very big and mature eco system of multiple sw stacks, kernel subsystem, drivers, and nics for several years now.
>
> > A drawback of using switch is that it introduces dependencies.
> >
> Feature is not a dependency. :)
>
> > > A virtio switch object can be part of a existing virtio device or a new virtio
> > device type in itself.
> >
> > That's fine.
> >
> > >
> > > Xuan,
> > > As we discussed, since the owner device packets also needs to be
> > > filtered, potentially outside of the owner device itself,
> >
> > This seems the admin request out of the scope of virtio.
> >
> Not really, it could be virto switch device that manage PF also.
> At that point, there may be two functions, PF and switching PF, switching PF filters the traffic of the PF.
>
> Anyways, I am just not finding it useful enough at current point in time for us as far mature alternatives exist that users are comfortable with.
> I would like to listen to Xuan if they really have use case for having switching PF as virtio object or not.

So the switching PF is that there is a switch under all devices?

If we use the switch rx/tx filter for the ip restriction, the PF with switch is
enough,  we do not need the ip restirction for the PF.

Thanks.


>
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] RE: RE: RE: RE: RE: RE: [virtio-comment] RE: RE: RE: RE: [RFC] virtio-net: support access and control the member devices

[Parav Pandit]

> From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> Sent: Tuesday, August 8, 2023 12:15 PM

> > At that point, there may be two functions, PF and switching PF, switching PF
> filters the traffic of the PF.
> >
> > Anyways, I am just not finding it useful enough at current point in time for us
> as far mature alternatives exist that users are comfortable with.
> > I would like to listen to Xuan if they really have use case for having switching
> PF as virtio object or not.
> 
> So the switching PF is that there is a switch under all devices?
> 
Right. Switching PF controls the ports of the PF and VFs.

> If we use the switch rx/tx filter for the ip restriction, the PF with switch is
> enough,  we do not need the ip restirction for the PF.
I understood that you for some reason do not need restrictions for the PF.
I do not know why you don't need it. :)
Most cloud setups that I came across so far, needs it, but ok...
The design for the switch object needs to cover the PF as well, even though it may not be done initially.
(hint: an abstraction of switch port to be done, instead of doing things directly on the group member id).

We are seeing use cases reducing of having switch located on the PF for its VFs.
So please reconsider.
I remember you mentioned in past in other thread, that mac etc is controlled from the infrastructure side.
So, I repeatedly ask if you _really_ need to have the switch object as part of the owner PF or not.
Which sort of contradicts with locating the administrative switch on the owner PF.

If it does, flow filters vq that is being worked with Heng, Satananda, David and others seems right direction to implement simple->complex switch object progressively.

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org