Re: [PATCH 1/4] vdpa/mlx5: Fix rule forwarding VLAN to TIR

Re: [PATCH 1/4] vdpa/mlx5: Fix rule forwarding VLAN to TIR

[Si-Wei Liu]

Hi Eli,

It's not for this patch but something related, so just a friendly 
heads-up. I haven't validated the VLAN tagging behavior yet for mlx5 
vdpa, but from my quick read of the code it doesn't seem it 
differentiates the case with and without VIRTIO_NET_F_CTRL_VLAN, to be 
compatible/compliant with what's been implemented in QEMU software (a 
spec addendum was filed as requested by Michael):

https://github.com/oasis-tcs/virtio-spec/issues/147

- when VIRTIO_NET_F_CTRL_VLAN is negotiated, the device starts with
all VLANs filtered (meaning only untagged traffic can be received,
and traffic with VLAN tag will be dropped).

- when VIRTIO_NET_F_CTRL_VLAN is not negotiated, all traffic including
untagged and tagged can be received.

Can you please help check if we need further fix in terms of VLAN tagging?

Thanks,
-Siwei


On 10/18/2022 4:12 AM, Eli Cohen wrote:
> Set the VLAN id to the header values field instead of overwriting the
> headers criteria field.
>
> Before this fix, VLAN filtering would not really work and tagged packets
> would be forwarded unfiltered to the TIR.
>
> Fixes: baf2ad3f6a98 ("vdpa/mlx5: Add RX MAC VLAN filter support")
>
> Signed-off-by: Eli Cohen <elic@nvidia.com>
Reviewed-by: Si-Wei Liu <si-wei.liu@oracle.com>

> ---
>   drivers/vdpa/mlx5/net/mlx5_vnet.c | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c
> index 90913365def4..dd29fdfc24ed 100644
> --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c
> +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c
> @@ -1472,7 +1472,7 @@ static int mlx5_vdpa_add_mac_vlan_rules(struct mlx5_vdpa_net *ndev, u8 *mac,
>   	if (tagged) {
>   		MLX5_SET(fte_match_set_lyr_2_4, headers_v, cvlan_tag, 1);
>   		MLX5_SET_TO_ONES(fte_match_set_lyr_2_4, headers_c, first_vid);
> -		MLX5_SET(fte_match_set_lyr_2_4, headers_c, first_vid, vid);
> +		MLX5_SET(fte_match_set_lyr_2_4, headers_v, first_vid, vid);
>   	}
>   	flow_act.action = MLX5_FLOW_CONTEXT_ACTION_FWD_DEST;
>   	dest.type = MLX5_FLOW_DESTINATION_TYPE_TIR;

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH 1/4] vdpa/mlx5: Fix rule forwarding VLAN to TIR

[Michael S. Tsirkin]

On Wed, Oct 19, 2022 at 05:31:48AM +0000, Eli Cohen wrote:
> > From: Si-Wei Liu <si-wei.liu@oracle.com>
> > Sent: Tuesday, 18 October 2022 22:21
> > To: Eli Cohen <elic@nvidia.com>; mst@redhat.com; jasowang@redhat.com;
> > linux-kernel@vger.kernel.org; virtualization@lists.linux-foundation.org
> > Cc: eperezma@redhat.com; lulu@redhat.com
> > Subject: Re: [PATCH 1/4] vdpa/mlx5: Fix rule forwarding VLAN to TIR
> > 
> > Hi Eli,
> > 
> > It's not for this patch but something related, so just a friendly
> > heads-up. I haven't validated the VLAN tagging behavior yet for mlx5
> > vdpa, but from my quick read of the code it doesn't seem it
> > differentiates the case with and without VIRTIO_NET_F_CTRL_VLAN, to be
> > compatible/compliant with what's been implemented in QEMU software (a
> > spec addendum was filed as requested by Michael):
> > 
> > https://github.com/oasis-tcs/virtio-spec/issues/147
> > 
> > - when VIRTIO_NET_F_CTRL_VLAN is negotiated, the device starts with
> > all VLANs filtered (meaning only untagged traffic can be received,
> > and traffic with VLAN tag will be dropped).
> > 
> > - when VIRTIO_NET_F_CTRL_VLAN is not negotiated, all traffic including
> > untagged and tagged can be received.
> > 
> > Can you please help check if we need further fix in terms of VLAN tagging?
> > 
> 
> Sure. It's broken today. I will fix this to conform to the above requirements and send V1.

Did you mean v2?

> > Thanks,
> > -Siwei
> > 
> > 
> > On 10/18/2022 4:12 AM, Eli Cohen wrote:
> > > Set the VLAN id to the header values field instead of overwriting the
> > > headers criteria field.
> > >
> > > Before this fix, VLAN filtering would not really work and tagged packets
> > > would be forwarded unfiltered to the TIR.
> > >
> > > Fixes: baf2ad3f6a98 ("vdpa/mlx5: Add RX MAC VLAN filter support")
> > >
> > > Signed-off-by: Eli Cohen <elic@nvidia.com>
> > Reviewed-by: Si-Wei Liu <si-wei.liu@oracle.com>
> > 
> > > ---
> > >   drivers/vdpa/mlx5/net/mlx5_vnet.c | 2 +-
> > >   1 file changed, 1 insertion(+), 1 deletion(-)
> > >
> > > diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > b/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > > index 90913365def4..dd29fdfc24ed 100644
> > > --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > > +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > > @@ -1472,7 +1472,7 @@ static int mlx5_vdpa_add_mac_vlan_rules(struct
> > mlx5_vdpa_net *ndev, u8 *mac,
> > >   	if (tagged) {
> > >   		MLX5_SET(fte_match_set_lyr_2_4, headers_v, cvlan_tag, 1);
> > >   		MLX5_SET_TO_ONES(fte_match_set_lyr_2_4, headers_c,
> > first_vid);
> > > -		MLX5_SET(fte_match_set_lyr_2_4, headers_c, first_vid, vid);
> > > +		MLX5_SET(fte_match_set_lyr_2_4, headers_v, first_vid, vid);
> > >   	}
> > >   	flow_act.action = MLX5_FLOW_CONTEXT_ACTION_FWD_DEST;
> > >   	dest.type = MLX5_FLOW_DESTINATION_TYPE_TIR;
> 

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH 1/4] vdpa/mlx5: Fix rule forwarding VLAN to TIR

[Michael S. Tsirkin]

On Wed, Oct 19, 2022 at 05:36:17AM +0000, Eli Cohen wrote:
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Wednesday, 19 October 2022 8:34
> > To: Eli Cohen <elic@nvidia.com>
> > Cc: Si-Wei Liu <si-wei.liu@oracle.com>; jasowang@redhat.com; linux-
> > kernel@vger.kernel.org; virtualization@lists.linux-foundation.org;
> > eperezma@redhat.com; lulu@redhat.com
> > Subject: Re: [PATCH 1/4] vdpa/mlx5: Fix rule forwarding VLAN to TIR
> > 
> > On Wed, Oct 19, 2022 at 05:31:48AM +0000, Eli Cohen wrote:
> > > > From: Si-Wei Liu <si-wei.liu@oracle.com>
> > > > Sent: Tuesday, 18 October 2022 22:21
> > > > To: Eli Cohen <elic@nvidia.com>; mst@redhat.com; jasowang@redhat.com;
> > > > linux-kernel@vger.kernel.org; virtualization@lists.linux-foundation.org
> > > > Cc: eperezma@redhat.com; lulu@redhat.com
> > > > Subject: Re: [PATCH 1/4] vdpa/mlx5: Fix rule forwarding VLAN to TIR
> > > >
> > > > Hi Eli,
> > > >
> > > > It's not for this patch but something related, so just a friendly
> > > > heads-up. I haven't validated the VLAN tagging behavior yet for mlx5
> > > > vdpa, but from my quick read of the code it doesn't seem it
> > > > differentiates the case with and without VIRTIO_NET_F_CTRL_VLAN, to be
> > > > compatible/compliant with what's been implemented in QEMU software (a
> > > > spec addendum was filed as requested by Michael):
> > > >
> > > > https://github.com/oasis-tcs/virtio-spec/issues/147
> > > >
> > > > - when VIRTIO_NET_F_CTRL_VLAN is negotiated, the device starts with
> > > > all VLANs filtered (meaning only untagged traffic can be received,
> > > > and traffic with VLAN tag will be dropped).
> > > >
> > > > - when VIRTIO_NET_F_CTRL_VLAN is not negotiated, all traffic including
> > > > untagged and tagged can be received.
> > > >
> > > > Can you please help check if we need further fix in terms of VLAN tagging?
> > > >
> > >
> > > Sure. It's broken today. I will fix this to conform to the above requirements
> > and send V1.
> > 
> > Did you mean v2?
> > 
> 
> I count from 0 and have been following this scheme but I can make it v2 if that's the norm.

Yes, most people seem to count patches from 1 so [PATCH] is followed by
[PATCH v2].  I don't know why. But it doesn't matter much - I just
wanted to understand whether you will be sending a new version of this
patchset. I know now.


> > > > Thanks,
> > > > -Siwei
> > > >
> > > >
> > > > On 10/18/2022 4:12 AM, Eli Cohen wrote:
> > > > > Set the VLAN id to the header values field instead of overwriting the
> > > > > headers criteria field.
> > > > >
> > > > > Before this fix, VLAN filtering would not really work and tagged packets
> > > > > would be forwarded unfiltered to the TIR.
> > > > >
> > > > > Fixes: baf2ad3f6a98 ("vdpa/mlx5: Add RX MAC VLAN filter support")
> > > > >
> > > > > Signed-off-by: Eli Cohen <elic@nvidia.com>
> > > > Reviewed-by: Si-Wei Liu <si-wei.liu@oracle.com>
> > > >
> > > > > ---
> > > > >   drivers/vdpa/mlx5/net/mlx5_vnet.c | 2 +-
> > > > >   1 file changed, 1 insertion(+), 1 deletion(-)
> > > > >
> > > > > diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > > > b/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > > > > index 90913365def4..dd29fdfc24ed 100644
> > > > > --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > > > > +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > > > > @@ -1472,7 +1472,7 @@ static int
> > mlx5_vdpa_add_mac_vlan_rules(struct
> > > > mlx5_vdpa_net *ndev, u8 *mac,
> > > > >   	if (tagged) {
> > > > >   		MLX5_SET(fte_match_set_lyr_2_4, headers_v, cvlan_tag, 1);
> > > > >   		MLX5_SET_TO_ONES(fte_match_set_lyr_2_4, headers_c,
> > > > first_vid);
> > > > > -		MLX5_SET(fte_match_set_lyr_2_4, headers_c, first_vid, vid);
> > > > > +		MLX5_SET(fte_match_set_lyr_2_4, headers_v, first_vid, vid);
> > > > >   	}
> > > > >   	flow_act.action = MLX5_FLOW_CONTEXT_ACTION_FWD_DEST;
> > > > >   	dest.type = MLX5_FLOW_DESTINATION_TYPE_TIR;
> > >
> 

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH 2/4] vdpa/mlx5: Move some definitions to a new header file

[Si-Wei Liu]

[-- Attachment #1.1: Type: text/plain, Size: 5009 bytes --]



On 10/18/2022 4:12 AM, Eli Cohen wrote:
> Move some definitions from mlx5_vnet.c to newly added header file
> mlx5_vnet.h. We need these definitions for the following patches that
> add debugfs tree to expose information vital for debug.
>
> Signed-off-by: Eli Cohen<elic@nvidia.com>
Reviewed-by: Si-Wei Liu <si-wei.liu@oracle.com>
> ---
>   drivers/vdpa/mlx5/net/mlx5_vnet.c | 45 +------------------------
>   drivers/vdpa/mlx5/net/mlx5_vnet.h | 55 +++++++++++++++++++++++++++++++
>   2 files changed, 56 insertions(+), 44 deletions(-)
>   create mode 100644 drivers/vdpa/mlx5/net/mlx5_vnet.h
>
> diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c
> index dd29fdfc24ed..64fdb940380f 100644
> --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c
> +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c
> @@ -18,15 +18,12 @@
>   #include <linux/mlx5/mlx5_ifc_vdpa.h>
>   #include <linux/mlx5/mpfs.h>
>   #include "mlx5_vdpa.h"
> +#include "mlx5_vnet.h"
>   
>   MODULE_AUTHOR("Eli Cohen<eli@mellanox.com>");
>   MODULE_DESCRIPTION("Mellanox VDPA driver");
>   MODULE_LICENSE("Dual BSD/GPL");
>   
> -#define to_mlx5_vdpa_ndev(__mvdev)                                             \
> -	container_of(__mvdev, struct mlx5_vdpa_net, mvdev)
> -#define to_mvdev(__vdev) container_of((__vdev), struct mlx5_vdpa_dev, vdev)
> -
>   #define VALID_FEATURES_MASK                                                                        \
>   	(BIT_ULL(VIRTIO_NET_F_CSUM) | BIT_ULL(VIRTIO_NET_F_GUEST_CSUM) |                                   \
>   	 BIT_ULL(VIRTIO_NET_F_CTRL_GUEST_OFFLOADS) | BIT_ULL(VIRTIO_NET_F_MTU) | BIT_ULL(VIRTIO_NET_F_MAC) |   \
> @@ -50,14 +47,6 @@ MODULE_LICENSE("Dual BSD/GPL");
>   
>   #define MLX5V_UNTAGGED 0x1000
>   
> -struct mlx5_vdpa_net_resources {
> -	u32 tisn;
> -	u32 tdn;
> -	u32 tirn;
> -	u32 rqtn;
> -	bool valid;
> -};
> -
>   struct mlx5_vdpa_cq_buf {
>   	struct mlx5_frag_buf_ctrl fbc;
>   	struct mlx5_frag_buf frag_buf;
> @@ -146,38 +135,6 @@ static bool is_index_valid(struct mlx5_vdpa_dev *mvdev, u16 idx)
>   	return idx <= mvdev->max_idx;
>   }
>   
> -#define MLX5V_MACVLAN_SIZE 256
> -
> -struct mlx5_vdpa_net {
> -	struct mlx5_vdpa_dev mvdev;
> -	struct mlx5_vdpa_net_resources res;
> -	struct virtio_net_config config;
> -	struct mlx5_vdpa_virtqueue *vqs;
> -	struct vdpa_callback *event_cbs;
> -
> -	/* Serialize vq resources creation and destruction. This is required
> -	 * since memory map might change and we need to destroy and create
> -	 * resources while driver in operational.
> -	 */
> -	struct rw_semaphore reslock;
> -	struct mlx5_flow_table *rxft;
> -	bool setup;
> -	u32 cur_num_vqs;
> -	u32 rqt_size;
> -	bool nb_registered;
> -	struct notifier_block nb;
> -	struct vdpa_callback config_cb;
> -	struct mlx5_vdpa_wq_ent cvq_ent;
> -	struct hlist_head macvlan_hash[MLX5V_MACVLAN_SIZE];
> -};
> -
> -struct macvlan_node {
> -	struct hlist_node hlist;
> -	struct mlx5_flow_handle *ucast_rule;
> -	struct mlx5_flow_handle *mcast_rule;
> -	u64 macvlan;
> -};
> -
>   static void free_resources(struct mlx5_vdpa_net *ndev);
>   static void init_mvqs(struct mlx5_vdpa_net *ndev);
>   static int setup_driver(struct mlx5_vdpa_dev *mvdev);
> diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.h b/drivers/vdpa/mlx5/net/mlx5_vnet.h
> new file mode 100644
> index 000000000000..6691c879a6ca
> --- /dev/null
> +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.h
> @@ -0,0 +1,55 @@
> +/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
> +/* Copyright (c) 2022, NVIDIA CORPORATION & AFFILIATES. All rights reserved. */
> +
> +#ifndef __MLX5_VNET_H__
> +#define __MLX5_VNET_H__
> +
> +#include "mlx5_vdpa.h"
> +
> +#define to_mlx5_vdpa_ndev(__mvdev)                                             \
> +	container_of(__mvdev, struct mlx5_vdpa_net, mvdev)
> +#define to_mvdev(__vdev) container_of((__vdev), struct mlx5_vdpa_dev, vdev)
> +
> +struct mlx5_vdpa_net_resources {
> +	u32 tisn;
> +	u32 tdn;
> +	u32 tirn;
> +	u32 rqtn;
> +	bool valid;
> +};
> +
> +#define MLX5V_MACVLAN_SIZE 256
> +
> +struct mlx5_vdpa_net {
> +	struct mlx5_vdpa_dev mvdev;
> +	struct mlx5_vdpa_net_resources res;
> +	struct virtio_net_config config;
> +	struct mlx5_vdpa_virtqueue *vqs;
> +	struct vdpa_callback *event_cbs;
> +
> +	/* Serialize vq resources creation and destruction. This is required
> +	 * since memory map might change and we need to destroy and create
> +	 * resources while driver in operational.
> +	 */
> +	struct rw_semaphore reslock;
> +	struct mlx5_flow_table *rxft;
> +	struct dentry *rx_dent;
> +	struct dentry *rx_table_dent;
> +	bool setup;
> +	u32 cur_num_vqs;
> +	u32 rqt_size;
> +	bool nb_registered;
> +	struct notifier_block nb;
> +	struct vdpa_callback config_cb;
> +	struct mlx5_vdpa_wq_ent cvq_ent;
> +	struct hlist_head macvlan_hash[MLX5V_MACVLAN_SIZE];
> +};
> +
> +struct macvlan_node {
> +	struct hlist_node hlist;
> +	struct mlx5_flow_handle *ucast_rule;
> +	struct mlx5_flow_handle *mcast_rule;
> +	u64 macvlan;
> +};
> +
> +#endif /* __MLX5_VNET_H__ */

[-- Attachment #1.2: Type: text/html, Size: 5502 bytes --]

[-- Attachment #2: Type: text/plain, Size: 183 bytes --]

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH 4/4] vdpa/mlx5: Add RX counters to debugfs

[Si-Wei Liu]

[-- Attachment #1.1: Type: text/plain, Size: 13635 bytes --]



On 10/18/2022 4:12 AM, Eli Cohen wrote:
> For each interface, either VLAN tagged or untagged, add two hardware
> counters: one for unicast and another for multicast. The counters count
> RX packets and bytes and can be read through debugfs:
>
> $ cat /sys/kernel/debug/mlx5/mlx5_core.sf.1/vdpa-0/rx/untagged/mcast/packets
> $ cat /sys/kernel/debug/mlx5/mlx5_core.sf.1/vdpa-0/rx/untagged/ucast/bytes
>
> This feature is controlled via the config option
> MLX5_VDPA_STEERING_DEBUG. It is off by default as it may have some
> impact on performance.
>
> Signed-off-by: Eli Cohen<elic@nvidia.com>
> ---
>   drivers/vdpa/Kconfig              | 12 ++++
>   drivers/vdpa/mlx5/net/debug.c     | 82 +++++++++++++++++++++++++++
>   drivers/vdpa/mlx5/net/mlx5_vnet.c | 94 +++++++++++++++++++++++--------
>   drivers/vdpa/mlx5/net/mlx5_vnet.h | 30 ++++++++++
>   4 files changed, 195 insertions(+), 23 deletions(-)
>
> diff --git a/drivers/vdpa/Kconfig b/drivers/vdpa/Kconfig
> index 50f45d037611..43b716ec2d18 100644
> --- a/drivers/vdpa/Kconfig
> +++ b/drivers/vdpa/Kconfig
> @@ -71,6 +71,18 @@ config MLX5_VDPA_NET
>   	  be executed by the hardware. It also supports a variety of stateless
>   	  offloads depending on the actual device used and firmware version.
>   
> +config MLX5_VDPA_STEERING_DEBUG
> +	bool "expose steering counters on debugfs"
> +	select MLX5_VDPA
> +	help
> +	  Expose RX steering counters in debugfs to aid in debugging. For each VLAN
> +	  or non VLAN interface, two hardware counters are added to the RX flow
> +	  table: one for unicast and one for multicast.
> +	  The counters counts the number of packets and bytes and exposes them in
> +	  debugfs. Once can read the counters using, e.g.:
> +	  cat /sys/kernel/debug/mlx5/mlx5_core.sf.1/vdpa-0/rx/untagged/ucast/packets
> +	  cat /sys/kernel/debug/mlx5/mlx5_core.sf.1/vdpa-0/rx/untagged/mcast/bytes
> +
>   config VP_VDPA
>   	tristate "Virtio PCI bridge vDPA driver"
>   	select VIRTIO_PCI_LIB
> diff --git a/drivers/vdpa/mlx5/net/debug.c b/drivers/vdpa/mlx5/net/debug.c
> index 95e4801df211..35137a36433a 100644
> --- a/drivers/vdpa/mlx5/net/debug.c
> +++ b/drivers/vdpa/mlx5/net/debug.c
> @@ -49,6 +49,88 @@ void mlx5_vdpa_add_rx_flow_table(struct mlx5_vdpa_net *ndev)
>   						  ndev, &rx_flow_table_fops);
>   }
>   
> +#if defined(CONFIG_MLX5_VDPA_STEERING_DEBUG)
> +static int packets_show(struct seq_file *file, void *priv)
> +{
> +	struct mlx5_vdpa_counter *counter = file->private;
> +	u64 packets;
> +	u64 bytes;
> +	int err;
> +
> +	err = mlx5_fc_query(counter->mdev, counter->counter, &packets, &bytes);
> +	if (err)
> +		return err;
> +
> +	seq_printf(file, "0x%llx\n", packets);
> +	return 0;
> +}
> +
> +static int bytes_show(struct seq_file *file, void *priv)
> +{
> +	struct mlx5_vdpa_counter *counter = file->private;
> +	u64 packets;
> +	u64 bytes;
> +	int err;
> +
> +	err = mlx5_fc_query(counter->mdev, counter->counter, &packets, &bytes);
> +	if (err)
> +		return err;
> +
> +	seq_printf(file, "0x%llx\n", bytes);
> +	return 0;
> +}
> +
> +DEFINE_SHOW_ATTRIBUTE(packets);
> +DEFINE_SHOW_ATTRIBUTE(bytes);
> +
> +static void add_counter_node(struct mlx5_vdpa_counter *counter,
> +			     struct dentry *parent)
> +{
> +	debugfs_create_file("packets", 0444, parent, counter,
> +			    &packets_fops);
> +	debugfs_create_file("bytes", 0444, parent, counter,
> +			    &bytes_fops);
> +}
> +
> +void mlx5_vdpa_add_rx_counters(struct mlx5_vdpa_net *ndev,
> +			       struct macvlan_node *node)
> +{
> +	static const char *ut = "untagged";
> +	char vidstr[9];
> +	u16 vid;
> +
> +	if (node->tagged) {
> +		vid = key2vid(node->macvlan);
> +		snprintf(vidstr, sizeof(vidstr), "0x%x", vid);
> +	} else {
> +		strcpy(vidstr, ut);
> +	}
> +
> +	node->dent = debugfs_create_dir(vidstr, ndev->rx_dent);
> +	if (IS_ERR(node->dent))
> +		return;
> +
> +	node->ucast_counter.dent = debugfs_create_dir("ucast", node->dent);
> +	if (IS_ERR(node->ucast_counter.dent))
> +		return;
> +
> +	add_counter_node(&node->ucast_counter, node->ucast_counter.dent);
> +
> +	node->mcast_counter.dent = debugfs_create_dir("mcast", node->dent);
> +	if (IS_ERR(node->mcast_counter.dent))
> +		return;
> +
> +	add_counter_node(&node->mcast_counter, node->mcast_counter.dent);
> +}
> +
> +void mlx5_vdpa_remove_rx_counters(struct mlx5_vdpa_net *ndev,
> +				  struct macvlan_node *node)
> +{
> +	if (ndev->debugfs)
> +		debugfs_remove_recursive(node->dent);
> +}
> +#endif
> +
>   void mlx5_vdpa_add_debugfs(struct mlx5_vdpa_net *ndev)
>   {
>   	struct mlx5_core_dev *mdev;
> diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c
> index ee50da33e25b..1b850d0aee99 100644
> --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c
> +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c
> @@ -1404,12 +1404,16 @@ static void destroy_tir(struct mlx5_vdpa_net *ndev)
>   #define MAX_STEERING_ENT 0x8000
>   #define MAX_STEERING_GROUPS 2
>   
> +#if defined(CONFIG_MLX5_VDPA_STEERING_DEBUG)
> +       #define NUM_DESTS 2
> +#else
> +       #define NUM_DESTS 1
> +#endif
> +
>   static int mlx5_vdpa_add_mac_vlan_rules(struct mlx5_vdpa_net *ndev, u8 *mac,
> -					u16 vid, bool tagged,
> -					struct mlx5_flow_handle **ucast,
> -					struct mlx5_flow_handle **mcast)
> +					struct macvlan_node *node)
>   {
> -	struct mlx5_flow_destination dest = {};
> +	struct mlx5_flow_destination dests[NUM_DESTS] = {};
>   	struct mlx5_flow_act flow_act = {};
>   	struct mlx5_flow_handle *rule;
>   	struct mlx5_flow_spec *spec;
> @@ -1418,11 +1422,13 @@ static int mlx5_vdpa_add_mac_vlan_rules(struct mlx5_vdpa_net *ndev, u8 *mac,
>   	u8 *dmac_c;
>   	u8 *dmac_v;
>   	int err;
> +	u16 vid;
>   
>   	spec = kvzalloc(sizeof(*spec), GFP_KERNEL);
>   	if (!spec)
>   		return -ENOMEM;
>   
> +	vid = key2vid(node->macvlan);
>   	spec->match_criteria_enable = MLX5_MATCH_OUTER_HEADERS;
>   	headers_c = MLX5_ADDR_OF(fte_match_param, spec->match_criteria, outer_headers);
>   	headers_v = MLX5_ADDR_OF(fte_match_param, spec->match_value, outer_headers);
> @@ -1431,45 +1437,78 @@ static int mlx5_vdpa_add_mac_vlan_rules(struct mlx5_vdpa_net *ndev, u8 *mac,
>   	eth_broadcast_addr(dmac_c);
>   	ether_addr_copy(dmac_v, mac);
>   	MLX5_SET(fte_match_set_lyr_2_4, headers_c, cvlan_tag, 1);
> -	if (tagged) {
> +	if (node->tagged) {
>   		MLX5_SET(fte_match_set_lyr_2_4, headers_v, cvlan_tag, 1);
>   		MLX5_SET_TO_ONES(fte_match_set_lyr_2_4, headers_c, first_vid);
>   		MLX5_SET(fte_match_set_lyr_2_4, headers_v, first_vid, vid);
>   	}
>   	flow_act.action = MLX5_FLOW_CONTEXT_ACTION_FWD_DEST;
> -	dest.type = MLX5_FLOW_DESTINATION_TYPE_TIR;
> -	dest.tir_num = ndev->res.tirn;
> -	rule = mlx5_add_flow_rules(ndev->rxft, spec, &flow_act, &dest, 1);
> +	dests[0].type = MLX5_FLOW_DESTINATION_TYPE_TIR;
> +	dests[0].tir_num = ndev->res.tirn;
> +#if defined(CONFIG_MLX5_VDPA_STEERING_DEBUG)
> +	node->ucast_counter.counter = mlx5_fc_create(ndev->mvdev.mdev, false);
> +	if (IS_ERR(node->ucast_counter.counter)) {
> +		err = PTR_ERR(node->ucast_counter.counter);
> +		goto err_ucast_counter;
> +	}
> +	node->mcast_counter.counter = mlx5_fc_create(ndev->mvdev.mdev, false);
> +	if (IS_ERR(node->mcast_counter.counter)) {
> +		err = PTR_ERR(node->mcast_counter.counter);
> +		goto err_mcast_counter;
> +	}
> +
> +	dests[1].type = MLX5_FLOW_DESTINATION_TYPE_COUNTER;
> +	dests[1].counter_id = mlx5_fc_id(node->ucast_counter.counter);
> +	flow_act.action |= MLX5_FLOW_CONTEXT_ACTION_COUNT;
> +#endif
> +	node->ucast_rule = mlx5_add_flow_rules(ndev->rxft, spec, &flow_act, dests, NUM_DESTS);
>   	if (IS_ERR(rule))
>   		return PTR_ERR(rule);
There'll be leakage on counters and flow spec if using direct return.

>   
> -	*ucast = rule;
> +#if defined(CONFIG_MLX5_VDPA_STEERING_DEBUG)
> +	dests[1].counter_id = mlx5_fc_id(node->mcast_counter.counter);
> +#endif
>   
>   	memset(dmac_c, 0, ETH_ALEN);
>   	memset(dmac_v, 0, ETH_ALEN);
>   	dmac_c[0] = 1;
>   	dmac_v[0] = 1;
> -	rule = mlx5_add_flow_rules(ndev->rxft, spec, &flow_act, &dest, 1);
> +	node->mcast_rule = mlx5_add_flow_rules(ndev->rxft, spec, &flow_act, dests, NUM_DESTS);
>   	kvfree(spec);
Move kvfree a couple of lines under (after the IS_ERR check).
>   	if (IS_ERR(rule)) {
>   		err = PTR_ERR(rule);
>   		goto err_mcast;
>   	}
>   
> -	*mcast = rule;
> +#if defined(CONFIG_MLX5_VDPA_STEERING_DEBUG)
> +	node->ucast_counter.mdev = ndev->mvdev.mdev;
> +	node->mcast_counter.mdev = ndev->mvdev.mdev;
> +#endif
> +	mlx5_vdpa_add_rx_counters(ndev, node);
> +
>   	return 0;
>   
>   err_mcast:
> -	mlx5_del_flow_rules(*ucast);
> +	mlx5_del_flow_rules(node->ucast_rule);
> +#if defined(CONFIG_MLX5_VDPA_STEERING_DEBUG)
No corresponding mlx5_fc_destroy() for mcast_counter.
> +err_mcast_counter:
> +	mlx5_fc_destroy(ndev->mvdev.mdev, node->ucast_counter.counter);
> +err_ucast_counter:
> +	kfree(spec);
kvfree?

Regards,
-Siwei
> +#endif
>   	return err;
>   }
>   
>   static void mlx5_vdpa_del_mac_vlan_rules(struct mlx5_vdpa_net *ndev,
> -					 struct mlx5_flow_handle *ucast,
> -					 struct mlx5_flow_handle *mcast)
> +					 struct macvlan_node *node)
>   {
> -	mlx5_del_flow_rules(ucast);
> -	mlx5_del_flow_rules(mcast);
> +#if defined(CONFIG_MLX5_VDPA_STEERING_DEBUG)
> +	if (node->dent && !IS_ERR(node->dent))
> +		mlx5_vdpa_remove_rx_counters(ndev, node);
> +#endif
> +
> +	mlx5_del_flow_rules(node->ucast_rule);
> +	mlx5_del_flow_rules(node->mcast_rule);
>   }
>   
>   static u64 search_val(u8 *mac, u16 vlan, bool tagged)
> @@ -1503,14 +1542,14 @@ static struct macvlan_node *mac_vlan_lookup(struct mlx5_vdpa_net *ndev, u64 valu
>   	return NULL;
>   }
>   
> -static int mac_vlan_add(struct mlx5_vdpa_net *ndev, u8 *mac, u16 vlan, bool tagged) // vlan -> vid
> +static int mac_vlan_add(struct mlx5_vdpa_net *ndev, u8 *mac, u16 vid, bool tagged)
>   {
>   	struct macvlan_node *ptr;
>   	u64 val;
>   	u32 idx;
>   	int err;
>   
> -	val = search_val(mac, vlan, tagged);
> +	val = search_val(mac, vid, tagged);
>   	if (mac_vlan_lookup(ndev, val))
>   		return -EEXIST;
>   
> @@ -1518,12 +1557,13 @@ static int mac_vlan_add(struct mlx5_vdpa_net *ndev, u8 *mac, u16 vlan, bool tagg
>   	if (!ptr)
>   		return -ENOMEM;
>   
> -	err = mlx5_vdpa_add_mac_vlan_rules(ndev, ndev->config.mac, vlan, tagged,
> -					   &ptr->ucast_rule, &ptr->mcast_rule);
> +	ptr->tagged = tagged;
> +	ptr->macvlan = val;
> +	ptr->ndev = ndev;
> +	err = mlx5_vdpa_add_mac_vlan_rules(ndev, ndev->config.mac, ptr);
>   	if (err)
>   		goto err_add;
>   
> -	ptr->macvlan = val;
>   	idx = hash_64(val, 8);
>   	hlist_add_head(&ptr->hlist, &ndev->macvlan_hash[idx]);
>   	return 0;
> @@ -1542,7 +1582,11 @@ static void mac_vlan_del(struct mlx5_vdpa_net *ndev, u8 *mac, u16 vlan, bool tag
>   		return;
>   
>   	hlist_del(&ptr->hlist);
> -	mlx5_vdpa_del_mac_vlan_rules(ndev, ptr->ucast_rule, ptr->mcast_rule);
> +	mlx5_vdpa_del_mac_vlan_rules(ndev, ptr);
> +#if defined(CONFIG_MLX5_VDPA_STEERING_DEBUG)
> +			mlx5_fc_destroy(ndev->mvdev.mdev, ptr->mcast_counter.counter);
> +			mlx5_fc_destroy(ndev->mvdev.mdev, ptr->ucast_counter.counter);
> +#endif
>   	kfree(ptr);
>   }
>   
> @@ -1555,7 +1599,11 @@ static void clear_mac_vlan_table(struct mlx5_vdpa_net *ndev)
>   	for (i = 0; i < MLX5V_MACVLAN_SIZE; i++) {
>   		hlist_for_each_entry_safe(pos, n, &ndev->macvlan_hash[i], hlist) {
>   			hlist_del(&pos->hlist);
> -			mlx5_vdpa_del_mac_vlan_rules(ndev, pos->ucast_rule, pos->mcast_rule);
> +			mlx5_vdpa_del_mac_vlan_rules(ndev, pos);
> +#if defined(CONFIG_MLX5_VDPA_STEERING_DEBUG)
> +			mlx5_fc_destroy(ndev->mvdev.mdev, pos->mcast_counter.counter);
> +			mlx5_fc_destroy(ndev->mvdev.mdev, pos->ucast_counter.counter);
> +#endif
>   			kfree(pos);
>   		}
>   	}
> diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.h b/drivers/vdpa/mlx5/net/mlx5_vnet.h
> index f2cef3925e5b..706c83016b71 100644
> --- a/drivers/vdpa/mlx5/net/mlx5_vnet.h
> +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.h
> @@ -21,6 +21,11 @@ struct mlx5_vdpa_net_resources {
>   
>   #define MLX5V_MACVLAN_SIZE 256
>   
> +static inline u16 key2vid(u64 key)
> +{
> +	return (u16)(key >> 48);
> +}
> +
>   struct mlx5_vdpa_net {
>   	struct mlx5_vdpa_dev mvdev;
>   	struct mlx5_vdpa_net_resources res;
> @@ -47,11 +52,24 @@ struct mlx5_vdpa_net {
>   	struct dentry *debugfs;
>   };
>   
> +struct mlx5_vdpa_counter {
> +	struct mlx5_fc *counter;
> +	struct dentry *dent;
> +	struct mlx5_core_dev *mdev;
> +};
> +
>   struct macvlan_node {
>   	struct hlist_node hlist;
>   	struct mlx5_flow_handle *ucast_rule;
>   	struct mlx5_flow_handle *mcast_rule;
>   	u64 macvlan;
> +	struct mlx5_vdpa_net *ndev;
> +	bool tagged;
> +#if defined(CONFIG_MLX5_VDPA_STEERING_DEBUG)
> +	struct dentry *dent;
> +	struct mlx5_vdpa_counter ucast_counter;
> +	struct mlx5_vdpa_counter mcast_counter;
> +#endif
>   };
>   
>   void mlx5_vdpa_add_debugfs(struct mlx5_vdpa_net *ndev);
> @@ -60,5 +78,17 @@ void mlx5_vdpa_add_rx_flow_table(struct mlx5_vdpa_net *ndev);
>   void mlx5_vdpa_remove_rx_flow_table(struct mlx5_vdpa_net *ndev);
>   void mlx5_vdpa_add_tirn(struct mlx5_vdpa_net *ndev);
>   void mlx5_vdpa_remove_tirn(struct mlx5_vdpa_net *ndev);
> +#if defined(CONFIG_MLX5_VDPA_STEERING_DEBUG)
> +void mlx5_vdpa_add_rx_counters(struct mlx5_vdpa_net *ndev,
> +			       struct macvlan_node *node);
> +void mlx5_vdpa_remove_rx_counters(struct mlx5_vdpa_net *ndev,
> +				  struct macvlan_node *node);
> +#else
> +static inline void mlx5_vdpa_add_rx_counters(struct mlx5_vdpa_net *ndev,
> +					     struct macvlan_node *node) {}
> +static inline void mlx5_vdpa_remove_rx_counters(struct mlx5_vdpa_net *ndev,
> +						struct macvlan_node *node) {}
> +#endif
> +
>   
>   #endif /* __MLX5_VNET_H__ */

[-- Attachment #1.2: Type: text/html, Size: 14254 bytes --]

[-- Attachment #2: Type: text/plain, Size: 183 bytes --]

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH 3/4] vdpa/mlx5: Add debugfs subtree

[Si-Wei Liu]

On 10/18/2022 4:12 AM, Eli Cohen wrote:
> Add debugfs subtree and expose flow table ID and TIR number. This
> information can be used by external tools to do extended
> troubleshooting.
>
> The information can be retrieved like so:
> $ cat /sys/kernel/debug/mlx5/mlx5_core.sf.1/vdpa-0/rx/table_id
> $ cat /sys/kernel/debug/mlx5/mlx5_core.sf.1/vdpa-0/rx/tirn
>
> Signed-off-by: Eli Cohen <elic@nvidia.com>
LGTM

Reviewed-by: Si-Wei Liu <si-wei.liu@oracle.com>
> ---
>   drivers/vdpa/mlx5/Makefile        |  2 +-
>   drivers/vdpa/mlx5/net/debug.c     | 66 +++++++++++++++++++++++++++++++
>   drivers/vdpa/mlx5/net/mlx5_vnet.c | 11 ++++++
>   drivers/vdpa/mlx5/net/mlx5_vnet.h |  9 +++++
>   4 files changed, 87 insertions(+), 1 deletion(-)
>   create mode 100644 drivers/vdpa/mlx5/net/debug.c
>
> diff --git a/drivers/vdpa/mlx5/Makefile b/drivers/vdpa/mlx5/Makefile
> index f717978c83bf..e791394c33e3 100644
> --- a/drivers/vdpa/mlx5/Makefile
> +++ b/drivers/vdpa/mlx5/Makefile
> @@ -1,4 +1,4 @@
>   subdir-ccflags-y += -I$(srctree)/drivers/vdpa/mlx5/core
>   
>   obj-$(CONFIG_MLX5_VDPA_NET) += mlx5_vdpa.o
> -mlx5_vdpa-$(CONFIG_MLX5_VDPA_NET) += net/mlx5_vnet.o core/resources.o core/mr.o
> +mlx5_vdpa-$(CONFIG_MLX5_VDPA_NET) += net/mlx5_vnet.o core/resources.o core/mr.o net/debug.o
> diff --git a/drivers/vdpa/mlx5/net/debug.c b/drivers/vdpa/mlx5/net/debug.c
> new file mode 100644
> index 000000000000..95e4801df211
> --- /dev/null
> +++ b/drivers/vdpa/mlx5/net/debug.c
> @@ -0,0 +1,66 @@
> +// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
> +/* Copyright (c) 2022, NVIDIA CORPORATION & AFFILIATES. All rights reserved. */
> +
> +#include <linux/debugfs.h>
> +#include <linux/mlx5/fs.h>
> +#include "mlx5_vnet.h"
> +
> +static int tirn_show(struct seq_file *file, void *priv)
> +{
> +	struct mlx5_vdpa_net *ndev = file->private;
> +
> +	seq_printf(file, "0x%x\n", ndev->res.tirn);
> +	return 0;
> +}
> +
> +DEFINE_SHOW_ATTRIBUTE(tirn);
> +
> +void mlx5_vdpa_remove_tirn(struct mlx5_vdpa_net *ndev)
> +{
> +	if (ndev->debugfs)
> +		debugfs_remove(ndev->res.tirn_dent);
> +}
> +
> +void mlx5_vdpa_add_tirn(struct mlx5_vdpa_net *ndev)
> +{
> +	ndev->res.tirn_dent = debugfs_create_file("tirn", 0444, ndev->rx_dent,
> +						  ndev, &tirn_fops);
> +}
> +
> +static int rx_flow_table_show(struct seq_file *file, void *priv)
> +{
> +	struct mlx5_vdpa_net *ndev = file->private;
> +
> +	seq_printf(file, "0x%x\n", mlx5_flow_table_id(ndev->rxft));
> +	return 0;
> +}
> +
> +DEFINE_SHOW_ATTRIBUTE(rx_flow_table);
> +
> +void mlx5_vdpa_remove_rx_flow_table(struct mlx5_vdpa_net *ndev)
> +{
> +	if (ndev->debugfs)
> +		debugfs_remove(ndev->rx_table_dent);
> +}
> +
> +void mlx5_vdpa_add_rx_flow_table(struct mlx5_vdpa_net *ndev)
> +{
> +	ndev->rx_table_dent = debugfs_create_file("table_id", 0444, ndev->rx_dent,
> +						  ndev, &rx_flow_table_fops);
> +}
> +
> +void mlx5_vdpa_add_debugfs(struct mlx5_vdpa_net *ndev)
> +{
> +	struct mlx5_core_dev *mdev;
> +
> +	mdev = ndev->mvdev.mdev;
> +	ndev->debugfs = debugfs_create_dir(dev_name(&ndev->mvdev.vdev.dev),
> +					   mlx5_debugfs_get_dev_root(mdev));
> +	if (!IS_ERR(ndev->debugfs))
> +		ndev->rx_dent = debugfs_create_dir("rx", ndev->debugfs);
> +}
> +
> +void mlx5_vdpa_remove_debugfs(struct dentry *dbg)
> +{
> +	debugfs_remove_recursive(dbg);
> +}
> diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c
> index 64fdb940380f..ee50da33e25b 100644
> --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c
> +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c
> @@ -1388,11 +1388,16 @@ static int create_tir(struct mlx5_vdpa_net *ndev)
>   
>   	err = mlx5_vdpa_create_tir(&ndev->mvdev, in, &ndev->res.tirn);
>   	kfree(in);
> +	if (err)
> +		return err;
> +
> +	mlx5_vdpa_add_tirn(ndev);
>   	return err;
>   }
>   
>   static void destroy_tir(struct mlx5_vdpa_net *ndev)
>   {
> +	mlx5_vdpa_remove_tirn(ndev);
>   	mlx5_vdpa_destroy_tir(&ndev->mvdev, ndev->res.tirn);
>   }
>   
> @@ -1576,6 +1581,7 @@ static int setup_steering(struct mlx5_vdpa_net *ndev)
>   		mlx5_vdpa_warn(&ndev->mvdev, "failed to create flow table\n");
>   		return PTR_ERR(ndev->rxft);
>   	}
> +	mlx5_vdpa_add_rx_flow_table(ndev);
>   
>   	err = mac_vlan_add(ndev, ndev->config.mac, 0, false);
>   	if (err)
> @@ -1584,6 +1590,7 @@ static int setup_steering(struct mlx5_vdpa_net *ndev)
>   	return 0;
>   
>   err_add:
> +	mlx5_vdpa_remove_rx_flow_table(ndev);
>   	mlx5_destroy_flow_table(ndev->rxft);
>   	return err;
>   }
> @@ -1591,6 +1598,7 @@ static int setup_steering(struct mlx5_vdpa_net *ndev)
>   static void teardown_steering(struct mlx5_vdpa_net *ndev)
>   {
>   	clear_mac_vlan_table(ndev);
> +	mlx5_vdpa_remove_rx_flow_table(ndev);
>   	mlx5_destroy_flow_table(ndev->rxft);
>   }
>   
> @@ -3167,6 +3175,7 @@ static int mlx5_vdpa_dev_add(struct vdpa_mgmt_dev *v_mdev, const char *name,
>   	if (err)
>   		goto err_reg;
>   
> +	mlx5_vdpa_add_debugfs(ndev);
>   	mgtdev->ndev = ndev;
>   	return 0;
>   
> @@ -3193,6 +3202,8 @@ static void mlx5_vdpa_dev_del(struct vdpa_mgmt_dev *v_mdev, struct vdpa_device *
>   	struct mlx5_vdpa_net *ndev = to_mlx5_vdpa_ndev(mvdev);
>   	struct workqueue_struct *wq;
>   
> +	mlx5_vdpa_remove_debugfs(ndev->debugfs);
> +	ndev->debugfs = NULL;
>   	if (ndev->nb_registered) {
>   		mlx5_notifier_unregister(mvdev->mdev, &ndev->nb);
>   		ndev->nb_registered = false;
> diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.h b/drivers/vdpa/mlx5/net/mlx5_vnet.h
> index 6691c879a6ca..f2cef3925e5b 100644
> --- a/drivers/vdpa/mlx5/net/mlx5_vnet.h
> +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.h
> @@ -16,6 +16,7 @@ struct mlx5_vdpa_net_resources {
>   	u32 tirn;
>   	u32 rqtn;
>   	bool valid;
> +	struct dentry *tirn_dent;
>   };
>   
>   #define MLX5V_MACVLAN_SIZE 256
> @@ -43,6 +44,7 @@ struct mlx5_vdpa_net {
>   	struct vdpa_callback config_cb;
>   	struct mlx5_vdpa_wq_ent cvq_ent;
>   	struct hlist_head macvlan_hash[MLX5V_MACVLAN_SIZE];
> +	struct dentry *debugfs;
>   };
>   
>   struct macvlan_node {
> @@ -52,4 +54,11 @@ struct macvlan_node {
>   	u64 macvlan;
>   };
>   
> +void mlx5_vdpa_add_debugfs(struct mlx5_vdpa_net *ndev);
> +void mlx5_vdpa_remove_debugfs(struct dentry *dbg);
> +void mlx5_vdpa_add_rx_flow_table(struct mlx5_vdpa_net *ndev);
> +void mlx5_vdpa_remove_rx_flow_table(struct mlx5_vdpa_net *ndev);
> +void mlx5_vdpa_add_tirn(struct mlx5_vdpa_net *ndev);
> +void mlx5_vdpa_remove_tirn(struct mlx5_vdpa_net *ndev);
> +
>   #endif /* __MLX5_VNET_H__ */

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH 3/4] vdpa/mlx5: Add debugfs subtree

[Jason Wang]

在 2022/10/18 19:12, Eli Cohen 写道:
> Add debugfs subtree and expose flow table ID and TIR number. This
> information can be used by external tools to do extended
> troubleshooting.
>
> The information can be retrieved like so:
> $ cat /sys/kernel/debug/mlx5/mlx5_core.sf.1/vdpa-0/rx/table_id
> $ cat /sys/kernel/debug/mlx5/mlx5_core.sf.1/vdpa-0/rx/tirn


I wonder if we need a config option for this.

And if it's better to use vendor stats.

Thanks


>
> Signed-off-by: Eli Cohen <elic@nvidia.com>
> ---
>   drivers/vdpa/mlx5/Makefile        |  2 +-
>   drivers/vdpa/mlx5/net/debug.c     | 66 +++++++++++++++++++++++++++++++
>   drivers/vdpa/mlx5/net/mlx5_vnet.c | 11 ++++++
>   drivers/vdpa/mlx5/net/mlx5_vnet.h |  9 +++++
>   4 files changed, 87 insertions(+), 1 deletion(-)
>   create mode 100644 drivers/vdpa/mlx5/net/debug.c
>
> diff --git a/drivers/vdpa/mlx5/Makefile b/drivers/vdpa/mlx5/Makefile
> index f717978c83bf..e791394c33e3 100644
> --- a/drivers/vdpa/mlx5/Makefile
> +++ b/drivers/vdpa/mlx5/Makefile
> @@ -1,4 +1,4 @@
>   subdir-ccflags-y += -I$(srctree)/drivers/vdpa/mlx5/core
>   
>   obj-$(CONFIG_MLX5_VDPA_NET) += mlx5_vdpa.o
> -mlx5_vdpa-$(CONFIG_MLX5_VDPA_NET) += net/mlx5_vnet.o core/resources.o core/mr.o
> +mlx5_vdpa-$(CONFIG_MLX5_VDPA_NET) += net/mlx5_vnet.o core/resources.o core/mr.o net/debug.o
> diff --git a/drivers/vdpa/mlx5/net/debug.c b/drivers/vdpa/mlx5/net/debug.c
> new file mode 100644
> index 000000000000..95e4801df211
> --- /dev/null
> +++ b/drivers/vdpa/mlx5/net/debug.c
> @@ -0,0 +1,66 @@
> +// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
> +/* Copyright (c) 2022, NVIDIA CORPORATION & AFFILIATES. All rights reserved. */
> +
> +#include <linux/debugfs.h>
> +#include <linux/mlx5/fs.h>
> +#include "mlx5_vnet.h"
> +
> +static int tirn_show(struct seq_file *file, void *priv)
> +{
> +	struct mlx5_vdpa_net *ndev = file->private;
> +
> +	seq_printf(file, "0x%x\n", ndev->res.tirn);
> +	return 0;
> +}
> +
> +DEFINE_SHOW_ATTRIBUTE(tirn);
> +
> +void mlx5_vdpa_remove_tirn(struct mlx5_vdpa_net *ndev)
> +{
> +	if (ndev->debugfs)
> +		debugfs_remove(ndev->res.tirn_dent);
> +}
> +
> +void mlx5_vdpa_add_tirn(struct mlx5_vdpa_net *ndev)
> +{
> +	ndev->res.tirn_dent = debugfs_create_file("tirn", 0444, ndev->rx_dent,
> +						  ndev, &tirn_fops);
> +}
> +
> +static int rx_flow_table_show(struct seq_file *file, void *priv)
> +{
> +	struct mlx5_vdpa_net *ndev = file->private;
> +
> +	seq_printf(file, "0x%x\n", mlx5_flow_table_id(ndev->rxft));
> +	return 0;
> +}
> +
> +DEFINE_SHOW_ATTRIBUTE(rx_flow_table);
> +
> +void mlx5_vdpa_remove_rx_flow_table(struct mlx5_vdpa_net *ndev)
> +{
> +	if (ndev->debugfs)
> +		debugfs_remove(ndev->rx_table_dent);
> +}
> +
> +void mlx5_vdpa_add_rx_flow_table(struct mlx5_vdpa_net *ndev)
> +{
> +	ndev->rx_table_dent = debugfs_create_file("table_id", 0444, ndev->rx_dent,
> +						  ndev, &rx_flow_table_fops);
> +}
> +
> +void mlx5_vdpa_add_debugfs(struct mlx5_vdpa_net *ndev)
> +{
> +	struct mlx5_core_dev *mdev;
> +
> +	mdev = ndev->mvdev.mdev;
> +	ndev->debugfs = debugfs_create_dir(dev_name(&ndev->mvdev.vdev.dev),
> +					   mlx5_debugfs_get_dev_root(mdev));
> +	if (!IS_ERR(ndev->debugfs))
> +		ndev->rx_dent = debugfs_create_dir("rx", ndev->debugfs);
> +}
> +
> +void mlx5_vdpa_remove_debugfs(struct dentry *dbg)
> +{
> +	debugfs_remove_recursive(dbg);
> +}
> diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c
> index 64fdb940380f..ee50da33e25b 100644
> --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c
> +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c
> @@ -1388,11 +1388,16 @@ static int create_tir(struct mlx5_vdpa_net *ndev)
>   
>   	err = mlx5_vdpa_create_tir(&ndev->mvdev, in, &ndev->res.tirn);
>   	kfree(in);
> +	if (err)
> +		return err;
> +
> +	mlx5_vdpa_add_tirn(ndev);
>   	return err;
>   }
>   
>   static void destroy_tir(struct mlx5_vdpa_net *ndev)
>   {
> +	mlx5_vdpa_remove_tirn(ndev);
>   	mlx5_vdpa_destroy_tir(&ndev->mvdev, ndev->res.tirn);
>   }
>   
> @@ -1576,6 +1581,7 @@ static int setup_steering(struct mlx5_vdpa_net *ndev)
>   		mlx5_vdpa_warn(&ndev->mvdev, "failed to create flow table\n");
>   		return PTR_ERR(ndev->rxft);
>   	}
> +	mlx5_vdpa_add_rx_flow_table(ndev);
>   
>   	err = mac_vlan_add(ndev, ndev->config.mac, 0, false);
>   	if (err)
> @@ -1584,6 +1590,7 @@ static int setup_steering(struct mlx5_vdpa_net *ndev)
>   	return 0;
>   
>   err_add:
> +	mlx5_vdpa_remove_rx_flow_table(ndev);
>   	mlx5_destroy_flow_table(ndev->rxft);
>   	return err;
>   }
> @@ -1591,6 +1598,7 @@ static int setup_steering(struct mlx5_vdpa_net *ndev)
>   static void teardown_steering(struct mlx5_vdpa_net *ndev)
>   {
>   	clear_mac_vlan_table(ndev);
> +	mlx5_vdpa_remove_rx_flow_table(ndev);
>   	mlx5_destroy_flow_table(ndev->rxft);
>   }
>   
> @@ -3167,6 +3175,7 @@ static int mlx5_vdpa_dev_add(struct vdpa_mgmt_dev *v_mdev, const char *name,
>   	if (err)
>   		goto err_reg;
>   
> +	mlx5_vdpa_add_debugfs(ndev);
>   	mgtdev->ndev = ndev;
>   	return 0;
>   
> @@ -3193,6 +3202,8 @@ static void mlx5_vdpa_dev_del(struct vdpa_mgmt_dev *v_mdev, struct vdpa_device *
>   	struct mlx5_vdpa_net *ndev = to_mlx5_vdpa_ndev(mvdev);
>   	struct workqueue_struct *wq;
>   
> +	mlx5_vdpa_remove_debugfs(ndev->debugfs);
> +	ndev->debugfs = NULL;
>   	if (ndev->nb_registered) {
>   		mlx5_notifier_unregister(mvdev->mdev, &ndev->nb);
>   		ndev->nb_registered = false;
> diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.h b/drivers/vdpa/mlx5/net/mlx5_vnet.h
> index 6691c879a6ca..f2cef3925e5b 100644
> --- a/drivers/vdpa/mlx5/net/mlx5_vnet.h
> +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.h
> @@ -16,6 +16,7 @@ struct mlx5_vdpa_net_resources {
>   	u32 tirn;
>   	u32 rqtn;
>   	bool valid;
> +	struct dentry *tirn_dent;
>   };
>   
>   #define MLX5V_MACVLAN_SIZE 256
> @@ -43,6 +44,7 @@ struct mlx5_vdpa_net {
>   	struct vdpa_callback config_cb;
>   	struct mlx5_vdpa_wq_ent cvq_ent;
>   	struct hlist_head macvlan_hash[MLX5V_MACVLAN_SIZE];
> +	struct dentry *debugfs;
>   };
>   
>   struct macvlan_node {
> @@ -52,4 +54,11 @@ struct macvlan_node {
>   	u64 macvlan;
>   };
>   
> +void mlx5_vdpa_add_debugfs(struct mlx5_vdpa_net *ndev);
> +void mlx5_vdpa_remove_debugfs(struct dentry *dbg);
> +void mlx5_vdpa_add_rx_flow_table(struct mlx5_vdpa_net *ndev);
> +void mlx5_vdpa_remove_rx_flow_table(struct mlx5_vdpa_net *ndev);
> +void mlx5_vdpa_add_tirn(struct mlx5_vdpa_net *ndev);
> +void mlx5_vdpa_remove_tirn(struct mlx5_vdpa_net *ndev);
> +
>   #endif /* __MLX5_VNET_H__ */

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH 3/4] vdpa/mlx5: Add debugfs subtree

[Jason Wang]

On Wed, Oct 19, 2022 at 1:38 PM Eli Cohen <elic@nvidia.com> wrote:
>
> > From: Jason Wang <jasowang@redhat.com>
> > Sent: Wednesday, 19 October 2022 8:14
> > To: Eli Cohen <elic@nvidia.com>; mst@redhat.com; linux-
> > kernel@vger.kernel.org; virtualization@lists.linux-foundation.org
> > Cc: si-wei.liu@oracle.com; eperezma@redhat.com; lulu@redhat.com
> > Subject: Re: [PATCH 3/4] vdpa/mlx5: Add debugfs subtree
> >
> >
> > 在 2022/10/18 19:12, Eli Cohen 写道:
> > > Add debugfs subtree and expose flow table ID and TIR number. This
> > > information can be used by external tools to do extended
> > > troubleshooting.
> > >
> > > The information can be retrieved like so:
> > > $ cat /sys/kernel/debug/mlx5/mlx5_core.sf.1/vdpa-0/rx/table_id
> > > $ cat /sys/kernel/debug/mlx5/mlx5_core.sf.1/vdpa-0/rx/tirn
> >
> >
> > I wonder if we need a config option for this.
> >
> > And if it's better to use vendor stats.
> >
>
> I am not sure. TIRs and flow tables has loose association VDPA. I would keep it in
> debugfs which for mlx5 there's lots of other information there.

That's fine.

Thanks

>
> > Thanks
> >
> >
> > >
> > > Signed-off-by: Eli Cohen <elic@nvidia.com>
> > > ---
> > >   drivers/vdpa/mlx5/Makefile        |  2 +-
> > >   drivers/vdpa/mlx5/net/debug.c     | 66
> > +++++++++++++++++++++++++++++++
> > >   drivers/vdpa/mlx5/net/mlx5_vnet.c | 11 ++++++
> > >   drivers/vdpa/mlx5/net/mlx5_vnet.h |  9 +++++
> > >   4 files changed, 87 insertions(+), 1 deletion(-)
> > >   create mode 100644 drivers/vdpa/mlx5/net/debug.c
> > >
> > > diff --git a/drivers/vdpa/mlx5/Makefile b/drivers/vdpa/mlx5/Makefile
> > > index f717978c83bf..e791394c33e3 100644
> > > --- a/drivers/vdpa/mlx5/Makefile
> > > +++ b/drivers/vdpa/mlx5/Makefile
> > > @@ -1,4 +1,4 @@
> > >   subdir-ccflags-y += -I$(srctree)/drivers/vdpa/mlx5/core
> > >
> > >   obj-$(CONFIG_MLX5_VDPA_NET) += mlx5_vdpa.o
> > > -mlx5_vdpa-$(CONFIG_MLX5_VDPA_NET) += net/mlx5_vnet.o
> > core/resources.o core/mr.o
> > > +mlx5_vdpa-$(CONFIG_MLX5_VDPA_NET) += net/mlx5_vnet.o
> > core/resources.o core/mr.o net/debug.o
> > > diff --git a/drivers/vdpa/mlx5/net/debug.c b/drivers/vdpa/mlx5/net/debug.c
> > > new file mode 100644
> > > index 000000000000..95e4801df211
> > > --- /dev/null
> > > +++ b/drivers/vdpa/mlx5/net/debug.c
> > > @@ -0,0 +1,66 @@
> > > +// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
> > > +/* Copyright (c) 2022, NVIDIA CORPORATION & AFFILIATES. All rights
> > reserved. */
> > > +
> > > +#include <linux/debugfs.h>
> > > +#include <linux/mlx5/fs.h>
> > > +#include "mlx5_vnet.h"
> > > +
> > > +static int tirn_show(struct seq_file *file, void *priv)
> > > +{
> > > +   struct mlx5_vdpa_net *ndev = file->private;
> > > +
> > > +   seq_printf(file, "0x%x\n", ndev->res.tirn);
> > > +   return 0;
> > > +}
> > > +
> > > +DEFINE_SHOW_ATTRIBUTE(tirn);
> > > +
> > > +void mlx5_vdpa_remove_tirn(struct mlx5_vdpa_net *ndev)
> > > +{
> > > +   if (ndev->debugfs)
> > > +           debugfs_remove(ndev->res.tirn_dent);
> > > +}
> > > +
> > > +void mlx5_vdpa_add_tirn(struct mlx5_vdpa_net *ndev)
> > > +{
> > > +   ndev->res.tirn_dent = debugfs_create_file("tirn", 0444, ndev-
> > >rx_dent,
> > > +                                             ndev, &tirn_fops);
> > > +}
> > > +
> > > +static int rx_flow_table_show(struct seq_file *file, void *priv)
> > > +{
> > > +   struct mlx5_vdpa_net *ndev = file->private;
> > > +
> > > +   seq_printf(file, "0x%x\n", mlx5_flow_table_id(ndev->rxft));
> > > +   return 0;
> > > +}
> > > +
> > > +DEFINE_SHOW_ATTRIBUTE(rx_flow_table);
> > > +
> > > +void mlx5_vdpa_remove_rx_flow_table(struct mlx5_vdpa_net *ndev)
> > > +{
> > > +   if (ndev->debugfs)
> > > +           debugfs_remove(ndev->rx_table_dent);
> > > +}
> > > +
> > > +void mlx5_vdpa_add_rx_flow_table(struct mlx5_vdpa_net *ndev)
> > > +{
> > > +   ndev->rx_table_dent = debugfs_create_file("table_id", 0444, ndev-
> > >rx_dent,
> > > +                                             ndev, &rx_flow_table_fops);
> > > +}
> > > +
> > > +void mlx5_vdpa_add_debugfs(struct mlx5_vdpa_net *ndev)
> > > +{
> > > +   struct mlx5_core_dev *mdev;
> > > +
> > > +   mdev = ndev->mvdev.mdev;
> > > +   ndev->debugfs = debugfs_create_dir(dev_name(&ndev-
> > >mvdev.vdev.dev),
> > > +
> > mlx5_debugfs_get_dev_root(mdev));
> > > +   if (!IS_ERR(ndev->debugfs))
> > > +           ndev->rx_dent = debugfs_create_dir("rx", ndev->debugfs);
> > > +}
> > > +
> > > +void mlx5_vdpa_remove_debugfs(struct dentry *dbg)
> > > +{
> > > +   debugfs_remove_recursive(dbg);
> > > +}
> > > diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > b/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > > index 64fdb940380f..ee50da33e25b 100644
> > > --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > > +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > > @@ -1388,11 +1388,16 @@ static int create_tir(struct mlx5_vdpa_net
> > *ndev)
> > >
> > >     err = mlx5_vdpa_create_tir(&ndev->mvdev, in, &ndev->res.tirn);
> > >     kfree(in);
> > > +   if (err)
> > > +           return err;
> > > +
> > > +   mlx5_vdpa_add_tirn(ndev);
> > >     return err;
> > >   }
> > >
> > >   static void destroy_tir(struct mlx5_vdpa_net *ndev)
> > >   {
> > > +   mlx5_vdpa_remove_tirn(ndev);
> > >     mlx5_vdpa_destroy_tir(&ndev->mvdev, ndev->res.tirn);
> > >   }
> > >
> > > @@ -1576,6 +1581,7 @@ static int setup_steering(struct mlx5_vdpa_net
> > *ndev)
> > >             mlx5_vdpa_warn(&ndev->mvdev, "failed to create flow
> > table\n");
> > >             return PTR_ERR(ndev->rxft);
> > >     }
> > > +   mlx5_vdpa_add_rx_flow_table(ndev);
> > >
> > >     err = mac_vlan_add(ndev, ndev->config.mac, 0, false);
> > >     if (err)
> > > @@ -1584,6 +1590,7 @@ static int setup_steering(struct mlx5_vdpa_net
> > *ndev)
> > >     return 0;
> > >
> > >   err_add:
> > > +   mlx5_vdpa_remove_rx_flow_table(ndev);
> > >     mlx5_destroy_flow_table(ndev->rxft);
> > >     return err;
> > >   }
> > > @@ -1591,6 +1598,7 @@ static int setup_steering(struct mlx5_vdpa_net
> > *ndev)
> > >   static void teardown_steering(struct mlx5_vdpa_net *ndev)
> > >   {
> > >     clear_mac_vlan_table(ndev);
> > > +   mlx5_vdpa_remove_rx_flow_table(ndev);
> > >     mlx5_destroy_flow_table(ndev->rxft);
> > >   }
> > >
> > > @@ -3167,6 +3175,7 @@ static int mlx5_vdpa_dev_add(struct
> > vdpa_mgmt_dev *v_mdev, const char *name,
> > >     if (err)
> > >             goto err_reg;
> > >
> > > +   mlx5_vdpa_add_debugfs(ndev);
> > >     mgtdev->ndev = ndev;
> > >     return 0;
> > >
> > > @@ -3193,6 +3202,8 @@ static void mlx5_vdpa_dev_del(struct
> > vdpa_mgmt_dev *v_mdev, struct vdpa_device *
> > >     struct mlx5_vdpa_net *ndev = to_mlx5_vdpa_ndev(mvdev);
> > >     struct workqueue_struct *wq;
> > >
> > > +   mlx5_vdpa_remove_debugfs(ndev->debugfs);
> > > +   ndev->debugfs = NULL;
> > >     if (ndev->nb_registered) {
> > >             mlx5_notifier_unregister(mvdev->mdev, &ndev->nb);
> > >             ndev->nb_registered = false;
> > > diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.h
> > b/drivers/vdpa/mlx5/net/mlx5_vnet.h
> > > index 6691c879a6ca..f2cef3925e5b 100644
> > > --- a/drivers/vdpa/mlx5/net/mlx5_vnet.h
> > > +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.h
> > > @@ -16,6 +16,7 @@ struct mlx5_vdpa_net_resources {
> > >     u32 tirn;
> > >     u32 rqtn;
> > >     bool valid;
> > > +   struct dentry *tirn_dent;
> > >   };
> > >
> > >   #define MLX5V_MACVLAN_SIZE 256
> > > @@ -43,6 +44,7 @@ struct mlx5_vdpa_net {
> > >     struct vdpa_callback config_cb;
> > >     struct mlx5_vdpa_wq_ent cvq_ent;
> > >     struct hlist_head macvlan_hash[MLX5V_MACVLAN_SIZE];
> > > +   struct dentry *debugfs;
> > >   };
> > >
> > >   struct macvlan_node {
> > > @@ -52,4 +54,11 @@ struct macvlan_node {
> > >     u64 macvlan;
> > >   };
> > >
> > > +void mlx5_vdpa_add_debugfs(struct mlx5_vdpa_net *ndev);
> > > +void mlx5_vdpa_remove_debugfs(struct dentry *dbg);
> > > +void mlx5_vdpa_add_rx_flow_table(struct mlx5_vdpa_net *ndev);
> > > +void mlx5_vdpa_remove_rx_flow_table(struct mlx5_vdpa_net *ndev);
> > > +void mlx5_vdpa_add_tirn(struct mlx5_vdpa_net *ndev);
> > > +void mlx5_vdpa_remove_tirn(struct mlx5_vdpa_net *ndev);
> > > +
> > >   #endif /* __MLX5_VNET_H__ */
>

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization