[PATCH 0/4] vdpa:/mlx5: Add debugfs subtree

[PATCH 0/4] vdpa:/mlx5: Add debugfs subtree

[Eli Cohen]

First patch fixes a copy paste error causing VLAN tagged packets to no
be filtered.

Following three patches add debugfs subtree.
First one just moves definitions to a new header file. These definitions
are needed for following patches.
Second patch adds entries to read flow table number and TIR number.
Third patch adds multicast and unicast packets and bytes counters for
for each untagged or VLAN tagged incoming traffic. This is depends on
MLX5_VDPA_STEERING_DEBUG config option and is off by default.

Eli Cohen (4):
  vdpa/mlx5: Fix rule forwarding VLAN to TIR
  vdpa/mlx5: Move some definitions to a new header file
  vdpa/mlx5: Add debugfs subtree
  vdpa/mlx5: Add RX counters to debugfs

 drivers/vdpa/Kconfig              |  12 +++
 drivers/vdpa/mlx5/Makefile        |   2 +-
 drivers/vdpa/mlx5/net/debug.c     | 148 +++++++++++++++++++++++++++++
 drivers/vdpa/mlx5/net/mlx5_vnet.c | 152 +++++++++++++++++-------------
 drivers/vdpa/mlx5/net/mlx5_vnet.h |  94 ++++++++++++++++++
 5 files changed, 339 insertions(+), 69 deletions(-)
 create mode 100644 drivers/vdpa/mlx5/net/debug.c
 create mode 100644 drivers/vdpa/mlx5/net/mlx5_vnet.h

-- 
2.35.1

[PATCH 1/4] vdpa/mlx5: Fix rule forwarding VLAN to TIR

[Eli Cohen]

Set the VLAN id to the header values field instead of overwriting the
headers criteria field.

Before this fix, VLAN filtering would not really work and tagged packets
would be forwarded unfiltered to the TIR.

Fixes: baf2ad3f6a98 ("vdpa/mlx5: Add RX MAC VLAN filter support")

Signed-off-by: Eli Cohen <elic@nvidia.com>
---
 drivers/vdpa/mlx5/net/mlx5_vnet.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c
index 90913365def4..dd29fdfc24ed 100644
--- a/drivers/vdpa/mlx5/net/mlx5_vnet.c
+++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c
@@ -1472,7 +1472,7 @@ static int mlx5_vdpa_add_mac_vlan_rules(struct mlx5_vdpa_net *ndev, u8 *mac,
 	if (tagged) {
 		MLX5_SET(fte_match_set_lyr_2_4, headers_v, cvlan_tag, 1);
 		MLX5_SET_TO_ONES(fte_match_set_lyr_2_4, headers_c, first_vid);
-		MLX5_SET(fte_match_set_lyr_2_4, headers_c, first_vid, vid);
+		MLX5_SET(fte_match_set_lyr_2_4, headers_v, first_vid, vid);
 	}
 	flow_act.action = MLX5_FLOW_CONTEXT_ACTION_FWD_DEST;
 	dest.type = MLX5_FLOW_DESTINATION_TYPE_TIR;
-- 
2.35.1

Re: [PATCH 1/4] vdpa/mlx5: Fix rule forwarding VLAN to TIR

[Si-Wei Liu]

Hi Eli,

It's not for this patch but something related, so just a friendly 
heads-up. I haven't validated the VLAN tagging behavior yet for mlx5 
vdpa, but from my quick read of the code it doesn't seem it 
differentiates the case with and without VIRTIO_NET_F_CTRL_VLAN, to be 
compatible/compliant with what's been implemented in QEMU software (a 
spec addendum was filed as requested by Michael):

https://github.com/oasis-tcs/virtio-spec/issues/147

- when VIRTIO_NET_F_CTRL_VLAN is negotiated, the device starts with
all VLANs filtered (meaning only untagged traffic can be received,
and traffic with VLAN tag will be dropped).

- when VIRTIO_NET_F_CTRL_VLAN is not negotiated, all traffic including
untagged and tagged can be received.

Can you please help check if we need further fix in terms of VLAN tagging?

Thanks,
-Siwei


On 10/18/2022 4:12 AM, Eli Cohen wrote:
> Set the VLAN id to the header values field instead of overwriting the
> headers criteria field.
>
> Before this fix, VLAN filtering would not really work and tagged packets
> would be forwarded unfiltered to the TIR.
>
> Fixes: baf2ad3f6a98 ("vdpa/mlx5: Add RX MAC VLAN filter support")
>
> Signed-off-by: Eli Cohen <elic@nvidia.com>
Reviewed-by: Si-Wei Liu <si-wei.liu@oracle.com>

> ---
>   drivers/vdpa/mlx5/net/mlx5_vnet.c | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c
> index 90913365def4..dd29fdfc24ed 100644
> --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c
> +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c
> @@ -1472,7 +1472,7 @@ static int mlx5_vdpa_add_mac_vlan_rules(struct mlx5_vdpa_net *ndev, u8 *mac,
>   	if (tagged) {
>   		MLX5_SET(fte_match_set_lyr_2_4, headers_v, cvlan_tag, 1);
>   		MLX5_SET_TO_ONES(fte_match_set_lyr_2_4, headers_c, first_vid);
> -		MLX5_SET(fte_match_set_lyr_2_4, headers_c, first_vid, vid);
> +		MLX5_SET(fte_match_set_lyr_2_4, headers_v, first_vid, vid);
>   	}
>   	flow_act.action = MLX5_FLOW_CONTEXT_ACTION_FWD_DEST;
>   	dest.type = MLX5_FLOW_DESTINATION_TYPE_TIR;

RE: [PATCH 1/4] vdpa/mlx5: Fix rule forwarding VLAN to TIR

[Eli Cohen]

> From: Si-Wei Liu <si-wei.liu@oracle.com>
> Sent: Tuesday, 18 October 2022 22:21
> To: Eli Cohen <elic@nvidia.com>; mst@redhat.com; jasowang@redhat.com;
> linux-kernel@vger.kernel.org; virtualization@lists.linux-foundation.org
> Cc: eperezma@redhat.com; lulu@redhat.com
> Subject: Re: [PATCH 1/4] vdpa/mlx5: Fix rule forwarding VLAN to TIR
> 
> Hi Eli,
> 
> It's not for this patch but something related, so just a friendly
> heads-up. I haven't validated the VLAN tagging behavior yet for mlx5
> vdpa, but from my quick read of the code it doesn't seem it
> differentiates the case with and without VIRTIO_NET_F_CTRL_VLAN, to be
> compatible/compliant with what's been implemented in QEMU software (a
> spec addendum was filed as requested by Michael):
> 
> https://github.com/oasis-tcs/virtio-spec/issues/147
> 
> - when VIRTIO_NET_F_CTRL_VLAN is negotiated, the device starts with
> all VLANs filtered (meaning only untagged traffic can be received,
> and traffic with VLAN tag will be dropped).
> 
> - when VIRTIO_NET_F_CTRL_VLAN is not negotiated, all traffic including
> untagged and tagged can be received.
> 
> Can you please help check if we need further fix in terms of VLAN tagging?
> 

Sure. It's broken today. I will fix this to conform to the above requirements and send V1.

> Thanks,
> -Siwei
> 
> 
> On 10/18/2022 4:12 AM, Eli Cohen wrote:
> > Set the VLAN id to the header values field instead of overwriting the
> > headers criteria field.
> >
> > Before this fix, VLAN filtering would not really work and tagged packets
> > would be forwarded unfiltered to the TIR.
> >
> > Fixes: baf2ad3f6a98 ("vdpa/mlx5: Add RX MAC VLAN filter support")
> >
> > Signed-off-by: Eli Cohen <elic@nvidia.com>
> Reviewed-by: Si-Wei Liu <si-wei.liu@oracle.com>
> 
> > ---
> >   drivers/vdpa/mlx5/net/mlx5_vnet.c | 2 +-
> >   1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c
> b/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > index 90913365def4..dd29fdfc24ed 100644
> > --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > @@ -1472,7 +1472,7 @@ static int mlx5_vdpa_add_mac_vlan_rules(struct
> mlx5_vdpa_net *ndev, u8 *mac,
> >   	if (tagged) {
> >   		MLX5_SET(fte_match_set_lyr_2_4, headers_v, cvlan_tag, 1);
> >   		MLX5_SET_TO_ONES(fte_match_set_lyr_2_4, headers_c,
> first_vid);
> > -		MLX5_SET(fte_match_set_lyr_2_4, headers_c, first_vid, vid);
> > +		MLX5_SET(fte_match_set_lyr_2_4, headers_v, first_vid, vid);
> >   	}
> >   	flow_act.action = MLX5_FLOW_CONTEXT_ACTION_FWD_DEST;
> >   	dest.type = MLX5_FLOW_DESTINATION_TYPE_TIR;

Re: [PATCH 1/4] vdpa/mlx5: Fix rule forwarding VLAN to TIR

[Michael S. Tsirkin]

On Wed, Oct 19, 2022 at 05:31:48AM +0000, Eli Cohen wrote:
> > From: Si-Wei Liu <si-wei.liu@oracle.com>
> > Sent: Tuesday, 18 October 2022 22:21
> > To: Eli Cohen <elic@nvidia.com>; mst@redhat.com; jasowang@redhat.com;
> > linux-kernel@vger.kernel.org; virtualization@lists.linux-foundation.org
> > Cc: eperezma@redhat.com; lulu@redhat.com
> > Subject: Re: [PATCH 1/4] vdpa/mlx5: Fix rule forwarding VLAN to TIR
> > 
> > Hi Eli,
> > 
> > It's not for this patch but something related, so just a friendly
> > heads-up. I haven't validated the VLAN tagging behavior yet for mlx5
> > vdpa, but from my quick read of the code it doesn't seem it
> > differentiates the case with and without VIRTIO_NET_F_CTRL_VLAN, to be
> > compatible/compliant with what's been implemented in QEMU software (a
> > spec addendum was filed as requested by Michael):
> > 
> > https://github.com/oasis-tcs/virtio-spec/issues/147
> > 
> > - when VIRTIO_NET_F_CTRL_VLAN is negotiated, the device starts with
> > all VLANs filtered (meaning only untagged traffic can be received,
> > and traffic with VLAN tag will be dropped).
> > 
> > - when VIRTIO_NET_F_CTRL_VLAN is not negotiated, all traffic including
> > untagged and tagged can be received.
> > 
> > Can you please help check if we need further fix in terms of VLAN tagging?
> > 
> 
> Sure. It's broken today. I will fix this to conform to the above requirements and send V1.

Did you mean v2?

> > Thanks,
> > -Siwei
> > 
> > 
> > On 10/18/2022 4:12 AM, Eli Cohen wrote:
> > > Set the VLAN id to the header values field instead of overwriting the
> > > headers criteria field.
> > >
> > > Before this fix, VLAN filtering would not really work and tagged packets
> > > would be forwarded unfiltered to the TIR.
> > >
> > > Fixes: baf2ad3f6a98 ("vdpa/mlx5: Add RX MAC VLAN filter support")
> > >
> > > Signed-off-by: Eli Cohen <elic@nvidia.com>
> > Reviewed-by: Si-Wei Liu <si-wei.liu@oracle.com>
> > 
> > > ---
> > >   drivers/vdpa/mlx5/net/mlx5_vnet.c | 2 +-
> > >   1 file changed, 1 insertion(+), 1 deletion(-)
> > >
> > > diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > b/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > > index 90913365def4..dd29fdfc24ed 100644
> > > --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > > +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > > @@ -1472,7 +1472,7 @@ static int mlx5_vdpa_add_mac_vlan_rules(struct
> > mlx5_vdpa_net *ndev, u8 *mac,
> > >   	if (tagged) {
> > >   		MLX5_SET(fte_match_set_lyr_2_4, headers_v, cvlan_tag, 1);
> > >   		MLX5_SET_TO_ONES(fte_match_set_lyr_2_4, headers_c,
> > first_vid);
> > > -		MLX5_SET(fte_match_set_lyr_2_4, headers_c, first_vid, vid);
> > > +		MLX5_SET(fte_match_set_lyr_2_4, headers_v, first_vid, vid);
> > >   	}
> > >   	flow_act.action = MLX5_FLOW_CONTEXT_ACTION_FWD_DEST;
> > >   	dest.type = MLX5_FLOW_DESTINATION_TYPE_TIR;
> 

RE: [PATCH 1/4] vdpa/mlx5: Fix rule forwarding VLAN to TIR

[Eli Cohen]

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Wednesday, 19 October 2022 8:34
> To: Eli Cohen <elic@nvidia.com>
> Cc: Si-Wei Liu <si-wei.liu@oracle.com>; jasowang@redhat.com; linux-
> kernel@vger.kernel.org; virtualization@lists.linux-foundation.org;
> eperezma@redhat.com; lulu@redhat.com
> Subject: Re: [PATCH 1/4] vdpa/mlx5: Fix rule forwarding VLAN to TIR
> 
> On Wed, Oct 19, 2022 at 05:31:48AM +0000, Eli Cohen wrote:
> > > From: Si-Wei Liu <si-wei.liu@oracle.com>
> > > Sent: Tuesday, 18 October 2022 22:21
> > > To: Eli Cohen <elic@nvidia.com>; mst@redhat.com; jasowang@redhat.com;
> > > linux-kernel@vger.kernel.org; virtualization@lists.linux-foundation.org
> > > Cc: eperezma@redhat.com; lulu@redhat.com
> > > Subject: Re: [PATCH 1/4] vdpa/mlx5: Fix rule forwarding VLAN to TIR
> > >
> > > Hi Eli,
> > >
> > > It's not for this patch but something related, so just a friendly
> > > heads-up. I haven't validated the VLAN tagging behavior yet for mlx5
> > > vdpa, but from my quick read of the code it doesn't seem it
> > > differentiates the case with and without VIRTIO_NET_F_CTRL_VLAN, to be
> > > compatible/compliant with what's been implemented in QEMU software (a
> > > spec addendum was filed as requested by Michael):
> > >
> > > https://github.com/oasis-tcs/virtio-spec/issues/147
> > >
> > > - when VIRTIO_NET_F_CTRL_VLAN is negotiated, the device starts with
> > > all VLANs filtered (meaning only untagged traffic can be received,
> > > and traffic with VLAN tag will be dropped).
> > >
> > > - when VIRTIO_NET_F_CTRL_VLAN is not negotiated, all traffic including
> > > untagged and tagged can be received.
> > >
> > > Can you please help check if we need further fix in terms of VLAN tagging?
> > >
> >
> > Sure. It's broken today. I will fix this to conform to the above requirements
> and send V1.
> 
> Did you mean v2?
> 

I count from 0 and have been following this scheme but I can make it v2 if that's the norm.

> > > Thanks,
> > > -Siwei
> > >
> > >
> > > On 10/18/2022 4:12 AM, Eli Cohen wrote:
> > > > Set the VLAN id to the header values field instead of overwriting the
> > > > headers criteria field.
> > > >
> > > > Before this fix, VLAN filtering would not really work and tagged packets
> > > > would be forwarded unfiltered to the TIR.
> > > >
> > > > Fixes: baf2ad3f6a98 ("vdpa/mlx5: Add RX MAC VLAN filter support")
> > > >
> > > > Signed-off-by: Eli Cohen <elic@nvidia.com>
> > > Reviewed-by: Si-Wei Liu <si-wei.liu@oracle.com>
> > >
> > > > ---
> > > >   drivers/vdpa/mlx5/net/mlx5_vnet.c | 2 +-
> > > >   1 file changed, 1 insertion(+), 1 deletion(-)
> > > >
> > > > diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > > b/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > > > index 90913365def4..dd29fdfc24ed 100644
> > > > --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > > > +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > > > @@ -1472,7 +1472,7 @@ static int
> mlx5_vdpa_add_mac_vlan_rules(struct
> > > mlx5_vdpa_net *ndev, u8 *mac,
> > > >   	if (tagged) {
> > > >   		MLX5_SET(fte_match_set_lyr_2_4, headers_v, cvlan_tag, 1);
> > > >   		MLX5_SET_TO_ONES(fte_match_set_lyr_2_4, headers_c,
> > > first_vid);
> > > > -		MLX5_SET(fte_match_set_lyr_2_4, headers_c, first_vid, vid);
> > > > +		MLX5_SET(fte_match_set_lyr_2_4, headers_v, first_vid, vid);
> > > >   	}
> > > >   	flow_act.action = MLX5_FLOW_CONTEXT_ACTION_FWD_DEST;
> > > >   	dest.type = MLX5_FLOW_DESTINATION_TYPE_TIR;
> >

Re: [PATCH 1/4] vdpa/mlx5: Fix rule forwarding VLAN to TIR

[Michael S. Tsirkin]

On Wed, Oct 19, 2022 at 05:36:17AM +0000, Eli Cohen wrote:
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Wednesday, 19 October 2022 8:34
> > To: Eli Cohen <elic@nvidia.com>
> > Cc: Si-Wei Liu <si-wei.liu@oracle.com>; jasowang@redhat.com; linux-
> > kernel@vger.kernel.org; virtualization@lists.linux-foundation.org;
> > eperezma@redhat.com; lulu@redhat.com
> > Subject: Re: [PATCH 1/4] vdpa/mlx5: Fix rule forwarding VLAN to TIR
> > 
> > On Wed, Oct 19, 2022 at 05:31:48AM +0000, Eli Cohen wrote:
> > > > From: Si-Wei Liu <si-wei.liu@oracle.com>
> > > > Sent: Tuesday, 18 October 2022 22:21
> > > > To: Eli Cohen <elic@nvidia.com>; mst@redhat.com; jasowang@redhat.com;
> > > > linux-kernel@vger.kernel.org; virtualization@lists.linux-foundation.org
> > > > Cc: eperezma@redhat.com; lulu@redhat.com
> > > > Subject: Re: [PATCH 1/4] vdpa/mlx5: Fix rule forwarding VLAN to TIR
> > > >
> > > > Hi Eli,
> > > >
> > > > It's not for this patch but something related, so just a friendly
> > > > heads-up. I haven't validated the VLAN tagging behavior yet for mlx5
> > > > vdpa, but from my quick read of the code it doesn't seem it
> > > > differentiates the case with and without VIRTIO_NET_F_CTRL_VLAN, to be
> > > > compatible/compliant with what's been implemented in QEMU software (a
> > > > spec addendum was filed as requested by Michael):
> > > >
> > > > https://github.com/oasis-tcs/virtio-spec/issues/147
> > > >
> > > > - when VIRTIO_NET_F_CTRL_VLAN is negotiated, the device starts with
> > > > all VLANs filtered (meaning only untagged traffic can be received,
> > > > and traffic with VLAN tag will be dropped).
> > > >
> > > > - when VIRTIO_NET_F_CTRL_VLAN is not negotiated, all traffic including
> > > > untagged and tagged can be received.
> > > >
> > > > Can you please help check if we need further fix in terms of VLAN tagging?
> > > >
> > >
> > > Sure. It's broken today. I will fix this to conform to the above requirements
> > and send V1.
> > 
> > Did you mean v2?
> > 
> 
> I count from 0 and have been following this scheme but I can make it v2 if that's the norm.

Yes, most people seem to count patches from 1 so [PATCH] is followed by
[PATCH v2].  I don't know why. But it doesn't matter much - I just
wanted to understand whether you will be sending a new version of this
patchset. I know now.


> > > > Thanks,
> > > > -Siwei
> > > >
> > > >
> > > > On 10/18/2022 4:12 AM, Eli Cohen wrote:
> > > > > Set the VLAN id to the header values field instead of overwriting the
> > > > > headers criteria field.
> > > > >
> > > > > Before this fix, VLAN filtering would not really work and tagged packets
> > > > > would be forwarded unfiltered to the TIR.
> > > > >
> > > > > Fixes: baf2ad3f6a98 ("vdpa/mlx5: Add RX MAC VLAN filter support")
> > > > >
> > > > > Signed-off-by: Eli Cohen <elic@nvidia.com>
> > > > Reviewed-by: Si-Wei Liu <si-wei.liu@oracle.com>
> > > >
> > > > > ---
> > > > >   drivers/vdpa/mlx5/net/mlx5_vnet.c | 2 +-
> > > > >   1 file changed, 1 insertion(+), 1 deletion(-)
> > > > >
> > > > > diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > > > b/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > > > > index 90913365def4..dd29fdfc24ed 100644
> > > > > --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > > > > +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > > > > @@ -1472,7 +1472,7 @@ static int
> > mlx5_vdpa_add_mac_vlan_rules(struct
> > > > mlx5_vdpa_net *ndev, u8 *mac,
> > > > >   	if (tagged) {
> > > > >   		MLX5_SET(fte_match_set_lyr_2_4, headers_v, cvlan_tag, 1);
> > > > >   		MLX5_SET_TO_ONES(fte_match_set_lyr_2_4, headers_c,
> > > > first_vid);
> > > > > -		MLX5_SET(fte_match_set_lyr_2_4, headers_c, first_vid, vid);
> > > > > +		MLX5_SET(fte_match_set_lyr_2_4, headers_v, first_vid, vid);
> > > > >   	}
> > > > >   	flow_act.action = MLX5_FLOW_CONTEXT_ACTION_FWD_DEST;
> > > > >   	dest.type = MLX5_FLOW_DESTINATION_TYPE_TIR;
> > >
> 

[PATCH 2/4] vdpa/mlx5: Move some definitions to a new header file

[Eli Cohen]

Move some definitions from mlx5_vnet.c to newly added header file
mlx5_vnet.h. We need these definitions for the following patches that
add debugfs tree to expose information vital for debug.

Signed-off-by: Eli Cohen <elic@nvidia.com>
---
 drivers/vdpa/mlx5/net/mlx5_vnet.c | 45 +------------------------
 drivers/vdpa/mlx5/net/mlx5_vnet.h | 55 +++++++++++++++++++++++++++++++
 2 files changed, 56 insertions(+), 44 deletions(-)
 create mode 100644 drivers/vdpa/mlx5/net/mlx5_vnet.h

diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c
index dd29fdfc24ed..64fdb940380f 100644
--- a/drivers/vdpa/mlx5/net/mlx5_vnet.c
+++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c
@@ -18,15 +18,12 @@
 #include <linux/mlx5/mlx5_ifc_vdpa.h>
 #include <linux/mlx5/mpfs.h>
 #include "mlx5_vdpa.h"
+#include "mlx5_vnet.h"
 
 MODULE_AUTHOR("Eli Cohen <eli@mellanox.com>");
 MODULE_DESCRIPTION("Mellanox VDPA driver");
 MODULE_LICENSE("Dual BSD/GPL");
 
-#define to_mlx5_vdpa_ndev(__mvdev)                                             \
-	container_of(__mvdev, struct mlx5_vdpa_net, mvdev)
-#define to_mvdev(__vdev) container_of((__vdev), struct mlx5_vdpa_dev, vdev)
-
 #define VALID_FEATURES_MASK                                                                        \
 	(BIT_ULL(VIRTIO_NET_F_CSUM) | BIT_ULL(VIRTIO_NET_F_GUEST_CSUM) |                                   \
 	 BIT_ULL(VIRTIO_NET_F_CTRL_GUEST_OFFLOADS) | BIT_ULL(VIRTIO_NET_F_MTU) | BIT_ULL(VIRTIO_NET_F_MAC) |   \
@@ -50,14 +47,6 @@ MODULE_LICENSE("Dual BSD/GPL");
 
 #define MLX5V_UNTAGGED 0x1000
 
-struct mlx5_vdpa_net_resources {
-	u32 tisn;
-	u32 tdn;
-	u32 tirn;
-	u32 rqtn;
-	bool valid;
-};
-
 struct mlx5_vdpa_cq_buf {
 	struct mlx5_frag_buf_ctrl fbc;
 	struct mlx5_frag_buf frag_buf;
@@ -146,38 +135,6 @@ static bool is_index_valid(struct mlx5_vdpa_dev *mvdev, u16 idx)
 	return idx <= mvdev->max_idx;
 }
 
-#define MLX5V_MACVLAN_SIZE 256
-
-struct mlx5_vdpa_net {
-	struct mlx5_vdpa_dev mvdev;
-	struct mlx5_vdpa_net_resources res;
-	struct virtio_net_config config;
-	struct mlx5_vdpa_virtqueue *vqs;
-	struct vdpa_callback *event_cbs;
-
-	/* Serialize vq resources creation and destruction. This is required
-	 * since memory map might change and we need to destroy and create
-	 * resources while driver in operational.
-	 */
-	struct rw_semaphore reslock;
-	struct mlx5_flow_table *rxft;
-	bool setup;
-	u32 cur_num_vqs;
-	u32 rqt_size;
-	bool nb_registered;
-	struct notifier_block nb;
-	struct vdpa_callback config_cb;
-	struct mlx5_vdpa_wq_ent cvq_ent;
-	struct hlist_head macvlan_hash[MLX5V_MACVLAN_SIZE];
-};
-
-struct macvlan_node {
-	struct hlist_node hlist;
-	struct mlx5_flow_handle *ucast_rule;
-	struct mlx5_flow_handle *mcast_rule;
-	u64 macvlan;
-};
-
 static void free_resources(struct mlx5_vdpa_net *ndev);
 static void init_mvqs(struct mlx5_vdpa_net *ndev);
 static int setup_driver(struct mlx5_vdpa_dev *mvdev);
diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.h b/drivers/vdpa/mlx5/net/mlx5_vnet.h
new file mode 100644
index 000000000000..6691c879a6ca
--- /dev/null
+++ b/drivers/vdpa/mlx5/net/mlx5_vnet.h
@@ -0,0 +1,55 @@
+/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
+/* Copyright (c) 2022, NVIDIA CORPORATION & AFFILIATES. All rights reserved. */
+
+#ifndef __MLX5_VNET_H__
+#define __MLX5_VNET_H__
+
+#include "mlx5_vdpa.h"
+
+#define to_mlx5_vdpa_ndev(__mvdev)                                             \
+	container_of(__mvdev, struct mlx5_vdpa_net, mvdev)
+#define to_mvdev(__vdev) container_of((__vdev), struct mlx5_vdpa_dev, vdev)
+
+struct mlx5_vdpa_net_resources {
+	u32 tisn;
+	u32 tdn;
+	u32 tirn;
+	u32 rqtn;
+	bool valid;
+};
+
+#define MLX5V_MACVLAN_SIZE 256
+
+struct mlx5_vdpa_net {
+	struct mlx5_vdpa_dev mvdev;
+	struct mlx5_vdpa_net_resources res;
+	struct virtio_net_config config;
+	struct mlx5_vdpa_virtqueue *vqs;
+	struct vdpa_callback *event_cbs;
+
+	/* Serialize vq resources creation and destruction. This is required
+	 * since memory map might change and we need to destroy and create
+	 * resources while driver in operational.
+	 */
+	struct rw_semaphore reslock;
+	struct mlx5_flow_table *rxft;
+	struct dentry *rx_dent;
+	struct dentry *rx_table_dent;
+	bool setup;
+	u32 cur_num_vqs;
+	u32 rqt_size;
+	bool nb_registered;
+	struct notifier_block nb;
+	struct vdpa_callback config_cb;
+	struct mlx5_vdpa_wq_ent cvq_ent;
+	struct hlist_head macvlan_hash[MLX5V_MACVLAN_SIZE];
+};
+
+struct macvlan_node {
+	struct hlist_node hlist;
+	struct mlx5_flow_handle *ucast_rule;
+	struct mlx5_flow_handle *mcast_rule;
+	u64 macvlan;
+};
+
+#endif /* __MLX5_VNET_H__ */
-- 
2.35.1

[PATCH 3/4] vdpa/mlx5: Add debugfs subtree

[Eli Cohen]

Add debugfs subtree and expose flow table ID and TIR number. This
information can be used by external tools to do extended
troubleshooting.

The information can be retrieved like so:
$ cat /sys/kernel/debug/mlx5/mlx5_core.sf.1/vdpa-0/rx/table_id
$ cat /sys/kernel/debug/mlx5/mlx5_core.sf.1/vdpa-0/rx/tirn

Signed-off-by: Eli Cohen <elic@nvidia.com>
---
 drivers/vdpa/mlx5/Makefile        |  2 +-
 drivers/vdpa/mlx5/net/debug.c     | 66 +++++++++++++++++++++++++++++++
 drivers/vdpa/mlx5/net/mlx5_vnet.c | 11 ++++++
 drivers/vdpa/mlx5/net/mlx5_vnet.h |  9 +++++
 4 files changed, 87 insertions(+), 1 deletion(-)
 create mode 100644 drivers/vdpa/mlx5/net/debug.c

diff --git a/drivers/vdpa/mlx5/Makefile b/drivers/vdpa/mlx5/Makefile
index f717978c83bf..e791394c33e3 100644
--- a/drivers/vdpa/mlx5/Makefile
+++ b/drivers/vdpa/mlx5/Makefile
@@ -1,4 +1,4 @@
 subdir-ccflags-y += -I$(srctree)/drivers/vdpa/mlx5/core
 
 obj-$(CONFIG_MLX5_VDPA_NET) += mlx5_vdpa.o
-mlx5_vdpa-$(CONFIG_MLX5_VDPA_NET) += net/mlx5_vnet.o core/resources.o core/mr.o
+mlx5_vdpa-$(CONFIG_MLX5_VDPA_NET) += net/mlx5_vnet.o core/resources.o core/mr.o net/debug.o
diff --git a/drivers/vdpa/mlx5/net/debug.c b/drivers/vdpa/mlx5/net/debug.c
new file mode 100644
index 000000000000..95e4801df211
--- /dev/null
+++ b/drivers/vdpa/mlx5/net/debug.c
@@ -0,0 +1,66 @@
+// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
+/* Copyright (c) 2022, NVIDIA CORPORATION & AFFILIATES. All rights reserved. */
+
+#include <linux/debugfs.h>
+#include <linux/mlx5/fs.h>
+#include "mlx5_vnet.h"
+
+static int tirn_show(struct seq_file *file, void *priv)
+{
+	struct mlx5_vdpa_net *ndev = file->private;
+
+	seq_printf(file, "0x%x\n", ndev->res.tirn);
+	return 0;
+}
+
+DEFINE_SHOW_ATTRIBUTE(tirn);
+
+void mlx5_vdpa_remove_tirn(struct mlx5_vdpa_net *ndev)
+{
+	if (ndev->debugfs)
+		debugfs_remove(ndev->res.tirn_dent);
+}
+
+void mlx5_vdpa_add_tirn(struct mlx5_vdpa_net *ndev)
+{
+	ndev->res.tirn_dent = debugfs_create_file("tirn", 0444, ndev->rx_dent,
+						  ndev, &tirn_fops);
+}
+
+static int rx_flow_table_show(struct seq_file *file, void *priv)
+{
+	struct mlx5_vdpa_net *ndev = file->private;
+
+	seq_printf(file, "0x%x\n", mlx5_flow_table_id(ndev->rxft));
+	return 0;
+}
+
+DEFINE_SHOW_ATTRIBUTE(rx_flow_table);
+
+void mlx5_vdpa_remove_rx_flow_table(struct mlx5_vdpa_net *ndev)
+{
+	if (ndev->debugfs)
+		debugfs_remove(ndev->rx_table_dent);
+}
+
+void mlx5_vdpa_add_rx_flow_table(struct mlx5_vdpa_net *ndev)
+{
+	ndev->rx_table_dent = debugfs_create_file("table_id", 0444, ndev->rx_dent,
+						  ndev, &rx_flow_table_fops);
+}
+
+void mlx5_vdpa_add_debugfs(struct mlx5_vdpa_net *ndev)
+{
+	struct mlx5_core_dev *mdev;
+
+	mdev = ndev->mvdev.mdev;
+	ndev->debugfs = debugfs_create_dir(dev_name(&ndev->mvdev.vdev.dev),
+					   mlx5_debugfs_get_dev_root(mdev));
+	if (!IS_ERR(ndev->debugfs))
+		ndev->rx_dent = debugfs_create_dir("rx", ndev->debugfs);
+}
+
+void mlx5_vdpa_remove_debugfs(struct dentry *dbg)
+{
+	debugfs_remove_recursive(dbg);
+}
diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c
index 64fdb940380f..ee50da33e25b 100644
--- a/drivers/vdpa/mlx5/net/mlx5_vnet.c
+++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c
@@ -1388,11 +1388,16 @@ static int create_tir(struct mlx5_vdpa_net *ndev)
 
 	err = mlx5_vdpa_create_tir(&ndev->mvdev, in, &ndev->res.tirn);
 	kfree(in);
+	if (err)
+		return err;
+
+	mlx5_vdpa_add_tirn(ndev);
 	return err;
 }
 
 static void destroy_tir(struct mlx5_vdpa_net *ndev)
 {
+	mlx5_vdpa_remove_tirn(ndev);
 	mlx5_vdpa_destroy_tir(&ndev->mvdev, ndev->res.tirn);
 }
 
@@ -1576,6 +1581,7 @@ static int setup_steering(struct mlx5_vdpa_net *ndev)
 		mlx5_vdpa_warn(&ndev->mvdev, "failed to create flow table\n");
 		return PTR_ERR(ndev->rxft);
 	}
+	mlx5_vdpa_add_rx_flow_table(ndev);
 
 	err = mac_vlan_add(ndev, ndev->config.mac, 0, false);
 	if (err)
@@ -1584,6 +1590,7 @@ static int setup_steering(struct mlx5_vdpa_net *ndev)
 	return 0;
 
 err_add:
+	mlx5_vdpa_remove_rx_flow_table(ndev);
 	mlx5_destroy_flow_table(ndev->rxft);
 	return err;
 }
@@ -1591,6 +1598,7 @@ static int setup_steering(struct mlx5_vdpa_net *ndev)
 static void teardown_steering(struct mlx5_vdpa_net *ndev)
 {
 	clear_mac_vlan_table(ndev);
+	mlx5_vdpa_remove_rx_flow_table(ndev);
 	mlx5_destroy_flow_table(ndev->rxft);
 }
 
@@ -3167,6 +3175,7 @@ static int mlx5_vdpa_dev_add(struct vdpa_mgmt_dev *v_mdev, const char *name,
 	if (err)
 		goto err_reg;
 
+	mlx5_vdpa_add_debugfs(ndev);
 	mgtdev->ndev = ndev;
 	return 0;
 
@@ -3193,6 +3202,8 @@ static void mlx5_vdpa_dev_del(struct vdpa_mgmt_dev *v_mdev, struct vdpa_device *
 	struct mlx5_vdpa_net *ndev = to_mlx5_vdpa_ndev(mvdev);
 	struct workqueue_struct *wq;
 
+	mlx5_vdpa_remove_debugfs(ndev->debugfs);
+	ndev->debugfs = NULL;
 	if (ndev->nb_registered) {
 		mlx5_notifier_unregister(mvdev->mdev, &ndev->nb);
 		ndev->nb_registered = false;
diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.h b/drivers/vdpa/mlx5/net/mlx5_vnet.h
index 6691c879a6ca..f2cef3925e5b 100644
--- a/drivers/vdpa/mlx5/net/mlx5_vnet.h
+++ b/drivers/vdpa/mlx5/net/mlx5_vnet.h
@@ -16,6 +16,7 @@ struct mlx5_vdpa_net_resources {
 	u32 tirn;
 	u32 rqtn;
 	bool valid;
+	struct dentry *tirn_dent;
 };
 
 #define MLX5V_MACVLAN_SIZE 256
@@ -43,6 +44,7 @@ struct mlx5_vdpa_net {
 	struct vdpa_callback config_cb;
 	struct mlx5_vdpa_wq_ent cvq_ent;
 	struct hlist_head macvlan_hash[MLX5V_MACVLAN_SIZE];
+	struct dentry *debugfs;
 };
 
 struct macvlan_node {
@@ -52,4 +54,11 @@ struct macvlan_node {
 	u64 macvlan;
 };
 
+void mlx5_vdpa_add_debugfs(struct mlx5_vdpa_net *ndev);
+void mlx5_vdpa_remove_debugfs(struct dentry *dbg);
+void mlx5_vdpa_add_rx_flow_table(struct mlx5_vdpa_net *ndev);
+void mlx5_vdpa_remove_rx_flow_table(struct mlx5_vdpa_net *ndev);
+void mlx5_vdpa_add_tirn(struct mlx5_vdpa_net *ndev);
+void mlx5_vdpa_remove_tirn(struct mlx5_vdpa_net *ndev);
+
 #endif /* __MLX5_VNET_H__ */
-- 
2.35.1

Re: [PATCH 3/4] vdpa/mlx5: Add debugfs subtree

[Si-Wei Liu]

On 10/18/2022 4:12 AM, Eli Cohen wrote:
> Add debugfs subtree and expose flow table ID and TIR number. This
> information can be used by external tools to do extended
> troubleshooting.
>
> The information can be retrieved like so:
> $ cat /sys/kernel/debug/mlx5/mlx5_core.sf.1/vdpa-0/rx/table_id
> $ cat /sys/kernel/debug/mlx5/mlx5_core.sf.1/vdpa-0/rx/tirn
>
> Signed-off-by: Eli Cohen <elic@nvidia.com>
LGTM

Reviewed-by: Si-Wei Liu <si-wei.liu@oracle.com>
> ---
>   drivers/vdpa/mlx5/Makefile        |  2 +-
>   drivers/vdpa/mlx5/net/debug.c     | 66 +++++++++++++++++++++++++++++++
>   drivers/vdpa/mlx5/net/mlx5_vnet.c | 11 ++++++
>   drivers/vdpa/mlx5/net/mlx5_vnet.h |  9 +++++
>   4 files changed, 87 insertions(+), 1 deletion(-)
>   create mode 100644 drivers/vdpa/mlx5/net/debug.c
>
> diff --git a/drivers/vdpa/mlx5/Makefile b/drivers/vdpa/mlx5/Makefile
> index f717978c83bf..e791394c33e3 100644
> --- a/drivers/vdpa/mlx5/Makefile
> +++ b/drivers/vdpa/mlx5/Makefile
> @@ -1,4 +1,4 @@
>   subdir-ccflags-y += -I$(srctree)/drivers/vdpa/mlx5/core
>   
>   obj-$(CONFIG_MLX5_VDPA_NET) += mlx5_vdpa.o
> -mlx5_vdpa-$(CONFIG_MLX5_VDPA_NET) += net/mlx5_vnet.o core/resources.o core/mr.o
> +mlx5_vdpa-$(CONFIG_MLX5_VDPA_NET) += net/mlx5_vnet.o core/resources.o core/mr.o net/debug.o
> diff --git a/drivers/vdpa/mlx5/net/debug.c b/drivers/vdpa/mlx5/net/debug.c
> new file mode 100644
> index 000000000000..95e4801df211
> --- /dev/null
> +++ b/drivers/vdpa/mlx5/net/debug.c
> @@ -0,0 +1,66 @@
> +// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
> +/* Copyright (c) 2022, NVIDIA CORPORATION & AFFILIATES. All rights reserved. */
> +
> +#include <linux/debugfs.h>
> +#include <linux/mlx5/fs.h>
> +#include "mlx5_vnet.h"
> +
> +static int tirn_show(struct seq_file *file, void *priv)
> +{
> +	struct mlx5_vdpa_net *ndev = file->private;
> +
> +	seq_printf(file, "0x%x\n", ndev->res.tirn);
> +	return 0;
> +}
> +
> +DEFINE_SHOW_ATTRIBUTE(tirn);
> +
> +void mlx5_vdpa_remove_tirn(struct mlx5_vdpa_net *ndev)
> +{
> +	if (ndev->debugfs)
> +		debugfs_remove(ndev->res.tirn_dent);
> +}
> +
> +void mlx5_vdpa_add_tirn(struct mlx5_vdpa_net *ndev)
> +{
> +	ndev->res.tirn_dent = debugfs_create_file("tirn", 0444, ndev->rx_dent,
> +						  ndev, &tirn_fops);
> +}
> +
> +static int rx_flow_table_show(struct seq_file *file, void *priv)
> +{
> +	struct mlx5_vdpa_net *ndev = file->private;
> +
> +	seq_printf(file, "0x%x\n", mlx5_flow_table_id(ndev->rxft));
> +	return 0;
> +}
> +
> +DEFINE_SHOW_ATTRIBUTE(rx_flow_table);
> +
> +void mlx5_vdpa_remove_rx_flow_table(struct mlx5_vdpa_net *ndev)
> +{
> +	if (ndev->debugfs)
> +		debugfs_remove(ndev->rx_table_dent);
> +}
> +
> +void mlx5_vdpa_add_rx_flow_table(struct mlx5_vdpa_net *ndev)
> +{
> +	ndev->rx_table_dent = debugfs_create_file("table_id", 0444, ndev->rx_dent,
> +						  ndev, &rx_flow_table_fops);
> +}
> +
> +void mlx5_vdpa_add_debugfs(struct mlx5_vdpa_net *ndev)
> +{
> +	struct mlx5_core_dev *mdev;
> +
> +	mdev = ndev->mvdev.mdev;
> +	ndev->debugfs = debugfs_create_dir(dev_name(&ndev->mvdev.vdev.dev),
> +					   mlx5_debugfs_get_dev_root(mdev));
> +	if (!IS_ERR(ndev->debugfs))
> +		ndev->rx_dent = debugfs_create_dir("rx", ndev->debugfs);
> +}
> +
> +void mlx5_vdpa_remove_debugfs(struct dentry *dbg)
> +{
> +	debugfs_remove_recursive(dbg);
> +}
> diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c
> index 64fdb940380f..ee50da33e25b 100644
> --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c
> +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c
> @@ -1388,11 +1388,16 @@ static int create_tir(struct mlx5_vdpa_net *ndev)
>   
>   	err = mlx5_vdpa_create_tir(&ndev->mvdev, in, &ndev->res.tirn);
>   	kfree(in);
> +	if (err)
> +		return err;
> +
> +	mlx5_vdpa_add_tirn(ndev);
>   	return err;
>   }
>   
>   static void destroy_tir(struct mlx5_vdpa_net *ndev)
>   {
> +	mlx5_vdpa_remove_tirn(ndev);
>   	mlx5_vdpa_destroy_tir(&ndev->mvdev, ndev->res.tirn);
>   }
>   
> @@ -1576,6 +1581,7 @@ static int setup_steering(struct mlx5_vdpa_net *ndev)
>   		mlx5_vdpa_warn(&ndev->mvdev, "failed to create flow table\n");
>   		return PTR_ERR(ndev->rxft);
>   	}
> +	mlx5_vdpa_add_rx_flow_table(ndev);
>   
>   	err = mac_vlan_add(ndev, ndev->config.mac, 0, false);
>   	if (err)
> @@ -1584,6 +1590,7 @@ static int setup_steering(struct mlx5_vdpa_net *ndev)
>   	return 0;
>   
>   err_add:
> +	mlx5_vdpa_remove_rx_flow_table(ndev);
>   	mlx5_destroy_flow_table(ndev->rxft);
>   	return err;
>   }
> @@ -1591,6 +1598,7 @@ static int setup_steering(struct mlx5_vdpa_net *ndev)
>   static void teardown_steering(struct mlx5_vdpa_net *ndev)
>   {
>   	clear_mac_vlan_table(ndev);
> +	mlx5_vdpa_remove_rx_flow_table(ndev);
>   	mlx5_destroy_flow_table(ndev->rxft);
>   }
>   
> @@ -3167,6 +3175,7 @@ static int mlx5_vdpa_dev_add(struct vdpa_mgmt_dev *v_mdev, const char *name,
>   	if (err)
>   		goto err_reg;
>   
> +	mlx5_vdpa_add_debugfs(ndev);
>   	mgtdev->ndev = ndev;
>   	return 0;
>   
> @@ -3193,6 +3202,8 @@ static void mlx5_vdpa_dev_del(struct vdpa_mgmt_dev *v_mdev, struct vdpa_device *
>   	struct mlx5_vdpa_net *ndev = to_mlx5_vdpa_ndev(mvdev);
>   	struct workqueue_struct *wq;
>   
> +	mlx5_vdpa_remove_debugfs(ndev->debugfs);
> +	ndev->debugfs = NULL;
>   	if (ndev->nb_registered) {
>   		mlx5_notifier_unregister(mvdev->mdev, &ndev->nb);
>   		ndev->nb_registered = false;
> diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.h b/drivers/vdpa/mlx5/net/mlx5_vnet.h
> index 6691c879a6ca..f2cef3925e5b 100644
> --- a/drivers/vdpa/mlx5/net/mlx5_vnet.h
> +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.h
> @@ -16,6 +16,7 @@ struct mlx5_vdpa_net_resources {
>   	u32 tirn;
>   	u32 rqtn;
>   	bool valid;
> +	struct dentry *tirn_dent;
>   };
>   
>   #define MLX5V_MACVLAN_SIZE 256
> @@ -43,6 +44,7 @@ struct mlx5_vdpa_net {
>   	struct vdpa_callback config_cb;
>   	struct mlx5_vdpa_wq_ent cvq_ent;
>   	struct hlist_head macvlan_hash[MLX5V_MACVLAN_SIZE];
> +	struct dentry *debugfs;
>   };
>   
>   struct macvlan_node {
> @@ -52,4 +54,11 @@ struct macvlan_node {
>   	u64 macvlan;
>   };
>   
> +void mlx5_vdpa_add_debugfs(struct mlx5_vdpa_net *ndev);
> +void mlx5_vdpa_remove_debugfs(struct dentry *dbg);
> +void mlx5_vdpa_add_rx_flow_table(struct mlx5_vdpa_net *ndev);
> +void mlx5_vdpa_remove_rx_flow_table(struct mlx5_vdpa_net *ndev);
> +void mlx5_vdpa_add_tirn(struct mlx5_vdpa_net *ndev);
> +void mlx5_vdpa_remove_tirn(struct mlx5_vdpa_net *ndev);
> +
>   #endif /* __MLX5_VNET_H__ */

Re: [PATCH 3/4] vdpa/mlx5: Add debugfs subtree

[Jason Wang]

在 2022/10/18 19:12, Eli Cohen 写道:
> Add debugfs subtree and expose flow table ID and TIR number. This
> information can be used by external tools to do extended
> troubleshooting.
>
> The information can be retrieved like so:
> $ cat /sys/kernel/debug/mlx5/mlx5_core.sf.1/vdpa-0/rx/table_id
> $ cat /sys/kernel/debug/mlx5/mlx5_core.sf.1/vdpa-0/rx/tirn


I wonder if we need a config option for this.

And if it's better to use vendor stats.

Thanks


>
> Signed-off-by: Eli Cohen <elic@nvidia.com>
> ---
>   drivers/vdpa/mlx5/Makefile        |  2 +-
>   drivers/vdpa/mlx5/net/debug.c     | 66 +++++++++++++++++++++++++++++++
>   drivers/vdpa/mlx5/net/mlx5_vnet.c | 11 ++++++
>   drivers/vdpa/mlx5/net/mlx5_vnet.h |  9 +++++
>   4 files changed, 87 insertions(+), 1 deletion(-)
>   create mode 100644 drivers/vdpa/mlx5/net/debug.c
>
> diff --git a/drivers/vdpa/mlx5/Makefile b/drivers/vdpa/mlx5/Makefile
> index f717978c83bf..e791394c33e3 100644
> --- a/drivers/vdpa/mlx5/Makefile
> +++ b/drivers/vdpa/mlx5/Makefile
> @@ -1,4 +1,4 @@
>   subdir-ccflags-y += -I$(srctree)/drivers/vdpa/mlx5/core
>   
>   obj-$(CONFIG_MLX5_VDPA_NET) += mlx5_vdpa.o
> -mlx5_vdpa-$(CONFIG_MLX5_VDPA_NET) += net/mlx5_vnet.o core/resources.o core/mr.o
> +mlx5_vdpa-$(CONFIG_MLX5_VDPA_NET) += net/mlx5_vnet.o core/resources.o core/mr.o net/debug.o
> diff --git a/drivers/vdpa/mlx5/net/debug.c b/drivers/vdpa/mlx5/net/debug.c
> new file mode 100644
> index 000000000000..95e4801df211
> --- /dev/null
> +++ b/drivers/vdpa/mlx5/net/debug.c
> @@ -0,0 +1,66 @@
> +// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
> +/* Copyright (c) 2022, NVIDIA CORPORATION & AFFILIATES. All rights reserved. */
> +
> +#include <linux/debugfs.h>
> +#include <linux/mlx5/fs.h>
> +#include "mlx5_vnet.h"
> +
> +static int tirn_show(struct seq_file *file, void *priv)
> +{
> +	struct mlx5_vdpa_net *ndev = file->private;
> +
> +	seq_printf(file, "0x%x\n", ndev->res.tirn);
> +	return 0;
> +}
> +
> +DEFINE_SHOW_ATTRIBUTE(tirn);
> +
> +void mlx5_vdpa_remove_tirn(struct mlx5_vdpa_net *ndev)
> +{
> +	if (ndev->debugfs)
> +		debugfs_remove(ndev->res.tirn_dent);
> +}
> +
> +void mlx5_vdpa_add_tirn(struct mlx5_vdpa_net *ndev)
> +{
> +	ndev->res.tirn_dent = debugfs_create_file("tirn", 0444, ndev->rx_dent,
> +						  ndev, &tirn_fops);
> +}
> +
> +static int rx_flow_table_show(struct seq_file *file, void *priv)
> +{
> +	struct mlx5_vdpa_net *ndev = file->private;
> +
> +	seq_printf(file, "0x%x\n", mlx5_flow_table_id(ndev->rxft));
> +	return 0;
> +}
> +
> +DEFINE_SHOW_ATTRIBUTE(rx_flow_table);
> +
> +void mlx5_vdpa_remove_rx_flow_table(struct mlx5_vdpa_net *ndev)
> +{
> +	if (ndev->debugfs)
> +		debugfs_remove(ndev->rx_table_dent);
> +}
> +
> +void mlx5_vdpa_add_rx_flow_table(struct mlx5_vdpa_net *ndev)
> +{
> +	ndev->rx_table_dent = debugfs_create_file("table_id", 0444, ndev->rx_dent,
> +						  ndev, &rx_flow_table_fops);
> +}
> +
> +void mlx5_vdpa_add_debugfs(struct mlx5_vdpa_net *ndev)
> +{
> +	struct mlx5_core_dev *mdev;
> +
> +	mdev = ndev->mvdev.mdev;
> +	ndev->debugfs = debugfs_create_dir(dev_name(&ndev->mvdev.vdev.dev),
> +					   mlx5_debugfs_get_dev_root(mdev));
> +	if (!IS_ERR(ndev->debugfs))
> +		ndev->rx_dent = debugfs_create_dir("rx", ndev->debugfs);
> +}
> +
> +void mlx5_vdpa_remove_debugfs(struct dentry *dbg)
> +{
> +	debugfs_remove_recursive(dbg);
> +}
> diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c
> index 64fdb940380f..ee50da33e25b 100644
> --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c
> +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c
> @@ -1388,11 +1388,16 @@ static int create_tir(struct mlx5_vdpa_net *ndev)
>   
>   	err = mlx5_vdpa_create_tir(&ndev->mvdev, in, &ndev->res.tirn);
>   	kfree(in);
> +	if (err)
> +		return err;
> +
> +	mlx5_vdpa_add_tirn(ndev);
>   	return err;
>   }
>   
>   static void destroy_tir(struct mlx5_vdpa_net *ndev)
>   {
> +	mlx5_vdpa_remove_tirn(ndev);
>   	mlx5_vdpa_destroy_tir(&ndev->mvdev, ndev->res.tirn);
>   }
>   
> @@ -1576,6 +1581,7 @@ static int setup_steering(struct mlx5_vdpa_net *ndev)
>   		mlx5_vdpa_warn(&ndev->mvdev, "failed to create flow table\n");
>   		return PTR_ERR(ndev->rxft);
>   	}
> +	mlx5_vdpa_add_rx_flow_table(ndev);
>   
>   	err = mac_vlan_add(ndev, ndev->config.mac, 0, false);
>   	if (err)
> @@ -1584,6 +1590,7 @@ static int setup_steering(struct mlx5_vdpa_net *ndev)
>   	return 0;
>   
>   err_add:
> +	mlx5_vdpa_remove_rx_flow_table(ndev);
>   	mlx5_destroy_flow_table(ndev->rxft);
>   	return err;
>   }
> @@ -1591,6 +1598,7 @@ static int setup_steering(struct mlx5_vdpa_net *ndev)
>   static void teardown_steering(struct mlx5_vdpa_net *ndev)
>   {
>   	clear_mac_vlan_table(ndev);
> +	mlx5_vdpa_remove_rx_flow_table(ndev);
>   	mlx5_destroy_flow_table(ndev->rxft);
>   }
>   
> @@ -3167,6 +3175,7 @@ static int mlx5_vdpa_dev_add(struct vdpa_mgmt_dev *v_mdev, const char *name,
>   	if (err)
>   		goto err_reg;
>   
> +	mlx5_vdpa_add_debugfs(ndev);
>   	mgtdev->ndev = ndev;
>   	return 0;
>   
> @@ -3193,6 +3202,8 @@ static void mlx5_vdpa_dev_del(struct vdpa_mgmt_dev *v_mdev, struct vdpa_device *
>   	struct mlx5_vdpa_net *ndev = to_mlx5_vdpa_ndev(mvdev);
>   	struct workqueue_struct *wq;
>   
> +	mlx5_vdpa_remove_debugfs(ndev->debugfs);
> +	ndev->debugfs = NULL;
>   	if (ndev->nb_registered) {
>   		mlx5_notifier_unregister(mvdev->mdev, &ndev->nb);
>   		ndev->nb_registered = false;
> diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.h b/drivers/vdpa/mlx5/net/mlx5_vnet.h
> index 6691c879a6ca..f2cef3925e5b 100644
> --- a/drivers/vdpa/mlx5/net/mlx5_vnet.h
> +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.h
> @@ -16,6 +16,7 @@ struct mlx5_vdpa_net_resources {
>   	u32 tirn;
>   	u32 rqtn;
>   	bool valid;
> +	struct dentry *tirn_dent;
>   };
>   
>   #define MLX5V_MACVLAN_SIZE 256
> @@ -43,6 +44,7 @@ struct mlx5_vdpa_net {
>   	struct vdpa_callback config_cb;
>   	struct mlx5_vdpa_wq_ent cvq_ent;
>   	struct hlist_head macvlan_hash[MLX5V_MACVLAN_SIZE];
> +	struct dentry *debugfs;
>   };
>   
>   struct macvlan_node {
> @@ -52,4 +54,11 @@ struct macvlan_node {
>   	u64 macvlan;
>   };
>   
> +void mlx5_vdpa_add_debugfs(struct mlx5_vdpa_net *ndev);
> +void mlx5_vdpa_remove_debugfs(struct dentry *dbg);
> +void mlx5_vdpa_add_rx_flow_table(struct mlx5_vdpa_net *ndev);
> +void mlx5_vdpa_remove_rx_flow_table(struct mlx5_vdpa_net *ndev);
> +void mlx5_vdpa_add_tirn(struct mlx5_vdpa_net *ndev);
> +void mlx5_vdpa_remove_tirn(struct mlx5_vdpa_net *ndev);
> +
>   #endif /* __MLX5_VNET_H__ */

RE: [PATCH 3/4] vdpa/mlx5: Add debugfs subtree

[Eli Cohen]

> From: Jason Wang <jasowang@redhat.com>
> Sent: Wednesday, 19 October 2022 8:14
> To: Eli Cohen <elic@nvidia.com>; mst@redhat.com; linux-
> kernel@vger.kernel.org; virtualization@lists.linux-foundation.org
> Cc: si-wei.liu@oracle.com; eperezma@redhat.com; lulu@redhat.com
> Subject: Re: [PATCH 3/4] vdpa/mlx5: Add debugfs subtree
> 
> 
> 在 2022/10/18 19:12, Eli Cohen 写道:
> > Add debugfs subtree and expose flow table ID and TIR number. This
> > information can be used by external tools to do extended
> > troubleshooting.
> >
> > The information can be retrieved like so:
> > $ cat /sys/kernel/debug/mlx5/mlx5_core.sf.1/vdpa-0/rx/table_id
> > $ cat /sys/kernel/debug/mlx5/mlx5_core.sf.1/vdpa-0/rx/tirn
> 
> 
> I wonder if we need a config option for this.
> 
> And if it's better to use vendor stats.
> 

I am not sure. TIRs and flow tables has loose association VDPA. I would keep it in
debugfs which for mlx5 there's lots of other information there.

> Thanks
> 
> 
> >
> > Signed-off-by: Eli Cohen <elic@nvidia.com>
> > ---
> >   drivers/vdpa/mlx5/Makefile        |  2 +-
> >   drivers/vdpa/mlx5/net/debug.c     | 66
> +++++++++++++++++++++++++++++++
> >   drivers/vdpa/mlx5/net/mlx5_vnet.c | 11 ++++++
> >   drivers/vdpa/mlx5/net/mlx5_vnet.h |  9 +++++
> >   4 files changed, 87 insertions(+), 1 deletion(-)
> >   create mode 100644 drivers/vdpa/mlx5/net/debug.c
> >
> > diff --git a/drivers/vdpa/mlx5/Makefile b/drivers/vdpa/mlx5/Makefile
> > index f717978c83bf..e791394c33e3 100644
> > --- a/drivers/vdpa/mlx5/Makefile
> > +++ b/drivers/vdpa/mlx5/Makefile
> > @@ -1,4 +1,4 @@
> >   subdir-ccflags-y += -I$(srctree)/drivers/vdpa/mlx5/core
> >
> >   obj-$(CONFIG_MLX5_VDPA_NET) += mlx5_vdpa.o
> > -mlx5_vdpa-$(CONFIG_MLX5_VDPA_NET) += net/mlx5_vnet.o
> core/resources.o core/mr.o
> > +mlx5_vdpa-$(CONFIG_MLX5_VDPA_NET) += net/mlx5_vnet.o
> core/resources.o core/mr.o net/debug.o
> > diff --git a/drivers/vdpa/mlx5/net/debug.c b/drivers/vdpa/mlx5/net/debug.c
> > new file mode 100644
> > index 000000000000..95e4801df211
> > --- /dev/null
> > +++ b/drivers/vdpa/mlx5/net/debug.c
> > @@ -0,0 +1,66 @@
> > +// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
> > +/* Copyright (c) 2022, NVIDIA CORPORATION & AFFILIATES. All rights
> reserved. */
> > +
> > +#include <linux/debugfs.h>
> > +#include <linux/mlx5/fs.h>
> > +#include "mlx5_vnet.h"
> > +
> > +static int tirn_show(struct seq_file *file, void *priv)
> > +{
> > +	struct mlx5_vdpa_net *ndev = file->private;
> > +
> > +	seq_printf(file, "0x%x\n", ndev->res.tirn);
> > +	return 0;
> > +}
> > +
> > +DEFINE_SHOW_ATTRIBUTE(tirn);
> > +
> > +void mlx5_vdpa_remove_tirn(struct mlx5_vdpa_net *ndev)
> > +{
> > +	if (ndev->debugfs)
> > +		debugfs_remove(ndev->res.tirn_dent);
> > +}
> > +
> > +void mlx5_vdpa_add_tirn(struct mlx5_vdpa_net *ndev)
> > +{
> > +	ndev->res.tirn_dent = debugfs_create_file("tirn", 0444, ndev-
> >rx_dent,
> > +						  ndev, &tirn_fops);
> > +}
> > +
> > +static int rx_flow_table_show(struct seq_file *file, void *priv)
> > +{
> > +	struct mlx5_vdpa_net *ndev = file->private;
> > +
> > +	seq_printf(file, "0x%x\n", mlx5_flow_table_id(ndev->rxft));
> > +	return 0;
> > +}
> > +
> > +DEFINE_SHOW_ATTRIBUTE(rx_flow_table);
> > +
> > +void mlx5_vdpa_remove_rx_flow_table(struct mlx5_vdpa_net *ndev)
> > +{
> > +	if (ndev->debugfs)
> > +		debugfs_remove(ndev->rx_table_dent);
> > +}
> > +
> > +void mlx5_vdpa_add_rx_flow_table(struct mlx5_vdpa_net *ndev)
> > +{
> > +	ndev->rx_table_dent = debugfs_create_file("table_id", 0444, ndev-
> >rx_dent,
> > +						  ndev, &rx_flow_table_fops);
> > +}
> > +
> > +void mlx5_vdpa_add_debugfs(struct mlx5_vdpa_net *ndev)
> > +{
> > +	struct mlx5_core_dev *mdev;
> > +
> > +	mdev = ndev->mvdev.mdev;
> > +	ndev->debugfs = debugfs_create_dir(dev_name(&ndev-
> >mvdev.vdev.dev),
> > +
> mlx5_debugfs_get_dev_root(mdev));
> > +	if (!IS_ERR(ndev->debugfs))
> > +		ndev->rx_dent = debugfs_create_dir("rx", ndev->debugfs);
> > +}
> > +
> > +void mlx5_vdpa_remove_debugfs(struct dentry *dbg)
> > +{
> > +	debugfs_remove_recursive(dbg);
> > +}
> > diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c
> b/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > index 64fdb940380f..ee50da33e25b 100644
> > --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > @@ -1388,11 +1388,16 @@ static int create_tir(struct mlx5_vdpa_net
> *ndev)
> >
> >   	err = mlx5_vdpa_create_tir(&ndev->mvdev, in, &ndev->res.tirn);
> >   	kfree(in);
> > +	if (err)
> > +		return err;
> > +
> > +	mlx5_vdpa_add_tirn(ndev);
> >   	return err;
> >   }
> >
> >   static void destroy_tir(struct mlx5_vdpa_net *ndev)
> >   {
> > +	mlx5_vdpa_remove_tirn(ndev);
> >   	mlx5_vdpa_destroy_tir(&ndev->mvdev, ndev->res.tirn);
> >   }
> >
> > @@ -1576,6 +1581,7 @@ static int setup_steering(struct mlx5_vdpa_net
> *ndev)
> >   		mlx5_vdpa_warn(&ndev->mvdev, "failed to create flow
> table\n");
> >   		return PTR_ERR(ndev->rxft);
> >   	}
> > +	mlx5_vdpa_add_rx_flow_table(ndev);
> >
> >   	err = mac_vlan_add(ndev, ndev->config.mac, 0, false);
> >   	if (err)
> > @@ -1584,6 +1590,7 @@ static int setup_steering(struct mlx5_vdpa_net
> *ndev)
> >   	return 0;
> >
> >   err_add:
> > +	mlx5_vdpa_remove_rx_flow_table(ndev);
> >   	mlx5_destroy_flow_table(ndev->rxft);
> >   	return err;
> >   }
> > @@ -1591,6 +1598,7 @@ static int setup_steering(struct mlx5_vdpa_net
> *ndev)
> >   static void teardown_steering(struct mlx5_vdpa_net *ndev)
> >   {
> >   	clear_mac_vlan_table(ndev);
> > +	mlx5_vdpa_remove_rx_flow_table(ndev);
> >   	mlx5_destroy_flow_table(ndev->rxft);
> >   }
> >
> > @@ -3167,6 +3175,7 @@ static int mlx5_vdpa_dev_add(struct
> vdpa_mgmt_dev *v_mdev, const char *name,
> >   	if (err)
> >   		goto err_reg;
> >
> > +	mlx5_vdpa_add_debugfs(ndev);
> >   	mgtdev->ndev = ndev;
> >   	return 0;
> >
> > @@ -3193,6 +3202,8 @@ static void mlx5_vdpa_dev_del(struct
> vdpa_mgmt_dev *v_mdev, struct vdpa_device *
> >   	struct mlx5_vdpa_net *ndev = to_mlx5_vdpa_ndev(mvdev);
> >   	struct workqueue_struct *wq;
> >
> > +	mlx5_vdpa_remove_debugfs(ndev->debugfs);
> > +	ndev->debugfs = NULL;
> >   	if (ndev->nb_registered) {
> >   		mlx5_notifier_unregister(mvdev->mdev, &ndev->nb);
> >   		ndev->nb_registered = false;
> > diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.h
> b/drivers/vdpa/mlx5/net/mlx5_vnet.h
> > index 6691c879a6ca..f2cef3925e5b 100644
> > --- a/drivers/vdpa/mlx5/net/mlx5_vnet.h
> > +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.h
> > @@ -16,6 +16,7 @@ struct mlx5_vdpa_net_resources {
> >   	u32 tirn;
> >   	u32 rqtn;
> >   	bool valid;
> > +	struct dentry *tirn_dent;
> >   };
> >
> >   #define MLX5V_MACVLAN_SIZE 256
> > @@ -43,6 +44,7 @@ struct mlx5_vdpa_net {
> >   	struct vdpa_callback config_cb;
> >   	struct mlx5_vdpa_wq_ent cvq_ent;
> >   	struct hlist_head macvlan_hash[MLX5V_MACVLAN_SIZE];
> > +	struct dentry *debugfs;
> >   };
> >
> >   struct macvlan_node {
> > @@ -52,4 +54,11 @@ struct macvlan_node {
> >   	u64 macvlan;
> >   };
> >
> > +void mlx5_vdpa_add_debugfs(struct mlx5_vdpa_net *ndev);
> > +void mlx5_vdpa_remove_debugfs(struct dentry *dbg);
> > +void mlx5_vdpa_add_rx_flow_table(struct mlx5_vdpa_net *ndev);
> > +void mlx5_vdpa_remove_rx_flow_table(struct mlx5_vdpa_net *ndev);
> > +void mlx5_vdpa_add_tirn(struct mlx5_vdpa_net *ndev);
> > +void mlx5_vdpa_remove_tirn(struct mlx5_vdpa_net *ndev);
> > +
> >   #endif /* __MLX5_VNET_H__ */

Re: [PATCH 3/4] vdpa/mlx5: Add debugfs subtree

[Jason Wang]

On Wed, Oct 19, 2022 at 1:38 PM Eli Cohen <elic@nvidia.com> wrote:
>
> > From: Jason Wang <jasowang@redhat.com>
> > Sent: Wednesday, 19 October 2022 8:14
> > To: Eli Cohen <elic@nvidia.com>; mst@redhat.com; linux-
> > kernel@vger.kernel.org; virtualization@lists.linux-foundation.org
> > Cc: si-wei.liu@oracle.com; eperezma@redhat.com; lulu@redhat.com
> > Subject: Re: [PATCH 3/4] vdpa/mlx5: Add debugfs subtree
> >
> >
> > 在 2022/10/18 19:12, Eli Cohen 写道:
> > > Add debugfs subtree and expose flow table ID and TIR number. This
> > > information can be used by external tools to do extended
> > > troubleshooting.
> > >
> > > The information can be retrieved like so:
> > > $ cat /sys/kernel/debug/mlx5/mlx5_core.sf.1/vdpa-0/rx/table_id
> > > $ cat /sys/kernel/debug/mlx5/mlx5_core.sf.1/vdpa-0/rx/tirn
> >
> >
> > I wonder if we need a config option for this.
> >
> > And if it's better to use vendor stats.
> >
>
> I am not sure. TIRs and flow tables has loose association VDPA. I would keep it in
> debugfs which for mlx5 there's lots of other information there.

That's fine.

Thanks

>
> > Thanks
> >
> >
> > >
> > > Signed-off-by: Eli Cohen <elic@nvidia.com>
> > > ---
> > >   drivers/vdpa/mlx5/Makefile        |  2 +-
> > >   drivers/vdpa/mlx5/net/debug.c     | 66
> > +++++++++++++++++++++++++++++++
> > >   drivers/vdpa/mlx5/net/mlx5_vnet.c | 11 ++++++
> > >   drivers/vdpa/mlx5/net/mlx5_vnet.h |  9 +++++
> > >   4 files changed, 87 insertions(+), 1 deletion(-)
> > >   create mode 100644 drivers/vdpa/mlx5/net/debug.c
> > >
> > > diff --git a/drivers/vdpa/mlx5/Makefile b/drivers/vdpa/mlx5/Makefile
> > > index f717978c83bf..e791394c33e3 100644
> > > --- a/drivers/vdpa/mlx5/Makefile
> > > +++ b/drivers/vdpa/mlx5/Makefile
> > > @@ -1,4 +1,4 @@
> > >   subdir-ccflags-y += -I$(srctree)/drivers/vdpa/mlx5/core
> > >
> > >   obj-$(CONFIG_MLX5_VDPA_NET) += mlx5_vdpa.o
> > > -mlx5_vdpa-$(CONFIG_MLX5_VDPA_NET) += net/mlx5_vnet.o
> > core/resources.o core/mr.o
> > > +mlx5_vdpa-$(CONFIG_MLX5_VDPA_NET) += net/mlx5_vnet.o
> > core/resources.o core/mr.o net/debug.o
> > > diff --git a/drivers/vdpa/mlx5/net/debug.c b/drivers/vdpa/mlx5/net/debug.c
> > > new file mode 100644
> > > index 000000000000..95e4801df211
> > > --- /dev/null
> > > +++ b/drivers/vdpa/mlx5/net/debug.c
> > > @@ -0,0 +1,66 @@
> > > +// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
> > > +/* Copyright (c) 2022, NVIDIA CORPORATION & AFFILIATES. All rights
> > reserved. */
> > > +
> > > +#include <linux/debugfs.h>
> > > +#include <linux/mlx5/fs.h>
> > > +#include "mlx5_vnet.h"
> > > +
> > > +static int tirn_show(struct seq_file *file, void *priv)
> > > +{
> > > +   struct mlx5_vdpa_net *ndev = file->private;
> > > +
> > > +   seq_printf(file, "0x%x\n", ndev->res.tirn);
> > > +   return 0;
> > > +}
> > > +
> > > +DEFINE_SHOW_ATTRIBUTE(tirn);
> > > +
> > > +void mlx5_vdpa_remove_tirn(struct mlx5_vdpa_net *ndev)
> > > +{
> > > +   if (ndev->debugfs)
> > > +           debugfs_remove(ndev->res.tirn_dent);
> > > +}
> > > +
> > > +void mlx5_vdpa_add_tirn(struct mlx5_vdpa_net *ndev)
> > > +{
> > > +   ndev->res.tirn_dent = debugfs_create_file("tirn", 0444, ndev-
> > >rx_dent,
> > > +                                             ndev, &tirn_fops);
> > > +}
> > > +
> > > +static int rx_flow_table_show(struct seq_file *file, void *priv)
> > > +{
> > > +   struct mlx5_vdpa_net *ndev = file->private;
> > > +
> > > +   seq_printf(file, "0x%x\n", mlx5_flow_table_id(ndev->rxft));
> > > +   return 0;
> > > +}
> > > +
> > > +DEFINE_SHOW_ATTRIBUTE(rx_flow_table);
> > > +
> > > +void mlx5_vdpa_remove_rx_flow_table(struct mlx5_vdpa_net *ndev)
> > > +{
> > > +   if (ndev->debugfs)
> > > +           debugfs_remove(ndev->rx_table_dent);
> > > +}
> > > +
> > > +void mlx5_vdpa_add_rx_flow_table(struct mlx5_vdpa_net *ndev)
> > > +{
> > > +   ndev->rx_table_dent = debugfs_create_file("table_id", 0444, ndev-
> > >rx_dent,
> > > +                                             ndev, &rx_flow_table_fops);
> > > +}
> > > +
> > > +void mlx5_vdpa_add_debugfs(struct mlx5_vdpa_net *ndev)
> > > +{
> > > +   struct mlx5_core_dev *mdev;
> > > +
> > > +   mdev = ndev->mvdev.mdev;
> > > +   ndev->debugfs = debugfs_create_dir(dev_name(&ndev-
> > >mvdev.vdev.dev),
> > > +
> > mlx5_debugfs_get_dev_root(mdev));
> > > +   if (!IS_ERR(ndev->debugfs))
> > > +           ndev->rx_dent = debugfs_create_dir("rx", ndev->debugfs);
> > > +}
> > > +
> > > +void mlx5_vdpa_remove_debugfs(struct dentry *dbg)
> > > +{
> > > +   debugfs_remove_recursive(dbg);
> > > +}
> > > diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > b/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > > index 64fdb940380f..ee50da33e25b 100644
> > > --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > > +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > > @@ -1388,11 +1388,16 @@ static int create_tir(struct mlx5_vdpa_net
> > *ndev)
> > >
> > >     err = mlx5_vdpa_create_tir(&ndev->mvdev, in, &ndev->res.tirn);
> > >     kfree(in);
> > > +   if (err)
> > > +           return err;
> > > +
> > > +   mlx5_vdpa_add_tirn(ndev);
> > >     return err;
> > >   }
> > >
> > >   static void destroy_tir(struct mlx5_vdpa_net *ndev)
> > >   {
> > > +   mlx5_vdpa_remove_tirn(ndev);
> > >     mlx5_vdpa_destroy_tir(&ndev->mvdev, ndev->res.tirn);
> > >   }
> > >
> > > @@ -1576,6 +1581,7 @@ static int setup_steering(struct mlx5_vdpa_net
> > *ndev)
> > >             mlx5_vdpa_warn(&ndev->mvdev, "failed to create flow
> > table\n");
> > >             return PTR_ERR(ndev->rxft);
> > >     }
> > > +   mlx5_vdpa_add_rx_flow_table(ndev);
> > >
> > >     err = mac_vlan_add(ndev, ndev->config.mac, 0, false);
> > >     if (err)
> > > @@ -1584,6 +1590,7 @@ static int setup_steering(struct mlx5_vdpa_net
> > *ndev)
> > >     return 0;
> > >
> > >   err_add:
> > > +   mlx5_vdpa_remove_rx_flow_table(ndev);
> > >     mlx5_destroy_flow_table(ndev->rxft);
> > >     return err;
> > >   }
> > > @@ -1591,6 +1598,7 @@ static int setup_steering(struct mlx5_vdpa_net
> > *ndev)
> > >   static void teardown_steering(struct mlx5_vdpa_net *ndev)
> > >   {
> > >     clear_mac_vlan_table(ndev);
> > > +   mlx5_vdpa_remove_rx_flow_table(ndev);
> > >     mlx5_destroy_flow_table(ndev->rxft);
> > >   }
> > >
> > > @@ -3167,6 +3175,7 @@ static int mlx5_vdpa_dev_add(struct
> > vdpa_mgmt_dev *v_mdev, const char *name,
> > >     if (err)
> > >             goto err_reg;
> > >
> > > +   mlx5_vdpa_add_debugfs(ndev);
> > >     mgtdev->ndev = ndev;
> > >     return 0;
> > >
> > > @@ -3193,6 +3202,8 @@ static void mlx5_vdpa_dev_del(struct
> > vdpa_mgmt_dev *v_mdev, struct vdpa_device *
> > >     struct mlx5_vdpa_net *ndev = to_mlx5_vdpa_ndev(mvdev);
> > >     struct workqueue_struct *wq;
> > >
> > > +   mlx5_vdpa_remove_debugfs(ndev->debugfs);
> > > +   ndev->debugfs = NULL;
> > >     if (ndev->nb_registered) {
> > >             mlx5_notifier_unregister(mvdev->mdev, &ndev->nb);
> > >             ndev->nb_registered = false;
> > > diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.h
> > b/drivers/vdpa/mlx5/net/mlx5_vnet.h
> > > index 6691c879a6ca..f2cef3925e5b 100644
> > > --- a/drivers/vdpa/mlx5/net/mlx5_vnet.h
> > > +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.h
> > > @@ -16,6 +16,7 @@ struct mlx5_vdpa_net_resources {
> > >     u32 tirn;
> > >     u32 rqtn;
> > >     bool valid;
> > > +   struct dentry *tirn_dent;
> > >   };
> > >
> > >   #define MLX5V_MACVLAN_SIZE 256
> > > @@ -43,6 +44,7 @@ struct mlx5_vdpa_net {
> > >     struct vdpa_callback config_cb;
> > >     struct mlx5_vdpa_wq_ent cvq_ent;
> > >     struct hlist_head macvlan_hash[MLX5V_MACVLAN_SIZE];
> > > +   struct dentry *debugfs;
> > >   };
> > >
> > >   struct macvlan_node {
> > > @@ -52,4 +54,11 @@ struct macvlan_node {
> > >     u64 macvlan;
> > >   };
> > >
> > > +void mlx5_vdpa_add_debugfs(struct mlx5_vdpa_net *ndev);
> > > +void mlx5_vdpa_remove_debugfs(struct dentry *dbg);
> > > +void mlx5_vdpa_add_rx_flow_table(struct mlx5_vdpa_net *ndev);
> > > +void mlx5_vdpa_remove_rx_flow_table(struct mlx5_vdpa_net *ndev);
> > > +void mlx5_vdpa_add_tirn(struct mlx5_vdpa_net *ndev);
> > > +void mlx5_vdpa_remove_tirn(struct mlx5_vdpa_net *ndev);
> > > +
> > >   #endif /* __MLX5_VNET_H__ */
>

[PATCH 4/4] vdpa/mlx5: Add RX counters to debugfs

[Eli Cohen]

For each interface, either VLAN tagged or untagged, add two hardware
counters: one for unicast and another for multicast. The counters count
RX packets and bytes and can be read through debugfs:

$ cat /sys/kernel/debug/mlx5/mlx5_core.sf.1/vdpa-0/rx/untagged/mcast/packets
$ cat /sys/kernel/debug/mlx5/mlx5_core.sf.1/vdpa-0/rx/untagged/ucast/bytes

This feature is controlled via the config option
MLX5_VDPA_STEERING_DEBUG. It is off by default as it may have some
impact on performance.

Signed-off-by: Eli Cohen <elic@nvidia.com>
---
 drivers/vdpa/Kconfig              | 12 ++++
 drivers/vdpa/mlx5/net/debug.c     | 82 +++++++++++++++++++++++++++
 drivers/vdpa/mlx5/net/mlx5_vnet.c | 94 +++++++++++++++++++++++--------
 drivers/vdpa/mlx5/net/mlx5_vnet.h | 30 ++++++++++
 4 files changed, 195 insertions(+), 23 deletions(-)

diff --git a/drivers/vdpa/Kconfig b/drivers/vdpa/Kconfig
index 50f45d037611..43b716ec2d18 100644
--- a/drivers/vdpa/Kconfig
+++ b/drivers/vdpa/Kconfig
@@ -71,6 +71,18 @@ config MLX5_VDPA_NET
 	  be executed by the hardware. It also supports a variety of stateless
 	  offloads depending on the actual device used and firmware version.
 
+config MLX5_VDPA_STEERING_DEBUG
+	bool "expose steering counters on debugfs"
+	select MLX5_VDPA
+	help
+	  Expose RX steering counters in debugfs to aid in debugging. For each VLAN
+	  or non VLAN interface, two hardware counters are added to the RX flow
+	  table: one for unicast and one for multicast.
+	  The counters counts the number of packets and bytes and exposes them in
+	  debugfs. Once can read the counters using, e.g.:
+	  cat /sys/kernel/debug/mlx5/mlx5_core.sf.1/vdpa-0/rx/untagged/ucast/packets
+	  cat /sys/kernel/debug/mlx5/mlx5_core.sf.1/vdpa-0/rx/untagged/mcast/bytes
+
 config VP_VDPA
 	tristate "Virtio PCI bridge vDPA driver"
 	select VIRTIO_PCI_LIB
diff --git a/drivers/vdpa/mlx5/net/debug.c b/drivers/vdpa/mlx5/net/debug.c
index 95e4801df211..35137a36433a 100644
--- a/drivers/vdpa/mlx5/net/debug.c
+++ b/drivers/vdpa/mlx5/net/debug.c
@@ -49,6 +49,88 @@ void mlx5_vdpa_add_rx_flow_table(struct mlx5_vdpa_net *ndev)
 						  ndev, &rx_flow_table_fops);
 }
 
+#if defined(CONFIG_MLX5_VDPA_STEERING_DEBUG)
+static int packets_show(struct seq_file *file, void *priv)
+{
+	struct mlx5_vdpa_counter *counter = file->private;
+	u64 packets;
+	u64 bytes;
+	int err;
+
+	err = mlx5_fc_query(counter->mdev, counter->counter, &packets, &bytes);
+	if (err)
+		return err;
+
+	seq_printf(file, "0x%llx\n", packets);
+	return 0;
+}
+
+static int bytes_show(struct seq_file *file, void *priv)
+{
+	struct mlx5_vdpa_counter *counter = file->private;
+	u64 packets;
+	u64 bytes;
+	int err;
+
+	err = mlx5_fc_query(counter->mdev, counter->counter, &packets, &bytes);
+	if (err)
+		return err;
+
+	seq_printf(file, "0x%llx\n", bytes);
+	return 0;
+}
+
+DEFINE_SHOW_ATTRIBUTE(packets);
+DEFINE_SHOW_ATTRIBUTE(bytes);
+
+static void add_counter_node(struct mlx5_vdpa_counter *counter,
+			     struct dentry *parent)
+{
+	debugfs_create_file("packets", 0444, parent, counter,
+			    &packets_fops);
+	debugfs_create_file("bytes", 0444, parent, counter,
+			    &bytes_fops);
+}
+
+void mlx5_vdpa_add_rx_counters(struct mlx5_vdpa_net *ndev,
+			       struct macvlan_node *node)
+{
+	static const char *ut = "untagged";
+	char vidstr[9];
+	u16 vid;
+
+	if (node->tagged) {
+		vid = key2vid(node->macvlan);
+		snprintf(vidstr, sizeof(vidstr), "0x%x", vid);
+	} else {
+		strcpy(vidstr, ut);
+	}
+
+	node->dent = debugfs_create_dir(vidstr, ndev->rx_dent);
+	if (IS_ERR(node->dent))
+		return;
+
+	node->ucast_counter.dent = debugfs_create_dir("ucast", node->dent);
+	if (IS_ERR(node->ucast_counter.dent))
+		return;
+
+	add_counter_node(&node->ucast_counter, node->ucast_counter.dent);
+
+	node->mcast_counter.dent = debugfs_create_dir("mcast", node->dent);
+	if (IS_ERR(node->mcast_counter.dent))
+		return;
+
+	add_counter_node(&node->mcast_counter, node->mcast_counter.dent);
+}
+
+void mlx5_vdpa_remove_rx_counters(struct mlx5_vdpa_net *ndev,
+				  struct macvlan_node *node)
+{
+	if (ndev->debugfs)
+		debugfs_remove_recursive(node->dent);
+}
+#endif
+
 void mlx5_vdpa_add_debugfs(struct mlx5_vdpa_net *ndev)
 {
 	struct mlx5_core_dev *mdev;
diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c
index ee50da33e25b..1b850d0aee99 100644
--- a/drivers/vdpa/mlx5/net/mlx5_vnet.c
+++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c
@@ -1404,12 +1404,16 @@ static void destroy_tir(struct mlx5_vdpa_net *ndev)
 #define MAX_STEERING_ENT 0x8000
 #define MAX_STEERING_GROUPS 2
 
+#if defined(CONFIG_MLX5_VDPA_STEERING_DEBUG)
+       #define NUM_DESTS 2
+#else
+       #define NUM_DESTS 1
+#endif
+
 static int mlx5_vdpa_add_mac_vlan_rules(struct mlx5_vdpa_net *ndev, u8 *mac,
-					u16 vid, bool tagged,
-					struct mlx5_flow_handle **ucast,
-					struct mlx5_flow_handle **mcast)
+					struct macvlan_node *node)
 {
-	struct mlx5_flow_destination dest = {};
+	struct mlx5_flow_destination dests[NUM_DESTS] = {};
 	struct mlx5_flow_act flow_act = {};
 	struct mlx5_flow_handle *rule;
 	struct mlx5_flow_spec *spec;
@@ -1418,11 +1422,13 @@ static int mlx5_vdpa_add_mac_vlan_rules(struct mlx5_vdpa_net *ndev, u8 *mac,
 	u8 *dmac_c;
 	u8 *dmac_v;
 	int err;
+	u16 vid;
 
 	spec = kvzalloc(sizeof(*spec), GFP_KERNEL);
 	if (!spec)
 		return -ENOMEM;
 
+	vid = key2vid(node->macvlan);
 	spec->match_criteria_enable = MLX5_MATCH_OUTER_HEADERS;
 	headers_c = MLX5_ADDR_OF(fte_match_param, spec->match_criteria, outer_headers);
 	headers_v = MLX5_ADDR_OF(fte_match_param, spec->match_value, outer_headers);
@@ -1431,45 +1437,78 @@ static int mlx5_vdpa_add_mac_vlan_rules(struct mlx5_vdpa_net *ndev, u8 *mac,
 	eth_broadcast_addr(dmac_c);
 	ether_addr_copy(dmac_v, mac);
 	MLX5_SET(fte_match_set_lyr_2_4, headers_c, cvlan_tag, 1);
-	if (tagged) {
+	if (node->tagged) {
 		MLX5_SET(fte_match_set_lyr_2_4, headers_v, cvlan_tag, 1);
 		MLX5_SET_TO_ONES(fte_match_set_lyr_2_4, headers_c, first_vid);
 		MLX5_SET(fte_match_set_lyr_2_4, headers_v, first_vid, vid);
 	}
 	flow_act.action = MLX5_FLOW_CONTEXT_ACTION_FWD_DEST;
-	dest.type = MLX5_FLOW_DESTINATION_TYPE_TIR;
-	dest.tir_num = ndev->res.tirn;
-	rule = mlx5_add_flow_rules(ndev->rxft, spec, &flow_act, &dest, 1);
+	dests[0].type = MLX5_FLOW_DESTINATION_TYPE_TIR;
+	dests[0].tir_num = ndev->res.tirn;
+#if defined(CONFIG_MLX5_VDPA_STEERING_DEBUG)
+	node->ucast_counter.counter = mlx5_fc_create(ndev->mvdev.mdev, false);
+	if (IS_ERR(node->ucast_counter.counter)) {
+		err = PTR_ERR(node->ucast_counter.counter);
+		goto err_ucast_counter;
+	}
+	node->mcast_counter.counter = mlx5_fc_create(ndev->mvdev.mdev, false);
+	if (IS_ERR(node->mcast_counter.counter)) {
+		err = PTR_ERR(node->mcast_counter.counter);
+		goto err_mcast_counter;
+	}
+
+	dests[1].type = MLX5_FLOW_DESTINATION_TYPE_COUNTER;
+	dests[1].counter_id = mlx5_fc_id(node->ucast_counter.counter);
+	flow_act.action |= MLX5_FLOW_CONTEXT_ACTION_COUNT;
+#endif
+	node->ucast_rule = mlx5_add_flow_rules(ndev->rxft, spec, &flow_act, dests, NUM_DESTS);
 	if (IS_ERR(rule))
 		return PTR_ERR(rule);
 
-	*ucast = rule;
+#if defined(CONFIG_MLX5_VDPA_STEERING_DEBUG)
+	dests[1].counter_id = mlx5_fc_id(node->mcast_counter.counter);
+#endif
 
 	memset(dmac_c, 0, ETH_ALEN);
 	memset(dmac_v, 0, ETH_ALEN);
 	dmac_c[0] = 1;
 	dmac_v[0] = 1;
-	rule = mlx5_add_flow_rules(ndev->rxft, spec, &flow_act, &dest, 1);
+	node->mcast_rule = mlx5_add_flow_rules(ndev->rxft, spec, &flow_act, dests, NUM_DESTS);
 	kvfree(spec);
 	if (IS_ERR(rule)) {
 		err = PTR_ERR(rule);
 		goto err_mcast;
 	}
 
-	*mcast = rule;
+#if defined(CONFIG_MLX5_VDPA_STEERING_DEBUG)
+	node->ucast_counter.mdev = ndev->mvdev.mdev;
+	node->mcast_counter.mdev = ndev->mvdev.mdev;
+#endif
+	mlx5_vdpa_add_rx_counters(ndev, node);
+
 	return 0;
 
 err_mcast:
-	mlx5_del_flow_rules(*ucast);
+	mlx5_del_flow_rules(node->ucast_rule);
+#if defined(CONFIG_MLX5_VDPA_STEERING_DEBUG)
+err_mcast_counter:
+	mlx5_fc_destroy(ndev->mvdev.mdev, node->ucast_counter.counter);
+err_ucast_counter:
+	kfree(spec);
+#endif
 	return err;
 }
 
 static void mlx5_vdpa_del_mac_vlan_rules(struct mlx5_vdpa_net *ndev,
-					 struct mlx5_flow_handle *ucast,
-					 struct mlx5_flow_handle *mcast)
+					 struct macvlan_node *node)
 {
-	mlx5_del_flow_rules(ucast);
-	mlx5_del_flow_rules(mcast);
+#if defined(CONFIG_MLX5_VDPA_STEERING_DEBUG)
+	if (node->dent && !IS_ERR(node->dent))
+		mlx5_vdpa_remove_rx_counters(ndev, node);
+#endif
+
+	mlx5_del_flow_rules(node->ucast_rule);
+	mlx5_del_flow_rules(node->mcast_rule);
 }
 
 static u64 search_val(u8 *mac, u16 vlan, bool tagged)
@@ -1503,14 +1542,14 @@ static struct macvlan_node *mac_vlan_lookup(struct mlx5_vdpa_net *ndev, u64 valu
 	return NULL;
 }
 
-static int mac_vlan_add(struct mlx5_vdpa_net *ndev, u8 *mac, u16 vlan, bool tagged) // vlan -> vid
+static int mac_vlan_add(struct mlx5_vdpa_net *ndev, u8 *mac, u16 vid, bool tagged)
 {
 	struct macvlan_node *ptr;
 	u64 val;
 	u32 idx;
 	int err;
 
-	val = search_val(mac, vlan, tagged);
+	val = search_val(mac, vid, tagged);
 	if (mac_vlan_lookup(ndev, val))
 		return -EEXIST;
 
@@ -1518,12 +1557,13 @@ static int mac_vlan_add(struct mlx5_vdpa_net *ndev, u8 *mac, u16 vlan, bool tagg
 	if (!ptr)
 		return -ENOMEM;
 
-	err = mlx5_vdpa_add_mac_vlan_rules(ndev, ndev->config.mac, vlan, tagged,
-					   &ptr->ucast_rule, &ptr->mcast_rule);
+	ptr->tagged = tagged;
+	ptr->macvlan = val;
+	ptr->ndev = ndev;
+	err = mlx5_vdpa_add_mac_vlan_rules(ndev, ndev->config.mac, ptr);
 	if (err)
 		goto err_add;
 
-	ptr->macvlan = val;
 	idx = hash_64(val, 8);
 	hlist_add_head(&ptr->hlist, &ndev->macvlan_hash[idx]);
 	return 0;
@@ -1542,7 +1582,11 @@ static void mac_vlan_del(struct mlx5_vdpa_net *ndev, u8 *mac, u16 vlan, bool tag
 		return;
 
 	hlist_del(&ptr->hlist);
-	mlx5_vdpa_del_mac_vlan_rules(ndev, ptr->ucast_rule, ptr->mcast_rule);
+	mlx5_vdpa_del_mac_vlan_rules(ndev, ptr);
+#if defined(CONFIG_MLX5_VDPA_STEERING_DEBUG)
+			mlx5_fc_destroy(ndev->mvdev.mdev, ptr->mcast_counter.counter);
+			mlx5_fc_destroy(ndev->mvdev.mdev, ptr->ucast_counter.counter);
+#endif
 	kfree(ptr);
 }
 
@@ -1555,7 +1599,11 @@ static void clear_mac_vlan_table(struct mlx5_vdpa_net *ndev)
 	for (i = 0; i < MLX5V_MACVLAN_SIZE; i++) {
 		hlist_for_each_entry_safe(pos, n, &ndev->macvlan_hash[i], hlist) {
 			hlist_del(&pos->hlist);
-			mlx5_vdpa_del_mac_vlan_rules(ndev, pos->ucast_rule, pos->mcast_rule);
+			mlx5_vdpa_del_mac_vlan_rules(ndev, pos);
+#if defined(CONFIG_MLX5_VDPA_STEERING_DEBUG)
+			mlx5_fc_destroy(ndev->mvdev.mdev, pos->mcast_counter.counter);
+			mlx5_fc_destroy(ndev->mvdev.mdev, pos->ucast_counter.counter);
+#endif
 			kfree(pos);
 		}
 	}
diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.h b/drivers/vdpa/mlx5/net/mlx5_vnet.h
index f2cef3925e5b..706c83016b71 100644
--- a/drivers/vdpa/mlx5/net/mlx5_vnet.h
+++ b/drivers/vdpa/mlx5/net/mlx5_vnet.h
@@ -21,6 +21,11 @@ struct mlx5_vdpa_net_resources {
 
 #define MLX5V_MACVLAN_SIZE 256
 
+static inline u16 key2vid(u64 key)
+{
+	return (u16)(key >> 48);
+}
+
 struct mlx5_vdpa_net {
 	struct mlx5_vdpa_dev mvdev;
 	struct mlx5_vdpa_net_resources res;
@@ -47,11 +52,24 @@ struct mlx5_vdpa_net {
 	struct dentry *debugfs;
 };
 
+struct mlx5_vdpa_counter {
+	struct mlx5_fc *counter;
+	struct dentry *dent;
+	struct mlx5_core_dev *mdev;
+};
+
 struct macvlan_node {
 	struct hlist_node hlist;
 	struct mlx5_flow_handle *ucast_rule;
 	struct mlx5_flow_handle *mcast_rule;
 	u64 macvlan;
+	struct mlx5_vdpa_net *ndev;
+	bool tagged;
+#if defined(CONFIG_MLX5_VDPA_STEERING_DEBUG)
+	struct dentry *dent;
+	struct mlx5_vdpa_counter ucast_counter;
+	struct mlx5_vdpa_counter mcast_counter;
+#endif
 };
 
 void mlx5_vdpa_add_debugfs(struct mlx5_vdpa_net *ndev);
@@ -60,5 +78,17 @@ void mlx5_vdpa_add_rx_flow_table(struct mlx5_vdpa_net *ndev);
 void mlx5_vdpa_remove_rx_flow_table(struct mlx5_vdpa_net *ndev);
 void mlx5_vdpa_add_tirn(struct mlx5_vdpa_net *ndev);
 void mlx5_vdpa_remove_tirn(struct mlx5_vdpa_net *ndev);
+#if defined(CONFIG_MLX5_VDPA_STEERING_DEBUG)
+void mlx5_vdpa_add_rx_counters(struct mlx5_vdpa_net *ndev,
+			       struct macvlan_node *node);
+void mlx5_vdpa_remove_rx_counters(struct mlx5_vdpa_net *ndev,
+				  struct macvlan_node *node);
+#else
+static inline void mlx5_vdpa_add_rx_counters(struct mlx5_vdpa_net *ndev,
+					     struct macvlan_node *node) {}
+static inline void mlx5_vdpa_remove_rx_counters(struct mlx5_vdpa_net *ndev,
+						struct macvlan_node *node) {}
+#endif
+
 
 #endif /* __MLX5_VNET_H__ */
-- 
2.35.1