Re: [PATCH 0/7] vdpa/mlx5: Add debugfs subtree and fixes

Re: [PATCH 0/7] vdpa/mlx5: Add debugfs subtree and fixes

[Michael S. Tsirkin]

On Sun, Nov 13, 2022 at 03:44:35PM +0200, Eli Cohen wrote:
> This series is a resend of previously sent patch list. It adds a few
> fixes so I treat as a v0 of a new series.
> 
> It adds a kernel config param CONFIG_MLX5_VDPA_STEERING_DEBUG that when
> eabled allows to read rx unicast and multicast counters per tagged or untagged traffic.
> 
> Examples:
> $ cat /sys/kernel/debug/mlx5/mlx5_core.sf.1/vdpa-0/rx/untagged/mcast/packets
> $ cat /sys/kernel/debug/mlx5/mlx5_core.sf.1/vdpa-0/rx/untagged/ucast/bytes


Thanks!
Is this patchset intended for 6.2 then?

> 
> Eli Cohen (7):
>   vdpa/mlx5: Fix rule forwarding VLAN to TIR
>   vdpa/mlx5: Fix wrong mac address deletion
>   vdpa/mlx5: Move some definitions to a new header file
>   vdpa/mlx5: Add debugfs subtree
>   vdpa/mlx5: Add RX counters to debugfs
>   vdpa/mlx5: Avoid using reslock in event_handler
>   vdpa/mlx5: Avoid overwriting CVQ iotlb
> 
>  drivers/vdpa/Kconfig               |  12 ++
>  drivers/vdpa/mlx5/Makefile         |   2 +-
>  drivers/vdpa/mlx5/core/mlx5_vdpa.h |   5 +-
>  drivers/vdpa/mlx5/core/mr.c        |  44 ++---
>  drivers/vdpa/mlx5/net/debug.c      | 152 ++++++++++++++++++
>  drivers/vdpa/mlx5/net/mlx5_vnet.c  | 248 ++++++++++++++---------------
>  drivers/vdpa/mlx5/net/mlx5_vnet.h  |  94 +++++++++++
>  7 files changed, 411 insertions(+), 146 deletions(-)
>  create mode 100644 drivers/vdpa/mlx5/net/debug.c
>  create mode 100644 drivers/vdpa/mlx5/net/mlx5_vnet.h
> 
> -- 
> 2.38.1

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH 0/7] vdpa/mlx5: Add debugfs subtree and fixes

[Michael S. Tsirkin]

On Sun, Nov 13, 2022 at 01:56:00PM +0000, Eli Cohen wrote:
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Sunday, 13 November 2022 15:52
> > To: Eli Cohen <elic@nvidia.com>
> > Cc: jasowang@redhat.com; linux-kernel@vger.kernel.org;
> > virtualization@lists.linux-foundation.org; si-wei.liu@oracle.com;
> > eperezma@redhat.com; lulu@redhat.com
> > Subject: Re: [PATCH 0/7] vdpa/mlx5: Add debugfs subtree and fixes
> > 
> > On Sun, Nov 13, 2022 at 03:44:35PM +0200, Eli Cohen wrote:
> > > This series is a resend of previously sent patch list. It adds a few
> > > fixes so I treat as a v0 of a new series.
> > >
> > > It adds a kernel config param CONFIG_MLX5_VDPA_STEERING_DEBUG that
> > when
> > > eabled allows to read rx unicast and multicast counters per tagged or
> > untagged traffic.
> > >
> > > Examples:
> > > $ cat /sys/kernel/debug/mlx5/mlx5_core.sf.1/vdpa-
> > 0/rx/untagged/mcast/packets
> > > $ cat /sys/kernel/debug/mlx5/mlx5_core.sf.1/vdpa-
> > 0/rx/untagged/ucast/bytes
> > 
> > 
> > Thanks!
> > Is this patchset intended for 6.2 then?
> > 
> 
> The debugfs part can go to 6.2 but maybe the fixes should go to 6.1.


Tat's fine - do they apply cleanly if I just drop 4/7?

> > >
> > > Eli Cohen (7):
> > >   vdpa/mlx5: Fix rule forwarding VLAN to TIR
> > >   vdpa/mlx5: Fix wrong mac address deletion
> > >   vdpa/mlx5: Move some definitions to a new header file
> > >   vdpa/mlx5: Add debugfs subtree
> > >   vdpa/mlx5: Add RX counters to debugfs
> > >   vdpa/mlx5: Avoid using reslock in event_handler
> > >   vdpa/mlx5: Avoid overwriting CVQ iotlb
> > >
> > >  drivers/vdpa/Kconfig               |  12 ++
> > >  drivers/vdpa/mlx5/Makefile         |   2 +-
> > >  drivers/vdpa/mlx5/core/mlx5_vdpa.h |   5 +-
> > >  drivers/vdpa/mlx5/core/mr.c        |  44 ++---
> > >  drivers/vdpa/mlx5/net/debug.c      | 152 ++++++++++++++++++
> > >  drivers/vdpa/mlx5/net/mlx5_vnet.c  | 248 ++++++++++++++---------------
> > >  drivers/vdpa/mlx5/net/mlx5_vnet.h  |  94 +++++++++++
> > >  7 files changed, 411 insertions(+), 146 deletions(-)
> > >  create mode 100644 drivers/vdpa/mlx5/net/debug.c
> > >  create mode 100644 drivers/vdpa/mlx5/net/mlx5_vnet.h
> > >
> > > --
> > > 2.38.1
> 

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH 1/7] vdpa/mlx5: Fix rule forwarding VLAN to TIR

[Jason Wang]

On Sun, Nov 13, 2022 at 9:45 PM Eli Cohen <elic@nvidia.com> wrote:
>
> Set the VLAN id to the header values field instead of overwriting the
> headers criteria field.
>
> Before this fix, VLAN filtering would not really work and tagged packets
> would be forwarded unfiltered to the TIR.
>
> In addition modify the logic so that VLAN filtering is enforced only
> when VIRTIO_NET_F_CTRL_VLAN is negotiated. When not negotiated, all
> incoming traffic is accepted as long as it is targeting the net device's
> MAC address.
>
> Fixes: baf2ad3f6a98 ("vdpa/mlx5: Add RX MAC VLAN filter support")
>
> Signed-off-by: Eli Cohen <elic@nvidia.com>
> ---
>  drivers/vdpa/mlx5/net/mlx5_vnet.c | 11 ++++++++---
>  1 file changed, 8 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c
> index 90913365def4..ea95081eca0c 100644
> --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c
> +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c
> @@ -1468,11 +1468,13 @@ static int mlx5_vdpa_add_mac_vlan_rules(struct mlx5_vdpa_net *ndev, u8 *mac,
>         dmac_v = MLX5_ADDR_OF(fte_match_param, headers_v, outer_headers.dmac_47_16);
>         eth_broadcast_addr(dmac_c);
>         ether_addr_copy(dmac_v, mac);
> -       MLX5_SET(fte_match_set_lyr_2_4, headers_c, cvlan_tag, 1);
> +       if (ndev->mvdev.actual_features & VIRTIO_NET_F_CTRL_VLAN) {
> +               MLX5_SET(fte_match_set_lyr_2_4, headers_c, cvlan_tag, 1);
> +               MLX5_SET_TO_ONES(fte_match_set_lyr_2_4, headers_c, first_vid);
> +       }
>         if (tagged) {
>                 MLX5_SET(fte_match_set_lyr_2_4, headers_v, cvlan_tag, 1);
> -               MLX5_SET_TO_ONES(fte_match_set_lyr_2_4, headers_c, first_vid);
> -               MLX5_SET(fte_match_set_lyr_2_4, headers_c, first_vid, vid);
> +               MLX5_SET(fte_match_set_lyr_2_4, headers_v, first_vid, vid);
>         }
>         flow_act.action = MLX5_FLOW_CONTEXT_ACTION_FWD_DEST;
>         dest.type = MLX5_FLOW_DESTINATION_TYPE_TIR;
> @@ -1821,6 +1823,9 @@ static virtio_net_ctrl_ack handle_ctrl_vlan(struct mlx5_vdpa_dev *mvdev, u8 cmd)
>         size_t read;
>         u16 id;
>
> +       if (!(ndev->mvdev.actual_features & BIT_ULL(VIRTIO_NET_F_CTRL_VLAN)))
> +               return status;

Nit: this seems unrelated to the patch.

Other than this.

Acked-by: Jason Wang <jasowang@redhat.com>

> +
>         switch (cmd) {
>         case VIRTIO_NET_CTRL_VLAN_ADD:
>                 read = vringh_iov_pull_iotlb(&cvq->vring, &cvq->riov, &vlan, sizeof(vlan));
> --
> 2.38.1
>

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH 1/7] vdpa/mlx5: Fix rule forwarding VLAN to TIR

[Michael S. Tsirkin]

On Mon, Nov 14, 2022 at 06:44:30AM +0000, Eli Cohen wrote:
> > From: Jason Wang <jasowang@redhat.com>
> > Sent: Monday, 14 November 2022 6:39
> > To: Eli Cohen <elic@nvidia.com>
> > Cc: mst@redhat.com; linux-kernel@vger.kernel.org; virtualization@lists.linux-
> > foundation.org; si-wei.liu@oracle.com; eperezma@redhat.com;
> > lulu@redhat.com
> > Subject: Re: [PATCH 1/7] vdpa/mlx5: Fix rule forwarding VLAN to TIR
> > 
> > On Sun, Nov 13, 2022 at 9:45 PM Eli Cohen <elic@nvidia.com> wrote:
> > >
> > > Set the VLAN id to the header values field instead of overwriting the
> > > headers criteria field.
> > >
> > > Before this fix, VLAN filtering would not really work and tagged packets
> > > would be forwarded unfiltered to the TIR.
> > >
> > > In addition modify the logic so that VLAN filtering is enforced only
> > > when VIRTIO_NET_F_CTRL_VLAN is negotiated. When not negotiated, all
> > > incoming traffic is accepted as long as it is targeting the net device's
> > > MAC address.
> > >
> > > Fixes: baf2ad3f6a98 ("vdpa/mlx5: Add RX MAC VLAN filter support")
> > >
> > > Signed-off-by: Eli Cohen <elic@nvidia.com>
> > > ---
> > >  drivers/vdpa/mlx5/net/mlx5_vnet.c | 11 ++++++++---
> > >  1 file changed, 8 insertions(+), 3 deletions(-)
> > >
> > > diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > b/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > > index 90913365def4..ea95081eca0c 100644
> > > --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > > +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > > @@ -1468,11 +1468,13 @@ static int mlx5_vdpa_add_mac_vlan_rules(struct
> > mlx5_vdpa_net *ndev, u8 *mac,
> > >         dmac_v = MLX5_ADDR_OF(fte_match_param, headers_v,
> > outer_headers.dmac_47_16);
> > >         eth_broadcast_addr(dmac_c);
> > >         ether_addr_copy(dmac_v, mac);
> > > -       MLX5_SET(fte_match_set_lyr_2_4, headers_c, cvlan_tag, 1);
> > > +       if (ndev->mvdev.actual_features & VIRTIO_NET_F_CTRL_VLAN) {
> > > +               MLX5_SET(fte_match_set_lyr_2_4, headers_c, cvlan_tag, 1);
> > > +               MLX5_SET_TO_ONES(fte_match_set_lyr_2_4, headers_c, first_vid);
> > > +       }
> > >         if (tagged) {
> > >                 MLX5_SET(fte_match_set_lyr_2_4, headers_v, cvlan_tag, 1);
> > > -               MLX5_SET_TO_ONES(fte_match_set_lyr_2_4, headers_c, first_vid);
> > > -               MLX5_SET(fte_match_set_lyr_2_4, headers_c, first_vid, vid);
> > > +               MLX5_SET(fte_match_set_lyr_2_4, headers_v, first_vid, vid);
> > >         }
> > >         flow_act.action = MLX5_FLOW_CONTEXT_ACTION_FWD_DEST;
> > >         dest.type = MLX5_FLOW_DESTINATION_TYPE_TIR;
> > > @@ -1821,6 +1823,9 @@ static virtio_net_ctrl_ack handle_ctrl_vlan(struct
> > mlx5_vdpa_dev *mvdev, u8 cmd)
> > >         size_t read;
> > >         u16 id;
> > >
> > > +       if (!(ndev->mvdev.actual_features &
> > BIT_ULL(VIRTIO_NET_F_CTRL_VLAN)))
> > > +               return status;
> > 
> > Nit: this seems unrelated to the patch.
> > 
> Will put in another patch


OK since you are doing a new version, please split fixes and
features in two patchsets. Thanks!

> > Other than this.
> > 
> > Acked-by: Jason Wang <jasowang@redhat.com>
> > 
> > > +
> > >         switch (cmd) {
> > >         case VIRTIO_NET_CTRL_VLAN_ADD:
> > >                 read = vringh_iov_pull_iotlb(&cvq->vring, &cvq->riov, &vlan,
> > sizeof(vlan));
> > > --
> > > 2.38.1
> > >
> 

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH 2/7] vdpa/mlx5: Fix wrong mac address deletion

[Jason Wang]

On Sun, Nov 13, 2022 at 9:45 PM Eli Cohen <elic@nvidia.com> wrote:
>
> Delete the old MAC from the table and not the new one which is not there
> yet.
>
> Fixes: baf2ad3f6a98 ("vdpa/mlx5: Add RX MAC VLAN filter support")
> Signed-off-by: Eli Cohen <elic@nvidia.com>

Acked-by: Jason Wang <jasowang@redhat.com>

> ---
>  drivers/vdpa/mlx5/net/mlx5_vnet.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c
> index ea95081eca0c..c54d5088ed4f 100644
> --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c
> +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c
> @@ -1686,7 +1686,7 @@ static virtio_net_ctrl_ack handle_ctrl_mac(struct mlx5_vdpa_dev *mvdev, u8 cmd)
>
>                 /* Need recreate the flow table entry, so that the packet could forward back
>                  */
> -               mac_vlan_del(ndev, ndev->config.mac, 0, false);
> +               mac_vlan_del(ndev, mac_back, 0, false);
>
>                 if (mac_vlan_add(ndev, ndev->config.mac, 0, false)) {
>                         mlx5_vdpa_warn(mvdev, "failed to insert forward rules, try to restore\n");
> --
> 2.38.1
>

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH 3/7] vdpa/mlx5: Move some definitions to a new header file

[Jason Wang]

On Sun, Nov 13, 2022 at 9:45 PM Eli Cohen <elic@nvidia.com> wrote:
>
> Move some definitions from mlx5_vnet.c to newly added header file
> mlx5_vnet.h. We need these definitions for the following patches that
> add debugfs tree to expose information vital for debug.
>
> Reviewed-by: Si-Wei Liu <si-wei.liu@oracle.com>
> Signed-off-by: Eli Cohen <elic@nvidia.com>

Acked-by: Jason Wang <jasowang@redhat.com>

Thanks

> ---
>  drivers/vdpa/mlx5/net/mlx5_vnet.c | 45 +------------------------
>  drivers/vdpa/mlx5/net/mlx5_vnet.h | 55 +++++++++++++++++++++++++++++++
>  2 files changed, 56 insertions(+), 44 deletions(-)
>  create mode 100644 drivers/vdpa/mlx5/net/mlx5_vnet.h
>
> diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c
> index c54d5088ed4f..e45b83527b41 100644
> --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c
> +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c
> @@ -18,15 +18,12 @@
>  #include <linux/mlx5/mlx5_ifc_vdpa.h>
>  #include <linux/mlx5/mpfs.h>
>  #include "mlx5_vdpa.h"
> +#include "mlx5_vnet.h"
>
>  MODULE_AUTHOR("Eli Cohen <eli@mellanox.com>");
>  MODULE_DESCRIPTION("Mellanox VDPA driver");
>  MODULE_LICENSE("Dual BSD/GPL");
>
> -#define to_mlx5_vdpa_ndev(__mvdev)                                             \
> -       container_of(__mvdev, struct mlx5_vdpa_net, mvdev)
> -#define to_mvdev(__vdev) container_of((__vdev), struct mlx5_vdpa_dev, vdev)
> -
>  #define VALID_FEATURES_MASK                                                                        \
>         (BIT_ULL(VIRTIO_NET_F_CSUM) | BIT_ULL(VIRTIO_NET_F_GUEST_CSUM) |                                   \
>          BIT_ULL(VIRTIO_NET_F_CTRL_GUEST_OFFLOADS) | BIT_ULL(VIRTIO_NET_F_MTU) | BIT_ULL(VIRTIO_NET_F_MAC) |   \
> @@ -50,14 +47,6 @@ MODULE_LICENSE("Dual BSD/GPL");
>
>  #define MLX5V_UNTAGGED 0x1000
>
> -struct mlx5_vdpa_net_resources {
> -       u32 tisn;
> -       u32 tdn;
> -       u32 tirn;
> -       u32 rqtn;
> -       bool valid;
> -};
> -
>  struct mlx5_vdpa_cq_buf {
>         struct mlx5_frag_buf_ctrl fbc;
>         struct mlx5_frag_buf frag_buf;
> @@ -146,38 +135,6 @@ static bool is_index_valid(struct mlx5_vdpa_dev *mvdev, u16 idx)
>         return idx <= mvdev->max_idx;
>  }
>
> -#define MLX5V_MACVLAN_SIZE 256
> -
> -struct mlx5_vdpa_net {
> -       struct mlx5_vdpa_dev mvdev;
> -       struct mlx5_vdpa_net_resources res;
> -       struct virtio_net_config config;
> -       struct mlx5_vdpa_virtqueue *vqs;
> -       struct vdpa_callback *event_cbs;
> -
> -       /* Serialize vq resources creation and destruction. This is required
> -        * since memory map might change and we need to destroy and create
> -        * resources while driver in operational.
> -        */
> -       struct rw_semaphore reslock;
> -       struct mlx5_flow_table *rxft;
> -       bool setup;
> -       u32 cur_num_vqs;
> -       u32 rqt_size;
> -       bool nb_registered;
> -       struct notifier_block nb;
> -       struct vdpa_callback config_cb;
> -       struct mlx5_vdpa_wq_ent cvq_ent;
> -       struct hlist_head macvlan_hash[MLX5V_MACVLAN_SIZE];
> -};
> -
> -struct macvlan_node {
> -       struct hlist_node hlist;
> -       struct mlx5_flow_handle *ucast_rule;
> -       struct mlx5_flow_handle *mcast_rule;
> -       u64 macvlan;
> -};
> -
>  static void free_resources(struct mlx5_vdpa_net *ndev);
>  static void init_mvqs(struct mlx5_vdpa_net *ndev);
>  static int setup_driver(struct mlx5_vdpa_dev *mvdev);
> diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.h b/drivers/vdpa/mlx5/net/mlx5_vnet.h
> new file mode 100644
> index 000000000000..6691c879a6ca
> --- /dev/null
> +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.h
> @@ -0,0 +1,55 @@
> +/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
> +/* Copyright (c) 2022, NVIDIA CORPORATION & AFFILIATES. All rights reserved. */
> +
> +#ifndef __MLX5_VNET_H__
> +#define __MLX5_VNET_H__
> +
> +#include "mlx5_vdpa.h"
> +
> +#define to_mlx5_vdpa_ndev(__mvdev)                                             \
> +       container_of(__mvdev, struct mlx5_vdpa_net, mvdev)
> +#define to_mvdev(__vdev) container_of((__vdev), struct mlx5_vdpa_dev, vdev)
> +
> +struct mlx5_vdpa_net_resources {
> +       u32 tisn;
> +       u32 tdn;
> +       u32 tirn;
> +       u32 rqtn;
> +       bool valid;
> +};
> +
> +#define MLX5V_MACVLAN_SIZE 256
> +
> +struct mlx5_vdpa_net {
> +       struct mlx5_vdpa_dev mvdev;
> +       struct mlx5_vdpa_net_resources res;
> +       struct virtio_net_config config;
> +       struct mlx5_vdpa_virtqueue *vqs;
> +       struct vdpa_callback *event_cbs;
> +
> +       /* Serialize vq resources creation and destruction. This is required
> +        * since memory map might change and we need to destroy and create
> +        * resources while driver in operational.
> +        */
> +       struct rw_semaphore reslock;
> +       struct mlx5_flow_table *rxft;
> +       struct dentry *rx_dent;
> +       struct dentry *rx_table_dent;
> +       bool setup;
> +       u32 cur_num_vqs;
> +       u32 rqt_size;
> +       bool nb_registered;
> +       struct notifier_block nb;
> +       struct vdpa_callback config_cb;
> +       struct mlx5_vdpa_wq_ent cvq_ent;
> +       struct hlist_head macvlan_hash[MLX5V_MACVLAN_SIZE];
> +};
> +
> +struct macvlan_node {
> +       struct hlist_node hlist;
> +       struct mlx5_flow_handle *ucast_rule;
> +       struct mlx5_flow_handle *mcast_rule;
> +       u64 macvlan;
> +};
> +
> +#endif /* __MLX5_VNET_H__ */
> --
> 2.38.1
>

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH 4/7] vdpa/mlx5: Add debugfs subtree

[Jason Wang]

On Sun, Nov 13, 2022 at 9:45 PM Eli Cohen <elic@nvidia.com> wrote:
>
> Add debugfs subtree and expose flow table ID and TIR number. This
> information can be used by external tools to do extended
> troubleshooting.
>
> The information can be retrieved like so:
> $ cat /sys/kernel/debug/mlx5/mlx5_core.sf.1/vdpa-0/rx/table_id
> $ cat /sys/kernel/debug/mlx5/mlx5_core.sf.1/vdpa-0/rx/tirn
>
> Reviewed-by: Si-Wei Liu <si-wei.liu@oracle.com>
> Signed-off-by: Eli Cohen <elic@nvidia.com>

Acked-by: Jason Wang <jasowang@redhat.com>

> ---
>  drivers/vdpa/mlx5/Makefile        |  2 +-
>  drivers/vdpa/mlx5/net/debug.c     | 66 +++++++++++++++++++++++++++++++
>  drivers/vdpa/mlx5/net/mlx5_vnet.c | 11 ++++++
>  drivers/vdpa/mlx5/net/mlx5_vnet.h |  9 +++++
>  4 files changed, 87 insertions(+), 1 deletion(-)
>  create mode 100644 drivers/vdpa/mlx5/net/debug.c
>
> diff --git a/drivers/vdpa/mlx5/Makefile b/drivers/vdpa/mlx5/Makefile
> index f717978c83bf..e791394c33e3 100644
> --- a/drivers/vdpa/mlx5/Makefile
> +++ b/drivers/vdpa/mlx5/Makefile
> @@ -1,4 +1,4 @@
>  subdir-ccflags-y += -I$(srctree)/drivers/vdpa/mlx5/core
>
>  obj-$(CONFIG_MLX5_VDPA_NET) += mlx5_vdpa.o
> -mlx5_vdpa-$(CONFIG_MLX5_VDPA_NET) += net/mlx5_vnet.o core/resources.o core/mr.o
> +mlx5_vdpa-$(CONFIG_MLX5_VDPA_NET) += net/mlx5_vnet.o core/resources.o core/mr.o net/debug.o
> diff --git a/drivers/vdpa/mlx5/net/debug.c b/drivers/vdpa/mlx5/net/debug.c
> new file mode 100644
> index 000000000000..95e4801df211
> --- /dev/null
> +++ b/drivers/vdpa/mlx5/net/debug.c
> @@ -0,0 +1,66 @@
> +// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
> +/* Copyright (c) 2022, NVIDIA CORPORATION & AFFILIATES. All rights reserved. */
> +
> +#include <linux/debugfs.h>
> +#include <linux/mlx5/fs.h>
> +#include "mlx5_vnet.h"
> +
> +static int tirn_show(struct seq_file *file, void *priv)
> +{
> +       struct mlx5_vdpa_net *ndev = file->private;
> +
> +       seq_printf(file, "0x%x\n", ndev->res.tirn);
> +       return 0;
> +}
> +
> +DEFINE_SHOW_ATTRIBUTE(tirn);
> +
> +void mlx5_vdpa_remove_tirn(struct mlx5_vdpa_net *ndev)
> +{
> +       if (ndev->debugfs)
> +               debugfs_remove(ndev->res.tirn_dent);
> +}
> +
> +void mlx5_vdpa_add_tirn(struct mlx5_vdpa_net *ndev)
> +{
> +       ndev->res.tirn_dent = debugfs_create_file("tirn", 0444, ndev->rx_dent,
> +                                                 ndev, &tirn_fops);
> +}
> +
> +static int rx_flow_table_show(struct seq_file *file, void *priv)
> +{
> +       struct mlx5_vdpa_net *ndev = file->private;
> +
> +       seq_printf(file, "0x%x\n", mlx5_flow_table_id(ndev->rxft));
> +       return 0;
> +}
> +
> +DEFINE_SHOW_ATTRIBUTE(rx_flow_table);
> +
> +void mlx5_vdpa_remove_rx_flow_table(struct mlx5_vdpa_net *ndev)
> +{
> +       if (ndev->debugfs)
> +               debugfs_remove(ndev->rx_table_dent);
> +}
> +
> +void mlx5_vdpa_add_rx_flow_table(struct mlx5_vdpa_net *ndev)
> +{
> +       ndev->rx_table_dent = debugfs_create_file("table_id", 0444, ndev->rx_dent,
> +                                                 ndev, &rx_flow_table_fops);
> +}
> +
> +void mlx5_vdpa_add_debugfs(struct mlx5_vdpa_net *ndev)
> +{
> +       struct mlx5_core_dev *mdev;
> +
> +       mdev = ndev->mvdev.mdev;
> +       ndev->debugfs = debugfs_create_dir(dev_name(&ndev->mvdev.vdev.dev),
> +                                          mlx5_debugfs_get_dev_root(mdev));
> +       if (!IS_ERR(ndev->debugfs))
> +               ndev->rx_dent = debugfs_create_dir("rx", ndev->debugfs);
> +}
> +
> +void mlx5_vdpa_remove_debugfs(struct dentry *dbg)
> +{
> +       debugfs_remove_recursive(dbg);
> +}
> diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c
> index e45b83527b41..ce2f478a93f5 100644
> --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c
> +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c
> @@ -1388,11 +1388,16 @@ static int create_tir(struct mlx5_vdpa_net *ndev)
>
>         err = mlx5_vdpa_create_tir(&ndev->mvdev, in, &ndev->res.tirn);
>         kfree(in);
> +       if (err)
> +               return err;
> +
> +       mlx5_vdpa_add_tirn(ndev);
>         return err;
>  }
>
>  static void destroy_tir(struct mlx5_vdpa_net *ndev)
>  {
> +       mlx5_vdpa_remove_tirn(ndev);
>         mlx5_vdpa_destroy_tir(&ndev->mvdev, ndev->res.tirn);
>  }
>
> @@ -1578,6 +1583,7 @@ static int setup_steering(struct mlx5_vdpa_net *ndev)
>                 mlx5_vdpa_warn(&ndev->mvdev, "failed to create flow table\n");
>                 return PTR_ERR(ndev->rxft);
>         }
> +       mlx5_vdpa_add_rx_flow_table(ndev);
>
>         err = mac_vlan_add(ndev, ndev->config.mac, 0, false);
>         if (err)
> @@ -1586,6 +1592,7 @@ static int setup_steering(struct mlx5_vdpa_net *ndev)
>         return 0;
>
>  err_add:
> +       mlx5_vdpa_remove_rx_flow_table(ndev);
>         mlx5_destroy_flow_table(ndev->rxft);
>         return err;
>  }
> @@ -1593,6 +1600,7 @@ static int setup_steering(struct mlx5_vdpa_net *ndev)
>  static void teardown_steering(struct mlx5_vdpa_net *ndev)
>  {
>         clear_mac_vlan_table(ndev);
> +       mlx5_vdpa_remove_rx_flow_table(ndev);
>         mlx5_destroy_flow_table(ndev->rxft);
>  }
>
> @@ -3172,6 +3180,7 @@ static int mlx5_vdpa_dev_add(struct vdpa_mgmt_dev *v_mdev, const char *name,
>         if (err)
>                 goto err_reg;
>
> +       mlx5_vdpa_add_debugfs(ndev);
>         mgtdev->ndev = ndev;
>         return 0;
>
> @@ -3198,6 +3207,8 @@ static void mlx5_vdpa_dev_del(struct vdpa_mgmt_dev *v_mdev, struct vdpa_device *
>         struct mlx5_vdpa_net *ndev = to_mlx5_vdpa_ndev(mvdev);
>         struct workqueue_struct *wq;
>
> +       mlx5_vdpa_remove_debugfs(ndev->debugfs);
> +       ndev->debugfs = NULL;
>         if (ndev->nb_registered) {
>                 mlx5_notifier_unregister(mvdev->mdev, &ndev->nb);
>                 ndev->nb_registered = false;
> diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.h b/drivers/vdpa/mlx5/net/mlx5_vnet.h
> index 6691c879a6ca..f2cef3925e5b 100644
> --- a/drivers/vdpa/mlx5/net/mlx5_vnet.h
> +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.h
> @@ -16,6 +16,7 @@ struct mlx5_vdpa_net_resources {
>         u32 tirn;
>         u32 rqtn;
>         bool valid;
> +       struct dentry *tirn_dent;
>  };
>
>  #define MLX5V_MACVLAN_SIZE 256
> @@ -43,6 +44,7 @@ struct mlx5_vdpa_net {
>         struct vdpa_callback config_cb;
>         struct mlx5_vdpa_wq_ent cvq_ent;
>         struct hlist_head macvlan_hash[MLX5V_MACVLAN_SIZE];
> +       struct dentry *debugfs;
>  };
>
>  struct macvlan_node {
> @@ -52,4 +54,11 @@ struct macvlan_node {
>         u64 macvlan;
>  };
>
> +void mlx5_vdpa_add_debugfs(struct mlx5_vdpa_net *ndev);
> +void mlx5_vdpa_remove_debugfs(struct dentry *dbg);
> +void mlx5_vdpa_add_rx_flow_table(struct mlx5_vdpa_net *ndev);
> +void mlx5_vdpa_remove_rx_flow_table(struct mlx5_vdpa_net *ndev);
> +void mlx5_vdpa_add_tirn(struct mlx5_vdpa_net *ndev);
> +void mlx5_vdpa_remove_tirn(struct mlx5_vdpa_net *ndev);
> +
>  #endif /* __MLX5_VNET_H__ */
> --
> 2.38.1
>

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH 5/7] vdpa/mlx5: Add RX counters to debugfs

[Jason Wang]

On Sun, Nov 13, 2022 at 9:45 PM Eli Cohen <elic@nvidia.com> wrote:
>
> For each interface, either VLAN tagged or untagged, add two hardware
> counters: one for unicast and another for multicast. The counters count
> RX packets and bytes and can be read through debugfs:
>
> $ cat /sys/kernel/debug/mlx5/mlx5_core.sf.1/vdpa-0/rx/untagged/mcast/packets
> $ cat /sys/kernel/debug/mlx5/mlx5_core.sf.1/vdpa-0/rx/untagged/ucast/bytes
>
> This feature is controlled via the config option
> MLX5_VDPA_STEERING_DEBUG. It is off by default as it may have some
> impact on performance.
>
> Signed-off-by: Eli Cohen <elic@nvidia.com>
> ---
>  drivers/vdpa/Kconfig              |  12 +++
>  drivers/vdpa/mlx5/net/debug.c     |  86 ++++++++++++++++++++++
>  drivers/vdpa/mlx5/net/mlx5_vnet.c | 118 +++++++++++++++++++++++-------
>  drivers/vdpa/mlx5/net/mlx5_vnet.h |  30 ++++++++
>  4 files changed, 218 insertions(+), 28 deletions(-)
>

[...]

> +       vid = key2vid(node->macvlan);
>         spec->match_criteria_enable = MLX5_MATCH_OUTER_HEADERS;
>         headers_c = MLX5_ADDR_OF(fte_match_param, spec->match_criteria, outer_headers);
>         headers_v = MLX5_ADDR_OF(fte_match_param, spec->match_value, outer_headers);
> @@ -1430,48 +1475,62 @@ static int mlx5_vdpa_add_mac_vlan_rules(struct mlx5_vdpa_net *ndev, u8 *mac,
>         dmac_v = MLX5_ADDR_OF(fte_match_param, headers_v, outer_headers.dmac_47_16);
>         eth_broadcast_addr(dmac_c);
>         ether_addr_copy(dmac_v, mac);
> -       if (ndev->mvdev.actual_features & VIRTIO_NET_F_CTRL_VLAN) {
> +       if (ndev->mvdev.actual_features & BIT_ULL(VIRTIO_NET_F_CTRL_VLAN)) {


This seems like a fix for patch 1? If yes, let's just squash this.

Thanks

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH 6/7] vdpa/mlx5: Avoid using reslock in event_handler

[Jason Wang]

On Sun, Nov 13, 2022 at 9:45 PM Eli Cohen <elic@nvidia.com> wrote:
>
> event_handler runs under atomic context and may not acquire reslock. We
> can still guarantee that the handler won't be called after suspend by
> clearing nb_registered, unregistering the handler and flushing the
> workqueue.
>
> Signed-off-by: Eli Cohen <elic@nvidia.com>
> ---
>  drivers/vdpa/mlx5/net/mlx5_vnet.c | 14 +++-----------
>  1 file changed, 3 insertions(+), 11 deletions(-)
>
> diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c
> index 6e6490c85be2..bebfba530247 100644
> --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c
> +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c
> @@ -2872,8 +2872,8 @@ static int mlx5_vdpa_suspend(struct vdpa_device *vdev)
>         int i;
>
>         down_write(&ndev->reslock);
> -       mlx5_notifier_unregister(mvdev->mdev, &ndev->nb);
>         ndev->nb_registered = false;
> +       mlx5_notifier_unregister(mvdev->mdev, &ndev->nb);

I wonder why this can help anything. And if it does, we have simliar
logic in mlx5_vdpa_dev_del() do we need to fix that as well?

Thanks

>         flush_workqueue(ndev->mvdev.wq);
>         for (i = 0; i < ndev->cur_num_vqs; i++) {
>                 mvq = &ndev->vqs[i];
> @@ -3051,7 +3051,7 @@ static void update_carrier(struct work_struct *work)
>         else
>                 ndev->config.status &= cpu_to_mlx5vdpa16(mvdev, ~VIRTIO_NET_S_LINK_UP);
>
> -       if (ndev->config_cb.callback)
> +       if (ndev->nb_registered && ndev->config_cb.callback)
>                 ndev->config_cb.callback(ndev->config_cb.private);
>
>         kfree(wqent);
> @@ -3068,21 +3068,13 @@ static int event_handler(struct notifier_block *nb, unsigned long event, void *p
>                 switch (eqe->sub_type) {
>                 case MLX5_PORT_CHANGE_SUBTYPE_DOWN:
>                 case MLX5_PORT_CHANGE_SUBTYPE_ACTIVE:
> -                       down_read(&ndev->reslock);
> -                       if (!ndev->nb_registered) {
> -                               up_read(&ndev->reslock);
> -                               return NOTIFY_DONE;
> -                       }
>                         wqent = kzalloc(sizeof(*wqent), GFP_ATOMIC);
> -                       if (!wqent) {
> -                               up_read(&ndev->reslock);
> +                       if (!wqent)
>                                 return NOTIFY_DONE;
> -                       }
>
>                         wqent->mvdev = &ndev->mvdev;
>                         INIT_WORK(&wqent->work, update_carrier);
>                         queue_work(ndev->mvdev.wq, &wqent->work);
> -                       up_read(&ndev->reslock);
>                         ret = NOTIFY_OK;
>                         break;
>                 default:
> --
> 2.38.1
>

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH 6/7] vdpa/mlx5: Avoid using reslock in event_handler

[Jason Wang]

On Mon, Nov 14, 2022 at 4:58 PM Eli Cohen <elic@nvidia.com> wrote:
>
> > From: Jason Wang <jasowang@redhat.com>
> > Sent: Monday, 14 November 2022 9:53
> > To: Eli Cohen <elic@nvidia.com>
> > Cc: mst@redhat.com; linux-kernel@vger.kernel.org; virtualization@lists.linux-
> > foundation.org; si-wei.liu@oracle.com; eperezma@redhat.com;
> > lulu@redhat.com
> > Subject: Re: [PATCH 6/7] vdpa/mlx5: Avoid using reslock in event_handler
> >
> > On Sun, Nov 13, 2022 at 9:45 PM Eli Cohen <elic@nvidia.com> wrote:
> > >
> > > event_handler runs under atomic context and may not acquire reslock. We
> > > can still guarantee that the handler won't be called after suspend by
> > > clearing nb_registered, unregistering the handler and flushing the
> > > workqueue.
> > >
> > > Signed-off-by: Eli Cohen <elic@nvidia.com>
> > > ---
> > >  drivers/vdpa/mlx5/net/mlx5_vnet.c | 14 +++-----------
> > >  1 file changed, 3 insertions(+), 11 deletions(-)
> > >
> > > diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > b/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > > index 6e6490c85be2..bebfba530247 100644
> > > --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > > +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > > @@ -2872,8 +2872,8 @@ static int mlx5_vdpa_suspend(struct vdpa_device
> > *vdev)
> > >         int i;
> > >
> > >         down_write(&ndev->reslock);
> > > -       mlx5_notifier_unregister(mvdev->mdev, &ndev->nb);
> > >         ndev->nb_registered = false;
> > > +       mlx5_notifier_unregister(mvdev->mdev, &ndev->nb);
> >
> > I wonder why this can help anything.
> I think you were concerned that async events will come when the device was suspended. Since we can't take reslock, I think this guarantees that we won't get any events after suspension.
>

Ok, I see.

> > And if it does, we have simliar
> > logic in mlx5_vdpa_dev_del() do we need to fix that as well?
> >
> We have the same construct there only that I set nb_registered = false after unregistering the notifier. So I probably need to move it before mlx5_notifier_unregister().

Right.

Thanks

>
> > Thanks
> >
> > >         flush_workqueue(ndev->mvdev.wq);
> > >         for (i = 0; i < ndev->cur_num_vqs; i++) {
> > >                 mvq = &ndev->vqs[i];
> > > @@ -3051,7 +3051,7 @@ static void update_carrier(struct work_struct
> > *work)
> > >         else
> > >                 ndev->config.status &= cpu_to_mlx5vdpa16(mvdev,
> > ~VIRTIO_NET_S_LINK_UP);
> > >
> > > -       if (ndev->config_cb.callback)
> > > +       if (ndev->nb_registered && ndev->config_cb.callback)
> > >                 ndev->config_cb.callback(ndev->config_cb.private);
> > >
> > >         kfree(wqent);
> > > @@ -3068,21 +3068,13 @@ static int event_handler(struct notifier_block
> > *nb, unsigned long event, void *p
> > >                 switch (eqe->sub_type) {
> > >                 case MLX5_PORT_CHANGE_SUBTYPE_DOWN:
> > >                 case MLX5_PORT_CHANGE_SUBTYPE_ACTIVE:
> > > -                       down_read(&ndev->reslock);
> > > -                       if (!ndev->nb_registered) {
> > > -                               up_read(&ndev->reslock);
> > > -                               return NOTIFY_DONE;
> > > -                       }
> > >                         wqent = kzalloc(sizeof(*wqent), GFP_ATOMIC);
> > > -                       if (!wqent) {
> > > -                               up_read(&ndev->reslock);
> > > +                       if (!wqent)
> > >                                 return NOTIFY_DONE;
> > > -                       }
> > >
> > >                         wqent->mvdev = &ndev->mvdev;
> > >                         INIT_WORK(&wqent->work, update_carrier);
> > >                         queue_work(ndev->mvdev.wq, &wqent->work);
> > > -                       up_read(&ndev->reslock);
> > >                         ret = NOTIFY_OK;
> > >                         break;
> > >                 default:
> > > --
> > > 2.38.1
> > >
>

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH 6/7] vdpa/mlx5: Avoid using reslock in event_handler

[Jason Wang]

On Wed, Nov 23, 2022 at 12:00 PM Jason Wang <jasowang@redhat.com> wrote:
>
> On Mon, Nov 14, 2022 at 4:58 PM Eli Cohen <elic@nvidia.com> wrote:
> >
> > > From: Jason Wang <jasowang@redhat.com>
> > > Sent: Monday, 14 November 2022 9:53
> > > To: Eli Cohen <elic@nvidia.com>
> > > Cc: mst@redhat.com; linux-kernel@vger.kernel.org; virtualization@lists.linux-
> > > foundation.org; si-wei.liu@oracle.com; eperezma@redhat.com;
> > > lulu@redhat.com
> > > Subject: Re: [PATCH 6/7] vdpa/mlx5: Avoid using reslock in event_handler
> > >
> > > On Sun, Nov 13, 2022 at 9:45 PM Eli Cohen <elic@nvidia.com> wrote:
> > > >
> > > > event_handler runs under atomic context and may not acquire reslock. We
> > > > can still guarantee that the handler won't be called after suspend by
> > > > clearing nb_registered, unregistering the handler and flushing the
> > > > workqueue.
> > > >
> > > > Signed-off-by: Eli Cohen <elic@nvidia.com>
> > > > ---
> > > >  drivers/vdpa/mlx5/net/mlx5_vnet.c | 14 +++-----------
> > > >  1 file changed, 3 insertions(+), 11 deletions(-)
> > > >
> > > > diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > > b/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > > > index 6e6490c85be2..bebfba530247 100644
> > > > --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > > > +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > > > @@ -2872,8 +2872,8 @@ static int mlx5_vdpa_suspend(struct vdpa_device
> > > *vdev)
> > > >         int i;
> > > >
> > > >         down_write(&ndev->reslock);
> > > > -       mlx5_notifier_unregister(mvdev->mdev, &ndev->nb);
> > > >         ndev->nb_registered = false;
> > > > +       mlx5_notifier_unregister(mvdev->mdev, &ndev->nb);
> > >
> > > I wonder why this can help anything.
> > I think you were concerned that async events will come when the device was suspended. Since we can't take reslock, I think this guarantees that we won't get any events after suspension.
> >
>
> Ok, I see.
>
> > > And if it does, we have simliar
> > > logic in mlx5_vdpa_dev_del() do we need to fix that as well?
> > >
> > We have the same construct there only that I set nb_registered = false after unregistering the notifier. So I probably need to move it before mlx5_notifier_unregister().
>
> Right.
>
> Thanks

So I'm fine with this patch.

Acked-by: Jason Wang <jasowang@redhat.com>

Thanks

>
> >
> > > Thanks
> > >
> > > >         flush_workqueue(ndev->mvdev.wq);
> > > >         for (i = 0; i < ndev->cur_num_vqs; i++) {
> > > >                 mvq = &ndev->vqs[i];
> > > > @@ -3051,7 +3051,7 @@ static void update_carrier(struct work_struct
> > > *work)
> > > >         else
> > > >                 ndev->config.status &= cpu_to_mlx5vdpa16(mvdev,
> > > ~VIRTIO_NET_S_LINK_UP);
> > > >
> > > > -       if (ndev->config_cb.callback)
> > > > +       if (ndev->nb_registered && ndev->config_cb.callback)
> > > >                 ndev->config_cb.callback(ndev->config_cb.private);
> > > >
> > > >         kfree(wqent);
> > > > @@ -3068,21 +3068,13 @@ static int event_handler(struct notifier_block
> > > *nb, unsigned long event, void *p
> > > >                 switch (eqe->sub_type) {
> > > >                 case MLX5_PORT_CHANGE_SUBTYPE_DOWN:
> > > >                 case MLX5_PORT_CHANGE_SUBTYPE_ACTIVE:
> > > > -                       down_read(&ndev->reslock);
> > > > -                       if (!ndev->nb_registered) {
> > > > -                               up_read(&ndev->reslock);
> > > > -                               return NOTIFY_DONE;
> > > > -                       }
> > > >                         wqent = kzalloc(sizeof(*wqent), GFP_ATOMIC);
> > > > -                       if (!wqent) {
> > > > -                               up_read(&ndev->reslock);
> > > > +                       if (!wqent)
> > > >                                 return NOTIFY_DONE;
> > > > -                       }
> > > >
> > > >                         wqent->mvdev = &ndev->mvdev;
> > > >                         INIT_WORK(&wqent->work, update_carrier);
> > > >                         queue_work(ndev->mvdev.wq, &wqent->work);
> > > > -                       up_read(&ndev->reslock);
> > > >                         ret = NOTIFY_OK;
> > > >                         break;
> > > >                 default:
> > > > --
> > > > 2.38.1
> > > >
> >

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization