[PATCH RFC 00/12] RDMA: Support link status events dispatching in ib_core

[PATCH RFC 00/12] RDMA: Support link status events dispatching in ib_core

[Junxian Huang]

This series is to integrate a common link status event handler in
ib_core as this functionality is needed by most drivers and
implemented in very similar patterns. This is not a new issue but
a restart of the previous work of our colleagues from several years
ago, please see [1] and [2].

[1]: https://lore.kernel.org/linux-rdma/1570184954-21384-1-git-send-email-liweihang@hisilicon.com/
[2]: https://lore.kernel.org/linux-rdma/20200204082408.18728-1-liweihang@huawei.com/

With this series, ib_core can handle netdev events of link status,
i.e. NETDEV_UP, NETDEV_DOWN and NETDEV_CHANGE, and dispatch ib port
events to ULPs instead of drivers. However some drivers currently
have some private processing in their handler, rather than simply
dispatching events. For these drivers, this series provides a new
ops report_port_event(). If this ops is set, ib_core will call it
and the events will still be handled in the driver.

Events of LAG devices are also not handled in ib_core as currently
there is no way to obtain ibdev from upper netdev in ib_core. This
can be a TODO work after the core have more support for LAG. For
now mlx5 is the only driver that supports RoCE LAG, and the events
handling of mlx5 RoCE LAG will remain in mlx5 driver.

In this series:

Patch #1 adds a new helper to query the port num of a netdev
associated with an ibdev. This is used in the following patch.

Patch #2 adds support for link status events dispatching in ib_core.

Patch #3-#7 removes link status event handler in several drivers.
The port state setting in erdma, rxe and siw are replaced with
ib_get_curr_port_state(), so their handler can be totally removed.

Patch #8-#10 add support for report_port_event() ops in usnic, mlx4
and pvrdma as their current handler cannot be perfectly replaced by
the ib_core handler in patch #2.

Patch #11 adds a check in mlx5 that only events of RoCE LAG will be
handled in mlx5 driver.

Patch #12 adds a fast path for link-down events dispatching in hns by
getting notified from hns3 nic driver directly.

Yuyu Li (12):
  RDMA/core: Add ib_query_netdev_port() to query netdev port by IB
    device.
  RDMA/core: Support link status events dispatching
  RDMA/bnxt_re: Remove deliver net device event
  RDMA/erdma: Remove deliver net device event
  RDMA/irdma: Remove deliver net device event
  RDMA/rxe: Remove deliver net device event
  RDMA/siw: Remove deliver net device event
  RDMA/usnic: Support report_port_event() ops
  RDMA/mlx4: Support report_port_event() ops
  RDMA/pvrdma: Support report_port_event() ops
  RDMA/mlx5: Handle link status event only for LAG device
  RDMA/hns: Support fast path for link-down events dispatching

 drivers/infiniband/core/device.c              | 99 +++++++++++++++++--
 drivers/infiniband/hw/bnxt_re/main.c          | 59 -----------
 drivers/infiniband/hw/erdma/erdma.h           |  2 -
 drivers/infiniband/hw/erdma/erdma_main.c      |  8 --
 drivers/infiniband/hw/erdma/erdma_verbs.c     |  8 +-
 drivers/infiniband/hw/hns/hns_roce_hw_v2.c    | 13 +++
 drivers/infiniband/hw/irdma/utils.c           |  3 -
 drivers/infiniband/hw/mlx4/main.c             | 58 +++++------
 drivers/infiniband/hw/mlx5/main.c             |  3 +
 drivers/infiniband/hw/usnic/usnic_ib_main.c   | 71 +++++++------
 .../infiniband/hw/vmw_pvrdma/pvrdma_main.c    | 42 +++++---
 drivers/infiniband/sw/rxe/rxe_net.c           | 22 +----
 drivers/infiniband/sw/rxe/rxe_verbs.c         |  1 +
 drivers/infiniband/sw/siw/siw.h               |  3 -
 drivers/infiniband/sw/siw/siw_main.c          | 16 ---
 drivers/infiniband/sw/siw/siw_verbs.c         |  6 +-
 include/rdma/ib_verbs.h                       | 19 ++++
 17 files changed, 239 insertions(+), 194 deletions(-)

--
2.33.0

[PATCH RFC 01/12] RDMA/core: Add ib_query_netdev_port() to query netdev port by IB device.

[Junxian Huang]

From: Yuyu Li <liyuyu6@huawei.com>

Query the port number of a netdev associated with an ibdev.

Signed-off-by: Yuyu Li <liyuyu6@huawei.com>
Signed-off-by: Junxian Huang <huangjunxian6@hisilicon.com>
---
 drivers/infiniband/core/device.c | 39 ++++++++++++++++++++++++++------
 include/rdma/ib_verbs.h          |  2 ++
 2 files changed, 34 insertions(+), 7 deletions(-)

diff --git a/drivers/infiniband/core/device.c b/drivers/infiniband/core/device.c
index ca9b956c034d..c60dd50d434f 100644
--- a/drivers/infiniband/core/device.c
+++ b/drivers/infiniband/core/device.c
@@ -2295,6 +2295,33 @@ struct net_device *ib_device_get_netdev(struct ib_device *ib_dev,
 }
 EXPORT_SYMBOL(ib_device_get_netdev);
 
+/**
+ * ib_query_netdev_port - Query the port number of a net_device
+ * associated with an ibdev
+ * @ibdev: IB device
+ * @ndev: Network device
+ * @port: IB port the net_device is connected to
+ */
+int ib_query_netdev_port(struct ib_device *ibdev, struct net_device *ndev,
+			 u32 *port)
+{
+	struct net_device *ib_ndev;
+	u32 port_num;
+
+	rdma_for_each_port(ibdev, port_num) {
+		ib_ndev = ib_device_get_netdev(ibdev, port_num);
+		if (ndev == ib_ndev) {
+			*port = port_num;
+			dev_put(ib_ndev);
+			return 0;
+		}
+		dev_put(ib_ndev);
+	}
+
+	return -ENOENT;
+}
+EXPORT_SYMBOL(ib_query_netdev_port);
+
 /**
  * ib_device_get_by_netdev - Find an IB device associated with a netdev
  * @ndev: netdev to locate
@@ -2858,7 +2885,6 @@ static int ib_netdevice_event(struct notifier_block *this,
 			      unsigned long event, void *ptr)
 {
 	struct net_device *ndev = netdev_notifier_info_to_dev(ptr);
-	struct net_device *ib_ndev;
 	struct ib_device *ibdev;
 	u32 port;
 
@@ -2868,13 +2894,12 @@ static int ib_netdevice_event(struct notifier_block *this,
 		if (!ibdev)
 			return NOTIFY_DONE;
 
-		rdma_for_each_port(ibdev, port) {
-			ib_ndev = ib_device_get_netdev(ibdev, port);
-			if (ndev == ib_ndev)
-				rdma_nl_notify_event(ibdev, port,
-						     RDMA_NETDEV_RENAME_EVENT);
-			dev_put(ib_ndev);
+		if (ib_query_netdev_port(ibdev, ndev, &port)) {
+			ib_device_put(ibdev);
+			break;
 		}
+
+		rdma_nl_notify_event(ibdev, port, RDMA_NETDEV_RENAME_EVENT);
 		ib_device_put(ibdev);
 		break;
 	default:
diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index 3417636da960..b5ee5e748a47 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -4469,6 +4469,8 @@ int ib_device_set_netdev(struct ib_device *ib_dev, struct net_device *ndev,
 			 unsigned int port);
 struct net_device *ib_device_get_netdev(struct ib_device *ib_dev,
 					u32 port);
+int ib_query_netdev_port(struct ib_device *ibdev, struct net_device *ndev,
+			 u32 *port);
 struct ib_wq *ib_create_wq(struct ib_pd *pd,
 			   struct ib_wq_init_attr *init_attr);
 int ib_destroy_wq_user(struct ib_wq *wq, struct ib_udata *udata);
-- 
2.33.0

[PATCH RFC 02/12] RDMA/core: Support link status events dispatching

[Junxian Huang]

From: Yuyu Li <liyuyu6@huawei.com>

Currently the dispatching of link status events is implemented by
each RDMA driver independently, and most of them have very similar
patterns. Add support for this in ib_core so that we can get rid
of duplicate codes in each driver.

A new last_port_state is added in ib_port_cache to cache the port
state of the last link status events dispatching. The original
port_state in ib_port_cache is not used here because it will be
updated when ib_dispatch_event() is called, which means it may
be changed between two link status events, and may lead to a loss
of event dispatching.

Some drivers currently have some private stuff in their link status
events handler in addition to event dispatching, and cannot be
perfectly integrated into the ib_core handling process. For these
drivers, add a new ops report_port_event() so that they can keep
their current processing.

Finally, events of LAG devices are not supported yet in this patch
as currently there is no way to obtain ibdev from upper netdev in
ib_core. This can be a TODO work after the core have more support
for LAG.

Signed-off-by: Yuyu Li <liyuyu6@huawei.com>
Signed-off-by: Junxian Huang <huangjunxian6@hisilicon.com>
---
 drivers/infiniband/core/device.c | 60 ++++++++++++++++++++++++++++++++
 include/rdma/ib_verbs.h          | 17 +++++++++
 2 files changed, 77 insertions(+)

diff --git a/drivers/infiniband/core/device.c b/drivers/infiniband/core/device.c
index c60dd50d434f..0838a423ac82 100644
--- a/drivers/infiniband/core/device.c
+++ b/drivers/infiniband/core/device.c
@@ -2788,6 +2788,7 @@ void ib_set_device_ops(struct ib_device *dev, const struct ib_device_ops *ops)
 	SET_DEVICE_OP(dev_ops, set_vf_guid);
 	SET_DEVICE_OP(dev_ops, set_vf_link_state);
 	SET_DEVICE_OP(dev_ops, ufile_hw_cleanup);
+	SET_DEVICE_OP(dev_ops, report_port_event);
 
 	SET_OBJ_SIZE(dev_ops, ib_ah);
 	SET_OBJ_SIZE(dev_ops, ib_counters);
@@ -2881,6 +2882,58 @@ static const struct rdma_nl_cbs ibnl_ls_cb_table[RDMA_NL_LS_NUM_OPS] = {
 	},
 };
 
+void ib_dispatch_port_state_event(struct ib_device *ibdev, struct net_device *ndev)
+{
+	enum ib_port_state curr_state;
+	struct ib_event ibevent = {};
+	u32 port;
+
+	if (ib_query_netdev_port(ibdev, ndev, &port))
+		return;
+
+	curr_state = ib_get_curr_port_state(ndev);
+
+	write_lock_irq(&ibdev->cache_lock);
+	if (ibdev->port_data[port].cache.last_port_state == curr_state) {
+		write_unlock_irq(&ibdev->cache_lock);
+		return;
+	}
+	ibdev->port_data[port].cache.last_port_state = curr_state;
+	write_unlock_irq(&ibdev->cache_lock);
+
+	ibevent.event = (curr_state == IB_PORT_DOWN) ?
+					IB_EVENT_PORT_ERR : IB_EVENT_PORT_ACTIVE;
+	ibevent.device = ibdev;
+	ibevent.element.port_num = port;
+	ib_dispatch_event(&ibevent);
+}
+EXPORT_SYMBOL(ib_dispatch_port_state_event);
+
+static void handle_port_event(struct net_device *ndev, unsigned long event)
+{
+	struct ib_device *ibdev;
+
+	/* Currently, link events in bonding scenarios are still
+	 * reported by drivers that support bonding.
+	 */
+	if (netif_is_lag_master(ndev) || netif_is_lag_port(ndev))
+		return;
+
+	ibdev = ib_device_get_by_netdev(ndev, RDMA_DRIVER_UNKNOWN);
+	if (!ibdev)
+		return;
+
+	if (ibdev->ops.report_port_event) {
+		ibdev->ops.report_port_event(ibdev, ndev, event);
+		goto put_ibdev;
+	}
+
+	ib_dispatch_port_state_event(ibdev, ndev);
+
+put_ibdev:
+	ib_device_put(ibdev);
+};
+
 static int ib_netdevice_event(struct notifier_block *this,
 			      unsigned long event, void *ptr)
 {
@@ -2902,6 +2955,13 @@ static int ib_netdevice_event(struct notifier_block *this,
 		rdma_nl_notify_event(ibdev, port, RDMA_NETDEV_RENAME_EVENT);
 		ib_device_put(ibdev);
 		break;
+
+	case NETDEV_UP:
+	case NETDEV_CHANGE:
+	case NETDEV_DOWN:
+		handle_port_event(ndev, event);
+		break;
+
 	default:
 		break;
 	}
diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index b5ee5e748a47..bc70299c42e9 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -2177,6 +2177,7 @@ struct ib_port_cache {
 	struct ib_gid_table   *gid;
 	u8                     lmc;
 	enum ib_port_state     port_state;
+	enum ib_port_state     last_port_state;
 };
 
 struct ib_port_immutable {
@@ -2681,6 +2682,13 @@ struct ib_device_ops {
 	 */
 	void (*ufile_hw_cleanup)(struct ib_uverbs_file *ufile);
 
+	/**
+	 * report_port_event - Drivers need to implement this if they have
+	 * some private stuff to handle when link status changes.
+	 */
+	void (*report_port_event)(struct ib_device *ibdev,
+				  struct net_device *ndev, unsigned long event);
+
 	DECLARE_RDMA_OBJ_SIZE(ib_ah);
 	DECLARE_RDMA_OBJ_SIZE(ib_counters);
 	DECLARE_RDMA_OBJ_SIZE(ib_cq);
@@ -4471,6 +4479,15 @@ struct net_device *ib_device_get_netdev(struct ib_device *ib_dev,
 					u32 port);
 int ib_query_netdev_port(struct ib_device *ibdev, struct net_device *ndev,
 			 u32 *port);
+
+static inline enum ib_port_state ib_get_curr_port_state(struct net_device *net_dev)
+{
+	return (netif_running(net_dev) && netif_carrier_ok(net_dev)) ?
+		IB_PORT_ACTIVE : IB_PORT_DOWN;
+}
+
+void ib_dispatch_port_state_event(struct ib_device *ibdev,
+				  struct net_device *ndev);
 struct ib_wq *ib_create_wq(struct ib_pd *pd,
 			   struct ib_wq_init_attr *init_attr);
 int ib_destroy_wq_user(struct ib_wq *wq, struct ib_udata *udata);
-- 
2.33.0

[PATCH RFC 03/12] RDMA/bnxt_re: Remove deliver net device event

[Junxian Huang]

From: Yuyu Li <liyuyu6@huawei.com>

Since the netdev events of link status is now handled in ib_core,
remove the related code in drivers.

Signed-off-by: Yuyu Li <liyuyu6@huawei.com>
Signed-off-by: Junxian Huang <huangjunxian6@hisilicon.com>
---
 drivers/infiniband/hw/bnxt_re/main.c | 59 ----------------------------
 1 file changed, 59 deletions(-)

diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c
index 298c848f3a4d..973c1ecde4cf 100644
--- a/drivers/infiniband/hw/bnxt_re/main.c
+++ b/drivers/infiniband/hw/bnxt_re/main.c
@@ -81,8 +81,6 @@ static DEFINE_MUTEX(bnxt_re_mutex);
 
 static void bnxt_re_stop_irq(void *handle);
 static void bnxt_re_dev_stop(struct bnxt_re_dev *rdev);
-static int bnxt_re_netdev_event(struct notifier_block *notifier,
-				unsigned long event, void *ptr);
 static struct bnxt_re_dev *bnxt_re_from_netdev(struct net_device *netdev);
 static void bnxt_re_dev_uninit(struct bnxt_re_dev *rdev, u8 op_type);
 static int bnxt_re_hwrm_qcaps(struct bnxt_re_dev *rdev);
@@ -2169,14 +2167,6 @@ static int bnxt_re_add_device(struct auxiliary_device *adev, u8 op_type)
 		goto re_dev_uninit;
 	}
 
-	rdev->nb.notifier_call = bnxt_re_netdev_event;
-	rc = register_netdevice_notifier(&rdev->nb);
-	if (rc) {
-		rdev->nb.notifier_call = NULL;
-		pr_err("%s: Cannot register to netdevice_notifier",
-		       ROCE_DRV_MODULE_NAME);
-		return rc;
-	}
 	bnxt_re_setup_cc(rdev, true);
 
 	return 0;
@@ -2214,55 +2204,6 @@ static void bnxt_re_setup_cc(struct bnxt_re_dev *rdev, bool enable)
 		ibdev_err(&rdev->ibdev, "Failed to setup CC enable = %d\n", enable);
 }
 
-/*
- * "Notifier chain callback can be invoked for the same chain from
- * different CPUs at the same time".
- *
- * For cases when the netdev is already present, our call to the
- * register_netdevice_notifier() will actually get the rtnl_lock()
- * before sending NETDEV_REGISTER and (if up) NETDEV_UP
- * events.
- *
- * But for cases when the netdev is not already present, the notifier
- * chain is subjected to be invoked from different CPUs simultaneously.
- *
- * This is protected by the netdev_mutex.
- */
-static int bnxt_re_netdev_event(struct notifier_block *notifier,
-				unsigned long event, void *ptr)
-{
-	struct net_device *real_dev, *netdev = netdev_notifier_info_to_dev(ptr);
-	struct bnxt_re_dev *rdev;
-
-	real_dev = rdma_vlan_dev_real_dev(netdev);
-	if (!real_dev)
-		real_dev = netdev;
-
-	if (real_dev != netdev)
-		goto exit;
-
-	rdev = bnxt_re_from_netdev(real_dev);
-	if (!rdev)
-		return NOTIFY_DONE;
-
-
-	switch (event) {
-	case NETDEV_UP:
-	case NETDEV_DOWN:
-	case NETDEV_CHANGE:
-		bnxt_re_dispatch_event(&rdev->ibdev, NULL, 1,
-					netif_carrier_ok(real_dev) ?
-					IB_EVENT_PORT_ACTIVE :
-					IB_EVENT_PORT_ERR);
-		break;
-	default:
-		break;
-	}
-	ib_device_put(&rdev->ibdev);
-exit:
-	return NOTIFY_DONE;
-}
-
 #define BNXT_ADEV_NAME "bnxt_en"
 
 static void bnxt_re_remove_device(struct bnxt_re_dev *rdev, u8 op_type,
-- 
2.33.0

[PATCH RFC 04/12] RDMA/erdma: Remove deliver net device event

[Junxian Huang]

From: Yuyu Li <liyuyu6@huawei.com>

Since the netdev events of link status is now handled in ib_core,
remove the related code in drivers.

In addition, remove dev->state as it is only used in erdma_query_port(),
and it can be replaced by ib_get_curr_port_state().

Signed-off-by: Yuyu Li <liyuyu6@huawei.com>
Signed-off-by: Junxian Huang <huangjunxian6@hisilicon.com>
---
 drivers/infiniband/hw/erdma/erdma.h       | 2 --
 drivers/infiniband/hw/erdma/erdma_main.c  | 8 --------
 drivers/infiniband/hw/erdma/erdma_verbs.c | 8 ++------
 3 files changed, 2 insertions(+), 16 deletions(-)

diff --git a/drivers/infiniband/hw/erdma/erdma.h b/drivers/infiniband/hw/erdma/erdma.h
index 3c166359448d..7ba554da992d 100644
--- a/drivers/infiniband/hw/erdma/erdma.h
+++ b/drivers/infiniband/hw/erdma/erdma.h
@@ -192,8 +192,6 @@ struct erdma_dev {
 	u8 __iomem *func_bar;
 
 	struct erdma_devattr attrs;
-	/* physical port state (only one port per device) */
-	enum ib_port_state state;
 	u32 mtu;
 
 	/* cmdq and aeq use the same msix vector */
diff --git a/drivers/infiniband/hw/erdma/erdma_main.c b/drivers/infiniband/hw/erdma/erdma_main.c
index 62f497a71004..c5cdf7a4aa2d 100644
--- a/drivers/infiniband/hw/erdma/erdma_main.c
+++ b/drivers/infiniband/hw/erdma/erdma_main.c
@@ -26,14 +26,6 @@ static int erdma_netdev_event(struct notifier_block *nb, unsigned long event,
 		goto done;
 
 	switch (event) {
-	case NETDEV_UP:
-		dev->state = IB_PORT_ACTIVE;
-		erdma_port_event(dev, IB_EVENT_PORT_ACTIVE);
-		break;
-	case NETDEV_DOWN:
-		dev->state = IB_PORT_DOWN;
-		erdma_port_event(dev, IB_EVENT_PORT_ERR);
-		break;
 	case NETDEV_CHANGEMTU:
 		if (dev->mtu != netdev->mtu) {
 			erdma_set_mtu(dev, netdev->mtu);
diff --git a/drivers/infiniband/hw/erdma/erdma_verbs.c b/drivers/infiniband/hw/erdma/erdma_verbs.c
index 51d619edb6c5..65a65416343c 100644
--- a/drivers/infiniband/hw/erdma/erdma_verbs.c
+++ b/drivers/infiniband/hw/erdma/erdma_verbs.c
@@ -377,14 +377,10 @@ int erdma_query_port(struct ib_device *ibdev, u32 port,
 	ib_get_eth_speed(ibdev, port, &attr->active_speed, &attr->active_width);
 	attr->max_mtu = ib_mtu_int_to_enum(ndev->mtu);
 	attr->active_mtu = ib_mtu_int_to_enum(ndev->mtu);
-	if (netif_running(ndev) && netif_carrier_ok(ndev))
-		dev->state = IB_PORT_ACTIVE;
-	else
-		dev->state = IB_PORT_DOWN;
-	attr->state = dev->state;
+	attr->state = ib_get_curr_port_state(ndev);
 
 out:
-	if (dev->state == IB_PORT_ACTIVE)
+	if (attr->state == IB_PORT_ACTIVE)
 		attr->phys_state = IB_PORT_PHYS_STATE_LINK_UP;
 	else
 		attr->phys_state = IB_PORT_PHYS_STATE_DISABLED;
-- 
2.33.0

[PATCH RFC 05/12] RDMA/irdma: Remove deliver net device event

[Junxian Huang]

From: Yuyu Li <liyuyu6@huawei.com>

Since the netdev events of link status is now handled in ib_core,
remove the related code in drivers.

Signed-off-by: Yuyu Li <liyuyu6@huawei.com>
Signed-off-by: Junxian Huang <huangjunxian6@hisilicon.com>
---
 drivers/infiniband/hw/irdma/utils.c | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/drivers/infiniband/hw/irdma/utils.c b/drivers/infiniband/hw/irdma/utils.c
index 0422787592d8..1ebacc1533ce 100644
--- a/drivers/infiniband/hw/irdma/utils.c
+++ b/drivers/infiniband/hw/irdma/utils.c
@@ -320,9 +320,6 @@ int irdma_netdevice_event(struct notifier_block *notifier, unsigned long event,
 	case NETDEV_DOWN:
 		iwdev->iw_status = 0;
 		fallthrough;
-	case NETDEV_UP:
-		irdma_port_ibevent(iwdev);
-		break;
 	default:
 		break;
 	}
-- 
2.33.0

[PATCH RFC 06/12] RDMA/rxe: Remove deliver net device event

[Junxian Huang]

From: Yuyu Li <liyuyu6@huawei.com>

Since the netdev events of link status is now handled in ib_core,
remove the related code in drivers.

In addition, remove the setting of port->attr.state in rxe_port_up()
and rxe_port_down(), as it is only used in rxe_query_port(), and it
can be replaced by ib_get_curr_port_state().

Signed-off-by: Yuyu Li <liyuyu6@huawei.com>
Signed-off-by: Junxian Huang <huangjunxian6@hisilicon.com>
---
 drivers/infiniband/sw/rxe/rxe_net.c   | 22 ++++------------------
 drivers/infiniband/sw/rxe/rxe_verbs.c |  1 +
 2 files changed, 5 insertions(+), 18 deletions(-)

diff --git a/drivers/infiniband/sw/rxe/rxe_net.c b/drivers/infiniband/sw/rxe/rxe_net.c
index 75d1407db52d..0fde8974d6c6 100644
--- a/drivers/infiniband/sw/rxe/rxe_net.c
+++ b/drivers/infiniband/sw/rxe/rxe_net.c
@@ -563,11 +563,6 @@ static void rxe_port_event(struct rxe_dev *rxe,
 /* Caller must hold net_info_lock */
 void rxe_port_up(struct rxe_dev *rxe)
 {
-	struct rxe_port *port;
-
-	port = &rxe->port;
-	port->attr.state = IB_PORT_ACTIVE;
-
 	rxe_port_event(rxe, IB_EVENT_PORT_ACTIVE);
 	dev_info(&rxe->ib_dev.dev, "set active\n");
 }
@@ -575,11 +570,6 @@ void rxe_port_up(struct rxe_dev *rxe)
 /* Caller must hold net_info_lock */
 void rxe_port_down(struct rxe_dev *rxe)
 {
-	struct rxe_port *port;
-
-	port = &rxe->port;
-	port->attr.state = IB_PORT_DOWN;
-
 	rxe_port_event(rxe, IB_EVENT_PORT_ERR);
 	rxe_counter_inc(rxe, RXE_CNT_LINK_DOWNED);
 	dev_info(&rxe->ib_dev.dev, "set down\n");
@@ -587,7 +577,7 @@ void rxe_port_down(struct rxe_dev *rxe)
 
 void rxe_set_port_state(struct rxe_dev *rxe)
 {
-	if (netif_running(rxe->ndev) && netif_carrier_ok(rxe->ndev))
+	if (ib_get_curr_port_state(rxe->ndev) == IB_PORT_ACTIVE)
 		rxe_port_up(rxe);
 	else
 		rxe_port_down(rxe);
@@ -607,18 +597,14 @@ static int rxe_notify(struct notifier_block *not_blk,
 	case NETDEV_UNREGISTER:
 		ib_unregister_device_queued(&rxe->ib_dev);
 		break;
-	case NETDEV_UP:
-		rxe_port_up(rxe);
-		break;
-	case NETDEV_DOWN:
-		rxe_port_down(rxe);
-		break;
 	case NETDEV_CHANGEMTU:
 		rxe_dbg_dev(rxe, "%s changed mtu to %d\n", ndev->name, ndev->mtu);
 		rxe_set_mtu(rxe, ndev->mtu);
 		break;
+	case NETDEV_DOWN:
 	case NETDEV_CHANGE:
-		rxe_set_port_state(rxe);
+		if (ib_get_curr_port_state(rxe->ndev) == IB_PORT_DOWN)
+			rxe_counter_inc(rxe, RXE_CNT_LINK_DOWNED);
 		break;
 	case NETDEV_REBOOT:
 	case NETDEV_GOING_DOWN:
diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.c b/drivers/infiniband/sw/rxe/rxe_verbs.c
index 5c18f7e342f2..0101e4d4d694 100644
--- a/drivers/infiniband/sw/rxe/rxe_verbs.c
+++ b/drivers/infiniband/sw/rxe/rxe_verbs.c
@@ -55,6 +55,7 @@ static int rxe_query_port(struct ib_device *ibdev,
 	ret = ib_get_eth_speed(ibdev, port_num, &attr->active_speed,
 			       &attr->active_width);
 
+	attr->state = ib_get_curr_port_state(rxe->ndev);
 	if (attr->state == IB_PORT_ACTIVE)
 		attr->phys_state = IB_PORT_PHYS_STATE_LINK_UP;
 	else if (dev_get_flags(rxe->ndev) & IFF_UP)
-- 
2.33.0

[PATCH RFC 07/12] RDMA/siw: Remove deliver net device event

[Junxian Huang]

From: Yuyu Li <liyuyu6@huawei.com>

Since the netdev events of link status is now handled in ib_core,
remove the related code in drivers.

In addition, remove sdev->state as it is only used in siw_query_port(),
and it can be replaced by ib_get_curr_port_state().

Signed-off-by: Yuyu Li <liyuyu6@huawei.com>
Signed-off-by: Junxian Huang <huangjunxian6@hisilicon.com>
---
 drivers/infiniband/sw/siw/siw.h       |  3 ---
 drivers/infiniband/sw/siw/siw_main.c  | 16 ----------------
 drivers/infiniband/sw/siw/siw_verbs.c |  6 +++---
 3 files changed, 3 insertions(+), 22 deletions(-)

diff --git a/drivers/infiniband/sw/siw/siw.h b/drivers/infiniband/sw/siw/siw.h
index 86d4d6a2170e..f5dc4b3e0e60 100644
--- a/drivers/infiniband/sw/siw/siw.h
+++ b/drivers/infiniband/sw/siw/siw.h
@@ -76,9 +76,6 @@ struct siw_device {
 	int numa_node;
 	char raw_gid[ETH_ALEN];
 
-	/* physical port state (only one port per device) */
-	enum ib_port_state state;
-
 	spinlock_t lock;
 
 	struct xarray qp_xa;
diff --git a/drivers/infiniband/sw/siw/siw_main.c b/drivers/infiniband/sw/siw/siw_main.c
index 17abef48abcd..a9dc20f241ec 100644
--- a/drivers/infiniband/sw/siw/siw_main.c
+++ b/drivers/infiniband/sw/siw/siw_main.c
@@ -380,16 +380,6 @@ static int siw_netdev_event(struct notifier_block *nb, unsigned long event,
 	sdev = to_siw_dev(base_dev);
 
 	switch (event) {
-	case NETDEV_UP:
-		sdev->state = IB_PORT_ACTIVE;
-		siw_port_event(sdev, 1, IB_EVENT_PORT_ACTIVE);
-		break;
-
-	case NETDEV_DOWN:
-		sdev->state = IB_PORT_DOWN;
-		siw_port_event(sdev, 1, IB_EVENT_PORT_ERR);
-		break;
-
 	case NETDEV_REGISTER:
 		/*
 		 * Device registration now handled only by
@@ -410,7 +400,6 @@ static int siw_netdev_event(struct notifier_block *nb, unsigned long event,
 	 * Todo: Below netdev events are currently not handled.
 	 */
 	case NETDEV_CHANGEMTU:
-	case NETDEV_CHANGE:
 		break;
 
 	default:
@@ -443,11 +432,6 @@ static int siw_newlink(const char *basedev_name, struct net_device *netdev)
 	if (sdev) {
 		dev_dbg(&netdev->dev, "siw: new device\n");
 
-		if (netif_running(netdev) && netif_carrier_ok(netdev))
-			sdev->state = IB_PORT_ACTIVE;
-		else
-			sdev->state = IB_PORT_DOWN;
-
 		ib_mark_name_assigned_by_user(&sdev->base_dev);
 		rv = siw_device_register(sdev, basedev_name);
 		if (rv)
diff --git a/drivers/infiniband/sw/siw/siw_verbs.c b/drivers/infiniband/sw/siw/siw_verbs.c
index 986666c19378..137819184b3b 100644
--- a/drivers/infiniband/sw/siw/siw_verbs.c
+++ b/drivers/infiniband/sw/siw/siw_verbs.c
@@ -182,10 +182,10 @@ int siw_query_port(struct ib_device *base_dev, u32 port,
 	attr->max_msg_sz = -1;
 	attr->max_mtu = ib_mtu_int_to_enum(sdev->netdev->mtu);
 	attr->active_mtu = ib_mtu_int_to_enum(sdev->netdev->mtu);
-	attr->phys_state = sdev->state == IB_PORT_ACTIVE ?
-		IB_PORT_PHYS_STATE_LINK_UP : IB_PORT_PHYS_STATE_DISABLED;
 	attr->port_cap_flags = IB_PORT_CM_SUP | IB_PORT_DEVICE_MGMT_SUP;
-	attr->state = sdev->state;
+	attr->state = ib_get_curr_port_state(sdev->ndev);
+	attr->phys_state = attr->state == IB_PORT_ACTIVE ?
+		IB_PORT_PHYS_STATE_LINK_UP : IB_PORT_PHYS_STATE_DISABLED;
 	/*
 	 * All zero
 	 *
-- 
2.33.0

[PATCH RFC 08/12] RDMA/usnic: Support report_port_event() ops

[Junxian Huang]

From: Yuyu Li <liyuyu6@huawei.com>

In addition to dispatching event, some private stuffs need to be
done in this driver's link status event handler. Implement the new
report_port_event() ops with the link status event codes.

Signed-off-by: Yuyu Li <liyuyu6@huawei.com>
Signed-off-by: Junxian Huang <huangjunxian6@hisilicon.com>
---
 drivers/infiniband/hw/usnic/usnic_ib_main.c | 71 +++++++++++++--------
 1 file changed, 43 insertions(+), 28 deletions(-)

diff --git a/drivers/infiniband/hw/usnic/usnic_ib_main.c b/drivers/infiniband/hw/usnic/usnic_ib_main.c
index 13b654ddd3cc..5ad7fe7e662f 100644
--- a/drivers/infiniband/hw/usnic/usnic_ib_main.c
+++ b/drivers/infiniband/hw/usnic/usnic_ib_main.c
@@ -151,34 +151,6 @@ static void usnic_ib_handle_usdev_event(struct usnic_ib_dev *us_ibdev,
 		ib_event.element.port_num = 1;
 		ib_dispatch_event(&ib_event);
 		break;
-	case NETDEV_UP:
-	case NETDEV_DOWN:
-	case NETDEV_CHANGE:
-		if (!us_ibdev->ufdev->link_up &&
-				netif_carrier_ok(netdev)) {
-			usnic_fwd_carrier_up(us_ibdev->ufdev);
-			usnic_info("Link UP on %s\n",
-				   dev_name(&us_ibdev->ib_dev.dev));
-			ib_event.event = IB_EVENT_PORT_ACTIVE;
-			ib_event.device = &us_ibdev->ib_dev;
-			ib_event.element.port_num = 1;
-			ib_dispatch_event(&ib_event);
-		} else if (us_ibdev->ufdev->link_up &&
-				!netif_carrier_ok(netdev)) {
-			usnic_fwd_carrier_down(us_ibdev->ufdev);
-			usnic_info("Link DOWN on %s\n",
-				   dev_name(&us_ibdev->ib_dev.dev));
-			usnic_ib_qp_grp_modify_active_to_err(us_ibdev);
-			ib_event.event = IB_EVENT_PORT_ERR;
-			ib_event.device = &us_ibdev->ib_dev;
-			ib_event.element.port_num = 1;
-			ib_dispatch_event(&ib_event);
-		} else {
-			usnic_dbg("Ignoring %s on %s\n",
-					netdev_cmd_to_name(event),
-					dev_name(&us_ibdev->ib_dev.dev));
-		}
-		break;
 	case NETDEV_CHANGEADDR:
 		if (!memcmp(us_ibdev->ufdev->mac, netdev->dev_addr,
 				sizeof(us_ibdev->ufdev->mac))) {
@@ -218,6 +190,48 @@ static void usnic_ib_handle_usdev_event(struct usnic_ib_dev *us_ibdev,
 	mutex_unlock(&us_ibdev->usdev_lock);
 }
 
+static void usnic_ib_handle_port_event(struct ib_device *ibdev,
+				       struct net_device *netdev,
+				       unsigned long event);
+{
+	struct usnic_ib_dev *us_ibdev =
+			container_of(ibdev, struct usnic_ib_dev, ib_dev);
+	mutex_lock(&us_ibdev->usdev_lock);
+	switch (event) {
+	case NETDEV_UP:
+	case NETDEV_DOWN:
+	case NETDEV_CHANGE:
+		if (!us_ibdev->ufdev->link_up &&
+		    netif_carrier_ok(netdev)) {
+			usnic_fwd_carrier_up(us_ibdev->ufdev);
+			usnic_info("Link UP on %s\n",
+				   dev_name(&us_ibdev->ib_dev.dev));
+			ib_event.event = IB_EVENT_PORT_ACTIVE;
+			ib_event.device = &us_ibdev->ib_dev;
+			ib_event.element.port_num = 1;
+			ib_dispatch_event(&ib_event);
+		} else if (us_ibdev->ufdev->link_up &&
+			   !netif_carrier_ok(netdev)) {
+			usnic_fwd_carrier_down(us_ibdev->ufdev);
+			usnic_info("Link DOWN on %s\n",
+				   dev_name(&us_ibdev->ib_dev.dev));
+			usnic_ib_qp_grp_modify_active_to_err(us_ibdev);
+			ib_event.event = IB_EVENT_PORT_ERR;
+			ib_event.device = &us_ibdev->ib_dev;
+			ib_event.element.port_num = 1;
+			ib_dispatch_event(&ib_event);
+		} else {
+			usnic_dbg("Ignoring %s on %s\n",
+				  netdev_cmd_to_name(event),
+				  dev_name(&us_ibdev->ib_dev.dev));
+		}
+		break;
+	default:
+		break;
+	}
+	mutex_unlock(&us_ibdev->usdev_lock);
+}
+
 static int usnic_ib_netdevice_event(struct notifier_block *notifier,
 					unsigned long event, void *ptr)
 {
@@ -358,6 +372,7 @@ static const struct ib_device_ops usnic_dev_ops = {
 	.query_port = usnic_ib_query_port,
 	.query_qp = usnic_ib_query_qp,
 	.reg_user_mr = usnic_ib_reg_mr,
+	.report_port_event = usnic_ib_handle_port_event,
 	INIT_RDMA_OBJ_SIZE(ib_pd, usnic_ib_pd, ibpd),
 	INIT_RDMA_OBJ_SIZE(ib_cq, usnic_ib_cq, ibcq),
 	INIT_RDMA_OBJ_SIZE(ib_qp, usnic_ib_qp_grp, ibqp),
-- 
2.33.0

[PATCH RFC 09/12] RDMA/mlx4: Support report_port_event() ops

[Junxian Huang]

From: Yuyu Li <liyuyu6@huawei.com>

In addition to dispatching event, some private stuffs need to be
done in this driver's link status event handler. Implement the new
report_port_event() ops with the link status event codes.

Signed-off-by: Yuyu Li <liyuyu6@huawei.com>
Signed-off-by: Junxian Huang <huangjunxian6@hisilicon.com>
---
 drivers/infiniband/hw/mlx4/main.c | 58 ++++++++++++++++---------------
 1 file changed, 30 insertions(+), 28 deletions(-)

diff --git a/drivers/infiniband/hw/mlx4/main.c b/drivers/infiniband/hw/mlx4/main.c
index 529db874d67c..8bc390eaa921 100644
--- a/drivers/infiniband/hw/mlx4/main.c
+++ b/drivers/infiniband/hw/mlx4/main.c
@@ -2341,39 +2341,40 @@ static void mlx4_ib_scan_netdev(struct mlx4_ib_dev *ibdev,
 
 	iboe->netdevs[dev->dev_port] = event != NETDEV_UNREGISTER ? dev : NULL;
 
-	if (event == NETDEV_UP || event == NETDEV_DOWN) {
-		enum ib_port_state port_state;
-		struct ib_event ibev = { };
-
-		if (ib_get_cached_port_state(&ibdev->ib_dev, dev->dev_port + 1,
-					     &port_state))
-			goto iboe_out;
-
-		if (event == NETDEV_UP &&
-		    (port_state != IB_PORT_ACTIVE ||
-		     iboe->last_port_state[dev->dev_port] != IB_PORT_DOWN))
-			goto iboe_out;
-		if (event == NETDEV_DOWN &&
-		    (port_state != IB_PORT_DOWN ||
-		     iboe->last_port_state[dev->dev_port] != IB_PORT_ACTIVE))
-			goto iboe_out;
-		iboe->last_port_state[dev->dev_port] = port_state;
-
-		ibev.device = &ibdev->ib_dev;
-		ibev.element.port_num = dev->dev_port + 1;
-		ibev.event = event == NETDEV_UP ? IB_EVENT_PORT_ACTIVE :
-						  IB_EVENT_PORT_ERR;
-		ib_dispatch_event(&ibev);
-	}
-
-iboe_out:
 	spin_unlock_bh(&iboe->lock);
 
-	if (event == NETDEV_CHANGEADDR || event == NETDEV_REGISTER ||
-	    event == NETDEV_UP || event == NETDEV_CHANGE)
+	if (event == NETDEV_CHANGEADDR || event == NETDEV_REGISTER)
 		mlx4_ib_update_qps(ibdev, dev, dev->dev_port + 1);
 }
 
+static void mlx4_ib_port_event(struct ib_device *ibdev, struct net_device *ndev,
+			       unsigned long event)
+{
+	struct mlx4_ib_dev *mlx4_ibdev =
+		container_of(ibdev, struct mlx4_ib_dev, ib_dev);
+	struct mlx4_ib_iboe *iboe = &mlx4_ibdev->iboe;
+
+	if (!net_eq(dev_net(ndev), &init_net))
+		return;
+
+	ASSERT_RTNL();
+
+	if (ndev->dev.parent != mlx4_ibdev->ib_dev.dev.parent)
+		return;
+
+	spin_lock_bh(&iboe->lock);
+
+	iboe->netdevs[ndev->dev_port] = event != NETDEV_UNREGISTER ? ndev : NULL;
+
+	if (event == NETDEV_UP || event == NETDEV_DOWN)
+		ib_dispatch_port_state_event(&mlx4_ibdev->ib_dev, ndev);
+
+	spin_unlock_bh(&iboe->lock);
+
+	if (event == NETDEV_UP || event == NETDEV_CHANGE)
+		mlx4_ib_update_qps(mlx4_ibdev, ndev, ndev->dev_port + 1);
+}
+
 static int mlx4_ib_netdev_event(struct notifier_block *this,
 				unsigned long event, void *ptr)
 {
@@ -2569,6 +2570,7 @@ static const struct ib_device_ops mlx4_ib_dev_ops = {
 	.req_notify_cq = mlx4_ib_arm_cq,
 	.rereg_user_mr = mlx4_ib_rereg_user_mr,
 	.resize_cq = mlx4_ib_resize_cq,
+	.report_port_event = mlx4_ib_port_event,
 
 	INIT_RDMA_OBJ_SIZE(ib_ah, mlx4_ib_ah, ibah),
 	INIT_RDMA_OBJ_SIZE(ib_cq, mlx4_ib_cq, ibcq),
-- 
2.33.0

[PATCH RFC 10/12] RDMA/pvrdma: Support report_port_event() ops

[Junxian Huang]

From: Yuyu Li <liyuyu6@huawei.com>

In addition to dispatching event, some private stuffs need to be
done in this driver's link status event handler. Implement the new
report_port_event() ops with the link status event codes.

Signed-off-by: Yuyu Li <liyuyu6@huawei.com>
Signed-off-by: Junxian Huang <huangjunxian6@hisilicon.com>
---
 .../infiniband/hw/vmw_pvrdma/pvrdma_main.c    | 42 +++++++++++++------
 1 file changed, 29 insertions(+), 13 deletions(-)

diff --git a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_main.c b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_main.c
index 768aad364c89..4bf6c7b682b5 100644
--- a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_main.c
+++ b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_main.c
@@ -181,6 +181,7 @@ static const struct ib_device_ops pvrdma_dev_ops = {
 	.query_qp = pvrdma_query_qp,
 	.reg_user_mr = pvrdma_reg_user_mr,
 	.req_notify_cq = pvrdma_req_notify_cq,
+	.report_port_event = pvrdma_report_event_handle,
 
 	INIT_RDMA_OBJ_SIZE(ib_ah, pvrdma_ah, ibah),
 	INIT_RDMA_OBJ_SIZE(ib_cq, pvrdma_cq, ibcq),
@@ -666,21 +667,8 @@ static void pvrdma_netdevice_event_handle(struct pvrdma_dev *dev,
 
 	switch (event) {
 	case NETDEV_REBOOT:
-	case NETDEV_DOWN:
 		pvrdma_dispatch_event(dev, 1, IB_EVENT_PORT_ERR);
 		break;
-	case NETDEV_UP:
-		pvrdma_write_reg(dev, PVRDMA_REG_CTL,
-				 PVRDMA_DEVICE_CTL_UNQUIESCE);
-
-		mb();
-
-		if (pvrdma_read_reg(dev, PVRDMA_REG_ERR))
-			dev_err(&dev->pdev->dev,
-				"failed to activate device during link up\n");
-		else
-			pvrdma_dispatch_event(dev, 1, IB_EVENT_PORT_ACTIVE);
-		break;
 	case NETDEV_UNREGISTER:
 		ib_device_set_netdev(&dev->ib_dev, NULL, 1);
 		dev_put(dev->netdev);
@@ -708,6 +696,34 @@ static void pvrdma_netdevice_event_handle(struct pvrdma_dev *dev,
 	}
 }
 
+static void pvrdma_report_event_handle(struct ib_device *ibdev,
+				       struct net_device *ndev,
+				       unsigned long event)
+{
+	struct pvrdma_dev *dev = container_of(ibdev, struct pvrdma_dev, ib_dev);
+
+	switch (event) {
+	case NETDEV_DOWN:
+		pvrdma_dispatch_event(dev, 1, IB_EVENT_PORT_ERR);
+		break;
+	case NETDEV_UP:
+		pvrdma_write_reg(dev, PVRDMA_REG_CTL,
+				 PVRDMA_DEVICE_CTL_UNQUIESCE);
+
+		mb();
+
+		if (pvrdma_read_reg(dev, PVRDMA_REG_ERR))
+			dev_err(&dev->pdev->dev,
+				"failed to activate device during link up\n");
+		else
+			pvrdma_dispatch_event(dev, 1, IB_EVENT_PORT_ACTIVE);
+		break;
+
+	default:
+		break;
+	}
+}
+
 static void pvrdma_netdevice_event_work(struct work_struct *work)
 {
 	struct pvrdma_netdevice_work *netdev_work;
-- 
2.33.0

[PATCH RFC 11/12] RDMA/mlx5: Handle link status event only for LAG device

[Junxian Huang]

From: Yuyu Li <liyuyu6@huawei.com>

The link status events of non-LAG devices are now handled in ib_core,
so only LAG device events need to be handled in driver.

Signed-off-by: Yuyu Li <liyuyu6@huawei.com>
Signed-off-by: Junxian Huang <huangjunxian6@hisilicon.com>
---
 drivers/infiniband/hw/mlx5/main.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c
index bc7930d0c564..e4010f871865 100644
--- a/drivers/infiniband/hw/mlx5/main.c
+++ b/drivers/infiniband/hw/mlx5/main.c
@@ -242,6 +242,9 @@ static int mlx5_netdev_event(struct notifier_block *this,
 	case NETDEV_DOWN: {
 		struct net_device *upper = NULL;
 
+		if (!netif_is_lag_master(ndev) && !netif_is_lag_port(ndev))
+			return NOTIFY_DONE;
+
 		if (mlx5_lag_is_roce(mdev) || mlx5_lag_is_sriov(mdev)) {
 			struct net_device *lag_ndev;
 
-- 
2.33.0

[PATCH RFC 12/12] RDMA/hns: Support fast path for link-down events dispatching

[Junxian Huang]

From: Yuyu Li <liyuyu6@huawei.com>

hns3 NIC driver can directly notify the RoCE driver about link status
events bypassing the netdev notifier. This can provide more timely
event dispatching for ULPs.

Signed-off-by: Yuyu Li <liyuyu6@huawei.com>
Signed-off-by: Junxian Huang <huangjunxian6@hisilicon.com>
---
 drivers/infiniband/hw/hns/hns_roce_hw_v2.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/drivers/infiniband/hw/hns/hns_roce_hw_v2.c b/drivers/infiniband/hw/hns/hns_roce_hw_v2.c
index 697b17cca02e..5c911d1def03 100644
--- a/drivers/infiniband/hw/hns/hns_roce_hw_v2.c
+++ b/drivers/infiniband/hw/hns/hns_roce_hw_v2.c
@@ -7178,9 +7178,22 @@ static int hns_roce_hw_v2_reset_notify(struct hnae3_handle *handle,
 	return ret;
 }
 
+static void hns_roce_hw_v2_link_status_change(struct hnae3_handle *handle,
+					      bool linkup)
+{
+	struct hns_roce_dev *hr_dev = (struct hns_roce_dev *)handle->priv;
+	struct net_device *netdev = handle->rinfo.netdev;
+
+	if (linkup || !hr_dev)
+		return;
+
+	ib_dispatch_port_state_event(&hr_dev->ib_dev, netdev);
+}
+
 static const struct hnae3_client_ops hns_roce_hw_v2_ops = {
 	.init_instance = hns_roce_hw_v2_init_instance,
 	.uninit_instance = hns_roce_hw_v2_uninit_instance,
+	.link_status_change = hns_roce_hw_v2_link_status_change,
 	.reset_notify = hns_roce_hw_v2_reset_notify,
 };
 
-- 
2.33.0

Re: [PATCH RFC 12/12] RDMA/hns: Support fast path for link-down events dispatching

[Leon Romanovsky]

On Fri, Nov 22, 2024 at 06:53:08PM +0800, Junxian Huang wrote:
> From: Yuyu Li <liyuyu6@huawei.com>
> 
> hns3 NIC driver can directly notify the RoCE driver about link status
> events bypassing the netdev notifier. This can provide more timely
> event dispatching for ULPs.

It is unlikely that it matters for ULPs and better would be if you don't
open-code existing netdev functionality (netdev notifiers).

Thanks

> 
> Signed-off-by: Yuyu Li <liyuyu6@huawei.com>
> Signed-off-by: Junxian Huang <huangjunxian6@hisilicon.com>
> ---
>  drivers/infiniband/hw/hns/hns_roce_hw_v2.c | 13 +++++++++++++
>  1 file changed, 13 insertions(+)
> 
> diff --git a/drivers/infiniband/hw/hns/hns_roce_hw_v2.c b/drivers/infiniband/hw/hns/hns_roce_hw_v2.c
> index 697b17cca02e..5c911d1def03 100644
> --- a/drivers/infiniband/hw/hns/hns_roce_hw_v2.c
> +++ b/drivers/infiniband/hw/hns/hns_roce_hw_v2.c
> @@ -7178,9 +7178,22 @@ static int hns_roce_hw_v2_reset_notify(struct hnae3_handle *handle,
>  	return ret;
>  }
>  
> +static void hns_roce_hw_v2_link_status_change(struct hnae3_handle *handle,
> +					      bool linkup)
> +{
> +	struct hns_roce_dev *hr_dev = (struct hns_roce_dev *)handle->priv;
> +	struct net_device *netdev = handle->rinfo.netdev;
> +
> +	if (linkup || !hr_dev)
> +		return;
> +
> +	ib_dispatch_port_state_event(&hr_dev->ib_dev, netdev);
> +}
> +
>  static const struct hnae3_client_ops hns_roce_hw_v2_ops = {
>  	.init_instance = hns_roce_hw_v2_init_instance,
>  	.uninit_instance = hns_roce_hw_v2_uninit_instance,
> +	.link_status_change = hns_roce_hw_v2_link_status_change,
>  	.reset_notify = hns_roce_hw_v2_reset_notify,
>  };
>  
> -- 
> 2.33.0
> 

Re: [PATCH RFC 00/12] RDMA: Support link status events dispatching in ib_core

[Leon Romanovsky]

On Fri, 22 Nov 2024 18:52:56 +0800, Junxian Huang wrote:
> This series is to integrate a common link status event handler in
> ib_core as this functionality is needed by most drivers and
> implemented in very similar patterns. This is not a new issue but
> a restart of the previous work of our colleagues from several years
> ago, please see [1] and [2].
> 
> [1]: https://lore.kernel.org/linux-rdma/1570184954-21384-1-git-send-email-liweihang@hisilicon.com/
> [2]: https://lore.kernel.org/linux-rdma/20200204082408.18728-1-liweihang@huawei.com/
> 
> [...]

Applied, thanks!

[01/12] RDMA/core: Add ib_query_netdev_port() to query netdev port by IB device.
        https://git.kernel.org/rdma/rdma/c/0c039a57b68dfb
[02/12] RDMA/core: Support link status events dispatching
        https://git.kernel.org/rdma/rdma/c/1fb0644c3899b2
[03/12] RDMA/bnxt_re: Remove deliver net device event
        https://git.kernel.org/rdma/rdma/c/6ed44e35108526
[04/12] RDMA/erdma: Remove deliver net device event
        https://git.kernel.org/rdma/rdma/c/0cd5fba400d225
[05/12] RDMA/irdma: Remove deliver net device event
        https://git.kernel.org/rdma/rdma/c/504c1945bc0625
[06/12] RDMA/rxe: Remove deliver net device event
        https://git.kernel.org/rdma/rdma/c/970a1c37dd0476
[07/12] RDMA/siw: Remove deliver net device event
        https://git.kernel.org/rdma/rdma/c/8ec9b334d8c17c
[08/12] RDMA/usnic: Support report_port_event() ops
        https://git.kernel.org/rdma/rdma/c/a9112cb29ee36a
[09/12] RDMA/mlx4: Support report_port_event() ops
        https://git.kernel.org/rdma/rdma/c/0b15bb46e71c44
[10/12] RDMA/pvrdma: Support report_port_event() ops
        https://git.kernel.org/rdma/rdma/c/d9e3c4d7eb50ad
[11/12] RDMA/mlx5: Handle link status event only for LAG device
        https://git.kernel.org/rdma/rdma/c/01b6e181a551d3
[12/12] RDMA/hns: Support fast path for link-down events dispatching
        https://git.kernel.org/rdma/rdma/c/7e83fbdd9ca37f

Best regards,
-- 
Leon Romanovsky <leon@kernel.org>

Re: [PATCH RFC 00/12] RDMA: Support link status events dispatching in ib_core

[Leon Romanovsky]

On Fri, Nov 22, 2024 at 06:52:56PM +0800, Junxian Huang wrote:
> This series is to integrate a common link status event handler in
> ib_core as this functionality is needed by most drivers and
> implemented in very similar patterns. This is not a new issue but
> a restart of the previous work of our colleagues from several years
> ago, please see [1] and [2].
> 
> [1]: https://lore.kernel.org/linux-rdma/1570184954-21384-1-git-send-email-liweihang@hisilicon.com/
> [2]: https://lore.kernel.org/linux-rdma/20200204082408.18728-1-liweihang@huawei.com/
> 
> With this series, ib_core can handle netdev events of link status,
> i.e. NETDEV_UP, NETDEV_DOWN and NETDEV_CHANGE, and dispatch ib port
> events to ULPs instead of drivers. However some drivers currently
> have some private processing in their handler, rather than simply
> dispatching events. For these drivers, this series provides a new
> ops report_port_event(). If this ops is set, ib_core will call it
> and the events will still be handled in the driver.
> 
> Events of LAG devices are also not handled in ib_core as currently
> there is no way to obtain ibdev from upper netdev in ib_core. This
> can be a TODO work after the core have more support for LAG. For
> now mlx5 is the only driver that supports RoCE LAG, and the events
> handling of mlx5 RoCE LAG will remain in mlx5 driver.
> 
> In this series:
> 
> Patch #1 adds a new helper to query the port num of a netdev
> associated with an ibdev. This is used in the following patch.
> 
> Patch #2 adds support for link status events dispatching in ib_core.
> 
> Patch #3-#7 removes link status event handler in several drivers.
> The port state setting in erdma, rxe and siw are replaced with
> ib_get_curr_port_state(), so their handler can be totally removed.
> 
> Patch #8-#10 add support for report_port_event() ops in usnic, mlx4
> and pvrdma as their current handler cannot be perfectly replaced by
> the ib_core handler in patch #2.
> 
> Patch #11 adds a check in mlx5 that only events of RoCE LAG will be
> handled in mlx5 driver.
> 
> Patch #12 adds a fast path for link-down events dispatching in hns by
> getting notified from hns3 nic driver directly.
> 
> Yuyu Li (12):
>   RDMA/core: Add ib_query_netdev_port() to query netdev port by IB
>     device.
>   RDMA/core: Support link status events dispatching
>   RDMA/bnxt_re: Remove deliver net device event
>   RDMA/erdma: Remove deliver net device event
>   RDMA/irdma: Remove deliver net device event
>   RDMA/rxe: Remove deliver net device event
>   RDMA/siw: Remove deliver net device event
>   RDMA/usnic: Support report_port_event() ops
>   RDMA/mlx4: Support report_port_event() ops
>   RDMA/pvrdma: Support report_port_event() ops
>   RDMA/mlx5: Handle link status event only for LAG device
>   RDMA/hns: Support fast path for link-down events dispatching

I took the series as it is good thing to remove code duplication
and we waited enough.

However, I'm disappointed to see "RDMA/hns: Support fast path for
link-down events dispatching" patch and would like you to use
netdev notifiers instead. I doubt that this "fast-path" is needed.

Thanks

Re: [PATCH RFC 00/12] RDMA: Support link status events dispatching in ib_core

[Junxian Huang]

On 2024/12/24 18:32, Leon Romanovsky wrote:
> On Fri, Nov 22, 2024 at 06:52:56PM +0800, Junxian Huang wrote:
>> This series is to integrate a common link status event handler in
>> ib_core as this functionality is needed by most drivers and
>> implemented in very similar patterns. This is not a new issue but
>> a restart of the previous work of our colleagues from several years
>> ago, please see [1] and [2].
>>
>> [1]: https://lore.kernel.org/linux-rdma/1570184954-21384-1-git-send-email-liweihang@hisilicon.com/
>> [2]: https://lore.kernel.org/linux-rdma/20200204082408.18728-1-liweihang@huawei.com/
>>
>> With this series, ib_core can handle netdev events of link status,
>> i.e. NETDEV_UP, NETDEV_DOWN and NETDEV_CHANGE, and dispatch ib port
>> events to ULPs instead of drivers. However some drivers currently
>> have some private processing in their handler, rather than simply
>> dispatching events. For these drivers, this series provides a new
>> ops report_port_event(). If this ops is set, ib_core will call it
>> and the events will still be handled in the driver.
>>
>> Events of LAG devices are also not handled in ib_core as currently
>> there is no way to obtain ibdev from upper netdev in ib_core. This
>> can be a TODO work after the core have more support for LAG. For
>> now mlx5 is the only driver that supports RoCE LAG, and the events
>> handling of mlx5 RoCE LAG will remain in mlx5 driver.
>>
>> In this series:
>>
>> Patch #1 adds a new helper to query the port num of a netdev
>> associated with an ibdev. This is used in the following patch.
>>
>> Patch #2 adds support for link status events dispatching in ib_core.
>>
>> Patch #3-#7 removes link status event handler in several drivers.
>> The port state setting in erdma, rxe and siw are replaced with
>> ib_get_curr_port_state(), so their handler can be totally removed.
>>
>> Patch #8-#10 add support for report_port_event() ops in usnic, mlx4
>> and pvrdma as their current handler cannot be perfectly replaced by
>> the ib_core handler in patch #2.
>>
>> Patch #11 adds a check in mlx5 that only events of RoCE LAG will be
>> handled in mlx5 driver.
>>
>> Patch #12 adds a fast path for link-down events dispatching in hns by
>> getting notified from hns3 nic driver directly.
>>
>> Yuyu Li (12):
>>   RDMA/core: Add ib_query_netdev_port() to query netdev port by IB
>>     device.
>>   RDMA/core: Support link status events dispatching
>>   RDMA/bnxt_re: Remove deliver net device event
>>   RDMA/erdma: Remove deliver net device event
>>   RDMA/irdma: Remove deliver net device event
>>   RDMA/rxe: Remove deliver net device event
>>   RDMA/siw: Remove deliver net device event
>>   RDMA/usnic: Support report_port_event() ops
>>   RDMA/mlx4: Support report_port_event() ops
>>   RDMA/pvrdma: Support report_port_event() ops
>>   RDMA/mlx5: Handle link status event only for LAG device
>>   RDMA/hns: Support fast path for link-down events dispatching
> 
> I took the series as it is good thing to remove code duplication
> and we waited enough.
> 

Thanks Leon.

The kernel test robot has reported one warning and one error for
this series:

https://lore.kernel.org/oe-kbuild-all/202411251625.VrcLuTRx-lkp@intel.com/
https://lore.kernel.org/oe-kbuild-all/202411251727.RFxtcpiI-lkp@intel.com/

I was planning to fix them when I could send the formal patches,
but since you have applied these RFC patches，could you please
fix them on your wip branch, or should I send separate patches
to fix them?

Junxian

Re: [PATCH RFC 00/12] RDMA: Support link status events dispatching in ib_core

[Leon Romanovsky]

On Tue, Dec 24, 2024 at 08:05:26PM +0800, Junxian Huang wrote:
> 
> 
> On 2024/12/24 18:32, Leon Romanovsky wrote:
> > On Fri, Nov 22, 2024 at 06:52:56PM +0800, Junxian Huang wrote:
> >> This series is to integrate a common link status event handler in
> >> ib_core as this functionality is needed by most drivers and
> >> implemented in very similar patterns. This is not a new issue but
> >> a restart of the previous work of our colleagues from several years
> >> ago, please see [1] and [2].
> >>
> >> [1]: https://lore.kernel.org/linux-rdma/1570184954-21384-1-git-send-email-liweihang@hisilicon.com/
> >> [2]: https://lore.kernel.org/linux-rdma/20200204082408.18728-1-liweihang@huawei.com/
> >>
> >> With this series, ib_core can handle netdev events of link status,
> >> i.e. NETDEV_UP, NETDEV_DOWN and NETDEV_CHANGE, and dispatch ib port
> >> events to ULPs instead of drivers. However some drivers currently
> >> have some private processing in their handler, rather than simply
> >> dispatching events. For these drivers, this series provides a new
> >> ops report_port_event(). If this ops is set, ib_core will call it
> >> and the events will still be handled in the driver.
> >>
> >> Events of LAG devices are also not handled in ib_core as currently
> >> there is no way to obtain ibdev from upper netdev in ib_core. This
> >> can be a TODO work after the core have more support for LAG. For
> >> now mlx5 is the only driver that supports RoCE LAG, and the events
> >> handling of mlx5 RoCE LAG will remain in mlx5 driver.
> >>
> >> In this series:
> >>
> >> Patch #1 adds a new helper to query the port num of a netdev
> >> associated with an ibdev. This is used in the following patch.
> >>
> >> Patch #2 adds support for link status events dispatching in ib_core.
> >>
> >> Patch #3-#7 removes link status event handler in several drivers.
> >> The port state setting in erdma, rxe and siw are replaced with
> >> ib_get_curr_port_state(), so their handler can be totally removed.
> >>
> >> Patch #8-#10 add support for report_port_event() ops in usnic, mlx4
> >> and pvrdma as their current handler cannot be perfectly replaced by
> >> the ib_core handler in patch #2.
> >>
> >> Patch #11 adds a check in mlx5 that only events of RoCE LAG will be
> >> handled in mlx5 driver.
> >>
> >> Patch #12 adds a fast path for link-down events dispatching in hns by
> >> getting notified from hns3 nic driver directly.
> >>
> >> Yuyu Li (12):
> >>   RDMA/core: Add ib_query_netdev_port() to query netdev port by IB
> >>     device.
> >>   RDMA/core: Support link status events dispatching
> >>   RDMA/bnxt_re: Remove deliver net device event
> >>   RDMA/erdma: Remove deliver net device event
> >>   RDMA/irdma: Remove deliver net device event
> >>   RDMA/rxe: Remove deliver net device event
> >>   RDMA/siw: Remove deliver net device event
> >>   RDMA/usnic: Support report_port_event() ops
> >>   RDMA/mlx4: Support report_port_event() ops
> >>   RDMA/pvrdma: Support report_port_event() ops
> >>   RDMA/mlx5: Handle link status event only for LAG device
> >>   RDMA/hns: Support fast path for link-down events dispatching
> > 
> > I took the series as it is good thing to remove code duplication
> > and we waited enough.
> > 
> 
> Thanks Leon.
> 
> The kernel test robot has reported one warning and one error for
> this series:
> 
> https://lore.kernel.org/oe-kbuild-all/202411251625.VrcLuTRx-lkp@intel.com/
> https://lore.kernel.org/oe-kbuild-all/202411251727.RFxtcpiI-lkp@intel.com/
> 
> I was planning to fix them when I could send the formal patches,
> but since you have applied these RFC patches，could you please
> fix them on your wip branch, or should I send separate patches
> to fix them?

This is how I fixed it. Is it ok?

diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c
index 4286fd4a9324..b886fe2922ae 100644
--- a/drivers/infiniband/hw/bnxt_re/main.c
+++ b/drivers/infiniband/hw/bnxt_re/main.c
@@ -822,17 +822,6 @@ static void bnxt_re_disassociate_ucontext(struct ib_ucontext *ibcontext)
 }
 
 /* Device */
-
-static struct bnxt_re_dev *bnxt_re_from_netdev(struct net_device *netdev)
-{
-	struct ib_device *ibdev =
-		ib_device_get_by_netdev(netdev, RDMA_DRIVER_BNXT_RE);
-	if (!ibdev)
-		return NULL;
-
-	return container_of(ibdev, struct bnxt_re_dev, ibdev);
-}
-
 static ssize_t hw_rev_show(struct device *device, struct device_attribute *attr,
 			   char *buf)
 {
diff --git a/drivers/infiniband/hw/usnic/usnic_ib_main.c b/drivers/infiniband/hw/usnic/usnic_ib_main.c
index 5ad7fe7e662f..4ddcd5860e0f 100644
--- a/drivers/infiniband/hw/usnic/usnic_ib_main.c
+++ b/drivers/infiniband/hw/usnic/usnic_ib_main.c
@@ -192,10 +192,12 @@ static void usnic_ib_handle_usdev_event(struct usnic_ib_dev *us_ibdev,
 
 static void usnic_ib_handle_port_event(struct ib_device *ibdev,
 				       struct net_device *netdev,
-				       unsigned long event);
+				       unsigned long event)
 {
 	struct usnic_ib_dev *us_ibdev =
 			container_of(ibdev, struct usnic_ib_dev, ib_dev);
+	struct ib_event ib_event;
+
 	mutex_lock(&us_ibdev->usdev_lock);
 	switch (event) {
 	case NETDEV_UP:
diff --git a/drivers/infiniband/sw/siw/siw_verbs.c b/drivers/infiniband/sw/siw/siw_verbs.c
index 137819184b3b..6b24438df917 100644
--- a/drivers/infiniband/sw/siw/siw_verbs.c
+++ b/drivers/infiniband/sw/siw/siw_verbs.c
@@ -172,6 +172,7 @@ int siw_query_port(struct ib_device *base_dev, u32 port,
 		   struct ib_port_attr *attr)
 {
 	struct siw_device *sdev = to_siw_dev(base_dev);
+	struct net_device *ndev;
 	int rv;
 
 	memset(attr, 0, sizeof(*attr));
@@ -183,7 +184,12 @@ int siw_query_port(struct ib_device *base_dev, u32 port,
 	attr->max_mtu = ib_mtu_int_to_enum(sdev->netdev->mtu);
 	attr->active_mtu = ib_mtu_int_to_enum(sdev->netdev->mtu);
 	attr->port_cap_flags = IB_PORT_CM_SUP | IB_PORT_DEVICE_MGMT_SUP;
-	attr->state = ib_get_curr_port_state(sdev->ndev);
+	ndev = ib_device_get_netdev(base_dev, port);
+	if (ndev)
+		attr->state = ib_get_curr_port_state(ndev);
+	else
+		attr->state = IB_PORT_DOWN;
+	dev_put(ndev);
 	attr->phys_state = attr->state == IB_PORT_ACTIVE ?
 		IB_PORT_PHYS_STATE_LINK_UP : IB_PORT_PHYS_STATE_DISABLED;
 	/*


> 
> Junxian

Re: [PATCH RFC 00/12] RDMA: Support link status events dispatching in ib_core

[Junxian Huang]

On 2024/12/24 21:38, Leon Romanovsky wrote:
> On Tue, Dec 24, 2024 at 08:05:26PM +0800, Junxian Huang wrote:
>>
>>
>> On 2024/12/24 18:32, Leon Romanovsky wrote:
>>> On Fri, Nov 22, 2024 at 06:52:56PM +0800, Junxian Huang wrote:
>>>> This series is to integrate a common link status event handler in
>>>> ib_core as this functionality is needed by most drivers and
>>>> implemented in very similar patterns. This is not a new issue but
>>>> a restart of the previous work of our colleagues from several years
>>>> ago, please see [1] and [2].
>>>>
>>>> [1]: https://lore.kernel.org/linux-rdma/1570184954-21384-1-git-send-email-liweihang@hisilicon.com/
>>>> [2]: https://lore.kernel.org/linux-rdma/20200204082408.18728-1-liweihang@huawei.com/
>>>>
>>>> With this series, ib_core can handle netdev events of link status,
>>>> i.e. NETDEV_UP, NETDEV_DOWN and NETDEV_CHANGE, and dispatch ib port
>>>> events to ULPs instead of drivers. However some drivers currently
>>>> have some private processing in their handler, rather than simply
>>>> dispatching events. For these drivers, this series provides a new
>>>> ops report_port_event(). If this ops is set, ib_core will call it
>>>> and the events will still be handled in the driver.
>>>>
>>>> Events of LAG devices are also not handled in ib_core as currently
>>>> there is no way to obtain ibdev from upper netdev in ib_core. This
>>>> can be a TODO work after the core have more support for LAG. For
>>>> now mlx5 is the only driver that supports RoCE LAG, and the events
>>>> handling of mlx5 RoCE LAG will remain in mlx5 driver.
>>>>
>>>> In this series:
>>>>
>>>> Patch #1 adds a new helper to query the port num of a netdev
>>>> associated with an ibdev. This is used in the following patch.
>>>>
>>>> Patch #2 adds support for link status events dispatching in ib_core.
>>>>
>>>> Patch #3-#7 removes link status event handler in several drivers.
>>>> The port state setting in erdma, rxe and siw are replaced with
>>>> ib_get_curr_port_state(), so their handler can be totally removed.
>>>>
>>>> Patch #8-#10 add support for report_port_event() ops in usnic, mlx4
>>>> and pvrdma as their current handler cannot be perfectly replaced by
>>>> the ib_core handler in patch #2.
>>>>
>>>> Patch #11 adds a check in mlx5 that only events of RoCE LAG will be
>>>> handled in mlx5 driver.
>>>>
>>>> Patch #12 adds a fast path for link-down events dispatching in hns by
>>>> getting notified from hns3 nic driver directly.
>>>>
>>>> Yuyu Li (12):
>>>>   RDMA/core: Add ib_query_netdev_port() to query netdev port by IB
>>>>     device.
>>>>   RDMA/core: Support link status events dispatching
>>>>   RDMA/bnxt_re: Remove deliver net device event
>>>>   RDMA/erdma: Remove deliver net device event
>>>>   RDMA/irdma: Remove deliver net device event
>>>>   RDMA/rxe: Remove deliver net device event
>>>>   RDMA/siw: Remove deliver net device event
>>>>   RDMA/usnic: Support report_port_event() ops
>>>>   RDMA/mlx4: Support report_port_event() ops
>>>>   RDMA/pvrdma: Support report_port_event() ops
>>>>   RDMA/mlx5: Handle link status event only for LAG device
>>>>   RDMA/hns: Support fast path for link-down events dispatching
>>>
>>> I took the series as it is good thing to remove code duplication
>>> and we waited enough.
>>>
>>
>> Thanks Leon.
>>
>> The kernel test robot has reported one warning and one error for
>> this series:
>>
>> https://lore.kernel.org/oe-kbuild-all/202411251625.VrcLuTRx-lkp@intel.com/
>> https://lore.kernel.org/oe-kbuild-all/202411251727.RFxtcpiI-lkp@intel.com/
>>
>> I was planning to fix them when I could send the formal patches,
>> but since you have applied these RFC patches，could you please
>> fix them on your wip branch, or should I send separate patches
>> to fix them?
> 
> This is how I fixed it. Is it ok?
> 
> diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c
> index 4286fd4a9324..b886fe2922ae 100644
> --- a/drivers/infiniband/hw/bnxt_re/main.c
> +++ b/drivers/infiniband/hw/bnxt_re/main.c
> @@ -822,17 +822,6 @@ static void bnxt_re_disassociate_ucontext(struct ib_ucontext *ibcontext)
>  }
>  
>  /* Device */
> -
> -static struct bnxt_re_dev *bnxt_re_from_netdev(struct net_device *netdev)
> -{
> -	struct ib_device *ibdev =
> -		ib_device_get_by_netdev(netdev, RDMA_DRIVER_BNXT_RE);
> -	if (!ibdev)
> -		return NULL;
> -
> -	return container_of(ibdev, struct bnxt_re_dev, ibdev);
> -}
> -
>  static ssize_t hw_rev_show(struct device *device, struct device_attribute *attr,
>  			   char *buf)
>  {
> diff --git a/drivers/infiniband/hw/usnic/usnic_ib_main.c b/drivers/infiniband/hw/usnic/usnic_ib_main.c
> index 5ad7fe7e662f..4ddcd5860e0f 100644
> --- a/drivers/infiniband/hw/usnic/usnic_ib_main.c
> +++ b/drivers/infiniband/hw/usnic/usnic_ib_main.c
> @@ -192,10 +192,12 @@ static void usnic_ib_handle_usdev_event(struct usnic_ib_dev *us_ibdev,
>  
>  static void usnic_ib_handle_port_event(struct ib_device *ibdev,
>  				       struct net_device *netdev,
> -				       unsigned long event);
> +				       unsigned long event)
>  {
>  	struct usnic_ib_dev *us_ibdev =
>  			container_of(ibdev, struct usnic_ib_dev, ib_dev);
> +	struct ib_event ib_event;
> +
>  	mutex_lock(&us_ibdev->usdev_lock);
>  	switch (event) {
>  	case NETDEV_UP:
> diff --git a/drivers/infiniband/sw/siw/siw_verbs.c b/drivers/infiniband/sw/siw/siw_verbs.c
> index 137819184b3b..6b24438df917 100644
> --- a/drivers/infiniband/sw/siw/siw_verbs.c
> +++ b/drivers/infiniband/sw/siw/siw_verbs.c
> @@ -172,6 +172,7 @@ int siw_query_port(struct ib_device *base_dev, u32 port,
>  		   struct ib_port_attr *attr)
>  {
>  	struct siw_device *sdev = to_siw_dev(base_dev);
> +	struct net_device *ndev;
>  	int rv;
>  
>  	memset(attr, 0, sizeof(*attr));
> @@ -183,7 +184,12 @@ int siw_query_port(struct ib_device *base_dev, u32 port,
>  	attr->max_mtu = ib_mtu_int_to_enum(sdev->netdev->mtu);
>  	attr->active_mtu = ib_mtu_int_to_enum(sdev->netdev->mtu);
>  	attr->port_cap_flags = IB_PORT_CM_SUP | IB_PORT_DEVICE_MGMT_SUP;
> -	attr->state = ib_get_curr_port_state(sdev->ndev);
> +	ndev = ib_device_get_netdev(base_dev, port);
> +	if (ndev)
> +		attr->state = ib_get_curr_port_state(ndev);
> +	else
> +		attr->state = IB_PORT_DOWN;
> +	dev_put(ndev);

I think this is a simpler way:

attr->state = ib_get_curr_port_state(sdev->netdev);

But overall LGTM, thanks.

BTW, it seems the kernel test robot has reported some more warnings
after you applied these patches (and solved the conflicts I guess?)

Thanks,
Junxian

>  	attr->phys_state = attr->state == IB_PORT_ACTIVE ?
>  		IB_PORT_PHYS_STATE_LINK_UP : IB_PORT_PHYS_STATE_DISABLED;
>  	/*
> 
> 
>>
>> Junxian
> 

Re: [PATCH RFC 00/12] RDMA: Support link status events dispatching in ib_core

[Leon Romanovsky]

On Wed, Dec 25, 2024 at 02:12:58PM +0800, Junxian Huang wrote:
> 
> 
> On 2024/12/24 21:38, Leon Romanovsky wrote:
> > On Tue, Dec 24, 2024 at 08:05:26PM +0800, Junxian Huang wrote:
> >>
> >>
> >> On 2024/12/24 18:32, Leon Romanovsky wrote:
> >>> On Fri, Nov 22, 2024 at 06:52:56PM +0800, Junxian Huang wrote:
> >>>> This series is to integrate a common link status event handler in
> >>>> ib_core as this functionality is needed by most drivers and
> >>>> implemented in very similar patterns. This is not a new issue but
> >>>> a restart of the previous work of our colleagues from several years
> >>>> ago, please see [1] and [2].
> >>>>
> >>>> [1]: https://lore.kernel.org/linux-rdma/1570184954-21384-1-git-send-email-liweihang@hisilicon.com/
> >>>> [2]: https://lore.kernel.org/linux-rdma/20200204082408.18728-1-liweihang@huawei.com/
> >>>>
> >>>> With this series, ib_core can handle netdev events of link status,
> >>>> i.e. NETDEV_UP, NETDEV_DOWN and NETDEV_CHANGE, and dispatch ib port
> >>>> events to ULPs instead of drivers. However some drivers currently
> >>>> have some private processing in their handler, rather than simply
> >>>> dispatching events. For these drivers, this series provides a new
> >>>> ops report_port_event(). If this ops is set, ib_core will call it
> >>>> and the events will still be handled in the driver.
> >>>>
> >>>> Events of LAG devices are also not handled in ib_core as currently
> >>>> there is no way to obtain ibdev from upper netdev in ib_core. This
> >>>> can be a TODO work after the core have more support for LAG. For
> >>>> now mlx5 is the only driver that supports RoCE LAG, and the events
> >>>> handling of mlx5 RoCE LAG will remain in mlx5 driver.
> >>>>
> >>>> In this series:
> >>>>
> >>>> Patch #1 adds a new helper to query the port num of a netdev
> >>>> associated with an ibdev. This is used in the following patch.
> >>>>
> >>>> Patch #2 adds support for link status events dispatching in ib_core.
> >>>>
> >>>> Patch #3-#7 removes link status event handler in several drivers.
> >>>> The port state setting in erdma, rxe and siw are replaced with
> >>>> ib_get_curr_port_state(), so their handler can be totally removed.
> >>>>
> >>>> Patch #8-#10 add support for report_port_event() ops in usnic, mlx4
> >>>> and pvrdma as their current handler cannot be perfectly replaced by
> >>>> the ib_core handler in patch #2.
> >>>>
> >>>> Patch #11 adds a check in mlx5 that only events of RoCE LAG will be
> >>>> handled in mlx5 driver.
> >>>>
> >>>> Patch #12 adds a fast path for link-down events dispatching in hns by
> >>>> getting notified from hns3 nic driver directly.
> >>>>
> >>>> Yuyu Li (12):
> >>>>   RDMA/core: Add ib_query_netdev_port() to query netdev port by IB
> >>>>     device.
> >>>>   RDMA/core: Support link status events dispatching
> >>>>   RDMA/bnxt_re: Remove deliver net device event
> >>>>   RDMA/erdma: Remove deliver net device event
> >>>>   RDMA/irdma: Remove deliver net device event
> >>>>   RDMA/rxe: Remove deliver net device event
> >>>>   RDMA/siw: Remove deliver net device event
> >>>>   RDMA/usnic: Support report_port_event() ops
> >>>>   RDMA/mlx4: Support report_port_event() ops
> >>>>   RDMA/pvrdma: Support report_port_event() ops
> >>>>   RDMA/mlx5: Handle link status event only for LAG device
> >>>>   RDMA/hns: Support fast path for link-down events dispatching
> >>>
> >>> I took the series as it is good thing to remove code duplication
> >>> and we waited enough.
> >>>
> >>
> >> Thanks Leon.
> >>
> >> The kernel test robot has reported one warning and one error for
> >> this series:
> >>
> >> https://lore.kernel.org/oe-kbuild-all/202411251625.VrcLuTRx-lkp@intel.com/
> >> https://lore.kernel.org/oe-kbuild-all/202411251727.RFxtcpiI-lkp@intel.com/
> >>
> >> I was planning to fix them when I could send the formal patches,
> >> but since you have applied these RFC patches，could you please
> >> fix them on your wip branch, or should I send separate patches
> >> to fix them?
> > 
> > This is how I fixed it. Is it ok?
> > 
> > diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c
> > index 4286fd4a9324..b886fe2922ae 100644
> > --- a/drivers/infiniband/hw/bnxt_re/main.c
> > +++ b/drivers/infiniband/hw/bnxt_re/main.c
> > @@ -822,17 +822,6 @@ static void bnxt_re_disassociate_ucontext(struct ib_ucontext *ibcontext)
> >  }
> >  
> >  /* Device */
> > -
> > -static struct bnxt_re_dev *bnxt_re_from_netdev(struct net_device *netdev)
> > -{
> > -	struct ib_device *ibdev =
> > -		ib_device_get_by_netdev(netdev, RDMA_DRIVER_BNXT_RE);
> > -	if (!ibdev)
> > -		return NULL;
> > -
> > -	return container_of(ibdev, struct bnxt_re_dev, ibdev);
> > -}
> > -
> >  static ssize_t hw_rev_show(struct device *device, struct device_attribute *attr,
> >  			   char *buf)
> >  {
> > diff --git a/drivers/infiniband/hw/usnic/usnic_ib_main.c b/drivers/infiniband/hw/usnic/usnic_ib_main.c
> > index 5ad7fe7e662f..4ddcd5860e0f 100644
> > --- a/drivers/infiniband/hw/usnic/usnic_ib_main.c
> > +++ b/drivers/infiniband/hw/usnic/usnic_ib_main.c
> > @@ -192,10 +192,12 @@ static void usnic_ib_handle_usdev_event(struct usnic_ib_dev *us_ibdev,
> >  
> >  static void usnic_ib_handle_port_event(struct ib_device *ibdev,
> >  				       struct net_device *netdev,
> > -				       unsigned long event);
> > +				       unsigned long event)
> >  {
> >  	struct usnic_ib_dev *us_ibdev =
> >  			container_of(ibdev, struct usnic_ib_dev, ib_dev);
> > +	struct ib_event ib_event;
> > +
> >  	mutex_lock(&us_ibdev->usdev_lock);
> >  	switch (event) {
> >  	case NETDEV_UP:
> > diff --git a/drivers/infiniband/sw/siw/siw_verbs.c b/drivers/infiniband/sw/siw/siw_verbs.c
> > index 137819184b3b..6b24438df917 100644
> > --- a/drivers/infiniband/sw/siw/siw_verbs.c
> > +++ b/drivers/infiniband/sw/siw/siw_verbs.c
> > @@ -172,6 +172,7 @@ int siw_query_port(struct ib_device *base_dev, u32 port,
> >  		   struct ib_port_attr *attr)
> >  {
> >  	struct siw_device *sdev = to_siw_dev(base_dev);
> > +	struct net_device *ndev;
> >  	int rv;
> >  
> >  	memset(attr, 0, sizeof(*attr));
> > @@ -183,7 +184,12 @@ int siw_query_port(struct ib_device *base_dev, u32 port,
> >  	attr->max_mtu = ib_mtu_int_to_enum(sdev->netdev->mtu);
> >  	attr->active_mtu = ib_mtu_int_to_enum(sdev->netdev->mtu);
> >  	attr->port_cap_flags = IB_PORT_CM_SUP | IB_PORT_DEVICE_MGMT_SUP;
> > -	attr->state = ib_get_curr_port_state(sdev->ndev);
> > +	ndev = ib_device_get_netdev(base_dev, port);
> > +	if (ndev)
> > +		attr->state = ib_get_curr_port_state(ndev);
> > +	else
> > +		attr->state = IB_PORT_DOWN;
> > +	dev_put(ndev);
> 
> I think this is a simpler way:
> 
> attr->state = ib_get_curr_port_state(sdev->netdev);
> 
> But overall LGTM, thanks.
> 
> BTW, it seems the kernel test robot has reported some more warnings
> after you applied these patches (and solved the conflicts I guess?)

I'll fix them, this is why we have wip/* branches :). 

Thanks

> 
> Thanks,
> Junxian
> 
> >  	attr->phys_state = attr->state == IB_PORT_ACTIVE ?
> >  		IB_PORT_PHYS_STATE_LINK_UP : IB_PORT_PHYS_STATE_DISABLED;
> >  	/*
> > 
> > 
> >>
> >> Junxian
> >