Re: [RFC PATCH 1/2] RDMA/rxe: drive UDP tunnel socket lifetime from the GID table

[Zhu Yanjun <yanjun.zhu@linux.dev>]

在 2026/7/18 7:26, Serhat Kumral 写道:
> rxe creates the shared per-netns UDP tunnel sockets from rxe_newlink()
> and releases them from several teardown paths (dellink, the
> NETDEV_UNREGISTER notifier and a pernet exit hook), with driver side
> bookkeeping deciding when the last user is gone. Keeping that state in
> sync with the core GID table has been a recurring source of lifetime
> bugs, most recently a refcount underflow / use-after-free reported by
> syzkaller when those paths raced.
> 
> Implement the add_gid/del_gid device ops and drive the socket lifetime
> from the GID entries instead: the first RoCEv2 GID entry of an address
> family in a netns creates the wildcard socket, the last one releases
> it. The GID table guarantees one del_gid for every successfully
> installed GID entry, so the counters have a single owner and cannot
> underflow. When an address, a netdev or the whole netns goes away,
> the core removes the GID entries and the sockets follow.
> 
> The per-netns state lives in a small global list keyed by netns.
> Event-triggered GID removal is asynchronous, so a netns can be torn
> down before the last del_gid has run, e.g. after its netdev was
> moved to another netns; a pernet exit hook closes the sockets of
> such a dying netns before the net is freed. The hook is registered
> as a pernet subsys so it runs after the netdev cleanup of a dying
> netns, which already waits for the del_gid calls a normal teardown
> triggers; it has no pernet storage and takes no part in the normal
> accounting. The netdev notifier now handles only device unregister,
> MTU and port state events. The old rxe_ns.[ch] bookkeeping, the
> sk_refcnt magic in rxe_net_del(), and the .dellink hook are gone.
> The RX data path is unchanged once a GID entry has its sockets.
> 
> The main user visible change: socket setup errors no longer fail
> "rdma link add". The core does not propagate GID add errors back
> through device registration and does not retry them later, so a busy
> port 4791, an allocation failure or a vanishing netns now shows up
> as a missing GID entry plus the GID cache warning in the kernel log
> instead of a failed link add. Re-adding the address or recreating
> the link retries the socket setup.
> 
> Suggested-by: Jason Gunthorpe <jgg@ziepe.ca>
> Fixes: f1327abd6abe ("RDMA/rxe: Support RDMA link creation and destruction per net namespace")
> Reported-by: syzbot+8c9eede336e3a843750e@syzkaller.appspotmail.com
> Closes: https://syzkaller.appspot.com/bug?extid=8c9eede336e3a843750e
> Assisted-by: Claude:claude-fable-5
> Signed-off-by: Serhat Kumral <serhatkumral1@gmail.com>
> ---
> Sending as RFC after reworking the socket lifetime around the GID
> table, as Jason suggested.
> 
> I first tried the literal reading, one bound socket per GID entry,
> but real addresses get in the way: default GIDs derived from the MAC
> are not bindable addresses, IPv6 GIDs are added while still tentative
> so bind() fails and nothing re-adds them after DAD, and multicast RX
> cannot match address-bound sockets. So this version keeps the
> wildcard sockets as they are today and only moves their lifetime to
> add_gid/del_gid, which is where the actual bugs were.
> 
> The pernet exit hook survives in a reduced form, as a backstop only:
> event-triggered GID removal is asynchronous, so a netns can be gone
> before the last del_gid runs (most easily by moving the netdev to
> another netns and deleting the old one), and something has to close
> the kernel sockets before the net they live in is freed. It is
> registered as a subsys so that a normal netns teardown still
> releases the sockets through del_gid; the hook only picks up what
> GID removal could not get to, and it has no pernet storage and takes
> no part in the normal accounting.
> 
> The global list is linear and its mutex is held across socket
> creation and release; both are rare slow paths and the number of
> netns with RoCEv2 GIDs is small, so I kept it simple instead of
> using a hashtable.
> 
> Tested with KASAN and lockdep under concurrent device/GID teardown,
> network namespace destruction, address churn and port 4791 conflicts.
> No splats or leaked tunnel sockets were observed.
> 
>   drivers/infiniband/sw/rxe/Makefile    |   3 +-
>   drivers/infiniband/sw/rxe/rxe.c       |  26 +-
>   drivers/infiniband/sw/rxe/rxe_net.c   | 341 ++++++++++++++++----------
>   drivers/infiniband/sw/rxe/rxe_net.h   |   5 +-
>   drivers/infiniband/sw/rxe/rxe_ns.c    | 124 ----------
>   drivers/infiniband/sw/rxe/rxe_ns.h    |  26 --
>   drivers/infiniband/sw/rxe/rxe_verbs.c |   2 +
>   7 files changed, 223 insertions(+), 304 deletions(-)
>   delete mode 100644 drivers/infiniband/sw/rxe/rxe_ns.c
>   delete mode 100644 drivers/infiniband/sw/rxe/rxe_ns.h
> 
> diff --git a/drivers/infiniband/sw/rxe/Makefile b/drivers/infiniband/sw/rxe/Makefile
> index e097c1ca1874..18ff60542744 100644
> --- a/drivers/infiniband/sw/rxe/Makefile
> +++ b/drivers/infiniband/sw/rxe/Makefile
> @@ -23,7 +23,6 @@ rdma_rxe-y := \
>   	rxe_task.o \
>   	rxe_net.o \
>   	rxe_hw_counters.o \
> -	rxe_mad.o \
> -	rxe_ns.o
> +	rxe_mad.o
>   
>   rdma_rxe-$(CONFIG_INFINIBAND_ON_DEMAND_PAGING) += rxe_odp.o
> diff --git a/drivers/infiniband/sw/rxe/rxe.c b/drivers/infiniband/sw/rxe/rxe.c
> index af39209d0fcf..66d5986dac99 100644
> --- a/drivers/infiniband/sw/rxe/rxe.c
> +++ b/drivers/infiniband/sw/rxe/rxe.c
> @@ -9,7 +9,6 @@
>   #include "rxe.h"
>   #include "rxe_loc.h"
>   #include "rxe_net.h"
> -#include "rxe_ns.h"
>   
>   MODULE_AUTHOR("Bob Pearson, Frank Zago, John Groves, Kamal Heib");
>   MODULE_DESCRIPTION("Soft RDMA transport");
> @@ -202,8 +201,6 @@ void rxe_set_mtu(struct rxe_dev *rxe, unsigned int ndev_mtu)
>   	port->mtu_cap = ib_mtu_enum_to_int(mtu);
>   }
>   
> -static struct rdma_link_ops rxe_link_ops;
> -
>   /* called by ifc layer to create new rxe device.
>    * The caller should allocate memory for rxe by calling ib_alloc_device.
>    */
> @@ -212,7 +209,6 @@ int rxe_add(struct rxe_dev *rxe, unsigned int mtu, const char *ibdev_name,
>   {
>   	rxe_init(rxe, ndev);
>   	rxe_set_mtu(rxe, mtu);
> -	rxe->ib_dev.link_ops = &rxe_link_ops;
>   
>   	return rxe_register_device(rxe, ibdev_name, ndev);
>   }
> @@ -236,10 +232,6 @@ static int rxe_newlink(const char *ibdev_name, struct net_device *ndev)
>   		goto err;
>   	}
>   
> -	err = rxe_net_init(ndev);
> -	if (err)
> -		return err;
> -
>   	err = rxe_net_add(ibdev_name, ndev);
>   	if (err) {
>   		rxe_err("failed to add %s\n", ndev->name);
> @@ -249,17 +241,9 @@ static int rxe_newlink(const char *ibdev_name, struct net_device *ndev)
>   	return err;
>   }
>   
> -static int rxe_dellink(struct ib_device *dev)
> -{
> -	rxe_net_del(dev);
> -
> -	return 0;
> -}
> -
>   static struct rdma_link_ops rxe_link_ops = {
>   	.type = "rxe",
>   	.newlink = rxe_newlink,
> -	.dellink = rxe_dellink,
>   };
>   
>   static int __init rxe_module_init(void)
> @@ -270,21 +254,15 @@ static int __init rxe_module_init(void)
>   	if (err)
>   		return err;
>   
> -	err = rxe_namespace_init();
> -	if (err)
> -		goto err_destroy_wq;
> -
>   	err = rxe_register_notifier();
>   	if (err)
> -		goto err_namespace_exit;
> +		goto err_destroy_wq;
>   
>   	rdma_link_register(&rxe_link_ops);
>   
>   	pr_info("loaded\n");
>   	return 0;
>   
> -err_namespace_exit:
> -	rxe_namespace_exit();
>   err_destroy_wq:
>   	rxe_destroy_wq();
>   	return err;
> @@ -297,8 +275,6 @@ static void __exit rxe_module_exit(void)
>   	rxe_net_exit();
>   	rxe_destroy_wq();
>   
> -	rxe_namespace_exit();
> -
>   	pr_info("unloaded\n");
>   }
>   
> diff --git a/drivers/infiniband/sw/rxe/rxe_net.c b/drivers/infiniband/sw/rxe/rxe_net.c
> index 3741b2c4b0bb..1c1119e71615 100644
> --- a/drivers/infiniband/sw/rxe/rxe_net.c
> +++ b/drivers/infiniband/sw/rxe/rxe_net.c
> @@ -17,11 +17,6 @@
>   #include "rxe.h"
>   #include "rxe_net.h"
>   #include "rxe_loc.h"
> -#include "rxe_ns.h"
> -
> -#ifndef SK_REF_FOR_TUNNEL
> -#define SK_REF_FOR_TUNNEL	2
> -#endif
>   
>   #ifdef CONFIG_DEBUG_LOCK_ALLOC
>   /*
> @@ -81,9 +76,10 @@ static inline void rxe_reclassify_recv_socket(struct socket *sock)
>   	 * from being called and 'rmmod rdma_rxe'
>   	 * is refused because of the references.
>   	 *
> -	 * For the global sockets in recv_sockets,
> -	 * we are sure that rxe_net_exit() will call
> -	 * rxe_release_udp_tunnel -> udp_tunnel_sock_release.
> +	 * For the shared tunnel sockets, we are sure
> +	 * that udp_tunnel_sock_release is called,
> +	 * normally by the last rxe_del_gid() and at
> +	 * the latest by the pernet exit backstop.
>   	 *
>   	 * So we don't need the additional reference to
>   	 * our own (THIS_MODULE).
> @@ -141,7 +137,7 @@ static struct dst_entry *rxe_find_route6(struct rxe_qp *qp,
>   	memcpy(&fl6.daddr, daddr, sizeof(*daddr));
>   	fl6.flowi6_proto = IPPROTO_UDP;
>   
> -	ndst = ip6_dst_lookup_flow(net, rxe_ns_pernet_sk6(net), &fl6, NULL);
> +	ndst = ip6_dst_lookup_flow(net, NULL, &fl6, NULL);
>   	if (IS_ERR(ndst)) {
>   		rxe_dbg_qp(qp, "no route to %pI6\n", daddr);
>   		return NULL;
> @@ -288,10 +284,204 @@ static struct socket *rxe_setup_udp_tunnel(struct net *net, __be16 port,
>   	return sock;
>   }
>   
> -static void rxe_release_udp_tunnel(struct sock *sk)
> +/*
> + * The wildcard tunnel sockets are shared by every RoCEv2 GID entry in
> + * a netns. Their lifetime is driven by add_gid/del_gid: the first GID
> + * entry of an address family creates the socket and normally the last
> + * one releases it; the pernet exit hook below can get there first
> + * when the netns dies. The GID table guarantees one del_gid for
> + * every successfully installed GID entry, so the counts cannot
> + * underflow and nothing else needs to manage the sockets: when an
> + * address, a netdev or the whole netns goes away, the core removes
> + * the GID entries and the sockets follow.
> + *
> + * The per netns state lives in a small global list instead of pernet
> + * storage: an entry exists only while the netns has RoCEv2 GID
> + * entries. Event-triggered GID removal is asynchronous, so a netns
> + * can be torn down before the last del_gid has run; the pernet exit
> + * hook closes the sockets before the net is freed and invalidates
> + * the entry's key so it cannot match a recycled net pointer, and the
> + * remaining del_gid calls only drop the counts.
> + */
> +struct rxe_ns_sock {
> +	struct list_head node;
> +	struct net *net;
> +	struct socket *sk4;
> +	struct socket *sk6;
> +	int nr4;
> +	int nr6;
> +};
> +
> +static DEFINE_MUTEX(rxe_ns_lock);
> +static LIST_HEAD(rxe_ns_list);
> +
> +static struct rxe_ns_sock *rxe_ns_find(struct net *net)
>   {
> -	if (sk)
> -		udp_tunnel_sock_release(sk);
> +	struct rxe_ns_sock *ns;
> +
> +	lockdep_assert_held(&rxe_ns_lock);
> +
> +	list_for_each_entry(ns, &rxe_ns_list, node)
> +		if (ns->net == net)
> +			return ns;
> +
> +	return NULL;
> +}
> +
> +/*
> + * Close the sockets of a dying netns before the net is freed. This
> + * only matters when GID entries still reference the netns at that
> + * point, e.g. after a netdev was moved to another netns and the
> + * queued removal of its GID entries has not run yet. Clearing
> + * ns->net keeps the entry from matching a recycled net pointer; the
> + * outstanding del_gid calls drop the counts and free it.
> + */
> +static void __net_exit rxe_ns_exit(struct net *net)
> +{
> +	struct rxe_ns_sock *ns;
> +
> +	mutex_lock(&rxe_ns_lock);
> +	ns = rxe_ns_find(net);
> +	if (ns) {
> +		if (ns->sk4)
> +			udp_tunnel_sock_release(ns->sk4->sk);
> +		if (ns->sk6)
> +			udp_tunnel_sock_release(ns->sk6->sk);
> +		ns->sk4 = NULL;
> +		ns->sk6 = NULL;
> +		ns->net = NULL;
> +	}
> +	mutex_unlock(&rxe_ns_lock);
> +}
> +
> +static struct pernet_operations rxe_pernet_ops = {
> +	.exit = rxe_ns_exit,
> +};
> +
> +int rxe_add_gid(const struct ib_gid_attr *attr, void **context)
> +{
> +	bool ipv6 = rdma_gid_attr_network_type(attr) == RDMA_NETWORK_IPV6;
> +	__be16 port = htons(ROCE_V2_UDP_DPORT);
> +	struct net_device *ndev;
> +	struct rxe_ns_sock *ns;
> +	struct socket *sock;
> +	struct net *net;
> +	int err = 0;
> +
> +	*context = NULL;
> +
> +	/* Only RoCEv2 GIDs use a UDP tunnel socket. */
> +	if (attr->gid_type != IB_GID_TYPE_ROCE_UDP_ENCAP)
> +		return 0;
> +
> +	/*
> +	 * Hold the netns across the socket setup; this also fails
> +	 * instead of binding a new socket in a netns that is already
> +	 * being dismantled.
> +	 */
> +	rcu_read_lock();
> +	ndev = rcu_dereference(attr->ndev);
> +	net = ndev ? maybe_get_net(dev_net_rcu(ndev)) : NULL;
> +	rcu_read_unlock();
> +	if (!net)
> +		return -ENODEV;
> +
> +	mutex_lock(&rxe_ns_lock);
> +	ns = rxe_ns_find(net);
> +	if (!ns) {
> +		ns = kzalloc_obj(*ns);
> +		if (!ns) {
> +			err = -ENOMEM;
> +			goto out_unlock;
> +		}
> +		ns->net = net;
> +		list_add(&ns->node, &rxe_ns_list);
> +	}
> +
> +	if (ipv6) {
> +		if (!ns->nr6) {
> +			sock = rxe_setup_udp_tunnel(net, port, true);
> +			if (IS_ERR(sock)) {
> +				err = PTR_ERR(sock);
> +				/*
> +				 * No IPv6 support: leave this GID entry
> +				 * without a socket and without a count;
> +				 * rxe_del_gid() skips a NULL context.
> +				 */
> +				if (err == -EAFNOSUPPORT ||
> +				    err == -EPFNOSUPPORT)
> +					err = 0;
> +				goto out_free;
> +			}
> +			ns->sk6 = sock;
> +		}
> +		ns->nr6++;
> +	} else {
> +		if (!ns->nr4) {
> +			sock = rxe_setup_udp_tunnel(net, port, false);
> +			if (IS_ERR(sock)) {
> +				err = PTR_ERR(sock);
> +				goto out_free;
> +			}
> +			ns->sk4 = sock;
> +		}
> +		ns->nr4++;
> +	}
> +	mutex_unlock(&rxe_ns_lock);
> +
> +	put_net(net);
> +	*context = ns;
> +	return 0;
> +
> +out_free:
> +	if (!ns->nr4 && !ns->nr6) {
> +		list_del(&ns->node);
> +		kfree(ns);
> +	}
> +out_unlock:
> +	mutex_unlock(&rxe_ns_lock);
> +	put_net(net);
> +	return err;
> +}
> +
> +int rxe_del_gid(const struct ib_gid_attr *attr, void **context)
> +{
> +	bool ipv6 = rdma_gid_attr_network_type(attr) == RDMA_NETWORK_IPV6;
> +	struct rxe_ns_sock *ns = *context;
> +	struct socket *sock = NULL;
> +
> +	if (!ns)
> +		return 0;
> +
> +	*context = NULL;
> +
> +	mutex_lock(&rxe_ns_lock);
> +	if (ipv6) {
> +		if (!WARN_ON_ONCE(!ns->nr6) && !--ns->nr6) {
> +			sock = ns->sk6;
> +			ns->sk6 = NULL;
> +		}
> +	} else {
> +		if (!WARN_ON_ONCE(!ns->nr4) && !--ns->nr4) {
> +			sock = ns->sk4;
> +			ns->sk4 = NULL;
> +		}
> +	}
> +	/*
> +	 * Release under the lock: a concurrent rxe_add_gid() must not
> +	 * see a zero count while the old socket still holds the port,
> +	 * or its bind() fails with -EADDRINUSE.
> +	 */
> +	if (sock)
> +		udp_tunnel_sock_release(sock->sk);
> +
> +	if (!ns->nr4 && !ns->nr6) {
> +		list_del(&ns->node);
> +		kfree(ns);
> +	}
> +	mutex_unlock(&rxe_ns_lock);
> +
> +	return 0;
>   }
>   
>   static void prepare_udp_hdr(struct sk_buff *skb, __be16 src_port,
> @@ -631,42 +821,6 @@ int rxe_net_add(const char *ibdev_name, struct net_device *ndev)
>   	return 0;
>   }
>   
> -static void rxe_sock_put(struct sock *sk,
> -					void (*set_sk)(struct net *, struct sock *),
> -					struct net *net)
> -{
> -	if (refcount_read(&sk->sk_refcnt) > SK_REF_FOR_TUNNEL) {
> -		__sock_put(sk);
> -	} else {
> -		rxe_release_udp_tunnel(sk);
> -		sk = NULL;
> -		set_sk(net, sk);
> -	}
> -}
> -
> -void rxe_net_del(struct ib_device *dev)
> -{
> -	struct net_device *ndev;
> -	struct sock *sk;
> -	struct net *net;
> -
> -	ndev = ib_device_get_netdev(dev, 1);
> -	if (!ndev)
> -		return;
> -
> -	net = dev_net(ndev);
> -
> -	sk = rxe_ns_pernet_sk4(net);
> -	if (sk)
> -		rxe_sock_put(sk, rxe_ns_pernet_set_sk4, net);
> -
> -	sk = rxe_ns_pernet_sk6(net);
> -	if (sk)
> -		rxe_sock_put(sk, rxe_ns_pernet_set_sk6, net);
> -
> -	dev_put(ndev);
> -}
> -
>   static void rxe_port_event(struct rxe_dev *rxe,
>   			   enum ib_event_type event)
>   {
> @@ -723,7 +877,6 @@ static int rxe_notify(struct notifier_block *not_blk,
>   	switch (event) {
>   	case NETDEV_UNREGISTER:
>   		ib_unregister_device_queued(&rxe->ib_dev);
> -		rxe_net_del(&rxe->ib_dev);
>   		break;
>   	case NETDEV_CHANGEMTU:
>   		rxe_dbg_dev(rxe, "%s changed mtu to %d\n", ndev->name, ndev->mtu);
> @@ -753,63 +906,24 @@ static struct notifier_block rxe_net_notifier = {
>   	.notifier_call = rxe_notify,
>   };
>   
> -static int rxe_net_ipv4_init(struct net *net)
> -{
> -	struct sock *sk;
> -	struct socket *sock;
> -
> -	sk = rxe_ns_pernet_sk4(net);
> -	if (sk) {
> -		sock_hold(sk);
> -		return 0;
> -	}
> -
> -	sock = rxe_setup_udp_tunnel(net, htons(ROCE_V2_UDP_DPORT), false);
> -	if (IS_ERR(sock)) {
> -		pr_err("Failed to create IPv4 UDP tunnel\n");
> -		return -1;
> -	}
> -	rxe_ns_pernet_set_sk4(net, sock->sk);
> -
> -	return 0;
> -}
> -
> -static int rxe_net_ipv6_init(struct net *net)
> -{
> -#if IS_ENABLED(CONFIG_IPV6)
> -	struct sock *sk;
> -	struct socket *sock;
> -
> -	sk = rxe_ns_pernet_sk6(net);
> -	if (sk) {
> -		sock_hold(sk);
> -		return 0;
> -	}
> -
> -	sock = rxe_setup_udp_tunnel(net, htons(ROCE_V2_UDP_DPORT), true);
> -	if (PTR_ERR(sock) == -EAFNOSUPPORT) {
> -		pr_warn("IPv6 is not supported, can not create a UDPv6 socket\n");
> -		return 0;
> -	}
> -
> -	if (IS_ERR(sock)) {
> -		pr_err("Failed to create IPv6 UDP tunnel\n");
> -		return -1;
> -	}
> -
> -	rxe_ns_pernet_set_sk6(net, sock->sk);
> -
> -#endif
> -	return 0;
> -}
> -
>   int rxe_register_notifier(void)
>   {
>   	int err;
>   
> +	/*
> +	 * A pernet subsys, not a pernet device: its exit hook must run
> +	 * after the netdev cleanup of a dying netns, which waits for
> +	 * the GID entries' netdev references and thus for the del_gid
> +	 * calls that normally release the sockets.
> +	 */
> +	err = register_pernet_subsys(&rxe_pernet_ops);
> +	if (err)
> +		return err;
> +
>   	err = register_netdevice_notifier(&rxe_net_notifier);
>   	if (err) {
>   		pr_err("Failed to register netdev notifier\n");
> +		unregister_pernet_subsys(&rxe_pernet_ops);
>   		return -1;
>   	}
>   
> @@ -819,31 +933,8 @@ int rxe_register_notifier(void)
>   void rxe_net_exit(void)
>   {
>   	unregister_netdevice_notifier(&rxe_net_notifier);
> -}
> -
> -int rxe_net_init(struct net_device *ndev)
> -{
> -	struct net *net;
> -	struct sock *sk;
> -	int err;
> -
> -	net = dev_net(ndev);
> -
> -	err = rxe_net_ipv4_init(net);
> -	if (err)
> -		return err;
> -
> -	err = rxe_net_ipv6_init(net);
> -	if (err)
> -		goto err_out;
> +	unregister_pernet_subsys(&rxe_pernet_ops);
>   
> -	return 0;
> -
> -err_out:
> -	/* If ipv6 error, release ipv4 resource */
> -	sk = rxe_ns_pernet_sk4(net);
> -	if (sk)
> -		rxe_sock_put(sk, rxe_ns_pernet_set_sk4, net);
> -
> -	return err;
> +	/* all devices and thus all GID entries are gone by now */
> +	WARN_ON(!list_empty(&rxe_ns_list));
>   }
> diff --git a/drivers/infiniband/sw/rxe/rxe_net.h b/drivers/infiniband/sw/rxe/rxe_net.h
> index 56249677d692..b564ff5c83f3 100644
> --- a/drivers/infiniband/sw/rxe/rxe_net.h
> +++ b/drivers/infiniband/sw/rxe/rxe_net.h
> @@ -12,10 +12,11 @@
>   #include <linux/module.h>
>   
>   int rxe_net_add(const char *ibdev_name, struct net_device *ndev);
> -void rxe_net_del(struct ib_device *dev);
> +
> +int rxe_add_gid(const struct ib_gid_attr *attr, void **context);
> +int rxe_del_gid(const struct ib_gid_attr *attr, void **context);
>   
>   int rxe_register_notifier(void);
> -int rxe_net_init(struct net_device *ndev);
>   void rxe_net_exit(void);
>   
>   #endif /* RXE_NET_H */
> diff --git a/drivers/infiniband/sw/rxe/rxe_ns.c b/drivers/infiniband/sw/rxe/rxe_ns.c
> deleted file mode 100644
> index 64621c89f8bf..000000000000
> --- a/drivers/infiniband/sw/rxe/rxe_ns.c
> +++ /dev/null
> @@ -1,124 +0,0 @@
> -/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
> -
> -#include <net/sock.h>
> -#include <net/netns/generic.h>
> -#include <net/net_namespace.h>
> -#include <linux/module.h>
> -#include <linux/skbuff.h>
> -#include <linux/pid_namespace.h>
> -#include <net/udp_tunnel.h>
> -
> -#include "rxe_ns.h"
> -
> -/*
> - * Per network namespace data
> - */
> -struct rxe_ns_sock {
> -	struct sock __rcu *rxe_sk4;
> -	struct sock __rcu *rxe_sk6;
> -};
> -
> -/*
> - * Index to store custom data for each network namespace.
> - */
> -static unsigned int rxe_pernet_id;
> -
> -/*
> - * Called for every existing and added network namespaces
> - */
> -static int rxe_ns_init(struct net *net)
> -{
> -	/* defer socket create in the namespace to the first
> -	 * device create.
> -	 */
> -
> -	return 0;
> -}
> -
> -static void rxe_ns_exit(struct net *net)
> -{
> -	/* called when the network namespace is removed
> -	 */
> -	struct rxe_ns_sock *ns_sk = net_generic(net, rxe_pernet_id);
> -	struct sock *sk;
> -
> -	rcu_read_lock();
> -	sk = rcu_dereference(ns_sk->rxe_sk4);
> -	rcu_read_unlock();
> -	if (sk) {
> -		rcu_assign_pointer(ns_sk->rxe_sk4, NULL);
> -		udp_tunnel_sock_release(sk);
> -	}
> -
> -#if IS_ENABLED(CONFIG_IPV6)
> -	rcu_read_lock();
> -	sk = rcu_dereference(ns_sk->rxe_sk6);
> -	rcu_read_unlock();
> -	if (sk) {
> -		rcu_assign_pointer(ns_sk->rxe_sk6, NULL);
> -		udp_tunnel_sock_release(sk);
> -	}
> -#endif
> -}
> -
> -/*
> - * callback to make the module network namespace aware
> - */
> -static struct pernet_operations rxe_net_ops = {
> -	.init = rxe_ns_init,
> -	.exit = rxe_ns_exit,
> -	.id = &rxe_pernet_id,
> -	.size = sizeof(struct rxe_ns_sock),
> -};
> -
> -struct sock *rxe_ns_pernet_sk4(struct net *net)
> -{
> -	struct rxe_ns_sock *ns_sk = net_generic(net, rxe_pernet_id);
> -	struct sock *sk;
> -
> -	rcu_read_lock();
> -	sk = rcu_dereference(ns_sk->rxe_sk4);
> -	rcu_read_unlock();
> -
> -	return sk;
> -}
> -
> -void rxe_ns_pernet_set_sk4(struct net *net, struct sock *sk)
> -{
> -	struct rxe_ns_sock *ns_sk = net_generic(net, rxe_pernet_id);
> -
> -	rcu_assign_pointer(ns_sk->rxe_sk4, sk);
> -	synchronize_rcu();
> -}
> -
> -#if IS_ENABLED(CONFIG_IPV6)
> -struct sock *rxe_ns_pernet_sk6(struct net *net)
> -{
> -	struct rxe_ns_sock *ns_sk = net_generic(net, rxe_pernet_id);
> -	struct sock *sk;
> -
> -	rcu_read_lock();
> -	sk = rcu_dereference(ns_sk->rxe_sk6);
> -	rcu_read_unlock();
> -
> -	return sk;
> -}
> -

Hi Serhat,

While testing your recent RFC patch ("RDMA/rxe: Drive UDP tunnel socket
lifetime from GID table"), I encountered a deterministic kernel hang
when running the RDMA kselftests.

# make -C tools/testing/selftests/ TARGETS=rdma run_tests
make: Entering directory '/root/Development/linux/tools/testing/selftests'
make[1]: Nothing to be done for 'all'.
# timeout set to 45
# selftests: rdma: rxe_rping_between_netns.sh
[Kernel Hangs Here]

The execution freezes permanently during the rxe_rping_between_netns.sh
testcase, which heavily exercises the creation, link configuration,
and teardown of Soft-RoCE devices across multiple network namespaces.

Unfortunately, my local KVM test environment is currently suffering from
severe performance degradation, preventing me from capturing a reliable
kdump or crash log to extract the exact stack trace/lockdep splat.

Given that this specific selftest directly targets the asynchronous
netns deletion and GID churn path that your patch refactored, it is 
highly likely related to the new rxe_ns_exit pernet subsys backstop or 
the global rxe_ns_lock synchronization logic.

Could you please run this selftest in your environment to see if you can
reproduce the deadlock/hang and capture the stack trace?

Thanks,

Zhu Yanjun

> -void rxe_ns_pernet_set_sk6(struct net *net, struct sock *sk)
> -{
> -	struct rxe_ns_sock *ns_sk = net_generic(net, rxe_pernet_id);
> -
> -	rcu_assign_pointer(ns_sk->rxe_sk6, sk);
> -	synchronize_rcu();
> -}
> -#endif /* IPV6 */
> -
> -int rxe_namespace_init(void)
> -{
> -	return register_pernet_subsys(&rxe_net_ops);
> -}
> -
> -void rxe_namespace_exit(void)
> -{
> -	unregister_pernet_subsys(&rxe_net_ops);
> -}
> diff --git a/drivers/infiniband/sw/rxe/rxe_ns.h b/drivers/infiniband/sw/rxe/rxe_ns.h
> deleted file mode 100644
> index 4da2709e6b71..000000000000
> --- a/drivers/infiniband/sw/rxe/rxe_ns.h
> +++ /dev/null
> @@ -1,26 +0,0 @@
> -/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
> -
> -#ifndef RXE_NS_H
> -#define RXE_NS_H
> -
> -struct sock *rxe_ns_pernet_sk4(struct net *net);
> -void rxe_ns_pernet_set_sk4(struct net *net, struct sock *sk);
> -
> -#if IS_ENABLED(CONFIG_IPV6)
> -void rxe_ns_pernet_set_sk6(struct net *net, struct sock *sk);
> -struct sock *rxe_ns_pernet_sk6(struct net *net);
> -#else /* IPv6 */
> -static inline struct sock *rxe_ns_pernet_sk6(struct net *net)
> -{
> -	return NULL;
> -}
> -
> -static inline void rxe_ns_pernet_set_sk6(struct net *net, struct sock *sk)
> -{
> -}
> -#endif /* IPv6 */
> -
> -int rxe_namespace_init(void);
> -void rxe_namespace_exit(void);
> -
> -#endif /* RXE_NS_H */
> diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.c b/drivers/infiniband/sw/rxe/rxe_verbs.c
> index 1ec130fee8ea..7954c2f2fd47 100644
> --- a/drivers/infiniband/sw/rxe/rxe_verbs.c
> +++ b/drivers/infiniband/sw/rxe/rxe_verbs.c
> @@ -1456,6 +1456,7 @@ static const struct ib_device_ops rxe_dev_ops = {
>   	.driver_id = RDMA_DRIVER_RXE,
>   	.uverbs_abi_ver = RXE_UVERBS_ABI_VERSION,
>   
> +	.add_gid = rxe_add_gid,
>   	.alloc_hw_port_stats = rxe_ib_alloc_hw_port_stats,
>   	.alloc_mr = rxe_alloc_mr,
>   	.alloc_mw = rxe_alloc_mw,
> @@ -1471,6 +1472,7 @@ static const struct ib_device_ops rxe_dev_ops = {
>   	.dealloc_mw = rxe_dealloc_mw,
>   	.dealloc_pd = rxe_dealloc_pd,
>   	.dealloc_ucontext = rxe_dealloc_ucontext,
> +	.del_gid = rxe_del_gid,
>   	.dereg_mr = rxe_dereg_mr,
>   	.destroy_ah = rxe_destroy_ah,
>   	.destroy_cq = rxe_destroy_cq,