[PATCH] RDMA/rxe: let rxe_reclassify_recv_socket() call sk_owner_put()

[PATCH] RDMA/rxe: let rxe_reclassify_recv_socket() call sk_owner_put()

[Stefan Metzmacher]

On kernels build with CONFIG_PROVE_LOCKING, CONFIG_MODULES
and CONFIG_DEBUG_LOCK_ALLOC 'rmmod rdma_rxe' is no longer
possible.

For the global recv sockets rxe_net_exit() is where we
call rxe_release_udp_tunnel-> udp_tunnel_sock_release(),
which means the sockets are destroyed before 'rmmod rdma_rxe'
finishes, so there's no need to protect against
rxe_recv_slock_key and rxe_recv_sk_key disappearing
while the sockets are still alive.

Fixes: 80a85a771deb ("RDMA/rxe: reclassify sockets in order to avoid false positives from lockdep")
Cc: Zhu Yanjun <zyjzyj2000@gmail.com>
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: Leon Romanovsky <leon@kernel.org>
Cc: Shinichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Cc: linux-rdma@vger.kernel.org
Cc: netdev@vger.kernel.org
Cc: linux-cifs@vger.kernel.org
Signed-off-by: Stefan Metzmacher <metze@samba.org>
---
 drivers/infiniband/sw/rxe/rxe_net.c | 32 +++++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)

diff --git a/drivers/infiniband/sw/rxe/rxe_net.c b/drivers/infiniband/sw/rxe/rxe_net.c
index 0195d361e5e3..0bd0902b11f7 100644
--- a/drivers/infiniband/sw/rxe/rxe_net.c
+++ b/drivers/infiniband/sw/rxe/rxe_net.c
@@ -64,7 +64,39 @@ static inline void rxe_reclassify_recv_socket(struct socket *sock)
 		break;
 	default:
 		WARN_ON_ONCE(1);
+		return;
 	}
+	/*
+	 * sock_lock_init_class_and_name() calls
+	 * sk_owner_set(sk, THIS_MODULE); in order
+	 * to make sure the referenced global
+	 * variables rxe_recv_slock_key and
+	 * rxe_recv_sk_key are not removed
+	 * before the socket is closed.
+	 *
+	 * However this prevents rxe_net_exit()
+	 * from being called and 'rmmod rdma_rxe'
+	 * is refused because of the references.
+	 *
+	 * For the global sockets in recv_sockets,
+	 * we are sure that rxe_net_exit() will call
+	 * rxe_release_udp_tunnel -> udp_tunnel_sock_release.
+	 *
+	 * So we don't need the additional reference to
+	 * our own (THIS_MODULE).
+	 */
+	sk_owner_put(sk);
+	/*
+	 * We also call sk_owner_clear() otherwise
+	 * sk_owner_put(sk) in sk_prot_free will
+	 * fail, which is called via
+	 * sk_free -> __sk_free -> sk_destruct
+	 * and sk_destruct calls __sk_destruct
+	 * directly or via call_rcu()
+	 * so sk_prot_free() might be called
+	 * after rxe_net_exit().
+	 */
+	sk_owner_clear(sk);
 #endif /* CONFIG_DEBUG_LOCK_ALLOC */
 }
 
-- 
2.43.0

Re: [PATCH] RDMA/rxe: let rxe_reclassify_recv_socket() call sk_owner_put()

[Zhu Yanjun]

在 2025/12/19 6:04, Stefan Metzmacher 写道:
> On kernels build with CONFIG_PROVE_LOCKING, CONFIG_MODULES
> and CONFIG_DEBUG_LOCK_ALLOC 'rmmod rdma_rxe' is no longer
> possible.
> 
> For the global recv sockets rxe_net_exit() is where we
> call rxe_release_udp_tunnel-> udp_tunnel_sock_release(),
> which means the sockets are destroyed before 'rmmod rdma_rxe'
> finishes, so there's no need to protect against
> rxe_recv_slock_key and rxe_recv_sk_key disappearing
> while the sockets are still alive.
> 
> Fixes: 80a85a771deb ("RDMA/rxe: reclassify sockets in order to avoid false positives from lockdep")
> Cc: Zhu Yanjun <zyjzyj2000@gmail.com>
> Cc: Jason Gunthorpe <jgg@ziepe.ca>
> Cc: Leon Romanovsky <leon@kernel.org>
> Cc: Shinichiro Kawasaki <shinichiro.kawasaki@wdc.com>
> Cc: linux-rdma@vger.kernel.org
> Cc: netdev@vger.kernel.org
> Cc: linux-cifs@vger.kernel.org
> Signed-off-by: Stefan Metzmacher <metze@samba.org>

Thanks a lot. IIRC, there is a similar commit for SIW driver. Thus, I am 
not sure if there is a similar problem in SIW driver or not.

Reviewed-by: Zhu Yanjun <yanjun.zhu@linux.dev>

Zhu Yanjun

> ---
>   drivers/infiniband/sw/rxe/rxe_net.c | 32 +++++++++++++++++++++++++++++
>   1 file changed, 32 insertions(+)
> 
> diff --git a/drivers/infiniband/sw/rxe/rxe_net.c b/drivers/infiniband/sw/rxe/rxe_net.c
> index 0195d361e5e3..0bd0902b11f7 100644
> --- a/drivers/infiniband/sw/rxe/rxe_net.c
> +++ b/drivers/infiniband/sw/rxe/rxe_net.c
> @@ -64,7 +64,39 @@ static inline void rxe_reclassify_recv_socket(struct socket *sock)
>   		break;
>   	default:
>   		WARN_ON_ONCE(1);
> +		return;
>   	}
> +	/*
> +	 * sock_lock_init_class_and_name() calls
> +	 * sk_owner_set(sk, THIS_MODULE); in order
> +	 * to make sure the referenced global
> +	 * variables rxe_recv_slock_key and
> +	 * rxe_recv_sk_key are not removed
> +	 * before the socket is closed.
> +	 *
> +	 * However this prevents rxe_net_exit()
> +	 * from being called and 'rmmod rdma_rxe'
> +	 * is refused because of the references.
> +	 *
> +	 * For the global sockets in recv_sockets,
> +	 * we are sure that rxe_net_exit() will call
> +	 * rxe_release_udp_tunnel -> udp_tunnel_sock_release.
> +	 *
> +	 * So we don't need the additional reference to
> +	 * our own (THIS_MODULE).
> +	 */
> +	sk_owner_put(sk);
> +	/*
> +	 * We also call sk_owner_clear() otherwise
> +	 * sk_owner_put(sk) in sk_prot_free will
> +	 * fail, which is called via
> +	 * sk_free -> __sk_free -> sk_destruct
> +	 * and sk_destruct calls __sk_destruct
> +	 * directly or via call_rcu()
> +	 * so sk_prot_free() might be called
> +	 * after rxe_net_exit().
> +	 */
> +	sk_owner_clear(sk);
>   #endif /* CONFIG_DEBUG_LOCK_ALLOC */
>   }
>   

Re: [PATCH] RDMA/rxe: let rxe_reclassify_recv_socket() call sk_owner_put()

[Shinichiro Kawasaki]

On 12/19/25 11:04 PM, Stefan Metzmacher wrote:
> On kernels build with CONFIG_PROVE_LOCKING, CONFIG_MODULES
> and CONFIG_DEBUG_LOCK_ALLOC 'rmmod rdma_rxe' is no longer
> possible.
> 
> For the global recv sockets rxe_net_exit() is where we
> call rxe_release_udp_tunnel-> udp_tunnel_sock_release(),
> which means the sockets are destroyed before 'rmmod rdma_rxe'
> finishes, so there's no need to protect against
> rxe_recv_slock_key and rxe_recv_sk_key disappearing
> while the sockets are still alive.
> 
> Fixes: 80a85a771deb ("RDMA/rxe: reclassify sockets in order to avoid false positives from lockdep")
> Cc: Zhu Yanjun <zyjzyj2000@gmail.com>
> Cc: Jason Gunthorpe <jgg@ziepe.ca>
> Cc: Leon Romanovsky <leon@kernel.org>
> Cc: Shinichiro Kawasaki <shinichiro.kawasaki@wdc.com>
> Cc: linux-rdma@vger.kernel.org
> Cc: netdev@vger.kernel.org
> Cc: linux-cifs@vger.kernel.org
> Signed-off-by: Stefan Metzmacher <metze@samba.org>

I applied this patch on top of v6.19-rc1 kernel, and confirmed the
failure I had reported [1] is no longer observed. Thank you :)

Tested-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>

[1] https://lore.kernel.org/linux-rdma/170e3191-7e15-4af8-948f-14904fe260cc@wdc.com/

Re: [PATCH] RDMA/rxe: let rxe_reclassify_recv_socket() call sk_owner_put()

[Stefan Metzmacher]

Am 20.12.25 um 04:51 schrieb Zhu Yanjun:
> 在 2025/12/19 6:04, Stefan Metzmacher 写道:
>> On kernels build with CONFIG_PROVE_LOCKING, CONFIG_MODULES
>> and CONFIG_DEBUG_LOCK_ALLOC 'rmmod rdma_rxe' is no longer
>> possible.
>>
>> For the global recv sockets rxe_net_exit() is where we
>> call rxe_release_udp_tunnel-> udp_tunnel_sock_release(),
>> which means the sockets are destroyed before 'rmmod rdma_rxe'
>> finishes, so there's no need to protect against
>> rxe_recv_slock_key and rxe_recv_sk_key disappearing
>> while the sockets are still alive.
>>
>> Fixes: 80a85a771deb ("RDMA/rxe: reclassify sockets in order to avoid false positives from lockdep")
>> Cc: Zhu Yanjun <zyjzyj2000@gmail.com>
>> Cc: Jason Gunthorpe <jgg@ziepe.ca>
>> Cc: Leon Romanovsky <leon@kernel.org>
>> Cc: Shinichiro Kawasaki <shinichiro.kawasaki@wdc.com>
>> Cc: linux-rdma@vger.kernel.org
>> Cc: netdev@vger.kernel.org
>> Cc: linux-cifs@vger.kernel.org
>> Signed-off-by: Stefan Metzmacher <metze@samba.org>
> 
> Thanks a lot. IIRC, there is a similar commit for SIW driver. Thus, I am not sure if there is a similar problem in SIW driver or not.

I don't think so, siw and the other place in rxe  are attached to specific connections
and there the reference is ok and needed.

The problem was only related to the two global sockets with the lifetime
the rdma_rxe is loaded.

metze

Re: [PATCH] RDMA/rxe: let rxe_reclassify_recv_socket() call sk_owner_put()

[Zhu Yanjun]

在 2025/12/20 5:05, Stefan Metzmacher 写道:
> Am 20.12.25 um 04:51 schrieb Zhu Yanjun:
>> 在 2025/12/19 6:04, Stefan Metzmacher 写道:
>>> On kernels build with CONFIG_PROVE_LOCKING, CONFIG_MODULES
>>> and CONFIG_DEBUG_LOCK_ALLOC 'rmmod rdma_rxe' is no longer
>>> possible.
>>>
>>> For the global recv sockets rxe_net_exit() is where we
>>> call rxe_release_udp_tunnel-> udp_tunnel_sock_release(),
>>> which means the sockets are destroyed before 'rmmod rdma_rxe'
>>> finishes, so there's no need to protect against
>>> rxe_recv_slock_key and rxe_recv_sk_key disappearing
>>> while the sockets are still alive.
>>>
>>> Fixes: 80a85a771deb ("RDMA/rxe: reclassify sockets in order to avoid 
>>> false positives from lockdep")
>>> Cc: Zhu Yanjun <zyjzyj2000@gmail.com>
>>> Cc: Jason Gunthorpe <jgg@ziepe.ca>
>>> Cc: Leon Romanovsky <leon@kernel.org>
>>> Cc: Shinichiro Kawasaki <shinichiro.kawasaki@wdc.com>
>>> Cc: linux-rdma@vger.kernel.org
>>> Cc: netdev@vger.kernel.org
>>> Cc: linux-cifs@vger.kernel.org
>>> Signed-off-by: Stefan Metzmacher <metze@samba.org>
>>
>> Thanks a lot. IIRC, there is a similar commit for SIW driver. Thus, I 
>> am not sure if there is a similar problem in SIW driver or not.
>
> I don't think so, siw and the other place in rxe  are attached to 
> specific connections
> and there the reference is ok and needed.


Make sense. Thanks.

Yanjun.Zhu


>
> The problem was only related to the two global sockets with the lifetime
> the rdma_rxe is loaded.
>
> metze

-- 
Best Regards,
Yanjun.Zhu

Re: [PATCH] RDMA/rxe: let rxe_reclassify_recv_socket() call sk_owner_put()

[Leon Romanovsky]

On Fri, 19 Dec 2025 15:04:08 +0100, Stefan Metzmacher wrote:
> On kernels build with CONFIG_PROVE_LOCKING, CONFIG_MODULES
> and CONFIG_DEBUG_LOCK_ALLOC 'rmmod rdma_rxe' is no longer
> possible.
> 
> For the global recv sockets rxe_net_exit() is where we
> call rxe_release_udp_tunnel-> udp_tunnel_sock_release(),
> which means the sockets are destroyed before 'rmmod rdma_rxe'
> finishes, so there's no need to protect against
> rxe_recv_slock_key and rxe_recv_sk_key disappearing
> while the sockets are still alive.
> 
> [...]

Applied, thanks!

[1/1] RDMA/rxe: let rxe_reclassify_recv_socket() call sk_owner_put()
      https://git.kernel.org/rdma/rdma/c/de41cbc64d02ae

Best regards,
-- 
Leon Romanovsky <leon@kernel.org>