* [PATCH] RDMA/cma: add rdma_destroy_id_wq() and use it from RDS
@ 2026-11-14 15:05 Aibing Chen
2026-11-15 0:18 ` Jason Gunthorpe
` (2 more replies)
0 siblings, 3 replies; 10+ messages in thread
From: Aibing Chen @ 2026-11-14 15:05 UTC (permalink / raw)
To: jgg, leon, achender
Cc: davem, edumazet, kuba, pabeni, horms, linux-rdma, netdev,
rds-devel, linux-kernel, hdanton, hackerowskiy,
syzbot+ccdb5058c3df2d94811c, Aibing Chen
syzbot reported a possible circular locking dependency between
rds_trans_sem and the RDMA CM handler mutex:
CPU0 (bind path) CPU1 (CM callback path)
---- ----
down_read(&rds_trans_sem) mutex_lock(&handler_mutex)
rds_ib_laddr_check() cma_ib_req_handler()
rdma_destroy_id() rds_ib_cm_handle_connect()
mutex_lock(&handler_mutex) rds_conn_create()
rds_trans_get_preferred()
down_read(&rds_trans_sem)
rds_ib_laddr_check_cm() destroys the CM id from inside
rds_trans_get_preferred()'s read-side critical section, while CM
callbacks create RDS connections (taking the same read lock) from
under handler_mutex.
Break the cycle by destroying the CM id asynchronously: add
rdma_destroy_id_wq(), which queues the destruction on cma_wq, and
switch rds_ib_laddr_check_cm() to it. The destroy work is embedded
in struct rdma_id_private, which keeps the id alive until the work
runs; rds does not touch cm_id after the destroy call.
This formalizes the approach proposed and syzbot-tested by Hillf
Danton; Eduard Zateev additionally verified the fix on a real QEMU
system (KASAN + lockdep + RDMA_RXE, active and passive RDS
connections established without any lockdep output).
Reported-by: syzbot+ccdb5058c3df2d94811c@syzkaller.appspotmail.com
Closes: https://syzkaller.appspot.com/bug?extid=ccdb5058c3df2d94811c
Suggested-by: Hillf Danton <hdanton@sina.com>
Tested-by: syzbot+ccdb5058c3df2d94811c@syzkaller.appspotmail.com
Tested-by: Eduard Zateev <hackerowskiy@gmail.com>
Signed-off-by: Aibing Chen <chenaibing@cmss.chinamobile.com>
---
drivers/infiniband/core/cma.c | 19 +++++++++++++++++++
drivers/infiniband/core/cma_priv.h | 1 +
include/rdma/rdma_cm.h | 1 +
net/rds/ib.c | 2 +-
4 files changed, 22 insertions(+), 1 deletion(-)
diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c
index 337a49d1a..d10aaf51e 100644
--- a/drivers/infiniband/core/cma.c
+++ b/drivers/infiniband/core/cma.c
@@ -2126,6 +2126,25 @@ void rdma_destroy_id(struct rdma_cm_id *id)
}
EXPORT_SYMBOL(rdma_destroy_id);
+static void destroy_id_work(struct work_struct *work)
+{
+ struct rdma_id_private *id_priv =
+ container_of(work, struct rdma_id_private, destroy_work);
+
+ mutex_lock(&id_priv->handler_mutex);
+ destroy_id_handler_unlock(id_priv);
+}
+
+void rdma_destroy_id_wq(struct rdma_cm_id *id)
+{
+ struct rdma_id_private *id_priv =
+ container_of(id, struct rdma_id_private, id);
+
+ INIT_WORK(&id_priv->destroy_work, destroy_id_work);
+ queue_work(cma_wq, &id_priv->destroy_work);
+}
+EXPORT_SYMBOL(rdma_destroy_id_wq);
+
static int cma_rep_recv(struct rdma_id_private *id_priv)
{
int ret;
diff --git a/drivers/infiniband/core/cma_priv.h b/drivers/infiniband/core/cma_priv.h
index 04332eb82..95ff8bbd4 100644
--- a/drivers/infiniband/core/cma_priv.h
+++ b/drivers/infiniband/core/cma_priv.h
@@ -110,6 +110,7 @@ struct rdma_id_private {
*/
struct rdma_restrack_entry res;
struct rdma_ucm_ece ece;
+ struct work_struct destroy_work;
};
#if IS_ENABLED(CONFIG_INFINIBAND_ADDR_TRANS_CONFIGFS)
diff --git a/include/rdma/rdma_cm.h b/include/rdma/rdma_cm.h
index d639ff889..ac1c1d917 100644
--- a/include/rdma/rdma_cm.h
+++ b/include/rdma/rdma_cm.h
@@ -167,6 +167,7 @@ struct rdma_cm_id *rdma_create_user_id(rdma_cm_event_handler event_handler,
* asynchronous operations associated with the id.
*/
void rdma_destroy_id(struct rdma_cm_id *id);
+void rdma_destroy_id_wq(struct rdma_cm_id *id);
/**
* rdma_restrict_node_type - Restrict an RDMA identifier to specific
diff --git a/net/rds/ib.c b/net/rds/ib.c
index 4ea9838d0..1290747cd 100644
--- a/net/rds/ib.c
+++ b/net/rds/ib.c
@@ -484,7 +484,7 @@ static int rds_ib_laddr_check_cm(struct net *net, const struct in6_addr *addr,
cm_id->device ? cm_id->device->node_type : -1);
out:
- rdma_destroy_id(cm_id);
+ rdma_destroy_id_wq(cm_id);
return ret;
}
--
2.50.1.windows.1
^ permalink raw reply related [flat|nested] 10+ messages in thread* Re: [PATCH] RDMA/cma: add rdma_destroy_id_wq() and use it from RDS 2026-11-14 15:05 [PATCH] RDMA/cma: add rdma_destroy_id_wq() and use it from RDS Aibing Chen @ 2026-11-15 0:18 ` Jason Gunthorpe 2026-11-16 6:47 ` Allison Henderson 2026-11-15 15:05 ` sashiko-bot 2026-11-16 7:54 ` [PATCH net v2] net/rds: defer CM id destruction to a workqueue Aibing Chen 2 siblings, 1 reply; 10+ messages in thread From: Jason Gunthorpe @ 2026-11-15 0:18 UTC (permalink / raw) To: Aibing Chen Cc: leon, achender, davem, edumazet, kuba, pabeni, horms, linux-rdma, netdev, rds-devel, linux-kernel, hdanton, hackerowskiy, syzbot+ccdb5058c3df2d94811c On Sat, Nov 14, 2026 at 11:05:41PM +0800, Aibing Chen wrote: > syzbot reported a possible circular locking dependency between > rds_trans_sem and the RDMA CM handler mutex: > > CPU0 (bind path) CPU1 (CM callback path) > ---- ---- > down_read(&rds_trans_sem) mutex_lock(&handler_mutex) > rds_ib_laddr_check() cma_ib_req_handler() > rdma_destroy_id() rds_ib_cm_handle_connect() > mutex_lock(&handler_mutex) rds_conn_create() > rds_trans_get_preferred() > down_read(&rds_trans_sem) > > rds_ib_laddr_check_cm() destroys the CM id from inside > rds_trans_get_preferred()'s read-side critical section, while CM > callbacks create RDS connections (taking the same read lock) from > under handler_mutex. > > Break the cycle by destroying the CM id asynchronously: add > rdma_destroy_id_wq(), which queues the destruction on cma_wq, and > switch rds_ib_laddr_check_cm() to it. The destroy work is embedded > in struct rdma_id_private, which keeps the id alive until the work > runs; rds does not touch cm_id after the destroy call. This doesn't seem like it should be the core codes problem? Why can't rds do the workqueue? Jason ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH] RDMA/cma: add rdma_destroy_id_wq() and use it from RDS 2026-11-15 0:18 ` Jason Gunthorpe @ 2026-11-16 6:47 ` Allison Henderson 0 siblings, 0 replies; 10+ messages in thread From: Allison Henderson @ 2026-11-16 6:47 UTC (permalink / raw) To: Jason Gunthorpe, Aibing Chen Cc: leon, davem, edumazet, kuba, pabeni, horms, linux-rdma, netdev, rds-devel, linux-kernel, hdanton, hackerowskiy, syzbot+ccdb5058c3df2d94811c On Sat, 2026-11-14 at 20:18 -0400, Jason Gunthorpe wrote: > On Sat, Nov 14, 2026 at 11:05:41PM +0800, Aibing Chen wrote: > > syzbot reported a possible circular locking dependency between > > rds_trans_sem and the RDMA CM handler mutex: > > > > CPU0 (bind path) CPU1 (CM callback path) > > ---- ---- > > down_read(&rds_trans_sem) mutex_lock(&handler_mutex) > > rds_ib_laddr_check() cma_ib_req_handler() > > rdma_destroy_id() rds_ib_cm_handle_connect() > > mutex_lock(&handler_mutex) rds_conn_create() > > rds_trans_get_preferred() > > down_read(&rds_trans_sem) > > > > rds_ib_laddr_check_cm() destroys the CM id from inside > > rds_trans_get_preferred()'s read-side critical section, while CM > > callbacks create RDS connections (taking the same read lock) from > > under handler_mutex. > > > > Break the cycle by destroying the CM id asynchronously: add > > rdma_destroy_id_wq(), which queues the destruction on cma_wq, and > > switch rds_ib_laddr_check_cm() to it. The destroy work is embedded > > in struct rdma_id_private, which keeps the id alive until the work > > runs; rds does not touch cm_id after the destroy call. > > This doesn't seem like it should be the core codes problem? Why can't > rds do the workqueue? > > Jason Hi Aibing, Aibing, thanks for chasing this down. I think Jason is right, this should probably be in the rds side. This solution doesnt quite cover all the locking issues though. rds_conn_create calls rds_trans_get_preferred() which falls through to rds_ib_laddr_check_cm() when not local. And then we get a rdma_destroy_id while the listen id's handler_mutex is still held. I think a better way to do this might be to create a new work item and then deffer the rdma_destroy_id to be completed later by a worker. The work item would just hold the cm_id queued on rds_wq, and then the handler just calls rdma_destroy_id() and frees itself. alloc the new work item at the top of rds_ib_laddr_check_cm and then queue the new work item in place of the rdma_destroy_id. You'll also need to add another flush_workqueue(rds_wq) after rds_trans_unregister() in rds_ib_exit(). Other nits: this patch should have a fixes tag, and target the net tree like this: "[PATCH net v2] net/rds ...". And I think the fixes tag is: ec16227e1414 ("RDS/IB: Infiniband transport"). Drop the tested-by until it is tested by syzbot and/or Eduard, and be sure to update your change log below the dashed line. Thanks for working on this! Allion ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH] RDMA/cma: add rdma_destroy_id_wq() and use it from RDS 2026-11-14 15:05 [PATCH] RDMA/cma: add rdma_destroy_id_wq() and use it from RDS Aibing Chen 2026-11-15 0:18 ` Jason Gunthorpe @ 2026-11-15 15:05 ` sashiko-bot 2026-11-16 7:54 ` [PATCH net v2] net/rds: defer CM id destruction to a workqueue Aibing Chen 2 siblings, 0 replies; 10+ messages in thread From: sashiko-bot @ 2026-11-15 15:05 UTC (permalink / raw) To: Aibing Chen; +Cc: linux-rdma Thank you for your contribution! Sashiko AI review found 5 potential issues to consider: - [High] RDMA/cma: deadlock on ordered cma_wq in destroy_id_work() - [Medium] net/rds: UAF on rds_rdma unload with pending rdma_destroy_id_wq() - [Medium] RDMA/cma: cma_cleanup() tears down cma_pernet before draining cma_wq - [Low] RDMA/cma: missing Fixes trailer on RDS lockdep bug fix - [Low] RDMA/cma: missing documentation for exported rdma_destroy_id_wq() API -- commit 8d1d784abda4c543f5197cf89fc2119bf91e1d62 Author: Aibing Chen <chenaibing@cmss.chinamobile.com> > RDMA/cma: add rdma_destroy_id_wq() and use it from RDS > > syzbot reported a possible circular locking dependency between > rds_trans_sem and the RDMA CM handler mutex: > > CPU0 (bind path) CPU1 (CM callback path) > ---- ---- > down_read(&rds_trans_sem) mutex_lock(&handler_mutex) > rds_ib_laddr_check() cma_ib_req_handler() > rdma_destroy_id() rds_ib_cm_handle_connect() > mutex_lock(&handler_mutex) rds_conn_create() > rds_trans_get_preferred() > down_read(&rds_trans_sem) > > rds_ib_laddr_check_cm() destroys the CM id from inside > rds_trans_get_preferred()'s read-side critical section, while CM > callbacks create RDS connections (taking the same read lock) from > under handler_mutex. > > Break the cycle by destroying the CM id asynchronously: add > rdma_destroy_id_wq(), which queues the destruction on cma_wq, and > switch rds_ib_laddr_check_cm() to it. The destroy work is embedded > in struct rdma_id_private, which keeps the id alive until the work > runs; rds does not touch cm_id after the destroy call. > > This formalizes the approach proposed and syzbot-tested by Hillf > Danton; Eduard Zateev additionally verified the fix on a real QEMU > system (KASAN + lockdep + RDMA_RXE, active and passive RDS > connections established without any lockdep output). > > Reported-by: syzbot+ccdb5058c3df2d94811c@syzkaller.appspotmail.com > Closes: https://syzkaller.appspot.com/bug?extid=ccdb5058c3df2d94811c > Suggested-by: Hillf Danton <hdanton@sina.com> > Tested-by: syzbot+ccdb5058c3df2d94811c@syzkaller.appspotmail.com > Tested-by: Eduard Zateev <hackerowskiy@gmail.com> > Signed-off-by: Aibing Chen <chenaibing@cmss.chinamobile.com> [Severity: Low] Since this fixes a lockdep cycle in the RDS/IB transport, should it include: Fixes: ec16227e1414 ("RDS/IB: Infiniband transport") > diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c > index 337a49d1acf72..d10aaf51ee86b 100644 > --- a/drivers/infiniband/core/cma.c > +++ b/drivers/infiniband/core/cma.c > @@ -2126,6 +2126,25 @@ void rdma_destroy_id(struct rdma_cm_id *id) > } > EXPORT_SYMBOL(rdma_destroy_id); > > +static void destroy_id_work(struct work_struct *work) > +{ > + struct rdma_id_private *id_priv = > + container_of(work, struct rdma_id_private, destroy_work); > + > + mutex_lock(&id_priv->handler_mutex); > + destroy_id_handler_unlock(id_priv); > +} > + > +void rdma_destroy_id_wq(struct rdma_cm_id *id) > +{ > + struct rdma_id_private *id_priv = > + container_of(id, struct rdma_id_private, id); > + > + INIT_WORK(&id_priv->destroy_work, destroy_id_work); > + queue_work(cma_wq, &id_priv->destroy_work); > +} > +EXPORT_SYMBOL(rdma_destroy_id_wq); [Severity: High] Can destroy_id_work() deadlock the ordered cma_wq workqueue when another event queues work for the same id_priv behind destroy_work? cma_init() creates cma_wq with alloc_ordered_workqueue(), so only one work item runs at a time in FIFO order. After rdma_bind_addr() attaches id_priv to cma_dev->id_list, rdma_destroy_id_wq() queues destroy_work without removing id_priv from cma_dev->id_list or marking it RDMA_CM_DESTROYING: CPU0 CPU1 cma_wq rds_ib_laddr_check_cm() rdma_bind_addr(cm_id, ...) rdma_destroy_id_wq(cm_id) queue_work(cma_wq, &destroy_work) cma_netdev_change() cma_id_get(id_priv) queue_work(cma_wq, &work->work) destroy_id_work() ... _destroy_id() cma_id_put() wait_for_completion() <-- hangs forever Because cma_work_handler() is queued behind destroy_id_work() on the same ordered workqueue, it can never run to call cma_id_put(id_priv) and complete id_priv->comp, wedging cma_wq system-wide. [Severity: Medium] Can queuing _destroy_id() on cma_wq race with cma_cleanup() during module unload and access freed cma_pernet memory? In cma_cleanup(), unregister_pernet_subsys(&cma_pernet_operations) runs before destroy_workqueue(cma_wq): drivers/infiniband/core/cma.c:cma_cleanup() { ... ib_unregister_client(&cma_client); unregister_pernet_subsys(&cma_pernet_operations); destroy_workqueue(cma_wq); ... } If rdma_bind_addr() bound a wildcard address (leaving id_priv->cma_dev NULL while allocating a bind_list entry in pernet->tcp_ps), or if _destroy_id() finishes cma_release_dev(id_priv) just before cma_release_port(id_priv), ib_unregister_client() will not wait for destroy_id_work() to finish. cma_exit_net() can then hit WARN_ON(!xa_empty(&pernet->tcp_ps)) and free the pernet structure before destroy_id_work() calls cma_release_port() -> cma_ps_remove(), causing a use-after-free. [ ... ] > diff --git a/include/rdma/rdma_cm.h b/include/rdma/rdma_cm.h > index d639ff889e649..ac1c1d9177e2e 100644 > --- a/include/rdma/rdma_cm.h > +++ b/include/rdma/rdma_cm.h > @@ -167,6 +167,7 @@ struct rdma_cm_id *rdma_create_user_id(rdma_cm_event_handler event_handler, > * asynchronous operations associated with the id. > */ > void rdma_destroy_id(struct rdma_cm_id *id); > +void rdma_destroy_id_wq(struct rdma_cm_id *id); [Severity: Low] This isn't a bug, but rdma_destroy_id_wq() is a newly exported API placed right under the kernel-doc comment for rdma_destroy_id() without documentation of its own. Unlike rdma_destroy_id(), it returns before in-flight callbacks and operations associated with the id are canceled or drained, so documenting its asynchronous lifetime and callback constraints would help callers use it safely. > diff --git a/net/rds/ib.c b/net/rds/ib.c > index 4ea9838d090c3..1290747cdd991 100644 > --- a/net/rds/ib.c > +++ b/net/rds/ib.c > @@ -484,7 +484,7 @@ static int rds_ib_laddr_check_cm(struct net *net, const struct in6_addr *addr, > cm_id->device ? cm_id->device->node_type : -1); > > out: > - rdma_destroy_id(cm_id); > + rdma_destroy_id_wq(cm_id); > > return ret; > } [Severity: Medium] Can unloading rds_rdma while destroy_id_work() is still pending on cma_wq cause a use-after-free in rds_rdma's code and rodata? rds_ib_laddr_check_cm() creates cm_id with event_handler set to rds_rdma_cm_event_handler and restrack kern_name set to KBUILD_MODNAME. Previously, rdma_destroy_id() waited for cm_id teardown inside rds_trans_get_preferred() under rds_trans_sem, which serialized against rds_trans_unregister() in rds_ib_exit(). With rdma_destroy_id_wq(), rds_ib_laddr_check_cm() returns while cm_id is still attached to cma_dev->id_list and restrack, and rds_ib_exit() does not drain cma_wq: CPU0 CPU1 rds_trans_get_preferred() rds_ib_laddr_check_cm() rdma_create_id(..., rds_rdma_cm_event_handler, ...) rdma_bind_addr(cm_id, sa) rdma_destroy_id_wq(cm_id) up_read(&rds_trans_sem) rds_ib_exit() ... rds_trans_unregister() /* rds_rdma module freed */ cma_send_device_removal_put() id_priv->id.event_handler() <-- calls freed module .text A concurrent netlink RDMA_NLDEV_CMD_RES_CM_ID_GET dump in fill_res_name_pid() can likewise read res->kern_name from freed module rodata before destroy_id_work() runs. -- Sashiko AI review · https://sashiko.dev/#/patchset/20261114150541.1908-1-chenaibing@cmss.chinamobile.com?part=1 ^ permalink raw reply [flat|nested] 10+ messages in thread
* [PATCH net v2] net/rds: defer CM id destruction to a workqueue 2026-11-14 15:05 [PATCH] RDMA/cma: add rdma_destroy_id_wq() and use it from RDS Aibing Chen 2026-11-15 0:18 ` Jason Gunthorpe 2026-11-15 15:05 ` sashiko-bot @ 2026-11-16 7:54 ` Aibing Chen 2026-11-17 6:15 ` Allison Henderson ` (3 more replies) 2 siblings, 4 replies; 10+ messages in thread From: Aibing Chen @ 2026-11-16 7:54 UTC (permalink / raw) To: Allison Henderson Cc: David S . Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni, Simon Horman, netdev, linux-rdma, rds-devel, linux-kernel rds_ib_laddr_check_cm() destroys the RDMA CM id it created while rds_trans_get_preferred() holds rds_trans_sem for reading. The bind path then takes rds_trans_sem and afterwards handler_mutex in rdma_destroy_id(), while the CM callback path takes handler_mutex in cma_ib_req_handler() and afterwards rds_trans_sem through rds_conn_create(). lockdep reports the resulting cycle. Queue the destruction on rds_wq instead, so that rds_trans_sem is released before handler_mutex is taken. The work item allocated here owns the cm_id until rdma_destroy_id() has run; if the allocation fails, the id is destroyed synchronously as before. rds_ib_exit() flushes rds_wq after rds_trans_unregister() so that no queued destroy runs after the module text is gone. Reported-by: syzbot+ccdb5058c3df2d94811c@syzkaller.appspotmail.com Closes: https://syzkaller.appspot.com/bug?extid=ccdb5058c3df2d94811c Suggested-by: Hillf Danton <hdanton@sina.com> Suggested-by: Jason Gunthorpe <jgg@ziepe.ca> Suggested-by: Allison Henderson <achender@kernel.org> Fixes: ec16227e1414 ("RDS/IB: Infiniband transport") Cc: stable@vger.kernel.org Signed-off-by: Aibing Chen <chenaibing@cmss.chinamobile.com> --- v2: - move the workqueue from RDMA core into RDS, as suggested in review. - flush rds_wq in rds_ib_exit() so a queued destroy cannot run after the module text is freed. - add the Fixes tag, drop the Tested-by tags. net/rds/ib.c | 37 ++++++++++++++++++++++++++++++++++++- 1 file changed, 36 insertions(+), 1 deletion(-) diff --git a/net/rds/ib.c b/net/rds/ib.c index 4ea9838d0..66af4a72a 100644 --- a/net/rds/ib.c +++ b/net/rds/ib.c @@ -391,6 +391,20 @@ static void rds6_ib_ic_info(struct socket *sock, unsigned int len, } #endif +struct rds_ib_destroy_work { + struct work_struct work; + struct rdma_cm_id *cm_id; +}; + +static void rds_ib_destroy_id_work(struct work_struct *work) +{ + struct rds_ib_destroy_work *w = + container_of(work, struct rds_ib_destroy_work, work); + + rdma_destroy_id(w->cm_id); + kfree(w); +} + /* * Early RDS/IB was built to only bind to an address if there is an IPoIB * device with that address set. @@ -411,6 +425,7 @@ static int rds_ib_laddr_check_cm(struct net *net, const struct in6_addr *addr, #endif struct sockaddr_in sin; struct sockaddr *sa; + struct rds_ib_destroy_work *w; bool isv4; isv4 = ipv6_addr_v4mapped(addr); @@ -419,6 +434,15 @@ static int rds_ib_laddr_check_cm(struct net *net, const struct in6_addr *addr, NULL, RDMA_PS_TCP, IB_QPT_RC); if (IS_ERR(cm_id)) return PTR_ERR(cm_id); + + /* + * The id is destroyed from rds_wq: rds_trans_get_preferred() calls this + * function with rds_trans_sem held for reading, and rdma_destroy_id() + * takes handler_mutex, which the CM callback path acquires before + * taking rds_trans_sem. If the allocation fails, fall back to the + * synchronous destroy. + */ + w = kzalloc_obj(*w, GFP_KERNEL); ret = rdma_restrict_node_type(cm_id, RDMA_NODE_IB_CA); if (ret) goto out; @@ -484,7 +508,13 @@ static int rds_ib_laddr_check_cm(struct net *net, const struct in6_addr *addr, cm_id->device ? cm_id->device->node_type : -1); out: - rdma_destroy_id(cm_id); + if (w) { + INIT_WORK(&w->work, rds_ib_destroy_id_work); + w->cm_id = cm_id; + queue_work(rds_wq, &w->work); + } else { + rdma_destroy_id(cm_id); + } return ret; } @@ -542,6 +572,11 @@ void rds_ib_exit(void) rds_ib_sysctl_exit(); rds_ib_recv_exit(); rds_trans_unregister(&rds_ib_transport); + /* + * rds_ib_laddr_check_cm() may have queued rdma_destroy_id() on rds_wq; + * wait for it before the module text goes away. + */ + flush_workqueue(rds_wq); rds_ib_mr_exit(); } ^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: [PATCH net v2] net/rds: defer CM id destruction to a workqueue 2026-11-16 7:54 ` [PATCH net v2] net/rds: defer CM id destruction to a workqueue Aibing Chen @ 2026-11-17 6:15 ` Allison Henderson 2026-11-17 7:54 ` sashiko-bot ` (2 subsequent siblings) 3 siblings, 0 replies; 10+ messages in thread From: Allison Henderson @ 2026-11-17 6:15 UTC (permalink / raw) To: Aibing Chen Cc: David S . Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni, Simon Horman, netdev, linux-rdma, rds-devel, linux-kernel On Mon, 2026-11-16 at 15:54 +0800, Aibing Chen wrote: > rds_ib_laddr_check_cm() destroys the RDMA CM id it created while > rds_trans_get_preferred() holds rds_trans_sem for reading. The bind > path then takes rds_trans_sem and afterwards handler_mutex in > rdma_destroy_id(), while the CM callback path takes handler_mutex in > cma_ib_req_handler() and afterwards rds_trans_sem through > rds_conn_create(). lockdep reports the resulting cycle. > > Queue the destruction on rds_wq instead, so that rds_trans_sem is > released before handler_mutex is taken. The work item allocated here > owns the cm_id until rdma_destroy_id() has run; if the allocation fails, > the id is destroyed synchronously as before. > > rds_ib_exit() flushes rds_wq after rds_trans_unregister() so that no > queued destroy runs after the module text is gone. > > Reported-by: syzbot+ccdb5058c3df2d94811c@syzkaller.appspotmail.com > Closes: https://syzkaller.appspot.com/bug?extid=ccdb5058c3df2d94811c > Suggested-by: Hillf Danton <hdanton@sina.com> > Suggested-by: Jason Gunthorpe <jgg@ziepe.ca> > Suggested-by: Allison Henderson <achender@kernel.org> > Fixes: ec16227e1414 ("RDS/IB: Infiniband transport") > Cc: stable@vger.kernel.org > Signed-off-by: Aibing Chen <chenaibing@cmss.chinamobile.com> > --- > v2: > - move the workqueue from RDMA core into RDS, as suggested in review. > - flush rds_wq in rds_ib_exit() so a queued destroy cannot run after > the module text is freed. > - add the Fixes tag, drop the Tested-by tags. > > net/rds/ib.c | 37 ++++++++++++++++++++++++++++++++++++- > 1 file changed, 36 insertions(+), 1 deletion(-) > > diff --git a/net/rds/ib.c b/net/rds/ib.c > index 4ea9838d0..66af4a72a 100644 > --- a/net/rds/ib.c > +++ b/net/rds/ib.c > @@ -391,6 +391,20 @@ static void rds6_ib_ic_info(struct socket *sock, unsigned int len, > } > #endif > > +struct rds_ib_destroy_work { > + struct work_struct work; > + struct rdma_cm_id *cm_id; > +}; > + > +static void rds_ib_destroy_id_work(struct work_struct *work) > +{ > + struct rds_ib_destroy_work *w = > + container_of(work, struct rds_ib_destroy_work, work); > + > + rdma_destroy_id(w->cm_id); > + kfree(w); > +} > + > /* > * Early RDS/IB was built to only bind to an address if there is an IPoIB > * device with that address set. > @@ -411,6 +425,7 @@ static int rds_ib_laddr_check_cm(struct net *net, const struct in6_addr *addr, > #endif > struct sockaddr_in sin; > struct sockaddr *sa; > + struct rds_ib_destroy_work *w; > bool isv4; > Hi Aibing, Thanks for the quick turn around, this is looking much better. Just a few things to get this through: we cant fall back to the synchronous destroy if alloc fails, or the same bug will come back under memory pressure. Lets hoist the alloc up here before the rdma_create_id(). If alloc fails, just return -ENOMEM right away. > isv4 = ipv6_addr_v4mapped(addr); > @@ -419,6 +434,15 @@ static int rds_ib_laddr_check_cm(struct net *net, const struct in6_addr *addr, > NULL, RDMA_PS_TCP, IB_QPT_RC); > if (IS_ERR(cm_id)) Then add a kfree(w) here before returning on this error case > return PTR_ERR(cm_id); > + > + /* > + * The id is destroyed from rds_wq: rds_trans_get_preferred() calls this > + * function with rds_trans_sem held for reading, and rdma_destroy_id() > + * takes handler_mutex, which the CM callback path acquires before > + * taking rds_trans_sem. If the allocation fails, fall back to the > + * synchronous destroy. > + */ > + w = kzalloc_obj(*w, GFP_KERNEL); > ret = rdma_restrict_node_type(cm_id, RDMA_NODE_IB_CA); > if (ret) > goto out; > @@ -484,7 +508,13 @@ static int rds_ib_laddr_check_cm(struct net *net, const struct in6_addr *addr, > cm_id->device ? cm_id->device->node_type : -1); > > out: > - rdma_destroy_id(cm_id); > + if (w) { > + INIT_WORK(&w->work, rds_ib_destroy_id_work); > + w->cm_id = cm_id; > + queue_work(rds_wq, &w->work); > + } else { > + rdma_destroy_id(cm_id); > + } And then this else part can be removed since w is guaranteed here Other than that, this looking really good. Thanks for working on this! Allison > > return ret; > } > @@ -542,6 +572,11 @@ void rds_ib_exit(void) > rds_ib_sysctl_exit(); > rds_ib_recv_exit(); > rds_trans_unregister(&rds_ib_transport); > + /* > + * rds_ib_laddr_check_cm() may have queued rdma_destroy_id() on rds_wq; > + * wait for it before the module text goes away. > + */ > + flush_workqueue(rds_wq); > rds_ib_mr_exit(); > } > > > > ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH net v2] net/rds: defer CM id destruction to a workqueue 2026-11-16 7:54 ` [PATCH net v2] net/rds: defer CM id destruction to a workqueue Aibing Chen 2026-11-17 6:15 ` Allison Henderson @ 2026-11-17 7:54 ` sashiko-bot 2026-11-17 8:21 ` [PATCH net v3] " Aibing Chen 2026-11-17 8:35 ` [PATCH net v2] " netdev-bot+sashiko 3 siblings, 0 replies; 10+ messages in thread From: sashiko-bot @ 2026-11-17 7:54 UTC (permalink / raw) To: Aibing Chen; +Cc: linux-rdma Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [Medium] net/rds: synchronous rdma_destroy_id() fallback in rds_ib_laddr_check_cm() -- commit 26cfa811c05ca14cedb9fc00446b35dbd44dc87e Author: Aibing Chen <chenaibing@cmss.chinamobile.com> net/rds: defer CM id destruction to a workqueue This commit defers rdma_destroy_id() from rds_ib_laddr_check_cm() to rds_wq to avoid taking handler_mutex while rds_trans_sem is held, and flushes rds_wq in rds_ib_exit(). > diff --git a/net/rds/ib.c b/net/rds/ib.c > index 4ea9838d090c3..66af4a72a5595 100644 > --- a/net/rds/ib.c > +++ b/net/rds/ib.c [ ... ] > @@ -419,6 +434,15 @@ static int rds_ib_laddr_check_cm(struct net *net, const struct in6_addr *addr, > NULL, RDMA_PS_TCP, IB_QPT_RC); > if (IS_ERR(cm_id)) > return PTR_ERR(cm_id); > + > + /* > + * The id is destroyed from rds_wq: rds_trans_get_preferred() calls this > + * function with rds_trans_sem held for reading, and rdma_destroy_id() > + * takes handler_mutex, which the CM callback path acquires before > + * taking rds_trans_sem. If the allocation fails, fall back to the > + * synchronous destroy. > + */ > + w = kzalloc_obj(*w, GFP_KERNEL); > ret = rdma_restrict_node_type(cm_id, RDMA_NODE_IB_CA); > if (ret) > goto out; > @@ -484,7 +508,13 @@ static int rds_ib_laddr_check_cm(struct net *net, const struct in6_addr *addr, > cm_id->device ? cm_id->device->node_type : -1); > > out: > - rdma_destroy_id(cm_id); > + if (w) { > + INIT_WORK(&w->work, rds_ib_destroy_id_work); > + w->cm_id = cm_id; > + queue_work(rds_wq, &w->work); > + } else { > + rdma_destroy_id(cm_id); > + } > > return ret; > } [Severity: Medium] Can the fallback to synchronous rdma_destroy_id(cm_id) when kzalloc_obj(*w, GFP_KERNEL) fails still trigger the rds_trans_sem -> handler_mutex ABBA deadlock this patch fixes? Allocating w before rdma_create_id() (returning -ENOMEM if !w and freeing w if IS_ERR(cm_id)) would ensure cm_id is never created without a work item to destroy it. -- Sashiko AI review · https://sashiko.dev/#/patchset/20261116075423.782-1-chenaibing@cmss.chinamobile.com?part=1 ^ permalink raw reply [flat|nested] 10+ messages in thread
* [PATCH net v3] net/rds: defer CM id destruction to a workqueue 2026-11-16 7:54 ` [PATCH net v2] net/rds: defer CM id destruction to a workqueue Aibing Chen 2026-11-17 6:15 ` Allison Henderson 2026-11-17 7:54 ` sashiko-bot @ 2026-11-17 8:21 ` Aibing Chen 2026-11-17 8:33 ` sashiko-bot 2026-11-17 8:35 ` [PATCH net v2] " netdev-bot+sashiko 3 siblings, 1 reply; 10+ messages in thread From: Aibing Chen @ 2026-11-17 8:21 UTC (permalink / raw) To: Allison Henderson Cc: David S . Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni, Simon Horman, netdev, linux-rdma, rds-devel, linux-kernel rds_ib_laddr_check_cm() destroys the RDMA CM id it created while rds_trans_get_preferred() holds rds_trans_sem for reading. The bind path then takes rds_trans_sem and afterwards handler_mutex in rdma_destroy_id(), while the CM callback path takes handler_mutex in cma_ib_req_handler() and afterwards rds_trans_sem through rds_conn_create(). lockdep reports the resulting cycle. Queue the destruction on rds_wq instead, so that rds_trans_sem is released before handler_mutex is taken. The work item is allocated before the CM id is created and owns the id until rdma_destroy_id() has run; if the allocation fails, the address check fails with -ENOMEM rather than falling back to the synchronous destroy, which would bring the deadlock back under memory pressure. rds_ib_exit() flushes rds_wq after rds_trans_unregister() so that no queued destroy runs after the module text is gone. Reported-by: syzbot+ccdb5058c3df2d94811c@syzkaller.appspotmail.com Closes: https://syzkaller.appspot.com/bug?extid=ccdb5058c3df2d94811c Suggested-by: Hillf Danton <hdanton@sina.com> Suggested-by: Jason Gunthorpe <jgg@ziepe.ca> Suggested-by: Allison Henderson <achender@kernel.org> Fixes: ec16227e1414 ("RDS/IB: Infiniband transport") Cc: stable@vger.kernel.org Signed-off-by: Aibing Chen <chenaibing@cmss.chinamobile.com> --- v3: - allocate the work item before rdma_create_id() and fail with -ENOMEM instead of falling back to the synchronous destroy, which would bring the deadlock back under memory pressure. - free the work item when rdma_create_id() fails and drop the now redundant fallback in the out path. v2: - move the workqueue from RDMA core into RDS, as suggested in review. - flush rds_wq in rds_ib_exit() so a queued destroy cannot run after the module text is freed. - add the Fixes tag, drop the Tested-by tags. net/rds/ib.c | 39 +++++++++++++++++++++++++++++++++++++-- 1 file changed, 37 insertions(+), 2 deletions(-) diff --git a/net/rds/ib.c b/net/rds/ib.c index 4ea9838d0..44b46defa 100644 --- a/net/rds/ib.c +++ b/net/rds/ib.c @@ -391,6 +391,20 @@ static void rds6_ib_ic_info(struct socket *sock, unsigned int len, } #endif +struct rds_ib_destroy_work { + struct work_struct work; + struct rdma_cm_id *cm_id; +}; + +static void rds_ib_destroy_id_work(struct work_struct *work) +{ + struct rds_ib_destroy_work *w = + container_of(work, struct rds_ib_destroy_work, work); + + rdma_destroy_id(w->cm_id); + kfree(w); +} + /* * Early RDS/IB was built to only bind to an address if there is an IPoIB * device with that address set. @@ -411,14 +425,28 @@ static int rds_ib_laddr_check_cm(struct net *net, const struct in6_addr *addr, #endif struct sockaddr_in sin; struct sockaddr *sa; + struct rds_ib_destroy_work *w; bool isv4; isv4 = ipv6_addr_v4mapped(addr); + + /* + * The id is destroyed from rds_wq: rds_trans_get_preferred() calls this + * function with rds_trans_sem held for reading, and rdma_destroy_id() + * takes handler_mutex, which the CM callback path acquires before + * taking rds_trans_sem. + */ + w = kzalloc_obj(*w, GFP_KERNEL); + if (!w) + return -ENOMEM; + /* Create a CMA ID restricted to IB devices and try to bind it. */ cm_id = rdma_create_id(&init_net, rds_rdma_cm_event_handler, NULL, RDMA_PS_TCP, IB_QPT_RC); - if (IS_ERR(cm_id)) + if (IS_ERR(cm_id)) { + kfree(w); return PTR_ERR(cm_id); + } ret = rdma_restrict_node_type(cm_id, RDMA_NODE_IB_CA); if (ret) goto out; @@ -484,7 +512,9 @@ static int rds_ib_laddr_check_cm(struct net *net, const struct in6_addr *addr, cm_id->device ? cm_id->device->node_type : -1); out: - rdma_destroy_id(cm_id); + INIT_WORK(&w->work, rds_ib_destroy_id_work); + w->cm_id = cm_id; + queue_work(rds_wq, &w->work); return ret; } @@ -542,6 +572,11 @@ void rds_ib_exit(void) rds_ib_sysctl_exit(); rds_ib_recv_exit(); rds_trans_unregister(&rds_ib_transport); + /* + * rds_ib_laddr_check_cm() may have queued rdma_destroy_id() on rds_wq; + * wait for it before the module text goes away. + */ + flush_workqueue(rds_wq); rds_ib_mr_exit(); } ^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: [PATCH net v3] net/rds: defer CM id destruction to a workqueue 2026-11-17 8:21 ` [PATCH net v3] " Aibing Chen @ 2026-11-17 8:33 ` sashiko-bot 0 siblings, 0 replies; 10+ messages in thread From: sashiko-bot @ 2026-11-17 8:33 UTC (permalink / raw) To: Aibing Chen; +Cc: linux-rdma > rds_ib_laddr_check_cm() destroys the RDMA CM id it created while > rds_trans_get_preferred() holds rds_trans_sem for reading. The bind > path then takes rds_trans_sem and afterwards handler_mutex in > rdma_destroy_id(), while the CM callback path takes handler_mutex in > cma_ib_req_handler() and afterwards rds_trans_sem through > rds_conn_create(). lockdep reports the resulting cycle. > > Queue the destruction on rds_wq instead, so that rds_trans_sem is > released before handler_mutex is taken. The work item is allocated > before the CM id is created and owns the id until rdma_destroy_id() has > run; if the allocation fails, the address check fails with -ENOMEM > rather than falling back to the synchronous destroy, which would bring > the deadlock back under memory pressure. > > rds_ib_exit() flushes rds_wq after rds_trans_unregister() so that no > queued destroy runs after the module text is gone. > > Reported-by: syzbot+ccdb5058c3df2d94811c@syzkaller.appspotmail.com > Closes: https://syzkaller.appspot.com/bug?extid=ccdb5058c3df2d94811c > Suggested-by: Hillf Danton <hdanton@sina.com> > Suggested-by: Jason Gunthorpe <jgg@ziepe.ca> > Suggested-by: Allison Henderson <achender@kernel.org> > Fixes: ec16227e1414 ("RDS/IB: Infiniband transport") > Cc: stable@vger.kernel.org > Signed-off-by: Aibing Chen <chenaibing@cmss.chinamobile.com> Sashiko has reviewed this patch and found no issues. -- Sashiko AI review · https://sashiko.dev/#/patchset/20261117082145.1737-1-chenaibing@cmss.chinamobile.com?part=1 ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH net v2] net/rds: defer CM id destruction to a workqueue 2026-11-16 7:54 ` [PATCH net v2] net/rds: defer CM id destruction to a workqueue Aibing Chen ` (2 preceding siblings ...) 2026-11-17 8:21 ` [PATCH net v3] " Aibing Chen @ 2026-11-17 8:35 ` netdev-bot+sashiko 3 siblings, 0 replies; 10+ messages in thread From: netdev-bot+sashiko @ 2026-11-17 8:35 UTC (permalink / raw) To: chenaibing Cc: achender, davem, edumazet, kuba, pabeni, horms, netdev, linux-rdma, rds-devel, linux-kernel Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider. Critical: 0 · High: 1 · Medium: 1 · Low: 0 If you believe none of the issues are legitimate, please reply to each issue of Medium severity or higher and explain why the AI is wrong. If any of them are legitimate and you plan to respin, please let patchwork know by including "pw-bot: cr" as a separate line at the end of your reply (one such reply per series is enough). - [High] The lock cycle this patch is meant to fix can still happen. - [Medium] After this patch, each probe RDMA CM id created by rds_ib_laddr_check_cm() stays alive after bind() returns, until… -- net/rds: defer CM id destruction to a workqueue The patch moves rdma_destroy_id() on the probe cm_id in rds_ib_laddr_check_cm() onto rds_wq. The goal is to break the lockdep cycle between rds_trans_sem and handler_mutex. If the work item cannot be allocated, the id is still destroyed synchronously, and rds_ib_exit() now flushes rds_wq. Closes: https://syzkaller.appspot.com/bug?extid=ccdb5058c3df2d94811c > diff --git a/net/rds/ib.c b/net/rds/ib.c > index 4ea9838d090c3..66af4a72a5595 100644 > --- a/net/rds/ib.c > +++ b/net/rds/ib.c [ ... ] > @@ -419,6 +434,15 @@ static int rds_ib_laddr_check_cm(struct net *net, const struct in6_addr *addr, > NULL, RDMA_PS_TCP, IB_QPT_RC); > if (IS_ERR(cm_id)) > return PTR_ERR(cm_id); > + > + /* > + * The id is destroyed from rds_wq: rds_trans_get_preferred() calls this > + * function with rds_trans_sem held for reading, and rdma_destroy_id() > + * takes handler_mutex, which the CM callback path acquires before > + * taking rds_trans_sem. If the allocation fails, fall back to the > + * synchronous destroy. > + */ > + w = kzalloc_obj(*w, GFP_KERNEL); [Severity: High] If kzalloc_obj() fails here, the out: label calls rdma_destroy_id() directly. Doesn't that still run inside the down_read(&rds_trans_sem) section of rds_trans_get_preferred()? rds_bind() rds_trans_get_preferred() down_read(&rds_trans_sem) rds_ib_laddr_check() rds_ib_laddr_check_cm() kzalloc_obj() fails rdma_destroy_id() mutex_lock(&id_priv->handler_mutex) That is the same rds_trans_sem -> handler_mutex dependency that lockdep reported. The other half of the cycle is handler_mutex -> rds_trans_sem, taken via cma_ib_req_handler() -> rds_conn_create(). If a writer such as rds_trans_register() or rds_trans_unregister() is queued on the rwsem, this can also hang. The commit message says: if the allocation fails, the id is destroyed synchronously as before. "As before" is the behaviour that produced the original report. The allocation failure is also easy to trigger with failslab, which is what syzbot uses. Could the work item be allocated before rdma_create_id(), with -ENOMEM returned if that fails? rds_trans_get_preferred() already treats any non-zero laddr_check() result as "not this transport". rds_bind() already maps that to -EADDRNOTAVAIL or -ENOPROTOOPT, so no new error handling should be needed. > ret = rdma_restrict_node_type(cm_id, RDMA_NODE_IB_CA); > if (ret) > goto out; > @@ -484,7 +508,13 @@ static int rds_ib_laddr_check_cm(struct net *net, const struct in6_addr *addr, > cm_id->device ? cm_id->device->node_type : -1); > > out: > - rdma_destroy_id(cm_id); > + if (w) { > + INIT_WORK(&w->work, rds_ib_destroy_id_work); > + w->cm_id = cm_id; > + queue_work(rds_wq, &w->work); [Severity: Medium] With the destroy deferred, each probe cm_id stays alive after bind() returns, until rds_ib_destroy_id_work() runs. What limits how many of these can be pending? rds_threads_init() creates rds_wq with create_singlethread_workqueue("krdsd"). That makes it an ordered queue with one worker, shared with the connection, shutdown and free work for all of RDS. An unprivileged user can call bind() on AF_RDS sockets in a loop. The call reaches laddr_check() either through rs_transport (set via SO_RDS_TRANSPORT) or through rds_trans_get_preferred(). A failed bind leaves the socket unbound. Each attempt with an IPv6 address, or with an IPv4 address not yet in rds_ibdev->ipaddr_list, queues another item. Producers on several CPUs feed one consumer. That consumer also contends on the global cma lock mutex and sits behind any slow RDS work. Each pending item holds the rdma_id_private and the struct rds_ib_destroy_work. The latter is allocated with GFP_KERNEL and no __GFP_ACCOUNT, so it is not charged to the caller's memcg. If rdma_bind_addr_dst() got far enough, the item also holds two more things: - a cma_dev reference from cma_acquire_dev_by_src_ip() - an ephemeral port in the shared RDMA_PS_TCP space from cma_get_port() Before this patch, all of these were released before the syscall returned. Could a backlog here pin memory, use up ephemeral RDMA_PS_TCP ports needed by other in-kernel users (NVMe-oF/RDMA, iSER, NFS/RDMA), and delay IB device removal? cma_process_remove() waits here until every pending destroy has run: wait_for_completion(&cma_dev->comp); > + } else { > + rdma_destroy_id(cm_id); > + } > > return ret; > } [ ... ] -- Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20261116075423.782-1-chenaibing%40cmss.chinamobile.com ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2026-11-17 8:35 UTC | newest] Thread overview: 10+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2026-11-14 15:05 [PATCH] RDMA/cma: add rdma_destroy_id_wq() and use it from RDS Aibing Chen 2026-11-15 0:18 ` Jason Gunthorpe 2026-11-16 6:47 ` Allison Henderson 2026-11-15 15:05 ` sashiko-bot 2026-11-16 7:54 ` [PATCH net v2] net/rds: defer CM id destruction to a workqueue Aibing Chen 2026-11-17 6:15 ` Allison Henderson 2026-11-17 7:54 ` sashiko-bot 2026-11-17 8:21 ` [PATCH net v3] " Aibing Chen 2026-11-17 8:33 ` sashiko-bot 2026-11-17 8:35 ` [PATCH net v2] " netdev-bot+sashiko
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox