Netdev List
 help / color / mirror / Atom feed
* [PATCH] RDMA/cma: add rdma_destroy_id_wq() and use it from RDS
@ 2026-11-14 15:05 Aibing Chen
  2026-11-15  0:18 ` Jason Gunthorpe
  2026-11-16  7:54 ` [PATCH net v2] net/rds: defer CM id destruction to a workqueue Aibing Chen
  0 siblings, 2 replies; 7+ messages in thread
From: Aibing Chen @ 2026-11-14 15:05 UTC (permalink / raw)
  To: jgg, leon, achender
  Cc: davem, edumazet, kuba, pabeni, horms, linux-rdma, netdev,
	rds-devel, linux-kernel, hdanton, hackerowskiy,
	syzbot+ccdb5058c3df2d94811c, Aibing Chen

syzbot reported a possible circular locking dependency between
rds_trans_sem and the RDMA CM handler mutex:

  CPU0 (bind path)                    CPU1 (CM callback path)
  ----                                ----
  down_read(&rds_trans_sem)           mutex_lock(&handler_mutex)
  rds_ib_laddr_check()                cma_ib_req_handler()
    rdma_destroy_id()                   rds_ib_cm_handle_connect()
      mutex_lock(&handler_mutex)          rds_conn_create()
                                            rds_trans_get_preferred()
                                              down_read(&rds_trans_sem)

rds_ib_laddr_check_cm() destroys the CM id from inside
rds_trans_get_preferred()'s read-side critical section, while CM
callbacks create RDS connections (taking the same read lock) from
under handler_mutex.

Break the cycle by destroying the CM id asynchronously: add
rdma_destroy_id_wq(), which queues the destruction on cma_wq, and
switch rds_ib_laddr_check_cm() to it.  The destroy work is embedded
in struct rdma_id_private, which keeps the id alive until the work
runs; rds does not touch cm_id after the destroy call.

This formalizes the approach proposed and syzbot-tested by Hillf
Danton; Eduard Zateev additionally verified the fix on a real QEMU
system (KASAN + lockdep + RDMA_RXE, active and passive RDS
connections established without any lockdep output).

Reported-by: syzbot+ccdb5058c3df2d94811c@syzkaller.appspotmail.com
Closes: https://syzkaller.appspot.com/bug?extid=ccdb5058c3df2d94811c
Suggested-by: Hillf Danton <hdanton@sina.com>
Tested-by: syzbot+ccdb5058c3df2d94811c@syzkaller.appspotmail.com
Tested-by: Eduard Zateev <hackerowskiy@gmail.com>
Signed-off-by: Aibing Chen <chenaibing@cmss.chinamobile.com>
---
 drivers/infiniband/core/cma.c      | 19 +++++++++++++++++++
 drivers/infiniband/core/cma_priv.h |  1 +
 include/rdma/rdma_cm.h             |  1 +
 net/rds/ib.c                       |  2 +-
 4 files changed, 22 insertions(+), 1 deletion(-)

diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c
index 337a49d1a..d10aaf51e 100644
--- a/drivers/infiniband/core/cma.c
+++ b/drivers/infiniband/core/cma.c
@@ -2126,6 +2126,25 @@ void rdma_destroy_id(struct rdma_cm_id *id)
 }
 EXPORT_SYMBOL(rdma_destroy_id);
 
+static void destroy_id_work(struct work_struct *work)
+{
+	struct rdma_id_private *id_priv =
+		container_of(work, struct rdma_id_private, destroy_work);
+
+	mutex_lock(&id_priv->handler_mutex);
+	destroy_id_handler_unlock(id_priv);
+}
+
+void rdma_destroy_id_wq(struct rdma_cm_id *id)
+{
+	struct rdma_id_private *id_priv =
+		container_of(id, struct rdma_id_private, id);
+
+	INIT_WORK(&id_priv->destroy_work, destroy_id_work);
+	queue_work(cma_wq, &id_priv->destroy_work);
+}
+EXPORT_SYMBOL(rdma_destroy_id_wq);
+
 static int cma_rep_recv(struct rdma_id_private *id_priv)
 {
 	int ret;
diff --git a/drivers/infiniband/core/cma_priv.h b/drivers/infiniband/core/cma_priv.h
index 04332eb82..95ff8bbd4 100644
--- a/drivers/infiniband/core/cma_priv.h
+++ b/drivers/infiniband/core/cma_priv.h
@@ -110,6 +110,7 @@ struct rdma_id_private {
 	 */
 	struct rdma_restrack_entry     res;
 	struct rdma_ucm_ece ece;
+	struct work_struct destroy_work;
 };
 
 #if IS_ENABLED(CONFIG_INFINIBAND_ADDR_TRANS_CONFIGFS)
diff --git a/include/rdma/rdma_cm.h b/include/rdma/rdma_cm.h
index d639ff889..ac1c1d917 100644
--- a/include/rdma/rdma_cm.h
+++ b/include/rdma/rdma_cm.h
@@ -167,6 +167,7 @@ struct rdma_cm_id *rdma_create_user_id(rdma_cm_event_handler event_handler,
   * asynchronous operations associated with the id.
   */
 void rdma_destroy_id(struct rdma_cm_id *id);
+void rdma_destroy_id_wq(struct rdma_cm_id *id);
 
 /**
  * rdma_restrict_node_type - Restrict an RDMA identifier to specific
diff --git a/net/rds/ib.c b/net/rds/ib.c
index 4ea9838d0..1290747cd 100644
--- a/net/rds/ib.c
+++ b/net/rds/ib.c
@@ -484,7 +484,7 @@ static int rds_ib_laddr_check_cm(struct net *net, const struct in6_addr *addr,
 		 cm_id->device ? cm_id->device->node_type : -1);
 
 out:
-	rdma_destroy_id(cm_id);
+	rdma_destroy_id_wq(cm_id);
 
 	return ret;
 }
-- 
2.50.1.windows.1




^ permalink raw reply related	[flat|nested] 7+ messages in thread

* Re: [PATCH] RDMA/cma: add rdma_destroy_id_wq() and use it from RDS
  2026-11-14 15:05 [PATCH] RDMA/cma: add rdma_destroy_id_wq() and use it from RDS Aibing Chen
@ 2026-11-15  0:18 ` Jason Gunthorpe
  2026-11-16  6:47   ` Allison Henderson
  2026-11-16  7:54 ` [PATCH net v2] net/rds: defer CM id destruction to a workqueue Aibing Chen
  1 sibling, 1 reply; 7+ messages in thread
From: Jason Gunthorpe @ 2026-11-15  0:18 UTC (permalink / raw)
  To: Aibing Chen
  Cc: leon, achender, davem, edumazet, kuba, pabeni, horms, linux-rdma,
	netdev, rds-devel, linux-kernel, hdanton, hackerowskiy,
	syzbot+ccdb5058c3df2d94811c

On Sat, Nov 14, 2026 at 11:05:41PM +0800, Aibing Chen wrote:
> syzbot reported a possible circular locking dependency between
> rds_trans_sem and the RDMA CM handler mutex:
> 
>   CPU0 (bind path)                    CPU1 (CM callback path)
>   ----                                ----
>   down_read(&rds_trans_sem)           mutex_lock(&handler_mutex)
>   rds_ib_laddr_check()                cma_ib_req_handler()
>     rdma_destroy_id()                   rds_ib_cm_handle_connect()
>       mutex_lock(&handler_mutex)          rds_conn_create()
>                                             rds_trans_get_preferred()
>                                               down_read(&rds_trans_sem)
> 
> rds_ib_laddr_check_cm() destroys the CM id from inside
> rds_trans_get_preferred()'s read-side critical section, while CM
> callbacks create RDS connections (taking the same read lock) from
> under handler_mutex.
> 
> Break the cycle by destroying the CM id asynchronously: add
> rdma_destroy_id_wq(), which queues the destruction on cma_wq, and
> switch rds_ib_laddr_check_cm() to it.  The destroy work is embedded
> in struct rdma_id_private, which keeps the id alive until the work
> runs; rds does not touch cm_id after the destroy call.

This doesn't seem like it should be the core codes problem? Why can't
rds do the workqueue?

Jason

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [PATCH] RDMA/cma: add rdma_destroy_id_wq() and use it from RDS
  2026-11-15  0:18 ` Jason Gunthorpe
@ 2026-11-16  6:47   ` Allison Henderson
  0 siblings, 0 replies; 7+ messages in thread
From: Allison Henderson @ 2026-11-16  6:47 UTC (permalink / raw)
  To: Jason Gunthorpe, Aibing Chen
  Cc: leon, davem, edumazet, kuba, pabeni, horms, linux-rdma, netdev,
	rds-devel, linux-kernel, hdanton, hackerowskiy,
	syzbot+ccdb5058c3df2d94811c

On Sat, 2026-11-14 at 20:18 -0400, Jason Gunthorpe wrote:
> On Sat, Nov 14, 2026 at 11:05:41PM +0800, Aibing Chen wrote:
> > syzbot reported a possible circular locking dependency between
> > rds_trans_sem and the RDMA CM handler mutex:
> > 
> >   CPU0 (bind path)                    CPU1 (CM callback path)
> >   ----                                ----
> >   down_read(&rds_trans_sem)           mutex_lock(&handler_mutex)
> >   rds_ib_laddr_check()                cma_ib_req_handler()
> >     rdma_destroy_id()                   rds_ib_cm_handle_connect()
> >       mutex_lock(&handler_mutex)          rds_conn_create()
> >                                             rds_trans_get_preferred()
> >                                               down_read(&rds_trans_sem)
> > 
> > rds_ib_laddr_check_cm() destroys the CM id from inside
> > rds_trans_get_preferred()'s read-side critical section, while CM
> > callbacks create RDS connections (taking the same read lock) from
> > under handler_mutex.
> > 
> > Break the cycle by destroying the CM id asynchronously: add
> > rdma_destroy_id_wq(), which queues the destruction on cma_wq, and
> > switch rds_ib_laddr_check_cm() to it.  The destroy work is embedded
> > in struct rdma_id_private, which keeps the id alive until the work
> > runs; rds does not touch cm_id after the destroy call.
> 
> This doesn't seem like it should be the core codes problem? Why can't
> rds do the workqueue?
> 
> Jason

Hi Aibing,

Aibing, thanks for chasing this down.  I think Jason is right, this
should probably be in the rds side.  This solution doesnt quite cover
all the locking issues though.  rds_conn_create calls
rds_trans_get_preferred() which falls through to rds_ib_laddr_check_cm()
when not local.  And then we get a rdma_destroy_id while the listen id's
handler_mutex is still held.

I think a better way to do this might be to create a new work item and
then deffer the rdma_destroy_id to be completed later by a worker.  The
work item would just hold the cm_id queued on rds_wq, and then the handler
just calls rdma_destroy_id() and frees itself.  alloc the new work item at
the top of rds_ib_laddr_check_cm and then queue the new work item in place of
the rdma_destroy_id.  You'll also need to add another flush_workqueue(rds_wq)
after rds_trans_unregister() in rds_ib_exit().

Other nits:  this patch should have a fixes tag, and target the net tree
like this: "[PATCH net v2] net/rds ...".  And I think the fixes tag is:
ec16227e1414 ("RDS/IB: Infiniband transport").  Drop the tested-by until it
is tested by syzbot and/or Eduard, and be sure to update your change log
below the dashed line.

Thanks for working on this!
Allion

^ permalink raw reply	[flat|nested] 7+ messages in thread

* [PATCH net v2] net/rds: defer CM id destruction to a workqueue
  2026-11-14 15:05 [PATCH] RDMA/cma: add rdma_destroy_id_wq() and use it from RDS Aibing Chen
  2026-11-15  0:18 ` Jason Gunthorpe
@ 2026-11-16  7:54 ` Aibing Chen
  2026-11-17  6:15   ` Allison Henderson
                     ` (2 more replies)
  1 sibling, 3 replies; 7+ messages in thread
From: Aibing Chen @ 2026-11-16  7:54 UTC (permalink / raw)
  To: Allison Henderson
  Cc: David S . Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
	Simon Horman, netdev, linux-rdma, rds-devel, linux-kernel

rds_ib_laddr_check_cm() destroys the RDMA CM id it created while
rds_trans_get_preferred() holds rds_trans_sem for reading.  The bind
path then takes rds_trans_sem and afterwards handler_mutex in
rdma_destroy_id(), while the CM callback path takes handler_mutex in
cma_ib_req_handler() and afterwards rds_trans_sem through
rds_conn_create().  lockdep reports the resulting cycle.

Queue the destruction on rds_wq instead, so that rds_trans_sem is
released before handler_mutex is taken.  The work item allocated here
owns the cm_id until rdma_destroy_id() has run; if the allocation fails,
the id is destroyed synchronously as before.

rds_ib_exit() flushes rds_wq after rds_trans_unregister() so that no
queued destroy runs after the module text is gone.

Reported-by: syzbot+ccdb5058c3df2d94811c@syzkaller.appspotmail.com
Closes: https://syzkaller.appspot.com/bug?extid=ccdb5058c3df2d94811c
Suggested-by: Hillf Danton <hdanton@sina.com>
Suggested-by: Jason Gunthorpe <jgg@ziepe.ca>
Suggested-by: Allison Henderson <achender@kernel.org>
Fixes: ec16227e1414 ("RDS/IB: Infiniband transport")
Cc: stable@vger.kernel.org
Signed-off-by: Aibing Chen <chenaibing@cmss.chinamobile.com>
---
v2:
- move the workqueue from RDMA core into RDS, as suggested in review.
- flush rds_wq in rds_ib_exit() so a queued destroy cannot run after
  the module text is freed.
- add the Fixes tag, drop the Tested-by tags.

 net/rds/ib.c | 37 ++++++++++++++++++++++++++++++++++++-
 1 file changed, 36 insertions(+), 1 deletion(-)

diff --git a/net/rds/ib.c b/net/rds/ib.c
index 4ea9838d0..66af4a72a 100644
--- a/net/rds/ib.c
+++ b/net/rds/ib.c
@@ -391,6 +391,20 @@ static void rds6_ib_ic_info(struct socket *sock, unsigned int len,
 }
 #endif
 
+struct rds_ib_destroy_work {
+	struct work_struct work;
+	struct rdma_cm_id *cm_id;
+};
+
+static void rds_ib_destroy_id_work(struct work_struct *work)
+{
+	struct rds_ib_destroy_work *w =
+		container_of(work, struct rds_ib_destroy_work, work);
+
+	rdma_destroy_id(w->cm_id);
+	kfree(w);
+}
+
 /*
  * Early RDS/IB was built to only bind to an address if there is an IPoIB
  * device with that address set.
@@ -411,6 +425,7 @@ static int rds_ib_laddr_check_cm(struct net *net, const struct in6_addr *addr,
 #endif
 	struct sockaddr_in sin;
 	struct sockaddr *sa;
+	struct rds_ib_destroy_work *w;
 	bool isv4;
 
 	isv4 = ipv6_addr_v4mapped(addr);
@@ -419,6 +434,15 @@ static int rds_ib_laddr_check_cm(struct net *net, const struct in6_addr *addr,
 			       NULL, RDMA_PS_TCP, IB_QPT_RC);
 	if (IS_ERR(cm_id))
 		return PTR_ERR(cm_id);
+
+	/*
+	 * The id is destroyed from rds_wq: rds_trans_get_preferred() calls this
+	 * function with rds_trans_sem held for reading, and rdma_destroy_id()
+	 * takes handler_mutex, which the CM callback path acquires before
+	 * taking rds_trans_sem.  If the allocation fails, fall back to the
+	 * synchronous destroy.
+	 */
+	w = kzalloc_obj(*w, GFP_KERNEL);
 	ret = rdma_restrict_node_type(cm_id, RDMA_NODE_IB_CA);
 	if (ret)
 		goto out;
@@ -484,7 +508,13 @@ static int rds_ib_laddr_check_cm(struct net *net, const struct in6_addr *addr,
 		 cm_id->device ? cm_id->device->node_type : -1);
 
 out:
-	rdma_destroy_id(cm_id);
+	if (w) {
+		INIT_WORK(&w->work, rds_ib_destroy_id_work);
+		w->cm_id = cm_id;
+		queue_work(rds_wq, &w->work);
+	} else {
+		rdma_destroy_id(cm_id);
+	}
 
 	return ret;
 }
@@ -542,6 +572,11 @@ void rds_ib_exit(void)
 	rds_ib_sysctl_exit();
 	rds_ib_recv_exit();
 	rds_trans_unregister(&rds_ib_transport);
+	/*
+	 * rds_ib_laddr_check_cm() may have queued rdma_destroy_id() on rds_wq;
+	 * wait for it before the module text goes away.
+	 */
+	flush_workqueue(rds_wq);
 	rds_ib_mr_exit();
 }
 



^ permalink raw reply related	[flat|nested] 7+ messages in thread

* Re: [PATCH net v2] net/rds: defer CM id destruction to a workqueue
  2026-11-16  7:54 ` [PATCH net v2] net/rds: defer CM id destruction to a workqueue Aibing Chen
@ 2026-11-17  6:15   ` Allison Henderson
  2026-11-17  8:21   ` [PATCH net v3] " Aibing Chen
  2026-11-17  8:35   ` [PATCH net v2] " netdev-bot+sashiko
  2 siblings, 0 replies; 7+ messages in thread
From: Allison Henderson @ 2026-11-17  6:15 UTC (permalink / raw)
  To: Aibing Chen
  Cc: David S . Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
	Simon Horman, netdev, linux-rdma, rds-devel, linux-kernel

On Mon, 2026-11-16 at 15:54 +0800, Aibing Chen wrote:
> rds_ib_laddr_check_cm() destroys the RDMA CM id it created while
> rds_trans_get_preferred() holds rds_trans_sem for reading.  The bind
> path then takes rds_trans_sem and afterwards handler_mutex in
> rdma_destroy_id(), while the CM callback path takes handler_mutex in
> cma_ib_req_handler() and afterwards rds_trans_sem through
> rds_conn_create().  lockdep reports the resulting cycle.
> 
> Queue the destruction on rds_wq instead, so that rds_trans_sem is
> released before handler_mutex is taken.  The work item allocated here
> owns the cm_id until rdma_destroy_id() has run; if the allocation fails,
> the id is destroyed synchronously as before.
> 
> rds_ib_exit() flushes rds_wq after rds_trans_unregister() so that no
> queued destroy runs after the module text is gone.
> 
> Reported-by: syzbot+ccdb5058c3df2d94811c@syzkaller.appspotmail.com
> Closes: https://syzkaller.appspot.com/bug?extid=ccdb5058c3df2d94811c
> Suggested-by: Hillf Danton <hdanton@sina.com>
> Suggested-by: Jason Gunthorpe <jgg@ziepe.ca>
> Suggested-by: Allison Henderson <achender@kernel.org>
> Fixes: ec16227e1414 ("RDS/IB: Infiniband transport")
> Cc: stable@vger.kernel.org
> Signed-off-by: Aibing Chen <chenaibing@cmss.chinamobile.com>
> ---
> v2:
> - move the workqueue from RDMA core into RDS, as suggested in review.
> - flush rds_wq in rds_ib_exit() so a queued destroy cannot run after
>   the module text is freed.
> - add the Fixes tag, drop the Tested-by tags.
> 
>  net/rds/ib.c | 37 ++++++++++++++++++++++++++++++++++++-
>  1 file changed, 36 insertions(+), 1 deletion(-)
> 
> diff --git a/net/rds/ib.c b/net/rds/ib.c
> index 4ea9838d0..66af4a72a 100644
> --- a/net/rds/ib.c
> +++ b/net/rds/ib.c
> @@ -391,6 +391,20 @@ static void rds6_ib_ic_info(struct socket *sock, unsigned int len,
>  }
>  #endif
>  
> +struct rds_ib_destroy_work {
> +	struct work_struct work;
> +	struct rdma_cm_id *cm_id;
> +};
> +
> +static void rds_ib_destroy_id_work(struct work_struct *work)
> +{
> +	struct rds_ib_destroy_work *w =
> +		container_of(work, struct rds_ib_destroy_work, work);
> +
> +	rdma_destroy_id(w->cm_id);
> +	kfree(w);
> +}
> +
>  /*
>   * Early RDS/IB was built to only bind to an address if there is an IPoIB
>   * device with that address set.
> @@ -411,6 +425,7 @@ static int rds_ib_laddr_check_cm(struct net *net, const struct in6_addr *addr,
>  #endif
>  	struct sockaddr_in sin;
>  	struct sockaddr *sa;
> +	struct rds_ib_destroy_work *w;
>  	bool isv4;
>  
Hi Aibing,

Thanks for the quick turn around, this is looking much better.  Just a few things to get this
through: we cant fall back to the synchronous destroy if alloc fails, or the same bug
will come back under memory pressure.  Lets hoist the alloc up here before the
rdma_create_id().  If alloc fails, just return -ENOMEM right away.

>  	isv4 = ipv6_addr_v4mapped(addr);
> @@ -419,6 +434,15 @@ static int rds_ib_laddr_check_cm(struct net *net, const struct in6_addr *addr,
>  			       NULL, RDMA_PS_TCP, IB_QPT_RC);
>  	if (IS_ERR(cm_id))
Then add a kfree(w) here before returning on this error case

>  		return PTR_ERR(cm_id);
> +
> +	/*
> +	 * The id is destroyed from rds_wq: rds_trans_get_preferred() calls this
> +	 * function with rds_trans_sem held for reading, and rdma_destroy_id()
> +	 * takes handler_mutex, which the CM callback path acquires before
> +	 * taking rds_trans_sem.  If the allocation fails, fall back to the
> +	 * synchronous destroy.
> +	 */
> +	w = kzalloc_obj(*w, GFP_KERNEL);
>  	ret = rdma_restrict_node_type(cm_id, RDMA_NODE_IB_CA);
>  	if (ret)
>  		goto out;
> @@ -484,7 +508,13 @@ static int rds_ib_laddr_check_cm(struct net *net, const struct in6_addr *addr,
>  		 cm_id->device ? cm_id->device->node_type : -1);
>  
>  out:
> -	rdma_destroy_id(cm_id);
> +	if (w) {
> +		INIT_WORK(&w->work, rds_ib_destroy_id_work);
> +		w->cm_id = cm_id;
> +		queue_work(rds_wq, &w->work);
> +	} else {
> +		rdma_destroy_id(cm_id);
> +	}
And then this else part can be removed since w is guaranteed here

Other than that, this looking really good.
Thanks for working on this!
Allison

>  
>  	return ret;
>  }
> @@ -542,6 +572,11 @@ void rds_ib_exit(void)
>  	rds_ib_sysctl_exit();
>  	rds_ib_recv_exit();
>  	rds_trans_unregister(&rds_ib_transport);
> +	/*
> +	 * rds_ib_laddr_check_cm() may have queued rdma_destroy_id() on rds_wq;
> +	 * wait for it before the module text goes away.
> +	 */
> +	flush_workqueue(rds_wq);
>  	rds_ib_mr_exit();
>  }
>  
> 
> 
> 


^ permalink raw reply	[flat|nested] 7+ messages in thread

* [PATCH net v3] net/rds: defer CM id destruction to a workqueue
  2026-11-16  7:54 ` [PATCH net v2] net/rds: defer CM id destruction to a workqueue Aibing Chen
  2026-11-17  6:15   ` Allison Henderson
@ 2026-11-17  8:21   ` Aibing Chen
  2026-11-17  8:35   ` [PATCH net v2] " netdev-bot+sashiko
  2 siblings, 0 replies; 7+ messages in thread
From: Aibing Chen @ 2026-11-17  8:21 UTC (permalink / raw)
  To: Allison Henderson
  Cc: David S . Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
	Simon Horman, netdev, linux-rdma, rds-devel, linux-kernel

rds_ib_laddr_check_cm() destroys the RDMA CM id it created while
rds_trans_get_preferred() holds rds_trans_sem for reading.  The bind
path then takes rds_trans_sem and afterwards handler_mutex in
rdma_destroy_id(), while the CM callback path takes handler_mutex in
cma_ib_req_handler() and afterwards rds_trans_sem through
rds_conn_create().  lockdep reports the resulting cycle.

Queue the destruction on rds_wq instead, so that rds_trans_sem is
released before handler_mutex is taken.  The work item is allocated
before the CM id is created and owns the id until rdma_destroy_id() has
run; if the allocation fails, the address check fails with -ENOMEM
rather than falling back to the synchronous destroy, which would bring
the deadlock back under memory pressure.

rds_ib_exit() flushes rds_wq after rds_trans_unregister() so that no
queued destroy runs after the module text is gone.

Reported-by: syzbot+ccdb5058c3df2d94811c@syzkaller.appspotmail.com
Closes: https://syzkaller.appspot.com/bug?extid=ccdb5058c3df2d94811c
Suggested-by: Hillf Danton <hdanton@sina.com>
Suggested-by: Jason Gunthorpe <jgg@ziepe.ca>
Suggested-by: Allison Henderson <achender@kernel.org>
Fixes: ec16227e1414 ("RDS/IB: Infiniband transport")
Cc: stable@vger.kernel.org
Signed-off-by: Aibing Chen <chenaibing@cmss.chinamobile.com>
---
v3:
- allocate the work item before rdma_create_id() and fail with -ENOMEM
  instead of falling back to the synchronous destroy, which would bring
  the deadlock back under memory pressure.
- free the work item when rdma_create_id() fails and drop the now
  redundant fallback in the out path.

v2:
- move the workqueue from RDMA core into RDS, as suggested in review.
- flush rds_wq in rds_ib_exit() so a queued destroy cannot run after
  the module text is freed.
- add the Fixes tag, drop the Tested-by tags.

 net/rds/ib.c | 39 +++++++++++++++++++++++++++++++++++++--
 1 file changed, 37 insertions(+), 2 deletions(-)

diff --git a/net/rds/ib.c b/net/rds/ib.c
index 4ea9838d0..44b46defa 100644
--- a/net/rds/ib.c
+++ b/net/rds/ib.c
@@ -391,6 +391,20 @@ static void rds6_ib_ic_info(struct socket *sock, unsigned int len,
 }
 #endif
 
+struct rds_ib_destroy_work {
+	struct work_struct work;
+	struct rdma_cm_id *cm_id;
+};
+
+static void rds_ib_destroy_id_work(struct work_struct *work)
+{
+	struct rds_ib_destroy_work *w =
+		container_of(work, struct rds_ib_destroy_work, work);
+
+	rdma_destroy_id(w->cm_id);
+	kfree(w);
+}
+
 /*
  * Early RDS/IB was built to only bind to an address if there is an IPoIB
  * device with that address set.
@@ -411,14 +425,28 @@ static int rds_ib_laddr_check_cm(struct net *net, const struct in6_addr *addr,
 #endif
 	struct sockaddr_in sin;
 	struct sockaddr *sa;
+	struct rds_ib_destroy_work *w;
 	bool isv4;
 
 	isv4 = ipv6_addr_v4mapped(addr);
+
+	/*
+	 * The id is destroyed from rds_wq: rds_trans_get_preferred() calls this
+	 * function with rds_trans_sem held for reading, and rdma_destroy_id()
+	 * takes handler_mutex, which the CM callback path acquires before
+	 * taking rds_trans_sem.
+	 */
+	w = kzalloc_obj(*w, GFP_KERNEL);
+	if (!w)
+		return -ENOMEM;
+
 	/* Create a CMA ID restricted to IB devices and try to bind it. */
 	cm_id = rdma_create_id(&init_net, rds_rdma_cm_event_handler,
 			       NULL, RDMA_PS_TCP, IB_QPT_RC);
-	if (IS_ERR(cm_id))
+	if (IS_ERR(cm_id)) {
+		kfree(w);
 		return PTR_ERR(cm_id);
+	}
 	ret = rdma_restrict_node_type(cm_id, RDMA_NODE_IB_CA);
 	if (ret)
 		goto out;
@@ -484,7 +512,9 @@ static int rds_ib_laddr_check_cm(struct net *net, const struct in6_addr *addr,
 		 cm_id->device ? cm_id->device->node_type : -1);
 
 out:
-	rdma_destroy_id(cm_id);
+	INIT_WORK(&w->work, rds_ib_destroy_id_work);
+	w->cm_id = cm_id;
+	queue_work(rds_wq, &w->work);
 
 	return ret;
 }
@@ -542,6 +572,11 @@ void rds_ib_exit(void)
 	rds_ib_sysctl_exit();
 	rds_ib_recv_exit();
 	rds_trans_unregister(&rds_ib_transport);
+	/*
+	 * rds_ib_laddr_check_cm() may have queued rdma_destroy_id() on rds_wq;
+	 * wait for it before the module text goes away.
+	 */
+	flush_workqueue(rds_wq);
 	rds_ib_mr_exit();
 }
 



^ permalink raw reply related	[flat|nested] 7+ messages in thread

* Re: [PATCH net v2] net/rds: defer CM id destruction to a workqueue
  2026-11-16  7:54 ` [PATCH net v2] net/rds: defer CM id destruction to a workqueue Aibing Chen
  2026-11-17  6:15   ` Allison Henderson
  2026-11-17  8:21   ` [PATCH net v3] " Aibing Chen
@ 2026-11-17  8:35   ` netdev-bot+sashiko
  2 siblings, 0 replies; 7+ messages in thread
From: netdev-bot+sashiko @ 2026-11-17  8:35 UTC (permalink / raw)
  To: chenaibing
  Cc: achender, davem, edumazet, kuba, pabeni, horms, netdev,
	linux-rdma, rds-devel, linux-kernel

Thank you for your contribution! Sashiko AI review found 2 potential
issue(s) to consider.

Critical: 0 · High: 1 · Medium: 1 · Low: 0

If you believe none of the issues are legitimate, please reply to each
issue of Medium severity or higher and explain why the AI is wrong.
If any of them are legitimate and you plan to respin, please let patchwork
know by including "pw-bot: cr" as a separate line at the end of your reply
(one such reply per series is enough).

- [High] The lock cycle this patch is meant to fix can still happen.
- [Medium] After this patch, each probe RDMA CM id created by
  rds_ib_laddr_check_cm() stays alive after bind() returns, until…

--

net/rds: defer CM id destruction to a workqueue

The patch moves rdma_destroy_id() on the probe cm_id in
rds_ib_laddr_check_cm() onto rds_wq. The goal is to break the lockdep
cycle between rds_trans_sem and handler_mutex. If the work item cannot
be allocated, the id is still destroyed synchronously, and rds_ib_exit()
now flushes rds_wq.

Closes: https://syzkaller.appspot.com/bug?extid=ccdb5058c3df2d94811c

> diff --git a/net/rds/ib.c b/net/rds/ib.c
> index 4ea9838d090c3..66af4a72a5595 100644
> --- a/net/rds/ib.c
> +++ b/net/rds/ib.c

[ ... ]

> @@ -419,6 +434,15 @@ static int rds_ib_laddr_check_cm(struct net *net, const struct in6_addr *addr,
>  			       NULL, RDMA_PS_TCP, IB_QPT_RC);
>  	if (IS_ERR(cm_id))
>  		return PTR_ERR(cm_id);
> +
> +	/*
> +	 * The id is destroyed from rds_wq: rds_trans_get_preferred() calls this
> +	 * function with rds_trans_sem held for reading, and rdma_destroy_id()
> +	 * takes handler_mutex, which the CM callback path acquires before
> +	 * taking rds_trans_sem.  If the allocation fails, fall back to the
> +	 * synchronous destroy.
> +	 */
> +	w = kzalloc_obj(*w, GFP_KERNEL);

[Severity: High]
If kzalloc_obj() fails here, the out: label calls rdma_destroy_id()
directly. Doesn't that still run inside the down_read(&rds_trans_sem)
section of rds_trans_get_preferred()?

rds_bind()
  rds_trans_get_preferred()
    down_read(&rds_trans_sem)
    rds_ib_laddr_check()
      rds_ib_laddr_check_cm()
        kzalloc_obj() fails
        rdma_destroy_id()
          mutex_lock(&id_priv->handler_mutex)

That is the same rds_trans_sem -> handler_mutex dependency that lockdep
reported. The other half of the cycle is handler_mutex -> rds_trans_sem,
taken via cma_ib_req_handler() -> rds_conn_create(). If a writer such as
rds_trans_register() or rds_trans_unregister() is queued on the rwsem,
this can also hang.

The commit message says:

    if the allocation fails, the id is destroyed synchronously as before.

"As before" is the behaviour that produced the original report. The
allocation failure is also easy to trigger with failslab, which is what
syzbot uses.

Could the work item be allocated before rdma_create_id(), with -ENOMEM
returned if that fails? rds_trans_get_preferred() already treats any
non-zero laddr_check() result as "not this transport". rds_bind()
already maps that to -EADDRNOTAVAIL or -ENOPROTOOPT, so no new error
handling should be needed.

>  	ret = rdma_restrict_node_type(cm_id, RDMA_NODE_IB_CA);
>  	if (ret)
>  		goto out;
> @@ -484,7 +508,13 @@ static int rds_ib_laddr_check_cm(struct net *net, const struct in6_addr *addr,
>  		 cm_id->device ? cm_id->device->node_type : -1);
>  
>  out:
> -	rdma_destroy_id(cm_id);
> +	if (w) {
> +		INIT_WORK(&w->work, rds_ib_destroy_id_work);
> +		w->cm_id = cm_id;
> +		queue_work(rds_wq, &w->work);

[Severity: Medium]
With the destroy deferred, each probe cm_id stays alive after bind()
returns, until rds_ib_destroy_id_work() runs. What limits how many of
these can be pending?

rds_threads_init() creates rds_wq with
create_singlethread_workqueue("krdsd"). That makes it an ordered queue
with one worker, shared with the connection, shutdown and free work for
all of RDS.

An unprivileged user can call bind() on AF_RDS sockets in a loop. The
call reaches laddr_check() either through rs_transport (set via
SO_RDS_TRANSPORT) or through rds_trans_get_preferred().

A failed bind leaves the socket unbound. Each attempt with an IPv6
address, or with an IPv4 address not yet in rds_ibdev->ipaddr_list,
queues another item. Producers on several CPUs feed one consumer. That
consumer also contends on the global cma lock mutex and sits behind any
slow RDS work.

Each pending item holds the rdma_id_private and the
struct rds_ib_destroy_work. The latter is allocated with GFP_KERNEL and
no __GFP_ACCOUNT, so it is not charged to the caller's memcg.

If rdma_bind_addr_dst() got far enough, the item also holds two more
things:
- a cma_dev reference from cma_acquire_dev_by_src_ip()
- an ephemeral port in the shared RDMA_PS_TCP space from cma_get_port()

Before this patch, all of these were released before the syscall
returned.

Could a backlog here pin memory, use up ephemeral RDMA_PS_TCP ports
needed by other in-kernel users (NVMe-oF/RDMA, iSER, NFS/RDMA), and
delay IB device removal? cma_process_remove() waits here until every
pending destroy has run:

	wait_for_completion(&cma_dev->comp);

> +	} else {
> +		rdma_destroy_id(cm_id);
> +	}
>  
>  	return ret;
>  }

[ ... ]

-- 
Sashiko AI review · https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20261116075423.782-1-chenaibing%40cmss.chinamobile.com

^ permalink raw reply	[flat|nested] 7+ messages in thread

end of thread, other threads:[~2026-11-17  8:35 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-14 15:05 [PATCH] RDMA/cma: add rdma_destroy_id_wq() and use it from RDS Aibing Chen
2026-11-15  0:18 ` Jason Gunthorpe
2026-11-16  6:47   ` Allison Henderson
2026-11-16  7:54 ` [PATCH net v2] net/rds: defer CM id destruction to a workqueue Aibing Chen
2026-11-17  6:15   ` Allison Henderson
2026-11-17  8:21   ` [PATCH net v3] " Aibing Chen
2026-11-17  8:35   ` [PATCH net v2] " netdev-bot+sashiko

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox