From: Allison Henderson <achender@kernel.org>
To: netdev-bot+sashiko@kernel.org, chenaibing@cmss.chinamobile.com
Cc: davem@davemloft.net, edumazet@kernel.org, kuba@kernel.org,
pabeni@redhat.com, horms@kernel.org, netdev@vger.kernel.org,
linux-rdma@vger.kernel.org, rds-devel@oss.oracle.com,
linux-kernel@vger.kernel.org
Subject: Re: [PATCH net v2] net/rds: defer CM id destruction to a workqueue
Date: Wed, 18 Nov 2026 23:33:47 -0700 [thread overview]
Message-ID: <1065d21dbab9fdcbe3a34e2f3b1deac934b4a5ce.camel@kernel.org> (raw)
In-Reply-To: <179490452051.3136.18322831263977968329@kernel.org>
On Tue, 2026-11-17 at 08:35 +0000, netdev-bot+sashiko@kernel.org wrote:
> Thank you for your contribution! Sashiko AI review found 2 potential
> issue(s) to consider.
>
> Critical: 0 · High: 1 · Medium: 1 · Low: 0
>
> If you believe none of the issues are legitimate, please reply to each
> issue of Medium severity or higher and explain why the AI is wrong.
> If any of them are legitimate and you plan to respin, please let patchwork
> know by including "pw-bot: cr" as a separate line at the end of your reply
> (one such reply per series is enough).
>
> - [High] The lock cycle this patch is meant to fix can still happen.
> - [Medium] After this patch, each probe RDMA CM id created by
> rds_ib_laddr_check_cm() stays alive after bind() returns, until…
>
> --
>
> net/rds: defer CM id destruction to a workqueue
>
> The patch moves rdma_destroy_id() on the probe cm_id in
> rds_ib_laddr_check_cm() onto rds_wq. The goal is to break the lockdep
> cycle between rds_trans_sem and handler_mutex. If the work item cannot
> be allocated, the id is still destroyed synchronously, and rds_ib_exit()
> now flushes rds_wq.
>
> Closes: https://syzkaller.appspot.com/bug?extid=ccdb5058c3df2d94811c
>
> > diff --git a/net/rds/ib.c b/net/rds/ib.c
> > index 4ea9838d090c3..66af4a72a5595 100644
> > --- a/net/rds/ib.c
> > +++ b/net/rds/ib.c
>
> [ ... ]
>
> > @@ -419,6 +434,15 @@ static int rds_ib_laddr_check_cm(struct net *net, const struct in6_addr *addr,
> > NULL, RDMA_PS_TCP, IB_QPT_RC);
> > if (IS_ERR(cm_id))
> > return PTR_ERR(cm_id);
> > +
> > + /*
> > + * The id is destroyed from rds_wq: rds_trans_get_preferred() calls this
> > + * function with rds_trans_sem held for reading, and rdma_destroy_id()
> > + * takes handler_mutex, which the CM callback path acquires before
> > + * taking rds_trans_sem. If the allocation fails, fall back to the
> > + * synchronous destroy.
> > + */
> > + w = kzalloc_obj(*w, GFP_KERNEL);
>
> [Severity: High]
> If kzalloc_obj() fails here, the out: label calls rdma_destroy_id()
> directly. Doesn't that still run inside the down_read(&rds_trans_sem)
> section of rds_trans_get_preferred()?
>
> rds_bind()
> rds_trans_get_preferred()
> down_read(&rds_trans_sem)
> rds_ib_laddr_check()
> rds_ib_laddr_check_cm()
> kzalloc_obj() fails
> rdma_destroy_id()
> mutex_lock(&id_priv->handler_mutex)
>
> That is the same rds_trans_sem -> handler_mutex dependency that lockdep
> reported. The other half of the cycle is handler_mutex -> rds_trans_sem,
> taken via cma_ib_req_handler() -> rds_conn_create(). If a writer such as
> rds_trans_register() or rds_trans_unregister() is queued on the rwsem,
> this can also hang.
>
> The commit message says:
>
> if the allocation fails, the id is destroyed synchronously as before.
>
> "As before" is the behaviour that produced the original report. The
> allocation failure is also easy to trigger with failslab, which is what
> syzbot uses.
>
> Could the work item be allocated before rdma_create_id(), with -ENOMEM
> returned if that fails? rds_trans_get_preferred() already treats any
> non-zero laddr_check() result as "not this transport". rds_bind()
> already maps that to -EADDRNOTAVAIL or -ENOPROTOOPT, so no new error
> handling should be needed.
>
> > ret = rdma_restrict_node_type(cm_id, RDMA_NODE_IB_CA);
> > if (ret)
> > goto out;
> > @@ -484,7 +508,13 @@ static int rds_ib_laddr_check_cm(struct net *net, const struct in6_addr *addr,
> > cm_id->device ? cm_id->device->node_type : -1);
> >
> > out:
> > - rdma_destroy_id(cm_id);
> > + if (w) {
> > + INIT_WORK(&w->work, rds_ib_destroy_id_work);
> > + w->cm_id = cm_id;
> > + queue_work(rds_wq, &w->work);
>
> [Severity: Medium]
> With the destroy deferred, each probe cm_id stays alive after bind()
> returns, until rds_ib_destroy_id_work() runs. What limits how many of
> these can be pending?
>
> rds_threads_init() creates rds_wq with
> create_singlethread_workqueue("krdsd"). That makes it an ordered queue
> with one worker, shared with the connection, shutdown and free work for
> all of RDS.
>
> An unprivileged user can call bind() on AF_RDS sockets in a loop. The
> call reaches laddr_check() either through rs_transport (set via
> SO_RDS_TRANSPORT) or through rds_trans_get_preferred().
>
> A failed bind leaves the socket unbound. Each attempt with an IPv6
> address, or with an IPv4 address not yet in rds_ibdev->ipaddr_list,
> queues another item. Producers on several CPUs feed one consumer. That
> consumer also contends on the global cma lock mutex and sits behind any
> slow RDS work.
>
> Each pending item holds the rdma_id_private and the
> struct rds_ib_destroy_work. The latter is allocated with GFP_KERNEL and
> no __GFP_ACCOUNT, so it is not charged to the caller's memcg.
>
> If rdma_bind_addr_dst() got far enough, the item also holds two more
> things:
> - a cma_dev reference from cma_acquire_dev_by_src_ip()
> - an ephemeral port in the shared RDMA_PS_TCP space from cma_get_port()
>
> Before this patch, all of these were released before the syscall
> returned.
>
> Could a backlog here pin memory, use up ephemeral RDMA_PS_TCP ports
> needed by other in-kernel users (NVMe-oF/RDMA, iSER, NFS/RDMA), and
> delay IB device removal? cma_process_remove() waits here until every
> pending destroy has run:
>
Hi Aibing,
Thanks for the quick response on this, v3 takes care of the fallback,
but the second sashiko finding on v2 applies to v3 as well since nothing
limits the pending probe ids there either. So likely we'll need to
bound it somehow.
I think the easiest fix would be an atomic counter that is incremented
when a probe is queued and decremented in the worker. Check it at the
top of rds_ib_laddr_check_cm(), before the allocation, and return
-EAGAIN if it's above a defined cap (say 64 pending probes). Remember
to decrement it on the -ENOMEM and rdma_create_id() error paths too.
If you'd like to take it a step further, the probes could go on their
own workqueue so they don't get stuck behind slow connection shutdowns
on rds_wq. rds_ib_exit() could then call destroy_workqueue() on it
after rds_trans_unregister() instead of flushing rds_wq.
Please also mention the port exposure and the cap in the commit
message, so the reasoning stays with the patch.
The sashiko review for v3 is still queued, so let's wait for it to
land before sending v4. That way we can address anything else it finds
in the same version:
https://patchwork.kernel.org/project/netdevbpf/patch/20261117082145.1737-1-chenaibing@cmss.chinamobile.com/
Thanks for working on this!
Allison
> wait_for_completion(&cma_dev->comp);
>
> > + } else {
> > + rdma_destroy_id(cm_id);
> > + }
> >
> > return ret;
> > }
>
> [ ... ]
>
prev parent reply other threads:[~2026-11-19 6:33 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-14 15:05 [PATCH] RDMA/cma: add rdma_destroy_id_wq() and use it from RDS Aibing Chen
2026-11-15 0:18 ` Jason Gunthorpe
2026-11-16 6:47 ` Allison Henderson
2026-11-16 7:54 ` [PATCH net v2] net/rds: defer CM id destruction to a workqueue Aibing Chen
2026-11-17 6:15 ` Allison Henderson
2026-11-17 8:21 ` [PATCH net v3] " Aibing Chen
2026-11-17 8:35 ` [PATCH net v2] " netdev-bot+sashiko
2026-11-19 6:33 ` Allison Henderson [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1065d21dbab9fdcbe3a34e2f3b1deac934b4a5ce.camel@kernel.org \
--to=achender@kernel.org \
--cc=chenaibing@cmss.chinamobile.com \
--cc=davem@davemloft.net \
--cc=edumazet@kernel.org \
--cc=horms@kernel.org \
--cc=kuba@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=netdev-bot+sashiko@kernel.org \
--cc=netdev@vger.kernel.org \
--cc=pabeni@redhat.com \
--cc=rds-devel@oss.oracle.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox