From: Slawomir Stepien <sst@poczta.fm>
To: Jeff Layton <jlayton@kernel.org>
Cc: syzbot <syzbot@kernel.org>,
syzkaller-bugs@googlegroups.com, Chuck Lever <cel@kernel.org>,
linux-nfs@vger.kernel.org, Lorenzo Bianconi <lorenzo@kernel.org>,
Dai.Ngo@oracle.com, linux-kernel@vger.kernel.org,
neil@brown.name, okorniev@redhat.com, syzbot@lists.linux.dev,
tom@talpey.com
Subject: Re: [PATCH] nfsd: prevent hung task in nfsd_nl_listener_set_doit()
Date: Sun, 23 Aug 2026 22:27:56 +0200 [thread overview]
Message-ID: <aotXzLmrd-B89YRg@nr200> (raw)
In-Reply-To: <699bacb503758fbe2ebf01f3e8aca95e3eee4128.camel@kernel.org>
On sie 20, 2026 07:37, Jeff Layton wrote:
> On Thu, 2026-08-20 at 06:55 +0000, syzbot wrote:
> > From: Slawomir Stepien <sst@poczta.fm>
> >
> > In nfsd_nl_listener_set_doit(), the kernel iterates over all
> > NFSD_A_SERVER_SOCK_ADDR attributes provided in a netlink message to
> > configure NFS server listeners. There is currently no limit on the number
> > of attributes a user can send.
> >
> > For each attribute, svc_xprt_create_from_sa() is called, which may
> > synchronously invoke request_module() to load the corresponding transport
> > module. If a user provides a large number of invalid transport names,
> > request_module() is called sequentially for each, taking a massive amount
> > of time. Since this entire process occurs while holding the global
> > nfsd_mutex, it blocks other tasks attempting to acquire the mutex and
> > triggers a hung task timeout:
> >
>
> My LLM latched onto the same explanation, but the request module
> upcalls are actually quite quick and have some other gates on them that
> make this problem hard to hit in practice.
Oh, OK. Thanks for clearing that up!
> The real problem is rpcbind registration, which can be quite slow. The
> fix for that is to make the registration asynchronous, and to not
> trigger errors on rpcbind registration failure. That's a bigger
> project, but let us know if you're interested!
I think I'm not that good to do it correctly and in some reasonable time. I might take a 2nd look at
this and understand the root cause (also, based on your comments in your series) but I think in the
end it might be too hard for me.
> In any case, I've sent a series that includes a similar fix to this
> one, along with some others:
>
> https://lore.kernel.org/linux-nfs/20260811-nfsd-nl-hang-v2-0-c0c92b3953c3@kernel.org/
>
> I'll try to get the follow-on series out later today or tomorrow.
Nice series, I've learnt something from it! I will be looking for v3!
> > INFO: task blocked for more than 10 seconds.
> > ...
> > Call Trace:
> > <TASK>
> > __schedule+0x17e7/0x5630 kernel/sched/core.c:7234
> > schedule+0x164/0x2b0 kernel/sched/core.c:7326
> > schedule_preempt_disabled+0x13/0x30 kernel/sched/core.c:7383
> > __mutex_lock_common kernel/locking/mutex.c:726 [inline]
> > __mutex_lock+0x7bf/0x1550 kernel/locking/mutex.c:821
> > nfsd_nl_version_get_doit+0x17c/0xd20 fs/nfsd/nfsctl.c:1889
> > genl_family_rcv_msg_doit+0x233/0x340 net/netlink/genetlink.c:1114
> > genl_family_rcv_msg net/netlink/genetlink.c:1194 [inline]
> > genl_rcv_msg+0x614/0x7a0 net/netlink/genetlink.c:1209
> > ...
> > 2 locks held by task/5864:
> > #0: ffffffff90114168 (cb_lock){++++}-{4:4}, at: genl_rcv+0x19/0x40
> > net/netlink/genetlink.c:1217
> > #1: ffffffff8eeb1a60 (nfsd_mutex){+.+.}-{4:4}, at:
> > nfsd_nl_listener_set_doit+0x135/0x1750 fs/nfsd/nfsctl.c:1964
> >
> > Furthermore, the function does not break out of the loop if
> > svc_xprt_create_from_sa() fails, and it suffers from an O(N^2) complexity
> > issue because svc_find_listener() iterates over the serv->sv_permsocks list
> > for each attribute.
> >
> > Address this by introducing a hard limit of 128 NFSD_A_SERVER_SOCK_ADDR
> > attributes per netlink message. The limit is enforced before acquiring the
> > nfsd_mutex to prevent lock contention. Additionally, modify the loop to
> > break immediately if listener creation fails, avoiding needless sequential
> > request_module() calls for invalid configurations.
> >
> > Fixes: 16a471177496 ("NFSD: add listener-{set,get} netlink command")
> > Assisted-by: Gemini:gemini-3.6-flash Gemini:gemini-3.1-pro-preview syzbot
> > Reported-by: syzbot+41bc60511c2884783c27@syzkaller.appspotmail.com
> > Closes: https://syzkaller.appspot.com/bug?extid=41bc60511c2884783c27
> > Link: https://syzkaller.appspot.com/ai_job?id=30871f87-7a83-46b8-ab83-919894b0787b
> > Signed-off-by: Slawomir Stepien <sst@poczta.fm>
> >
> > ---
> > diff --git a/fs/nfsd/nfsctl.c b/fs/nfsd/nfsctl.c
> > index fa92e31d1..516c04c14 100644
> > --- a/fs/nfsd/nfsctl.c
> > +++ b/fs/nfsd/nfsctl.c
> > @@ -1943,6 +1943,8 @@ int nfsd_nl_version_get_doit(struct sk_buff *skb, struct genl_info *info)
> > return err;
> > }
> >
> > +#define NFSD_MAX_LISTENERS 128
> > +
> > /**
> > * nfsd_nl_listener_set_doit - set the nfs running sockets
> > * @skb: reply buffer
> > @@ -1955,12 +1957,19 @@ int nfsd_nl_listener_set_doit(struct sk_buff *skb, struct genl_info *info)
> > struct net *net = genl_info_net(info);
> > struct svc_xprt *xprt, *tmp;
> > const struct nlattr *attr;
> > + unsigned int count = 0;
> > struct svc_serv *serv;
> > LIST_HEAD(permsocks);
> > struct nfsd_net *nn;
> > bool delete = false;
> > int err, rem;
> >
> > + nlmsg_for_each_attr_type(attr, NFSD_A_SERVER_SOCK_ADDR, info->nlhdr,
> > + GENL_HDRLEN, rem) {
> > + if (++count > NFSD_MAX_LISTENERS)
> > + return -EINVAL;
> > + }
> > +
> > mutex_lock(&nfsd_mutex);
> >
> > err = nfsd_create_serv(net);
> > @@ -2073,8 +2082,10 @@ int nfsd_nl_listener_set_doit(struct sk_buff *skb, struct genl_info *info)
> > ret = svc_xprt_create_from_sa(serv, xcl_name, net, sa, 0,
> > current_cred());
> > /* always save the latest error */
> > - if (ret < 0)
> > + if (ret < 0) {
> > err = ret;
> > + break;
> > + }
> > }
> >
> > if (!serv->sv_nrthreads && list_empty(&nn->nfsd_serv->sv_permsocks))
> >
> >
> > base-commit: 075b74841bd0065a3bda3440873c747938e69b68
>
> I have a patch series in progress for this already, which also
--
Slawomir Stepien
next prev parent reply other threads:[~2026-08-23 20:29 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-08-20 6:55 [PATCH] nfsd: prevent hung task in nfsd_nl_listener_set_doit() syzbot
2026-08-20 11:37 ` Jeff Layton
2026-08-23 20:27 ` Slawomir Stepien [this message]
2026-08-24 12:00 ` Jeff Layton
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=aotXzLmrd-B89YRg@nr200 \
--to=sst@poczta.fm \
--cc=Dai.Ngo@oracle.com \
--cc=cel@kernel.org \
--cc=jlayton@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-nfs@vger.kernel.org \
--cc=lorenzo@kernel.org \
--cc=neil@brown.name \
--cc=okorniev@redhat.com \
--cc=syzbot@kernel.org \
--cc=syzbot@lists.linux.dev \
--cc=syzkaller-bugs@googlegroups.com \
--cc=tom@talpey.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox