From: Jeff Layton <jlayton@kernel.org>
To: syzbot <syzbot@kernel.org>,
syzkaller-bugs@googlegroups.com, Slawomir Stepien <sst@poczta.fm>,
Chuck Lever <cel@kernel.org>,
linux-nfs@vger.kernel.org, Lorenzo Bianconi <lorenzo@kernel.org>
Cc: Dai.Ngo@oracle.com, linux-kernel@vger.kernel.org,
neil@brown.name, okorniev@redhat.com, syzbot@lists.linux.dev,
tom@talpey.com
Subject: Re: [PATCH] nfsd: prevent hung task in nfsd_nl_listener_set_doit()
Date: Thu, 20 Aug 2026 07:37:12 -0400 [thread overview]
Message-ID: <699bacb503758fbe2ebf01f3e8aca95e3eee4128.camel@kernel.org> (raw)
In-Reply-To: <4601ffd5-c662-4ab1-ab2f-50a67a0a990e@mail.kernel.org>
On Thu, 2026-08-20 at 06:55 +0000, syzbot wrote:
> From: Slawomir Stepien <sst@poczta.fm>
>
> In nfsd_nl_listener_set_doit(), the kernel iterates over all
> NFSD_A_SERVER_SOCK_ADDR attributes provided in a netlink message to
> configure NFS server listeners. There is currently no limit on the number
> of attributes a user can send.
>
> For each attribute, svc_xprt_create_from_sa() is called, which may
> synchronously invoke request_module() to load the corresponding transport
> module. If a user provides a large number of invalid transport names,
> request_module() is called sequentially for each, taking a massive amount
> of time. Since this entire process occurs while holding the global
> nfsd_mutex, it blocks other tasks attempting to acquire the mutex and
> triggers a hung task timeout:
>
My LLM latched onto the same explanation, but the request module
upcalls are actually quite quick and have some other gates on them that
make this problem hard to hit in practice.
The real problem is rpcbind registration, which can be quite slow. The
fix for that is to make the registration asynchronous, and to not
trigger errors on rpcbind registration failure. That's a bigger
project, but let us know if you're interested!
In any case, I've sent a series that includes a similar fix to this
one, along with some others:
https://lore.kernel.org/linux-nfs/20260811-nfsd-nl-hang-v2-0-c0c92b3953c3@kernel.org/
I'll try to get the follow-on series out later today or tomorrow.
> INFO: task blocked for more than 10 seconds.
> ...
> Call Trace:
> <TASK>
> __schedule+0x17e7/0x5630 kernel/sched/core.c:7234
> schedule+0x164/0x2b0 kernel/sched/core.c:7326
> schedule_preempt_disabled+0x13/0x30 kernel/sched/core.c:7383
> __mutex_lock_common kernel/locking/mutex.c:726 [inline]
> __mutex_lock+0x7bf/0x1550 kernel/locking/mutex.c:821
> nfsd_nl_version_get_doit+0x17c/0xd20 fs/nfsd/nfsctl.c:1889
> genl_family_rcv_msg_doit+0x233/0x340 net/netlink/genetlink.c:1114
> genl_family_rcv_msg net/netlink/genetlink.c:1194 [inline]
> genl_rcv_msg+0x614/0x7a0 net/netlink/genetlink.c:1209
> ...
> 2 locks held by task/5864:
> #0: ffffffff90114168 (cb_lock){++++}-{4:4}, at: genl_rcv+0x19/0x40
> net/netlink/genetlink.c:1217
> #1: ffffffff8eeb1a60 (nfsd_mutex){+.+.}-{4:4}, at:
> nfsd_nl_listener_set_doit+0x135/0x1750 fs/nfsd/nfsctl.c:1964
>
> Furthermore, the function does not break out of the loop if
> svc_xprt_create_from_sa() fails, and it suffers from an O(N^2) complexity
> issue because svc_find_listener() iterates over the serv->sv_permsocks list
> for each attribute.
>
> Address this by introducing a hard limit of 128 NFSD_A_SERVER_SOCK_ADDR
> attributes per netlink message. The limit is enforced before acquiring the
> nfsd_mutex to prevent lock contention. Additionally, modify the loop to
> break immediately if listener creation fails, avoiding needless sequential
> request_module() calls for invalid configurations.
>
> Fixes: 16a471177496 ("NFSD: add listener-{set,get} netlink command")
> Assisted-by: Gemini:gemini-3.6-flash Gemini:gemini-3.1-pro-preview syzbot
> Reported-by: syzbot+41bc60511c2884783c27@syzkaller.appspotmail.com
> Closes: https://syzkaller.appspot.com/bug?extid=41bc60511c2884783c27
> Link: https://syzkaller.appspot.com/ai_job?id=30871f87-7a83-46b8-ab83-919894b0787b
> Signed-off-by: Slawomir Stepien <sst@poczta.fm>
>
> ---
> diff --git a/fs/nfsd/nfsctl.c b/fs/nfsd/nfsctl.c
> index fa92e31d1..516c04c14 100644
> --- a/fs/nfsd/nfsctl.c
> +++ b/fs/nfsd/nfsctl.c
> @@ -1943,6 +1943,8 @@ int nfsd_nl_version_get_doit(struct sk_buff *skb, struct genl_info *info)
> return err;
> }
>
> +#define NFSD_MAX_LISTENERS 128
> +
> /**
> * nfsd_nl_listener_set_doit - set the nfs running sockets
> * @skb: reply buffer
> @@ -1955,12 +1957,19 @@ int nfsd_nl_listener_set_doit(struct sk_buff *skb, struct genl_info *info)
> struct net *net = genl_info_net(info);
> struct svc_xprt *xprt, *tmp;
> const struct nlattr *attr;
> + unsigned int count = 0;
> struct svc_serv *serv;
> LIST_HEAD(permsocks);
> struct nfsd_net *nn;
> bool delete = false;
> int err, rem;
>
> + nlmsg_for_each_attr_type(attr, NFSD_A_SERVER_SOCK_ADDR, info->nlhdr,
> + GENL_HDRLEN, rem) {
> + if (++count > NFSD_MAX_LISTENERS)
> + return -EINVAL;
> + }
> +
> mutex_lock(&nfsd_mutex);
>
> err = nfsd_create_serv(net);
> @@ -2073,8 +2082,10 @@ int nfsd_nl_listener_set_doit(struct sk_buff *skb, struct genl_info *info)
> ret = svc_xprt_create_from_sa(serv, xcl_name, net, sa, 0,
> current_cred());
> /* always save the latest error */
> - if (ret < 0)
> + if (ret < 0) {
> err = ret;
> + break;
> + }
> }
>
> if (!serv->sv_nrthreads && list_empty(&nn->nfsd_serv->sv_permsocks))
>
>
> base-commit: 075b74841bd0065a3bda3440873c747938e69b68
I have a patch series in progress for this already, which also
--
Jeff Layton <jlayton@kernel.org>
prev parent reply other threads:[~2026-08-20 11:37 UTC|newest]
Thread overview: 2+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-08-20 6:55 [PATCH] nfsd: prevent hung task in nfsd_nl_listener_set_doit() syzbot
2026-08-20 11:37 ` Jeff Layton [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=699bacb503758fbe2ebf01f3e8aca95e3eee4128.camel@kernel.org \
--to=jlayton@kernel.org \
--cc=Dai.Ngo@oracle.com \
--cc=cel@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-nfs@vger.kernel.org \
--cc=lorenzo@kernel.org \
--cc=neil@brown.name \
--cc=okorniev@redhat.com \
--cc=sst@poczta.fm \
--cc=syzbot@kernel.org \
--cc=syzbot@lists.linux.dev \
--cc=syzkaller-bugs@googlegroups.com \
--cc=tom@talpey.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox