The Linux Kernel Mailing List
 help / color / mirror / Atom feed
From: Slawomir Stepien <sst@poczta.fm>
To: Jeff Layton <jlayton@kernel.org>
Cc: syzbot <syzbot@kernel.org>,
	syzkaller-bugs@googlegroups.com, Chuck Lever <cel@kernel.org>,
	linux-nfs@vger.kernel.org, Lorenzo Bianconi <lorenzo@kernel.org>,
	Dai.Ngo@oracle.com, linux-kernel@vger.kernel.org,
	neil@brown.name, okorniev@redhat.com, syzbot@lists.linux.dev,
	tom@talpey.com
Subject: Re: [PATCH] nfsd: prevent hung task in nfsd_nl_listener_set_doit()
Date: Sun, 23 Aug 2026 22:27:56 +0200	[thread overview]
Message-ID: <aotXzLmrd-B89YRg@nr200> (raw)
In-Reply-To: <699bacb503758fbe2ebf01f3e8aca95e3eee4128.camel@kernel.org>

On sie 20, 2026 07:37, Jeff Layton wrote:
> On Thu, 2026-08-20 at 06:55 +0000, syzbot wrote:
> > From: Slawomir Stepien <sst@poczta.fm>
> > 
> > In nfsd_nl_listener_set_doit(), the kernel iterates over all
> > NFSD_A_SERVER_SOCK_ADDR attributes provided in a netlink message to
> > configure NFS server listeners. There is currently no limit on the number
> > of attributes a user can send.
> > 
> > For each attribute, svc_xprt_create_from_sa() is called, which may
> > synchronously invoke request_module() to load the corresponding transport
> > module. If a user provides a large number of invalid transport names,
> > request_module() is called sequentially for each, taking a massive amount
> > of time. Since this entire process occurs while holding the global
> > nfsd_mutex, it blocks other tasks attempting to acquire the mutex and
> > triggers a hung task timeout:
> > 
> 
> My LLM latched onto the same explanation, but the request module
> upcalls are actually quite quick and have some other gates on them that
> make this problem hard to hit in practice.

Oh, OK. Thanks for clearing that up!

> The real problem is rpcbind registration, which can be quite slow. The
> fix for that is to make the registration asynchronous, and to not
> trigger errors on rpcbind registration failure. That's a bigger
> project, but let us know if you're interested!

I think I'm not that good to do it correctly and in some reasonable time. I might take a 2nd look at
this and understand the root cause (also, based on your comments in your series) but I think in the
end it might be too hard for me.

> In any case, I've sent a series that includes a similar fix to this
> one, along with some others:
> 
>     https://lore.kernel.org/linux-nfs/20260811-nfsd-nl-hang-v2-0-c0c92b3953c3@kernel.org/
> 
> I'll try to get the follow-on series out later today or tomorrow.

Nice series, I've learnt something from it! I will be looking for v3!

> >   INFO: task blocked for more than 10 seconds.
> >   ...
> >   Call Trace:
> >    <TASK>
> >    __schedule+0x17e7/0x5630 kernel/sched/core.c:7234
> >    schedule+0x164/0x2b0 kernel/sched/core.c:7326
> >    schedule_preempt_disabled+0x13/0x30 kernel/sched/core.c:7383
> >    __mutex_lock_common kernel/locking/mutex.c:726 [inline]
> >    __mutex_lock+0x7bf/0x1550 kernel/locking/mutex.c:821
> >    nfsd_nl_version_get_doit+0x17c/0xd20 fs/nfsd/nfsctl.c:1889
> >    genl_family_rcv_msg_doit+0x233/0x340 net/netlink/genetlink.c:1114
> >    genl_family_rcv_msg net/netlink/genetlink.c:1194 [inline]
> >    genl_rcv_msg+0x614/0x7a0 net/netlink/genetlink.c:1209
> >   ...
> >   2 locks held by task/5864:
> >    #0: ffffffff90114168 (cb_lock){++++}-{4:4}, at: genl_rcv+0x19/0x40
> >    net/netlink/genetlink.c:1217
> >    #1: ffffffff8eeb1a60 (nfsd_mutex){+.+.}-{4:4}, at:
> >    nfsd_nl_listener_set_doit+0x135/0x1750 fs/nfsd/nfsctl.c:1964
> > 
> > Furthermore, the function does not break out of the loop if
> > svc_xprt_create_from_sa() fails, and it suffers from an O(N^2) complexity
> > issue because svc_find_listener() iterates over the serv->sv_permsocks list
> > for each attribute.
> > 
> > Address this by introducing a hard limit of 128 NFSD_A_SERVER_SOCK_ADDR
> > attributes per netlink message. The limit is enforced before acquiring the
> > nfsd_mutex to prevent lock contention. Additionally, modify the loop to
> > break immediately if listener creation fails, avoiding needless sequential
> > request_module() calls for invalid configurations.
> > 
> > Fixes: 16a471177496 ("NFSD: add listener-{set,get} netlink command")
> > Assisted-by: Gemini:gemini-3.6-flash Gemini:gemini-3.1-pro-preview syzbot
> > Reported-by: syzbot+41bc60511c2884783c27@syzkaller.appspotmail.com
> > Closes: https://syzkaller.appspot.com/bug?extid=41bc60511c2884783c27
> > Link: https://syzkaller.appspot.com/ai_job?id=30871f87-7a83-46b8-ab83-919894b0787b
> > Signed-off-by: Slawomir Stepien <sst@poczta.fm>
> >
> > ---
> > diff --git a/fs/nfsd/nfsctl.c b/fs/nfsd/nfsctl.c
> > index fa92e31d1..516c04c14 100644
> > --- a/fs/nfsd/nfsctl.c
> > +++ b/fs/nfsd/nfsctl.c
> > @@ -1943,6 +1943,8 @@ int nfsd_nl_version_get_doit(struct sk_buff *skb, struct genl_info *info)
> >  	return err;
> >  }
> >  
> > +#define NFSD_MAX_LISTENERS 128
> > +
> >  /**
> >   * nfsd_nl_listener_set_doit - set the nfs running sockets
> >   * @skb: reply buffer
> > @@ -1955,12 +1957,19 @@ int nfsd_nl_listener_set_doit(struct sk_buff *skb, struct genl_info *info)
> >  	struct net *net = genl_info_net(info);
> >  	struct svc_xprt *xprt, *tmp;
> >  	const struct nlattr *attr;
> > +	unsigned int count = 0;
> >  	struct svc_serv *serv;
> >  	LIST_HEAD(permsocks);
> >  	struct nfsd_net *nn;
> >  	bool delete = false;
> >  	int err, rem;
> >  
> > +	nlmsg_for_each_attr_type(attr, NFSD_A_SERVER_SOCK_ADDR, info->nlhdr,
> > +				 GENL_HDRLEN, rem) {
> > +		if (++count > NFSD_MAX_LISTENERS)
> > +			return -EINVAL;
> > +	}
> > +
> >  	mutex_lock(&nfsd_mutex);
> >  
> >  	err = nfsd_create_serv(net);
> > @@ -2073,8 +2082,10 @@ int nfsd_nl_listener_set_doit(struct sk_buff *skb, struct genl_info *info)
> >  		ret = svc_xprt_create_from_sa(serv, xcl_name, net, sa, 0,
> >  					      current_cred());
> >  		/* always save the latest error */
> > -		if (ret < 0)
> > +		if (ret < 0) {
> >  			err = ret;
> > +			break;
> > +		}
> >  	}
> >  
> >  	if (!serv->sv_nrthreads && list_empty(&nn->nfsd_serv->sv_permsocks))
> > 
> > 
> > base-commit: 075b74841bd0065a3bda3440873c747938e69b68
> 
> I have a patch series in progress for this already, which also 

-- 
Slawomir Stepien

  reply	other threads:[~2026-08-23 20:29 UTC|newest]

Thread overview: 4+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-08-20  6:55 [PATCH] nfsd: prevent hung task in nfsd_nl_listener_set_doit() syzbot
2026-08-20 11:37 ` Jeff Layton
2026-08-23 20:27   ` Slawomir Stepien [this message]
2026-08-24 12:00     ` Jeff Layton

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=aotXzLmrd-B89YRg@nr200 \
    --to=sst@poczta.fm \
    --cc=Dai.Ngo@oracle.com \
    --cc=cel@kernel.org \
    --cc=jlayton@kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-nfs@vger.kernel.org \
    --cc=lorenzo@kernel.org \
    --cc=neil@brown.name \
    --cc=okorniev@redhat.com \
    --cc=syzbot@kernel.org \
    --cc=syzbot@lists.linux.dev \
    --cc=syzkaller-bugs@googlegroups.com \
    --cc=tom@talpey.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox