All of lore.kernel.org
 help / color / mirror / Atom feed
From: Slawomir Stepien <sst@poczta.fm>
To: Jeff Layton <jlayton@kernel.org>
Cc: syzbot <syzbot@kernel.org>,
	syzkaller-bugs@googlegroups.com, Chuck Lever <cel@kernel.org>,
	linux-nfs@vger.kernel.org, Lorenzo Bianconi <lorenzo@kernel.org>,
	Dai.Ngo@oracle.com, linux-kernel@vger.kernel.org,
	neil@brown.name, okorniev@redhat.com, syzbot@lists.linux.dev,
	tom@talpey.com
Subject: Re: [PATCH] nfsd: prevent hung task in nfsd_nl_listener_set_doit()
Date: Sun, 23 Aug 2026 22:27:56 +0200	[thread overview]
Message-ID: <aotXzLmrd-B89YRg@nr200> (raw)
In-Reply-To: <699bacb503758fbe2ebf01f3e8aca95e3eee4128.camel@kernel.org>

On sie 20, 2026 07:37, Jeff Layton wrote:
> On Thu, 2026-08-20 at 06:55 +0000, syzbot wrote:
> > From: Slawomir Stepien <sst@poczta.fm>
> > 
> > In nfsd_nl_listener_set_doit(), the kernel iterates over all
> > NFSD_A_SERVER_SOCK_ADDR attributes provided in a netlink message to
> > configure NFS server listeners. There is currently no limit on the number
> > of attributes a user can send.
> > 
> > For each attribute, svc_xprt_create_from_sa() is called, which may
> > synchronously invoke request_module() to load the corresponding transport
> > module. If a user provides a large number of invalid transport names,
> > request_module() is called sequentially for each, taking a massive amount
> > of time. Since this entire process occurs while holding the global
> > nfsd_mutex, it blocks other tasks attempting to acquire the mutex and
> > triggers a hung task timeout:
> > 
> 
> My LLM latched onto the same explanation, but the request module
> upcalls are actually quite quick and have some other gates on them that
> make this problem hard to hit in practice.

Oh, OK. Thanks for clearing that up!

> The real problem is rpcbind registration, which can be quite slow. The
> fix for that is to make the registration asynchronous, and to not
> trigger errors on rpcbind registration failure. That's a bigger
> project, but let us know if you're interested!

I think I'm not that good to do it correctly and in some reasonable time. I might take a 2nd look at
this and understand the root cause (also, based on your comments in your series) but I think in the
end it might be too hard for me.

> In any case, I've sent a series that includes a similar fix to this
> one, along with some others:
> 
>     https://lore.kernel.org/linux-nfs/20260811-nfsd-nl-hang-v2-0-c0c92b3953c3@kernel.org/
> 
> I'll try to get the follow-on series out later today or tomorrow.

Nice series, I've learnt something from it! I will be looking for v3!

> >   INFO: task blocked for more than 10 seconds.
> >   ...
> >   Call Trace:
> >    <TASK>
> >    __schedule+0x17e7/0x5630 kernel/sched/core.c:7234
> >    schedule+0x164/0x2b0 kernel/sched/core.c:7326
> >    schedule_preempt_disabled+0x13/0x30 kernel/sched/core.c:7383
> >    __mutex_lock_common kernel/locking/mutex.c:726 [inline]
> >    __mutex_lock+0x7bf/0x1550 kernel/locking/mutex.c:821
> >    nfsd_nl_version_get_doit+0x17c/0xd20 fs/nfsd/nfsctl.c:1889
> >    genl_family_rcv_msg_doit+0x233/0x340 net/netlink/genetlink.c:1114
> >    genl_family_rcv_msg net/netlink/genetlink.c:1194 [inline]
> >    genl_rcv_msg+0x614/0x7a0 net/netlink/genetlink.c:1209
> >   ...
> >   2 locks held by task/5864:
> >    #0: ffffffff90114168 (cb_lock){++++}-{4:4}, at: genl_rcv+0x19/0x40
> >    net/netlink/genetlink.c:1217
> >    #1: ffffffff8eeb1a60 (nfsd_mutex){+.+.}-{4:4}, at:
> >    nfsd_nl_listener_set_doit+0x135/0x1750 fs/nfsd/nfsctl.c:1964
> > 
> > Furthermore, the function does not break out of the loop if
> > svc_xprt_create_from_sa() fails, and it suffers from an O(N^2) complexity
> > issue because svc_find_listener() iterates over the serv->sv_permsocks list
> > for each attribute.
> > 
> > Address this by introducing a hard limit of 128 NFSD_A_SERVER_SOCK_ADDR
> > attributes per netlink message. The limit is enforced before acquiring the
> > nfsd_mutex to prevent lock contention. Additionally, modify the loop to
> > break immediately if listener creation fails, avoiding needless sequential
> > request_module() calls for invalid configurations.
> > 
> > Fixes: 16a471177496 ("NFSD: add listener-{set,get} netlink command")
> > Assisted-by: Gemini:gemini-3.6-flash Gemini:gemini-3.1-pro-preview syzbot
> > Reported-by: syzbot+41bc60511c2884783c27@syzkaller.appspotmail.com
> > Closes: https://syzkaller.appspot.com/bug?extid=41bc60511c2884783c27
> > Link: https://syzkaller.appspot.com/ai_job?id=30871f87-7a83-46b8-ab83-919894b0787b
> > Signed-off-by: Slawomir Stepien <sst@poczta.fm>
> >
> > ---
> > diff --git a/fs/nfsd/nfsctl.c b/fs/nfsd/nfsctl.c
> > index fa92e31d1..516c04c14 100644
> > --- a/fs/nfsd/nfsctl.c
> > +++ b/fs/nfsd/nfsctl.c
> > @@ -1943,6 +1943,8 @@ int nfsd_nl_version_get_doit(struct sk_buff *skb, struct genl_info *info)
> >  	return err;
> >  }
> >  
> > +#define NFSD_MAX_LISTENERS 128
> > +
> >  /**
> >   * nfsd_nl_listener_set_doit - set the nfs running sockets
> >   * @skb: reply buffer
> > @@ -1955,12 +1957,19 @@ int nfsd_nl_listener_set_doit(struct sk_buff *skb, struct genl_info *info)
> >  	struct net *net = genl_info_net(info);
> >  	struct svc_xprt *xprt, *tmp;
> >  	const struct nlattr *attr;
> > +	unsigned int count = 0;
> >  	struct svc_serv *serv;
> >  	LIST_HEAD(permsocks);
> >  	struct nfsd_net *nn;
> >  	bool delete = false;
> >  	int err, rem;
> >  
> > +	nlmsg_for_each_attr_type(attr, NFSD_A_SERVER_SOCK_ADDR, info->nlhdr,
> > +				 GENL_HDRLEN, rem) {
> > +		if (++count > NFSD_MAX_LISTENERS)
> > +			return -EINVAL;
> > +	}
> > +
> >  	mutex_lock(&nfsd_mutex);
> >  
> >  	err = nfsd_create_serv(net);
> > @@ -2073,8 +2082,10 @@ int nfsd_nl_listener_set_doit(struct sk_buff *skb, struct genl_info *info)
> >  		ret = svc_xprt_create_from_sa(serv, xcl_name, net, sa, 0,
> >  					      current_cred());
> >  		/* always save the latest error */
> > -		if (ret < 0)
> > +		if (ret < 0) {
> >  			err = ret;
> > +			break;
> > +		}
> >  	}
> >  
> >  	if (!serv->sv_nrthreads && list_empty(&nn->nfsd_serv->sv_permsocks))
> > 
> > 
> > base-commit: 075b74841bd0065a3bda3440873c747938e69b68
> 
> I have a patch series in progress for this already, which also 

-- 
Slawomir Stepien

      reply	other threads:[~2026-08-23 20:28 UTC|newest]

Thread overview: 3+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-08-20  6:55 [PATCH] nfsd: prevent hung task in nfsd_nl_listener_set_doit() syzbot
2026-08-20 11:37 ` Jeff Layton
2026-08-23 20:27   ` Slawomir Stepien [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=aotXzLmrd-B89YRg@nr200 \
    --to=sst@poczta.fm \
    --cc=Dai.Ngo@oracle.com \
    --cc=cel@kernel.org \
    --cc=jlayton@kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-nfs@vger.kernel.org \
    --cc=lorenzo@kernel.org \
    --cc=neil@brown.name \
    --cc=okorniev@redhat.com \
    --cc=syzbot@kernel.org \
    --cc=syzbot@lists.linux.dev \
    --cc=syzkaller-bugs@googlegroups.com \
    --cc=tom@talpey.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.