Netdev List
 help / color / mirror / Atom feed
From: Jeff Layton <jlayton@kernel.org>
To: Chuck Lever <cel@kernel.org>, NeilBrown <neil@brown.name>,
	 Olga Kornievskaia <okorniev@redhat.com>,
	Dai Ngo <Dai.Ngo@oracle.com>,  Tom Talpey <tom@talpey.com>,
	Trond Myklebust <trondmy@kernel.org>,
	 Anna Schumaker <anna@kernel.org>,
	"David S. Miller" <davem@davemloft.net>,
	 Eric Dumazet <edumazet@google.com>,
	Jakub Kicinski <kuba@kernel.org>,
	 Paolo Abeni <pabeni@redhat.com>, Simon Horman <horms@kernel.org>,
	 Shuah Khan <shuah@kernel.org>
Cc: Slawomir Stepien <sst@poczta.fm>,
	linux-nfs@vger.kernel.org,  linux-kernel@vger.kernel.org,
	netdev@vger.kernel.org,  Trond Myklebust <trondmy@gmail.com>,
	linux-kselftest@vger.kernel.org,
	 Jeff Layton <jlayton@kernel.org>
Subject: [PATCH v3 04/14] SUNRPC: bound the local rpcbind client timeout to 1s
Date: Fri, 28 Aug 2026 12:37:34 -0400	[thread overview]
Message-ID: <20260828-nfsd-nl-hang-v3-4-55026685c75d@kernel.org> (raw)
In-Reply-To: <20260828-nfsd-nl-hang-v3-0-55026685c75d@kernel.org>

The kernel's local rpcbind client uses the transport defaults: a 10s major
timeout for AF_LOCAL, and 60s for the loopback TCP fallback.
xprt_calc_majortimeo() returns to_initval when to_increment is 0.

Those calls are synchronous, and they run under nfsd_mutex. One operation
makes several of them. rpcb_create_local() tries up to three client
creations, and svc_register() sends one call for each program and version.
A local rpcbind that accepts the connection but never replies stalls every
one of these calls. The accumulated hold is long enough to trip the
hung-task watchdog on other NFSD netlink operations. The holder itself
waits killably and escapes the watchdog:

  INFO: task hung in nfsd_nl_cache_flush_doit

The local rpcbind is on loopback or on an AF_LOCAL socket, and it answers
in microseconds. Bound its client to one attempt of 1s.

This shortens the stall. It does not remove the stall, and it is not free.
Registration stays synchronous and stays fatal. An rpcb_create_local()
failure aborts nfsd_create_serv() through svc_bind(), and an
svc_register() failure makes svc_setup_socket() fail. An rpcbind that is
merely slow to be scheduled can therefore now fail server startup, where
it succeeded before. The real fix is to make the registration
asynchronous.

Assisted-by: LLM
Link: https://syzkaller.appspot.com/bug?extid=c7eae0eb80858a2dba0f
Signed-off-by: Jeff Layton <jlayton@kernel.org>
---
 net/sunrpc/rpcb_clnt.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/net/sunrpc/rpcb_clnt.c b/net/sunrpc/rpcb_clnt.c
index 6aa372188c86..0aa376b82a52 100644
--- a/net/sunrpc/rpcb_clnt.c
+++ b/net/sunrpc/rpcb_clnt.c
@@ -221,6 +221,16 @@ static void rpcb_set_local(struct net *net, struct rpc_clnt *clnt,
 # define SUN_LEN(ptr) (offsetof(struct sockaddr_un, sun_path)		\
 		      + 1 + strlen((ptr)->sun_path + 1))
 
+/*
+ * The kernel's rpcbind client talks only to the local rpcbind, over loopback
+ * or a local AF_LOCAL socket, where a healthy rpcbind answers in microseconds.
+ */
+static const struct rpc_timeout rpcb_local_timeout = {
+	.to_initval	= 1 * HZ,
+	.to_maxval	= 1 * HZ,
+	.to_retries	= 0,
+};
+
 /*
  * Returns zero on success, otherwise a negative errno value
  * is returned.
@@ -238,6 +248,7 @@ static int rpcb_create_af_local(struct net *net,
 		.version	= RPCBVERS_2,
 		.authflavor	= RPC_AUTH_NULL,
 		.cred		= current_cred(),
+		.timeout	= &rpcb_local_timeout,
 		/*
 		 * We turn off the idle timeout to prevent the kernel
 		 * from automatically disconnecting the socket.
@@ -312,6 +323,7 @@ static int rpcb_create_local_net(struct net *net)
 		.version	= RPCBVERS_2,
 		.authflavor	= RPC_AUTH_UNIX,
 		.cred		= current_cred(),
+		.timeout	= &rpcb_local_timeout,
 		.flags		= RPC_CLNT_CREATE_NOPING,
 	};
 	struct rpc_clnt *clnt, *clnt4;

-- 
2.55.0


  parent reply	other threads:[~2026-08-28 16:37 UTC|newest]

Thread overview: 17+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-08-28 16:37 [PATCH v3 00/14] nfsd/sunrpc: harden the netlink listener set interface Jeff Layton
2026-08-28 16:37 ` [PATCH v3 01/14] NFSD: cap the number of listeners accepted in listener_set Jeff Layton
2026-08-28 16:37 ` [PATCH v3 02/14] NFSD: validate transport name in listener_set before serv creation Jeff Layton
2026-08-28 16:37 ` [PATCH v3 03/14] SUNRPC: keep the first error in svc_register() Jeff Layton
2026-08-28 16:37 ` Jeff Layton [this message]
2026-08-28 16:37 ` [PATCH v3 05/14] NFSD: report listener creation failures through extack Jeff Layton
2026-08-28 16:37 ` [PATCH v3 06/14] SUNRPC: report local rpcbind calls that get no answer Jeff Layton
2026-08-30 15:51   ` Chuck Lever
2026-08-31 12:05     ` Jeff Layton
2026-08-28 16:37 ` [PATCH v3 07/14] SUNRPC: stop svc_register() once rpcbind stops answering Jeff Layton
2026-08-28 16:37 ` [PATCH v3 08/14] SUNRPC: stop the svc_unregister() sweep " Jeff Layton
2026-08-28 16:37 ` [PATCH v3 09/14] SUNRPC: stop unregistering listeners " Jeff Layton
2026-08-28 16:37 ` [PATCH v3 10/14] NFSD: stop registering with rpcbind after a failure in listener_set Jeff Layton
2026-08-28 16:37 ` [PATCH v3 11/14] selftests/nfsd: exercise listener_set request validation Jeff Layton
2026-08-28 16:37 ` [PATCH v3 12/14] selftests/nfsd: add a per-netns rpcbind stub and the listener round-trips Jeff Layton
2026-08-28 16:37 ` [PATCH v3 13/14] selftests/nfsd: check that listener_set asks rpcbind once Jeff Layton
2026-08-28 16:37 ` [PATCH v3 14/14] selftests/nfsd: check that listener removal " Jeff Layton

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20260828-nfsd-nl-hang-v3-4-55026685c75d@kernel.org \
    --to=jlayton@kernel.org \
    --cc=Dai.Ngo@oracle.com \
    --cc=anna@kernel.org \
    --cc=cel@kernel.org \
    --cc=davem@davemloft.net \
    --cc=edumazet@google.com \
    --cc=horms@kernel.org \
    --cc=kuba@kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-kselftest@vger.kernel.org \
    --cc=linux-nfs@vger.kernel.org \
    --cc=neil@brown.name \
    --cc=netdev@vger.kernel.org \
    --cc=okorniev@redhat.com \
    --cc=pabeni@redhat.com \
    --cc=shuah@kernel.org \
    --cc=sst@poczta.fm \
    --cc=tom@talpey.com \
    --cc=trondmy@gmail.com \
    --cc=trondmy@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox