Netdev List
 help / color / mirror / Atom feed
From: "Anna Schumaker" <anna@kernel.org>
To: "Andrew Pam" <andrew@sericyb.com.au>,
	"Trond Myklebust" <trondmy@kernel.org>
Cc: "Chuck Lever" <cel@kernel.org>,
	"Jeff Layton" <jlayton@kernel.org>, NeilBrown <neil@brown.name>,
	"Olga Kornievskaia" <okorniev@redhat.com>,
	"Dai Ngo" <Dai.Ngo@oracle.com>, "Tom Talpey" <tom@talpey.com>,
	"Rafael J. Wysocki" <rafael@kernel.org>,
	"Mike Snitzer" <snitzer@kernel.org>,
	linux-nfs@vger.kernel.org, linux-pm@vger.kernel.org,
	netdev@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: Re: [PATCH v2 3/3] SUNRPC: reconnect client transports after resuming from hibernation
Date: Mon, 16 Nov 2026 14:34:54 -0500	[thread overview]
Message-ID: <51a2b5d3-4322-42ed-a2ee-e17c125865fa@app.fastmail.com> (raw)
In-Reply-To: <20261116131428.1024230-4-andrew@sericyb.com.au>

Hi Andrew,

On Mon, Nov 16, 2026, at 8:14 AM, Andrew Pam wrote:
> With rpciod and xprtiod frozen across hibernation, no RPC is sent after
> the snapshot. TCP can still acknowledge data the server sends then,
> though: typically a reply that was in flight when the snapshot was
> taken. The restored image has not seen that data, and the server, which
> saw it acknowledged, will not send it again. The client then waits
> indefinitely for replies that never arrive.
>
> Captured in a QEMU test (below): on the one cycle of six that hung, the
> client's TCP acknowledgement went backwards by 200 bytes after the
> restore, and the requests outstanding (WRITE, COMMIT, GETATTR) stayed
> queued on xprt_pending with no further progress.
>
> After resuming, disconnect every client transport. Requests in flight
> are retransmitted on the new connection, and the server answers any it
> has already executed from its reply cache. Because nothing was sent
> after the snapshot, NFSv4.1 session slot sequence ids still agree on
> both sides.
>
> The disconnect is done from a PM notifier at PM_POST_HIBERNATION, which
> runs in the restored kernel once everything has thawed. It is also sent
> when hibernation fails, where the reconnect is unnecessary but harmless.
> xprt_put() may sleep, so references to the transports are taken under
> the client list lock and RCU into an array, and released afterwards. The
> array is sized by a counting pass first, so it is allocated with
> GFP_KERNEL rather than under the locks, and the walk is repeated if
> transports were added in between: one left out would stay stalled.
>
> Tested in the QEMU setup described in the first patch of this series,
> with 20 hibernations in test_resume mode back to back while NFS writes
> (8 MB, fsync) and reads ran continuously. Without the series the client
> hung on the first cycle; with only the first two patches, on the sixth.
> With this patch all 20 cycles completed and I/O progressed on every
> one. The capture shows the client's acknowledgement going backwards on
> two of the old connections, and the client reconnecting and carrying on
> both times. The single-hibernation test is unchanged apart from one
> extra connection per resume: nothing reached the server during the
> image write, and the client resumed without errors.
>
> Drafted with an AI coding assistant (Claude), which also ran the tests.
>
> Signed-off-by: Andrew Pam <andrew@sericyb.com.au>
> ---
>  net/sunrpc/clnt.c        | 112 +++++++++++++++++++++++++++++++++++++++
>  net/sunrpc/sunrpc.h      |   2 +
>  net/sunrpc/sunrpc_syms.c |   7 +++
>  3 files changed, 121 insertions(+)
>
> diff --git a/net/sunrpc/clnt.c b/net/sunrpc/clnt.c
> index 2f076b4bfd54..f9c31c49dc0b 100644
> --- a/net/sunrpc/clnt.c
> +++ b/net/sunrpc/clnt.c
> @@ -32,6 +32,8 @@
>  #include <linux/in.h>
>  #include <linux/in6.h>
>  #include <linux/un.h>
> +#include <linux/suspend.h>
> +#include <net/net_namespace.h>
> 
>  #include <linux/sunrpc/clnt.h>
>  #include <linux/sunrpc/addr.h>
> @@ -881,6 +883,116 @@ int rpc_clnt_iterate_for_each_xprt(struct rpc_clnt *clnt,
>  }
>  EXPORT_SYMBOL_GPL(rpc_clnt_iterate_for_each_xprt);
> 
> +/*
> + * After hibernation takes its snapshot, the kernel keeps running with the
> + * network up while it writes the image. rpciod and xprtiod are frozen by
> + * then, so no RPC is sent, but TCP still acknowledges anything the server
> + * sends, such as a reply that was in flight when the snapshot was taken.
> + * The restored image has not seen that data, and the server will not send
> + * it again, so the connection cannot make progress.
> + *
> + * Once the system has resumed, disconnect every client transport. Requests
> + * in flight are retransmitted on a new connection, and the server answers
> + * any it has already executed from its reply cache. Because nothing was
> + * sent after the snapshot, NFSv4.1 session slot sequence ids still agree.
> + */
> +/*
> + * Take a reference to each client transport in every network namespace and
> + * store it in @xprts, up to @max of them. Returns the number of transports
> + * found, which exceeds @max if the array was too small. Called with
> + * net_rwsem held for reading.
> + */

What's with the weird double comment here? I like the second one better.

> +static unsigned int rpc_pm_get_xprts(struct rpc_xprt **xprts,
> +				     unsigned int max)
> +{
> +	unsigned int n = 0;
> +	struct net *net;
> +
> +	for_each_net(net) {
> +		struct sunrpc_net *sn = net_generic(net, sunrpc_net_id);
> +		struct rpc_clnt *clnt;
> +
> +		spin_lock(&sn->rpc_client_lock);
> +		rcu_read_lock();
> +		list_for_each_entry(clnt, &sn->all_clients, cl_clients) {
> +			struct rpc_xprt_switch *xps;
> +			struct rpc_xprt *xprt;
> +
> +			xps = rcu_dereference(clnt->cl_xpi.xpi_xpswitch);
> +			if (!xps)
> +				continue;
> +			list_for_each_entry_rcu(xprt, &xps->xps_xprt_list,
> +						xprt_switch) {
> +				if (n < max) {
> +					xprts[n] = xprt_get(xprt);
> +					if (!xprts[n])
> +						continue;
> +				}
> +				n++;
> +			}
> +		}
> +		rcu_read_unlock();
> +		spin_unlock(&sn->rpc_client_lock);
> +	}
> +	return n;
> +}
> +
> +static int rpc_pm_notify(struct notifier_block *nb, unsigned long action,
> +			 void *data)
> +{
> +	struct rpc_xprt **xprts = NULL;
> +	unsigned int i, n, max = 0;
> +
> +	if (action != PM_POST_HIBERNATION)
> +		return NOTIFY_DONE;
> +
> +	/*
> +	 * The references are taken under spinlocks, but xprt_put() may sleep,
> +	 * so collect them in an array and disconnect after the locks are
> +	 * dropped. Count first so that the array can be allocated without
> +	 * GFP_ATOMIC, and recount if transports were added in the meantime:
> +	 * a transport left out would keep the stalled connection.
> +	 */

Is there a way to do this without the double loop? The rpc_xprt_switch has a
"xps_nxprts" field that should hold the number of xprts in the switch. Using
that might be more straightforward than looping twice and counting.

Thanks,
Anna

> +	for (;;) {
> +		down_read(&net_rwsem);
> +		n = rpc_pm_get_xprts(xprts, max);
> +		up_read(&net_rwsem);
> +		if (n <= max)
> +			break;
> +
> +		for (i = 0; i < max; i++)
> +			xprt_put(xprts[i]);
> +		kfree(xprts);
> +		max = n + 8;
> +		xprts = kmalloc_objs(*xprts, max);
> +		if (!xprts) {
> +			pr_warn("RPC: no memory to reset transports after resume\n");
> +			return NOTIFY_DONE;
> +		}
> +	}
> +
> +	for (i = 0; i < n; i++) {
> +		xprt_force_disconnect(xprts[i]);
> +		xprt_put(xprts[i]);
> +	}
> +	kfree(xprts);
> +	return NOTIFY_DONE;
> +}
> +
> +static struct notifier_block rpc_pm_nb = {
> +	.notifier_call = rpc_pm_notify,
> +};
> +
> +int rpc_pm_notifier_register(void)
> +{
> +	return register_pm_notifier(&rpc_pm_nb);
> +}
> +
> +void rpc_pm_notifier_unregister(void)
> +{
> +	unregister_pm_notifier(&rpc_pm_nb);
> +}
> +
>  /*
>   * Kill all tasks for the given client.
>   * XXX: kill their descendants as well?
> diff --git a/net/sunrpc/sunrpc.h b/net/sunrpc/sunrpc.h
> index 11b9e5cead8e..1833b167616c 100644
> --- a/net/sunrpc/sunrpc.h
> +++ b/net/sunrpc/sunrpc.h
> @@ -43,6 +43,8 @@ struct rpc_task;
>  struct rpc_xprt;
>  int rpc_clients_notifier_register(void);
>  void rpc_clients_notifier_unregister(void);
> +int rpc_pm_notifier_register(void);
> +void rpc_pm_notifier_unregister(void);
>  void auth_domain_cleanup(void);
>  void svc_sock_update_bufs(struct svc_serv *serv);
>  enum svc_auth_status svc_authenticate(struct svc_rqst *rqstp);
> diff --git a/net/sunrpc/sunrpc_syms.c b/net/sunrpc/sunrpc_syms.c
> index 1a3884a0376a..d32786f51ae5 100644
> --- a/net/sunrpc/sunrpc_syms.c
> +++ b/net/sunrpc/sunrpc_syms.c
> @@ -116,6 +116,10 @@ init_sunrpc(void)
>  	if (err)
>  		goto out6;
> 
> +	err = rpc_pm_notifier_register();
> +	if (err)
> +		goto out7;
> +
>  	sunrpc_debugfs_init();
>  #if IS_ENABLED(CONFIG_SUNRPC_DEBUG)
>  	rpc_register_sysctl();
> @@ -124,6 +128,8 @@ init_sunrpc(void)
>  	init_socket_xprt();	/* clnt sock transport */
>  	return 0;
> 
> +out7:
> +	genl_unregister_family(&sunrpc_nl_family);
>  out6:
>  	rpc_sysfs_exit();
>  out5:
> @@ -141,6 +147,7 @@ init_sunrpc(void)
>  static void __exit
>  cleanup_sunrpc(void)
>  {
> +	rpc_pm_notifier_unregister();
>  	genl_unregister_family(&sunrpc_nl_family);
>  	rpc_sysfs_exit();
>  	rpc_cleanup_clids();
> -- 
> 2.53.0

      reply	other threads:[~2026-11-16 19:35 UTC|newest]

Thread overview: 6+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-16 13:14 [PATCH v2 0/3] SUNRPC: keep NFS client connections consistent across hibernation Andrew Pam
2026-11-16 13:14 ` [PATCH v2 1/3] SUNRPC: freeze rpciod and xprtiod across system sleep Andrew Pam
2026-11-16 19:32   ` Anna Schumaker
2026-11-16 13:14 ` [PATCH v2 2/3] NFSv4: run lease renewal on a freezable workqueue Andrew Pam
2026-11-16 13:14 ` [PATCH v2 3/3] SUNRPC: reconnect client transports after resuming from hibernation Andrew Pam
2026-11-16 19:34   ` Anna Schumaker [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=51a2b5d3-4322-42ed-a2ee-e17c125865fa@app.fastmail.com \
    --to=anna@kernel.org \
    --cc=Dai.Ngo@oracle.com \
    --cc=andrew@sericyb.com.au \
    --cc=cel@kernel.org \
    --cc=jlayton@kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-nfs@vger.kernel.org \
    --cc=linux-pm@vger.kernel.org \
    --cc=neil@brown.name \
    --cc=netdev@vger.kernel.org \
    --cc=okorniev@redhat.com \
    --cc=rafael@kernel.org \
    --cc=snitzer@kernel.org \
    --cc=tom@talpey.com \
    --cc=trondmy@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox