Netdev List
 help / color / mirror / Atom feed
From: "Anna Schumaker" <anna@kernel.org>
To: "Andrew Pam" <andrew@sericyb.com.au>,
	"Trond Myklebust" <trondmy@kernel.org>
Cc: "Chuck Lever" <cel@kernel.org>,
	"Jeff Layton" <jlayton@kernel.org>, NeilBrown <neil@brown.name>,
	"Olga Kornievskaia" <okorniev@redhat.com>,
	"Dai Ngo" <Dai.Ngo@oracle.com>, "Tom Talpey" <tom@talpey.com>,
	"Rafael J. Wysocki" <rafael@kernel.org>,
	"Mike Snitzer" <snitzer@kernel.org>,
	linux-nfs@vger.kernel.org, linux-pm@vger.kernel.org,
	netdev@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: Re: [PATCH v2 1/3] SUNRPC: freeze rpciod and xprtiod across system sleep
Date: Mon, 16 Nov 2026 14:32:00 -0500	[thread overview]
Message-ID: <0d9f6196-6737-4936-a8fd-97192d96ad3f@app.fastmail.com> (raw)
In-Reply-To: <20261116131428.1024230-2-andrew@sericyb.com.au>

Hi Andrew,

On Mon, Nov 16, 2026, at 8:14 AM, Andrew Pam wrote:
> After hibernation takes its snapshot, the kernel thaws devices to write
> the image, and the NIC with them. Freezable kernel threads and
> workqueues stay frozen until the image is restored, but rpciod and
> xprtiod are not freezable, so RPCs continue to be transmitted, and
> transports reconnected, until power-off. The server acts on them; the
> restored image knows nothing of them.
>
> With NFSv4.1 this wedges the mount after resume. Captured between a
> client that hibernated and its server (Linux 7.0 client, 6.12 server):
>
>   - 05:06:02 last exchange before the snapshot; client snd_nxt
>     2711544997.
>   - 05:06:15 hibernation entry.
>   - 05:06:44 and 05:07:48, after the snapshot: a lease-renewal
>     SEQUENCE on slot 0 (seqid 192219), the next (192220), and
>     DELEGRETURNs on slots 1-5. The server executes all of them and
>     acknowledges up to TCP sequence 2711546645.
>   - 14 hours later, on resume: the client retransmits the slot 0
>     SEQUENCE from TCP sequence 2711544997. The server discards it as a
>     duplicate and acknowledges 2711546645, which the client ignores as
>     acknowledging unsent data. Nothing more crosses the connection.
>   - On reconnecting, slot 0 gets NFS4ERR_SEQ_MISORDERED (server at
>     192220, client at 192219) every 15 seconds indefinitely; the state
>     manager waits in nfs4_begin_drain_session() for that task, and
>     every other RPC waits behind it.
>
> Other resumes produced the same rollback with different symptoms:
> requests accepted by the server's TCP but never reaching nfsd, or a
> partial overlap that left nfsd reading a session ID as a record marker
> ("svc_tcp_read_marker nfsd RPC fragment too large"). Restarting the
> server is the only recovery.
>
> Make both workqueues freezable, as the SMB client already does for all
> of its workqueues (WQ_FREEZABLE | WQ_MEM_RECLAIM). Work queued while
> frozen runs after thaw_processes(), in the restored kernel, which is
> consistent with the server. Memory preallocation for the image happens
> before freeze_kernel_threads(), so writeback through rpciod is still
> available then, and pm_restrict_gfp_mask() keeps reclaim from issuing
> I/O while the image is written.
>
> WQ_FREEZABLE is consulted only when freezing and thawing, not when work
> is queued or run, so normal operation is unaffected. Entering system
> sleep now waits for in-flight rpciod and xprtiod items to finish; these
> are short state-machine steps with non-blocking sends and connects. If
> one ever blocked for longer than freeze_timeout_msecs, suspend or
> hibernation would fail, naming the busy workqueue, where today it
> proceeds. The SMB client and XFS accept the same trade-off for their
> reclaim workqueues.
>
> This does not stop the network stack from acknowledging data the
> server sends after the snapshot, such as a callback or a reply to a
> request still in flight when the snapshot was taken; that remains
> a general hibernation and TCP problem.
>
> No Fixes: tag: rpciod has never been freezable, so no single commit
> introduced this.

Can you take a human editing-pass over this commit message, please? Try
to really focus it on what this patch does and why it is important. The
"No Fixes: tag:" part really sticks out to me as an unnecessary addition,
and the testing stuff (below) should be in the cover letter and not the
patch itself.

>
> Tested in QEMU/KVM: an NFSv4.2 client hibernating in test_resume mode
> (the image is written, then restored at once) against a 7.4-rc2 nfsd
> with a 10 second lease, with 1.2 GB of incompressible data in memory and
> the swap device throttled so that writing the image takes 81 seconds.
> Without this patch the server received 12 SEQUENCE renewals during the
> image write, none of which the restored client knew of; after the
> restore the client's mountstats show a SEQUENCE error, two extra
> connections and 10 seconds of queueing before it recovered. With this
> series the server received nothing from the snapshot until the restore,
> and the client resumed without errors. The indefinite hang seen on real
> hardware was not reproduced in the VM: it also needs a request holding
> a session slot across the restore.
>
> Drafted with an AI coding assistant (Claude), which analysed the packet
> capture, wrote the change and this description, and ran the test above.
>
> Signed-off-by: Andrew Pam <andrew@sericyb.com.au>
> ---
>  net/sunrpc/sched.c | 11 ++++++++---
>  1 file changed, 8 insertions(+), 3 deletions(-)
>
> diff --git a/net/sunrpc/sched.c b/net/sunrpc/sched.c
> index 016f16ca5779..41f25309b8fb 100644
> --- a/net/sunrpc/sched.c
> +++ b/net/sunrpc/sched.c
> @@ -1276,13 +1276,18 @@ static int rpciod_start(void)
>  	struct workqueue_struct *wq;
> 
>  	/*
> -	 * Create the rpciod thread and wait for it to start.
> +	 * Both workqueues are freezable so that no RPC is transmitted, and
> +	 * no connection is made, between taking a hibernation snapshot and
> +	 * powering off. Anything sent then changes the server's state but is
> +	 * missing from the image, and is forgotten when the image is restored.

Let's leave the original comment alone, please.

>  	 */
> -	wq = alloc_workqueue("rpciod", WQ_MEM_RECLAIM | WQ_UNBOUND, 0);
> +	wq = alloc_workqueue("rpciod",
> +			     WQ_MEM_RECLAIM | WQ_UNBOUND | WQ_FREEZABLE, 0);

Note that this change is going to conflict with Chuck's "NFS: Set WQ_SYSFS on nfsiod"
patch that I'm expecting to go into v7.5:  https://lore.kernel.org/linux-nfs/20260915-performance-v3-3-ae26d460bfd3@kernel.org/

Thanks,
Anna

>  	if (!wq)
>  		goto out_failed;
>  	rpciod_workqueue = wq;
> -	wq = alloc_workqueue("xprtiod", WQ_UNBOUND | WQ_MEM_RECLAIM, 0);
> +	wq = alloc_workqueue("xprtiod",
> +			     WQ_UNBOUND | WQ_MEM_RECLAIM | WQ_FREEZABLE, 0);
>  	if (!wq)
>  		goto free_rpciod;
>  	xprtiod_workqueue = wq;
> -- 
> 2.53.0

  reply	other threads:[~2026-11-16 19:32 UTC|newest]

Thread overview: 6+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-16 13:14 [PATCH v2 0/3] SUNRPC: keep NFS client connections consistent across hibernation Andrew Pam
2026-11-16 13:14 ` [PATCH v2 1/3] SUNRPC: freeze rpciod and xprtiod across system sleep Andrew Pam
2026-11-16 19:32   ` Anna Schumaker [this message]
2026-11-16 13:14 ` [PATCH v2 2/3] NFSv4: run lease renewal on a freezable workqueue Andrew Pam
2026-11-16 13:14 ` [PATCH v2 3/3] SUNRPC: reconnect client transports after resuming from hibernation Andrew Pam
2026-11-16 19:34   ` Anna Schumaker

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=0d9f6196-6737-4936-a8fd-97192d96ad3f@app.fastmail.com \
    --to=anna@kernel.org \
    --cc=Dai.Ngo@oracle.com \
    --cc=andrew@sericyb.com.au \
    --cc=cel@kernel.org \
    --cc=jlayton@kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-nfs@vger.kernel.org \
    --cc=linux-pm@vger.kernel.org \
    --cc=neil@brown.name \
    --cc=netdev@vger.kernel.org \
    --cc=okorniev@redhat.com \
    --cc=rafael@kernel.org \
    --cc=snitzer@kernel.org \
    --cc=tom@talpey.com \
    --cc=trondmy@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox