From: "Anna Schumaker" <anna@kernel.org>
To: "Andrew Pam" <andrew@sericyb.com.au>,
"Trond Myklebust" <trondmy@kernel.org>
Cc: "Chuck Lever" <cel@kernel.org>,
"Jeff Layton" <jlayton@kernel.org>, NeilBrown <neil@brown.name>,
"Olga Kornievskaia" <okorniev@redhat.com>,
"Dai Ngo" <Dai.Ngo@oracle.com>, "Tom Talpey" <tom@talpey.com>,
"Rafael J. Wysocki" <rafael@kernel.org>,
"Mike Snitzer" <snitzer@kernel.org>,
linux-nfs@vger.kernel.org, linux-pm@vger.kernel.org,
netdev@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: Re: [PATCH v2 1/3] SUNRPC: freeze rpciod and xprtiod across system sleep
Date: Mon, 16 Nov 2026 14:32:00 -0500 [thread overview]
Message-ID: <0d9f6196-6737-4936-a8fd-97192d96ad3f@app.fastmail.com> (raw)
In-Reply-To: <20261116131428.1024230-2-andrew@sericyb.com.au>
Hi Andrew,
On Mon, Nov 16, 2026, at 8:14 AM, Andrew Pam wrote:
> After hibernation takes its snapshot, the kernel thaws devices to write
> the image, and the NIC with them. Freezable kernel threads and
> workqueues stay frozen until the image is restored, but rpciod and
> xprtiod are not freezable, so RPCs continue to be transmitted, and
> transports reconnected, until power-off. The server acts on them; the
> restored image knows nothing of them.
>
> With NFSv4.1 this wedges the mount after resume. Captured between a
> client that hibernated and its server (Linux 7.0 client, 6.12 server):
>
> - 05:06:02 last exchange before the snapshot; client snd_nxt
> 2711544997.
> - 05:06:15 hibernation entry.
> - 05:06:44 and 05:07:48, after the snapshot: a lease-renewal
> SEQUENCE on slot 0 (seqid 192219), the next (192220), and
> DELEGRETURNs on slots 1-5. The server executes all of them and
> acknowledges up to TCP sequence 2711546645.
> - 14 hours later, on resume: the client retransmits the slot 0
> SEQUENCE from TCP sequence 2711544997. The server discards it as a
> duplicate and acknowledges 2711546645, which the client ignores as
> acknowledging unsent data. Nothing more crosses the connection.
> - On reconnecting, slot 0 gets NFS4ERR_SEQ_MISORDERED (server at
> 192220, client at 192219) every 15 seconds indefinitely; the state
> manager waits in nfs4_begin_drain_session() for that task, and
> every other RPC waits behind it.
>
> Other resumes produced the same rollback with different symptoms:
> requests accepted by the server's TCP but never reaching nfsd, or a
> partial overlap that left nfsd reading a session ID as a record marker
> ("svc_tcp_read_marker nfsd RPC fragment too large"). Restarting the
> server is the only recovery.
>
> Make both workqueues freezable, as the SMB client already does for all
> of its workqueues (WQ_FREEZABLE | WQ_MEM_RECLAIM). Work queued while
> frozen runs after thaw_processes(), in the restored kernel, which is
> consistent with the server. Memory preallocation for the image happens
> before freeze_kernel_threads(), so writeback through rpciod is still
> available then, and pm_restrict_gfp_mask() keeps reclaim from issuing
> I/O while the image is written.
>
> WQ_FREEZABLE is consulted only when freezing and thawing, not when work
> is queued or run, so normal operation is unaffected. Entering system
> sleep now waits for in-flight rpciod and xprtiod items to finish; these
> are short state-machine steps with non-blocking sends and connects. If
> one ever blocked for longer than freeze_timeout_msecs, suspend or
> hibernation would fail, naming the busy workqueue, where today it
> proceeds. The SMB client and XFS accept the same trade-off for their
> reclaim workqueues.
>
> This does not stop the network stack from acknowledging data the
> server sends after the snapshot, such as a callback or a reply to a
> request still in flight when the snapshot was taken; that remains
> a general hibernation and TCP problem.
>
> No Fixes: tag: rpciod has never been freezable, so no single commit
> introduced this.
Can you take a human editing-pass over this commit message, please? Try
to really focus it on what this patch does and why it is important. The
"No Fixes: tag:" part really sticks out to me as an unnecessary addition,
and the testing stuff (below) should be in the cover letter and not the
patch itself.
>
> Tested in QEMU/KVM: an NFSv4.2 client hibernating in test_resume mode
> (the image is written, then restored at once) against a 7.4-rc2 nfsd
> with a 10 second lease, with 1.2 GB of incompressible data in memory and
> the swap device throttled so that writing the image takes 81 seconds.
> Without this patch the server received 12 SEQUENCE renewals during the
> image write, none of which the restored client knew of; after the
> restore the client's mountstats show a SEQUENCE error, two extra
> connections and 10 seconds of queueing before it recovered. With this
> series the server received nothing from the snapshot until the restore,
> and the client resumed without errors. The indefinite hang seen on real
> hardware was not reproduced in the VM: it also needs a request holding
> a session slot across the restore.
>
> Drafted with an AI coding assistant (Claude), which analysed the packet
> capture, wrote the change and this description, and ran the test above.
>
> Signed-off-by: Andrew Pam <andrew@sericyb.com.au>
> ---
> net/sunrpc/sched.c | 11 ++++++++---
> 1 file changed, 8 insertions(+), 3 deletions(-)
>
> diff --git a/net/sunrpc/sched.c b/net/sunrpc/sched.c
> index 016f16ca5779..41f25309b8fb 100644
> --- a/net/sunrpc/sched.c
> +++ b/net/sunrpc/sched.c
> @@ -1276,13 +1276,18 @@ static int rpciod_start(void)
> struct workqueue_struct *wq;
>
> /*
> - * Create the rpciod thread and wait for it to start.
> + * Both workqueues are freezable so that no RPC is transmitted, and
> + * no connection is made, between taking a hibernation snapshot and
> + * powering off. Anything sent then changes the server's state but is
> + * missing from the image, and is forgotten when the image is restored.
Let's leave the original comment alone, please.
> */
> - wq = alloc_workqueue("rpciod", WQ_MEM_RECLAIM | WQ_UNBOUND, 0);
> + wq = alloc_workqueue("rpciod",
> + WQ_MEM_RECLAIM | WQ_UNBOUND | WQ_FREEZABLE, 0);
Note that this change is going to conflict with Chuck's "NFS: Set WQ_SYSFS on nfsiod"
patch that I'm expecting to go into v7.5: https://lore.kernel.org/linux-nfs/20260915-performance-v3-3-ae26d460bfd3@kernel.org/
Thanks,
Anna
> if (!wq)
> goto out_failed;
> rpciod_workqueue = wq;
> - wq = alloc_workqueue("xprtiod", WQ_UNBOUND | WQ_MEM_RECLAIM, 0);
> + wq = alloc_workqueue("xprtiod",
> + WQ_UNBOUND | WQ_MEM_RECLAIM | WQ_FREEZABLE, 0);
> if (!wq)
> goto free_rpciod;
> xprtiod_workqueue = wq;
> --
> 2.53.0
next prev parent reply other threads:[~2026-11-16 19:32 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-16 13:14 [PATCH v2 0/3] SUNRPC: keep NFS client connections consistent across hibernation Andrew Pam
2026-11-16 13:14 ` [PATCH v2 1/3] SUNRPC: freeze rpciod and xprtiod across system sleep Andrew Pam
2026-11-16 19:32 ` Anna Schumaker [this message]
2026-11-16 13:14 ` [PATCH v2 2/3] NFSv4: run lease renewal on a freezable workqueue Andrew Pam
2026-11-16 13:14 ` [PATCH v2 3/3] SUNRPC: reconnect client transports after resuming from hibernation Andrew Pam
2026-11-16 19:34 ` Anna Schumaker
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=0d9f6196-6737-4936-a8fd-97192d96ad3f@app.fastmail.com \
--to=anna@kernel.org \
--cc=Dai.Ngo@oracle.com \
--cc=andrew@sericyb.com.au \
--cc=cel@kernel.org \
--cc=jlayton@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-nfs@vger.kernel.org \
--cc=linux-pm@vger.kernel.org \
--cc=neil@brown.name \
--cc=netdev@vger.kernel.org \
--cc=okorniev@redhat.com \
--cc=rafael@kernel.org \
--cc=snitzer@kernel.org \
--cc=tom@talpey.com \
--cc=trondmy@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox