Linux NFS development
 help / color / mirror / Atom feed
* non-stop kworker NFS/RPC write traffic even after unmount
@ 2024-12-15 12:38 Rik Theys
  2024-12-16  0:34 ` Trond Myklebust
  2025-04-01 12:05 ` Daniel Kobras
  0 siblings, 2 replies; 17+ messages in thread
From: Rik Theys @ 2024-12-15 12:38 UTC (permalink / raw)
  To: Linux Nfs

Hi,

We are experiencing an issue on our Rocky 9 NFS server and Rocky 8, 
Rocky 9 and Fedora 41 clients.

The server is (now) running upstream Linux 6.11.11 and the Fedora 41 
clients are running the Fedora 6.11.11 kernel. The Rocky 8 and 9 
machines are running the latest Rocky 8/9 kernels.

Suddenly, a number of clients start to send an abnormal amount of NFS 
traffic to the server that saturates their link and never seems to stop. 
Running iotop on the clients shows kworker-{rpciod,nfsiod,xprtiod} 
processes generating the write traffic. On the server side, the system 
seems to process the traffic as the disks are processing the write requests.

This behavior continues even after stopping all user processes on the 
clients and unmounting the NFS mount on the client. Is this normal? I 
was under the impression that once the NFS mount is unmounted no further 
traffic to the server should be visible?

Not all clients seem to trigger this issue. On a Fedora 41 client that 
(auto)mounts home directories from the NFS server the behavior seems to 
be triggered when I start Thunderbird and let it process a lot of new 
mail (mail from the IMAP server is stored in the thunderbird cache 
that's stored in the nfs-mounted home directory). This triggers the high 
write traffic of the kworker threads. At first, thunderbird behaves 
normally but gets really slow over time. Stopping thunderbird does not 
stop the kworker threads and they keep sending a lot of traffic to the 
server.

Can you point me to some steps to further diagnose this? Where can I 
find what triggers the creation of these kworker threads? Why does iotop 
show the write traffic with these threads, and not the thunderbird threads?

There haven't been many changes to our kernels on the Rocky side 
recently. Is it possible a Fedora 41 client running a more recent kernel 
somehow triggers a behavior on the server that results in Rocky clients 
to start to misbehave?

Regards,

Rik

-- 
Rik Theys
System Engineer
KU Leuven - Dept. Elektrotechniek (ESAT)
Kasteelpark Arenberg 10 bus 2440  - B-3001 Leuven-Heverlee
+32(0)16/32.11.07
----------------------------------------------------------------
<<Any errors in spelling, tact or fact are transmission errors>>


^ permalink raw reply	[flat|nested] 17+ messages in thread

end of thread, other threads:[~2025-05-16 13:35 UTC | newest]

Thread overview: 17+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-12-15 12:38 non-stop kworker NFS/RPC write traffic even after unmount Rik Theys
2024-12-16  0:34 ` Trond Myklebust
2024-12-16  7:35   ` Rik Theys
2024-12-16  8:49   ` Rik Theys
2025-04-01 12:05 ` Daniel Kobras
2025-04-01 12:15   ` Rik Theys
2025-04-01 12:20     ` Rik Theys
2025-04-18 13:31     ` Daniel Kobras
2025-05-16  5:51       ` Rik Theys
2025-05-16  6:17         ` Rik Theys
2025-05-16  9:47           ` Rik Theys
2025-05-16 11:32             ` Rik Theys
2025-05-16 12:19               ` Chuck Lever
2025-05-16 12:36                 ` Rik Theys
2025-05-16 12:59                   ` Chuck Lever
2025-05-16 13:09                     ` Rik Theys
2025-05-16 13:33                       ` Chuck Lever

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox