Re: non-stop kworker NFS/RPC write traffic even after unmount

[Daniel Kobras <kobras@puzzle-itc.de>]

Hi Rik!

Am 01.04.25 um 14:15 schrieb Rik Theys:
> On 4/1/25 2:05 PM, Daniel Kobras wrote:
>> Am 15.12.24 um 13:38 schrieb Rik Theys:
>>> Suddenly, a number of clients start to send an abnormal amount of NFS 
>>> traffic to the server that saturates their link and never seems to 
>>> stop. Running iotop on the clients shows kworker- 
>>> {rpciod,nfsiod,xprtiod} processes generating the write traffic. On 
>>> the server side, the system seems to process the traffic as the disks 
>>> are processing the write requests.
>>>
>>> This behavior continues even after stopping all user processes on the 
>>> clients and unmounting the NFS mount on the client. Is this normal? I 
>>> was under the impression that once the NFS mount is unmounted no 
>>> further traffic to the server should be visible?
>>
>> I'm currently looking at an issue that resembles your description 
>> above (excess traffic to the server for data that was already written 
>> and committed), and part of the packet capture also looks roughly 
>> similar to what you've sent in a followup. Before I dig any deeper: 
>> Did you manage to pinpoint or resolve the problem in the meantime?
> 
> Our server is currently running the 6.12 LTS kernel and we haven't had 
> this specific issue any more. But we were never able to reproduce it, so 
> unfortunately I can't say for sure if it's fixed, or what fixed it :-/.

Thanks for the update! Indeed, in the meantime the affected environment 
here stopped showing the reported behavior as well after a few days, and 
I don't have a clear indication what might have been the fix, either.

When the issue still occurred, it could (once) be provoked by dd'ing 4GB 
of /dev/zero to a test file on an NFSv4.2 mount. The network trace shows 
that the file is completely written at wire speed. But after a five 
second pause, the client then starts sending the same file again in 
smaller chunks of a few hundred MB at five second intervals. So it 
appears that the file's pages are background-flushed to storage again, 
even though they've already been written out. On the NFS layer, none of 
the passes look conspicuous to me: WRITE and COMMIT operations all get 
NFS4_OK'ed by the server.

> Which kernel version(s) are your server and clients running?

The systems in the affected environment run Debian-packaged kernels. The 
servers are on Debian's 6.1.0-32 which corresponds to upstream's 
6.1.129. The issues was seen on clients running the same kernel version, 
but also on older systems running Debian's 5.10.0-33, corresponding to 
5.10.226 upstream. I've skimmed the list of patches that went into 
either of these kernel versions, but nothing stood out as clearly related.

Kind regards,

Daniel
-- 
Daniel Kobras
Principal Architect
Puzzle ITC Deutschland
+49 7071 14316 0
www.puzzle-itc.de

-- 
Puzzle ITC Deutschland GmbH
Sitz der Gesellschaft: Eisenbahnstraße 1, 72072 
Tübingen

Eingetragen am Amtsgericht Stuttgart HRB 765802
Geschäftsführer: 
Lukas Kallies, Daniel Kobras, Mark Pröhl