NFS data corruption on congested network

[Jacek Tomaka <Jacek.Tomaka@poczta.fm>]

Hello,
I ran into an issue where the NFS file ends up being corrupted on disk. We started noticing it on certain, quite old hardware after upgrading OS from Centos 6 to Rocky 9.2. We do see it on Rocky 9.3 but not on 9.1.

After some investigation we have reasons to believe that the change was introduced by the following commit: 
https://github.com/torvalds/linux/commit/6df25e58532be7a4cd6fb15bcd85805947402d91

We write a number of files on a single thread. Each file is up to 4GB. Before closing we call fdatasync. Sometimes the file ends up being corrupted. The corruptions is in a form of a number ( more than 3k pages in one case) of zero filled pages.
When this happens the file cannot be deleted from the client machine which created the file, even when the process which wrote the file completed successfully.

The machines have about 128GB of memory, i think and probably network that leaves to be desired.

My reproducer is currently tied up to our internal software, but i suspect setting the write_congested flag randomly should allow to reproduce the issue.

Regards.
Jacek Tomaka