parts of pages on NFS being replaced by swaths of NULs

parts of pages on NFS being replaced by swaths of NULs

[Jeff Layton]

[-- Attachment #1: Type: text/plain, Size: 7625 bytes --]

We've been seeing a rather nasty bit of data corruption with NFS in our
environment. The clients in this env run a patched v6.9 kernel (mostly
due to GPU driver requirements). Most of the patches are NFS
containerization fixes.

The workload is python scripts writing JSONL files sequentially using
bog-standard buffered write() calls. We're fairly certain that userland
is not seeking so there should be no gaps in the data written.

The problem is that we see ranges of written files being replaced by
NULs. The length of the file seemingly doesn't change from what it
should be, but a chunk of it will be zeroed-out. Looking at the offsets
of the zeroed out ranges, the front part of one page is fine, but the
data from some random offset in the page to the end of the page is
zeroes.

We have a reproducer but we have to run it in a heavily parallel
configuration to make it happen, so it's evidently a tight race of some
sort.

We've turned up some tracepoints and reproduced this twice. What we see
in both cases is that the client just doesn't write some section of the
file.

In the first trace, there was is a gap of 2201 bytes between these two
writes on the wire:

 kworker/u1038:1-2597138 [106] ..... 46138.516795: nfs_initiate_write: fileid=00:aa:10056165185 fhandle=0x6bd94d55 offset=53248 count=1895 stable=UNSTABLE
 oil-localfs-252-2605046 [163] ..... 46138.551459: nfs_initiate_write: fileid=00:aa:10056165185 fhandle=0x6bd94d55 offset=57344 count=443956 stable=FILE_SYNC

The zeroed-out range is from 55143-57344. At the same time that the
file is growing from 53248 to 55143 (due to sequential write()
activity), the client is kicking off writeback for the range up to
55143. It's issuing 2 writes, one for 0-53248 and one for 53248-55143
(note that I've filtered out all but one of the DS filehandles for
brevity):

 oil-localfs-252-2605046 [162] ..... 46138.516414: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=49152 newsize=50130
 oil-localfs-252-2605046 [162] ..... 46138.516593: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=50130 newsize=53248
 kworker/u1038:1-2597138 [106] ..... 46138.516740: nfs_initiate_write: fileid=00:aa:10056165185 fhandle=0x6bd94d55 offset=0 count=53248 stable=UNSTABLE
 oil-localfs-252-2605046 [162] ..... 46138.516753: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=53248 newsize=55143
 kworker/u1038:1-2597138 [106] ..... 46138.516795: nfs_initiate_write: fileid=00:aa:10056165185 fhandle=0x6bd94d55 offset=53248 count=1895 stable=UNSTABLE
 kworker/u1037:2-2871862 [097] ..... 46138.517659: nfs4_pnfs_write: error=0 (OK) fileid=00:aa:10056165185 fhandle=0x6bd94d55 offset=0 count=53248 res=53248 stateid=1:0x79a9c471 layoutstateid=1:0xcbd8aaad
 kworker/u1037:2-2871862 [097] ..... 46138.517662: nfs_writeback_done: error=53248 fileid=00:aa:10056165185 fhandle=0x6bd94d55 offset=0 count=53248 res=53248 stable=UNSTABLE verifier=5199cdae2816c899
 kworker/u1037:5-2593935 [226] ..... 46138.517669: nfs4_pnfs_write: error=0 (OK) fileid=00:aa:10056165185 fhandle=0x6bd94d55 offset=53248 count=1895 res=1895 stateid=1:0x79a9c471 layoutstateid=1:0xcbd8aaad
 kworker/u1037:5-2593935 [226] ..... 46138.517672: nfs_writeback_done: error=1895 fileid=00:aa:10056165185 fhandle=0x6bd94d55 offset=53248 count=1895 res=1895 stable=UNSTABLE verifier=5199cdae2816c899
 oil-localfs-252-2605046 [162] ..... 46138.518360: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=55143 newsize=57344
 oil-localfs-252-2605046 [162] ..... 46138.518556: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=57344 newsize=60156

...and just after writeback completes, we see the file size grow from
55143 to the end of the page (57344).

The second trace has similar symptoms. There is a lot more (smaller)
write activity (due to memory pressure?). There is a gap of 3791 bytes
between these on-the-wire writes, however:

 kworker/u1036:0-2339252 [217] ..... 479572.054622: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232 offset=221184 count=4401 stable=UNSTABLE
 kworker/u1030:1-2297876 [042] ..... 479572.074194: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232 offset=229376 count=261898 stable=UNSTABLE

Same situation -- the at page at offset 53248 has 305 bytes on it, and
the remaining is zeroed. This trace shows similar racing write() and
writeback activity as in Friday's trace. At around the same time as the
client was growing the file over the affected range, writeback was
kicking off for everything up to the affected range (this has some
other wb related calls filtered for brevity):

  oil-localfs-86-727850  [215] ..... 479572.053987: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=217088 newsize=220572                                                                     
 kworker/u1036:8-2339326 [088] ..... 479572.054008: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232 offset=217088 count=3484 stable=UNSTABLE                                                                                 
  oil-localfs-86-727850  [215] ..... 479572.054405: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=220572 newsize=221184                                                                     
 kworker/u1036:1-2297875 [217] ..... 479572.054418: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232 offset=220572 count=612 stable=UNSTABLE                                                                                  
  oil-localfs-86-727850  [215] ..... 479572.054581: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=221184 newsize=225280                                                                     
  oil-localfs-86-727850  [215] ..... 479572.054584: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=225280 newsize=225585                                                                     
 kworker/u1036:0-2339252 [217] ..... 479572.054622: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232 offset=221184 count=4401 stable=UNSTABLE                                                                                 
  oil-localfs-86-727850  [215] ..... 479572.054997: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=225585 newsize=229376                                                                     
  oil-localfs-86-727850  [215] ..... 479572.055190: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=229376 newsize=230598                                                                     

Could this be a race between extending an existing dirty page, and
writeback kicking off for the pre-extension range on the page? Maybe
the client is clearing the dirty bit, thinking that the write covers
the dirty range, but it has an outdated idea about what that range is
or doesn't properly check?

Traces for both events, filtered on the relevant fileid are attached.
I've rolled patches for some new tracepoints that I'm going to attempt
to turn up next, but I thought that this was a good point to solicit
ideas.

Happy to entertain other thoughts or patches!
-- 
Jeff Layton <jlayton@kernel.org>

[-- Attachment #2: 20250729.txt --]
[-- Type: text/plain, Size: 166129 bytes --]

  oil-localfs-86-727850  [215] ..... 479572.008843: nfs_getattr_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
  oil-localfs-86-727850  [215] ..... 479572.008844: nfs_getattr_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=0 cache_validity=0x0 () nfs_flags=0x4 (ACL_LRU_SET)
  oil-localfs-86-727850  [215] ..... 479572.008856: nfs_getattr_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
  oil-localfs-86-727850  [215] ..... 479572.008857: nfs_getattr_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=0 cache_validity=0x0 () nfs_flags=0x4 (ACL_LRU_SET)
  oil-localfs-86-727850  [215] ..... 479572.021714: nfs_getattr_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
  oil-localfs-86-727850  [215] ..... 479572.021715: nfs_getattr_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=0 cache_validity=0x0 () nfs_flags=0x4 (ACL_LRU_SET)
  oil-localfs-86-727850  [215] ..... 479572.021722: nfs_getattr_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
  oil-localfs-86-727850  [215] ..... 479572.021722: nfs_getattr_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=0 cache_validity=0x0 () nfs_flags=0x4 (ACL_LRU_SET)
  oil-localfs-86-727850  [215] ..... 479572.032497: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=0 newsize=4096
  oil-localfs-86-727850  [215] ..... 479572.032504: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=4096 newsize=5013
  oil-localfs-86-727850  [215] ..... 479572.032891: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=5013 newsize=8192
  oil-localfs-86-727850  [215] ..... 479572.033185: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=8192 newsize=10026
  oil-localfs-86-727850  [215] ..... 479572.033466: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=10026 newsize=12288
  oil-localfs-86-727850  [215] ..... 479572.033739: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=12288 newsize=15039
  oil-localfs-86-727850  [215] ..... 479572.033988: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=15039 newsize=16384
 kworker/u1027:7-2369635 [023] ..... 479572.034060: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232 offset=0 count=16384 stable=UNSTABLE
 kworker/u1027:7-2369635 [023] ..... 479572.034069: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0x6af4272f offset=0 count=16384 stable=UNSTABLE
 kworker/u1027:7-2369635 [023] ..... 479572.034073: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xf88816fa offset=0 count=16384 stable=UNSTABLE
  oil-localfs-86-727850  [215] ..... 479572.034248: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=16384 newsize=20052
 kworker/u1027:1-2286624 [148] ..... 479572.034332: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232 offset=16384 count=3668 stable=UNSTABLE
 kworker/u1027:1-2286624 [148] ..... 479572.034337: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0x6af4272f offset=16384 count=3668 stable=UNSTABLE
 kworker/u1027:1-2286624 [148] ..... 479572.034340: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xf88816fa offset=16384 count=3668 stable=UNSTABLE
 kworker/u1033:2-2344691 [197] ..... 479572.034425: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0xf88816fa offset=0 count=16384 res=16384 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1033:2-2344691 [197] ..... 479572.034430: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=20052 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:2-2344691 [197] ..... 479572.034432: nfs_writeback_done: error=16384 fileid=00:96:10067193438 fhandle=0xf88816fa offset=0 count=16384 res=16384 stable=UNSTABLE verifier=a5cd34d9a5d3efdb
 kworker/u1037:1-2369493 [230] ..... 479572.034453: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0xc9992232 offset=0 count=16384 res=16384 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1037:1-2369493 [230] ..... 479572.034457: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=20052 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:1-2369493 [230] ..... 479572.034459: nfs_writeback_done: error=16384 fileid=00:96:10067193438 fhandle=0xc9992232 offset=0 count=16384 res=16384 stable=UNSTABLE verifier=e7a66eedc4f3c318
 kworker/u1037:0-2348081 [103] ..... 479572.034466: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0x6af4272f offset=0 count=16384 res=16384 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1037:0-2348081 [103] ..... 479572.034467: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=20052 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:0-2348081 [103] ..... 479572.034468: nfs_writeback_done: error=16384 fileid=00:96:10067193438 fhandle=0x6af4272f offset=0 count=16384 res=16384 stable=UNSTABLE verifier=7fb1042b4dd46a93
 kworker/u1033:6-2344693 [070] ..... 479572.034478: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0xf88816fa offset=16384 count=3668 res=3668 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1033:6-2344693 [070] ..... 479572.034480: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=20052 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:6-2344693 [070] ..... 479572.034481: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1033:6-2344693 [070] ..... 479572.034482: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=20052 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:6-2344693 [070] ..... 479572.034482: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=20052 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:6-2344693 [070] ..... 479572.034483: nfs_writeback_done: error=3668 fileid=00:96:10067193438 fhandle=0xf88816fa offset=16384 count=3668 res=3668 stable=UNSTABLE verifier=a5cd34d9a5d3efdb
 kworker/u1037:4-2348084 [225] ..... 479572.034518: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0xc9992232 offset=16384 count=3668 res=3668 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1037:4-2348084 [225] ..... 479572.034520: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=20052 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:4-2348084 [225] ..... 479572.034521: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1037:4-2348084 [225] ..... 479572.034522: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=20052 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:4-2348084 [225] ..... 479572.034522: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=20052 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:4-2348084 [225] ..... 479572.034523: nfs_writeback_done: error=3668 fileid=00:96:10067193438 fhandle=0xc9992232 offset=16384 count=3668 res=3668 stable=UNSTABLE verifier=e7a66eedc4f3c318
 kworker/u1037:4-2348084 [225] ..... 479572.034526: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0x6af4272f offset=16384 count=3668 res=3668 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1037:4-2348084 [225] ..... 479572.034527: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=20052 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:4-2348084 [225] ..... 479572.034527: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1037:4-2348084 [225] ..... 479572.034527: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=20052 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:4-2348084 [225] ..... 479572.034527: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=20052 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:4-2348084 [225] ..... 479572.034527: nfs_writeback_done: error=3668 fileid=00:96:10067193438 fhandle=0x6af4272f offset=16384 count=3668 res=3668 stable=UNSTABLE verifier=7fb1042b4dd46a93
  oil-localfs-86-727850  [215] ..... 479572.035090: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=20052 newsize=20480
 kworker/u1027:4-2369632 [145] ..... 479572.035142: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232 offset=20052 count=428 stable=UNSTABLE
 kworker/u1027:4-2369632 [145] ..... 479572.035145: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0x6af4272f offset=20052 count=428 stable=UNSTABLE
 kworker/u1027:4-2369632 [145] ..... 479572.035147: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xf88816fa offset=20052 count=428 stable=UNSTABLE
 kworker/u1033:6-2344693 [070] ..... 479572.035252: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0xf88816fa offset=20052 count=428 res=428 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1033:6-2344693 [070] ..... 479572.035253: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=20480 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:6-2344693 [070] ..... 479572.035253: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1033:6-2344693 [070] ..... 479572.035254: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=20480 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:6-2344693 [070] ..... 479572.035254: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=20480 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:6-2344693 [070] ..... 479572.035254: nfs_writeback_done: error=428 fileid=00:96:10067193438 fhandle=0xf88816fa offset=20052 count=428 res=428 stable=UNSTABLE verifier=a5cd34d9a5d3efdb
 kworker/u1037:1-2369493 [230] ..... 479572.035264: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0xc9992232 offset=20052 count=428 res=428 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1037:1-2369493 [230] ..... 479572.035264: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=20480 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:1-2369493 [230] ..... 479572.035265: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1037:1-2369493 [230] ..... 479572.035265: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=20480 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:1-2369493 [230] ..... 479572.035266: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=20480 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:1-2369493 [230] ..... 479572.035266: nfs_writeback_done: error=428 fileid=00:96:10067193438 fhandle=0xc9992232 offset=20052 count=428 res=428 stable=UNSTABLE verifier=e7a66eedc4f3c318
 kworker/u1037:7-2298146 [100] ..... 479572.035267: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0x6af4272f offset=20052 count=428 res=428 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1037:7-2298146 [100] ..... 479572.035268: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=20480 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:7-2298146 [100] ..... 479572.035269: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1037:7-2298146 [100] ..... 479572.035269: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=20480 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:7-2298146 [100] ..... 479572.035269: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=20480 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:7-2298146 [100] ..... 479572.035270: nfs_writeback_done: error=428 fileid=00:96:10067193438 fhandle=0x6af4272f offset=20052 count=428 res=428 stable=UNSTABLE verifier=7fb1042b4dd46a93
  oil-localfs-86-727850  [215] ..... 479572.035342: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=20480 newsize=24576
  oil-localfs-86-727850  [215] ..... 479572.035345: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=24576 newsize=25065
 kworker/u1027:7-2369635 [023] ..... 479572.035385: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232 offset=20480 count=4585 stable=UNSTABLE
 kworker/u1027:7-2369635 [023] ..... 479572.035388: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0x6af4272f offset=20480 count=4585 stable=UNSTABLE
 kworker/u1027:7-2369635 [023] ..... 479572.035389: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xf88816fa offset=20480 count=4585 stable=UNSTABLE
 kworker/u1033:6-2344693 [070] ..... 479572.035536: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0xf88816fa offset=20480 count=4585 res=4585 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1033:6-2344693 [070] ..... 479572.035536: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=25065 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:6-2344693 [070] ..... 479572.035537: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1033:6-2344693 [070] ..... 479572.035537: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=25065 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:6-2344693 [070] ..... 479572.035537: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=25065 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:6-2344693 [070] ..... 479572.035538: nfs_writeback_done: error=4585 fileid=00:96:10067193438 fhandle=0xf88816fa offset=20480 count=4585 res=4585 stable=UNSTABLE verifier=a5cd34d9a5d3efdb
 kworker/u1037:7-2298146 [100] ..... 479572.035558: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0xc9992232 offset=20480 count=4585 res=4585 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1037:7-2298146 [100] ..... 479572.035558: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=25065 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:7-2298146 [100] ..... 479572.035558: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1037:4-2348084 [225] ..... 479572.035559: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0x6af4272f offset=20480 count=4585 res=4585 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1037:7-2298146 [100] ..... 479572.035559: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=25065 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:7-2298146 [100] ..... 479572.035559: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=25065 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:7-2298146 [100] ..... 479572.035559: nfs_writeback_done: error=4585 fileid=00:96:10067193438 fhandle=0xc9992232 offset=20480 count=4585 res=4585 stable=UNSTABLE verifier=e7a66eedc4f3c318
 kworker/u1037:4-2348084 [225] ..... 479572.035559: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=25065 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:4-2348084 [225] ..... 479572.035560: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1037:4-2348084 [225] ..... 479572.035560: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=25065 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:4-2348084 [225] ..... 479572.035560: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=25065 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:4-2348084 [225] ..... 479572.035560: nfs_writeback_done: error=4585 fileid=00:96:10067193438 fhandle=0x6af4272f offset=20480 count=4585 res=4585 stable=UNSTABLE verifier=7fb1042b4dd46a93
  oil-localfs-86-727850  [215] ..... 479572.035904: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=25065 newsize=28672
 kworker/u1027:1-2369517 [146] ..... 479572.035932: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232 offset=24576 count=4096 stable=UNSTABLE
 kworker/u1027:1-2369517 [146] ..... 479572.035935: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0x6af4272f offset=24576 count=4096 stable=UNSTABLE
 kworker/u1027:1-2369517 [146] ..... 479572.035936: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xf88816fa offset=24576 count=4096 stable=UNSTABLE
 kworker/u1037:4-2348084 [225] ..... 479572.036041: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0xc9992232 offset=24576 count=4096 res=4096 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1037:4-2348084 [225] ..... 479572.036041: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=28672 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:4-2348084 [225] ..... 479572.036042: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1037:4-2348084 [225] ..... 479572.036042: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=28672 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:4-2348084 [225] ..... 479572.036042: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=28672 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:4-2348084 [225] ..... 479572.036043: nfs_writeback_done: error=4096 fileid=00:96:10067193438 fhandle=0xc9992232 offset=24576 count=4096 res=4096 stable=UNSTABLE verifier=e7a66eedc4f3c318
 kworker/u1033:6-2344693 [070] ..... 479572.036045: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0xf88816fa offset=24576 count=4096 res=4096 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1033:6-2344693 [070] ..... 479572.036045: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=28672 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:6-2344693 [070] ..... 479572.036046: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1033:6-2344693 [070] ..... 479572.036046: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=28672 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:6-2344693 [070] ..... 479572.036046: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=28672 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:6-2344693 [070] ..... 479572.036047: nfs_writeback_done: error=4096 fileid=00:96:10067193438 fhandle=0xf88816fa offset=24576 count=4096 res=4096 stable=UNSTABLE verifier=a5cd34d9a5d3efdb
 kworker/u1037:4-2348084 [225] ..... 479572.036052: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0x6af4272f offset=24576 count=4096 res=4096 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1037:4-2348084 [225] ..... 479572.036053: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=28672 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:4-2348084 [225] ..... 479572.036053: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1037:4-2348084 [225] ..... 479572.036053: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=28672 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:4-2348084 [225] ..... 479572.036053: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=28672 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:4-2348084 [225] ..... 479572.036053: nfs_writeback_done: error=4096 fileid=00:96:10067193438 fhandle=0x6af4272f offset=24576 count=4096 res=4096 stable=UNSTABLE verifier=7fb1042b4dd46a93
  oil-localfs-86-727850  [215] ..... 479572.036116: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=28672 newsize=30078
 kworker/u1027:1-2392846 [150] ..... 479572.036153: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232 offset=28672 count=1406 stable=UNSTABLE
 kworker/u1027:1-2392846 [150] ..... 479572.036156: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0x6af4272f offset=28672 count=1406 stable=UNSTABLE
 kworker/u1027:1-2392846 [150] ..... 479572.036158: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xf88816fa offset=28672 count=1406 stable=UNSTABLE
 kworker/u1033:2-2344691 [197] ..... 479572.036275: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0xf88816fa offset=28672 count=1406 res=1406 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1033:2-2344691 [197] ..... 479572.036276: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=30078 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:2-2344691 [197] ..... 479572.036277: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1033:2-2344691 [197] ..... 479572.036277: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=30078 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:2-2344691 [197] ..... 479572.036277: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=30078 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:2-2344691 [197] ..... 479572.036277: nfs_writeback_done: error=1406 fileid=00:96:10067193438 fhandle=0xf88816fa offset=28672 count=1406 res=1406 stable=UNSTABLE verifier=a5cd34d9a5d3efdb
 kworker/u1037:7-2298146 [100] ..... 479572.036318: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0xc9992232 offset=28672 count=1406 res=1406 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1037:7-2298146 [100] ..... 479572.036319: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=30078 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:7-2298146 [100] ..... 479572.036319: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1037:7-2298146 [100] ..... 479572.036320: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=30078 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:7-2298146 [100] ..... 479572.036320: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=30078 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:7-2298146 [100] ..... 479572.036320: nfs_writeback_done: error=1406 fileid=00:96:10067193438 fhandle=0xc9992232 offset=28672 count=1406 res=1406 stable=UNSTABLE verifier=e7a66eedc4f3c318
 kworker/u1037:7-2298146 [100] ..... 479572.036387: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0x6af4272f offset=28672 count=1406 res=1406 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1037:7-2298146 [100] ..... 479572.036389: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=30078 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:7-2298146 [100] ..... 479572.036389: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1037:7-2298146 [100] ..... 479572.036390: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=30078 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:7-2298146 [100] ..... 479572.036390: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=30078 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:7-2298146 [100] ..... 479572.036390: nfs_writeback_done: error=1406 fileid=00:96:10067193438 fhandle=0x6af4272f offset=28672 count=1406 res=1406 stable=UNSTABLE verifier=7fb1042b4dd46a93
  oil-localfs-86-727850  [215] ..... 479572.036835: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=30078 newsize=32768
  oil-localfs-86-727850  [215] ..... 479572.037058: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=32768 newsize=35091
  oil-localfs-86-727850  [215] ..... 479572.037269: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=35091 newsize=36864
  oil-localfs-86-727850  [215] ..... 479572.037481: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=36864 newsize=40104
  oil-localfs-86-727850  [215] ..... 479572.037688: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=40104 newsize=40960
  oil-localfs-86-727850  [215] ..... 479572.037872: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=40960 newsize=45056
  oil-localfs-86-727850  [215] ..... 479572.037875: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=45056 newsize=45117
  oil-localfs-86-727850  [215] ..... 479572.038094: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=45117 newsize=49152
  oil-localfs-86-727850  [215] ..... 479572.038283: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=49152 newsize=50130
  oil-localfs-86-727850  [215] ..... 479572.038486: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=50130 newsize=53248
  oil-localfs-86-727850  [215] ..... 479572.038701: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=53248 newsize=55143
  oil-localfs-86-727850  [215] ..... 479572.038916: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=55143 newsize=57344
  oil-localfs-86-727850  [215] ..... 479572.039094: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=57344 newsize=60156
  oil-localfs-86-727850  [215] ..... 479572.039298: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=60156 newsize=61440
  oil-localfs-86-727850  [215] ..... 479572.039521: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=61440 newsize=65169
  oil-localfs-86-727850  [215] ..... 479572.039721: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=65169 newsize=65536
  oil-localfs-86-727850  [215] ..... 479572.039922: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=65536 newsize=69632
  oil-localfs-86-727850  [215] ..... 479572.039924: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=69632 newsize=70182
  oil-localfs-86-727850  [215] ..... 479572.040135: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=70182 newsize=73728
  oil-localfs-86-727850  [215] ..... 479572.040301: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=73728 newsize=75195
  oil-localfs-86-727850  [215] ..... 479572.040525: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=75195 newsize=77824
  oil-localfs-86-727850  [215] ..... 479572.040723: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=77824 newsize=80208
  oil-localfs-86-727850  [215] ..... 479572.040923: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=80208 newsize=81920
  oil-localfs-86-727850  [215] ..... 479572.041122: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=81920 newsize=85221
  oil-localfs-86-727850  [215] ..... 479572.041324: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=85221 newsize=86016
  oil-localfs-86-727850  [215] ..... 479572.041512: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=86016 newsize=90112
  oil-localfs-86-727850  [215] ..... 479572.041514: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=90112 newsize=90234
  oil-localfs-86-727850  [215] ..... 479572.041731: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=90234 newsize=94208
  oil-localfs-86-727850  [215] ..... 479572.041923: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=94208 newsize=95247
  oil-localfs-86-727850  [215] ..... 479572.042116: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=95247 newsize=98304
  oil-localfs-86-727850  [215] ..... 479572.042308: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=98304 newsize=100260
  oil-localfs-86-727850  [215] ..... 479572.042522: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=100260 newsize=102400
  oil-localfs-86-727850  [215] ..... 479572.042711: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=102400 newsize=105273
  oil-localfs-86-727850  [215] ..... 479572.042918: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=105273 newsize=106496
  oil-localfs-86-727850  [215] ..... 479572.043105: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=106496 newsize=110286
  oil-localfs-86-727850  [215] ..... 479572.043292: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=110286 newsize=110592
  oil-localfs-86-727850  [215] ..... 479572.043505: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=110592 newsize=114688
  oil-localfs-86-727850  [215] ..... 479572.043507: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=114688 newsize=115299
  oil-localfs-86-727850  [215] ..... 479572.043738: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=115299 newsize=118784
  oil-localfs-86-727850  [215] ..... 479572.043925: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=118784 newsize=120312
  oil-localfs-86-727850  [215] ..... 479572.044130: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=120312 newsize=122880
  oil-localfs-86-727850  [215] ..... 479572.044327: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=122880 newsize=125325
  oil-localfs-86-727850  [215] ..... 479572.044533: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=125325 newsize=126976
  oil-localfs-86-727850  [215] ..... 479572.044734: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=126976 newsize=130338
  oil-localfs-86-727850  [215] ..... 479572.044931: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=130338 newsize=131072
  oil-localfs-86-727850  [215] ..... 479572.045105: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=131072 newsize=135168
  oil-localfs-86-727850  [215] ..... 479572.045107: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=135168 newsize=135351
  oil-localfs-86-727850  [215] ..... 479572.045322: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=135351 newsize=139264
  oil-localfs-86-727850  [215] ..... 479572.045523: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=139264 newsize=140364
  oil-localfs-86-727850  [215] ..... 479572.045729: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=140364 newsize=143360
  oil-localfs-86-727850  [215] ..... 479572.045936: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=143360 newsize=145377
  oil-localfs-86-727850  [215] ..... 479572.046143: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=145377 newsize=147456
  oil-localfs-86-727850  [215] ..... 479572.046329: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=147456 newsize=150390
  oil-localfs-86-727850  [215] ..... 479572.046548: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=150390 newsize=151552
  oil-localfs-86-727850  [215] ..... 479572.046736: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=151552 newsize=155403
  oil-localfs-86-727850  [215] ..... 479572.046915: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=155403 newsize=155648
  oil-localfs-86-727850  [215] ..... 479572.047096: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=155648 newsize=159744
  oil-localfs-86-727850  [215] ..... 479572.047098: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=159744 newsize=160416
  oil-localfs-86-727850  [215] ..... 479572.047295: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=160416 newsize=163840
  oil-localfs-86-727850  [215] ..... 479572.047477: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=163840 newsize=165429
 kworker/u1036:9-2181405 [092] ..... 479572.047561: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232 offset=28672 count=136757 stable=UNSTABLE
 kworker/u1036:9-2181405 [092] ..... 479572.047570: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0x6af4272f offset=28672 count=136757 stable=UNSTABLE
 kworker/u1036:9-2181405 [092] ..... 479572.047576: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xf88816fa offset=28672 count=136757 stable=UNSTABLE
 kworker/u1033:2-2344691 [197] ..... 479572.047942: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0xf88816fa offset=28672 count=136757 res=136757 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1033:2-2344691 [197] ..... 479572.047945: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=165429 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:2-2344691 [197] ..... 479572.047946: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1033:2-2344691 [197] ..... 479572.047947: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=165429 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:2-2344691 [197] ..... 479572.047947: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=165429 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:2-2344691 [197] ..... 479572.047947: nfs_writeback_done: error=136757 fileid=00:96:10067193438 fhandle=0xf88816fa offset=28672 count=136757 res=136757 stable=UNSTABLE verifier=a5cd34d9a5d3efdb
 kworker/u1037:7-2298146 [100] ..... 479572.048038: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0x6af4272f offset=28672 count=136757 res=136757 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1037:7-2298146 [100] ..... 479572.048040: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=165429 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:7-2298146 [100] ..... 479572.048041: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1037:7-2298146 [100] ..... 479572.048042: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=165429 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:7-2298146 [100] ..... 479572.048042: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=165429 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:7-2298146 [100] ..... 479572.048042: nfs_writeback_done: error=136757 fileid=00:96:10067193438 fhandle=0x6af4272f offset=28672 count=136757 res=136757 stable=UNSTABLE verifier=7fb1042b4dd46a93
 kworker/u1037:7-2298146 [100] ..... 479572.048174: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0xc9992232 offset=28672 count=136757 res=136757 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1037:7-2298146 [100] ..... 479572.048175: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=165429 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:7-2298146 [100] ..... 479572.048175: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1037:7-2298146 [100] ..... 479572.048175: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=165429 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:7-2298146 [100] ..... 479572.048175: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=165429 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:7-2298146 [100] ..... 479572.048176: nfs_writeback_done: error=136757 fileid=00:96:10067193438 fhandle=0xc9992232 offset=28672 count=136757 res=136757 stable=UNSTABLE verifier=e7a66eedc4f3c318
  oil-localfs-86-727850  [215] ..... 479572.048629: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=165429 newsize=167936
 kworker/u1036:1-2339253 [216] ..... 479572.048651: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232 offset=165429 count=2507 stable=UNSTABLE
 kworker/u1036:1-2339253 [216] ..... 479572.048654: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0x6af4272f offset=165429 count=2507 stable=UNSTABLE
 kworker/u1036:1-2339253 [216] ..... 479572.048655: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xf88816fa offset=165429 count=2507 stable=UNSTABLE
 kworker/u1037:7-2298146 [100] ..... 479572.048770: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0x6af4272f offset=165429 count=2507 res=2507 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1037:7-2298146 [100] ..... 479572.048771: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=167936 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:7-2298146 [100] ..... 479572.048771: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1037:7-2298146 [100] ..... 479572.048771: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=167936 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:7-2298146 [100] ..... 479572.048771: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=167936 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:7-2298146 [100] ..... 479572.048772: nfs_writeback_done: error=2507 fileid=00:96:10067193438 fhandle=0x6af4272f offset=165429 count=2507 res=2507 stable=UNSTABLE verifier=7fb1042b4dd46a93
 kworker/u1037:1-2369493 [230] ..... 479572.048778: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0xc9992232 offset=165429 count=2507 res=2507 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1037:1-2369493 [230] ..... 479572.048779: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=167936 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:1-2369493 [230] ..... 479572.048779: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1037:1-2369493 [230] ..... 479572.048779: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=167936 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:1-2369493 [230] ..... 479572.048779: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=167936 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:1-2369493 [230] ..... 479572.048779: nfs_writeback_done: error=2507 fileid=00:96:10067193438 fhandle=0xc9992232 offset=165429 count=2507 res=2507 stable=UNSTABLE verifier=e7a66eedc4f3c318
 kworker/u1033:6-2344693 [070] ..... 479572.048796: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0xf88816fa offset=165429 count=2507 res=2507 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1033:6-2344693 [070] ..... 479572.048797: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=167936 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:6-2344693 [070] ..... 479572.048797: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1033:6-2344693 [070] ..... 479572.048798: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=167936 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:6-2344693 [070] ..... 479572.048798: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=167936 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:6-2344693 [070] ..... 479572.048798: nfs_writeback_done: error=2507 fileid=00:96:10067193438 fhandle=0xf88816fa offset=165429 count=2507 res=2507 stable=UNSTABLE verifier=a5cd34d9a5d3efdb
  oil-localfs-86-727850  [215] ..... 479572.048828: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=167936 newsize=170442
 kworker/u1036:7-2339325 [220] ..... 479572.048836: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232 offset=167936 count=2506 stable=UNSTABLE
 kworker/u1036:7-2339325 [220] ..... 479572.048839: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0x6af4272f offset=167936 count=2506 stable=UNSTABLE
 kworker/u1036:7-2339325 [220] ..... 479572.048840: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xf88816fa offset=167936 count=2506 stable=UNSTABLE
 kworker/u1037:4-2348084 [225] ..... 479572.048959: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0x6af4272f offset=167936 count=2506 res=2506 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1037:4-2348084 [225] ..... 479572.048960: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=170442 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:4-2348084 [225] ..... 479572.048960: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1037:4-2348084 [225] ..... 479572.048961: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=170442 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:4-2348084 [225] ..... 479572.048961: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=170442 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:4-2348084 [225] ..... 479572.048961: nfs_writeback_done: error=2506 fileid=00:96:10067193438 fhandle=0x6af4272f offset=167936 count=2506 res=2506 stable=UNSTABLE verifier=7fb1042b4dd46a93
 kworker/u1037:1-2369493 [230] ..... 479572.049095: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0xc9992232 offset=167936 count=2506 res=2506 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1037:1-2369493 [230] ..... 479572.049096: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=170442 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:1-2369493 [230] ..... 479572.049096: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1037:1-2369493 [230] ..... 479572.049096: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=170442 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:1-2369493 [230] ..... 479572.049097: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=170442 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:1-2369493 [230] ..... 479572.049097: nfs_writeback_done: error=2506 fileid=00:96:10067193438 fhandle=0xc9992232 offset=167936 count=2506 res=2506 stable=UNSTABLE verifier=e7a66eedc4f3c318
 kworker/u1033:6-2344693 [070] ..... 479572.049131: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0xf88816fa offset=167936 count=2506 res=2506 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1033:6-2344693 [070] ..... 479572.049132: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=170442 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:6-2344693 [070] ..... 479572.049132: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1033:6-2344693 [070] ..... 479572.049133: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=170442 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:6-2344693 [070] ..... 479572.049133: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=170442 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:6-2344693 [070] ..... 479572.049133: nfs_writeback_done: error=2506 fileid=00:96:10067193438 fhandle=0xf88816fa offset=167936 count=2506 res=2506 stable=UNSTABLE verifier=a5cd34d9a5d3efdb
  oil-localfs-86-727850  [215] ..... 479572.049579: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=170442 newsize=172032
 kworker/u1036:1-2369512 [218] ..... 479572.049590: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232 offset=167936 count=4096 stable=UNSTABLE
 kworker/u1036:1-2369512 [218] ..... 479572.049593: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0x6af4272f offset=167936 count=4096 stable=UNSTABLE
 kworker/u1036:1-2369512 [218] ..... 479572.049595: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xf88816fa offset=167936 count=4096 stable=UNSTABLE
 kworker/u1037:7-2298146 [100] ..... 479572.049708: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0x6af4272f offset=167936 count=4096 res=4096 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1037:7-2298146 [100] ..... 479572.049709: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=172032 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:7-2298146 [100] ..... 479572.049709: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1037:7-2298146 [100] ..... 479572.049710: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=172032 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:7-2298146 [100] ..... 479572.049710: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=172032 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:7-2298146 [100] ..... 479572.049711: nfs_writeback_done: error=4096 fileid=00:96:10067193438 fhandle=0x6af4272f offset=167936 count=4096 res=4096 stable=UNSTABLE verifier=7fb1042b4dd46a93
 kworker/u1037:7-2298146 [100] ..... 479572.049724: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0xc9992232 offset=167936 count=4096 res=4096 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1037:7-2298146 [100] ..... 479572.049725: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=172032 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:7-2298146 [100] ..... 479572.049725: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1037:7-2298146 [100] ..... 479572.049725: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=172032 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:7-2298146 [100] ..... 479572.049725: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=172032 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:7-2298146 [100] ..... 479572.049725: nfs_writeback_done: error=4096 fileid=00:96:10067193438 fhandle=0xc9992232 offset=167936 count=4096 res=4096 stable=UNSTABLE verifier=e7a66eedc4f3c318
 kworker/u1033:6-2344693 [070] ..... 479572.049728: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0xf88816fa offset=167936 count=4096 res=4096 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1033:6-2344693 [070] ..... 479572.049729: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=172032 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:6-2344693 [070] ..... 479572.049730: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1033:6-2344693 [070] ..... 479572.049731: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=172032 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:6-2344693 [070] ..... 479572.049731: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=172032 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:6-2344693 [070] ..... 479572.049731: nfs_writeback_done: error=4096 fileid=00:96:10067193438 fhandle=0xf88816fa offset=167936 count=4096 res=4096 stable=UNSTABLE verifier=a5cd34d9a5d3efdb
  oil-localfs-86-727850  [215] ..... 479572.049779: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=172032 newsize=175455
 kworker/u1036:1-2339253 [216] ..... 479572.049794: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232 offset=172032 count=3423 stable=UNSTABLE
 kworker/u1036:1-2339253 [216] ..... 479572.049796: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0x6af4272f offset=172032 count=3423 stable=UNSTABLE
 kworker/u1036:1-2339253 [216] ..... 479572.049800: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xf88816fa offset=172032 count=3423 stable=UNSTABLE
 kworker/u1037:7-2298146 [100] ..... 479572.049943: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0xc9992232 offset=172032 count=3423 res=3423 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1037:7-2298146 [100] ..... 479572.049944: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=175455 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:7-2298146 [100] ..... 479572.049944: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1037:7-2298146 [100] ..... 479572.049944: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=175455 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:7-2298146 [100] ..... 479572.049945: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=175455 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:7-2298146 [100] ..... 479572.049945: nfs_writeback_done: error=3423 fileid=00:96:10067193438 fhandle=0xc9992232 offset=172032 count=3423 res=3423 stable=UNSTABLE verifier=e7a66eedc4f3c318
 kworker/u1033:1-2209763 [064] ..... 479572.049954: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0xf88816fa offset=172032 count=3423 res=3423 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1033:1-2209763 [064] ..... 479572.049955: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=175455 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:1-2209763 [064] ..... 479572.049955: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1033:1-2209763 [064] ..... 479572.049956: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=175455 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:1-2209763 [064] ..... 479572.049956: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=175455 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:1-2209763 [064] ..... 479572.049956: nfs_writeback_done: error=3423 fileid=00:96:10067193438 fhandle=0xf88816fa offset=172032 count=3423 res=3423 stable=UNSTABLE verifier=a5cd34d9a5d3efdb
 kworker/u1037:7-2298146 [100] ..... 479572.049976: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0x6af4272f offset=172032 count=3423 res=3423 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1037:7-2298146 [100] ..... 479572.049976: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=175455 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:7-2298146 [100] ..... 479572.049977: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1037:7-2298146 [100] ..... 479572.049977: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=175455 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:7-2298146 [100] ..... 479572.049977: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=175455 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:7-2298146 [100] ..... 479572.049977: nfs_writeback_done: error=3423 fileid=00:96:10067193438 fhandle=0x6af4272f offset=172032 count=3423 res=3423 stable=UNSTABLE verifier=7fb1042b4dd46a93
  oil-localfs-86-727850  [215] ..... 479572.050224: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=175455 newsize=176128
  oil-localfs-86-727850  [215] ..... 479572.050391: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=176128 newsize=180224
  oil-localfs-86-727850  [215] ..... 479572.050393: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=180224 newsize=180468
  oil-localfs-86-727850  [215] ..... 479572.050619: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=180468 newsize=184320
  oil-localfs-86-727850  [215] ..... 479572.050797: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=184320 newsize=185481
  oil-localfs-86-727850  [215] ..... 479572.050984: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=185481 newsize=188416
  oil-localfs-86-727850  [215] ..... 479572.051172: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=188416 newsize=190494
  oil-localfs-86-727850  [215] ..... 479572.051363: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=190494 newsize=192512
  oil-localfs-86-727850  [215] ..... 479572.051529: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=192512 newsize=195507
  oil-localfs-86-727850  [215] ..... 479572.051716: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=195507 newsize=196608
  oil-localfs-86-727850  [215] ..... 479572.051891: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=196608 newsize=200520
  oil-localfs-86-727850  [215] ..... 479572.052054: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=200520 newsize=200704
  oil-localfs-86-727850  [215] ..... 479572.052238: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=200704 newsize=204800
  oil-localfs-86-727850  [215] ..... 479572.052240: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=204800 newsize=205533
  oil-localfs-86-727850  [215] ..... 479572.052448: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=205533 newsize=208896
 kworker/u1027:1-2286620 [017] ..... 479572.052495: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232 offset=175455 count=33441 stable=UNSTABLE
 kworker/u1027:1-2286620 [017] ..... 479572.052499: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0x6af4272f offset=175455 count=33441 stable=UNSTABLE
 kworker/u1027:1-2286620 [017] ..... 479572.052501: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xf88816fa offset=175455 count=33441 stable=UNSTABLE
  oil-localfs-86-727850  [215] ..... 479572.052617: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=208896 newsize=210546
 kworker/u1037:4-2348084 [225] ..... 479572.052662: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0xc9992232 offset=175455 count=33441 res=33441 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1037:4-2348084 [225] ..... 479572.052663: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=210546 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:4-2348084 [225] ..... 479572.052663: nfs_writeback_done: error=33441 fileid=00:96:10067193438 fhandle=0xc9992232 offset=175455 count=33441 res=33441 stable=UNSTABLE verifier=e7a66eedc4f3c318
 kworker/u1033:6-2344693 [070] ..... 479572.052672: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0xf88816fa offset=175455 count=33441 res=33441 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1033:6-2344693 [070] ..... 479572.052673: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=210546 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:6-2344693 [070] ..... 479572.052674: nfs_writeback_done: error=33441 fileid=00:96:10067193438 fhandle=0xf88816fa offset=175455 count=33441 res=33441 stable=UNSTABLE verifier=a5cd34d9a5d3efdb
 kworker/u1027:5-2369633 [022] ..... 479572.052691: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232 offset=208896 count=1650 stable=UNSTABLE
 kworker/u1027:5-2369633 [022] ..... 479572.052695: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0x6af4272f offset=208896 count=1650 stable=UNSTABLE
 kworker/u1027:5-2369633 [022] ..... 479572.052696: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xf88816fa offset=208896 count=1650 stable=UNSTABLE
 kworker/u1037:4-2348084 [225] ..... 479572.052700: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0x6af4272f offset=175455 count=33441 res=33441 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1037:4-2348084 [225] ..... 479572.052700: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=210546 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:4-2348084 [225] ..... 479572.052701: nfs_writeback_done: error=33441 fileid=00:96:10067193438 fhandle=0x6af4272f offset=175455 count=33441 res=33441 stable=UNSTABLE verifier=7fb1042b4dd46a93
 kworker/u1033:2-2344691 [197] ..... 479572.052788: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0xf88816fa offset=208896 count=1650 res=1650 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1033:2-2344691 [197] ..... 479572.052788: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=210546 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:2-2344691 [197] ..... 479572.052789: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1033:2-2344691 [197] ..... 479572.052789: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=210546 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:2-2344691 [197] ..... 479572.052789: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=210546 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:2-2344691 [197] ..... 479572.052790: nfs_writeback_done: error=1650 fileid=00:96:10067193438 fhandle=0xf88816fa offset=208896 count=1650 res=1650 stable=UNSTABLE verifier=a5cd34d9a5d3efdb
 kworker/u1037:1-2369493 [230] ..... 479572.052802: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0xc9992232 offset=208896 count=1650 res=1650 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1037:1-2369493 [230] ..... 479572.052802: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=210546 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:1-2369493 [230] ..... 479572.052803: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1037:1-2369493 [230] ..... 479572.052803: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=210546 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:1-2369493 [230] ..... 479572.052803: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=210546 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:1-2369493 [230] ..... 479572.052804: nfs_writeback_done: error=1650 fileid=00:96:10067193438 fhandle=0xc9992232 offset=208896 count=1650 res=1650 stable=UNSTABLE verifier=e7a66eedc4f3c318
 kworker/u1037:4-2348084 [225] ..... 479572.052804: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0x6af4272f offset=208896 count=1650 res=1650 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1037:4-2348084 [225] ..... 479572.052804: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=210546 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:4-2348084 [225] ..... 479572.052805: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1037:4-2348084 [225] ..... 479572.052805: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=210546 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:4-2348084 [225] ..... 479572.052805: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=210546 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:4-2348084 [225] ..... 479572.052805: nfs_writeback_done: error=1650 fileid=00:96:10067193438 fhandle=0x6af4272f offset=208896 count=1650 res=1650 stable=UNSTABLE verifier=7fb1042b4dd46a93
  oil-localfs-86-727850  [215] ..... 479572.053111: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=210546 newsize=212992
 kworker/u1027:1-2392847 [151] ..... 479572.053134: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232 offset=208896 count=4096 stable=UNSTABLE
 kworker/u1027:1-2392847 [151] ..... 479572.053137: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0x6af4272f offset=208896 count=4096 stable=UNSTABLE
 kworker/u1027:1-2392847 [151] ..... 479572.053138: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xf88816fa offset=208896 count=4096 stable=UNSTABLE
 kworker/u1037:4-2348084 [225] ..... 479572.053247: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0xc9992232 offset=208896 count=4096 res=4096 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1037:4-2348084 [225] ..... 479572.053247: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=212992 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:4-2348084 [225] ..... 479572.053248: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1037:4-2348084 [225] ..... 479572.053248: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=212992 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:4-2348084 [225] ..... 479572.053249: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=212992 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:4-2348084 [225] ..... 479572.053249: nfs_writeback_done: error=4096 fileid=00:96:10067193438 fhandle=0xc9992232 offset=208896 count=4096 res=4096 stable=UNSTABLE verifier=e7a66eedc4f3c318
 kworker/u1033:1-2209763 [064] ..... 479572.053255: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0xf88816fa offset=208896 count=4096 res=4096 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1033:1-2209763 [064] ..... 479572.053255: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=212992 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:1-2209763 [064] ..... 479572.053256: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1037:1-2369493 [230] ..... 479572.053256: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0x6af4272f offset=208896 count=4096 res=4096 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1033:1-2209763 [064] ..... 479572.053256: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=212992 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:1-2209763 [064] ..... 479572.053257: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=212992 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:1-2209763 [064] ..... 479572.053257: nfs_writeback_done: error=4096 fileid=00:96:10067193438 fhandle=0xf88816fa offset=208896 count=4096 res=4096 stable=UNSTABLE verifier=a5cd34d9a5d3efdb
 kworker/u1037:1-2369493 [230] ..... 479572.053257: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=212992 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:1-2369493 [230] ..... 479572.053257: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1037:1-2369493 [230] ..... 479572.053258: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=212992 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:1-2369493 [230] ..... 479572.053258: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=212992 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:1-2369493 [230] ..... 479572.053258: nfs_writeback_done: error=4096 fileid=00:96:10067193438 fhandle=0x6af4272f offset=208896 count=4096 res=4096 stable=UNSTABLE verifier=7fb1042b4dd46a93
  oil-localfs-86-727850  [215] ..... 479572.053377: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=212992 newsize=215559
 kworker/u1027:1-2286624 [148] ..... 479572.053402: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232 offset=212992 count=2567 stable=UNSTABLE
 kworker/u1027:1-2286624 [148] ..... 479572.053406: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0x6af4272f offset=212992 count=2567 stable=UNSTABLE
 kworker/u1027:1-2286624 [148] ..... 479572.053408: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xf88816fa offset=212992 count=2567 stable=UNSTABLE
 kworker/u1033:6-2344693 [070] ..... 479572.053523: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0xf88816fa offset=212992 count=2567 res=2567 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1033:6-2344693 [070] ..... 479572.053523: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=215559 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:6-2344693 [070] ..... 479572.053524: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1033:6-2344693 [070] ..... 479572.053524: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=215559 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:6-2344693 [070] ..... 479572.053524: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=215559 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:6-2344693 [070] ..... 479572.053524: nfs_writeback_done: error=2567 fileid=00:96:10067193438 fhandle=0xf88816fa offset=212992 count=2567 res=2567 stable=UNSTABLE verifier=a5cd34d9a5d3efdb
 kworker/u1037:7-2298146 [100] ..... 479572.053527: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0xc9992232 offset=212992 count=2567 res=2567 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1037:7-2298146 [100] ..... 479572.053527: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=215559 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:7-2298146 [100] ..... 479572.053528: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1037:7-2298146 [100] ..... 479572.053528: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=215559 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:7-2298146 [100] ..... 479572.053528: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=215559 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:7-2298146 [100] ..... 479572.053529: nfs_writeback_done: error=2567 fileid=00:96:10067193438 fhandle=0xc9992232 offset=212992 count=2567 res=2567 stable=UNSTABLE verifier=e7a66eedc4f3c318
 kworker/u1037:7-2298146 [100] ..... 479572.053554: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0x6af4272f offset=212992 count=2567 res=2567 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1037:7-2298146 [100] ..... 479572.053554: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=215559 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:7-2298146 [100] ..... 479572.053555: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1037:7-2298146 [100] ..... 479572.053555: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=215559 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:7-2298146 [100] ..... 479572.053555: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=215559 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:7-2298146 [100] ..... 479572.053555: nfs_writeback_done: error=2567 fileid=00:96:10067193438 fhandle=0x6af4272f offset=212992 count=2567 res=2567 stable=UNSTABLE verifier=7fb1042b4dd46a93
  oil-localfs-86-727850  [215] ..... 479572.053789: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=215559 newsize=217088
 kworker/u1036:0-2339252 [222] ..... 479572.053806: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232 offset=212992 count=4096 stable=UNSTABLE
 kworker/u1036:0-2339252 [222] ..... 479572.053809: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0x6af4272f offset=212992 count=4096 stable=UNSTABLE
 kworker/u1036:0-2339252 [222] ..... 479572.053810: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xf88816fa offset=212992 count=4096 stable=UNSTABLE
 kworker/u1037:4-2348084 [225] ..... 479572.053919: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0xc9992232 offset=212992 count=4096 res=4096 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1037:4-2348084 [225] ..... 479572.053920: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=217088 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:4-2348084 [225] ..... 479572.053920: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1037:4-2348084 [225] ..... 479572.053920: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=217088 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:1-2209763 [064] ..... 479572.053920: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0xf88816fa offset=212992 count=4096 res=4096 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1037:4-2348084 [225] ..... 479572.053921: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=217088 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:4-2348084 [225] ..... 479572.053921: nfs_writeback_done: error=4096 fileid=00:96:10067193438 fhandle=0xc9992232 offset=212992 count=4096 res=4096 stable=UNSTABLE verifier=e7a66eedc4f3c318
 kworker/u1033:1-2209763 [064] ..... 479572.053921: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=217088 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:1-2209763 [064] ..... 479572.053921: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1033:1-2209763 [064] ..... 479572.053922: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=217088 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:1-2209763 [064] ..... 479572.053922: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=217088 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:1-2209763 [064] ..... 479572.053922: nfs_writeback_done: error=4096 fileid=00:96:10067193438 fhandle=0xf88816fa offset=212992 count=4096 res=4096 stable=UNSTABLE verifier=a5cd34d9a5d3efdb
 kworker/u1037:4-2348084 [225] ..... 479572.053933: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0x6af4272f offset=212992 count=4096 res=4096 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1037:4-2348084 [225] ..... 479572.053934: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=217088 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:4-2348084 [225] ..... 479572.053934: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1037:4-2348084 [225] ..... 479572.053935: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=217088 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:4-2348084 [225] ..... 479572.053935: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=217088 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:4-2348084 [225] ..... 479572.053935: nfs_writeback_done: error=4096 fileid=00:96:10067193438 fhandle=0x6af4272f offset=212992 count=4096 res=4096 stable=UNSTABLE verifier=7fb1042b4dd46a93
  oil-localfs-86-727850  [215] ..... 479572.053987: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=217088 newsize=220572
 kworker/u1036:8-2339326 [088] ..... 479572.054008: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232 offset=217088 count=3484 stable=UNSTABLE
 kworker/u1036:8-2339326 [088] ..... 479572.054011: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0x6af4272f offset=217088 count=3484 stable=UNSTABLE
 kworker/u1036:8-2339326 [088] ..... 479572.054013: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xf88816fa offset=217088 count=3484 stable=UNSTABLE
 kworker/u1037:1-2369493 [230] ..... 479572.054142: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0xc9992232 offset=217088 count=3484 res=3484 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1037:1-2369493 [230] ..... 479572.054143: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=220572 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:1-2369493 [230] ..... 479572.054143: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1037:1-2369493 [230] ..... 479572.054144: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=220572 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:1-2369493 [230] ..... 479572.054144: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=220572 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:1-2369493 [230] ..... 479572.054144: nfs_writeback_done: error=3484 fileid=00:96:10067193438 fhandle=0xc9992232 offset=217088 count=3484 res=3484 stable=UNSTABLE verifier=e7a66eedc4f3c318
 kworker/u1033:6-2344693 [070] ..... 479572.054145: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0xf88816fa offset=217088 count=3484 res=3484 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1033:6-2344693 [070] ..... 479572.054145: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=220572 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:6-2344693 [070] ..... 479572.054145: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1033:6-2344693 [070] ..... 479572.054146: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=220572 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:6-2344693 [070] ..... 479572.054146: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=220572 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:6-2344693 [070] ..... 479572.054146: nfs_writeback_done: error=3484 fileid=00:96:10067193438 fhandle=0xf88816fa offset=217088 count=3484 res=3484 stable=UNSTABLE verifier=a5cd34d9a5d3efdb
 kworker/u1037:1-2369493 [230] ..... 479572.054154: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0x6af4272f offset=217088 count=3484 res=3484 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1037:1-2369493 [230] ..... 479572.054154: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=220572 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:1-2369493 [230] ..... 479572.054155: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1037:1-2369493 [230] ..... 479572.054155: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=220572 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:1-2369493 [230] ..... 479572.054155: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=220572 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:1-2369493 [230] ..... 479572.054155: nfs_writeback_done: error=3484 fileid=00:96:10067193438 fhandle=0x6af4272f offset=217088 count=3484 res=3484 stable=UNSTABLE verifier=7fb1042b4dd46a93
  oil-localfs-86-727850  [215] ..... 479572.054405: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=220572 newsize=221184
 kworker/u1036:1-2297875 [217] ..... 479572.054418: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232 offset=220572 count=612 stable=UNSTABLE
 kworker/u1036:1-2297875 [217] ..... 479572.054420: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0x6af4272f offset=220572 count=612 stable=UNSTABLE
 kworker/u1036:1-2297875 [217] ..... 479572.054422: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xf88816fa offset=220572 count=612 stable=UNSTABLE
 kworker/u1037:7-2298146 [100] ..... 479572.054522: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0xc9992232 offset=220572 count=612 res=612 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1037:7-2298146 [100] ..... 479572.054523: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=221184 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:7-2298146 [100] ..... 479572.054524: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1037:7-2298146 [100] ..... 479572.054524: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=221184 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:7-2298146 [100] ..... 479572.054524: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=221184 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:7-2298146 [100] ..... 479572.054525: nfs_writeback_done: error=612 fileid=00:96:10067193438 fhandle=0xc9992232 offset=220572 count=612 res=612 stable=UNSTABLE verifier=e7a66eedc4f3c318
 kworker/u1037:4-2348084 [225] ..... 479572.054527: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0x6af4272f offset=220572 count=612 res=612 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1037:4-2348084 [225] ..... 479572.054528: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=221184 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:4-2348084 [225] ..... 479572.054528: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1037:4-2348084 [225] ..... 479572.054528: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=221184 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:4-2348084 [225] ..... 479572.054528: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=221184 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:4-2348084 [225] ..... 479572.054528: nfs_writeback_done: error=612 fileid=00:96:10067193438 fhandle=0x6af4272f offset=220572 count=612 res=612 stable=UNSTABLE verifier=7fb1042b4dd46a93
 kworker/u1033:1-2209763 [064] ..... 479572.054537: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0xf88816fa offset=220572 count=612 res=612 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1033:1-2209763 [064] ..... 479572.054538: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=221184 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:1-2209763 [064] ..... 479572.054538: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1033:1-2209763 [064] ..... 479572.054538: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=221184 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:1-2209763 [064] ..... 479572.054539: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=221184 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:1-2209763 [064] ..... 479572.054539: nfs_writeback_done: error=612 fileid=00:96:10067193438 fhandle=0xf88816fa offset=220572 count=612 res=612 stable=UNSTABLE verifier=a5cd34d9a5d3efdb
  oil-localfs-86-727850  [215] ..... 479572.054581: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=221184 newsize=225280
  oil-localfs-86-727850  [215] ..... 479572.054584: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=225280 newsize=225585
 kworker/u1036:0-2339252 [217] ..... 479572.054622: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232 offset=221184 count=4401 stable=UNSTABLE
 kworker/u1036:0-2339252 [217] ..... 479572.054624: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0x6af4272f offset=221184 count=4401 stable=UNSTABLE
 kworker/u1036:0-2339252 [217] ..... 479572.054626: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xf88816fa offset=221184 count=4401 stable=UNSTABLE
 kworker/u1037:4-2348084 [225] ..... 479572.054752: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0x6af4272f offset=221184 count=4401 res=4401 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1037:4-2348084 [225] ..... 479572.054753: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=225585 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:4-2348084 [225] ..... 479572.054753: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1037:4-2348084 [225] ..... 479572.054753: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=225585 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:4-2348084 [225] ..... 479572.054753: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=225585 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:4-2348084 [225] ..... 479572.054754: nfs_writeback_done: error=4401 fileid=00:96:10067193438 fhandle=0x6af4272f offset=221184 count=4401 res=4401 stable=UNSTABLE verifier=7fb1042b4dd46a93
 kworker/u1037:7-2298146 [100] ..... 479572.054754: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0xc9992232 offset=221184 count=4401 res=4401 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1037:7-2298146 [100] ..... 479572.054754: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=225585 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:7-2298146 [100] ..... 479572.054755: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1037:7-2298146 [100] ..... 479572.054755: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=225585 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:7-2298146 [100] ..... 479572.054755: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=225585 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:7-2298146 [100] ..... 479572.054755: nfs_writeback_done: error=4401 fileid=00:96:10067193438 fhandle=0xc9992232 offset=221184 count=4401 res=4401 stable=UNSTABLE verifier=e7a66eedc4f3c318
 kworker/u1033:0-2344690 [066] ..... 479572.054766: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0xf88816fa offset=221184 count=4401 res=4401 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1033:0-2344690 [066] ..... 479572.054767: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=225585 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:0-2344690 [066] ..... 479572.054767: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1033:0-2344690 [066] ..... 479572.054768: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=225585 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:0-2344690 [066] ..... 479572.054768: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=225585 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:0-2344690 [066] ..... 479572.054768: nfs_writeback_done: error=4401 fileid=00:96:10067193438 fhandle=0xf88816fa offset=221184 count=4401 res=4401 stable=UNSTABLE verifier=a5cd34d9a5d3efdb
  oil-localfs-86-727850  [215] ..... 479572.054997: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=225585 newsize=229376
  oil-localfs-86-727850  [215] ..... 479572.055190: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=229376 newsize=230598
  oil-localfs-86-727850  [215] ..... 479572.055379: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=230598 newsize=233472
  oil-localfs-86-727850  [215] ..... 479572.055562: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=233472 newsize=235611
  oil-localfs-86-727850  [215] ..... 479572.055752: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=235611 newsize=237568
  oil-localfs-86-727850  [215] ..... 479572.055913: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=237568 newsize=240624
  oil-localfs-86-727850  [215] ..... 479572.056101: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=240624 newsize=241664
  oil-localfs-86-727850  [215] ..... 479572.056274: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=241664 newsize=245637
  oil-localfs-86-727850  [215] ..... 479572.056466: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=245637 newsize=245760
  oil-localfs-86-727850  [215] ..... 479572.056657: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=245760 newsize=249856
  oil-localfs-86-727850  [215] ..... 479572.056659: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=249856 newsize=250650
  oil-localfs-86-727850  [215] ..... 479572.056858: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=250650 newsize=253952
  oil-localfs-86-727850  [215] ..... 479572.057009: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=253952 newsize=255663
  oil-localfs-86-727850  [215] ..... 479572.057227: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=255663 newsize=258048
  oil-localfs-86-727850  [215] ..... 479572.057417: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=258048 newsize=260676
  oil-localfs-86-727850  [215] ..... 479572.057606: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=260676 newsize=262144
  oil-localfs-86-727850  [215] ..... 479572.057789: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=262144 newsize=265689
  oil-localfs-86-727850  [215] ..... 479572.057976: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=265689 newsize=266240
  oil-localfs-86-727850  [215] ..... 479572.058143: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=266240 newsize=270336
  oil-localfs-86-727850  [215] ..... 479572.058150: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=270336 newsize=270702
  oil-localfs-86-727850  [215] ..... 479572.058348: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=270702 newsize=274432
  oil-localfs-86-727850  [215] ..... 479572.058524: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=274432 newsize=275715
  oil-localfs-86-727850  [215] ..... 479572.058699: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=275715 newsize=278528
  oil-localfs-86-727850  [215] ..... 479572.058874: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=278528 newsize=280728
  oil-localfs-86-727850  [215] ..... 479572.059053: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=280728 newsize=282624
  oil-localfs-86-727850  [215] ..... 479572.059213: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=282624 newsize=285741
  oil-localfs-86-727850  [215] ..... 479572.059414: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=285741 newsize=286720
  oil-localfs-86-727850  [215] ..... 479572.059594: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=286720 newsize=290754
  oil-localfs-86-727850  [215] ..... 479572.059771: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=290754 newsize=290816
  oil-localfs-86-727850  [215] ..... 479572.059949: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=290816 newsize=294912
  oil-localfs-86-727850  [215] ..... 479572.059950: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=294912 newsize=295767
  oil-localfs-86-727850  [215] ..... 479572.060130: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=295767 newsize=299008
  oil-localfs-86-727850  [215] ..... 479572.060302: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=299008 newsize=300780
  oil-localfs-86-727850  [215] ..... 479572.060512: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=300780 newsize=303104
  oil-localfs-86-727850  [215] ..... 479572.060688: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=303104 newsize=305793
  oil-localfs-86-727850  [215] ..... 479572.060855: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=305793 newsize=307200
  oil-localfs-86-727850  [215] ..... 479572.061027: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=307200 newsize=310806
  oil-localfs-86-727850  [215] ..... 479572.061214: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=310806 newsize=311296
  oil-localfs-86-727850  [215] ..... 479572.061383: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=311296 newsize=315392
  oil-localfs-86-727850  [215] ..... 479572.061386: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=315392 newsize=315819
  oil-localfs-86-727850  [215] ..... 479572.061584: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=315819 newsize=319488
  oil-localfs-86-727850  [215] ..... 479572.061762: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=319488 newsize=320832
  oil-localfs-86-727850  [215] ..... 479572.061940: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=320832 newsize=323584
  oil-localfs-86-727850  [215] ..... 479572.062126: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=323584 newsize=325845
  oil-localfs-86-727850  [215] ..... 479572.062318: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=325845 newsize=327680
  oil-localfs-86-727850  [215] ..... 479572.062485: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=327680 newsize=330858
  oil-localfs-86-727850  [215] ..... 479572.062670: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=330858 newsize=331776
  oil-localfs-86-727850  [215] ..... 479572.062846: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=331776 newsize=335871
  oil-localfs-86-727850  [215] ..... 479572.063011: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=335871 newsize=335872
  oil-localfs-86-727850  [215] ..... 479572.063185: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=335872 newsize=339968
  oil-localfs-86-727850  [215] ..... 479572.063187: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=339968 newsize=340884
  oil-localfs-86-727850  [215] ..... 479572.063385: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=340884 newsize=344064
  oil-localfs-86-727850  [215] ..... 479572.063547: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=344064 newsize=345897
  oil-localfs-86-727850  [215] ..... 479572.063732: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=345897 newsize=348160
  oil-localfs-86-727850  [215] ..... 479572.063898: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=348160 newsize=350910
  oil-localfs-86-727850  [215] ..... 479572.064063: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=350910 newsize=352256
  oil-localfs-86-727850  [215] ..... 479572.064239: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=352256 newsize=355923
  oil-localfs-86-727850  [215] ..... 479572.064446: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=355923 newsize=356352
  oil-localfs-86-727850  [215] ..... 479572.064631: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=356352 newsize=360448
  oil-localfs-86-727850  [215] ..... 479572.064633: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=360448 newsize=360936
  oil-localfs-86-727850  [215] ..... 479572.064845: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=360936 newsize=364544
  oil-localfs-86-727850  [215] ..... 479572.065028: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=364544 newsize=365949
  oil-localfs-86-727850  [215] ..... 479572.065207: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=365949 newsize=368640
  oil-localfs-86-727850  [215] ..... 479572.065388: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=368640 newsize=370962
  oil-localfs-86-727850  [215] ..... 479572.065583: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=370962 newsize=372736
  oil-localfs-86-727850  [215] ..... 479572.065751: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=372736 newsize=375975
  oil-localfs-86-727850  [215] ..... 479572.065935: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=375975 newsize=376832
  oil-localfs-86-727850  [215] ..... 479572.066106: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=376832 newsize=380928
  oil-localfs-86-727850  [215] ..... 479572.066108: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=380928 newsize=380988
  oil-localfs-86-727850  [215] ..... 479572.066283: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=380988 newsize=385024
  oil-localfs-86-727850  [215] ..... 479572.066462: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=385024 newsize=386001
  oil-localfs-86-727850  [215] ..... 479572.066661: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=386001 newsize=389120
  oil-localfs-86-727850  [215] ..... 479572.066824: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=389120 newsize=391014
  oil-localfs-86-727850  [215] ..... 479572.067012: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=391014 newsize=393216
  oil-localfs-86-727850  [215] ..... 479572.067180: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=393216 newsize=396027
  oil-localfs-86-727850  [215] ..... 479572.067361: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=396027 newsize=397312
  oil-localfs-86-727850  [215] ..... 479572.067545: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=397312 newsize=401040
  oil-localfs-86-727850  [215] ..... 479572.067730: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=401040 newsize=401408
  oil-localfs-86-727850  [215] ..... 479572.067888: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=401408 newsize=405504
  oil-localfs-86-727850  [215] ..... 479572.067889: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=405504 newsize=406053
  oil-localfs-86-727850  [215] ..... 479572.068084: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=406053 newsize=409600
  oil-localfs-86-727850  [215] ..... 479572.068252: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=409600 newsize=411066
  oil-localfs-86-727850  [215] ..... 479572.068432: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=411066 newsize=413696
  oil-localfs-86-727850  [215] ..... 479572.068631: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=413696 newsize=416079
  oil-localfs-86-727850  [215] ..... 479572.068820: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=416079 newsize=417792
  oil-localfs-86-727850  [215] ..... 479572.068986: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=417792 newsize=421092
  oil-localfs-86-727850  [215] ..... 479572.069169: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=421092 newsize=421888
  oil-localfs-86-727850  [215] ..... 479572.069328: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=421888 newsize=425984
  oil-localfs-86-727850  [215] ..... 479572.069329: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=425984 newsize=426105
  oil-localfs-86-727850  [215] ..... 479572.069511: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=426105 newsize=430080
  oil-localfs-86-727850  [215] ..... 479572.069685: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=430080 newsize=431118
  oil-localfs-86-727850  [215] ..... 479572.069873: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=431118 newsize=434176
  oil-localfs-86-727850  [215] ..... 479572.070030: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=434176 newsize=436131
  oil-localfs-86-727850  [215] ..... 479572.070213: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=436131 newsize=438272
  oil-localfs-86-727850  [215] ..... 479572.070390: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=438272 newsize=441144
  oil-localfs-86-727850  [215] ..... 479572.070574: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=441144 newsize=442368
  oil-localfs-86-727850  [215] ..... 479572.070768: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=442368 newsize=446157
  oil-localfs-86-727850  [215] ..... 479572.070956: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=446157 newsize=446464
  oil-localfs-86-727850  [215] ..... 479572.071126: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=446464 newsize=450560
  oil-localfs-86-727850  [215] ..... 479572.071128: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=450560 newsize=451170
  oil-localfs-86-727850  [215] ..... 479572.071326: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=451170 newsize=454656
  oil-localfs-86-727850  [215] ..... 479572.071503: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=454656 newsize=456183
  oil-localfs-86-727850  [215] ..... 479572.071670: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=456183 newsize=458752
  oil-localfs-86-727850  [215] ..... 479572.071843: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=458752 newsize=461196
  oil-localfs-86-727850  [215] ..... 479572.072022: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=461196 newsize=462848
  oil-localfs-86-727850  [215] ..... 479572.072197: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=462848 newsize=466209
  oil-localfs-86-727850  [215] ..... 479572.072396: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=466209 newsize=466944
  oil-localfs-86-727850  [215] ..... 479572.072571: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=466944 newsize=471040
  oil-localfs-86-727850  [215] ..... 479572.072573: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=471040 newsize=471222
  oil-localfs-86-727850  [215] ..... 479572.072748: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=471222 newsize=475136
  oil-localfs-86-727850  [215] ..... 479572.072932: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=475136 newsize=476235
  oil-localfs-86-727850  [215] ..... 479572.073118: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=476235 newsize=479232
  oil-localfs-86-727850  [215] ..... 479572.073303: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=479232 newsize=481248
  oil-localfs-86-727850  [215] ..... 479572.073503: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=481248 newsize=483328
  oil-localfs-86-727850  [215] ..... 479572.073686: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=483328 newsize=486261
  oil-localfs-86-727850  [215] ..... 479572.073857: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=486261 newsize=487424
  oil-localfs-86-727850  [215] ..... 479572.074044: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=487424 newsize=491274
 kworker/u1030:1-2297876 [042] ..... 479572.074194: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232 offset=229376 count=261898 stable=UNSTABLE
 kworker/u1030:1-2297876 [042] ..... 479572.074225: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0x6af4272f offset=229376 count=261898 stable=UNSTABLE
 kworker/u1030:1-2297876 [042] ..... 479572.074230: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xf88816fa offset=229376 count=261898 stable=UNSTABLE
 kworker/u1037:4-2348084 [225] ..... 479572.074944: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0xc9992232 offset=229376 count=261898 res=261898 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1037:4-2348084 [225] ..... 479572.074945: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=491274 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:4-2348084 [225] ..... 479572.074946: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1037:4-2348084 [225] ..... 479572.074946: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=491274 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:1-2209763 [068] ..... 479572.074947: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0xf88816fa offset=229376 count=261898 res=261898 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1037:4-2348084 [225] ..... 479572.074947: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=491274 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:4-2348084 [225] ..... 479572.074947: nfs_writeback_done: error=261898 fileid=00:96:10067193438 fhandle=0xc9992232 offset=229376 count=261898 res=261898 stable=UNSTABLE verifier=e7a66eedc4f3c318
 kworker/u1033:1-2209763 [068] ..... 479572.074949: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=491274 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:1-2209763 [068] ..... 479572.074950: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1033:1-2209763 [068] ..... 479572.074951: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=491274 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:1-2209763 [068] ..... 479572.074951: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=491274 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:1-2209763 [068] ..... 479572.074951: nfs_writeback_done: error=261898 fileid=00:96:10067193438 fhandle=0xf88816fa offset=229376 count=261898 res=261898 stable=UNSTABLE verifier=a5cd34d9a5d3efdb
 kworker/u1037:4-2348084 [225] ..... 479572.075079: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0x6af4272f offset=229376 count=261898 res=261898 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1037:4-2348084 [225] ..... 479572.075079: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=491274 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:4-2348084 [225] ..... 479572.075080: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1037:4-2348084 [225] ..... 479572.075080: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=491274 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:4-2348084 [225] ..... 479572.075080: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=491274 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:4-2348084 [225] ..... 479572.075080: nfs_writeback_done: error=261898 fileid=00:96:10067193438 fhandle=0x6af4272f offset=229376 count=261898 res=261898 stable=UNSTABLE verifier=7fb1042b4dd46a93
  oil-localfs-86-727850  [226] ..... 479572.075704: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=491274 newsize=491520
 kworker/u1036:2-2339254 [092] ..... 479572.075719: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232 offset=487424 count=4096 stable=UNSTABLE
 kworker/u1036:2-2339254 [092] ..... 479572.075722: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0x6af4272f offset=487424 count=4096 stable=UNSTABLE
 kworker/u1036:2-2339254 [092] ..... 479572.075725: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xf88816fa offset=487424 count=4096 stable=UNSTABLE
 kworker/u1037:4-2348084 [096] ..... 479572.075847: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0xc9992232 offset=487424 count=4096 res=4096 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1033:1-2209763 [068] ..... 479572.075847: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0xf88816fa offset=487424 count=4096 res=4096 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1037:4-2348084 [096] ..... 479572.075854: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=491520 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:4-2348084 [096] ..... 479572.075855: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1037:4-2348084 [096] ..... 479572.075855: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=491520 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:4-2348084 [096] ..... 479572.075855: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=491520 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:4-2348084 [096] ..... 479572.075856: nfs_writeback_done: error=4096 fileid=00:96:10067193438 fhandle=0xc9992232 offset=487424 count=4096 res=4096 stable=UNSTABLE verifier=e7a66eedc4f3c318
 kworker/u1033:1-2209763 [068] ..... 479572.075856: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=491520 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:1-2209763 [068] ..... 479572.075856: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1033:1-2209763 [068] ..... 479572.075857: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=491520 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:1-2209763 [068] ..... 479572.075857: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=491520 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:1-2209763 [068] ..... 479572.075857: nfs_writeback_done: error=4096 fileid=00:96:10067193438 fhandle=0xf88816fa offset=487424 count=4096 res=4096 stable=UNSTABLE verifier=a5cd34d9a5d3efdb
 kworker/u1037:1-2369493 [230] ..... 479572.075859: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0x6af4272f offset=487424 count=4096 res=4096 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1037:1-2369493 [230] ..... 479572.075860: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=491520 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:1-2369493 [230] ..... 479572.075860: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1037:1-2369493 [230] ..... 479572.075860: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=491520 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:1-2369493 [230] ..... 479572.075861: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=491520 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:1-2369493 [230] ..... 479572.075861: nfs_writeback_done: error=4096 fileid=00:96:10067193438 fhandle=0x6af4272f offset=487424 count=4096 res=4096 stable=UNSTABLE verifier=7fb1042b4dd46a93
  oil-localfs-86-727850  [226] ..... 479572.075971: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=491520 newsize=495616
  oil-localfs-86-727850  [226] ..... 479572.075975: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=495616 newsize=496287
 kworker/u1036:3-2339255 [223] ..... 479572.075997: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232 offset=491520 count=4767 stable=UNSTABLE
 kworker/u1036:3-2339255 [223] ..... 479572.076001: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0x6af4272f offset=491520 count=4767 stable=UNSTABLE
 kworker/u1036:3-2339255 [223] ..... 479572.076004: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xf88816fa offset=491520 count=4767 stable=UNSTABLE
 kworker/u1037:7-2298146 [100] ..... 479572.076132: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0x6af4272f offset=491520 count=4767 res=4767 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1037:7-2298146 [100] ..... 479572.076132: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=496287 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:7-2298146 [100] ..... 479572.076133: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1037:7-2298146 [100] ..... 479572.076133: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=496287 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:7-2298146 [100] ..... 479572.076133: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=496287 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:7-2298146 [100] ..... 479572.076133: nfs_writeback_done: error=4767 fileid=00:96:10067193438 fhandle=0x6af4272f offset=491520 count=4767 res=4767 stable=UNSTABLE verifier=7fb1042b4dd46a93
 kworker/u1037:1-2369493 [230] ..... 479572.076134: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0xc9992232 offset=491520 count=4767 res=4767 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1037:1-2369493 [230] ..... 479572.076134: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=496287 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:1-2369493 [230] ..... 479572.076134: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1037:1-2369493 [230] ..... 479572.076135: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=496287 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:1-2369493 [230] ..... 479572.076135: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=496287 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:1-2369493 [230] ..... 479572.076135: nfs_writeback_done: error=4767 fileid=00:96:10067193438 fhandle=0xc9992232 offset=491520 count=4767 res=4767 stable=UNSTABLE verifier=e7a66eedc4f3c318
 kworker/u1033:0-2344690 [068] ..... 479572.076144: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0xf88816fa offset=491520 count=4767 res=4767 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1033:0-2344690 [068] ..... 479572.076144: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=496287 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:0-2344690 [068] ..... 479572.076145: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1033:0-2344690 [068] ..... 479572.076145: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=496287 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:0-2344690 [068] ..... 479572.076145: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=496287 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:0-2344690 [068] ..... 479572.076145: nfs_writeback_done: error=4767 fileid=00:96:10067193438 fhandle=0xf88816fa offset=491520 count=4767 res=4767 stable=UNSTABLE verifier=a5cd34d9a5d3efdb
  oil-localfs-86-727850  [226] ..... 479572.076404: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=496287 newsize=499712
 kworker/u1036:0-2339252 [217] ..... 479572.076423: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232 offset=496287 count=3425 stable=UNSTABLE
 kworker/u1036:0-2339252 [217] ..... 479572.076425: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0x6af4272f offset=496287 count=3425 stable=UNSTABLE
 kworker/u1036:0-2339252 [217] ..... 479572.076427: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xf88816fa offset=496287 count=3425 stable=UNSTABLE
 kworker/u1037:7-2298146 [100] ..... 479572.076535: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0x6af4272f offset=496287 count=3425 res=3425 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1037:7-2298146 [100] ..... 479572.076535: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=499712 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:7-2298146 [100] ..... 479572.076536: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1037:7-2298146 [100] ..... 479572.076536: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=499712 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:7-2298146 [100] ..... 479572.076537: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=499712 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:7-2298146 [100] ..... 479572.076537: nfs_writeback_done: error=3425 fileid=00:96:10067193438 fhandle=0x6af4272f offset=496287 count=3425 res=3425 stable=UNSTABLE verifier=7fb1042b4dd46a93
 kworker/u1033:0-2344690 [068] ..... 479572.076541: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0xf88816fa offset=496287 count=3425 res=3425 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1033:0-2344690 [068] ..... 479572.076542: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=499712 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:0-2344690 [068] ..... 479572.076542: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1033:0-2344690 [068] ..... 479572.076543: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=499712 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:0-2344690 [068] ..... 479572.076543: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=499712 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:0-2344690 [068] ..... 479572.076543: nfs_writeback_done: error=3425 fileid=00:96:10067193438 fhandle=0xf88816fa offset=496287 count=3425 res=3425 stable=UNSTABLE verifier=a5cd34d9a5d3efdb
 kworker/u1037:7-2298146 [100] ..... 479572.076557: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0xc9992232 offset=496287 count=3425 res=3425 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1037:7-2298146 [100] ..... 479572.076558: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=499712 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:7-2298146 [100] ..... 479572.076558: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1037:7-2298146 [100] ..... 479572.076558: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=499712 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:7-2298146 [100] ..... 479572.076559: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=499712 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:7-2298146 [100] ..... 479572.076559: nfs_writeback_done: error=3425 fileid=00:96:10067193438 fhandle=0xc9992232 offset=496287 count=3425 res=3425 stable=UNSTABLE verifier=e7a66eedc4f3c318
  oil-localfs-86-727850  [226] ..... 479572.076610: nfs_size_grow: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 cursize=499712 newsize=501300
 kworker/u1036:1-2044368 [221] ..... 479572.076628: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232 offset=499712 count=1588 stable=UNSTABLE
 kworker/u1036:1-2044368 [221] ..... 479572.076631: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0x6af4272f offset=499712 count=1588 stable=UNSTABLE
 kworker/u1036:1-2044368 [221] ..... 479572.076632: nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xf88816fa offset=499712 count=1588 stable=UNSTABLE
 kworker/u1033:1-2209763 [194] ..... 479572.076797: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0xf88816fa offset=499712 count=1588 res=1588 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1033:1-2209763 [194] ..... 479572.076798: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=501300 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:1-2209763 [194] ..... 479572.076799: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1033:1-2209763 [194] ..... 479572.076799: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=501300 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:1-2209763 [194] ..... 479572.076799: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=501300 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1033:1-2209763 [194] ..... 479572.076800: nfs_writeback_done: error=1588 fileid=00:96:10067193438 fhandle=0xf88816fa offset=499712 count=1588 res=1588 stable=UNSTABLE verifier=a5cd34d9a5d3efdb
 kworker/u1037:4-2348084 [096] ..... 479572.076809: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0x6af4272f offset=499712 count=1588 res=1588 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1037:4-2348084 [096] ..... 479572.076810: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=501300 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:4-2348084 [096] ..... 479572.076810: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1037:4-2348084 [096] ..... 479572.076810: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=501300 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:4-2348084 [096] ..... 479572.076811: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=501300 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:4-2348084 [096] ..... 479572.076811: nfs_writeback_done: error=1588 fileid=00:96:10067193438 fhandle=0x6af4272f offset=499712 count=1588 res=1588 stable=UNSTABLE verifier=7fb1042b4dd46a93
 kworker/u1037:8-2095961 [101] ..... 479572.076817: nfs4_pnfs_write: error=0 (OK) fileid=00:96:10067193438 fhandle=0xc9992232 offset=499712 count=1588 res=1588 stateid=1:0xd68163a6 layoutstateid=1:0xaa2ed7d2
 kworker/u1037:8-2095961 [101] ..... 479572.076818: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=501300 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:8-2095961 [101] ..... 479572.076819: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
 kworker/u1037:8-2095961 [101] ..... 479572.076819: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=501300 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:8-2095961 [101] ..... 479572.076820: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=501300 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:8-2095961 [101] ..... 479572.076820: nfs_writeback_done: error=1588 fileid=00:96:10067193438 fhandle=0xc9992232 offset=499712 count=1588 res=1588 stable=UNSTABLE verifier=e7a66eedc4f3c318
  oil-localfs-86-727850  [226] ..... 479572.076832: nfs_fsync_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
  oil-localfs-86-727850  [226] ..... 479572.077046: nfs_fsync_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=501300 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
  oil-localfs-86-727850  [226] ..... 479572.077328: nfs_writeback_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
  oil-localfs-86-727850  [226] ..... 479572.077331: nfs_writeback_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598774309300 size=501300 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
           <...>-1706339 [212] ..... 479572.178837: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598774309300 
           <...>-1706339 [212] ..... 479572.178839: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598933970106 size=501300 cache_validity=0x3101a (INVALID_DATA|INVALID_ACCESS|INVALID_ACL|INVALID_OTHER|INVALID_NLINK|INVALID_MODE) nfs_flags=0x4 (ACL_LRU_SET)
           <...>-1706339 [212] ..... 479572.178839: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598933970106 size=501300 cache_validity=0x3101a (INVALID_DATA|INVALID_ACCESS|INVALID_ACL|INVALID_OTHER|INVALID_NLINK|INVALID_MODE) nfs_flags=0x4 (ACL_LRU_SET)
  oil-localfs-86-727850  [227] ..... 479572.181992: nfs_writeback_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598933970106 
  oil-localfs-86-727850  [227] ..... 479572.181999: nfs_writeback_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598933970106 size=501300 cache_validity=0x3101a (INVALID_DATA|INVALID_ACCESS|INVALID_ACL|INVALID_OTHER|INVALID_NLINK|INVALID_MODE) nfs_flags=0x4 (ACL_LRU_SET)
  oil-localfs-86-727850  [227] ..... 479572.213513: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598933970106 size=501300 cache_validity=0x31b1a (INVALID_DATA|INVALID_ACCESS|INVALID_ACL|INVALID_CHANGE|INVALID_CTIME|INVALID_SIZE|INVALID_OTHER|INVALID_NLINK|INVALID_MODE) nfs_flags=0x4 (ACL_LRU_SET)
  oil-localfs-86-727850  [227] ..... 479572.314116: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598933970106 
  oil-localfs-86-727850  [227] ..... 479572.314120: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598933970106 size=501300 cache_validity=0x1a (INVALID_DATA|INVALID_ACCESS|INVALID_ACL) nfs_flags=0x4 (ACL_LRU_SET)
  oil-localfs-86-727850  [227] ..... 479572.314120: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598933970106 size=501300 cache_validity=0x1a (INVALID_DATA|INVALID_ACCESS|INVALID_ACL) nfs_flags=0x4 (ACL_LRU_SET)
  oil-localfs-86-727850  [227] ..... 479572.314229: nfs_getattr_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598933970106 
  oil-localfs-86-727850  [227] ..... 479572.314241: nfs_getattr_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598933970106 size=501300 cache_validity=0x1a (INVALID_DATA|INVALID_ACCESS|INVALID_ACL) nfs_flags=0x4 (ACL_LRU_SET)
  oil-localfs-86-727850  [227] ..... 479572.314256: nfs_getattr_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598933970106 
  oil-localfs-86-727850  [227] ..... 479572.314257: nfs_getattr_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598933970106 size=501300 cache_validity=0x1a (INVALID_DATA|INVALID_ACCESS|INVALID_ACL) nfs_flags=0x4 (ACL_LRU_SET)
  oil-localfs-86-727850  [227] ..... 479572.324902: nfs_invalidate_mapping_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598933970106 
  oil-localfs-86-727850  [227] ..... 479572.324903: nfs_writeback_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598933970106 
  oil-localfs-86-727850  [227] ..... 479572.324908: nfs_writeback_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598933970106 size=501300 cache_validity=0x18 (INVALID_ACCESS|INVALID_ACL) nfs_flags=0xc (ACL_LRU_SET|INVALIDATING)
  oil-localfs-86-727850  [227] ..... 479572.324968: nfs_invalidate_mapping_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598933970106 size=501300 cache_validity=0x18 (INVALID_ACCESS|INVALID_ACL) nfs_flags=0xc (ACL_LRU_SET|INVALIDATING)
 kworker/u1037:9-2298147 [098] ..... 479572.325798: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598933970106 size=501300 cache_validity=0x1c (INVALID_ATIME|INVALID_ACCESS|INVALID_ACL) nfs_flags=0x4 (ACL_LRU_SET)
 SR_TLSConnectio-727357  [154] ..... 479573.117824: nfs_writeback_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598933970106 
 SR_TLSConnectio-727357  [154] ..... 479573.117825: nfs_writeback_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598933970106 size=501300 cache_validity=0x1c (INVALID_ATIME|INVALID_ACCESS|INVALID_ACL) nfs_flags=0x4 (ACL_LRU_SET)
           <...>-1706339 [212] ..... 479573.176810: nfs_refresh_inode_enter: fileid=00:96:10067193438 fhandle=0x14c40498 version=1753823598933970106 
           <...>-1706339 [212] ..... 479573.176811: nfs_set_cache_invalid: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598933970106 size=501300 cache_validity=0x18 (INVALID_ACCESS|INVALID_ACL) nfs_flags=0x4 (ACL_LRU_SET)
           <...>-1706339 [212] ..... 479573.176811: nfs_refresh_inode_exit: error=0 (OK) fileid=00:96:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598933970106 size=501300 cache_validity=0x18 (INVALID_ACCESS|INVALID_ACL) nfs_flags=0x4 (ACL_LRU_SET)
 oil-localfs-254-2395524 [179] ..... 479897.213099: nfs_getattr_enter: fileid=00:7f:10067193438 fhandle=0x14c40498 version=1753823598933970106 
 oil-localfs-254-2395524 [179] ..... 479897.213100: nfs_getattr_exit: error=0 (OK) fileid=00:7f:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598933970106 size=501300 cache_validity=0x0 () nfs_flags=0x0 ()
 oil-localfs-254-2395524 [051] ..... 479897.977995: nfs_refresh_inode_enter: fileid=00:7f:10067193438 fhandle=0x14c40498 version=1753823598933970106 
 oil-localfs-254-2395524 [051] ..... 479897.977995: nfs_set_cache_invalid: error=0 (OK) fileid=00:7f:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598933970106 size=501300 cache_validity=0x0 () nfs_flags=0x0 ()
 oil-localfs-254-2395524 [051] ..... 479897.977995: nfs_refresh_inode_exit: error=0 (OK) fileid=00:7f:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598933970106 size=501300 cache_validity=0x0 () nfs_flags=0x0 ()
 oil-localfs-254-2395524 [051] ..... 479898.108026: nfs_getattr_enter: fileid=00:7f:10067193438 fhandle=0x14c40498 version=1753823598933970106 
 oil-localfs-254-2395524 [051] ..... 479898.108027: nfs_getattr_exit: error=0 (OK) fileid=00:7f:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598933970106 size=501300 cache_validity=0x0 () nfs_flags=0x0 ()
 oil-localfs-254-2395524 [055] ..... 479912.629681: nfs_refresh_inode_enter: fileid=00:7f:10067193438 fhandle=0x14c40498 version=1753823598933970106 
 oil-localfs-254-2395524 [055] ..... 479912.629681: nfs_set_cache_invalid: error=0 (OK) fileid=00:7f:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598933970106 size=501300 cache_validity=0x0 () nfs_flags=0x0 ()
 oil-localfs-254-2395524 [055] ..... 479912.629681: nfs_refresh_inode_exit: error=0 (OK) fileid=00:7f:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598933970106 size=501300 cache_validity=0x0 () nfs_flags=0x0 ()
 oil-localfs-254-2395524 [055] ..... 479912.942894: nfs_getattr_enter: fileid=00:7f:10067193438 fhandle=0x14c40498 version=1753823598933970106 
 oil-localfs-254-2395524 [055] ..... 479912.942895: nfs_getattr_exit: error=0 (OK) fileid=00:7f:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598933970106 size=501300 cache_validity=0x0 () nfs_flags=0x0 ()
 oil-localfs-254-2395524 [055] ..... 479918.331078: nfs_refresh_inode_enter: fileid=00:7f:10067193438 fhandle=0x14c40498 version=1753823598933970106 
 oil-localfs-254-2395524 [055] ..... 479918.331079: nfs_set_cache_invalid: error=0 (OK) fileid=00:7f:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598933970106 size=501300 cache_validity=0x0 () nfs_flags=0x0 ()
 oil-localfs-254-2395524 [055] ..... 479918.331079: nfs_refresh_inode_exit: error=0 (OK) fileid=00:7f:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598933970106 size=501300 cache_validity=0x0 () nfs_flags=0x0 ()
 oil-localfs-254-2395524 [055] ..... 479918.679960: nfs_getattr_enter: fileid=00:7f:10067193438 fhandle=0x14c40498 version=1753823598933970106 
 oil-localfs-254-2395524 [055] ..... 479918.679960: nfs_getattr_exit: error=0 (OK) fileid=00:7f:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598933970106 size=501300 cache_validity=0x0 () nfs_flags=0x0 ()
 oil-localfs-254-2395524 [055] ..... 479920.956689: nfs_refresh_inode_enter: fileid=00:7f:10067193438 fhandle=0x14c40498 version=1753823598933970106 
 oil-localfs-254-2395524 [055] ..... 479920.956689: nfs_set_cache_invalid: error=0 (OK) fileid=00:7f:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598933970106 size=501300 cache_validity=0x0 () nfs_flags=0x0 ()
 oil-localfs-254-2395524 [055] ..... 479920.956689: nfs_refresh_inode_exit: error=0 (OK) fileid=00:7f:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598933970106 size=501300 cache_validity=0x0 () nfs_flags=0x0 ()
 oil-localfs-254-2395524 [055] ..... 479921.674574: nfs_getattr_enter: fileid=00:7f:10067193438 fhandle=0x14c40498 version=1753823598933970106 
 oil-localfs-254-2395524 [055] ..... 479921.674574: nfs_getattr_exit: error=0 (OK) fileid=00:7f:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598933970106 size=501300 cache_validity=0x0 () nfs_flags=0x0 ()
 oil-localfs-246-2395516 [150] ..... 479924.619307: nfs_refresh_inode_enter: fileid=00:7f:10067193438 fhandle=0x14c40498 version=1753823598933970106 
 oil-localfs-246-2395516 [150] ..... 479924.619307: nfs_set_cache_invalid: error=0 (OK) fileid=00:7f:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598933970106 size=501300 cache_validity=0x0 () nfs_flags=0x0 ()
 oil-localfs-246-2395516 [150] ..... 479924.619307: nfs_refresh_inode_exit: error=0 (OK) fileid=00:7f:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598933970106 size=501300 cache_validity=0x0 () nfs_flags=0x0 ()
 oil-localfs-246-2395516 [150] ..... 479925.121897: nfs_getattr_enter: fileid=00:7f:10067193438 fhandle=0x14c40498 version=1753823598933970106 
 oil-localfs-246-2395516 [150] ..... 479925.121897: nfs_getattr_exit: error=0 (OK) fileid=00:7f:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598933970106 size=501300 cache_validity=0x0 () nfs_flags=0x0 ()
 oil-localfs-246-2395516 [150] ..... 479928.930828: nfs_refresh_inode_enter: fileid=00:7f:10067193438 fhandle=0x14c40498 version=1753823598933970106 
 oil-localfs-246-2395516 [150] ..... 479928.930828: nfs_set_cache_invalid: error=0 (OK) fileid=00:7f:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598933970106 size=501300 cache_validity=0x0 () nfs_flags=0x0 ()
 oil-localfs-246-2395516 [150] ..... 479928.930828: nfs_refresh_inode_exit: error=0 (OK) fileid=00:7f:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598933970106 size=501300 cache_validity=0x0 () nfs_flags=0x0 ()
 oil-localfs-246-2395516 [150] ..... 479929.198609: nfs_getattr_enter: fileid=00:7f:10067193438 fhandle=0x14c40498 version=1753823598933970106 
 oil-localfs-246-2395516 [150] ..... 479929.198610: nfs_getattr_exit: error=0 (OK) fileid=00:7f:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598933970106 size=501300 cache_validity=0x0 () nfs_flags=0x0 ()
 oil-localfs-254-2395524 [055] ..... 479933.090988: nfs_refresh_inode_enter: fileid=00:7f:10067193438 fhandle=0x14c40498 version=1753823598933970106 
 oil-localfs-254-2395524 [055] ..... 479933.090988: nfs_set_cache_invalid: error=0 (OK) fileid=00:7f:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598933970106 size=501300 cache_validity=0x0 () nfs_flags=0x0 ()
 oil-localfs-254-2395524 [055] ..... 479933.090988: nfs_refresh_inode_exit: error=0 (OK) fileid=00:7f:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598933970106 size=501300 cache_validity=0x0 () nfs_flags=0x0 ()
 oil-localfs-254-2395524 [055] ..... 479933.470110: nfs_getattr_enter: fileid=00:7f:10067193438 fhandle=0x14c40498 version=1753823598933970106 
 oil-localfs-254-2395524 [055] ..... 479933.470111: nfs_getattr_exit: error=0 (OK) fileid=00:7f:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598933970106 size=501300 cache_validity=0x0 () nfs_flags=0x0 ()
 oil-localfs-246-2395516 [150] ..... 479937.701874: nfs_refresh_inode_enter: fileid=00:7f:10067193438 fhandle=0x14c40498 version=1753823598933970106 
 oil-localfs-246-2395516 [150] ..... 479937.701874: nfs_set_cache_invalid: error=0 (OK) fileid=00:7f:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598933970106 size=501300 cache_validity=0x0 () nfs_flags=0x0 ()
 oil-localfs-246-2395516 [150] ..... 479937.701875: nfs_refresh_inode_exit: error=0 (OK) fileid=00:7f:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598933970106 size=501300 cache_validity=0x0 () nfs_flags=0x0 ()
 oil-localfs-246-2395516 [150] ..... 479938.419888: nfs_getattr_enter: fileid=00:7f:10067193438 fhandle=0x14c40498 version=1753823598933970106 
 oil-localfs-246-2395516 [150] ..... 479938.419889: nfs_getattr_exit: error=0 (OK) fileid=00:7f:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598933970106 size=501300 cache_validity=0x0 () nfs_flags=0x0 ()
 oil-localfs-246-2395516 [150] ..... 479942.768481: nfs_refresh_inode_enter: fileid=00:7f:10067193438 fhandle=0x14c40498 version=1753823598933970106 
 oil-localfs-246-2395516 [150] ..... 479942.768481: nfs_set_cache_invalid: error=0 (OK) fileid=00:7f:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598933970106 size=501300 cache_validity=0x0 () nfs_flags=0x0 ()
 oil-localfs-246-2395516 [150] ..... 479942.768481: nfs_refresh_inode_exit: error=0 (OK) fileid=00:7f:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598933970106 size=501300 cache_validity=0x0 () nfs_flags=0x0 ()
 oil-localfs-246-2395516 [150] ..... 479943.573332: nfs_getattr_enter: fileid=00:7f:10067193438 fhandle=0x14c40498 version=1753823598933970106 
 oil-localfs-246-2395516 [150] ..... 479943.573332: nfs_getattr_exit: error=0 (OK) fileid=00:7f:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598933970106 size=501300 cache_validity=0x0 () nfs_flags=0x0 ()
 oil-localfs-254-2395524 [055] ..... 479948.195171: nfs_refresh_inode_enter: fileid=00:7f:10067193438 fhandle=0x14c40498 version=1753823598933970106 
 oil-localfs-254-2395524 [055] ..... 479948.195171: nfs_set_cache_invalid: error=0 (OK) fileid=00:7f:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598933970106 size=501300 cache_validity=0x0 () nfs_flags=0x0 ()
 oil-localfs-254-2395524 [055] ..... 479948.195171: nfs_refresh_inode_exit: error=0 (OK) fileid=00:7f:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598933970106 size=501300 cache_validity=0x0 () nfs_flags=0x0 ()
 oil-localfs-254-2395524 [055] ..... 479948.704686: nfs_getattr_enter: fileid=00:7f:10067193438 fhandle=0x14c40498 version=1753823598933970106 
 oil-localfs-254-2395524 [055] ..... 479948.704687: nfs_getattr_exit: error=0 (OK) fileid=00:7f:10067193438 fhandle=0x14c40498 type=8 (REG) version=1753823598933970106 size=501300 cache_validity=0x0 () nfs_flags=0x0 ()


[-- Attachment #3: 20250725.txt --]
[-- Type: text/plain, Size: 57239 bytes --]

[root@ocloud0094.03.oas1 ~]# grep 10056165185  /sys/kernel/debug/tracing/trace
 oil-localfs-252-2605046 [162] ..... 46138.503632: nfs_getattr_enter: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 
 oil-localfs-252-2605046 [162] ..... 46138.503633: nfs_getattr_exit: error=0 (OK) fileid=00:aa:10056165185 fhandle=0x8bfc64c9 type=8 (REG) version=1753485366409158129 size=0 cache_validity=0x0 () nfs_flags=0x4 (ACL_LRU_SET)
 oil-localfs-252-2605046 [162] ..... 46138.503654: nfs_getattr_enter: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 
 oil-localfs-252-2605046 [162] ..... 46138.503654: nfs_getattr_exit: error=0 (OK) fileid=00:aa:10056165185 fhandle=0x8bfc64c9 type=8 (REG) version=1753485366409158129 size=0 cache_validity=0x0 () nfs_flags=0x4 (ACL_LRU_SET)
 oil-localfs-252-2605046 [162] ..... 46138.504437: nfs_getattr_enter: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 
 oil-localfs-252-2605046 [162] ..... 46138.504437: nfs_getattr_exit: error=0 (OK) fileid=00:aa:10056165185 fhandle=0x8bfc64c9 type=8 (REG) version=1753485366409158129 size=0 cache_validity=0x0 () nfs_flags=0x4 (ACL_LRU_SET)
 oil-localfs-252-2605046 [162] ..... 46138.504441: nfs_getattr_enter: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 
 oil-localfs-252-2605046 [162] ..... 46138.504441: nfs_getattr_exit: error=0 (OK) fileid=00:aa:10056165185 fhandle=0x8bfc64c9 type=8 (REG) version=1753485366409158129 size=0 cache_validity=0x0 () nfs_flags=0x4 (ACL_LRU_SET)
 oil-localfs-252-2605046 [162] ..... 46138.512974: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=0 newsize=4096
 oil-localfs-252-2605046 [162] ..... 46138.512983: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=4096 newsize=5013
 oil-localfs-252-2605046 [162] ..... 46138.513247: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=5013 newsize=8192
 oil-localfs-252-2605046 [162] ..... 46138.513434: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=8192 newsize=10026
 oil-localfs-252-2605046 [162] ..... 46138.513671: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=10026 newsize=12288
 oil-localfs-252-2605046 [162] ..... 46138.513879: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=12288 newsize=15039
 oil-localfs-252-2605046 [162] ..... 46138.514064: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=15039 newsize=16384
 oil-localfs-252-2605046 [162] ..... 46138.514240: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=16384 newsize=20052
 oil-localfs-252-2605046 [162] ..... 46138.514436: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=20052 newsize=20480
 oil-localfs-252-2605046 [162] ..... 46138.514593: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=20480 newsize=24576
 oil-localfs-252-2605046 [162] ..... 46138.514597: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=24576 newsize=25065
 oil-localfs-252-2605046 [162] ..... 46138.514802: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=25065 newsize=28672
 oil-localfs-252-2605046 [162] ..... 46138.514971: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=28672 newsize=30078
 oil-localfs-252-2605046 [162] ..... 46138.515150: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=30078 newsize=32768
 oil-localfs-252-2605046 [162] ..... 46138.515299: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=32768 newsize=35091
 oil-localfs-252-2605046 [162] ..... 46138.515481: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=35091 newsize=36864
 oil-localfs-252-2605046 [162] ..... 46138.515689: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=36864 newsize=40104
 oil-localfs-252-2605046 [162] ..... 46138.515871: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=40104 newsize=40960
 oil-localfs-252-2605046 [162] ..... 46138.516080: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=40960 newsize=45056
 oil-localfs-252-2605046 [162] ..... 46138.516085: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=45056 newsize=45117
 oil-localfs-252-2605046 [162] ..... 46138.516255: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=45117 newsize=49152
 oil-localfs-252-2605046 [162] ..... 46138.516414: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=49152 newsize=50130
 oil-localfs-252-2605046 [162] ..... 46138.516593: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=50130 newsize=53248
 kworker/u1038:1-2597138 [106] ..... 46138.516740: nfs_initiate_write: fileid=00:aa:10056165185 fhandle=0x6bd94d55 offset=0 count=53248 stable=UNSTABLE
 oil-localfs-252-2605046 [162] ..... 46138.516753: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=53248 newsize=55143
 kworker/u1038:1-2597138 [106] ..... 46138.516759: nfs_initiate_write: fileid=00:aa:10056165185 fhandle=0xbb9c42be offset=0 count=53248 stable=UNSTABLE
 kworker/u1038:1-2597138 [106] ..... 46138.516778: nfs_initiate_write: fileid=00:aa:10056165185 fhandle=0x297952b1 offset=0 count=53248 stable=UNSTABLE
 kworker/u1038:1-2597138 [106] ..... 46138.516795: nfs_initiate_write: fileid=00:aa:10056165185 fhandle=0x6bd94d55 offset=53248 count=1895 stable=UNSTABLE
 kworker/u1038:1-2597138 [106] ..... 46138.516796: nfs_initiate_write: fileid=00:aa:10056165185 fhandle=0xbb9c42be offset=53248 count=1895 stable=UNSTABLE
 kworker/u1038:1-2597138 [106] ..... 46138.516798: nfs_initiate_write: fileid=00:aa:10056165185 fhandle=0x297952b1 offset=53248 count=1895 stable=UNSTABLE
 kworker/u1038:1-2593545 [108] ..... 46138.517622: nfs4_pnfs_write: error=0 (OK) fileid=00:aa:10056165185 fhandle=0xbb9c42be offset=0 count=53248 res=53248 stateid=1:0x79a9c471 layoutstateid=1:0xcbd8aaad
 kworker/u1038:1-2593545 [108] ..... 46138.517623: nfs_set_cache_invalid: error=0 (OK) fileid=00:aa:10056165185 fhandle=0x8bfc64c9 type=8 (REG) version=1753485366409158129 size=55143 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1038:1-2593545 [108] ..... 46138.517623: nfs_writeback_done: error=53248 fileid=00:aa:10056165185 fhandle=0xbb9c42be offset=0 count=53248 res=53248 stable=UNSTABLE verifier=d807dfba72196253
 kworker/u1037:2-2871862 [097] ..... 46138.517659: nfs4_pnfs_write: error=0 (OK) fileid=00:aa:10056165185 fhandle=0x6bd94d55 offset=0 count=53248 res=53248 stateid=1:0x79a9c471 layoutstateid=1:0xcbd8aaad
 kworker/u1037:1-2779638 [099] ..... 46138.517659: nfs4_pnfs_write: error=0 (OK) fileid=00:aa:10056165185 fhandle=0x297952b1 offset=0 count=53248 res=53248 stateid=1:0x79a9c471 layoutstateid=1:0xcbd8aaad
 kworker/u1037:2-2871862 [097] ..... 46138.517661: nfs_set_cache_invalid: error=0 (OK) fileid=00:aa:10056165185 fhandle=0x8bfc64c9 type=8 (REG) version=1753485366409158129 size=55143 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:2-2871862 [097] ..... 46138.517662: nfs_writeback_done: error=53248 fileid=00:aa:10056165185 fhandle=0x6bd94d55 offset=0 count=53248 res=53248 stable=UNSTABLE verifier=5199cdae2816c899
 kworker/u1037:1-2779638 [099] ..... 46138.517662: nfs_set_cache_invalid: error=0 (OK) fileid=00:aa:10056165185 fhandle=0x8bfc64c9 type=8 (REG) version=1753485366409158129 size=55143 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:1-2779638 [099] ..... 46138.517663: nfs_writeback_done: error=53248 fileid=00:aa:10056165185 fhandle=0x297952b1 offset=0 count=53248 res=53248 stable=UNSTABLE verifier=a5cd34d9a5d3efdb
 kworker/u1038:1-2593545 [108] ..... 46138.517665: nfs4_pnfs_write: error=0 (OK) fileid=00:aa:10056165185 fhandle=0xbb9c42be offset=53248 count=1895 res=1895 stateid=1:0x79a9c471 layoutstateid=1:0xcbd8aaad
 kworker/u1038:1-2593545 [108] ..... 46138.517666: nfs_set_cache_invalid: error=0 (OK) fileid=00:aa:10056165185 fhandle=0x8bfc64c9 type=8 (REG) version=1753485366409158129 size=55143 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1038:1-2593545 [108] ..... 46138.517666: nfs_refresh_inode_enter: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 
 kworker/u1038:1-2593545 [108] ..... 46138.517667: nfs_set_cache_invalid: error=0 (OK) fileid=00:aa:10056165185 fhandle=0x8bfc64c9 type=8 (REG) version=1753485366409158129 size=55143 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1038:1-2593545 [108] ..... 46138.517668: nfs_refresh_inode_exit: error=0 (OK) fileid=00:aa:10056165185 fhandle=0x8bfc64c9 type=8 (REG) version=1753485366409158129 size=55143 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1038:1-2593545 [108] ..... 46138.517668: nfs_writeback_done: error=1895 fileid=00:aa:10056165185 fhandle=0xbb9c42be offset=53248 count=1895 res=1895 stable=UNSTABLE verifier=d807dfba72196253
 kworker/u1037:5-2593935 [226] ..... 46138.517669: nfs4_pnfs_write: error=0 (OK) fileid=00:aa:10056165185 fhandle=0x6bd94d55 offset=53248 count=1895 res=1895 stateid=1:0x79a9c471 layoutstateid=1:0xcbd8aaad
 kworker/u1037:5-2593935 [226] ..... 46138.517670: nfs_set_cache_invalid: error=0 (OK) fileid=00:aa:10056165185 fhandle=0x8bfc64c9 type=8 (REG) version=1753485366409158129 size=55143 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:5-2593935 [226] ..... 46138.517671: nfs_refresh_inode_enter: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 
 kworker/u1037:5-2593935 [226] ..... 46138.517671: nfs_set_cache_invalid: error=0 (OK) fileid=00:aa:10056165185 fhandle=0x8bfc64c9 type=8 (REG) version=1753485366409158129 size=55143 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:5-2593935 [226] ..... 46138.517672: nfs_refresh_inode_exit: error=0 (OK) fileid=00:aa:10056165185 fhandle=0x8bfc64c9 type=8 (REG) version=1753485366409158129 size=55143 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:5-2593935 [226] ..... 46138.517672: nfs_writeback_done: error=1895 fileid=00:aa:10056165185 fhandle=0x6bd94d55 offset=53248 count=1895 res=1895 stable=UNSTABLE verifier=5199cdae2816c899
 kworker/u1037:4-2593530 [096] ..... 46138.517688: nfs4_pnfs_write: error=0 (OK) fileid=00:aa:10056165185 fhandle=0x297952b1 offset=53248 count=1895 res=1895 stateid=1:0x79a9c471 layoutstateid=1:0xcbd8aaad
 kworker/u1037:4-2593530 [096] ..... 46138.517689: nfs_set_cache_invalid: error=0 (OK) fileid=00:aa:10056165185 fhandle=0x8bfc64c9 type=8 (REG) version=1753485366409158129 size=55143 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:4-2593530 [096] ..... 46138.517690: nfs_refresh_inode_enter: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 
 kworker/u1037:4-2593530 [096] ..... 46138.517690: nfs_set_cache_invalid: error=0 (OK) fileid=00:aa:10056165185 fhandle=0x8bfc64c9 type=8 (REG) version=1753485366409158129 size=55143 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:4-2593530 [096] ..... 46138.517690: nfs_refresh_inode_exit: error=0 (OK) fileid=00:aa:10056165185 fhandle=0x8bfc64c9 type=8 (REG) version=1753485366409158129 size=55143 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:4-2593530 [096] ..... 46138.517691: nfs_writeback_done: error=1895 fileid=00:aa:10056165185 fhandle=0x297952b1 offset=53248 count=1895 res=1895 stable=UNSTABLE verifier=a5cd34d9a5d3efdb
 oil-localfs-252-2605046 [162] ..... 46138.518360: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=55143 newsize=57344
 oil-localfs-252-2605046 [162] ..... 46138.518556: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=57344 newsize=60156
 oil-localfs-252-2605046 [162] ..... 46138.518760: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=60156 newsize=61440
 oil-localfs-252-2605046 [162] ..... 46138.518932: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=61440 newsize=65169
 oil-localfs-252-2605046 [162] ..... 46138.519123: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=65169 newsize=65536
 oil-localfs-252-2605046 [162] ..... 46138.519308: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=65536 newsize=69632
 oil-localfs-252-2605046 [162] ..... 46138.519312: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=69632 newsize=70182
 oil-localfs-252-2605046 [162] ..... 46138.519499: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=70182 newsize=73728
 oil-localfs-252-2605046 [162] ..... 46138.519662: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=73728 newsize=75195
 oil-localfs-252-2605046 [162] ..... 46138.519854: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=75195 newsize=77824
 oil-localfs-252-2605046 [162] ..... 46138.520022: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=77824 newsize=80208
 oil-localfs-252-2605046 [162] ..... 46138.520208: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=80208 newsize=81920
 oil-localfs-252-2605046 [162] ..... 46138.520379: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=81920 newsize=85221
 oil-localfs-252-2605046 [162] ..... 46138.520575: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=85221 newsize=86016
 oil-localfs-252-2605046 [162] ..... 46138.520767: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=86016 newsize=90112
 oil-localfs-252-2605046 [162] ..... 46138.520771: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=90112 newsize=90234
 oil-localfs-252-2605046 [162] ..... 46138.520963: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=90234 newsize=94208
 oil-localfs-252-2605046 [162] ..... 46138.521123: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=94208 newsize=95247
 oil-localfs-252-2605046 [162] ..... 46138.521311: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=95247 newsize=98304
 oil-localfs-252-2605046 [162] ..... 46138.521452: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=98304 newsize=100260
 oil-localfs-252-2605046 [162] ..... 46138.521624: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=100260 newsize=102400
 oil-localfs-252-2605046 [162] ..... 46138.521794: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=102400 newsize=105273
 oil-localfs-252-2605046 [162] ..... 46138.521977: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=105273 newsize=106496
 oil-localfs-252-2605046 [162] ..... 46138.522180: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=106496 newsize=110286
 oil-localfs-252-2605046 [162] ..... 46138.522365: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=110286 newsize=110592
 oil-localfs-252-2605046 [162] ..... 46138.522531: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=110592 newsize=114688
 oil-localfs-252-2605046 [162] ..... 46138.522535: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=114688 newsize=115299
 oil-localfs-252-2605046 [162] ..... 46138.522746: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=115299 newsize=118784
 oil-localfs-252-2605046 [162] ..... 46138.522907: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=118784 newsize=120312
 oil-localfs-252-2605046 [162] ..... 46138.523094: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=120312 newsize=122880
 oil-localfs-252-2605046 [162] ..... 46138.523251: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=122880 newsize=125325
 oil-localfs-252-2605046 [162] ..... 46138.523438: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=125325 newsize=126976
 oil-localfs-252-2605046 [162] ..... 46138.523619: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=126976 newsize=130338
 oil-localfs-252-2605046 [162] ..... 46138.523828: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=130338 newsize=131072
 oil-localfs-252-2605046 [162] ..... 46138.524037: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=131072 newsize=135168
 oil-localfs-252-2605046 [162] ..... 46138.524041: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=135168 newsize=135351
 oil-localfs-252-2605046 [162] ..... 46138.524245: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=135351 newsize=139264
 oil-localfs-252-2605046 [162] ..... 46138.524423: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=139264 newsize=140364
 oil-localfs-252-2605046 [162] ..... 46138.524631: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=140364 newsize=143360
 oil-localfs-252-2605046 [162] ..... 46138.524788: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=143360 newsize=145377
 oil-localfs-252-2605046 [162] ..... 46138.525001: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=145377 newsize=147456
 oil-localfs-252-2605046 [162] ..... 46138.525180: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=147456 newsize=150390
 oil-localfs-252-2605046 [162] ..... 46138.525366: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=150390 newsize=151552
 oil-localfs-252-2605046 [162] ..... 46138.525529: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=151552 newsize=155403
 oil-localfs-252-2605046 [162] ..... 46138.525750: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=155403 newsize=155648
 oil-localfs-252-2605046 [162] ..... 46138.525947: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=155648 newsize=159744
 oil-localfs-252-2605046 [162] ..... 46138.525952: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=159744 newsize=160416
 oil-localfs-252-2605046 [162] ..... 46138.526172: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=160416 newsize=163840
 oil-localfs-252-2605046 [162] ..... 46138.526354: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=163840 newsize=165429
 oil-localfs-252-2605046 [162] ..... 46138.526542: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=165429 newsize=167936
 oil-localfs-252-2605046 [162] ..... 46138.526713: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=167936 newsize=170442
 oil-localfs-252-2605046 [162] ..... 46138.526932: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=170442 newsize=172032
 oil-localfs-252-2605046 [162] ..... 46138.527095: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=172032 newsize=175455
 oil-localfs-252-2605046 [162] ..... 46138.527301: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=175455 newsize=176128
 oil-localfs-252-2605046 [162] ..... 46138.527564: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=176128 newsize=180224
 oil-localfs-252-2605046 [162] ..... 46138.527568: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=180224 newsize=180468
 oil-localfs-252-2605046 [162] ..... 46138.527774: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=180468 newsize=184320
 oil-localfs-252-2605046 [162] ..... 46138.527933: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=184320 newsize=185481
 oil-localfs-252-2605046 [162] ..... 46138.528148: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=185481 newsize=188416
 oil-localfs-252-2605046 [162] ..... 46138.528297: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=188416 newsize=190494
 oil-localfs-252-2605046 [162] ..... 46138.528498: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=190494 newsize=192512
 oil-localfs-252-2605046 [162] ..... 46138.528748: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=192512 newsize=195507
 oil-localfs-252-2605046 [162] ..... 46138.528935: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=195507 newsize=196608
 oil-localfs-252-2605046 [162] ..... 46138.529090: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=196608 newsize=200520
 oil-localfs-252-2605046 [162] ..... 46138.529296: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=200520 newsize=200704
 oil-localfs-252-2605046 [162] ..... 46138.529451: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=200704 newsize=204800
 oil-localfs-252-2605046 [162] ..... 46138.529455: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=204800 newsize=205533
 oil-localfs-252-2605046 [162] ..... 46138.529654: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=205533 newsize=208896
 oil-localfs-252-2605046 [162] ..... 46138.529827: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=208896 newsize=210546
 oil-localfs-252-2605046 [162] ..... 46138.530021: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=210546 newsize=212992
 oil-localfs-252-2605046 [162] ..... 46138.530182: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=212992 newsize=215559
 oil-localfs-252-2605046 [162] ..... 46138.530401: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=215559 newsize=217088
 oil-localfs-252-2605046 [162] ..... 46138.530562: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=217088 newsize=220572
 oil-localfs-252-2605046 [162] ..... 46138.530790: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=220572 newsize=221184
 oil-localfs-252-2605046 [162] ..... 46138.530985: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=221184 newsize=225280
 oil-localfs-252-2605046 [162] ..... 46138.530990: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=225280 newsize=225585
 oil-localfs-252-2605046 [162] ..... 46138.531177: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=225585 newsize=229376
 oil-localfs-252-2605046 [162] ..... 46138.531336: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=229376 newsize=230598
 oil-localfs-252-2605046 [162] ..... 46138.531544: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=230598 newsize=233472
 oil-localfs-252-2605046 [162] ..... 46138.531714: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=233472 newsize=235611
 oil-localfs-252-2605046 [162] ..... 46138.531917: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=235611 newsize=237568
 oil-localfs-252-2605046 [162] ..... 46138.532144: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=237568 newsize=240624
 oil-localfs-252-2605046 [162] ..... 46138.532345: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=240624 newsize=241664
 oil-localfs-252-2605046 [162] ..... 46138.532541: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=241664 newsize=245637
 oil-localfs-252-2605046 [162] ..... 46138.532782: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=245637 newsize=245760
 oil-localfs-252-2605046 [162] ..... 46138.532978: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=245760 newsize=249856
 oil-localfs-252-2605046 [162] ..... 46138.532982: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=249856 newsize=250650
 oil-localfs-252-2605046 [162] ..... 46138.533177: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=250650 newsize=253952
 oil-localfs-252-2605046 [162] ..... 46138.533365: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=253952 newsize=255663
 oil-localfs-252-2605046 [162] ..... 46138.533553: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=255663 newsize=258048
 oil-localfs-252-2605046 [162] ..... 46138.533715: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=258048 newsize=260676
 oil-localfs-252-2605046 [162] ..... 46138.533926: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=260676 newsize=262144
 oil-localfs-252-2605046 [162] ..... 46138.534105: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=262144 newsize=265689
 oil-localfs-252-2605046 [162] ..... 46138.534309: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=265689 newsize=266240
 oil-localfs-252-2605046 [162] ..... 46138.534575: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=266240 newsize=270336
 oil-localfs-252-2605046 [162] ..... 46138.534579: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=270336 newsize=270702
 oil-localfs-252-2605046 [162] ..... 46138.534783: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=270702 newsize=274432
 oil-localfs-252-2605046 [162] ..... 46138.534940: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=274432 newsize=275715
 oil-localfs-252-2605046 [162] ..... 46138.535155: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=275715 newsize=278528
 oil-localfs-252-2605046 [162] ..... 46138.535297: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=278528 newsize=280728
 oil-localfs-252-2605046 [162] ..... 46138.535484: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=280728 newsize=282624
 oil-localfs-252-2605046 [162] ..... 46138.535695: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=282624 newsize=285741
 oil-localfs-252-2605046 [162] ..... 46138.535879: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=285741 newsize=286720
 oil-localfs-252-2605046 [162] ..... 46138.536042: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=286720 newsize=290754
 oil-localfs-252-2605046 [162] ..... 46138.536255: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=290754 newsize=290816
 oil-localfs-252-2605046 [162] ..... 46138.536419: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=290816 newsize=294912
 oil-localfs-252-2605046 [162] ..... 46138.536422: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=294912 newsize=295767
 oil-localfs-252-2605046 [162] ..... 46138.536626: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=295767 newsize=299008
 oil-localfs-252-2605046 [162] ..... 46138.536816: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=299008 newsize=300780
 oil-localfs-252-2605046 [162] ..... 46138.537011: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=300780 newsize=303104
 oil-localfs-252-2605046 [162] ..... 46138.537184: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=303104 newsize=305793
 oil-localfs-252-2605046 [163] ..... 46138.537434: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=305793 newsize=307200
 oil-localfs-252-2605046 [163] ..... 46138.537623: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=307200 newsize=310806
 oil-localfs-252-2605046 [163] ..... 46138.537848: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=310806 newsize=311296
 oil-localfs-252-2605046 [163] ..... 46138.538046: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=311296 newsize=315392
 oil-localfs-252-2605046 [163] ..... 46138.538049: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=315392 newsize=315819
 oil-localfs-252-2605046 [163] ..... 46138.538235: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=315819 newsize=319488
 oil-localfs-252-2605046 [163] ..... 46138.538408: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=319488 newsize=320832
 oil-localfs-252-2605046 [163] ..... 46138.538621: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=320832 newsize=323584
 oil-localfs-252-2605046 [163] ..... 46138.538837: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=323584 newsize=325845
 oil-localfs-252-2605046 [163] ..... 46138.539032: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=325845 newsize=327680
 oil-localfs-252-2605046 [163] ..... 46138.539214: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=327680 newsize=330858
 oil-localfs-252-2605046 [163] ..... 46138.539379: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=330858 newsize=331776
 oil-localfs-252-2605046 [163] ..... 46138.539552: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=331776 newsize=335871
 oil-localfs-252-2605046 [163] ..... 46138.539750: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=335871 newsize=335872
 oil-localfs-252-2605046 [163] ..... 46138.539936: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=335872 newsize=339968
 oil-localfs-252-2605046 [163] ..... 46138.539940: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=339968 newsize=340884
 oil-localfs-252-2605046 [163] ..... 46138.540117: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=340884 newsize=344064
 oil-localfs-252-2605046 [163] ..... 46138.540299: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=344064 newsize=345897
 oil-localfs-252-2605046 [163] ..... 46138.540454: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=345897 newsize=348160
 oil-localfs-252-2605046 [163] ..... 46138.540611: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=348160 newsize=350910
 oil-localfs-252-2605046 [163] ..... 46138.540820: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=350910 newsize=352256
 oil-localfs-252-2605046 [163] ..... 46138.540974: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=352256 newsize=355923
 oil-localfs-252-2605046 [163] ..... 46138.541159: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=355923 newsize=356352
 oil-localfs-252-2605046 [163] ..... 46138.541345: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=356352 newsize=360448
 oil-localfs-252-2605046 [163] ..... 46138.541348: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=360448 newsize=360936
 oil-localfs-252-2605046 [163] ..... 46138.541514: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=360936 newsize=364544
 oil-localfs-252-2605046 [163] ..... 46138.541686: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=364544 newsize=365949
 oil-localfs-252-2605046 [163] ..... 46138.541893: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=365949 newsize=368640
 oil-localfs-252-2605046 [163] ..... 46138.542049: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=368640 newsize=370962
 oil-localfs-252-2605046 [163] ..... 46138.542229: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=370962 newsize=372736
 oil-localfs-252-2605046 [163] ..... 46138.542400: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=372736 newsize=375975
 oil-localfs-252-2605046 [163] ..... 46138.542559: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=375975 newsize=376832
 oil-localfs-252-2605046 [163] ..... 46138.542736: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=376832 newsize=380928
 oil-localfs-252-2605046 [163] ..... 46138.542739: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=380928 newsize=380988
 oil-localfs-252-2605046 [163] ..... 46138.542939: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=380988 newsize=385024
 oil-localfs-252-2605046 [163] ..... 46138.543093: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=385024 newsize=386001
 oil-localfs-252-2605046 [163] ..... 46138.543270: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=386001 newsize=389120
 oil-localfs-252-2605046 [163] ..... 46138.543450: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=389120 newsize=391014
 oil-localfs-252-2605046 [163] ..... 46138.543613: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=391014 newsize=393216
 oil-localfs-252-2605046 [163] ..... 46138.543782: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=393216 newsize=396027
 oil-localfs-252-2605046 [163] ..... 46138.543977: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=396027 newsize=397312
 oil-localfs-252-2605046 [163] ..... 46138.544167: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=397312 newsize=401040
 oil-localfs-252-2605046 [163] ..... 46138.544339: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=401040 newsize=401408
 oil-localfs-252-2605046 [163] ..... 46138.544504: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=401408 newsize=405504
 oil-localfs-252-2605046 [163] ..... 46138.544507: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=405504 newsize=406053
 oil-localfs-252-2605046 [163] ..... 46138.544677: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=406053 newsize=409600
 oil-localfs-252-2605046 [163] ..... 46138.544838: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=409600 newsize=411066
 oil-localfs-252-2605046 [163] ..... 46138.545022: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=411066 newsize=413696
 oil-localfs-252-2605046 [163] ..... 46138.545171: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=413696 newsize=416079
 oil-localfs-252-2605046 [163] ..... 46138.545350: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=416079 newsize=417792
 oil-localfs-252-2605046 [163] ..... 46138.545526: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=417792 newsize=421092
 oil-localfs-252-2605046 [163] ..... 46138.545695: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=421092 newsize=421888
 oil-localfs-252-2605046 [163] ..... 46138.545863: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=421888 newsize=425984
 oil-localfs-252-2605046 [163] ..... 46138.545869: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=425984 newsize=426105
 oil-localfs-252-2605046 [163] ..... 46138.546055: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=426105 newsize=430080
 oil-localfs-252-2605046 [163] ..... 46138.546201: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=430080 newsize=431118
 oil-localfs-252-2605046 [163] ..... 46138.546382: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=431118 newsize=434176
 oil-localfs-252-2605046 [163] ..... 46138.546545: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=434176 newsize=436131
 oil-localfs-252-2605046 [163] ..... 46138.546715: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=436131 newsize=438272
 oil-localfs-252-2605046 [163] ..... 46138.546890: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=438272 newsize=441144
 oil-localfs-252-2605046 [163] ..... 46138.547083: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=441144 newsize=442368
 oil-localfs-252-2605046 [163] ..... 46138.547249: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=442368 newsize=446157
 oil-localfs-252-2605046 [163] ..... 46138.547435: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=446157 newsize=446464
 oil-localfs-252-2605046 [163] ..... 46138.547613: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=446464 newsize=450560
 oil-localfs-252-2605046 [163] ..... 46138.547616: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=450560 newsize=451170
 oil-localfs-252-2605046 [163] ..... 46138.547796: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=451170 newsize=454656
 oil-localfs-252-2605046 [163] ..... 46138.547962: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=454656 newsize=456183
 oil-localfs-252-2605046 [163] ..... 46138.548143: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=456183 newsize=458752
 oil-localfs-252-2605046 [163] ..... 46138.548293: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=458752 newsize=461196
 oil-localfs-252-2605046 [163] ..... 46138.548461: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=461196 newsize=462848
 oil-localfs-252-2605046 [163] ..... 46138.548620: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=462848 newsize=466209
 oil-localfs-252-2605046 [163] ..... 46138.548787: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=466209 newsize=466944
 oil-localfs-252-2605046 [163] ..... 46138.548967: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=466944 newsize=471040
 oil-localfs-252-2605046 [163] ..... 46138.548970: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=471040 newsize=471222
 oil-localfs-252-2605046 [163] ..... 46138.549151: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=471222 newsize=475136
 oil-localfs-252-2605046 [163] ..... 46138.549297: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=475136 newsize=476235
 oil-localfs-252-2605046 [163] ..... 46138.549473: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=476235 newsize=479232
 oil-localfs-252-2605046 [163] ..... 46138.549639: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=479232 newsize=481248
 oil-localfs-252-2605046 [163] ..... 46138.549812: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=481248 newsize=483328
 oil-localfs-252-2605046 [163] ..... 46138.549963: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=483328 newsize=486261
 oil-localfs-252-2605046 [163] ..... 46138.550144: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=486261 newsize=487424
 oil-localfs-252-2605046 [163] ..... 46138.550325: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=487424 newsize=491274
 oil-localfs-252-2605046 [163] ..... 46138.550494: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=491274 newsize=491520
 oil-localfs-252-2605046 [163] ..... 46138.550680: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=491520 newsize=495616
 oil-localfs-252-2605046 [163] ..... 46138.550684: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=495616 newsize=496287
 oil-localfs-252-2605046 [163] ..... 46138.550845: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=496287 newsize=499712
 oil-localfs-252-2605046 [163] ..... 46138.551001: nfs_size_grow: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 cursize=499712 newsize=501300
 oil-localfs-252-2605046 [163] ..... 46138.551210: nfs_fsync_enter: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 
 oil-localfs-252-2605046 [163] ..... 46138.551459: nfs_initiate_write: fileid=00:aa:10056165185 fhandle=0x6bd94d55 offset=57344 count=443956 stable=FILE_SYNC
 oil-localfs-252-2605046 [163] ..... 46138.551594: nfs_initiate_write: fileid=00:aa:10056165185 fhandle=0xbb9c42be offset=57344 count=443956 stable=FILE_SYNC
 oil-localfs-252-2605046 [163] ..... 46138.551731: nfs_initiate_write: fileid=00:aa:10056165185 fhandle=0x297952b1 offset=57344 count=443956 stable=FILE_SYNC
 kworker/u1037:5-2593935 [226] ..... 46138.552991: nfs4_pnfs_write: error=0 (OK) fileid=00:aa:10056165185 fhandle=0x6bd94d55 offset=57344 count=443956 res=443956 stateid=1:0x79a9c471 layoutstateid=1:0xcbd8aaad
 kworker/u1037:5-2593935 [226] ..... 46138.552993: nfs_set_cache_invalid: error=0 (OK) fileid=00:aa:10056165185 fhandle=0x8bfc64c9 type=8 (REG) version=1753485366409158129 size=501300 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:5-2593935 [226] ..... 46138.552994: nfs_refresh_inode_enter: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 
 kworker/u1037:5-2593935 [226] ..... 46138.552995: nfs_set_cache_invalid: error=0 (OK) fileid=00:aa:10056165185 fhandle=0x8bfc64c9 type=8 (REG) version=1753485366409158129 size=501300 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:5-2593935 [226] ..... 46138.552995: nfs_refresh_inode_exit: error=0 (OK) fileid=00:aa:10056165185 fhandle=0x8bfc64c9 type=8 (REG) version=1753485366409158129 size=501300 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:5-2593935 [226] ..... 46138.552995: nfs_writeback_done: error=443956 fileid=00:aa:10056165185 fhandle=0x6bd94d55 offset=57344 count=443956 res=443956 stable=FILE_SYNC verifier=5199cdae2816c899
 kworker/u1037:5-2593935 [226] ..... 46138.553238: nfs4_pnfs_write: error=0 (OK) fileid=00:aa:10056165185 fhandle=0x297952b1 offset=57344 count=443956 res=443956 stateid=1:0x79a9c471 layoutstateid=1:0xcbd8aaad
 kworker/u1037:5-2593935 [226] ..... 46138.553239: nfs_set_cache_invalid: error=0 (OK) fileid=00:aa:10056165185 fhandle=0x8bfc64c9 type=8 (REG) version=1753485366409158129 size=501300 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:5-2593935 [226] ..... 46138.553239: nfs_refresh_inode_enter: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 
 kworker/u1037:5-2593935 [226] ..... 46138.553240: nfs_set_cache_invalid: error=0 (OK) fileid=00:aa:10056165185 fhandle=0x8bfc64c9 type=8 (REG) version=1753485366409158129 size=501300 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:5-2593935 [226] ..... 46138.553240: nfs_refresh_inode_exit: error=0 (OK) fileid=00:aa:10056165185 fhandle=0x8bfc64c9 type=8 (REG) version=1753485366409158129 size=501300 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1037:5-2593935 [226] ..... 46138.553240: nfs_writeback_done: error=443956 fileid=00:aa:10056165185 fhandle=0x297952b1 offset=57344 count=443956 res=443956 stable=FILE_SYNC verifier=a5cd34d9a5d3efdb
 kworker/u1038:1-2593545 [108] ..... 46138.553322: nfs4_pnfs_write: error=0 (OK) fileid=00:aa:10056165185 fhandle=0xbb9c42be offset=57344 count=443956 res=443956 stateid=1:0x79a9c471 layoutstateid=1:0xcbd8aaad
 kworker/u1038:1-2593545 [108] ..... 46138.553324: nfs_set_cache_invalid: error=0 (OK) fileid=00:aa:10056165185 fhandle=0x8bfc64c9 type=8 (REG) version=1753485366409158129 size=501300 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1038:1-2593545 [108] ..... 46138.553325: nfs_refresh_inode_enter: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 
 kworker/u1038:1-2593545 [108] ..... 46138.553326: nfs_set_cache_invalid: error=0 (OK) fileid=00:aa:10056165185 fhandle=0x8bfc64c9 type=8 (REG) version=1753485366409158129 size=501300 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1038:1-2593545 [108] ..... 46138.553326: nfs_refresh_inode_exit: error=0 (OK) fileid=00:aa:10056165185 fhandle=0x8bfc64c9 type=8 (REG) version=1753485366409158129 size=501300 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1038:1-2593545 [108] ..... 46138.553326: nfs_writeback_done: error=443956 fileid=00:aa:10056165185 fhandle=0xbb9c42be offset=57344 count=443956 res=443956 stable=FILE_SYNC verifier=d807dfba72196253
 oil-localfs-252-2605046 [163] ..... 46138.553519: nfs_fsync_exit: error=0 (OK) fileid=00:aa:10056165185 fhandle=0x8bfc64c9 type=8 (REG) version=1753485366409158129 size=501300 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 oil-localfs-252-2605046 [163] ..... 46138.553740: nfs_writeback_inode_enter: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 
 oil-localfs-252-2605046 [163] ..... 46138.553743: nfs_writeback_inode_exit: error=0 (OK) fileid=00:aa:10056165185 fhandle=0x8bfc64c9 type=8 (REG) version=1753485366409158129 size=501300 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1025:1-1058132 [006] ..... 46138.584970: nfs_refresh_inode_enter: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366409158129 
 kworker/u1025:1-1058132 [006] ..... 46138.584973: nfs_set_cache_invalid: error=0 (OK) fileid=00:aa:10056165185 fhandle=0x8bfc64c9 type=8 (REG) version=1753485366506226295 size=501300 cache_validity=0x3101a (INVALID_DATA|INVALID_ACCESS|INVALID_ACL|INVALID_OTHER|INVALID_NLINK|INVALID_MODE) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1025:1-1058132 [006] ..... 46138.584973: nfs_refresh_inode_exit: error=0 (OK) fileid=00:aa:10056165185 fhandle=0x8bfc64c9 type=8 (REG) version=1753485366506226295 size=501300 cache_validity=0x3101a (INVALID_DATA|INVALID_ACCESS|INVALID_ACL|INVALID_OTHER|INVALID_NLINK|INVALID_MODE) nfs_flags=0x4 (ACL_LRU_SET)
 oil-localfs-252-2605046 [007] ..... 46138.591925: nfs_writeback_inode_enter: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366506226295 
 oil-localfs-252-2605046 [007] ..... 46138.591932: nfs_writeback_inode_exit: error=0 (OK) fileid=00:aa:10056165185 fhandle=0x8bfc64c9 type=8 (REG) version=1753485366506226295 size=501300 cache_validity=0x3101a (INVALID_DATA|INVALID_ACCESS|INVALID_ACL|INVALID_OTHER|INVALID_NLINK|INVALID_MODE) nfs_flags=0x4 (ACL_LRU_SET)
 oil-localfs-252-2605046 [007] ..... 46138.629684: nfs_set_cache_invalid: error=0 (OK) fileid=00:aa:10056165185 fhandle=0x8bfc64c9 type=8 (REG) version=1753485366506226295 size=501300 cache_validity=0x31b1a (INVALID_DATA|INVALID_ACCESS|INVALID_ACL|INVALID_CHANGE|INVALID_CTIME|INVALID_SIZE|INVALID_OTHER|INVALID_NLINK|INVALID_MODE) nfs_flags=0x4 (ACL_LRU_SET)
 oil-localfs-252-2605046 [007] ..... 46138.728413: nfs_refresh_inode_enter: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366506226295 
 oil-localfs-252-2605046 [007] ..... 46138.728415: nfs_set_cache_invalid: error=0 (OK) fileid=00:aa:10056165185 fhandle=0x8bfc64c9 type=8 (REG) version=1753485366506226295 size=501300 cache_validity=0x1a (INVALID_DATA|INVALID_ACCESS|INVALID_ACL) nfs_flags=0x4 (ACL_LRU_SET)
 oil-localfs-252-2605046 [007] ..... 46138.728416: nfs_refresh_inode_exit: error=0 (OK) fileid=00:aa:10056165185 fhandle=0x8bfc64c9 type=8 (REG) version=1753485366506226295 size=501300 cache_validity=0x1a (INVALID_DATA|INVALID_ACCESS|INVALID_ACL) nfs_flags=0x4 (ACL_LRU_SET)
 oil-localfs-252-2605046 [007] ..... 46138.728486: nfs_getattr_enter: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366506226295 
 oil-localfs-252-2605046 [007] ..... 46138.728492: nfs_getattr_exit: error=0 (OK) fileid=00:aa:10056165185 fhandle=0x8bfc64c9 type=8 (REG) version=1753485366506226295 size=501300 cache_validity=0x1a (INVALID_DATA|INVALID_ACCESS|INVALID_ACL) nfs_flags=0x4 (ACL_LRU_SET)
 oil-localfs-252-2605046 [007] ..... 46138.728500: nfs_getattr_enter: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366506226295 
 oil-localfs-252-2605046 [007] ..... 46138.728501: nfs_getattr_exit: error=0 (OK) fileid=00:aa:10056165185 fhandle=0x8bfc64c9 type=8 (REG) version=1753485366506226295 size=501300 cache_validity=0x1a (INVALID_DATA|INVALID_ACCESS|INVALID_ACL) nfs_flags=0x4 (ACL_LRU_SET)
 oil-localfs-252-2605046 [007] ..... 46138.735689: nfs_invalidate_mapping_enter: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366506226295 
 oil-localfs-252-2605046 [007] ..... 46138.735690: nfs_writeback_inode_enter: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366506226295 
 oil-localfs-252-2605046 [007] ..... 46138.735694: nfs_writeback_inode_exit: error=0 (OK) fileid=00:aa:10056165185 fhandle=0x8bfc64c9 type=8 (REG) version=1753485366506226295 size=501300 cache_validity=0x18 (INVALID_ACCESS|INVALID_ACL) nfs_flags=0xc (ACL_LRU_SET|INVALIDATING)
 oil-localfs-252-2605046 [007] ..... 46138.735752: nfs_invalidate_mapping_exit: error=0 (OK) fileid=00:aa:10056165185 fhandle=0x8bfc64c9 type=8 (REG) version=1753485366506226295 size=501300 cache_validity=0x18 (INVALID_ACCESS|INVALID_ACL) nfs_flags=0xc (ACL_LRU_SET|INVALIDATING)
 kworker/u1037:3-2593501 [101] ..... 46138.737212: nfs_set_cache_invalid: error=0 (OK) fileid=00:aa:10056165185 fhandle=0x8bfc64c9 type=8 (REG) version=1753485366506226295 size=501300 cache_validity=0x1c (INVALID_ATIME|INVALID_ACCESS|INVALID_ACL) nfs_flags=0x4 (ACL_LRU_SET)
      ThriftIO86-2603963 [075] ..... 46139.390716: nfs_writeback_inode_enter: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366506226295 
      ThriftIO86-2603963 [075] ..... 46139.390717: nfs_writeback_inode_exit: error=0 (OK) fileid=00:aa:10056165185 fhandle=0x8bfc64c9 type=8 (REG) version=1753485366506226295 size=501300 cache_validity=0x1c (INVALID_ATIME|INVALID_ACCESS|INVALID_ACL) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1025:1-1058132 [006] ..... 46139.439373: nfs_refresh_inode_enter: fileid=00:aa:10056165185 fhandle=0x8bfc64c9 version=1753485366506226295 
 kworker/u1025:1-1058132 [006] ..... 46139.439375: nfs_set_cache_invalid: error=0 (OK) fileid=00:aa:10056165185 fhandle=0x8bfc64c9 type=8 (REG) version=1753485366506226295 size=501300 cache_validity=0x18 (INVALID_ACCESS|INVALID_ACL) nfs_flags=0x4 (ACL_LRU_SET)
 kworker/u1025:1-1058132 [006] ..... 46139.439376: nfs_refresh_inode_exit: error=0 (OK) fileid=00:aa:10056165185 fhandle=0x8bfc64c9 type=8 (REG) version=1753485366506226295 size=501300 cache_validity=0x18 (INVALID_ACCESS|INVALID_ACL) nfs_flags=0x4 (ACL_LRU_SET)
timed out waiting for input: auto-logout
Connection to ocloud0094.03.oas1.facebook.com closed.

Re: parts of pages on NFS being replaced by swaths of NULs

[Trond Myklebust]

On Wed, 2025-07-30 at 10:52 -0400, Jeff Layton wrote:
> We've been seeing a rather nasty bit of data corruption with NFS in
> our
> environment. The clients in this env run a patched v6.9 kernel
> (mostly
> due to GPU driver requirements). Most of the patches are NFS
> containerization fixes.
> 
> The workload is python scripts writing JSONL files sequentially using
> bog-standard buffered write() calls. We're fairly certain that
> userland
> is not seeking so there should be no gaps in the data written.
> 
> The problem is that we see ranges of written files being replaced by
> NULs. The length of the file seemingly doesn't change from what it
> should be, but a chunk of it will be zeroed-out. Looking at the
> offsets
> of the zeroed out ranges, the front part of one page is fine, but the
> data from some random offset in the page to the end of the page is
> zeroes.
> 
> We have a reproducer but we have to run it in a heavily parallel
> configuration to make it happen, so it's evidently a tight race of
> some
> sort.
> 
> We've turned up some tracepoints and reproduced this twice. What we
> see
> in both cases is that the client just doesn't write some section of
> the
> file.
> 
> In the first trace, there was is a gap of 2201 bytes between these
> two
> writes on the wire:
> 
>  kworker/u1038:1-2597138 [106] ..... 46138.516795:
> nfs_initiate_write: fileid=00:aa:10056165185 fhandle=0x6bd94d55
> offset=53248 count=1895 stable=UNSTABLE
>  oil-localfs-252-2605046 [163] ..... 46138.551459:
> nfs_initiate_write: fileid=00:aa:10056165185 fhandle=0x6bd94d55
> offset=57344 count=443956 stable=FILE_SYNC
> 
> The zeroed-out range is from 55143-57344. At the same time that the
> file is growing from 53248 to 55143 (due to sequential write()
> activity), the client is kicking off writeback for the range up to
> 55143. It's issuing 2 writes, one for 0-53248 and one for 53248-55143
> (note that I've filtered out all but one of the DS filehandles for
> brevity):
> 
>  oil-localfs-252-2605046 [162] ..... 46138.516414: nfs_size_grow:
> fileid=00:aa:10056165185 fhandle=0x8bfc64c9
> version=1753485366409158129 cursize=49152 newsize=50130
>  oil-localfs-252-2605046 [162] ..... 46138.516593: nfs_size_grow:
> fileid=00:aa:10056165185 fhandle=0x8bfc64c9
> version=1753485366409158129 cursize=50130 newsize=53248
>  kworker/u1038:1-2597138 [106] ..... 46138.516740:
> nfs_initiate_write: fileid=00:aa:10056165185 fhandle=0x6bd94d55
> offset=0 count=53248 stable=UNSTABLE
>  oil-localfs-252-2605046 [162] ..... 46138.516753: nfs_size_grow:
> fileid=00:aa:10056165185 fhandle=0x8bfc64c9
> version=1753485366409158129 cursize=53248 newsize=55143
>  kworker/u1038:1-2597138 [106] ..... 46138.516795:
> nfs_initiate_write: fileid=00:aa:10056165185 fhandle=0x6bd94d55
> offset=53248 count=1895 stable=UNSTABLE
>  kworker/u1037:2-2871862 [097] ..... 46138.517659: nfs4_pnfs_write:
> error=0 (OK) fileid=00:aa:10056165185 fhandle=0x6bd94d55 offset=0
> count=53248 res=53248 stateid=1:0x79a9c471 layoutstateid=1:0xcbd8aaad
>  kworker/u1037:2-2871862 [097] ..... 46138.517662:
> nfs_writeback_done: error=53248 fileid=00:aa:10056165185
> fhandle=0x6bd94d55 offset=0 count=53248 res=53248 stable=UNSTABLE
> verifier=5199cdae2816c899
>  kworker/u1037:5-2593935 [226] ..... 46138.517669: nfs4_pnfs_write:
> error=0 (OK) fileid=00:aa:10056165185 fhandle=0x6bd94d55 offset=53248
> count=1895 res=1895 stateid=1:0x79a9c471 layoutstateid=1:0xcbd8aaad
>  kworker/u1037:5-2593935 [226] ..... 46138.517672:
> nfs_writeback_done: error=1895 fileid=00:aa:10056165185
> fhandle=0x6bd94d55 offset=53248 count=1895 res=1895 stable=UNSTABLE
> verifier=5199cdae2816c899
>  oil-localfs-252-2605046 [162] ..... 46138.518360: nfs_size_grow:
> fileid=00:aa:10056165185 fhandle=0x8bfc64c9
> version=1753485366409158129 cursize=55143 newsize=57344
>  oil-localfs-252-2605046 [162] ..... 46138.518556: nfs_size_grow:
> fileid=00:aa:10056165185 fhandle=0x8bfc64c9
> version=1753485366409158129 cursize=57344 newsize=60156
> 
> ...and just after writeback completes, we see the file size grow from
> 55143 to the end of the page (57344).
> 
> The second trace has similar symptoms. There is a lot more (smaller)
> write activity (due to memory pressure?). There is a gap of 3791
> bytes
> between these on-the-wire writes, however:
> 
>  kworker/u1036:0-2339252 [217] ..... 479572.054622:
> nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232
> offset=221184 count=4401 stable=UNSTABLE
>  kworker/u1030:1-2297876 [042] ..... 479572.074194:
> nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232
> offset=229376 count=261898 stable=UNSTABLE
> 
> Same situation -- the at page at offset 53248 has 305 bytes on it,
> and
> the remaining is zeroed. This trace shows similar racing write() and
> writeback activity as in Friday's trace. At around the same time as
> the
> client was growing the file over the affected range, writeback was
> kicking off for everything up to the affected range (this has some
> other wb related calls filtered for brevity):
> 
>   oil-localfs-86-727850  [215] ..... 479572.053987: nfs_size_grow:
> fileid=00:96:10067193438 fhandle=0x14c40498
> version=1753823598774309300 cursize=217088
> newsize=220572                                                       
>               
>  kworker/u1036:8-2339326 [088] ..... 479572.054008:
> nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232
> offset=217088 count=3484
> stable=UNSTABLE                                                      
>                            
>   oil-localfs-86-727850  [215] ..... 479572.054405: nfs_size_grow:
> fileid=00:96:10067193438 fhandle=0x14c40498
> version=1753823598774309300 cursize=220572
> newsize=221184                                                       
>               
>  kworker/u1036:1-2297875 [217] ..... 479572.054418:
> nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232
> offset=220572 count=612
> stable=UNSTABLE                                                      
>                             
>   oil-localfs-86-727850  [215] ..... 479572.054581: nfs_size_grow:
> fileid=00:96:10067193438 fhandle=0x14c40498
> version=1753823598774309300 cursize=221184
> newsize=225280                                                       
>               
>   oil-localfs-86-727850  [215] ..... 479572.054584: nfs_size_grow:
> fileid=00:96:10067193438 fhandle=0x14c40498
> version=1753823598774309300 cursize=225280
> newsize=225585                                                       
>               
>  kworker/u1036:0-2339252 [217] ..... 479572.054622:
> nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232
> offset=221184 count=4401
> stable=UNSTABLE                                                      
>                            
>   oil-localfs-86-727850  [215] ..... 479572.054997: nfs_size_grow:
> fileid=00:96:10067193438 fhandle=0x14c40498
> version=1753823598774309300 cursize=225585
> newsize=229376                                                       
>               
>   oil-localfs-86-727850  [215] ..... 479572.055190: nfs_size_grow:
> fileid=00:96:10067193438 fhandle=0x14c40498
> version=1753823598774309300 cursize=229376
> newsize=230598                                                       
>               
> 
> Could this be a race between extending an existing dirty page, and
> writeback kicking off for the pre-extension range on the page? Maybe
> the client is clearing the dirty bit, thinking that the write covers
> the dirty range, but it has an outdated idea about what that range is
> or doesn't properly check?
> 
> Traces for both events, filtered on the relevant fileid are attached.
> I've rolled patches for some new tracepoints that I'm going to
> attempt
> to turn up next, but I thought that this was a good point to solicit
> ideas.
> 
> Happy to entertain other thoughts or patches!

So... The fact that we are seeing a nfs_size_grow() for the hole at
offset 55143 means that either an existing request was updated, or a
new one was created in order to cover that hole, and it must have been
marked as dirty.

I'm not seeing anything in the NFS code that can lose that request
without triggering either the nfs_write_error tracepoint, the
nfs_commit_error tracepoint, the nfs_invalidate_folio tracepoint or
else completing the write.

The only other way I can see this data being lost is if something is
corrupting folio->private, or if the page cache is somehow managing to
throw away a dirty folio.
Of the two, there was for a while a netfs bug which would corrupt
folio->private, but I assume you're not using cachefs?

-- 
Trond Myklebust
Linux NFS client maintainer, Hammerspace
trondmy@kernel.org, trond.myklebust@hammerspace.com

Re: parts of pages on NFS being replaced by swaths of NULs

[Jeff Layton]

On Thu, 2025-07-31 at 17:56 -0400, Trond Myklebust wrote:
> On Wed, 2025-07-30 at 10:52 -0400, Jeff Layton wrote:
> > We've been seeing a rather nasty bit of data corruption with NFS in
> > our
> > environment. The clients in this env run a patched v6.9 kernel
> > (mostly
> > due to GPU driver requirements). Most of the patches are NFS
> > containerization fixes.
> > 
> > The workload is python scripts writing JSONL files sequentially using
> > bog-standard buffered write() calls. We're fairly certain that
> > userland
> > is not seeking so there should be no gaps in the data written.
> > 
> > The problem is that we see ranges of written files being replaced by
> > NULs. The length of the file seemingly doesn't change from what it
> > should be, but a chunk of it will be zeroed-out. Looking at the
> > offsets
> > of the zeroed out ranges, the front part of one page is fine, but the
> > data from some random offset in the page to the end of the page is
> > zeroes.
> > 
> > We have a reproducer but we have to run it in a heavily parallel
> > configuration to make it happen, so it's evidently a tight race of
> > some
> > sort.
> > 
> > We've turned up some tracepoints and reproduced this twice. What we
> > see
> > in both cases is that the client just doesn't write some section of
> > the
> > file.
> > 
> > In the first trace, there was is a gap of 2201 bytes between these
> > two
> > writes on the wire:
> > 
> >  kworker/u1038:1-2597138 [106] ..... 46138.516795:
> > nfs_initiate_write: fileid=00:aa:10056165185 fhandle=0x6bd94d55
> > offset=53248 count=1895 stable=UNSTABLE
> >  oil-localfs-252-2605046 [163] ..... 46138.551459:
> > nfs_initiate_write: fileid=00:aa:10056165185 fhandle=0x6bd94d55
> > offset=57344 count=443956 stable=FILE_SYNC
> > 
> > The zeroed-out range is from 55143-57344. At the same time that the
> > file is growing from 53248 to 55143 (due to sequential write()
> > activity), the client is kicking off writeback for the range up to
> > 55143. It's issuing 2 writes, one for 0-53248 and one for 53248-55143
> > (note that I've filtered out all but one of the DS filehandles for
> > brevity):
> > 
> >  oil-localfs-252-2605046 [162] ..... 46138.516414: nfs_size_grow:
> > fileid=00:aa:10056165185 fhandle=0x8bfc64c9
> > version=1753485366409158129 cursize=49152 newsize=50130
> >  oil-localfs-252-2605046 [162] ..... 46138.516593: nfs_size_grow:
> > fileid=00:aa:10056165185 fhandle=0x8bfc64c9
> > version=1753485366409158129 cursize=50130 newsize=53248
> >  kworker/u1038:1-2597138 [106] ..... 46138.516740:
> > nfs_initiate_write: fileid=00:aa:10056165185 fhandle=0x6bd94d55
> > offset=0 count=53248 stable=UNSTABLE
> >  oil-localfs-252-2605046 [162] ..... 46138.516753: nfs_size_grow:
> > fileid=00:aa:10056165185 fhandle=0x8bfc64c9
> > version=1753485366409158129 cursize=53248 newsize=55143
> >  kworker/u1038:1-2597138 [106] ..... 46138.516795:
> > nfs_initiate_write: fileid=00:aa:10056165185 fhandle=0x6bd94d55
> > offset=53248 count=1895 stable=UNSTABLE
> >  kworker/u1037:2-2871862 [097] ..... 46138.517659: nfs4_pnfs_write:
> > error=0 (OK) fileid=00:aa:10056165185 fhandle=0x6bd94d55 offset=0
> > count=53248 res=53248 stateid=1:0x79a9c471 layoutstateid=1:0xcbd8aaad
> >  kworker/u1037:2-2871862 [097] ..... 46138.517662:
> > nfs_writeback_done: error=53248 fileid=00:aa:10056165185
> > fhandle=0x6bd94d55 offset=0 count=53248 res=53248 stable=UNSTABLE
> > verifier=5199cdae2816c899
> >  kworker/u1037:5-2593935 [226] ..... 46138.517669: nfs4_pnfs_write:
> > error=0 (OK) fileid=00:aa:10056165185 fhandle=0x6bd94d55 offset=53248
> > count=1895 res=1895 stateid=1:0x79a9c471 layoutstateid=1:0xcbd8aaad
> >  kworker/u1037:5-2593935 [226] ..... 46138.517672:
> > nfs_writeback_done: error=1895 fileid=00:aa:10056165185
> > fhandle=0x6bd94d55 offset=53248 count=1895 res=1895 stable=UNSTABLE
> > verifier=5199cdae2816c899
> >  oil-localfs-252-2605046 [162] ..... 46138.518360: nfs_size_grow:
> > fileid=00:aa:10056165185 fhandle=0x8bfc64c9
> > version=1753485366409158129 cursize=55143 newsize=57344
> >  oil-localfs-252-2605046 [162] ..... 46138.518556: nfs_size_grow:
> > fileid=00:aa:10056165185 fhandle=0x8bfc64c9
> > version=1753485366409158129 cursize=57344 newsize=60156
> > 
> > ...and just after writeback completes, we see the file size grow from
> > 55143 to the end of the page (57344).
> > 
> > The second trace has similar symptoms. There is a lot more (smaller)
> > write activity (due to memory pressure?). There is a gap of 3791
> > bytes
> > between these on-the-wire writes, however:
> > 
> >  kworker/u1036:0-2339252 [217] ..... 479572.054622:
> > nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232
> > offset=221184 count=4401 stable=UNSTABLE
> >  kworker/u1030:1-2297876 [042] ..... 479572.074194:
> > nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232
> > offset=229376 count=261898 stable=UNSTABLE
> > 
> > Same situation -- the at page at offset 53248 has 305 bytes on it,
> > and
> > the remaining is zeroed. This trace shows similar racing write() and
> > writeback activity as in Friday's trace. At around the same time as
> > the
> > client was growing the file over the affected range, writeback was
> > kicking off for everything up to the affected range (this has some
> > other wb related calls filtered for brevity):
> > 
> >   oil-localfs-86-727850  [215] ..... 479572.053987: nfs_size_grow:
> > fileid=00:96:10067193438 fhandle=0x14c40498
> > version=1753823598774309300 cursize=217088
> > newsize=220572                                                       
> >               
> >  kworker/u1036:8-2339326 [088] ..... 479572.054008:
> > nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232
> > offset=217088 count=3484
> > stable=UNSTABLE                                                      
> >                            
> >   oil-localfs-86-727850  [215] ..... 479572.054405: nfs_size_grow:
> > fileid=00:96:10067193438 fhandle=0x14c40498
> > version=1753823598774309300 cursize=220572
> > newsize=221184                                                       
> >               
> >  kworker/u1036:1-2297875 [217] ..... 479572.054418:
> > nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232
> > offset=220572 count=612
> > stable=UNSTABLE                                                      
> >                             
> >   oil-localfs-86-727850  [215] ..... 479572.054581: nfs_size_grow:
> > fileid=00:96:10067193438 fhandle=0x14c40498
> > version=1753823598774309300 cursize=221184
> > newsize=225280                                                       
> >               
> >   oil-localfs-86-727850  [215] ..... 479572.054584: nfs_size_grow:
> > fileid=00:96:10067193438 fhandle=0x14c40498
> > version=1753823598774309300 cursize=225280
> > newsize=225585                                                       
> >               
> >  kworker/u1036:0-2339252 [217] ..... 479572.054622:
> > nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232
> > offset=221184 count=4401
> > stable=UNSTABLE                                                      
> >                            
> >   oil-localfs-86-727850  [215] ..... 479572.054997: nfs_size_grow:
> > fileid=00:96:10067193438 fhandle=0x14c40498
> > version=1753823598774309300 cursize=225585
> > newsize=229376                                                       
> >               
> >   oil-localfs-86-727850  [215] ..... 479572.055190: nfs_size_grow:
> > fileid=00:96:10067193438 fhandle=0x14c40498
> > version=1753823598774309300 cursize=229376
> > newsize=230598                                                       
> >               
> > 
> > Could this be a race between extending an existing dirty page, and
> > writeback kicking off for the pre-extension range on the page? Maybe
> > the client is clearing the dirty bit, thinking that the write covers
> > the dirty range, but it has an outdated idea about what that range is
> > or doesn't properly check?
> > 
> > Traces for both events, filtered on the relevant fileid are attached.
> > I've rolled patches for some new tracepoints that I'm going to
> > attempt
> > to turn up next, but I thought that this was a good point to solicit
> > ideas.
> > 
> > Happy to entertain other thoughts or patches!
> 
> So... The fact that we are seeing a nfs_size_grow() for the hole at
> offset 55143 means that either an existing request was updated, or a
> new one was created in order to cover that hole, and it must have been
> marked as dirty.
> 
> I'm not seeing anything in the NFS code that can lose that request
> without triggering either the nfs_write_error tracepoint, the
> nfs_commit_error tracepoint, the nfs_invalidate_folio tracepoint or
> else completing the write.
> 

I've been looking for a couple of days too and haven't spotted any
obvious bugs yet.

> The only other way I can see this data being lost is if something is
> corrupting folio->private, or if the page cache is somehow managing to
> throw away a dirty folio.
> Of the two, there was for a while a netfs bug which would corrupt
> folio->private, but I assume you're not using cachefs?

No, no fscache. Mount options are:

    proto=tcp6,port=20492,vers=4.2,nosharecache,timeo=30,nconnect=16,noresvport

-- 
Jeff Layton <jlayton@kernel.org>

Re: parts of pages on NFS being replaced by swaths of NULs

[Jeff Layton]

On Thu, 2025-07-31 at 17:56 -0400, Trond Myklebust wrote:
> On Wed, 2025-07-30 at 10:52 -0400, Jeff Layton wrote:
> > We've been seeing a rather nasty bit of data corruption with NFS in
> > our
> > environment. The clients in this env run a patched v6.9 kernel
> > (mostly
> > due to GPU driver requirements). Most of the patches are NFS
> > containerization fixes.
> > 
> > The workload is python scripts writing JSONL files sequentially using
> > bog-standard buffered write() calls. We're fairly certain that
> > userland
> > is not seeking so there should be no gaps in the data written.
> > 
> > The problem is that we see ranges of written files being replaced by
> > NULs. The length of the file seemingly doesn't change from what it
> > should be, but a chunk of it will be zeroed-out. Looking at the
> > offsets
> > of the zeroed out ranges, the front part of one page is fine, but the
> > data from some random offset in the page to the end of the page is
> > zeroes.
> > 
> > We have a reproducer but we have to run it in a heavily parallel
> > configuration to make it happen, so it's evidently a tight race of
> > some
> > sort.
> > 
> > We've turned up some tracepoints and reproduced this twice. What we
> > see
> > in both cases is that the client just doesn't write some section of
> > the
> > file.
> > 
> > In the first trace, there was is a gap of 2201 bytes between these
> > two
> > writes on the wire:
> > 
> >  kworker/u1038:1-2597138 [106] ..... 46138.516795:
> > nfs_initiate_write: fileid=00:aa:10056165185 fhandle=0x6bd94d55
> > offset=53248 count=1895 stable=UNSTABLE
> >  foo-localfs-252-2605046 [163] ..... 46138.551459:
> > nfs_initiate_write: fileid=00:aa:10056165185 fhandle=0x6bd94d55
> > offset=57344 count=443956 stable=FILE_SYNC
> > 
> > The zeroed-out range is from 55143-57344. At the same time that the
> > file is growing from 53248 to 55143 (due to sequential write()
> > activity), the client is kicking off writeback for the range up to
> > 55143. It's issuing 2 writes, one for 0-53248 and one for 53248-55143
> > (note that I've filtered out all but one of the DS filehandles for
> > brevity):
> > 
> >  foo-localfs-252-2605046 [162] ..... 46138.516414: nfs_size_grow:
> > fileid=00:aa:10056165185 fhandle=0x8bfc64c9
> > version=1753485366409158129 cursize=49152 newsize=50130
> >  foo-localfs-252-2605046 [162] ..... 46138.516593: nfs_size_grow:
> > fileid=00:aa:10056165185 fhandle=0x8bfc64c9
> > version=1753485366409158129 cursize=50130 newsize=53248
> >  kworker/u1038:1-2597138 [106] ..... 46138.516740:
> > nfs_initiate_write: fileid=00:aa:10056165185 fhandle=0x6bd94d55
> > offset=0 count=53248 stable=UNSTABLE
> >  foo-localfs-252-2605046 [162] ..... 46138.516753: nfs_size_grow:
> > fileid=00:aa:10056165185 fhandle=0x8bfc64c9
> > version=1753485366409158129 cursize=53248 newsize=55143
> >  kworker/u1038:1-2597138 [106] ..... 46138.516795:
> > nfs_initiate_write: fileid=00:aa:10056165185 fhandle=0x6bd94d55
> > offset=53248 count=1895 stable=UNSTABLE
> >  kworker/u1037:2-2871862 [097] ..... 46138.517659: nfs4_pnfs_write:
> > error=0 (OK) fileid=00:aa:10056165185 fhandle=0x6bd94d55 offset=0
> > count=53248 res=53248 stateid=1:0x79a9c471 layoutstateid=1:0xcbd8aaad
> >  kworker/u1037:2-2871862 [097] ..... 46138.517662:
> > nfs_writeback_done: error=53248 fileid=00:aa:10056165185
> > fhandle=0x6bd94d55 offset=0 count=53248 res=53248 stable=UNSTABLE
> > verifier=5199cdae2816c899
> >  kworker/u1037:5-2593935 [226] ..... 46138.517669: nfs4_pnfs_write:
> > error=0 (OK) fileid=00:aa:10056165185 fhandle=0x6bd94d55 offset=53248
> > count=1895 res=1895 stateid=1:0x79a9c471 layoutstateid=1:0xcbd8aaad
> >  kworker/u1037:5-2593935 [226] ..... 46138.517672:
> > nfs_writeback_done: error=1895 fileid=00:aa:10056165185
> > fhandle=0x6bd94d55 offset=53248 count=1895 res=1895 stable=UNSTABLE
> > verifier=5199cdae2816c899
> >  foo-localfs-252-2605046 [162] ..... 46138.518360: nfs_size_grow:
> > fileid=00:aa:10056165185 fhandle=0x8bfc64c9
> > version=1753485366409158129 cursize=55143 newsize=57344
> >  foo-localfs-252-2605046 [162] ..... 46138.518556: nfs_size_grow:
> > fileid=00:aa:10056165185 fhandle=0x8bfc64c9
> > version=1753485366409158129 cursize=57344 newsize=60156
> > 
> > ...and just after writeback completes, we see the file size grow from
> > 55143 to the end of the page (57344).
> > 
> > The second trace has similar symptoms. There is a lot more (smaller)
> > write activity (due to memory pressure?). There is a gap of 3791
> > bytes
> > between these on-the-wire writes, however:
> > 
> >  kworker/u1036:0-2339252 [217] ..... 479572.054622:
> > nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232
> > offset=221184 count=4401 stable=UNSTABLE
> >  kworker/u1030:1-2297876 [042] ..... 479572.074194:
> > nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232
> > offset=229376 count=261898 stable=UNSTABLE
> > 
> > Same situation -- the at page at offset 53248 has 305 bytes on it,
> > and
> > the remaining is zeroed. This trace shows similar racing write() and
> > writeback activity as in Friday's trace. At around the same time as
> > the
> > client was growing the file over the affected range, writeback was
> > kicking off for everything up to the affected range (this has some
> > other wb related calls filtered for brevity):
> > 
> >   foo-localfs-86-727850  [215] ..... 479572.053987: nfs_size_grow:
> > fileid=00:96:10067193438 fhandle=0x14c40498
> > version=1753823598774309300 cursize=217088
> > newsize=220572                                                       
> >               
> >  kworker/u1036:8-2339326 [088] ..... 479572.054008:
> > nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232
> > offset=217088 count=3484
> > stable=UNSTABLE                                                      
> >                            
> >   foo-localfs-86-727850  [215] ..... 479572.054405: nfs_size_grow:
> > fileid=00:96:10067193438 fhandle=0x14c40498
> > version=1753823598774309300 cursize=220572
> > newsize=221184                                                       
> >               
> >  kworker/u1036:1-2297875 [217] ..... 479572.054418:
> > nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232
> > offset=220572 count=612
> > stable=UNSTABLE                                                      
> >                             
> >   foo-localfs-86-727850  [215] ..... 479572.054581: nfs_size_grow:
> > fileid=00:96:10067193438 fhandle=0x14c40498
> > version=1753823598774309300 cursize=221184
> > newsize=225280                                                       
> >               
> >   foo-localfs-86-727850  [215] ..... 479572.054584: nfs_size_grow:
> > fileid=00:96:10067193438 fhandle=0x14c40498
> > version=1753823598774309300 cursize=225280
> > newsize=225585                                                       
> >               
> >  kworker/u1036:0-2339252 [217] ..... 479572.054622:
> > nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232
> > offset=221184 count=4401
> > stable=UNSTABLE                                                      
> >                            
> >   foo-localfs-86-727850  [215] ..... 479572.054997: nfs_size_grow:
> > fileid=00:96:10067193438 fhandle=0x14c40498
> > version=1753823598774309300 cursize=225585
> > newsize=229376                                                       
> >               
> >   foo-localfs-86-727850  [215] ..... 479572.055190: nfs_size_grow:
> > fileid=00:96:10067193438 fhandle=0x14c40498
> > version=1753823598774309300 cursize=229376
> > newsize=230598                                                       
> >               
> > 
> > Could this be a race between extending an existing dirty page, and
> > writeback kicking off for the pre-extension range on the page? Maybe
> > the client is clearing the dirty bit, thinking that the write covers
> > the dirty range, but it has an outdated idea about what that range is
> > or doesn't properly check?
> > 
> > Traces for both events, filtered on the relevant fileid are attached.
> > I've rolled patches for some new tracepoints that I'm going to
> > attempt
> > to turn up next, but I thought that this was a good point to solicit
> > ideas.
> > 
> > Happy to entertain other thoughts or patches!
> 
> So... The fact that we are seeing a nfs_size_grow() for the hole at
> offset 55143 means that either an existing request was updated, or a
> new one was created in order to cover that hole, and it must have been
> marked as dirty.
> 
> I'm not seeing anything in the NFS code that can lose that request
> without triggering either the nfs_write_error tracepoint, the
> nfs_commit_error tracepoint, the nfs_invalidate_folio tracepoint or
> else completing the write.
> 
> The only other way I can see this data being lost is if something is
> corrupting folio->private, or if the page cache is somehow managing to
> throw away a dirty folio.
> Of the two, there was for a while a netfs bug which would corrupt
> folio->private, but I assume you're not using cachefs?

We reproduced this again, this time with some extra tracepoints that I
added. I'll post patches for those soon. I may need to add more.

Here's the writeup of it I did this morning. Some names changed to
protect the paranoid. Let me know if anyone has ideas:

In this case, the corruption happened fairly early. There is a 2262
byte hole between 10026 and 12288:

 kworker/u1025:9-3435794 [000] ..... 42066.127742: nfs_initiate_write: fileid=00:82:10087279963 fhandle=0x483d45e0 offset=0 count=10026 stable=UNSTABLE
 foo-localfs-161-1326370 [135] ..... 42066.169637: nfs_initiate_write: fileid=00:82:10087279963 fhandle=0x483d45e0 offset=12288 count=489012 stable=FILE_SYNC

The reproducer is a python script doing 5013 byte write() calls
exclusively. There is another layer between that and NFS though. It's
implemented as a FUSE fs that passes reads and writes through to NFS.
This layer alters the I/O pattern in an interesting way:

The first write from userland is 5013 bytes. (Side q: Does foofs use
io_uring to do these writes?):

 foo-localfs-161-1326370 [135] ..... 42066.127165: nfs_file_write: fileid=00:82:10087279963 fhandle=0xa6491ad0 version=1754454982197296994 offset=0 count=5013 ki_flags=

...but then it writes everything up to the end of the second page in
the next write() call:

 foo-localfs-161-1326370 [135] ..... 42066.127486: nfs_file_write: fileid=00:82:10087279963 fhandle=0xa6491ad0 version=1754454982197296994 offset=5013 count=3179 ki_flags=

 ...then it writes the rest of that 5013 byte write:

 foo-localfs-161-1326370 [135] ..... 42066.127717: nfs_file_write: fileid=00:82:10087279963 fhandle=0xa6491ad0 version=1754454982197296994 offset=8192 count=1834 ki_flags=

 ...so now we have 2 complete lines. The next write from userland is to
the end of the page:

 foo-localfs-161-1326370 [135] ..... 42066.127954: nfs_file_write: fileid=00:82:10087279963 fhandle=0xa6491ad0 version=1754454982197296994 offset=10026 count=2262 ki_flags=
 
...and the pattern continues...
 
 foo-localfs-161-1326370 [135] ..... 42066.129411: nfs_file_write: fileid=00:82:10087279963 fhandle=0xa6491ad0 version=1754454982197296994 offset=12288 count=2751 ki_flags=
 foo-localfs-161-1326370 [135] ..... 42066.129749: nfs_file_write: fileid=00:82:10087279963 fhandle=0xa6491ad0 version=1754454982197296994 offset=15039 count=1345 ki_flags=
 foo-localfs-161-1326370 [135] ..... 42066.130020: nfs_file_write: fileid=00:82:10087279963 fhandle=0xa6491ad0 version=1754454982197296994 offset=16384 count=3668 ki_flags=

This weird I/O pattern may help explain why it reproduces easier with
this layer in the mix. This gives the NFS client ample opportunity to
have to extend existing pages. I plan to change my attempted reproducer
to mimic this pattern.

The new tracepoints give us a slightly clearer picture of the race (I
filtered out all but one of the DS filehandles for simplicity). 

Here's the end of the write from 8192-10026. At this point the file
size is 10026 (everything is normal at this point, AFAICT):

 foo-localfs-161-1326370 [135] ..... 42066.127720: nfs_size_grow: fileid=00:82:10087279963 fhandle=0xa6491ad0 version=1754454982197296994 cursize=8192 newsize=10026

Writeback for everything in the cache so far is kicked off:

 kworker/u1025:9-3435794 [000] ..... 42066.127742: nfs_initiate_write: fileid=00:82:10087279963 fhandle=0x483d45e0 offset=0 count=10026 stable=UNSTABLE

A write comes in from foofs for 2262 bytes, and it calls into
write_begin. I suspect at this point nfs_write_begin is blocked on the
page lock:

 foo-localfs-161-1326370 [135] ..... 42066.127954: nfs_file_write: fileid=00:82:10087279963 fhandle=0xa6491ad0 version=1754454982197296994 offset=10026 count=2262 ki_flags=
 foo-localfs-161-1326370 [135] ..... 42066.127954: nfs_write_begin: fileid=00:82:10087279963 fhandle=0xa6491ad0 version=1754454982197296994 offset=10026 count=2262

Writeback to the DS's proceeds:

 kworker/u1035:0-3302696 [209] ..... 42066.128388: nfs4_pnfs_write: error=0 (OK) fileid=00:82:10087279963 fhandle=0x483d45e0 offset=0 count=10026 res=10026 stateid=1:0x428af712 layoutstateid=1:0x47c699b3
 kworker/u1035:0-3302696 [209] ..... 42066.128396: nfs_writeback_done: error=10026 fileid=00:82:10087279963 fhandle=0x483d45e0 offset=0 count=10026 res=10026 stable=UNSTABLE verifier=a376459679d60091

Eventually, foofs gets the page lock, and write_end is called. It
updates the folio via nfs_try_to_update_request():

 foo-localfs-161-1326370 [135] ..... 42066.128429: nfs_write_end: fileid=00:82:10087279963 fhandle=0xa6491ad0 version=1754454982197296994 offset=10026 count=2262
 foo-localfs-161-1326370 [135] ..... 42066.128429: nfs_update_folio: fileid=00:82:10087279963 fhandle=0xa6491ad0 version=1754454982197296994 offset=1834 count=2262
 foo-localfs-161-1326370 [135] ..... 42066.128429: nfs_try_to_update_request: fileid=00:82:10087279963 fhandle=0xa6491ad0 version=1754454982197296994 offset=1834 count=2262

...COMMITs go out for the data just written (side q: there was only a
single WRITE call on the wire before this. Why didn't it just do a SYNC
write instead?):

 kworker/u1028:3-3432139 [031] ..... 42066.128431: nfs_initiate_commit: fileid=00:82:10087279963 fhandle=0x483d45e0 offset=0 count=0
 kworker/u1035:0-3302696 [209] ..... 42066.129158: nfs_commit_done: error=0 fileid=00:82:10087279963 fhandle=0x483d45e0 offset=0 stable=FILE_SYNC verifier=a376459679d60091

 ...the file size in the inode is grown:

 foo-localfs-161-1326370 [135] ..... 42066.129179: nfs_size_grow: fileid=00:82:10087279963 fhandle=0xa6491ad0 version=1754454982197296994 cursize=10026 newsize=12288

 ...and then the next userland write happens:

 foo-localfs-161-1326370 [135] ..... 42066.129411: nfs_file_write: fileid=00:82:10087279963 fhandle=0xa6491ad0 version=1754454982197296994 offset=12288 count=2751 ki_flags=

...and eventually the next on-the-wire write occurs due to an fsync, which skips the partial page:

 foo-localfs-161-1326370 [135] ..... 42066.169262: nfs_size_grow: fileid=00:82:10087279963 fhandle=0xa6491ad0 version=1754454982197296994 cursize=499712 newsize=501300
 foo-localfs-161-1326370 [135] ..... 42066.169517: nfs_fsync_enter: fileid=00:82:10087279963 fhandle=0xa6491ad0 version=1754454982197296994 cache_validity=0x4700 (INVALID_CHANGE|INVALID_CTIME|INVALID_MTIME|INVALID_BLOCKS)
 foo-localfs-161-1326370 [135] ..... 42066.169637: nfs_initiate_write: fileid=00:82:10087279963 fhandle=0x483d45e0 offset=12288 count=489012 stable=FILE_SYNC

It's still not 100% clear why part of the page didn't get written back.
I have a couple of theories at this point:

1/ the dirty bit is somehow either not being set properly in the first
place for the 2262 byte write, or is getting cleared inappropriately
during the WRITE that includes the first part of the page.

2/ The file size doesn't grow until quite late in the process. Perhaps
we could be clamping the writeback range to the old file size and
tossing out the rest of the page?

I think #1 is most probable.

My thinking at this point is that since we know the userland writes are
contiguous in this case, I can make nfs_writepages look for holes in
the data to be written back, and dump info about the page / nfs_page
with the hole.

Happy to entertain other ideas too.

Cheers,
-- 
Jeff Layton <jlayton@kernel.org>

Re: parts of pages on NFS being replaced by swaths of NULs

[Jeff Layton]

On Thu, 2025-07-31 at 17:56 -0400, Trond Myklebust wrote:
> On Wed, 2025-07-30 at 10:52 -0400, Jeff Layton wrote:
> > We've been seeing a rather nasty bit of data corruption with NFS in
> > our
> > environment. The clients in this env run a patched v6.9 kernel
> > (mostly
> > due to GPU driver requirements). Most of the patches are NFS
> > containerization fixes.
> > 
> > The workload is python scripts writing JSONL files sequentially using
> > bog-standard buffered write() calls. We're fairly certain that
> > userland
> > is not seeking so there should be no gaps in the data written.
> > 
> > The problem is that we see ranges of written files being replaced by
> > NULs. The length of the file seemingly doesn't change from what it
> > should be, but a chunk of it will be zeroed-out. Looking at the
> > offsets
> > of the zeroed out ranges, the front part of one page is fine, but the
> > data from some random offset in the page to the end of the page is
> > zeroes.
> > 
> > We have a reproducer but we have to run it in a heavily parallel
> > configuration to make it happen, so it's evidently a tight race of
> > some
> > sort.
> > 
> > We've turned up some tracepoints and reproduced this twice. What we
> > see
> > in both cases is that the client just doesn't write some section of
> > the
> > file.
> > 
> > In the first trace, there was is a gap of 2201 bytes between these
> > two
> > writes on the wire:
> > 
> >  kworker/u1038:1-2597138 [106] ..... 46138.516795:
> > nfs_initiate_write: fileid=00:aa:10056165185 fhandle=0x6bd94d55
> > offset=53248 count=1895 stable=UNSTABLE
> >  oil-localfs-252-2605046 [163] ..... 46138.551459:
> > nfs_initiate_write: fileid=00:aa:10056165185 fhandle=0x6bd94d55
> > offset=57344 count=443956 stable=FILE_SYNC
> > 
> > The zeroed-out range is from 55143-57344. At the same time that the
> > file is growing from 53248 to 55143 (due to sequential write()
> > activity), the client is kicking off writeback for the range up to
> > 55143. It's issuing 2 writes, one for 0-53248 and one for 53248-55143
> > (note that I've filtered out all but one of the DS filehandles for
> > brevity):
> > 
> >  oil-localfs-252-2605046 [162] ..... 46138.516414: nfs_size_grow:
> > fileid=00:aa:10056165185 fhandle=0x8bfc64c9
> > version=1753485366409158129 cursize=49152 newsize=50130
> >  oil-localfs-252-2605046 [162] ..... 46138.516593: nfs_size_grow:
> > fileid=00:aa:10056165185 fhandle=0x8bfc64c9
> > version=1753485366409158129 cursize=50130 newsize=53248
> >  kworker/u1038:1-2597138 [106] ..... 46138.516740:
> > nfs_initiate_write: fileid=00:aa:10056165185 fhandle=0x6bd94d55
> > offset=0 count=53248 stable=UNSTABLE
> >  oil-localfs-252-2605046 [162] ..... 46138.516753: nfs_size_grow:
> > fileid=00:aa:10056165185 fhandle=0x8bfc64c9
> > version=1753485366409158129 cursize=53248 newsize=55143
> >  kworker/u1038:1-2597138 [106] ..... 46138.516795:
> > nfs_initiate_write: fileid=00:aa:10056165185 fhandle=0x6bd94d55
> > offset=53248 count=1895 stable=UNSTABLE
> >  kworker/u1037:2-2871862 [097] ..... 46138.517659: nfs4_pnfs_write:
> > error=0 (OK) fileid=00:aa:10056165185 fhandle=0x6bd94d55 offset=0
> > count=53248 res=53248 stateid=1:0x79a9c471 layoutstateid=1:0xcbd8aaad
> >  kworker/u1037:2-2871862 [097] ..... 46138.517662:
> > nfs_writeback_done: error=53248 fileid=00:aa:10056165185
> > fhandle=0x6bd94d55 offset=0 count=53248 res=53248 stable=UNSTABLE
> > verifier=5199cdae2816c899
> >  kworker/u1037:5-2593935 [226] ..... 46138.517669: nfs4_pnfs_write:
> > error=0 (OK) fileid=00:aa:10056165185 fhandle=0x6bd94d55 offset=53248
> > count=1895 res=1895 stateid=1:0x79a9c471 layoutstateid=1:0xcbd8aaad
> >  kworker/u1037:5-2593935 [226] ..... 46138.517672:
> > nfs_writeback_done: error=1895 fileid=00:aa:10056165185
> > fhandle=0x6bd94d55 offset=53248 count=1895 res=1895 stable=UNSTABLE
> > verifier=5199cdae2816c899
> >  oil-localfs-252-2605046 [162] ..... 46138.518360: nfs_size_grow:
> > fileid=00:aa:10056165185 fhandle=0x8bfc64c9
> > version=1753485366409158129 cursize=55143 newsize=57344
> >  oil-localfs-252-2605046 [162] ..... 46138.518556: nfs_size_grow:
> > fileid=00:aa:10056165185 fhandle=0x8bfc64c9
> > version=1753485366409158129 cursize=57344 newsize=60156
> > 
> > ...and just after writeback completes, we see the file size grow from
> > 55143 to the end of the page (57344).
> > 
> > The second trace has similar symptoms. There is a lot more (smaller)
> > write activity (due to memory pressure?). There is a gap of 3791
> > bytes
> > between these on-the-wire writes, however:
> > 
> >  kworker/u1036:0-2339252 [217] ..... 479572.054622:
> > nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232
> > offset=221184 count=4401 stable=UNSTABLE
> >  kworker/u1030:1-2297876 [042] ..... 479572.074194:
> > nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232
> > offset=229376 count=261898 stable=UNSTABLE
> > 
> > Same situation -- the at page at offset 53248 has 305 bytes on it,
> > and
> > the remaining is zeroed. This trace shows similar racing write() and
> > writeback activity as in Friday's trace. At around the same time as
> > the
> > client was growing the file over the affected range, writeback was
> > kicking off for everything up to the affected range (this has some
> > other wb related calls filtered for brevity):
> > 
> >   oil-localfs-86-727850  [215] ..... 479572.053987: nfs_size_grow:
> > fileid=00:96:10067193438 fhandle=0x14c40498
> > version=1753823598774309300 cursize=217088
> > newsize=220572                                                       
> >               
> >  kworker/u1036:8-2339326 [088] ..... 479572.054008:
> > nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232
> > offset=217088 count=3484
> > stable=UNSTABLE                                                      
> >                            
> >   oil-localfs-86-727850  [215] ..... 479572.054405: nfs_size_grow:
> > fileid=00:96:10067193438 fhandle=0x14c40498
> > version=1753823598774309300 cursize=220572
> > newsize=221184                                                       
> >               
> >  kworker/u1036:1-2297875 [217] ..... 479572.054418:
> > nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232
> > offset=220572 count=612
> > stable=UNSTABLE                                                      
> >                             
> >   oil-localfs-86-727850  [215] ..... 479572.054581: nfs_size_grow:
> > fileid=00:96:10067193438 fhandle=0x14c40498
> > version=1753823598774309300 cursize=221184
> > newsize=225280                                                       
> >               
> >   oil-localfs-86-727850  [215] ..... 479572.054584: nfs_size_grow:
> > fileid=00:96:10067193438 fhandle=0x14c40498
> > version=1753823598774309300 cursize=225280
> > newsize=225585                                                       
> >               
> >  kworker/u1036:0-2339252 [217] ..... 479572.054622:
> > nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232
> > offset=221184 count=4401
> > stable=UNSTABLE                                                      
> >                            
> >   oil-localfs-86-727850  [215] ..... 479572.054997: nfs_size_grow:
> > fileid=00:96:10067193438 fhandle=0x14c40498
> > version=1753823598774309300 cursize=225585
> > newsize=229376                                                       
> >               
> >   oil-localfs-86-727850  [215] ..... 479572.055190: nfs_size_grow:
> > fileid=00:96:10067193438 fhandle=0x14c40498
> > version=1753823598774309300 cursize=229376
> > newsize=230598                                                       
> >               
> > 
> > Could this be a race between extending an existing dirty page, and
> > writeback kicking off for the pre-extension range on the page? Maybe
> > the client is clearing the dirty bit, thinking that the write covers
> > the dirty range, but it has an outdated idea about what that range is
> > or doesn't properly check?
> > 
> > Traces for both events, filtered on the relevant fileid are attached.
> > I've rolled patches for some new tracepoints that I'm going to
> > attempt
> > to turn up next, but I thought that this was a good point to solicit
> > ideas.
> > 
> > Happy to entertain other thoughts or patches!
> 
> So... The fact that we are seeing a nfs_size_grow() for the hole at
> offset 55143 means that either an existing request was updated, or a
> new one was created in order to cover that hole, and it must have been
> marked as dirty.
> 
> I'm not seeing anything in the NFS code that can lose that request
> without triggering either the nfs_write_error tracepoint, the
> nfs_commit_error tracepoint, the nfs_invalidate_folio tracepoint or
> else completing the write.
> 
> The only other way I can see this data being lost is if something is
> corrupting folio->private, or if the page cache is somehow managing to
> throw away a dirty folio.
> Of the two, there was for a while a netfs bug which would corrupt
> folio->private, but I assume you're not using cachefs?

After staring at this code a lot, I have a theory. But, it seems like
we'd be seeing this a lot more if it were correct, so I must be
overlooking something.

Here's the scenario:

--------------8<--------------

Userland has written some of a file and the last folio is not full.

Writeback has kicked off for the inode and is successful.
nfs_write_completion() calls nfs_page_end_writeback(). That will unlock
the nfs_page (clear PG_BUSY) and leave it attached to the folio, and on
the commit list.

Next a write from userland comes in to extend the file to the end of
the page (and beyond). nfs_try_to_update_request() merges the write
into the original request and re-marks the page dirty.

Later the commit runs successfully and the write verifier matches.
nfs_commit_release_pages() runs and nfs_inode_remove_request() is
called which detaches the nfs_page from the folio.

Eventually, writeback starts up again and the folio is picked up and
submitted by nfs_writepages(), but folio->private is now NULL, and it's
ignored.

But...like I said I feel like we'd hit this all the time if it were
possible, even though I don't see what prevents it. If this is a
possibility, then the patch may be as simple as something like this?

diff --git a/fs/nfs/write.c b/fs/nfs/write.c
index bb0e78644ffb..72402208fa33 100644
--- a/fs/nfs/write.c
+++ b/fs/nfs/write.c
@@ -1867,7 +1867,7 @@ static void nfs_commit_release_pages(struct
nfs_commit_data *data)
                 * returned by the server against all stored verfs. */
                if (nfs_write_match_verf(verf, req)) {
                        /* We have a match */
-                       if (folio)
+                       if (folio && !folio_test_dirty(folio))
                                nfs_inode_remove_request(req);
                        dprintk_cont(" OK\n");
                        goto next;

Re: parts of pages on NFS being replaced by swaths of NULs

[Trond Myklebust]

On Tue, 2025-08-12 at 07:58 -0400, Jeff Layton wrote:
> On Thu, 2025-07-31 at 17:56 -0400, Trond Myklebust wrote:
> > On Wed, 2025-07-30 at 10:52 -0400, Jeff Layton wrote:
> > > We've been seeing a rather nasty bit of data corruption with NFS
> > > in
> > > our
> > > environment. The clients in this env run a patched v6.9 kernel
> > > (mostly
> > > due to GPU driver requirements). Most of the patches are NFS
> > > containerization fixes.
> > > 
> > > The workload is python scripts writing JSONL files sequentially
> > > using
> > > bog-standard buffered write() calls. We're fairly certain that
> > > userland
> > > is not seeking so there should be no gaps in the data written.
> > > 
> > > The problem is that we see ranges of written files being replaced
> > > by
> > > NULs. The length of the file seemingly doesn't change from what
> > > it
> > > should be, but a chunk of it will be zeroed-out. Looking at the
> > > offsets
> > > of the zeroed out ranges, the front part of one page is fine, but
> > > the
> > > data from some random offset in the page to the end of the page
> > > is
> > > zeroes.
> > > 
> > > We have a reproducer but we have to run it in a heavily parallel
> > > configuration to make it happen, so it's evidently a tight race
> > > of
> > > some
> > > sort.
> > > 
> > > We've turned up some tracepoints and reproduced this twice. What
> > > we
> > > see
> > > in both cases is that the client just doesn't write some section
> > > of
> > > the
> > > file.
> > > 
> > > In the first trace, there was is a gap of 2201 bytes between
> > > these
> > > two
> > > writes on the wire:
> > > 
> > >  kworker/u1038:1-2597138 [106] ..... 46138.516795:
> > > nfs_initiate_write: fileid=00:aa:10056165185 fhandle=0x6bd94d55
> > > offset=53248 count=1895 stable=UNSTABLE
> > >  oil-localfs-252-2605046 [163] ..... 46138.551459:
> > > nfs_initiate_write: fileid=00:aa:10056165185 fhandle=0x6bd94d55
> > > offset=57344 count=443956 stable=FILE_SYNC
> > > 
> > > The zeroed-out range is from 55143-57344. At the same time that
> > > the
> > > file is growing from 53248 to 55143 (due to sequential write()
> > > activity), the client is kicking off writeback for the range up
> > > to
> > > 55143. It's issuing 2 writes, one for 0-53248 and one for 53248-
> > > 55143
> > > (note that I've filtered out all but one of the DS filehandles
> > > for
> > > brevity):
> > > 
> > >  oil-localfs-252-2605046 [162] ..... 46138.516414: nfs_size_grow:
> > > fileid=00:aa:10056165185 fhandle=0x8bfc64c9
> > > version=1753485366409158129 cursize=49152 newsize=50130
> > >  oil-localfs-252-2605046 [162] ..... 46138.516593: nfs_size_grow:
> > > fileid=00:aa:10056165185 fhandle=0x8bfc64c9
> > > version=1753485366409158129 cursize=50130 newsize=53248
> > >  kworker/u1038:1-2597138 [106] ..... 46138.516740:
> > > nfs_initiate_write: fileid=00:aa:10056165185 fhandle=0x6bd94d55
> > > offset=0 count=53248 stable=UNSTABLE
> > >  oil-localfs-252-2605046 [162] ..... 46138.516753: nfs_size_grow:
> > > fileid=00:aa:10056165185 fhandle=0x8bfc64c9
> > > version=1753485366409158129 cursize=53248 newsize=55143
> > >  kworker/u1038:1-2597138 [106] ..... 46138.516795:
> > > nfs_initiate_write: fileid=00:aa:10056165185 fhandle=0x6bd94d55
> > > offset=53248 count=1895 stable=UNSTABLE
> > >  kworker/u1037:2-2871862 [097] ..... 46138.517659:
> > > nfs4_pnfs_write:
> > > error=0 (OK) fileid=00:aa:10056165185 fhandle=0x6bd94d55 offset=0
> > > count=53248 res=53248 stateid=1:0x79a9c471
> > > layoutstateid=1:0xcbd8aaad
> > >  kworker/u1037:2-2871862 [097] ..... 46138.517662:
> > > nfs_writeback_done: error=53248 fileid=00:aa:10056165185
> > > fhandle=0x6bd94d55 offset=0 count=53248 res=53248 stable=UNSTABLE
> > > verifier=5199cdae2816c899
> > >  kworker/u1037:5-2593935 [226] ..... 46138.517669:
> > > nfs4_pnfs_write:
> > > error=0 (OK) fileid=00:aa:10056165185 fhandle=0x6bd94d55
> > > offset=53248
> > > count=1895 res=1895 stateid=1:0x79a9c471
> > > layoutstateid=1:0xcbd8aaad
> > >  kworker/u1037:5-2593935 [226] ..... 46138.517672:
> > > nfs_writeback_done: error=1895 fileid=00:aa:10056165185
> > > fhandle=0x6bd94d55 offset=53248 count=1895 res=1895
> > > stable=UNSTABLE
> > > verifier=5199cdae2816c899
> > >  oil-localfs-252-2605046 [162] ..... 46138.518360: nfs_size_grow:
> > > fileid=00:aa:10056165185 fhandle=0x8bfc64c9
> > > version=1753485366409158129 cursize=55143 newsize=57344
> > >  oil-localfs-252-2605046 [162] ..... 46138.518556: nfs_size_grow:
> > > fileid=00:aa:10056165185 fhandle=0x8bfc64c9
> > > version=1753485366409158129 cursize=57344 newsize=60156
> > > 
> > > ...and just after writeback completes, we see the file size grow
> > > from
> > > 55143 to the end of the page (57344).
> > > 
> > > The second trace has similar symptoms. There is a lot more
> > > (smaller)
> > > write activity (due to memory pressure?). There is a gap of 3791
> > > bytes
> > > between these on-the-wire writes, however:
> > > 
> > >  kworker/u1036:0-2339252 [217] ..... 479572.054622:
> > > nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232
> > > offset=221184 count=4401 stable=UNSTABLE
> > >  kworker/u1030:1-2297876 [042] ..... 479572.074194:
> > > nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232
> > > offset=229376 count=261898 stable=UNSTABLE
> > > 
> > > Same situation -- the at page at offset 53248 has 305 bytes on
> > > it,
> > > and
> > > the remaining is zeroed. This trace shows similar racing write()
> > > and
> > > writeback activity as in Friday's trace. At around the same time
> > > as
> > > the
> > > client was growing the file over the affected range, writeback
> > > was
> > > kicking off for everything up to the affected range (this has
> > > some
> > > other wb related calls filtered for brevity):
> > > 
> > >   oil-localfs-86-727850  [215] ..... 479572.053987:
> > > nfs_size_grow:
> > > fileid=00:96:10067193438 fhandle=0x14c40498
> > > version=1753823598774309300 cursize=217088
> > > newsize=220572                                                   
> > >     
> > >               
> > >  kworker/u1036:8-2339326 [088] ..... 479572.054008:
> > > nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232
> > > offset=217088 count=3484
> > > stable=UNSTABLE                                                  
> > >     
> > >                            
> > >   oil-localfs-86-727850  [215] ..... 479572.054405:
> > > nfs_size_grow:
> > > fileid=00:96:10067193438 fhandle=0x14c40498
> > > version=1753823598774309300 cursize=220572
> > > newsize=221184                                                   
> > >     
> > >               
> > >  kworker/u1036:1-2297875 [217] ..... 479572.054418:
> > > nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232
> > > offset=220572 count=612
> > > stable=UNSTABLE                                                  
> > >     
> > >                             
> > >   oil-localfs-86-727850  [215] ..... 479572.054581:
> > > nfs_size_grow:
> > > fileid=00:96:10067193438 fhandle=0x14c40498
> > > version=1753823598774309300 cursize=221184
> > > newsize=225280                                                   
> > >     
> > >               
> > >   oil-localfs-86-727850  [215] ..... 479572.054584:
> > > nfs_size_grow:
> > > fileid=00:96:10067193438 fhandle=0x14c40498
> > > version=1753823598774309300 cursize=225280
> > > newsize=225585                                                   
> > >     
> > >               
> > >  kworker/u1036:0-2339252 [217] ..... 479572.054622:
> > > nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232
> > > offset=221184 count=4401
> > > stable=UNSTABLE                                                  
> > >     
> > >                            
> > >   oil-localfs-86-727850  [215] ..... 479572.054997:
> > > nfs_size_grow:
> > > fileid=00:96:10067193438 fhandle=0x14c40498
> > > version=1753823598774309300 cursize=225585
> > > newsize=229376                                                   
> > >     
> > >               
> > >   oil-localfs-86-727850  [215] ..... 479572.055190:
> > > nfs_size_grow:
> > > fileid=00:96:10067193438 fhandle=0x14c40498
> > > version=1753823598774309300 cursize=229376
> > > newsize=230598                                                   
> > >     
> > >               
> > > 
> > > Could this be a race between extending an existing dirty page,
> > > and
> > > writeback kicking off for the pre-extension range on the page?
> > > Maybe
> > > the client is clearing the dirty bit, thinking that the write
> > > covers
> > > the dirty range, but it has an outdated idea about what that
> > > range is
> > > or doesn't properly check?
> > > 
> > > Traces for both events, filtered on the relevant fileid are
> > > attached.
> > > I've rolled patches for some new tracepoints that I'm going to
> > > attempt
> > > to turn up next, but I thought that this was a good point to
> > > solicit
> > > ideas.
> > > 
> > > Happy to entertain other thoughts or patches!
> > 
> > So... The fact that we are seeing a nfs_size_grow() for the hole at
> > offset 55143 means that either an existing request was updated, or
> > a
> > new one was created in order to cover that hole, and it must have
> > been
> > marked as dirty.
> > 
> > I'm not seeing anything in the NFS code that can lose that request
> > without triggering either the nfs_write_error tracepoint, the
> > nfs_commit_error tracepoint, the nfs_invalidate_folio tracepoint or
> > else completing the write.
> > 
> > The only other way I can see this data being lost is if something
> > is
> > corrupting folio->private, or if the page cache is somehow managing
> > to
> > throw away a dirty folio.
> > Of the two, there was for a while a netfs bug which would corrupt
> > folio->private, but I assume you're not using cachefs?
> 
> After staring at this code a lot, I have a theory. But, it seems like
> we'd be seeing this a lot more if it were correct, so I must be
> overlooking something.
> 
> Here's the scenario:
> 
> --------------8<--------------
> 
> Userland has written some of a file and the last folio is not full.
> 
> Writeback has kicked off for the inode and is successful.
> nfs_write_completion() calls nfs_page_end_writeback(). That will
> unlock
> the nfs_page (clear PG_BUSY) and leave it attached to the folio, and
> on
> the commit list.
> 
> Next a write from userland comes in to extend the file to the end of
> the page (and beyond). nfs_try_to_update_request() merges the write
> into the original request and re-marks the page dirty.
> 
> Later the commit runs successfully and the write verifier matches.
> nfs_commit_release_pages() runs and nfs_inode_remove_request() is
> called which detaches the nfs_page from the folio.
> 
> Eventually, writeback starts up again and the folio is picked up and
> submitted by nfs_writepages(), but folio->private is now NULL, and
> it's
> ignored.
> 
> But...like I said I feel like we'd hit this all the time if it were
> possible, even though I don't see what prevents it. If this is a
> possibility, then the patch may be as simple as something like this?
> 
> diff --git a/fs/nfs/write.c b/fs/nfs/write.c
> index bb0e78644ffb..72402208fa33 100644
> --- a/fs/nfs/write.c
> +++ b/fs/nfs/write.c
> @@ -1867,7 +1867,7 @@ static void nfs_commit_release_pages(struct
> nfs_commit_data *data)
>                  * returned by the server against all stored verfs.
> */
>                 if (nfs_write_match_verf(verf, req)) {
>                         /* We have a match */
> -                       if (folio)
> +                       if (folio && !folio_test_dirty(folio))
>                                 nfs_inode_remove_request(req);
>                         dprintk_cont(" OK\n");
>                         goto next;

The call to nfs_clear_request_commit() in nfs_join_page_group() should
be taking care of removing the page before a COMMIT is sent.

During both the writeback and the commit, the nfs_page is locked, so
won't be available to be updated by nfs_try_to_update_request().

-- 
Trond Myklebust
Linux NFS client maintainer, Hammerspace
trondmy@kernel.org, trond.myklebust@hammerspace.com

Re: parts of pages on NFS being replaced by swaths of NULs

[Jeff Layton]

On Tue, 2025-08-12 at 09:58 -0700, Trond Myklebust wrote:
> On Tue, 2025-08-12 at 07:58 -0400, Jeff Layton wrote:
> > On Thu, 2025-07-31 at 17:56 -0400, Trond Myklebust wrote:
> > > On Wed, 2025-07-30 at 10:52 -0400, Jeff Layton wrote:
> > > > We've been seeing a rather nasty bit of data corruption with NFS
> > > > in
> > > > our
> > > > environment. The clients in this env run a patched v6.9 kernel
> > > > (mostly
> > > > due to GPU driver requirements). Most of the patches are NFS
> > > > containerization fixes.
> > > > 
> > > > The workload is python scripts writing JSONL files sequentially
> > > > using
> > > > bog-standard buffered write() calls. We're fairly certain that
> > > > userland
> > > > is not seeking so there should be no gaps in the data written.
> > > > 
> > > > The problem is that we see ranges of written files being replaced
> > > > by
> > > > NULs. The length of the file seemingly doesn't change from what
> > > > it
> > > > should be, but a chunk of it will be zeroed-out. Looking at the
> > > > offsets
> > > > of the zeroed out ranges, the front part of one page is fine, but
> > > > the
> > > > data from some random offset in the page to the end of the page
> > > > is
> > > > zeroes.
> > > > 
> > > > We have a reproducer but we have to run it in a heavily parallel
> > > > configuration to make it happen, so it's evidently a tight race
> > > > of
> > > > some
> > > > sort.
> > > > 
> > > > We've turned up some tracepoints and reproduced this twice. What
> > > > we
> > > > see
> > > > in both cases is that the client just doesn't write some section
> > > > of
> > > > the
> > > > file.
> > > > 
> > > > In the first trace, there was is a gap of 2201 bytes between
> > > > these
> > > > two
> > > > writes on the wire:
> > > > 
> > > >  kworker/u1038:1-2597138 [106] ..... 46138.516795:
> > > > nfs_initiate_write: fileid=00:aa:10056165185 fhandle=0x6bd94d55
> > > > offset=53248 count=1895 stable=UNSTABLE
> > > >  oil-localfs-252-2605046 [163] ..... 46138.551459:
> > > > nfs_initiate_write: fileid=00:aa:10056165185 fhandle=0x6bd94d55
> > > > offset=57344 count=443956 stable=FILE_SYNC
> > > > 
> > > > The zeroed-out range is from 55143-57344. At the same time that
> > > > the
> > > > file is growing from 53248 to 55143 (due to sequential write()
> > > > activity), the client is kicking off writeback for the range up
> > > > to
> > > > 55143. It's issuing 2 writes, one for 0-53248 and one for 53248-
> > > > 55143
> > > > (note that I've filtered out all but one of the DS filehandles
> > > > for
> > > > brevity):
> > > > 
> > > >  oil-localfs-252-2605046 [162] ..... 46138.516414: nfs_size_grow:
> > > > fileid=00:aa:10056165185 fhandle=0x8bfc64c9
> > > > version=1753485366409158129 cursize=49152 newsize=50130
> > > >  oil-localfs-252-2605046 [162] ..... 46138.516593: nfs_size_grow:
> > > > fileid=00:aa:10056165185 fhandle=0x8bfc64c9
> > > > version=1753485366409158129 cursize=50130 newsize=53248
> > > >  kworker/u1038:1-2597138 [106] ..... 46138.516740:
> > > > nfs_initiate_write: fileid=00:aa:10056165185 fhandle=0x6bd94d55
> > > > offset=0 count=53248 stable=UNSTABLE
> > > >  oil-localfs-252-2605046 [162] ..... 46138.516753: nfs_size_grow:
> > > > fileid=00:aa:10056165185 fhandle=0x8bfc64c9
> > > > version=1753485366409158129 cursize=53248 newsize=55143
> > > >  kworker/u1038:1-2597138 [106] ..... 46138.516795:
> > > > nfs_initiate_write: fileid=00:aa:10056165185 fhandle=0x6bd94d55
> > > > offset=53248 count=1895 stable=UNSTABLE
> > > >  kworker/u1037:2-2871862 [097] ..... 46138.517659:
> > > > nfs4_pnfs_write:
> > > > error=0 (OK) fileid=00:aa:10056165185 fhandle=0x6bd94d55 offset=0
> > > > count=53248 res=53248 stateid=1:0x79a9c471
> > > > layoutstateid=1:0xcbd8aaad
> > > >  kworker/u1037:2-2871862 [097] ..... 46138.517662:
> > > > nfs_writeback_done: error=53248 fileid=00:aa:10056165185
> > > > fhandle=0x6bd94d55 offset=0 count=53248 res=53248 stable=UNSTABLE
> > > > verifier=5199cdae2816c899
> > > >  kworker/u1037:5-2593935 [226] ..... 46138.517669:
> > > > nfs4_pnfs_write:
> > > > error=0 (OK) fileid=00:aa:10056165185 fhandle=0x6bd94d55
> > > > offset=53248
> > > > count=1895 res=1895 stateid=1:0x79a9c471
> > > > layoutstateid=1:0xcbd8aaad
> > > >  kworker/u1037:5-2593935 [226] ..... 46138.517672:
> > > > nfs_writeback_done: error=1895 fileid=00:aa:10056165185
> > > > fhandle=0x6bd94d55 offset=53248 count=1895 res=1895
> > > > stable=UNSTABLE
> > > > verifier=5199cdae2816c899
> > > >  oil-localfs-252-2605046 [162] ..... 46138.518360: nfs_size_grow:
> > > > fileid=00:aa:10056165185 fhandle=0x8bfc64c9
> > > > version=1753485366409158129 cursize=55143 newsize=57344
> > > >  oil-localfs-252-2605046 [162] ..... 46138.518556: nfs_size_grow:
> > > > fileid=00:aa:10056165185 fhandle=0x8bfc64c9
> > > > version=1753485366409158129 cursize=57344 newsize=60156
> > > > 
> > > > ...and just after writeback completes, we see the file size grow
> > > > from
> > > > 55143 to the end of the page (57344).
> > > > 
> > > > The second trace has similar symptoms. There is a lot more
> > > > (smaller)
> > > > write activity (due to memory pressure?). There is a gap of 3791
> > > > bytes
> > > > between these on-the-wire writes, however:
> > > > 
> > > >  kworker/u1036:0-2339252 [217] ..... 479572.054622:
> > > > nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232
> > > > offset=221184 count=4401 stable=UNSTABLE
> > > >  kworker/u1030:1-2297876 [042] ..... 479572.074194:
> > > > nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232
> > > > offset=229376 count=261898 stable=UNSTABLE
> > > > 
> > > > Same situation -- the at page at offset 53248 has 305 bytes on
> > > > it,
> > > > and
> > > > the remaining is zeroed. This trace shows similar racing write()
> > > > and
> > > > writeback activity as in Friday's trace. At around the same time
> > > > as
> > > > the
> > > > client was growing the file over the affected range, writeback
> > > > was
> > > > kicking off for everything up to the affected range (this has
> > > > some
> > > > other wb related calls filtered for brevity):
> > > > 
> > > >   oil-localfs-86-727850  [215] ..... 479572.053987:
> > > > nfs_size_grow:
> > > > fileid=00:96:10067193438 fhandle=0x14c40498
> > > > version=1753823598774309300 cursize=217088
> > > > newsize=220572                                                   
> > > >     
> > > >               
> > > >  kworker/u1036:8-2339326 [088] ..... 479572.054008:
> > > > nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232
> > > > offset=217088 count=3484
> > > > stable=UNSTABLE                                                  
> > > >     
> > > >                            
> > > >   oil-localfs-86-727850  [215] ..... 479572.054405:
> > > > nfs_size_grow:
> > > > fileid=00:96:10067193438 fhandle=0x14c40498
> > > > version=1753823598774309300 cursize=220572
> > > > newsize=221184                                                   
> > > >     
> > > >               
> > > >  kworker/u1036:1-2297875 [217] ..... 479572.054418:
> > > > nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232
> > > > offset=220572 count=612
> > > > stable=UNSTABLE                                                  
> > > >     
> > > >                             
> > > >   oil-localfs-86-727850  [215] ..... 479572.054581:
> > > > nfs_size_grow:
> > > > fileid=00:96:10067193438 fhandle=0x14c40498
> > > > version=1753823598774309300 cursize=221184
> > > > newsize=225280                                                   
> > > >     
> > > >               
> > > >   oil-localfs-86-727850  [215] ..... 479572.054584:
> > > > nfs_size_grow:
> > > > fileid=00:96:10067193438 fhandle=0x14c40498
> > > > version=1753823598774309300 cursize=225280
> > > > newsize=225585                                                   
> > > >     
> > > >               
> > > >  kworker/u1036:0-2339252 [217] ..... 479572.054622:
> > > > nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232
> > > > offset=221184 count=4401
> > > > stable=UNSTABLE                                                  
> > > >     
> > > >                            
> > > >   oil-localfs-86-727850  [215] ..... 479572.054997:
> > > > nfs_size_grow:
> > > > fileid=00:96:10067193438 fhandle=0x14c40498
> > > > version=1753823598774309300 cursize=225585
> > > > newsize=229376                                                   
> > > >     
> > > >               
> > > >   oil-localfs-86-727850  [215] ..... 479572.055190:
> > > > nfs_size_grow:
> > > > fileid=00:96:10067193438 fhandle=0x14c40498
> > > > version=1753823598774309300 cursize=229376
> > > > newsize=230598                                                   
> > > >     
> > > >               
> > > > 
> > > > Could this be a race between extending an existing dirty page,
> > > > and
> > > > writeback kicking off for the pre-extension range on the page?
> > > > Maybe
> > > > the client is clearing the dirty bit, thinking that the write
> > > > covers
> > > > the dirty range, but it has an outdated idea about what that
> > > > range is
> > > > or doesn't properly check?
> > > > 
> > > > Traces for both events, filtered on the relevant fileid are
> > > > attached.
> > > > I've rolled patches for some new tracepoints that I'm going to
> > > > attempt
> > > > to turn up next, but I thought that this was a good point to
> > > > solicit
> > > > ideas.
> > > > 
> > > > Happy to entertain other thoughts or patches!
> > > 
> > > So... The fact that we are seeing a nfs_size_grow() for the hole at
> > > offset 55143 means that either an existing request was updated, or
> > > a
> > > new one was created in order to cover that hole, and it must have
> > > been
> > > marked as dirty.
> > > 
> > > I'm not seeing anything in the NFS code that can lose that request
> > > without triggering either the nfs_write_error tracepoint, the
> > > nfs_commit_error tracepoint, the nfs_invalidate_folio tracepoint or
> > > else completing the write.
> > > 
> > > The only other way I can see this data being lost is if something
> > > is
> > > corrupting folio->private, or if the page cache is somehow managing
> > > to
> > > throw away a dirty folio.
> > > Of the two, there was for a while a netfs bug which would corrupt
> > > folio->private, but I assume you're not using cachefs?
> > 
> > After staring at this code a lot, I have a theory. But, it seems like
> > we'd be seeing this a lot more if it were correct, so I must be
> > overlooking something.
> > 
> > Here's the scenario:
> > 
> > --------------8<--------------
> > 
> > Userland has written some of a file and the last folio is not full.
> > 
> > Writeback has kicked off for the inode and is successful.
> > nfs_write_completion() calls nfs_page_end_writeback(). That will
> > unlock
> > the nfs_page (clear PG_BUSY) and leave it attached to the folio, and
> > on
> > the commit list.
> > 
> > Next a write from userland comes in to extend the file to the end of
> > the page (and beyond). nfs_try_to_update_request() merges the write
> > into the original request and re-marks the page dirty.
> > 
> > Later the commit runs successfully and the write verifier matches.
> > nfs_commit_release_pages() runs and nfs_inode_remove_request() is
> > called which detaches the nfs_page from the folio.
> > 
> > Eventually, writeback starts up again and the folio is picked up and
> > submitted by nfs_writepages(), but folio->private is now NULL, and
> > it's
> > ignored.
> > 
> > But...like I said I feel like we'd hit this all the time if it were
> > possible, even though I don't see what prevents it. If this is a
> > possibility, then the patch may be as simple as something like this?
> > 
> > diff --git a/fs/nfs/write.c b/fs/nfs/write.c
> > index bb0e78644ffb..72402208fa33 100644
> > --- a/fs/nfs/write.c
> > +++ b/fs/nfs/write.c
> > @@ -1867,7 +1867,7 @@ static void nfs_commit_release_pages(struct
> > nfs_commit_data *data)
> >                  * returned by the server against all stored verfs.
> > */
> >                 if (nfs_write_match_verf(verf, req)) {
> >                         /* We have a match */
> > -                       if (folio)
> > +                       if (folio && !folio_test_dirty(folio))
> >                                 nfs_inode_remove_request(req);
> >                         dprintk_cont(" OK\n");
> >                         goto next;
> 
> The call to nfs_clear_request_commit() in nfs_join_page_group() should
> be taking care of removing the page before a COMMIT is sent.
> 

Got it, thanks. Yeah, that should prevent the commit callback from
touching the page.

> During both the writeback and the commit, the nfs_page is locked, so
> won't be available to be updated by nfs_try_to_update_request().

Yes. I was thinking of the window between the two, but I think you're
right that nfs_clear_request_commit() should prevent the commit from
touching the page after it gets merged in.

Still, the timing of the trace messages suggests that there is race of
some sort:

 oil-localfs-161-1326370 [135] ..... 42066.128429: nfs_write_end: fileid=00:82:10087279963 fhandle=0xa6491ad0 version=1754454982197296994 offset=10026 count=2262
 oil-localfs-161-1326370 [135] ..... 42066.128429: nfs_update_folio: fileid=00:82:10087279963 fhandle=0xa6491ad0 version=1754454982197296994 offset=1834 count=2262
 oil-localfs-161-1326370 [135] ..... 42066.128429: nfs_try_to_update_request: fileid=00:82:10087279963 fhandle=0xa6491ad0 version=1754454982197296994 offset=1834 count=2262

Here it ends up blocked in nfs_lock_request() while the commit runs.

 kworker/u1028:3-3432139 [031] ..... 42066.128431: nfs_initiate_commit: fileid=00:82:10087279963 fhandle=0x483d45e0 offset=0 count=0
 kworker/u1028:3-3432139 [031] ..... 42066.128435: nfs_initiate_commit: fileid=00:82:10087279963 fhandle=0x7619eec9 offset=0 count=0
 kworker/u1028:3-3432139 [031] ..... 42066.128437: nfs_initiate_commit: fileid=00:82:10087279963 fhandle=0xd0cc79e6 offset=0 count=0
 kworker/u1037:1-3477598 [226] ..... 42066.129140: nfs_commit_done: error=0 fileid=00:82:10087279963 fhandle=0x7619eec9 offset=0 stable=FILE_SYNC verifier=615487ab86b79ab3
 kworker/u1035:0-3302696 [209] ..... 42066.129158: nfs_commit_done: error=0 fileid=00:82:10087279963 fhandle=0x483d45e0 offset=0 stable=FILE_SYNC verifier=a376459679d60091
 kworker/u1028:3-3432139 [031] ..... 42066.129163: nfs_commit_done: error=0 fileid=00:82:10087279963 fhandle=0xd0cc79e6 offset=0 stable=FILE_SYNC verifier=ff1ddeecd9737f82

kworker drops the lock, nfs_try_to_update_request() runs to completion and eventually the nfs_size_grow() runs.

 oil-localfs-161-1326370 [135] ..... 42066.129179: nfs_size_grow: fileid=00:82:10087279963 fhandle=0xa6491ad0 version=1754454982197296994 cursize=10026 newsize=12288

FWIW, I have a bpftrace script that I'm hoping might give us a bit more
info about why the page didn't get written back when this happens. Just
waiting for some time to run it in our test env.

I'll keep looking!
-- 
Jeff Layton <jlayton@kernel.org>

Re: parts of pages on NFS being replaced by swaths of NULs

[Jeff Layton]

[-- Attachment #1: Type: text/plain, Size: 16275 bytes --]

On Tue, 2025-08-12 at 09:58 -0700, Trond Myklebust wrote:
> On Tue, 2025-08-12 at 07:58 -0400, Jeff Layton wrote:
> > On Thu, 2025-07-31 at 17:56 -0400, Trond Myklebust wrote:
> > > On Wed, 2025-07-30 at 10:52 -0400, Jeff Layton wrote:
> > > > We've been seeing a rather nasty bit of data corruption with NFS
> > > > in
> > > > our
> > > > environment. The clients in this env run a patched v6.9 kernel
> > > > (mostly
> > > > due to GPU driver requirements). Most of the patches are NFS
> > > > containerization fixes.
> > > > 
> > > > The workload is python scripts writing JSONL files sequentially
> > > > using
> > > > bog-standard buffered write() calls. We're fairly certain that
> > > > userland
> > > > is not seeking so there should be no gaps in the data written.
> > > > 
> > > > The problem is that we see ranges of written files being replaced
> > > > by
> > > > NULs. The length of the file seemingly doesn't change from what
> > > > it
> > > > should be, but a chunk of it will be zeroed-out. Looking at the
> > > > offsets
> > > > of the zeroed out ranges, the front part of one page is fine, but
> > > > the
> > > > data from some random offset in the page to the end of the page
> > > > is
> > > > zeroes.
> > > > 
> > > > We have a reproducer but we have to run it in a heavily parallel
> > > > configuration to make it happen, so it's evidently a tight race
> > > > of
> > > > some
> > > > sort.
> > > > 
> > > > We've turned up some tracepoints and reproduced this twice. What
> > > > we
> > > > see
> > > > in both cases is that the client just doesn't write some section
> > > > of
> > > > the
> > > > file.
> > > > 
> > > > In the first trace, there was is a gap of 2201 bytes between
> > > > these
> > > > two
> > > > writes on the wire:
> > > > 
> > > >  kworker/u1038:1-2597138 [106] ..... 46138.516795:
> > > > nfs_initiate_write: fileid=00:aa:10056165185 fhandle=0x6bd94d55
> > > > offset=53248 count=1895 stable=UNSTABLE
> > > >  oil-localfs-252-2605046 [163] ..... 46138.551459:
> > > > nfs_initiate_write: fileid=00:aa:10056165185 fhandle=0x6bd94d55
> > > > offset=57344 count=443956 stable=FILE_SYNC
> > > > 
> > > > The zeroed-out range is from 55143-57344. At the same time that
> > > > the
> > > > file is growing from 53248 to 55143 (due to sequential write()
> > > > activity), the client is kicking off writeback for the range up
> > > > to
> > > > 55143. It's issuing 2 writes, one for 0-53248 and one for 53248-
> > > > 55143
> > > > (note that I've filtered out all but one of the DS filehandles
> > > > for
> > > > brevity):
> > > > 
> > > >  oil-localfs-252-2605046 [162] ..... 46138.516414: nfs_size_grow:
> > > > fileid=00:aa:10056165185 fhandle=0x8bfc64c9
> > > > version=1753485366409158129 cursize=49152 newsize=50130
> > > >  oil-localfs-252-2605046 [162] ..... 46138.516593: nfs_size_grow:
> > > > fileid=00:aa:10056165185 fhandle=0x8bfc64c9
> > > > version=1753485366409158129 cursize=50130 newsize=53248
> > > >  kworker/u1038:1-2597138 [106] ..... 46138.516740:
> > > > nfs_initiate_write: fileid=00:aa:10056165185 fhandle=0x6bd94d55
> > > > offset=0 count=53248 stable=UNSTABLE
> > > >  oil-localfs-252-2605046 [162] ..... 46138.516753: nfs_size_grow:
> > > > fileid=00:aa:10056165185 fhandle=0x8bfc64c9
> > > > version=1753485366409158129 cursize=53248 newsize=55143
> > > >  kworker/u1038:1-2597138 [106] ..... 46138.516795:
> > > > nfs_initiate_write: fileid=00:aa:10056165185 fhandle=0x6bd94d55
> > > > offset=53248 count=1895 stable=UNSTABLE
> > > >  kworker/u1037:2-2871862 [097] ..... 46138.517659:
> > > > nfs4_pnfs_write:
> > > > error=0 (OK) fileid=00:aa:10056165185 fhandle=0x6bd94d55 offset=0
> > > > count=53248 res=53248 stateid=1:0x79a9c471
> > > > layoutstateid=1:0xcbd8aaad
> > > >  kworker/u1037:2-2871862 [097] ..... 46138.517662:
> > > > nfs_writeback_done: error=53248 fileid=00:aa:10056165185
> > > > fhandle=0x6bd94d55 offset=0 count=53248 res=53248 stable=UNSTABLE
> > > > verifier=5199cdae2816c899
> > > >  kworker/u1037:5-2593935 [226] ..... 46138.517669:
> > > > nfs4_pnfs_write:
> > > > error=0 (OK) fileid=00:aa:10056165185 fhandle=0x6bd94d55
> > > > offset=53248
> > > > count=1895 res=1895 stateid=1:0x79a9c471
> > > > layoutstateid=1:0xcbd8aaad
> > > >  kworker/u1037:5-2593935 [226] ..... 46138.517672:
> > > > nfs_writeback_done: error=1895 fileid=00:aa:10056165185
> > > > fhandle=0x6bd94d55 offset=53248 count=1895 res=1895
> > > > stable=UNSTABLE
> > > > verifier=5199cdae2816c899
> > > >  oil-localfs-252-2605046 [162] ..... 46138.518360: nfs_size_grow:
> > > > fileid=00:aa:10056165185 fhandle=0x8bfc64c9
> > > > version=1753485366409158129 cursize=55143 newsize=57344
> > > >  oil-localfs-252-2605046 [162] ..... 46138.518556: nfs_size_grow:
> > > > fileid=00:aa:10056165185 fhandle=0x8bfc64c9
> > > > version=1753485366409158129 cursize=57344 newsize=60156
> > > > 
> > > > ...and just after writeback completes, we see the file size grow
> > > > from
> > > > 55143 to the end of the page (57344).
> > > > 
> > > > The second trace has similar symptoms. There is a lot more
> > > > (smaller)
> > > > write activity (due to memory pressure?). There is a gap of 3791
> > > > bytes
> > > > between these on-the-wire writes, however:
> > > > 
> > > >  kworker/u1036:0-2339252 [217] ..... 479572.054622:
> > > > nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232
> > > > offset=221184 count=4401 stable=UNSTABLE
> > > >  kworker/u1030:1-2297876 [042] ..... 479572.074194:
> > > > nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232
> > > > offset=229376 count=261898 stable=UNSTABLE
> > > > 
> > > > Same situation -- the at page at offset 53248 has 305 bytes on
> > > > it,
> > > > and
> > > > the remaining is zeroed. This trace shows similar racing write()
> > > > and
> > > > writeback activity as in Friday's trace. At around the same time
> > > > as
> > > > the
> > > > client was growing the file over the affected range, writeback
> > > > was
> > > > kicking off for everything up to the affected range (this has
> > > > some
> > > > other wb related calls filtered for brevity):
> > > > 
> > > >   oil-localfs-86-727850  [215] ..... 479572.053987:
> > > > nfs_size_grow:
> > > > fileid=00:96:10067193438 fhandle=0x14c40498
> > > > version=1753823598774309300 cursize=217088
> > > > newsize=220572                                                   
> > > >     
> > > >               
> > > >  kworker/u1036:8-2339326 [088] ..... 479572.054008:
> > > > nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232
> > > > offset=217088 count=3484
> > > > stable=UNSTABLE                                                  
> > > >     
> > > >                            
> > > >   oil-localfs-86-727850  [215] ..... 479572.054405:
> > > > nfs_size_grow:
> > > > fileid=00:96:10067193438 fhandle=0x14c40498
> > > > version=1753823598774309300 cursize=220572
> > > > newsize=221184                                                   
> > > >     
> > > >               
> > > >  kworker/u1036:1-2297875 [217] ..... 479572.054418:
> > > > nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232
> > > > offset=220572 count=612
> > > > stable=UNSTABLE                                                  
> > > >     
> > > >                             
> > > >   oil-localfs-86-727850  [215] ..... 479572.054581:
> > > > nfs_size_grow:
> > > > fileid=00:96:10067193438 fhandle=0x14c40498
> > > > version=1753823598774309300 cursize=221184
> > > > newsize=225280                                                   
> > > >     
> > > >               
> > > >   oil-localfs-86-727850  [215] ..... 479572.054584:
> > > > nfs_size_grow:
> > > > fileid=00:96:10067193438 fhandle=0x14c40498
> > > > version=1753823598774309300 cursize=225280
> > > > newsize=225585                                                   
> > > >     
> > > >               
> > > >  kworker/u1036:0-2339252 [217] ..... 479572.054622:
> > > > nfs_initiate_write: fileid=00:96:10067193438 fhandle=0xc9992232
> > > > offset=221184 count=4401
> > > > stable=UNSTABLE                                                  
> > > >     
> > > >                            
> > > >   oil-localfs-86-727850  [215] ..... 479572.054997:
> > > > nfs_size_grow:
> > > > fileid=00:96:10067193438 fhandle=0x14c40498
> > > > version=1753823598774309300 cursize=225585
> > > > newsize=229376                                                   
> > > >     
> > > >               
> > > >   oil-localfs-86-727850  [215] ..... 479572.055190:
> > > > nfs_size_grow:
> > > > fileid=00:96:10067193438 fhandle=0x14c40498
> > > > version=1753823598774309300 cursize=229376
> > > > newsize=230598                                                   
> > > >     
> > > >               
> > > > 
> > > > Could this be a race between extending an existing dirty page,
> > > > and
> > > > writeback kicking off for the pre-extension range on the page?
> > > > Maybe
> > > > the client is clearing the dirty bit, thinking that the write
> > > > covers
> > > > the dirty range, but it has an outdated idea about what that
> > > > range is
> > > > or doesn't properly check?
> > > > 
> > > > Traces for both events, filtered on the relevant fileid are
> > > > attached.
> > > > I've rolled patches for some new tracepoints that I'm going to
> > > > attempt
> > > > to turn up next, but I thought that this was a good point to
> > > > solicit
> > > > ideas.
> > > > 
> > > > Happy to entertain other thoughts or patches!
> > > 
> > > So... The fact that we are seeing a nfs_size_grow() for the hole at
> > > offset 55143 means that either an existing request was updated, or
> > > a
> > > new one was created in order to cover that hole, and it must have
> > > been
> > > marked as dirty.
> > > 
> > > I'm not seeing anything in the NFS code that can lose that request
> > > without triggering either the nfs_write_error tracepoint, the
> > > nfs_commit_error tracepoint, the nfs_invalidate_folio tracepoint or
> > > else completing the write.
> > > 
> > > The only other way I can see this data being lost is if something
> > > is
> > > corrupting folio->private, or if the page cache is somehow managing
> > > to
> > > throw away a dirty folio.
> > > Of the two, there was for a while a netfs bug which would corrupt
> > > folio->private, but I assume you're not using cachefs?
> > 
> > After staring at this code a lot, I have a theory. But, it seems like
> > we'd be seeing this a lot more if it were correct, so I must be
> > overlooking something.
> > 
> > Here's the scenario:
> > 
> > --------------8<--------------
> > 
> > Userland has written some of a file and the last folio is not full.
> > 
> > Writeback has kicked off for the inode and is successful.
> > nfs_write_completion() calls nfs_page_end_writeback(). That will
> > unlock
> > the nfs_page (clear PG_BUSY) and leave it attached to the folio, and
> > on
> > the commit list.
> > 
> > Next a write from userland comes in to extend the file to the end of
> > the page (and beyond). nfs_try_to_update_request() merges the write
> > into the original request and re-marks the page dirty.
> > 
> > Later the commit runs successfully and the write verifier matches.
> > nfs_commit_release_pages() runs and nfs_inode_remove_request() is
> > called which detaches the nfs_page from the folio.
> > 
> > Eventually, writeback starts up again and the folio is picked up and
> > submitted by nfs_writepages(), but folio->private is now NULL, and
> > it's
> > ignored.
> > 
> > But...like I said I feel like we'd hit this all the time if it were
> > possible, even though I don't see what prevents it. If this is a
> > possibility, then the patch may be as simple as something like this?
> > 
> > diff --git a/fs/nfs/write.c b/fs/nfs/write.c
> > index bb0e78644ffb..72402208fa33 100644
> > --- a/fs/nfs/write.c
> > +++ b/fs/nfs/write.c
> > @@ -1867,7 +1867,7 @@ static void nfs_commit_release_pages(struct
> > nfs_commit_data *data)
> >                  * returned by the server against all stored verfs.
> > */
> >                 if (nfs_write_match_verf(verf, req)) {
> >                         /* We have a match */
> > -                       if (folio)
> > +                       if (folio && !folio_test_dirty(folio))
> >                                 nfs_inode_remove_request(req);
> >                         dprintk_cont(" OK\n");
> >                         goto next;
> 
> The call to nfs_clear_request_commit() in nfs_join_page_group() should
> be taking care of removing the page before a COMMIT is sent.
> 
> During both the writeback and the commit, the nfs_page is locked, so
> won't be available to be updated by nfs_try_to_update_request().

I finally caught something concrete today. I had the attached bpftrace
script running while running the reproducer on a dozen or so machines,
and it detected a hole in some data being written:

-------------8<---------------
Attached 2 probes
Missing nfs_page: ino=10122173116 idx=2 flags=0x15ffff0000000029
Hole: ino=10122173116 idx=3 off=10026 size=2262
Prev folio: idx=2 flags=0x15ffff0000000028 pgbase=0 bytes=4096 req=0 prevreq=0xffff8955b2f55980
-------------8<---------------

What this tells us is that the page at idx=2 got submitted to
nfs_do_writepage() (so it was marked dirty in the pagecache), but when
it got there, folio->private was NULL and it was ignored.

The kernel in this case is based on v6.9, so it's (just) pre-large-
folio support. It has a fair number of NFS patches, but not much to
this portion of the code. Most of them are are containerization fixes.

I'm looking askance at nfs_inode_remove_request(). It does this:

        if (nfs_page_group_sync_on_bit(req, PG_REMOVE)) {
                struct folio *folio = nfs_page_to_folio(req->wb_head);
                struct address_space *mapping = folio->mapping;

                spin_lock(&mapping->i_private_lock);
                if (likely(folio)) {
                        folio->private = NULL;
                        folio_clear_private(folio);
                        clear_bit(PG_MAPPED, &req->wb_head->wb_flags);
                }
                spin_unlock(&mapping->i_private_lock);
        }

If nfs_page_group_sync_on_bit() returns true, then the nfs_page gets
detached from the folio. Meanwhile, if a new write request comes in
just after that, nfs_lock_and_join_requests() will call
nfs_cancel_remove_inode() to try to "cancel" PG_REMOVE:

static int
nfs_cancel_remove_inode(struct nfs_page *req, struct inode *inode)
{
        int ret;

        if (!test_bit(PG_REMOVE, &req->wb_flags))
                return 0;
        ret = nfs_page_group_lock(req);
        if (ret)
                return ret;
        if (test_and_clear_bit(PG_REMOVE, &req->wb_flags))
                nfs_page_set_inode_ref(req, inode);
        nfs_page_group_unlock(req);                          
        return 0;                                    
}                     

...but that does not reattach the nfs_page to the folio. Should it?
                         
-- 
Jeff Layton <jlayton@kernel.org>

[-- Attachment #2: nfs_write_hole.bt --]
[-- Type: text/x-csrc, Size: 2262 bytes --]

#include <linux/nfs_fs.h>
#include <linux/nfs4.h>
#include <linux/nfs_page.h>

/*
 * This does generate some false positives if writeback goes out of ascending order. That
 * shouldn't happen much. Mostly we're interested in unaligned gaps.
 */

config = {
        print_maps_on_exit = false;
        max_map_keys = 16384;
}

fentry:nfs:nfs_do_writepage {
        $folio = args.folio;
        $ino = $folio->mapping->host->i_ino;
        $idx = $folio->index;
        $req = (struct nfs_page *)$folio->private;
        if ($req == 0) {
                printf("Missing nfs_page: ino=%llu idx=%llu flags=0x%lx\n", $ino, $idx, $folio->page.flags);
                return;
        }
        $start = ($idx * 4096) + $req->wb_pgbase;
        $end = $start + $req->wb_bytes;
        $gapsize = $start - @lastend[$ino];
        if (($idx == @lastidx[$ino] + 1) && ($gapsize % 4096)) {
                printf("Hole: ino=%llu idx=%lu off=%llu size=%llu\n", $ino, $idx, @lastend[$ino], $gapsize);
                $prevfolio = @lastfolio[$ino];
                $prevflags = $prevfolio->page.flags;
                $prevreq = @lastreq[$ino];
                $pgbase = (uint32)0;
                $bytes = (uint32)0;
                if ($prevreq) {
                        $pgbase = $prevreq->wb_pgbase;
                        $bytes = $prevreq->wb_bytes;
                }
                printf("Prev folio: idx=%lu flags=0x%lx pgbase=%llu bytes=%llu req=%p prevreq=%p\n", @lastidx[$ino], $prevflags, $pgbase, $bytes, $prevfolio->private, $prevreq);
        }
        @lastidx[$ino] = $idx;
        @lastfolio[$ino] = $folio;
        @lastreq[$ino] = $req;
        @lastend[$ino] = $end;
}

fentry:nfs:nfs_file_release {
        $inode = args.inode;
        $file = args.filp;
        $ino = $inode->i_ino;

        // FMODE_WRITE == 0x2
        if (!($file->f_mode & 0x2)) {
                return;
        }

        if (has_key(@lastidx, $ino)) {
                delete(@lastidx, $ino);
        }
        if (has_key(@lastfolio, $ino)) {
                delete(@lastfolio, $ino);
        }
        if (has_key(@lastreq, $ino)) {
                delete(@lastreq, $ino);
        }
        if (has_key(@lastend, $ino)) {
                delete(@lastend, $ino);
        }
}

Re: parts of pages on NFS being replaced by swaths of NULs

[Trond Myklebust]

On Sat, 2025-08-16 at 09:01 -0400, Jeff Layton wrote:
> 
> I finally caught something concrete today. I had the attached
> bpftrace
> script running while running the reproducer on a dozen or so
> machines,
> and it detected a hole in some data being written:
> 
> -------------8<---------------
> Attached 2 probes
> Missing nfs_page: ino=10122173116 idx=2 flags=0x15ffff0000000029
> Hole: ino=10122173116 idx=3 off=10026 size=2262
> Prev folio: idx=2 flags=0x15ffff0000000028 pgbase=0 bytes=4096 req=0
> prevreq=0xffff8955b2f55980
> -------------8<---------------
> 
> What this tells us is that the page at idx=2 got submitted to
> nfs_do_writepage() (so it was marked dirty in the pagecache), but
> when
> it got there, folio->private was NULL and it was ignored.
> 
> The kernel in this case is based on v6.9, so it's (just) pre-large-
> folio support. It has a fair number of NFS patches, but not much to
> this portion of the code. Most of them are are containerization
> fixes.
> 
> I'm looking askance at nfs_inode_remove_request(). It does this:
> 
>         if (nfs_page_group_sync_on_bit(req, PG_REMOVE)) {
>                 struct folio *folio = nfs_page_to_folio(req-
> >wb_head);
>                 struct address_space *mapping = folio->mapping;
> 
>                 spin_lock(&mapping->i_private_lock);
>                 if (likely(folio)) {
>                         folio->private = NULL;
>                         folio_clear_private(folio);
>                         clear_bit(PG_MAPPED, &req->wb_head-
> >wb_flags);
>                 }
>                 spin_unlock(&mapping->i_private_lock);
>         }
> 
> If nfs_page_group_sync_on_bit() returns true, then the nfs_page gets
> detached from the folio. Meanwhile, if a new write request comes in
> just after that, nfs_lock_and_join_requests() will call
> nfs_cancel_remove_inode() to try to "cancel" PG_REMOVE:
> 
> static int
> nfs_cancel_remove_inode(struct nfs_page *req, struct inode *inode)
> {
>         int ret;
> 
>         if (!test_bit(PG_REMOVE, &req->wb_flags))
>                 return 0;
>         ret = nfs_page_group_lock(req);
>         if (ret)
>                 return ret;
>         if (test_and_clear_bit(PG_REMOVE, &req->wb_flags))
>                 nfs_page_set_inode_ref(req, inode);
>         nfs_page_group_unlock(req);                          
>         return 0;                                    
> }                     
> 
> ...but that does not reattach the nfs_page to the folio. Should it?
>                         

That's not sufficient AFAICS. Does the following patch work?

8<------------------------------------------------------------
From fc9690dda01f001c6cd11665701394da8ebba1ab Mon Sep 17 00:00:00 2001
Message-ID: <fc9690dda01f001c6cd11665701394da8ebba1ab.1755355810.git.trond.myklebust@hammerspace.com>
From: Trond Myklebust <trond.myklebust@hammerspace.com>
Date: Sat, 16 Aug 2025 07:25:20 -0700
Subject: [PATCH] NFS: Fix a race when updating an existing write

After nfs_lock_and_join_requests() tests for whether the request is
still attached to the mapping, nothing prevents a call to
nfs_inode_remove_request() from succeeding until we actually lock the
page group.
The reason is that whoever called nfs_inode_remove_request() doesn't
necessarily have a lock on the page group head.

So in order to avoid races, let's take the page group lock earlier in
nfs_lock_and_join_requests(), and hold it across the removal of the
request in nfs_inode_remove_request().

Reported-by: Jeff Layton <jlayton@kernel.org>
Fixes: c3f2235782c3 ("nfs: fold nfs_folio_find_and_lock_request into nfs_lock_and_join_requests")
Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
---
 fs/nfs/pagelist.c        |  9 +++++----
 fs/nfs/write.c           | 29 ++++++++++-------------------
 include/linux/nfs_page.h |  1 +
 3 files changed, 16 insertions(+), 23 deletions(-)

diff --git a/fs/nfs/pagelist.c b/fs/nfs/pagelist.c
index 11968dcb7243..6e69ce43a13f 100644
--- a/fs/nfs/pagelist.c
+++ b/fs/nfs/pagelist.c
@@ -253,13 +253,14 @@ nfs_page_group_unlock(struct nfs_page *req)
 	nfs_page_clear_headlock(req);
 }
 
-/*
- * nfs_page_group_sync_on_bit_locked
+/**
+ * nfs_page_group_sync_on_bit_locked - Test if all requests have @bit set
+ * @req: request in page group
+ * @bit: PG_* bit that is used to sync page group
  *
  * must be called with page group lock held
  */
-static bool
-nfs_page_group_sync_on_bit_locked(struct nfs_page *req, unsigned int bit)
+bool nfs_page_group_sync_on_bit_locked(struct nfs_page *req, unsigned int bit)
 {
 	struct nfs_page *head = req->wb_head;
 	struct nfs_page *tmp;
diff --git a/fs/nfs/write.c b/fs/nfs/write.c
index fa5c41d0989a..8b7c04737967 100644
--- a/fs/nfs/write.c
+++ b/fs/nfs/write.c
@@ -153,20 +153,10 @@ nfs_page_set_inode_ref(struct nfs_page *req, struct inode *inode)
 	}
 }
 
-static int
-nfs_cancel_remove_inode(struct nfs_page *req, struct inode *inode)
+static void nfs_cancel_remove_inode(struct nfs_page *req, struct inode *inode)
 {
-	int ret;
-
-	if (!test_bit(PG_REMOVE, &req->wb_flags))
-		return 0;
-	ret = nfs_page_group_lock(req);
-	if (ret)
-		return ret;
 	if (test_and_clear_bit(PG_REMOVE, &req->wb_flags))
 		nfs_page_set_inode_ref(req, inode);
-	nfs_page_group_unlock(req);
-	return 0;
 }
 
 /**
@@ -585,19 +575,18 @@ static struct nfs_page *nfs_lock_and_join_requests(struct folio *folio)
 		}
 	}
 
+	ret = nfs_page_group_lock(head);
+	if (ret < 0)
+		goto out_unlock;
+
 	/* Ensure that nobody removed the request before we locked it */
 	if (head != folio->private) {
+		nfs_page_group_unlock(head);
 		nfs_unlock_and_release_request(head);
 		goto retry;
 	}
 
-	ret = nfs_cancel_remove_inode(head, inode);
-	if (ret < 0)
-		goto out_unlock;
-
-	ret = nfs_page_group_lock(head);
-	if (ret < 0)
-		goto out_unlock;
+	nfs_cancel_remove_inode(head, inode);
 
 	/* lock each request in the page group */
 	for (subreq = head->wb_this_page;
@@ -786,7 +775,8 @@ static void nfs_inode_remove_request(struct nfs_page *req)
 {
 	struct nfs_inode *nfsi = NFS_I(nfs_page_to_inode(req));
 
-	if (nfs_page_group_sync_on_bit(req, PG_REMOVE)) {
+	nfs_page_group_lock(req);
+	if (nfs_page_group_sync_on_bit_locked(req, PG_REMOVE)) {
 		struct folio *folio = nfs_page_to_folio(req->wb_head);
 		struct address_space *mapping = folio->mapping;
 
@@ -798,6 +788,7 @@ static void nfs_inode_remove_request(struct nfs_page *req)
 		}
 		spin_unlock(&mapping->i_private_lock);
 	}
+	nfs_page_group_unlock(req);
 
 	if (test_and_clear_bit(PG_INODE_REF, &req->wb_flags)) {
 		atomic_long_dec(&nfsi->nrequests);
diff --git a/include/linux/nfs_page.h b/include/linux/nfs_page.h
index 169b4ae30ff4..9aed39abc94b 100644
--- a/include/linux/nfs_page.h
+++ b/include/linux/nfs_page.h
@@ -160,6 +160,7 @@ extern void nfs_join_page_group(struct nfs_page *head,
 extern int nfs_page_group_lock(struct nfs_page *);
 extern void nfs_page_group_unlock(struct nfs_page *);
 extern bool nfs_page_group_sync_on_bit(struct nfs_page *, unsigned int);
+extern bool nfs_page_group_sync_on_bit_locked(struct nfs_page *, unsigned int);
 extern	int nfs_page_set_headlock(struct nfs_page *req);
 extern void nfs_page_clear_headlock(struct nfs_page *req);
 extern bool nfs_async_iocounter_wait(struct rpc_task *, struct nfs_lock_context *);
-- 
2.50.1

Re: parts of pages on NFS being replaced by swaths of NULs

[Jeff Layton]

On Sat, 2025-08-16 at 07:51 -0700, Trond Myklebust wrote:
> On Sat, 2025-08-16 at 09:01 -0400, Jeff Layton wrote:
> > 
> > I finally caught something concrete today. I had the attached
> > bpftrace
> > script running while running the reproducer on a dozen or so
> > machines,
> > and it detected a hole in some data being written:
> > 
> > -------------8<---------------
> > Attached 2 probes
> > Missing nfs_page: ino=10122173116 idx=2 flags=0x15ffff0000000029
> > Hole: ino=10122173116 idx=3 off=10026 size=2262
> > Prev folio: idx=2 flags=0x15ffff0000000028 pgbase=0 bytes=4096 req=0
> > prevreq=0xffff8955b2f55980
> > -------------8<---------------
> > 
> > What this tells us is that the page at idx=2 got submitted to
> > nfs_do_writepage() (so it was marked dirty in the pagecache), but
> > when
> > it got there, folio->private was NULL and it was ignored.
> > 
> > The kernel in this case is based on v6.9, so it's (just) pre-large-
> > folio support. It has a fair number of NFS patches, but not much to
> > this portion of the code. Most of them are are containerization
> > fixes.
> > 
> > I'm looking askance at nfs_inode_remove_request(). It does this:
> > 
> >         if (nfs_page_group_sync_on_bit(req, PG_REMOVE)) {
> >                 struct folio *folio = nfs_page_to_folio(req-
> > > wb_head);
> >                 struct address_space *mapping = folio->mapping;
> > 
> >                 spin_lock(&mapping->i_private_lock);
> >                 if (likely(folio)) {
> >                         folio->private = NULL;
> >                         folio_clear_private(folio);
> >                         clear_bit(PG_MAPPED, &req->wb_head-
> > > wb_flags);
> >                 }
> >                 spin_unlock(&mapping->i_private_lock);
> >         }
> > 
> > If nfs_page_group_sync_on_bit() returns true, then the nfs_page gets
> > detached from the folio. Meanwhile, if a new write request comes in
> > just after that, nfs_lock_and_join_requests() will call
> > nfs_cancel_remove_inode() to try to "cancel" PG_REMOVE:
> > 
> > static int
> > nfs_cancel_remove_inode(struct nfs_page *req, struct inode *inode)
> > {
> >         int ret;
> > 
> >         if (!test_bit(PG_REMOVE, &req->wb_flags))
> >                 return 0;
> >         ret = nfs_page_group_lock(req);
> >         if (ret)
> >                 return ret;
> >         if (test_and_clear_bit(PG_REMOVE, &req->wb_flags))
> >                 nfs_page_set_inode_ref(req, inode);
> >         nfs_page_group_unlock(req);                          
> >         return 0;                                    
> > }                     
> > 
> > ...but that does not reattach the nfs_page to the folio. Should it?
> >                         
> 
> That's not sufficient AFAICS. Does the following patch work?
> 

The patch looks good to me and it seems like it should close the race.
I'll have to look at backporting it to v6.9, and we'll have to build
images, etc. Hopefully we will have some results in a week or so. I'll
keep you posted!

Thanks,

> 8<------------------------------------------------------------
> From fc9690dda01f001c6cd11665701394da8ebba1ab Mon Sep 17 00:00:00 2001
> Message-ID: <fc9690dda01f001c6cd11665701394da8ebba1ab.1755355810.git.trond.myklebust@hammerspace.com>
> From: Trond Myklebust <trond.myklebust@hammerspace.com>
> Date: Sat, 16 Aug 2025 07:25:20 -0700
> Subject: [PATCH] NFS: Fix a race when updating an existing write
> 
> After nfs_lock_and_join_requests() tests for whether the request is
> still attached to the mapping, nothing prevents a call to
> nfs_inode_remove_request() from succeeding until we actually lock the
> page group.
> The reason is that whoever called nfs_inode_remove_request() doesn't
> necessarily have a lock on the page group head.
> 
> So in order to avoid races, let's take the page group lock earlier in
> nfs_lock_and_join_requests(), and hold it across the removal of the
> request in nfs_inode_remove_request().
> 
> Reported-by: Jeff Layton <jlayton@kernel.org>
> Fixes: c3f2235782c3 ("nfs: fold nfs_folio_find_and_lock_request into nfs_lock_and_join_requests")
> Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
> ---
>  fs/nfs/pagelist.c        |  9 +++++----
>  fs/nfs/write.c           | 29 ++++++++++-------------------
>  include/linux/nfs_page.h |  1 +
>  3 files changed, 16 insertions(+), 23 deletions(-)
> 
> diff --git a/fs/nfs/pagelist.c b/fs/nfs/pagelist.c
> index 11968dcb7243..6e69ce43a13f 100644
> --- a/fs/nfs/pagelist.c
> +++ b/fs/nfs/pagelist.c
> @@ -253,13 +253,14 @@ nfs_page_group_unlock(struct nfs_page *req)
>  	nfs_page_clear_headlock(req);
>  }
>  
> -/*
> - * nfs_page_group_sync_on_bit_locked
> +/**
> + * nfs_page_group_sync_on_bit_locked - Test if all requests have @bit set
> + * @req: request in page group
> + * @bit: PG_* bit that is used to sync page group
>   *
>   * must be called with page group lock held
>   */
> -static bool
> -nfs_page_group_sync_on_bit_locked(struct nfs_page *req, unsigned int bit)
> +bool nfs_page_group_sync_on_bit_locked(struct nfs_page *req, unsigned int bit)
>  {
>  	struct nfs_page *head = req->wb_head;
>  	struct nfs_page *tmp;
> diff --git a/fs/nfs/write.c b/fs/nfs/write.c
> index fa5c41d0989a..8b7c04737967 100644
> --- a/fs/nfs/write.c
> +++ b/fs/nfs/write.c
> @@ -153,20 +153,10 @@ nfs_page_set_inode_ref(struct nfs_page *req, struct inode *inode)
>  	}
>  }
>  
> -static int
> -nfs_cancel_remove_inode(struct nfs_page *req, struct inode *inode)
> +static void nfs_cancel_remove_inode(struct nfs_page *req, struct inode *inode)
>  {
> -	int ret;
> -
> -	if (!test_bit(PG_REMOVE, &req->wb_flags))
> -		return 0;
> -	ret = nfs_page_group_lock(req);
> -	if (ret)
> -		return ret;
>  	if (test_and_clear_bit(PG_REMOVE, &req->wb_flags))
>  		nfs_page_set_inode_ref(req, inode);
> -	nfs_page_group_unlock(req);
> -	return 0;
>  }
>  
>  /**
> @@ -585,19 +575,18 @@ static struct nfs_page *nfs_lock_and_join_requests(struct folio *folio)
>  		}
>  	}
>  
> +	ret = nfs_page_group_lock(head);
> +	if (ret < 0)
> +		goto out_unlock;
> +
>  	/* Ensure that nobody removed the request before we locked it */
>  	if (head != folio->private) {
> +		nfs_page_group_unlock(head);
>  		nfs_unlock_and_release_request(head);
>  		goto retry;
>  	}
>  
> -	ret = nfs_cancel_remove_inode(head, inode);
> -	if (ret < 0)
> -		goto out_unlock;
> -
> -	ret = nfs_page_group_lock(head);
> -	if (ret < 0)
> -		goto out_unlock;
> +	nfs_cancel_remove_inode(head, inode);
>  
>  	/* lock each request in the page group */
>  	for (subreq = head->wb_this_page;
> @@ -786,7 +775,8 @@ static void nfs_inode_remove_request(struct nfs_page *req)
>  {
>  	struct nfs_inode *nfsi = NFS_I(nfs_page_to_inode(req));
>  
> -	if (nfs_page_group_sync_on_bit(req, PG_REMOVE)) {
> +	nfs_page_group_lock(req);
> +	if (nfs_page_group_sync_on_bit_locked(req, PG_REMOVE)) {
>  		struct folio *folio = nfs_page_to_folio(req->wb_head);
>  		struct address_space *mapping = folio->mapping;
>  
> @@ -798,6 +788,7 @@ static void nfs_inode_remove_request(struct nfs_page *req)
>  		}
>  		spin_unlock(&mapping->i_private_lock);
>  	}
> +	nfs_page_group_unlock(req);
>  
>  	if (test_and_clear_bit(PG_INODE_REF, &req->wb_flags)) {
>  		atomic_long_dec(&nfsi->nrequests);
> diff --git a/include/linux/nfs_page.h b/include/linux/nfs_page.h
> index 169b4ae30ff4..9aed39abc94b 100644
> --- a/include/linux/nfs_page.h
> +++ b/include/linux/nfs_page.h
> @@ -160,6 +160,7 @@ extern void nfs_join_page_group(struct nfs_page *head,
>  extern int nfs_page_group_lock(struct nfs_page *);
>  extern void nfs_page_group_unlock(struct nfs_page *);
>  extern bool nfs_page_group_sync_on_bit(struct nfs_page *, unsigned int);
> +extern bool nfs_page_group_sync_on_bit_locked(struct nfs_page *, unsigned int);
>  extern	int nfs_page_set_headlock(struct nfs_page *req);
>  extern void nfs_page_clear_headlock(struct nfs_page *req);
>  extern bool nfs_async_iocounter_wait(struct rpc_task *, struct nfs_lock_context *);

-- 
Jeff Layton <jlayton@kernel.org>

Re: parts of pages on NFS being replaced by swaths of NULs

[Jeff Layton]

On Sat, 2025-08-16 at 07:51 -0700, Trond Myklebust wrote:
> On Sat, 2025-08-16 at 09:01 -0400, Jeff Layton wrote:
> > 
> > I finally caught something concrete today. I had the attached
> > bpftrace
> > script running while running the reproducer on a dozen or so
> > machines,
> > and it detected a hole in some data being written:
> > 
> > -------------8<---------------
> > Attached 2 probes
> > Missing nfs_page: ino=10122173116 idx=2 flags=0x15ffff0000000029
> > Hole: ino=10122173116 idx=3 off=10026 size=2262
> > Prev folio: idx=2 flags=0x15ffff0000000028 pgbase=0 bytes=4096 req=0
> > prevreq=0xffff8955b2f55980
> > -------------8<---------------
> > 
> > What this tells us is that the page at idx=2 got submitted to
> > nfs_do_writepage() (so it was marked dirty in the pagecache), but
> > when
> > it got there, folio->private was NULL and it was ignored.
> > 
> > The kernel in this case is based on v6.9, so it's (just) pre-large-
> > folio support. It has a fair number of NFS patches, but not much to
> > this portion of the code. Most of them are are containerization
> > fixes.
> > 
> > I'm looking askance at nfs_inode_remove_request(). It does this:
> > 
> >         if (nfs_page_group_sync_on_bit(req, PG_REMOVE)) {
> >                 struct folio *folio = nfs_page_to_folio(req-
> > > wb_head);
> >                 struct address_space *mapping = folio->mapping;
> > 
> >                 spin_lock(&mapping->i_private_lock);
> >                 if (likely(folio)) {
> >                         folio->private = NULL;
> >                         folio_clear_private(folio);
> >                         clear_bit(PG_MAPPED, &req->wb_head-
> > > wb_flags);
> >                 }
> >                 spin_unlock(&mapping->i_private_lock);
> >         }
> > 
> > If nfs_page_group_sync_on_bit() returns true, then the nfs_page gets
> > detached from the folio. Meanwhile, if a new write request comes in
> > just after that, nfs_lock_and_join_requests() will call
> > nfs_cancel_remove_inode() to try to "cancel" PG_REMOVE:
> > 
> > static int
> > nfs_cancel_remove_inode(struct nfs_page *req, struct inode *inode)
> > {
> >         int ret;
> > 
> >         if (!test_bit(PG_REMOVE, &req->wb_flags))
> >                 return 0;
> >         ret = nfs_page_group_lock(req);
> >         if (ret)
> >                 return ret;
> >         if (test_and_clear_bit(PG_REMOVE, &req->wb_flags))
> >                 nfs_page_set_inode_ref(req, inode);
> >         nfs_page_group_unlock(req);                          
> >         return 0;                                    
> > }                     
> > 
> > ...but that does not reattach the nfs_page to the folio. Should it?
> >                         
> 
> That's not sufficient AFAICS. Does the following patch work?
> 
> 8<------------------------------------------------------------
> From fc9690dda01f001c6cd11665701394da8ebba1ab Mon Sep 17 00:00:00 2001
> Message-ID: <fc9690dda01f001c6cd11665701394da8ebba1ab.1755355810.git.trond.myklebust@hammerspace.com>
> From: Trond Myklebust <trond.myklebust@hammerspace.com>
> Date: Sat, 16 Aug 2025 07:25:20 -0700
> Subject: [PATCH] NFS: Fix a race when updating an existing write
> 
> After nfs_lock_and_join_requests() tests for whether the request is
> still attached to the mapping, nothing prevents a call to
> nfs_inode_remove_request() from succeeding until we actually lock the
> page group.
> The reason is that whoever called nfs_inode_remove_request() doesn't
> necessarily have a lock on the page group head.
> 
> So in order to avoid races, let's take the page group lock earlier in
> nfs_lock_and_join_requests(), and hold it across the removal of the
> request in nfs_inode_remove_request().
> 
> Reported-by: Jeff Layton <jlayton@kernel.org>
> Fixes: c3f2235782c3 ("nfs: fold nfs_folio_find_and_lock_request into nfs_lock_and_join_requests")

One comment here: The above patch went into v6.11. The kernel we've
been reproducing this on doesn't have it, so I don't think that's the
correct Fixes tag.

Also, I got a second occurrence with the bpftrace script. Looks like
the same problem (just for confirmation):

Missing nfs_page: ino=10123250003 idx=23 flags=0x5ffff0000000009
Hole: ino=10123250003 idx=24 off=95247 size=3057
Prev folio: idx=23 flags=0x5ffff0000000008 pgbase=0 bytes=1956 req=0 prevreq=0xffff8882ab9b6a00


> Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
> ---
>  fs/nfs/pagelist.c        |  9 +++++----
>  fs/nfs/write.c           | 29 ++++++++++-------------------
>  include/linux/nfs_page.h |  1 +
>  3 files changed, 16 insertions(+), 23 deletions(-)
> 
> diff --git a/fs/nfs/pagelist.c b/fs/nfs/pagelist.c
> index 11968dcb7243..6e69ce43a13f 100644
> --- a/fs/nfs/pagelist.c
> +++ b/fs/nfs/pagelist.c
> @@ -253,13 +253,14 @@ nfs_page_group_unlock(struct nfs_page *req)
>  	nfs_page_clear_headlock(req);
>  }
>  
> -/*
> - * nfs_page_group_sync_on_bit_locked
> +/**
> + * nfs_page_group_sync_on_bit_locked - Test if all requests have @bit set
> + * @req: request in page group
> + * @bit: PG_* bit that is used to sync page group
>   *
>   * must be called with page group lock held
>   */
> -static bool
> -nfs_page_group_sync_on_bit_locked(struct nfs_page *req, unsigned int bit)
> +bool nfs_page_group_sync_on_bit_locked(struct nfs_page *req, unsigned int bit)
>  {
>  	struct nfs_page *head = req->wb_head;
>  	struct nfs_page *tmp;
> diff --git a/fs/nfs/write.c b/fs/nfs/write.c
> index fa5c41d0989a..8b7c04737967 100644
> --- a/fs/nfs/write.c
> +++ b/fs/nfs/write.c
> @@ -153,20 +153,10 @@ nfs_page_set_inode_ref(struct nfs_page *req, struct inode *inode)
>  	}
>  }
>  
> -static int
> -nfs_cancel_remove_inode(struct nfs_page *req, struct inode *inode)
> +static void nfs_cancel_remove_inode(struct nfs_page *req, struct inode *inode)
>  {
> -	int ret;
> -
> -	if (!test_bit(PG_REMOVE, &req->wb_flags))
> -		return 0;
> -	ret = nfs_page_group_lock(req);
> -	if (ret)
> -		return ret;
>  	if (test_and_clear_bit(PG_REMOVE, &req->wb_flags))
>  		nfs_page_set_inode_ref(req, inode);
> -	nfs_page_group_unlock(req);
> -	return 0;
>  }
>  
>  /**
> @@ -585,19 +575,18 @@ static struct nfs_page *nfs_lock_and_join_requests(struct folio *folio)
>  		}
>  	}
>  
> +	ret = nfs_page_group_lock(head);
> +	if (ret < 0)
> +		goto out_unlock;
> +
>  	/* Ensure that nobody removed the request before we locked it */
>  	if (head != folio->private) {
> +		nfs_page_group_unlock(head);
>  		nfs_unlock_and_release_request(head);
>  		goto retry;
>  	}
>  
> -	ret = nfs_cancel_remove_inode(head, inode);
> -	if (ret < 0)
> -		goto out_unlock;
> -
> -	ret = nfs_page_group_lock(head);
> -	if (ret < 0)
> -		goto out_unlock;
> +	nfs_cancel_remove_inode(head, inode);
>  
>  	/* lock each request in the page group */
>  	for (subreq = head->wb_this_page;
> @@ -786,7 +775,8 @@ static void nfs_inode_remove_request(struct nfs_page *req)
>  {
>  	struct nfs_inode *nfsi = NFS_I(nfs_page_to_inode(req));
>  
> -	if (nfs_page_group_sync_on_bit(req, PG_REMOVE)) {
> +	nfs_page_group_lock(req);
> +	if (nfs_page_group_sync_on_bit_locked(req, PG_REMOVE)) {
>  		struct folio *folio = nfs_page_to_folio(req->wb_head);
>  		struct address_space *mapping = folio->mapping;
>  
> @@ -798,6 +788,7 @@ static void nfs_inode_remove_request(struct nfs_page *req)
>  		}
>  		spin_unlock(&mapping->i_private_lock);
>  	}
> +	nfs_page_group_unlock(req);
>  
>  	if (test_and_clear_bit(PG_INODE_REF, &req->wb_flags)) {
>  		atomic_long_dec(&nfsi->nrequests);
> diff --git a/include/linux/nfs_page.h b/include/linux/nfs_page.h
> index 169b4ae30ff4..9aed39abc94b 100644
> --- a/include/linux/nfs_page.h
> +++ b/include/linux/nfs_page.h
> @@ -160,6 +160,7 @@ extern void nfs_join_page_group(struct nfs_page *head,
>  extern int nfs_page_group_lock(struct nfs_page *);
>  extern void nfs_page_group_unlock(struct nfs_page *);
>  extern bool nfs_page_group_sync_on_bit(struct nfs_page *, unsigned int);
> +extern bool nfs_page_group_sync_on_bit_locked(struct nfs_page *, unsigned int);
>  extern	int nfs_page_set_headlock(struct nfs_page *req);
>  extern void nfs_page_clear_headlock(struct nfs_page *req);
>  extern bool nfs_async_iocounter_wait(struct rpc_task *, struct nfs_lock_context *);

-- 
Jeff Layton <jlayton@kernel.org>

Re: parts of pages on NFS being replaced by swaths of NULs

[Mike Snitzer]

[-- Attachment #1: Type: text/plain, Size: 8628 bytes --]

On Sat, Aug 16, 2025 at 07:51:17AM -0700, Trond Myklebust wrote:
> On Sat, 2025-08-16 at 09:01 -0400, Jeff Layton wrote:
> > 
> > I finally caught something concrete today. I had the attached
> > bpftrace
> > script running while running the reproducer on a dozen or so
> > machines,
> > and it detected a hole in some data being written:
> > 
> > -------------8<---------------
> > Attached 2 probes
> > Missing nfs_page: ino=10122173116 idx=2 flags=0x15ffff0000000029
> > Hole: ino=10122173116 idx=3 off=10026 size=2262
> > Prev folio: idx=2 flags=0x15ffff0000000028 pgbase=0 bytes=4096 req=0
> > prevreq=0xffff8955b2f55980
> > -------------8<---------------
> > 
> > What this tells us is that the page at idx=2 got submitted to
> > nfs_do_writepage() (so it was marked dirty in the pagecache), but
> > when
> > it got there, folio->private was NULL and it was ignored.
> > 
> > The kernel in this case is based on v6.9, so it's (just) pre-large-
> > folio support. It has a fair number of NFS patches, but not much to
> > this portion of the code. Most of them are are containerization
> > fixes.
> > 
> > I'm looking askance at nfs_inode_remove_request(). It does this:
> > 
> >         if (nfs_page_group_sync_on_bit(req, PG_REMOVE)) {
> >                 struct folio *folio = nfs_page_to_folio(req-
> > >wb_head);
> >                 struct address_space *mapping = folio->mapping;
> > 
> >                 spin_lock(&mapping->i_private_lock);
> >                 if (likely(folio)) {
> >                         folio->private = NULL;
> >                         folio_clear_private(folio);
> >                         clear_bit(PG_MAPPED, &req->wb_head-
> > >wb_flags);
> >                 }
> >                 spin_unlock(&mapping->i_private_lock);
> >         }
> > 
> > If nfs_page_group_sync_on_bit() returns true, then the nfs_page gets
> > detached from the folio. Meanwhile, if a new write request comes in
> > just after that, nfs_lock_and_join_requests() will call
> > nfs_cancel_remove_inode() to try to "cancel" PG_REMOVE:
> > 
> > static int
> > nfs_cancel_remove_inode(struct nfs_page *req, struct inode *inode)
> > {
> >         int ret;
> > 
> >         if (!test_bit(PG_REMOVE, &req->wb_flags))
> >                 return 0;
> >         ret = nfs_page_group_lock(req);
> >         if (ret)
> >                 return ret;
> >         if (test_and_clear_bit(PG_REMOVE, &req->wb_flags))
> >                 nfs_page_set_inode_ref(req, inode);
> >         nfs_page_group_unlock(req);                          
> >         return 0;                                    
> > }                     
> > 
> > ...but that does not reattach the nfs_page to the folio. Should it?
> >                         
> 
> That's not sufficient AFAICS. Does the following patch work?
> 
> 8<------------------------------------------------------------
> From fc9690dda01f001c6cd11665701394da8ebba1ab Mon Sep 17 00:00:00 2001
> Message-ID: <fc9690dda01f001c6cd11665701394da8ebba1ab.1755355810.git.trond.myklebust@hammerspace.com>
> From: Trond Myklebust <trond.myklebust@hammerspace.com>
> Date: Sat, 16 Aug 2025 07:25:20 -0700
> Subject: [PATCH] NFS: Fix a race when updating an existing write
> 
> After nfs_lock_and_join_requests() tests for whether the request is
> still attached to the mapping, nothing prevents a call to
> nfs_inode_remove_request() from succeeding until we actually lock the
> page group.
> The reason is that whoever called nfs_inode_remove_request() doesn't
> necessarily have a lock on the page group head.
> 
> So in order to avoid races, let's take the page group lock earlier in
> nfs_lock_and_join_requests(), and hold it across the removal of the
> request in nfs_inode_remove_request().
> 
> Reported-by: Jeff Layton <jlayton@kernel.org>
> Fixes: c3f2235782c3 ("nfs: fold nfs_folio_find_and_lock_request into nfs_lock_and_join_requests")
> Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
> ---
>  fs/nfs/pagelist.c        |  9 +++++----
>  fs/nfs/write.c           | 29 ++++++++++-------------------
>  include/linux/nfs_page.h |  1 +
>  3 files changed, 16 insertions(+), 23 deletions(-)
> 
> diff --git a/fs/nfs/pagelist.c b/fs/nfs/pagelist.c
> index 11968dcb7243..6e69ce43a13f 100644
> --- a/fs/nfs/pagelist.c
> +++ b/fs/nfs/pagelist.c
> @@ -253,13 +253,14 @@ nfs_page_group_unlock(struct nfs_page *req)
>  	nfs_page_clear_headlock(req);
>  }
>  
> -/*
> - * nfs_page_group_sync_on_bit_locked
> +/**
> + * nfs_page_group_sync_on_bit_locked - Test if all requests have @bit set
> + * @req: request in page group
> + * @bit: PG_* bit that is used to sync page group
>   *
>   * must be called with page group lock held
>   */
> -static bool
> -nfs_page_group_sync_on_bit_locked(struct nfs_page *req, unsigned int bit)
> +bool nfs_page_group_sync_on_bit_locked(struct nfs_page *req, unsigned int bit)
>  {
>  	struct nfs_page *head = req->wb_head;
>  	struct nfs_page *tmp;
> diff --git a/fs/nfs/write.c b/fs/nfs/write.c
> index fa5c41d0989a..8b7c04737967 100644
> --- a/fs/nfs/write.c
> +++ b/fs/nfs/write.c
> @@ -153,20 +153,10 @@ nfs_page_set_inode_ref(struct nfs_page *req, struct inode *inode)
>  	}
>  }
>  
> -static int
> -nfs_cancel_remove_inode(struct nfs_page *req, struct inode *inode)
> +static void nfs_cancel_remove_inode(struct nfs_page *req, struct inode *inode)
>  {
> -	int ret;
> -
> -	if (!test_bit(PG_REMOVE, &req->wb_flags))
> -		return 0;
> -	ret = nfs_page_group_lock(req);
> -	if (ret)
> -		return ret;
>  	if (test_and_clear_bit(PG_REMOVE, &req->wb_flags))
>  		nfs_page_set_inode_ref(req, inode);
> -	nfs_page_group_unlock(req);
> -	return 0;
>  }
>  
>  /**
> @@ -585,19 +575,18 @@ static struct nfs_page *nfs_lock_and_join_requests(struct folio *folio)
>  		}
>  	}
>  
> +	ret = nfs_page_group_lock(head);
> +	if (ret < 0)
> +		goto out_unlock;
> +
>  	/* Ensure that nobody removed the request before we locked it */
>  	if (head != folio->private) {
> +		nfs_page_group_unlock(head);
>  		nfs_unlock_and_release_request(head);
>  		goto retry;
>  	}
>  
> -	ret = nfs_cancel_remove_inode(head, inode);
> -	if (ret < 0)
> -		goto out_unlock;
> -
> -	ret = nfs_page_group_lock(head);
> -	if (ret < 0)
> -		goto out_unlock;
> +	nfs_cancel_remove_inode(head, inode);
>  
>  	/* lock each request in the page group */
>  	for (subreq = head->wb_this_page;
> @@ -786,7 +775,8 @@ static void nfs_inode_remove_request(struct nfs_page *req)
>  {
>  	struct nfs_inode *nfsi = NFS_I(nfs_page_to_inode(req));
>  
> -	if (nfs_page_group_sync_on_bit(req, PG_REMOVE)) {
> +	nfs_page_group_lock(req);
> +	if (nfs_page_group_sync_on_bit_locked(req, PG_REMOVE)) {
>  		struct folio *folio = nfs_page_to_folio(req->wb_head);
>  		struct address_space *mapping = folio->mapping;
>  
> @@ -798,6 +788,7 @@ static void nfs_inode_remove_request(struct nfs_page *req)
>  		}
>  		spin_unlock(&mapping->i_private_lock);
>  	}
> +	nfs_page_group_unlock(req);
>  
>  	if (test_and_clear_bit(PG_INODE_REF, &req->wb_flags)) {
>  		atomic_long_dec(&nfsi->nrequests);
> diff --git a/include/linux/nfs_page.h b/include/linux/nfs_page.h
> index 169b4ae30ff4..9aed39abc94b 100644
> --- a/include/linux/nfs_page.h
> +++ b/include/linux/nfs_page.h
> @@ -160,6 +160,7 @@ extern void nfs_join_page_group(struct nfs_page *head,
>  extern int nfs_page_group_lock(struct nfs_page *);
>  extern void nfs_page_group_unlock(struct nfs_page *);
>  extern bool nfs_page_group_sync_on_bit(struct nfs_page *, unsigned int);
> +extern bool nfs_page_group_sync_on_bit_locked(struct nfs_page *, unsigned int);
>  extern	int nfs_page_set_headlock(struct nfs_page *req);
>  extern void nfs_page_clear_headlock(struct nfs_page *req);
>  extern bool nfs_async_iocounter_wait(struct rpc_task *, struct nfs_lock_context *);
> -- 
> 2.50.1
> 
> 


Trond,

You're the best! ;)

Your patch fixes corruption I've been chasing for the past week
relative to NFS DIRECT, specifically with:
echo Y > /sys/module/nfs/parameters/localio_O_DIRECT_align_misaligned_IO

So you need my latest NFS DIRECT patchset:
https://lore.kernel.org/linux-nfs/20250815233003.55071-1-snitzer@kernel.org/

With it, writes would be corrupted when using the attached reproducer
(from Jon Flynn, with the assistance of ChatGPT) that pulls out the
subset of MLperf unet3d test (when ran in buffered IO mode, so
entirely misaligned relative to DIO-alignment requirements) that we've
seen npz CRC compare failure with.

I tested my patchset with your patch applied and it all "just works".

Ship it all!

Thanks,
Mike

ps. running the attached reproducer is as simple as:
./mlperf_npz_tool.py --npz-path /mnt/share1/sample_a.npz

[-- Attachment #2: mlperf_npz_tool.py --]
[-- Type: text/plain, Size: 7770 bytes --]

#!/usr/bin/env python3
import argparse, math, os, struct, zlib
from pathlib import Path
import numpy as np
import zipfile

# -----------------------
# Defaults (from your YAML)
# -----------------------
DEFAULT_MEAN_BYTES  = 146_600_628
DEFAULT_STDEV_BYTES = 68_341_808
DEFAULT_RESIZE      = 2_097_152     # 2 MiB
DEFAULT_SAMPLES     = 1
DEFAULT_DTYPE       = "uint8"
DEFAULT_SEED        = 10

# -----------------------
# Helpers
# -----------------------
DTYPE_SIZES = {
    "uint8": 1, "int8": 1,
    "uint16": 2, "int16": 2,
    "uint32": 4, "int32": 4,
    "float32": 4, "float64": 8,
}

def choose_target_bytes(mean_b, stdev_b, resize, randomize):
    if randomize and stdev_b > 0:
        draw = int(np.random.normal(loc=mean_b, scale=stdev_b))
        draw = max(draw, 1)
    else:
        draw = int(mean_b)
    # Round to nearest multiple of resize
    return int(round(draw / resize) * resize)

def choose_hw_for_bytes(total_bytes, samples, dtype_size):
    """
    Choose H, W making H*W*samples*dtype_size == total_bytes.
    We factor total elements and spread powers of two across H and W
    to avoid super-skinny arrays.
    """
    total_elems = total_bytes // (dtype_size * samples)
    if total_elems == 0:
        raise ValueError("Total elements computed as 0; check inputs.")
    n = total_elems
    # Factor out powers of two
    exp2 = (n & -n).bit_length() - 1
    odd  = n >> exp2
    h = 1 << (exp2 // 2)
    w = (1 << (exp2 - exp2 // 2)) * odd
    return int(h), int(w)

def save_npz(out_path: Path, *, mean_bytes, stdev_bytes, resize_bytes,
             samples, dtype_name, seed, compress, randomize):
    dtype = getattr(np, dtype_name)
    dtype_size = DTYPE_SIZES[dtype_name]

    np.random.seed(seed)
    target_bytes = choose_target_bytes(mean_bytes, stdev_bytes, resize_bytes, randomize)
    # Ensure divisibility:
    elems_per_sample = target_bytes // dtype_size // samples
    if elems_per_sample * dtype_size * samples != target_bytes:
        raise ValueError("Target bytes not divisible by dtype_size*samples; adjust params.")

    h, w = choose_hw_for_bytes(target_bytes, samples, dtype_size)

    x = np.random.randint(255, size=(h, w, samples), dtype=dtype if dtype_name == "uint8" else np.uint8)
    if dtype_name != "uint8":
        x = x.astype(dtype, copy=False)
    y = np.zeros((samples,), dtype=np.uint8)  # matches DLIO NPZ generator convention

    out_path.parent.mkdir(parents=True, exist_ok=True)
    if compress:
        np.savez_compressed(out_path, x=x, y=y)
    else:
        np.savez(out_path, x=x, y=y)

    print(f"✅ Wrote {out_path}")
    try:
        sz = out_path.stat().st_size
        print(f"   size={sz} bytes, x.shape={x.shape}, dtype={x.dtype}, samples={samples}")
    except FileNotFoundError:
        pass

def list_and_crc(npz_path: Path, deep=False):
    print(f"📂 File: {npz_path}")
    with zipfile.ZipFile(npz_path, "r") as zf:
        names = zf.namelist()
        print(f"📦 Files in archive: {names}\n")
        for name in names:
            info = zf.getinfo(name)
            print(f"--- {name} ---")
            print(f"Stored CRC32       : 0x{info.CRC:08x}")
            print(f"Compressed Size    : {info.compress_size}")
            print(f"Uncompressed Size  : {info.file_size}")
            try:
                with zf.open(info) as f:
                    _ = f.read()                 # will raise if CRC mismatch
                print("✅ CRC verified by zipfile.\n")
            except zipfile.BadZipFile as e:
                print(f"⚠️ CRC error via zipfile: {e}")
                if deep:
                    ok = deep_crc_check(npz_path, info)
                    print("🔎 Deep check      :", "✅ OK\n" if ok else "❌ Mismatch\n")
                else:
                    print("ℹ️  Re-run with --deep-check to diagnose.\n")
            except Exception as e:
                print(f"❌ Unexpected error: {e}\n")

def deep_crc_check(npz_path: Path, info: zipfile.ZipInfo) -> bool:
    """
    Manual CRC of the *uncompressed* payload.
    Parse the local file header to find the compressed bytes, then
    decompress and compute CRC32 of the uncompressed stream.
    """
    with npz_path.open("rb") as fh:
        fh.seek(info.header_offset)
        local = fh.read(30)  # fixed part of local header
        # local file header sig 'PK\x03\x04'
        if local[:4] != b'PK\x03\x04':
            return False
        # filename length, extra length
        name_len, extra_len = struct.unpack("<HH", local[26:30])
        fh.seek(info.header_offset + 30 + name_len + extra_len)
        comp = fh.read(info.compress_size)

    # Decompress if needed
    if info.compress_type == zipfile.ZIP_STORED:
        data = comp
    elif info.compress_type == zipfile.ZIP_DEFLATED:
        # ZIP uses raw DEFLATE stream (no zlib header): wbits = -15
        try:
            data = zlib.decompress(comp, -15)
        except zlib.error:
            # Some writers include zlib headers; try normal
            data = zlib.decompress(comp)
    else:
        # Other methods not handled here
        return False

    crc = zlib.crc32(data) & 0xFFFFFFFF
    print(f"Computed CRC32     : 0x{crc:08x}")
    return crc == info.CRC

# -----------------------
# CLI
# -----------------------
def parse_args():
    p = argparse.ArgumentParser(description="MLPerf-style NPZ save & check tool")
    mode = p.add_mutually_exclusive_group()
    mode.add_argument("--save-only", action="store_true", help="Only save the NPZ")
    mode.add_argument("--check-only", action="store_true", help="Only verify/show the NPZ")

    p.add_argument("--npz-path", type=Path, help="Full output/check path to NPZ (overrides --out-dir/--name)")
    p.add_argument("--out-dir", type=Path, default=Path("/mnt/hs_test"), help="Directory for output NPZ")
    p.add_argument("--name", default="sample_000000.npz", help="Filename for output NPZ")
    p.add_argument("--compress", action="store_true", help="Use compressed NPZ (deflate)")

    # Size/dtype controls
    p.add_argument("--mean-bytes", type=int, default=DEFAULT_MEAN_BYTES, help="record_length_bytes")
    p.add_argument("--stdev-bytes", type=int, default=DEFAULT_STDEV_BYTES, help="record_length_bytes_stdev")
    p.add_argument("--resize-bytes", type=int, default=DEFAULT_RESIZE, help="record_length_bytes_resize multiple")
    p.add_argument("--randomize", action="store_true", help="Draw size from N(mean,stdev) before rounding")
    p.add_argument("--samples", type=int, default=DEFAULT_SAMPLES, help="num_samples_per_file")
    p.add_argument("--dtype", choices=DTYPE_SIZES.keys(), default=DEFAULT_DTYPE, help="Data dtype for 'x'")
    p.add_argument("--seed", type=int, default=DEFAULT_SEED, help="RNG seed for reproducibility")

    # Check controls
    p.add_argument("--deep-check", action="store_true", help="When checking, manually parse & CRC the member data")
    return p.parse_args()

def main():
    args = parse_args()
    out_path = args.npz_path or (args.out_dir / args.name)

    did_save = False
    if not args.check_only:
        save_npz(
            out_path=out_path,
            mean_bytes=args.mean_bytes,
            stdev_bytes=args.stdev_bytes,
            resize_bytes=args.resize_bytes,
            samples=args.samples,
            dtype_name=args.dtype,
            seed=args.seed,
            compress=args.compress,
            randomize=args.randomize,
        )
        did_save = True

    if not args.save_only:
        if not out_path.exists():
            raise SystemExit(f"File not found for check: {out_path}")
        list_and_crc(out_path, deep=args.deep_check)
    elif did_save:
        # echo path for easy piping
        print(out_path)

if __name__ == "__main__":
    main()

Re: parts of pages on NFS being replaced by swaths of NULs

[Jeff Layton]

On Sat, 2025-08-16 at 07:51 -0700, Trond Myklebust wrote:
> On Sat, 2025-08-16 at 09:01 -0400, Jeff Layton wrote:
> > 
> > I finally caught something concrete today. I had the attached
> > bpftrace
> > script running while running the reproducer on a dozen or so
> > machines,
> > and it detected a hole in some data being written:
> > 
> > -------------8<---------------
> > Attached 2 probes
> > Missing nfs_page: ino=10122173116 idx=2 flags=0x15ffff0000000029
> > Hole: ino=10122173116 idx=3 off=10026 size=2262
> > Prev folio: idx=2 flags=0x15ffff0000000028 pgbase=0 bytes=4096 req=0
> > prevreq=0xffff8955b2f55980
> > -------------8<---------------
> > 
> > What this tells us is that the page at idx=2 got submitted to
> > nfs_do_writepage() (so it was marked dirty in the pagecache), but
> > when
> > it got there, folio->private was NULL and it was ignored.
> > 
> > The kernel in this case is based on v6.9, so it's (just) pre-large-
> > folio support. It has a fair number of NFS patches, but not much to
> > this portion of the code. Most of them are are containerization
> > fixes.
> > 
> > I'm looking askance at nfs_inode_remove_request(). It does this:
> > 
> >         if (nfs_page_group_sync_on_bit(req, PG_REMOVE)) {
> >                 struct folio *folio = nfs_page_to_folio(req-
> > > wb_head);
> >                 struct address_space *mapping = folio->mapping;
> > 
> >                 spin_lock(&mapping->i_private_lock);
> >                 if (likely(folio)) {
> >                         folio->private = NULL;
> >                         folio_clear_private(folio);
> >                         clear_bit(PG_MAPPED, &req->wb_head-
> > > wb_flags);
> >                 }
> >                 spin_unlock(&mapping->i_private_lock);
> >         }
> > 
> > If nfs_page_group_sync_on_bit() returns true, then the nfs_page gets
> > detached from the folio. Meanwhile, if a new write request comes in
> > just after that, nfs_lock_and_join_requests() will call
> > nfs_cancel_remove_inode() to try to "cancel" PG_REMOVE:
> > 
> > static int
> > nfs_cancel_remove_inode(struct nfs_page *req, struct inode *inode)
> > {
> >         int ret;
> > 
> >         if (!test_bit(PG_REMOVE, &req->wb_flags))
> >                 return 0;
> >         ret = nfs_page_group_lock(req);
> >         if (ret)
> >                 return ret;
> >         if (test_and_clear_bit(PG_REMOVE, &req->wb_flags))
> >                 nfs_page_set_inode_ref(req, inode);
> >         nfs_page_group_unlock(req);                          
> >         return 0;                                    
> > }                     
> > 
> > ...but that does not reattach the nfs_page to the folio. Should it?
> >                         
> 
> That's not sufficient AFAICS. Does the following patch work?
> 
> 8<------------------------------------------------------------
> From fc9690dda01f001c6cd11665701394da8ebba1ab Mon Sep 17 00:00:00 2001
> Message-ID: <fc9690dda01f001c6cd11665701394da8ebba1ab.1755355810.git.trond.myklebust@hammerspace.com>
> From: Trond Myklebust <trond.myklebust@hammerspace.com>
> Date: Sat, 16 Aug 2025 07:25:20 -0700
> Subject: [PATCH] NFS: Fix a race when updating an existing write
> 
> After nfs_lock_and_join_requests() tests for whether the request is
> still attached to the mapping, nothing prevents a call to
> nfs_inode_remove_request() from succeeding until we actually lock the
> page group.
> The reason is that whoever called nfs_inode_remove_request() doesn't
> necessarily have a lock on the page group head.
> 
> So in order to avoid races, let's take the page group lock earlier in
> nfs_lock_and_join_requests(), and hold it across the removal of the
> request in nfs_inode_remove_request().
> 
> Reported-by: Jeff Layton <jlayton@kernel.org>
> Fixes: c3f2235782c3 ("nfs: fold nfs_folio_find_and_lock_request into nfs_lock_and_join_requests")
> Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
> ---
>  fs/nfs/pagelist.c        |  9 +++++----
>  fs/nfs/write.c           | 29 ++++++++++-------------------
>  include/linux/nfs_page.h |  1 +
>  3 files changed, 16 insertions(+), 23 deletions(-)
> 
> diff --git a/fs/nfs/pagelist.c b/fs/nfs/pagelist.c
> index 11968dcb7243..6e69ce43a13f 100644
> --- a/fs/nfs/pagelist.c
> +++ b/fs/nfs/pagelist.c
> @@ -253,13 +253,14 @@ nfs_page_group_unlock(struct nfs_page *req)
>  	nfs_page_clear_headlock(req);
>  }
>  
> -/*
> - * nfs_page_group_sync_on_bit_locked
> +/**
> + * nfs_page_group_sync_on_bit_locked - Test if all requests have @bit set
> + * @req: request in page group
> + * @bit: PG_* bit that is used to sync page group
>   *
>   * must be called with page group lock held
>   */
> -static bool
> -nfs_page_group_sync_on_bit_locked(struct nfs_page *req, unsigned int bit)
> +bool nfs_page_group_sync_on_bit_locked(struct nfs_page *req, unsigned int bit)
>  {
>  	struct nfs_page *head = req->wb_head;
>  	struct nfs_page *tmp;
> diff --git a/fs/nfs/write.c b/fs/nfs/write.c
> index fa5c41d0989a..8b7c04737967 100644
> --- a/fs/nfs/write.c
> +++ b/fs/nfs/write.c
> @@ -153,20 +153,10 @@ nfs_page_set_inode_ref(struct nfs_page *req, struct inode *inode)
>  	}
>  }
>  
> -static int
> -nfs_cancel_remove_inode(struct nfs_page *req, struct inode *inode)
> +static void nfs_cancel_remove_inode(struct nfs_page *req, struct inode *inode)
>  {
> -	int ret;
> -
> -	if (!test_bit(PG_REMOVE, &req->wb_flags))
> -		return 0;
> -	ret = nfs_page_group_lock(req);
> -	if (ret)
> -		return ret;
>  	if (test_and_clear_bit(PG_REMOVE, &req->wb_flags))
>  		nfs_page_set_inode_ref(req, inode);
> -	nfs_page_group_unlock(req);
> -	return 0;
>  }
>  
>  /**
> @@ -585,19 +575,18 @@ static struct nfs_page *nfs_lock_and_join_requests(struct folio *folio)
>  		}
>  	}
>  
> +	ret = nfs_page_group_lock(head);
> +	if (ret < 0)
> +		goto out_unlock;
> +
>  	/* Ensure that nobody removed the request before we locked it */
>  	if (head != folio->private) {
> +		nfs_page_group_unlock(head);
>  		nfs_unlock_and_release_request(head);
>  		goto retry;
>  	}
>  
> -	ret = nfs_cancel_remove_inode(head, inode);
> -	if (ret < 0)
> -		goto out_unlock;
> -
> -	ret = nfs_page_group_lock(head);
> -	if (ret < 0)
> -		goto out_unlock;
> +	nfs_cancel_remove_inode(head, inode);
>  
>  	/* lock each request in the page group */
>  	for (subreq = head->wb_this_page;
> @@ -786,7 +775,8 @@ static void nfs_inode_remove_request(struct nfs_page *req)
>  {
>  	struct nfs_inode *nfsi = NFS_I(nfs_page_to_inode(req));
>  
> -	if (nfs_page_group_sync_on_bit(req, PG_REMOVE)) {
> +	nfs_page_group_lock(req);
> +	if (nfs_page_group_sync_on_bit_locked(req, PG_REMOVE)) {
>  		struct folio *folio = nfs_page_to_folio(req->wb_head);
>  		struct address_space *mapping = folio->mapping;
>  
> @@ -798,6 +788,7 @@ static void nfs_inode_remove_request(struct nfs_page *req)
>  		}
>  		spin_unlock(&mapping->i_private_lock);
>  	}
> +	nfs_page_group_unlock(req);
>  
>  	if (test_and_clear_bit(PG_INODE_REF, &req->wb_flags)) {
>  		atomic_long_dec(&nfsi->nrequests);
> diff --git a/include/linux/nfs_page.h b/include/linux/nfs_page.h
> index 169b4ae30ff4..9aed39abc94b 100644
> --- a/include/linux/nfs_page.h
> +++ b/include/linux/nfs_page.h
> @@ -160,6 +160,7 @@ extern void nfs_join_page_group(struct nfs_page *head,
>  extern int nfs_page_group_lock(struct nfs_page *);
>  extern void nfs_page_group_unlock(struct nfs_page *);
>  extern bool nfs_page_group_sync_on_bit(struct nfs_page *, unsigned int);
> +extern bool nfs_page_group_sync_on_bit_locked(struct nfs_page *, unsigned int);
>  extern	int nfs_page_set_headlock(struct nfs_page *req);
>  extern void nfs_page_clear_headlock(struct nfs_page *req);
>  extern bool nfs_async_iocounter_wait(struct rpc_task *, struct nfs_lock_context *);

I backported this patch to the kernel we've been using to reproduce
this and have had the test running for almost 24 hours now. The longest
it's taken to reproduce on this test rig is about 12 hours. So, the
initial signs are good.

The patch also looks good to me. This one took a while to track down,
and I needed a lot of help to set up the test rig. Can you add these?

Tested-by: Joe Quanaim <jdq@meta.com>
Tested-by: Andrew Steffen <aksteffen@meta.com>
Reviewed-by: Jeff Layton <jlayton@kernel.org>

Joe and Andrew spent a lot of time getting us a reproducer.

I assume we also want to send this to stable? I'm pretty sure the
Fixes: tag is wrong. The kernel we were using didn't have that patch. I
took a look at some earlier releases, and AFAICT, this bug has been
present for a long time -- at least since v6.0 and probably well
before.

Thanks!
-- 
Jeff Layton <jlayton@kernel.org>

Re: parts of pages on NFS being replaced by swaths of NULs

[Trond Myklebust]

On Tue, 2025-08-19 at 13:10 -0400, Jeff Layton wrote:
> On Sat, 2025-08-16 at 07:51 -0700, Trond Myklebust wrote:
> > On Sat, 2025-08-16 at 09:01 -0400, Jeff Layton wrote:
> > > 
> > > I finally caught something concrete today. I had the attached
> > > bpftrace
> > > script running while running the reproducer on a dozen or so
> > > machines,
> > > and it detected a hole in some data being written:
> > > 
> > > -------------8<---------------
> > > Attached 2 probes
> > > Missing nfs_page: ino=10122173116 idx=2 flags=0x15ffff0000000029
> > > Hole: ino=10122173116 idx=3 off=10026 size=2262
> > > Prev folio: idx=2 flags=0x15ffff0000000028 pgbase=0 bytes=4096
> > > req=0
> > > prevreq=0xffff8955b2f55980
> > > -------------8<---------------
> > > 
> > > What this tells us is that the page at idx=2 got submitted to
> > > nfs_do_writepage() (so it was marked dirty in the pagecache), but
> > > when
> > > it got there, folio->private was NULL and it was ignored.
> > > 
> > > The kernel in this case is based on v6.9, so it's (just) pre-
> > > large-
> > > folio support. It has a fair number of NFS patches, but not much
> > > to
> > > this portion of the code. Most of them are are containerization
> > > fixes.
> > > 
> > > I'm looking askance at nfs_inode_remove_request(). It does this:
> > > 
> > >         if (nfs_page_group_sync_on_bit(req, PG_REMOVE)) {
> > >                 struct folio *folio = nfs_page_to_folio(req-
> > > > wb_head);
> > >                 struct address_space *mapping = folio->mapping;
> > > 
> > >                 spin_lock(&mapping->i_private_lock);
> > >                 if (likely(folio)) {
> > >                         folio->private = NULL;
> > >                         folio_clear_private(folio);
> > >                         clear_bit(PG_MAPPED, &req->wb_head-
> > > > wb_flags);
> > >                 }
> > >                 spin_unlock(&mapping->i_private_lock);
> > >         }
> > > 
> > > If nfs_page_group_sync_on_bit() returns true, then the nfs_page
> > > gets
> > > detached from the folio. Meanwhile, if a new write request comes
> > > in
> > > just after that, nfs_lock_and_join_requests() will call
> > > nfs_cancel_remove_inode() to try to "cancel" PG_REMOVE:
> > > 
> > > static int
> > > nfs_cancel_remove_inode(struct nfs_page *req, struct inode
> > > *inode)
> > > {
> > >         int ret;
> > > 
> > >         if (!test_bit(PG_REMOVE, &req->wb_flags))
> > >                 return 0;
> > >         ret = nfs_page_group_lock(req);
> > >         if (ret)
> > >                 return ret;
> > >         if (test_and_clear_bit(PG_REMOVE, &req->wb_flags))
> > >                 nfs_page_set_inode_ref(req, inode);
> > >         nfs_page_group_unlock(req);                          
> > >         return 0;                                    
> > > }                     
> > > 
> > > ...but that does not reattach the nfs_page to the folio. Should
> > > it?
> > >                         
> > 
> > That's not sufficient AFAICS. Does the following patch work?
> > 
> > 8<------------------------------------------------------------
> > From fc9690dda01f001c6cd11665701394da8ebba1ab Mon Sep 17 00:00:00
> > 2001
> > Message-ID:
> > <fc9690dda01f001c6cd11665701394da8ebba1ab.1755355810.git.trond.mykl
> > ebust@hammerspace.com>
> > From: Trond Myklebust <trond.myklebust@hammerspace.com>
> > Date: Sat, 16 Aug 2025 07:25:20 -0700
> > Subject: [PATCH] NFS: Fix a race when updating an existing write
> > 
> > After nfs_lock_and_join_requests() tests for whether the request is
> > still attached to the mapping, nothing prevents a call to
> > nfs_inode_remove_request() from succeeding until we actually lock
> > the
> > page group.
> > The reason is that whoever called nfs_inode_remove_request()
> > doesn't
> > necessarily have a lock on the page group head.
> > 
> > So in order to avoid races, let's take the page group lock earlier
> > in
> > nfs_lock_and_join_requests(), and hold it across the removal of the
> > request in nfs_inode_remove_request().
> > 
> > Reported-by: Jeff Layton <jlayton@kernel.org>
> > Fixes: c3f2235782c3 ("nfs: fold nfs_folio_find_and_lock_request
> > into nfs_lock_and_join_requests")
> > Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
> > ---
> >  fs/nfs/pagelist.c        |  9 +++++----
> >  fs/nfs/write.c           | 29 ++++++++++-------------------
> >  include/linux/nfs_page.h |  1 +
> >  3 files changed, 16 insertions(+), 23 deletions(-)
> > 
> > diff --git a/fs/nfs/pagelist.c b/fs/nfs/pagelist.c
> > index 11968dcb7243..6e69ce43a13f 100644
> > --- a/fs/nfs/pagelist.c
> > +++ b/fs/nfs/pagelist.c
> > @@ -253,13 +253,14 @@ nfs_page_group_unlock(struct nfs_page *req)
> >  	nfs_page_clear_headlock(req);
> >  }
> >  
> > -/*
> > - * nfs_page_group_sync_on_bit_locked
> > +/**
> > + * nfs_page_group_sync_on_bit_locked - Test if all requests have
> > @bit set
> > + * @req: request in page group
> > + * @bit: PG_* bit that is used to sync page group
> >   *
> >   * must be called with page group lock held
> >   */
> > -static bool
> > -nfs_page_group_sync_on_bit_locked(struct nfs_page *req, unsigned
> > int bit)
> > +bool nfs_page_group_sync_on_bit_locked(struct nfs_page *req,
> > unsigned int bit)
> >  {
> >  	struct nfs_page *head = req->wb_head;
> >  	struct nfs_page *tmp;
> > diff --git a/fs/nfs/write.c b/fs/nfs/write.c
> > index fa5c41d0989a..8b7c04737967 100644
> > --- a/fs/nfs/write.c
> > +++ b/fs/nfs/write.c
> > @@ -153,20 +153,10 @@ nfs_page_set_inode_ref(struct nfs_page *req,
> > struct inode *inode)
> >  	}
> >  }
> >  
> > -static int
> > -nfs_cancel_remove_inode(struct nfs_page *req, struct inode *inode)
> > +static void nfs_cancel_remove_inode(struct nfs_page *req, struct
> > inode *inode)
> >  {
> > -	int ret;
> > -
> > -	if (!test_bit(PG_REMOVE, &req->wb_flags))
> > -		return 0;
> > -	ret = nfs_page_group_lock(req);
> > -	if (ret)
> > -		return ret;
> >  	if (test_and_clear_bit(PG_REMOVE, &req->wb_flags))
> >  		nfs_page_set_inode_ref(req, inode);
> > -	nfs_page_group_unlock(req);
> > -	return 0;
> >  }
> >  
> >  /**
> > @@ -585,19 +575,18 @@ static struct nfs_page
> > *nfs_lock_and_join_requests(struct folio *folio)
> >  		}
> >  	}
> >  
> > +	ret = nfs_page_group_lock(head);
> > +	if (ret < 0)
> > +		goto out_unlock;
> > +
> >  	/* Ensure that nobody removed the request before we locked
> > it */
> >  	if (head != folio->private) {
> > +		nfs_page_group_unlock(head);
> >  		nfs_unlock_and_release_request(head);
> >  		goto retry;
> >  	}
> >  
> > -	ret = nfs_cancel_remove_inode(head, inode);
> > -	if (ret < 0)
> > -		goto out_unlock;
> > -
> > -	ret = nfs_page_group_lock(head);
> > -	if (ret < 0)
> > -		goto out_unlock;
> > +	nfs_cancel_remove_inode(head, inode);
> >  
> >  	/* lock each request in the page group */
> >  	for (subreq = head->wb_this_page;
> > @@ -786,7 +775,8 @@ static void nfs_inode_remove_request(struct
> > nfs_page *req)
> >  {
> >  	struct nfs_inode *nfsi = NFS_I(nfs_page_to_inode(req));
> >  
> > -	if (nfs_page_group_sync_on_bit(req, PG_REMOVE)) {
> > +	nfs_page_group_lock(req);
> > +	if (nfs_page_group_sync_on_bit_locked(req, PG_REMOVE)) {
> >  		struct folio *folio = nfs_page_to_folio(req-
> > >wb_head);
> >  		struct address_space *mapping = folio->mapping;
> >  
> > @@ -798,6 +788,7 @@ static void nfs_inode_remove_request(struct
> > nfs_page *req)
> >  		}
> >  		spin_unlock(&mapping->i_private_lock);
> >  	}
> > +	nfs_page_group_unlock(req);
> >  
> >  	if (test_and_clear_bit(PG_INODE_REF, &req->wb_flags)) {
> >  		atomic_long_dec(&nfsi->nrequests);
> > diff --git a/include/linux/nfs_page.h b/include/linux/nfs_page.h
> > index 169b4ae30ff4..9aed39abc94b 100644
> > --- a/include/linux/nfs_page.h
> > +++ b/include/linux/nfs_page.h
> > @@ -160,6 +160,7 @@ extern void nfs_join_page_group(struct nfs_page
> > *head,
> >  extern int nfs_page_group_lock(struct nfs_page *);
> >  extern void nfs_page_group_unlock(struct nfs_page *);
> >  extern bool nfs_page_group_sync_on_bit(struct nfs_page *, unsigned
> > int);
> > +extern bool nfs_page_group_sync_on_bit_locked(struct nfs_page *,
> > unsigned int);
> >  extern	int nfs_page_set_headlock(struct nfs_page *req);
> >  extern void nfs_page_clear_headlock(struct nfs_page *req);
> >  extern bool nfs_async_iocounter_wait(struct rpc_task *, struct
> > nfs_lock_context *);
> 
> I backported this patch to the kernel we've been using to reproduce
> this and have had the test running for almost 24 hours now. The
> longest
> it's taken to reproduce on this test rig is about 12 hours. So, the
> initial signs are good.
> 
> The patch also looks good to me. This one took a while to track down,
> and I needed a lot of help to set up the test rig. Can you add these?
> 
> Tested-by: Joe Quanaim <jdq@meta.com>
> Tested-by: Andrew Steffen <aksteffen@meta.com>
> Reviewed-by: Jeff Layton <jlayton@kernel.org>
> 
> Joe and Andrew spent a lot of time getting us a reproducer.
> 
> I assume we also want to send this to stable? I'm pretty sure the
> Fixes: tag is wrong. The kernel we were using didn't have that patch.
> I
> took a look at some earlier releases, and AFAICT, this bug has been
> present for a long time -- at least since v6.0 and probably well
> before.
> 

I've set
Fixes: bd37d6fce184 ("NFSv4: Convert nfs_lock_and_join_requests() to use nfs_page_find_head_request()")

on the very latest commit tags. That's about as far back as I can trace
it before going cross-eyed.
-- 
Trond Myklebust
Linux NFS client maintainer, Hammerspace
trondmy@kernel.org, trond.myklebust@hammerspace.com

Re: parts of pages on NFS being replaced by swaths of NULs

[Mike Snitzer]

On Sat, Aug 16, 2025 at 05:25:28PM -0400, Mike Snitzer wrote:
> On Sat, Aug 16, 2025 at 07:51:17AM -0700, Trond Myklebust wrote:
> > 
> > That's not sufficient AFAICS. Does the following patch work?
> > 
> > 8<------------------------------------------------------------
> > From fc9690dda01f001c6cd11665701394da8ebba1ab Mon Sep 17 00:00:00 2001
> > Message-ID: <fc9690dda01f001c6cd11665701394da8ebba1ab.1755355810.git.trond.myklebust@hammerspace.com>
> > From: Trond Myklebust <trond.myklebust@hammerspace.com>
> > Date: Sat, 16 Aug 2025 07:25:20 -0700
> > Subject: [PATCH] NFS: Fix a race when updating an existing write
> > 
> > After nfs_lock_and_join_requests() tests for whether the request is
> > still attached to the mapping, nothing prevents a call to
> > nfs_inode_remove_request() from succeeding until we actually lock the
> > page group.
> > The reason is that whoever called nfs_inode_remove_request() doesn't
> > necessarily have a lock on the page group head.
> > 
> > So in order to avoid races, let's take the page group lock earlier in
> > nfs_lock_and_join_requests(), and hold it across the removal of the
> > request in nfs_inode_remove_request().
> > 
> > Reported-by: Jeff Layton <jlayton@kernel.org>
> > Fixes: c3f2235782c3 ("nfs: fold nfs_folio_find_and_lock_request into nfs_lock_and_join_requests")
> > Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
> > ---
> >  fs/nfs/pagelist.c        |  9 +++++----
> >  fs/nfs/write.c           | 29 ++++++++++-------------------
> >  include/linux/nfs_page.h |  1 +
> >  3 files changed, 16 insertions(+), 23 deletions(-)
> > 
> > diff --git a/fs/nfs/pagelist.c b/fs/nfs/pagelist.c
> > index 11968dcb7243..6e69ce43a13f 100644
> > --- a/fs/nfs/pagelist.c
> > +++ b/fs/nfs/pagelist.c
> > @@ -253,13 +253,14 @@ nfs_page_group_unlock(struct nfs_page *req)
> >  	nfs_page_clear_headlock(req);
> >  }
> >  
> > -/*
> > - * nfs_page_group_sync_on_bit_locked
> > +/**
> > + * nfs_page_group_sync_on_bit_locked - Test if all requests have @bit set
> > + * @req: request in page group
> > + * @bit: PG_* bit that is used to sync page group
> >   *
> >   * must be called with page group lock held
> >   */
> > -static bool
> > -nfs_page_group_sync_on_bit_locked(struct nfs_page *req, unsigned int bit)
> > +bool nfs_page_group_sync_on_bit_locked(struct nfs_page *req, unsigned int bit)
> >  {
> >  	struct nfs_page *head = req->wb_head;
> >  	struct nfs_page *tmp;
> > diff --git a/fs/nfs/write.c b/fs/nfs/write.c
> > index fa5c41d0989a..8b7c04737967 100644
> > --- a/fs/nfs/write.c
> > +++ b/fs/nfs/write.c
> > @@ -153,20 +153,10 @@ nfs_page_set_inode_ref(struct nfs_page *req, struct inode *inode)
> >  	}
> >  }
> >  
> > -static int
> > -nfs_cancel_remove_inode(struct nfs_page *req, struct inode *inode)
> > +static void nfs_cancel_remove_inode(struct nfs_page *req, struct inode *inode)
> >  {
> > -	int ret;
> > -
> > -	if (!test_bit(PG_REMOVE, &req->wb_flags))
> > -		return 0;
> > -	ret = nfs_page_group_lock(req);
> > -	if (ret)
> > -		return ret;
> >  	if (test_and_clear_bit(PG_REMOVE, &req->wb_flags))
> >  		nfs_page_set_inode_ref(req, inode);
> > -	nfs_page_group_unlock(req);
> > -	return 0;
> >  }
> >  
> >  /**
> > @@ -585,19 +575,18 @@ static struct nfs_page *nfs_lock_and_join_requests(struct folio *folio)
> >  		}
> >  	}
> >  
> > +	ret = nfs_page_group_lock(head);
> > +	if (ret < 0)
> > +		goto out_unlock;
> > +
> >  	/* Ensure that nobody removed the request before we locked it */
> >  	if (head != folio->private) {
> > +		nfs_page_group_unlock(head);
> >  		nfs_unlock_and_release_request(head);
> >  		goto retry;
> >  	}
> >  
> > -	ret = nfs_cancel_remove_inode(head, inode);
> > -	if (ret < 0)
> > -		goto out_unlock;
> > -
> > -	ret = nfs_page_group_lock(head);
> > -	if (ret < 0)
> > -		goto out_unlock;
> > +	nfs_cancel_remove_inode(head, inode);
> >  
> >  	/* lock each request in the page group */
> >  	for (subreq = head->wb_this_page;
> > @@ -786,7 +775,8 @@ static void nfs_inode_remove_request(struct nfs_page *req)
> >  {
> >  	struct nfs_inode *nfsi = NFS_I(nfs_page_to_inode(req));
> >  
> > -	if (nfs_page_group_sync_on_bit(req, PG_REMOVE)) {
> > +	nfs_page_group_lock(req);
> > +	if (nfs_page_group_sync_on_bit_locked(req, PG_REMOVE)) {
> >  		struct folio *folio = nfs_page_to_folio(req->wb_head);
> >  		struct address_space *mapping = folio->mapping;
> >  
> > @@ -798,6 +788,7 @@ static void nfs_inode_remove_request(struct nfs_page *req)
> >  		}
> >  		spin_unlock(&mapping->i_private_lock);
> >  	}
> > +	nfs_page_group_unlock(req);
> >  
> >  	if (test_and_clear_bit(PG_INODE_REF, &req->wb_flags)) {
> >  		atomic_long_dec(&nfsi->nrequests);
> > diff --git a/include/linux/nfs_page.h b/include/linux/nfs_page.h
> > index 169b4ae30ff4..9aed39abc94b 100644
> > --- a/include/linux/nfs_page.h
> > +++ b/include/linux/nfs_page.h
> > @@ -160,6 +160,7 @@ extern void nfs_join_page_group(struct nfs_page *head,
> >  extern int nfs_page_group_lock(struct nfs_page *);
> >  extern void nfs_page_group_unlock(struct nfs_page *);
> >  extern bool nfs_page_group_sync_on_bit(struct nfs_page *, unsigned int);
> > +extern bool nfs_page_group_sync_on_bit_locked(struct nfs_page *, unsigned int);
> >  extern	int nfs_page_set_headlock(struct nfs_page *req);
> >  extern void nfs_page_clear_headlock(struct nfs_page *req);
> >  extern bool nfs_async_iocounter_wait(struct rpc_task *, struct nfs_lock_context *);
> > -- 
> > 2.50.1
> > 
> > 
> 
> 
> Trond,
> 
> You're the best! ;)
> 
> Your patch fixes corruption I've been chasing for the past week
> relative to NFS DIRECT, specifically with:
> echo Y > /sys/module/nfs/parameters/localio_O_DIRECT_align_misaligned_IO
> 
> So you need my latest NFS DIRECT patchset:
> https://lore.kernel.org/linux-nfs/20250815233003.55071-1-snitzer@kernel.org/
> 
> With it, writes would be corrupted when using the attached reproducer
> (from Jon Flynn, with the assistance of ChatGPT) that pulls out the
> subset of MLperf unet3d test (when ran in buffered IO mode, so
> entirely misaligned relative to DIO-alignment requirements) that we've
> seen npz CRC compare failure with.
> 
> I tested my patchset with your patch applied and it all "just works".
> 
> Ship it all!
> 
> Thanks,
> Mike
> 
> ps. running the attached reproducer is as simple as:
> ./mlperf_npz_tool.py --npz-path /mnt/share1/sample_a.npz

So even with this patch there is something still causing data
corruption on much faster systems.  This mlperf_npz_tool.py reproducer
works fine on a slower VMware based testbed, but once I load the same
kernel (with Trond's fix applied) on a very fast baremetal system I
still see CRC mismatch due to write corruption if
localio_O_DIRECT_align_misaligned_IO=Y (from my NFS DIRECT patchset).

Again, both of my latest NFSD DIRECT and NFS DIRECT patchsets are
applied:
https://lore.kernel.org/linux-nfs/20250815144607.50967-1-snitzer@kernel.org/
https://lore.kernel.org/linux-nfs/20250815233003.55071-1-snitzer@kernel.org/

And I'm hacking nfs4 to impose O_DIRECT on buffered IO (reproducer
doesn't open its files with O_DIRECT but NFS client acts like it did).
I can dig out the patch to achieve that if others are interested.

Mike

Re: parts of pages on NFS being replaced by swaths of NULs

[Jeff Layton]

On Tue, 2025-08-19 at 11:28 -0700, Trond Myklebust wrote:
> On Tue, 2025-08-19 at 13:10 -0400, Jeff Layton wrote:
> > On Sat, 2025-08-16 at 07:51 -0700, Trond Myklebust wrote:
> > > On Sat, 2025-08-16 at 09:01 -0400, Jeff Layton wrote:
> > > > 
> > > > I finally caught something concrete today. I had the attached
> > > > bpftrace
> > > > script running while running the reproducer on a dozen or so
> > > > machines,
> > > > and it detected a hole in some data being written:
> > > > 
> > > > -------------8<---------------
> > > > Attached 2 probes
> > > > Missing nfs_page: ino=10122173116 idx=2 flags=0x15ffff0000000029
> > > > Hole: ino=10122173116 idx=3 off=10026 size=2262
> > > > Prev folio: idx=2 flags=0x15ffff0000000028 pgbase=0 bytes=4096
> > > > req=0
> > > > prevreq=0xffff8955b2f55980
> > > > -------------8<---------------
> > > > 
> > > > What this tells us is that the page at idx=2 got submitted to
> > > > nfs_do_writepage() (so it was marked dirty in the pagecache), but
> > > > when
> > > > it got there, folio->private was NULL and it was ignored.
> > > > 
> > > > The kernel in this case is based on v6.9, so it's (just) pre-
> > > > large-
> > > > folio support. It has a fair number of NFS patches, but not much
> > > > to
> > > > this portion of the code. Most of them are are containerization
> > > > fixes.
> > > > 
> > > > I'm looking askance at nfs_inode_remove_request(). It does this:
> > > > 
> > > >         if (nfs_page_group_sync_on_bit(req, PG_REMOVE)) {
> > > >                 struct folio *folio = nfs_page_to_folio(req-
> > > > > wb_head);
> > > >                 struct address_space *mapping = folio->mapping;
> > > > 
> > > >                 spin_lock(&mapping->i_private_lock);
> > > >                 if (likely(folio)) {
> > > >                         folio->private = NULL;
> > > >                         folio_clear_private(folio);
> > > >                         clear_bit(PG_MAPPED, &req->wb_head-
> > > > > wb_flags);
> > > >                 }
> > > >                 spin_unlock(&mapping->i_private_lock);
> > > >         }
> > > > 
> > > > If nfs_page_group_sync_on_bit() returns true, then the nfs_page
> > > > gets
> > > > detached from the folio. Meanwhile, if a new write request comes
> > > > in
> > > > just after that, nfs_lock_and_join_requests() will call
> > > > nfs_cancel_remove_inode() to try to "cancel" PG_REMOVE:
> > > > 
> > > > static int
> > > > nfs_cancel_remove_inode(struct nfs_page *req, struct inode
> > > > *inode)
> > > > {
> > > >         int ret;
> > > > 
> > > >         if (!test_bit(PG_REMOVE, &req->wb_flags))
> > > >                 return 0;
> > > >         ret = nfs_page_group_lock(req);
> > > >         if (ret)
> > > >                 return ret;
> > > >         if (test_and_clear_bit(PG_REMOVE, &req->wb_flags))
> > > >                 nfs_page_set_inode_ref(req, inode);
> > > >         nfs_page_group_unlock(req);                          
> > > >         return 0;                                    
> > > > }                     
> > > > 
> > > > ...but that does not reattach the nfs_page to the folio. Should
> > > > it?
> > > >                         
> > > 
> > > That's not sufficient AFAICS. Does the following patch work?
> > > 
> > > 8<------------------------------------------------------------
> > > From fc9690dda01f001c6cd11665701394da8ebba1ab Mon Sep 17 00:00:00
> > > 2001
> > > Message-ID:
> > > <fc9690dda01f001c6cd11665701394da8ebba1ab.1755355810.git.trond.mykl
> > > ebust@hammerspace.com>
> > > From: Trond Myklebust <trond.myklebust@hammerspace.com>
> > > Date: Sat, 16 Aug 2025 07:25:20 -0700
> > > Subject: [PATCH] NFS: Fix a race when updating an existing write
> > > 
> > > After nfs_lock_and_join_requests() tests for whether the request is
> > > still attached to the mapping, nothing prevents a call to
> > > nfs_inode_remove_request() from succeeding until we actually lock
> > > the
> > > page group.
> > > The reason is that whoever called nfs_inode_remove_request()
> > > doesn't
> > > necessarily have a lock on the page group head.
> > > 
> > > So in order to avoid races, let's take the page group lock earlier
> > > in
> > > nfs_lock_and_join_requests(), and hold it across the removal of the
> > > request in nfs_inode_remove_request().
> > > 
> > > Reported-by: Jeff Layton <jlayton@kernel.org>
> > > Fixes: c3f2235782c3 ("nfs: fold nfs_folio_find_and_lock_request
> > > into nfs_lock_and_join_requests")
> > > Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
> > > ---
> > >  fs/nfs/pagelist.c        |  9 +++++----
> > >  fs/nfs/write.c           | 29 ++++++++++-------------------
> > >  include/linux/nfs_page.h |  1 +
> > >  3 files changed, 16 insertions(+), 23 deletions(-)
> > > 
> > > diff --git a/fs/nfs/pagelist.c b/fs/nfs/pagelist.c
> > > index 11968dcb7243..6e69ce43a13f 100644
> > > --- a/fs/nfs/pagelist.c
> > > +++ b/fs/nfs/pagelist.c
> > > @@ -253,13 +253,14 @@ nfs_page_group_unlock(struct nfs_page *req)
> > >  	nfs_page_clear_headlock(req);
> > >  }
> > >  
> > > -/*
> > > - * nfs_page_group_sync_on_bit_locked
> > > +/**
> > > + * nfs_page_group_sync_on_bit_locked - Test if all requests have
> > > @bit set
> > > + * @req: request in page group
> > > + * @bit: PG_* bit that is used to sync page group
> > >   *
> > >   * must be called with page group lock held
> > >   */
> > > -static bool
> > > -nfs_page_group_sync_on_bit_locked(struct nfs_page *req, unsigned
> > > int bit)
> > > +bool nfs_page_group_sync_on_bit_locked(struct nfs_page *req,
> > > unsigned int bit)
> > >  {
> > >  	struct nfs_page *head = req->wb_head;
> > >  	struct nfs_page *tmp;
> > > diff --git a/fs/nfs/write.c b/fs/nfs/write.c
> > > index fa5c41d0989a..8b7c04737967 100644
> > > --- a/fs/nfs/write.c
> > > +++ b/fs/nfs/write.c
> > > @@ -153,20 +153,10 @@ nfs_page_set_inode_ref(struct nfs_page *req,
> > > struct inode *inode)
> > >  	}
> > >  }
> > >  
> > > -static int
> > > -nfs_cancel_remove_inode(struct nfs_page *req, struct inode *inode)
> > > +static void nfs_cancel_remove_inode(struct nfs_page *req, struct
> > > inode *inode)
> > >  {
> > > -	int ret;
> > > -
> > > -	if (!test_bit(PG_REMOVE, &req->wb_flags))
> > > -		return 0;
> > > -	ret = nfs_page_group_lock(req);
> > > -	if (ret)
> > > -		return ret;
> > >  	if (test_and_clear_bit(PG_REMOVE, &req->wb_flags))
> > >  		nfs_page_set_inode_ref(req, inode);
> > > -	nfs_page_group_unlock(req);
> > > -	return 0;
> > >  }
> > >  
> > >  /**
> > > @@ -585,19 +575,18 @@ static struct nfs_page
> > > *nfs_lock_and_join_requests(struct folio *folio)
> > >  		}
> > >  	}
> > >  
> > > +	ret = nfs_page_group_lock(head);
> > > +	if (ret < 0)
> > > +		goto out_unlock;
> > > +
> > >  	/* Ensure that nobody removed the request before we locked
> > > it */
> > >  	if (head != folio->private) {
> > > +		nfs_page_group_unlock(head);
> > >  		nfs_unlock_and_release_request(head);
> > >  		goto retry;
> > >  	}
> > >  
> > > -	ret = nfs_cancel_remove_inode(head, inode);
> > > -	if (ret < 0)
> > > -		goto out_unlock;
> > > -
> > > -	ret = nfs_page_group_lock(head);
> > > -	if (ret < 0)
> > > -		goto out_unlock;
> > > +	nfs_cancel_remove_inode(head, inode);
> > >  
> > >  	/* lock each request in the page group */
> > >  	for (subreq = head->wb_this_page;
> > > @@ -786,7 +775,8 @@ static void nfs_inode_remove_request(struct
> > > nfs_page *req)
> > >  {
> > >  	struct nfs_inode *nfsi = NFS_I(nfs_page_to_inode(req));
> > >  
> > > -	if (nfs_page_group_sync_on_bit(req, PG_REMOVE)) {
> > > +	nfs_page_group_lock(req);
> > > +	if (nfs_page_group_sync_on_bit_locked(req, PG_REMOVE)) {
> > >  		struct folio *folio = nfs_page_to_folio(req-
> > > > wb_head);
> > >  		struct address_space *mapping = folio->mapping;
> > >  
> > > @@ -798,6 +788,7 @@ static void nfs_inode_remove_request(struct
> > > nfs_page *req)
> > >  		}
> > >  		spin_unlock(&mapping->i_private_lock);
> > >  	}
> > > +	nfs_page_group_unlock(req);
> > >  
> > >  	if (test_and_clear_bit(PG_INODE_REF, &req->wb_flags)) {
> > >  		atomic_long_dec(&nfsi->nrequests);
> > > diff --git a/include/linux/nfs_page.h b/include/linux/nfs_page.h
> > > index 169b4ae30ff4..9aed39abc94b 100644
> > > --- a/include/linux/nfs_page.h
> > > +++ b/include/linux/nfs_page.h
> > > @@ -160,6 +160,7 @@ extern void nfs_join_page_group(struct nfs_page
> > > *head,
> > >  extern int nfs_page_group_lock(struct nfs_page *);
> > >  extern void nfs_page_group_unlock(struct nfs_page *);
> > >  extern bool nfs_page_group_sync_on_bit(struct nfs_page *, unsigned
> > > int);
> > > +extern bool nfs_page_group_sync_on_bit_locked(struct nfs_page *,
> > > unsigned int);
> > >  extern	int nfs_page_set_headlock(struct nfs_page *req);
> > >  extern void nfs_page_clear_headlock(struct nfs_page *req);
> > >  extern bool nfs_async_iocounter_wait(struct rpc_task *, struct
> > > nfs_lock_context *);
> > 
> > I backported this patch to the kernel we've been using to reproduce
> > this and have had the test running for almost 24 hours now. The
> > longest
> > it's taken to reproduce on this test rig is about 12 hours. So, the
> > initial signs are good.
> > 
> > The patch also looks good to me. This one took a while to track down,
> > and I needed a lot of help to set up the test rig. Can you add these?
> > 
> > Tested-by: Joe Quanaim <jdq@meta.com>
> > Tested-by: Andrew Steffen <aksteffen@meta.com>
> > Reviewed-by: Jeff Layton <jlayton@kernel.org>
> > 
> > Joe and Andrew spent a lot of time getting us a reproducer.
> > 
> > I assume we also want to send this to stable? I'm pretty sure the
> > Fixes: tag is wrong. The kernel we were using didn't have that patch.
> > I
> > took a look at some earlier releases, and AFAICT, this bug has been
> > present for a long time -- at least since v6.0 and probably well
> > before.
> > 
> 
> I've set
> Fixes: bd37d6fce184 ("NFSv4: Convert nfs_lock_and_join_requests() to use nfs_page_find_head_request()")
> 
> on the very latest commit tags. That's about as far back as I can trace
> it before going cross-eyed.

Thanks. Looks like that went into v4.14. That's probably far enough. :)
-- 
Jeff Layton <jlayton@kernel.org>

Re: parts of pages on NFS being replaced by swaths of NULs

[Jeff Layton]

On Tue, 2025-08-19 at 14:38 -0400, Mike Snitzer wrote:
> On Sat, Aug 16, 2025 at 05:25:28PM -0400, Mike Snitzer wrote:
> > On Sat, Aug 16, 2025 at 07:51:17AM -0700, Trond Myklebust wrote:
> > > 
> > > That's not sufficient AFAICS. Does the following patch work?
> > > 
> > > 8<------------------------------------------------------------
> > > From fc9690dda01f001c6cd11665701394da8ebba1ab Mon Sep 17 00:00:00 2001
> > > Message-ID: <fc9690dda01f001c6cd11665701394da8ebba1ab.1755355810.git.trond.myklebust@hammerspace.com>
> > > From: Trond Myklebust <trond.myklebust@hammerspace.com>
> > > Date: Sat, 16 Aug 2025 07:25:20 -0700
> > > Subject: [PATCH] NFS: Fix a race when updating an existing write
> > > 
> > > After nfs_lock_and_join_requests() tests for whether the request is
> > > still attached to the mapping, nothing prevents a call to
> > > nfs_inode_remove_request() from succeeding until we actually lock the
> > > page group.
> > > The reason is that whoever called nfs_inode_remove_request() doesn't
> > > necessarily have a lock on the page group head.
> > > 
> > > So in order to avoid races, let's take the page group lock earlier in
> > > nfs_lock_and_join_requests(), and hold it across the removal of the
> > > request in nfs_inode_remove_request().
> > > 
> > > Reported-by: Jeff Layton <jlayton@kernel.org>
> > > Fixes: c3f2235782c3 ("nfs: fold nfs_folio_find_and_lock_request into nfs_lock_and_join_requests")
> > > Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
> > > ---
> > >  fs/nfs/pagelist.c        |  9 +++++----
> > >  fs/nfs/write.c           | 29 ++++++++++-------------------
> > >  include/linux/nfs_page.h |  1 +
> > >  3 files changed, 16 insertions(+), 23 deletions(-)
> > > 
> > > diff --git a/fs/nfs/pagelist.c b/fs/nfs/pagelist.c
> > > index 11968dcb7243..6e69ce43a13f 100644
> > > --- a/fs/nfs/pagelist.c
> > > +++ b/fs/nfs/pagelist.c
> > > @@ -253,13 +253,14 @@ nfs_page_group_unlock(struct nfs_page *req)
> > >  	nfs_page_clear_headlock(req);
> > >  }
> > >  
> > > -/*
> > > - * nfs_page_group_sync_on_bit_locked
> > > +/**
> > > + * nfs_page_group_sync_on_bit_locked - Test if all requests have @bit set
> > > + * @req: request in page group
> > > + * @bit: PG_* bit that is used to sync page group
> > >   *
> > >   * must be called with page group lock held
> > >   */
> > > -static bool
> > > -nfs_page_group_sync_on_bit_locked(struct nfs_page *req, unsigned int bit)
> > > +bool nfs_page_group_sync_on_bit_locked(struct nfs_page *req, unsigned int bit)
> > >  {
> > >  	struct nfs_page *head = req->wb_head;
> > >  	struct nfs_page *tmp;
> > > diff --git a/fs/nfs/write.c b/fs/nfs/write.c
> > > index fa5c41d0989a..8b7c04737967 100644
> > > --- a/fs/nfs/write.c
> > > +++ b/fs/nfs/write.c
> > > @@ -153,20 +153,10 @@ nfs_page_set_inode_ref(struct nfs_page *req, struct inode *inode)
> > >  	}
> > >  }
> > >  
> > > -static int
> > > -nfs_cancel_remove_inode(struct nfs_page *req, struct inode *inode)
> > > +static void nfs_cancel_remove_inode(struct nfs_page *req, struct inode *inode)
> > >  {
> > > -	int ret;
> > > -
> > > -	if (!test_bit(PG_REMOVE, &req->wb_flags))
> > > -		return 0;
> > > -	ret = nfs_page_group_lock(req);
> > > -	if (ret)
> > > -		return ret;
> > >  	if (test_and_clear_bit(PG_REMOVE, &req->wb_flags))
> > >  		nfs_page_set_inode_ref(req, inode);
> > > -	nfs_page_group_unlock(req);
> > > -	return 0;
> > >  }
> > >  
> > >  /**
> > > @@ -585,19 +575,18 @@ static struct nfs_page *nfs_lock_and_join_requests(struct folio *folio)
> > >  		}
> > >  	}
> > >  
> > > +	ret = nfs_page_group_lock(head);
> > > +	if (ret < 0)
> > > +		goto out_unlock;
> > > +
> > >  	/* Ensure that nobody removed the request before we locked it */
> > >  	if (head != folio->private) {
> > > +		nfs_page_group_unlock(head);
> > >  		nfs_unlock_and_release_request(head);
> > >  		goto retry;
> > >  	}
> > >  
> > > -	ret = nfs_cancel_remove_inode(head, inode);
> > > -	if (ret < 0)
> > > -		goto out_unlock;
> > > -
> > > -	ret = nfs_page_group_lock(head);
> > > -	if (ret < 0)
> > > -		goto out_unlock;
> > > +	nfs_cancel_remove_inode(head, inode);
> > >  
> > >  	/* lock each request in the page group */
> > >  	for (subreq = head->wb_this_page;
> > > @@ -786,7 +775,8 @@ static void nfs_inode_remove_request(struct nfs_page *req)
> > >  {
> > >  	struct nfs_inode *nfsi = NFS_I(nfs_page_to_inode(req));
> > >  
> > > -	if (nfs_page_group_sync_on_bit(req, PG_REMOVE)) {
> > > +	nfs_page_group_lock(req);
> > > +	if (nfs_page_group_sync_on_bit_locked(req, PG_REMOVE)) {
> > >  		struct folio *folio = nfs_page_to_folio(req->wb_head);
> > >  		struct address_space *mapping = folio->mapping;
> > >  
> > > @@ -798,6 +788,7 @@ static void nfs_inode_remove_request(struct nfs_page *req)
> > >  		}
> > >  		spin_unlock(&mapping->i_private_lock);
> > >  	}
> > > +	nfs_page_group_unlock(req);
> > >  
> > >  	if (test_and_clear_bit(PG_INODE_REF, &req->wb_flags)) {
> > >  		atomic_long_dec(&nfsi->nrequests);
> > > diff --git a/include/linux/nfs_page.h b/include/linux/nfs_page.h
> > > index 169b4ae30ff4..9aed39abc94b 100644
> > > --- a/include/linux/nfs_page.h
> > > +++ b/include/linux/nfs_page.h
> > > @@ -160,6 +160,7 @@ extern void nfs_join_page_group(struct nfs_page *head,
> > >  extern int nfs_page_group_lock(struct nfs_page *);
> > >  extern void nfs_page_group_unlock(struct nfs_page *);
> > >  extern bool nfs_page_group_sync_on_bit(struct nfs_page *, unsigned int);
> > > +extern bool nfs_page_group_sync_on_bit_locked(struct nfs_page *, unsigned int);
> > >  extern	int nfs_page_set_headlock(struct nfs_page *req);
> > >  extern void nfs_page_clear_headlock(struct nfs_page *req);
> > >  extern bool nfs_async_iocounter_wait(struct rpc_task *, struct nfs_lock_context *);
> > > -- 
> > > 2.50.1
> > > 
> > > 
> > 
> > 
> > Trond,
> > 
> > You're the best! ;)
> > 
> > Your patch fixes corruption I've been chasing for the past week
> > relative to NFS DIRECT, specifically with:
> > echo Y > /sys/module/nfs/parameters/localio_O_DIRECT_align_misaligned_IO
> > 
> > So you need my latest NFS DIRECT patchset:
> > https://lore.kernel.org/linux-nfs/20250815233003.55071-1-snitzer@kernel.org/
> > 
> > With it, writes would be corrupted when using the attached reproducer
> > (from Jon Flynn, with the assistance of ChatGPT) that pulls out the
> > subset of MLperf unet3d test (when ran in buffered IO mode, so
> > entirely misaligned relative to DIO-alignment requirements) that we've
> > seen npz CRC compare failure with.
> > 
> > I tested my patchset with your patch applied and it all "just works".
> > 
> > Ship it all!
> > 
> > Thanks,
> > Mike
> > 
> > ps. running the attached reproducer is as simple as:
> > ./mlperf_npz_tool.py --npz-path /mnt/share1/sample_a.npz
> 
> So even with this patch there is something still causing data
> corruption on much faster systems.  This mlperf_npz_tool.py reproducer
> works fine on a slower VMware based testbed, but once I load the same
> kernel (with Trond's fix applied) on a very fast baremetal system I
> still see CRC mismatch due to write corruption if
> localio_O_DIRECT_align_misaligned_IO=Y (from my NFS DIRECT patchset).
> 
> Again, both of my latest NFSD DIRECT and NFS DIRECT patchsets are
> applied:
> https://lore.kernel.org/linux-nfs/20250815144607.50967-1-snitzer@kernel.org/
> https://lore.kernel.org/linux-nfs/20250815233003.55071-1-snitzer@kernel.org/
> 
> And I'm hacking nfs4 to impose O_DIRECT on buffered IO (reproducer
> doesn't open its files with O_DIRECT but NFS client acts like it did).
> I can dig out the patch to achieve that if others are interested.
> 
> Mike

Can you compare a corrupt file with the expected output? That might
give us a clue about the nature of this bug.

-- 
Jeff Layton <jlayton@kernel.org>

Re: parts of pages on NFS being replaced by swaths of NULs

[Mike Snitzer]

On Tue, Aug 19, 2025 at 03:48:32PM -0400, Jeff Layton wrote:
> On Tue, 2025-08-19 at 11:28 -0700, Trond Myklebust wrote:
> > On Tue, 2025-08-19 at 13:10 -0400, Jeff Layton wrote:
> > > On Sat, 2025-08-16 at 07:51 -0700, Trond Myklebust wrote:
> > > > On Sat, 2025-08-16 at 09:01 -0400, Jeff Layton wrote:
> > > > > 
> > > > > I finally caught something concrete today. I had the attached
> > > > > bpftrace
> > > > > script running while running the reproducer on a dozen or so
> > > > > machines,
> > > > > and it detected a hole in some data being written:
> > > > > 
> > > > > -------------8<---------------
> > > > > Attached 2 probes
> > > > > Missing nfs_page: ino=10122173116 idx=2 flags=0x15ffff0000000029
> > > > > Hole: ino=10122173116 idx=3 off=10026 size=2262
> > > > > Prev folio: idx=2 flags=0x15ffff0000000028 pgbase=0 bytes=4096
> > > > > req=0
> > > > > prevreq=0xffff8955b2f55980
> > > > > -------------8<---------------
> > > > > 
> > > > > What this tells us is that the page at idx=2 got submitted to
> > > > > nfs_do_writepage() (so it was marked dirty in the pagecache), but
> > > > > when
> > > > > it got there, folio->private was NULL and it was ignored.
> > > > > 
> > > > > The kernel in this case is based on v6.9, so it's (just) pre-
> > > > > large-
> > > > > folio support. It has a fair number of NFS patches, but not much
> > > > > to
> > > > > this portion of the code. Most of them are are containerization
> > > > > fixes.
> > > > > 
> > > > > I'm looking askance at nfs_inode_remove_request(). It does this:
> > > > > 
> > > > >         if (nfs_page_group_sync_on_bit(req, PG_REMOVE)) {
> > > > >                 struct folio *folio = nfs_page_to_folio(req-
> > > > > > wb_head);
> > > > >                 struct address_space *mapping = folio->mapping;
> > > > > 
> > > > >                 spin_lock(&mapping->i_private_lock);
> > > > >                 if (likely(folio)) {
> > > > >                         folio->private = NULL;
> > > > >                         folio_clear_private(folio);
> > > > >                         clear_bit(PG_MAPPED, &req->wb_head-
> > > > > > wb_flags);
> > > > >                 }
> > > > >                 spin_unlock(&mapping->i_private_lock);
> > > > >         }
> > > > > 
> > > > > If nfs_page_group_sync_on_bit() returns true, then the nfs_page
> > > > > gets
> > > > > detached from the folio. Meanwhile, if a new write request comes
> > > > > in
> > > > > just after that, nfs_lock_and_join_requests() will call
> > > > > nfs_cancel_remove_inode() to try to "cancel" PG_REMOVE:
> > > > > 
> > > > > static int
> > > > > nfs_cancel_remove_inode(struct nfs_page *req, struct inode
> > > > > *inode)
> > > > > {
> > > > >         int ret;
> > > > > 
> > > > >         if (!test_bit(PG_REMOVE, &req->wb_flags))
> > > > >                 return 0;
> > > > >         ret = nfs_page_group_lock(req);
> > > > >         if (ret)
> > > > >                 return ret;
> > > > >         if (test_and_clear_bit(PG_REMOVE, &req->wb_flags))
> > > > >                 nfs_page_set_inode_ref(req, inode);
> > > > >         nfs_page_group_unlock(req);                          
> > > > >         return 0;                                    
> > > > > }                     
> > > > > 
> > > > > ...but that does not reattach the nfs_page to the folio. Should
> > > > > it?
> > > > >                         
> > > > 
> > > > That's not sufficient AFAICS. Does the following patch work?
> > > > 
> > > > 8<------------------------------------------------------------
> > > > From fc9690dda01f001c6cd11665701394da8ebba1ab Mon Sep 17 00:00:00
> > > > 2001
> > > > Message-ID:
> > > > <fc9690dda01f001c6cd11665701394da8ebba1ab.1755355810.git.trond.mykl
> > > > ebust@hammerspace.com>
> > > > From: Trond Myklebust <trond.myklebust@hammerspace.com>
> > > > Date: Sat, 16 Aug 2025 07:25:20 -0700
> > > > Subject: [PATCH] NFS: Fix a race when updating an existing write
> > > > 
> > > > After nfs_lock_and_join_requests() tests for whether the request is
> > > > still attached to the mapping, nothing prevents a call to
> > > > nfs_inode_remove_request() from succeeding until we actually lock
> > > > the
> > > > page group.
> > > > The reason is that whoever called nfs_inode_remove_request()
> > > > doesn't
> > > > necessarily have a lock on the page group head.
> > > > 
> > > > So in order to avoid races, let's take the page group lock earlier
> > > > in
> > > > nfs_lock_and_join_requests(), and hold it across the removal of the
> > > > request in nfs_inode_remove_request().
> > > > 
> > > > Reported-by: Jeff Layton <jlayton@kernel.org>
> > > > Fixes: c3f2235782c3 ("nfs: fold nfs_folio_find_and_lock_request
> > > > into nfs_lock_and_join_requests")
> > > > Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
> > > > ---
> > > >  fs/nfs/pagelist.c        |  9 +++++----
> > > >  fs/nfs/write.c           | 29 ++++++++++-------------------
> > > >  include/linux/nfs_page.h |  1 +
> > > >  3 files changed, 16 insertions(+), 23 deletions(-)
> > > > 
> > > > diff --git a/fs/nfs/pagelist.c b/fs/nfs/pagelist.c
> > > > index 11968dcb7243..6e69ce43a13f 100644
> > > > --- a/fs/nfs/pagelist.c
> > > > +++ b/fs/nfs/pagelist.c
> > > > @@ -253,13 +253,14 @@ nfs_page_group_unlock(struct nfs_page *req)
> > > >  	nfs_page_clear_headlock(req);
> > > >  }
> > > >  
> > > > -/*
> > > > - * nfs_page_group_sync_on_bit_locked
> > > > +/**
> > > > + * nfs_page_group_sync_on_bit_locked - Test if all requests have
> > > > @bit set
> > > > + * @req: request in page group
> > > > + * @bit: PG_* bit that is used to sync page group
> > > >   *
> > > >   * must be called with page group lock held
> > > >   */
> > > > -static bool
> > > > -nfs_page_group_sync_on_bit_locked(struct nfs_page *req, unsigned
> > > > int bit)
> > > > +bool nfs_page_group_sync_on_bit_locked(struct nfs_page *req,
> > > > unsigned int bit)
> > > >  {
> > > >  	struct nfs_page *head = req->wb_head;
> > > >  	struct nfs_page *tmp;
> > > > diff --git a/fs/nfs/write.c b/fs/nfs/write.c
> > > > index fa5c41d0989a..8b7c04737967 100644
> > > > --- a/fs/nfs/write.c
> > > > +++ b/fs/nfs/write.c
> > > > @@ -153,20 +153,10 @@ nfs_page_set_inode_ref(struct nfs_page *req,
> > > > struct inode *inode)
> > > >  	}
> > > >  }
> > > >  
> > > > -static int
> > > > -nfs_cancel_remove_inode(struct nfs_page *req, struct inode *inode)
> > > > +static void nfs_cancel_remove_inode(struct nfs_page *req, struct
> > > > inode *inode)
> > > >  {
> > > > -	int ret;
> > > > -
> > > > -	if (!test_bit(PG_REMOVE, &req->wb_flags))
> > > > -		return 0;
> > > > -	ret = nfs_page_group_lock(req);
> > > > -	if (ret)
> > > > -		return ret;
> > > >  	if (test_and_clear_bit(PG_REMOVE, &req->wb_flags))
> > > >  		nfs_page_set_inode_ref(req, inode);
> > > > -	nfs_page_group_unlock(req);
> > > > -	return 0;
> > > >  }
> > > >  
> > > >  /**
> > > > @@ -585,19 +575,18 @@ static struct nfs_page
> > > > *nfs_lock_and_join_requests(struct folio *folio)
> > > >  		}
> > > >  	}
> > > >  
> > > > +	ret = nfs_page_group_lock(head);
> > > > +	if (ret < 0)
> > > > +		goto out_unlock;
> > > > +
> > > >  	/* Ensure that nobody removed the request before we locked
> > > > it */
> > > >  	if (head != folio->private) {
> > > > +		nfs_page_group_unlock(head);
> > > >  		nfs_unlock_and_release_request(head);
> > > >  		goto retry;
> > > >  	}
> > > >  
> > > > -	ret = nfs_cancel_remove_inode(head, inode);
> > > > -	if (ret < 0)
> > > > -		goto out_unlock;
> > > > -
> > > > -	ret = nfs_page_group_lock(head);
> > > > -	if (ret < 0)
> > > > -		goto out_unlock;
> > > > +	nfs_cancel_remove_inode(head, inode);
> > > >  
> > > >  	/* lock each request in the page group */
> > > >  	for (subreq = head->wb_this_page;
> > > > @@ -786,7 +775,8 @@ static void nfs_inode_remove_request(struct
> > > > nfs_page *req)
> > > >  {
> > > >  	struct nfs_inode *nfsi = NFS_I(nfs_page_to_inode(req));
> > > >  
> > > > -	if (nfs_page_group_sync_on_bit(req, PG_REMOVE)) {
> > > > +	nfs_page_group_lock(req);
> > > > +	if (nfs_page_group_sync_on_bit_locked(req, PG_REMOVE)) {
> > > >  		struct folio *folio = nfs_page_to_folio(req-
> > > > > wb_head);
> > > >  		struct address_space *mapping = folio->mapping;
> > > >  
> > > > @@ -798,6 +788,7 @@ static void nfs_inode_remove_request(struct
> > > > nfs_page *req)
> > > >  		}
> > > >  		spin_unlock(&mapping->i_private_lock);
> > > >  	}
> > > > +	nfs_page_group_unlock(req);
> > > >  
> > > >  	if (test_and_clear_bit(PG_INODE_REF, &req->wb_flags)) {
> > > >  		atomic_long_dec(&nfsi->nrequests);
> > > > diff --git a/include/linux/nfs_page.h b/include/linux/nfs_page.h
> > > > index 169b4ae30ff4..9aed39abc94b 100644
> > > > --- a/include/linux/nfs_page.h
> > > > +++ b/include/linux/nfs_page.h
> > > > @@ -160,6 +160,7 @@ extern void nfs_join_page_group(struct nfs_page
> > > > *head,
> > > >  extern int nfs_page_group_lock(struct nfs_page *);
> > > >  extern void nfs_page_group_unlock(struct nfs_page *);
> > > >  extern bool nfs_page_group_sync_on_bit(struct nfs_page *, unsigned
> > > > int);
> > > > +extern bool nfs_page_group_sync_on_bit_locked(struct nfs_page *,
> > > > unsigned int);
> > > >  extern	int nfs_page_set_headlock(struct nfs_page *req);
> > > >  extern void nfs_page_clear_headlock(struct nfs_page *req);
> > > >  extern bool nfs_async_iocounter_wait(struct rpc_task *, struct
> > > > nfs_lock_context *);
> > > 
> > > I backported this patch to the kernel we've been using to reproduce
> > > this and have had the test running for almost 24 hours now. The
> > > longest
> > > it's taken to reproduce on this test rig is about 12 hours. So, the
> > > initial signs are good.
> > > 
> > > The patch also looks good to me. This one took a while to track down,
> > > and I needed a lot of help to set up the test rig. Can you add these?
> > > 
> > > Tested-by: Joe Quanaim <jdq@meta.com>
> > > Tested-by: Andrew Steffen <aksteffen@meta.com>
> > > Reviewed-by: Jeff Layton <jlayton@kernel.org>
> > > 
> > > Joe and Andrew spent a lot of time getting us a reproducer.
> > > 
> > > I assume we also want to send this to stable? I'm pretty sure the
> > > Fixes: tag is wrong. The kernel we were using didn't have that patch.
> > > I
> > > took a look at some earlier releases, and AFAICT, this bug has been
> > > present for a long time -- at least since v6.0 and probably well
> > > before.
> > > 
> > 
> > I've set
> > Fixes: bd37d6fce184 ("NFSv4: Convert nfs_lock_and_join_requests() to use nfs_page_find_head_request()")
> > 
> > on the very latest commit tags. That's about as far back as I can trace
> > it before going cross-eyed.
> 
> Thanks. Looks like that went into v4.14. That's probably far enough. :)

Indeed, but we'll need to provide specialized backports for each
stable@ kernel the further back it needed.  So your 6.9 backport might
help with 6.6.  The other active older stable trees are 6.1 and 6.12.

Gregkh et al will quickly defer to domain experts to see that a fix
like this lands everywhere it needed.

Mike

seeing CRC mismatch with NFS DIRECT WRITE when localio_O_DIRECT_align_misaligned_IO=Y

[Mike Snitzer]

On Tue, Aug 19, 2025 at 04:13:16PM -0400, Jeff Layton wrote:
> On Tue, 2025-08-19 at 14:38 -0400, Mike Snitzer wrote:
> > On Sat, Aug 16, 2025 at 05:25:28PM -0400, Mike Snitzer wrote:
> > > On Sat, Aug 16, 2025 at 07:51:17AM -0700, Trond Myklebust wrote:
> > > > 
> > > > That's not sufficient AFAICS. Does the following patch work?
> > > > 
> > > > 8<------------------------------------------------------------
> > > > From fc9690dda01f001c6cd11665701394da8ebba1ab Mon Sep 17 00:00:00 2001
> > > > Message-ID: <fc9690dda01f001c6cd11665701394da8ebba1ab.1755355810.git.trond.myklebust@hammerspace.com>
> > > > From: Trond Myklebust <trond.myklebust@hammerspace.com>
> > > > Date: Sat, 16 Aug 2025 07:25:20 -0700
> > > > Subject: [PATCH] NFS: Fix a race when updating an existing write
> > > > 
> > > > After nfs_lock_and_join_requests() tests for whether the request is
> > > > still attached to the mapping, nothing prevents a call to
> > > > nfs_inode_remove_request() from succeeding until we actually lock the
> > > > page group.
> > > > The reason is that whoever called nfs_inode_remove_request() doesn't
> > > > necessarily have a lock on the page group head.
> > > > 
> > > > So in order to avoid races, let's take the page group lock earlier in
> > > > nfs_lock_and_join_requests(), and hold it across the removal of the
> > > > request in nfs_inode_remove_request().
> > > > 
> > > > Reported-by: Jeff Layton <jlayton@kernel.org>
> > > > Fixes: c3f2235782c3 ("nfs: fold nfs_folio_find_and_lock_request into nfs_lock_and_join_requests")
> > > > Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
> > > > ---
> > > >  fs/nfs/pagelist.c        |  9 +++++----
> > > >  fs/nfs/write.c           | 29 ++++++++++-------------------
> > > >  include/linux/nfs_page.h |  1 +
> > > >  3 files changed, 16 insertions(+), 23 deletions(-)
> > > > 
> > > > diff --git a/fs/nfs/pagelist.c b/fs/nfs/pagelist.c
> > > > index 11968dcb7243..6e69ce43a13f 100644
> > > > --- a/fs/nfs/pagelist.c
> > > > +++ b/fs/nfs/pagelist.c
> > > > @@ -253,13 +253,14 @@ nfs_page_group_unlock(struct nfs_page *req)
> > > >  	nfs_page_clear_headlock(req);
> > > >  }
> > > >  
> > > > -/*
> > > > - * nfs_page_group_sync_on_bit_locked
> > > > +/**
> > > > + * nfs_page_group_sync_on_bit_locked - Test if all requests have @bit set
> > > > + * @req: request in page group
> > > > + * @bit: PG_* bit that is used to sync page group
> > > >   *
> > > >   * must be called with page group lock held
> > > >   */
> > > > -static bool
> > > > -nfs_page_group_sync_on_bit_locked(struct nfs_page *req, unsigned int bit)
> > > > +bool nfs_page_group_sync_on_bit_locked(struct nfs_page *req, unsigned int bit)
> > > >  {
> > > >  	struct nfs_page *head = req->wb_head;
> > > >  	struct nfs_page *tmp;
> > > > diff --git a/fs/nfs/write.c b/fs/nfs/write.c
> > > > index fa5c41d0989a..8b7c04737967 100644
> > > > --- a/fs/nfs/write.c
> > > > +++ b/fs/nfs/write.c
> > > > @@ -153,20 +153,10 @@ nfs_page_set_inode_ref(struct nfs_page *req, struct inode *inode)
> > > >  	}
> > > >  }
> > > >  
> > > > -static int
> > > > -nfs_cancel_remove_inode(struct nfs_page *req, struct inode *inode)
> > > > +static void nfs_cancel_remove_inode(struct nfs_page *req, struct inode *inode)
> > > >  {
> > > > -	int ret;
> > > > -
> > > > -	if (!test_bit(PG_REMOVE, &req->wb_flags))
> > > > -		return 0;
> > > > -	ret = nfs_page_group_lock(req);
> > > > -	if (ret)
> > > > -		return ret;
> > > >  	if (test_and_clear_bit(PG_REMOVE, &req->wb_flags))
> > > >  		nfs_page_set_inode_ref(req, inode);
> > > > -	nfs_page_group_unlock(req);
> > > > -	return 0;
> > > >  }
> > > >  
> > > >  /**
> > > > @@ -585,19 +575,18 @@ static struct nfs_page *nfs_lock_and_join_requests(struct folio *folio)
> > > >  		}
> > > >  	}
> > > >  
> > > > +	ret = nfs_page_group_lock(head);
> > > > +	if (ret < 0)
> > > > +		goto out_unlock;
> > > > +
> > > >  	/* Ensure that nobody removed the request before we locked it */
> > > >  	if (head != folio->private) {
> > > > +		nfs_page_group_unlock(head);
> > > >  		nfs_unlock_and_release_request(head);
> > > >  		goto retry;
> > > >  	}
> > > >  
> > > > -	ret = nfs_cancel_remove_inode(head, inode);
> > > > -	if (ret < 0)
> > > > -		goto out_unlock;
> > > > -
> > > > -	ret = nfs_page_group_lock(head);
> > > > -	if (ret < 0)
> > > > -		goto out_unlock;
> > > > +	nfs_cancel_remove_inode(head, inode);
> > > >  
> > > >  	/* lock each request in the page group */
> > > >  	for (subreq = head->wb_this_page;
> > > > @@ -786,7 +775,8 @@ static void nfs_inode_remove_request(struct nfs_page *req)
> > > >  {
> > > >  	struct nfs_inode *nfsi = NFS_I(nfs_page_to_inode(req));
> > > >  
> > > > -	if (nfs_page_group_sync_on_bit(req, PG_REMOVE)) {
> > > > +	nfs_page_group_lock(req);
> > > > +	if (nfs_page_group_sync_on_bit_locked(req, PG_REMOVE)) {
> > > >  		struct folio *folio = nfs_page_to_folio(req->wb_head);
> > > >  		struct address_space *mapping = folio->mapping;
> > > >  
> > > > @@ -798,6 +788,7 @@ static void nfs_inode_remove_request(struct nfs_page *req)
> > > >  		}
> > > >  		spin_unlock(&mapping->i_private_lock);
> > > >  	}
> > > > +	nfs_page_group_unlock(req);
> > > >  
> > > >  	if (test_and_clear_bit(PG_INODE_REF, &req->wb_flags)) {
> > > >  		atomic_long_dec(&nfsi->nrequests);
> > > > diff --git a/include/linux/nfs_page.h b/include/linux/nfs_page.h
> > > > index 169b4ae30ff4..9aed39abc94b 100644
> > > > --- a/include/linux/nfs_page.h
> > > > +++ b/include/linux/nfs_page.h
> > > > @@ -160,6 +160,7 @@ extern void nfs_join_page_group(struct nfs_page *head,
> > > >  extern int nfs_page_group_lock(struct nfs_page *);
> > > >  extern void nfs_page_group_unlock(struct nfs_page *);
> > > >  extern bool nfs_page_group_sync_on_bit(struct nfs_page *, unsigned int);
> > > > +extern bool nfs_page_group_sync_on_bit_locked(struct nfs_page *, unsigned int);
> > > >  extern	int nfs_page_set_headlock(struct nfs_page *req);
> > > >  extern void nfs_page_clear_headlock(struct nfs_page *req);
> > > >  extern bool nfs_async_iocounter_wait(struct rpc_task *, struct nfs_lock_context *);
> > > > -- 
> > > > 2.50.1
> > > > 
> > > > 
> > > 
> > > 
> > > Trond,
> > > 
> > > You're the best! ;)
> > > 
> > > Your patch fixes corruption I've been chasing for the past week
> > > relative to NFS DIRECT, specifically with:
> > > echo Y > /sys/module/nfs/parameters/localio_O_DIRECT_align_misaligned_IO
> > > 
> > > So you need my latest NFS DIRECT patchset:
> > > https://lore.kernel.org/linux-nfs/20250815233003.55071-1-snitzer@kernel.org/
> > > 
> > > With it, writes would be corrupted when using the attached reproducer
> > > (from Jon Flynn, with the assistance of ChatGPT) that pulls out the
> > > subset of MLperf unet3d test (when ran in buffered IO mode, so
> > > entirely misaligned relative to DIO-alignment requirements) that we've
> > > seen npz CRC compare failure with.
> > > 
> > > I tested my patchset with your patch applied and it all "just works".
> > > 
> > > Ship it all!
> > > 
> > > Thanks,
> > > Mike
> > > 
> > > ps. running the attached reproducer is as simple as:
> > > ./mlperf_npz_tool.py --npz-path /mnt/share1/sample_a.npz
> > 
> > So even with this patch there is something still causing data
> > corruption on much faster systems.  This mlperf_npz_tool.py reproducer
> > works fine on a slower VMware based testbed, but once I load the same
> > kernel (with Trond's fix applied) on a very fast baremetal system I
> > still see CRC mismatch due to write corruption if
> > localio_O_DIRECT_align_misaligned_IO=Y (from my NFS DIRECT patchset).
> > 
> > Again, both of my latest NFSD DIRECT and NFS DIRECT patchsets are
> > applied:
> > https://lore.kernel.org/linux-nfs/20250815144607.50967-1-snitzer@kernel.org/
> > https://lore.kernel.org/linux-nfs/20250815233003.55071-1-snitzer@kernel.org/
> > 
> > And I'm hacking nfs4 to impose O_DIRECT on buffered IO (reproducer
> > doesn't open its files with O_DIRECT but NFS client acts like it did).
> > I can dig out the patch to achieve that if others are interested.
> > 
> > Mike
> 
> Can you compare a corrupt file with the expected output? That might
> give us a clue about the nature of this bug.

While it _is_ interesting that "speed kills" my particular lithmus
test: I've changed the $SUBJECT because this issue could easily be
unique to my new NFS DIRECT code or the test itself.  So best to
divorce my issue from the original issue you reported that was the
catalyst for Trond's fix (that did seem to help my testcase).

To be clear, this may well be some bug in the python application (that
happens to be part of the MLperf "industry benchmark")...

Using other tools to verify data integrity have all passed.  In
particular, dt isn't able to see any issues (but dt is more of an open
secret in the utility belt of companies like NetApp, Red Hat, now
Hammerspace, etc): https://github.com/RobinTMiller/dt.git
This patch header in the NFSD DIRECT series shows how dt proved very
useful:
https://lore.kernel.org/linux-nfs/20250815144607.50967-6-snitzer@kernel.org/

But back to the mlperf_npz_tool.py, I can easily generate good vs bad
files, any words of wisdom for useful tools to analyze the
differences?  ChatGPT?  Other AIs?

hexdump isn't showing useful or approachable differences, basically
the entire good vs bad file is different:

# hexdump /tmp/good.npz > /root/analyze_this/good.hd
# hexdump /tmp/bad.npz > /root/analyze_this/bad.hd
# diff -Naur /root/analyze_this/good.hd /root/analyze_this/bad.hd > hd.diff

# ls -alh
total 1.7G
drwxr-xr-x   2 root root   50 Aug 19 20:53 .
dr-xr-x---. 25 root root 4.0K Aug 19 20:56 ..
-rw-r--r--   1 root root 421M Aug 19 20:52 bad.hd
-rw-r--r--   1 root root 421M Aug 19 20:51 good.hd
-rw-r--r--   1 root root 803M Aug 19 20:54 hd.diff

# less hd.diff
--- /root/analyze_this/good.hd  2025-08-19 20:51:55.797022245 +0000
+++ /root/analyze_this/bad.hd   2025-08-19 20:52:14.868669416 +0000
@@ -254,983046 +254,983046 @@
 0000fd0 a73d eae0 f2b0 df27 6098 16be b990 f5b0
 0000fe0 e2d1 857f 1843 6b2f 079b f319 5386 b6f7
 0000ff0 e5ac ec09 29cd 17e9 1341 8a18 e54b 5057
-0001000 7711 ef46 982e fe27 a0b5 0d2f 0f05 0130
-0001010 d6e2 4a4c 74be f4cc a1ed cd98 fe09 0ba0
-0001020 65b5 05a1 426f 7658 c9d9 0381 8ed5 b4ad
...
+0001000 2325 8099 b007 9fd9 8596 01fe ae93 d384
+0001010 422e 7685 abab 0d9b 0a05 b4e9 b774 f619
+0001020 10fc 37d0 9e72 795c 82cd c9c8 9cac 4665
...
 8b01000 ce50 6a33 b82d f7ce 0699 5948 1a7e 4e33
 8b01010 ed0c f087 853e 952b 77c4 6e51 2790 26af
 8b01020 d588 4bc7 313e d729 c20f 4412 8844 8fa6

Mike

Re: seeing CRC mismatch with NFS DIRECT WRITE when localio_O_DIRECT_align_misaligned_IO=Y

[Jeff Layton]

On Tue, 2025-08-19 at 17:38 -0400, Mike Snitzer wrote:
> On Tue, Aug 19, 2025 at 04:13:16PM -0400, Jeff Layton wrote:
> > On Tue, 2025-08-19 at 14:38 -0400, Mike Snitzer wrote:
> > > On Sat, Aug 16, 2025 at 05:25:28PM -0400, Mike Snitzer wrote:
> > > > On Sat, Aug 16, 2025 at 07:51:17AM -0700, Trond Myklebust wrote:
> > > > > 
> > > > > That's not sufficient AFAICS. Does the following patch work?
> > > > > 
> > > > > 8<------------------------------------------------------------
> > > > > From fc9690dda01f001c6cd11665701394da8ebba1ab Mon Sep 17 00:00:00 2001
> > > > > Message-ID: <fc9690dda01f001c6cd11665701394da8ebba1ab.1755355810.git.trond.myklebust@hammerspace.com>
> > > > > From: Trond Myklebust <trond.myklebust@hammerspace.com>
> > > > > Date: Sat, 16 Aug 2025 07:25:20 -0700
> > > > > Subject: [PATCH] NFS: Fix a race when updating an existing write
> > > > > 
> > > > > After nfs_lock_and_join_requests() tests for whether the request is
> > > > > still attached to the mapping, nothing prevents a call to
> > > > > nfs_inode_remove_request() from succeeding until we actually lock the
> > > > > page group.
> > > > > The reason is that whoever called nfs_inode_remove_request() doesn't
> > > > > necessarily have a lock on the page group head.
> > > > > 
> > > > > So in order to avoid races, let's take the page group lock earlier in
> > > > > nfs_lock_and_join_requests(), and hold it across the removal of the
> > > > > request in nfs_inode_remove_request().
> > > > > 
> > > > > Reported-by: Jeff Layton <jlayton@kernel.org>
> > > > > Fixes: c3f2235782c3 ("nfs: fold nfs_folio_find_and_lock_request into nfs_lock_and_join_requests")
> > > > > Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
> > > > > ---
> > > > >  fs/nfs/pagelist.c        |  9 +++++----
> > > > >  fs/nfs/write.c           | 29 ++++++++++-------------------
> > > > >  include/linux/nfs_page.h |  1 +
> > > > >  3 files changed, 16 insertions(+), 23 deletions(-)
> > > > > 
> > > > > diff --git a/fs/nfs/pagelist.c b/fs/nfs/pagelist.c
> > > > > index 11968dcb7243..6e69ce43a13f 100644
> > > > > --- a/fs/nfs/pagelist.c
> > > > > +++ b/fs/nfs/pagelist.c
> > > > > @@ -253,13 +253,14 @@ nfs_page_group_unlock(struct nfs_page *req)
> > > > >  	nfs_page_clear_headlock(req);
> > > > >  }
> > > > >  
> > > > > -/*
> > > > > - * nfs_page_group_sync_on_bit_locked
> > > > > +/**
> > > > > + * nfs_page_group_sync_on_bit_locked - Test if all requests have @bit set
> > > > > + * @req: request in page group
> > > > > + * @bit: PG_* bit that is used to sync page group
> > > > >   *
> > > > >   * must be called with page group lock held
> > > > >   */
> > > > > -static bool
> > > > > -nfs_page_group_sync_on_bit_locked(struct nfs_page *req, unsigned int bit)
> > > > > +bool nfs_page_group_sync_on_bit_locked(struct nfs_page *req, unsigned int bit)
> > > > >  {
> > > > >  	struct nfs_page *head = req->wb_head;
> > > > >  	struct nfs_page *tmp;
> > > > > diff --git a/fs/nfs/write.c b/fs/nfs/write.c
> > > > > index fa5c41d0989a..8b7c04737967 100644
> > > > > --- a/fs/nfs/write.c
> > > > > +++ b/fs/nfs/write.c
> > > > > @@ -153,20 +153,10 @@ nfs_page_set_inode_ref(struct nfs_page *req, struct inode *inode)
> > > > >  	}
> > > > >  }
> > > > >  
> > > > > -static int
> > > > > -nfs_cancel_remove_inode(struct nfs_page *req, struct inode *inode)
> > > > > +static void nfs_cancel_remove_inode(struct nfs_page *req, struct inode *inode)
> > > > >  {
> > > > > -	int ret;
> > > > > -
> > > > > -	if (!test_bit(PG_REMOVE, &req->wb_flags))
> > > > > -		return 0;
> > > > > -	ret = nfs_page_group_lock(req);
> > > > > -	if (ret)
> > > > > -		return ret;
> > > > >  	if (test_and_clear_bit(PG_REMOVE, &req->wb_flags))
> > > > >  		nfs_page_set_inode_ref(req, inode);
> > > > > -	nfs_page_group_unlock(req);
> > > > > -	return 0;
> > > > >  }
> > > > >  
> > > > >  /**
> > > > > @@ -585,19 +575,18 @@ static struct nfs_page *nfs_lock_and_join_requests(struct folio *folio)
> > > > >  		}
> > > > >  	}
> > > > >  
> > > > > +	ret = nfs_page_group_lock(head);
> > > > > +	if (ret < 0)
> > > > > +		goto out_unlock;
> > > > > +
> > > > >  	/* Ensure that nobody removed the request before we locked it */
> > > > >  	if (head != folio->private) {
> > > > > +		nfs_page_group_unlock(head);
> > > > >  		nfs_unlock_and_release_request(head);
> > > > >  		goto retry;
> > > > >  	}
> > > > >  
> > > > > -	ret = nfs_cancel_remove_inode(head, inode);
> > > > > -	if (ret < 0)
> > > > > -		goto out_unlock;
> > > > > -
> > > > > -	ret = nfs_page_group_lock(head);
> > > > > -	if (ret < 0)
> > > > > -		goto out_unlock;
> > > > > +	nfs_cancel_remove_inode(head, inode);
> > > > >  
> > > > >  	/* lock each request in the page group */
> > > > >  	for (subreq = head->wb_this_page;
> > > > > @@ -786,7 +775,8 @@ static void nfs_inode_remove_request(struct nfs_page *req)
> > > > >  {
> > > > >  	struct nfs_inode *nfsi = NFS_I(nfs_page_to_inode(req));
> > > > >  
> > > > > -	if (nfs_page_group_sync_on_bit(req, PG_REMOVE)) {
> > > > > +	nfs_page_group_lock(req);
> > > > > +	if (nfs_page_group_sync_on_bit_locked(req, PG_REMOVE)) {
> > > > >  		struct folio *folio = nfs_page_to_folio(req->wb_head);
> > > > >  		struct address_space *mapping = folio->mapping;
> > > > >  
> > > > > @@ -798,6 +788,7 @@ static void nfs_inode_remove_request(struct nfs_page *req)
> > > > >  		}
> > > > >  		spin_unlock(&mapping->i_private_lock);
> > > > >  	}
> > > > > +	nfs_page_group_unlock(req);
> > > > >  
> > > > >  	if (test_and_clear_bit(PG_INODE_REF, &req->wb_flags)) {
> > > > >  		atomic_long_dec(&nfsi->nrequests);
> > > > > diff --git a/include/linux/nfs_page.h b/include/linux/nfs_page.h
> > > > > index 169b4ae30ff4..9aed39abc94b 100644
> > > > > --- a/include/linux/nfs_page.h
> > > > > +++ b/include/linux/nfs_page.h
> > > > > @@ -160,6 +160,7 @@ extern void nfs_join_page_group(struct nfs_page *head,
> > > > >  extern int nfs_page_group_lock(struct nfs_page *);
> > > > >  extern void nfs_page_group_unlock(struct nfs_page *);
> > > > >  extern bool nfs_page_group_sync_on_bit(struct nfs_page *, unsigned int);
> > > > > +extern bool nfs_page_group_sync_on_bit_locked(struct nfs_page *, unsigned int);
> > > > >  extern	int nfs_page_set_headlock(struct nfs_page *req);
> > > > >  extern void nfs_page_clear_headlock(struct nfs_page *req);
> > > > >  extern bool nfs_async_iocounter_wait(struct rpc_task *, struct nfs_lock_context *);
> > > > > -- 
> > > > > 2.50.1
> > > > > 
> > > > > 
> > > > 
> > > > 
> > > > Trond,
> > > > 
> > > > You're the best! ;)
> > > > 
> > > > Your patch fixes corruption I've been chasing for the past week
> > > > relative to NFS DIRECT, specifically with:
> > > > echo Y > /sys/module/nfs/parameters/localio_O_DIRECT_align_misaligned_IO
> > > > 
> > > > So you need my latest NFS DIRECT patchset:
> > > > https://lore.kernel.org/linux-nfs/20250815233003.55071-1-snitzer@kernel.org/
> > > > 
> > > > With it, writes would be corrupted when using the attached reproducer
> > > > (from Jon Flynn, with the assistance of ChatGPT) that pulls out the
> > > > subset of MLperf unet3d test (when ran in buffered IO mode, so
> > > > entirely misaligned relative to DIO-alignment requirements) that we've
> > > > seen npz CRC compare failure with.
> > > > 
> > > > I tested my patchset with your patch applied and it all "just works".
> > > > 
> > > > Ship it all!
> > > > 
> > > > Thanks,
> > > > Mike
> > > > 
> > > > ps. running the attached reproducer is as simple as:
> > > > ./mlperf_npz_tool.py --npz-path /mnt/share1/sample_a.npz
> > > 
> > > So even with this patch there is something still causing data
> > > corruption on much faster systems.  This mlperf_npz_tool.py reproducer
> > > works fine on a slower VMware based testbed, but once I load the same
> > > kernel (with Trond's fix applied) on a very fast baremetal system I
> > > still see CRC mismatch due to write corruption if
> > > localio_O_DIRECT_align_misaligned_IO=Y (from my NFS DIRECT patchset).
> > > 
> > > Again, both of my latest NFSD DIRECT and NFS DIRECT patchsets are
> > > applied:
> > > https://lore.kernel.org/linux-nfs/20250815144607.50967-1-snitzer@kernel.org/
> > > https://lore.kernel.org/linux-nfs/20250815233003.55071-1-snitzer@kernel.org/
> > > 
> > > And I'm hacking nfs4 to impose O_DIRECT on buffered IO (reproducer
> > > doesn't open its files with O_DIRECT but NFS client acts like it did).
> > > I can dig out the patch to achieve that if others are interested.
> > > 
> > > Mike
> > 
> > Can you compare a corrupt file with the expected output? That might
> > give us a clue about the nature of this bug.
> 
> While it _is_ interesting that "speed kills" my particular lithmus
> test: I've changed the $SUBJECT because this issue could easily be
> unique to my new NFS DIRECT code or the test itself.  So best to
> divorce my issue from the original issue you reported that was the
> catalyst for Trond's fix (that did seem to help my testcase).
> 
> To be clear, this may well be some bug in the python application (that
> happens to be part of the MLperf "industry benchmark")...
> 
> Using other tools to verify data integrity have all passed.  In
> particular, dt isn't able to see any issues (but dt is more of an open
> secret in the utility belt of companies like NetApp, Red Hat, now
> Hammerspace, etc): https://github.com/RobinTMiller/dt.git
> This patch header in the NFSD DIRECT series shows how dt proved very
> useful:
> https://lore.kernel.org/linux-nfs/20250815144607.50967-6-snitzer@kernel.org/
> 
> But back to the mlperf_npz_tool.py, I can easily generate good vs bad
> files, any words of wisdom for useful tools to analyze the
> differences?  ChatGPT?  Other AIs?
>
> hexdump isn't showing useful or approachable differences, basically
> the entire good vs bad file is different:
> 
> # hexdump /tmp/good.npz > /root/analyze_this/good.hd
> # hexdump /tmp/bad.npz > /root/analyze_this/bad.hd
> # diff -Naur /root/analyze_this/good.hd /root/analyze_this/bad.hd > hd.diff
> 
> # ls -alh
> total 1.7G
> drwxr-xr-x   2 root root   50 Aug 19 20:53 .
> dr-xr-x---. 25 root root 4.0K Aug 19 20:56 ..
> -rw-r--r--   1 root root 421M Aug 19 20:52 bad.hd
> -rw-r--r--   1 root root 421M Aug 19 20:51 good.hd
> -rw-r--r--   1 root root 803M Aug 19 20:54 hd.diff
> 
> # less hd.diff
> --- /root/analyze_this/good.hd  2025-08-19 20:51:55.797022245 +0000
> +++ /root/analyze_this/bad.hd   2025-08-19 20:52:14.868669416 +0000
> @@ -254,983046 +254,983046 @@
>  0000fd0 a73d eae0 f2b0 df27 6098 16be b990 f5b0
>  0000fe0 e2d1 857f 1843 6b2f 079b f319 5386 b6f7
>  0000ff0 e5ac ec09 29cd 17e9 1341 8a18 e54b 5057
> -0001000 7711 ef46 982e fe27 a0b5 0d2f 0f05 0130
> -0001010 d6e2 4a4c 74be f4cc a1ed cd98 fe09 0ba0
> -0001020 65b5 05a1 426f 7658 c9d9 0381 8ed5 b4ad
> ...
> +0001000 2325 8099 b007 9fd9 8596 01fe ae93 d384
> +0001010 422e 7685 abab 0d9b 0a05 b4e9 b774 f619
> +0001020 10fc 37d0 9e72 795c 82cd c9c8 9cac 4665
> ...
>  8b01000 ce50 6a33 b82d f7ce 0699 5948 1a7e 4e33
>  8b01010 ed0c f087 853e 952b 77c4 6e51 2790 26af
>  8b01020 d588 4bc7 313e d729 c20f 4412 8844 8fa6
> 

2 interesting things there:

1/ the corruption doesn't appear to be zeroes, so this is definitely a
different bug than the one we saw before.

2/ It looks like the corruption starts on a page boundary. How far does
it extend? If it's all page aligned, then that's probably an important
clue.

-- 
Jeff Layton <jlayton@kernel.org>