Re: [PATCH 0/6] NFSD: add enable-dontcache and initially use it to add DIO support

[Mike Snitzer <snitzer@kernel.org>]

On Thu, Jun 12, 2025 at 12:39:44AM -0700, Christoph Hellwig wrote:
> 
> Another thing is that using the page cache for reads is probably
> rather pointless.  I've been wondering if we should just change
> the direct I/O read code to read from the page cache if there are
> cached pages and otherwise go direct to the device.  That would make
> a setup using buffered writes (without or without the dontcache
> flag) and direct I/O reads safe.

Yes, that sounds like a good idea.  Just an idea at this point or have
you tried to implement it?

I'll start looking at associated code, but may slip until next week.

FYI, I mentioned this earlier at one point in this thread but I was
thinking the IOR "hard" benchmark would offer a solid test for
invalidating page cache vs O_DIRECT read when ran against NFS/NFSD
with this NFSD O_DIRECT series applied: which causes NFSD's misaligned
IO to use buffered IO for writes and O_DIRECT for reads.  NFSD issuing
the misaligned write to XFS will force RMW when writing, creating
pages that must be invalidated for any subsequent NFSD read.

Turns out IOR "hard" does in fact fail spectacularly with:
  WARNING: Incorrect data on read (6640830 errors found).

It doesn't fail if the 6th patch in this series isn't used:
https://lore.kernel.org/linux-nfs/20250610205737.63343-7-snitzer@kernel.org/

Could be a bug in that patch but I think it more likely IOR-hard is
teasing out the invalidation race you mentioned at the start.

We're retesting with RWF_SYNC set for the buffered write IO (which
Jeff suggested earlier in this thread).

But your idea seems important to pursue.

I won't be posting a v2 for this series until I can find/fix this
IOR-hard testcase.

Mike