Re: performance r nfsd with RWF_DONTCACHE and larger wsizes

[Dave Chinner <david@fromorbit.com>]

On Wed, May 07, 2025 at 05:50:14PM -0400, Mike Snitzer wrote:
> Hey Dave,
> 
> Thanks for providing your thoughts on all this.  More inlined below.
> 
> On Wed, May 07, 2025 at 12:50:20PM +1000, Dave Chinner wrote:
> > On Tue, May 06, 2025 at 08:06:51PM -0400, Jeff Layton wrote:
> > > On Wed, 2025-05-07 at 08:31 +1000, Dave Chinner wrote:
> > > > What are the fio control parameters of the IO you are doing? (e.g.
> > > > is this single threaded IO, does it use the psync, libaio or iouring
> > > > engine, etc)
> > > > 
> > > 
> > > 
> > > ; fio-seq-write.job for fiotest
> > > 
> > > [global]
> > > name=fio-seq-write
> > > filename=fio-seq-write
> > > rw=write
> > > bs=256K
> > > direct=0
> > > numjobs=1
> > > time_based
> > > runtime=900
> > > 
> > > [file1]
> > > size=10G
> > > ioengine=libaio
> > > iodepth=16
> >
> > Ok, so we are doing AIO writes on the client side, so we have ~16
> > writes on the wire from the client at any given time.
> 
> Jeff's workload is really underwhelming given he is operating well
> within available memory (so avoiding reclaim, etc).  As such this test
> is really not testing what RWF_DONTCACHE is meant to address (and to
> answer Chuck's question of "what do you hope to get from
> RWF_DONTCACHE?"): the ability to reach steady state where even if
> memory is oversubscribed the network pipes and NVMe devices are as
> close to 100% utilization as possible.

Right.

However, one of the things that has to be kept in mind is that we
don't have 100% of the CPU dedicated to servicing RWF_DONTCACHE IO
like the fio microbenchmarks have.

Applications are going to take a chunk of CPU time to
create/marshall/process the data that we we are doing IO on, so any
time we spend on doing IO is less time that the applications have to
do their work. If you can saturate the storage without saturating
CPUs, then RWF_DONTCACHE should allow that steady state to be
maintained indefinitely.

However, RWF_DONTCACHE does not remove the data copy overhead of
buffered IO, whilst it adds IO submission overhead to each IO. Hence
it will require more CPU time to saturate the storage devices than
normal buffered IO. If you've got CPU to spare, great. If you don't,
then overall performance will be reduced.

> > This also means they are likely not being received by the NFS server
> > in sequential order, and the NFS server is going to be processing
> > roughly 16 write RPCs to the same file concurrently using
> > RWF_DONTCACHE IO.
> > 
> > These are not going to be exactly sequential - the server side IO
> > pattern to the filesystem is quasi-sequential, with random IOs being
> > out of order and leaving temporary holes in the file until the OO
> > write is processed.
> > 
> > XFS should handle this fine via the speculative preallocation beyond
> > EOF that is triggered by extending writes (it was designed to
> > mitigate the fragmentation this NFS behaviour causes). However, we
> > should always keep in mind that while client side IO is sequential,
> > what the server is doing to the underlying filesystem needs to be
> > treated as "concurrent IO to a single file" rather than "sequential
> > IO".
> 
> Hammerspace has definitely seen that 1MB IO coming off the wire is
> fragmented by the time it XFS issues it to underlying storage; so much
> so that IOPs bound devices (e.g. AWS devices that are capped at ~10K
> IOPs) are choking due to all the small IO.

That should not happen in the general case. Can you start a separate
thread to triage the issue so we can try to understand why that is
happening?

> So yeah, minimizing the fragmentation is critical (and largely *not*
> solved at this point... hacks like sync mount from NFS client or using
> O_DIRECT at the client, which sets sync bit, helps reduce the
> fragmentation but as soon as you go full buffered the N=16+ IOs on the
> wire will fragment each other).

Fragmentation mitigation for NFS server IO is generally only
addressable at the filesystem level - it's not really something you
can mitigate at the NFS server or client.

> Do you recommend any particular tuning to help XFS's speculative
> preallocation work for many competing "sequential" IO threads?

I can suggest lots of things, but without knowing the IO pattern,
the fragmentation pattern, the filesystem state, what triggers the
fragmentation, etc, I'd just be guessing as to which knob might make
the problem go away (hence the request to separate that out).

-Dave.
-- 
Dave Chinner
david@fromorbit.com