Re: A comparison of the new nfsd iomodes (and an experimental one)

[Mike Snitzer <snitzer@kernel.org>]

On Fri, Mar 27, 2026 at 09:19:07AM -0400, Chuck Lever wrote:
> On 3/27/26 7:32 AM, Jeff Layton wrote:
> > On Thu, 2026-03-26 at 16:48 -0400, Mike Snitzer wrote:
> >
> >> Your bandwidth for 1MB sequential IO of 793 MB/s for O_DIRECT and
> >> 4,952 MB/s for buffered and dontcache is considerably less than the 72
> >> GB/s offered in Jon's testbed.  Your testing isn't exposing the
> >> bottlenecks (contention) of the MM subsystem for buffered IO... not
> >> yet put my finger on _why_ that is.
> > 
> > That may very well be, but not everyone has a box as large as the one
> > you and Jon were working with.
>
> Right, and this is kind of a blocker for us. Practically speaking, Jon's
> results are not reproducible.
> 
> It would be immensely helpful if the MM-tipover behavior could be
> reproduced on smaller systems. Reduced physical memory size, lower
> network and storage speed, and so on, so that Jeff and I can study the
> issue on our own systems.

Hammerspace still has the same performance lab setup, so we can
certainly reproduce if needed (and try to scale it down, etc) but
unfortunately its currently tied up with other important work. Jon
Flynn's testing was "only" with 2 systems (the NFS server has 16 fast
NVMe, client system connecting over 400 GbE with RDMA). But I'll see
about shaking a couple systems loose...

Might be useful for us to document the setup of the NFS server side
and give pointers for how to mount from the client system and then run
fio commandline.

I think Jens has a couple beefy servers with lots of fast NVMe. Maybe
he'd be open to testing NFS server scalability when he isn't touring
around the country watching his kid win motorsports events.. JENS!? ;)

Mike