Re: [PATCH] nfs: add 'noextend' option for lock-less 'lost writes' prevention

[Trond Myklebust <trondmy@hammerspace.com>]

On Wed, 2024-06-19 at 11:37 +0300, Sagi Grimberg wrote:
> 
> 
> On 18/06/2024 21:59, Trond Myklebust wrote:
> > Hi Dan,
> > 
> > On Tue, 2024-06-18 at 18:33 +0300, Dan Aloni wrote:
> > > There are some applications that write to predefined non-
> > > overlapping
> > > file offsets from multiple clients and therefore don't need to
> > > rely
> > > on
> > > file locking. However, NFS file system behavior of extending
> > > writes
> > > to
> > > to deal with write fragmentation, causes those clients to corrupt
> > > each
> > > other's data.
> > > 
> > > To help these applications, this change adds the `noextend`
> > > parameter
> > > to
> > > the mount options, and handles this case in
> > > `nfs_can_extend_write`.
> > > 
> > > Clients can additionally add the 'noac' option to ensure page
> > > cache
> > > flush on read for modified files.
> > I'm not overly enamoured of the name "noextend". To me that sounds
> > like
> > it might have something to do with preventing appends. Can we find
> > something that is a bit more descriptive?
> 
> nopbw (No page boundary writes) ?
> 
> > 
> > That said, and given your last comment about reads. Wouldn't it be
> > better to have the application use O_DIRECT for these workloads?
> > Turning off attribute caching is both racy and an inefficient way
> > to
> > manage page cache consistency. It forces the client to bombard the
> > server with GETATTR requests in order to check that the page cache
> > is
> > in synch, whereas your description of the workload appears to
> > suggest
> > that the correct assumption should be that it is not in synch.
> > 
> > IOW: I'm asking if the better solution might not be to rather
> > implement
> > something akin to Solaris' "forcedirectio"?
> 
> This access pattern represents a common case in HPC where different
> workers
> write records to a shared output file which do not necessarily align
> to 
> a page boundary.
> 
> This is not everything that the app is doing nor the only file it is 
> accessing, so IMO forcing
> directio universally is may penalize the application.

Worse than forcing an attribute revalidation on every read?

BTW: We've been asked about the same issue from some of our customers,
and are planning on solving the problem by adding a new per-file
attribute to the NFSv4.2 protocol.

The detection of that NOCACHE attribute would cause the client to
automatically choose O_DIRECT on file open, overriding the default
buffered I/O model. So this would allow the user or sysadmin to specify
at file creation time that this file will be used for purposes that are
incompatible with caching.

If set on a directory, the same attribute would cause the client not to
cache the READDIR contents. This is useful when dealing with
directories where a Windows sysadmin may have set an Access Based
Enumeration property.

-- 
Trond Myklebust
Linux NFS client maintainer, Hammerspace
trond.myklebust@hammerspace.com