Re: fixing redundant network opens on Linux file creation

[Andreas Dilger <adilger@clusterfs.com>]

On Jan 06, 2003  11:42 -0800, Bryan Henderson wrote:
> >There is a lookup intent patch from lustre group. It can be found
> >somewhere in the archives. Pushing that (or something along that lines)
> >to mainline and using that would be IMHO most beneficial
> 
> "Intent," as it's generally understood, is not a promise of future activity
> -- it's either a hint to improve efficiency or it's a promise to restrict
> future activity, but it should be possible simply to bail out before
> exercising that intent.  E.g. you can't open a file at the same time as it
> is looked up just because the looker upper says he intends to open it
> later.

In our code, the lookup-with-intent actually performs both of the operations
on the server, and it is up to the client methods to detect that the operation
was done and deal with it appropriately.  We have very well-tested code for
2.4 and 2.5 code is mostly functional (2.5 is a lot neater implementation but
the changes mean that it isn't yet as functional as the 2.4 code).

In the Lustre code, the premise is that the lookup-with-intent operation
(called lookup2 for now) does one of:
1) the lookup + operation on the server in one RPC (i.e. lookup+create[+open],
   lookup+unlink, lookup+rename) and tells the client "I just did this
   for you, here are the attributes of the new entry and a lock on it if
   necessary, please fix up your local state to match", and the actual VFS
   operations are only doing the post-facto state cleanup.

2) OR it returns a lock to the client that grants the client exclusive
   control over the item(s) in question (normally the parent dir(s)) and
   lets the client do the operations locally and send the operations to
   the server separately.

We currently implement (1) only right now, but the goal is to implement
(2) in the future (which would be back to nearly what the VFS currently
does, except that we are now granted the locks in advance) so that
a client can do many operations locally without the need for getting
lots of locks.  For example, in the future, a Lustre client creating a new
directory could be granted the lock on that directory, and it could then
create files in that directory without further RPCs to the server very
efficiently (e.g.  untarring a file) until another client revokes the
lock(s) and forces the client to flush all of its updates to the server.

For an updated version of the intent patch, see:

ftp://ftp.lustre.org/pub/kernels/patches/2.4.18-hp1_pnnl18_l5.patch
ftp://ftp.lustre.org/pub/kernels/patches/37chaos-l5.patch

The first patch is good for vanilla kernels, and the second for RH
2.4.18-17ish kernels.  There is a bit of extra stuff therein which
isn't really related to the intent changes.

Cheers, Andreas

PS - I've added lustre-devel to this thread so that the Lustre developers
     also see any discussion related to the intent changes.
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/