when should the client request a directory delegation?

when should the client request a directory delegation?

[Jeff Layton]

I've started work on a patchset to add support for directory delegations
to the Linux kernel client and server. It's still too rough to post at
this point, and for now, I'm just cobbling in a ioctl to drive it.

As I started working on some of the client bits, however, I realized
that I don't really have a clear picture as to when the client should
request a delegation on a directory. It seems like there are a lot of
things we could do:

One idea: request one on an initial directory readdir. So maybe when the
offset is 0 and we don't have a dir delegation already, do:

	PUTFH:GET_DIR_DELEGATION:READDIR

Or, maybe just do it on any readdir when we haven't requested one in a
little while?

We could also do one on every lookup, when we expect that the result
will be a directory. I'm not sure if LOOKUP_DIRECTORY would be a
sufficient indicator or if we'd need the vfs to indicate that with a new
flag.

Would we also want to request one after a mkdir?

	PUTFH:CREATE:GET_DIR_DELEGATION:GETFH:GET_DIR_DELEGATION:...

Assuming we can get this all working, what should drive the client to
issues GET_DIR_DELEGATION ops?
-- 
Jeff Layton <jlayton@kernel.org>

Re: when should the client request a directory delegation?

[Trond Myklebust]

On Wed, 2024-02-07 at 08:34 -0500, Jeff Layton wrote:
> I've started work on a patchset to add support for directory
> delegations
> to the Linux kernel client and server. It's still too rough to post
> at
> this point, and for now, I'm just cobbling in a ioctl to drive it.
> 
> As I started working on some of the client bits, however, I realized
> that I don't really have a clear picture as to when the client should
> request a delegation on a directory. It seems like there are a lot of
> things we could do:
> 
> One idea: request one on an initial directory readdir. So maybe when
> the
> offset is 0 and we don't have a dir delegation already, do:
> 
> 	PUTFH:GET_DIR_DELEGATION:READDIR
> 
> Or, maybe just do it on any readdir when we haven't requested one in
> a
> little while?
> 
> We could also do one on every lookup, when we expect that the result
> will be a directory. I'm not sure if LOOKUP_DIRECTORY would be a
> sufficient indicator or if we'd need the vfs to indicate that with a
> new
> flag.
> 
> Would we also want to request one after a mkdir?
> 
> 	PUTFH:CREATE:GET_DIR_DELEGATION:GETFH:GET_DIR_DELEGATION:...
> 
> Assuming we can get this all working, what should drive the client to
> issues GET_DIR_DELEGATION ops?

As far as I'm concerned, the main case to be made for directory
delegations in the client is for reducing the number of revalidations
on said directory, particularly during path lookups.
i.e. the goal is to eliminate the need to constantly poll the directory
change attribute, and to eliminate the need to constantly revalidate
the dentries (and negative dentries!) contained in the directory after
a change.

Perhaps that means we should focus on adding a request for a directory
delegation to the function nfs_lookup_revalidate() since that would
seem to indicate that we're going through the same directory multiple
times? The other call site to consider would be nfs_check_verifier().

-- 
Trond Myklebust
Linux NFS client maintainer, Hammerspace
trond.myklebust@hammerspace.com

Re: when should the client request a directory delegation?

[Jeff Layton]

On Wed, 2024-02-07 at 14:21 +0000, Trond Myklebust wrote:
> On Wed, 2024-02-07 at 08:34 -0500, Jeff Layton wrote:
> > I've started work on a patchset to add support for directory
> > delegations
> > to the Linux kernel client and server. It's still too rough to post
> > at
> > this point, and for now, I'm just cobbling in a ioctl to drive it.
> > 
> > As I started working on some of the client bits, however, I realized
> > that I don't really have a clear picture as to when the client should
> > request a delegation on a directory. It seems like there are a lot of
> > things we could do:
> > 
> > One idea: request one on an initial directory readdir. So maybe when
> > the
> > offset is 0 and we don't have a dir delegation already, do:
> > 
> > 	PUTFH:GET_DIR_DELEGATION:READDIR
> > 
> > Or, maybe just do it on any readdir when we haven't requested one in
> > a
> > little while?
> > 
> > We could also do one on every lookup, when we expect that the result
> > will be a directory. I'm not sure if LOOKUP_DIRECTORY would be a
> > sufficient indicator or if we'd need the vfs to indicate that with a
> > new
> > flag.
> > 
> > Would we also want to request one after a mkdir?
> > 
> > 	PUTFH:CREATE:GET_DIR_DELEGATION:GETFH:GET_DIR_DELEGATION:...
> > 
> > Assuming we can get this all working, what should drive the client to
> > issues GET_DIR_DELEGATION ops?
> 
> As far as I'm concerned, the main case to be made for directory
> delegations in the client is for reducing the number of revalidations
> on said directory, particularly during path lookups.
> i.e. the goal is to eliminate the need to constantly poll the directory
> change attribute, and to eliminate the need to constantly revalidate
> the dentries (and negative dentries!) contained in the directory after
> a change.
> 
> Perhaps that means we should focus on adding a request for a directory
> delegation to the function nfs_lookup_revalidate() since that would
> seem to indicate that we're going through the same directory multiple
> times? The other call site to consider would be nfs_check_verifier().
> 

Sounds good. I'm not nearly at the point where I'm modifying client
behavior yet, but I'll plan to try wiring it up in the revalidate
codepaths first.
-- 
Jeff Layton <jlayton@kernel.org>

Re: when should the client request a directory delegation?

[Trond Myklebust]

On Wed, 2024-02-07 at 09:56 -0500, Jeff Layton wrote:
> On Wed, 2024-02-07 at 14:21 +0000, Trond Myklebust wrote:
> > On Wed, 2024-02-07 at 08:34 -0500, Jeff Layton wrote:
> > > I've started work on a patchset to add support for directory
> > > delegations
> > > to the Linux kernel client and server. It's still too rough to
> > > post
> > > at
> > > this point, and for now, I'm just cobbling in a ioctl to drive
> > > it.
> > > 
> > > As I started working on some of the client bits, however, I
> > > realized
> > > that I don't really have a clear picture as to when the client
> > > should
> > > request a delegation on a directory. It seems like there are a
> > > lot of
> > > things we could do:
> > > 
> > > One idea: request one on an initial directory readdir. So maybe
> > > when
> > > the
> > > offset is 0 and we don't have a dir delegation already, do:
> > > 
> > > 	PUTFH:GET_DIR_DELEGATION:READDIR
> > > 
> > > Or, maybe just do it on any readdir when we haven't requested one
> > > in
> > > a
> > > little while?
> > > 
> > > We could also do one on every lookup, when we expect that the
> > > result
> > > will be a directory. I'm not sure if LOOKUP_DIRECTORY would be a
> > > sufficient indicator or if we'd need the vfs to indicate that
> > > with a
> > > new
> > > flag.
> > > 
> > > Would we also want to request one after a mkdir?
> > > 
> > > 	PUTFH:CREATE:GET_DIR_DELEGATION:GETFH:GET_DIR_DELEGATION
> > > :...
> > > 
> > > Assuming we can get this all working, what should drive the
> > > client to
> > > issues GET_DIR_DELEGATION ops?
> > 
> > As far as I'm concerned, the main case to be made for directory
> > delegations in the client is for reducing the number of
> > revalidations
> > on said directory, particularly during path lookups.
> > i.e. the goal is to eliminate the need to constantly poll the
> > directory
> > change attribute, and to eliminate the need to constantly
> > revalidate
> > the dentries (and negative dentries!) contained in the directory
> > after
> > a change.
> > 
> > Perhaps that means we should focus on adding a request for a
> > directory
> > delegation to the function nfs_lookup_revalidate() since that would
> > seem to indicate that we're going through the same directory
> > multiple
> > times? The other call site to consider would be
> > nfs_check_verifier().
> > 
> 
> Sounds good. I'm not nearly at the point where I'm modifying client
> behavior yet, but I'll plan to try wiring it up in the revalidate
> codepaths first.

Understood, but you appeared to be asking which COMPOUNDs to modify. I
think a discussion around the goals of introducing directory
delegations needs to inform that choice.

-- 
Trond Myklebust
Linux NFS client maintainer, Hammerspace
trond.myklebust@hammerspace.com

Re: when should the client request a directory delegation?

[Jeff Layton]

On Wed, 2024-02-07 at 15:02 +0000, Trond Myklebust wrote:
> On Wed, 2024-02-07 at 09:56 -0500, Jeff Layton wrote:
> > On Wed, 2024-02-07 at 14:21 +0000, Trond Myklebust wrote:
> > > On Wed, 2024-02-07 at 08:34 -0500, Jeff Layton wrote:
> > > > I've started work on a patchset to add support for directory
> > > > delegations
> > > > to the Linux kernel client and server. It's still too rough to
> > > > post
> > > > at
> > > > this point, and for now, I'm just cobbling in a ioctl to drive
> > > > it.
> > > > 
> > > > As I started working on some of the client bits, however, I
> > > > realized
> > > > that I don't really have a clear picture as to when the client
> > > > should
> > > > request a delegation on a directory. It seems like there are a
> > > > lot of
> > > > things we could do:
> > > > 
> > > > One idea: request one on an initial directory readdir. So maybe
> > > > when
> > > > the
> > > > offset is 0 and we don't have a dir delegation already, do:
> > > > 
> > > > 	PUTFH:GET_DIR_DELEGATION:READDIR
> > > > 
> > > > Or, maybe just do it on any readdir when we haven't requested one
> > > > in
> > > > a
> > > > little while?
> > > > 
> > > > We could also do one on every lookup, when we expect that the
> > > > result
> > > > will be a directory. I'm not sure if LOOKUP_DIRECTORY would be a
> > > > sufficient indicator or if we'd need the vfs to indicate that
> > > > with a
> > > > new
> > > > flag.
> > > > 
> > > > Would we also want to request one after a mkdir?
> > > > 
> > > > 	PUTFH:CREATE:GET_DIR_DELEGATION:GETFH:GET_DIR_DELEGATION
> > > > :...
> > > > 
> > > > Assuming we can get this all working, what should drive the
> > > > client to
> > > > issues GET_DIR_DELEGATION ops?
> > > 
> > > As far as I'm concerned, the main case to be made for directory
> > > delegations in the client is for reducing the number of
> > > revalidations
> > > on said directory, particularly during path lookups.
> > > i.e. the goal is to eliminate the need to constantly poll the
> > > directory
> > > change attribute, and to eliminate the need to constantly
> > > revalidate
> > > the dentries (and negative dentries!) contained in the directory
> > > after
> > > a change.
> > > 
> > > Perhaps that means we should focus on adding a request for a
> > > directory
> > > delegation to the function nfs_lookup_revalidate() since that would
> > > seem to indicate that we're going through the same directory
> > > multiple
> > > times? The other call site to consider would be
> > > nfs_check_verifier().
> > > 
> > 
> > Sounds good. I'm not nearly at the point where I'm modifying client
> > behavior yet, but I'll plan to try wiring it up in the revalidate
> > codepaths first.
> 
> Understood, but you appeared to be asking which COMPOUNDs to modify. I
> think a discussion around the goals of introducing directory
> delegations needs to inform that choice.
> 

The goal is to improve lookup performance, and reduce the GETATTR load
on directories. What sort of userland behavior should trigger the client
to get a dir_deleg? Trying to improve repeated lookups in the same dir
does seem like the biggest initial win for this.

Plumbing one into readdir might also be reasonable. Someone doing a
readdir is probably interested in the contents. If we get a deleg first,
then that might allow the client to mark the directory "complete" once
it has read every entry.
-- 
Jeff Layton <jlayton@kernel.org>

Re: when should the client request a directory delegation?

[Trond Myklebust]

On Wed, 2024-02-07 at 14:21 +0000, Trond Myklebust wrote:
> On Wed, 2024-02-07 at 08:34 -0500, Jeff Layton wrote:
> > I've started work on a patchset to add support for directory
> > delegations
> > to the Linux kernel client and server. It's still too rough to post
> > at
> > this point, and for now, I'm just cobbling in a ioctl to drive it.
> > 
> > As I started working on some of the client bits, however, I
> > realized
> > that I don't really have a clear picture as to when the client
> > should
> > request a delegation on a directory. It seems like there are a lot
> > of
> > things we could do:
> > 
> > One idea: request one on an initial directory readdir. So maybe
> > when
> > the
> > offset is 0 and we don't have a dir delegation already, do:
> > 
> > 	PUTFH:GET_DIR_DELEGATION:READDIR
> > 
> > Or, maybe just do it on any readdir when we haven't requested one
> > in
> > a
> > little while?
> > 
> > We could also do one on every lookup, when we expect that the
> > result
> > will be a directory. I'm not sure if LOOKUP_DIRECTORY would be a
> > sufficient indicator or if we'd need the vfs to indicate that with
> > a
> > new
> > flag.
> > 
> > Would we also want to request one after a mkdir?
> > 
> > 	PUTFH:CREATE:GET_DIR_DELEGATION:GETFH:GET_DIR_DELEGATION:.
> > ..
> > 
> > Assuming we can get this all working, what should drive the client
> > to
> > issues GET_DIR_DELEGATION ops?
> 
> As far as I'm concerned, the main case to be made for directory
> delegations in the client is for reducing the number of revalidations
> on said directory, particularly during path lookups.
> i.e. the goal is to eliminate the need to constantly poll the
> directory
> change attribute, and to eliminate the need to constantly revalidate
> the dentries (and negative dentries!) contained in the directory
> after
> a change.
> 
> Perhaps that means we should focus on adding a request for a
> directory
> delegation to the function nfs_lookup_revalidate() since that would
> seem to indicate that we're going through the same directory multiple
> times? The other call site to consider would be nfs_check_verifier().

Note: if you disagree with the above argument, and think that improving
that caching READDIR of results is more important, then consider the
whole discussion we had in the thread started by Tigran here:
https://lore.kernel.org/all/CAGue13pe_ZH_Eto-jL3mLjTNGFK26izTarnZdjs3eL82A2Z37w@mail.gmail.com/

-- 
Trond Myklebust
Linux NFS client maintainer, Hammerspace
trond.myklebust@hammerspace.com

Re: when should the client request a directory delegation?

[Jeff Layton]

On Wed, 2024-02-07 at 14:56 +0000, Trond Myklebust wrote:
> On Wed, 2024-02-07 at 14:21 +0000, Trond Myklebust wrote:
> > On Wed, 2024-02-07 at 08:34 -0500, Jeff Layton wrote:
> > > I've started work on a patchset to add support for directory
> > > delegations
> > > to the Linux kernel client and server. It's still too rough to post
> > > at
> > > this point, and for now, I'm just cobbling in a ioctl to drive it.
> > > 
> > > As I started working on some of the client bits, however, I
> > > realized
> > > that I don't really have a clear picture as to when the client
> > > should
> > > request a delegation on a directory. It seems like there are a lot
> > > of
> > > things we could do:
> > > 
> > > One idea: request one on an initial directory readdir. So maybe
> > > when
> > > the
> > > offset is 0 and we don't have a dir delegation already, do:
> > > 
> > > 	PUTFH:GET_DIR_DELEGATION:READDIR
> > > 
> > > Or, maybe just do it on any readdir when we haven't requested one
> > > in
> > > a
> > > little while?
> > > 
> > > We could also do one on every lookup, when we expect that the
> > > result
> > > will be a directory. I'm not sure if LOOKUP_DIRECTORY would be a
> > > sufficient indicator or if we'd need the vfs to indicate that with
> > > a
> > > new
> > > flag.
> > > 
> > > Would we also want to request one after a mkdir?
> > > 
> > > 	PUTFH:CREATE:GET_DIR_DELEGATION:GETFH:GET_DIR_DELEGATION:.
> > > ..
> > > 
> > > Assuming we can get this all working, what should drive the client
> > > to
> > > issues GET_DIR_DELEGATION ops?
> > 
> > As far as I'm concerned, the main case to be made for directory
> > delegations in the client is for reducing the number of revalidations
> > on said directory, particularly during path lookups.
> > i.e. the goal is to eliminate the need to constantly poll the
> > directory
> > change attribute, and to eliminate the need to constantly revalidate
> > the dentries (and negative dentries!) contained in the directory
> > after
> > a change.
> > 
> > Perhaps that means we should focus on adding a request for a
> > directory
> > delegation to the function nfs_lookup_revalidate() since that would
> > seem to indicate that we're going through the same directory multiple
> > times? The other call site to consider would be nfs_check_verifier().
> 
> Note: if you disagree with the above argument, and think that improving
> that caching READDIR of results is more important, then consider the
> whole discussion we had in the thread started by Tigran here:
> https://lore.kernel.org/all/CAGue13pe_ZH_Eto-jL3mLjTNGFK26izTarnZdjs3eL82A2Z37w@mail.gmail.com/
> 

I think you're correct about the main use-case. I certainly don't plan
to try for readdir caching in the initial stages (there's enough to do
just to get to a minimum-viable feature).  Long term though, it might be
possible to do something:

The ceph protocol has had directory caps for a long time. The client can
mark directories as "complete" when it has every dentry in the
directory, and also as "ordered" when they are correctly ordered in the
dcache.

So if it has the right caps, "complete" allows them to satisfy any
lookup in the dir w/o going to the MDS, and "ordered" allows them to
satisfy readdir using dcache_readdir() (also w/o going to the MDS).

It may be possible for us to leverage dir delegations to do something
like that in NFS too.

-- 
Jeff Layton <jlayton@kernel.org>