NFS delegations behavior analysis

NFS delegations behavior analysis

[Piyush Sachdeva]

Hi,
Lately I have been running micro benchmarks around the `ls` command and
reading through the code documentation of the NFS client to better
understand the client side caching behavior with and without
delegations.

Understanding so far:
Delegations (both file and directory) are granted by the server to the
client, indefinitely (until revoked or under the watermark) to cache
attributes. The caching of data is a result of the attribute
cache. Hence forth, a directory delegation will cache the directory
attributes and the names of the files in the directory, and a file
delegation will cache the attributes of the file and the file data.

Workload run:
I focused on the 2 workloads below, doing 2 passes of a large flat
directory (with close to 100K files) - 
a cold pass, and warm pass using the cache from the cold pass:
- lslr - ls -lR on both runs
- lsmix - ls -R (cold) and then ls -lR (warm)

I also played with the rdirplus behavior using both the default
heuristic behavior and the `rdirplus=force` set at mount time.

Numbers:
actimeo=5s, rdirplus=force, ACLs off, flat_dir
==================================================================

                 |         LSLR          |         LSMIX
                 |  (ls -lR cold / warm) |  (p1 ls -R / p2 ls -lR)
Operation        |  flat cold  | flat warm |   flat p1   | flat p2
-----------------+-------------+-----------+-------------+---------
READDIR calls    |    27       |     0     |   27        |    0
READDIR recv B   | 23,603,024  |     0     | 23,603,024  |    0
   call type     | readdirplus |    --     | readdirplus |    --
LOOKUP           |     1       |     0     |    1        |    0
GETATTR          |     3       |  100,000  |    2        | 100,001
ACCESS           |     2       |     0     |    2        |    0
-----------------+-------------+-----------+-------------+---------
Elapsed (age)    |  ~14 s      |  ~62 s    |   ~16 s     |  ~63 s


Observations:
When doing `ls` or `ls -l` on a directory, due to the open(2) on the
directory, the client gets a directory delegation - caching the
directory attributes and file names. However, as we don't have file
delegations due to no open(2) calls to any of the files. Henceforth,
the cache of file attributes is governed by `actimeo`.
Now here is the interesting bit, if the next `ls -l` is issued after
the `actimeo`, a massive GETATTR storm hits the server, doing stat()
calls for every file in the directory. As a result, the performance of
this warm `ls -l` run ends up being worse than the cold pass. I am
guessing this is most likely due to the compounded "rdirplus" being more
efficient than stat() calls.


Proposal:
For large directories, this ends up being a massive problem, taking 1-2
minutes when enumerating a directory on the warm passes.
- An easier way to tackle this could be to do a rdirplus=[auto | forced]
  instead of issuing the stat(2) storm to the server: When the client
  notices that there are cache misses, which would be the case of file
  attributes, instead of fetching file names from the directory-delegation
  cache and attributes from GETATTR, the client does a READDIRPLUS to
  the server, nonetheless.
- A more tedious would be the to cache file attributes as well, as a part
  of the directory delegation. This would end up requiring a change in the
  NFS protocol spec though.
- Bulk GETATTR calls: I am uncertain of the feasibility of this, but
  what if, the client could do 1 GETATTR call for getting attributes
  for multiple files. 

-- 
Piyush

Re: NFS delegations behavior analysis

[Jeff Layton]

On Tue, 2026-06-23 at 15:31 +0530, Piyush Sachdeva wrote:
> Hi,
> Lately I have been running micro benchmarks around the `ls` command and
> reading through the code documentation of the NFS client to better
> understand the client side caching behavior with and without
> delegations.
> 
> Understanding so far:
> Delegations (both file and directory) are granted by the server to the
> client, indefinitely (until revoked or under the watermark) to cache
> attributes. The caching of data is a result of the attribute
> cache. Hence forth, a directory delegation will cache the directory
> attributes and the names of the files in the directory, and a file
> delegation will cache the attributes of the file and the file data.
> 
> Workload run:
> I focused on the 2 workloads below, doing 2 passes of a large flat
> directory (with close to 100K files) - 
> a cold pass, and warm pass using the cache from the cold pass:
> - lslr - ls -lR on both runs
> - lsmix - ls -R (cold) and then ls -lR (warm)
> 
> I also played with the rdirplus behavior using both the default
> heuristic behavior and the `rdirplus=force` set at mount time.
> 
> Numbers:
> actimeo=5s, rdirplus=force, ACLs off, flat_dir
> ==================================================================
> 
>                  |         LSLR          |         LSMIX
>                  |  (ls -lR cold / warm) |  (p1 ls -R / p2 ls -lR)
> Operation        |  flat cold  | flat warm |   flat p1   | flat p2
> -----------------+-------------+-----------+-------------+---------
> READDIR calls    |    27       |     0     |   27        |    0
> READDIR recv B   | 23,603,024  |     0     | 23,603,024  |    0
>    call type     | readdirplus |    --     | readdirplus |    --
> LOOKUP           |     1       |     0     |    1        |    0
> GETATTR          |     3       |  100,000  |    2        | 100,001
> ACCESS           |     2       |     0     |    2        |    0
> -----------------+-------------+-----------+-------------+---------
> Elapsed (age)    |  ~14 s      |  ~62 s    |   ~16 s     |  ~63 s
> 
> 
> Observations:
> When doing `ls` or `ls -l` on a directory, due to the open(2) on the
> directory, the client gets a directory delegation - caching the
> directory attributes and file names. However, as we don't have file
> delegations due to no open(2) calls to any of the files. Henceforth,
> the cache of file attributes is governed by `actimeo`.
> Now here is the interesting bit, if the next `ls -l` is issued after
> the `actimeo`, a massive GETATTR storm hits the server, doing stat()
> calls for every file in the directory. As a result, the performance of
> this warm `ls -l` run ends up being worse than the cold pass. I am
> guessing this is most likely due to the compounded "rdirplus" being more
> efficient than stat() calls.
> 
> 
> Proposal:
> For large directories, this ends up being a massive problem, taking 1-2
> minutes when enumerating a directory on the warm passes.
> - An easier way to tackle this could be to do a rdirplus=[auto | forced]
>   instead of issuing the stat(2) storm to the server: When the client
>   notices that there are cache misses, which would be the case of file
>   attributes, instead of fetching file names from the directory-delegation
>   cache and attributes from GETATTR, the client does a READDIRPLUS to
>   the server, nonetheless.
> - A more tedious would be the to cache file attributes as well, as a part
>   of the directory delegation. This would end up requiring a change in the
>   NFS protocol spec though.
> - Bulk GETATTR calls: I am uncertain of the feasibility of this, but
>   what if, the client could do 1 GETATTR call for getting attributes
>   for multiple files. 


ls is such a hard workload to get right, because we don't really get an
indication in the kernel of what userland's intentions are. It's
basically a readdir() call followed by a bunch of stat()'s, but at the
point where we're getting the readdir() call, we don't know if userland
intends to stat() those files or not. We have to make a guess about
that intention.

In this case, it sounds like the directory cache was valid, so the
client decided it didn't need to do a READDIR at all, but the
individual files had caches that timed out.

So imagine you're the kernel client and have been given that second
readdir() call: Why should you decide to do a READDIRPLUS at that point
instead of a regular READDIR?
-- 
Jeff Layton <jlayton@kernel.org>

Re: NFS delegations behavior analysis

[Mkrtchyan, Tigran]

[-- Attachment #1: Type: text/plain, Size: 5344 bytes --]



----- Original Message -----
> From: "Jeff Layton" <jlayton@kernel.org>
> To: "Piyush Sachdeva" <s.piyush1024@gmail.com>, "linux-nfs" <linux-nfs@vger.kernel.org>, "Chuck Lever" <cel@kernel.org>,
> "trondmy" <trondmy@kernel.org>, sfrench@samba.org, sprasad@microsoft.com
> Cc: vaibsharma@microsoft.com
> Sent: Tuesday, 23 June, 2026 12:50:16
> Subject: Re: NFS delegations behavior analysis

> On Tue, 2026-06-23 at 15:31 +0530, Piyush Sachdeva wrote:
>> Hi,
>> Lately I have been running micro benchmarks around the `ls` command and
>> reading through the code documentation of the NFS client to better
>> understand the client side caching behavior with and without
>> delegations.
>> 
>> Understanding so far:
>> Delegations (both file and directory) are granted by the server to the
>> client, indefinitely (until revoked or under the watermark) to cache
>> attributes. The caching of data is a result of the attribute
>> cache. Hence forth, a directory delegation will cache the directory
>> attributes and the names of the files in the directory, and a file
>> delegation will cache the attributes of the file and the file data.
>> 
>> Workload run:
>> I focused on the 2 workloads below, doing 2 passes of a large flat
>> directory (with close to 100K files) -
>> a cold pass, and warm pass using the cache from the cold pass:
>> - lslr - ls -lR on both runs
>> - lsmix - ls -R (cold) and then ls -lR (warm)
>> 
>> I also played with the rdirplus behavior using both the default
>> heuristic behavior and the `rdirplus=force` set at mount time.
>> 
>> Numbers:
>> actimeo=5s, rdirplus=force, ACLs off, flat_dir
>> ==================================================================
>> 
>>                  |         LSLR          |         LSMIX
>>                  |  (ls -lR cold / warm) |  (p1 ls -R / p2 ls -lR)
>> Operation        |  flat cold  | flat warm |   flat p1   | flat p2
>> -----------------+-------------+-----------+-------------+---------
>> READDIR calls    |    27       |     0     |   27        |    0
>> READDIR recv B   | 23,603,024  |     0     | 23,603,024  |    0
>>    call type     | readdirplus |    --     | readdirplus |    --
>> LOOKUP           |     1       |     0     |    1        |    0
>> GETATTR          |     3       |  100,000  |    2        | 100,001
>> ACCESS           |     2       |     0     |    2        |    0
>> -----------------+-------------+-----------+-------------+---------
>> Elapsed (age)    |  ~14 s      |  ~62 s    |   ~16 s     |  ~63 s
>> 
>> 
>> Observations:
>> When doing `ls` or `ls -l` on a directory, due to the open(2) on the
>> directory, the client gets a directory delegation - caching the
>> directory attributes and file names. However, as we don't have file
>> delegations due to no open(2) calls to any of the files. Henceforth,
>> the cache of file attributes is governed by `actimeo`.
>> Now here is the interesting bit, if the next `ls -l` is issued after
>> the `actimeo`, a massive GETATTR storm hits the server, doing stat()
>> calls for every file in the directory. As a result, the performance of
>> this warm `ls -l` run ends up being worse than the cold pass. I am
>> guessing this is most likely due to the compounded "rdirplus" being more
>> efficient than stat() calls.
>> 
>> 
>> Proposal:
>> For large directories, this ends up being a massive problem, taking 1-2
>> minutes when enumerating a directory on the warm passes.
>> - An easier way to tackle this could be to do a rdirplus=[auto | forced]
>>   instead of issuing the stat(2) storm to the server: When the client
>>   notices that there are cache misses, which would be the case of file
>>   attributes, instead of fetching file names from the directory-delegation
>>   cache and attributes from GETATTR, the client does a READDIRPLUS to
>>   the server, nonetheless.
>> - A more tedious would be the to cache file attributes as well, as a part
>>   of the directory delegation. This would end up requiring a change in the
>>   NFS protocol spec though.
>> - Bulk GETATTR calls: I am uncertain of the feasibility of this, but
>>   what if, the client could do 1 GETATTR call for getting attributes
>>   for multiple files.
> 
> 
> ls is such a hard workload to get right, because we don't really get an

100% agree. And there were a couple of attempts to address this issue
(second ls that is slow).

> indication in the kernel of what userland's intentions are. It's
> basically a readdir() call followed by a bunch of stat()'s, but at the
> point where we're getting the readdir() call, we don't know if userland
> intends to stat() those files or not. We have to make a guess about
> that intention.
> 
> In this case, it sounds like the directory cache was valid, so the
> client decided it didn't need to do a READDIR at all, but the
> individual files had caches that timed out.
> 
> So imagine you're the kernel client and have been given that second
> readdir() call: Why should you decide to do a READDIRPLUS at that point
> instead of a regular READDIR?

May we need some kind of client-side heuristics, like on the server side
for open-delegations, where after seeing some `stats` for files in the
In the same directory, the client will decide to switch to READDIR (v4)
to get all attributes in one go.

Best regards,
   Tigran.

> --
> Jeff Layton <jlayton@kernel.org>

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 2309 bytes --]

Re: NFS delegations behavior analysis

[Jeff Layton]

On Tue, 2026-06-23 at 13:04 +0200, Mkrtchyan, Tigran wrote:
> 
> ----- Original Message -----
> > From: "Jeff Layton" <jlayton@kernel.org>
> > To: "Piyush Sachdeva" <s.piyush1024@gmail.com>, "linux-nfs" <linux-nfs@vger.kernel.org>, "Chuck Lever" <cel@kernel.org>,
> > "trondmy" <trondmy@kernel.org>, sfrench@samba.org, sprasad@microsoft.com
> > Cc: vaibsharma@microsoft.com
> > Sent: Tuesday, 23 June, 2026 12:50:16
> > Subject: Re: NFS delegations behavior analysis
> 
> > On Tue, 2026-06-23 at 15:31 +0530, Piyush Sachdeva wrote:
> > > Hi,
> > > Lately I have been running micro benchmarks around the `ls` command and
> > > reading through the code documentation of the NFS client to better
> > > understand the client side caching behavior with and without
> > > delegations.
> > > 
> > > Understanding so far:
> > > Delegations (both file and directory) are granted by the server to the
> > > client, indefinitely (until revoked or under the watermark) to cache
> > > attributes. The caching of data is a result of the attribute
> > > cache. Hence forth, a directory delegation will cache the directory
> > > attributes and the names of the files in the directory, and a file
> > > delegation will cache the attributes of the file and the file data.
> > > 
> > > Workload run:
> > > I focused on the 2 workloads below, doing 2 passes of a large flat
> > > directory (with close to 100K files) -
> > > a cold pass, and warm pass using the cache from the cold pass:
> > > - lslr - ls -lR on both runs
> > > - lsmix - ls -R (cold) and then ls -lR (warm)
> > > 
> > > I also played with the rdirplus behavior using both the default
> > > heuristic behavior and the `rdirplus=force` set at mount time.
> > > 
> > > Numbers:
> > > actimeo=5s, rdirplus=force, ACLs off, flat_dir
> > > ==================================================================
> > > 
> > >                  |         LSLR          |         LSMIX
> > >                  |  (ls -lR cold / warm) |  (p1 ls -R / p2 ls -lR)
> > > Operation        |  flat cold  | flat warm |   flat p1   | flat p2
> > > -----------------+-------------+-----------+-------------+---------
> > > READDIR calls    |    27       |     0     |   27        |    0
> > > READDIR recv B   | 23,603,024  |     0     | 23,603,024  |    0
> > >    call type     | readdirplus |    --     | readdirplus |    --
> > > LOOKUP           |     1       |     0     |    1        |    0
> > > GETATTR          |     3       |  100,000  |    2        | 100,001
> > > ACCESS           |     2       |     0     |    2        |    0
> > > -----------------+-------------+-----------+-------------+---------
> > > Elapsed (age)    |  ~14 s      |  ~62 s    |   ~16 s     |  ~63 s
> > > 
> > > 
> > > Observations:
> > > When doing `ls` or `ls -l` on a directory, due to the open(2) on the
> > > directory, the client gets a directory delegation - caching the
> > > directory attributes and file names. However, as we don't have file
> > > delegations due to no open(2) calls to any of the files. Henceforth,
> > > the cache of file attributes is governed by `actimeo`.
> > > Now here is the interesting bit, if the next `ls -l` is issued after
> > > the `actimeo`, a massive GETATTR storm hits the server, doing stat()
> > > calls for every file in the directory. As a result, the performance of
> > > this warm `ls -l` run ends up being worse than the cold pass. I am
> > > guessing this is most likely due to the compounded "rdirplus" being more
> > > efficient than stat() calls.
> > > 
> > > 
> > > Proposal:
> > > For large directories, this ends up being a massive problem, taking 1-2
> > > minutes when enumerating a directory on the warm passes.
> > > - An easier way to tackle this could be to do a rdirplus=[auto | forced]
> > >   instead of issuing the stat(2) storm to the server: When the client
> > >   notices that there are cache misses, which would be the case of file
> > >   attributes, instead of fetching file names from the directory-delegation
> > >   cache and attributes from GETATTR, the client does a READDIRPLUS to
> > >   the server, nonetheless.
> > > - A more tedious would be the to cache file attributes as well, as a part
> > >   of the directory delegation. This would end up requiring a change in the
> > >   NFS protocol spec though.
> > > - Bulk GETATTR calls: I am uncertain of the feasibility of this, but
> > >   what if, the client could do 1 GETATTR call for getting attributes
> > >   for multiple files.
> > 
> > 
> > ls is such a hard workload to get right, because we don't really get an
> 
> 100% agree. And there were a couple of attempts to address this issue
> (second ls that is slow).
> 
> > indication in the kernel of what userland's intentions are. It's
> > basically a readdir() call followed by a bunch of stat()'s, but at the
> > point where we're getting the readdir() call, we don't know if userland
> > intends to stat() those files or not. We have to make a guess about
> > that intention.
> > 
> > In this case, it sounds like the directory cache was valid, so the
> > client decided it didn't need to do a READDIR at all, but the
> > individual files had caches that timed out.
> > 
> > So imagine you're the kernel client and have been given that second
> > readdir() call: Why should you decide to do a READDIRPLUS at that point
> > instead of a regular READDIR?
> 
> May we need some kind of client-side heuristics, like on the server side
> for open-delegations, where after seeing some `stats` for files in the
> In the same directory, the client will decide to switch to READDIR (v4)
> to get all attributes in one go.
> 

Yeah, we definitely do. I'm just not sure what those heuristics look
like.

I think Ben did the latest pass of trying to tune the heuristics here.
Any thoughts on how we could do this better (and whether there are
particular ls-ish workloads that we don't want to regress)?
-- 
Jeff Layton <jlayton@kernel.org>

Re: NFS delegations behavior analysis

[Anna Schumaker]

On Tue, Jun 23, 2026, at 7:04 AM, Mkrtchyan, Tigran wrote:
> ----- Original Message -----
>> From: "Jeff Layton" <jlayton@kernel.org>
>> To: "Piyush Sachdeva" <s.piyush1024@gmail.com>, "linux-nfs" <linux-nfs@vger.kernel.org>, "Chuck Lever" <cel@kernel.org>,
>> "trondmy" <trondmy@kernel.org>, sfrench@samba.org, sprasad@microsoft.com
>> Cc: vaibsharma@microsoft.com
>> Sent: Tuesday, 23 June, 2026 12:50:16
>> Subject: Re: NFS delegations behavior analysis
>
>> On Tue, 2026-06-23 at 15:31 +0530, Piyush Sachdeva wrote:
>>> Hi,
>>> Lately I have been running micro benchmarks around the `ls` command and
>>> reading through the code documentation of the NFS client to better
>>> understand the client side caching behavior with and without
>>> delegations.
>>> 
>>> Understanding so far:
>>> Delegations (both file and directory) are granted by the server to the
>>> client, indefinitely (until revoked or under the watermark) to cache
>>> attributes. The caching of data is a result of the attribute
>>> cache. Hence forth, a directory delegation will cache the directory
>>> attributes and the names of the files in the directory, and a file
>>> delegation will cache the attributes of the file and the file data.
>>> 
>>> Workload run:
>>> I focused on the 2 workloads below, doing 2 passes of a large flat
>>> directory (with close to 100K files) -
>>> a cold pass, and warm pass using the cache from the cold pass:
>>> - lslr - ls -lR on both runs
>>> - lsmix - ls -R (cold) and then ls -lR (warm)
>>> 
>>> I also played with the rdirplus behavior using both the default
>>> heuristic behavior and the `rdirplus=force` set at mount time.
>>> 
>>> Numbers:
>>> actimeo=5s, rdirplus=force, ACLs off, flat_dir
>>> ==================================================================
>>> 
>>>                  |         LSLR          |         LSMIX
>>>                  |  (ls -lR cold / warm) |  (p1 ls -R / p2 ls -lR)
>>> Operation        |  flat cold  | flat warm |   flat p1   | flat p2
>>> -----------------+-------------+-----------+-------------+---------
>>> READDIR calls    |    27       |     0     |   27        |    0
>>> READDIR recv B   | 23,603,024  |     0     | 23,603,024  |    0
>>>    call type     | readdirplus |    --     | readdirplus |    --
>>> LOOKUP           |     1       |     0     |    1        |    0
>>> GETATTR          |     3       |  100,000  |    2        | 100,001
>>> ACCESS           |     2       |     0     |    2        |    0
>>> -----------------+-------------+-----------+-------------+---------
>>> Elapsed (age)    |  ~14 s      |  ~62 s    |   ~16 s     |  ~63 s
>>> 
>>> 
>>> Observations:
>>> When doing `ls` or `ls -l` on a directory, due to the open(2) on the
>>> directory, the client gets a directory delegation - caching the
>>> directory attributes and file names. However, as we don't have file
>>> delegations due to no open(2) calls to any of the files. Henceforth,
>>> the cache of file attributes is governed by `actimeo`.
>>> Now here is the interesting bit, if the next `ls -l` is issued after
>>> the `actimeo`, a massive GETATTR storm hits the server, doing stat()
>>> calls for every file in the directory. As a result, the performance of
>>> this warm `ls -l` run ends up being worse than the cold pass. I am
>>> guessing this is most likely due to the compounded "rdirplus" being more
>>> efficient than stat() calls.
>>> 
>>> 
>>> Proposal:
>>> For large directories, this ends up being a massive problem, taking 1-2
>>> minutes when enumerating a directory on the warm passes.
>>> - An easier way to tackle this could be to do a rdirplus=[auto | forced]
>>>   instead of issuing the stat(2) storm to the server: When the client
>>>   notices that there are cache misses, which would be the case of file
>>>   attributes, instead of fetching file names from the directory-delegation
>>>   cache and attributes from GETATTR, the client does a READDIRPLUS to
>>>   the server, nonetheless.
>>> - A more tedious would be the to cache file attributes as well, as a part
>>>   of the directory delegation. This would end up requiring a change in the
>>>   NFS protocol spec though.
>>> - Bulk GETATTR calls: I am uncertain of the feasibility of this, but
>>>   what if, the client could do 1 GETATTR call for getting attributes
>>>   for multiple files.
>> 
>> 
>> ls is such a hard workload to get right, because we don't really get an
>
> 100% agree. And there were a couple of attempts to address this issue
> (second ls that is slow).
>
>> indication in the kernel of what userland's intentions are. It's
>> basically a readdir() call followed by a bunch of stat()'s, but at the
>> point where we're getting the readdir() call, we don't know if userland
>> intends to stat() those files or not. We have to make a guess about
>> that intention.
>> 
>> In this case, it sounds like the directory cache was valid, so the
>> client decided it didn't need to do a READDIR at all, but the
>> individual files had caches that timed out.
>> 
>> So imagine you're the kernel client and have been given that second
>> readdir() call: Why should you decide to do a READDIRPLUS at that point
>> instead of a regular READDIR?
>
> May we need some kind of client-side heuristics, like on the server side
> for open-delegations, where after seeing some `stats` for files in the
> In the same directory, the client will decide to switch to READDIR (v4)
> to get all attributes in one go.

We do something like that already. I don't think we'll ever have a readdir
plus heuristic that makes everybody happy without userspace somehow telling
us their intentions. I know a statx-based readdir plus system call probably
sounds crazy, but something like that would go a long way to take all the
guesswork out of things on our end.

Anna

>
> Best regards,
>    Tigran.
>
>> --
>> Jeff Layton <jlayton@kernel.org>
>
> Attachments:
> * smime.p7s

Re: NFS delegations behavior analysis

[Benjamin Coddington]

On 23 Jun 2026, at 7:10, Jeff Layton wrote:

> I think Ben did the latest pass of trying to tune the heuristics here.
> Any thoughts on how we could do this better (and whether there are
> particular ls-ish workloads that we don't want to regress)?

I haven't (shame) thought about READDIR in the context of directory
delegations.

But during my time at Red Hat we worked hard to optimize some readdir
problems and I learned that almost any change we made ended up making
someone's workload regress.  We also found that our performance benchmarks
rarely matched the most common real-world workloads.  We made the mistake of
trying to improve the benchmark which resulted in performance regressions
for real-world users.

Jeff, you've already touched on the core issue regarding fixing this with
bulk GETATTR calls - the kernel doesn't know what syscall pattern the
userspace process is going to use next.  The `ls -l` command and `find` and
friends have complex history and branching logic, they do different lookup
and getattr patterns based on their own goals, and NFS cannot optimize for
any one case.

I think the last time we discussed additional improvements there were some
ideas about teaching the readdir code to respond to fadvise flags, but then
you'd also need to teach the utilities how to use them as well, and those
utilities try to be filesystem-agnostic.

Its a tough problem, and sometimes the simplest thing might be to just use
more directories on NFS.

**coffee!**

.... er - so with directory delegations, can we simply re-hydrate the dentry
cache from the directory page mappings if the delegation is still valid?
Does the directory delegation pin the mapping?  Clearly I need to look at
the code..

Ben

Re: NFS delegations behavior analysis

[Daire Byrne]

On Tue, 23 Jun 2026 at 14:12, Benjamin Coddington
<ben.coddington@hammerspace.com> wrote:
>
> On 23 Jun 2026, at 7:10, Jeff Layton wrote:
>
> > I think Ben did the latest pass of trying to tune the heuristics here.
> > Any thoughts on how we could do this better (and whether there are
> > particular ls-ish workloads that we don't want to regress)?
>
> I haven't (shame) thought about READDIR in the context of directory
> delegations.
>
> But during my time at Red Hat we worked hard to optimize some readdir
> problems and I learned that almost any change we made ended up making
> someone's workload regress.  We also found that our performance benchmarks
> rarely matched the most common real-world workloads.  We made the mistake of
> trying to improve the benchmark which resulted in performance regressions
> for real-world users.
>
> Jeff, you've already touched on the core issue regarding fixing this with
> bulk GETATTR calls - the kernel doesn't know what syscall pattern the
> userspace process is going to use next.  The `ls -l` command and `find` and
> friends have complex history and branching logic, they do different lookup
> and getattr patterns based on their own goals, and NFS cannot optimize for
> any one case.
>
> I think the last time we discussed additional improvements there were some
> ideas about teaching the readdir code to respond to fadvise flags, but then
> you'd also need to teach the utilities how to use them as well, and those
> utilities try to be filesystem-agnostic.
>
> Its a tough problem, and sometimes the simplest thing might be to just use
> more directories on NFS.
>
> **coffee!**
>
> .... er - so with directory delegations, can we simply re-hydrate the dentry
> cache from the directory page mappings if the delegation is still valid?
> Does the directory delegation pin the mapping?  Clearly I need to look at
> the code..
>
> Ben
>

I have also long hoped for a way to cache or speed up "negative"
lookups using something akin to directory delegations.

So if you have your software (e.g. PYTHONPATH) on an NFS share, then
negative lookups are a huge scalability problem. It would be nice if a
client that got the directory contents could also serve negative
lookups until the server told it that something had changed.

But as has been explained to me multiple times before, there is no
mechanism to say that a client "has the complete cache of directory
contents". But I will forever live in hope!

This kind of path search walking is also something that is unlikely to
ever trigger readdirplus as it does a file and then tries the same
file in another directory etc etc.

Daire

Re: NFS delegations behavior analysis

[Benjamin Coddington]

On 23 Jun 2026, at 9:11, Benjamin Coddington wrote:

> .... er - so with directory delegations, can we simply re-hydrate the dentry
> cache from the directory page mappings if the delegation is still valid?
> Does the directory delegation pin the mapping?  Clearly I need to look at
> the code..

.. right - we don't keep the file attributes in the mappings today.  And,
more to the point - the directory delegation doesn't protect those file
attributes either.  We'd need NOTIFY4_CHANGE_CHILD_ATTRIBUTES implemented.

Ben

Re: NFS delegations behavior analysis

[Jeff Layton]

On Tue, 2026-06-23 at 09:11 -0400, Benjamin Coddington wrote:
> On 23 Jun 2026, at 7:10, Jeff Layton wrote:
> 
> > I think Ben did the latest pass of trying to tune the heuristics here.
> > Any thoughts on how we could do this better (and whether there are
> > particular ls-ish workloads that we don't want to regress)?
> 
> I haven't (shame) thought about READDIR in the context of directory
> delegations.
> 
> But during my time at Red Hat we worked hard to optimize some readdir
> problems and I learned that almost any change we made ended up making
> someone's workload regress.  We also found that our performance benchmarks
> rarely matched the most common real-world workloads.  We made the mistake of
> trying to improve the benchmark which resulted in performance regressions
> for real-world users.
> 
> Jeff, you've already touched on the core issue regarding fixing this with
> bulk GETATTR calls - the kernel doesn't know what syscall pattern the
> userspace process is going to use next.  The `ls -l` command and `find` and
> friends have complex history and branching logic, they do different lookup
> and getattr patterns based on their own goals, and NFS cannot optimize for
> any one case.
> 
> I think the last time we discussed additional improvements there were some
> ideas about teaching the readdir code to respond to fadvise flags, but then
> you'd also need to teach the utilities how to use them as well, and those
> utilities try to be filesystem-agnostic.
> 
> Its a tough problem, and sometimes the simplest thing might be to just use
> more directories on NFS.
> 
> **coffee!**
> 
> .... er - so with directory delegations, can we simply re-hydrate the dentry
> cache from the directory page mappings if the delegation is still valid?
> Does the directory delegation pin the mapping?  Clearly I need to look at
> the code..
> 

I think the main problem is not the dcache, but the attributes on the
inodes. The first pass uses a full-attrs READDIR (aka readdirplus) and
everything goes reasonably quick.

Then, on the second pass, the client skips checking directory
attributes before trusting the dcache because of the delegation (which
is good), but the inodes those dentries point to have an attrcache
timeout, and we end up doing a GETATTR for each statx() call when a
readdirplus would have been cheaper.

I guess after you see a few statx() calls from the same pid on the same
directory in a short timeframe you could switch to doing READDIR, but
that sounds horrid to get right and keep working.

-- 
Jeff Layton <jlayton@kernel.org>

Re: NFS delegations behavior analysis

[Jeff Layton]

On Tue, 2026-06-23 at 09:32 -0400, Benjamin Coddington wrote:
> On 23 Jun 2026, at 9:11, Benjamin Coddington wrote:
> 
> > .... er - so with directory delegations, can we simply re-hydrate the dentry
> > cache from the directory page mappings if the delegation is still valid?
> > Does the directory delegation pin the mapping?  Clearly I need to look at
> > the code..
> 
> .. right - we don't keep the file attributes in the mappings today.  And,
> more to the point - the directory delegation doesn't protect those file
> attributes either.  We'd need NOTIFY4_CHANGE_CHILD_ATTRIBUTES implemented.
> 

Which I think could be done, at least on the Linux server. fsnotify
does support watching for child attribute changes (FS_EVENT_ON_CHILD |
FS_ATTRIB).

That could be very chatty though. We would need to do the work to make
nfsd send callbacks >1 page in order to keep up, I imagine.
-- 
Jeff Layton <jlayton@kernel.org>

Re: NFS delegations behavior analysis

[Benjamin Coddington]

On 23 Jun 2026, at 9:40, Jeff Layton wrote:

> On Tue, 2026-06-23 at 09:32 -0400, Benjamin Coddington wrote:
>> On 23 Jun 2026, at 9:11, Benjamin Coddington wrote:
>>
>>> .... er - so with directory delegations, can we simply re-hydrate the dentry
>>> cache from the directory page mappings if the delegation is still valid?
>>> Does the directory delegation pin the mapping?  Clearly I need to look at
>>> the code..
>>
>> .. right - we don't keep the file attributes in the mappings today.  And,
>> more to the point - the directory delegation doesn't protect those file
>> attributes either.  We'd need NOTIFY4_CHANGE_CHILD_ATTRIBUTES implemented.
>>
>
> Which I think could be done, at least on the Linux server. fsnotify
> does support watching for child attribute changes (FS_EVENT_ON_CHILD |
> FS_ATTRIB).
>
> That could be very chatty though. We would need to do the work to make
> nfsd send callbacks >1 page in order to keep up, I imagine.

I agree - it could easily end up having the opposite effect of optimizing
performance by amplifying op counts for information the client might not
actually use.

Ben

Re: NFS delegations behavior analysis

[Trond Myklebust]

On Tue, 2026-06-23 at 09:32 -0400, Benjamin Coddington wrote:
> On 23 Jun 2026, at 9:11, Benjamin Coddington wrote:
> 
> > .... er - so with directory delegations, can we simply re-hydrate
> > the dentry
> > cache from the directory page mappings if the delegation is still
> > valid?
> > Does the directory delegation pin the mapping?  Clearly I need to
> > look at
> > the code..
> 
> .. right - we don't keep the file attributes in the mappings today. 
> And,
> more to the point - the directory delegation doesn't protect those
> file
> attributes either.  We'd need NOTIFY4_CHANGE_CHILD_ATTRIBUTES
> implemented.
> 
> Ben

Unlike delegations, notifications are asynchronous by design. Upon
reception of a notification, you may deduce that you should revalidate
the attributes on a file. However the absence of that notification is
insufficient to deduce that it is safe not to revalidate those
attributes.

So when it comes to informing the client whether to use READDIR or
READDIRPLUS, I'm sceptical concerning the value of throwing
NOTIFY4_CHANGE_CHILD_ATTRS at the problem.

-- 
Trond Myklebust
Linux NFS client maintainer, Hammerspace
trondmy@kernel.org, trond.myklebust@hammerspace.com

Re: NFS delegations behavior analysis

[Piyush Sachdeva]

"Anna Schumaker" <anna@nowheycreamery.com> writes:

> On Tue, Jun 23, 2026, at 7:04 AM, Mkrtchyan, Tigran wrote:
>> ----- Original Message -----
>>> From: "Jeff Layton" <jlayton@kernel.org>
>>> To: "Piyush Sachdeva" <s.piyush1024@gmail.com>, "linux-nfs" <linux-nfs@vger.kernel.org>, "Chuck Lever" <cel@kernel.org>,
>>> "trondmy" <trondmy@kernel.org>, sfrench@samba.org, sprasad@microsoft.com
>>> Cc: vaibsharma@microsoft.com
>>> Sent: Tuesday, 23 June, 2026 12:50:16
>>> Subject: Re: NFS delegations behavior analysis
>>
>>> On Tue, 2026-06-23 at 15:31 +0530, Piyush Sachdeva wrote:
>>>> Hi,
>>>> Lately I have been running micro benchmarks around the `ls` command and
>>>> reading through the code documentation of the NFS client to better
>>>> understand the client side caching behavior with and without
>>>> delegations.
>>>> 
>>>> Understanding so far:
>>>> Delegations (both file and directory) are granted by the server to the
>>>> client, indefinitely (until revoked or under the watermark) to cache
>>>> attributes. The caching of data is a result of the attribute
>>>> cache. Hence forth, a directory delegation will cache the directory
>>>> attributes and the names of the files in the directory, and a file
>>>> delegation will cache the attributes of the file and the file data.
>>>> 
>>>> Workload run:
>>>> I focused on the 2 workloads below, doing 2 passes of a large flat
>>>> directory (with close to 100K files) -
>>>> a cold pass, and warm pass using the cache from the cold pass:
>>>> - lslr - ls -lR on both runs
>>>> - lsmix - ls -R (cold) and then ls -lR (warm)
>>>> 
>>>> I also played with the rdirplus behavior using both the default
>>>> heuristic behavior and the `rdirplus=force` set at mount time.
>>>> 
>>>> Numbers:
>>>> actimeo=5s, rdirplus=force, ACLs off, flat_dir
>>>> ==================================================================
>>>> 
>>>>                  |         LSLR          |         LSMIX
>>>>                  |  (ls -lR cold / warm) |  (p1 ls -R / p2 ls -lR)
>>>> Operation        |  flat cold  | flat warm |   flat p1   | flat p2
>>>> -----------------+-------------+-----------+-------------+---------
>>>> READDIR calls    |    27       |     0     |   27        |    0
>>>> READDIR recv B   | 23,603,024  |     0     | 23,603,024  |    0
>>>>    call type     | readdirplus |    --     | readdirplus |    --
>>>> LOOKUP           |     1       |     0     |    1        |    0
>>>> GETATTR          |     3       |  100,000  |    2        | 100,001
>>>> ACCESS           |     2       |     0     |    2        |    0
>>>> -----------------+-------------+-----------+-------------+---------
>>>> Elapsed (age)    |  ~14 s      |  ~62 s    |   ~16 s     |  ~63 s
>>>> 
>>>> 
>>>> Observations:
>>>> When doing `ls` or `ls -l` on a directory, due to the open(2) on the
>>>> directory, the client gets a directory delegation - caching the
>>>> directory attributes and file names. However, as we don't have file
>>>> delegations due to no open(2) calls to any of the files. Henceforth,
>>>> the cache of file attributes is governed by `actimeo`.
>>>> Now here is the interesting bit, if the next `ls -l` is issued after
>>>> the `actimeo`, a massive GETATTR storm hits the server, doing stat()
>>>> calls for every file in the directory. As a result, the performance of
>>>> this warm `ls -l` run ends up being worse than the cold pass. I am
>>>> guessing this is most likely due to the compounded "rdirplus" being more
>>>> efficient than stat() calls.
>>>> 
>>>> 
>>>> Proposal:
>>>> For large directories, this ends up being a massive problem, taking 1-2
>>>> minutes when enumerating a directory on the warm passes.
>>>> - An easier way to tackle this could be to do a rdirplus=[auto | forced]
>>>>   instead of issuing the stat(2) storm to the server: When the client
>>>>   notices that there are cache misses, which would be the case of file
>>>>   attributes, instead of fetching file names from the directory-delegation
>>>>   cache and attributes from GETATTR, the client does a READDIRPLUS to
>>>>   the server, nonetheless.
>>>> - A more tedious would be the to cache file attributes as well, as a part
>>>>   of the directory delegation. This would end up requiring a change in the
>>>>   NFS protocol spec though.
>>>> - Bulk GETATTR calls: I am uncertain of the feasibility of this, but
>>>>   what if, the client could do 1 GETATTR call for getting attributes
>>>>   for multiple files.
>>> 
>>> 
>>> ls is such a hard workload to get right, because we don't really get an
>>
>> 100% agree. And there were a couple of attempts to address this issue
>> (second ls that is slow).
>>
>>> indication in the kernel of what userland's intentions are. It's
>>> basically a readdir() call followed by a bunch of stat()'s, but at the
>>> point where we're getting the readdir() call, we don't know if userland
>>> intends to stat() those files or not. We have to make a guess about
>>> that intention.
>>> 
>>> In this case, it sounds like the directory cache was valid, so the
>>> client decided it didn't need to do a READDIR at all, but the
>>> individual files had caches that timed out.
>>> 
>>> So imagine you're the kernel client and have been given that second
>>> readdir() call: Why should you decide to do a READDIRPLUS at that point
>>> instead of a regular READDIR?
>>
>> May we need some kind of client-side heuristics, like on the server side
>> for open-delegations, where after seeing some `stats` for files in the
>> In the same directory, the client will decide to switch to READDIR (v4)
>> to get all attributes in one go.
>
> We do something like that already. I don't think we'll ever have a readdir
> plus heuristic that makes everybody happy without userspace somehow telling
> us their intentions. I know a statx-based readdir plus system call probably
> sounds crazy, but something like that would go a long way to take all the
> guesswork out of things on our end.
>
> Anna
>
>>
>> Best regards,
>>    Tigran.
>>
>>> --
>>> Jeff Layton <jlayton@kernel.org>
>>
>> Attachments:
>> * smime.p7s

Hi,
I believe the problem from this discussion has been the cost associated
with GETATTRs during directory enumerations, compounded by the fact that
the kernel can't know whether more GETATTRs on the same directory are
coming.

Since the readdirplus heuristic rework in v5.18, plaing READDIRs
fetching the file type in v6.8, and introduction of rdirplus=force in
v6.14, the client has become quite performant especially for the
ls-style workloads. find and du, still suffer from GETATTR storms for
large-enough directories, as they enumerate the directory once then
stat() every child. Note that nfs_getattr() already feeds the
cache_hit/cache_miss counters (since v4.10), but that signal lives on
the open-directory context and is consumed only inside nfs_readdir().
This is where the performance hurts the most.

As a benchmark, I prototyped a solution for older kernels (v4.18, v5.14,
v6.4-based distros). An eBPF-based detector recognizes GETATTR/LOOKUP
storms on large flat directories and triggers a userspace helper to
enumerate the directory (READDIR(PLUS), start -> EOF). The returned
entries refresh the attribute/dcache so subsequent GETATTR/LOOKUP calls
are served from cache.  We measured directory-enumeration time and
GETATTR/LOOKUP counts drop by ~50–80x on large directories, for ls,
find, and du.

With that as evidence, I'd like to propose exploring two in-kernel
directions and get your opinion on feasibility and desirability:
1. Extend the existing getattr miss-tracking: when cache_misses on a
   directory cross a threshold, trigger a deferred, background
   READDIRPLUS to refresh that directory's child attributes. This gives
   the find / du case a consumer for the miss signal it already
   generates.
2. A batched-GETATTR path so tools like du /find can request many
   children's attributes in fewer round-trips. I recognize there's no
   batched-stat() interface today and that this spans userspace and the
   kernel, so I'm raising it mainly to ask whether a metadata-batching
   interface (or wiring stat readahead to an NFSv4 multi-GETATTR
   compound) is something the community would consider.

I'd appreciate your thoughts on both, particularly whether (1) is best
framed as an extension of the v5.18 heuristic.

-- 
Best,
Piyush Sachdeva

Re: NFS delegations behavior analysis

[Piyush Sachdeva]

Piyush Sachdeva <s.piyush1024@gmail.com> writes:

> "Anna Schumaker" <anna@nowheycreamery.com> writes:
>
>> On Tue, Jun 23, 2026, at 7:04 AM, Mkrtchyan, Tigran wrote:
>>> ----- Original Message -----
>>>> From: "Jeff Layton" <jlayton@kernel.org>
>>>> To: "Piyush Sachdeva" <s.piyush1024@gmail.com>, "linux-nfs" <linux-nfs@vger.kernel.org>, "Chuck Lever" <cel@kernel.org>,
>>>> "trondmy" <trondmy@kernel.org>, sfrench@samba.org, sprasad@microsoft.com
>>>> Cc: vaibsharma@microsoft.com
>>>> Sent: Tuesday, 23 June, 2026 12:50:16
>>>> Subject: Re: NFS delegations behavior analysis
>>>
>>>> On Tue, 2026-06-23 at 15:31 +0530, Piyush Sachdeva wrote:
>>>>> Hi,
>>>>> Lately I have been running micro benchmarks around the `ls` command and
>>>>> reading through the code documentation of the NFS client to better
>>>>> understand the client side caching behavior with and without
>>>>> delegations.
>>>>> 
>>>>> Understanding so far:
>>>>> Delegations (both file and directory) are granted by the server to the
>>>>> client, indefinitely (until revoked or under the watermark) to cache
>>>>> attributes. The caching of data is a result of the attribute
>>>>> cache. Hence forth, a directory delegation will cache the directory
>>>>> attributes and the names of the files in the directory, and a file
>>>>> delegation will cache the attributes of the file and the file data.
>>>>> 
>>>>> Workload run:
>>>>> I focused on the 2 workloads below, doing 2 passes of a large flat
>>>>> directory (with close to 100K files) -
>>>>> a cold pass, and warm pass using the cache from the cold pass:
>>>>> - lslr - ls -lR on both runs
>>>>> - lsmix - ls -R (cold) and then ls -lR (warm)
>>>>> 
>>>>> I also played with the rdirplus behavior using both the default
>>>>> heuristic behavior and the `rdirplus=force` set at mount time.
>>>>> 
>>>>> Numbers:
>>>>> actimeo=5s, rdirplus=force, ACLs off, flat_dir
>>>>> ==================================================================
>>>>> 
>>>>>                  |         LSLR          |         LSMIX
>>>>>                  |  (ls -lR cold / warm) |  (p1 ls -R / p2 ls -lR)
>>>>> Operation        |  flat cold  | flat warm |   flat p1   | flat p2
>>>>> -----------------+-------------+-----------+-------------+---------
>>>>> READDIR calls    |    27       |     0     |   27        |    0
>>>>> READDIR recv B   | 23,603,024  |     0     | 23,603,024  |    0
>>>>>    call type     | readdirplus |    --     | readdirplus |    --
>>>>> LOOKUP           |     1       |     0     |    1        |    0
>>>>> GETATTR          |     3       |  100,000  |    2        | 100,001
>>>>> ACCESS           |     2       |     0     |    2        |    0
>>>>> -----------------+-------------+-----------+-------------+---------
>>>>> Elapsed (age)    |  ~14 s      |  ~62 s    |   ~16 s     |  ~63 s
>>>>> 
>>>>> 
>>>>> Observations:
>>>>> When doing `ls` or `ls -l` on a directory, due to the open(2) on the
>>>>> directory, the client gets a directory delegation - caching the
>>>>> directory attributes and file names. However, as we don't have file
>>>>> delegations due to no open(2) calls to any of the files. Henceforth,
>>>>> the cache of file attributes is governed by `actimeo`.
>>>>> Now here is the interesting bit, if the next `ls -l` is issued after
>>>>> the `actimeo`, a massive GETATTR storm hits the server, doing stat()
>>>>> calls for every file in the directory. As a result, the performance of
>>>>> this warm `ls -l` run ends up being worse than the cold pass. I am
>>>>> guessing this is most likely due to the compounded "rdirplus" being more
>>>>> efficient than stat() calls.
>>>>> 
>>>>> 
>>>>> Proposal:
>>>>> For large directories, this ends up being a massive problem, taking 1-2
>>>>> minutes when enumerating a directory on the warm passes.
>>>>> - An easier way to tackle this could be to do a rdirplus=[auto | forced]
>>>>>   instead of issuing the stat(2) storm to the server: When the client
>>>>>   notices that there are cache misses, which would be the case of file
>>>>>   attributes, instead of fetching file names from the directory-delegation
>>>>>   cache and attributes from GETATTR, the client does a READDIRPLUS to
>>>>>   the server, nonetheless.
>>>>> - A more tedious would be the to cache file attributes as well, as a part
>>>>>   of the directory delegation. This would end up requiring a change in the
>>>>>   NFS protocol spec though.
>>>>> - Bulk GETATTR calls: I am uncertain of the feasibility of this, but
>>>>>   what if, the client could do 1 GETATTR call for getting attributes
>>>>>   for multiple files.
>>>> 
>>>> 
>>>> ls is such a hard workload to get right, because we don't really get an
>>>
>>> 100% agree. And there were a couple of attempts to address this issue
>>> (second ls that is slow).
>>>
>>>> indication in the kernel of what userland's intentions are. It's
>>>> basically a readdir() call followed by a bunch of stat()'s, but at the
>>>> point where we're getting the readdir() call, we don't know if userland
>>>> intends to stat() those files or not. We have to make a guess about
>>>> that intention.
>>>> 
>>>> In this case, it sounds like the directory cache was valid, so the
>>>> client decided it didn't need to do a READDIR at all, but the
>>>> individual files had caches that timed out.
>>>> 
>>>> So imagine you're the kernel client and have been given that second
>>>> readdir() call: Why should you decide to do a READDIRPLUS at that point
>>>> instead of a regular READDIR?
>>>
>>> May we need some kind of client-side heuristics, like on the server side
>>> for open-delegations, where after seeing some `stats` for files in the
>>> In the same directory, the client will decide to switch to READDIR (v4)
>>> to get all attributes in one go.
>>
>> We do something like that already. I don't think we'll ever have a readdir
>> plus heuristic that makes everybody happy without userspace somehow telling
>> us their intentions. I know a statx-based readdir plus system call probably
>> sounds crazy, but something like that would go a long way to take all the
>> guesswork out of things on our end.
>>
>> Anna
>>
>>>
>>> Best regards,
>>>    Tigran.
>>>
>>>> --
>>>> Jeff Layton <jlayton@kernel.org>
>>>
>>> Attachments:
>>> * smime.p7s
>
> Hi,
> I believe the problem from this discussion has been the cost associated
> with GETATTRs during directory enumerations, compounded by the fact that
> the kernel can't know whether more GETATTRs on the same directory are
> coming.
>
> Since the readdirplus heuristic rework in v5.18, plaing READDIRs
> fetching the file type in v6.8, and introduction of rdirplus=force in
> v6.14, the client has become quite performant especially for the
> ls-style workloads. find and du, still suffer from GETATTR storms for
> large-enough directories, as they enumerate the directory once then
> stat() every child. Note that nfs_getattr() already feeds the
> cache_hit/cache_miss counters (since v4.10), but that signal lives on
> the open-directory context and is consumed only inside nfs_readdir().
> This is where the performance hurts the most.
>
> As a benchmark, I prototyped a solution for older kernels (v4.18, v5.14,
> v6.4-based distros). An eBPF-based detector recognizes GETATTR/LOOKUP
> storms on large flat directories and triggers a userspace helper to
> enumerate the directory (READDIR(PLUS), start -> EOF). The returned
> entries refresh the attribute/dcache so subsequent GETATTR/LOOKUP calls
> are served from cache.  We measured directory-enumeration time and
> GETATTR/LOOKUP counts drop by ~50–80x on large directories, for ls,
> find, and du.
>
> With that as evidence, I'd like to propose exploring two in-kernel
> directions and get your opinion on feasibility and desirability:
> 1. Extend the existing getattr miss-tracking: when cache_misses on a
>    directory cross a threshold, trigger a deferred, background
>    READDIRPLUS to refresh that directory's child attributes. This gives
>    the find / du case a consumer for the miss signal it already
>    generates.
> 2. A batched-GETATTR path so tools like du /find can request many
>    children's attributes in fewer round-trips. I recognize there's no
>    batched-stat() interface today and that this spans userspace and the
>    kernel, so I'm raising it mainly to ask whether a metadata-batching
>    interface (or wiring stat readahead to an NFSv4 multi-GETATTR
>    compound) is something the community would consider.
>
> I'd appreciate your thoughts on both, particularly whether (1) is best
> framed as an extension of the v5.18 heuristic.
>
> -- 
> Best,
> Piyush Sachdeva

I am posting the benchmark numbers of the directory enumeration workload
both with and without the eBPF-based tool for reference.

Different workloads:
- ls_r: 2 iterations of - ls -R "$TARGET_DIR"
- ls_lr: 2 iterations of - ls -lR "$TARGET_DIR"
- lsmix: ls -R "$TARGET_DIR" followed by ls -lR "$TARGET_DIR"
- find: 2 iterations of - find "$TARGET_DIR" -type f -size +0c
- du: 2 iterations of - du "$TARGET_DIR"

- The share gets remounted in between each workload runs.
- Each workload has 2 successive iterations (no gap) - a cold run that
  populates the cache, and after, a warm run that re-uses the cache.
- Distro(kernel): SLES 15 SP5 (v5.14)
- $TARGET_DIR - Flat large directory; file count - 1 Million files
- actimeo=30

Numbers:
- Iteration 1 (cold) - without eBPF / with eBPF

op               ls_r                  ls_lr                 lsmix                 du                      find
---------------  --------------------  --------------------  --------------------  ----------------------  ----------------------
READDIR calls    217 / 455             263 / 263             217 / 454             127 / 422               127 / 421
READDIR recv(B)  210,687,496 /         248,029,456 /         210,687,496 /         115,395,048 /           115,395,048 /
                 440,573,116           248,029,456           439,524,444           405,689,200             404,640,792
LOOKUP           30,178 / 6,161        1 / 1                 30,178 / 6,691        733,573 / 19,465        733,573 / 5,837
GETATTR          61 / 5                1 / 1                 63 / 5                177,623 / 5             177,622 / 4
ACCESS           2 / 2                 2 / 2                 2 / 2                 2 / 2                   2 / 2
elapsed (ms)     919,684 / 40,657      23,938 / 24,984       947,506 / 41,133      2,643,906 / 78,684      2,627,536 / 38,337


- Iteration 2 (warm) - without eBPF / with eBPF

op               ls_r                  ls_lr                 lsmix                 du                      find
---------------  --------------------  --------------------  --------------------  ----------------------  ----------------------
READDIR calls    217 / 0               0 / 0                 258 / 256             124 / 317               123 / 417
READDIR recv(B)  208,601,368 / 0       0 / 0                 248,909,816 /         109,134,232 /           109,134,232 /
                                                             248,909,816           398,379,976             398,379,976
LOOKUP           0 / 0                 0 / 0                 0 / 0                 0 / 0                   0 / 0
GETATTR          301,214 / 1           1 / 1                 127 / 127             822,266 / 26,329        822,267 / 25,611
ACCESS           0 / 0                 0 / 0                 0 / 0                 0 / 0                   0 / 0
elapsed (ms)     550,011 / 2,648       5,231 / 6,419         23,264 / 24,090       1,494,821 / 65,103      1,527,773 / 64,990


You can see the reduction in total time, GETATTRs and LOOKUPs. 

-- 
Best,
Piyush Sachdeva <s.piyush1024@gmail.com>

Re: NFS delegations behavior analysis

[Jeff Layton]

On Wed, 2026-09-09 at 15:40 +0530, Piyush Sachdeva wrote:
> "Anna Schumaker" <anna@nowheycreamery.com> writes:
> 
> > On Tue, Jun 23, 2026, at 7:04 AM, Mkrtchyan, Tigran wrote:
> > > ----- Original Message -----
> > > > From: "Jeff Layton" <jlayton@kernel.org>
> > > > To: "Piyush Sachdeva" <s.piyush1024@gmail.com>, "linux-nfs" <linux-nfs@vger.kernel.org>, "Chuck Lever" <cel@kernel.org>,
> > > > "trondmy" <trondmy@kernel.org>, sfrench@samba.org, sprasad@microsoft.com
> > > > Cc: vaibsharma@microsoft.com
> > > > Sent: Tuesday, 23 June, 2026 12:50:16
> > > > Subject: Re: NFS delegations behavior analysis
> > > 
> > > > On Tue, 2026-06-23 at 15:31 +0530, Piyush Sachdeva wrote:
> > > > > Hi,
> > > > > Lately I have been running micro benchmarks around the `ls` command and
> > > > > reading through the code documentation of the NFS client to better
> > > > > understand the client side caching behavior with and without
> > > > > delegations.
> > > > > 
> > > > > Understanding so far:
> > > > > Delegations (both file and directory) are granted by the server to the
> > > > > client, indefinitely (until revoked or under the watermark) to cache
> > > > > attributes. The caching of data is a result of the attribute
> > > > > cache. Hence forth, a directory delegation will cache the directory
> > > > > attributes and the names of the files in the directory, and a file
> > > > > delegation will cache the attributes of the file and the file data.
> > > > > 
> > > > > Workload run:
> > > > > I focused on the 2 workloads below, doing 2 passes of a large flat
> > > > > directory (with close to 100K files) -
> > > > > a cold pass, and warm pass using the cache from the cold pass:
> > > > > - lslr - ls -lR on both runs
> > > > > - lsmix - ls -R (cold) and then ls -lR (warm)
> > > > > 
> > > > > I also played with the rdirplus behavior using both the default
> > > > > heuristic behavior and the `rdirplus=force` set at mount time.
> > > > > 
> > > > > Numbers:
> > > > > actimeo=5s, rdirplus=force, ACLs off, flat_dir
> > > > > ==================================================================
> > > > > 
> > > > >                  |         LSLR          |         LSMIX
> > > > >                  |  (ls -lR cold / warm) |  (p1 ls -R / p2 ls -lR)
> > > > > Operation        |  flat cold  | flat warm |   flat p1   | flat p2
> > > > > -----------------+-------------+-----------+-------------+---------
> > > > > READDIR calls    |    27       |     0     |   27        |    0
> > > > > READDIR recv B   | 23,603,024  |     0     | 23,603,024  |    0
> > > > >    call type     | readdirplus |    --     | readdirplus |    --
> > > > > LOOKUP           |     1       |     0     |    1        |    0
> > > > > GETATTR          |     3       |  100,000  |    2        | 100,001
> > > > > ACCESS           |     2       |     0     |    2        |    0
> > > > > -----------------+-------------+-----------+-------------+---------
> > > > > Elapsed (age)    |  ~14 s      |  ~62 s    |   ~16 s     |  ~63 s
> > > > > 
> > > > > 
> > > > > Observations:
> > > > > When doing `ls` or `ls -l` on a directory, due to the open(2) on the
> > > > > directory, the client gets a directory delegation - caching the
> > > > > directory attributes and file names. However, as we don't have file
> > > > > delegations due to no open(2) calls to any of the files. Henceforth,
> > > > > the cache of file attributes is governed by `actimeo`.
> > > > > Now here is the interesting bit, if the next `ls -l` is issued after
> > > > > the `actimeo`, a massive GETATTR storm hits the server, doing stat()
> > > > > calls for every file in the directory. As a result, the performance of
> > > > > this warm `ls -l` run ends up being worse than the cold pass. I am
> > > > > guessing this is most likely due to the compounded "rdirplus" being more
> > > > > efficient than stat() calls.
> > > > > 
> > > > > 
> > > > > Proposal:
> > > > > For large directories, this ends up being a massive problem, taking 1-2
> > > > > minutes when enumerating a directory on the warm passes.
> > > > > - An easier way to tackle this could be to do a rdirplus=[auto | forced]
> > > > >   instead of issuing the stat(2) storm to the server: When the client
> > > > >   notices that there are cache misses, which would be the case of file
> > > > >   attributes, instead of fetching file names from the directory-delegation
> > > > >   cache and attributes from GETATTR, the client does a READDIRPLUS to
> > > > >   the server, nonetheless.
> > > > > - A more tedious would be the to cache file attributes as well, as a part
> > > > >   of the directory delegation. This would end up requiring a change in the
> > > > >   NFS protocol spec though.
> > > > > - Bulk GETATTR calls: I am uncertain of the feasibility of this, but
> > > > >   what if, the client could do 1 GETATTR call for getting attributes
> > > > >   for multiple files.
> > > > 
> > > > 
> > > > ls is such a hard workload to get right, because we don't really get an
> > > 
> > > 100% agree. And there were a couple of attempts to address this issue
> > > (second ls that is slow).
> > > 
> > > > indication in the kernel of what userland's intentions are. It's
> > > > basically a readdir() call followed by a bunch of stat()'s, but at the
> > > > point where we're getting the readdir() call, we don't know if userland
> > > > intends to stat() those files or not. We have to make a guess about
> > > > that intention.
> > > > 
> > > > In this case, it sounds like the directory cache was valid, so the
> > > > client decided it didn't need to do a READDIR at all, but the
> > > > individual files had caches that timed out.
> > > > 
> > > > So imagine you're the kernel client and have been given that second
> > > > readdir() call: Why should you decide to do a READDIRPLUS at that point
> > > > instead of a regular READDIR?
> > > 
> > > May we need some kind of client-side heuristics, like on the server side
> > > for open-delegations, where after seeing some `stats` for files in the
> > > In the same directory, the client will decide to switch to READDIR (v4)
> > > to get all attributes in one go.
> > 
> > We do something like that already. I don't think we'll ever have a readdir
> > plus heuristic that makes everybody happy without userspace somehow telling
> > us their intentions. I know a statx-based readdir plus system call probably
> > sounds crazy, but something like that would go a long way to take all the
> > guesswork out of things on our end.
> > 
> > Anna
> > 
> > > 
> > > Best regards,
> > >    Tigran.
> > > 
> > > > --
> > > > Jeff Layton <jlayton@kernel.org>
> > > 
> > > Attachments:
> > > * smime.p7s
> 
> Hi,
> I believe the problem from this discussion has been the cost associated
> with GETATTRs during directory enumerations, compounded by the fact that
> the kernel can't know whether more GETATTRs on the same directory are
> coming.
> 
> Since the readdirplus heuristic rework in v5.18, plaing READDIRs
> fetching the file type in v6.8, and introduction of rdirplus=force in
> v6.14, the client has become quite performant especially for the
> ls-style workloads. find and du, still suffer from GETATTR storms for
> large-enough directories, as they enumerate the directory once then
> stat() every child. Note that nfs_getattr() already feeds the
> cache_hit/cache_miss counters (since v4.10), but that signal lives on
> the open-directory context and is consumed only inside nfs_readdir().
> This is where the performance hurts the most.
> 
> As a benchmark, I prototyped a solution for older kernels (v4.18, v5.14,
> v6.4-based distros). An eBPF-based detector recognizes GETATTR/LOOKUP
> storms on large flat directories and triggers a userspace helper to
> enumerate the directory (READDIR(PLUS), start -> EOF). The returned
> entries refresh the attribute/dcache so subsequent GETATTR/LOOKUP calls
> are served from cache.  We measured directory-enumeration time and
> GETATTR/LOOKUP counts drop by ~50–80x on large directories, for ls,
> find, and du.
> 
> With that as evidence, I'd like to propose exploring two in-kernel
> directions and get your opinion on feasibility and desirability:
> 1. Extend the existing getattr miss-tracking: when cache_misses on a
>    directory cross a threshold, trigger a deferred, background
>    READDIRPLUS to refresh that directory's child attributes. This gives
>    the find / du case a consumer for the miss signal it already
>    generates.

Tough call. This adds load to the server for attributes that will never
be used. My gut feeling is that that might help some workloads, but
hurt others.

> 2. A batched-GETATTR path so tools like du /find can request many
>    children's attributes in fewer round-trips. I recognize there's no
>    batched-stat() interface today and that this spans userspace and the
>    kernel, so I'm raising it mainly to ask whether a metadata-batching
>    interface (or wiring stat readahead to an NFSv4 multi-GETATTR
>    compound) is something the community would consider.
> 
> I'd appreciate your thoughts on both, particularly whether (1) is best
> framed as an extension of the v5.18 heuristic.


#2 sounds like a mess to get right....

What might be more reasonable is adding a real readdirplus() style-
system call, like Anna mentioned. Something where userland can do a
readdir() and pass down a statx() mask to tell the kernel what
attributes it's interested in, and get back a list of filenames and
their inodes' attributes in one shot.

That too might not be accepted upstream (it has been attempted before),
but that seems like it would be a fairly straightforward thing for
userland, and would work well for "ls -l" and the like.
-- 
Jeff Layton <jlayton@kernel.org>