Re: NFS delegations behavior analysis

["Mkrtchyan, Tigran" <tigran.mkrtchyan@desy.de>]

[-- Attachment #1: Type: text/plain, Size: 5344 bytes --]



----- Original Message -----
> From: "Jeff Layton" <jlayton@kernel.org>
> To: "Piyush Sachdeva" <s.piyush1024@gmail.com>, "linux-nfs" <linux-nfs@vger.kernel.org>, "Chuck Lever" <cel@kernel.org>,
> "trondmy" <trondmy@kernel.org>, sfrench@samba.org, sprasad@microsoft.com
> Cc: vaibsharma@microsoft.com
> Sent: Tuesday, 23 June, 2026 12:50:16
> Subject: Re: NFS delegations behavior analysis

> On Tue, 2026-06-23 at 15:31 +0530, Piyush Sachdeva wrote:
>> Hi,
>> Lately I have been running micro benchmarks around the `ls` command and
>> reading through the code documentation of the NFS client to better
>> understand the client side caching behavior with and without
>> delegations.
>> 
>> Understanding so far:
>> Delegations (both file and directory) are granted by the server to the
>> client, indefinitely (until revoked or under the watermark) to cache
>> attributes. The caching of data is a result of the attribute
>> cache. Hence forth, a directory delegation will cache the directory
>> attributes and the names of the files in the directory, and a file
>> delegation will cache the attributes of the file and the file data.
>> 
>> Workload run:
>> I focused on the 2 workloads below, doing 2 passes of a large flat
>> directory (with close to 100K files) -
>> a cold pass, and warm pass using the cache from the cold pass:
>> - lslr - ls -lR on both runs
>> - lsmix - ls -R (cold) and then ls -lR (warm)
>> 
>> I also played with the rdirplus behavior using both the default
>> heuristic behavior and the `rdirplus=force` set at mount time.
>> 
>> Numbers:
>> actimeo=5s, rdirplus=force, ACLs off, flat_dir
>> ==================================================================
>> 
>>                  |         LSLR          |         LSMIX
>>                  |  (ls -lR cold / warm) |  (p1 ls -R / p2 ls -lR)
>> Operation        |  flat cold  | flat warm |   flat p1   | flat p2
>> -----------------+-------------+-----------+-------------+---------
>> READDIR calls    |    27       |     0     |   27        |    0
>> READDIR recv B   | 23,603,024  |     0     | 23,603,024  |    0
>>    call type     | readdirplus |    --     | readdirplus |    --
>> LOOKUP           |     1       |     0     |    1        |    0
>> GETATTR          |     3       |  100,000  |    2        | 100,001
>> ACCESS           |     2       |     0     |    2        |    0
>> -----------------+-------------+-----------+-------------+---------
>> Elapsed (age)    |  ~14 s      |  ~62 s    |   ~16 s     |  ~63 s
>> 
>> 
>> Observations:
>> When doing `ls` or `ls -l` on a directory, due to the open(2) on the
>> directory, the client gets a directory delegation - caching the
>> directory attributes and file names. However, as we don't have file
>> delegations due to no open(2) calls to any of the files. Henceforth,
>> the cache of file attributes is governed by `actimeo`.
>> Now here is the interesting bit, if the next `ls -l` is issued after
>> the `actimeo`, a massive GETATTR storm hits the server, doing stat()
>> calls for every file in the directory. As a result, the performance of
>> this warm `ls -l` run ends up being worse than the cold pass. I am
>> guessing this is most likely due to the compounded "rdirplus" being more
>> efficient than stat() calls.
>> 
>> 
>> Proposal:
>> For large directories, this ends up being a massive problem, taking 1-2
>> minutes when enumerating a directory on the warm passes.
>> - An easier way to tackle this could be to do a rdirplus=[auto | forced]
>>   instead of issuing the stat(2) storm to the server: When the client
>>   notices that there are cache misses, which would be the case of file
>>   attributes, instead of fetching file names from the directory-delegation
>>   cache and attributes from GETATTR, the client does a READDIRPLUS to
>>   the server, nonetheless.
>> - A more tedious would be the to cache file attributes as well, as a part
>>   of the directory delegation. This would end up requiring a change in the
>>   NFS protocol spec though.
>> - Bulk GETATTR calls: I am uncertain of the feasibility of this, but
>>   what if, the client could do 1 GETATTR call for getting attributes
>>   for multiple files.
> 
> 
> ls is such a hard workload to get right, because we don't really get an

100% agree. And there were a couple of attempts to address this issue
(second ls that is slow).

> indication in the kernel of what userland's intentions are. It's
> basically a readdir() call followed by a bunch of stat()'s, but at the
> point where we're getting the readdir() call, we don't know if userland
> intends to stat() those files or not. We have to make a guess about
> that intention.
> 
> In this case, it sounds like the directory cache was valid, so the
> client decided it didn't need to do a READDIR at all, but the
> individual files had caches that timed out.
> 
> So imagine you're the kernel client and have been given that second
> readdir() call: Why should you decide to do a READDIRPLUS at that point
> instead of a regular READDIR?

May we need some kind of client-side heuristics, like on the server side
for open-delegations, where after seeing some `stats` for files in the
In the same directory, the client will decide to switch to READDIR (v4)
to get all attributes in one go.

Best regards,
   Tigran.

> --
> Jeff Layton <jlayton@kernel.org>

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 2309 bytes --]