NFS client low performance in concurrent environment.

["Mkrtchyan, Tigran" <tigran.mkrtchyan@desy.de>]

[-- Attachment #1: Type: text/plain, Size: 3865 bytes --]


Dear NFS fellows,

As part of research, we have adopted a well-known in the HPC community, IOR[1],
to support libnfs[2]. After running a bunch of tests, our observation is that the
multiple clients in userspace have a higher throughput than the in-kernel
client (or server).

In the test below, nfs server runs on RHEL9 with kernel 5.14.0-503.23.1.el9_5.x86_64
exporting /mnt. The results are in operations per second, thus, higher numbers are better.

The client is an 80-core single host, running RHEL9 with kernel 5.14.0-427.26.1.el9_4.x86_64.
We used NFSv3 in the test to eliminate NFSv4's open/close overhead on zero-byte files.


TEST 1: libnfs
```
$ mpirun -n 128 --map-by :OVERSUBSCRIBE  ./mdtest  -a LIBNFS --libnfs.url='nfs://lab008/mnt/?uid=0&gid=0&version=3' -w 0 -I 128 -i 10 -z 0 -b 0 -F -d /test
-- started at 04/03/2025 14:39:30 --

mdtest-4.1.0+dev was launched with 128 total task(s) on 1 node(s)
Command line used: ./mdtest '-a' 'LIBNFS' '--libnfs.url=nfs://lab008/mnt/version=3' '-w' '0' '-I' '128' '-i' '10' '-z' '0' '-b' '0' '-F' '-d' '/test'
Nodemap: 11111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111
Path                : /test
FS                  : 38.2 GiB   Used FS: 41.3%   Inodes: 2.4 Mi   Used Inodes: 5.8%
128 tasks, 16384 files

SUMMARY rate (in ops/sec): (of 10 iterations)
   Operation                     Max            Min           Mean        Std Dev
   ---------                     ---            ---           ----        -------
   File creation                7147.432       6789.531       6996.044        132.149
   File stat                   97175.603      57844.142      91063.340      12000.718
   File read                   97004.685      48234.620      89099.077      14715.699
   File removal                25172.919      23405.880      24424.384        577.264
   Tree creation                2375.031        555.537       1982.139        561.013
   Tree removal                   99.443         95.475         97.632          1.266
-- finished at 04/03/2025 14:40:05 --
```


TEST 2: in-kernel client
```
$ mpirun -n 128 --map-by :OVERSUBSCRIBE  ./mdtest  -w 0 -I 128 -i 10 -z 0 -b 0 -F -d /mnt/test
-- started at 04/03/2025 14:36:09 --

mdtest-4.1.0+dev was launched with 128 total task(s) on 1 node(s)
Nodemap: 11111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111
Path                : /mnt/test
FS                  : 38.2 GiB   Used FS: 41.3%   Inodes: 2.4 Mi   Used Inodes: 5.8%
128 tasks, 16384 files

SUMMARY rate (in ops/sec): (of 10 iterations)
   Operation                     Max            Min           Mean        Std Dev
   ---------                     ---            ---           ----        -------
   File creation                2301.914       2046.406       2203.859         88.793
   File stat                  101396.240      77386.014      91270.677       6229.657
   File read                   43631.081      36858.229      40800.066       2534.255
   File removal                 3102.328       2647.649       2840.170        153.959
   Tree creation                2142.137        253.739       1710.416        620.293
   Tree removal                   42.922         25.670         36.604          4.820
-- finished at 04/03/2025 14:38:28 --
```


Obviously, the kernel client shares the TCP connection. So, either (a) this is an expected behavior;
(b) client thread starvation; and (c) server thread starvation. The last option is unlikely, as we
first observed the behavior with the dCache NFS server implementation before falling back to
the linux kernel nfsd.

Best regards,
   Tigran.


[1]: https://github.com/hpc/ior
[2]: https://github.com/sahlberg/libnfs

-----------------------------
DESY-IT, Scientific Computing

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 2826 bytes --]