Unexpected low pNFS IO performance with parallel workload

["Mkrtchyan, Tigran" <tigran.mkrtchyan@desy.de>]

[-- Attachment #1: Type: text/plain, Size: 1767 bytes --]


Dear NFS fellows,

During HPC workloads on we notice that Linux NFS4.2/pNFS client menonstraits unexpected low performance.
The application opens 55 files parallel reads the data with multiple threads. The server issues flexfile
layout with tighly coupled NFSv4.1 DSes.

Oservations:

- despite 1MB rsize/wsize returned by layout, client never issues reads bigger that 512k (offten much smaller)
- client always uses slot 0 on DS, and
- reads happen sequentialy, i.e. only one in-flight READ requests
- following reads often just read the next 512k block
- If instead of parallel application a simple dd is called, that multiple slots and 1MB READs are sent

$ dd if=/pnfs/xxxx/00054.h5 of=/dev/null
45753381+1 records in
45753381+1 records out
23425731171 bytes (23 GB, 22 GiB) copied, 69.702 s, 336 MB/s


The client has 80 cores on 2 sockets, 512BG of RAM and runs REHL 9.4

$ uname -r
5.14.0-427.26.1.el9_4.x86_64

$ free -g
               total        used        free      shared  buff/cache   available
Mem:             503          84         392           0          29         419

$ lscpu | head
Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Address sizes:                      46 bits physical, 48 bits virtual
Byte Order:                         Little Endian
CPU(s):                             80
On-line CPU(s) list:                0-79
Vendor ID:                          GenuineIntel
BIOS Vendor ID:                     Intel(R) Corporation
Model name:                         Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz
BIOS Model name:                    Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz

The client and all DSes equiped  with 10GB/s NICs.

Any ideas where to look?

Best regards,
   Tigran.

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 2826 bytes --]