File create performance between NFSv3 and NFSv4.

File create performance between NFSv3 and NFSv4.

[Pradeep]

Hello,

While testing a performance issue, I noticed that the number of
creates that an NFS client can do over NFSv4 is significantly lower
compared to NFSv3. I'm using the test below:

https://github.com/distributed-system-analysis/smallfile

Command run:
smallfile_cli.py  --operation create --threads 128 --file-size 1
--files 1024 --top <nfs-dir>
[ 128 threads, each doing 1024 file creates of 1K size]

This gives around 1169 creates/sec with NFSv4.1 and 8073 creates/sec
with NFSv3. This is with the exact same client and server. NFS server
is tuned to use 200 threads.

When I looked at tcpdump, I noticed that over NFSv3 multiple parallel
requests are being sent and NFSv4 is pretty much serial. Is there
anything that I can tune to improve NFSv4 performance w.r.t to file
creations?

This also shows up in benchmarks like SpecFS SWBUILD's INIT phase
where millions of files are getting created.

Thanks,
Pradeep

Re: File create performance between NFSv3 and NFSv4.

[Mkrtchyan, Tigran]

Hi Pradeep,

When you talk performance always specify which version
client and server you use, as older kernels might have
issues and newer kernels might hare regressions.

There are couple of reasons why you can observe it. First of all,
NFS v3 is state less, thus less operations sent over the wire.
The NFS v4.1 is more efficient than v4.0.

Running your test as: ./smallfile_cli.py  --operation create --threads 24 --file-size 0 --files 1024 --top /mnt/test

I get:

v4.0: 2530
v4.1: 13191 
v3  : 14728

This is with 5.6.16-300 client, but a non linux server.


Second, if you use NFSv4.1 you should check that client have
enough session slots configured to process requests in parallel.
You can check/set the value on the client as:

cat /sys/module/nfs/parameters/max_session_slots
echo 128 > /sys/module/nfs/parameters/max_session_slots

IFAIK, server has hard coded max value of 160.


Regards,
   Tigran.

----- Original Message -----
> From: "Pradeep" <pradeep.thomas@gmail.com>
> To: "linux-nfs" <linux-nfs@vger.kernel.org>
> Sent: Wednesday, July 1, 2020 2:55:15 AM
> Subject: File create performance between NFSv3 and NFSv4.

> Hello,
> 
> While testing a performance issue, I noticed that the number of
> creates that an NFS client can do over NFSv4 is significantly lower
> compared to NFSv3. I'm using the test below:
> 
> https://github.com/distributed-system-analysis/smallfile
> 
> Command run:
> smallfile_cli.py  --operation create --threads 128 --file-size 1
> --files 1024 --top <nfs-dir>
> [ 128 threads, each doing 1024 file creates of 1K size]
> 
> This gives around 1169 creates/sec with NFSv4.1 and 8073 creates/sec
> with NFSv3. This is with the exact same client and server. NFS server
> is tuned to use 200 threads.
> 
> When I looked at tcpdump, I noticed that over NFSv3 multiple parallel
> requests are being sent and NFSv4 is pretty much serial. Is there
> anything that I can tune to improve NFSv4 performance w.r.t to file
> creations?
> 
> This also shows up in benchmarks like SpecFS SWBUILD's INIT phase
> where millions of files are getting created.
> 
> Thanks,
> Pradeep