NFS/UDP/IP performance - 2.4.19 v/s 2.4.20, 2.4.20-pre3

NFS/UDP/IP performance - 2.4.19 v/s 2.4.20, 2.4.20-pre3

[Arun Dharankar]

Greetings.

There seem to be a remarkable performance difference
between 2.4.19 and 2.4.20/2.4.21-pre3 in regards to
NFS writes/reads. I am not sure, but the problem may not
in NFS but somewhere lower (UDP/IP or core).



For example, in my kernel and network configuration a 
write to a new file over NFS on 2.4.19 for 5MB takes 2.5
seconds or so. With everything same (including kernel
configuration) 2.4.20 and 2.4.21-pre3 the same takes
11 or more seconds.

Also, when this file write is in progress, the system
time goes up to 15% on 2.4.19, whereas on 2.4.20/21-pre3,
it is about 4%. (I use sar/sysstat for this).


Memory accesses dont seem to be the issue either. Test
program to check this show same times and are ok (as I
expect on the board I use).


"netstat -s" or ifconfig or tcpdump traces dont seem to
point to dropped messages, collisions, retransmissions
etc.


The hardware configuration is PowerPC based, and there
are no changes in the board specific IO subsystem between 
2.4.19 and 2.4.20/21-pre3. The same compiler is used for 
building both the kernels, and have tried this even with 
GCC 3.2, with same results. 

So, I dont suspect this is either board or compiler
related issue.


Also, I see some differences in handling of the bottom
halves in net/core/dev.c between 2.4.19 and 2.4.20/21-pre3.
Although, I have not gone through these in details to
assert that this is indeed the problem area.



Questions:

  - Has anyone seen this? Perhaps on other platforms (x86 etc)?
    Is there some tunable that has been added (or is different)
    after 2.4.19, and which needs to be tuned?


  - I have tried to enable kernel profiling to find any
    potential problem code areas. But given the low cpu
    utilization during these copies, I am not sure if this
    can give any useful info.

    Could anyone offer any ideas to debug this?



I would appreciate if you copy me on any responses to this post, I
dont subscribe to this list.

Best regards,
-Arun.
ADharankar@ATTBI.Com

Re: NFS/UDP/IP performance - 2.4.19 v/s 2.4.20, 2.4.20-pre3

[Arun Dharankar]

Hello...

I had posted question yesterday. That post can be seen at
    http://www.lkml.org/archive/2003/1/25/4/index.html

After doing some more testing with standalone UDP program, 
the transfer rates are normal and as expected on 2.4.20.
The same performance can be seen on 2.4.19.


There is a similar posting at:
    http://www.uwsg.iu.edu/hypermail/linux/kernel/0210.0/1116.html

--- Re: linux-2.4.20-pre8-ac3: NFS performance regression
--- From: Alan Cox (alan@lxorguk.ukuu.org.uk)
--- Date: Thu Oct 03 2002 - 17:13:04 EST 
--- In reply to: Andreas Pfaller: "linux-2.4.20-pre8-ac3: 
---    NFS performance regression"
--- On Thu, 2002-10-03 at 19:32, Andreas Pfaller wrote: 
--- > However I noticed a significant NFS performance drop with 
--- > 2.4.20-pre8-ac3. Other network throughput is not affected. 
---
--- I see this with all recent 2.4.20pre and 2.4.20pre-ac 
--- kernels. I've not had time to retest with Trond's fixes to 
--- recheck it all


So, the problem may be with NFS. NFS is not an issue for me 
and I can do with this problem. Hope this helps whoever is
testing or working on related areas.


Best regards,
-Arun.